Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 447192 - app-admin/logstash - a tool for managing events and logs
Summary: app-admin/logstash - a tool for managing events and logs
Status: RESOLVED OBSOLETE
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: New packages (show other bugs)
Hardware: All Linux
: Normal enhancement (vote)
Assignee: Default Assignee for New Packages
URL: http://logstash.net/
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-12-14 06:11 UTC by Travis Hansen
Modified: 2019-05-14 21:40 UTC (History)
6 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
logstash-bin-1.4.2.ebuild (logstash-bin-1.4.2.ebuild,1.15 KB, text/plain)
2015-04-14 09:40 UTC, Tomáš Mózes
Details
agent.conf.sample (agent.conf.sample,336 bytes, text/plain)
2015-04-14 09:41 UTC, Tomáš Mózes
Details
logstash.confd (logstash.confd,37 bytes, text/plain)
2015-04-14 09:41 UTC, Tomáš Mózes
Details
logstash.initd (logstash.initd,1007 bytes, text/plain)
2015-04-14 09:41 UTC, Tomáš Mózes
Details
logstash.logrotate (logstash.logrotate,103 bytes, text/plain)
2015-04-14 09:41 UTC, Tomáš Mózes
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Travis Hansen 2012-12-14 06:11:03 UTC
logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs.
Comment 1 Travis Hansen 2013-02-02 04:42:43 UTC
I finally got around to putting together an ebuild for this. Available in my overlay:

https://github.com/travisghansen/chaos
Comment 2 Tomáš Mózes 2013-06-07 14:00:19 UTC
Thank you Travis, I was looking out for logstash and I see your work on the ebuild and the scripts, really thank you for sharing and doing it :)
Comment 3 Travis Hansen 2013-06-08 03:45:45 UTC
(In reply to Tomas Mozes from comment #2)
> Thank you Travis, I was looking out for logstash and I see your work on the
> ebuild and the scripts, really thank you for sharing and doing it :)

Thanks!  I haven't gotten around to putting together an ebuild for kibana (version 3 specifically) http://three.kibana.org/ but I've been using it in the datacenter and the graphs/functionality is awesome.

Lastly, you may want to check out using https://github.com/travisghansen/chaos/tree/master/sys-apps/lumberjack for logs that can't be sent through syslog.

If you bump into any issues let me know.
Comment 4 Tomáš Mózes 2013-06-08 06:50:24 UTC
I would like to try the syslog, lumberjack, logstash, redis, elasticsearch, graphite, statd, graphital, kibana stack, but it will certainly take some time to discover all of those projects and write the ebuilds ;) However, thanks to your contribution, it will be easier!
Comment 5 Tomáš Mózes 2014-05-23 13:31:50 UTC
Version 1.4.1 is out:
http://logstash.net/docs/1.4.1/release-notes

I've tried https://github.com/travisghansen/chaos/blob/master/sys-apps/logstash/logstash-1.4.0.ebuild, but it failed to start, here is a patch for 1.4.1 (tested on amd64):

--- logstash-1.4.0.ebuild       2014-05-21 11:56:49.721760335 +0000
+++ logstash-1.4.1.ebuild       2014-05-23 13:29:55.258940751 +0000
@@ -37,9 +37,6 @@
        dodir "${LS_ROOT_PATH}"
        cp -R "${S}"/* "${D}/${LS_ROOT_PATH}/"
 
-       # symlink launcher
-       dosym "${LS_ROOT_PATH}/bin/${PN}" "/usr/bin/${PN}"
-
        # requires pyes
        # https://logstash.jira.com/browse/LOGSTASH-211
        dobin ${FILESDIR}/logstash_index_cleaner.py


--- files/logstash.init
+++ files/logstash.init
@@ -28,7 +28,7 @@ else
        JAVA=java
 fi
 
-server_command="/usr/bin/logstash"
+server_command="/opt/logstash/bin/logstash"
 
 depend() {
        use net
Comment 6 Travis Hansen 2014-05-23 15:54:13 UTC
Tomas

Sorry about that.  I started 1.4.0 and never got it working on my setup.

I've just pushed an updated ebuild for 1.4.1 but it still doesn't work on my machine.

 /usr/bin/logstash
LoadError: no such file to load -- auto_gem
  require at org/jruby/RubyKernel.java:1085
  require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55

If this works for you then I'll know something is strange with my environment and not the ebuild itself.  Can you try it out and let me know?
Comment 7 Tomáš Mózes 2014-05-23 16:14:43 UTC
Probably because you have dev-lang/ruby installed and ruby sets:
RUBYOPT=-rauto_gem
Comment 8 Travis Hansen 2014-05-23 17:16:09 UTC
Tomas

Thanks so much for the feedback.  I've fixed the issue here:

https://github.com/travisghansen/chaos/commit/4ac33f98aae6e3c5870991f29e23fb548a2b7151
Comment 9 Tomáš Mózes 2014-07-01 09:48:27 UTC
Yes, that seems to work fine, tested on version 1.4.2.

logstash_index_cleaner.py is dead, replaced by Curator. A sample ebuild that we use:
https://github.com/hydrapolic/gentoo/blob/master/dev-python/elasticsearch-curator/elasticsearch-curator-1.0.0.ebuild
Comment 10 Amadeusz Żołnowski (RETIRED) gentoo-dev 2014-07-21 19:53:24 UTC
@java, @ruby teams: could you take a look at the ebuild? Could you add this to the tree?
Comment 11 Johann Schmitz (ercpe) (RETIRED) gentoo-dev 2014-07-22 04:47:47 UTC
This is a binary ebuild which bundles a lot of stuff.
Some time ago i started to work on a source version; most of the dependencies are in the last hope overlay: https://github.com/ercpe/lh-overlay.

Unfortunately, it turned out that jruby 1.9 (?) is a hard requirement. Due to the work required to support jruby through all the deps i have put this on hold.
Comment 12 Hans de Graaff gentoo-dev Security 2014-07-22 06:02:34 UTC
(In reply to Johann Schmitz (ercpe) from comment #11)

> Unfortunately, it turned out that jruby 1.9 (?) is a hard requirement. Due
> to the work required to support jruby through all the deps i have put this
> on hold.

I assume this should be jruby 1.7? That is tracked in bug 442230
Comment 13 Hans de Graaff gentoo-dev Security 2014-07-22 06:04:25 UTC
From the ruby side we will not pick this up as maintainers because this is a specific application. We are happy to look at some of the dependencies needed on the ruby side, though.
Comment 14 Tomáš Mózes 2015-04-14 09:40:49 UTC
Created attachment 401212 [details]
logstash-bin-1.4.2.ebuild
Comment 15 Tomáš Mózes 2015-04-14 09:41:15 UTC
Created attachment 401214 [details]
agent.conf.sample
Comment 16 Tomáš Mózes 2015-04-14 09:41:37 UTC
Created attachment 401216 [details]
logstash.confd
Comment 17 Tomáš Mózes 2015-04-14 09:41:48 UTC
Created attachment 401218 [details]
logstash.initd
Comment 18 Tomáš Mózes 2015-04-14 09:41:57 UTC
Created attachment 401220 [details]
logstash.logrotate
Comment 19 Ian Delaney (RETIRED) gentoo-dev 2015-04-14 13:50:06 UTC
added a bin version of this package  by Tomas, but this bug is for a full source version, which I doubt portage will ever be capable of supporting
Comment 20 Amadeusz Żołnowski (RETIRED) gentoo-dev 2015-08-16 10:09:31 UTC
So... maybe just change title to app-admin/logstash-bin and close as RESO FIXED? (-: It really looks like it's too much hassle to decompose logstash package.
Comment 21 James Le Cuirot gentoo-dev 2015-08-16 11:16:49 UTC
We use Logstash at work but on CentOS, I'm afraid. ;) If you only need the client then I recommend Beaver instead, which is written in Python and has a much smaller footprint. I like JRuby but it's a real hog and I think it was totally the wrong tool to use for a client that is supposed live quietly alongside some larger application.
Comment 22 Amadeusz Żołnowski (RETIRED) gentoo-dev 2015-08-16 12:09:12 UTC
You can also use app-admin/logstash-forwarder (written in Go) for forwarding logs to host with logstash+eleasticsearch setup.
Comment 23 Tomáš Mózes 2015-11-09 09:38:55 UTC
Seems like https://github.com/elastic/filebeat replaces logstash-forwarder.
Comment 24 Jesse Adelman 2017-05-17 22:01:11 UTC
FYI, logstash-bin is in the tree. I think this bug can be closed.
Comment 25 Tomáš Mózes 2017-05-18 12:18:52 UTC
(In reply to Jesse Adelman from comment #24)
> FYI, logstash-bin is in the tree. I think this bug can be closed.

Not really. We have a binary version in the tree, but the ultimate goal is to have a version built from source.
Comment 26 Patrice Clement gentoo-dev 2019-05-14 21:40:10 UTC
Feel free to file a PR if someone wants logstash (not the binary version) added to the Portage tree.