logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs.
I finally got around to putting together an ebuild for this. Available in my overlay: https://github.com/travisghansen/chaos
Thank you Travis, I was looking out for logstash and I see your work on the ebuild and the scripts, really thank you for sharing and doing it :)
(In reply to Tomas Mozes from comment #2) > Thank you Travis, I was looking out for logstash and I see your work on the > ebuild and the scripts, really thank you for sharing and doing it :) Thanks! I haven't gotten around to putting together an ebuild for kibana (version 3 specifically) http://three.kibana.org/ but I've been using it in the datacenter and the graphs/functionality is awesome. Lastly, you may want to check out using https://github.com/travisghansen/chaos/tree/master/sys-apps/lumberjack for logs that can't be sent through syslog. If you bump into any issues let me know.
I would like to try the syslog, lumberjack, logstash, redis, elasticsearch, graphite, statd, graphital, kibana stack, but it will certainly take some time to discover all of those projects and write the ebuilds ;) However, thanks to your contribution, it will be easier!
Version 1.4.1 is out: http://logstash.net/docs/1.4.1/release-notes I've tried https://github.com/travisghansen/chaos/blob/master/sys-apps/logstash/logstash-1.4.0.ebuild, but it failed to start, here is a patch for 1.4.1 (tested on amd64): --- logstash-1.4.0.ebuild 2014-05-21 11:56:49.721760335 +0000 +++ logstash-1.4.1.ebuild 2014-05-23 13:29:55.258940751 +0000 @@ -37,9 +37,6 @@ dodir "${LS_ROOT_PATH}" cp -R "${S}"/* "${D}/${LS_ROOT_PATH}/" - # symlink launcher - dosym "${LS_ROOT_PATH}/bin/${PN}" "/usr/bin/${PN}" - # requires pyes # https://logstash.jira.com/browse/LOGSTASH-211 dobin ${FILESDIR}/logstash_index_cleaner.py --- files/logstash.init +++ files/logstash.init @@ -28,7 +28,7 @@ else JAVA=java fi -server_command="/usr/bin/logstash" +server_command="/opt/logstash/bin/logstash" depend() { use net
Tomas Sorry about that. I started 1.4.0 and never got it working on my setup. I've just pushed an updated ebuild for 1.4.1 but it still doesn't work on my machine. /usr/bin/logstash LoadError: no such file to load -- auto_gem require at org/jruby/RubyKernel.java:1085 require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55 If this works for you then I'll know something is strange with my environment and not the ebuild itself. Can you try it out and let me know?
Probably because you have dev-lang/ruby installed and ruby sets: RUBYOPT=-rauto_gem
Tomas Thanks so much for the feedback. I've fixed the issue here: https://github.com/travisghansen/chaos/commit/4ac33f98aae6e3c5870991f29e23fb548a2b7151
Yes, that seems to work fine, tested on version 1.4.2. logstash_index_cleaner.py is dead, replaced by Curator. A sample ebuild that we use: https://github.com/hydrapolic/gentoo/blob/master/dev-python/elasticsearch-curator/elasticsearch-curator-1.0.0.ebuild
@java, @ruby teams: could you take a look at the ebuild? Could you add this to the tree?
This is a binary ebuild which bundles a lot of stuff. Some time ago i started to work on a source version; most of the dependencies are in the last hope overlay: https://github.com/ercpe/lh-overlay. Unfortunately, it turned out that jruby 1.9 (?) is a hard requirement. Due to the work required to support jruby through all the deps i have put this on hold.
(In reply to Johann Schmitz (ercpe) from comment #11) > Unfortunately, it turned out that jruby 1.9 (?) is a hard requirement. Due > to the work required to support jruby through all the deps i have put this > on hold. I assume this should be jruby 1.7? That is tracked in bug 442230
From the ruby side we will not pick this up as maintainers because this is a specific application. We are happy to look at some of the dependencies needed on the ruby side, though.
Created attachment 401212 [details] logstash-bin-1.4.2.ebuild
Created attachment 401214 [details] agent.conf.sample
Created attachment 401216 [details] logstash.confd
Created attachment 401218 [details] logstash.initd
Created attachment 401220 [details] logstash.logrotate
added a bin version of this package by Tomas, but this bug is for a full source version, which I doubt portage will ever be capable of supporting
So... maybe just change title to app-admin/logstash-bin and close as RESO FIXED? (-: It really looks like it's too much hassle to decompose logstash package.
We use Logstash at work but on CentOS, I'm afraid. ;) If you only need the client then I recommend Beaver instead, which is written in Python and has a much smaller footprint. I like JRuby but it's a real hog and I think it was totally the wrong tool to use for a client that is supposed live quietly alongside some larger application.
You can also use app-admin/logstash-forwarder (written in Go) for forwarding logs to host with logstash+eleasticsearch setup.
Seems like https://github.com/elastic/filebeat replaces logstash-forwarder.
FYI, logstash-bin is in the tree. I think this bug can be closed.
(In reply to Jesse Adelman from comment #24) > FYI, logstash-bin is in the tree. I think this bug can be closed. Not really. We have a binary version in the tree, but the ultimate goal is to have a version built from source.
Feel free to file a PR if someone wants logstash (not the binary version) added to the Portage tree.
