I tried to use libpcap-ringbuffer as a replacement for net-libs/libpcap. I've discovered several issues. 1) both actual versions (1.0.20041001 1.0.20050129-r2) shouldn't provide virtual/libpcap due to undefined symbols : mrness@a-nastac ~ $ objdump --dynamic-syms /usr/lib/libpcap.so | grep finddevs 00000000 D *UND* 00000000 pcap_platform_finddevs The function does not exist because pcap-ring.c fails to implement it. Of course, you could choose something like --with-pcap=linux but it defeats the purpose of libpcap-ringbuffer package (pcap support via MMAP). Note: pcap_platform_finddevs is used by ethereal. 2) in the event you decide to leave this package in the tree, you should either mark 1.0.20050129-r2 as stable on x86 or delete version 1.0.20041001. Also, on homepage you will find a version (0.9.20050810b) that seems to be even newer than 1.0.20050129, despite its 0.9 prefix! 3) If I'm not mistaken, libpcap-ringbuffer and libpcap are the only packages that provides virtual/libpcap. In this case, virtual/libpcap should be deleted and all the reverse dependencies should be actualized to net-libs/libpcap. In conclusion, my 2 eurocents: erase it from the tree.
Suggestion accepted. net-libs/libpcap-ringbuffer will be masked shortly and be removed a month or so after that. With libpcap-ringbuffer gone, we can get rid of virtual/libpcap. This means that packages that depend on virtual/libpcap should be updated to depend on net-libs/libpcap instead. The following scripts (hacked of other Gentoo devs' scripts) can be used to generate a list of offending packages: http://dev.gentoo.org/~vanquirius/files/deprecated-vlibpcap-scan.py http://dev.gentoo.org/~vanquirius/files/deprecated-vlibpcap-maintainers.sh List of offending packages as of today: http://dev.gentoo.org/~vanquirius/deprecated-vlibpcap-20061202.txt http://dev.gentoo.org/~vanquirius/deprecated-vlibpcap-maintainers-20061202.txt
i'd like take look at this package and fix the problems (i'd fixed some stuff before), please hold off on the removal from portage.
Fair enough :-). But I'm keeping it masked until then.
this package shouldn't exist at all because it is derived from the original libpcap (99.99% of the code is shared). if you want the MMAP thing, why not create a patch against current libpcap?
(In reply to comment #4) > this package shouldn't exist at all because it is derived from the original > libpcap (99.99% of the code is shared). > if you want the MMAP thing, why not create a patch against current libpcap? while i agree with you in principle, i'm not the right person to ask for this task given the raw diff size between 0.9.3 and 0.9.20050810b, nor am i the emissary to negotiate between the two upstreams (i guess there's a reason that *they* haven't merged). for a start, 0.9.20050810b uses automake already, then that 99.99% sharing is wide off the mark, the 0.9.3 *.[ch] size comes out at 850k whereas 0.9.20050810b is over 1.1 MB (even if i ignore scanner.c that can be regenerated from scanner.l, it's a 150k diff just in the source files). so a trivial task it isn't.
since the mmap code is implemented through pcap-ring.c, I guess you need to make a patch which import pcap-ring.c functionality to pcap-linux.c. however you're right, it isn't a trivial task. otherwise I would have done it myself :)
Created attachment 80008 [details, diff] new ebuild for libpcap-ringbuffer here's my first cut which has actually been lying on my disk for the previous version for some time now, i don't know what happened, i thought i'd sent it to someone already when i submitted the mmap(PROT_EXEC) fix, oh well. being the first cut, there're a few issues and i'll need some help to sort them out (hence the added CC's ;-). 1. the ringbuffer code uses memory barriers on linux (the mb() macro in particular) which presents a problem. it either comes from the system include file /usr/include/asm/system.h (on 2.6 based systems) or its copy in pcap-ring.h. in either case this causes breakage because of the use of alternative(). one breakage is that the resulting binary will have a nice textrel (and scanelf barfs on it as it's not in .text), the other bug is that there's noone in userland to actually apply the alternative (that mechanism is for the kernel, not userland), and the whole idea of having alternative() exposed to userland is just wrong. the way i fixed it locally is that i modified mb() and rmb() to not use alternative() at all, that compiles and works nicely (tested on tcpdump). so before this ebuild can go into portage, 1. scanelf should be fixed, 2. system headers (and optionally pcap-ring.h) should be cleaned of alternative(). 2. for some reason make install creates a dummy xxx file (which is actually a copy of the gzip'ped manpage), no idea what to do with it, probably the makefile needs a patch, but considering my lazyness factor, i'd be happy if gentoo's install system could be told to simply ignore it ;-). 3. make install creates both usr/lib/libpcap-0.9.3.so and usr/lib/libpcap.so.0.9.3 (as full files, i.e., we get two copies instead of the former being a symlink to the latter), no idea how to fix it the simplest way (lazyness factor applies again).
Any comments netmon? Can you review/comment/commit? This package would be a shame to lose.
I have to agree that this would be a shame to lose. I think it was put there for a reason, and it doesn't look like it's going to be put in upstream anytime soon. Until that happens I think the virtual should stay around. In fact it'd be nice if snort had the virtual depend instead of specifically net-libs/pcap...
Ok scratch what I said about snort - now I know why it doesn't doh :( Initializing Network Interface eth0 snort: symbol lookup error: snort: undefined symbol: pcap_open_live [0.00 0.05 0.03][root]@achilles:~# snort -dev snort: symbol lookup error: /usr/lib/libpcap.so.0: undefined symbol: pcap_platform_finddevs
(In reply to comment #10) > Ok scratch what I said about snort - now I know why it doesn't doh :( > > Initializing Network Interface eth0 > snort: symbol lookup error: snort: undefined symbol: pcap_open_live > [0.00 0.05 0.03][root]@achilles:~# snort -dev > snort: symbol lookup error: /usr/lib/libpcap.so.0: undefined symbol: > pcap_platform_finddevs ...and if you had actually tried my ebuild: # readelf -s /usr/lib/libpcap.so.0.9.3 | egrep pcap_open_live\|pcap_platform_finddevs 76: 00007ff0 71 FUNC GLOBAL DEFAULT 10 pcap_platform_finddevs 82: 00008dc0 5093 FUNC GLOBAL DEFAULT 10 pcap_open_live
(In reply to comment #7) > Created an attachment (id=80008) [edit] > new ebuild for libpcap-ringbuffer > > here's my first cut which has actually been lying on my disk for the previous > version for some time now, i don't know what happened, i thought i'd sent it to > someone already when i submitted the mmap(PROT_EXEC) fix, oh well. being the > first cut, there're a few issues and i'll need some help to sort them out > (hence the added CC's ;-). > > 1. the ringbuffer code uses memory barriers on linux (the mb() macro in > particular) which presents a problem. it either comes from the system include > file /usr/include/asm/system.h (on 2.6 based systems) or its copy in > pcap-ring.h. in either case this causes breakage because of the use of > alternative(). one breakage is that the resulting binary will have a nice > textrel (and scanelf barfs on it as it's not in .text), the other bug is that > there's noone in userland to actually apply the alternative (that mechanism is > for the kernel, not userland), and the whole idea of having alternative() > exposed to userland is just wrong. the way i fixed it locally is that i > modified mb() and rmb() to not use alternative() at all, that compiles and > works nicely (tested on tcpdump). so before this ebuild can go into portage, 1. > scanelf should be fixed, 2. system headers (and optionally pcap-ring.h) should > be cleaned of alternative(). > > 2. for some reason make install creates a dummy xxx file (which is actually a > copy of the gzip'ped manpage), no idea what to do with it, probably the > makefile needs a patch, but considering my lazyness factor, i'd be happy if > gentoo's install system could be told to simply ignore it ;-). > > 3. make install creates both usr/lib/libpcap-0.9.3.so and > usr/lib/libpcap.so.0.9.3 (as full files, i.e., we get two copies instead of the > former being a symlink to the latter), no idea how to fix it the simplest way > (lazyness factor applies again). > The problem with the libpcap-ringbuffer ebuilds currently masked in portage is that they run configure with the '--with-pcap=ring' argument. I think the correct way to run it is with plain old '--with-pcap=linux', as this PaXTeam ebuild does. That seems to bring in the pcap-ring.c code automatically and allows for the switching between classic libpcap and MMAP/ringbuffer libpcap with the absence/presence of the environment variable PCAP_FRAMES. (If I understand the READMEs correctly, that is.) I have verified that PaXTeam's ebuild compiles and installs cleanly with libpcap-0.9.20060417.tar.gz from Phil Wood's website (http://public.lanl.gov/cpw). I did a 'make install' to a clean DESTDIR and did not see any unnecessary files installed...so apparently points 2 and 3 from the comment above are fixed. Also, I didn't get any complaints of TEXTREL's (point 1 above), so perhaps those mb() and alternative() calls have been fixed, too. Two suggestions for discussion: 1) Add ~amd64 to the ebuild since Phil Wood indicates that it works on x86_64 since late July 2005 and since I am currently using it on AMD64. :) 2) Add a file to be put in /etc/env.d with "PCAP_FRAMES=max" so that the pcap-ring.c code is called by default. I figure that if folks are emerging this package, they probably want the MMAP/ringbuffer code on by default; otherwise they would just use plain old libpcap from tcpdump.org. If several other people can verify that this ebuild works for them, perhaps we can unmask this version (0.9.20060417) of the package and put it in the testing branches.
Created attachment 87049 [details] ebuild for libpcap-ringbuffer-0.9.20060417
Created attachment 87050 [details, diff] small patch for the new ebuild
this new ebuild + patch is in response to comment #12, with a caveat or two. 1. the memory barrier problem is solved in that gentoo's linux-headers fixes the use of alternative() in mb() and rmb(), but it still doesn't fix wmb() and possibly others, i didn't look further. also the whole alternative() remains userland accessible. i guess we'll have to wait till someone upstream shows up to take up the linux-headers package again and hope that this will get fixed too. 2. the upstream makefile got apparently fixed up over time so it makes more use of automake and installs properly, except for the header files which i fixed in the attached makefile patch (i also fixed a dummy make target dependency), this should probably go upstream. 3. the library installation is also fixed by better using automake facilities, however it raises another potential problem that i think we can't easily solve ourselves. namely, it's about library versioning. as it is, libpcap.so has a soname of libpcap-0.9.3.so, which is a compromise solution used by libtools when one specifies -release instead of -version. the previous upstream 'solution' for this versioning issue was that the makefile explicitly created symlinks for previous versions, which is no longer done, hence users may need revdep-rebuild. now, all this is the wrong approach of course, the proper way would be to finally start making use of the ELF library versioning facilities, both in using soname properly and perhaps symbol versioning. this however needs upstream's help/decision as they're in the best position to know when the API changes in ways that require bumping various parts of the library version. another related issue is that i don't know how closely the ringbuffer fork tracks the original libpcap, in particular, when the ringbuffer mode is disabled, is one guaranteed to get back the behaviour of any particular libpcap version (this in turn would affect how versioning can be used to allow replacing one libpcap library with a forked one)? 4. the ebuild enables ~amd64 as suggested by Brett, but i haven't verified it myself (as in, i don't know if it even compiles, although i don't see any reason why it shouldn't, at least the mb() stuff is fixed on amd64 as well). 5. i didn't do anything for the environment variable part as i don't know gentoo's policy on this, the CC'd folks are more than welcome to chime in ;-). an alternative could be a patch that enables the mmap code by default.
Created attachment 87057 [details] ebuild for Phil Wood's libpcap ver. 0.9.20060417 Coming a little late to this but this packager should deffinitly not be dropped and it should provide virtual/libpcap. Anyone watching the IDS mailing list will agree. Well it looks like The PaX Team and I spent some thime doing the same thing. I'll attach my ebuild anyways. I also added the ~amd64, but am unable to test it. I can confirm that setting PCAP_FRAMES=0 will disable use of the ringbuffer and revert back to the old pcap style capture. I handeled the header file issue in the ebuild with doins instead of patching the make file. Not sure with method is perfered or standard. doins seemed cleaner to me. I added some dosyms to set up symlinks simular to the ones set in the regular libpcap ebuild. This lets a user uninstall libpcap and install libpcap-ringbuffer with out having to rebuild snort and tcpdump. Testing showed: That tcpdump, snort, and ethereal will compile against libpcap-ringbuffer from this ebuild. They do not need to be recompiled when switching from libpcap to libpcap-framebuffer, but ethereal will not work with the PCAP_FRAMES= set.It needs to be called like 'PCAP_FRAMES=0 ethereal' or 'PCAP_FRAMES=0 tethereal'. I added the PCAP_FRAMES= env to /etc/env.d/77ringbuffer. In testing I found that setting PCAP_FRAMES=max does not always work, but setting it to PCAP_FRAMES=32768 (the equivilent to max) does always work, so that is what I used. Added einfo stuff to explain the PCAP_FRAMES= env var and explain the (t)ethereal issue. I think that is it... Wally
We have been using libpcap-ringbuffer with tcpdump to provide archives of network data and it has help a lot. Packet loss dropped from around 15% to maybe a couple hundred packets (out of around 75GB+ of traffic per-day). I can also confirm that setting PCAP_FRAMES to 0 allows tools that will not work with the ringbuffer to run correctly. IMO it would be nice to see this as a USE to libpcap but like someone said the diffs are big, or change ebuilds that require a pcap driver to check for virtual/libpcap (like snort) instead of net-libs/libpcap (like ethereal and tcpdump) Using Jasons ebuild everything went smoothly.
If no netmon team member is going to solve this, I will. Consider this comment as a one week notice.
I've commited a merged version between these 2 proposals. I've tested this on x86 and amd64, but... The generated library contains DT_TEXTREL: TYPE TEXTRELS FILE scanelf: scanelf_file_textrels(): ELF libpcap.so has TEXTREL markings but doesnt appear to have any real TEXTREL's !? ET_DYN libpcap.so @solar : please help me with this. I will keep it masked until you/we solve this issue.
(In reply to comment #19) > I've commited a merged version between these 2 proposals. I've tested this on > x86 and amd64, but... > The generated library contains DT_TEXTREL: > TYPE TEXTRELS FILE > scanelf: scanelf_file_textrels(): ELF libpcap.so has TEXTREL markings but > doesnt appear to have any real TEXTREL's !? > ET_DYN libpcap.so have you read comment #7? it talks about the 'ghost' textrels coming from the memory barrier code. if you fix that in /usr/include then pcap builds without textrels. on another note, is anyone familiar with http://www.mail-archive.com/ntop-misc@listgateway.unipi.it/msg00320.html ? is it worse/better than this fork? should one be preferred over the other?
Okay, sorry for the delay, gonna pick this one up now(In reply to comment #20) > (In reply to comment #19) > on another note, is anyone familiar with > http://www.mail-archive.com/ntop-misc@listgateway.unipi.it/msg00320.html ? is > it worse/better than this fork? should one be preferred over the other? Would be nice to see if it works as well. Can you take care of some testing?
added self to cc list
I've been looked at pfring. It is rumored to give slightly better performance then ringbuffer. The setup/install is a little harry. You need to I think it would require 2 ebuilds. one for a kernel module/patch and one for the actual libpcap. I'm not sure yet if a module can be built or if you actually have to recompile the whole kernel. take a look here... http://wiki.ntop.org/mediawiki/index.php/PF_RING first one has a more detailed description. --Wallace
can you provide ebuilds for it?
(In reply to comment #24) > can you provide ebuilds for it? > I'm currently working on a huge pfring project at work. Once that is finished I plan on creating ebuilds for both pfring and the modified version of libpcap it requires. There is one issue I have not figured out yet with regards to ebuilds. Pfring requires a patch to the kernel. The process of creating the patch is rather convoluted. They do not provide a patch but rather they provide a script that creates a patch based on your kernel sources. Does anyone know of any precedence for creating an ebuild that patches an existing kernel source installed on the system? I'm thinking that some kernel module ebuilds might do this. If anyone has a good example let me know. Wally
Well if the kernel patch just means building an additional module then it is really easy deeper patching is a bit hairy....
(In reply to comment #26) > Well if the kernel patch just means building an additional module then it is > really easy > > deeper patching is a bit hairy.... > Then an ebuild for PF_RING would be a bit hairy. PF_RING changes several core kernel networking device functions (netif_*). Perhaps this is something that could be included in the gentoo-sources patchsets? I'm not at all familiar with how patches get approved for inclusion in the gentoo-sources patchsets, perhaps someone could enlighten us?
> > Then an ebuild for PF_RING would be a bit hairy. > > PF_RING changes several core kernel networking device functions (netif_*). > Perhaps this is something that could be included in the gentoo-sources > patchsets? I'm not at all familiar with how patches get approved for inclusion > in the gentoo-sources patchsets, perhaps someone could enlighten us? > Yes it would be hairy that is why I haven't done this yet. From looking at the gentoo-sources dev page it only looks like they add stuff that will most likely be added to the development kernel... http://dev.gentoo.org/~dsd/genpatches/faq.htm "If it's a serious and non-intrusive bugfix that has been accepted into the upstream development kernel, then usually yes (please open a bug). Otherwise, generally the answer is no - instead of getting new features added to our kernel, we'd prefer you to contact the authors of that code and get them to finish off their work and submit it to the mainline kernel so that everyone can benefit." Although I would call this project "serious" I'm not sure how close it comes to being added to the dev kernel or if that is something they are striving for and it is definitely not "non-intrusive", so I don't think it is a candidate right now for inclusion in gentoo-sources. If I'm wrong by all means let me know. I think patching a currently installed source tree could be done and could be done safely... 1. either tar up a copy of the sources in /usr/src/linux and suck it in to the build sandbox, generate a pfring patch, and install it as a new kernel 2. mimic a gentoo-sources up to the point where all the current patches are applied, generate a pfring patch, apply the patch and install it as a new kernel but I'd really like to see an example of this before I go down that road. There is always the option of starting a new kernel sources (sensor-sources) specifically designed for IDS/IPS/packet capture sensors, but I really don't have time to maintain a project that size. Any other ideas or if someone knows of an existing example of the "hairy" options let me know? Wally
Doesn't matter if kernel team add your patch or not. It is possible to add a info message in pkg_postinst instructing users to download and apply that patch against their kernel, ya know...
I have added ebuilds for libpfring (Bug 167108) and libpcap-pfring (Bug 167110). libpcap-pfring depends on libpfring. I handled the kernel issue by adding a check to the libpfring ebuild that checks to see if the user is using a PF_RING enabled kernel and if not the ebuild exits with a link to a wiki page I created outlining how to create a PF_RING kernel (http://gentoo-wiki.com/Pfring). If there are any questions related to the two ebuild please comment at their bugs and not here. If you find any issues with the wiki feel free to email me directly. Wally
latest suggestion being broken with newer kernels + only cvs available Feel free to reopen once upstream has supported tarballs in place, marking LATER until then
libpcap-0.9.8.20080206.tar.gz released http://public.lanl.gov/cpw/ - I haven't tested it.
