[ebuild R ] sys-apps/pciutils-3.1.7-r1 USE="zlib -network-cron -static-libs" 0 kB [ebuild R ] sys-fs/udev-164-r2 USE="extras (-selinux) -test" 0 kB [ .. ] checking for acl_init in -lacl... yes checking acl/libacl.h usability... yes checking acl/libacl.h presence... yes checking for acl/libacl.h... yes checking for LIBUSB... yes checking for USBUTILS... yes checking for /usr/share/pci.ids... no checking for /usr/share/hwdata/pci.ids... no checking for /usr/share/misc/pci.ids... no configure: error: pci.ids not found, try --with-pci-ids-path= [ .. ] And by looking at code in extras/usb-db/usb-db.c, it doesn't look like it has support for gzipped copy. I've masked sys-apps/pciutils-3.1.7-r1 for now.
package.mask: # Samuli Suominen <ssuominen@gentoo.org> (28 Mar 2011) # Masked temporarily until bug 360849 is solved so people # can emerge udev. =sys-apps/pciutils-3.1.7-r1
bug in udev, not pciutils. it shouldnt be poking the pci.ids file directly when libpci already provides funcs for parsing it.
Old patch here: http://www.spinics.net/lists/hotplug/msg02729.html But if you look at the discussion in same thread these have been refused in past. GregKH could you please take a look? We would want to compress pci.ids and usb.ids, and not install the uncompressed copies at all.
Created attachment 267563 [details, diff] zlib support This is sam'ish with the one in http://www.spinics.net/lists/hotplug/msg02729.html (As in, I'm not really the author) Needs also --with-pci-ids-path=/usr/share/misc/pci.ids.gz passed to ./configure so it doesn't bail out.
*** Bug 388027 has been marked as a duplicate of this bug. ***
Now pciutils-3.1.8 is masked too for same broken behavior with udev.
FYI: emerge udev-171-r2 does compile with pciutils-3.1.8 -- so why mask ?
also udev-171 is no longer in the tree udev-171-r1 {extras ..} also compiles with pciutils-3.1.8 {zlib}..
pciutils-3.1.8 installs /usr/share/misc/pci.ids.gz udev-171-r2 is passing argument --with-pci-ids-path=/usr/share/misc/pci.ids it doesn't fail to compile, but the database file isn't there either, and obviously it doesn't work
so in essence udev requires pciutils to be compiled as USE="-zlib" and up to pciutils-3.1.7 a patch has been added to co-install the uncompressed file when USE="zlib" .. this was dropped in pciutils-3.1.7-r1 which causes failure of udev to work properly as it loses access to pci.id when it is installed compressed .. so how many other programs use pci.id uncompressed/compressed? do we know ?
I notice that usbutils has a cron script that regularly updates usb.ids/usb.ids.gz files .. usbutils also co-installs usb.ids and usb.ids.gz with +zlib .. should pciutils get a cron update script as well? .. it does have the update script and it works okay in 3.1.7 but with 3.1.8 it clobbers the pci.ids file if it is present..( had been added manually for testing) .. I suspect the old patch fixes update-pciids as well as the install of pci.ids initially
the man page for update-pciids .. is erronous as with +zlib it does not install pci.ids but pci.ids.gz .. should this go to a pciutils bug .. I was looking for a work arround to the udev pci.ids requirement when id found this out .. hoped that update-pciids would install pci.ids but it does not.
re comment 11 -- I see I missed the network-cron flag in the ebuild of pciutils .. sorry for the noise ..
I see that this issue just won't die ..Bug 166790 .. can we not just revert b ack the update-pciids patch and get on with business ..
(In reply to comment #3) > Old patch here: > > http://www.spinics.net/lists/hotplug/msg02729.html > > But if you look at the discussion in same thread these have been refused in > past. > > GregKH could you please take a look? We would want to compress pci.ids and > usb.ids, and not install the uncompressed copies at all. Why compress at all ? What are we gaining here, half a megabyte ?
Please refrain from posting useless comments if you don't have anything contructive, like patch to udev, to provide.
(In reply to comment #16) > Please refrain from posting useless comments if you don't have anything > contructive, like patch to udev, to provide. Certainly not for you to judge, do you know the answer ? Why do we bother to compress at all ?
turns out, udev only cares about the pci db when USE=hwdb. so we can restrict the udev-inflicted-pain significantly by having it depend on pciutils[-zlib] when people have USE=hwdb: http://sources.gentoo.org/sys-fs/udev/udev-171-r3.ebuild?r1=1.1&r2=1.2 http://sources.gentoo.org/sys-fs/udev/udev-175.ebuild?r1=1.6&r2=1.7 http://sources.gentoo.org/sys-fs/udev/udev-175-r1.ebuild?r1=1.1&r2=1.2 which means i've also dropped the pciutils mask: http://sources.gentoo.org/profiles/package.mask?r1=1.13312&r2=1.13313 as for the original error, udev-171+ ebuilds have this fixed by explicitly specifying the path: --with-pci-ids-path=/usr/share/misc/pci.ids we want that anyways for cross-compiling setups among other things because we can't search the / tree and assume the filesystem layout and a future note: if any other package ends up trying to read the ids database directly, people need to *not* package mask pciutils. update that package to depend on pciutils[-zlib] and then file a bug for that package.
Created attachment 295319 [details, diff] zlib support for udev[hwdb] bah, sorry Samuli, didn't realize you already wrote a patch. here's one that should be less invasive (at the cost of more indirection), and make the zlib dep optional. it also has a minor fix -- the constant free(line) is unnecessary. i slapped together this patch so udev can optionally support compressed pci/usb databases. associated ebuild support: --- udev-175-r1.ebuild +++ udev-175-r1.ebuild @@ -34,7 +34,7 @@ LICENSE="GPL-2" SLOT="0" IUSE="build selinux debug +rule_generator hwdb acl gudev introspection - keymap floppy edd doc" + keymap floppy edd doc zlib" COMMON_DEPEND="selinux? ( sys-libs/libselinux ) acl? ( sys-apps/acl dev-libs/glib:2 ) @@ -62,7 +62,12 @@ else fi RDEPEND="${COMMON_DEPEND} - hwdb? ( >=sys-apps/usbutils-0.82 sys-apps/pciutils[-zlib] ) + hwdb? ( + >=sys-apps/usbutils-0.82 + sys-apps/pciutils + zlib? ( sys-libs/zlib ) + !zlib? ( sys-apps/pciutils[-zlib] ) + ) acl? ( sys-apps/coreutils[acl] ) !sys-apps/coldplug !<sys-fs/lvm2-2.02.45 @@ -132,6 +137,8 @@ EPATCH_FORCE="yes" epatch fi + use hwdb && use zlib && epatch "${FILESDIR}"/${PN}-175-zlib.patch + # change rules back to group uucp instead of dialout for now sed -e 's/GROUP="dialout"/GROUP="uucp"/' \ -i rules/{rules.d,arch}/*.rules \
To make things more interesting, a few days ago udev turned usb-id/pci-db into builtins: http://git.kernel.org/?p=linux/hotplug/udev.git;a=commit;h=3cf5266b029c1ec4522edc6b42d874d80407a246. It's a bit annoying, that udev slowly turns into systemd backend, instead of being used as one.
i've added the simple zlib patch. feel free to ping me if future versions need an updated patch and i'll take care of it. http://sources.gentoo.org/sys-fs/udev/udev-175-r1.ebuild?r1=1.3&r2=1.4 http://sources.gentoo.org/sys-fs/udev/files/udev-175-zlib.patch?rev=1.1
@vapier: Has this been sent upstream so that we can get it into udev proper before 176 is released?
no ... i'm assuming upstream doesn't care about zlib support. the hal guys didn't care when we tried with them.
@vapier: After looking at this patch, I don't think it should be forwarded upstream, but, I think it should be re-worked either by you, me or someone else, so that it is possible to enable/disable zlib when running the configure script. Once that happens, I think we might have a shot at getting this in upstream.
At least I don't think there would be a reason to refuse it if we propose it in a way that it is off by default.
if upstream is open to the idea of adding support, then i have no problem reworking the patch to be a standard configure option/etc... but if upstream doesn't care, then i'm not going to waste time on it.
I am re-opening this bug, because I feel that we need a better fix. Here are my thoughts on what should happen. 1) Mike/Samuli, can you please write a patch against current udev git that makes this feature controlable by a configure switch and attach it to this bug? 2) At that point, I will submit the patch myself to the udev mailing list and see what Kay says about it. I need to know one thing however. Does upstream pciutils compress the pci and usb id databases and only install the compressed copy when zlib support is on? If they do, that may be a good argument to get him to support zlib in udev. 3) If the support is rejected upstream, I feel that this should fall back on pciutils to be sure that there are always uncompressed copies of the databases available for udev to read. Thanks, William
I spoke with Kay, the udev author, on irc today. I asked him what he thought about zlib support for reading the hardware databases in udev. Here is his response: < kay> WilliamH: don't think that helps _anything_. people should stop optimizing at the wrong corner < kay> WilliamH: that astuff should be properly indexed, not compressed < kay> WilliamH: we will get there this year, i hope < kay> WilliamH: the dbs, as they are today, will just go away < WilliamH> kay: ok, thanks, I'll pass that on. It appears that upstream is working on taking this a different direction.
so the status-quo remains unchanged. i disagree with (3).
Based on my conversation with Kay, I am dropping this patch to udev-175 and assigning this bug to base-system to fix pciutils.
(In reply to comment #29) > so the status-quo remains unchanged. i disagree with (3). I'm confused here, what do you disagree with?
I read the bug referred to in comment #14, and this combined with my conversation with Kay earlier today, makes it pretty clear that upstream is not interested in supporting a compressed pci.ids file. Usbutils doesn't seem to remove the uncompressed usb.ids file, so I'm thinking that it would be good for pciutils to install the uncompressed database as well.
i'm not changing pciutils. usbutils not installing both is an oversight that'll get fixed.
Mike, I do not have an issue with carrying a custom patch for a few versions of a product if upstream is considering encorporating that feature. In this case, Kay has made it clear that he does not see how this feature helps anything, and he has stated that the databases will go away at some point. Carrying custom patches for udev, pciutils and usbutils is not a good long term solution for this issue. Imo we need to convince these upstreams to agree on something, or come up with a solution that doesn't involve patches.
(In reply to comment #34) > Carrying custom patches for udev, pciutils and usbutils is not a good > long term solution for this issue. The "custom patch" for pciutils was the one that made it install compressed and uncompressed pci.ids in the first place. Upstream pciutils does *not* install an uncompressed pci.ids if zlib is enabled.
Hi Nick, (In reply to comment #35) > The "custom patch" for pciutils was the one that made it install compressed > and uncompressed pci.ids in the first place. > Upstream pciutils does *not* install an uncompressed pci.ids if zlib is > enabled. Upstream udev does *not* support zlib at all and has no intention of doing so (see comment #28 for upstream's response to this). So, we are just trading one custom patch for another or udev can just require zlib to be disabled on pciutils and usbutils when it stops installing the uncompressed db when zlib is enabled.
Here is a summary of what we found on this bug: 1) Udev upstream does not want to support zlib compression for the hardware databases (see comment #28). 2) when zlib is active for pciutils and usbutils, they will only install compressed hardware databases. There will need to be a custom patch to udev to support zlib compressed databases, or custom patches to make pciutils and usbutils install both compressed and uncompressed databases. I understand that base-system doesn't want to custom patch pci and usb utils. Since udev upstream has rejected support for reading compressed databases, the udev maintainers (myself and Robin Johnson <robbat2@gentoo.org>) agree that the best way forward is to package.use.mask zlib for pciutils and usbutils until either the upstreams agree or sys-fs/udev moves away from using the databases these packages provide.
I'm re-opening this due to ssuominon approaching me on irc. The conflict is because other packages in the tree use pciutils[zlib], and if that happens udev is broken because it doesn't support compression and when pciutils has compression enabled it does not keep the flat file around. Also pciutils is moving that direction.
Latest update: I just spoke with Kay on #udev. He made it abundently clear that upstream is not going to support zlib. So, we are back to a custom patch for udev.
This has been re-applied for now.
i can add another USE flag to pciutils/usbutils to split the logic between "i want to support compressed files (USE=zlib)" and "i want my files to be compressed". that way all the people who want to save <50ms can. iirc, that's what my local tests showed -- reading the entire pci db 50 times with stdio line by line vs zlib stdio took slightly longer when the file was compressed (on the order of milliseconds).
(In reply to comment #41) > i can add another USE flag to pciutils/usbutils to split the logic between "i > want to support compressed files (USE=zlib)" and "i want my files to be > compressed". that way all the people who want to save <50ms can. iirc, that's > what my local tests showed -- reading the entire pci db 50 times with stdio > line by line vs zlib stdio took slightly longer when the file was compressed > (on the order of milliseconds). That might be the best choice since udev doesn't want its database files compressed. That way, I could make udev use dep on your packages with that use flag off.
I am missing something here. I understand the hwdb USE flag requiring sys-apps/pciutils[-zlib]. BUT /usr/portage/sys-fs/udev/udev-171-r5.ebuild shows: hwdb? ( >=sys-apps/usbutils-0.82 sys-apps/pciutils[-zlib] ) extras? ( >=sys-apps/usbutils-0.82 sys-apps/pciutils ) and (further down): src_configure() { econf \ [...] $(use_extras hwdb) \ so with the extras USE flag, hwdb gets set anyway. Shouldn't the extras USE flag also require sys-apps/pciutils[-zlib]?
(In reply to comment #42) that is an option, but personally, i'd still prefer to maintain this zlib patch in udev. i have no problem being listed as the maintainer and if it doesn't apply to new versions, just drop it and file a report for me. in the uncompressed db case, i don't think there are any perf issues w/the patch enabled.
