After upgrading from rc62 to rc63, portage keeps reinstalling all packages that were updated along with the portage update on every subsequent world update. Results in a bunch of meaningless reinstalls on every world update. This bug seems to trigger only if the system is using binary packages. On the source system providing the binpkgs, everything seems fine. Downgrading to rc62 makes the reinstalls go away. Upgrade back to rc63 and they're there again. Reproducible: Always Steps to Reproduce:
addendum: It must be something in the criteria for determining if a tbz2 needs to be reinstalled. When I run portage without -g (and without FEATURES=getbinpkg), it works correctly. Whenever I do emerge -guDavN world, it will reinstall those packages over and over. However I've got no idea what determines the set of packages that are affected by this. They're definitely the ones that were updated when portage was first updated from 2.2_rc62 to 2.2_rc63. So they probably were in the resume list at the time, but dunno how that might be persisted in this strange kind of way...
Created attachment 221037 [details, diff] trigger --rebuilt-binarie with --usepkgonly instead of --usepkg It may be related to the new --rebuilt-binaries option which is triggered automatically with --usepkg/--getbinpkg --update --deep. You can use --rebuilt-binaries=n to forcefully disable it. In svn (patch attached) I've changed the behavior so that it's only triggered automatically with --usepkgonly or --getbinpkgonly.
Yup, running the world update with --rebuilt-binaries=n makes the useless reinstalls go away. So I guess there's some breakage in the comparison of the metadata that makes it trigger every time. Not sure why it keeps futzing around on the pkgs from the resume list, though. I have now upgraded and downgraded portage several times from and to rc62 and rc63, but it still wants to reinstall the binpkgs from the resume list when I first updated to rc63. Where does it save that resume list, btw?
The resume list is saved in /var/cache/edb/mtimedb. You can purge it with `emaint --fix cleanresume`.
(In reply to comment #4) > The resume list is saved in /var/cache/edb/mtimedb. You can purge it with > `emaint --fix cleanresume`. > Hum. Doing that doesn't help. However I noticed that the Packages index on the server side has those BUILD_TIME entries whereas /var/cache/edb/binhost/horst.phuk.ath.cx/packages/pentium-m/Packages differs from it exactly in *not* having them. Diff looks like this 344d343 < BUILD_TIME: 1266867281 412d410 < BUILD_TIME: 1266851415 527d524 < BUILD_TIME: 1266861057 812d808 < BUILD_TIME: 1266867559 876d871 < BUILD_TIME: 1266865732 891d885 < BUILD_TIME: 1266866806 1241d1234 < BUILD_TIME: 1266868373 1286d1278 < BUILD_TIME: 1266719707 Dunno if that means anything...
(In reply to comment #5) > Hum. Doing that doesn't help. However I noticed that the Packages index on the > server side has those BUILD_TIME entries whereas > /var/cache/edb/binhost/horst.phuk.ath.cx/packages/pentium-m/Packages differs > from it exactly in *not* having them. I think the local copy doesn't have them because it got downloaded by an older version of portage. If you removed the cached copy on the client then it should solve the problem. It's not being downloaded again because the timestamp in the header hasn't changed since the last time it was downloaded.
> I think the local copy doesn't have them because it got downloaded by an older > version of portage. If you removed the cached copy on the client then it should > solve the problem. It's not being downloaded again because the timestamp in the > header hasn't changed since the last time it was downloaded. > Nope. Deleted it and now the Package indices are the same, but so is the set of tbz2's it wants to reinstall over and over.
Can you check if there is any difference between /var/db/pkg/*/*/BUILD_TIME and the corresponding BUILD_TIME entries in the cached Packages file from the binhost? It should only reinstall if those are different for some reason.
ok, picking media-gfx/ufraw-0.16: # cat /var/db/pkg/media-gfx/ufraw-0.16/BUILD_TIME 1266705894 # grep -B 1 "CPV: media-gfx/ufraw-0.16" /var/cache/edb/binhost/horst.phuk.ath.cx/packages/pentium-m/Packages BUILD_TIME: 1266705894 CPV: media-gfx/ufraw-0.16
Created attachment 221045 [details] bogus emergelist btw, this is the output of "emerge -guDpvN world", whereas when using --rebuilt-binaries=n, no packages are merged. One more BUILD_TIME example from the list: # cat /var/db/pkg/media-libs/jpeg-8/BUILD_TIME 1266701067 # grep -B 1 "CPV: media-libs/jpeg-8" /var/cache/edb/binhost/horst.phuk.ath.cx/packages/pentium-m/Packages BUILD_TIME: 1266701067 CPV: media-libs/jpeg-8
Maybe there's something wrong with the /var/db/pkg metadata cache. You can check the cached BUILD_TIME value like this: portageq metadata / installed media-gfx/ufraw-0.16 BUILD_TIME
(In reply to comment #11) > Maybe there's something wrong with the /var/db/pkg metadata cache. You can > check the cached BUILD_TIME value like this: > > portageq metadata / installed media-gfx/ufraw-0.16 BUILD_TIME > # portageq metadata / installed media-gfx/ufraw-0.16 BUILD_TIME 1266705894 # portageq metadata / installed media-libs/jpeg-8 BUILD_TIME 1266701067 good thing I learn a bit more about portage's binpkg handling this way^^
Strangely, those BUILD_TIME look like they all match. What about in $PKGDIR/Packages? Does it contain correct BUILD_TIME values? You can also query them like this: portageq metadata / binary media-gfx/ufraw-0.16 BUILD_TIME
(In reply to comment #13) > Strangely, those BUILD_TIME look like they all match. What about in > $PKGDIR/Packages? Does it contain correct BUILD_TIME values? You can also query > them like this: > > portageq metadata / binary media-gfx/ufraw-0.16 BUILD_TIME > Ah, now this looks like something, as in, nothing: # portageq metadata / binary media-gfx/ufraw-0.16 BUILD_TIME # portageq metadata / binary media-libs/jpeg-8 BUILD_TIME (it outputs an empty line)... So let's clean that package index: # mv /usr/portage/packages/Packages ~ and retry: # emerge -guDpvN world These are the packages that would be merged, in order: Calculating dependencies... done! Total: 0 packages, Size of downloads: 0 kB Fair enough. Now the question remains: Why are there two copies of that Package index and why weren't they updated?
I think we should make absolutely sure we have the latest index file when using --rebuilt-binaries=y. That is, just download it. Thinking of that, I remember that I did have some issues with Package indices not getting updated a few months ago. I also resolved them by removing the local copy, but I hadn't seen these problems in a while, so I didn't bother to report. But maybe the code for (not) downloading the index file should get some attention.
(In reply to comment #14) > Fair enough. Now the question remains: Why are there two copies of that Package > index and why weren't they updated? One copy is for $PKGDIR and the other is for the remote binhost. They were both missing BUILD_TIME metadata because they were generated by older portage. I'm not sure how to fix it yet.
> One copy is for $PKGDIR and the other is for the remote binhost. Shouldn't those two ideally be always in sync? As far as I can see the client $PKGDIR copy also contains all binpkgs that exist on the server-side, and not just the ones that have in fact been downloaded. As I understand it, the local state is in /var/db/pkg, and the remote state is in $PKGDIR/Packages. Or does the Packages index also reflect some kind of local state (given I don't build binpkgs on the client side)? BTW, is there any more technical documentation available on this? I find the docs I've read so far to be somewhat... well... shallow, on the topic of binary package use.
(In reply to comment #17) > > One copy is for $PKGDIR and the other is for the remote binhost. > > Shouldn't those two ideally be always in sync? As far as I can see the client > $PKGDIR copy also contains all binpkgs that exist on the server-side, and not > just the ones that have in fact been downloaded. > As I understand it, the local state is in /var/db/pkg, and the remote state is > in $PKGDIR/Packages. Or does the Packages index also reflect some kind of local > state (given I don't build binpkgs on the client side)? $PKGDIR/Packages is strictly for caching local package metadata. Remote packages get merged into it as soon as they are downloaded. > BTW, is there any more technical documentation available on this? I find the > docs I've read so far to be somewhat... well... shallow, on the topic of binary > package use. No, it's almost completely undocumented at this time.
My plan is to write an upgrade script that will run in pkg_postinst when upgrading from a version of portage that doesn't support BUILD_TIME. The script will purge /var/cache/edb/binhost/ and add missing BUILD_TIME entries to $PKGDIR/Packages.
Created attachment 221751 [details, diff] only reinstall if binary package has non-empty BUILD_TIME field This is a really simple fix, and should handle all relevant cases.
This is fixed in 2.2_rc64.
