I don't know when it started. But after every sync, I see every updates file being processed. Excuse the slim bug report...
Seems to be after changes to the repository support for Portage. I don't know what kind of heuristic it used to have before.
@dev-portage, does HEAD have this problem* too? * The problem is, after a emerge-webrsync, I see messages for EVERY update file being processed.
The code that triggers it it the portage.update.grab_updates() function. It compares mtimes from the mtimedb and triggers updates if the timestamps have changed since the last time portage ran. Generally, you should only have updates triggered when the mtimes of files change inside $PORTDIR/profiles/updates. It can also be triggered by things such as changing the PORTDIR setting to a new path, since the mtime is recorded for the particular path. Also, changing from rsync tree to cvs tree or back again is a common way to trigger it since generally the two trees will have completely different profiles/updates mtimes.
(In reply to comment #2) > @dev-portage, does HEAD have this problem* too? I never use emerge-webrsync, but it should preserve mtimes. Maybe tar is broken? The only recent related change in portage did not come until January 25 which was after you reported this bug: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=d055cc9b6e46f37193841ef89843493e0d61ee7b
Thanks Zac. Removing you from CC. Found the problem in tar-1.25 on the client side. It is doing something wrongly with timestamps because the server generating the tarballs is preserving the timestamps correctly. Once the files get to PORTAGE_TMPDIR, the timestamps are wrong and that is done with tar. After that ther eare rsync'd to PORTDIR. @base-system: Tested tar-1.25{,-r1} to have this problem. tar-1.23-r4 does not. What can we do about it?
(1) come up with a simple testcase (2) see if upstream tar git HEAD has same problem (3) either report or find commit that fixes it
oh wow - i hit that yesterday, and thought it was interix specific ;) i don't have a solution yet though - still investigating here.
this commit looks suspicious: http://git.savannah.gnu.org/gitweb/?p=tar.git;a=commitdiff;h=57b11473b0135b1e092cf045f94635c934c7c63f
indeed, yes, that looks suspicious. i reverted the commit on my 1.25 tree and tried, but no luck... :( also my suspicion that the utime() call would be effectless because close() right after it would re-write modification times is not true, so i'm back to cloning the git repo and reverting commits up until i have a working version...
is there a reproducible test case for linux? it is a pain to test this with interix (long turn around times), and it seems both vanilla and gentoo-patched 1.25 work on my gentoo box :(
(In reply to comment #10) > is there a reproducible test case for linux? it is a pain to test this with > interix (long turn around times), and it seems both vanilla and gentoo-patched > 1.25 work on my gentoo box :( > Markus, no. I'm not able to devise a 'simple' testcase for this issue. Although emerge-webrsync can reproduce it readily (everytime) here. I imagine a git bisect would reveal the problem commit, if I can ever find time for that =/
hmm... i did some more debugging on friday. it seems the problem is within gnulib, not tar. Also, setting a breakpoint in any of the gnulib utime related functions makes the problem disappear... a heisenbug - great.
hmm... interesting: if i set a breakpoint right _after_ setting utime of the file, it's correct. if i then kill tar, the stamp is wrong..?! darn... haubi said he had problems with close() resetting timestamps, but i was unable to pin the problem on that before...
Created attachment 261159 [details, diff] set utime after closing file. huh?! i re-tried moving the set_stat call after the close for each file, and suddenly it works... hmm.. could you try tar with the attached patch whether it does the trick for you too? thanks
(In reply to comment #14) > Created an attachment (id=261159) [details] > set utime after closing file. > > huh?! i re-tried moving the set_stat call after the close for each file, and > suddenly it works... > > hmm.. could you try tar with the attached patch whether it does the trick for > you too? thanks > confirmed, patch works for me.
[1] suggests, that both the patch and our filesystems are broken :) which fs have you tested on? [1] http://www.mail-archive.com/bug-tar@gnu.org/msg03143.html
Created attachment 261397 [details, diff] fsync() before close() to sync timestamps to disk. upstream still thinks our filesystems are broken, as close() seems to get the wrong timestamp. the suggested solution (fsync() before close()) does work for me. can you test the new patch on linux too please? and tell me which filesystem your using please.
(In reply to comment #17) > Created attachment 261397 [details, diff] > fsync() before close() to sync timestamps to disk. > > upstream still thinks our filesystems are broken, as close() seems to get the > wrong timestamp. the suggested solution (fsync() before close()) does work for > me. can you test the new patch on linux too please? and tell me which > filesystem your using please. Patch doesn't work for tar-1.26, only tar-1.25-r{0,1}. I can't answer what underlying fs I am on. NFS on NetApp Filer behind closed gates.
@jeremy: could you try defining HAVE_BUGGY_NFS_TIME_STAMPS during the tar build? that should fix your NFS problems. regarding interix: the problem is to be solved in libsuacomp; i'm currently working on implementing the sync() and futimes() system calls. see [1] [1] http://lists.gnu.org/archive/html/bug-gnulib/2011-05/msg00305.html
Closing this bug, Interix is gone, and later tar are reported to be ok.
