After a recent tree signing, I'm facing an issue with "emerge --sync" command it once in while. The error is below: * - timestamp: 2018-07-26 23:38:35 UTC * Verifying /usr/portage ...!!! Manifest verification failed: Manifest mismatch for metadata/Manifest.gz __size__: expected: 1985, have: 1983 I'm using the official mirror sync-uri=rsync://rsync.asia.gentoo.org/gentoo-portage The second attempt to sync after few min syncs ok usually. So that sounds like a race of conditions during syncing a mirror with the main tree and a client who is syncing with a mirror during that perioud. I suggest to introduce a locking mechanism to prevent this from happening. That gives a wrong impression that I'm under MITM attack and we should fix this problem.
I'm increasing priority because it happens almost every time I sync using a mirror.
It may be possible to fix this issue by using the new feature (hardlinks): https://bugs.gentoo.org/660410
I ran into the same problem before enabling the hardlinks configuration setting. The count is typically off by two. I compared the output of `ls` in both /usr/portage and /usr/portage/.tmp-unverified-download-quarantine after it happened, and it's tripping up over /usr/portage having two more directories: distfiles and packages.
This bug has been resolved or I don't face it any more at least. So the bug #660410 might have been a solution. Closing.
I have just faced for other mirrors so reopening again
ok, still happening once in awhile sent 98.33K bytes received 16.24M bytes 117.14K bytes/sec total size is 208.29M speedup is 12.75 * Manifest timestamp: 2020-05-16 00:08:25 UTC * Valid OpenPGP signature found: * - primary key: DCD05B71EAB94199527F44ACDB6B8C1F96D8BF6D * - subkey: E1D6ABB63BFCFB4BA02FDF1CEC590EEAC9189250 * - timestamp: 2020-05-16 00:08:25 UTC * Verifying /usr/portage/.tmp-unverified-download-quarantine ...!!! Manifest verification failed: Manifest mismatch for metadata/Manifest.gz __size__: expected: 2828, have: 2824 Any progress on this bug?
It started happening for me about 7 days ago. I anacron an emerge --sync very night, they started failing. The actual reason is not in any of the logs but if I run emerge --sync --quiet=n or emaint sync --repo=gentoo I see that there is a random (apparently) failure in the size or checksum of something in the temporary tree. This is an rsync --sync and it has become, apparently, permanent. I note another complaint "fixed" the problem with web-rsync, which implies a permanently corrupted file in the tree, but it looks like a race to me. I did check (via dmesg) for issues with file integrity; none reported, but it is a RAID SSD file system on Linux 5.8.9 so the obvious suspicion is that it is a Linux bug. Alas I've deleted my 5.7 builds so this will take a while to check.
Replying to my own comment... The problem "disappeared" overnight; when I checked after my normal 3am --sync it had worked correctly. It may somehow be network speed related; the previous nights there had been substantial other download activity at 3am, however I also saw consistent failures during the day and running an "emaint sync -A" now works just fine. Could it be that there are temporary inconsistencies on the gentoo repo itself, and all its clones?
(In reply to John Bowler from comment #8) > Replying to my own comment... The problem "disappeared" overnight; when I > checked after my normal 3am --sync it had worked correctly. It may somehow > be network speed related; the previous nights there had been substantial > other download activity at 3am, however I also saw consistent failures > during the day and running an "emaint sync -A" now works just fine. > > Could it be that there are temporary inconsistencies on the gentoo repo > itself, and all its clones? If you are using rsync, there is no guarantee you get an atomic copy of the repo; so races definitely exist. We are looking to replace rsync entirely with something else that hopefully avoids this problem; I'm not aware of any plans to 'fix' rsync. -A
Hello Alec, what do you recommend we use in the meantime? Cheers.
Git is a good alternative that doesn't have any risk of these kinds of inconsistencies.
What about adding an option like --retry-sync N to portage to let it retry sync N times if this error occurs? I'm doing nightly emerge @world on a few hosts in CI and this often causes a failure.
I've been having a problem rsyncing my Dom0 to a VM I use as a mirror. I have had not problems using rsync with other machines and VM, yet I continued to be be plagued with the following error message: ... xfce-extra/xfce4-panel-profiles/Manifest xfce-extra/xfce4-panel-profiles/xfce4-panel-profiles-1.0.13.ebuild Number of files: 145,292 (reg: 118,233, dir: 27,059) Number of created files: 1,959 (reg: 1,903, dir: 56) Number of deleted files: 2,026 (reg: 1,980, dir: 46) Number of regular files transferred: 22,396 Total file size: 186.67M bytes Total transferred file size: 56.21M bytes Literal data: 56.21M bytes Matched data: 0 bytes File list size: 2.18M File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 455.48K Total bytes received: 32.41M sent 455.48K bytes received 32.41M bytes 305.70K bytes/sec total size is 186.67M speedup is 5.68 * Manifest timestamp: 2023-02-21 00:39:46 UTC * Valid OpenPGP signature found: * - primary key: DCD05B71EAB94199527F44ACDB6B8C1F96D8BF6D * - subkey: E1D6ABB63BFCFB4BA02FDF1CEC590EEAC9189250 * - timestamp: 2023-02-21 00:39:46 UTC * Verifying /var/db/repos/gentoo/.tmp-unverified-download-quarantine ...!!! Manifest verification failed: Manifest mismatch for gentoo-20230220.tar.xz.gpgsig __exists__: expected: False, have: True * IMPORTANT: 2 news items need reading for repository 'gentoo'. * Use eselect news read to view new items. Action: sync for repo: gentoo, returned code = 1 snucDom0 /etc/portage # There is a suggestion above to switch to git. Unfortunately, I am unable to find documentation on how to set up a local mirror using git. The current Gentoo Wiki is rsync-specific: https://wiki.gentoo.org/wiki/Local_Mirror. I have no druthers about whether I use git or rsync, but I do have many machines that currently use rsync successfully, but for this one: snucDom0. So if someone can suggest an instruction page on how to set up a git-based mirror, I be please to try such and reconfigure my Gentoo landscape. Otherwise, I'd like to troubleshoot and fix the current problem with rsync and wonder what I can provide that might give a clue as to what is causing the problem.
(In reply to John L. Poole from comment #13) > I've been having a problem rsyncing my Dom0 to a VM I use as a mirror. I > have had not problems using rsync with other machines and VM, yet I > continued to be be plagued with the following error message: > > ... > xfce-extra/xfce4-panel-profiles/Manifest > xfce-extra/xfce4-panel-profiles/xfce4-panel-profiles-1.0.13.ebuild > > Number of files: 145,292 (reg: 118,233, dir: 27,059) > Number of created files: 1,959 (reg: 1,903, dir: 56) > Number of deleted files: 2,026 (reg: 1,980, dir: 46) > Number of regular files transferred: 22,396 > Total file size: 186.67M bytes > Total transferred file size: 56.21M bytes > Literal data: 56.21M bytes > Matched data: 0 bytes > File list size: 2.18M > File list generation time: 0.001 seconds > File list transfer time: 0.000 seconds > Total bytes sent: 455.48K > Total bytes received: 32.41M > > sent 455.48K bytes received 32.41M bytes 305.70K bytes/sec > total size is 186.67M speedup is 5.68 > * Manifest timestamp: 2023-02-21 00:39:46 UTC > * Valid OpenPGP signature found: > * - primary key: DCD05B71EAB94199527F44ACDB6B8C1F96D8BF6D > * - subkey: E1D6ABB63BFCFB4BA02FDF1CEC590EEAC9189250 > * - timestamp: 2023-02-21 00:39:46 UTC > * Verifying /var/db/repos/gentoo/.tmp-unverified-download-quarantine > ...!!! Manifest verification failed: > Manifest mismatch for gentoo-20230220.tar.xz.gpgsig > __exists__: expected: False, have: True > > * IMPORTANT: 2 news items need reading for repository 'gentoo'. > * Use eselect news read to view new items. > > > Action: sync for repo: gentoo, returned code = 1 > > > snucDom0 /etc/portage # > > There is a suggestion above to switch to git. Unfortunately, I am unable to > find documentation on how to set up a local mirror using git. The current > Gentoo Wiki is rsync-specific: https://wiki.gentoo.org/wiki/Local_Mirror. I > have no druthers about whether I use git or rsync, but I do have many > machines that currently use rsync successfully, but for this one: snucDom0. > So if someone can suggest an instruction page on how to set up a git-based > mirror, I be please to try such and reconfigure my Gentoo landscape. > Otherwise, I'd like to troubleshoot and fix the current problem with rsync > and wonder what I can provide that might give a clue as to what is causing > the problem. I just got the rsync to work. Simply rem'ing statements was not enough, I tried using the opposite, e.g "0" when "1", and "no" when "yes". This worked for me: from my /etc/portage/repos.d/gentoo.conf: # # 2/7/23 jlpoole rem'd out four "verifys" below since eix-sync # failed. # #sync-rsync-verify-jobs = 1 # # 2/21/23 jlpoole: setting to "0" caused the sync to work in conjunction with sync-rsync-verify-metamanifest = yes # sync-rsync-verify-jobs = 0 #sync-rsync-verify-metamanifest = yes # # 2/21/23 jlpoole: setting to "no" caused synd to work in conjunction with sync-rsync-verify-jobs=0 above. # sync-rsync-verify-metamanifest = no
That's not really a fix.
Can't we kinda simulate atomic rsync repos by having a policy of only updating them every hour-ish? Ie. giving us clients about an hour window to do the sync? Maybe have a global policy to update on the hour, or have some way of clients to know when the repo was last updated, to schedule the local emerge --sync's better?
