Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 673496 - Portage full tree verification on community mirrors
Summary: Portage full tree verification on community mirrors
Status: UNCONFIRMED
Alias: None
Product: Mirrors
Classification: Unclassified
Component: Feature Request (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Mirror Admins
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-12-20 21:25 UTC by Fredrik Eriksson
Modified: 2023-07-06 07:55 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
example script that verifies tree before serving it (rsync-gentoo-portage.sh,2.20 KB, application/x-shellscript)
2018-12-20 21:27 UTC, Fredrik Eriksson
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Fredrik Eriksson 2018-12-20 21:25:41 UTC
I run the source mirror mirror.mdfnet.se. This server also has a public rsync mirror for the portage tree. After portage started to verify the tree after each sync (an awesome and much needed feature btw) my daily syncs against the mirror started failing a little now and then with messages like:

Manifest mismatch for metadata/Manifest.gz
  __size__: expected: 2148, have: 2147

I figured it probably had to do with bad verification of the tree on my mirror and/or that the mirror was halfway through syncing itself. 

I checked the documentation at https://wiki.gentoo.org/wiki/Project:Infrastructure/Mirrors/Rsync it looks like no tree verification is done even on the official rsync community mirrors. In my opinion a mirror should only serve data if it has verified that the entire tree is intact. A sync against a half-synced, or broken, mirror will just waste time and resources as the user will have to re-sync the tree a second time.

I created a heavy modified version of the sync-script which:
  * uses eix-sync to sync and verify against upstream (retries once if first sync fails)
  * does a local rsync to a non-active portage tree
  * updates a symlink to point to the newly synced portage tree
  * at next re-sync it syncs to the previous active tree, so it basically always has two ports trees: one active, and one inactive

I'm not proposing that this should be used as-is as it is just an ugly hack that can only run on gentoo machines (as it uses eix-sync) and is extremely untested, but I do this we should do *something* to improve the quality of the mirrors.

Reproducible: Sometimes

Steps to Reproduce:
1. sync against a community mirror

Actual Results:  
verification fails more often than reasonable

Expected Results:  
verification should pass
Comment 1 Fredrik Eriksson 2018-12-20 21:27:28 UTC
Created attachment 558234 [details]
example script that verifies tree before serving it