Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 557516 - Git gentoo-mirror breaks `emerge --sync` via rsync
Summary: Git gentoo-mirror breaks `emerge --sync` via rsync
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Core system (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL: https://github.com/gentoo-mirror/gent...
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-08-13 22:11 UTC by Anthony Basile
Modified: 2015-11-29 08:14 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Anthony Basile gentoo-dev 2015-08-13 22:11:13 UTC
The git mirror for sync-ing the portage tree with git at the above URL does not provide metadata/timestamp.chk.  While this is fine for the machine doing `emerge --sync`, if that box further exports the tree via rsyncd, `emerge --sync` fails on boxes trying to rsync off of it.  I hit this at home where I have one central server serving out the portage tree with rsyncd.  When I switch it to sync-ing by git, all hell broke loose on my dozens of vms that rsync off of it.  I could use git-daemon and switch them all, but ... ouch!

Here's the failing message:

# emerge --sync
>>> Syncing repository 'gentoo' into '/usr/portage'...
>>> Starting rsync with rsync://192.168.3.1/portage...
>>> Checking server timestamp ...
receiving incremental file list
rsync: link_stat "/metadata/timestamp.chk" (in portage) failed: No such file or directory (2)

As a workaround, manually creating metadata/timestamp.chk fixes things.  We should fix this in the emerge --sync code.  Its easy enough but I don't know why that timestamp is there so I'll let the portage team decide what to do with it.

Reproducible: Always
Comment 1 Anthony Basile gentoo-dev 2015-08-13 22:19:03 UTC
Actually, its even worse.  So sync-ing the first time works if you add the timestamp, but after that emerge --sync via rsync fails with

 # emerge --sync
>>> Syncing repository 'gentoo' into '/usr/portage'...
!!! /usr/portage appears to be under revision control (contains .git).
!!! Aborting rsync sync.


This is something of a mess.  I'm sure my situation isn't atypical.  This will force me to either 1) switch back to rsync on my central box or 2) switch over many vms to using git. to sync.
Comment 2 Brian Dolbec (RETIRED) gentoo-dev 2015-08-14 02:04:27 UTC
It looks like you need to add postsync hooks, on the serving one, one that creates the tiemstamp.  On the client machines one that removes the .git dir.

The alternative is to create an rsync tree on the server that does all those steps.

We'll have to see about configuring the .git/revision control check in the rsync module.
Comment 3 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2015-08-14 21:38:10 UTC
I'm going to look into adding metadata/timestamp.chk to mirrors. However, the problem is that it should be updated only when something actually changes in the repository, and pkgcore doesn't have any functionality to do that. So I'll probably have to find a good place for it in my scripts, and figure out when stuff actually changes.

(In reply to Brian Dolbec from comment #2)
> It looks like you need to add postsync hooks, on the serving one, one that
> creates the tiemstamp.  On the client machines one that removes the .git dir.

How about... --exclude option of rsync?
Comment 4 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2015-08-15 13:30:35 UTC
metadata/timestamp.chk now exists in all gentoo-mirror/ repositories (unless I failed at one of them :)). It should also be updated precisely when something changes (either in repo or in caches).
Comment 5 Matthias Maier gentoo-dev 2015-11-29 07:26:40 UTC
*** Bug 567074 has been marked as a duplicate of this bug. ***
Comment 6 Matthias Maier gentoo-dev 2015-11-29 07:29:45 UTC
Holy cow, it's late. Sorry for the noise.