Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 333701 - commit hooks/portage to rsync conversion scripts
Summary: commit hooks/portage to rsync conversion scripts
Status: CONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Unspecified (show other bugs)
Hardware: All Linux
: High normal with 2 votes (vote)
Assignee: Brian Harring (RETIRED)
URL:
Whiteboard:
Keywords:
Depends on: 436918
Blocks: 333531
  Show dependency tree
 
Reported: 2010-08-20 20:14 UTC by Thilo Bangert (RETIRED) (RETIRED)
Modified: 2018-08-31 16:07 UTC (History)
11 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Thilo Bangert (RETIRED) (RETIRED) gentoo-dev 2010-08-20 20:14:09 UTC
The hook scripts currently employed on the cvs server need to be ported to work with git and the git workflow.

Also the current cvs -> rsync conversion script needs to be ported to a git -> rsync conversion script.
Comment 1 Johannes Huber (RETIRED) gentoo-dev 2011-12-12 11:08:14 UTC
Whats the state of this task?
Comment 2 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2012-02-06 16:27:58 UTC
I originally volunteered to do this, months/years ago. To be honest, I probably have the most recent exp on the infra team to manage the rsync boxes. BUT, there is nothing to do at the moment with regards to this bug because there exists other tasks that need to be done before we even think about this task.
Comment 3 Richard Freeman gentoo-dev 2013-01-02 23:14:51 UTC
(In reply to comment #2)
> I originally volunteered to do this, months/years ago. To be honest, I
> probably have the most recent exp on the infra team to manage the rsync
> boxes. BUT, there is nothing to do at the moment with regards to this bug
> because there exists other tasks that need to be done before we even think
> about this task.

Are there any specific tasks that this depends on?  If so we should mark those as blockers to track the relationships.

We're actually pretty close on the actual conversion process.  I think we need to start pushing to work out the details on how we will use git and what that means for hooks/etc.
Comment 4 Christoph Junghans (RETIRED) gentoo-dev 2013-01-09 19:34:52 UTC
On a different thought, couldn't we use `git-cvsserver` to make the git repository look like CVS and trigger the original hook?
Comment 5 Peter Stuge 2013-06-20 11:33:42 UTC
(In reply to Jeremy Olexa (darkside) from comment #2)
> there is nothing to do at the moment with regards to this bug
> because there exists other tasks that need to be done before
> we even think about this task.

That reasoning doesn't make any sense to me. If someone wants to work on this, let them. If they do a good job, then there is one thing less to do before portage can be converted.

(In reply to Thilo Bangert from comment #1)
> The hook scripts currently employed on the cvs server

Where are those scripts available for looking at?

I hope they aren't accessible only for infra, that would be a sure way to exclude any potential contributors who might help get things done.

Thanks!
Comment 6 Brian Harring (RETIRED) gentoo-dev 2014-02-21 15:20:01 UTC
Jeremy- can you point me at the existing hook scripts we use?  I can sort the conversion for that if I have access to that code.  I'm assuming this is in reference to cvs hooks themselves, not the cvs -> rsync generation.

We still need to update the rsync snapshot generation- same thing, if I can get either access, or the scripts, I'll sort the rewrite (I have time right now to do so, hence sooner is better than later) ;)
Comment 7 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2014-02-22 02:29:00 UTC
What comment #2 should have said: "This bug is so low priority to the overall initiative that there shouldn't be anyone considering it a blocker, show me the git repo then we can talk" :)

Anyway, two things: 
- I've just now evaluated all the cvs pre/postcommit hooks in CVSROOT, and we are simply talking about the commit mailing lists / old CIA bot & XML checker for gentoo docs.
- The git->rsync issue is trivial. It is just moving files into one location, and "re-writing" it will take infrastructure team access/setup and an extra 10 minutes (ok, maybe another 10 minutes to verify). Here is the process that the script follows:

# 0) Make a backup incase there is something wrong with server side
# 1) Update rsync checkouts or exports from cvs.g.o::mastermirror-staging
# 1b) rsync EXPORTS/gentoo-x86 to STAGEDIR
# 1c) source (bash -n) eclasses to check for syntax errors
# 2) generate metadata (egencache)
# 3) place dtd info in STAGEDIR
# 4) place glsa's in STAGEDIR
# 5) place news in STAGEDIR
# 6) place herds.xml in STAGEDIR
# 7) rsync from STAGEDIR to FINALDIR

cvs is called zero times in that script and it should only be step 1 that needs modification.

I hope everyone now understands why I get tired of this bug and make perhaps wrongly interpreted comments! Can we get the git repo conversion done before "worrying" about this bug?
Comment 8 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2014-02-22 03:15:35 UTC
I've given Brian the in use pre/post hooks (pre: cvslogdate, checkxml.pl, checkgroup.pl. post: cvslogholder.pl, commitlog-doc.pl, commitlog-portage.pl). Also the cvs->rsync script alluded to in previous comment. I did not know there there was some manifest work done OUTSIDE the vcs. (critical disconnect in mindshare). So, while it will still take infrastructure team setup, Brian can see where this new manifest code plugs in. (I think in step 1d)

I don't want to make these files publically available at the moment. Please ask me/infra before sharing. Thanks.
Comment 9 Alexey Shvetsov archtester gentoo-dev 2014-03-19 11:47:11 UTC
Manifests seems like arent checked during conversion since sometime snapshots has br0ken manifests
Comment 10 Peter Stuge 2014-03-19 12:22:47 UTC
(In reply to Jeremy Olexa (darkside) from comment #7)
> - I've just now evaluated all the cvs pre/postcommit hooks in CVSROOT, and
> we are simply talking about the commit mailing lists / old CIA bot & XML
> checker for gentoo docs.

For mailing lists I simply use /usr/share/git/contrib/hooks/post-receive-email which is included in the git package. It works well.
Comment 11 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-09-14 08:39:41 UTC
I think we could go for two-stage model, with stage1 Funtoo-like and stage2 Gentoo-like.

That is, we have a public git repo for users to sync with. We pull changes from gx86, regen caches, update all other files, git add -A, git commit. So users can quickly and efficiently sync from git if they want to.

Then, optionally we can move files from this to rsync mirror, and generate ChangeLogs. Users can also use rsync if they want to.

I poorly understand why verification of data is done in rsync hook. IMO this belongs rather in main git repo's update hook, so we would prevent people from pushing obviously broken stuff.
Comment 12 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-09-14 10:50:29 UTC
Oh, one more thing occurred to me:

# 4) place glsa's in STAGEDIR
# 5) place news in STAGEDIR
# 6) place herds.xml in STAGEDIR

Doesn't it make sense to merge those three with the repo itself? I don't see a rationale to have them separate -- they reference packages in the repository, and are rather non-meaningful without it.