Bug 54150 – A way to make catalyst a bit faster?

• Product: Gentoo Hosted Projects
• Status: RESOLVED INVALID

[Description Marc Doughty 2004-06-16 18:11:07 UTC]

I think I have an idea to speed up snapshot creation in catalyst. Right now we're making a copy of /usr/portage and excluding a few directories. I think that if you  made a file from a list of /usr/portage, snipped the list items that would be excluded, and tarred items on the list you'd have a 'zero-copy' snapshotting system. Also, I haven't benchmarked it, but I think that if you reduce the 'blocking factor' in tar (defaults to 10KB) you could speed up the snapshotting process (less 'null-padding' for tar to inject and bzip2 to compress).

Reproducible: Always
Steps to Reproduce:
1. 'catalyst -f specs/snap.spec'
2. wait for a full copy of $PORTDIR
3. wait for the copy to get tbz2'd

Actual Results:  
I had time to smoke a cigarette.

Expected Results:  
I would have only enough time to pour another drink.

I've been thinking about a visual tool for catalyst, it could have a lot of
potential. Right now I've got a desk-full of flowcharts for making uber-enhanced
stages. Catalyst has a distinctly 'workflow' feel, and it would be nice to have
a gentoo utility that let you handle all it's features

[Comment 1 stephane germain 2004-06-18 21:09:19 UTC]

add a switch too keep the working directory when you restart the creation of a livecd (or whatever) following a build error.

[Comment 2 John Davis (zhen) (RETIRED) 2004-08-02 17:00:49 UTC]

good idea, but i think that it will cause more problems than it is worth. for example, every time the package list is changed, the snapshot will have to be recreated.

the second comment is being worked on as we speak. I am messing around with autoresuming.

[Comment 3 Marc Doughty 2004-08-13 21:32:21 UTC]

Well I don't know how other folks use catalyst; I use it to build my own stage3 systems for testing stuff (mostly pre-stage1 stuff, like NPTL when it wasn't built-in) and whipping-up current boot-CDs.

I don't see how it would cause any problems, and it would save several minutes off each snapshotting.

Maybe I'm missing something, but when you 'snap' something, it packs up the portage tree 'the wrong way' for me right now, whereas if you do the kung-fu I described, it takes less time and resources, with the same result.

[Comment 4 John Davis (zhen) (RETIRED) 2004-08-14 06:27:41 UTC]

try 1.9.0 in portage (it is ~arch masked). i cleaned up the snapshotting process a bit. after the first time, the time it takes to snapshot should be much improved. i didn't use the method that you recommended (i just tweaked rsync), but it should help. let me know what you think.