Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 129178 - subversion.eclass export sources speedup
Summary: subversion.eclass export sources speedup
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Eclasses (show other bugs)
Hardware: All Linux
: High enhancement (vote)
Assignee: Akinori Hattori
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2006-04-07 14:06 UTC by Ryan Hill (RETIRED)
Modified: 2006-04-30 02:21 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments
subversion.eclass.patch (subversion.eclass.patch,903 bytes, patch)
2006-04-07 14:06 UTC, Ryan Hill (RETIRED)
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Ryan Hill (RETIRED) gentoo-dev 2006-04-07 14:06:27 UTC
Currently, subversion.eclass exports the sources from ${ESVN_STORE_DIR}/${ESVN_CO_DIR} to ${S} by using a 'cp -pPR' command (at least until a fix for bug #119236 is available).  This copies everything in the svn-src/project/checkout directory to ${S}, including all the .svn directories, which it then removes with a 'find | xargs' operation.

This is inefficient in many ways, and generates a lot of unnecessary disk I/O and wasted time.  With a large SVN codebase on a slow laptop harddrive, you can really feel it.

Attached is a patch to subversion.eclass that instead pipes the svn source directory through tar using --exclude='*.svn'.  For testing, i used gcc's gcc-4_1-branch as the codebase (631Mib).  I disabled the 'svn update' bit of the eclass to make sure that the codebase would be the same on all runs and not be affected by network variables.  I then ran the following command three times with the patched and unpatched eclass.

# time ebuild ~/overlay/sys-devel/gcc/gcc-4.1.0_pre20060328.ebuild unpack

[original eclass w/ cp -pPR]

real    6m7.787s
user    0m14.022s
sys	0m56.946s

real	6m24.479s
user	0m13.926s
sys	0m56.728s

real	6m30.469s
user	0m13.988s
sys	0m56.933s

[ patched eclass w/ tar ]

real	2m14.513s
user	0m5.821s
sys	0m14.654s

real	1m32.750s
user	0m5.511s
sys	0m13.109s

real	1m15.995s
user	0m5.445s
sys	0m13.144s

-----

i don't like how this patch calls mkdir, but i couldn't find a way around it.  any suggestions are welcome.
Comment 1 Ryan Hill (RETIRED) gentoo-dev 2006-04-07 14:06:58 UTC
Created attachment 84162 [details, diff]
subversion.eclass.patch
Comment 2 Akinori Hattori gentoo-dev 2006-04-28 07:21:13 UTC
Fixed in CVS. (using "rsync -C")
Thank you for reporting.
Comment 3 Christian Birchinger (RETIRED) gentoo-dev 2006-04-29 13:13:20 UTC
The "cp -pPR" replacment was just fine. But the new "rsync -aC" version gives
me lots of troubles. To be exact its the "-C" option of rsync.

I have some dirs from repos not being copied now. First it was when .cvsignore
files existed with stuff like "Makefile.in" inside. It didn't rsync those
really important Makefile.in files.

Now i have all .cvsignore files removed and it still doesn't copy whole dirs
for some weird reason.

Since its about subversion repos, is this problematic CVS option really needed?


For example: http://svn.atheme.org/audacious/trunk/Plugins/Input/adplug/core
the whole "core" dir doesn't get copied with the -C option of rsync.

Comment 4 Christian Birchinger (RETIRED) gentoo-dev 2006-04-29 13:14:53 UTC
Reopening because rsync -C misses some important files (see previous comment)
Comment 5 Ryan Hill (RETIRED) gentoo-dev 2006-04-30 02:17:18 UTC
using tar doesn't have this problem, and it's faster than rsync from what i can tell.
Comment 6 Akinori Hattori gentoo-dev 2006-04-30 02:21:19 UTC
Oops, I didn't know such behavior of rsync -C.
It was replaced to rsync --exclude=".svn".