Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 554084 - unpack: improve speed by reducing number of chmods to perform using find -not -perm
Summary: unpack: improve speed by reducing number of chmods to perform using find -not...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - Ebuild Support (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on:
Blocks: 835380
  Show dependency tree
 
Reported: 2015-07-06 15:55 UTC by Michael Haubenwallner (RETIRED)
Modified: 2023-12-24 17:38 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
have gcc_quick_unpack do the unpack even more quickly (toolchain-quicker_unpack.diff,2.02 KB, patch)
2015-07-06 15:55 UTC, Michael Haubenwallner (RETIRED)
Details | Diff
unpack(): do chmod when needed only (0001-unpack-improve-chmod-speed-bug-554084.patch,839 bytes, patch)
2015-07-07 09:08 UTC, Michael Haubenwallner (RETIRED)
Details | Diff
unpack(): do chmod when needed only (0001-unpack-avoid-useless-chmods-to-improve-speed.patch,900 bytes, patch)
2015-07-07 10:44 UTC, Michael Haubenwallner (RETIRED)
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Haubenwallner (RETIRED) gentoo-dev 2015-07-06 15:55:45 UTC
Created attachment 406252 [details, diff]
have gcc_quick_unpack do the unpack even more quickly

Since unpack() does "chmod -R $WORKDIR/*" on each invocation, gcc_quick_unpack() may slow down on various operating-/file-systems, depending on the number of tarballs to unpack.

Instead, gcc_quick_unpack() better should invoke unpack() only once.

Thanks!
Comment 1 SpanKY gentoo-dev 2015-07-07 03:34:23 UTC
i think you missed that not all tarballs are unpacked into $WORKDIR

this should be mitigated in portage itself by using a `find -exec chmod` so it only operates on paths that need to be updated.  it still would walk all the paths, but it would be faster than `chmod -R`.
Comment 2 Michael Haubenwallner (RETIRED) gentoo-dev 2015-07-07 09:08:00 UTC
Created attachment 406294 [details, diff]
unpack(): do chmod when needed only

(In reply to SpanKY from comment #1)

Indeed, D

Some timings collected on x86_64-cygwin (Windows Server 2012r2):
>>> Unpacking gcc-4.9.3.tar.bz2 to ...
>>> Unpacking gcc-4.9.3-patches-1.0.tar.bz2 ...
>>> Unpacking gcc-4.9.3-uclibc-patches-1.0.tar.bz2 to ...
>>> Unpacking gcc-4.9.3-piepatches-v0.6.2.tar.bz2 to ...
>>> Unpacking gcc-4.4.3-specs-0.2.0.tar.bz2 to ...

multiple unpack, chmod -R   : ~6m  5s
  single unpack, chmod -R   : ~3m 50s
multiple unpack, find !-perm: ~3m 20s
  single unpack, find !-perm: ~3m  5s
Comment 3 SpanKY gentoo-dev 2015-07-07 09:39:59 UTC
Comment on attachment 406294 [details, diff]
unpack(): do chmod when needed only

looks nice.  would be good to change to -exec chmod ... + too.

 find . -mindepth 1 '!' -type l '!' -perm /a+rX,u+w,g-w,o-w \
     -exec chmod -f a+rX,u+w,g-w,o-w +
Comment 4 Michael Haubenwallner (RETIRED) gentoo-dev 2015-07-07 10:44:57 UTC
Created attachment 406306 [details, diff]
unpack(): do chmod when needed only
Comment 5 Brian Dolbec (RETIRED) gentoo-dev 2015-09-22 01:22:06 UTC
Released in portage-2.2.21
Comment 6 SpanKY gentoo-dev 2015-10-01 20:23:25 UTC
let's re-open since the change has been reverted
Comment 7 Zac Medico gentoo-dev 2015-10-01 20:38:46 UTC
As noted in bug 561368, comment #8, the find command fails as follows:

$ find . -mindepth 1 '!' -type l '!' -perm /a+rX,u+w,g-w,o-w
find: `./mod_auth_token': Permission denied

Maybe we should have an internal helper script, written in python, to check permissions with os.lstat and call os.chmod when needed.
Comment 8 Zac Medico gentoo-dev 2015-10-02 19:15:51 UTC
There's a patch in the following branch:

https://github.com/zmedico/portage/tree/bug_554084

You can test it like this:

echo '=sys-apps/portage-9999 **' >> /etc/portage/package.accept_keywords
portage_LIVE_BRANCH=bug_554084 \
portage_LIVE_REPO=https://github.com/zmedico/portage.git \
emerge -1 =sys-apps/portage-9999

I've posted it for review here:

https://archives.gentoo.org/gentoo-portage-dev/message/79c5e7cee2369123836d342ea78fed86
Comment 10 Zac Medico gentoo-dev 2016-03-14 03:02:39 UTC
Fixed in 2.2.23, including this related patch: 

https://gitweb.gentoo.org/proj/portage.git/commit/?id=597987aac1677e132b80ed2404697acf0188af7a