Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 97074 - etc-update calls portageq thrice, is very slow.
Summary: etc-update calls portageq thrice, is very slow.
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Core system (show other bugs)
Hardware: All All
: High enhancement (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on:
Blocks: 835380 136244
  Show dependency tree
 
Reported: 2005-06-25 14:59 UTC by Harald van Dijk (RETIRED)
Modified: 2023-05-20 07:44 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments
the patch (etc-update.patch,899 bytes, patch)
2005-06-25 15:04 UTC, Harald van Dijk (RETIRED)
Details | Diff
faster-etc-update (faster-etc-update,865 bytes, patch)
2005-12-10 11:31 UTC, Harald van Dijk (RETIRED)
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Harald van Dijk (RETIRED) gentoo-dev 2005-06-25 14:59:46 UTC
By wrapping etc-update's three portageq calls in a single python command, etc-update gets a huge speed boost.

# time echo -1 | ./etc-update >/dev/null

real    0m8.575s
user    0m6.675s
sys     0m1.899s
# time echo -1 | /usr/sbin/etc-update >/dev/null

real    0m21.541s
user    0m18.341s
sys     0m2.892s

Patch coming up...
Comment 1 Harald van Dijk (RETIRED) gentoo-dev 2005-06-25 15:04:55 UTC
Created attachment 61928 [details, diff]
the patch

It's safe to split by newlines here since CONFIG_PROTECT* and USERLAND can't
possibly contain any newlines themselves (CONFIG_PROTECT* are incremental, so
any whitespace will be changed to a single space, and USERLAND is set by
portage itself), but if you want to be safe in case etc-update will need more
variables later on, I can try to modify this to use '\0' as the separator
instead.
Comment 2 SpanKY gentoo-dev 2005-06-25 23:55:27 UTC
this would be even better i think:
eval $(python -c 'import portage
print "USERLAND=\""+portage.settings["USERLAND"]+"\";"
print "CONFIG_PROTECT=\""+portage.settings["CONFIG_PROTECT"]+"\";"
print "CONFIG_PROTECT_MASK=\""+portage.settings["CONFIG_PROTECT_MASK"]+"\";"
')

that way you dont have to worry about newlines or anything
Comment 3 Harald van Dijk (RETIRED) gentoo-dev 2005-06-26 02:12:11 UTC
That won't work correctly if any string contains a $, a \, a ", a `,
a newline or tab (they'll be replaced with spaces in your version,
which is not always correct); it's just as likely to work now, but
more likely to break in the future :)

But, it would be good, I think, if $(...) became "$(...)", if the
assignments became VAR='value', and if the values got a s/'/'\\''/g.
Comment 4 Jason Stubbs (RETIRED) gentoo-dev 2005-07-28 07:25:37 UTC
Putting a hold on feature requests for portage as they are drowning out the 
bugs. Most of these features should be available in the next major version of 
portage. But for the time being, they are just drowning out the major bugs and 
delaying the next version's progress. 
 
Any bugs that contain patches and any bugs for etc-update or dispatch-conf can 
be reopened. Sorry, I'm just not good enough with bugzilla. ;) 
Comment 5 Harald van Dijk (RETIRED) gentoo-dev 2005-07-28 11:18:45 UTC
> Any bugs that contain patches and any bugs for etc-update or dispatch-conf can 
be reopened.

Alrighty.
Comment 6 Harald van Dijk (RETIRED) gentoo-dev 2005-12-10 11:31:50 UTC
Created attachment 74448 [details, diff]
faster-etc-update

Updated patch for portage-2.0.53.
Comment 7 SpanKY gentoo-dev 2005-12-10 11:56:55 UTC
actually i just added support to portageq to handle multiple vars

so you could do `portageq envvar A B C D` and get all four back, each on a line
Comment 8 SpanKY gentoo-dev 2005-12-10 12:03:49 UTC
and actually, i just added support to trunk so you can do:
eval $(portageq envvar -v USERLAND CONFIG_PROTECT CONFIG_PROTECT_MASK)

$ portageq envvar -v ARCH PORTDIR USERLAND
ARCH='amd64'
PORTDIR='/usr/portage'
USERLAND='GNU'
Comment 9 Harald van Dijk (RETIRED) gentoo-dev 2005-12-10 12:31:27 UTC
> actually i just added support to portageq to handle multiple vars
> 
> so you could do `portageq envvar A B C D` and get all four back, each on a line

Is that a good idea? It has the same potential problem with newlines.

> and actually, i just added support to trunk so you can do:
> eval $(portageq envvar -v USERLAND CONFIG_PROTECT CONFIG_PROTECT_MASK)
> 
> $ portageq envvar -v ARCH PORTDIR USERLAND
> ARCH='amd64'
> PORTDIR='/usr/portage'
> USERLAND='GNU'

Great! Does this correctly change single quotes in variables too?
Comment 10 SpanKY gentoo-dev 2005-12-10 12:35:02 UTC
i dont see a problem ... portage strips out newlines from variables

> > $ portageq envvar -v ARCH PORTDIR USERLAND
> > ARCH='amd64'
> > PORTDIR='/usr/portage'
> > USERLAND='GNU'
> 
> Great! Does this correctly change single quotes in variables too?

no ... maybe it should print out " instead of '
Comment 11 Harald van Dijk (RETIRED) gentoo-dev 2005-12-10 12:36:51 UTC
(In reply to comment #10)
> > Great! Does this correctly change single quotes in variables too?
> 
> no ... maybe it should print out " instead of '

In that case, it should change double quotes, dollar signs, backticks and
backslashes in variables to be sure to get correct results. Probably easier to
stick with ' and change that.
Comment 12 SpanKY gentoo-dev 2005-12-10 12:43:03 UTC
> > no ... maybe it should print out " instead of '
> 
> In that case, it should change double quotes, dollar signs, backticks and
> backslashes in variables to be sure to get correct results.

right, which is why i picked ' instead of "

all in all, the quoting issue doesnt affect this bug as the variables we are
after should not have quotes in them
Comment 13 Harald van Dijk (RETIRED) gentoo-dev 2005-12-10 12:46:42 UTC
> i dont see a problem ... portage strips out newlines from variables

Sorry, overlooked this.

portage only strips out newlines from incremental variables. Try

$ mkdir $'/tmp/port\nage'
$ PORTAGE_TMPDIR=$'/tmp/port\nage' portageq envvar PORTAGE_TMPDIR

for example.

> right, which is why i picked ' instead of "
>
> all in all, the quoting issue doesnt affect this bug as the variables we are
> after should not have quotes in them

PORTAGE_TMPDIR can legitimately contain any character (think "/var/tmp/portage's
\`domain'" - sure, I wouldn't use it, but it's valid, isn't it?), and it's used
by etc-update.
Comment 14 Harald van Dijk (RETIRED) gentoo-dev 2005-12-10 12:53:41 UTC
Bleh, I just noticed etc-update already doesn't properly quote PORTAGE_TMPDIR...

> TMP="${PORTAGE_TMPDIR}/$$"
> rm -rf ${TMP} 2> /dev/null
> mkdir ${TMP} || die "failed mkdir command!" 1

So while I do personally consider it a bug, I guess it's not too big a deal if
it already doesn't work now.
Comment 15 SpanKY gentoo-dev 2005-12-10 14:03:52 UTC
> PORTAGE_TMPDIR can legitimately contain any character

yes, but atm, a hell of a lot more would break than just etc-update if there
were such things that required quoting, such as a significant number of ebuilds
which never quote $D or $S or $WORKDIR or ...
Comment 16 Marius Mauch (RETIRED) gentoo-dev 2006-07-19 04:00:54 UTC
in r3934
Comment 17 Zac Medico gentoo-dev 2006-07-20 14:04:00 UTC
This has been released in 2.1.1_pre3-r2.