Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 152681 - very slow metadata update after emerge --sync
Summary: very slow metadata update after emerge --sync
Status: RESOLVED NEEDINFO
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - Interface (emerge) (show other bugs)
Hardware: All Linux
: High normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2006-10-24 11:19 UTC by Radko Lazarov
Modified: 2006-11-06 23:13 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Radko Lazarov 2006-10-24 11:19:52 UTC
Hello,

I have noticed that from few last versions of portage when I do emerge --sync and 
metadata chace update start automatically it is very slow, it used to be much faster.

I deleted:
/usr/portage/metadata
and /var/cache/edb/dep/*

then I did emerge --sync and it's metadata update was fast as before.

But today I did again emerge --sync but it is slower again.
Any idea what is wrong?
Comment 1 Radko Lazarov 2006-10-24 11:20:08 UTC
emerge --info
Portage 2.1.2_pre3-r7 (default-linux/x86/2006.0, gcc-4.1.1, glibc-2.5-r0, 2.6.18-ck1-r1 i686)
=================================================================
System uname: 2.6.18-ck1-r1 i686 Intel(R) Pentium(R) M processor 1.80GHz
Gentoo Base System version 1.12.5
Last Sync: Tue, 24 Oct 2006 17:20:01 +0000
ccache version 2.4 [disabled]
app-admin/eselect-compiler: [Not Present]
dev-java/java-config: 1.3.7, 2.0.30
dev-lang/python:     2.4.3-r4
dev-python/pycrypto: 2.0.1-r5
dev-util/ccache:     2.4-r6
dev-util/confcache:  0.4.2-r1
sys-apps/sandbox:    1.2.18.1
sys-devel/autoconf:  2.13, 2.60
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10
sys-devel/binutils:  2.17
sys-devel/gcc-config: 1.3.14
sys-devel/libtool:   1.5.22
virtual/os-headers:  2.6.17-r1
ACCEPT_KEYWORDS="x86 ~x86"
AUTOCLEAN="yes"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O3 -march=pentium-m -pipe -fomit-frame-pointer -ftracer -fprefetch-loop-arrays"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="${SVCVARDIR} /etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/X11/xkb /usr/share/config"
CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/gconf /etc/java-config/vms/ /etc/revdep-rebuild /etc/splash /etc/terminfo"
CXXFLAGS="-O3 -march=pentium-m -pipe -fomit-frame-pointer -ftracer -fprefetch-loop-arrays -fvisibility-inlines-hidden"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--alphabetical"
FEATURES="autoconfig distlocks metadata-transfer parallel-fetch sandbox sfperms strict userpriv usersandbox"
GENTOO_MIRRORS="http://gentoo.itdnet.net/gentoo http://www.ibiblio.org/pub/Linux/distributions/gentoo http://gentoo.osuosl.org"
LDFLAGS="-Wl,-O1 -Wl,--sort-common -s"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://gentoo.ITDNet.net/gentoo-portage/"
USE="x86 X acl acpi alsa apache2 apm berkdb bitmap-fonts bzip2 cli cracklib crypt cups dga dlloader dri elibc_glibc emboss encode esd foomaticdb gdbm gif glibc-omitfp gnutls gpm gtk gtk2 imlib input_devices_keyboard input_devices_mouse input_devices_synaptics isdnlog ithreads jpeg kde kernel_linux ldap libg++ libwww mad mikmod mmx motif mp3 mpeg mppc mppe mysql ncurses nls nptl nptlonly ogg opengl oss pam pcre perl png postgres pppd python qt qt3 qt4 quicktime readline reflection rtc sdl session spell spl sse sse2 ssl tcpd tiff truetype truetype-fonts type1-fonts udev unicode userland_GNU userlocales video_cards_apm video_cards_ati video_cards_fbdev video_cards_fglrx video_cards_radeon video_cards_vesa vorbis xinetd xml xorg xv zlib"
Unset:  CTARGET, INSTALL_MASK, LANG, LC_ALL, LINGUAS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 2 Zac Medico gentoo-dev 2006-10-24 13:34:02 UTC
Do you have anything in /etc/portage/modules?
Comment 3 Simon Stelling (RETIRED) gentoo-dev 2006-10-24 15:23:57 UTC
wow, i fail at life
Comment 4 Radko Lazarov 2006-10-25 10:02:53 UTC
(In reply to comment #2)
> Do you have anything in /etc/portage/modules?
> 

ll /etc/portage/modules
ls: cannot access /etc/portage/modules: No such file or directory

Comment 5 Zac Medico gentoo-dev 2006-10-25 10:22:25 UTC
There's not really any reason in the portage code that should affect the speed of the metadata update.  Generally, available memory and IO load are the only things that should really affect it significantly.

Assuming that you can't reduce the load on your system, another option would be to try the metadata_overlay module that's documented in `man portage`.
Comment 6 Radko Lazarov 2006-10-28 06:00:44 UTC
(In reply to comment #5)
> There's not really any reason in the portage code that should affect the speed
> of the metadata update.  Generally, available memory and IO load are the only
> things that should really affect it significantly.
> 
> Assuming that you can't reduce the load on your system, another option would be
> to try the metadata_overlay module that's documented in `man portage`.
> 

Hi,
Thank you for the answer.
I agree with you, but memory, cpu and io usesage is almost the same.
I told you, when I did some clean up it is much faster... the speed of metadata-update used to be always almost the same.

I'll check what will be with that module, bue I have never used it before. So there should be some reason in my case.
Comment 7 Zac Medico gentoo-dev 2006-10-30 12:55:46 UTC
(In reply to comment #6)
> I'll check what will be with that module, bue I have never used it before. So
> there should be some reason in my case.

If you can demonstrate reproducible changes in --metadata runs that differ between portage-2.1.1 and portage-2.1.2, then I'd like to see it.  There haven't been any changes to the code that should cause any noticeable difference though, so I'd be surprised.
Comment 8 Radko Lazarov 2006-10-31 11:00:59 UTC
Hello,

I'm really sorry but I have never mentioned from which portage release it happened. And I'm afraid that I did not remember exactly ;-(

but do you think it is normal when do:
rm /usr/portage/metadata/*
rm /var/cache/edb/dep/*

and after that emerge --metadata 
or emerge --sync
to me few times faster?

Comment 9 Zac Medico gentoo-dev 2006-10-31 12:52:17 UTC
emerge --metadata won't work anymore if you remove /usr/portage/metadata/cache.  The way that it works is to transfer cache from /usr/portage/metadata/cache/ to /var/cache/edb/dep/${PORTDIR}.  For your tests, you'll probably want to switch bach and forth between at least 2 different snapshots of the portage tree and time --metadata runs for the switch several times with each version of portage.  Sincerely, I doubt that you'll fine any measurable difference between 2.1.1 and 2.1.2.
Comment 10 Radko Lazarov 2006-11-03 14:23:50 UTC
(In reply to comment #9)
> emerge --metadata won't work anymore if you remove /usr/portage/metadata/cache.
>  The way that it works is to transfer cache from /c
> to /var/cache/edb/dep/${PORTDIR}.  For your tests, you'll probably want to
> switch bach and forth between at least 2 different snapshots of the portage
> tree and time --metadata runs for the switch several times with each version of
> portage.  Sincerely, I doubt that you'll fine any measurable difference between
> 2.1.1 and 2.1.2.
> 

Hello,

actually you are right, there is no diff between 2.1.1 and 2.1.2, as I said you now it is slower that it was 1,2 or 3 months ago.
Actually when i delete /usr/portage/metadata/cache I do emerge --sync again
an metagade update is really faster than emerge --sync without to delete /usr/portage/metadata/cache. Do you have any idea?
Comment 11 Zac Medico gentoo-dev 2006-11-03 14:49:14 UTC
(In reply to comment #10)
> Actually when i delete /usr/portage/metadata/cache I do emerge --sync again
> an metagade update is really faster than emerge --sync without to delete
> /usr/portage/metadata/cache. Do you have any idea?

There's nothing I would expect to cause that behavior in the portage code.  If you want to get some more information about the metadata update, you can try a tool that I use:

http://dev.gentoo.org/~zmedico/portage/branches/2.1/bin/cache-tools.py

If you add FEATURES="-metadata-transfer" to make.conf, then the metadata update won't happen automatically after sync.  You can then use the above cache-tools script to perform the update.  I suggest something like this:

cache-tools.py --transfer --reportfile=-

After the transfer is complete, it will print information about the number of ccache entries that were updated and the number that were removed.
Comment 12 Radko Lazarov 2006-11-04 07:09:12 UTC
(In reply to comment #11)
> (In reply to comment #10)
> > Actually when i delete /usr/portage/metadata/cache I do emerge --sync again
> > an metagade update is really faster than emerge --sync without to delete
> > /usr/portage/metadata/cache. Do you have any idea?
> 
> There's nothing I would expect to cause that behavior in the portage code.  If
> you want to get some more information about the metadata update, you can try a
> tool that I use:
> 
> http://dev.gentoo.org/~zmedico/portage/branches/2.1/bin/cache-tools.py
> 
> If you add FEATURES="-metadata-transfer" to make.conf, then the metadata update
> won't happen automatically after sync.  You can then use the above cache-tools
> script to perform the update.  I suggest something like this:
> 
> cache-tools.py --transfer --reportfile=-
> 
> After the transfer is complete, it will print information about the number of
> ccache entries that were updated and the number that were removed.
> 

Hello,

I hope I'll be more helpful now, if you get the portage version which is included in http://gentoo.osuosl.org/releases/x86/2006.0/stages/stage3-i686-2006.0.tar.bz2  
you will see what I mean, it is much faster in medata-update than last version.

What does the medatataupdate do? is it just a copy or more?
Comment 13 Zac Medico gentoo-dev 2006-11-04 22:26:55 UTC
(In reply to comment #12)
> I hope I'll be more helpful now, if you get the portage version which is
> included in
> http://gentoo.osuosl.org/releases/x86/2006.0/stages/stage3-i686-2006.0.tar.bz2  
> you will see what I mean, it is much faster in medata-update than last version.

The 2006.0 stages have portage-2.0.x which uses a different cache format than >=portage-2.1 does.  Most people report dramatic improvements in metadata updates after they've switched to >=portage-2.1, so your report is the opposite of most people have observed.

> What does the medatataupdate do? is it just a copy or more?

It validates and copies new metadata cache, then it removes any stale metadata that may exist.
Comment 14 Radko Lazarov 2006-11-06 23:13:56 UTC
(In reply to comment #13)
> (In reply to comment #12)
> > I hope I'll be more helpful now, if you get the portage version which is
> > included in
> > http://gentoo.osuosl.org/releases/x86/2006.0/stages/stage3-i686-2006.0.tar.bz2  
> > you will see what I mean, it is much faster in medata-update than last version.
> 
> The 2006.0 stages have portage-2.0.x which uses a different cache format than
> >=portage-2.1 does.  Most people report dramatic improvements in metadata
> updates after they've switched to >=portage-2.1, so your report is the opposite
> of most people have observed.
> 
> > What does the medatataupdate do? is it just a copy or more?
> 
> It validates and copies new metadata cache, then it removes any stale metadata
> that may exist.
> 
Hello,

Sorry my mistake, you are absolutely right portage 2.1 is much faster than 2.0, actually I installed form stage 2006.0 in April or early May and immediately upgrade my portage to the last version it was 2.1 pre releases, so the metadata update was really fast then.
If you have an idea what was the version of portage at early May check it out.
I was trying to reproduce what I did that's why I misguided you, sorry for that.