Hello, I have noticed that from few last versions of portage when I do emerge --sync and metadata chace update start automatically it is very slow, it used to be much faster. I deleted: /usr/portage/metadata and /var/cache/edb/dep/* then I did emerge --sync and it's metadata update was fast as before. But today I did again emerge --sync but it is slower again. Any idea what is wrong?
emerge --info Portage 2.1.2_pre3-r7 (default-linux/x86/2006.0, gcc-4.1.1, glibc-2.5-r0, 2.6.18-ck1-r1 i686) ================================================================= System uname: 2.6.18-ck1-r1 i686 Intel(R) Pentium(R) M processor 1.80GHz Gentoo Base System version 1.12.5 Last Sync: Tue, 24 Oct 2006 17:20:01 +0000 ccache version 2.4 [disabled] app-admin/eselect-compiler: [Not Present] dev-java/java-config: 1.3.7, 2.0.30 dev-lang/python: 2.4.3-r4 dev-python/pycrypto: 2.0.1-r5 dev-util/ccache: 2.4-r6 dev-util/confcache: 0.4.2-r1 sys-apps/sandbox: 1.2.18.1 sys-devel/autoconf: 2.13, 2.60 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10 sys-devel/binutils: 2.17 sys-devel/gcc-config: 1.3.14 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.17-r1 ACCEPT_KEYWORDS="x86 ~x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-O3 -march=pentium-m -pipe -fomit-frame-pointer -ftracer -fprefetch-loop-arrays" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="${SVCVARDIR} /etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/X11/xkb /usr/share/config" CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/gconf /etc/java-config/vms/ /etc/revdep-rebuild /etc/splash /etc/terminfo" CXXFLAGS="-O3 -march=pentium-m -pipe -fomit-frame-pointer -ftracer -fprefetch-loop-arrays -fvisibility-inlines-hidden" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="--alphabetical" FEATURES="autoconfig distlocks metadata-transfer parallel-fetch sandbox sfperms strict userpriv usersandbox" GENTOO_MIRRORS="http://gentoo.itdnet.net/gentoo http://www.ibiblio.org/pub/Linux/distributions/gentoo http://gentoo.osuosl.org" LDFLAGS="-Wl,-O1 -Wl,--sort-common -s" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://gentoo.ITDNet.net/gentoo-portage/" USE="x86 X acl acpi alsa apache2 apm berkdb bitmap-fonts bzip2 cli cracklib crypt cups dga dlloader dri elibc_glibc emboss encode esd foomaticdb gdbm gif glibc-omitfp gnutls gpm gtk gtk2 imlib input_devices_keyboard input_devices_mouse input_devices_synaptics isdnlog ithreads jpeg kde kernel_linux ldap libg++ libwww mad mikmod mmx motif mp3 mpeg mppc mppe mysql ncurses nls nptl nptlonly ogg opengl oss pam pcre perl png postgres pppd python qt qt3 qt4 quicktime readline reflection rtc sdl session spell spl sse sse2 ssl tcpd tiff truetype truetype-fonts type1-fonts udev unicode userland_GNU userlocales video_cards_apm video_cards_ati video_cards_fbdev video_cards_fglrx video_cards_radeon video_cards_vesa vorbis xinetd xml xorg xv zlib" Unset: CTARGET, INSTALL_MASK, LANG, LC_ALL, LINGUAS, PORTAGE_RSYNC_EXTRA_OPTS
Do you have anything in /etc/portage/modules?
wow, i fail at life
(In reply to comment #2) > Do you have anything in /etc/portage/modules? > ll /etc/portage/modules ls: cannot access /etc/portage/modules: No such file or directory
There's not really any reason in the portage code that should affect the speed of the metadata update. Generally, available memory and IO load are the only things that should really affect it significantly. Assuming that you can't reduce the load on your system, another option would be to try the metadata_overlay module that's documented in `man portage`.
(In reply to comment #5) > There's not really any reason in the portage code that should affect the speed > of the metadata update. Generally, available memory and IO load are the only > things that should really affect it significantly. > > Assuming that you can't reduce the load on your system, another option would be > to try the metadata_overlay module that's documented in `man portage`. > Hi, Thank you for the answer. I agree with you, but memory, cpu and io usesage is almost the same. I told you, when I did some clean up it is much faster... the speed of metadata-update used to be always almost the same. I'll check what will be with that module, bue I have never used it before. So there should be some reason in my case.
(In reply to comment #6) > I'll check what will be with that module, bue I have never used it before. So > there should be some reason in my case. If you can demonstrate reproducible changes in --metadata runs that differ between portage-2.1.1 and portage-2.1.2, then I'd like to see it. There haven't been any changes to the code that should cause any noticeable difference though, so I'd be surprised.
Hello, I'm really sorry but I have never mentioned from which portage release it happened. And I'm afraid that I did not remember exactly ;-( but do you think it is normal when do: rm /usr/portage/metadata/* rm /var/cache/edb/dep/* and after that emerge --metadata or emerge --sync to me few times faster?
emerge --metadata won't work anymore if you remove /usr/portage/metadata/cache. The way that it works is to transfer cache from /usr/portage/metadata/cache/ to /var/cache/edb/dep/${PORTDIR}. For your tests, you'll probably want to switch bach and forth between at least 2 different snapshots of the portage tree and time --metadata runs for the switch several times with each version of portage. Sincerely, I doubt that you'll fine any measurable difference between 2.1.1 and 2.1.2.
(In reply to comment #9) > emerge --metadata won't work anymore if you remove /usr/portage/metadata/cache. > The way that it works is to transfer cache from /c > to /var/cache/edb/dep/${PORTDIR}. For your tests, you'll probably want to > switch bach and forth between at least 2 different snapshots of the portage > tree and time --metadata runs for the switch several times with each version of > portage. Sincerely, I doubt that you'll fine any measurable difference between > 2.1.1 and 2.1.2. > Hello, actually you are right, there is no diff between 2.1.1 and 2.1.2, as I said you now it is slower that it was 1,2 or 3 months ago. Actually when i delete /usr/portage/metadata/cache I do emerge --sync again an metagade update is really faster than emerge --sync without to delete /usr/portage/metadata/cache. Do you have any idea?
(In reply to comment #10) > Actually when i delete /usr/portage/metadata/cache I do emerge --sync again > an metagade update is really faster than emerge --sync without to delete > /usr/portage/metadata/cache. Do you have any idea? There's nothing I would expect to cause that behavior in the portage code. If you want to get some more information about the metadata update, you can try a tool that I use: http://dev.gentoo.org/~zmedico/portage/branches/2.1/bin/cache-tools.py If you add FEATURES="-metadata-transfer" to make.conf, then the metadata update won't happen automatically after sync. You can then use the above cache-tools script to perform the update. I suggest something like this: cache-tools.py --transfer --reportfile=- After the transfer is complete, it will print information about the number of ccache entries that were updated and the number that were removed.
(In reply to comment #11) > (In reply to comment #10) > > Actually when i delete /usr/portage/metadata/cache I do emerge --sync again > > an metagade update is really faster than emerge --sync without to delete > > /usr/portage/metadata/cache. Do you have any idea? > > There's nothing I would expect to cause that behavior in the portage code. If > you want to get some more information about the metadata update, you can try a > tool that I use: > > http://dev.gentoo.org/~zmedico/portage/branches/2.1/bin/cache-tools.py > > If you add FEATURES="-metadata-transfer" to make.conf, then the metadata update > won't happen automatically after sync. You can then use the above cache-tools > script to perform the update. I suggest something like this: > > cache-tools.py --transfer --reportfile=- > > After the transfer is complete, it will print information about the number of > ccache entries that were updated and the number that were removed. > Hello, I hope I'll be more helpful now, if you get the portage version which is included in http://gentoo.osuosl.org/releases/x86/2006.0/stages/stage3-i686-2006.0.tar.bz2 you will see what I mean, it is much faster in medata-update than last version. What does the medatataupdate do? is it just a copy or more?
(In reply to comment #12) > I hope I'll be more helpful now, if you get the portage version which is > included in > http://gentoo.osuosl.org/releases/x86/2006.0/stages/stage3-i686-2006.0.tar.bz2 > you will see what I mean, it is much faster in medata-update than last version. The 2006.0 stages have portage-2.0.x which uses a different cache format than >=portage-2.1 does. Most people report dramatic improvements in metadata updates after they've switched to >=portage-2.1, so your report is the opposite of most people have observed. > What does the medatataupdate do? is it just a copy or more? It validates and copies new metadata cache, then it removes any stale metadata that may exist.
(In reply to comment #13) > (In reply to comment #12) > > I hope I'll be more helpful now, if you get the portage version which is > > included in > > http://gentoo.osuosl.org/releases/x86/2006.0/stages/stage3-i686-2006.0.tar.bz2 > > you will see what I mean, it is much faster in medata-update than last version. > > The 2006.0 stages have portage-2.0.x which uses a different cache format than > >=portage-2.1 does. Most people report dramatic improvements in metadata > updates after they've switched to >=portage-2.1, so your report is the opposite > of most people have observed. > > > What does the medatataupdate do? is it just a copy or more? > > It validates and copies new metadata cache, then it removes any stale metadata > that may exist. > Hello, Sorry my mistake, you are absolutely right portage 2.1 is much faster than 2.0, actually I installed form stage 2006.0 in April or early May and immediately upgrade my portage to the last version it was 2.1 pre releases, so the metadata update was really fast then. If you have an idea what was the version of portage at early May check it out. I was trying to reproduce what I did that's why I misguided you, sorry for that.
