Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 563664 - sys-apps/portage-2.2.20.1 and emerge --fetchonly binary packages download loop
Summary: sys-apps/portage-2.2.20.1 and emerge --fetchonly binary packages download loop
Status: UNCONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Binary packages support (show other bugs)
Hardware: AMD64 Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
: 563820 (view as bug list)
Depends on:
Blocks:
 
Reported: 2015-10-21 10:36 UTC by marco
Modified: 2015-10-26 16:09 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
packages sample file (packages,2.98 KB, text/plain)
2015-10-22 07:10 UTC, marco
Details
debug patch (binhost_debug.patch,973 bytes, patch)
2015-10-23 00:02 UTC, Zac Medico
Details | Diff
Extract of Packages from the Builder (Packages.builder,10.06 KB, text/plain)
2015-10-23 08:07 UTC, marco
Details
Extract of Packages from the server (Packages.server,15.26 KB, text/plain)
2015-10-23 08:08 UTC, marco
Details
emerge log after patch (debug.log,153.22 KB, text/x-log)
2015-10-23 08:22 UTC, marco
Details

Note You need to log in before you can comment on or make changes to this bug.
Description marco 2015-10-21 10:36:31 UTC
We use a profile with these options:
metadata/layout.conf
masters = gentoo
repo-name = nucleus
profile-formats = portage-2 profile-set build-id

I want to download all the binary packages before install, then install the binary packages without the delay downloading the binary packages.

after upgrading to sys-apps/portage-2.2.20.1 and switching to profile-set build-id in our profile-formats
every time i do:
emerge -vauNDf world or a single package 

the binary packages are downloaded correctly.
if i run another time emerge -vauNDf world or a single package
all the binary packages are downloaded again.

If i install the packages , they are redownloaded again.

example of portage/packages/sys-libs/ncurses/
-rw-r--r-- 1 root root 1329555 Sep 22 14:04 ncurses-5.9-r5-1.xpak
-rw-r--r-- 1 root root 1329555 Sep 22 14:04 ncurses-5.9-r5-2.xpak
-rw-r--r-- 1 root root 1329555 Sep 22 14:04 ncurses-5.9-r5-3.xpak
-rw-r--r-- 1 root root 1329555 Sep 22 14:04 ncurses-5.9-r5-4.xpak
-rw-r--r-- 1 root root 1329555 Sep 22 14:04 ncurses-5.9-r5-5.xpak
-rw-r--r-- 1 root root 1329555 Sep 22 14:04 ncurses-5.9-r5-6.xpak
-rw-r--r-- 1 root root   15928 Sep 22 14:04 ncurses-5.9-r99-1.xpak
-rw-r--r-- 1 root root   15928 Sep 22 14:04 ncurses-5.9-r99-2.xpak
-rw-r--r-- 1 root root   15928 Sep 22 14:04 ncurses-5.9-r99-3.xpak
-rw-r--r-- 1 root root   15928 Sep 22 14:04 ncurses-5.9-r99-4.xpak
-rw-r--r-- 1 root root   15928 Sep 22 14:04 ncurses-5.9-r99-5.xpak
-rw-r--r-- 1 root root   15928 Sep 22 14:04 ncurses-5.9-r99-6.xpak

md5s are the same

af394baf8a2ee590bac265cc9c97895f  ncurses-5.9-r5-1.xpak
af394baf8a2ee590bac265cc9c97895f  ncurses-5.9-r5-2.xpak
af394baf8a2ee590bac265cc9c97895f  ncurses-5.9-r5-3.xpak
af394baf8a2ee590bac265cc9c97895f  ncurses-5.9-r5-4.xpak
af394baf8a2ee590bac265cc9c97895f  ncurses-5.9-r5-5.xpak
af394baf8a2ee590bac265cc9c97895f  ncurses-5.9-r5-6.xpak
03034a55ca52bcccb6e2eb099c368205  ncurses-5.9-r99-1.xpak
03034a55ca52bcccb6e2eb099c368205  ncurses-5.9-r99-2.xpak
03034a55ca52bcccb6e2eb099c368205  ncurses-5.9-r99-3.xpak
03034a55ca52bcccb6e2eb099c368205  ncurses-5.9-r99-4.xpak
03034a55ca52bcccb6e2eb099c368205  ncurses-5.9-r99-5.xpak
03034a55ca52bcccb6e2eb099c368205  ncurses-5.9-r99-6.xpak


why the binary packages are downloaded every time ?

Reproducible: Always



Expected Results:  
The binary packages must be downloaded only one time .
Comment 1 Zac Medico gentoo-dev 2015-10-22 05:34:33 UTC
The binary packages are compared using a tuple generated by the _instance_key_multi_instance method here:

https://gitweb.gentoo.org/proj/portage.git/tree/pym/portage/dbapi/virtual.py?h=v2.2.20.1#n55

Can you attach some of the duplicate entries from you $PKGDIR/Packages file? That will probably show us why _instance_key_multi_instance is returning a different result. If the package mtime changes or is not preserved for some reason, then that could trigger it.
Comment 2 marco 2015-10-22 07:10:53 UTC
Created attachment 415140 [details]
packages sample file
Comment 3 marco 2015-10-22 07:12:07 UTC
Packages file added
Comment 4 Zac Medico gentoo-dev 2015-10-22 16:07:44 UTC
(In reply to marco from comment #2)
> Created attachment 415140 [details]
> packages sample file

All of those entries appear to be identical. We should look at the server-side entry for the same package. Is it identical to these entries?
Comment 5 Zac Medico gentoo-dev 2015-10-23 00:02:46 UTC
Created attachment 415214 [details, diff]
debug patch

This patch will make emerge output some useful debugging information to stderr. Sample usage:

    emerge -pvg ncurses > debug.log 2>&1
Comment 6 marco 2015-10-23 08:07:24 UTC
Created attachment 415222 [details]
Extract of Packages from the Builder
Comment 7 marco 2015-10-23 08:08:05 UTC
Created attachment 415224 [details]
Extract of Packages from the server
Comment 8 marco 2015-10-23 08:22:05 UTC
Created attachment 415226 [details]
emerge log after patch
Comment 9 marco 2015-10-23 08:22:29 UTC
(In reply to Zac Medico from comment #5)
> Created attachment 415214 [details, diff] [details, diff]
> debug patch
> 
> This patch will make emerge output some useful debugging information to
> stderr. Sample usage:
> 
>     emerge -pvg ncurses > debug.log 2>&1

I have attached the debug.log file.

Thanks for the support
Comment 10 Zac Medico gentoo-dev 2015-10-23 16:35:42 UTC
(In reply to marco from comment #8)
> Created attachment 415226 [details]
> emerge log after patch

This shows that the instance key contains a None value for the BUILD_ID, while the local instance has BUILD_ID: 1 in the Packages file, so that explains why the comparison is failing. I'll have to investigate some more to find out what causes this.
Comment 11 Zac Medico gentoo-dev 2015-10-24 21:22:10 UTC
*** Bug 563820 has been marked as a duplicate of this bug. ***
Comment 12 Zac Medico gentoo-dev 2015-10-25 04:39:42 UTC
Does is make any difference if you remove the cached copy of the remote Packages file? You can remove all cached copies with this command:

   rm -rf /var/cache/edb/binhost
Comment 13 marco 2015-10-26 08:42:00 UTC
(In reply to Zac Medico from comment #12)
> Does is make any difference if you remove the cached copy of the remote
> Packages file? You can remove all cached copies with this command:
> 
>    rm -rf /var/cache/edb/binhost

Previously i tried to remove /usr/portage/packages/ but without success.

Perfect , once removed the edb/binhost cache all is ok .

Do you need something else to investigate ?
Comment 14 Zac Medico gentoo-dev 2015-10-26 16:09:25 UTC
(In reply to marco from comment #13)
> Perfect , once removed the edb/binhost cache all is ok .

Great!

> Do you need something else to investigate ?

Now that we have isolated the problem to the cache, I'll focus my attention on that area, and then I'll let you know if I need anything else.

@zerochaos: Does removing the cache also solve it for you?