Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 339642

Summary: <sys-apps/portage-2.1.9.13/2.2_rc89, emerge --sync either uses IPv4 or IPv6 hosts, never failover
Product: Portage Development Reporter: Sascha Hlusiak <bugs>
Component: UnclassifiedAssignee: Portage team <dev-portage>
Status: RESOLVED FIXED    
Severity: normal CC: esigra
Priority: High Keywords: InVCS
Version: unspecified   
Hardware: All   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 335925    
Attachments: display result of getaddrinfo(AF_UNSPEC)

Description Sascha Hlusiak 2010-10-03 23:05:12 UTC
Portage-2.2_rc88 does it's own name resolution and ordering. When doing an emerge --sync and DNS returns mixed A and AAAA records it will never use the IPv6 addresses. Source code reveales, that if not specified otherwise, AF_INET is the default:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=blob;f=pym/_emerge/actions.py;h=a5bb0e52cf9ccd68624838300643c2b01c3d7c3f;hb=HEAD#l2085

It can be overwritten by adding "-6" or "-4" (doesn't change anything) to make.conf: PORTAGE_RSYNC_EXTRA_OPTS. By default the IPv6 hosts will never be used, and if -6 is specified and the IPv6 connection is unreachable, it will never use IPv4. Is there a reason why the default isn't AF_UNSPEC to let the resolver decide what to return? There should be a check though to see if the system can reach the IPv6 destination so it doesn't exhaust PORTAGE_RSYNC_RETRIES with IPv6 hosts.

Current support of IPv6 rsync rotation isn't flexible and comfortable enough. I'd like to see portage use IPv6 mirrors when available without need to configure. Do you think that's possible?

Regards, Sascha


Reproducible: Always

Steps to Reproduce:
1. have SYNC point to a name with AAAA records (like rsync.de.gentoo.org)
2. have IPv6 connectivity
3. emerge --sync

Actual Results:  
Uses ONLY IPv4 addresses.
Setting PORTAGE_RSYNC_EXTRA_OPTS="-6" uses IPv6 addresses ONLY.

Expected Results:  
By default use IPv6 if available, else use IPv4.
Comment 1 Zac Medico gentoo-dev 2010-10-03 23:35:02 UTC
(In reply to comment #0)
> There should be a check though to see if the
> system can reach the IPv6 destination so it doesn't exhaust
> PORTAGE_RSYNC_RETRIES with IPv6 hosts.

This seems like the biggest obstacle. Maybe there's some kind of quick and simple test that can used to see if there's any point in trying any of the IPv6 hosts. It would be unfortunate if we had to test it by trying an actual connection to an IPv6 host, since that would be a waste of time if the client doesn't happen to have IPv6 connectivity at the moment that --sync is called.
Comment 2 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2010-10-04 06:47:28 UTC
(In reply to comment #1)
> This seems like the biggest obstacle. Maybe there's some kind of quick and
> simple test that can used to see if there's any point in trying any of the IPv6
> hosts. It would be unfortunate if we had to test it by trying an actual
> connection to an IPv6 host, since that would be a waste of time if the client
> doesn't happen to have IPv6 connectivity at the moment that --sync is called.

AFAIK it is pretty allowable to simply try IPv6. If the client doesn't have IPv6 connectivity, we'll intermediately get 'No route to host' error. If the client has broken IPv6, the user would probably have to wait till some timeout occurs but that's only his/her fault (and AFAIR Gentoo doesn't try to set up 6to4 blindly like some Windows versions do).
Comment 3 Sascha Hlusiak 2010-10-04 10:29:17 UTC
getaddrinfo() will already sort IPv6 addresses to the end of the list, if there is no IPv6 connection available or it will put them at the top, if there is. shuffle(ips) should be changed to shuffle the IPv4 and IPv6 separately though, to keep the family order returned by getaddrinfo(), that should make portage try IPv6 first only if IPv6 connectivity is assumed.

Leaving clients with broken IPv6 connectivity, that would exhaust PORTAGE_RSYNC_RETRIES before a healthy IPv4 can be tried. But why not just try all returned servers, instead of the first 3 only? Why should behaviour of emerge --sync differ from say "wget", which will try _all_ available addresses before giving up and decrease the waiting between tries?
Comment 4 Zac Medico gentoo-dev 2010-10-04 18:22:20 UTC
Created attachment 249562 [details]
display result of getaddrinfo(AF_UNSPEC)

When I run this script with 'rsync.gentoo.org' as the argument, it shows a list containing only IPv4 addresses. This is on a system that only has only IPv4 connectivity, but python has USE="ipv6" enabled. So, it seems that it's reasonable to use AF_UNSPEC in any case.

As for PORTAGE_RSYNC_RETRIES, I guess we can set that to -1 by default and make that mean that it should try all available addresses.
Comment 5 Zac Medico gentoo-dev 2010-10-04 18:44:30 UTC
(In reply to comment #4)
> As for PORTAGE_RSYNC_RETRIES, I guess we can set that to -1 by default and make
> that mean that it should try all available addresses.

This part is in git:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=5712d6c80f9be604abf2c293a86aedc8985d1916
Comment 6 Zac Medico gentoo-dev 2010-10-04 19:33:48 UTC
The AF_UNSPEC and shuffling part is here:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=78d28f182fbe2b2618be4652763849d0fc289b16
Comment 7 Sascha Hlusiak 2010-10-04 22:03:36 UTC
(In reply to comment #4)
> When I run this script with 'rsync.gentoo.org' as the argument, it shows a list
> containing only IPv4 addresses. 
That's because 'rsync.gentoo.org' only has IPv4 addresses. But run it on 'rsync.de.gentoo.org', and the script will put IPv6 addresses to the back of the list if you have no IPv6 connection. Otherwise they are ordered first.

I like the patch although I'd like the options '-4' and '-6' continue to exist, forcing the use of either ips_v4 or ips_v6, in case the user wants to overwrite the auto detect mechanism. I will give this some more testing soon...

Thanks a lot.
Comment 8 Zac Medico gentoo-dev 2010-10-04 22:13:26 UTC
(In reply to comment #7)

> I like the patch although I'd like the options '-4' and '-6' continue to exist,
> forcing the use of either ips_v4 or ips_v6, in case the user wants to overwrite
> the auto detect mechanism.

Okay, it's back:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=aa786192657f366d44ee7367e9ad69352a833048

Comment 9 Zac Medico gentoo-dev 2010-10-06 21:18:27 UTC
This is fixed in 2.2_rc90, but I'll leave this bug open until it's in an unmasked release.
Comment 10 Sascha Hlusiak 2010-10-07 07:01:36 UTC
I confirm that 2.2_rc90 shows the expected behaviour to me. Thanks, guys.
Comment 11 Zac Medico gentoo-dev 2010-10-08 02:36:15 UTC
(In reply to comment #10)
> I confirm that 2.2_rc90 shows the expected behaviour to me. Thanks, guys.

Thanks for testing. This is fixed in 2.1.9.14 too.