Bug 294791 – repoman should (optionally) check for 301 code for HOMEPAGE

• Product: Gentoo Linux
• Status: RESOLVED DUPLICATE

[Description Martin Walch 2009-11-27 07:25:27 UTC]

The homepage link

http://littlesvr.ca/asunder

is redirected (HTTP/1.1 301 Moved Permanently) to

http://littlesvr.ca/asunder/

Both are valid URIs (rfc2396), so I suggest adding the trailing slash in order
to avoid unnecessary redirection.

[Comment 1 Allen Brooker (AllenJB) 2009-11-27 08:05:11 UTC]

I'd suggest NOT making this change for 2 reasons:
1) It doesn't matter - the current URI works fine and invisibly redirects. This bug is just creating unnecessary load on the Gentoo developers.

2) The current URI allows upstream to invisibly move between a directory and a page without the need for the URI to be changed.

[Comment 2 Martin Walch 2009-11-27 08:44:42 UTC]

1) rfc2616 is clear about status code 301:
> The requested resource has been assigned a new permanent URI and any
> future references to this resource SHOULD use one of the returned URIs.

So it is primarily about conforming to HTTP/1.1 or not.

2) For this purpose, there are status codes 302 and 307.

[Comment 3 Martin Walch 2009-11-28 18:35:13 UTC]

I thought about this whole thing and came to the conclusion that it is indeed quite time consuming (especially my time, as during this year I have checked thousands of packages). Also it causes quite much noise with one bug report per 301 response.

So I have now written a (more or less) one liner in bash that checks each package for return code 301.

FORMAT="<category>/<name> <homepage>\n" eix | while read var; do name=`echo $var | cut -d\  -f1 -s`; value=`echo $var | cut -d\  -f2- -s`; for i in $value; do if [ ${i:0:4} = http ] && [ $(host $(echo $i | cut -d: -f2 | cut -d/ -f3)>/dev/null; echo $?) = 0 ] && [ ! "$(httping -g $i -c1 -G -m -o301 -e1)" = "1" -a $? = 0 ]; then echo $name: $i; fi; done; done

As it has become hard to read, I have tried unscrambling it a bit:

FORMAT="<category>/<name> <homepage>\n" eix | while read var
  do name=$(echo $var | cut -d\  -f1 -s)
     value=$(echo $var | cut -d\  -f2- -s)
     for i in $value
       do \
         host=$(echo $i | cut -d: -f2 | cut -d/ -f3)
         if [ ${i:0:4} = http ] \
         && [ $(host $host>/dev/null; echo $?) = 0 ] && \
         [ ! "$(httping $i -c1 -G -m -o301 -e1)" = "1" -a $? = 0 ]
           then echo $name: $i
         fi
     done
  done

It uses app-portage/eix, host from net-dns/bind-tools and >=net-analyzer/httping-1.30 (earlier versions of httping had an ugly bug).

Unfortunately, it is probably a bad idea to replace every link that is answered with "301 Moved Permanently" with the URI in the Location field, as some sites that have disappeared might now be redirected to placeholders or spam sites.

Maybe there is a maintainer who is willing to check all links in one pass and change (most of) the HOMEPAGE links accordingly? This will be much more efficient and will avoid much noise in the bug tracker.

Is this the right place for this suggestion?

[Comment 4 Martin Walch 2009-11-28 18:41:13 UTC]

Created attachment 211471 [details]
both versions of the small bash script from comment #3

Sorry, code in comment #3 broke when posting.

[Comment 5 Diego Elio Pettenò (RETIRED) 2009-11-29 17:41:42 UTC]

Open a feature request for repoman to check for that (optionally) might be a better idea.

[Comment 6 Sebastian Pipping 2010-03-11 22:53:04 UTC]

+  11 Mar 2010; Sebastian Pipping <[e-mail]> asunder-1.9.1.ebuild:
+  Fix homepage (bug #294791)
+

Now turning into repoman check as Diego suggested.

[Comment 7 Sebastian Pipping 2010-03-11 23:48:13 UTC]

*** This bug has been marked as a duplicate of bug 295194 ***