Bug 380801 – www-client/chromium-9999 needs to call src/build/download_nacl_toolchains.py

• Product: Gentoo Linux
• Status: RESOLVED FIXED

[Description Mike Gilbert 2011-08-27 01:11:59 UTC]

The hook in DEPS now calls src/build/download_nacl_toolchains.py in place of download_nacl_irt.py.

A version and some file hashes are passed as arguments.

When download_nacl_irt.py was called without arguments, it would parse DEPS itself and download the version specified there. The current ebuild exploits this feature to avoid parsing DEPS.

If download_nacl_toolchains.py is called without arguments, it will download the latest version available, rather than the version specified in DEPS.

Should we ask upstream to add an option to this script to make it parse the version from DEPS like the previous script did?

Alternatively, an easy hack is to patch the call to gyp_chromium out of DEPS, call gclient runhooks, and then restore the original DEPS. This feels both clever and dirty to me.

[Comment 1 Julien Sanchez 2011-08-29 12:41:35 UTC]

I'm sure you're right. I don't really understand chromium's build process. According to this commit http://git.chromium.org/gitweb/?p=chromium.git;a=commitdiff;h=cf07c4d6e826e749c42aef36336eb61f651d5074 , isn't download_nacl_irt.py simply dropped to be built with chrome instead. download_nacl_toolchains.py hook is now mandatory to build it (but skipped in previous ebuilds because of --no-hooks)?

I tried to comment '"$(PYTHON)" src/build/download_nacl_irt.py || die' 
and add '"$(PYTHON)" src/build/download_nacl_toolchains.py || die;'
in the ebuild but it fails with:

Syncing projects:  98% (77/78), done.ird_party/WebKit/Source
linux_x86: downloading latest version...
Download failed on http://commondatastorage.googleapis.com/nativeclient-archive2/x86_toolchain/rlatest/toolchain_linux_x86.tar.xz, retrying... (1)
Download failed on http://commondatastorage.googleapis.com/nativeclient-archive2/x86_toolchain/rlatest/toolchain_linux_x86.tar.xz, retrying... (2)
Download failed on http://commondatastorage.googleapis.com/nativeclient-archive2/x86_toolchain/rlatest/toolchain_linux_x86.tar.xz, retrying... (3)
Download failed on http://commondatastorage.googleapis.com/nativeclient-archive2/x86_toolchain/rlatest/toolchain_linux_x86.tar.xz, retrying... (4)
Download failed on http://commondatastorage.googleapis.com/nativeclient-archive2/x86_toolchain/rlatest/toolchain_linux_x86.tar.xz, retrying... (5)
Download failed on http://commondatastorage.googleapis.com/nativeclient-archive2/x86_toolchain/rlatest/toolchain_linux_x86.tar.xz, retrying... (6)
Download failed on http://commondatastorage.googleapis.com/nativeclient-archive2/x86_toolchain/rlatest/toolchain_linux_x86.tar.xz, retrying... (7)
Download failed on http://commondatastorage.googleapis.com/nativeclient-archive2/x86_toolchain/rlatest/toolchain_linux_x86.tar.xz, retrying... (8)
Download failed on http://commondatastorage.googleapis.com/nativeclient-archive2/x86_toolchain/rlatest/toolchain_linux_x86.tar.xz, retrying... (9)
Download failed on http://commondatastorage.googleapis.com/nativeclient-archive2/x86_toolchain/rlatest/toolchain_linux_x86.tar.xz, giving up.
Traceback (most recent call last):
  File "src/build/download_nacl_toolchains.py", line 42, in <module>
    Main(sys.argv[1:])
  File "src/build/download_nacl_toolchains.py", line 38, in Main
    download_toolchains.Main(args)
  File "/usr/portage/distfiles/svn-src/chromium/src/native_client/build/download_toolchains.py", line 148, in Main
    SyncFlavor(flavor, url, dst, hash_value)
  File "/usr/portage/distfiles/svn-src/chromium/src/native_client/build/download_toolchains.py", line 69, in SyncFlavor
    hash=hash)
  File "/usr/portage/distfiles/svn-src/chromium/src/native_client/build/sync_tgz.py", line 96, in SyncTgz
    username=username, password=password, verbose=verbose)
  File "/usr/portage/distfiles/svn-src/chromium/src/native_client/build/http_download.py", line 79, in HttpDownload
    src = urllib2.urlopen(url)
  File "/usr/lib64/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 400, in open
    response = meth(req, response)
  File "/usr/lib64/python2.7/urllib2.py", line 513, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.7/urllib2.py", line 438, in error
    return self._call_chain(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 372, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 521, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found
 * ERROR: www-client/chromium-9999-r1 failed (unpack phase):
 *   (no error message)
 * 
 * Call stack:
 *     ebuild.sh, line  56:  Called src_unpack
 *   environment, line 6577:  Called die
 * The specific snippet of code:
 *       "$(PYTHON)" src/build/download_nacl_toolchains.py || die;

As you say, it would be better to patch upstream (or with the ebuild?) download_nacl_toolchains.py to parse DEPS when no arguments.

[Comment 2 Julien Sanchez 2011-08-30 13:24:32 UTC]

It works and downloads latest version if we call directly:
"$(PYTHON)" src/build/download_nacl_toolchains.py --nacl-newlib-only || die

See attachement.

Otherwise if we care about file hash and version, is it acceptable to parse (sed/awk?) DEPS directly in a function of the ebuild to get parameters?

[Comment 3 Julien Sanchez 2011-08-30 13:25:20 UTC]

Created attachment 285083 [details, diff]
chromium 9999 ebuild patch

[Comment 4 Mike Gilbert 2011-08-30 23:54:57 UTC]

Thanks for testing and the suggestions Julien.

I would rather use the revisions specified in DEPS than the latest revs.

Pawel suggested that we might call gclient runhooks with GYP_DEFINES set instead of calling gyp_chromium manually. If we do that in src_unpack, we have to move all the logic into src_unpack. If we do it in src_configure, the nacl toolchain will be downloaded to ${WORKDIR} rather than ${DISTDIR}/svn-src/chromium.

For now, I am going with my "alternate" plan: use sed to replace gyp_chromium with "true" in DEPS.

[Comment 5 Jeff Kowalczyk 2011-08-31 02:04:26 UTC]

An ~amd64 gentoo-sources-3.0.x apparently returns a platform string (linux3) that the toolchain doesn't recognize yet:

________ running '/usr/bin/python2.7 src/build/download_nacl_toolchains.py --x86-version 6494 --nacl-newlib-only --file-hash mac_x86_newlib 1b0855435c03c435a011c6105a509624b2a4edaa --file-hash win_x86_newlib 5038a47b5a9a49acdc36cbe311aec7bce575c164 --file-hash linux_x86_newlib 01e245dc6dca16bea5cf840dbc77e3aa138f234f' in '/usr/portage/distfiles/svn-src/chromium'
Traceback (most recent call last):
  File "src/build/download_nacl_toolchains.py", line 42, in <module>
    Main(sys.argv[1:])
  File "src/build/download_nacl_toolchains.py", line 38, in Main
    download_toolchains.Main(args)
  File "/usr/portage/distfiles/svn-src/chromium/src/native_client/build/download_toolchains.py", line 112, in Main
    platform_fixed = download_utils.PlatformName()
  File "/usr/portage/distfiles/svn-src/chromium/src/native_client/build/download_utils.py", line 44, in PlatformName
    return PLATFORM_COLLAPSE[name]
KeyError: 'linux3'
Error: /usr/bin/python2.7 src/build/download_nacl_toolchains.py --x86-version 6494 --nacl-newlib-only --file-hash mac_x86_newlib 1b0855435c03c435a011c6105a509624b2a4edaa --file-hash win_x86_newlib 5038a47b5a9a49acdc36cbe311aec7bce575c164 --file-hash linux_x86_newlib 01e245dc6dca16bea5cf840dbc77e3aa138f234f in /usr/portage/distfiles/svn-src/chromium returned 1

[Comment 6 Mike Gilbert 2011-08-31 02:39:26 UTC]

(In reply to comment #5)

Jeff: Could you file a bug in the upstream bug tracker?

http://code.google.com/p/nativeclient/issues/list