Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 309825 - bootstrapping is broken due to very strict QA checking about invalid shebangs
Summary: bootstrapping is broken due to very strict QA checking about invalid shebangs
Status: RESOLVED FIXED
Alias: None
Product: Gentoo/Alt
Classification: Unclassified
Component: Prefix Support (show other bugs)
Hardware: All Linux
: High blocker (vote)
Assignee: Gentoo Prefix
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-03-16 17:15 UTC by Jeremy Olexa (darkside) (RETIRED)
Modified: 2010-03-18 13:14 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments
catch-bin-sh-case.patch (proposed patch) (catch-bin-sh-case.patch,2.23 KB, text/plain)
2010-03-17 15:12 UTC, Jeremy Olexa (darkside) (RETIRED)
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2010-03-16 17:15:18 UTC
In the solaris guide, you need ncurses to install bash. ncurses can't be installed because bash is not installed in EPREFIX yet. So, in other words, bash needs bash. (And you can't --nodeps bash either, because bashbug needs sh in EPREFIX).

The same is true for the osx guide according to a new user I have been working with in irc. (again, bash needs bash)

So, we need to either relax the invalid QA check or make it a FEATURE that is disabled during bootstrapping (mduft's idea). There is a symlink workaround but then you have to mess with collision-protect.
Comment 1 Fabian Groffen gentoo-dev 2010-03-16 17:19:29 UTC
I opt for changing the order to first --nodeps bash then wget.  This doesn't work on Interix, since bash doesn't compile for some reason, but on Solaris and OSX this should work, as bash could be bootstrapped as well.
Comment 2 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2010-03-16 17:20:56 UTC
(In reply to comment #1)
> I opt for changing the order to first --nodeps bash then wget.  This doesn't
> work on Interix, since bash doesn't compile for some reason, but on Solaris and
> OSX this should work, as bash could be bootstrapped as well.
> 

Nope, you can't do that. I said in comment #0 that bash needs bash. If sh is not in EPREFIX bash will fail to install. I already tried this =P
Comment 3 Markus Duft (RETIRED) gentoo-dev 2010-03-17 07:10:50 UTC
as i said before, i opt for a FEATURE. i must admit, i don't know the current manual bootstrap procedure, but with prefix-launcher, the FEATURE could be enabled before rebuilding @system. at this stage, everything is in place, and the rebuild is there to make the prefix self-contained, and cut all/most possibly remaining references to the outside world. it feels like this is the place where this check starts making sense.

during bootstrap i have outside references, and there is no way to get around this, so it is pointless to die if such a reference is found. correct me if you feel i'm wrong ;)
Comment 4 Fabian Groffen gentoo-dev 2010-03-17 09:23:03 UTC
Well, I just don't like that idea.  It seems the only things that are causing trouble are /bin/sh and /bin/bash.  We can -collision-protect install bash, provided we install the correct symlink.  However, I still don't understand why bash can't be installed on an empty system.  The check should consider ${ED} as well, since it obviously makes sense to check if it would work after the package has been installed.
Comment 5 Markus Duft (RETIRED) gentoo-dev 2010-03-17 09:33:07 UTC
interesting enough, i successfully bootstrapped on gentoo linux x86 17 hours ago in an automated nightly build. so it does work at least somewhere, it seems.
Comment 6 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2010-03-17 14:59:00 UTC
(In reply to comment #4)
> provided we install the correct symlink.  However, I still don't understand why
> bash can't be installed on an empty system.  The check should consider ${ED} as
> well, since it obviously makes sense to check if it would work after the
> package has been installed.

Ah ha, that is where we learn something.

Consider my debugging, below:

 * QA Notice: the following files use invalid (possible non-prefixed) shebangs:
 * fn: /home/olexa/test/var/tmp/portage/app-shells/bash-4.0_p37/image/home/olexa
/test/usr/bin/bashbug
 * pos: 1
 * line: #!/bin/sh -
 * ${EPREFIX}${line[0]}: /home/olexa/test/bin/sh
 * ${ED}${line[0]}: /home/olexa/test/var/tmp/portage/app-shells/bash-4.0_p37/image/home/olexa/test//bin/sh
 * fp: /home/olexa/test/usr/bin
 * home/olexa/test/usr/bin/bashbug:/bin/sh
 * ERROR: app-shells/bash-4.0_p37 failed:
 *   Aborting due to QA concerns: invalid shebangs found

Ok, that matches what you said (quoted above), the checks are working but automatic prefixing does NOT happen, why? Well, more debug:

"Does ${ED}${line[0]} exist?" NO!
%% ls /home/olexa/test/var/tmp/portage/app-shells/bash-4.0_p37/image/home/olexa/test//bin
bash*  rbash@

snippet from bash-4.0_p37.ebuild:

pkg_postinst() {
    # If /bin/sh does not exist, provide it
    if [[ ! -e ${EROOT}/bin/sh ]]; then
        ln -sf bash "${EROOT}"/bin/sh
    fi
}
Comment 7 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2010-03-17 15:12:58 UTC
Created attachment 224011 [details]
catch-bin-sh-case.patch (proposed patch)

Attached patch contains my debugging hints (which can be discarded) and most importantly the below snippet. This will allow bash to be automatically prefixed even though $EPREFIX/bin/sh doesn't exist yet. The only time this code will be evaluated is when bash does not yet exist (I think).

+       if [[ ${EPREFIX}${line[0]} == */bin/sh ]]; then
+           eqawarn "prefixing shebang of ${fn#${D}}"
+           sed -i -e '1s:^#! \?:#!'"${EPREFIX}"':' "${fn}"
+           continue
+       fi

Suggestions? Comments? anything?
Comment 8 Markus Duft (RETIRED) gentoo-dev 2010-03-17 15:19:47 UTC
so, maybe, is the shebang check in the wrong place then? should it be run later than now (after all the ebuild/profile provided postinstall hooks, etc.)?
Comment 9 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2010-03-17 15:30:10 UTC
(In reply to comment #8)
> so, maybe, is the shebang check in the wrong place then? should it be run later
> than now (after all the ebuild/profile provided postinstall hooks, etc.)?
> 

Nah, if you are going to change anything it has to be while in $ED still. From there portage makes binpkgs and does some checksum operation. Any modification after that would be bad, I'm sure.
Comment 10 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2010-03-17 21:08:20 UTC
Discussion was had in IRC.

We decided that since it is only this one package (bash) during this one time (when bash is being installed for the first time and no bash is in $EPREFIX), it would be best to NOT diverge portage away from mainline more-so. With that in mind, bashbug now has a bash shebang instead of a sh shebang, because sh isn't in $ED (as noted in comment #6). http://overlays.gentoo.org/proj/alt/changeset/57472

With that in place, I made a new tarball that users should download. http://overlays.gentoo.org/proj/alt/changeset/57473 The tarball was not bumped to a new date because the state of the tree is not healthy right now. Future snapshot bumps will be the normal naming convention.

I'll consider this closed for now since we feel that this shebang check is helpful and there is only this one edge case that is problematic.
Comment 11 Markus Duft (RETIRED) gentoo-dev 2010-03-18 06:40:06 UTC
uhm, what about bash being so buggy that bashbug cannot run the shell...? maybe /bin/sh (and really /bin/sh nothing from EPREFIX) was intentional here? just a thought ;p
Comment 12 Fabian Groffen gentoo-dev 2010-03-18 07:37:17 UTC
That crossed my mind also, but on many systems that bash is installed, /bin/sh IS that same /bin/bash.  So it doesn't really make things worse.

We may as well throw the script away, since it does things like this all over:

PATH=/bin:/usr/bin:/usr/local/bin:$PATH
export PATH

and it uses certain constructs of which I don't know if for example solaris /bin/sh groks it correctly.
Comment 13 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2010-03-18 13:14:29 UTC
In the end, we don't care about "bashbug" - it is a 271 line script to report bugs about bash to bug-bash@gnu.org - had this been a major tool that was used often I would consider your complaint.