Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 598527 - app-shells/push: installs stray push.sh to /usr/bin
Summary: app-shells/push: installs stray push.sh to /usr/bin
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal QA (vote)
Assignee: Martin Väth
URL:
Whiteboard:
Keywords: QAcanfix
: 416911 (view as bug list)
Depends on:
Blocks:
 
Reported: 2016-10-30 13:41 UTC by Michał Górny
Modified: 2017-02-10 09:47 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2016-10-30 13:41:56 UTC
$ head push.sh 
#! /bin/sh
# (C) Martin V\"ath <martin@mvath.de>
# This script is meant to be sourced; the first line is only for editors

If it's not meant to be executed, it has no place in /usr/bin. I know that bash searches PATH when sourcing files but that is no excuse to pollute it for everything.

Considering that it's used by eix only, just put it somewhere in /usr/lib, and use the full path.
Comment 1 Martin Väth 2016-10-30 18:46:50 UTC
(In reply to Michał Górny from comment #0)
> If it's not meant to be executed, it has no place in /usr/bin.

It is meant to be executed, namely by the shell interpreter.

> I know that bash searches PATH when sourcing files but that is no excuse
> to pollute it for everything.

Quite the opposite: Some self-invented "pollution" definition is no reason to make the tool artificially hard to use and to require users to write scripts which try random paths for random unix distributions. (See below).

If the executable bit would be set, I could agree that being in PATH _might_ be (harmlessly) polluting something (namely perhaps a completion function of a shell). But this is not the case.

Quite the opposite, after typing ". <TAB>" the user should _expect_ that this tool is presented to him for completion. Removing it from PATH would remove this intended functionality (among others).

> Considering that it's used by eix only

That's not correct. What is true is that 
_among the packets from the gentoo repository_
eix is (currently) the only consumer.

I have a dozens of scripts in the mv overlay and on my machines which rely on
 . push.sh
without special measurements to be working, and other users might have similar scripts (at least, I hope so, because that's why I wrote this tool):

The main intention of the tool is that it is used by users for scripts which are meant to run on different systems (if the shell has at least basic POSIX features) and which could not easily be written cleanly and compatible without it.

This is how it is documented and recommended to be used.

In fact, I had thought a long time before I (as being upstream) made this recommendation in the documentation: There is no other canonical place except PATH for a shell-must-source-it-to-use-it tool.

I intentionally want to avoid that some distribution installs it into /usr/lib, another into /usr/lib64, another into /usr/share, yet another into /usr/share/push, yet another into /usr/lib/push.sh or /usr/lib64/push or whatever (for a lot of these, one might find arguments, but none of these is really canonical and can be expected everywhere).

Namely, this would mean that the user who just wants to use the tool in some of his own (compatible) scripts - even if he installed push by the distribution - would have to write a very long script with a lot of magic guessing only to be able to _use_ a tool whose main intention is to _simplify_ writing compatible scripts.

Usage should be just as simple as ". push.sh" if you have a standard PATH.
Comment 2 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2016-10-30 18:57:46 UTC
(In reply to Martin Väth from comment #1)
> (In reply to Michał Górny from comment #0)
> > If it's not meant to be executed, it has no place in /usr/bin.
> 
> It is meant to be executed, namely by the shell interpreter.
> 
> > I know that bash searches PATH when sourcing files but that is no excuse
> > to pollute it for everything.
> 
> Quite the opposite: Some self-invented "pollution" definition is no reason
> to make the tool artificially hard to use and to require users to write
> scripts which try random paths for random unix distributions. (See below).

FHS:

| /usr/bin : Most user commands
|
| Purpose
|
| This is the primary directory of executable commands on the system.
                                              ^^^^^^^^

This is not a command. It won't work correctly when attempted to run as a command.
Comment 3 Ulrich Müller gentoo-dev 2016-10-30 19:07:08 UTC
+1

If this isn't a user command then it doesn't belong in /usr/bin/. It should go into /usr/share/${PN}/ or /usr/libexec/${PN}/.
Comment 4 Martin Väth 2016-10-30 20:29:59 UTC
It is as well a command as e.g. lesspipe.sh which is meant to be used in a particular way, only. And also that all executable commands _must_ be included in /usr/bin does not mean that other files must _not_. Moreover, /usr/lib would be false as well: FHS says about it
... that are not intended to be executed directly by users or shell scripts. 
But push.sh _is_ supposed to be run by shell scripts (or users).
On amd64 in gentoo, it is twice false, because /usr/lib _is_ /usr/lib64, and this should be reserved _only_ for binary libraries.

Anyway, whether splitting hairs gives one result or another is not important IMHO.

Important are the practical implications which rule over "philosophical" considerations. After all, the purpose of FHS is to simplify usage and help with writing compatible scripts/programs, not to hinder it.

Fact is that putting push.sh anywhere else than in PATH breaks the intended usage.

If having it in PATH is a bug, i.e., if it is actually causing some problems or breaking compatibility of something similar, then name this bug.
Otherwise, there is no reason to _introduce_ a bug and break the intended usage only for some abstract (and quite arguable) "philosophical ideals".
Comment 5 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2016-10-30 20:52:35 UTC
(In reply to Martin Väth from comment #4)
> It is as well a command as e.g. lesspipe.sh which is meant to be used in a
> particular way, only. And also that all executable commands _must_ be
> included in /usr/bin does not mean that other files must _not_.

You are bending the definition to its limits. So it would be fine to put random libraries and data files there, suddenly? Say, if it helped me avoid hacking on poorly written program that looks for its files relatively to the binary?

> Moreover,
> /usr/lib would be false as well: FHS says about it
> ... that are not intended to be executed directly by users or shell scripts. 
> But push.sh _is_ supposed to be run by shell scripts (or users).

source != execute. It doesn't have executable bit, it can't be executed.

Furthermore, your definition is outdated. I was talking about libexec-like /usr/lib.

> On amd64 in gentoo, it is twice false, because /usr/lib _is_ /usr/lib64, and
> this should be reserved _only_ for binary libraries.

Poor default filesystem layout is irrelevant.

> Anyway, whether splitting hairs gives one result or another is not important
> IMHO.
> 
> Important are the practical implications which rule over "philosophical"
> considerations. After all, the purpose of FHS is to simplify usage and help
> with writing compatible scripts/programs, not to hinder it.
> 
> Fact is that putting push.sh anywhere else than in PATH breaks the intended
> usage.
> 
> If having it in PATH is a bug, i.e., if it is actually causing some problems
> or breaking compatibility of something similar, then name this bug.
> Otherwise, there is no reason to _introduce_ a bug and break the intended
> usage only for some abstract (and quite arguable) "philosophical ideals".

Following specifications is not a bug. Introducing non-commands in PATH is.

Please follow the rules. Intentionally breaking the rules 'because I wanted it to be this way' doesn't render them incorrect.
Comment 6 Martin Väth 2016-10-30 23:05:12 UTC
(In reply to Michał Górny from comment #5)
> 
> You are bending the definition to its limits.

As you are with your definition of "command" which obviously is too restrictive for practice in this case.

> it would be fine to put random libraries

I am not going to speak about your irrelevant strawman.

> Poor default filesystem layout is irrelevant.

Suddenly it is irrelevant if an inappropriate file ends in a directory it definitely should not be according to FHS.
How far will you go only to break my tools?

> Following specifications is not a bug

That's a word of a lawyer. "A bug is what _I_ define to be a bug.
Not following some hair-splitted rule is - even if I know very well that it has absolutely no negative effect. But it is not a bug to intentionally render a tool useless and breaking existing user scripts just to follow some over-interpreted rule, which is obviously wrong in this case."
Lawyers perhaps do not care about the intention of some rules.
 
> Please follow the rules.

Please don't break my tools.
Comment 7 Ulrich Müller gentoo-dev 2016-10-31 07:36:26 UTC
*** Bug 416911 has been marked as a duplicate of this bug. ***
Comment 8 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2016-10-31 07:48:53 UTC
(In reply to Martin Väth from comment #6)
> (In reply to Michał Górny from comment #5)
> > 
> > You are bending the definition to its limits.
> 
> As you are with your definition of "command" which obviously is too
> restrictive for practice in this case.

It worked for many years. Then you came and extended it to your own purpose. And now you're saying the previous definition was wrong because it doesn't fit your purpose!

> > Poor default filesystem layout is irrelevant.
> 
> Suddenly it is irrelevant if an inappropriate file ends in a directory it
> definitely should not be according to FHS.
> How far will you go only to break my tools?

FYI, the lib/lib64 is problem is in process of being fixed. It's just a matter of pulling the plug, and telling people to migrate their systems.

> > Following specifications is not a bug
> 
> Lawyers perhaps do not care about the intention of some rules.

And do you? Because as far as I can see, you only care about things being your way.

And as we can see easily, there were multiple times when multiple people have pointed out the wrong of your ways. Why do you think you're special and have the power to redefine rules?
Comment 9 Ulrich Müller gentoo-dev 2016-10-31 07:57:49 UTC
<QA hat>
I see now that this has come up repeatedly since four years already, in bug 416911 and in bug 493000 (for eix-functions.sh). All of these scripts don't belong in /usr/bin but in a subdirectory of /usr/share. FHS is quite unambiguous there.

Please fix. (Doing this in the package is cleaner than downstream patching of eix.)
</QA hat>
Comment 10 Martin Väth 2016-10-31 09:55:28 UTC
> It worked for many years.

It had _never_ worked. Or can you name _any_ project which provides a shell function for general _compatible_ usage for user-provided POSIX shell script?

> Then you came and extended it to your own purpose

Then came I and made it work.
And after it worked for several years came one guy (and now another) nitpicking that one can word-by-word overly interpret some rules instead of understanding the _intention_ of the rules: The _intention_ of FHS is clearly to enhance compatibility for user programs and scripts, not to hinder it.

> you only care about things being your way.

I only care about my tools being _working_ and compatible over all distributions.

> multiple people

Let's be precise: Exactly 3 people: Spanky, you, and Ulrich (in timely order).
Just to recall: Spanky is the guy who insists on an overly word-by-word interpretation of the POSIX standard, even after being told by a member of a commission of that standard that his interpretation is not intended and that his interpretation would even hinder POSIX certification of unpatched Gentoo, see bug 590696.

> <QA hat>

Breaking package usage and user scripts has nothing to do with quality assurance - quite the opposite. Neither has "fixing" something which has absolutely no negative effect except not matching in some people's philosophy how things should be in an ideal world.
Comment 11 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2016-10-31 15:19:22 UTC
(In reply to Martin Väth from comment #10)
> > It worked for many years.
> 
> It had _never_ worked.

Err, are you saying that /usr/bin never worked? Thank you for saving us from years of inability to run anything that we didn't even notice!

> Or can you name _any_ project which provides a shell
> function for general _compatible_ usage for user-provided POSIX shell script?

gentoo-functions.

> > Then you came and extended it to your own purpose
> 
> Then came I and made it work.

Thank you for solving the problem that we suffered from for many years without even noticing!

> And after it worked for several years came one guy (and now another)
> nitpicking that one can word-by-word overly interpret some rules instead of
> understanding the _intention_ of the rules: The _intention_ of FHS is
> clearly to enhance compatibility for user programs and scripts, not to
> hinder it.

I count four already. Three of them involved with QA. I'm pretty sure there would be more if they noticed. There's also fifth who's actually working on QA checks that would actually catch this as '/usr/bin file without +x'.

> > you only care about things being your way.
> 
> I only care about my tools being _working_ and compatible over all
> distributions.

Excuse me but could you name one non-Gentoo distribution that has found your scripts useful and packaged them following your rules? Because I have serious doubts that e.g. Debian would have accepted stray files in /usr/bin.

> > multiple people
> 
> Let's be precise: Exactly 3 people: Spanky, you, and Ulrich (in timely
> order).

Four.

> Just to recall: Spanky is the guy who insists on an overly word-by-word
> interpretation of the POSIX standard, even after being told by a member of a
> commission of that standard that his interpretation is not intended and that
> his interpretation would even hinder POSIX certification of unpatched
> Gentoo, see bug 590696.

Personal attack to support your stance?

> > <QA hat>
> 
> Breaking package usage and user scripts has nothing to do with quality
> assurance - quite the opposite. Neither has "fixing" something which has
> absolutely no negative effect except not matching in some people's
> philosophy how things should be in an ideal world.

Now attack on the whole QA team and its purpose?
Comment 12 Martin Väth 2016-10-31 16:54:03 UTC
(In reply to Michał Górny from comment #11)
> Err, are you saying that /usr/bin never worked?

You are building a strawman again instead of answering:

: Or can you name _any_ project which provides a shell function for general _compatible_ usage for user-provided POSIX shell script?

> gentoo-functions.

This is indeed an example of a tool which has the main aim of being compatible among all linux or even unix distributions!

Actually even this poor example demonstrates how bad this path-is-fixed-at-a-random-location idea worked, even within the limited range of only one single distribution: When this file was moved between packages and locations, a lot of breakage occurred for many user. Not to speak about all the users who had to adapt their own scripts using gentoo-functions which suddenly all broke, too.
It is exactly such kind of disaster which I want to avoid (and in fact for a tool which aims to be much more general-purpose than gentoo-functions).

> There's also fifth who's actually working on QA checks that would actually catch this as '/usr/bin file without +x'.

Then its a good hint to him that this check - which I actually support, because it is certainly useful to catch unintentional installations into the wrong path - should be equipped with a proper way to mark justified intentional exceptions.

> > > <QA hat>
> >
> > Breaking package usage and user scripts has nothing to do with quality
> > assurance
>
> Now attack on the whole QA team and its purpose

Again a strawman. You are implying a very strange purpose of the QA team here. I stated exactly the opposite about its purpose and nothing about the QA team.

Your whole previous post consists only of personal attacks and strawmans.

Do you have any positive suggestion? Like e.g. how it might be possible to fulfill your wish without breaking the purpose of the tool?
Comment 13 Martin Väth 2016-10-31 19:01:09 UTC
I think that I found now a solution which hopefully makes everybody happy!

I had first thought about using a system-dependent binary which outputs the path of the actual script to be sourced, but now I got a much better idea:

Instead of sourcing, the better way is probably anyway to "eval" the output of some binary: This has the additional advantage for the user that he can verify in his shell script whether the package is installed (while sourcing would just stop the whole script if the package is not installed). The way it is implemented now also has the advantage that old scripts which are still sourcing will not break.
Thus: Improved functionality, no breakage of existing scripts, and in addition now only commands (meant to be used as such) are installed into /usr/bin

The new versions of push and quoter are already released (the push ebuild will need a modification); eix will still need some more testing, but I will make a new release when I find the time.
Comment 14 Ulrich Müller gentoo-dev 2016-11-01 06:18:43 UTC
Really, you are adding a "#!/usr/bin/env cat" header so that it now *formally* is an executable and can go into /usr/bin (from where it then still can be sourced, BTW)? Why don't you put "#!/bin/true" instead, it will achieve the same?

I see this simply as an attempt to circumvent FHS rules, and I am quite sure that QA is with me that we don't want such "commands".
Comment 15 Martin Väth 2016-11-01 08:08:51 UTC
(In reply to Ulrich Müller from comment #14)
> *formally* is an executable and can go into /usr/bin

Not only formally: It is now _meant_ to be executed to produce output for "eval", because this simplifies error handling. That's how it is documented and recommended now and indeed already used this way in eix and in further 12 of my project in the mv overlay.

> Why don't you put "#!/bin/true" instead, it will achieve the same?

How silly are your arguments now going to be? Using a command whose purpose is to output some command paths or simple code like pkg-config or lesspipe.sh is OK, but if it outputs a small function, it is not?
A _lot_ of unix commands do not do much except that they are convenient to use in shell scripts or other tools (/bin/true indeed is an example of such a command): That's how unix works!

> (from where it then still can be sourced, BTW)?

Sure, that's the advantage which I already pointed out and which every sane person should consider a big advantage: User scripts relying on the old method (with the inferior error handling) are not broken by the upgrade.

Of course, it contradicts your obvious _true_ intention of your involvement here: That it doesn't break my project:

> don't want such "commands".

Now your true intention becomes more obvious to everybody: You don't like my project (or perhaps rather me, judging from your past bullying against me on the developer mailing list).
You were already misusing your QA role to attempt to boycott a project which or whose author you do not like and are now trying to convince other members of QA to support your intrigue (that's why you already said "we" so that QA members cannot so easily contradict you if you already claimed to speak in the name of all QA members).
Your obvious attempt to boycott a project/person in the name of QA might become a case for COMREL.
Comment 16 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2016-11-01 08:34:02 UTC
I don't think this is discussion is worth continuing, as it will only result in more offensive statements and/or worse solutions being deployed.

I suggest the following course of action:

1. Mask app-shells/push and app-shells/quoter due to QA violations, lack of upstream collaboration and no revdeps in ::gentoo but for eix,

2. Bundle the both scripts in app-portage/eix, in /usr/lib/eix or like (move eix-functions.sh) there as well,

3. Update the scripts and eix to use Gentoo paths.

I would also suggest removing Martin from the position of proxied maintainer.

@Martin, if you believe there is a comrel case here, then please open an appropriate bug to comrel@, and mention this bug as evidence. But don't use comrel to threaten others.
Comment 17 Ulrich Müller gentoo-dev 2016-11-01 09:11:32 UTC
(In reply to Martin Väth from comment #15)
> > *formally* is an executable and can go into /usr/bin
> 
> Not only formally: It is now _meant_ to be executed to produce output for
> "eval", because this simplifies error handling. That's how it is documented
> and recommended now and indeed already used this way in eix and in further
> 12 of my project in the mv overlay.

And why would eval be superior to sourcing? In most cases usage of eval is an indication of broken code, especially if there is an alternative solution.

> [...] Using a command whose purpose is to output some command paths or
> simple code like pkg-config or lesspipe.sh is OK, but if it outputs a small
> function, it is not?

> How silly are your arguments now going to be? Using a command whose purpose
> is to output some command paths or simple code like pkg-config or
> lesspipe.sh is OK, but if it outputs a small function, it is not?

Indeed, it is not. A picture of a teletubbie (to reuse an example from the devmanual) remains a picture, even if one would put a #!/usr/bin/display shebang in front of it so that it will display itself (which will work for some image formats). Such a file has no place in /usr/bin. The same is the case for a quine script whose sole purpose is to output itself.

> Of course, it contradicts your obvious _true_ intention of your involvement
> here: That it doesn't break my project:
> 
> > don't want such "commands".
> 
> Now your true intention becomes more obvious to everybody: You don't like my
> project (or perhaps rather me, judging from your past bullying against me on
> the developer mailing list).
> You were already misusing your QA role to attempt to boycott a project which
> or whose author you do not like and are now trying to convince other members
> of QA to support your intrigue (that's why you already said "we" so that QA
> members cannot so easily contradict you if you already claimed to speak in
> the name of all QA members).

Sorry, but it is you who has just taken this discussion from a factual to a personal level. I won't tolerate your insinuating that I have any "intentions" besides those furthering the Gentoo project, or that I would misuse my QA role. Also, when and where have I bullied you in the past?

Note that by now three current (pinkbyte, mgorny, and myself) and two former (ssuominen and vapier) QA members have told you that your shell function libraries don't belong in /usr/bin. Indeed, I am the last of the five coming into play here. Also note that apart from pointing out that there is an issue, no QA action has taken place.

> Your obvious attempt to boycott a project/person in the name of QA might
> become a case for COMREL.

Yes, I am interested too what they would have to say about this, so feel free to put them into CC.
Comment 18 Martin Väth 2016-11-01 10:31:26 UTC
(In reply to Ulrich Müller from comment #17)
> And why would eval be superior to sourcing?

Because sourcing stops the script if the sourced file is not found.
As a "workaround" for this, one can do (and in fact, I had done this in the past) crazy things like

( . push.sh ) >/dev/null 2>&1 || { ... push.sh is not installed }
. push.sh

but this is unnecessarily redundant and, moreover, it cannot easily distinguish between an error in the sourcing or in the push.sh "execution" (of defining the function - e.g. if sourcing fails for some shell-internal reason like attempting to override a readonly variable or something similiar).

The separation between "getting the code" and "evaluating" it avoids all these problems and is IMHO indeed how such a thing should be solved: It corresponds very much to the pkg-config approach.

> In most cases usage of eval is an indication of broken code

Especially in pure POSIX, eval has many purposes: It is already needed for only variable indirection (push.sh internally makes intensive use of eval, anyway).
Sure, there are wrong usages of it, but IMHO this is not.

> Such a file has no place in /usr/bin.

Says which standard where?
I consider e.g. "quoter" (the C binary) and "quoter_pipe" or "Push" as the same type of commands, only that the latter two must for a technical reason not be run as a forked process and thus must be provided in some way as shell functions. But from their intention, they are clear commands, that's why they belong into /usr/bin. IMHO, they are still commands even if for a technical reason instead of "exec" they are "."d. But now even this is no longer the case.

> who has just taken this discussion from a factual to a personal level

Isn't it understandable that I do not want to see my projects crippled?
I have put quite some effort in them and consider them very useful for supporting to write _compatible_ POSIX scripts and hope that many users actually _use_ them. If they cannot be used easily and compatible, they loose their meaning and become pointless.

And seeing that the original complaint - which still consider invalid, because it is only a _formal_ issue without any technical need - which nevertheless is now satisfied (_formally_ as is the original issue), is now replaced by the argument "but I don't like ..." really looks like a personal reason.

It might be that you don't like an upstream project or how it solves certain issues. But kicking it out for QA reasons is a strong thing, especially if the original complain is no longer valid.

> > become a case for COMREL.
> 
> Yes, I am interested too what they would have to say about this, so feel
> free to put them into CC.

I had always thought, COMREL is the last resort after we first tried everything else to solve the issue among us. Of course, it is clear that I will not accept without any available form of protest that my projects are being kicked out (or crippled to unusability), even less that I am being kicked out as mgorny hinted.

If you really intend to go that step, please put COMREL on CC.

But I really suggest that you all (I am also including mgorny here) to take a few days and think about whether it is really good to risk loosing somebody who had always supported gentoo in the past - not only with eix, but also e.g. through my overlay or by contributing a lot of patches to numerous projects and bugs - only for the purpose to safe some disputable philosophical (and apparently home-brewn) idea about "commands".

If after thinking in quite you really decide to go to these limits, I think there is no other way than to let COMREL decide the case.
Comment 19 Martin Väth 2016-11-01 19:47:47 UTC
After a PM discussion, it turned out that I had a horrible mistake in my memory: I had confused postings of another person with those of Ulrich.

The "bullying" I had mentioned was indeed not from Ulrich!
In fact, Ulrich's postings in the thread I had in mind were very objective.

I am very sorry about the wrong accusation which - concerning Ulrich - was completely inappropriate and only due to my own misremembering of the author of the postings I had meant.
Comment 20 Ulrich Müller gentoo-dev 2016-11-01 20:18:06 UTC
(In reply to Martin Väth from comment #19)

Apology accepted.
Comment 21 Michael Weber (RETIRED) gentoo-dev 2017-02-10 09:47:51 UTC
commit ccbd433e0262b4840c974ee01d828d44e89e393d
Author: Michael Weber <xmw@gentoo.org>
Date:   Fri Feb 10 10:10:06 2017 +0100

    app-shells/push: move push.sh into /usr/share/push (bug 598527).
    
    Package-Manager: Portage-2.3.3, Repoman-2.3.1

app-shells/push/push-2.0-r1.ebuild