Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 787008

Summary: [Future EAPI] consider whether to enable 'lastpipe' shopt by default
Product: Gentoo Hosted Projects Reporter: Zac Medico <zmedico>
Component: PMS/EAPIAssignee: PMS/EAPI <pms>
Status: RESOLVED WONTFIX    
Severity: normal CC: chutzpah, esigra, flow, mgorny, sam
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: All   
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 174380    

Description Zac Medico gentoo-dev 2021-04-29 23:09:09 UTC
The lastpipe shopt is seems superior to <() in the sense that lastpipe returns exit status while <() always returns zero as demonstrated here:


$ (read -r; echo $REPLY) < <(echo hi; false); echo $?
hi
0
$ shopt -s lastpipe
$ echo hi | (read -r; echo $REPLY; false); echo $?
hi
1

Cons of having the lastpipe shopt automatically enabled by a future EAPI may include:

1) It could lead to confusion if lastpipe is silently enabled. People may ignore the lastpipe feature and use <() instead.

2) Since shell options are global, all bash code (including eclass functions) must be compatible with lastpipe. Maybe it's not that hard to be compatible though.
Comment 1 Ulrich Müller gentoo-dev 2021-04-30 06:52:26 UTC
(In reply to Zac Medico from comment #0)
> $ shopt -s lastpipe
> $ echo hi | (read -r; echo $REPLY; false); echo $?
> hi
> 1

Not sure if I understand the intention. Why is lastpipe needed there? The return status is 1 even if it isn't enabled:

   $ shopt lastpipe
   lastpipe        off
   $ echo hi | (read -r; echo $REPLY; false); echo $?
   hi
   1

(Also, why a subshell? echo hi | { read -r; echo $REPLY; false; } appears to do the same.)
Comment 2 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2021-04-30 07:58:14 UTC
The point of <() is really to avoid moving the meat into a subshell.

  printf '%s\n' foo bar baz | (while read x; do foo+=($x); done)

isn't going to work.
Comment 3 Zac Medico gentoo-dev 2021-04-30 17:12:20 UTC
(In reply to Ulrich Müller from comment #1)
> (In reply to Zac Medico from comment #0)
> > $ shopt -s lastpipe
> > $ echo hi | (read -r; echo $REPLY; false); echo $?
> > hi
> > 1
> 
> Not sure if I understand the intention. Why is lastpipe needed there? The
> return status is 1 even if it isn't enabled:

I suspect that some people will prefer the lastpipe style where the input source appears first rather than last. Here's a practical example applied to portage/bin/install-qa-check.d/05double-D:

> diff --git a/bin/install-qa-check.d/05double-D b/bin/install-qa-check.d/05double-D
> index 4b7737c0f..2554eb706 100644
> --- a/bin/install-qa-check.d/05double-D
> +++ b/bin/install-qa-check.d/05double-D
> @@ -1,15 +1,18 @@
>  # Check for accidential install into ${D}/${D}
>  
>  DD_check() {
> +	local lastpipe_state=$(shopt -p lastpipe)
> +	shopt -s lastpipe
>  	if [[ -d ${D%/}${D} ]] ; then
>  		eqawarn "QA Notice: files installed in \${D}/\${D}:"
>  		local files=()
> -		while read -r -d $'\0' i ; do
> +		find "${D%/}${D}" -print0 | while read -r -d $'\0' i ; do
>  			files+=( "${i#${D%/}${D}}" )
> -		done < <(find "${D%/}${D}" -print0)
> +		done
>  		eqatag -v double-D "${files[@]/#//}"
>  		die "Aborting due to QA concerns: ${#files[@]} files installed in ${D%/}${D}"
>  	fi
> +	eval "${lastpipe_state}"
>  }
>  
>  DD_check



>    $ shopt lastpipe
>    lastpipe        off
>    $ echo hi | (read -r; echo $REPLY; false); echo $?
>    hi
>    1
> 
> (Also, why a subshell? echo hi | { read -r; echo $REPLY; false; } appears to
> do the same.)

It's a very contrived example, so please refer to my practical example.

(In reply to Michał Górny from comment #2)
> The point of <() is really to avoid moving the meat into a subshell.
> 
>   printf '%s\n' foo bar baz | (while read x; do foo+=($x); done)
> 
> isn't going to work.

Yeah, please refer to the above refactor of portage/bin/install-qa-check.d/05double-D.
Comment 4 Zac Medico gentoo-dev 2021-04-30 17:20:12 UTC
(In reply to Zac Medico from comment #3)
> (In reply to Ulrich Müller from comment #1)
> > (In reply to Zac Medico from comment #0)
> > > $ shopt -s lastpipe
> > > $ echo hi | (read -r; echo $REPLY; false); echo $?
> > > hi
> > > 1
> > 
> > Not sure if I understand the intention. Why is lastpipe needed there? The
> > return status is 1 even if it isn't enabled:
> 
> I suspect that some people will prefer the lastpipe style where the input
> source appears first rather than last. Here's a practical example applied to
> portage/bin/install-qa-check.d/05double-D:
> 
> diff --git a/bin/install-qa-check.d/05double-D b/bin/install-qa-check.d/05double-D
> index 4b7737c0f..2554eb706 100644
> --- a/bin/install-qa-check.d/05double-D
> +++ b/bin/install-qa-check.d/05double-D
> @@ -1,15 +1,18 @@
>  # Check for accidential install into ${D}/${D}
>  
>  DD_check() {
> +	local lastpipe_state=$(shopt -p lastpipe)
> +	shopt -s lastpipe
>  	if [[ -d ${D%/}${D} ]] ; then
>  		eqawarn "QA Notice: files installed in \${D}/\${D}:"
>  		local files=()
> -		while read -r -d $'\0' i ; do
> +		find "${D%/}${D}" -print0 | while read -r -d $'\0' i ; do
>  			files+=( "${i#${D%/}${D}}" )
> -		done < <(find "${D%/}${D}" -print0)
> +		done
>  		eqatag -v double-D "${files[@]/#//}"
>  		die "Aborting due to QA concerns: ${#files[@]} files installed in ${D%/}${D}"
>  	fi
> +	eval "${lastpipe_state}"
>  }
>  
>  DD_check

To clarify the "superiority" mentioned in comment #0, we can use ${PIPESTATUS[@]} to check the status of the find command after the loop has completed.
Comment 5 Zac Medico gentoo-dev 2021-05-01 22:42:35 UTC
Since checking return values is important (it's the basis of all robust code), we could probably ban <() on the grounds that lastpipe is provably superior.
Comment 6 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2021-05-01 23:18:27 UTC
(In reply to Zac Medico from comment #5)
> Since checking return values is important (it's the basis of all robust
> code), we could probably ban <() on the grounds that lastpipe is provably
> superior.

No, we shouldn't.  You should use <(... || die).
Comment 7 Zac Medico gentoo-dev 2021-05-01 23:24:24 UTC
(In reply to Michał Górny from comment #6)
> (In reply to Zac Medico from comment #5)
> > Since checking return values is important (it's the basis of all robust
> > code), we could probably ban <() on the grounds that lastpipe is provably
> > superior.
> 
> No, we shouldn't.  You should use <(... || die).

That sounds reasonable, and I suppose that static analysis QA tools could assert that the <() status is checked in some way?
Comment 8 Ulrich Müller gentoo-dev 2021-05-02 07:44:38 UTC
(In reply to Zac Medico from comment #5)
> Since checking return values is important (it's the basis of all robust
> code), we could probably ban <() on the grounds that lastpipe is provably
> superior.

Banning <() is a separate issue (though lastpipe would be a prerequisite for it). Maybe we shouldn't discuss it here because it would complicate things. In any case, it would be an add-on QA policy but not be part of the spec.

Interestingly, POSIX doesn't specify the behaviour:
"each command of a multi-command pipeline is in a subshell environment; as an extension, however, any or all commands in a pipeline may be executed in the current environment."
https://pubs.opengroup.org/onlinepubs/7990989775/xcu/chap2.html#tag_001_012

So, are there any known adverse effects of enabling lastpipe?
Comment 9 Zac Medico gentoo-dev 2021-05-12 16:57:25 UTC
(In reply to Ulrich Müller from comment #8)
> Interestingly, POSIX doesn't specify the behaviour:
> "each command of a multi-command pipeline is in a subshell environment; as
> an extension, however, any or all commands in a pipeline may be executed in
> the current environment."
> https://pubs.opengroup.org/onlinepubs/7990989775/xcu/chap2.html#tag_001_012
> 
> So, are there any known adverse effects of enabling lastpipe?

The behavior of the ${PIPESTATUS[@]} is identical, so the difference is that variables from the last shell in the pipeline will persist while variables from the first shell in the pipeline will we lost. This has potential to cause confusion if lastpipe is silently enabled.
Comment 10 Zac Medico gentoo-dev 2021-05-22 21:29:58 UTC
(In reply to Zac Medico from comment #9)
> (In reply to Ulrich Müller from comment #8)
> > Interestingly, POSIX doesn't specify the behaviour:
> > "each command of a multi-command pipeline is in a subshell environment; as
> > an extension, however, any or all commands in a pipeline may be executed in
> > the current environment."
> > https://pubs.opengroup.org/onlinepubs/7990989775/xcu/chap2.html#tag_001_012
> > 
> > So, are there any known adverse effects of enabling lastpipe?
> 
> The behavior of the ${PIPESTATUS[@]} is identical, so the difference is that
> variables from the last shell in the pipeline will persist while variables
> from the first shell in the pipeline will we lost. This has potential to
> cause confusion if lastpipe is silently enabled.

Lately I've been using lastpipe in long-running event-driven bash scripts, and what I've found is that lastpipe may or may not be desired for any particular pipe. So, we can't expect a global lastpipe default to satisfy all use cases.
Comment 11 Zac Medico gentoo-dev 2021-05-22 21:33:45 UTC
(In reply to Zac Medico from comment #10)
> (In reply to Zac Medico from comment #9)
> > (In reply to Ulrich Müller from comment #8)
> > > Interestingly, POSIX doesn't specify the behaviour:
> > > "each command of a multi-command pipeline is in a subshell environment; as
> > > an extension, however, any or all commands in a pipeline may be executed in
> > > the current environment."
> > > https://pubs.opengroup.org/onlinepubs/7990989775/xcu/chap2.html#tag_001_012
> > > 
> > > So, are there any known adverse effects of enabling lastpipe?
> > 
> > The behavior of the ${PIPESTATUS[@]} is identical, so the difference is that
> > variables from the last shell in the pipeline will persist while variables
> > from the first shell in the pipeline will we lost. This has potential to
> > cause confusion if lastpipe is silently enabled.
> 
> Lately I've been using lastpipe in long-running event-driven bash scripts,
> and what I've found is that lastpipe may or may not be desired for any
> particular pipe. So, we can't expect a global lastpipe default to satisfy
> all use cases.

The decision about whether or not to use lastpipe for a particular pipe is governed  by whether or not the main shell is producing or consuming data.
Comment 12 Ulrich Müller gentoo-dev 2021-05-22 22:02:57 UTC
(In reply to Zac Medico from comment #10)
> Lately I've been using lastpipe in long-running event-driven bash scripts,
> and what I've found is that lastpipe may or may not be desired for any
> particular pipe. So, we can't expect a global lastpipe default to satisfy
> all use cases.

Closing, as discussed in #-portage.