Bug 647654 – sys-apps/portage: filter-bash-environment.py input is not buffered, it reads 1 byte at a time

• Product: Portage Development
• Status: RESOLVED FIXED

[Description Zac Medico 2018-02-14 19:50:54 UTC]

Using strace, we can see that iterating over lines with this code it reads 1 byte at a time:

	file_in = codecs.iterdecode(sys.stdin.buffer.raw,
		'utf_8', errors='replace')

This fixes it to use a read buffer:

	file_in = io.TextIOWrapper(sys.stdin.buffer.raw,
		'utf_8', errors='replace')

[Comment 1 Zac Medico 2018-02-14 19:59:21 UTC]

Using sys.stdin.buffer instead of sys.stdin.buffer.raw also solves the problem:

	file_in = codecs.iterdecode(sys.stdin.buffer,
		'utf_8', errors='replace')

[Comment 2 Zac Medico 2018-02-14 21:07:29 UTC]

Patch posted for review:

https://archives.gentoo.org/gentoo-portage-dev/message/5dcbb57f7b7fe8717ef9843909b296ba
https://github.com/gentoo/portage/pull/256

[Comment 3 Larry the Git Cow 2018-02-17 18:41:15 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=f1367a2eeb0c911a743b0da5f37a7e9f7de76488

commit f1367a2eeb0c911a743b0da5f37a7e9f7de76488
Author:     Zac Medico <[e-mail]>
AuthorDate: 2018-02-14 20:22:03 +0000
Commit:     Zac Medico <[e-mail]>
CommitDate: 2018-02-17 18:25:45 +0000

    filter-bash-environment.py: use buffered input, raw bytes (bug 647654)
    
    Use sys.stdin.buffer instead of sys.stdin.buffer.raw, for buffered input.
    Also use raw bytes instead of unicode strings, in order to avoid making
    assumptions about character encodings, and also to avoid overhead from
    unicode decoding/encoding.
    
    Since the % operator does not support bytes operands in python3.4, use
    the + operator to format strings of bytes.
    
    Bug: https://bugs.gentoo.org/647654

 bin/filter-bash-environment.py                |  47 +++++------
 pym/portage/tests/bin/test_filter_bash_env.py | 115 ++++++++++++++++++++++++++
 2 files changed, 137 insertions(+), 25 deletions(-)}

[Comment 4 Zac Medico 2018-07-02 18:44:00 UTC]

Fixed in portage-2.3.40-r1.