Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 647654 - sys-apps/portage: filter-bash-environment.py input is not buffered, it reads 1 byte at a time
Summary: sys-apps/portage: filter-bash-environment.py input is not buffered, it reads ...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - Ebuild Support (show other bugs)
Hardware: All All
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on:
Blocks: 651804
  Show dependency tree
 
Reported: 2018-02-14 19:50 UTC by Zac Medico
Modified: 2018-07-02 18:44 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Zac Medico gentoo-dev 2018-02-14 19:50:54 UTC
Using strace, we can see that iterating over lines with this code it reads 1 byte at a time:

	file_in = codecs.iterdecode(sys.stdin.buffer.raw,
		'utf_8', errors='replace')

This fixes it to use a read buffer:

	file_in = io.TextIOWrapper(sys.stdin.buffer.raw,
		'utf_8', errors='replace')
Comment 1 Zac Medico gentoo-dev 2018-02-14 19:59:21 UTC
Using sys.stdin.buffer instead of sys.stdin.buffer.raw also solves the problem:

	file_in = codecs.iterdecode(sys.stdin.buffer,
		'utf_8', errors='replace')
Comment 3 Larry the Git Cow gentoo-dev 2018-02-17 18:41:15 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=f1367a2eeb0c911a743b0da5f37a7e9f7de76488

commit f1367a2eeb0c911a743b0da5f37a7e9f7de76488
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-02-14 20:22:03 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-02-17 18:25:45 +0000

    filter-bash-environment.py: use buffered input, raw bytes (bug 647654)
    
    Use sys.stdin.buffer instead of sys.stdin.buffer.raw, for buffered input.
    Also use raw bytes instead of unicode strings, in order to avoid making
    assumptions about character encodings, and also to avoid overhead from
    unicode decoding/encoding.
    
    Since the % operator does not support bytes operands in python3.4, use
    the + operator to format strings of bytes.
    
    Bug: https://bugs.gentoo.org/647654

 bin/filter-bash-environment.py                |  47 +++++------
 pym/portage/tests/bin/test_filter_bash_env.py | 115 ++++++++++++++++++++++++++
 2 files changed, 137 insertions(+), 25 deletions(-)}
Comment 4 Zac Medico gentoo-dev 2018-07-02 18:44:00 UTC
Fixed in portage-2.3.40-r1.