Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 618042 - genlop fails with could not determine nature of (nonASCII) /var/log/emerge.log due to too many open files
Summary: genlop fails with could not determine nature of (nonASCII) /var/log/emerge.lo...
Status: UNCONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Tools (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage Tools Team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-05-09 20:02 UTC by clemenswaechter
Modified: 2022-12-13 11:37 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description clemenswaechter 2017-05-09 20:02:17 UTC
Recently I wanted to estimate the time needed for a full system build.
My system consists of ~1400 packages but genlop complained about emerge.log being nonASCII. 

Using strace revealed that genlop re-opens the emerge.log for each package which fails after the 1023th open file:

# emerge -ep @world | (strace genlop -p ) &> strace-genlop
# grep /var/log/emerge.log strace-genlop
open("/var/log/emerge.log", O_RDONLY)   = 1021
open("/var/log/emerge.log", O_RDONLY)   = 1021
stat("/var/log/emerge.log", {st_mode=S_IFREG|0660, st_size=1234588, ...}) = 0
open("/var/log/emerge.log", O_RDONLY)   = 1022
open("/var/log/emerge.log", O_RDONLY)   = 1022
stat("/var/log/emerge.log", {st_mode=S_IFREG|0660, st_size=1234588, ...}) = 0
open("/var/log/emerge.log", O_RDONLY)   = 1023
open("/var/log/emerge.log", O_RDONLY)   = 1023
stat("/var/log/emerge.log", {st_mode=S_IFREG|0660, st_size=1234588, ...}) = 0
open("/var/log/emerge.log", O_RDONLY)   = -1 EMFILE (Too many open files)
write(2, "could not determine nature of (n"..., 114could not determine nature of (nonASCII) /var/log/emerge.log at /usr/bin/genlop line 128, <_GEN_1020> line 13005.

This error can be prevented by setting a higher pre-process file limit using ulimit -n, nevertheless genlop should not need to open the log per entry and/or report the error correctly.
Comment 1 Nuno 2018-04-26 19:02:22 UTC
I have the same issue.


# emerge -ep @world | (strace genlop -p ) &> strace-genlop
# grep /var/log/emerge.log strace-genlop | tail -n 5
open("/var/log/emerge.log", O_RDONLY)   = 1023
open("/var/log/emerge.log", O_RDONLY)   = 1023
stat("/var/log/emerge.log", {st_mode=S_IFREG|0660, st_size=6950839, ...}) = 0
open("/var/log/emerge.log", O_RDONLY)   = -1 EMFILE (Too many open files)
write(2, "could not determine nature of (n"..., 114could not determine nature of (nonASCII) /var/log/emerge.log at /usr/bin/genlop line
128, <_GEN_1020> line 76024.
# ulimit -n
1024


`emerge -ep @world` gives 1301 packages to (re)emerge.
I also noticed that the error happens with packages that have never been emerged before (and thus are not in emerge.log).

Increasing the file limit to 2048 (as instructed by clemenswaechter) works.
Comment 2 Eduardo Coutinho Scalabrin 2018-08-10 17:02:22 UTC
I have the same error here using app-portage/genlop-0.30.9-r1, I did it 'emerge -pve1 @world | genlop --pretend' and I had it 'could not determine nature of (nonASCII) /var/log/emerge.log at /usr/bin/genlop line 128, <_GEN_1020> line 24712.'. I have installed on my system 2216 packages.
Comment 3 Vincent de Phily 2022-12-13 11:37:27 UTC
https://bugs.gentoo.org/289724 would probably fix this, by not reopening and reparsing the log file for each pretended merge. Note that without that patch, `genlop -p` for 1000+ packages is prohibitively slow anyway unless your emerge.log is tiny.

Genlop seems unmaintained at this stage (8 years since last commit, many open bugs), you might have better luck with emlop or qlop.