If I use eix-update on a Raspberry Pi with 256M ram (almost 200M free), eix-diff now gets killed by the OOM killer. I realize that running on systems with that little memory isn't typical and might constrain the design greatly to workaround. However, it might just be an oversight and fixable. Just thought I'd report :) Reproducible: Always
The way eix-diff works is that it needs both full trees (with all packages, versions, flags, etc.) in memory. This might be implemented differently, but it would need a ground-up rewrite of eix-diff.cc and of a lot of function it uses. Volunteers are welcome ;)
In the eix git master on github there is now an experimental reimplementation of eix-diff (eix-0.32.0). A corresponding live ebuild is in the mv overlay.
Wow thanks Martin! I build from master, and did a run on the system I noticed the issue on and it works fine now. Watching the process in top (I know, not scientific but fine for relative comparison), eix-update ~20% RAM usage emerge --sync hard to say as it forks, but definitely under 20% eix-diff now ~4% RAM Thanks for this!
The main problem is whether the new algorithm is correct. In fact, I was surprised how complex the coding is to implement the apparently trivial problem to read and compare two sorted lists from two files in a step-by-step manner... It could easily be that I overlooked some of the many cases and in certain exceptional situations some package is missed to print (or even there is crash because of a wrong double free or something similar). Let me know if you realize such a case. Probably some weeks of testing are needed, before this version is released...
(In reply to Martin Väth from comment #4) > The main problem is whether the new algorithm is correct. I restructured the class so that now a diff algorithm can be used which is as simple and straightforward as one should expect. (In reply to David Flogeras from comment #3) > eix-update ~20% RAM usage This number is suspiciously low. The "correct" way to get the memory required by some program under linux is command time -f "%M kB" eix-update For the gentoo tree, it should be around 400-500 MB, so over your limit unless you use some swap file/partition.
Interesting, that threw me for a bit, until I realized you need to use gnu time, not the bash builtin. Well, on this desktop machine, you are right, it reports just north of 500M. I'm not sure what happens on the Pi, it certainly doesn't have a swap file. Is it possible that when it cannot allocate memory it behaves in a degraded fashion? Or worse, misbehaves? I'll test master again on the Pi when I'm back to it next week.
(In reply to David Flogeras from comment #6) > you need to use gnu time That's why I had written "command time ..." > Is it possible that when it cannot allocate memory it behaves in a degraded > fashion? Or worse, misbehaves? eix does not explicitly use any C++ exceptions, so I suppose eix would die with some signal if some "new" command fails (or the OOM killer of the kernel does this). I don't know why it works on Pi. Perhaps some C++ library function uses more cache (or preallocates larger data structures) if there is more total memory.
Hi Martin, I saw you released 0.32.1 so I re-tested on the 256M Pi board. The good news is eix-diff seems to keep its memory usage quite modest. The not bad, but weird news, is when I do the following on the target: command time -f "%M kB" eix-update It reports: 334080 kB While the system is as follows: KiB Mem : 238304 total KiB Swap: 0 That one I cannot figure out, I know that total resident memory isn't exactly the real high-water mark, but it's almost 100MiB more than the total system RAM. Also, I know top is also no comparable measure, but it reports eix-update using at most 35% of MEM. In the end it completes, doesn't throw any exceptions, and doesn't invoke the OOM. So I would say this bug can be closed if you agree.
commit ec32266d801bed8d8e9ed57d4af1380129fae663 Author: Michael Weber <xmw@gentoo.org> Date: Wed Apr 19 13:45:17 2017 +0200 app-portage/eix: Remove old version (bug 600648, bug 597380). Package-Manager: Portage-2.3.5, Repoman-2.3.2 RepoMan-Options: --include-arches="arm arm64 ppc ppc64" app-portage/eix/Manifest app-portage/eix/eix-0.31.7-r1.ebuild app-portage/eix/metadata.xml
