Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 722666

Summary: emerge's speed can be improved in 3 ways, each massive
Product: Portage Development Reporter: Alexander Kern <trazytrace>
Component: CoreAssignee: Portage team <dev-portage>
Status: RESOLVED NEEDINFO    
Severity: enhancement CC: fturco, stijn+gentoo
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: All   
Whiteboard:
Package list:
Runtime testing required: ---

Description Alexander Kern 2020-05-12 10:39:57 UTC
Huge speed improvements possible:
1. by using other datastructures libraries
http://matthewrocklin.com/blog/work/2014/05/01/Fast-Data-Structures
1.1. using Numpy datastructures (huge speed improvement)
1.2 using CyToolz, even faster than Numpy
1.3. using Databases instead of Datastructures would also improve speed of portage emerge hugely

2. by not using brut force to solve dependencies, but that may be nothing new for you.

3. by make use of many cores of cpu's
3.1. maybe by using gpu cores, but that is maybe more insane than rational

Reproducible: Always

Steps to Reproduce:
emerge --search ....
emerge --searchdesc ....
emerge -whatsoever


Expected Results:  
to wait is even not neccessary
Comment 1 Enne Eziarc 2020-05-12 22:49:52 UTC
How have you actually benchmarked these changes? Show your work. Patches?

FWIW, Portage with the already existing sqlite backend doesn't show any appreciable speedup. How do you explain that?
Comment 2 Mike Gilbert gentoo-dev 2020-05-13 03:07:21 UTC
We would welcome patches to improve performance, but they are not going to write themselves. I would suggest sending some patches to the gentoo-portage-dev mailing list, along with some benchmarks to demonstrate their effect.