Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 722666 - emerge's speed can be improved in 3 ways, each massive
Summary: emerge's speed can be improved in 3 ways, each massive
Status: RESOLVED NEEDINFO
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All All
: Normal enhancement (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-05-12 10:39 UTC by Alexander Kern
Modified: 2022-03-16 01:04 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alexander Kern 2020-05-12 10:39:57 UTC
Huge speed improvements possible:
1. by using other datastructures libraries
http://matthewrocklin.com/blog/work/2014/05/01/Fast-Data-Structures
1.1. using Numpy datastructures (huge speed improvement)
1.2 using CyToolz, even faster than Numpy
1.3. using Databases instead of Datastructures would also improve speed of portage emerge hugely

2. by not using brut force to solve dependencies, but that may be nothing new for you.

3. by make use of many cores of cpu's
3.1. maybe by using gpu cores, but that is maybe more insane than rational

Reproducible: Always

Steps to Reproduce:
emerge --search ....
emerge --searchdesc ....
emerge -whatsoever


Expected Results:  
to wait is even not neccessary
Comment 1 Enne Eziarc 2020-05-12 22:49:52 UTC
How have you actually benchmarked these changes? Show your work. Patches?

FWIW, Portage with the already existing sqlite backend doesn't show any appreciable speedup. How do you explain that?
Comment 2 Mike Gilbert gentoo-dev 2020-05-13 03:07:21 UTC
We would welcome patches to improve performance, but they are not going to write themselves. I would suggest sending some patches to the gentoo-portage-dev mailing list, along with some benchmarks to demonstrate their effect.