Bug 660860 – [TRACKER] sys-apps/portage: use parallelization in dependency calculations

• Product: Portage Development
• Status: CONFIRMED ---

[Description Zac Medico 2018-07-10 17:23:53 UTC]

Some opportunities for parallelization might include:

* parallel construction of package instances
* parallel backtracking
* parallel resolution of independent parts of the dependency graph

[Comment 1 Zac Medico 2021-05-02 19:14:43 UTC]

Storing the dependency calculation in a shared DB might be a useful way to enable parallelization, by acting as means for concurrent processes to collaborate on a calculation.

[Comment 2 Zac Medico 2021-05-02 19:19:14 UTC]

Also, a DB can make dependency calculations more scalable by removing the need to store the whole calculation in memory at once.

[Comment 3 Zac Medico 2021-05-02 20:01:21 UTC]

LMDB is a candidate since it supports mmap, and we should beware that writemap=True with an undersized map_size value will trigger SIGBUS in concurrent processes: 

https://github.com/jnwatson/py-lmdb/issues/269#issuecomment-729750375

> What we have here is two bugs, one in py-lmdb and one in lmdb. The
> first bug is that a non-zero default value of map_size on Environment is
> inappropriate. Passing 0 is generally the right answer most of the time.
> 
> That bug triggers a second bug in the underlying lmdb where opening a
> database with write_map=True and explicitly specifying a map_size too
> small will ftruncate the file out from underneath another open process.

[Comment 4 Zac Medico 2024-02-19 01:52:36 UTC]

Maybe we can use threading for this, since PEP 703 plans to make the GIL optional in CPython 3.13.