Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 660860 - [TRACKER] sys-apps/portage: use parallelization in dependency calculations
Summary: [TRACKER] sys-apps/portage: use parallelization in dependency calculations
Status: CONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All All
: Normal normal with 1 vote (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: Tracker
Depends on:
Blocks: 155723 835380
  Show dependency tree
 
Reported: 2018-07-10 17:23 UTC by Zac Medico
Modified: 2024-02-19 01:52 UTC (History)
11 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Zac Medico gentoo-dev 2018-07-10 17:23:53 UTC
Some opportunities for parallelization might include:

* parallel construction of package instances
* parallel backtracking
* parallel resolution of independent parts of the dependency graph
Comment 1 Zac Medico gentoo-dev 2021-05-02 19:14:43 UTC
Storing the dependency calculation in a shared DB might be a useful way to enable parallelization, by acting as means for concurrent processes to collaborate on a calculation.
Comment 2 Zac Medico gentoo-dev 2021-05-02 19:19:14 UTC
Also, a DB can make dependency calculations more scalable by removing the need to store the whole calculation in memory at once.
Comment 3 Zac Medico gentoo-dev 2021-05-02 20:01:21 UTC
LMDB is a candidate since it supports mmap, and we should beware that writemap=True with an undersized map_size value will trigger SIGBUS in concurrent processes: 

https://github.com/jnwatson/py-lmdb/issues/269#issuecomment-729750375

> What we have here is two bugs, one in py-lmdb and one in lmdb. The
> first bug is that a non-zero default value of map_size on Environment is
> inappropriate. Passing 0 is generally the right answer most of the time.
> 
> That bug triggers a second bug in the underlying lmdb where opening a
> database with write_map=True and explicitly specifying a map_size too
> small will ftruncate the file out from underneath another open process.
Comment 4 Zac Medico gentoo-dev 2024-02-19 01:52:36 UTC
Maybe we can use threading for this, since PEP 703 plans to make the GIL optional in CPython 3.13.