Summary: | Portage 2.1.2 really slow | ||
---|---|---|---|
Product: | Portage Development | Reporter: | Edoardo Dusi <mailtoedo> |
Component: | Core | Assignee: | Portage team <dev-portage> |
Status: | RESOLVED FIXED | ||
Severity: | normal | CC: | pageexec |
Priority: | High | Keywords: | InVCS |
Version: | 2.1 | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Bug Depends on: | |||
Bug Blocks: | 147007 |
Description
Edoardo Dusi
2006-12-23 10:01:12 UTC
The performance penalty in 2.1.2 relative to 2.1.1 is due to the addition of several important features: 1) reverse blocker detection (bug 128809) 2) complete dependency graph (bug 147766) 3) installed packages that do not have matching ebuilds can satisfy dependencies (bug 48195) The extra time taken for dependency calculations depends on the number of packages installed, so users with more packages will notice more of a performance difference. Given the improvements in dependency calculations, I think the performance level in 2.1.2_rc3-r9 is acceptable. I'll add an explanation of these issues to the release notes in order to help clarify the situation for users. (In reply to comment #1) > Given the improvements in dependency calculations, I think the performance > level in 2.1.2_rc3-r9 is acceptable. I'll add an explanation of these issues > to the release notes in order to help clarify the situation for users. I think the general behaviour of portage 2.1.2_rc3-r9 is good, but in some situation the slowness is not completely acceptable. For me and many other users this is an important aspect of a package manager. (In reply to comment #0) > I found that new 2.1.2 portage is really slow with the first 3-4 operations of > the day, so it seems like a caching problem. It must be some variation of bug 124041. I see that you have 3 overlays. Each time that you sync, it's possible that some of the eclasses have been modified. When ebuilds from the overlay are pulled into the dependency graph, their metadata cache needs to be regenerated if they inherit any modified eclasses. (In reply to comment #3) You're right, without the overlays portage is a little bit faster, although there's not so much improvement on performances. Anyway I want my overlays to stay in portage tree, so you say that portage will remain so slow as it is now? (In reply to comment #4) > without the overlays portage is a little bit faster If this is like bug 124041, then then it should only be slow when it's generating metadata cache. After the cache has been generated, there's no problem. Do you have any eclasses in your overlay? This type of thing is never an issue for people that have small overlays with no eclasses in them. They don't have to generate metadata cache because it's already generated on our master mirror and transferred when they sync. We can change portage to avoid triggering cache generation for ebuilds that match installed packages, but that means the the dependency metadata from live ebuilds will no longer be used for installed packages. That would probably be fine in most cases but portage has never behaved that way before. It's always used the live ebuild metadata (that was the root of bug 48195). Now it uses the live ebuild metadata and falls back to the installed metadata if there is no matching ebuild in the tree. (In reply to comment #5) > If this is like bug 124041, then then it should only be slow when it's > generating metadata cache. After the cache has been generated, there's no > problem. Do you have any eclasses in your overlay? No I don't have eclasses in my overlays. And anyway they're small, two of them are the two from gentoo-xeffects and the third is a local overlay for ebuilds not in portage tree, and there's only 3 ebuilds in it. The general performances of my machine are good, but look, for example, at the first emerge of the day I do: # time emerge -p kdebase These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild R ] kde-base/kdebase-3.5.5-r3 real 1m32.194s user 0m2.640s sys 0m1.000s Isn't it a lot of time? And the second time is much faster: # time emerge -p kdebase These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild R ] kde-base/kdebase-3.5.5-r3 real 0m2.237s user 0m2.040s sys 0m0.108s Maybe the first time there's much more I/O operations? In svn r5402 I've added a metadata cache for the installed packages. This greatly improves the performance for loading the metadata when the pagecache is cold. In svn r5410 I've added a blocker cache for installed packages so that the dependency data doesn't have to be re-evaluated for every emerge invocation. This has been released in 2.1.2_rc4-r2. If you're interested in measuring the performance differences, you may want to know that emerge now creates 2 cache files when it's run as a superuser: /var/cache/edb/vdb_metadata.pickle /var/cache/edb/vdb_blockers.pickle Both files are disposable, so it's okay to remove them for testing purposes. The vdb_metadata.pickle is the one that will greatly improve performance for the first invocation of emerge. See http://linux-mm.org/Drop_Caches usage of /proc/sys/vm/drop_caches which is useful for testing it's effectiveness. Note that vdb_blockers.pickle will be invalidated every time that the set of installed packages or virtuals change, but it doesn't take very long to regenerate it. After an installation, this cache will be invalid, but any installation command run as a superuser (such as `emerge -p portage`) will cause it to regenerate and all users will be able to use it while it remains valid. |