It takes emerge -av <pkg> over 1 minute to figure out <pkg> does not exist on my beefcake system. This behavior is suboptimal to say the least. I guess stat("/var/db/pkg/dev-python/pypax-0.9.0", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/beautifulsoup-3.2.1-r1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/pyopenssl-0.13.1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/markupsafe-0.23", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/twisted-web-13.0.0-r1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/zope-interface-4.1.1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/pygobject-3.12.2", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/simplejson-3.6.4", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/docutils-0.12", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/cement-2.0.2", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/paramiko-1.13.0", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/pyxdg-0.25-r1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/pycrypto-2.6.1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/pycurl-7.19.3.1-r2", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/python-fastimport-0.9.2", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/m2crypto-0.21.1-r2", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/jinja-2.7.3", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/ecdsa-0.11-r1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/pygments-1.6_p20140324-r1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/setuptools-7.0", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/pyxattr-0.5.3", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/ply-3.4-r1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/urlgrabber-3.9.1-r2", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/twisted-core-13.0.0-r3", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/dev-python/pycairo-1.10.0-r4", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 stat("/var/db/pkg/games-fps", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 openat(AT_FDCWD, "/var/db/pkg/games-fps", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 6 getdents(6, /* 3 entries */, 32768) = 88 getdents(6, /* 0 entries */, 32768) = 0 close(6) = 0 has something to do with it. Reproducible: Always Steps to Reproduce: 1. echo 3 > /proc/sys/vm/drop_caches 2. time emerge -av xdg-basedir # assuming you don't have haskell overlay enabled. Actual Results: total time of 1m9 on my system. Expected Results: total time under 200 miliseconds.
Note it also makes no sense that emerge is displaying "calculating dependencies" during all that time.
We can do away with all those calls shown in comment #0 if we assume that the top-level directory mtime is bumped when appropriate as discussed in bug 290428. It could cause problems if an alternative package manager such as Paludis has been used to install packages, though.
In order for vardbapi to take advantage of the mtime updates from bug 290428, it will have to store the mtimes for the top-level and category-level directories in vdb_metadata.pickle and vdb_metadata_delta.json. Then it can use those mtimes to trigger cache invalidation, and avoid redundant listdir/stat calls when the cache is valid.
