portage/dbapi/porttree.py:309 seems to require the existence of metadata/cache before trying to create a pregen_auxdb from metadbmodule. I would like to experiment with the metadata cache residing in an external database, without a metadata/cache directory. Hence it would be nice if the metadbmodule would itself check for this directory's existance, instead of portdbapi doing it. This would allow for other modules without that requirement. Reproducible: Always
The metadata/cache directory an it's contents are specified by PMS, so it seems like you are extending the specification. In this case we'll probably want to add some kind of metadata to indicate that an extension is in use, perhaps using metadata/layout.conf as suggested in bug #331683, comment #5. Also, note that the portdbapi constructor currently passes the metadata/cache path to the portdbapi.metadbmodule constructor. Since you're experimenting, maybe you should just create the metadata/cache directory in the target repository. You could even put your cache data somewhere in there if you wanted to.
(In reply to comment #1) > The metadata/cache directory an it's contents are specified by PMS, so it seems > like you are extending the specification. I do not try to fiddle with the contents of that directory. It is merely that I have am trying to build/sync a pregenerated cache totally independend of this directory. Hence, to support different caching methods independend of a certain filesystem layout, I think it would be better for the metadbmodule to decide whether it can work on the given repository or not. > Since you're experimenting, maybe you should just create the metadata/cache > directory in the target repository. Yes, true. But I found it "bad" to create a dummy directory just to make portage load the metadbmodule.
(In reply to comment #2) > I do not try to fiddle with the contents of that directory. It is merely that I > have am trying to build/sync a pregenerated cache totally independend of this > directory. Hence, to support different caching methods independend of a certain > filesystem layout, I think it would be better for the metadbmodule to decide > whether it can work on the given repository or not. So, you want portage to instantiate a bunch of different modules so that each module can probe to see if it's cache format is available? Wouldn't it be a lot simpler if the repository specified the available formats in metadata/layout.conf?
(In reply to comment #3) > Wouldn't it be a lot simpler if the repository specified the available formats > in metadata/layout.conf? There are instances where the original repository contains no metadata (for various reason, e.g. because the maintainer does not care, or there is no infrastructure to reliably generate it, or because the repository comes directly from svn where metadata generation post-commit hooks are impractical, or ...). So it might be desired by the user to generate his own metadata independendly from the original repository.
(In reply to comment #4) > There are instances where the original repository contains no metadata (for > various reason, e.g. because the maintainer does not care, or there is no > infrastructure to reliably generate it, or because the repository comes > directly from svn where metadata generation post-commit hooks are impractical, > or ...). > So it might be desired by the user to generate his own metadata independendly > from the original repository. If the user is going to the trouble of generating metadata, then at the same time they can generate a metadata/layout.conf entry specifying which cache format(s) have been generated.
(In reply to comment #5) > If the user is going to the trouble of generating metadata, then at the same > time they can generate a metadata/layout.conf entry specifying which cache > format(s) have been generated. Agreed. Does this mean that auxdbmodule is selected from /etc/portage/modules, and metadbmodule comes from $REPOS/metadata/layout.conf? Is there a syntax for specifying it defined already? [1] suggests it is not, is that correct? [1] http://dev.gentoo.org/~zmedico/portage/doc/man/portage.5.html
Right, there's no syntax for it yet. Something like this should be fine: cache_formats = foo bar The "foo" and "bar" should be format identifiers that are independent of the class that implements them. So, portage will have an internal mapping that tells it which module to load in order to read format "foo". This mapping could be specified in /etc/portage/modules with settings like these: portdbapi.metadbmodule.foo = portage.cache.metadata_foo.database portdbapi.metadbmodule.bar = portage.cache.metadata_bar.database
(In reply to comment #7) > Right, there's no syntax for it yet. Something like this should be fine: > > cache_formats = foo bar When designing this, do not forget that some formats might require additional options, i.e. network addresses. This seems to fit there: --- cache_formats = foo foo.url = ... --- Or this: --- [cache] formats = foo [foo] url = ... ---
If the cache is distributed separately then you'll have to tag both the repo and the cache with UUIDs that you can compare them to make sure that they correspond to the same snapshot in time.
I have seen md5-cache recently. Does it implement this?
(In reply to comment #10) > I have seen md5-cache recently. Does it implement this? Yes, there's a cache-formats setting in metadata/layout.conf. The code is in pym/portage/repository/config.py.
The metadbmodule configuration that you mention in comment #0 was removed when we switched to using the metadata/layout.conf cache-formats in this commit: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=d4ea29bf6a3ce35d49e0f54f9173e3a6e42da2d6 In stable portage (2.1.11.51), the md5-cache format is now the default since this commit: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=e760c8d2a4ccc56e351ac37904c715f596b58e42
