Summary: | sys-apps/portage: varbapi aux_update transactions with write-ahead logging | ||
---|---|---|---|
Product: | Portage Development | Reporter: | Zac Medico <zmedico> |
Component: | Core | Assignee: | Portage team <dev-portage> |
Status: | CONFIRMED --- | ||
Severity: | enhancement | CC: | sam |
Priority: | Normal | Keywords: | PATCH |
Version: | unspecified | ||
Hardware: | All | ||
OS: | All | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Bug Depends on: | |||
Bug Blocks: | 193766, 605082 | ||
Attachments: |
Add write ahead log
Add write ahead log using aux_update Add write ahead log using aux_update - fixed |
Description
Zac Medico
2018-11-25 13:15:20 UTC
Created attachment 557478 [details, diff]
Add write ahead log
Proposed an implementation of Zac's suggestion.
What do you think of this patch?
(In reply to Sam from comment #1) > Created attachment 557478 [details, diff] [details, diff] > Add write ahead log > > Proposed an implementation of Zac's suggestion. > > What do you think of this patch? Thanks, I'll take a look at this soon. I'm thinking that we'll probably want to encapsulate the transaction implementation in a plugin so that it's possible to choose an implementation that's optimal for the user's environment. For example, the repository storage framework has multiple implementations that are encapsulated in plugins: https://gitweb.gentoo.org/proj/portage.git/tree/lib/portage/repository/storage I would like to try AcidFS to see how it performs: https://acidfs.readthedocs.io/en/latest/ https://github.com/Pylons/acidfs A nice feature of AcidFS is that readers can use AcidFS to obtain a consistent view even while transactions are in progress. The history is stored as a git repository, and each reader access an immutable snapshot (git tree object) via the AcidFS API. (In reply to Sam from comment #1) > Created attachment 557478 [details, diff] [details, diff] > Add write ahead log > > Proposed an implementation of Zac's suggestion. > > What do you think of this patch? Please use the aux_update method to apply the transaction. Okay, I'll have a got at it. Grepping through dbapi/ I couldn't find calls to vardbapi's aux_update. Does that mean I can change it freely? Or are there other users of it? (In reply to Sam from comment #4) > Okay, I'll have a got at it. Great, thanks! > Grepping through dbapi/ I couldn't find calls to vardbapi's aux_update. Does > that mean I can change it freely? Or are there other users of it? I should remain compatible, since we do have some consumers. It's mainly used in lib/portage/dbapi/__init__.py to apply metadata updates for package moves an renames. It's also used in these files: bin/quickpkg lib/_emerge/FakeVartree.py lib/_emerge/PackageVirtualDbapi.py lib/_emerge/resolver/DbapiProvidesIndex.py Created attachment 563338 [details, diff] Add write ahead log using aux_update Right, I've got something I'm happy with that now makes use of aux_update and encapsulates both updates to CONTENTS and updates to CONTENTS_*. (I'm not sure whether it's getting a bit muddy whether changes are in scope of bug 605082 or this bug.) The attached patch contains the following: - Move startContentsUpdate(), stopContentsUpdate() to dblink ('pkg'). - Add abortContentsUpdate(), which performs roll-back, to dblink. - Create a copy of aux_update names aux_update_pkg() which resides in dblink as opposed to the still existing aux_update() that resides in vardbapi. As aux_update only operates on a dblink and wants to know a cpv so that it can immediately fetch a dblink made me comfortable doing this. But maybe there are concerns I'm not aware of that keep it in vardbapi. - Move writeContentsToContentsFile() to dblink - Change writeContentsToContentsFile() to also write CONTENTS_* files and to move writes to pattern of catch {start(),update(),stop()} except {abort()}. - Change vardbapi.removeFromContents() to: 1) grab data using pkg.getcontents(), pkg.getContentsIndices(), pkg.getContentsMetadata() 2) call pkg.writeContentsToContentsFile to write out the updates to CONTENTS and CONTENTS_* files. (BTW; should't removeFromContents() also be moved to dblink?) - Change pkg._add_preserve_libs_to_contents() to make use of new pkg.writeContentsToContentsFile. - Miscellaneous small changes. Created attachment 563340 [details, diff] Add write ahead log using aux_update - fixed Small fix. Patch now applies to the vartree.py after applying patch of bug 605082. |