I am currently working on a project titled Gentoo for Pandora, which is creating bootable stage3(-like) tarballs for Pandora and similar devices, such as the BeagleBoard. We'd like to put /usr/portage into a squashfs file-system, but we want to do it in a way that is easily maintainable by the user, and is easily removed by simply running something like 'emerge -C sqaushfs-portage'. In order to maintain stability, we would like to be able to insert checks into the portage system that make sure everything is ship-shape before 'emerge' is allowed to continue. For example: is /usr/portage mounted? Currently, there is no way for us to insert such checks into portage, while keeping it all contained within an easily installable/removable package, without patching portage itself. My thoughts are, since portage already has this functionality partway hacked in (see /etc/portage/bin/post_sync, or http://www.mail-archive.com/gentoo-portage-dev@lists.gentoo.org/msg00523.html), why not go all the way and add an /etc/portage/hooks/ directory or similar? The basic idea is to simply check for the existence of {post,pre)-(run,sync,ebuild,configure,compile,install}.d directories (and maybe more, if anyone thinks of any) and execute each bash file within. I'm not sure what the procedure is for executing external files from within portage, but potentially these bash scripts could be given the ebuild.sh environment, if available, just like /etc/portage/bashrc. I've never really dealt with Portage development before now, so I'm not sure what your expectations are. I'm willing to write patches, and add Python to my programming repertoire. It doesn't look too hard... maybe. ;) Reproducible: Always Steps to Reproduce:
(The syntax errors in the summary line are bugging me.)
I still plan on writing the patch, by the way. I have not forgotten this bug. ;) In the meantime, I want to add a bit more to my explanation, since I get a lot of the same "why" questions, and I give this same answer: Having a ".d" directory, in which each hook is contained in its own file (for example: /etc/portage/hooks/pre-run.d/squashfs-portage) makes it extremely easy to install, update, and uninstall these rules without parsing monolithic bashrc files currently available in portage. It also fleshes out this already used, already partially-implemented feature into something that other developers could use for their own scripts in an organized fashion.
Not that hooks are a bad idea, but since you have as much control as you do, why not make a shell script wrapper around emerge that does what you want?
This is not easily compatible with software like eix, revdep-rebuild and anything else that might call emerge. I could move 'emerge' to 'emerge-bin' and then make 'emerge' a wrapper script, but if portage upgrades at all it'll overwrite that. The way around _that_ is an ebuild. In the end, the cleanest way is to have portage support this, rather than hack around it.
Update for the people who want to know: progress is being made. You can track it here: http://gitorious.org/neuvoo/portage Important files (so far) for those who have little time but some interest: http://gitorious.org/neuvoo/portage/blobs/neuvoo/trunk/pym/portage/hooks.py http://gitorious.org/neuvoo/portage/blobs/neuvoo/trunk/pym/portage/tests/hooks/test_HookDirectory.py I'll be posting patches here when I feel they are feature-complete and reasonably stable (as in, doesn't crash when I run my test suite and when I try it out in a chroot).
Created attachment 233213 [details, diff] Initial hooks patch, for review and comments The feature is reasonably implemented and stable. I would really appreciate review and commentary at this stage. While I will always maintain this feature in my own branch, the goal is to merge this into portage asap to ease maintenance burdens down the road. What has been implemented: * pre/post-sync * pre/post-run * pre-ebuild (but not post-ebuild yet, since there are over ten different ways an ebuild can leave the doebuild function). * a minimal test suite * documentation in both the man pages and docbook. I've attached a patch file using git's diffing utility. Once this is deemed acceptable, I'll upload a more precise patch (git-format-patch) that you guys can finally review and apply for good, and will include any additional changes I've made once I've put this feature through its paces.
(In reply to comment #6) > Created an attachment (id=233213) [details] > Initial hooks patch, for review and comments As I look at this, I realized I may not be clear in my intent. I still need to proof-read the documentation, and go over each change to be sure I'm not introducing totally unnecessary changes. This patch is supposed to be a rough draft of what I expect to be submitting in a week or so. However, comments about minor flaws are still accepted, because I might've missed something. :)
Created attachment 233889 [details, diff] Final hooks patch, ready for comments and commit If you guys would rather push in a generic patch into git, here is the full patch, which is diffed against the latest commit, b8be7aff8677b021dc01d24ed8772891f7a3db02.
Created attachment 233891 [details] Final hooks git-format-patch, ready for comments and commit This a .tar.bz2 file with all the git-format-patch files, which, when applied via git, retain author name and commit messages, which may be useful when reviewing this code at a future date.
Created attachment 233991 [details] Final hooks git-format-patch with small pre-run fix, ready for comments and commit The fix is in the last patch in this tarball. pre-run scripts are primarily meant to check or set up things before portage runs, so it makes sense to re-read settings after pre-run scripts execute.
Created attachment 233993 [details, diff] Final hooks patch, with small pre-run fix, ready for comments and commit
In order to accommodate people who want access to the ebuild's bash environment, I think we should be implementing the hooks on the bash side. I've been wanting to add a registration interface for ebuild phase hooks, and expose it to eclasses in a new EAPI. Something like this: add_phase_hook before pkg_setup my_pre_pkg_setup Something like this will be useful for moving eclass initialization code out of global scope. They can call the registration function (such as add_phase_hook) from global scope, and the code will be executed later. The sync hook is unrelated to the ebuild phase hooks, so the implementation and usage will be entirely different. Note that /etc/portage/bin/post_sync is already supported (portage-utils currently installs a shell script there to implement /etc/portage/postsync.d).
(In reply to comment #12) > In order to accommodate people who want access to the ebuild's bash > environment, I think we should be implementing the hooks on the bash side. I've > been wanting to add a registration interface for ebuild phase hooks, and expose > it to eclasses in a new EAPI. Something like this: > > add_phase_hook before pkg_setup my_pre_pkg_setup > > Something like this will be useful for moving eclass initialization code out of > global scope. They can call the registration function (such as add_phase_hook) > from global scope, and the code will be executed later. I'm leery of breaking up a unit of functionality (hooks) into multiple implementations based on execution time. At the same time, I want to keep code duplication to a minimum. I can understand wanting to modify the ebuild environment, though, and at this point I think it totally makes sense to push more of the hooks code into bash so ebuild.sh can also execute them. I'm going to do the following: * Any hook can modify the global environment portage is reading, and therefore ebuilds too. * pre-/post-ebuild hooks are executed within ebuild.sh, so they will have full access to ebuild.sh's (and therefore the ebuild's) local environment. * Hooks need to be able to communicate the changes to the environment back to python in some cases. I'm thinking it's easiest if the hooks simply modify their own environments, and then hooks code can read the changes and communicate that back to python. You mentioned you are working on some IPC features, which the hooks feature could use. The modifications could be sent via IPC, once IPC is ready. In the meantime, I will have the hooks code simply read bash variables to communicate the changes back to python. The IPC implementation can be added later, and if this is done right, hooks will never have to worry about API changes. > The sync hook is unrelated to the ebuild phase hooks, so the implementation and > usage will be entirely different. Note that /etc/portage/bin/post_sync is > already supported (portage-utils currently installs a shell script there to > implement /etc/portage/postsync.d). Since hook implementation will be as generic as possible, they will always be able to modify the environment portage is running in, even the sync ones, but it doesn't make much of a difference to portage anyway. The existing postsync.d implementation operates exactly the same way the hooks do, so portage-utils can move to the new hooks functionality by simply changing directories. The old postsync.d functionality hasn't been removed by myself. I highly recommend deprecation once the patch is applied, and I can even extend the hooks code to manage that old directory so code is still moving forward, but I'll leave that to you guys to decide. So, summary: * hooks need to be able to modify the bash environment. Non-ebuild hooks will modify the global environment, and ebuild hooks will modify the per-ebuild environment. * hooks will always communicate changes to the environment using the same API, but the API will first point to simple bash variable parsing until IPC is done. * You guys can decide what to do with existing postsync.d functionality. It can peacefully co-exist, but I highly recommend we deprecate it.
I don't know what you cases you intend to use the IPC for. The idea seems a alien to me because it seems like usually when people talk about hooks, everything that they need to do can be done from within the ebuild environment. I guess if you need some way to escape privilege dropping or the sandbox, then IPC would make sense for that. If your IPC mechanism only needs to be uni-directions, from bash to python, you can simply write messages into temporary files, for python to read when the phase exits (similar to how elog works).
(In reply to comment #14) > I don't know what you cases you intend to use the IPC for. The idea seems a > alien to me because it seems like usually when people talk about hooks, > everything that they need to do can be done from within the ebuild environment. That's true. So, I guess I'll just stick with... > I guess if you need some way to escape privilege dropping or the sandbox, then > IPC would make sense for that. If your IPC mechanism only needs to be > uni-directions, from bash to python, you can simply write messages into > temporary files, for python to read when the phase exits (similar to how elog > works). ...this idea. I've already implemented it for saving variables back into portage's settings dictionary. Thanks for your help! :)
Alright, the requested additions have been made. I've tested it as best as I can. I'll be happy to fix any bugs that come up from this bug, too. The latest can be had from the Neuvoo portage git repo: http://gitorious.org/neuvoo/portage
Created attachment 238177 [details, diff] Added phase hook changes, removed hackish bash->python API, and updated SUPPORTED_FEATURES Here's a patch that represents the current differences between official/master and neuvoo/portage-hooks in git. If you like the changes made in this patch, let me know what format you guys need this in, as I'm still fuzzy on that. (Or you can merge the revisions by simply merging with neuvoo/portage-hooks: http://gitorious.org/neuvoo/portage )
Created attachment 238181 [details, diff] Added phase hook changes, removed hackish bash->python API, and updated SUPPORTED_FEATURES Apologies for the noise. I found some small, harmless cleanups that I had left out when undoing bash->python API. I've reviewed this patch once again to make sure all the right changes are done.
Created attachment 238191 [details, diff] Added version check to ebuild phase hooks As requested by zmedico, a version check has been added to the ebuild phase hooks so future versions of this feature can be backwards compatible, or at least safely check for older versions.
Created attachment 238193 [details, diff] Added version check to ebuild phase hooks Forgot ebuild_phase existed. ;) Alright, ready for review and comments once again.
Comment on attachment 238193 [details, diff] Added version check to ebuild phase hooks Alright, I'm going to stop publishing updates to this bug (again). I've been a bit confused about what the best medium for reviewing changes should be. Everything will be updated here from here on out: http://gitorious.org/neuvoo/portage (branch neuvoo/portage-hooks)
I'd like to do another review of the code before this goes into trunk so I'm not a trouble-maker. Trunk/master has changed enough to cause issues already.
Status update: review still planned. Being a college student, free time is a precious thing. I hope to have this feature ready before September.
The feature is considered complete on my end, and ready for review.
Created attachment 299457 [details, diff] portage hooks diff This was created by asking git to diff the portage hooks branch against the official trunk. This functionality has been in use for at least 3 months by Neuvoo developers for some time now, to implement seamless squashfs support for /usr/portage. This has been fantastic to have for our embedded development, and it's a good use-case example for hooks. (Other ideas Neuvoo developers have been mulling over to add as hooks for portage are on-demand binary packages, automatic compile failure reports, patch overlays (versus ebuild overlays), etc.) The idea behind the hooks feature is to make it easy to develop additions to official portage, either for end-users or developers, without worrying about maintaining a separate portage branch and overlay. This patch remains backwards compatible with the /etc/portage/postsync.d directory that portage already has, but I recommend that the folder eventually be deprecated and replaced by /etc/portage/hooks/post-sync.d/ if this patch is accepted. So, long story short, pretty please accept this patch, with a cherry on top. ;)
Created attachment 299463 [details, diff] portage hooks diff The previous attachment was a .tar.bz2 of "git format-patch", which was accidentally selected. I really meant to upload this .diff instead. If you want this diff in some other format, let me know. I also forgot to re-mention that the git repo can be found here: gitorious.org/neuvoo/portage
Chromiumos inherits portage from Gentoo and the /etc/portage/hooks/install/ functionality exists within Chromiumos. Chromiumos uses portage version: Portage 2.1.10.11-r9. Adding an executable shell file in /etc/portage/hooks/install/ is executed in the install stage of the package and variables like ${D}, etc. are available in the shell file. However, functions like eqawarn are not available (though die is available). Would it be possible to export those functions and make them available to the install hook file?
(In reply to comment #27) > Chromiumos inherits portage from Gentoo and the /etc/portage/hooks/install/ > functionality exists within Chromiumos. Chromiumos uses portage version: > > Portage 2.1.10.11-r9. > > Adding an executable shell file in /etc/portage/hooks/install/ is executed > in the install stage of the package and variables like ${D}, etc. are > available in the shell file. However, functions like eqawarn are not > available (though die is available). Would it be possible to export those > functions and make them available to the install hook file? The hooks/install functionality was written by ferringb. I've CC'ed him in this bug. You may want to file a new bug report, since this bug is about a new feature, and your comment is about an existing feature.
(In reply to comment #27) > Chromiumos inherits portage from Gentoo and the /etc/portage/hooks/install/ > functionality exists within Chromiumos. Chromiumos uses portage version: > > Portage 2.1.10.11-r9. > > Adding an executable shell file in /etc/portage/hooks/install/ is executed > in the install stage of the package and variables like ${D}, etc. are > available in the shell file. However, functions like eqawarn are not > available (though die is available). Those hooks are simply treated as executables, so they don't inherit any functions, though they should inherit all helpers available in $PATH which is supposed to include eqawarn. You can see that 2.1.10.11 is supposed to have an eqawarn helper in the list of files in the ebuild-helpers directory here: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=tree;f=bin/ebuild-helpers;h=c542d5603863af620e5c202da3ce9923eec60d0d;hb=0ecf14f926554f1dab0e0bf577696d8e3d949da2 Note that 'die' is in that directory, so if you have die like you said, then you should also have eqawarn. Therefore, it's confusing that you report that die is available while eqawarn is not. > Would it be possible to export those > functions and make them available to the install hook file? I suppose that we could call them via the bash's 'source' command them if they have a shebang that looks something like '#!/bin/bash' or '#!/usr/bin/env bash'. Then they would inherit all functions in bash environment. Do you really need that though, considering that eqawarn should already be available?
