When using portage utils, specifically "qmerge -Kp <package>", the tool is able to correctly build a list of dependencies needed for <package>. When removing the pretend switch, the tool only downloads binary package for <package> and not its dependencies and soon after starts screaming horrible red things. However, it passes all merges correctly and records the dependencies as merged, making a mess out of the system. Tried (and broken) with both portage-utils-0.1.29 and portage-utils-0.2. Reproducible: Always Steps to Reproduce:
OK, since no reply or rejection of this bug has happened, i dug into the problem a bit deeper, and it is obviously pretty simple. The basic qmerge utility logic is that commandline arguments are being parsed, then the function pkg_fetch() is called, which basically takes all non-getopt arguments, treats them as package atoms, checks/fetches the binary package, and calls pkg_merge() on each of them. This is where the main issue is, because pkg_merge() then calls itself directly recursively on all of the packages dependencies, but does not even attempt to fetch them. The call stack looks like pkg_fetch()->pkg_merge()->pkg_merge()->..., whereas any other but the first pkg_merge() is doomed to fail unless the packages have been manually downloaded prior to running this. My fix works very easilly. Since pkg_fetch() has the arguments handy for calling pkg_merge() i split pkg_fetch() into 1) pkg_process() which takes the original arguments, and 2) pkg_fetch() which takes the same arguments as pkg_merge(). New pkg_process() keeps the logic of taking every commandline argument, resolving the atom, doing some checks, and then calls new pkg_fetch(). New pkg_fetch() is basically the for-cycle content of original function that receives a package atom as an argument, checks/fetches the binary package, and calls pkg_merge(). The above described is a mere refactoring. The function had a bad name and was unnecessarily huge anyway. But it makes possible for a simple fix, changing the pkg_merge() recursion into indirect, always calling pkg_fetch() first. So the call stack looks like this: pkg_process()->pkg_fetch()->pkg_merge()->pkg_fetch()->pkg_merge()->... It is really not much of a change, hopefully breaks nothing, and certainly fixes the described problem.
Created attachment 188756 [details, diff] Fetch binary packages recursively And the patch
Taking bug.
Uhhhhh, sorry, I think I reported this to the wrong target :/. I mean, tools-portage@gentoo.org sounds almost like utils-portage@gentoo.org. Besides, it said something like Portage Utils <tools-portage@gentoo.org>.
'- Refactor fetching logic (Zdenek Behan <behanz[@]seznam[dot]cz>). bug 266455' /var/cvsroot/gentoo-projects/portage-utils/qmerge.c,v <-- qmerge.c new revision: 1.83; previous revision: 1.82
Zdenek, Thanks for the patch. If you feel motivated there is much more refactoring to be done. Mianly in the area of better handling/parsing of the rdeps that could be done. Ideally the tool would build a deptree before merging the first package vs it's linear approach now.
Well, yes, that's what I wanted to point out. That if you "qmerge package itsdependency", then itsdependency will be merged twice, which is quite unnecessary. But this kind of issue would take a lot more than a simple fix. I can look into it, but my motivation for the patch was more practical than that I would be going about being bored looking for things to fix. I was working on deploying gentoo to series of identical embedded boxes which are just too slow to build all the stuff themselves (emerge system takes just over a day in the more faster cases), and rebuilding a stage3 tarball periodically to avoid having to overcome too steep system updates seemed like too much of a hassle. Instead, I built binary packages for everything, packaged statically linked busybox and portage-utils into a tiny "binary bootstrap" tarball, which has just over few megabytes large, will not need to be updated for a _long_ time, and even the update is pretty much trivial and straightfoward (USE=static ROOT=whereto/ emerge busybox portage-utils), and which can be used to download all of portage and its dependencies very smoothly and using a tiny script, with nearly no requirements for the host system (just wget and chroot, really). This bug was the only issue in the way.
