In planning Portage2, we should make sure that we don't miss any of the good ideas that Sorcerer Linux had. I know their dependency system did things that Portage doesn't, including some sort of "self-healing" ? Seemant, could you (and other Sorcerer Linux people you know) compile a list of things that are "missing" from Portage?
OK, as you might be aware - Sorcerer GNU/Linux (SGL) has gone through some major changes in the recent past. Kyle Sallee, inventor and thinker extraordinaire, has quit the project. So, Sorcerer has now divided into two distributions: Lunar-Penguin and Sorcerer linux. For the past two months, I have been using exclusively Gentoo, so my Sorcerer partition was drastically out of date. Amongst other things, the zlib update had to be performed. So, I rebooted in to Sorcerer this evening after reading through the documentation for updating the sorcerer source url info for updating (socererlinux.org). That went simply -- just changed one field in one file, and I was good to go. --- So, some observations. When you issue `sorcery update` the system creates a queue of files to be updated. At the end of this detection process, you are prompted to change the queue if you wish. After that, the system proceeds. Here's the interesting thing. After downloading the first package in the queue, another process takes over downloading the second package, while the first process is busy with configure/compile/install of the first process. That was kind of nice. The next observation: the user has the choice to be a voyeur or not. That means that you can put the updating process into the foreground and watch it as it unfolds, or you can stick into the background and go off to do other things. In the sorcery menu, you have the choice being e-mailed the updates and changes made. Additionally, a compilation log is stored in /var/log/sorcery/compile/package.tar.bz2 and a a tar.bz2 package is always created by default and stored in /var/cache/sorcery/package-$ARCH-$MACHTYPE-linux-gnu.tar.bz2 And now for the coup de grace. This was the thing that you were referring to, and that I e-mailed drobbins about a while ago. The thing about repairing itself. When the time came for the zlib update, it installed the new version, and then prompted me whether I wanted it to upgrade all the packages on my system that had been statically compiled. In particular, my screen looked like this: Creating: /var/log/sorcery/compile/zlib-1.1.4.tar.bz2 Creating: /var/log/sorcery/install/zlib-1.1.4 Creating: /var/cache/sorcery/zlib-1.1.4-i686-pc-linux-gnu.tar.bz2 zlib will now recompile software that is statically linked against it. This may take some time. If you stop it, disturb it, your machine hangs, or whatever, you can issue a /var/lib/sorcery/grimoire/archive/zlib/POST_INSTALL. --- And of course, POST_INSTALL is just a shell script which checks through the installed packages on the system for static zlib compiles and goes ahead and recompiles them. The following is what it looks like: ---
show_from() { cd $INSTALL_LOGS grep $1 * } find_libs() { LIB_LIST=`find /usr/lib \ ! -regex '.*/doc/.*' ! -regex '.*/man/.*' ! -regex '.*/include/.*' \ ! -regex '.*/src/.*' ! -regex '.*/share/.*' ! -regex '.*/fonts/.*' \ ! -regex '.*/info/.*' ! -regex '.*/perl5/.*' ! -regex '.*/site-packages/ .*' \ ! -regex '.*\.so.*' \ -exec file {} \; \ | egrep "current ar archive|statically linked" \ | cut -d ":" -f 1-1` AFFECTED=`nm --print-file-name $LIB_LIST | egrep "deflate|inflate" | cut -d " :" -f 1-1 ` for TARGET in $AFFECTED; do echo $TARGET >> $TMPFILE; done SORTED=`cat $TMPFILE | sort | grep -v libz | uniq` for SPELL in $SORTED; do for INSTALLED_SPELL in $INSTALLED_SPELLS; do if [ `show_from $SPELL | cut -d ":" -f 1-1 | grep $INSTALLED_SPELL` ]; then cast $INSTALLED_SPELL; fi done done } main() { echo "zlib will now recompile software that is statically linked against it." echo "This may take some time. If you stopped it, disturbed it, your machine " echo "hangs or whatever, you can issue a /var/lib/sorcery/grimoire/archive/zl ib/POST_INSTALL " echo "at any time to recompile the affected spells." echo if query "Do you want to recompile affected spells now ?" y; then find_libs; rm $TMPFILE else echo "Not recompiling affected spells - your system will be insecure unt il you " echo "issue a /var/lib/sorcery/grimoire/archive/zlib/POST_INSTALL" fi } . /etc/sorcery/config TMPFILE=/tmp/zlibfind.`uuidgen` INSTALLED_SPELLS=`gaze installed | cut -d ":" -f 1-1` main $*
One more thing which Sorcerer does well, that we do not do at all yet. When you issue a `sorcery update`, the sorcery system checks the integrity of installed packages. I assume this to mean that it at least does checksumming on vital system packages, and then reinstalls those that don't match. I think this would make a nice addition to portage's emerge world or emerge --update world. One more thing in the same vein, is maybe have emerge do a quick checksum check on listed DEPEND items which are already installed. Twice, I have had to remerge dependencies which already existed on my system.
Ah yes, the other nice thing is that Sorcerer fetches packages in the background while compiling in the foreground. T'would seem to be a nice feature that we could add into portage almost immediately...
I just remembered something else. Some common binaries in sorcery, if you haven't got them installed, are such that if you call them, it automatically casts the package and then runs the thing you called. spiffy but unimportant.
Sorcerer sets an e-mail variable in its equivalent make.conf file. Basically, this variable is for the sys-admin to put their own email address in, and if they like logging or error messages or informational messages, those get sent via e-mail either from the system or from their delta server.
After reading some of the comments .. the system integrety one struck my mind .. is it now worth creating md5 checksums during emerge and storing them somewhere and having a integrety check in cron once a week (user can change it) and then notify you if any packages has changed ... sort of like a simple build in tripwire for gentoo ? Comments ? Henti
I've just read this bug report "changelog" and .. now that some time has passed.. are any of the nice things about sorcery linux being developed or integrated into portage? things like: - anouncing things like that zlib case seemant talked about. - background downloading while ./configure && make && make install. I believe we allready have the "consistency" tests.. at least I recall something similar with qpkg. Miguel
I have an additional request that could go in the same vein for portage2: When you do an ebuild, that it records the version of the compiler that built it, not only flags. This way one can build a function that lets you find packages not rebuilt with the existing version. I.e. do an upgrade from gcc 3.1 to gcc 3.2, and then do something like "emerge --empty --new-compiler world". This is very cumbersome today, as you only can do "emerge --empty world" and watch it in one sitting, but with the new functionality you can do a few packages here and there, nifty for us laptop users that cant leave our machines running...
here are a few screenshots of ghow Sorcerer Linux tackles the compilier optins selection at install time. (very poorly coded ncurses interface from my expierence though). But it would be nice if we had somethign similar. Note Sorcerer doesn't give and descriptions for none of the options, expects you to know what each one does. If we get something like this I would really like to see a help for most if not all of the various options because not all users are going to know what to select. Also not sure if the flags that are dispalyed are arch specicifc, but that would generally be a good idea, we wouldn't want someone chosing host i686-linux-pc-gnu and a compilier option that was specifically for a sparc.
Created attachment 10578 [details] tarball of a few screenshots fomr the Sorcerer Linux insrtall pertaining too gcc options
Read about the job queue of packages to emerge for pipelining downloads, and would like to make a suggestion. It would be nice if a queue is to be developed if portage managed simultaneous builds as well, rather than relying entirely on 'make -jN', which is somewhat flakey where recursive make is used. Currently portage doesn't use SMP or uni-processor with distcc helpers particularly efficiently. Some parts of the dependancy tree are independant, so parallel building can interleave configure, compiles and linking. There could be a queue of pending jobs, some depend on source fetch, others waiting on pre-requisite builds to complete. When their dependencies are fulfilled, they would be moved to the runnables. Allowing both source fetch, and build queues able to process more than one item at a time, prevents everything idling waiting on one very long and slow fetch, and allows the steps that have the source present to be built immediately, even before the first download has completed. If the queue, at core of new portage system, manages the locking required, with the UI parts of portage communicating with this core, through a protocool. Then portage would inherit protection against current multi-tasking problems, where more than one administrator manage machines and may accidentally emerge simulataneously, risking breaking portage on that box. Secondly the protocol could allow simpler building of GUI interfaces, as well as CLI, and leave open the possibility for remote management, should it be required.
Note about previous comment: the next full rewrite of Portage will have very robust job management throughout.
Also a feature that should be great to have in portage-ng its like sorcerer have, download xdelta's so a user doesnt need to download a full package (Its hard to update gcc, glibc and kde on a dialup) =). There was a glep (#9) about this but the one who proposed it dissapeared so i think it should be rescued or dropped.
sourcemage too
Please don't add more to this bug... Combo bugs take too much effort to read through. Create a new bug and link it here via depend/block.
*** Bug 64610 has been marked as a duplicate of this bug. ***
Closing as there is no specific goal for this bug and I'm pretty sure most if not all the ideas are also on separate bugs.
