Bugzilla Bug 1184

In planning Portage2, we should make sure that we don't miss
any of the good ideas that Sorcerer Linux had.  I know their
dependency system did things that Portage doesn't, including
some sort of "self-healing" ?

Seemant, could you (and other Sorcerer Linux people you know)
compile a list of things that are "missing" from Portage?

OK, as you might be aware - Sorcerer GNU/Linux (SGL) has gone through some
major
changes in the recent past.  Kyle Sallee, inventor and thinker extraordinaire,
has quit the project.  So, Sorcerer has now divided into two distributions:
Lunar-Penguin and Sorcerer linux. For the past two months, I have been using
exclusively Gentoo, so my Sorcerer partition was drastically out of date. 
Amongst other things, the zlib update had to be performed.  So, I rebooted in
to
Sorcerer this evening after reading through the documentation for updating the
sorcerer source url info for updating (socererlinux.org).  That went simply --
just changed one field in one file, and I was good to go.
---

So, some observations.  When you issue `sorcery update` the system creates a
queue of files to be updated.  At the end of this detection process, you are
prompted to change the queue if you wish.  After that, the system proceeds. 
Here's the interesting thing.  After downloading the first package in the
queue,
another process takes over downloading the second package, while the first
process is busy with configure/compile/install of the first process.  That was
kind of nice.

The next observation: the user has the choice to be a voyeur or not.  That
means
that you can put the updating process into the foreground and watch it as it
unfolds, or you can stick into the background and go off to do other things. 
In
the sorcery menu, you have the choice being e-mailed the updates and changes
made.  Additionally, a compilation log is stored in
/var/log/sorcery/compile/package.tar.bz2 and a a tar.bz2 package is always
created by default and stored in
/var/cache/sorcery/package-$ARCH-$MACHTYPE-linux-gnu.tar.bz2

And now for the coup de grace.  This was the thing that you were referring to,
and that I e-mailed drobbins about a while ago.  The thing about repairing
itself.  When the time came for the zlib update, it installed the new version,
and then prompted me whether I wanted it to upgrade all the packages on my
system that had been statically compiled.  In particular, my screen looked like
this:

Creating: /var/log/sorcery/compile/zlib-1.1.4.tar.bz2
Creating: /var/log/sorcery/install/zlib-1.1.4
Creating: /var/cache/sorcery/zlib-1.1.4-i686-pc-linux-gnu.tar.bz2

zlib will now recompile  software that is statically linked against it.  This
may take some time.  If you stop it, disturb it, your machine hangs, or
whatever, you can issue a /var/lib/sorcery/grimoire/archive/zlib/POST_INSTALL.
---

And of course, POST_INSTALL is just a shell script which checks through the
installed packages on the system for static zlib compiles and goes ahead and
recompiles them.  The following is what it looks like:

---

show_from()  {

     cd  $INSTALL_LOGS
     grep  $1  *

}

find_libs() {

   LIB_LIST=`find /usr/lib   \
        ! -regex '.*/doc/.*'  ! -regex '.*/man/.*'   ! -regex '.*/include/.*'  \
        ! -regex '.*/src/.*'  ! -regex '.*/share/.*' ! -regex '.*/fonts/.*'    \
        ! -regex '.*/info/.*' ! -regex '.*/perl5/.*' ! -regex '.*/site-packages/
.*' \
        ! -regex '.*\.so.*'   \
        -exec file {} \;     \
             |  egrep "current ar archive|statically linked" \
             |  cut -d ":" -f 1-1`
             
   AFFECTED=`nm --print-file-name $LIB_LIST | egrep "deflate|inflate" | cut -d "
:" -f 1-1 `

   for TARGET in $AFFECTED; do echo $TARGET >> $TMPFILE; done
   SORTED=`cat $TMPFILE | sort | grep -v libz | uniq`
   for SPELL in $SORTED; 
   do 
      for INSTALLED_SPELL in $INSTALLED_SPELLS;
      do
         if [ `show_from $SPELL | cut -d ":" -f 1-1 | grep $INSTALLED_SPELL` ]; 
then
             cast $INSTALLED_SPELL; 
         fi
      done
   done
}

main() {
   echo "zlib will now recompile software that is statically linked against it."
   echo "This may take some time. If you stopped it, disturbed it, your machine 
"
   echo "hangs or whatever, you can issue a /var/lib/sorcery/grimoire/archive/zl
ib/POST_INSTALL "
   echo "at any time to recompile the affected spells." 
   echo
   if query "Do you want to recompile affected spells now ?" y; then
      find_libs;
      rm $TMPFILE
   else echo "Not recompiling affected spells - your system will be insecure unt
il you "
        echo "issue a /var/lib/sorcery/grimoire/archive/zlib/POST_INSTALL"
   fi
}

. /etc/sorcery/config
TMPFILE=/tmp/zlibfind.`uuidgen`
INSTALLED_SPELLS=`gaze installed | cut -d ":" -f 1-1`
main  $*

One more thing which Sorcerer does well, that we do not do at all yet.  When
you
issue a `sorcery update`, the sorcery system checks the integrity of installed
packages.  I assume this to mean that it at least does checksumming on vital
system packages, and then reinstalls those that don't match.
I think this would make a nice addition to portage's emerge world or emerge
--update world. 

One more thing in the same vein, is maybe have emerge do a quick checksum check
on listed DEPEND items which are already installed.  Twice, I have had to
remerge dependencies which already existed on my system.

Ah yes, the other nice thing is that Sorcerer fetches packages in the
background
while compiling in the foreground.  T'would seem to be a nice feature that we
could add into portage almost immediately...

I just remembered something else.  Some common binaries in sorcery, if you
haven't got them installed, are such that if you call them, it automatically
casts the package and then runs the thing you called.  spiffy but unimportant.

Sorcerer sets an e-mail variable in its equivalent make.conf file.  Basically,
this variable is for the sys-admin to put their own email address in, and if
they like logging or error messages or informational messages, those get sent
via e-mail either from the system or from their delta server.

After reading some of the comments .. the system integrety one struck my 
mind .. 
is it now worth creating md5 checksums during emerge and storing them somewhere 
and having a integrety check in cron once a week (user can change it) and then 
notify you if any packages has changed ... sort of like a simple build in 
tripwire for gentoo ? 

Comments ? 

Henti 

I've just read this bug report "changelog" and .. now that some time has
passed..
are any of the nice things about sorcery linux being developed or integrated
into portage?
things like:
- anouncing things like that zlib case seemant talked about.
- background downloading while ./configure && make && make install. 

I believe we allready have the "consistency" tests.. at least I recall
something
similar with qpkg.

Miguel

I have an additional request that could go in the same vein for portage2:
                                                                                
When you do an ebuild, that it records the version of the compiler that built it, not only flags. This way one can build a function that lets you find packages
not rebuilt with the existing version.
                                                                                
I.e. do an upgrade from gcc 3.1 to gcc 3.2, and then do something like "emerge --empty --new-compiler world". This is very cumbersome today, as you only can do
"emerge --empty world" and watch it in one sitting, but with the new functionality you can do a few packages here and there, nifty for us laptop users that cant leave our machines running...

here are a few screenshots of ghow Sorcerer Linux tackles the compilier optins
selection at install time. (very poorly coded ncurses interface from my
expierence though). But it would be nice if we had somethign similar. Note
Sorcerer doesn't give and descriptions for none of the options, expects you to
know what each one does. If we get something like this I would really like to
see a help for most if not all of the various options because not all users are
going to know what to select. Also not sure if the flags that are dispalyed are
arch specicifc, but that would generally be a good idea, we wouldn't want
someone chosing host i686-linux-pc-gnu and a compilier option that was
specifically for a sparc.

Created an attachment (id=10578) [details]
tarball of a few screenshots fomr the Sorcerer Linux insrtall pertaining too
gcc options

Read about the job queue of packages to emerge for pipelining downloads, and 
would like to make a suggestion.  It would be nice if a queue is to be 
developed if portage managed simultaneous builds as well, rather than relying 
entirely on 'make -jN', which is somewhat flakey where recursive make is used.  
Currently portage doesn't use SMP or uni-processor with distcc helpers 
particularly efficiently. 
 
Some parts of the dependancy tree are independant, so parallel building can 
interleave configure, compiles and linking.  There could be a queue of pending 
jobs, some depend on source fetch, others waiting on pre-requisite builds to 
complete.  When their dependencies are fulfilled, they would be moved to the 
runnables.  Allowing both source fetch, and build queues able to process more 
than one item at a time, prevents everything idling waiting on one very long 
and slow fetch, and allows the steps that have the source present to be built 
immediately, even before the first download has completed. 
 
If the queue, at core of new portage system, manages the locking required, with 
the UI parts of portage communicating with this core, through a protocool.  
Then portage would inherit protection against current multi-tasking problems, 
where more than one administrator manage machines and may accidentally emerge 
simulataneously, risking breaking portage on that box.  Secondly the protocol 
could allow simpler building of GUI interfaces, as well as CLI, and leave open 
the possibility for remote  management, should it be required. 

Note about previous comment: the next full rewrite of Portage will have very
robust job management throughout.

Also a feature that should be great to have in portage-ng its like sorcerer
have, download xdelta's so a user doesnt need to download a full package (Its
hard to update gcc, glibc and kde on a dialup) =).

There was a glep (#9) about this but the one who proposed it dissapeared so i
think it should be rescued or dropped.

sourcemage too

Please don't add more to this bug...
Combo bugs take too much effort to read through.

Create a new bug and link it here via depend/block.

*** Bug 64610 has been marked as a duplicate of this bug. ***

Closing as there is no specific goal for this bug and I'm pretty sure most if
not all the ideas are also on separate bugs.