Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 1184 - Ideas from Sorcerer Linux
Summary: Ideas from Sorcerer Linux
Status: RESOLVED CANTFIX
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Conceptual/Abstract Ideas (show other bugs)
Hardware: x86 Linux
: High enhancement (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
: 64610 (view as bug list)
Depends on:
Blocks: 1661 4552 8767
  Show dependency tree
 
Reported: 2002-03-16 15:11 UTC by Grant Goodyear (RETIRED)
Modified: 2004-10-04 06:56 UTC (History)
14 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
tarball of a few screenshots fomr the Sorcerer Linux insrtall pertaining too gcc options (gcc-options.tar.bz2,75.76 KB, application/octet-stream)
2003-04-13 03:30 UTC, teri-aka TrAns13nT
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Grant Goodyear (RETIRED) gentoo-dev 2002-03-16 15:11:29 UTC
In planning Portage2, we should make sure that we don't miss
any of the good ideas that Sorcerer Linux had.  I know their
dependency system did things that Portage doesn't, including
some sort of "self-healing" ?

Seemant, could you (and other Sorcerer Linux people you know)
compile a list of things that are "missing" from Portage?
Comment 1 Seemant Kulleen (RETIRED) gentoo-dev 2002-03-17 02:27:35 UTC
OK, as you might be aware - Sorcerer GNU/Linux (SGL) has gone through some major
changes in the recent past.  Kyle Sallee, inventor and thinker extraordinaire,
has quit the project.  So, Sorcerer has now divided into two distributions:
Lunar-Penguin and Sorcerer linux. For the past two months, I have been using
exclusively Gentoo, so my Sorcerer partition was drastically out of date. 
Amongst other things, the zlib update had to be performed.  So, I rebooted in to
Sorcerer this evening after reading through the documentation for updating the
sorcerer source url info for updating (socererlinux.org).  That went simply --
just changed one field in one file, and I was good to go.
---

So, some observations.  When you issue `sorcery update` the system creates a
queue of files to be updated.  At the end of this detection process, you are
prompted to change the queue if you wish.  After that, the system proceeds. 
Here's the interesting thing.  After downloading the first package in the queue,
another process takes over downloading the second package, while the first
process is busy with configure/compile/install of the first process.  That was
kind of nice.

The next observation: the user has the choice to be a voyeur or not.  That means
that you can put the updating process into the foreground and watch it as it
unfolds, or you can stick into the background and go off to do other things.  In
the sorcery menu, you have the choice being e-mailed the updates and changes
made.  Additionally, a compilation log is stored in
/var/log/sorcery/compile/package.tar.bz2 and a a tar.bz2 package is always
created by default and stored in
/var/cache/sorcery/package-$ARCH-$MACHTYPE-linux-gnu.tar.bz2

And now for the coup de grace.  This was the thing that you were referring to,
and that I e-mailed drobbins about a while ago.  The thing about repairing
itself.  When the time came for the zlib update, it installed the new version,
and then prompted me whether I wanted it to upgrade all the packages on my
system that had been statically compiled.  In particular, my screen looked like
this:

Creating: /var/log/sorcery/compile/zlib-1.1.4.tar.bz2
Creating: /var/log/sorcery/install/zlib-1.1.4
Creating: /var/cache/sorcery/zlib-1.1.4-i686-pc-linux-gnu.tar.bz2

zlib will now recompile  software that is statically linked against it.  This
may take some time.  If you stop it, disturb it, your machine hangs, or
whatever, you can issue a /var/lib/sorcery/grimoire/archive/zlib/POST_INSTALL.
---

And of course, POST_INSTALL is just a shell script which checks through the
installed packages on the system for static zlib compiles and goes ahead and
recompiles them.  The following is what it looks like:

---






Comment 2 Seemant Kulleen (RETIRED) gentoo-dev 2002-03-17 02:27:46 UTC
show_from()  {

     cd  $INSTALL_LOGS
     grep  $1  *

}

find_libs() {

   LIB_LIST=`find /usr/lib   \
        ! -regex '.*/doc/.*'  ! -regex '.*/man/.*'   ! -regex '.*/include/.*'  \
        ! -regex '.*/src/.*'  ! -regex '.*/share/.*' ! -regex '.*/fonts/.*'    \
        ! -regex '.*/info/.*' ! -regex '.*/perl5/.*' ! -regex '.*/site-packages/
.*' \
        ! -regex '.*\.so.*'   \
        -exec file {} \;     \
             |  egrep "current ar archive|statically linked" \
             |  cut -d ":" -f 1-1`
             
   AFFECTED=`nm --print-file-name $LIB_LIST | egrep "deflate|inflate" | cut -d "
:" -f 1-1 `

   for TARGET in $AFFECTED; do echo $TARGET >> $TMPFILE; done
   SORTED=`cat $TMPFILE | sort | grep -v libz | uniq`
   for SPELL in $SORTED; 
   do 
      for INSTALLED_SPELL in $INSTALLED_SPELLS;
      do
         if [ `show_from $SPELL | cut -d ":" -f 1-1 | grep $INSTALLED_SPELL` ]; 
then
             cast $INSTALLED_SPELL; 
         fi
      done
   done
}

main() {
   echo "zlib will now recompile software that is statically linked against it."
   echo "This may take some time. If you stopped it, disturbed it, your machine 
"
   echo "hangs or whatever, you can issue a /var/lib/sorcery/grimoire/archive/zl
ib/POST_INSTALL "
   echo "at any time to recompile the affected spells." 
   echo
   if query "Do you want to recompile affected spells now ?" y; then
      find_libs;
      rm $TMPFILE
   else echo "Not recompiling affected spells - your system will be insecure unt
il you "
        echo "issue a /var/lib/sorcery/grimoire/archive/zlib/POST_INSTALL"
   fi
}

. /etc/sorcery/config
TMPFILE=/tmp/zlibfind.`uuidgen`
INSTALLED_SPELLS=`gaze installed | cut -d ":" -f 1-1`
main  $*
Comment 3 Seemant Kulleen (RETIRED) gentoo-dev 2002-04-03 17:02:31 UTC
One more thing which Sorcerer does well, that we do not do at all yet.  When you
issue a `sorcery update`, the sorcery system checks the integrity of installed
packages.  I assume this to mean that it at least does checksumming on vital
system packages, and then reinstalls those that don't match.
I think this would make a nice addition to portage's emerge world or emerge
--update world. 

One more thing in the same vein, is maybe have emerge do a quick checksum check
on listed DEPEND items which are already installed.  Twice, I have had to
remerge dependencies which already existed on my system.
Comment 4 Seemant Kulleen (RETIRED) gentoo-dev 2002-04-10 16:56:16 UTC
Ah yes, the other nice thing is that Sorcerer fetches packages in the background
while compiling in the foreground.  T'would seem to be a nice feature that we
could add into portage almost immediately...
Comment 5 Seemant Kulleen (RETIRED) gentoo-dev 2002-07-03 21:12:57 UTC
I just remembered something else.  Some common binaries in sorcery, if you
haven't got them installed, are such that if you call them, it automatically
casts the package and then runs the thing you called.  spiffy but unimportant.
Comment 6 Seemant Kulleen (RETIRED) gentoo-dev 2002-09-08 17:22:38 UTC
Sorcerer sets an e-mail variable in its equivalent make.conf file.  Basically,
this variable is for the sys-admin to put their own email address in, and if
they like logging or error messages or informational messages, those get sent
via e-mail either from the system or from their delta server.
Comment 7 Henti Smith 2002-09-22 11:55:03 UTC
After reading some of the comments .. the system integrety one struck my 
mind .. 
is it now worth creating md5 checksums during emerge and storing them somewhere 
and having a integrety check in cron once a week (user can change it) and then 
notify you if any packages has changed ... sort of like a simple build in 
tripwire for gentoo ? 

Comments ? 

Henti 
Comment 8 Miguel Sousa Filipe 2002-12-17 22:43:31 UTC
I've just read this bug report "changelog" and .. now that some time has passed..
are any of the nice things about sorcery linux being developed or integrated
into portage?
things like:
- anouncing things like that zlib case seemant talked about.
- background downloading while ./configure && make && make install. 

I believe we allready have the "consistency" tests.. at least I recall something
similar with qpkg.

Miguel
Comment 9 Alexander Hoogerhuis 2003-02-02 23:29:45 UTC
I have an additional request that could go in the same vein for portage2:
                                                                                
When you do an ebuild, that it records the version of the compiler that built it, not only flags. This way one can build a function that lets you find packages
not rebuilt with the existing version.
                                                                                
I.e. do an upgrade from gcc 3.1 to gcc 3.2, and then do something like "emerge --empty --new-compiler world". This is very cumbersome today, as you only can do
"emerge --empty world" and watch it in one sitting, but with the new functionality you can do a few packages here and there, nifty for us laptop users that cant leave our machines running...
Comment 10 teri-aka TrAns13nT 2003-04-13 03:16:51 UTC
here are a few screenshots of ghow Sorcerer Linux tackles the compilier optins selection at install time. (very poorly coded ncurses interface from my expierence though). But it would be nice if we had somethign similar. Note Sorcerer doesn't give and descriptions for none of the options, expects you to know what each one does. If we get something like this I would really like to see a help for most if not all of the various options because not all users are going to know what to select. Also not sure if the flags that are dispalyed are arch specicifc, but that would generally be a good idea, we wouldn't want someone chosing host i686-linux-pc-gnu and a compilier option that was specifically for a sparc.
Comment 11 teri-aka TrAns13nT 2003-04-13 03:30:50 UTC
Created attachment 10578 [details]
tarball of a few screenshots fomr the Sorcerer Linux insrtall pertaining too gcc options
Comment 12 Rob Davies 2003-09-09 09:07:45 UTC
Read about the job queue of packages to emerge for pipelining downloads, and 
would like to make a suggestion.  It would be nice if a queue is to be 
developed if portage managed simultaneous builds as well, rather than relying 
entirely on 'make -jN', which is somewhat flakey where recursive make is used.  
Currently portage doesn't use SMP or uni-processor with distcc helpers 
particularly efficiently. 
 
Some parts of the dependancy tree are independant, so parallel building can 
interleave configure, compiles and linking.  There could be a queue of pending 
jobs, some depend on source fetch, others waiting on pre-requisite builds to 
complete.  When their dependencies are fulfilled, they would be moved to the 
runnables.  Allowing both source fetch, and build queues able to process more 
than one item at a time, prevents everything idling waiting on one very long 
and slow fetch, and allows the steps that have the source present to be built 
immediately, even before the first download has completed. 
 
If the queue, at core of new portage system, manages the locking required, with 
the UI parts of portage communicating with this core, through a protocool.  
Then portage would inherit protection against current multi-tasking problems, 
where more than one administrator manage machines and may accidentally emerge 
simulataneously, risking breaking portage on that box.  Secondly the protocol 
could allow simpler building of GUI interfaces, as well as CLI, and leave open 
the possibility for remote  management, should it be required. 
Comment 13 Daniel Robbins (RETIRED) gentoo-dev 2003-09-09 14:22:31 UTC
Note about previous comment: the next full rewrite of Portage will have very robust job management throughout.
Comment 14 Adrian Almenar 2004-02-09 07:44:38 UTC
Also a feature that should be great to have in portage-ng its like sorcerer have, download xdelta's so a user doesnt need to download a full package (Its hard to update gcc, glibc and kde on a dialup) =).

There was a glep (#9) about this but the one who proposed it dissapeared so i think it should be rescued or dropped.
Comment 15 Aaron Peterson 2004-03-01 02:16:29 UTC
sourcemage too
Comment 16 Nicholas Jones (RETIRED) gentoo-dev 2004-04-11 16:05:23 UTC
Please don't add more to this bug...
Combo bugs take too much effort to read through.

Create a new bug and link it here via depend/block.
Comment 17 SpanKY gentoo-dev 2004-09-19 15:08:50 UTC
*** Bug 64610 has been marked as a duplicate of this bug. ***
Comment 18 Jason Stubbs (RETIRED) gentoo-dev 2004-10-04 06:56:40 UTC
Closing as there is no specific goal for this bug and I'm pretty sure most if not all the ideas are also on separate bugs.