Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 90343 - sandbox logging of preinst and postinst file events
Summary: sandbox logging of preinst and postinst file events
Status: RESOLVED REMIND
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All All
: High enhancement (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 205312
  Show dependency tree
 
Reported: 2005-04-25 02:27 UTC by Zac Medico
Modified: 2009-02-12 07:39 UTC (History)
5 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
libsandsandbox file event logging support (sandbox-1.2.1.sbnotify.patch,7.29 KB, patch)
2005-04-25 02:30 UTC, Zac Medico
Details | Diff
Portage support for preinst and postinst file modification event logging (portage-2.0.51.20.sbnotify.patch,5.35 KB, patch)
2005-04-25 02:32 UTC, Zac Medico
Details | Diff
libsandbox file event logging support (sandbox-1.2.1.sbnotify.patch,7.12 KB, patch)
2005-04-25 10:52 UTC, Zac Medico
Details | Diff
sandbox-1.2.1.sbnotify.patch (sandbox-1.2.1.sbnotify.patch,7.53 KB, patch)
2005-04-26 22:48 UTC, Zac Medico
Details | Diff
portage-2.0.51.20.installwatch.patch (portage-2.0.51.20.installwatch.patch,5.96 KB, patch)
2005-04-27 17:55 UTC, Zac Medico
Details | Diff
portage-2.0.51.20.installwatch.patch (portage-2.0.51.20.installwatch.patch,9.56 KB, patch)
2005-04-28 21:38 UTC, Zac Medico
Details | Diff
orphan counts for 56 packages (report.txt,2.15 KB, text/plain)
2005-04-30 04:24 UTC, Zac Medico
Details
portage-2.0.51.21.installwatch.patch (portage-2.0.51.21.installwatch.patch,9.50 KB, patch)
2005-05-06 12:16 UTC, Zac Medico
Details | Diff
sandbox-1.2.11 redirect EINFO and EWARN to stderr (sandbox-1.2.11-printing-macro-stderr.patch,779 bytes, patch)
2005-07-21 19:32 UTC, Zac Medico
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Zac Medico gentoo-dev 2005-04-25 02:27:27 UTC
This feature is a libsandbox file event logging system (similar to InstallWatch) with convenient bitmask control over which kinds of events are logged.

It can be used to monitor live modification of system files during ebuild preinst and postinst.  The file modification logs are stored in the /var/db/pkg database as files named PREINST and POSTINST (similar to CONTENTS).  This feature is currently enabled only with "sbnotify" in portage FEATURES.

This allows many previously unaccounted files to be logged in the var db database so that their package ownership is known.  I am currently using these patches on my x86 desktop system and they seem to work well.  Here the first three lines from the openoffice-bin-1.9.95 POSTINST log:

147	/usr/share/applications/.mimeinfo.cache.TeJXQV	rename	0
267	/usr/share/applications/mimeinfo.cache	rename	0
35	/usr/share/mime/application/x-tgif.xml.new	fopen64	134587472
Comment 1 Zac Medico gentoo-dev 2005-04-25 02:30:23 UTC
Created attachment 57154 [details, diff]
libsandsandbox file event logging support
Comment 2 Zac Medico gentoo-dev 2005-04-25 02:32:07 UTC
Created attachment 57155 [details, diff]
Portage support for preinst and postinst file modification event logging
Comment 3 Zac Medico gentoo-dev 2005-04-25 02:35:46 UTC
Comment on attachment 57155 [details, diff]
Portage support for preinst and postinst file modification event logging

These patches are against sandbox-1.2.1-r3 adn portage-2.0.51.20-r4.
Comment 4 Jason Stubbs (RETIRED) gentoo-dev 2005-04-25 03:26:53 UTC
The patch looks quite good. A couple of small concerns though...

I haven't looked at the surrounding code yet, but is mode[1]=='+' okay? Is it guaranteed to fail before then if strlen(mode)==0?

Also, there's a couple of seemingly unrelated sections in the patches which I've copied below. What are the purpose of these?

--- sandbox-1.2.1.orig/libctest.c	2005-04-23 14:01:52.000000000 -0700
+++ sandbox-1.2.1.sbnotify/libctest.c	1969-12-31 16:00:00.000000000 -0800
@@ -1 +0,0 @@
-int main(void) { return 0; }

--- portage-2.0.51.20.orig/pym/portage.py	2005-04-23 00:39:43.000000000 -0700
+++ portage-2.0.51.20.sbnotify/pym/portage.py	2005-04-24 23:05:31.000000000 -0700
@@ -6514,13 +6520,6 @@
 			self.dbdir = self.dbtmpdir
 			print ">>> original instance of package unmerged safely."	
 
-		# We hold both directory locks.
-		self.dbdir = self.dbpkgdir
-		self.delete()
-		movefile(self.dbtmpdir, self.dbpkgdir, mysettings=self.settings)
-
-		self.unlockdb()
-
 		#write out our collection of md5sums
 		if cfgfiledict.has_key("IGNORE"):
 			del cfgfiledict["IGNORE"]
Comment 5 Zac Medico gentoo-dev 2005-04-25 10:52:27 UTC
Created attachment 57201 [details, diff]
libsandbox file event logging support

Okay, I think this is better:

int len = strlen(mode);
if ((len>0 && (mode[0]=='w' || mode[0]=='a')) || (len>1 && mode[1]=='+'))

libctest.c was just something unnecessary...

In portage.py the POSTINST log is written in self.dbtmpdir so this line should
go *after* postinst:

movefile(self.dbtmpdir, self.dbpkgdir, mysettings=self.settings)

I moved the whole block since it seemed to go together.  Is that okay?	It
seems to work.
Comment 6 Zac Medico gentoo-dev 2005-04-26 22:48:40 UTC
Created attachment 57355 [details, diff]
sandbox-1.2.1.sbnotify.patch

I've updated the libsandbox patch to parse the fopen mode with the same logic
as the preexising before_syscall_open_char() function.

This functonality would be quite useful if it is included in the main branch of
portage.  It would be of particular interest to those concerned about "cruft"
on their gentoo systems:

http://gentoo-wiki.com/TIP_Clean_Up_Cruft
Comment 7 Brian Harring gentoo-dev 2005-04-27 00:23:09 UTC
Or... we shoot the ebuild devs orphaning files to the fs?
Offhand, I'm not much for modifying the sandbox code to add this feature, simply because flipping on sandbox debug and greping the logs handles it already.  This is how confcache is implemented, for example.

Not much for feature anyways since it implies ebuild devs are doing things they (mostly) shouldn't :)
Comment 8 Zac Medico gentoo-dev 2005-04-27 17:55:35 UTC
Created attachment 57443 [details, diff]
portage-2.0.51.20.installwatch.patch

Thanks for the insight ferringb!  I've updated the portage patch to work with
sandbox debug (no sandbox patch necessary).  The feature is simply called
"installwatch" now.

True, ebuild devs (mostly) shouldn't orphan files in the fs.  But it happens
and maybe there are some circumstances where it's actually necessary. 
Particularly in postinstall where maybe the package needs to be installed
first?

This feature gives convenient feedback about ebuilds that create orphans.  When
possible the ebuild will be corrected.	In either case we will have a record of
the changes that were made.
Comment 9 Brian Harring gentoo-dev 2005-04-27 21:26:59 UTC
Like I said... what ebuilds are orpahning files?  Ebuilds that do so _should_ have a damn good reason, and be the exception (well aware the "right" view there may not be reality)
Comment 10 Jason Stubbs (RETIRED) gentoo-dev 2005-04-28 21:22:24 UTC
Portage itself was up until 2.0.51.20. Baselayout does. Shadow does. Many many do things that they shouldn't be in pkg_preinst and pkg_postinst.
Comment 11 Zac Medico gentoo-dev 2005-04-28 21:38:41 UTC
Created attachment 57542 [details, diff]
portage-2.0.51.20.installwatch.patch

I added some python code to clean up the sandbox debug log and format it
indentically to the CONTENTS files.

Right now I'm merging all my binary packages into a new root.  After that's
finished I'll do some cross referencing to weed out the files that are already
owned by other packages.  Then I'll make available some detailed statistics for
all the packages that I have installed...
Comment 12 Zac Medico gentoo-dev 2005-04-30 04:24:47 UTC
Created attachment 57669 [details]
orphan counts for 56 packages

In my data analysis I exclude all files that were touched by more than one
installation.  A tarball is available containing a complete list of all the
files used to compile the report:
http://home.socal.rr.com/zmedico/orphans.tar.bz2
Comment 13 Martin Schlemmer (RETIRED) gentoo-dev 2005-05-06 08:05:36 UTC
The question is still if we do care.  I am not too familiar with some of the other packages listed here, but most of the orphans by baselayout is because we _do_not_want_ those in CONTENTS (/etc/shadow, /etc/passwd, /etc/group, /etc/fstab, etc), as it creates problems (think many problems of novices overwriting /etc/passwd because there was an 'update' to it), or some needed empty directories being unmerged.
Comment 14 Zac Medico gentoo-dev 2005-05-06 12:16:46 UTC
Created attachment 58219 [details, diff]
portage-2.0.51.21.installwatch.patch

True, many of these files do not belong in CONTENTS.  That is why the patch
lists the files separately in PREINST and POSTINST.

The files detected by this feature fall into at least 3 categories:
1) Belongs to multiple packages or the whole system (/etc/passwd).
2) Belongs to a particular package and can be installed in the normal
src_install() (ebuild should be fixed).
3) Belongs to a specific package but needs to be installed during pkg_preinst()
or pkg_postinst().
Comment 15 Zac Medico gentoo-dev 2005-07-21 19:32:22 UTC
Created attachment 64027 [details, diff]
sandbox-1.2.11 redirect EINFO and EWARN to stderr

I'm attaching this here in case it's useful to anyone else who is using sandbox
logging.  It prevents sandbox from interferring with bash command substitution.
 This can be a major problem, for example, if it injects a file path into the
arguments of rm -fr.  I actually lost /bin/mktemp and /usr/bin/find this way
(maybe more)!
Comment 16 Martin Schlemmer (RETIRED) gentoo-dev 2005-07-22 01:16:05 UTC
It actually wont be a problem in normal environment without SANDBOX_DEBUG=1, but
yeah, I can understand the need to not have it to stdout.  Applied in svn.
Comment 17 Donnie Berkholz (RETIRED) gentoo-dev 2006-06-28 12:02:36 UTC
This seems really interesting. If this is in released sandbox now, perhaps the portage patch could be applied and this whole thing fixed?
Comment 18 Martin Schlemmer (RETIRED) gentoo-dev 2006-06-28 12:30:34 UTC
Personally I think its ugly (implementation wise):
- Not sure EINFO/EWARN should go to stderr by default - its not errors as such (nitpicking I know).  Besides, just set SANDBOX_VERBOSE=0 with newer sandbox.
- Really the big issue is loading libsandbox.so manually.  This makes the ties too close, and any radical internal changes in the future might break things.  I'd rather that it used the sandbox shell in some way, or fix that if it does not work.  NB: Please see the comment for SANDBOX_ACTIVE in the sources :/

Other possible issue that I am not sure about, is if it runs on all arch's now (haven't checked if its enabled by default, blah, blah ...).
Comment 19 Martin Schlemmer (RETIRED) gentoo-dev 2006-06-28 12:32:18 UTC
Hmm, I missed that I did apply the sandbox patch, but SANDBOX_VERBOSE will make things nice and clean.  I really just do not like manual preloading of libsandbox.so - that really needs to be looked at before adding it to portage.
Comment 20 Zac Medico gentoo-dev 2006-06-28 13:02:04 UTC
(In reply to comment #19)
> Hmm, I missed that I did apply the sandbox patch

As I mentioned in Comment #15, the stderr redirection is really quite essential.  Otherwise, shell substitutions can behave unexpectedly and cause bad thinks to happen.

(In reply to comment #19)
> I really just do not like manual preloading of
> libsandbox.so - that really needs to be looked at before adding it to portage.

A lot has changed since I wrote that patch (I wasn't a portage dev at the time).
Comment 21 Martin Schlemmer (RETIRED) gentoo-dev 2006-06-28 13:09:53 UTC
(In reply to comment #20)
> (In reply to comment #19)
> > Hmm, I missed that I did apply the sandbox patch
> 
> As I mentioned in Comment #15, the stderr redirection is really quite
> essential.  Otherwise, shell substitutions can behave unexpectedly and cause
> bad thinks to happen.
> 

Like I said - it is in latest version.  But also SANDBOX_VERBOSE, so just set that to 0, and there will be no output from sandbox (default is SANDBOX_VERBOSE=1).
Comment 22 SpanKY gentoo-dev 2009-02-12 07:27:32 UTC
so, is there any actual changes required here in sandbox ?  looks like sandbox-1.3 has all the verbose output you need ?
Comment 23 Zac Medico gentoo-dev 2009-02-12 07:39:14 UTC
(In reply to comment #22)
> so, is there any actual changes required here in sandbox ?  looks like
> sandbox-1.3 has all the verbose output you need ?

I haven't had a chance to look at it, and I've kind of lost interest in this feature. Thanks for taking care of the sandbox side. I guess I'll reassign it to dev-portage now since that's where remaining work would be.