Summary: | AutoRdependencies in ebuild | ||
---|---|---|---|
Product: | Portage Development | Reporter: | Francisco Gimeno <kikov> |
Component: | Sandbox | Assignee: | Portage team <dev-portage> |
Status: | RESOLVED INVALID | ||
Severity: | enhancement | CC: | azarah, dev-portage, kikov |
Priority: | High | Keywords: | Inclusion |
Version: | unspecified | ||
Hardware: | x86 | ||
OS: | Linux | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Attachments: |
Patch to get full dependencies on compiling
Small update patch for libsandbox.c to support tracing ebuild.sh changes to use the libsandbox patch |
Description
Francisco Gimeno
2002-05-28 21:00:24 UTC
Related, and mentioning here so I don't forget, would be to let sandbox write down all files that were read during the compile and in that way being able to tell what the building dependencies are Im not going to get into if I think this is needed or not, but can we either get it resolved or closed ? Ok, then I propose that we get it resolved :) I don't have time to work on this, but I think it is too important, so I don't want to close it. Anyway, some pointers in implementing this: To be totally sure of what dependencies were used in building a package, you need to keep a list of all files that are opened. The easiest way may be to just start strace -f -topen from within ebuild.sh, and then generate a list of used files from the output. The best way would be to have sandbox keep a hash of opened files and dump that at the end of the run. Takers welcome. Update: Here's how to get a package list that can help you write a DEPEND line for an ebuild: (using distcc as an example) Generate the list of files: # strace -qf -esignal=\!all -eopen -o distcc.out /usr/lib/portage/bin/ebuild /usr/portage/sys-devel/distcc/distcc-2.0.1-r1.ebuild compile [...] output deleted Get the unique list of successfully opened files that are important, and find out which packages they are from: # awk -F\" '/open/&&!/ENOENT/&&$2!~/\/tmp\//&&$2~/^\//{files[$2]++}END{for(f in files){print f}}' distcc.out | /usr/lib/portage/bin/find-packages baselayout-1.8.6.4-r1 fileutils-4.1.11 gcc-3.2.2-r1 glibc-2.3.2 popt-1.7-r1 portage-2.0.47-r10 zlib-1.1.4-r1 (zlib is probably a portage dependency or something like that. Running strace within ebuild.sh should give better results) And the RDEPEND could get hints from (after install) # find /var/tmp/portage/distcc-2.0.1-r1/image -type f -perm -100 | /usr/lib/portage/bin/find-requires|/usr/lib/portage/bin/find-packages glibc-2.3.2 popt-1.7-r1 To get more accurate results, the strace calls should be included in ebuild.sh. To do that, in the dyn_compile function, it should probably attach strace to the current process (strace -p $$ ...), run pkg_compile, and kill the strace. This could be controlled by a --find-depends option. The RDEPEND stuff could be run from the dyn_install function. It would be nicer if it was sandbox writing down the files, since it knows what files are inside the sandbox and which not, and since it would increase the speed. So here's a quick patch that makes ebuild tell you what packages were used during the compile. It still needs to be switched on conditionally, but it's a good first step, I'd say. I found for instance that distcc needs linux-headers to compile, which is missing from the DEPEND line. See, it already does useful stuff ;) Note that find-packages needs to have the little arrow removed in the grep for this to work properly. No idea why it's there. Now it just searches for symlinks. Also note that I needed to add exec tracing as well. So, please, someone take this ball and run with it. Still needed: - option --find-depends in the ebuild executable, to make it conditional - saving the results in a file - the abovementioned method to get an RDEPEND approximation - make it nicely integrated with portage. --- ebuild.sh.orig 2003-05-04 23:59:47.000000000 +0200 +++ ebuild.sh 2003-05-05 00:10:56.000000000 +0200 @@ -650,7 +650,12 @@ #scripts, so set it to $T. export TMP="${T}" export TMPDIR="${T}" - src_compile + strace -qf -e signal=\!all -e open,execve -o "${T}/openedfiles.log" -p $$ & + local strace_pid=$! + src_compile + kill $strace_pid + ewarn Packages that were used for compilation: + awk -F\" '!/ENOENT/&&$2!~/\/tmp\//&&$2~/^\//{files[$2]++}END{for(f in files){print f}}' "${T}/openedfiles.log" | /usr/lib/portage/bin/find-packages #|| abort_compile "fail" cd ${BUILDDIR} touch .compiled Created attachment 11510 [details, diff]
Patch to get full dependencies on compiling
Looks like I'm spending time on it after all :)
This patch doesn't need find-packages any more and is lots faster and more
accurate.
Still To Do:
- integrate in ebuild with a --find-deps option. Currently, you define
FINDDEPS=true.
- find the rdepends on installation
- save the results in a file, like COMPILE_DEPENDS or something like that.
- bugtest and tell people to use it/make the GRP builds use it.
Created attachment 11561 [details, diff]
Small update
Two little edits to make stuff more clear.
Created attachment 11872 [details, diff]
patch for libsandbox.c to support tracing
This will enable SANDBOX_TRACE and SANDBOX_TRACE_LOG to create the filelist
that is normally created with strace.
Advantages: paths are normalized, you only see the important paths, it's faster
than strace, and it doesn't use strace.
My feeling about this patch is that the code is very stable. Of course, testing
can never hurt :)
Carpaski, if you don't want to apply it, can you at least add the documentation
comments I added in front?
Created attachment 11873 [details, diff]
ebuild.sh changes to use the libsandbox patch
As you can see, very minimal.
In the long run, the sort -u should stay, and an external prog should be used
to generate the package list. This could then also be used to generate package
lists from stracing a running binary, giving a precise RDEPEND.
More ideas:
- put the tracing around all ebuild stages. This won't help for stages that
don't use sandbox, like postinstall, but for these the author knows what he
uses.
- Mark the packages that are in the system package list so that no dependencies
get added needlessly. (should go in that external app)
Comments very welcome...
It seems like nothing has happened on this since May? Why hasn't the patch been applied? It seems pretty harmless since it needs the FINDDEPS variable to be set. Technically and enhancement, marking it as such- Meanwhile, is this still even kicking? So... no longer kicking. Beyond that, reasons why this approach has major issues- How will dlopen trickery (say, kdelibs fex) be handled automatically? If an autogeneration of RDEPENDS is implemented, it pretty much has to be fool proof, 'coz it would become heavily relied upon. Beyond that, autogeneration of rdepends fails pretty miserably if the source makes any exec calls to external utilities- that's not even accounting trying to auto-determine what is required for scripts- quilt fex. |