Bugzilla Bug 164655

build a package of say xorg-server, and it produces a bum package

# grep /usr/lib/X11/xserver/ /var/db/pkg/x11-base/xorg-server-1.2.0/CONTENTS |
wc
    912    4337   65457
# quickpkg x11-base/xorg-server
 * Building package for xorg-server-1.2.0 ...

 * Packages now in /usr/portage/packages:
 * xorg-server-1.2.0: 8.8M
# cd /usr/portage/packages/x11-base
# tar tf xorg-server-1.2.0.tbz2 | grep /usr/lib/X11/xserver/ | wc
    247     247   16627


there are a ton of symlinks in /usr/lib/X11/xserver/ that point to
/usr/lib/X11/xserver/C and running quickpkg destroys them all

if we cant pull the files off correctly from /, then we should probably copy
them to a temp directory and create a tarball of that

should be pretty easy to translate CONTENTS into a file list for rsync and then
have rsync copy the files over

*** Bug 164271 has been marked as a duplicate of this bug. ***

(In reply to comment #1)
> if we cant pull the files off correctly from /, then we should probably copy
> them to a temp directory and create a tarball of that
> 
> should be pretty easy to translate CONTENTS into a file list for rsync and then
> have rsync copy the files over

I disagree. This will create a huge overhead for big and/or many packages.

Created an attachment (id=108933) [details]
Reimplementation of quickpkg.py

May I humbly suggest an alternative implementation of quickpkg? This is a
simple reimplementation I wrote when I noticed this problem the first time.
This implementation has the following features:
- It fixes bug #164655 by actually using the informations in CONTENTS
- It ignores bug #16162 silently (as do the official implementation)
- It's written in Python, so it can hook directly into Portage
- My initial benchmarkings indicates that it's a bit faster than the official
implementation (probably due to the overhead for calling portageq).

This implementation is missing:
- The ability to handle the '{PORTAGE_DB}/<CATEGORY>/<PKG-VERSION>/' -form
- Nice colors

However if anyone likes this implementation I might think about implementing
these features too.

(In reply to comment #4)
> Created an attachment (id=108933) [edit] [details]
> Reimplementation of quickpkg.py

I was thinking about rewriting it in python too.  That's probably the best
solution considering that it allows us to use the existing portage libraries
and python's tarfile module (to work around the inadequacies of the plain tar
command).

Instead of hard coding package paths in /usr/portage/packages, the script
should use the methods of the binarytree class like emerge does.  In 2.1.2,
emerge uses the binarytree.prevent_collision() method to prevent collisions
between two packages that have the same $PF (bug 133908) and it uses
binarytree.getname() to get the location that the package should be stored at.

(In reply to comment #5)
> (In reply to comment #4)
> > Created an attachment (id=108933) [edit] [details]
> > Reimplementation of quickpkg.py
> 
> I was thinking about rewriting it in python too.  That's probably the best
> solution considering that it allows us to use the existing portage libraries
> and python's tarfile module (to work around the inadequacies of the plain tar
> command).

Agreed.

> Instead of hard coding package paths in /usr/portage/packages, the script
> should use the methods of the binarytree class like emerge does.  In 2.1.2,
> emerge uses the binarytree.prevent_collision() method to prevent collisions
> between two packages that have the same $PF (bug 133908) and it uses
> binarytree.getname() to get the location that the package should be stored at.

Don't worry. This is not the final version. I just didn't want to spend too
much time on it until I knew that I had some support. I will tidy up the code
this weekend and post a new version.

Created an attachment (id=109243) [details]
A reimplementation of quickpkg

Here's a new and improved version of quickpkg. Among the new things are:
- Making use of Portage's EOutput for showing output
- Uses informations from bintree instead of hardcoded directories
- Encapsulated in a class for easy reuse (import) in other applications

Comments, complaints and requests are most welcome.

Worked where 'official' quickpkg failed, thanks.  (xorg-server install from
pkg)

It is missing the file size output compared to the official version ;)

(In reply to comment #8)
> It is missing the file size output compared to the official version ;)

Yeah, I know. Personally I think it a pretty useless feature, but I will
implement it later if someone asks for it. For now I have simply "prepared" the
code for it.

Created an attachment (id=110639) [details]
A reimplementation of quickpkg

Updated version. The differences are:
- Documentation (inline in epytext format, like Portage will be)
- Can raise a portage_exception.PackageNotFound exception
- Returns the path to the binary package, which could be usefull e.g. for
printing the size of the package.

Created an attachment (id=119261) [details]
A reimplementation of quickpkg

Minor update of my reimplementation of quickpkg. It now print the package size
like the official version does.

Comments, complaints, questions and suggestions are most welcome.

Created an attachment (id=119283) [details]
A reimplementation of quickpkg

Update of Martin Parm's quickpkg.

- now handles '{PORTAGE_DB}/<CATEGORY>/<PKG-VERSION>/' correctly.
- makes use of binarytree.prevent_collision()
- outputs unmatched packets like the original

> Update of Martin Parm's quickpkg.
> 
> - now handles '{PORTAGE_DB}/<CATEGORY>/<PKG-VERSION>/' correctly.
> - makes use of binarytree.prevent_collision()
> - outputs unmatched packets like the original
> 

Thanks for the update. It's good to see a little activity ^_^.
One question though: is there any particular reason for you to change my
codestyle (renaming variable, etc.)? Is there some official Gentoo codestyle
that you're trying to match? If so, could you please provide me with a
description/link.

Created an attachment (id=119332) [details]
A reimplementation of quickpkg (v0.7.1)

Some minor changes to Alexander Graf's update:
- The PID is read in __call__() instead of in __init__() thus allowing Quickpkg
to operate correctly from a child-process.
- Made use of os.path.realpath instead of doing the work manually with
os.readlink
- Check all package specification again the Portage API even when in the
/var/db/pkg/<CATEGORY>/<PKG-VERSION>/ form, thus preserving the original
intend: "Better integration with Portage by making direct use of the Portage
API, thus trying to avoid any kind of hardcoding or implicite knowledge."
- Check if binsize is empty before outputting it's content, thus making the
output more correct.

(In reply to comment #14)
> - The PID is read in __call__() instead of in __init__() thus allowing Quickpkg to operate correctly from a child-process.
> - Made use of os.path.realpath instead of doing the work manually with
> os.readlink

  I knew there was something easier ;)

  As for your/my codestyle: I don't know if there's an official gentoo
codestyle. It's just my style and I couldn't resist ;)
  But I think using variable names such as "cpv" is generally reasonable
because the same names are used in portage.

  One question: What is "the right thing" to do in line 110 if the "symlink" is
not removable (eg it's a folder or write protected or...)? Should we bail out
with an error or remove whatever there is using shutil.rmtree()?

Created an attachment (id=119926) [details]
A reimplementation of quickpkg (v0.7.2)

Some minor changes and bug fixes:

- added check for uid (root)
- added setting of umask
- fixed handling of name collisions (ie. quickpkg pecl-pdflib now works)
- fixed class import (os is now imported in __call__, not __init__ od Quickpkg)

Thanks, this is now in trunk r6728.

This has been released in 2.1.2.10.

*** Bug 193653 has been marked as a duplicate of this bug. ***