Bugzilla Bug 14983

quickpkg script cannot add a file with whitespaces in pathname to package. It
use cut -f 2 -d " ". So trying to quickpkg winex results in:
usr/lib/winex/.data/fake_windows/
tar: /usr/lib/winex/.data/fake_windows/Program: Cannot stat: No such file or
directory
tar: /usr/lib/winex/.data/fake_windows/Program: Cannot stat: No such file or
directory
Package is built, but not all files inside. portage version 2.0.46-r9.

Reproducible: Always
Steps to Reproduce:
1. emerge winex
2. quickpkg /var/db/pkg../winex..

nick: what if we change the delimiter to a tab ? i couldnt imagine a package
using a directory/filename with a tab in it ...

I wondered about how to solve that problem as well. The only surefire way is
delimiting with ", 
since that is not allowed in files, but \t practically doesn't happen either,
and it allows most tools 
to just work. 

This will still cause the tools which assume general whitespace delimitation to
fail, but those 
can be fixed. 

Good idea, let's do it :) 

This bug has been inactive for more than 180 days. 

winex ebuild should probably be removed - it died long ago anyway. I'm not sure where else to look to try to reproduce this bug.

Can somebody post an update please?

yes, most ebuilds are 'fixed', but the bug still exists

*** Bug 40744 has been marked as a duplicate of this bug. ***

Created an attachment (id=25157) [details]
double-spaces-fix.patch

Seems that this bug was already fixed for simple spaces in filenames. The
duplicate bug above was about double spaces: here is a minimalistic fix for
this issue. I know that was a very minor issue tho, but it's bugday and that's
what the "choose a random bug" algorithm returned :/

If a cleaner fix is needed, i can think of using backslashs to quote white
chars in filenames written to CONTENTS, and then to split the string with
shlex.split() instead of string.split(). Or also to split only the fields from
the end of the line, and leave the filenames untouched (we know how many fields
there should be by the type of contents entry).

Created an attachment (id=25188) [details]
Patch to provide proper CONTENTS file parsing

This is all... Well... Silly.

When I look at the CONTENTS file, I can see which parts are the filenames, and
which parts are the MD5 sums.  It's a very rigid structure, and is very easy to
parse.	The code that split on spaces should never have been written, frankly
;-)

If the line begins with 'dir', 'dev' or 'fif', it is very easy to parse:
everything after the type is the filename.  (On a side note, I've never seen a
package use 'dev' or 'fif'?)
If the line begins with 'obj', then the last two space-separated fields are the
md5 and mtime, and everything after the first space, and before the second last
space, is the filename.
This "everything" can contain spaces, commas, tabs, whatever you want.	It
matters not.

Using any separator, and relying on it, will cause problems one day, if a file
happens to want that separator in its filename.  The only way around this is to
use escape characters, which complexify the code unnecessarily.  Since we have
a fixed structure of the line already, we should use it.

Attached is a patch which uses regular expressions to do this easily.
(It's a patch to portage.py,v 1.385 2004/01/27 14:37:53)

It's difficult to test a part of portage, but my script that uses the same code
works ;-)

One thing that is in portage.py.orig that is not handled in my patch is some
odd code that looks like:

x=len(mydat)-1
if (x >= 13) and (mydat[-1][-1]==')'): # Old/Broken symlink entry
   mydat = mydat[:-10]+[mydat[-10:][ST_MTIME][:-1]]
   writemsg("FIXED SYMLINK LINE: %s\n" % mydat, 1)
   x=len(mydat)-1

It appears on lines 5035 to 5038 of portage.py, and I can't work it out, nor
can I find anything about it in bugs.gentoo.org.  I don't handle this
"Old/Broken symlink entry", but I'm not sure how to?

Created an attachment (id=33618) [details]
patch to .50_pre10 to use tabs for the delimiter, with a bit of display
cleanup.

Note that the old symlink fix has been moved out of portage; this is handled by
an external script that needs to be added to the portage ebuild post_pkginst
function.

Created an attachment (id=33619) [details]
update_vardb.py

This is a simple script that should be updated/included in portage versions;
it's intended use is for performing the necessary transformations on
/var/db/pkg/* as needed.
In this case, it is *only* working on CONTENTS, converting existing files to
tab based delimitors, also performing the old symlink fix for content entries.

Moving this out of portage, and into this file simplifies portages getcontents
function- main benefit is that this script can be added to pkg_postinst, and
perform the updates *once* when updating portage versions.

Either way, been tested, this file transforming /var/db/pkg/*/*/CONTENTS (note,
it's backwards compatable) and the patch posted above address and should fix
the issue.

This ought to be fixed- I'm attaching an ebuild that portage is still screwing
up on, which is fixed via these updates.

Created an attachment (id=33620) [details]
contents_demo-0.ebuild

Example ebuild that portage is still exhibiting broken behaviour- specifically,
the file is merged, but portage is unable to remove it due to space issues.

Err... one note; for update_vardb.py, it expect one arguement- the vardb base
directory work within.
so,  update_vardb.py /var/db/pkg is the appropriate way to execute it.

Created an attachment (id=33723) [details]
.50_pre10 tab based CONTENTS

Minor typo in the original patch, corrected now (inverted mtime/md5 positions)

*bump*, Did this ever get commited? *notes the numerous patches by ferringb*

*** Bug 147154 has been marked as a duplicate of this bug. ***

In portage-2.1.3.19 the CONTENTS parser is able to handle practically anything
except newlines in filenames. It seems like any CONTENTS format changes are
beyond the scope of this bug, so we can consider this fixed.