ca-certificates-20110502-r2 and -r3 both fail to emerge due to charset mapping problems for certificate file names containing non-ASCII chars. ca-certificates-20110502-r1 and before is fine. The emerge output is >>> Installing (3 of 4) app-misc/ca-certificates-20110502-r3 Traceback (most recent call last): File "/usr/lib64/portage/pym/portage/dbapi/_MergeProcess.py", line 246, in _spawn prev_mtimes=self.prev_mtimes, counter=counter) File "/usr/lib64/portage/pym/portage/dbapi/vartree.py", line 4288, in merge counter=counter) File "/usr/lib64/portage/pym/portage/dbapi/vartree.py", line 3599, in treewalk rval = self._merge_contents(srcroot, destroot, cfgfiledict) File "/usr/lib64/portage/pym/portage/dbapi/vartree.py", line 3879, in _merge_contents self.settings["EPREFIX"].lstrip(os.sep), cfgfiledict, mymtime): File "/usr/lib64/portage/pym/portage/dbapi/vartree.py", line 4167, in mergeme join(offset, x), cfgfiledict, thismtime): File "/usr/lib64/portage/pym/portage/dbapi/vartree.py", line 4167, in mergeme join(offset, x), cfgfiledict, thismtime): File "/usr/lib64/portage/pym/portage/dbapi/vartree.py", line 4167, in mergeme join(offset, x), cfgfiledict, thismtime): File "/usr/lib64/portage/pym/portage/dbapi/vartree.py", line 4016, in mergeme myabsto = abssymlink(mysrc) File "/usr/lib64/portage/pym/portage/__init__.py", line 396, in abssymlink mylink=os.readlink(symlink) File "/usr/lib64/portage/pym/portage/__init__.py", line 226, in __call__ rval = self._func(*wrapped_args, **wrapped_kwargs) OSError: [Errno 2] No such file or directory: '/var/portage/portage/app-misc/ca-certificates-20110502-r3/image/etc/ssl/certs/AC_Ra\xc3\x83\xc2\xadz_Certic\xc3\x83\xc2\xa1mara_S.A..pem' As far as I can tell, the problem is caused by python converting filenames from ISO8859-15 to unicode, but not back: The file which is not found contains two 4-byte escape sequences: '\xc3\x83\xc2\xad' and '\xc3\x83\xc2\xa1' (Unicode I believe) However, my system is *not* unicode (USE -nls -unicode, locale ISO8859-15), and the same filename on disk only contains two 2-byte non-ASCII codes: 'AC_Ra\303\255z_Certic\303\241mara_S.A..pem' So the install routine is actually looking up the file with a name differing from the original, and fails. If I remove all certificates with non-ASCII filenames, the error above goes away. However, the emerge fails immediately after that step: Python says that there is no valid mapping for charcode 011f.
*** This bug has been marked as a duplicate of bug 381629 ***
Please either reopen 381629 or undup this bug and reopen it: 381629 is "resolved fixed", and I have the version which should contain the fix for 381629, but I still get the error. Hence, either 381629 is not fixed, or this is a different problem.
(In reply to comment #0) > File "/usr/lib64/portage/pym/portage/dbapi/vartree.py", line 4016, in mergeme > myabsto = abssymlink(mysrc) > File "/usr/lib64/portage/pym/portage/__init__.py", line 396, in abssymlink > mylink=os.readlink(symlink) > File "/usr/lib64/portage/pym/portage/__init__.py", line 226, in __call__ > rval = self._func(*wrapped_args, **wrapped_kwargs) > OSError: [Errno 2] No such file or directory: This is fixed in git: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=eab5a6ee1abff1fbf142cf1558ba940b6d4b270a It should also fix the following error reported in bug #381629, comment #15: > File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 4025, in mergeme > myrealto = normalize_path(os.path.join(destroot, myabsto)) > File "/usr/lib/portage/pym/portage/__init__.py", line 218, in __call__ > for x in args] > File "/usr/lib/portage/pym/portage/__init__.py", line 218, in <listcomp> > for x in args] > File "/usr/lib/portage/pym/portage/__init__.py", line 179, in _unicode_encode > s = s.encode(encoding, errors) > File "/usr/lib/python3.2/encodings/iso8859_7.py", line 12, in encode > return codecs.charmap_encode(input,errors,encoding_table) > UnicodeEncodeError: 'charmap' codec can't encode character '\xdc' in position > 15: character maps to <undefined> This error is triggered by corrupt characters returned from the readlink call that is now bypassed.
Created attachment 285717 [details, diff] bypass abssymlink readlink call
Created attachment 285719 [details, diff] bypass abssymlink readlink call This fixes inverted logic in the previous patch. Save as /tmp/abssymlink.patch and apply as follows: cd /usr/lib/portage patch -p1 < /tmp/abssymlink.patch
This is fixed in 2.1.10.14 and 2.2.0_alpha54.
*** Bug 382087 has been marked as a duplicate of this bug. ***
Hmmm, much better but not yet completely fixed. Tested with portage 2.1.10.14 and ca-certificates-20110502-r4. Emerges correctly und unmerges correctly, so the main problem is fixed. However, /var/db/pkg/app-misc/ca-certificates-20110502-r4/CONTENTS contains the translated filenames which do not match the actual filenames on disk. This will certainly confuse third-party tools (like a security shell script of mine which compares all files on disk with all files listed in all CONTENTS files, finding orphaned and lost files).
(In reply to comment #8) > However, /var/db/pkg/app-misc/ca-certificates-20110502-r4/CONTENTS > contains the translated filenames which do not match the actual filenames > on disk. They match, but they are encoded differently. CONTENTS is encoded in UTF-8, and the file names on disk are encoded with your ISO8859-15 locale setting. I've tested this case with ca-certificates-20110502-r4, and emerge is able to correctly merge and unmerge these files. > This will certainly confuse third-party tools > (like a security shell script of mine which compares all files on disk > with all files listed in all CONTENTS files, finding orphaned and lost files). They won't get confused if they use the same codec translations as portage does. My goal is to keep CONTENTS encoded with UTF-8 encoding, and also to have portage to respect your ISO8859-15 locale setting. This is what it does, so I'd rather not change it. A possible alternative solution to your perceived problem would be to have the ca-certificates ebuild detect the ISO8859-15 locale setting and translate the encoding of the file names accordingly. If you want to take that approach, then you should file a new bug to have the ca-certificates ebuild translate the file names.
Since the OSError from comment #0 and the related UnicodeEncodeError from bug #381629, comment #15 are both fixed, let's consider this bug resolved. You can file a new bug for the other codec translation issue discussed in comment #9.
Continued in bug 382199.
*** Bug 394473 has been marked as a duplicate of this bug. ***
*** Bug 394641 has been marked as a duplicate of this bug. ***
*** Bug 394825 has been marked as a duplicate of this bug. ***
As the filer of bug 394825, the patch for this bug fails to apply. Please de-dup.
(In reply to comment #15) > As the filer of bug 394825, the patch for this bug fails to apply. Please > de-dup. Bug 382199 is also related. Anyway, you should use the latest ~arch version of portage rather than apply the attached patch. We plan to have it stabilized as soon as possible (bug 394695).
(In reply to comment #16) > (In reply to comment #15) > > As the filer of bug 394825, the patch for this bug fails to apply. Please > > de-dup. > > Bug 382199 is also related. Anyway, you should use the latest ~arch version of > portage rather than apply the attached patch. We plan to have it stabilized as > soon as possible (bug 394695). Thank you. However, I will wait until it is stabilized and mask the CA certificates package until then.
