Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 384147

Summary: Unpacking deb files with lzma compressed data is inconsistent
Product: Portage Development Reporter: Mike Gilbert <floppym>
Component: Core - External InteractionAssignee: PMS/EAPI <pms>
Status: RESOLVED FIXED    
Severity: minor CC: junghans
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
See Also: https://bugs.gentoo.org/show_bug.cgi?id=433577
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on: 384145    
Bug Blocks: 174380, 383027    

Description Mike Gilbert gentoo-dev 2011-09-22 22:22:36 UTC
If unpack is called on deb file containing lzma data (data.tar.lzma), 1 of 3 things will happen depending on the installed version of deb2targz:

1. app-arch/deb2targz not installed: files are successfully unpacked using ar

2. <app-arch/deb2targz-1-r1 is installed: unpack fails because deb2targz does not recognize data.tar.lzma

3. >=app-arch/deb2targz-1-r1 is installed: unpack fails because data.tar.lzma is extracted as data.tar.gz, but is not actually converted to gzip format. See bug 384145.
Comment 1 Zac Medico gentoo-dev 2011-09-23 04:38:02 UTC
(In reply to comment #0)
> If unpack is called on deb file containing lzma data (data.tar.lzma), 1 of 3
> things will happen depending on the installed version of deb2targz:
> 
> 1. app-arch/deb2targz not installed: files are successfully unpacked using ar

Good.

> 2. <app-arch/deb2targz-1-r1 is installed: unpack fails because deb2targz does
> not recognize data.tar.lzma
> 
> 3. >=app-arch/deb2targz-1-r1 is installed: unpack fails because data.tar.lzma
> is extracted as data.tar.gz, but is not actually converted to gzip format. See
> bug 384145.

Sounds like a bug. If it gets fixed in a new version, packages having data.tar.lzma should have a blocker against the old versions.
Comment 2 Mike Gilbert gentoo-dev 2011-09-23 06:25:32 UTC
I changed deb2targz so that it will keep the correct file extension on the output file.

This line in unpack() needs some tweaking. I'm not sure how to capture the file extension in bash.

mv -f "${y%.deb}".tar.gz data.tar.gz || die "$myfail"
Comment 3 Zac Medico gentoo-dev 2011-09-23 17:18:00 UTC
This should fix it:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=dd7c57ecb90949ce3306394cfe262566b87351ca

Diff looks like this:

-   mv -f "${y%.deb}".tar.gz data.tar.gz || die "$myfail"
+   local tarfile
+   for tarfile in "${y%.deb}".tar.* ; do
+           mv -f "${tarfile}" data.tar.${tarfile##*.} || \
+                   die "$myfail"
+   done
Comment 4 Zac Medico gentoo-dev 2011-09-23 17:22:36 UTC
@pms-bugs: Any package managers that hardcode data.tar.gz inside unpack will need a fix like the one shown in comment #3.
Comment 5 Ciaran McCreesh 2011-09-23 17:26:19 UTC
Shouldn't this be EAPI controlled? i.e. for current EAPIs we shouldn't support anything except data.tar.gz, and Portage should error rather than handle it?
Comment 6 Zac Medico gentoo-dev 2011-09-23 17:30:31 UTC
Either way is fine with me. I guess any packages that have this problem could just unpack the file manually until a supporting EAPI is available.
Comment 7 Ciaran McCreesh 2011-09-23 17:32:17 UTC
Well, it's a horribly backwards incompatible thing. PMS only doesn't restrict us to .gz because when we wrote it .gz was the only thing that existed, and we didn't realise there might be something else...
Comment 8 Mike Gilbert gentoo-dev 2011-09-23 17:39:51 UTC
I have to manually unpack the deb files for google-chrome until deb2targs-1-r2 is stable anyway. Waiting for an EAPI bump is no big deal.
Comment 9 Zac Medico gentoo-dev 2011-09-23 17:48:57 UTC
Okay, I've reverted the change in portage:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=6f4a5ef478b76ac24e6cd54a38728415fe852277
Comment 10 Christoph Junghans (RETIRED) gentoo-dev 2011-10-04 14:53:55 UTC
Same problem with media-sound/google-musicmanager (reported in bug #375969#c22).
Comment 11 Christoph Junghans (RETIRED) gentoo-dev 2012-01-12 16:50:11 UTC
@Zac: I had another thought about this, possible we should create an eclass "deb" (like rpm) which handles debian packages.

Debian packages anyway need a "double" uppack (*.deb -> data.tar.* -> content).
Comment 12 Zac Medico gentoo-dev 2012-01-12 17:08:22 UTC
Yeah, an eclass would be good for now. Then you can migrate away from that after we get this fixed in EAPI 5.
Comment 13 SpanKY gentoo-dev 2012-01-13 00:06:41 UTC
i'd rather create an unpacker.eclass, move unpack_{pdv,makeself} there, and then add a default src_unpack() that handles all these various things
Comment 14 Christoph Junghans (RETIRED) gentoo-dev 2012-01-15 01:32:15 UTC
(In reply to comment #12)
> Yeah, an eclass would be good for now. Then you can migrate away from that
> after we get this fixed in EAPI 5.
Here is a working deb.eclass:
<https://code.google.com/p/cj-overlay/source/browse/eclass/deb.eclass>

It avoids the workaround of bug#375969#c22.

(In reply to comment #13)
> i'd rather create an unpacker.eclass, move unpack_{pdv,makeself} there, and
> then add a default src_unpack() that handles all these various things
Sounds good. There should also be an automatic dependency function, I have something like DEPEND+="$(unpacker_deps ${A})" in mind.
Comment 15 Christoph Junghans (RETIRED) gentoo-dev 2012-01-15 01:38:38 UTC
And not to forget partial unpack!
Comment 16 SpanKY gentoo-dev 2012-01-15 02:20:51 UTC
i don't think that would work.  i don't believe ${A} is available at global scope source time.  you'd have to set SRC_URI before the inherit and have the code parse that instead.
Comment 17 Christoph Junghans (RETIRED) gentoo-dev 2012-01-15 03:05:57 UTC
(In reply to comment #16)
> i don't think that would work.  i don't believe ${A} is available at global
> scope source time.  you'd have to set SRC_URI before the inherit and have the
> code parse that instead.
True, so we could make it a strict qa check.
Comment 18 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2012-01-15 08:17:40 UTC
(In reply to comment #14)
> (In reply to comment #13)
> > i'd rather create an unpacker.eclass, move unpack_{pdv,makeself} there, and
> > then add a default src_unpack() that handles all these various things
> Sounds good. There should also be an automatic dependency function, I have
> something like DEPEND+="$(unpacker_deps ${A})" in mind.

${A} doesn't contain use-disabled sources, and thus is variable. It is unacceptable to have DEPEND vary like that.

While at it, you could make the code parse SRC_URI completely, detect and add correct DEPEND, including USE-conditionals in it and so on. But just letting ebuild author handle the dependency seems better IMO.
Comment 19 Mike Gilbert gentoo-dev 2012-01-15 09:50:10 UTC
A little off-topic, but deb2targz could be replaced with a function like this:

debexplode() {
    {
        # Global header
        read
        # File header
        while read filename timestamp uid gid mode size magic; do
            # File data
            head -c "$size" > "$filename"
        done
    } < "$1"
}

Based on the Wikipedia article for ar and a quick inspection of the deb2targz code.
Comment 20 Christoph Junghans (RETIRED) gentoo-dev 2012-01-15 19:05:34 UTC
(In reply to comment #13)
> i'd rather create an unpacker.eclass, move unpack_{pdv,makeself} there, and
> then add a default src_unpack() that handles all these various things

(In reply to comment #18)
> (In reply to comment #14)
> > (In reply to comment #13)
> > > i'd rather create an unpacker.eclass, move unpack_{pdv,makeself} there, and
> > > then add a default src_unpack() that handles all these various things
> > Sounds good. There should also be an automatic dependency function, I have
> > something like DEPEND+="$(unpacker_deps ${A})" in mind.
> 
> ${A} doesn't contain use-disabled sources, and thus is variable. It is
> unacceptable to have DEPEND vary like that.
> 
> While at it, you could make the code parse SRC_URI completely, detect and add
> correct DEPEND, including USE-conditionals in it and so on. But just letting
> ebuild author handle the dependency seems better IMO.

I moved the unpacker.eclass discussion in bug #399019.
Comment 21 Christoph Junghans (RETIRED) gentoo-dev 2012-01-15 19:09:27 UTC
(In reply to comment #19)
> A little off-topic, but deb2targz could be replaced with a function like this:
> 
> debexplode() {
>     {
>         # Global header
>         read
>         # File header
>         while read filename timestamp uid gid mode size magic; do
>             # File data
>             head -c "$size" > "$filename"
>         done
>     } < "$1"
> }
> 
> Based on the Wikipedia article for ar and a quick inspection of the deb2targz
> code.

Great, I will incorporate this function in deb.eclass. As it removed the deb2targz dependency for AIX.
Comment 22 Christoph Junghans (RETIRED) gentoo-dev 2012-01-15 21:57:20 UTC
(In reply to comment #21)
> (In reply to comment #19)
> > A little off-topic, but deb2targz could be replaced with a function like this:
> > 
> > debexplode() {
> >     {
> >         # Global header
> >         read
> >         # File header
> >         while read filename timestamp uid gid mode size magic; do
> >             # File data
> >             head -c "$size" > "$filename"
> >         done
> >     } < "$1"
> > }
> > 
> > Based on the Wikipedia article for ar and a quick inspection of the deb2targz
> > code.
> 
> Great, I will incorporate this function in deb.eclass. As it removed the
> deb2targz dependency for AIX.

Done. See:
<https://code.google.com/p/cj-overlay/source/browse/eclass/deb.eclass>

Any other suggestions?
Comment 23 Christoph Junghans (RETIRED) gentoo-dev 2012-02-08 01:54:35 UTC
I guess with unpacker.eclass beginning in the tree we can close this bug.

The rest of the story is filed under bug #399019.
Comment 24 Ulrich Müller gentoo-dev 2012-04-27 21:35:49 UTC
(In reply to comment #23)
> I guess with unpacker.eclass beginning in the tree we can close this bug.
> 
> The rest of the story is filed under bug #399019.

Closing. Feel free to reopen if you disagree.
Comment 25 Zac Medico gentoo-dev 2013-06-18 00:45:47 UTC
If people are just going to use unpacker.eclass, then maybe we should consider removing deb support from unpack in a future EAPI?
Comment 26 Christoph Junghans (RETIRED) gentoo-dev 2013-06-18 01:57:27 UTC
(In reply to Zac Medico from comment #25)
> If people are just going to use unpacker.eclass, then maybe we should
> consider removing deb support from unpack in a future EAPI?
Sounds good to me!