Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 603578 - dev-lang/perl-5.22.3_rc4 excessive decompression memory requirements for perl-5.22.3-RC4-patches-1.tar.xz
Summary: dev-lang/perl-5.22.3_rc4 excessive decompression memory requirements for perl...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal
Assignee: Gentoo Perl team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-12-23 12:27 UTC by Mark Davies
Modified: 2016-12-26 15:54 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
emerge --info '=dev-lang/perl-5.22.3_rc4::gentoo' output (emerge-info,4.83 KB, text/plain)
2016-12-23 12:27 UTC, Mark Davies
Details
build.log (build.log,2.50 KB, text/x-log)
2016-12-23 12:28 UTC, Mark Davies
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Mark Davies 2016-12-23 12:27:42 UTC
Created attachment 457282 [details]
emerge --info '=dev-lang/perl-5.22.3_rc4::gentoo' output

I run gentoo on a very old machine (emerge-info). Today the upgrade of perl to 5.22.4-RC4 failed with:

xz: /home/gentoo/tmp/portage/dev-lang/perl-5.22.3_rc4/distdir/perl-5.22.3-RC4-patches-1.tar.xz: Cannot allocate memory

Checking to see how much memory was required I get 513MiB:

$ unxz --memlimit-decompress=1MiB perl-5.22.3-RC4-patches-1.tar.xz
unxz: perl-5.22.3-RC4-patches-1.tar.xz: Memory usage limit reached
unxz: 513 MiB of memory is required. The limit is 1 MiB.

which seems a bit excessive for a file that uncompresses to 100k
$ xz -l perl-5.22.3-RC4-patches-1.tar.xz
Strms  Blocks   Compressed Uncompressed  Ratio  Check   Filename
    1       1     22.8 KiB    100.0 KiB  0.228  SHA-256 perl-5.22.3-RC4-patches-1.tar.xz

I uncompress the file on another machine and recompressed using the installed xz (app-arch/xz-utils-5.2.2). With xz -e -9 -C sha256 I get a file of the same size but it only need 65MiB to decompress which is the figure given in the man page.

So something weird has happened with this file or the way it was compressed.
Comment 1 Mark Davies 2016-12-23 12:28:33 UTC
Created attachment 457284 [details]
build.log
Comment 2 Mark Davies 2016-12-24 11:01:15 UTC
Found the problem - the file was compressed with --lzma2=dict=512MiB

Why?
Comment 3 Kent Fredric (IRC: kent\n) (RETIRED) gentoo-dev 2016-12-26 15:17:30 UTC
It inherited my defaults I use for other, much larger things from ENV unfortunately. 

Have set up a build script now so this mistake won't happen again, and sets a more frugal decompression limit of 1MB. ( which is still very generous considering how small these patches are :) )
Comment 4 Kent Fredric (IRC: kent\n) (RETIRED) gentoo-dev 2016-12-26 15:54:10 UTC
commit 2c1ec464e1c6d2d13c63715a34f8e0401c73a4fd
Author: Kent Fredric <kentnl@gentoo.org>
Date:   Tue Dec 27 04:20:27 2016 +1300

    dev-lang/perl: Use more frugal decompression limits re bug #603578
    
    In my user ENV I have some very expensive and large limits defined
    which leaked through into the release process.
    
    This commit ships re-generated tar.xz's with a more frugal memory
    limit of 1MB, which is *heaps* for this.
    
    Thanks to Mark Davies for reporting.
    
    Bug: https://bugs.gentoo.org/603578