Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 677496 - check-reqs.eclass - check for free space on PORTAGE_TMPDIR: take compression ratio into account for compressed filesystems
Summary: check-reqs.eclass - check for free space on PORTAGE_TMPDIR: take compression ...
Status: RESOLVED INVALID
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Eclasses (show other bugs)
Hardware: All Linux
: Normal normal
Assignee: Gentoo Quality Assurance Team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-02-08 12:53 UTC by segmentation fault
Modified: 2019-03-07 03:21 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description segmentation fault 2019-02-08 12:53:55 UTC
This is a general enhancement request: 

I have had a few 'pre-merge checks' about free space on PORTAGE_TMPDIR in the past, where the ebuild would refuse to continue if it needed, say 8GB, but the check returned 'only 5GB free'. These checks are correct on 'normal' filesystems, but fail to fullfill their purpose on compressed ones. Consider this scenario (there are certainly others, e.g. with compressed filesystems like btrfs etc.):

You have plenty of RAM, so you create a zram device - a compressed RAM disc - and mount it, say, under /zram - then you set PORTAGE_TMPDIR=/zram. You know that PORTAGE_TMPDIR will hold mainly text files, for which you can safely assume a 2:1 compression ratio. So setting a 5GB zram device should more than suffice for 8GB text files - it will probably be able to hold 12GB. Yet the pre-merge check for free space complains about not having 8GB at its disposal. 

This forces you to set 8GB *minimum* for the zram device - which is overkill, because now you have set up a device capable of storing 20GB of text. In cases where RAM is not copiously available, this forces you to abandon the idea of using a compressed RAM device for PORTAGE_TMPDIR - you are forced to use your (slow) discs.

For me, with 40GB of RAM at my disposal, this was just an annoyance - for others it will be more serious. With compression being more widely used, this will become a headache.

The enhancement might want to take into account the type of device POPRTAGE_TMPDIR is mounted on - if it is a compressed device, it should not be so strict and deny further execution, but should issue a warning instead. If that's too difficult to implement, then maybe the warning should become the default outcome.
Comment 1 Georgy Yakovlev archtester gentoo-dev 2019-02-08 21:21:06 UTC
your example is incorrect.

you create a fixed size filesystem. the fact that it sits atop of zram device does not mean that the filesystem will be able to fit more data, it's size is fixed.

it means that the data on zram device will use less memory.

you can't magically fit 8gb on a 5gb filesystem.

so check-reqs is correct. If a ebuild tries to write more than 4gb of data it'll fail.


modprobe zram
  zram: Added device: zram0
echo 4096M > /sys/block/zram0/disksize
  zram0: detected capacity change from 0 to 4294967296
mkfs.ext4 /dev/zram0 -m0
mount /dev/zram0 /mnt

df -m /mnt
Filesystem     1M-blocks  Used Available Use% Mounted on
/dev/zram0          3968    16      3936   1% /mnt

dd if=/dev/zero of=/mnt/zeroes bs=1M count=4096
dd: error writing '/mnt/zeroes': No space left on device
3936+0 records in
3935+0 records out
4126871552 bytes (4.1 GB, 3.8 GiB) copied, 2.17158 s, 1.9 GB/s


df -m /mnt
Filesystem     1M-blocks  Used Available Use% Mounted on
/dev/zram0          3968  3952         0 100% /mnt

du -m /mnt/zeroes 
3936    /mnt/zeroes

du -m --apparent-size /mnt/zeroes 
3936    /mnt/zeroes

File /sys/block/zram<id>/mm_stat

The stat file represents device's mm statistics. It consists of a single
line of text and contains the following stats separated by whitespace:
 orig_data_size   uncompressed size of data stored in this disk.
                  This excludes same-element-filled pages (same_pages) since
                  no memory is allocated for them.
                  Unit: bytes
 compr_data_size  compressed size of data stored in this disk
 mem_used_total   the amount of memory allocated for this disk. This
                  includes allocator fragmentation and metadata overhead,
                  allocated for this disk. So, allocator space efficiency
                  can be calculated using compr_data_size and this statistic.
                  Unit: bytes
 mem_limit        the maximum amount of memory ZRAM can use to store
                  the compressed data
 mem_used_max     the maximum amount of memory zram have consumed to
                  store the data
 same_pages       the number of same element filled pages written to this disk.
                  No memory is allocated for such pages.
 pages_compacted  the number of pages freed during compaction
 huge_pages       the number of incompressible pages


cat /sys/block/zram0/mm_stat 
4196306944    57531   159744        0   159744  1023924        0        0
Comment 2 segmentation fault 2019-02-13 11:03:20 UTC
You are right. The size passed to zram is the final size of the filesystem, not the site of memory to use (as I thought). Put otherwise, it is 'uncompresed size', not 'compressed'.

So you can set this to INVALID. 

Thank you for the answer. :-)
Comment 3 Georgy Yakovlev archtester gentoo-dev 2019-03-07 03:21:42 UTC
np, closing.