We are using mmap to map a text file into memory, that we then parse. We map a larger size than the file, and use the fact that bytes following the file are set to 0. This ensures that a \0 terminator exists, when we treat the memory as a char*. However, when the file size is a multiple of the page size (in our case, the file is 8192 bytes), the length argument to mmap is not respected. This results in bus errors when attempting to do any string handling of the returned pointer (e.g., strlen). Either the man page is wrong ("The length argument specifies the length of the mapping" and "The contents of a file mapping ... are initialized using length bytes" and "For a file that is not a multiple of the page size, the remaining memory is zeroed when mapped") or the implementation is wrong. Reproducible: Always Steps to Reproduce: 1. Compile and run the attached file: gcc -g test.c -o test ; ./test Actual Results: price@alala:/home/panstarrs/price/temp>./test Bus error (core dumped) Expected Results: The test program should behave like "cat", dumping the data file to stdout.
Created attachment 134462 [details] Test program to reproduce
Created attachment 134464 [details] Example text file
The strace of this code is really interesting. The second mmap() happens when passing either buf.st_size (8192) or buf.st_size + 1 (8193) to mmap(2) in the test example. It looks like glibc is trying to do the right thing by giving us a page of padding but for some reason this page is not being zero'd out and contains garbage. The results are consistent across multiple kernel revs so it looks like we're getting a recycled page and it's glibc's responsibility to zero it. open("ppImage.config", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=8192, ...}) = 0 fstat(3, {st_mode=S_IFREG|0644, st_size=8192, ...}) = 0 mmap(NULL, 8193, PROT_READ|PROT_WRITE, MAP_PRIVATE, 3, 0) = 0x2b7fea18c000 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b7fea18f000
the only time you're guaranteed the return from mmap() is zero-ed is when MAP_ANONYMOUS is used ... you are not using MAP_ANONYMOUS the mmap page states that when a fd is given, the specified file is used to initialize the mapping i see no mention of a guarantee that a non-anonymous mapping should be zero-filled beyond the length of the file given as backing to the mmap() really though, i think your mention of different kernels giving different behavior should be a hint ... glibc does not do zeroing *anywhere* with mmap() regardless of flags ... glibc merely passes your request on to the kernel and the kernel takes care of things
Please refer to "The Open Group Base Specifications Issue 6, IEEE Std 1003.1, 2004 Edition" definition of mmap (http://www.opengroup.org/onlinepubs/009695399/functions/mmap.html) where the following statements are made: * "The mmap() function shall establish a mapping between the address space of the process at an address pa for len bytes to the memory object represented by the file descriptor fildes at offset off for len bytes." * "while the argument len need not meet a size or alignment constraint, the implementation shall include, in any mapping operation, any partial page specified by the range [pa,pa+len)." * "The system shall always zero-fill any partial page at the end of an object." I submit that the glibc implementation of mmap does not result in a mapping "for len bytes". Instead, it continues only as far as the file size (these two are NOT the same). This is demonstrated by the attached test program, which attempts to access the mapping at a place between the length of the file and "len", resulting in SIGBUS. Making "len" larger than the size of the file should result in a mapping valid for "len" bytes (because "len" is not constrained according to the standard). Moreover, bytes beyond the end of the file should be zero-filled, since they belong to the partial page at the end of an object. Therefore, this implementation violates POSIX.1b. Please note further that we have NOT stated that different kernels give different results, but "The results are consistent across multiple kernel revs" (comment 3). Please reconsider the "INVALID" resolution.
the mmap() manpage does not mention that stipulation. however, i still dont think it is a glibc problem. look at the source code and you'll see that glibc merely calls the kernel. i'm pretty sure the second mmap() in your strace is irrelevant ... it isnt being done on the fd and it's MAP_ANONYMOUS which means there is no guarantee whatsoever as to the location of the mapping.
This issue is also present in sys-libs/glibc-2.6.1 which is the latest stable glibc for amd64.
The standard says the mapping will return zeros for any *partial* page following the end of the object. It also says: "References within the address range starting at pa and continuing for len bytes to whole pages following the end of an object shall result in delivery of a SIGBUS signal." In other words, if you read past the end of the file but within the final partial page you will get back zeros. If you read past the final partial page you will get a SIGBUS. Of course, in your test example, there isn't a partial page, since the file size is an exact multiple of the page size. Hence the string is unterminated and it blows up. Here is an old LKML post discussing this, BTW: http://marc.info/?l=linux-kernel&m=95954229317867&w=2 So I think this bug should indeed be closed as INVALID. However, see: http://bugs.gentoo.org/show_bug.cgi?id=197483
Fair enough, though I thought the standard seemed unclear about where the mapping is supposed to end: pa+len or pa+file_size.
The mapping itself does end at (pa + len), however it is not accessible beyond the last page containing (pa + file_size). Just because a memory region is mapped doesn't necessarily make it OK to access :)
