Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 607868 - sys-apps/portage: optimize file merge with copy_file_range (or sendfile in case of EXDEV)
Summary: sys-apps/portage: optimize file merge with copy_file_range (or sendfile in ca...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on:
Blocks: 612090 635020 835380 612780
  Show dependency tree
 
Reported: 2017-02-01 00:48 UTC by Zac Medico
Modified: 2024-03-15 21:17 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Zac Medico gentoo-dev 2017-02-01 00:48:47 UTC
An optimized implementation of dblink._merge_contents could improve performance significantly when merging a large number of binary packages. There could be a completely native implementation using a C extension, and possibly a hybrid implementation that calls copy_file_range via ctypes.
Comment 1 Zac Medico gentoo-dev 2017-02-01 00:50:31 UTC
This should support sparse files, using the lseek(2) SEEK_DATA and SEEK_HOLE operations as described in the copy_file_range man page.
Comment 2 Zac Medico gentoo-dev 2017-02-02 09:53:38 UTC
Actually copy_file_range returns EXDEV in cases where os.rename would fail, so it looks like we'll be using sendfile instead.
Comment 3 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2017-02-02 11:38:30 UTC
1. copy_file_range can take advantage of reflinking on btrfs, avoiding physically copying the data. sendfile can't, so it's worse.

2. sendfile requires outfd to be a socket on older kernels.
Comment 4 Zac Medico gentoo-dev 2017-02-02 16:52:48 UTC
If copy_file_range returns EXDEV then it will fallback to sendfile.
Comment 5 Zac Medico gentoo-dev 2017-02-02 17:05:10 UTC
There's also this mmap/memcpy trick that could be useful if sendfile doesn't work:

http://stackoverflow.com/questions/26582920/mmap-memcpy-to-copy-file-from-a-to-b
Comment 8 Zac Medico gentoo-dev 2017-05-20 18:23:46 UTC
Fixed in portage-2.3.5.

Also see bug 617778 and bug 618086.
Comment 9 Harry Lee 2017-08-09 07:36:32 UTC
(In reply to Zac Medico from comment #8)
> Fixed in portage-2.3.5.
> 
> Also see bug 617778 and bug 618086.

This cause a bug when I do emerge in a docker container. https://bugs.gentoo.org/show_bug.cgi?id=627374
Comment 10 Benda Xu gentoo-dev 2017-11-04 04:13:44 UTC
Hi Zac, there is a case when syscall_326 (timefd_gettime) is not implemented in the runtime and portage gets EINVAL error during qmerge.
Comment 11 Benda Xu gentoo-dev 2017-11-04 04:16:06 UTC
The relavant strace is

[pid  8651] lstat("/disk01/xmass/gentoo/usr/bin/ebump", {st_mode=S_IFREG|0755, st_size=8516, ...}) = 0                                       [pid  8651] open("/dev/shm/portage/app-portage/gentoolkit-0.4.0/image/disk01/xmass/gentoo/usr/bin/ebump", O_RDONLY) = 12                     [pid  8651] fstat(12, {st_mode=S_IFREG|0755, st_size=8518, ...}) = 0                                                                         [pid  8651] open("/disk01/xmass/gentoo/usr/bin/ebump#new", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 13                                              [pid  8651] fstat(13, {st_mode=S_IFREG|0666, st_size=0, ...}) = 0                                                                            [pid  8651] lseek(12, 0, SEEK_DATA)     = 0                                                                                                  [pid  8651] lseek(12, 0, SEEK_HOLE)     = 8518                                                                                               [pid  8651] lseek(12, 0, SEEK_SET)      = 0                                                                                                  [pid  8651] syscall_326(0xc, 0, 0xd, 0x7ffd7c2a6ec8, 0x2146, 0) = -1 (errno 38)                                                              [pid  8651] sendfile(13, 12, [0], 8518) = 8518                                                                                               [pid  8651] lseek(12, 8518, SEEK_DATA)  = -1 EINVAL (Invalid argument)                                                                       [pid  8651] close(13)                   = 0                                                                                                  [pid  8651] close(12)                   = 0
Comment 12 Zac Medico gentoo-dev 2017-11-04 05:50:31 UTC
(In reply to Benda Xu from comment #11)
> The relavant strace is
> 
> [pid  8651] lstat("/disk01/xmass/gentoo/usr/bin/ebump",
> {st_mode=S_IFREG|0755, st_size=8516, ...}) = 0                              
> [pid  8651]
> open("/dev/shm/portage/app-portage/gentoolkit-0.4.0/image/disk01/xmass/
> gentoo/usr/bin/ebump", O_RDONLY) = 12                     [pid  8651]
> fstat(12, {st_mode=S_IFREG|0755, st_size=8518, ...}) = 0                    
> [pid  8651] open("/disk01/xmass/gentoo/usr/bin/ebump#new",
> O_WRONLY|O_CREAT|O_TRUNC, 0666) = 13                                        
> [pid  8651] fstat(13, {st_mode=S_IFREG|0666, st_size=0, ...}) = 0           
> [pid  8651] lseek(12, 0, SEEK_DATA)     = 0                                 
> [pid  8651] lseek(12, 0, SEEK_HOLE)     = 8518                              
> [pid  8651] lseek(12, 0, SEEK_SET)      = 0                                 
> [pid  8651] syscall_326(0xc, 0, 0xd, 0x7ffd7c2a6ec8, 0x2146, 0) = -1 (errno
> 38)                                                              [pid  8651]v

The above errno 38 corresponds to ENOSYS, so it falls back to sendfile, as designed.

> sendfile(13, 12, [0], 8518) = 8518

The sendfile call succeeds.

> [pid  8651] lseek(12, 8518, SEEK_DATA)  = -1 EINVAL (Invalid argument)      

Normally the lseek error is ENXIO here, as shown in the lseek man page:

ENXIO  whence is SEEK_DATA or SEEK_HOLE, and the file offset is beyond the end of the file.

With which kernel versions and filesystem did you observe this EINVAL error from lseek?
Comment 13 Benda Xu gentoo-dev 2017-11-04 06:23:42 UTC
(In reply to Zac Medico from comment #12)
> (In reply to Benda Xu from comment #11)
> > The relavant strace is
> > 
> > [pid  8651] lstat("/disk01/xmass/gentoo/usr/bin/ebump",
> > {st_mode=S_IFREG|0755, st_size=8516, ...}) = 0                              
> > [pid  8651]
> > open("/dev/shm/portage/app-portage/gentoolkit-0.4.0/image/disk01/xmass/
> > gentoo/usr/bin/ebump", O_RDONLY) = 12                     [pid  8651]
> > fstat(12, {st_mode=S_IFREG|0755, st_size=8518, ...}) = 0                    
> > [pid  8651] open("/disk01/xmass/gentoo/usr/bin/ebump#new",
> > O_WRONLY|O_CREAT|O_TRUNC, 0666) = 13                                        
> > [pid  8651] fstat(13, {st_mode=S_IFREG|0666, st_size=0, ...}) = 0           
> > [pid  8651] lseek(12, 0, SEEK_DATA)     = 0                                 
> > [pid  8651] lseek(12, 0, SEEK_HOLE)     = 8518                              
> > [pid  8651] lseek(12, 0, SEEK_SET)      = 0                                 
> > [pid  8651] syscall_326(0xc, 0, 0xd, 0x7ffd7c2a6ec8, 0x2146, 0) = -1 (errno
> > 38)                                                              [pid  8651]v
> 
> The above errno 38 corresponds to ENOSYS, so it falls back to sendfile, as
> designed.
> 
> > sendfile(13, 12, [0], 8518) = 8518
> 
> The sendfile call succeeds.
> 
> > [pid  8651] lseek(12, 8518, SEEK_DATA)  = -1 EINVAL (Invalid argument)      
> 
> Normally the lseek error is ENXIO here, as shown in the lseek man page:
> 
> ENXIO  whence is SEEK_DATA or SEEK_HOLE, and the file offset is beyond the
> end of the file.
> 
> With which kernel versions and filesystem did you observe this EINVAL error
> from lseek?

Ah-ha, thanks.

Kernel is 3.10.0-327.22.2.el7.x86_64 from RHEL7 and the file system is Fujitsu FeFS (which is an enterprise derivation from Lustre.  I am also using PRoot (which is intercepting syscalls by ptrace).  I suspect PRoot is the cause of EINVAL here.
Comment 14 Zac Medico gentoo-dev 2017-11-04 07:45:05 UTC
(In reply to Benda Xu from comment #13)

Yeah that's pretty exotic. Let's close this bug and file a new one.