When I try to copy a large file (> 2 GB) like so: cp --debug file_a file_b output looks like this: 'file_a' -> 'file_b' cp: error copying 'file_a' to 'file_b': Invalid argument copy offload: unsupported, reflink: unsupported, sparse detection: no Afterwards file_b has a size of 2147479552 bytes (= 2G - 4K). On another system (with newer kernel version) it looks like this: cp --debug file_a file_b 'file_a' -> 'file_b' copy offload: yes, reflink: unsupported, sparse detection: no
Could you include emerge --info and the file system for the failing machine? Thanks.
With sys-apps/coreutils-9.1-r2 the issue is not present. Copying large files works as expected.
emerge --info Portage 3.0.44 (python 3.11.3-final-0, default/linux/amd64/17.1, gcc-12, glibc-2.36-r8, 4.19.80 x86_64) ================================================================= System uname: Linux-4.19.80-x86_64-Intel-R-_Core-TM-_i7-4790_CPU_@_3.60GHz-with-glibc2.36 KiB Mem: 16314388 total, 124560 free KiB Swap: 16762876 total, 16756732 free Timestamp of repository gentoo: Tue, 30 May 2023 06:30:01 +0000 Head commit of repository gentoo: 68d169f46e45ca303c79e0a4b9c803841835768c sh bash 5.1_p16-r4 ld GNU ld (Gentoo 2.39 p6) 2.39.0 app-misc/pax-utils: 1.3.5::gentoo app-shells/bash: 5.1_p16-r4::gentoo dev-lang/perl: 5.36.0-r2::gentoo dev-lang/python: 3.11.3::gentoo dev-util/cmake: 3.26.3::gentoo dev-util/meson: 1.0.1::gentoo sys-apps/baselayout: 2.13-r1::gentoo sys-apps/openrc: 0.46::gentoo sys-apps/sandbox: 2.29::gentoo sys-devel/autoconf: 2.71-r5::gentoo sys-devel/automake: 1.16.5::gentoo sys-devel/binutils: 2.39-r5::gentoo sys-devel/binutils-config: 5.5::gentoo sys-devel/gcc: 12.2.1_p20230428-r1::gentoo sys-devel/gcc-config: 2.10::gentoo sys-devel/libtool: 2.4.7-r1::gentoo sys-devel/make: 4.4.1-r1::gentoo sys-kernel/linux-headers: 6.1::gentoo (virtual/os-headers) sys-libs/glibc: 2.36-r8::gentoo Repositories: gentoo location: /usr/portage sync-type: rsync sync-uri: rsync://rsync.gentoo.org/gentoo-portage priority: -1000 volatile: True sync-rsync-verify-max-age: 24 sync-rsync-extra-opts: sync-rsync-verify-metamanifest: yes sync-rsync-verify-jobs: 1 ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=haswell -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=haswell -O2 -pipe" DISTDIR="/usr/portage/distfiles" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg-live config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LEX="flex" MAKEOPTS="-j8" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" SHELL="/bin/bash" USE="acl amd64 bash-completion bzip2 cli crypt dri fortran gdbm iconv ipv6 libtirpc multilib ncurses nls nptl openmp pam pcre perl python readline samba seccomp smp split-usr ssl test-rust threads unicode xattr zlib" ABI_X86="64" ADA_TARGET="gnat_2021" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 fma3 mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="pc" INPUT_DEVICES="libinput" KERNEL="linux" L10N="en" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-4 php8-0" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_11" PYTHON_TARGETS="python3_11" RUBY_TARGETS="ruby30 ruby31" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EMERGE_DEFAULT_OPTS, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS
File system is ext4.
The problem occurs also with sys-apps/coreutils-9.2-r2.
I git bisected the upstream source. The problem starts with this commit: https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=093a8b4bfaba60005f14493ce7ef11ed665a0176
Created attachment 862967 [details, diff] With this patch copying large files works again on affected systems. The attached patch fixes the problem, but I have no insight in the deeper reasons for this.
Could you strace the failing cp command so we can see the exact arguments passed to copy_file_range()?
execve("./cp", ["./cp", "file_a", "file_b"], 0x7fff39d79210 /* 35 vars */) = 0 brk(NULL) = 0x5622c2345000 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0406643000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=23153, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 23153, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f040663d000 close(3) = 0 openat(AT_FDCWD, "/lib64/libacl.so.1", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=39216, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 41456, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f0406632000 mmap(0x7f0406634000, 20480, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f0406634000 mmap(0x7f0406639000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7f0406639000 mmap(0x7f040663b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x7f040663b000 close(3) = 0 openat(AT_FDCWD, "/lib64/libattr.so.1", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=26808, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 29024, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f040662a000 mmap(0x7f040662c000, 12288, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f040662c000 mmap(0x7f040662f000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f040662f000 mmap(0x7f0406630000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f0406630000 close(3) = 0 openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\2204\2\0\0\0\0\0"..., 832) = 832 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=1876792, ...}, AT_EMPTY_PATH) = 0 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 mmap(NULL, 1903920, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f0406459000 mmap(0x7f040647b000, 1372160, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x22000) = 0x7f040647b000 mmap(0x7f04065ca000, 335872, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x171000) = 0x7f04065ca000 mmap(0x7f040661c000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c3000) = 0x7f040661c000 mmap(0x7f0406622000, 32048, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f0406622000 close(3) = 0 mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0406456000 arch_prctl(ARCH_SET_FS, 0x7f0406456740) = 0 set_tid_address(0x7f0406456a10) = 28586 set_robust_list(0x7f0406456a20, 24) = 0 rseq(0x7f0406457060, 0x20, 0, 0x53053053) = 0 mprotect(0x7f040661c000, 16384, PROT_READ) = 0 mprotect(0x7f0406630000, 4096, PROT_READ) = 0 mprotect(0x7f040663b000, 4096, PROT_READ) = 0 mprotect(0x5622c0c72000, 4096, PROT_READ) = 0 mprotect(0x7f0406676000, 8192, PROT_READ) = 0 prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0 munmap(0x7f040663d000, 23153) = 0 getrandom("\xf2\xea\x43\x89\xec\xdd\x23\x35", 8, GRND_NONBLOCK) = 8 brk(NULL) = 0x5622c2345000 brk(0x5622c2366000) = 0x5622c2366000 openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=4375232, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 4375232, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f0406029000 close(3) = 0 geteuid() = 131 openat(AT_FDCWD, "file_b", O_RDONLY|O_PATH|O_DIRECTORY) = -1 ENOENT (No such file or directory) newfstatat(AT_FDCWD, "file_a", {st_mode=S_IFREG|0754, st_size=3665008640, ...}, 0) = 0 openat(AT_FDCWD, "file_a", O_RDONLY) = 3 newfstatat(3, "", {st_mode=S_IFREG|0754, st_size=3665008640, ...}, AT_EMPTY_PATH) = 0 openat(AT_FDCWD, "file_b", O_WRONLY|O_CREAT|O_EXCL, 0754) = 4 ioctl(4, BTRFS_IOC_CLONE or FICLONE, 3) = -1 EOPNOTSUPP (Operation not supported) newfstatat(4, "", {st_mode=S_IFREG|0754, st_size=0, ...}, AT_EMPTY_PATH) = 0 fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 copy_file_range(3, NULL, 4, NULL, 9223372035781033984, 0) = 2147479552 copy_file_range(3, NULL, 4, NULL, 9223372035781033984, 0) = -1 EINVAL (Invalid argument) openat(AT_FDCWD, "/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 5 newfstatat(5, "", {st_mode=S_IFREG|0644, st_size=2998, ...}, AT_EMPTY_PATH) = 0 read(5, "# Locale name alias data base.\n#"..., 4096) = 2998 read(5, "", 4096) = 0 close(5) = 0 openat(AT_FDCWD, "/usr/local/share/locale/en_US/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/local/share/locale/en/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory) write(2, "./cp: ", 6./cp: ) = 6 write(2, "error copying 'file_a' to 'file_"..., 34error copying 'file_a' to 'file_b') = 34 openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) write(2, ": Invalid argument", 18: Invalid argument) = 18 write(2, "\n", 1 ) = 1 close(4) = 0 close(3) = 0 lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) close(0) = 0 close(1) = 0 close(2) = 0 exit_group(1) = ? +++ exited with 1 +++
So copy_file_range() gets called twice: The first call successfully copies 2147479552 bytes The second call returns -1 / EINVAL. This is unexpected, and not documented in the man page. This certainly seems like a kernel bug. I wonder if a fix has been backported to the 4.19 stable branch. Are you able to test with 4.19.284 (or the latest version)? I don't think there's really anything to be fixed in coreutils here, but feel free to report it upstream if you think otherwise.
No change with 4.19.284. This may be a kernel bug but it is still a regression in coreutils. The man page of copy_file_range states that "A major rework of the kernel implementation occurred in 5.3." So IMHO if coreutils breaks compatibility with kernel versions before 5.3 it should at least explicitly say so somewhere.
Again, you are welcome to report the issue upstream.
I went ahead and filed it upstream as https://debbugs.gnu.org/63850.
(In reply to Rojhalat Ibrahim from comment #11) > No change with 4.19.284. > > This may be a kernel bug but it is still a regression in coreutils. The man > page of copy_file_range states that "A major rework of the kernel > implementation occurred in 5.3." > > So IMHO if coreutils breaks compatibility with kernel versions before 5.3 it > should at least explicitly say so somewhere. Rojhalat, could you try the patch upstream give at https://debbugs.gnu.org/cgi/bugreport.cgi?bug=63850#8, and share the new output? Thanks.
(In reply to Sam James from comment #14) > (In reply to Rojhalat Ibrahim from comment #11) > > No change with 4.19.284. > > > > This may be a kernel bug but it is still a regression in coreutils. The man > > page of copy_file_range states that "A major rework of the kernel > > implementation occurred in 5.3." > > > > So IMHO if coreutils breaks compatibility with kernel versions before 5.3 it > > should at least explicitly say so somewhere. > > Rojhalat, could you try the patch upstream give at > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=63850#8, and share the new > output? Thanks. Actually https://debbugs.gnu.org/cgi/bugreport.cgi?bug=63850#11 (thanks floppym). Also, could you run (on the bad machine) "ebuild /var/db/repos/gentoo/sys-apps/coreutils/coreutils-9.3-r1.ebuild clean install", and upload the config.log & full build.log here as attachments? Thanks.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=a3d0fcb6d239631f19e03e704ff80c8bbfb6a70b commit a3d0fcb6d239631f19e03e704ff80c8bbfb6a70b Author: Sam James <sam@gentoo.org> AuthorDate: 2023-06-09 08:29:33 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-06-09 08:29:45 +0000 sys-apps/coreutils: backport fix for old kernels + copy_file_range Closes: https://bugs.gentoo.org/907474 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/coreutils/coreutils-9.3-r2.ebuild | 273 +++++++++++++++++++++ .../coreutils-9.3-old-kernel-copy_file_range.patch | 89 +++++++ 2 files changed, 362 insertions(+)
Sorry, was traveling and could not test. But it is working now with 9.3-r2. Thanks for the fix.
(In reply to Rojhalat Ibrahim from comment #17) > Sorry, was traveling and could not test. But it is working now with 9.3-r2. > Thanks for the fix. Thanks for confirming!
