Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bugzilla DB migration completed. Please report issues to Infra team via email via or IRC

Bug 500382

Summary: sys-libs/uclibc: pread/pwrite do not work on arm (breaks dev-vcs/git)
Product: Gentoo Linux Reporter: Steev Klimaszewski <steev>
Component: Current packagesAssignee: Embedded Gentoo Team <embedded>
Severity: major CC: blueness
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
Package list:
Runtime testing required: ---
Attachments: strace of git running on uclibc
uclibc savedconfig
ad hoc fix for non-atomic pread/write on armv7a-uclibc.

Description Steev Klimaszewski gentoo-dev 2014-02-05 15:19:45 UTC
Opening a bug to track this since 1.8 was stabilized and the old versions were removed, and blueness keeps bugging me to provide him with info about what's broken about it...

Reproducible: Always
Comment 1 Steev Klimaszewski gentoo-dev 2014-02-05 15:24:33 UTC
Created attachment 369614 [details]
strace of git running on uclibc

This was an attempt to run the following: 

git clone git:// -b odroid-3.8.y --depth=1    

Which is the kernel used on an odroid u3 device.

[I] dev-vcs/git
     Available versions: ~ ~ ~ **9999 {+blksha1 cgi +curl cvs doc emacs gnome-keyring +gpg gtk highlight +iconv mediawiki +nls +pcre +perl ppcsha1 +python subversion test +threads tk +webdav xinetd PYTHON_SINGLE_TARGET="python2_6 python2_7" PYTHON_TARGETS="python2_6 python2_7"}
     Installed versions: 02/05/14)(blksha1 gpg iconv pcre perl python threads -cgi -curl -cvs -doc -emacs -gnome-keyring -gtk -highlight -nls -ppcsha1 -subversion -test -tk -webdav -xinetd PYTHON_SINGLE_TARGET="python2_7 -python2_6" PYTHON_TARGETS="python2_7 -python2_6")

[I] sys-libs/uclibc
     Available versions:^s (~)^s^s^s^s (~)^s^s (~)^s^s (~)^s **9999^s {debug hardened iconv ipv6 nptl rpc savedconfig ssp uclibc-compat wordexp CROSSCOMPILE_OPTS="headers-only"}
     Installed versions:^s(12:21:38 02/05/14)(savedconfig -debug -hardened -iconv -ipv6 -nptl -rpc -ssp -uclibc-compat -wordexp CROSSCOMPILE_OPTS="-headers-only")
Comment 2 Steev Klimaszewski gentoo-dev 2014-02-05 15:25:42 UTC
emerge --info
Portage 2.2.7 (hardened/linux/uclibc/arm/armv7a, gcc-4.7.3, uclibc-, 3.4.74 armv7l)
System uname: Linux-3.4.74-armv7l-ARMv7_Processor_rev_3_-v7l-with-gentoo-2.2
KiB Mem:     1789860 total,    763028 free
KiB Swap:    2097148 total,   2096864 free
Timestamp of tree: Wed, 05 Feb 2014 00:00:01 +0000
ld GNU ld (GNU Binutils) 2.23.2
app-shells/bash:          4.2_p45
dev-lang/python:          2.7.5-r3, 3.2.5-r3, 3.3.3
dev-util/pkgconfig:       0.28
sys-apps/baselayout:      2.2
sys-apps/openrc:          0.12.4
sys-apps/sandbox:         2.6-r1
sys-devel/autoconf:       2.69
sys-devel/automake:       1.13.4
sys-devel/binutils:       2.23.2
sys-devel/gcc:            4.7.3-r1
sys-devel/gcc-config:     1.7.3
sys-devel/libtool:        2.4.2
sys-devel/make:           3.82-r4
sys-kernel/linux-headers: 3.9 (virtual/os-headers)
Repositories: gentoo
CFLAGS="-O2 -pipe -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard"
EMERGE_DEFAULT_OPTS="-j9 --load-average=9"
FCFLAGS="-O2 -pipe -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j9 -l9"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
USE="arm bindist cli cracklib crypt cxx dri fortran iconv ipv6 modules ncurses nptl openmp pax_kernel pcre pic readline session ssl tcpd uclibc unicode xattr zlib" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="uclibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3" RUBY_TARGETS="ruby19 ruby18" USERLAND="GNU" VIDEO_CARDS="dummy fbdev v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Comment 3 Steev Klimaszewski gentoo-dev 2014-02-05 17:29:28 UTC
So, here is some fun that I ran into - I updated uclibc on another machine, that still has git installed.

[I] sys-libs/uclibc
     Available versions:^s ~^s^s^s^s ~^s^s ~^s^s ~^s **9999^s {{crosscompile_opts_headers-only debug hardened iconv ipv6 nptl rpc savedconfig ssp uclibc-compat wordexp}}
     Installed versions:^s(14:36:13 02/05/14)(savedconfig -crosscompile_opts_headers-only -debug -hardened -iconv -ipv6 -nptl -rpc -ssp -uclibc-compat -wordexp)

(git version)
Installed versions: 01/24/13)(blksha1 curl gpg pcre perl python threads webdav -cgi -cvs -doc -emacs -gtk -highlight -iconv -nls -ppcsha1 -subversion -test -tk -xinetd)

kunlun ~ # git clone git://
Cloning into 'kali-arm-build-scripts'...
remote: Reusing existing pack: 191, done.
remote: Total 191 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (191/191), 151.07 KiB, done.
fatal: premature end of pack file, 47 bytes missing
fatal: index-pack failed

So now, on newer uclibc, even git 1.7 fails.

will attach the config that is used by these machines for uclibc, perhaps we're missing something on arm...?
Comment 4 Steev Klimaszewski gentoo-dev 2014-02-05 17:30:49 UTC
Created attachment 369626 [details]
uclibc savedconfig

uclibc config used (originally comes from (vanilla, not -rX))
Comment 5 Anthony Basile gentoo-dev 2014-02-16 22:49:18 UTC
(In reply to Steev Klimaszewski from comment #4)
> Created attachment 369626 [details]
> uclibc savedconfig
> uclibc config used (originally comes from (vanilla, not -rX))

You need the following from uclibc's master branch:

Unfortunately, the amount of work done on pread/pwrite since came out make it difficult to cleanly apply that patch to the head of the 0.9.33.x branch.  I'm going to have to hack a bit to port it over to the branch.

BTW, you don't need all that commit for arm, only the __UCLIBC_SYSCALL_ALIGN_64BIT__ stuff (hunk 1).  However, I'm not 100% sure why we need 64-bit alignment.
Comment 6 Anthony Basile gentoo-dev 2014-02-17 22:14:05 UTC
(In reply to Anthony Basile from comment #5)
> BTW, you don't need all that commit for arm, only the
> __UCLIBC_SYSCALL_ALIGN_64BIT__ stuff (hunk 1).  However, I'm not 100% sure
> why we need 64-bit alignment.

I narrowed it down to the difference between _syscall6 and _syscall5.  In uclibc-, pread/write uses _syscall5, but in uclibc-9999, __UCLIBC_SYSCALL_ALIGN_64BIT__ is defined in libc/sysdeps/linux/arm/bits/uClibc_arch_features.h and this triggers the use of _syscall6.

Why _syscall5 is not atomic but _syscall6 is, I'm not sure yet.  I'll give a quick-fix patch in my next post, but its just for analysis purpose or as an ad hoc fix for just armv7a, but it would break other arches.
Comment 7 Anthony Basile gentoo-dev 2014-02-17 22:15:12 UTC
Created attachment 370656 [details, diff]
ad hoc fix for non-atomic pread/write on armv7a-uclibc.
Comment 8 Anthony Basile gentoo-dev 2014-02-17 22:31:50 UTC
Okay here's why I'm confused: both _syscall5 and 6 expand the same way on arm_eabi:

1. in libc/sysdeps/linux/common/bits/syscalls-common.h _syscall{5,6} are defined as follows

#define _syscall5(args...)              SYSCALL_FUNC(5, args)
#define _syscall6(args...)              SYSCALL_FUNC(6, args)

2. SYSCALL_FUNC expands as follows 

#define SYSCALL_FUNC(nargs, type, name, args...)                        \
type name(C_DECL_ARGS_##nargs(args)) {                                  \
        return (type)INLINE_SYSCALL(name, nargs, C_ARGS_##nargs(args)); \

3. and INLINE_SYSCALL is defined as

define INLINE_SYSCALL(name, nr, args...) INLINE_SYSCALL_NCS(__NR_##name, nr, args)

# define INLINE_SYSCALL_NCS(name, nr, args...)                          \
(__extension__                                                          \
 ({                                                                     \
        INTERNAL_SYSCALL_DECL(__err);                                   \
        (__extension__                                                  \
         ({                                                             \
           long __res = INTERNAL_SYSCALL_NCS(name, __err, nr, args);    \
           if (unlikely(INTERNAL_SYSCALL_ERROR_P(__res, __err))) {      \
                __set_errno(INTERNAL_SYSCALL_ERRNO(__res, __err));      \
                __res = -1L;                                            \
           }                                                            \
           __res;                                                       \
          })                                                            \
        );                                                              \
  })                                                                    \

4.  INTERNAL_SYSCALL_NCS is defined in include/bits/syscalls.h where for __ARM_EABI__ it expands to

#define INTERNAL_SYSCALL_NCS(name, err, nr, args...)                    \
(__extension__ \
  ({unsigned int __internal_sys_result;                                 \
     {                                                                  \
       register int __a1 __asm__ ("r0"), _nr __asm__ ("r7");            \
       LOAD_ARGS_##nr (args)                                            \
       _nr = (name);                                                    \
       __asm__ __volatile__ ("swi       0x0     @ syscall " #name       \
                             : "=r" (__a1)                              \
                             : "r" (_nr) ASM_ARGS_##nr                  \
                             : "memory");                               \
               __internal_sys_result = __a1;                            \
     }                                                                  \
     (int) __internal_sys_result; }) \

5. So both _syscall5 and 6 expand the same way.  Can the non-atomicity of _syscall5 be a kernel issue?
Comment 10 SpanKY gentoo-dev 2014-02-18 01:05:36 UTC
(In reply to Anthony Basile from comment #5)

i'm afraid all of your leads seem to be off track.  check out the section "Architecture-specific requirements" of the syscall(2) man page as it explains things.
Comment 11 SpanKY gentoo-dev 2014-02-18 01:06:49 UTC
should be all set now in the tree; thanks for the report!

Commit message: Add more pread/pwrite fixes from upstream
Comment 12 Anthony Basile gentoo-dev 2014-02-18 12:26:09 UTC
(In reply to SpanKY from comment #10)
> (In reply to Anthony Basile from comment #5)
> i'm afraid all of your leads seem to be off track.  check out the section
> "Architecture-specific requirements" of the syscall(2) man page as it
> explains things.

Its because the 64-bit values need to be aligned to an EVEN register pair.  So the macro

 _syscall6(ssize_t, __syscall_pwrite, int, fd, const void *, buf,
                 size_t, count, int, dummy, off_t, offset_hi, off_t, offset_lo)

means somethng like

  syscall(SYS_pread64, fd, buf, count, dummy, offset_hi, offset_lo);

which are loaded as follows for arm_eabi:

  r0=fd  r1=buf  r2=count  r3=dummy  r4=offset_hi  r5=offset_lo

and the alignment is on r4/r5 --- and EVEN pair.  While 

  _syscall5(ssize_t, __syscall_pread, int, fd, void *, buf,
                 size_t, count, off_t, offset_hi, off_t, offset_lo)


   syscall(SYS_pread64, fd, buff, count, offset_hi, offset_lo)

and now we get r3=offset_hi, r4=offset_lo --- and ODD pair.

Do I pass?  Anyhow, I don't know arm that well, but this is like mips.
Comment 13 SpanKY gentoo-dev 2014-02-18 13:58:04 UTC
(In reply to Anthony Basile from comment #12)

that is correct.  you get a cookie ;).