Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bugzilla DB migration completed. Please report issues to Infra team via email via infra@gentoo.org or IRC

Bug 500382

Summary: sys-libs/uclibc: pread/pwrite do not work on arm (breaks dev-vcs/git)
Product: Gentoo Linux Reporter: Steev Klimaszewski <steev>
Component: Current packagesAssignee: Embedded Gentoo Team <embedded>
Status: RESOLVED FIXED    
Severity: major CC: blueness
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---
Attachments: strace of git running on uclibc
uclibc savedconfig
ad hoc fix for non-atomic pread/write on armv7a-uclibc.

Description Steev Klimaszewski gentoo-dev 2014-02-05 15:19:45 UTC
Opening a bug to track this since 1.8 was stabilized and the old versions were removed, and blueness keeps bugging me to provide him with info about what's broken about it...

Reproducible: Always
Comment 1 Steev Klimaszewski gentoo-dev 2014-02-05 15:24:33 UTC
Created attachment 369614 [details]
strace of git running on uclibc

This was an attempt to run the following: 

git clone git://github.com/hardkernel/linux -b odroid-3.8.y --depth=1    

Which is the kernel used on an odroid u3 device.

[I] dev-vcs/git
     Available versions:  1.8.1.5 1.8.1.5-r1 ~1.8.2.1 1.8.3.2-r1 ~1.8.4.5 ~1.8.5.3 **9999 {+blksha1 cgi +curl cvs doc emacs gnome-keyring +gpg gtk highlight +iconv mediawiki +nls +pcre +perl ppcsha1 +python subversion test +threads tk +webdav xinetd PYTHON_SINGLE_TARGET="python2_6 python2_7" PYTHON_TARGETS="python2_6 python2_7"}
     Installed versions:  1.8.3.2-r1(14:30:32 02/05/14)(blksha1 gpg iconv pcre perl python threads -cgi -curl -cvs -doc -emacs -gnome-keyring -gtk -highlight -nls -ppcsha1 -subversion -test -tk -webdav -xinetd PYTHON_SINGLE_TARGET="python2_7 -python2_6" PYTHON_TARGETS="python2_7 -python2_6")


[I] sys-libs/uclibc
     Available versions:  0.9.33.2^s (~)0.9.33.2-r1^s 0.9.33.2-r2^s 0.9.33.2-r3^s 0.9.33.2-r4^s (~)0.9.33.2-r5^s 0.9.33.2-r6^s (~)0.9.33.2-r7^s 0.9.33.2-r8^s (~)0.9.33.2-r9^s **9999^s {debug hardened iconv ipv6 nptl rpc savedconfig ssp uclibc-compat wordexp CROSSCOMPILE_OPTS="headers-only"}
     Installed versions:  0.9.33.2-r9^s(12:21:38 02/05/14)(savedconfig -debug -hardened -iconv -ipv6 -nptl -rpc -ssp -uclibc-compat -wordexp CROSSCOMPILE_OPTS="-headers-only")
Comment 2 Steev Klimaszewski gentoo-dev 2014-02-05 15:25:42 UTC
emerge --info
Portage 2.2.7 (hardened/linux/uclibc/arm/armv7a, gcc-4.7.3, uclibc-0.9.33.2-r9, 3.4.74 armv7l)
=================================================================
System uname: Linux-3.4.74-armv7l-ARMv7_Processor_rev_3_-v7l-with-gentoo-2.2
KiB Mem:     1789860 total,    763028 free
KiB Swap:    2097148 total,   2096864 free
Timestamp of tree: Wed, 05 Feb 2014 00:00:01 +0000
ld GNU ld (GNU Binutils) 2.23.2
app-shells/bash:          4.2_p45
dev-lang/python:          2.7.5-r3, 3.2.5-r3, 3.3.3
dev-util/pkgconfig:       0.28
sys-apps/baselayout:      2.2
sys-apps/openrc:          0.12.4
sys-apps/sandbox:         2.6-r1
sys-devel/autoconf:       2.69
sys-devel/automake:       1.13.4
sys-devel/binutils:       2.23.2
sys-devel/gcc:            4.7.3-r1
sys-devel/gcc-config:     1.7.3
sys-devel/libtool:        2.4.2
sys-devel/make:           3.82-r4
sys-kernel/linux-headers: 3.9 (virtual/os-headers)
sys-libs/uclibc:          0.9.33.2-r9
Repositories: gentoo
ACCEPT_KEYWORDS="arm"
ACCEPT_LICENSE="* -@EULA"
CBUILD="armv7a-hardfloat-linux-uclibceabi"
CFLAGS="-O2 -pipe -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard"
CHOST="armv7a-hardfloat-linux-uclibceabi"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="-j9 --load-average=9"
FCFLAGS="-O2 -pipe -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j9 -l9"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY=""
SYNC="rsync://xu/gentoo-portage"
USE="arm bindist cli cracklib crypt cxx dri fortran iconv ipv6 modules ncurses nptl openmp pax_kernel pcre pic readline session ssl tcpd uclibc unicode xattr zlib" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="uclibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3" RUBY_TARGETS="ruby19 ruby18" USERLAND="GNU" VIDEO_CARDS="dummy fbdev v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
Comment 3 Steev Klimaszewski gentoo-dev 2014-02-05 17:29:28 UTC
So, here is some fun that I ran into - I updated uclibc on another machine, that still has git 1.7.12.4 installed.

[I] sys-libs/uclibc
     Available versions:  0.9.33.2^s ~0.9.33.2-r1^s 0.9.33.2-r2^s 0.9.33.2-r3^s 0.9.33.2-r4^s ~0.9.33.2-r5^s 0.9.33.2-r6^s ~0.9.33.2-r7^s 0.9.33.2-r8^s ~0.9.33.2-r9^s **9999^s {{crosscompile_opts_headers-only debug hardened iconv ipv6 nptl rpc savedconfig ssp uclibc-compat wordexp}}
     Installed versions:  0.9.33.2-r8^s(14:36:13 02/05/14)(savedconfig -crosscompile_opts_headers-only -debug -hardened -iconv -ipv6 -nptl -rpc -ssp -uclibc-compat -wordexp)

(git version)
Installed versions:  1.7.12.4(04:02:25 01/24/13)(blksha1 curl gpg pcre perl python threads webdav -cgi -cvs -doc -emacs -gtk -highlight -iconv -nls -ppcsha1 -subversion -test -tk -xinetd)

kunlun ~ # git clone git://github.com/offensive-security/kali-arm-build-scripts
Cloning into 'kali-arm-build-scripts'...
remote: Reusing existing pack: 191, done.
remote: Total 191 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (191/191), 151.07 KiB, done.
fatal: premature end of pack file, 47 bytes missing
fatal: index-pack failed


So now, on newer uclibc, even git 1.7 fails.

will attach the config that is used by these machines for uclibc, perhaps we're missing something on arm...?
Comment 4 Steev Klimaszewski gentoo-dev 2014-02-05 17:30:49 UTC
Created attachment 369626 [details]
uclibc savedconfig

uclibc config used (originally comes from 0.9.33.2 (vanilla, not -rX))
Comment 5 Anthony Basile gentoo-dev 2014-02-16 22:49:18 UTC
(In reply to Steev Klimaszewski from comment #4)
> Created attachment 369626 [details]
> uclibc savedconfig
> 
> uclibc config used (originally comes from 0.9.33.2 (vanilla, not -rX))

You need the following from uclibc's master branch:

http://git.uclibc.org/uClibc/commit/?id=564a95241f8468145ff85998ef9173c46aed7636

Unfortunately, the amount of work done on pread/pwrite since 0.9.33.2 came out make it difficult to cleanly apply that patch to the head of the 0.9.33.x branch.  I'm going to have to hack a bit to port it over to the branch.

BTW, you don't need all that commit for arm, only the __UCLIBC_SYSCALL_ALIGN_64BIT__ stuff (hunk 1).  However, I'm not 100% sure why we need 64-bit alignment.
Comment 6 Anthony Basile gentoo-dev 2014-02-17 22:14:05 UTC
(In reply to Anthony Basile from comment #5)
> BTW, you don't need all that commit for arm, only the
> __UCLIBC_SYSCALL_ALIGN_64BIT__ stuff (hunk 1).  However, I'm not 100% sure
> why we need 64-bit alignment.

I narrowed it down to the difference between _syscall6 and _syscall5.  In uclibc-0.9.33.2, pread/write uses _syscall5, but in uclibc-9999, __UCLIBC_SYSCALL_ALIGN_64BIT__ is defined in libc/sysdeps/linux/arm/bits/uClibc_arch_features.h and this triggers the use of _syscall6.

Why _syscall5 is not atomic but _syscall6 is, I'm not sure yet.  I'll give a quick-fix patch in my next post, but its just for analysis purpose or as an ad hoc fix for just armv7a, but it would break other arches.
Comment 7 Anthony Basile gentoo-dev 2014-02-17 22:15:12 UTC
Created attachment 370656 [details, diff]
ad hoc fix for non-atomic pread/write on armv7a-uclibc.
Comment 8 Anthony Basile gentoo-dev 2014-02-17 22:31:50 UTC
Okay here's why I'm confused: both _syscall5 and 6 expand the same way on arm_eabi:

1. in libc/sysdeps/linux/common/bits/syscalls-common.h _syscall{5,6} are defined as follows

#define _syscall5(args...)              SYSCALL_FUNC(5, args)
#define _syscall6(args...)              SYSCALL_FUNC(6, args)


2. SYSCALL_FUNC expands as follows 

#define SYSCALL_FUNC(nargs, type, name, args...)                        \
type name(C_DECL_ARGS_##nargs(args)) {                                  \
        return (type)INLINE_SYSCALL(name, nargs, C_ARGS_##nargs(args)); \
}


3. and INLINE_SYSCALL is defined as


define INLINE_SYSCALL(name, nr, args...) INLINE_SYSCALL_NCS(__NR_##name, nr, args)

# define INLINE_SYSCALL_NCS(name, nr, args...)                          \
(__extension__                                                          \
 ({                                                                     \
        INTERNAL_SYSCALL_DECL(__err);                                   \
        (__extension__                                                  \
         ({                                                             \
           long __res = INTERNAL_SYSCALL_NCS(name, __err, nr, args);    \
           if (unlikely(INTERNAL_SYSCALL_ERROR_P(__res, __err))) {      \
                __set_errno(INTERNAL_SYSCALL_ERRNO(__res, __err));      \
                __res = -1L;                                            \
           }                                                            \
           __res;                                                       \
          })                                                            \
        );                                                              \
  })                                                                    \
)


4.  INTERNAL_SYSCALL_NCS is defined in include/bits/syscalls.h where for __ARM_EABI__ it expands to

#define INTERNAL_SYSCALL_NCS(name, err, nr, args...)                    \
(__extension__ \
  ({unsigned int __internal_sys_result;                                 \
     {                                                                  \
       register int __a1 __asm__ ("r0"), _nr __asm__ ("r7");            \
       LOAD_ARGS_##nr (args)                                            \
       _nr = (name);                                                    \
       __asm__ __volatile__ ("swi       0x0     @ syscall " #name       \
                             : "=r" (__a1)                              \
                             : "r" (_nr) ASM_ARGS_##nr                  \
                             : "memory");                               \
               __internal_sys_result = __a1;                            \
     }                                                                  \
     (int) __internal_sys_result; }) \
)



5. So both _syscall5 and 6 expand the same way.  Can the non-atomicity of _syscall5 be a kernel issue?
Comment 10 SpanKY gentoo-dev 2014-02-18 01:05:36 UTC
(In reply to Anthony Basile from comment #5)

i'm afraid all of your leads seem to be off track.  check out the section "Architecture-specific requirements" of the syscall(2) man page as it explains things.
Comment 11 SpanKY gentoo-dev 2014-02-18 01:06:49 UTC
should be all set now in the tree; thanks for the report!

Commit message: Add more pread/pwrite fixes from upstream
http://sources.gentoo.org/sys-libs/uclibc/uclibc-0.9.33.2-r10.ebuild?rev=1.1
Comment 12 Anthony Basile gentoo-dev 2014-02-18 12:26:09 UTC
(In reply to SpanKY from comment #10)
> (In reply to Anthony Basile from comment #5)
> 
> i'm afraid all of your leads seem to be off track.  check out the section
> "Architecture-specific requirements" of the syscall(2) man page as it
> explains things.

Its because the 64-bit values need to be aligned to an EVEN register pair.  So the macro

 _syscall6(ssize_t, __syscall_pwrite, int, fd, const void *, buf,
                 size_t, count, int, dummy, off_t, offset_hi, off_t, offset_lo)

means somethng like

  syscall(SYS_pread64, fd, buf, count, dummy, offset_hi, offset_lo);

which are loaded as follows for arm_eabi:

  r0=fd  r1=buf  r2=count  r3=dummy  r4=offset_hi  r5=offset_lo

and the alignment is on r4/r5 --- and EVEN pair.  While 

  _syscall5(ssize_t, __syscall_pread, int, fd, void *, buf,
                 size_t, count, off_t, offset_hi, off_t, offset_lo)

gives

   syscall(SYS_pread64, fd, buff, count, offset_hi, offset_lo)

and now we get r3=offset_hi, r4=offset_lo --- and ODD pair.

Do I pass?  Anyhow, I don't know arm that well, but this is like mips.
Comment 13 SpanKY gentoo-dev 2014-02-18 13:58:04 UTC
(In reply to Anthony Basile from comment #12)

that is correct.  you get a cookie ;).