Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 237974 - glibc time functionality broken with kernel 2.6.26 and later
Summary: glibc time functionality broken with kernel 2.6.26 and later
Status: RESOLVED UPSTREAM
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Core system (show other bugs)
Hardware: All Linux
: High normal (vote)
Assignee: Gentoo Toolchain Maintainers
URL: http://sources.redhat.com/bugzilla/sh...
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2008-09-18 00:20 UTC by Hal Engel
Modified: 2009-05-09 21:05 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
ntp-kernel-2.6.26.patch (ntp-kernel-2.6.26.patch,2.52 KB, patch)
2008-09-20 17:05 UTC, Hal Engel
Details | Diff
glibc-2.7-r2.ebuild that uses the nanokernal patch (glibc-2.7-r2.ebuild,12.41 KB, text/plain)
2008-09-20 17:07 UTC, Hal Engel
Details
glibc-2.8_p20080602.ebuild that uses the nanokernel patch (glibc-2.8_p20080602.ebuild,13.03 KB, text/plain)
2008-09-20 17:07 UTC, Hal Engel
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Hal Engel 2008-09-18 00:20:50 UTC
Kernel 2.6.26 changed it's time functionality from being microsecond based (microkernel) to being nanosecond based (nanokernel).  As a result there needs to be changes made to sysdeps/unix/sysv/linux/sys/timex.h and also to sysdeps/unix/sysv/linux/ntp_gettime.c to make glibc compatible with these newer kernel versions.  If this is not done ntp will not function correctly.  Please see http://sourceware.org/ml/libc-alpha/2008-03/msg00076.html for a set of patches based on glibc 2.7 that will correct this problem. 

Reproducible: Always

Actual Results:  
With the current code base ntp builds for a microkernel and it's functions to get the time and to set the clocks rate will be off by 3 orders of magnitude since the kernel is passing and expecting to get nanosecond values.  This results in the clock being unstable.  This is most noticeable when using a high quality reference clock like a GPS where it should be possible to get sub microsecond offsets with a properly functioning clock but with the current glibc code it is more typical to see around 100 microsecond offsets with the clock slewing between positive and negative offsets and sometimes being off by as much as 500 microseconds. 

Expected Results:  
ntp should build for a nanosecond kernel and ntpd should be able to hold near 1 microsecond offsets much of the time.

This is a wide spread problem most current Linux distros
Comment 1 cyrillic 2008-09-18 22:17:25 UTC
I am just curious.

If you recompile glibc against 2.6.26 kernel headers, does this improve the situation ?
Comment 2 Hal Engel 2008-09-18 22:56:54 UTC
No in order to get the right defines in /usr/include/sys/timex.h for a nanokernel you must use the patches that I provided a link to in the original post or something similar.  In addition the patch also makes a change to ntp_gettime.c to bring the glibc ntp_gettime() API into conformance with the new Linux time API (there was a change to a struct).  I agree with your thinking that the glibc build should look at /usr/include/linux/timex.h and do the right thing when it creates /usr/include.sys/timex.h but it does not.  The current timex.h from glibc is based on a Linux 2.2 kernel and has not been changed for a long time.

I have some hacked together ebuilds for glibc-2.7-r2 and glibc-2.8-20080602 that use this patch and when I get these working the way they should I will post them here.  The ebuilds check the linux-header version and if it is >=2.6.26 it applies the patch.  I am currently testing ntp with a patched version of 2.8-20080602 and it appears to be working OK but it has only been a few hours and I need to let ntp run long enough that the clock has settled down before knowing for sure how well it is working.  I am using a Motorola OnCore UT+ reference clock which puts out PPS pulses to within 50 nanoseconds of UTC.  When the ntp PPL corrections have caused the offset to cross 0 several times I expect things to stabilize near a 1 microsecond offset.  Currently it is at about a 23 microsecond offset.  It has not crossed 0 offset yet sonce the last restart.

# ntptime
ntp_gettime() returns code 0 (OK)
  time cc7d57f0.6dc8c90c  Thu, Sep 18 2008 15:42:56.428, (.428845451),
  maximum error 6257 us, estimated error 1 us, TAI offset 0
ntp_adjtime() returns code 0 (OK)
  modes 0x0 (),
  offset 22.849 us, frequency 87.470 ppm, interval 1 s,
  maximum error 6257 us, estimated error 1 us,
  status 0x2001 (PLL,NANO),
  time constant 4, precision 0.001 us, tolerance 500 ppm,

# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*GPS_ONCORE(0)   .GPS.            0 l    9   16  377    0.000    0.023   0.001

Comment 3 Hal Engel 2008-09-20 17:05:28 UTC
Created attachment 165913 [details, diff]
ntp-kernel-2.6.26.patch

glibc patch to sync time related functionality of glibc to kernels version >= 2.6.26.
Comment 4 Hal Engel 2008-09-20 17:07:07 UTC
Created attachment 165914 [details]
glibc-2.7-r2.ebuild that uses the nanokernal patch
Comment 5 Hal Engel 2008-09-20 17:07:52 UTC
Created attachment 165916 [details]
glibc-2.8_p20080602.ebuild that uses the nanokernel patch
Comment 6 Hal Engel 2008-09-20 17:18:46 UTC
Added attachments for a working nanokernel patch, glibc-2.7-r1.ebuild and glibc-2.8_p20080602.ebuild that have been modified to use the nanokernel patch.  The ebuilds are probably some what hackish since I am far from being an ebuild expert and I expect that the dev that maintains these will very likely do this in a totally different way.  

I have tested the glibc-2.8_p20080602.ebuild on an amd64 machine and have been running with it now for 2 days without issues.  NTP is working as it should and I have full nanosecond functionality.  The glibc-2.7-r2.ebuild has not been tested but the changes are identical to the tested patch and should be fine.

The ebuilds only applies the nanokernel patch if >=linux-headers-2.6.26 is installed.  So systems with older kernels (assuming that the right version of linux-headers is installed) will continue to work correctly.
Comment 7 SpanKY gentoo-dev 2008-12-07 22:27:03 UTC
the patch is not ok as is ... it changes the ABI (struct ntptimeval) which means all old binaries will hit memory corruption
Comment 8 Hal Engel 2008-12-28 20:58:42 UTC
Yes old binaries could have issues.  I would argue that glibc and ntp are both broken now without this patch even if they appear to be working for most users.  The kernel made these changes and glibc and ntp are broken now in gentoo and in fact in almost every distro with these newer kernels.  I think this only really affects ntp and related software so the number of "old" binaries that are affected is small.

Isn't there some way to figure out which apps use the affected struct and force those to rebuild?  I have been using these patches for some months now and so has almost everyone who is using the LinuxPPS kernel patches and after rebuilding ntp we have not had any problems caused by this glibc patch.

This issue will only get worse going forward since the LinuxPPS patches are now appearing in mm daily kernel snapshots and it appears that it is highly likely that kernel 2.6.29 will be shipped with the LinuxPPS patches.  This will allow more users to attach reference clocks to their machines and the issues with glibc and ntp and these newer kernels will become painfully apparent when these users do not get the clock accuracy that they were expecting.

My intension is not so much to insist that these patches be used in the in tree ebuilds right away but to create an awareness of the issue and to get the gentoo glibc devs talking to the glibc project to see how this should be handled so that it is eventually fixed.  On the flip side I created the ebuilds attached to this bug report so that other gentoo LinuxPPS users (and there are a number of us) would have a simple way to install a patched version of glibc until such time that this issue is fixed.

I decided to open a bug upstream as well http://sources.redhat.com/bugzilla/show_bug.cgi?id=9690
Comment 9 SpanKY gentoo-dev 2008-12-29 09:16:49 UTC
having broken runtime behavior is preferable to broken ABI
Comment 10 Hal Engel 2009-01-11 02:59:49 UTC
The following was sent to the LinuxPPS email list.  It describes a work around that could be used until glibc is fixed.  It also has a nano second fix for when ntp makes a call to the system to get the current time.

-------------------------------------------------

> I think the following posting is of interest to us. 
> I remember someone patched timex.h to use nano precision with LinuxPPS.

Patching timex.h alone is not enough. You essentially have to patch and 
recompile glibc, else you will get uninitialized values from the .tai 
field, as the poster experienced, because glibc copies the fields and 
messes around instead of just passing the whole structure to the system 
call. Since I did not want to recompile glibc, I built a different 
workaround: I modified timex.h to contain static functions that replace 
the broken ones from glibc. You can get it from
  http://www.mulder.franken.de/blogfiles/timex.h
Note: You do not need to replace your systems timex.h with that - it is
enough to create a subdirectory include/sys in the NTP source tree and 
place it there before compilation.
You will also need to tell configure to use librt (./configure 
LIBS="-lrt") - without librt, ntpd will resort to using gettimeofday for 
getting time from the kernel, and that function only returns 
microseconds because it is defined that way.
-- 
Michael Meier, HPC Services
Friedrich-Alexander-Universitaet Erlangen-Nuernberg
Regionales Rechenzentrum Erlangen
Martensstrasse 1, 91058 Erlangen, Germany
Comment 11 Samuel Thibault 2009-04-23 08:55:58 UTC
Eerrrr, if the kernel does not provide a backward-compatible interface,
it's the kernel that needs to be fixed first, since it's a kernel
regression. Then you can make glibc use the new interface when it
notices that the kernel supports it. Any other way to deal with API
changes will lead to everlasting bugs.
Comment 12 Hal Engel 2009-04-23 19:12:22 UTC
It is clear that this problem became apparent when the kernel time interface changed to a nanosecond interface.  The problem is that timex.h exists in both the kernel and glibc.  The glibc version of timex.h is basically a copy of the header file from kernel version 2.2.  This "worked" (IE. glossed over the underlaying issues) as long as the kernel time interfaces didn't change.  It has also been apparent for some time that at some point the kernel interface would be changed to support nanosecond resolution since there have been attempts to include this change since version 2.4 (the PPSKIT patch set also contained nanosecond patches along with PPS support) and this has been what the time community has wanted for some time.  I think it could be argued that the root of the problem was that glibc made their own static copy of timex.h at some point in the past rather than setting up the glibc build to base it's time interfaces on those of the installed kernel.    

"Then you can make glibc use the new interface when it notices that the kernel supports it."

I am not sure I understand this.  The kernel already has the new interface since it changed from microsecond resolution to nanosecond resolution in version 2.6.26.  It is glibc that does not currently use the new interface and the glibc build makes no attempt to see what the kernel time interface looks like (see #1 and #2).  So I am not sure how "..you can make glibc use the new interface when it notices that the kernel supports it" since the glibc build never attempts to notice what the kernel supports. In fact this is the root problem.    

At some point glibc needs to support nanosecond time keeping.  This bug has not only been posted here but also in the glibc bug tracker.  In addition, this bug has been the subject of threads on the ntp and several other time related email lists (specifically the time nuts and linuxpps email lists) and it is a widely known issue in the kernel, glibc, ntp and time communities.   It has been about 7 months now since this bug and the one in the glibc bug tracker were posted and over a year since the patch set was posted to the glibc email list.  The bug reports are still open and no resolution is in sight.  As far as I can tell we are already in a situation where we have "everlasting bugs."

Since the bug only affects ntp and the kernel and glibc folks don't seem to care that leaving this issue unresolved has broken ntp perhaps the best that distro developers can do is to use the approach suggested in post #10 which is to by pass glibc altogether and do the fix in the ntp build by creating a third copy of timex.h (ugh what a mess!).  By the way using this fix should allow ntp to continue working correctly with older microsecond kernels since ntp with this fix becomes smart enough to ask the kernel if it is a nano or micro second kernel and to adjust its behavior accordingly.

Perhaps this bug should be assigned to the ntp developers instead?  First there is a viable way for them to deal with the issue and second when the kernel and glibc folks finally resolve the issue ntp builds that used the suggested fix will continue to function correctly and new builds will work correctly without the fix.  In the mean time ntp builds that used a glibc build that does not use patches posted here or the fix suggested in comment #10 are broken (IE. have a bug) on any kernel with a version of 2.6.26 or newer.   I think that this means that most gentoo users (some users on some arches may still be using older kernels) are now using a broken version of ntp although most gentoo users that are "time nuts" (admittedly a small part of the gentoo community) will already be using one of these fixes. 

If it would help I can create an ebuild for ntp that contains the suggested fix in post #10 and post it here.
Comment 13 Samuel Thibault 2009-04-23 21:35:10 UTC
Which glibc is it?  I couldn't find it searching for ntp or time.  By "everlasting bug", I mean a bug where forever you have  - an old glibc with a new kernel fails - a new glibc with an old kernel fails  because it becomes a mess to know which can work with what.  Apparently, 2.6.26 changed the interface, _breaking_ glibc. That is NOT supposed to happen and the fix should NOT be in glibc, but in the kernel first, that's really a regression that should be fixed there. Also note that there is not only glibc on earth, there's µclibc, etc. etc.  Recompiling glibc after changing the structures is just wrong: old glibcs will still not work with newer kernels: _that_ is an everlasting bug, you should not be forced to upgrade your glibc just because you upgraded your kernel.  Now, glibc wants to take advantage of nanoseconds whenever they're available. For that to happen, the kernel must provide a way for glibc to know that the interface is available. But if glibc doesn't use it (because it's an old one), the kernel should fallback to the microsecond interface and not break old glibcs! 
Comment 14 Hal Engel 2009-04-24 00:49:00 UTC
Which glibc is it?  I couldn't find it searching for ntp or time.

Are you asking about the glibc bug report?  If so look in comment #8 for a link.

snip

"Now, glibc wants to take advantage of nanoseconds whenever they're available."

No this is not correct glibc does not know anything about microsecond vs. nanoseconds and it is currently missing every single declaration that is related to nanosecond functionality and these patches do not alter how it interfaces with the kernel other than adding the tai field to the ntptimeval structure.  As far as I can tell the kernel has added this field (tai) to be compatible with the nanokernel specifications that are on the ntp web site and that are also used by freeBSD but the kernel does not actually use it yet for backward compatibility reasons (IE. to not break old versions of glibc).

"For that to happen, the kernel must provide a way for glibc to know that the interface is available."

The kernel does in /usr/include/linux/timex.h but glibc does not even attempt to look there when it is being built and the kernel also can be queried at runtime to see if it is a nanosecond kernel and glibc does not do that either.  In other words glibc is completely unaware of even the possibility that the kernel might be a nanokernel on linux systems.  

FreeBSD has had a nanokernel for a long time.  How does glibc and ntp work correctly there?  Perhaps that would give us some clues about what needs to be done but I have not looked.

"But if glibc doesn't use it (because it's an old one), the kernel should fallback to the microsecond interface and not break old glibcs!" 

All glibc basically does is passes what the kernel gives it to any apps that make calls through its time interface (IE it is a middle man between apps and the kernel).  The only ABI issue related to these newer kernels is that the ntptimeval structure has had a field added (tai) and as far as I can tell the kernel is not using this field yet so that it remains compatible with older glibcs.  Removing this field from the kernel's timex.h file would likely fix the ABI issue but ntp would still not work correctly because the glibc timex.h file is missing all of the nanosecond declarations.  From a glibc perspective the nanosecond declarations in the patched timex.h file do not affect it in any way, they only tell other applications like ntp that there might be a nanosecond kernel available and that these other apps need to query the kernel to find out. 

If ntp uses a nanosecond correct timex.h header file when it is built it will query the kernel and ask it if it is a microkernel or a nanokernel and switch it's behavior as needed.  You can see this by running 

$ ntptime
ntp_gettime() returns code 0 (OK)
  time cd9b7266.bfed0f1c  Thu, Apr 23 2009 16:04:06.749, (.749711201),
  maximum error 7000 us, estimated error 0 us, TAI offset 0
ntp_adjtime() returns code 0 (OK)
  modes 0x0 (),
  offset 6.791 us, frequency -39.155 ppm, interval 1 s,
  maximum error 7000 us, estimated error 0 us,
  status 0x2001 (PLL,NANO),
  time constant 4, precision 0.001 us, tolerance 500 ppm,

As you can see the status includes NANO and precision is 0.001 us (IE. 1/1000 of a microsecond).  If I took this same ntp binary and placed it on a machine with an older kernel the status would say MICRO and precision would be 0.001 ms and the offset would also be in milliseconds instead of microseconds.

The problem is that when ntp is built using the existing glibc version of timex.h it finds no nanosecond defines and incorrectly assumes that the kernel is a microkernel no matter what it is capable of doing.  Because of this it never queries the kernel to check and the interface between it and the kernel is incorrect.  As far as I can tell the only app that is affected by this is ntp (there may be other ntp like or time related apps that also have this issue).

I really don't care who's fault it is or where it needs to be fixed.  But I do know that the patches and ebuilds posted here do fix the problem and do not appear to cause any on a machine where everything is built with these patches in place.  Of course modifying glibc is a big deal and it is probably best to recompile everything after making this or any change to any glibc interfaces.  But I know people with binary systems that only rebuilt glibc and ntp after making this change and they are reporting that their systems are totally stable and that ntp is working correctly with the nanokernel.  I think the reason for this is because the only thing this affects on most systems is ntp since it is really the only app that needs to do time stuff with the kernel at that level.   

Perhaps this glibc patch set is not the correct way to fix this and perhaps you know what the correct fix is.  In that case perhaps you could supply some patches or you know someone who could?  But those patches need to make ntp work correctly with the nanokernel since that is the issue that needs to be addressed.
Comment 15 Samuel Thibault 2009-04-26 23:14:51 UTC
> look in comment #8 for a link.  Ah right sorry.  > > "Now, glibc wants to take advantage of nanoseconds whenever they're > > available." > No this is not correct glibc does not know anything about microsecond > vs. nanoseconds  I'm talking about the _future_ glibc that _will_ want to take advantage etc., if you prefer that phrasing.  > > "For that to happen, the kernel must provide a way for glibc to know that the > > interface is available." >  > The kernel does in /usr/include/linux/timex.h  No, the kernel doesn't.  The kernel is something that _runs_ on the computer, it does not magically expose files in /usr/include according to whether it implements an interface or not.  A runtime test has to be done to know whether the currently running kernel does implement it or not.  Think of it as "as a user, I want to be able to boot my installed system with a 2.6.18 or with a shiny 2.6.30 kernel and everything should just work fine in both cases _without_ having to reinstall the glibc or the package that provides /usr/include/linux/timex.h and recompile glibc".  > If ntp uses a nanosecond correct timex.h header file when it is built > it will query the kernel and ask it if it is a microkernel or a > nanokernel and switch it's behavior as needed.  All right, but the thing is: a non-recompiled ntp _must_ also work with a recent kernel. The kernel is not supposed to break the interface. 
Comment 16 Hal Engel 2009-05-08 22:14:34 UTC
It looks like these changes are in glibc git now.  So newer versions of glibc should start to have the nanosecond declarations some time down the road.