Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 299390 - sys-libs/gdbm-1.8.3-r4: enabling LFS seems to break with db's generated by non-LFS gdbm
Summary: sys-libs/gdbm-1.8.3-r4: enabling LFS seems to break with db's generated by no...
Status: RESOLVED WORKSFORME
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Library (show other bugs)
Hardware: x86 Linux
: High major (vote)
Assignee: Gentoo's Team for Core System packages
URL:
Whiteboard:
Keywords:
: 299371 331719 (view as bug list)
Depends on:
Blocks:
 
Reported: 2010-01-02 20:28 UTC by Martin Froněk
Modified: 2010-08-09 07:45 UTC (History)
10 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
gdbm non-lfs to lfs db converter (convert.c,1.15 KB, text/plain)
2010-01-05 08:06 UTC, SpanKY
Details
propogate uid/gid ownership (convert.c,1.20 KB, text/plain)
2010-01-05 19:45 UTC, SpanKY
Details
Does all this stuff automagically (gdbm_convert.sh,1.03 KB, text/plain)
2010-03-07 20:45 UTC, Greg Turner
Details
fixed bugs, add x64 support (gdbm_convert.sh,1.15 KB, text/plain)
2010-03-07 21:03 UTC, Greg Turner
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Froněk 2010-01-02 20:28:18 UTC
some bug in sys-libs/gdbm-1.8.3-r4 causes that users cannot send any email using postfix with cyrrus-sasl2 auth
After downgrade, this bug disappear.

Maybe, this sys-libs/gdbm causes some other problems with another apps.

Reproducible: Always

Steps to Reproduce:
1. install sys-libs/gdbm-1.8.3-r4 to the fully function postfix system with cyrrus-sasl2 auth
2. restart postfix 
3.now, you cannot send any email, because the SMTP server cannot user auth and in the log file you can find this "warning: SASL authentication failure: Cou
ld not open /etc/sasl2/sasldb2: gdbm_errno=5"
4. after downgrade to sys-libs/gdbm-1.8.3-r3 this bug disappear.


Actual Results:  
downgrade to sys-libs/gdbm-1.8.3-r3 this bug disappear
Comment 1 Daniel Díaz 2010-01-03 08:24:25 UTC
I can confirm gdbm-1.8.3-r4 produces the errno 5. As reported, downgrading to gdbm-1.8.3-r3 helped getting thigns back to normal.
Comment 2 Ondrej Zary 2010-01-04 11:54:41 UTC
This also breaks openwebmail. It displays "Couldn't get read lock on /home/users/someone/something", logfile contains "db error - Couldn't update index db /home/users/someone/.openwebmail/db/something".
Comment 3 Ortwin Glueck 2010-01-04 13:42:19 UTC
This is caused by added large file support (LFS). As gdbm is now compiled with -D_FILE_OFFSET_BITS=64 any program using it must be (re-)compiled with -D_FILE_OFFSET_BITS=64 as well.
For postfix: just re-emerging postfix fixes the problem.
Comment 4 Ondrej Zary 2010-01-04 14:03:14 UTC
Open WebMail is written in Perl. Recompiling Perl did not help.
Comment 5 Ortwin Glueck 2010-01-04 14:11:47 UTC
Mixing packages with and without LFS support apparently may break ABI (see http://www.suse.de/~aj/linux_lfs.html). Also there are a couple of packages that have a local USE flag for this.

So it's necessary to find consensus on whether LFS support is a purely optional feature that needs to be controlled by a global USE flag, or if this is a feature that all packages need to support and must be enabled by default.

Maybe CC flameeyes on this?
Comment 6 Diego Elio Pettenò (RETIRED) gentoo-dev 2010-01-04 14:14:02 UTC
I'm already on this.

And I'd say largefile should just be enabled by default, if not for some small embedded cases.

But how much effort I'm going to pour on this is limited by what I think is a simpler method: on amd64 largefile is the default.
Comment 7 Ortwin Glueck 2010-01-04 17:09:26 UTC
So sys-libs/gdbm-1.8.3-r4 is incompatible with stable perl on x86. amd64 is not an option when your CPU is x86 only...
Comment 8 Martin Froněk 2010-01-04 19:21:35 UTC
(In reply to comment #3)
> This is caused by added large file support (LFS). As gdbm is now compiled with
> -D_FILE_OFFSET_BITS=64 any program using it must be (re-)compiled with
> -D_FILE_OFFSET_BITS=64 as well.
> For postfix: just re-emerging postfix fixes the problem.
> 

This was the first what I thought about, so I am afraid, recompiling postfix do not fix. So, I am still testing what is core of this bug.
Comment 9 SpanKY gentoo-dev 2010-01-05 04:37:40 UTC
we decided years ago that LFS is to enabled everywhere by default.  even on embedded systems, disabling LFS doesnt really gain much size.

the packages with the USE=largefile should probably have their flags dropped.  i checked net-p2p/mktorrent and it's pretty standard stuff (i.e. drop it).
Comment 10 SpanKY gentoo-dev 2010-01-05 04:37:44 UTC
*** Bug 299371 has been marked as a duplicate of this bug. ***
Comment 11 SpanKY gentoo-dev 2010-01-05 06:44:39 UTC
dare i ask if anyone has a simple test case that shows pass with r3 and fail with r4 ?
Comment 12 Bjorn Fahller 2010-01-05 07:03:01 UTC
(In reply to comment #11)
> dare i ask if anyone has a simple test case that shows pass with r3 and fail
> with r4 ?
> 

Install cyrus-sasl.
Enter: sasldblistusers2

With gdbm-1.8.4-r3, you get a list. With gdbm-1.8.4-r4 you get a failure. 

Running with strace shows that the failure comes from gdbm_open doing a very unreasonable seek.
Comment 13 SpanKY gentoo-dev 2010-01-05 07:19:57 UTC
that depends on the order of things run.  the simple test case is to gdbm_open(GDBM_WRCREAT) with a non-lfs gdbm and then gdbm_open(GDBM_READER) with a lfs gdbm.

$ cat test.c
#include <assert.h>
#include <unistd.h>
#include <gdbm.h>
int main(int argc, char *argv[])
{
  GDBM_FILE db;
  assert(argc == 2);
  db = gdbm_open(argv[1], 1024, GDBM_NOLOCK | (access(argv[1], R_OK) == 0 ? GDBM_READER : GDBM_WRCREAT), 0644, NULL);
  assert(db != NULL);
  gdbm_close(db);
  return 0;
}

$ rm -f db
$ ./non-lfs db
$ ./lfs db
lfs: test.c:9: main: Assertion `db != ((void *)0)' failed.
Comment 14 SpanKY gentoo-dev 2010-01-05 07:36:43 UTC
there is no ABI change in the gdbm library, so re-emerging packages wont change a thing.  well, unless someone does something stupid like creates an executable that links statically against gdbm and passes references to another shared library that is linked dynamically against gdbm.  but that's a broken package that needs fixing already.

the db file format changes though as it seems gdbm encodes off_t values into the databases it creates.  gdbm_file_header for example (which is the very first structure that sits at the start of the db) utilizes off_t a few times.

i'm probably just pointing out things people already know, but a gdbm database, once created, is tied to that endian/type-sizes of the system that it was created on.  values are read/written straight from/to the file without any portability tweaks.

that means the LFS upgrade for 32bit arches is a one way trip.  delete your old gdbm files and restart the package with the new LFS version.  i dont think in-place conversion is practical or even feasible.  you'd have to make a blind assumption that the values you're reading in are 32bit non-LFS which might be true for most people, but carries the possibility of screwing up a valid db already in LFS format.

it might be feasible to write a tiny app that is linked statically against a non-LFS gdbm and dynamically linked against a LFS gdbm so that people can run `gdbm-lfs-upgrade` on a db.  but it's easier to just say "delete your db's and start again".
Comment 15 SpanKY gentoo-dev 2010-01-05 08:06:58 UTC
Created attachment 215240 [details]
gdbm non-lfs to lfs db converter

i thought it'd be fun to write this converter ... never programmed gdbm before, so no idea if it works

to make this work:
emerge gdbm-1.8.3-r3
copy the libgdbm.a locally
emerge gdbm-1.8.3-r4
compile the converter: gcc -O2 -g convert.c libgdbm.a -ldl
run the converter on a db: ./converter /some/db/file

it'll create /some/db/file.upgrade.tmp which you should be able to replace the old db with.  a simple open/close works for me.
Comment 16 Ortwin Glueck 2010-01-05 09:48:56 UTC
Spanky, true, I made a mistake and didn't actually upgrade gdbm. So my claim that recompiling postfix fixed the problem is wrong. And true, the ABI doesn't change. It is "only" an incompatibility of the DB format with the library compiler flags (OMG what kind of "database" is that).

Thanks for the converter. It works!
Comment 17 Bjorn Fahller 2010-01-05 14:07:42 UTC
Bingo! Thanks
Comment 18 Jeremy Olexa (darkside) (RETIRED) archtester Gentoo Infrastructure gentoo-dev Security 2010-01-05 14:34:27 UTC
Yikes, broke my postfix email setup. Not fun. Thanks for the converter, as a tip make sure that the old and new file permissions are the same.
Comment 19 Konstantin Münning 2010-01-05 19:08:22 UTC
(In reply to comment #15)

Yes, that works for me as well. Nice tool. Interestingly old file was 385KB, converted file is now 3.5KB. Probably old file was full with garbage. For users of the converter: don't forget to set new file permissions :-). Or at least check.

By the way, this gdbm file incompatibility affects yellow pages as well. Fortunately it's db's are easily generated but this converter might help as well.
Comment 20 SpanKY gentoo-dev 2010-01-05 19:45:19 UTC
Created attachment 215326 [details]
propogate uid/gid ownership

file permissions should already be preserved (see how it does a stat and opens the new db with the old perms)

perhaps you guys actually mean file ownership ... this small tweak also transitions uid/gid ownership

if this is working for people, i'll include it in the ebuild with a warning and consider the issue "closed"
Comment 21 Ondrej Zary 2010-01-08 09:33:01 UTC
One question remains: how do I find all gdbm database files? Found that openwebmail uses them in /home/user/.openwebmail/db/ and in /var/www/localhost/cgi-bin/openwebmail/etc/maps (*.pag and *.dir files). Also spamassassin has auto-whitelist.pag file. Any other known users?
Comment 22 Ortwin Glueck 2010-01-08 09:59:19 UTC
I think Spamassassin uses Berkeley DB and not gdbm:
oglueck@gollum ~/.spamassassin $ equery u spamassassin
 U I
 + + berkdb   : Adds support for sys-libs/db (Berkeley DB for MySQL)
 - - doc      : Adds extra documentation (API, Javadoc, etc)
 + + ipv6     : Adds support for IP version 6
 + + ldap     : Adds LDAP support (Lightweight Directory Access Protocol)
 - - mysql    : Adds mySQL Database support
 - - postgres : Adds support for the postgresql database
 - - qmail    : Build qmail functionality and docs
 - - sqlite   : Adds support for sqlite - embedded sql database
 + + ssl      : Adds support for Secure Socket Layer connections
 + + tools    : Enables tools for spamassassin
Comment 23 Ondrej Zary 2010-01-08 10:25:54 UTC
You're right, the auto-whitelist files are old (2005) and should be probably deleted, not converted. There's a newer one that's berkeley DB (but still from 2005 so it seems to be broken somehow).
Comment 24 SpanKY gentoo-dev 2010-01-15 04:32:30 UTC
ive added some notes to the ebuild

http://sources.gentoo.org/sys-libs/gdbm/gdbm-1.8.3-r4.ebuild?r1=1.8&r2=1.9
Comment 25 Chris Ribble 2010-01-23 23:17:18 UTC
... only in Gentoo would the officially-sanctioned "solution" to a problem be:
 
"delete the database files"

or

"go grab the source of this c program", which really ends up being
"manually look up how to compile the c program, then install gbdm from the r3 ebuild, then copy the .a file to a tmp directory, then compile the c program and link in the .a file, then run the resulting binary on every gdbm-generated file, then move the files into the right place, then reinstall the r4 version of the ebuild and btw fuck you"

This didn't work for me. I am using ntop on a couple of routers and after doing all of these steps (which were ridiculous in the first place), ntop will still not start.

It fails with:

**ERROR** ....open of /var/lib/ntop/prefsCache.db failed: File seek error

This is before *and* after attempting to use the c program that is attached to this bug. This is an UNACCEPTABLE way to do things.

(In reply to comment #24)
> ive added some notes to the ebuild
> 
> http://sources.gentoo.org/sys-libs/gdbm/gdbm-1.8.3-r4.ebuild?r1=1.8&r2=1.9
> 

Comment 26 SpanKY gentoo-dev 2010-01-24 01:52:22 UTC
nothing in your comment was useful.  if you have nothing useful to contribute, feel free to refrain from commenting.  which is to say, do not comment.
Comment 27 Róbert Čerňanský 2010-01-28 18:26:27 UTC
For finding gdbm files the 'file' utility (sys-apps/file) can be used. It returns the "GNU dbm 1.x or ndbm database, little endian" string for gdbm files. So in combination with 'find' and 'grep' it should do the trick. Searching through /etc and /var should be enough in most cases:

find /etc /var -type f -exec file {} \; | grep 'GNU dbm 1\.x or ndbm database'

BTW, conversion utility works fine for ntop too.
Comment 28 SpanKY gentoo-dev 2010-01-28 20:52:54 UTC
thanks, i'll add that suggestion to the ebuild log
Comment 29 Rafał Mużyło 2010-02-06 02:40:23 UTC
Well, the important part of comment 25 was:
without looking at this bug, the added post-install
note doesn't make much sense.
Just how can a user divine, that a static version
of the old lib is required ?

At very least, the note should point to comment 15
for an explanation besides the correct attachment.
Comment 30 SpanKY gentoo-dev 2010-02-06 06:34:05 UTC
bitching and moaning quickly drowns out anything useful

ive added a link to the specific comment to the ebuild
Comment 31 Greg Turner 2010-03-07 20:43:18 UTC
(In reply to comment #30)
> bitching and moaning quickly drowns out anything useful
> 
> ive added a link to the specific comment to the ebuild

worked for me.  Since I am a nice guy and understand the desire to be lazy and not think about stuff like this...
Comment 32 Greg Turner 2010-03-07 20:45:41 UTC
Created attachment 222571 [details]
Does all this stuff automagically

Here is a script to do this automagically.  Just download and source. YMMV.
Comment 33 Greg Turner 2010-03-07 21:03:30 UTC
Created attachment 222581 [details]
fixed bugs, add x64 support

Oops.  Fixed a bug, and added support for amd64.  also, don't source this, as I suggested above. it uses "exit" which will make your shell disappear. save it, chmod +x it, and run it as in:

# wget -S -O gdbm_convert.sh "${whatever_the_url_of_this_attachment_is}"
# chmod +x gdbm_convert.sh
# ./gdbm_convert.sh
Comment 34 Greg Turner 2010-03-07 21:05:24 UTC
Comment on attachment 222581 [details]
fixed bugs, add x64 support

Oops.  Fixed a bug, and added support for amd64.  also, don't source this, as I
suggested above. it uses "exit" which will make your shell disappear. save it,
chmod +x it, and run it as in:

# wget -S -O gdbm_convert.sh "${http://bugs.gentoo.org/attachment.cgi?id=222581&action=view}"
# chmod +x gdbm_convert.sh
# ./gdbm_convert.sh
Comment 35 Greg Turner 2010-03-07 21:06:40 UTC
> # wget -S -O gdbm_convert.sh
> "${http://bugs.gentoo.org/attachment.cgi?id=222581&action=view}"

oh.  that didn't go as planned haha.

This should read

# wget -S -O gdbm_convert.sh \
  "http://bugs.gentoo.org/attachment.cgi?id=222581&action=view"
Comment 36 SpanKY gentoo-dev 2010-03-20 21:03:56 UTC
thanks, added a tip for that

http://sources.gentoo.org/sys-libs/gdbm/gdbm-1.8.3-r4.ebuild?r1=1.12&r2=1.13
Comment 37 Sven E. 2010-06-17 15:48:34 UTC
Doesn't work for me either.

After Downgrading to gdbm-1.83-r3 I am still getting:
Driver initialization failed for handler: gdbm: File seek error (using the gdbm file from within php)
Running the converter on the db file seems to work fine, a new file is created, files differ in size (slightly) and content (somewhere in the beginning).
Using the newly created file, I get:
Driver initialization failed for handler: gdbm: File read error

After upgrading to -r4 again I'm getting:
Driver initialization failed for handler: gdbm: File read error (for the converted file)
and
Driver initialization failed for handler: gdbm: File seek error (for the original file).

Seems this didn't help, any further suggestions?

BTW: I generally do agree, when databases (or whatsoever) changes physical data format, a sane and easy to use upgrade tool should be in place. Deleting and repopulating complete databases is just not an option.
Comment 38 SpanKY gentoo-dev 2010-08-09 07:45:39 UTC
*** Bug 331719 has been marked as a duplicate of this bug. ***