| Summary: | sys-libs/gdbm-1.8.3-r4: enabling LFS seems to break with db's generated by non-LFS gdbm | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Martin Froněk <martin> |
| Component: | [OLD] Library | Assignee: | Gentoo's Team for Core System packages <base-system> |
| Status: | RESOLVED WORKSFORME | ||
| Severity: | major | CC: | chris, dark, gmturner007, mgorny, net-mail+disabled, odi, ondrej, openhs, pva, vmatare+gbug, yosoy |
| Priority: | High | ||
| Version: | unspecified | ||
| Hardware: | x86 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Attachments: |
gdbm non-lfs to lfs db converter
propogate uid/gid ownership Does all this stuff automagically fixed bugs, add x64 support |
||
|
Description
Martin Froněk
2010-01-02 20:28:18 UTC
I can confirm gdbm-1.8.3-r4 produces the errno 5. As reported, downgrading to gdbm-1.8.3-r3 helped getting thigns back to normal. This also breaks openwebmail. It displays "Couldn't get read lock on /home/users/someone/something", logfile contains "db error - Couldn't update index db /home/users/someone/.openwebmail/db/something". This is caused by added large file support (LFS). As gdbm is now compiled with -D_FILE_OFFSET_BITS=64 any program using it must be (re-)compiled with -D_FILE_OFFSET_BITS=64 as well. For postfix: just re-emerging postfix fixes the problem. Open WebMail is written in Perl. Recompiling Perl did not help. Mixing packages with and without LFS support apparently may break ABI (see http://www.suse.de/~aj/linux_lfs.html). Also there are a couple of packages that have a local USE flag for this. So it's necessary to find consensus on whether LFS support is a purely optional feature that needs to be controlled by a global USE flag, or if this is a feature that all packages need to support and must be enabled by default. Maybe CC flameeyes on this? I'm already on this. And I'd say largefile should just be enabled by default, if not for some small embedded cases. But how much effort I'm going to pour on this is limited by what I think is a simpler method: on amd64 largefile is the default. So sys-libs/gdbm-1.8.3-r4 is incompatible with stable perl on x86. amd64 is not an option when your CPU is x86 only... (In reply to comment #3) > This is caused by added large file support (LFS). As gdbm is now compiled with > -D_FILE_OFFSET_BITS=64 any program using it must be (re-)compiled with > -D_FILE_OFFSET_BITS=64 as well. > For postfix: just re-emerging postfix fixes the problem. > This was the first what I thought about, so I am afraid, recompiling postfix do not fix. So, I am still testing what is core of this bug. we decided years ago that LFS is to enabled everywhere by default. even on embedded systems, disabling LFS doesnt really gain much size. the packages with the USE=largefile should probably have their flags dropped. i checked net-p2p/mktorrent and it's pretty standard stuff (i.e. drop it). *** Bug 299371 has been marked as a duplicate of this bug. *** dare i ask if anyone has a simple test case that shows pass with r3 and fail with r4 ? (In reply to comment #11) > dare i ask if anyone has a simple test case that shows pass with r3 and fail > with r4 ? > Install cyrus-sasl. Enter: sasldblistusers2 With gdbm-1.8.4-r3, you get a list. With gdbm-1.8.4-r4 you get a failure. Running with strace shows that the failure comes from gdbm_open doing a very unreasonable seek. that depends on the order of things run. the simple test case is to gdbm_open(GDBM_WRCREAT) with a non-lfs gdbm and then gdbm_open(GDBM_READER) with a lfs gdbm.
$ cat test.c
#include <assert.h>
#include <unistd.h>
#include <gdbm.h>
int main(int argc, char *argv[])
{
GDBM_FILE db;
assert(argc == 2);
db = gdbm_open(argv[1], 1024, GDBM_NOLOCK | (access(argv[1], R_OK) == 0 ? GDBM_READER : GDBM_WRCREAT), 0644, NULL);
assert(db != NULL);
gdbm_close(db);
return 0;
}
$ rm -f db
$ ./non-lfs db
$ ./lfs db
lfs: test.c:9: main: Assertion `db != ((void *)0)' failed.
there is no ABI change in the gdbm library, so re-emerging packages wont change a thing. well, unless someone does something stupid like creates an executable that links statically against gdbm and passes references to another shared library that is linked dynamically against gdbm. but that's a broken package that needs fixing already. the db file format changes though as it seems gdbm encodes off_t values into the databases it creates. gdbm_file_header for example (which is the very first structure that sits at the start of the db) utilizes off_t a few times. i'm probably just pointing out things people already know, but a gdbm database, once created, is tied to that endian/type-sizes of the system that it was created on. values are read/written straight from/to the file without any portability tweaks. that means the LFS upgrade for 32bit arches is a one way trip. delete your old gdbm files and restart the package with the new LFS version. i dont think in-place conversion is practical or even feasible. you'd have to make a blind assumption that the values you're reading in are 32bit non-LFS which might be true for most people, but carries the possibility of screwing up a valid db already in LFS format. it might be feasible to write a tiny app that is linked statically against a non-LFS gdbm and dynamically linked against a LFS gdbm so that people can run `gdbm-lfs-upgrade` on a db. but it's easier to just say "delete your db's and start again". Created attachment 215240 [details]
gdbm non-lfs to lfs db converter
i thought it'd be fun to write this converter ... never programmed gdbm before, so no idea if it works
to make this work:
emerge gdbm-1.8.3-r3
copy the libgdbm.a locally
emerge gdbm-1.8.3-r4
compile the converter: gcc -O2 -g convert.c libgdbm.a -ldl
run the converter on a db: ./converter /some/db/file
it'll create /some/db/file.upgrade.tmp which you should be able to replace the old db with. a simple open/close works for me.
Spanky, true, I made a mistake and didn't actually upgrade gdbm. So my claim that recompiling postfix fixed the problem is wrong. And true, the ABI doesn't change. It is "only" an incompatibility of the DB format with the library compiler flags (OMG what kind of "database" is that). Thanks for the converter. It works! Bingo! Thanks Yikes, broke my postfix email setup. Not fun. Thanks for the converter, as a tip make sure that the old and new file permissions are the same. (In reply to comment #15) Yes, that works for me as well. Nice tool. Interestingly old file was 385KB, converted file is now 3.5KB. Probably old file was full with garbage. For users of the converter: don't forget to set new file permissions :-). Or at least check. By the way, this gdbm file incompatibility affects yellow pages as well. Fortunately it's db's are easily generated but this converter might help as well. Created attachment 215326 [details]
propogate uid/gid ownership
file permissions should already be preserved (see how it does a stat and opens the new db with the old perms)
perhaps you guys actually mean file ownership ... this small tweak also transitions uid/gid ownership
if this is working for people, i'll include it in the ebuild with a warning and consider the issue "closed"
One question remains: how do I find all gdbm database files? Found that openwebmail uses them in /home/user/.openwebmail/db/ and in /var/www/localhost/cgi-bin/openwebmail/etc/maps (*.pag and *.dir files). Also spamassassin has auto-whitelist.pag file. Any other known users? I think Spamassassin uses Berkeley DB and not gdbm: oglueck@gollum ~/.spamassassin $ equery u spamassassin U I + + berkdb : Adds support for sys-libs/db (Berkeley DB for MySQL) - - doc : Adds extra documentation (API, Javadoc, etc) + + ipv6 : Adds support for IP version 6 + + ldap : Adds LDAP support (Lightweight Directory Access Protocol) - - mysql : Adds mySQL Database support - - postgres : Adds support for the postgresql database - - qmail : Build qmail functionality and docs - - sqlite : Adds support for sqlite - embedded sql database + + ssl : Adds support for Secure Socket Layer connections + + tools : Enables tools for spamassassin You're right, the auto-whitelist files are old (2005) and should be probably deleted, not converted. There's a newer one that's berkeley DB (but still from 2005 so it seems to be broken somehow). ive added some notes to the ebuild http://sources.gentoo.org/sys-libs/gdbm/gdbm-1.8.3-r4.ebuild?r1=1.8&r2=1.9 ... only in Gentoo would the officially-sanctioned "solution" to a problem be: "delete the database files" or "go grab the source of this c program", which really ends up being "manually look up how to compile the c program, then install gbdm from the r3 ebuild, then copy the .a file to a tmp directory, then compile the c program and link in the .a file, then run the resulting binary on every gdbm-generated file, then move the files into the right place, then reinstall the r4 version of the ebuild and btw fuck you" This didn't work for me. I am using ntop on a couple of routers and after doing all of these steps (which were ridiculous in the first place), ntop will still not start. It fails with: **ERROR** ....open of /var/lib/ntop/prefsCache.db failed: File seek error This is before *and* after attempting to use the c program that is attached to this bug. This is an UNACCEPTABLE way to do things. (In reply to comment #24) > ive added some notes to the ebuild > > http://sources.gentoo.org/sys-libs/gdbm/gdbm-1.8.3-r4.ebuild?r1=1.8&r2=1.9 > nothing in your comment was useful. if you have nothing useful to contribute, feel free to refrain from commenting. which is to say, do not comment. For finding gdbm files the 'file' utility (sys-apps/file) can be used. It returns the "GNU dbm 1.x or ndbm database, little endian" string for gdbm files. So in combination with 'find' and 'grep' it should do the trick. Searching through /etc and /var should be enough in most cases:
find /etc /var -type f -exec file {} \; | grep 'GNU dbm 1\.x or ndbm database'
BTW, conversion utility works fine for ntop too.
thanks, i'll add that suggestion to the ebuild log Well, the important part of comment 25 was: without looking at this bug, the added post-install note doesn't make much sense. Just how can a user divine, that a static version of the old lib is required ? At very least, the note should point to comment 15 for an explanation besides the correct attachment. bitching and moaning quickly drowns out anything useful ive added a link to the specific comment to the ebuild (In reply to comment #30) > bitching and moaning quickly drowns out anything useful > > ive added a link to the specific comment to the ebuild worked for me. Since I am a nice guy and understand the desire to be lazy and not think about stuff like this... Created attachment 222571 [details]
Does all this stuff automagically
Here is a script to do this automagically. Just download and source. YMMV.
Created attachment 222581 [details]
fixed bugs, add x64 support
Oops. Fixed a bug, and added support for amd64. also, don't source this, as I suggested above. it uses "exit" which will make your shell disappear. save it, chmod +x it, and run it as in:
# wget -S -O gdbm_convert.sh "${whatever_the_url_of_this_attachment_is}"
# chmod +x gdbm_convert.sh
# ./gdbm_convert.sh
Comment on attachment 222581 [details] fixed bugs, add x64 support Oops. Fixed a bug, and added support for amd64. also, don't source this, as I suggested above. it uses "exit" which will make your shell disappear. save it, chmod +x it, and run it as in: # wget -S -O gdbm_convert.sh "${http://bugs.gentoo.org/attachment.cgi?id=222581&action=view}" # chmod +x gdbm_convert.sh # ./gdbm_convert.sh > # wget -S -O gdbm_convert.sh > "${http://bugs.gentoo.org/attachment.cgi?id=222581&action=view}" oh. that didn't go as planned haha. This should read # wget -S -O gdbm_convert.sh \ "http://bugs.gentoo.org/attachment.cgi?id=222581&action=view" thanks, added a tip for that http://sources.gentoo.org/sys-libs/gdbm/gdbm-1.8.3-r4.ebuild?r1=1.12&r2=1.13 Doesn't work for me either. After Downgrading to gdbm-1.83-r3 I am still getting: Driver initialization failed for handler: gdbm: File seek error (using the gdbm file from within php) Running the converter on the db file seems to work fine, a new file is created, files differ in size (slightly) and content (somewhere in the beginning). Using the newly created file, I get: Driver initialization failed for handler: gdbm: File read error After upgrading to -r4 again I'm getting: Driver initialization failed for handler: gdbm: File read error (for the converted file) and Driver initialization failed for handler: gdbm: File seek error (for the original file). Seems this didn't help, any further suggestions? BTW: I generally do agree, when databases (or whatsoever) changes physical data format, a sane and easy to use upgrade tool should be in place. Deleting and repopulating complete databases is just not an option. *** Bug 331719 has been marked as a duplicate of this bug. *** |
