When upgrading to glibc-2.11.3, then rebuilding packages, in this case postgresql-base / postgresql-server 8.2.20 against the libraries, when starting the init script, we are getting segfault errors. This is happening on multiple machines that have the same combination, or even newer versions of Postgresql (8.4, 9.0). Reproducible: Always Steps to Reproduce: 1. Upgrade glibc and gcc to latest versions 2. Rebuild postgresql-server / postgresql-base (8.2.20) 3. Start server 4. run dmesg Actual Results: Init script fails to start or return properly, even though the server is running, init fails to update. Status of postgresql-8.2 is stopped. check_pgsql[13755]: segfault at 17d ip 00007f575d9c5899 sp 00007fff0ff69320 error 4 in libc-2.11.3.so[7f575d94e000+150000] Just about every time another application tries to access postgres, there is a segfault. Expected Results: No segfaulting!
1) Please post your `emerge --info' output too. 2) Attach the entire build log of the failing postgresql-base / postgresql-server.
3) Also, could you run `eselect postgresql list' and perhaps setting it to the desired version, before continuing to rebuild?
profit-jsi01 ~ # emerge --info Portage 2.1.9.42 (default/linux/amd64/10.0, gcc-4.4.5, glibc-2.11.3-r0, 2.6.27-gentoo-r8 x86_64) ================================================================= System uname: Linux-2.6.27-gentoo-r8-x86_64-Intel-R-_Xeon-R-_CPU_E5420_@_2.50GHz-with-gentoo-1.12.14 Timestamp of tree: Fri, 01 Apr 2011 08:30:01 +0000 distcc 3.1 x86_64-pc-linux-gnu [disabled] ccache version 2.4 [enabled] app-shells/bash: 4.1_p9 dev-java/java-config: 2.1.11-r3 dev-lang/python: 2.4.6, 2.5.2-r7, 2.6.6-r2, 3.1.3-r1 dev-util/ccache: 2.4-r9 sys-apps/baselayout: 1.12.14-r1 sys-apps/sandbox: 2.4 sys-devel/autoconf: 2.65-r1 sys-devel/automake: 1.7.9-r1, 1.8.5-r4, 1.9.6-r2, 1.10.3, 1.11.1 sys-devel/binutils: 2.20.1-r1 sys-devel/gcc: 4.1.2, 4.3.4, 4.4.5 sys-devel/gcc-config: 1.4.1 sys-devel/libtool: 2.2.10 sys-devel/make: 3.81-r2 virtual/os-headers: 2.6.36.1 (sys-kernel/linux-headers) ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=nocona -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/maven-bin-2.2/conf" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/eselect/postgresql /etc/fonts/fonts.conf /etc/gconf /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=nocona -O2 -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="assume-digests binpkg-logs ccache distlocks fixlafiles fixpackages news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch" FFLAGS="" GENTOO_MIRRORS="http://distfiles.farheap.com/ http://distfiles.gentoo.org" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="en" MAKEOPTS="-j4" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage/farheap" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="3dm acl amd64 apic async bash-completion berkdb bzip2 cli cracklib crypt cxx dri fortran gdbm graphs iconv ipmi ipv6 ldap ldap-sasl logrotate mmx modules mudflap multilib ncurses nls nptl nptlonly openmp pam pcre perl postfix postgres python readline sasl savvis server session snmp sqlite sse sse2 ssl swat symlink sysfs syslog tcpd threads unicode urandom vim vim-pager vim-syntax zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" APACHE2_MPMS="worker" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="fbdev glint intel mach64 mga neomagic nouveau nv r128 radeon savage sis tdfx trident vesa via vmware dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Postgresql-server / base is not actually failing on build. It installs correctly, so there is no build.log. The init script is failing to register for whatever reason, and logs a segfault message in dmesg when starting. However the actually process does start, but will eventually fail due to segfaults. Here is out put from the command line: profit-jsi01 ~ # /etc/init.d/postgresql-8.2 status * status: stopped profit-jsi01 ~ # ps aux|grep postgres root 7490 0.0 0.0 6092 636 pts/2 S+ 09:43 0:00 grep --colour=auto postgres postgres 25045 0.0 0.0 71652 4392 ? Ss 09:16 0:00 /usr/lib64/postgresql-8.2/bin/postgres --silent-mode=true -N 150 -B 300 postgres 25046 3.3 0.0 52436 1776 ? Ss 09:16 0:54 postgres: logger process postgres 25048 0.0 0.0 71760 4568 ? Ss 09:16 0:00 postgres: writer process postgres 25049 0.0 0.0 51956 1476 ? Ss 09:16 0:00 postgres: archiver process postgres 25050 1.0 0.0 52088 1500 ? Ss 09:16 0:16 postgres: stats collector process postgres 25051 82.4 2.0 245444 165964 ? Ss 09:16 22:14 postgres: autovacuum process profit_jsi_openmq root 25740 23.2 4.1 890760 335740 ? Sl 09:19 5:25 /etc/java-config-2/current-system-vm/bin/java -cp /opt/openmq/bin/../lib/imqbroker.jar:/opt/openmq/bin/../lib/imqutil.jar:/opt/openmq/bin/../lib/jsse.jar:/opt/openmq/bin/../lib/jnet.jar:/opt/openmq/bin/../lib/jcert.jar:/usr/lib/audit/Audit.jar:/opt/sun/mfwk/share/lib/jdmkrt.jar:/opt/sun/mfwk/share/lib/mfwk_instrum_tk.jar:/opt/SUNWhadb/4/lib/hadbjdbc4.jar:/opt/SUNWjavadb/derby.jar:/usr/share/jdbc-postgresql/lib/jdbc-postgresql.jar:/opt/openmq/bin/../lib/ext:/opt/openmq/bin/../lib/ext/postgresql-8.3-603.jdbc3.jar:/opt/openmq/bin/../lib/ext:/opt/openmq/bin/../lib/ext/postgresql-8.3-603.jdbc3.jar -Xms512m -Xmx512m -Xss128k -XX:MaxGCPauseMillis=5000 -verbose:gc -Xloggc:/var/log/jvm/openmq_gc.log -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dimq.home=/opt/openmq/bin/.. -Dimq.varhome=/var/imq -Dimq.etchome=/opt/openmq/bin/../etc -Dimq.libhome=/opt/openmq/bin/../lib com.sun.messaging.jmq.jmsserver.Broker -name jsimq -bgnd -startRmiRegistry -rmiRegistryPort 1098 -Dcom.sun.management.jmxremote postgres 25760 9.0 0.0 73188 7800 ? Rs 09:19 2:06 postgres: jsiopenmq profit_jsi_openmq 10.40.128.127(41102) UPDATE postgres 25761 9.6 0.1 73620 8336 ? Ss 09:19 2:15 postgres: jsiopenmq profit_jsi_openmq 10.40.128.127(41103) idle postgres 25762 8.8 0.0 73616 7948 ? Ss 09:19 2:04 postgres: jsiopenmq profit_jsi_openmq 10.40.128.127(41104) idle postgres 25763 8.1 0.0 73196 7764 ? Ss 09:19 1:53 postgres: jsiopenmq profit_jsi_openmq 10.40.128.127(41105) idle postgres 25764 8.2 0.0 73092 7672 ? Rs 09:19 1:55 postgres: jsiopenmq profit_jsi_openmq 10.40.128.127(41106) UPDATE postgres 26921 5.6 0.1 76812 11212 ? Ss 09:26 0:57 postgres: profitjsi profitjsi 10.40.128.127(55505) idle postgres 27589 0.2 0.0 73972 7892 ? Ss 09:27 0:02 postgres: profitjsi profitjsi 10.40.128.127(55935) idle postgres 29916 7.8 0.0 73964 8052 ? Ss 09:30 1:01 postgres: profitjsi profitjsi 10.40.128.127(57441) idle postgres 29939 11.0 0.0 73964 8068 ? Ss 09:30 1:25 postgres: profitjsi profitjsi 10.40.128.127(57448) idle in transaction postgres 30102 10.7 0.1 74632 8592 ? Ss 09:30 1:22 postgres: profitjsi profitjsi 10.40.128.127(57482) idle postgres 30115 4.3 0.0 73916 8028 ? Rs 09:30 0:33 postgres: profitjsi profitjsi 10.40.128.127(57490) SELECT postgres 30163 8.6 0.1 74500 8432 ? Ss 09:30 1:06 postgres: profitjsi profitjsi 10.40.128.127(57504) idle postgres 30307 10.2 0.0 73320 7872 ? Ss 09:30 1:18 postgres: profitjsi profitjsi 10.40.128.127(57626) idle
Please attach your /etc/conf.d/postgresql-${SLOT} and /etc/init.d/postgresql-${SLOT}
Also, attach /var/lib/postgresql/${SLOT}/data/postmaster.log
cat /etc/conf.d/postgresql-8.2 # PostgreSQL's Database Directory #PGDATA="/var/lib/postgresql/8.2/data" PGDATA="/var/lib/postgresql/data" # Options to pass to PostgreSQL's initdb. #PG_INITDB_OPTS="--locale=en_US.UTF-8" # PostgreSQL User PGUSER="postgres" # Which port and socket to bind PostgreSQL PGPORT="5432" # Extra options to run postmaster with, e.g.: # -N is the maximal number of client connections # -B is the number of shared buffers and has to be at least 2x the value for -N # Please read the man-page to postmaster for more options. Many of these options # can be set directly in the configuration-file. PGOPTS="-N 150 -B 300" # Allow server 60 seconds to start or stop. # Can disable waiting by changing to '-W' (capital W) WAIT_FOR_START="-w" WAIT_FOR_STOP="-w" # Ignore new connections and wait for clients to disconnect from server # before shutting down. # Set NICE_QUIT to "NO" to disable. NICE_QUIT="YES" # Forecfully disconnect clients from server and shut down. This is performed # after NICE_QUIT. Terminating a client results in a rollback of open # transactions for that client. # Set RUDE_QUIT to "NO" to disable. RUDE_QUIT="YES" # If the server still fails to shutdown, you can force it to quit by setting # this to yes and a recover-run will execute on the next startup. # Set FORCE_QUIT to "YES" to enable. FORCE_QUIT="NO" # Pass extra environment variables. If you have to export environment variables # for the database process, this can be done here. # Don't forget to escape quotes. #PG_EXTRA_ENV="PGPASSFILE=\"/path/to/.pgpass\""
Do you really want the log file? there's a lot of information in there, mostly queries, and they are all about 16M each.
(In reply to comment #8) > Do you really want the log file? there's a lot of information in there, mostly > queries, and they are all about 16M each. I'm interested in errors and fatals. A 'tail' output should be sufficient immediately following the attempted start. Also, it looks like you're not using the latest set of conf.d/init. And maybe not the latest revision of the ebuilds. An 'eselect postgresql list' would tell me for certain.
There are no errors, or fatals in the logs. Just query information really, and errors about bad queries. Here's the output from eselect: profit-jsi01 postgresql # eselect postgresql list Available postgresql installations 8.2 <-* <-S base-8.2.20 server-8.2.20 We are using the latest init and conf.d scripts, when rebuilding, there is nothing for etc-update to do and there are no notifications of files in etc to be updated.
There's a reason why pg_ctl is returning a non-zero status. We need to know why it is non-zero. There must be an error message to go with it. Edit the init and change: su -l ${PGUSER} \ -c "env PGDATA=\"${PGDATA}\" /usr/lib/postgresql-8.2/bin/pg_ctl start ${WAIT_FOR_START} -o '--silent-mode=true ${PGOPTS}'" To read: su -l ${PGUSER} \ -c "env PGDATA=\"${PGDATA}\" /usr/lib/postgresql-8.2/bin/pg_ctl start ${WAIT_FOR_START} -o '${PGOPTS}'" It'll keep the server from detaching properly, but it should reveal immediately what the problem is.
I made the edits to /etc/init.d/postgresql-8.2 and tried to start it. I got the following: profit-jsi01 init.d # /etc/init.d/postgresql-8.2 start * Starting PostgreSQL ... pg_ctl: no operation specified Try "pg_ctl --help" for more information. /bin/bash: line 1: 13567 Segmentation fault env PGDATA="/var/lib/postgresql/data" /usr/lib/postgresql-8.2/bin/pg_ctl /bin/bash: line 1: start: command not found
(In reply to comment #12) > I made the edits to /etc/init.d/postgresql-8.2 and tried to start it. I got > the following: > profit-jsi01 init.d # /etc/init.d/postgresql-8.2 start > * Starting PostgreSQL ... > pg_ctl: no operation specified > Try "pg_ctl --help" for more information. > /bin/bash: line 1: 13567 Segmentation fault env > PGDATA="/var/lib/postgresql/data" /usr/lib/postgresql-8.2/bin/pg_ctl > /bin/bash: line 1: start: command not found That's telling me that you've got something wrong in your init script. There's an unescaped newline following pg_ctl.
Created attachment 268659 [details] Postgres-8.2 init.d script I'm sorry, this is a production system and really shouldn't be mucking about with it's operations. I have attached the init script that you asked me to modify. I'm not sure if I was 100% clear on this, this is also affecting normal psql calls to the server as well. It's behaving as if revdep-rebuild needs to be run or something else, however, when running revdep-rebuild, I am not getting anything back when it checks the system.
$[PGOPTS} should be ${PGOPTS}. Are you using a bin host? What's the output of 'gcc-config -l'?
I'm sorry, I fixed that PGOPTS variable. Here is the output regarding gcc profit-jsi01 ~ # gcc-config -l [1] x86_64-pc-linux-gnu-4.1.2 [2] x86_64-pc-linux-gnu-4.3.4 [3] x86_64-pc-linux-gnu-4.4.5 * I have rebuilt the system, glibc, and postgresql against this version of gcc.
Are you using a binhost?
No, we are not using a binhost.
Sorry, I've forgotten about this. Does this issue still persist?
I can't reproduce this and the lack of others running into this issue makes it impossible for me to fix this.
