Bug 27969 – Unable to kill mysqld (/etc/init.d/mysql can not stop mysqld)

• Product: Gentoo Linux
• Status: RESOLVED FIXED

[Description Eric Pretorious 2003-09-04 18:47:40 UTC]

`/etc/init.d/mysql stop` doesn't stop mysql. (mysql-4.0.13-r3 on PPC)

Checking /usr/bin/mysql_safe (the --exec value passed to the start-stop-daemon)
reveals that signals '', 1, 2, 3, and 15 are caught to prevent mysql from being
kill'ed.

Shouldn't this behavior be configurable through the use of a configuration script?
e.g., /etc/conf.d/mysql

Reproducible: Always
Steps to Reproduce:
1. '/etc/init.d/mysql start'
2. '/etc/init.d/msyql stop'
3. 'ps -ax | grep mysql'

Actual Results:  
mysqld_safe did not stop.

Expected Results:  
mysqld_safe should have stoped.

Checking /usr/bin/mysql_safe (the --exec value passed to the start-stop-daemon)
reveals that signals '', 1, 2, 3, and 15 are caught to prevent mysql from being
kill'ed.

Shouldn't this behavior be configurable through the use of a configuration script?
e.g., /etc/conf.d/mysql

How is one supposed to stop mysqld? `/etc/init.d/msyql stop` *should* accomplish
this.

[Comment 1 SpanKY 2003-09-04 21:05:34 UTC]

happens on my x86 too 

[Comment 2 Eric Pretorious 2003-09-05 12:37:56 UTC]

I've developed a work-around to the signal trapping:
1. Create /etc/conf.d/mysql: Insert "MYSQL_EXEC=/usr/sbin/mysqld"
2. Edit /etc/init.d/msyql: Replace "--exec /usr/sbin/mysqld_safe" with "${MYSQL_EXEC}"

It's not 100% reliable - I've still got a bit of troubleshooting to do. Here's what I observed recently:

1. Start mysqld
   /etc/init.d/msyql start

2. Check processes:
   ps -ax | grep mysql
   <ONE process is started - The parent of *future* processes. Read on...>

3. Check .pid file:
   ls /var/run/mysql
   <NO .pid file is created>

4. Check IP interface
   netstat -ta
   <mysql is monitoring interface>

5. Re-check processes:
   ps -ax | grep mysql
   <FOUR process are now running - The parent and three children!>

6. Check .pid file:
   ls /var/run/mysql
   <.pid file has been created.>

7. Check contents of .pid file:
   cat /var/run/mysql/mysql.pid
   <.pid file contains pid of MIDDLE child-process.>

[Comment 3 Robin Johnson 2003-09-17 00:14:48 UTC]

please test again with 4.0.15
it should work properly now, as I have changed the stopping part of the init.d script slightly.

[Comment 4 Eric Pretorious 2003-10-02 15:05:07 UTC]

Since 4.0.15 is still marked as "unstable" I have not tested it. (This problem
still persists in dev-db/mysql-4.0.13-r3.)

[Comment 5 A Blamey 2005-10-11 07:12:09 UTC]

Please re-open this bug.

I'm seeing this with mysql 4.1.14 on ~x86

The /var/log/mysql/mysql.err shows:

051011 15:04:00  mysqld started        <-- This is where started mysql

Number of processes running now: 0     <-- This is where tried to stop mysql
051011 15:04:23  mysqld restarted

[Comment 6 Francesco R. (RETIRED) 2005-10-11 10:24:03 UTC]

please could you attach
- the output of "/etc/init.d/mysql start stop"
- the relevant content of the three files in /var/log/mysql
thanks

[Comment 7 A Blamey 2005-10-11 11:49:22 UTC]

OK - here is what I see when starting and stopping:

#ps -fe | grep mysql
root     24976 24688  0 19:37 pts/2    00:00:00 grep mysql

-- OK no mysql running

#/etc/init.d/mysql start
 * Starting mysqld (/etc/mysql/my.cnf) ...                                     
                                      [ ok ]

#ps -fe | grep mysql
root     25067     1  0 19:37 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe
--defaults-file=/etc/mysql/my.cnf
mysql    25105 25067  1 19:37 ?        00:00:00 /usr/sbin/mysqld
--defaults-file=/etc/mysql/my.cnf --basedir=/usr --datadir=/var/lib/mysql
--user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-locking --port=3306
--socket=/var/run/mysqld/mysqld.sock
root     25122 24688  0 19:37 pts/2    00:00:00 grep mysql

-- OK - now it's running ok

#/etc/init.d/mysql stop
 * Stopping mysqld (/etc/mysql/my.cnf) ...                                     
                                      [ ok ]

#ps -fe | grep mysql
root     25067     1  0 19:37 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe
--defaults-file=/etc/mysql/my.cnf
mysql    25298 25067  1 19:37 ?        00:00:00 /usr/sbin/mysqld
--defaults-file=/etc/mysql/my.cnf --basedir=/usr --datadir=/var/lib/mysql
--user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-locking --port=3306
--socket=/var/run/mysqld/mysqld.sock
root     25310 24688  0 19:37 pts/2    00:00:00 grep mysql

-- Aargh - it's still running - except the mysql PID has changed ?!

If I run mysqladmin -p<PASSWORD> shutdown, then it stops ok.

This is what I see in mysqld.err :

==========
051011 19:37:32  InnoDB: Started; log sequence number 0 13605859
/usr/sbin/mysqld: ready for connections.
Version: '4.1.14-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Gentoo
Linux mysql-4.1.14
051011 19:37:49 [Note] /usr/sbin/mysqld: Normal shutdown

051011 19:37:49  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
051011 19:37:49  InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 13605907.
InnoDB: Doing recovery: scanned up to log sequence number 0 13605907
InnoDB: Last MySQL binlog file position 0 79, file name ./cube-bin.000010
051011 19:37:49  InnoDB: Flushing modified pages from the buffer pool...
051011 19:37:49  InnoDB: Started; log sequence number 0 13605907
/usr/sbin/mysqld: ready for connections.
Version: '4.1.14-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Gentoo
Linux mysql-4.1.14
========

This is what I see in mysql.err:

=======
051011 19:37:31  mysqld started

Number of processes running now: 0
051011 19:37:49  mysqld restarted
========

Nothing is written to mysql.log


The PID file /var/run/mysqld/mysqld.pid contains 25298 - the PID of current
mysqld process (not that of the mysqld_safe process).

Also, as I'm running ~x86 I'm using baselayout-1.12.0_pre8-r2.

Is it possible that when start-stop-daemon kills the mysqld, the mysqld_safe
restarts a new one?  Does this mean that start-stop-daemon should instead try to
kill the mysqld_safe process?  Or should we be using the mysql shutdown command?

[Comment 8 Francesco R. (RETIRED) 2005-10-11 12:04:21 UTC]

look like there are problems with innodb.

if and *only* if you are not using innodb tables (the default for mysql is
myisam) try to:
- stop the database server (check with ps has you did before)
- remove "/var/lib/mysql/ib_logfile*" and "/var/lib/mysql/ibdata1"
- start and stop the server again to see if the problem still appear

[Comment 9 A Blamey 2005-10-11 12:52:23 UTC]

I'm using InnoDB (for mediawiki), so don't want to trash things if at all possible.
The DB applications are all working OK, and I have no errors when accessing data.
A mysqldump of all DBs works fine, so I don't think there is any corruption.

I've run some more tests and I think what happens is this: (looking at mysqld.err)

I RUN : /etc/init.d/mysql start

LOG SAYS:
051011 20:35:58  InnoDB: Started; log sequence number 0 13606003
/usr/sbin/mysqld: ready for connections.
Version: '4.1.14-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Gentoo
Linux mysql-4.1.14

PS SHOWS:
root     32490     1  0 20:35 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe
--defaults-file=/etc/mysql/my.cnf
mysql    32530 32490  0 20:35 ?        00:00:00 /usr/sbin/mysqld
--defaults-file=/etc/mysql/my.cnf --basedir=/usr --datadir=/var/lib/mysql
--user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-locking --port=3306
--socket=/var/run/mysqld/mysqld.sock

NOTE: mysqld_safe PID is 32490 and mysqld PID is 32530

I RUN: /etc/init.d/mysql stop

LOG SAYS:
051011 20:39:17 [Note] /usr/sbin/mysqld: Normal shutdown

051011 20:39:18  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
051011 20:39:18  InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 13606035.
InnoDB: Doing recovery: scanned up to log sequence number 0 13606035
InnoDB: Last MySQL binlog file position 0 79, file name ./cube-bin.000014
051011 20:39:18  InnoDB: Flushing modified pages from the buffer pool...
051011 20:39:18  InnoDB: Started; log sequence number 0 13606035
/usr/sbin/mysqld: ready for connections.
Version: '4.1.14-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Gentoo
Linux mysql-4.1.14

PS SHOWS:
root     32490     1  0 20:35 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe
--defaults-file=/etc/mysql/my.cnf
mysql    32761 32490  0 20:39 ?        00:00:00 /usr/sbin/mysqld
--defaults-file=/etc/mysql/my.cnf --basedir=/usr --datadir=/var/lib/mysql
--user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-locking --port=3306
--socket=/var/run/mysqld/mysqld.sock

NOTE: mysqld_safe PID is 32490 but mysqld PID is now 32761!

From what I can see in /etc/init.d/mysql, start-stop-daemon start is told to
start mysqld_safe, but given the pidfile for the mysqld process to stop.
This means that it kills the mysqld child process of mysqld_safe.

mysqld_safe will restart the mysqld child when it is killed.

Therefore, what I see in the log appears to be the newly started mysqld process
(by mysqld_safe) complaining that it's found data files that may need repairing
because start-stop-daemon had kill'd the previous mysqld process.

mysqld_safe seems to expect that the DB will be shutdown using the shutdown
command, and if the DB process dies in any other way (ie kill) then it will
restart it.

I don't think start-stop-daemon and mysqld_safe are playing nicely with each
other anymore.

[Comment 10 A Blamey 2005-10-11 12:59:29 UTC]

One more thing ... When I run mysqladmin -p<PASSWORD> shutdown I see this in the
mysqld.err log:

051011 20:54:44 [Note] /usr/sbin/mysqld: Normal shutdown

051011 20:54:44  InnoDB: Starting shutdown...
051011 20:54:47  InnoDB: Shutdown completed; log sequence number 0 13606067
051011 20:54:47 [Note] /usr/sbin/mysqld: Shutdown complete


I've tried the /etc/init.d/mysql stop, and mysqladmin shutdown commands a few
times, and the behaviour is consistent.