Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 537384 - dev-db/redis-2.8.19 - systemd[1]: redis.service start operation timed out. Terminating.
Summary: dev-db/redis-2.8.19 - systemd[1]: redis.service start operation timed out. Te...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Server (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Ultrabug
URL:
Whiteboard:
Keywords:
: 547304 (view as bug list)
Depends on:
Blocks: 539976
  Show dependency tree
 
Reported: 2015-01-22 19:05 UTC by MrSnivvel
Modified: 2015-08-28 10:12 UTC (History)
10 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
Run redis as a 'simple' systemd service (redis.service.patch,352 bytes, patch)
2015-05-04 08:02 UTC, Magnus Kessler
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description MrSnivvel 2015-01-22 19:05:24 UTC
Redis fails to start via systemd:

journalctl -xe:

Jan 22 13:02:00 alphaomega systemd[1]: redis.service start operation timed out. Terminating.
Jan 22 13:02:00 alphaomega systemd[1]: Failed to start A persistent key-value database.
-- Subject: Unit redis.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit redis.service has failed.
-- 
-- The result is failed.
Jan 22 13:02:00 alphaomega systemd[1]: Unit redis.service entered failed state.
Jan 22 13:02:00 alphaomega systemd[1]: redis.service failed.



redis.log:
[20339] 22 Jan 13:00:26.744 # Server started, Redis version 2.8.19
[20339] 22 Jan 13:00:26.744 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
[20339] 22 Jan 13:00:26.744 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
[20339] 22 Jan 13:00:26.745 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
[20339] 22 Jan 13:00:26.748 * The server is now ready to accept connections on port 6379
[20339 | signal handler] (1421953320) Received SIGTERM scheduling shutdown...
[20339] 22 Jan 13:02:00.603 # User requested shutdown...
[20339] 22 Jan 13:02:00.603 # Redis is now ready to exit, bye bye...

SystemD seems to not detect that the server started.

Running /usr/sbin/redis-server /etc/redis.conf from the command line works fine.

Reproducible: Always
Comment 1 Johan Bergström 2015-01-22 22:11:21 UTC
Can someone else that runs redis with systemd possibly chip in? I don't have any system with gentoo/systemd/redis.
Comment 2 Jérémie Horhant 2015-01-27 23:22:08 UTC
This errors is present in all redis versions in the portage tree.

In the redis.service the service type is forking so the redis-server must be launch with the daemonize option to yes.

You can set this option in /etc/redis.conf but it would be better to modify the .service file to force this option.
I also set the pidfile option to be sure that the config file doesn't set an incorrect value.

redis.service

[Unit]
Description=A persistent key-value database
After=syslog.target network.target

[Service]
Type=forking
PIDFile=/var/run/redis/redis.pid
ExecStart=/usr/sbin/redis-server /etc/redis.conf --daemonize yes --pidfile /var/run/redis/redis.pid
User=redis
Group=redis

[Install]
WantedBy=multi-user.target



In addition it would be good to add a comment at the end of these options in redis.conf to tell that theses options are ignored when redis is launch with systemd.
Comment 3 MrSnivvel 2015-02-05 23:03:00 UTC
Manually editing the redis.service file works for me.
Comment 4 MrSnivvel 2015-02-05 23:03:40 UTC
Manually editing the redis.service file as below works for me.

(In reply to Jérémie Horhant from comment #2)
> This errors is present in all redis versions in the portage tree.
> 
> In the redis.service the service type is forking so the redis-server must be
> launch with the daemonize option to yes.
> 
> You can set this option in /etc/redis.conf but it would be better to modify
> the .service file to force this option.
> I also set the pidfile option to be sure that the config file doesn't set an
> incorrect value.
> 
> redis.service
> 
> [Unit]
> Description=A persistent key-value database
> After=syslog.target network.target
> 
> [Service]
> Type=forking
> PIDFile=/var/run/redis/redis.pid
> ExecStart=/usr/sbin/redis-server /etc/redis.conf --daemonize yes --pidfile
> /var/run/redis/redis.pid
> User=redis
> Group=redis
> 
> [Install]
> WantedBy=multi-user.target
> 
> 
> 
> In addition it would be good to add a comment at the end of these options in
> redis.conf to tell that theses options are ignored when redis is launch with
> systemd.
Comment 5 Johan Bergström 2015-02-15 06:27:07 UTC
co-maintainers: are you running a systemd setup? I can't do much here and won't vouch for something that isn't properly reviewed.
Comment 6 Ultrabug gentoo-dev 2015-02-23 15:43:31 UTC
(In reply to Johan Bergström from comment #5)
> co-maintainers: are you running a systemd setup? I can't do much here and
> won't vouch for something that isn't properly reviewed.

I am not.

But I'm for trusting user experience, so can you guys attach a proper patch so I can make this happen ?
Comment 7 Johan Bergström 2015-04-27 08:13:05 UTC
I'll take a look at this tomorrow so we can proceed.
Comment 8 Andrew Savchenko gentoo-dev 2015-04-28 07:05:34 UTC
*** Bug 547304 has been marked as a duplicate of this bug. ***
Comment 9 Magnus Kessler 2015-05-04 08:02:54 UTC
Created attachment 402590 [details, diff]
Run redis as a 'simple' systemd service

The default redis configuration keeps running redis in the foreground. This is the default service type supported by systemd. No need to fork the process and deal with pid files.
Comment 10 Priit Laes (IRC: plaes) 2015-05-05 07:28:33 UTC
(In reply to Magnus Kessler from comment #9)
> Created attachment 402590 [details, diff] [details, diff]
> Run redis as a 'simple' systemd service
> 
> The default redis configuration keeps running redis in the foreground. This
> is the default service type supported by systemd. No need to fork the
> process and deal with pid files.

Thanks, this change works nicely!
Comment 11 Leonardo Amaral 2015-05-11 16:12:53 UTC
For me a merge of https://projects.archlinux.org/svntogit/community.git/tree/trunk/redis.service?h=packages/redis and gentoo config + "daemonize no" in /etc/redis.conf fixed the issue.

My diff:

manaira memcached # diff -Nu /usr/portage/dev-db/redis/files/redis.service /etc/systemd/system/multi-user.target.wants/redis.service 
--- /usr/portage/dev-db/redis/files/redis.service       2013-10-15 15:35:33.000000000 -0300
+++ /etc/systemd/system/multi-user.target.wants/redis.service   2015-05-11 13:08:40.960268556 -0300
@@ -3,11 +3,22 @@
 After=syslog.target network.target
 
 [Service]
-Type=forking
+Type=simple
 PIDFile=/var/run/redis/redis.pid
 ExecStart=/usr/sbin/redis-server /etc/redis.conf
+ExecStop=/usr/bin/redis-cli shutdown
+CapabilityBoundingSet=
+PrivateTmp=true
+PrivateDevices=true
+ProtectSystem=full
+ProtectHome=true
+NoNewPrivileges=true
+RuntimeDirectory=redis
+RuntimeDirectoryMode=755
 User=redis
 Group=redis
+# (open files)
+LimitNOFILE=30032
 
 [Install]
 WantedBy=multi-user.target
Comment 12 Ultrabug gentoo-dev 2015-05-12 16:51:07 UTC
(In reply to Johan Bergström from comment #7)
> I'll take a look at this tomorrow so we can proceed.

Johan, seeing the comments above, plz advise ?
Comment 13 Johan Bergström 2015-05-12 22:16:05 UTC
(In reply to Ultrabug from comment #12)
> (In reply to Johan Bergström from comment #7)
> > I'll take a look at this tomorrow so we can proceed.
> 
> Johan, seeing the comments above, plz advise ?

I'm not a big fan of changing daemonize in the default config. We've already run into trouble with it. Is there a way to avoid it with systemd?
Comment 14 Magnus Kessler 2015-05-15 07:12:14 UTC
(In reply to Johan Bergström from comment #13)
> (In reply to Ultrabug from comment #12)
> > (In reply to Johan Bergström from comment #7)
> > > I'll take a look at this tomorrow so we can proceed.
> > 
> > Johan, seeing the comments above, plz advise ?
> 
> I'm not a big fan of changing daemonize in the default config. We've already
> run into trouble with it. Is there a way to avoid it with systemd?

The default redis config has 'daemonize no', which requires us NOT to use 'Type=forking' in redis.service. That's at the heart of the patch I submitted earlier, where the main goal was NOT to change the default in the redis.conf file. Omitting the 'Type' key is the same a using 'Type=simple'. Systemd will stay attached to the child process and detect when this process exits. No need to monitor the PIDFile of a service. Systemd can also directly terminate the service without the need of a helper program.

The entire proposed redis.service file after my patch looks like this and even simpler than the current version:

##################################################
[Unit]
Description=A persistent key-value database
After=syslog.target network.target

[Service]
ExecStart=/usr/sbin/redis-server /etc/redis.conf
User=redis
Group=redis

[Install]
WantedBy=multi-user.target
##################################################

I cannot comment on most of the additional settings introduced in Leonardo's version; he may be able to explain why they are needed.
Comment 15 Johan Bergström 2015-05-17 03:44:48 UTC
(In reply to Magnus Kessler from comment #14)
> The default redis config has 'daemonize no', which requires us NOT to use
> 'Type=forking' in redis.service. That's at the heart of the patch I
> submitted earlier, where the main goal was NOT to change the default in the
> redis.conf file. Omitting the 'Type' key is the same a using 'Type=simple'.
> Systemd will stay attached to the child process and detect when this process
> exits. No need to monitor the PIDFile of a service. Systemd can also
> directly terminate the service without the need of a helper program.

Yeah, what I meant by default was rather that we've been using daemonize yes in gentoo.

> The entire proposed redis.service file after my patch looks like this and
> even simpler than the current version:
> 
> ##################################################
> [Unit]
> Description=A persistent key-value database
> After=syslog.target network.target
> 
> [Service]
> ExecStart=/usr/sbin/redis-server /etc/redis.conf
> User=redis
> Group=redis
> 
> [Install]
> WantedBy=multi-user.target
> ##################################################
> 
> I cannot comment on most of the additional settings introduced in Leonardo's
> version; he may be able to explain why they are needed.

Ok, lets wait for his input then.
Comment 16 Timo Rothenpieler 2015-05-29 19:42:28 UTC
I just ran into this problem today, and i think the proposed minimal service file is fine.
The only addition to it i'd make is to change the redis-server invocation to

ExecStart=/usr/sbin/redis-server /etc/redis.conf --daemonize no

That way, no matter how the server is configured, the unit will work.
Comment 17 David Novakovic 2015-06-15 07:07:41 UTC
Just dropping a note that Type=simple is a simple fix that worked for me too.
Comment 18 Ultrabug gentoo-dev 2015-07-08 09:58:44 UTC
FYI I just switched to Type=simple in the current systemd unit for redis-3.0.2

If someone could ACK that this simple fix is enough to close this bug, then we could end this never ending problem :)

Thanks

+*redis-3.0.2 (08 Jul 2015)
+
+  08 Jul 2015; Ultrabug <ultrabug@gentoo.org> -redis-3.0.0.ebuild,
+  +redis-3.0.2.ebuild, +files/redis.service-2:
+  version bump fix #551746, change systemd service wrt #537384, drop old
+
Comment 19 Christian Affolter 2015-07-09 08:22:56 UTC
(In reply to Ultrabug from comment #18)
> FYI I just switched to Type=simple in the current systemd unit for
> redis-3.0.2
> 
> If someone could ACK that this simple fix is enough to close this bug, then
> we could end this never ending problem :)
> 
> Thanks

Unfortunately, the service file gets installed as /usr/lib/systemd/system/redis.service-2 which results in an unknown service for systemd.

After manually copying the file to /usr/lib/systemd/system/redis.service Redis starts and stops correctly via systemd, so the configuration is OK.

Please install the file as "redis.service" rather than "redis.service-2", thanks.
Comment 20 Ultrabug gentoo-dev 2015-07-09 08:35:08 UTC
(In reply to Christian Affolter from comment #19)
> (In reply to Ultrabug from comment #18)
> > FYI I just switched to Type=simple in the current systemd unit for
> > redis-3.0.2
> > 
> > If someone could ACK that this simple fix is enough to close this bug, then
> > we could end this never ending problem :)
> > 
> > Thanks
> 
> Unfortunately, the service file gets installed as
> /usr/lib/systemd/system/redis.service-2 which results in an unknown service
> for systemd.
> 
> After manually copying the file to /usr/lib/systemd/system/redis.service
> Redis starts and stops correctly via systemd, so the configuration is OK.
> 
> Please install the file as "redis.service" rather than "redis.service-2",
> thanks.

Sorry Christian, you're right ... stupid me. fixed

+*redis-3.0.2-r1 (09 Jul 2015)
+
+  09 Jul 2015; Ultrabug <ultrabug@gentoo.org> -redis-3.0.2.ebuild,
+  +redis-3.0.2-r1.ebuild:
+  fix systemd unit install thx to Christian Affolter
+
Comment 21 Christian Affolter 2015-07-09 08:39:54 UTC
(In reply to Ultrabug from comment #20)
> (In reply to Christian Affolter from comment #19)
> > (In reply to Ultrabug from comment #18)
> > > FYI I just switched to Type=simple in the current systemd unit for
> > > redis-3.0.2
> > > 
> > > If someone could ACK that this simple fix is enough to close this bug, then
> > > we could end this never ending problem :)
> > > 
> > > Thanks
> > 
> > Unfortunately, the service file gets installed as
> > /usr/lib/systemd/system/redis.service-2 which results in an unknown service
> > for systemd.
> > 
> > After manually copying the file to /usr/lib/systemd/system/redis.service
> > Redis starts and stops correctly via systemd, so the configuration is OK.
> > 
> > Please install the file as "redis.service" rather than "redis.service-2",
> > thanks.
> 
> Sorry Christian, you're right ... stupid me. fixed
> 
> +*redis-3.0.2-r1 (09 Jul 2015)
> +
> +  09 Jul 2015; Ultrabug <ultrabug@gentoo.org> -redis-3.0.2.ebuild,
> +  +redis-3.0.2-r1.ebuild:
> +  fix systemd unit install thx to Christian Affolter
> +

Wow, you're ultra fast ;) Thanks a lot for fixing it!