Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 651246

Summary: sys-apps/openrc patch (for 0.34.11) to decrease openrc starting time
Product: Gentoo Hosted Projects Reporter: nobody <noreply>
Component: OpenRCAssignee: OpenRC Team <openrc>
Status: RESOLVED FIXED    
Severity: enhancement    
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
See Also: https://github.com/openrc/openrc/issues/85
Whiteboard:
Package list:
Runtime testing required: ---
Attachments: caching patch

Description nobody 2018-03-23 10:49:12 UTC
openrc generate a cache file for dependencies, making the bet (time cpu cycle + disk access to generate deptree > time to reload the deptree from the cache) ; sure a good bet.

But when it boot, openrc have no deptree cache available and must rebuild a new one, so on each boot, the cache file doesn't exist.

But you can have a way to hold the cache file, and openrc will have no need to rebuild it when booting, but just reuse it.
Which mean speed improvement get from using the cache WILL also be apply when you boot ; making openrc booting time decrease.



Reproducible: Always

Steps to Reproduce:
1.add patch to /etc/portage/patches/sys-apps/openrc-0.34.11 and rebuild openrc
2.enable rc_logging and reboot
3.cp /run/openrc/deptree /etc/rc_deptree_boot > /etc/local.d/openrc_cache.stop to enable the feature
4.reboot and compare results
Actual Results:  
These are results with my i7-4790K (when the "feature" is enable openrc message is change from "Caching" to "Reusing":
* Caching service dependencies ...
 [ ok ]
 * Time elapsed: 1.498604 
* Reusing service dependencies cache ...
 * Time elapsed: 0.473037 

So down ~1s!


How i do that, i simply add a .stop service that copy /run/openrc/deptree to /etc/rc_deptree_boot (the name is not important, the location in /etc is, because of /usr mount) on shutdown to copy the latest cache file and increase chance the cache will be valid on next boot, the patch doesn't remove openrc check for the cache validity, in case someone mess it in between the boot with some livecd or chroot (which will be worst scenario, because openrc will take time to copy cache file back, see it is dirty and rebuild it).

And when you boot, openrc check /run/openrc/deptree presence : if the file is not there, it check /etc/rc_deptree_boot presence, if the file is there, it then just copy it as /run/openrc/deptree and continue normally (openrc will check its validity...)

Note that the patch is more a proof of concept than a ready one to include, because i have add the timing ability in it (which makes openrc output dirty but more informative about timing taken) and changing the openrc message from "Caching" to "Reusing".
Comment 1 nobody 2018-03-23 10:51:11 UTC
Created attachment 524962 [details, diff]
caching patch
Comment 2 nobody 2018-03-23 11:13:30 UTC
In case it's not obvious, the patch change the bet to
(time cpu cycle + disk access to generate deptree > time to copy the file and reload the deptree from the cache)
Comment 3 nobody 2018-03-25 15:54:56 UTC
AntP point me to /etc/init.d/savecache that is suppose to do that, and /lib/rc/sh/init.sh that restore /lib/rc/cache content on boot.

Actually, yes, it's all there, and working, i made tests and if you let savecache saving the deptree file, init.sh will restore it when booting, and openrc will re-use it.

Result is then the same as my propose patch, except implementation is trivial now as we only need to tell savecache to also backup deptree :)

diff -u savecache.old savecache
--- savecache.old	2018-03-25 17:58:19.564654706 +0200
+++ savecache	2018-03-25 17:57:52.284995748 +0200
@@ -49,7 +49,7 @@
 	fi
 	ebegin "Saving dependency cache"
 	local rc=0 save=
-	for x in shutdowntime softlevel rc.log; do
+	for x in deptree shutdowntime softlevel rc.log; do
 		[ -e "$RC_SVCDIR/$x" ] && save="$save $RC_SVCDIR/$x"
 	done
 	if [ -n "$save" ]; then
Comment 4 nobody 2018-03-25 15:56:28 UTC
I made my tests for savecache changes with openrc-0.35.5
Comment 5 William Hubbs gentoo-dev 2018-03-26 17:04:53 UTC
My concern about caching dependencies, and the whole savecache service
for that matter, is it assumes that / is writable.

This is not always the case, that's the primary reason I generate the dependency
tree on bootup instead of caching it on shutdown.
Comment 6 Thomas Deutschmann (RETIRED) gentoo-dev 2018-03-26 17:25:28 UTC
(In reply to William Hubbs from comment #5)
> My concern about caching dependencies, and the whole savecache service
> for that matter, is it assumes that / is writable.
> 
> This is not always the case, that's the primary reason I generate the
> dependency
> tree on bootup instead of caching it on shutdown.
If this is the only concern you could test for it or make this only optional.

But if we provide a change like this we have to communicate that from now on, user have to call "rc-update --update" on their own after doing changes to /etc/init.d or /etc/conf.d. They can no longer assume that we will pick up the changes.
Comment 7 nobody 2018-03-27 01:31:31 UTC
(In reply to William Hubbs from comment #5)
> My concern about caching dependencies, and the whole savecache service
> for that matter, is it assumes that / is writable.
> 
> This is not always the case, that's the primary reason I generate the
> dependency
> tree on bootup instead of caching it on shutdown.

There's no / writable issue at all.
If / is RO on shutdown, then you cannot write the cache file, no change
If / is RO on boot, you will still be able to copy it from /lib/rc/cache location to the "always writable" /run/openrc
If / is RO on shutdown an a "dirty/bad" cache is already there, it's more a kind of racing issue (because / was RW before) and openrc has a dirty and rebuild detection system and will build a new one

So the real major issue is if someone shutdown / in RW, and later keep shutting down / in RO and latest cache file couldn't be write in /lib/rc/cache as it will always make openrc try use it, mark it dirty, and build a new one (which is worst of the worst case performance).
Comment 8 nobody 2018-03-27 01:34:31 UTC
(In reply to Thomas Deutschmann from comment #6)
> But if we provide a change like this we have to communicate that from now
> on, user have to call "rc-update --update" on their own after doing changes
> to /etc/init.d or /etc/conf.d. They can no longer assume that we will pick
> up the changes.

Even that, while of course it would be "sad" if they aren't aware of it, actually not updating it would just not enable the feature which just gave 100% of what openrc is doing today :)
I agree users should be warn to rc-update to benefit from it, but there's no real issue if someone doesn't.
Comment 9 William Hubbs gentoo-dev 2018-05-02 21:09:07 UTC
See the issue I added to see-also for more info on why we need to always
regenerate the deptree on boot.
Comment 10 William Hubbs gentoo-dev 2018-05-11 18:24:57 UTC
I have added a patch to 0.36 to save the cache again.
However, we always have to regenerate the cache when we boot because of
things like the related issue in see also.

https://github.com/openrc/openrc/commit/f0ad6473