tl;dr: Providing support for all 3 init systems (sysv-rc, Upstart and systemd) isn’t hard, and generating the init scripts / Upstart job / systemd using a template system is a lot easier than I previously thought.
As always, when writing this kind of blog post, I do expect that others will not like what I did. But that’s the point: give me your opinion in a constructive way (please be polite even if you don’t like what you see… I had too many times had to read harsh comments), and I’ll implement your ideas if I find it nice.
History of the implementation: how we came to the idea
I had no plan to do this. I don’t believe what I wrote can be generalized to all of the Debian archive. It’s just that I started doing things, and it made sense when I did it. Let me explain how it happened.
Since it’s clear that many, and especially the most advanced one, may have an opinion about which init system they prefer, and because I also support Ubuntu (at least Trusty), I though it was a good idea to support all the “main” init system: sysv-rc, Upstart and systemd. Though I have counted (for the sake of being exact in this blog) : OpenStack in Debian contains currently 64 init scripts to run daemons in total. That’s quite a lot. A way too much to just write them, all by hand. Though that’s what I was doing for the last years… until this the end of this last summer!
So, doing all by hand, I first started implementing Upstart. Its support was there only when building in Ubuntu (which isn’t the correct thing to do, this is now fixed, read further…). Then we thought about adding support for systemd. Gustavo Panizzo, one of the contributors in the OpenStack packages, started implementing it in Keystone (the auth server for OpenStack) for the Juno release which was released this October. He did that last summer, early enough so we didn’t expect anyone to use the Juno branch Keystone. After some experiments, we had kind of working. What he did was invoking “/etc/init.d/keystone start-systemd”, which was still using start-stop-daemon. Yes, that’s not perfect, and it’s better to use systemd foreground process handling, but at least, we had a unique place where to write the startup scripts, where we check the /etc/default for the logging configuration, configure the log file, and so on.
Then around in october, I took a step backward to see the whole picture with sysv-rc scripts, and saw the mess, with all the tiny, small difference between them. It became clear that I had to do something to make sure they were all the same, with the support for the same things (like which log system to use, where to store the PID, create /var/lib/project, /var/run/project and so on…).
Last, on this month of December, I was able to fix the remaining issues for systemd support, thanks to the awesome contribution of Mikael Cluseau on the Alioth OpenStack packaging list. Now, the systemd unit file is still invoking the init script, but it’s not using start-stop-daemon anymore, no PID file involved, and daemons are used as systemd foreground processes. Finally, daemons service files are also activated on installation (they were not previously).
Implementation
So I took the simplistic approach to use always the same template for the sysv-rc switch/case, and the start and stop functions, happening it at the end of all debian/*.init.in scripts. I started to try to reduce the number of variables, and I was surprised of the result: only a very small part of the init scripts need to change from daemon to daemon. For example, for nova-api, here’s the init script (LSB header stripped-out):
DESC="OpenStack Compute API"
PROJECT_NAME=nova
NAME=${PROJECT_NAME}-api
That is it: only 3 lines, defining only the name of the daemon, the name of the project it attaches (eg: nova, cinder, etc.), and a long description. There’s of course much more complicated init scripts (see the one for neutron-server in the Debian archive for example), but the vast majority only needs the above.
Here’s the sysv-rc init script template that I currently use:
#!/bin/sh
# The content after this line comes from openstack-pkg-tools
# and has been automatically added to a .init.in script, which
# contains only the descriptive part for the daemon. Everything
# else is standardized as a single unique script.
# Author: Thomas Goirand <zigo@debian.org>
# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin
if [ -z "${DAEMON}" ] ; then
DAEMON=/usr/bin/${NAME}
fi
PIDFILE=/var/run/${PROJECT_NAME}/${NAME}.pid
if [ -z "${SCRIPTNAME}" ] ; then
SCRIPTNAME=/etc/init.d/${NAME}
fi
if [ -z "${SYSTEM_USER}" ] ; then
SYSTEM_USER=${PROJECT_NAME}
fi
if [ -z "${SYSTEM_USER}" ] ; then
SYSTEM_GROUP=${PROJECT_NAME}
fi
if [ "${SYSTEM_USER}" != "root" ] ; then
STARTDAEMON_CHUID="--chuid ${SYSTEM_USER}:${SYSTEM_GROUP}"
fi
if [ -z "${CONFIG_FILE}" ] ; then
CONFIG_FILE=/etc/${PROJECT_NAME}/${PROJECT_NAME}.conf
fi
LOGFILE=/var/log/${PROJECT_NAME}/${NAME}.log
if [ -z "${NO_OPENSTACK_CONFIG_FILE_DAEMON_ARG}" ] ; then
DAEMON_ARGS="${DAEMON_ARGS} --config-file=${CONFIG_FILE}"
fi
# Exit if the package is not installed
[ -x $DAEMON ] || exit 0
# If ran as root, create /var/lock/X, /var/run/X, /var/lib/X and /var/log/X as needed
if [ "x$USER" = "xroot" ] ; then
for i in lock run log lib ; do
mkdir -p /var/$i/${PROJECT_NAME}
chown ${SYSTEM_USER} /var/$i/${PROJECT_NAME}
done
fi
# This defines init_is_upstart which we use later on (+ more...)
. /lib/lsb/init-functions
# Manage log options: logfile and/or syslog, depending on user's choosing
[ -r /etc/default/openstack ] && . /etc/default/openstack
[ -r /etc/default/$NAME ] && . /etc/default/$NAME
[ "x$USE_SYSLOG" = "xyes" ] && DAEMON_ARGS="$DAEMON_ARGS --use-syslog"
[ "x$USE_LOGFILE" != "xno" ] && DAEMON_ARGS="$DAEMON_ARGS --log-file=$LOGFILE"
do_start() {
start-stop-daemon --start --quiet --background ${STARTDAEMON_CHUID} --make-pidfile --pidfile ${PIDFILE} --chdir /var/lib/${PROJECT_NAME} --startas $DAEMON \
--test > /dev/null || return 1
start-stop-daemon --start --quiet --background ${STARTDAEMON_CHUID} --make-pidfile --pidfile ${PIDFILE} --chdir /var/lib/${PROJECT_NAME} --startas $DAEMON \
-- $DAEMON_ARGS || return 2
}
do_stop() {
start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE
RETVAL=$?
rm -f $PIDFILE
return "$RETVAL"
}
do_systemd_start() {
exec $DAEMON $DAEMON_ARGS
}
case "$1" in
start)
init_is_upstart > /dev/null 2>&1 && exit 1
log_daemon_msg "Starting $DESC" "$NAME"
do_start
case $? in
0|1) log_end_msg 0 ;;
2) log_end_msg 1 ;;
esac
;;
stop)
init_is_upstart > /dev/null 2>&1 && exit 0
log_daemon_msg "Stopping $DESC" "$NAME"
do_stop
case $? in
0|1) log_end_msg 0 ;;
2) log_end_msg 1 ;;
esac
;;
status)
status_of_proc "$DAEMON" "$NAME" && exit 0 || exit $?
;;
systemd-start)
do_systemd_start
;;
restart|force-reload)
init_is_upstart > /dev/null 2>&1 && exit 1
log_daemon_msg "Restarting $DESC" "$NAME"
do_stop
case $? in
0|1)
do_start
case $? in
0) log_end_msg 0 ;;
1) log_end_msg 1 ;; # Old process is still running
*) log_end_msg 1 ;; # Failed to start
esac
;;
*) log_end_msg 1 ;; # Failed to stop
esac
;;
*)
echo "Usage: $SCRIPTNAME {start|stop|status|restart|force-reload|systemd-start}" >&2
exit 3
;;
esac
exit 0
Nothing particularly fancy here… You’ll noticed that it’s really OpenStack centric (see the LOGFILE and CONFIGFILE things…). You may have also noticed the call to “init_is_upstart” which is needed for upstart support. I’m not sure if it’s at the correct place in the init script. Should I put that on top of the script? Was I right with the exit values for it? Please send me your comments…
Then I thought about generalizing all of this. Because not only the sysv-rc scripts needed to be squared-up, but also Upstart. The approach here was to source the sysv-rc script in debian/*.init.in, and then generate the Upstart job accordingly, using the above 3 variables (or more as needed). Here, the fun is that, instead of taking the approach of calculating everything at runtime with the sysv-rc, for Upstart jobs, many things are calculated at build time. For each debian/*.init.in script that the debian/rules finds, pkgos-gen-upstart-job is called. Here’s pkgos-gen-upstart-job:
#!/bin/sh
INIT_TEMPLATE=${1}
UPSTART_FILE=`echo ${INIT_TEMPLATE} | sed 's/.init.in/.upstart/'`
# Get the variables defined in the init template
. ${INIT_TEMPLATE}
## Find out what should go in After=
#SHOULD_START=`cat ${INIT_TEMPLATE} | grep "# Should-Start:" | sed 's/# Should-Start://'`
#
#if [ -n "${SHOULD_START}" ] ; then
# AFTER="After="
# for i in ${SHOULD_START} ; do
# AFTER="${AFTER}${i}.service "
# done
#fi
if [ -z "${DAEMON}" ] ; then
DAEMON=/usr/bin/${NAME}
fi
PIDFILE=/var/run/${PROJECT_NAME}/${NAME}.pid
if [ -z "${SCRIPTNAME}" ] ; then
SCRIPTNAME=/etc/init.d/${NAME}
fi
if [ -z "${SYSTEM_USER}" ] ; then
SYSTEM_USER=${PROJECT_NAME}
fi
if [ -z "${SYSTEM_GROUP}" ] ; then
SYSTEM_GROUP=${PROJECT_NAME}
fi
if [ "${SYSTEM_USER}" != "root" ] ; then
STARTDAEMON_CHUID="--chuid ${SYSTEM_USER}:${SYSTEM_GROUP}"
fi
if [ -z "${CONFIG_FILE}" ] ; then
CONFIG_FILE=/etc/${PROJECT_NAME}/${PROJECT_NAME}.conf
fi
LOGFILE=/var/log/${PROJECT_NAME}/${NAME}.log
DAEMON_ARGS="${DAEMON_ARGS} --config-file=${CONFIG_FILE}"
echo "description \"${DESC}\"
author \"Thomas Goirand <zigo@debian.org>\"
start on runlevel [2345]
stop on runlevel [!2345]
chdir /var/run
pre-start script
for i in lock run log lib ; do
mkdir -p /var/\$i/${PROJECT_NAME}
chown ${SYSTEM_USER} /var/\$i/${PROJECT_NAME}
done
end script
script
[ -x \"${DAEMON}\" ] || exit 0
DAEMON_ARGS=\"${DAEMON_ARGS}\"
[ -r /etc/default/openstack ] && . /etc/default/openstack
[ -r /etc/default/\$UPSTART_JOB ] && . /etc/default/\$UPSTART_JOB
[ \"x\$USE_SYSLOG\" = \"xyes\" ] && DAEMON_ARGS=\"\$DAEMON_ARGS --use-syslog\"
[ \"x\$USE_LOGFILE\" != \"xno\" ] && DAEMON_ARGS=\"\$DAEMON_ARGS --log-file=${LOGFILE}\"
exec start-stop-daemon --start --chdir /var/lib/${PROJECT_NAME} \\
${STARTDAEMON_CHUID} --make-pidfile --pidfile ${PIDFILE} \\
--exec ${DAEMON} -- --config-file=${CONFIG_FILE} \${DAEMON_ARGS}
end script
" >${UPSTART_FILE}
The only thing which I don’t know how to do, is how to implement the Should-Start / Should-Stop in an Upstart job. Can anyone shoot me a mail and tell me the solution?
Then, I wanted to add support for systemd. Here, we cheated, since we only just called the sysv-rc script from the systemd unit, however, the systemd-start target uses exec, so the process stays in the foreground. It’s also much smaller than the Upstart thing. However, here, I could implement the “After” thing, corresponding to the Should-Start:
#!/bin/sh
INIT_TEMPLATE=${1}
SERVICE_FILE=`echo ${INIT_TEMPLATE} | sed 's/.init.in/.service/'`
# Get the variables defined in the init template
. ${INIT_TEMPLATE}
if [ -z "${SCRIPTNAME}" ] ; then
SCRIPTNAME=/etc/init.d/${NAME}
fi
if [ -z "${SYSTEM_USER}" ] ; then
SYSTEM_USER=${PROJECT_NAME}
fi
if [ -z "${SYSTEM_GROUP}" ] ; then
SYSTEM_GROUP=${PROJECT_NAME}
fi
# Find out what should go in After=
SHOULD_START=`cat ${INIT_TEMPLATE} | grep "# Should-Start:" | sed 's/# Should-Start://'`
if [ -n "${SHOULD_START}" ] ; then
AFTER="After="
for i in ${SHOULD_START} ; do
AFTER="${AFTER}${i}.service "
done
fi
echo "[Unit]
Description=${DESC}
$AFTER
[Service]
User=${SYSTEM_USER}
Group=${SYSTEM_GROUP}
WorkingDirectory=/var/lib/${PROJECT_NAME}
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p /var/lock/${PROJECT_NAME} /var/log/${PROJECT_NAME} /var/lib/${PROJECT_NAME}
ExecStartPre=/bin/chown ${SYSTEM_USER}:${SYSTEM_GROUP} /var/lock/${PROJECT_NAME} /var/log/${PROJECT_NAME} /var/lib/${PROJECT_NAME}
ExecStart=${SCRIPTNAME} systemd-start
Restart=on-failure
[Install]
WantedBy=multi-user.target
" >${SERVICE_FILE}
As you can see, it’s calling /etc/init.d/${SCRIPTNAME} sytemd-start, which isn’t great. I’d be happy to have comments from systemd user / maintainers on how to fix it to make it better.
Integrating in debian/rules
To integrate with the Debian package build system, we only need had to write this:
override_dh_installinit: # Create the init scripts from the template for i in `ls -1 debian/*.init.in` ; do \ MYINIT=`echo $$i | sed s/.init.in//` ; \ cp $$i $$MYINIT.init ; \ cat /usr/share/openstack-pkg-tools/init-script-template >>$$MYINIT.init ; \ pkgos-gen-systemd-unit $$i ; \ done # If there's an upstart.in file, use that one instead of the generated one for i in `ls -1 debian/*.upstart.in` ; do \ MYPKG=`echo $$i | sed s/.upstart.in//` ; \ cp $$MYPKG.upstart.in $$MYPKG.upstart ; \ done # Generate the upstart job if there's no already existing .upstart.in for i in `ls debian/*.init.in` ; do \ MYINIT=`echo $$i | sed s/.init.in/.upstart.in/` ; \ if ! [ -e $$MYINIT ] ; then \ pkgos-gen-upstart-job $$i ; \ fi \ done dh_installinit --error-handler=true # Generate the systemd unit file # Note: because dh_systemd_enable is called by the # dh sequencer *before* dh_installinit, we have # to process it manually. for i in `ls debian/*.init.in` ; do \ pkgos-gen-systemd-unit $$i ; \ MYSERVICE=`echo $$i | sed 's/debian\///'` ; \ MYSERVICE=`echo $$MYSERVICE | sed 's/.init.in/.service/'` ; \ dh_systemd_enable $$MYSERVICE ; \ done
As you can see, it’s possible to use a debian/*.upstart.in and not use the templating system, in the more complicated case (I needed it mostly for neutron-server and neutron-plugin-openvswitch-agent).
Conclusion
I do not pretend that what I wrote in the openstack-pkg-tools is the ultimate solution. But I’m convince that it answers our own need as the OpenStack maintainers in Debian. There’s a lot of room for improvements (like implementing the Should-Start in Upstart jobs, or stop calling the sysv-rc script in the systemd units), but that this is a very good move that we did to use templates and generated scripts, as the init scripts are a way more easy to maintain now, in a much more unified way. As I’m not completely satisfied for the systemd and Upstart implementation, I’m sure that there’s already a huge improvements on the sysv-rc script maintainability.
Last and again: please send your comments and help improving the above! :)