OpenStack packaging activity, November 2014 to January 2015

November 2014:
Sunday 2nd:
– Travel from Moscow to Paris

Monday 3rd to Sunday 8th:
– Summit in Paris

Monday 10th:
– Uploaded python-rudolf to Sid (needed by Fuel)
– Uploaded python-invoke and python-invocations (needed to run fabric’s unit tests)
– Uploaded python-requests-kerberos/0.5-2 fixing CVE-2014-8650: failure to handle mutual authentication. Asked the release team for unblock.
– Uploaded openstack-pkg-tools version 19 fixing startup with systemd in Jessie (added RuntimeDirectory directive). Asked the release team for unblock.
– Opened ticket to remove TripleO, Tuskar and Ironic packages from Jessie. I don’t consider them ready for a Debian stable release, and there’s no long term support from upstream.
– Fixed Designate Juno dbsync process which prevented it from being installed.
– Fixed Ironic Juno unowned files after purge (policy 6.8, 10.8): /var/lib/ironic/{cache, ironicdb} (eg: purging these folders on purge)

Thuesday 11:
– Fixed nova-api “CVE-2014-3708: Nova network DoS through API filtering” in both the Juno and Icehouse release. Asked the release team to unblock the Icehouse version for Jessie. See: https://bugs.debian.org/769163
– Uploaded Cinder with Duch debconf translation fix and pt.po
– Uploaded python-django-pyscss with upstream patch for Django 1.7 support instead of the Debian one that I wrote 2 months ago. Asked the release team to unblock which they did.

Wednesday 12:
– Uploaded fix for horizon (see #769101) unowned files after purge (policy 6.8, 10.8). Now purging /usr/share/openstack-dashboard/openstack_dashboard on purge.
– Uploaded Ironic with Duch translations of debconf
– Uploaded Designate with Duch translations of Debconf screens
– Uploaded openstack-trove with Duch translations of Debconf screens
– Uploaded Tuskar with Duch translations of Debconf screens
– Updated python-oslotest in Experimental to version 1.2.0

Thursday 13:
– Uploaded new packages: python-oslo.middleware and python-oslo.concurrency.
– Opened a new packaging branch for Nova Kilo, and updated (build-)depends.
– Uploaded fix for Icehouse Cinder: “delete volume failed due to unicode problems”, and asked for unblock.
– Uploaded new package: python-pygit2 and python-xmlbuilder, needed for fuel-agent-ci.
– Uploaded sheepdog with Duch debconf translation.
– Uploaded python-daemonize to Sid (in FTP master NEW queue).
– Re-uploaded python-invoke after FTP master rejection (missing copyright information)

Friday 14:
– Uploaded liberasurecode & python-pyeclib to Sid, now in the FTP masters NEW queue waiting for approval. This will soon be needed by Swift.

Monday 17:
– Worked on the Cobbler packaging (all day long…)

Tuesday 18:
– Worked on backporting all of Fuel packages to Wheezy. Done with fuelclient already.
– Uploaded ruby-cstruct and ruby-rethtool to Sid (needed by nailgun-agent)

Wednesday 19:
– Uploaded pyeclib again, with fixes for the build-depends. Package is still in the NEW queue anyway.
– Built a Debian-based bootstrap hardware discovery image for Fuel, and … it seems that it works already (to be checked…)! \o/
To be added as packages in the ISO:
* nailgun-mcagents
* nailgun-net-check
* fuel-agent
* python-tasklib

Thursday 20:
– Uploaded python-tasklib to Sid (now in NEW queue…)
– Continued working on the discovery bootstrap ISO

Friday 21:
– Documented Sahara procedure in Debian in the official install-guide: https://review.openstack.org/136237
– Fixed oslo.messaging so it doesn’t use PROTOCOL_SSLv3 because its support has been removed from Debian (due to possible protocol downgrade attacks): https://review.openstack.org/136278 and uploaded fixed packages for Sid and Experimental.
– Uploaded fixed Neutron packages for CVE-2014-7821 in both Sid and Experimental (eg: Icehouse and Juno)

Monday 24:
– Uploaded new package: python-os-client-config (in NEW queue)
– Installed new Xen server to be used as my new Jenkins build machine
– Moved the juno-wheezy VM to it
– Finished to package python-pymysql and uploaded to Sid. It’s now running all unit tests successfully! \o/

Tuesday 25:
– Uploaded fix for openstack-debian-images to add the -o compat=1.0 option when building an image with Qemu > 1.0. Opened bug to the release team to have it unblocked.
– Continued working on unit tests for fuel-nailgun.

Wednesday 26:
– Uploaded python-os-net-config to Sid (new package)
– Worked briefly on python-cassandra-driver. It needs cassandra to be in, which is a LOT of work.
– Found a (not useable) hack to run nailgun unit tests. It works, however, it doesn’t seem like fuel-nailgun is designed to be able to use unix socket for the postgres connection in its unit tests.
– Uploaded python-pykmip to Sid (new package)
– Updated the Debian wheezy backport repository for libvirt to version 1.2.9 from official wheezy-backports. Removed policykit-1 and libusb from there too, as it broke stuff to use a backported version (X and usb were not useable on my Wheezy laptop when using it…).

Thursday 27 & Friday 28:
– Uploaded new Javascript packages or dependencies for Fuel: libjs-autonumeric, libjs-backbone-deep-model, libjs-backbone.stickit, libjs-cocktail, libjs-i18next, libjs-require-css, libjs-requirejs, libjs-requirejs-text

Sunday 30:
– Uploaded debian/copyright fixes for libjs-backbone-deep-model, libjs-backbone.stickit and libjs-cocktail after the packages were accepted by the FTP masters and they gave remarks about copyright.

DECEMBER 2014

Monday 01:
– Uploaded new Debian image to MOX, after I unerstood the issue was about the architecture field that I was wrongly filling. I’ll be able to use that for Tempest checking on my dev account.

Tuesday 02:
– Uploaded python-q-text-as-data to Sid (new awesome package!)
– Uploaded Horizon with some triggers mechanisms to start the compress when one of its JS depends is updated. That’s very important for security!
– Uploaded a fixed version of heat-cfntools to Sid (it was missing the /usr/lib/python* folder). Asked the release team for an unblock so it can reach Jessie.
– Fixed unit tests in fuel-nailgun, thanks to a patch from Sebastian Kalinowski. Now all unit tests are passing but one (for which I opened a launchpad bug: tests are trying to write in /var/log/nailgun, which is impossible at package build time).

Wednesday 03:
– Uploaded fixed version of ruby-rethtool after FTP master’s rejection and upstream correction of licensing files.
– Uploaded fixed version of libjs-require-css after FTP master’s rejection
– Fixed (in Git only) python-sysv-ipc missing build-depends on dh-python as per bug opened by James Page (this is not so important, but I did it still).
– Continued working on the tempest-ci scripts.
– Added to the image-guide docs about openstack-debian-images: https://review.openstack.org/#/c/138743/

Thursday 04:
– Uploaded new package: python-proliantutils. Send patch to upstream about an issue in indentation (mix-up with space and tabs) which made the package uninstallable with Python 3.4.

Friday 05:
– Worked on the package CI.

Monday 07:
– Worked on the package CI. All works now, up to all of the Tempest tests for Keystone. Now need to fix the neutron config.

Thuesday 08:
– Continued working on the CI.

Wednesday 09:
– Uploaded fix for FTBFS of python-tasklib (Closes: #772606)
– Uploaded fix for libjerasure-deb missing dependency on libgf-complete-dev, package already unblocked and will migrate to Jessie.
– Uploaded fix for Designate Juno fail to upgrade from Icehouse: this was due to the database_connection directive renamed to connection =.
– Uploaded fix for Designate purge in Sid (Icehouse release of Designate).
– Commited to git updates of the German debconf translation in both Icehouse and Juno.
– Updated nova to use libvirtd as init script dependency instead of libvirt-bin (this was renamed in the libvirt-daemon-system package).
– Do not touch the db connection directive if user didn’t ask for db handling by the package.

Thursday 10 to Saturday 13:
– Finally understood the issues with systemd service files not being activated by default. Fixed openstack-pkg-tools, and uploaded version 20 to Sid, after the release team accepted the changes.

Sunday 14:
– Uploaded Juno 2014.2.1 to Experimental: ceilometer, cinder, glance, python-glance-store, heat, horizon, keystone

Monday 15:
– Finished uploading Juno 2014.2.1 to Experimental: Nova, Neutron, Sahara

Tuesday 16:
– Added crontab to flush tokens in Icehouse Keystone
– Some more CI work

Wednesday 17:
– Uploaded keystone with systemd fix and crontab to flush the token table in Sid (eg: Icehouse).
– Uploaded nova Icehouse with a bunch of fixes in Sid.

Thursday 18:
– Updated some issues in Nova Icehouse (Sid/Jessie)

Friday 19:
– Started building a new Jenkins instance for building Kilo packages

Monday 22:
– Finished building the new Jenkins instance for building Kilo packages, and rebuilt every packages there, using Jessie as a base.

Tuesday 23:
– Updated version for the following packages: oslo.utils, oslo.middleware, stevedore, oslo.concurency, pecan, oslo.concurrency, python-oslo.vmware, python-glance-store
– Built so far: Ceilometer, Keystone, python-glanceclient, cinder, glance

Wednesday 24:
– Continued packaging Kilo beta 1. Updated: nova, designate, neutron
– Uploaded python-tempest-lib to Debian Unstable (new package)

Wednesday 31:
– Continued packaging Kilo beta 1. Updated: heat

JANUARY 2015

Thursday 01:
– Continued packaging Kilo beta 1. Updated: ironic, openstack-trove, openstack-doc-tools, ceilometer

Friday 02:
– Finished packaging Kilo beta 1. Updated: Sahara, Murano, Murano-dashboard, Murano-agent

Sunday 04:
– Started testing Kilo beta 1. Fixed a few issues on default configuration for Ceilometer and Glance.

Monday 05:
– Fixed openstack-pkg-tools which failed to create PID files at boot time, Uploaded to Sid, asked the release team for unblock.
– Uploaded ceilometer & cinder to Sid, rebuilt against openstack-pkg-tools 21.
– Did more testing of Kilo beta 1, fixed a few more minor issues.

Tuesday 06:
– Uploaded glance, neutron, nova, designate, keystone, heat, trove to Sid, so that all sysv-rc init scripts are fixed with the new openstack-pkg-tools 21. Designate, heat, keystone and trove contains other minor fixes reported to the Debian BTS.

Wednesday 07:
– Asked the Debian release team (open bugs with debdiff as attachment) for unblocks of glance, neutron, nova, designate, keystone, heat, trove so they migrate to Jessie.
– Fixed a few minor issues tracked in the Debian BTS on various packages.

Thesday 08:
– James Page from Canonical informed me that they are now using openstack-pkg-tools for maintaining their daemons in Nova, Cinder and Keystone in Ubuntu. That’s an awesome news : more QA for both platforms.
– James Page found out that dh_installinit *must* be called *after* the call of dh_systemd_enable, otherwise, daemons aren’t started automatically at the first install of packages, as the unmask of systemd happens after the invoke-rc.d.

Friday 09:
– Did some QA checks on the latest upload. Fixed Heat which broke because using the wrong template name (glance instead of heat).

Monday 12:
– Started re-running the automated openstack-deploy scrip in Icehouse, Juno and Kilo. Found out the issue in Keystone wasn’t fixed in Juno (but was fixed in other releases), and fixed it.
– Removed the use of ssl.PROTOCOL_SSLv3 from heat (removed form Debian). Uploaded the fixed package to Sid.
– All of openstack-deploy (debian/kilo branch) now works and succesfully installs OpenStack again.

If dh_installinit is called before, we have:

# Automatically added by dh_installinit
if [ -x "/etc/init.d/keystone" ]; then
update-rc.d keystone defaults >/dev/null
fi
if [ -x "/etc/init.d/keystone" ] || [ -e "/etc/init/keystone.conf" ]; then
invoke-rc.d keystone start || true
fi
# End automatically added section
# Automatically added by dh_systemd_enable
# This will only remove masks created by d-s-h on package removal.
deb-systemd-helper unmask keystone.service >/dev/null || true

# was-enabled defaults to true, so new installations run enable.
if deb-systemd-helper --quiet was-enabled keystone.service; then
# Enables the unit on first installation, creates new
# symlinks on upgrades if the unit file has changed.
deb-systemd-helper enable keystone.service >/dev/null || true
else
# Update the statefile to add new symlinks (if any), which need to be
# cleaned up on purge. Also remove old symlinks.
deb-systemd-helper update-state keystone.service >/dev/null || true
fi
# End automatically added section

If it’s called after dh_systemd_enable, we have:

# Automatically added by dh_systemd_enable
# This will only remove masks created by d-s-h on package removal.
deb-systemd-helper unmask keystone.service >/dev/null || true

# was-enabled defaults to true, so new installations run enable.
if deb-systemd-helper --quiet was-enabled keystone.service; then
# Enables the unit on first installation, creates new
# symlinks on upgrades if the unit file has changed.
deb-systemd-helper enable keystone.service >/dev/null || true
else
# Update the statefile to add new symlinks (if any), which need to be
# cleaned up on purge. Also remove old symlinks.
deb-systemd-helper update-state keystone.service >/dev/null || true
fi
# End automatically added section
# Automatically added by dh_installinit
if [ -x "/etc/init.d/keystone" ]; then
update-rc.d keystone defaults >/dev/null
fi
if [ -x "/etc/init.d/keystone" ] || [ -e "/etc/init/keystone.conf" ]; then
invoke-rc.d keystone start || true
fi
# End automatically added section

As a consequence, I have to re-upload version 22 of openstack-pkg-tools and also re-upload all OpenStack core packages to Debian Sid.

– Fixed a number of issues like:
* dbc_upgrade = true check which shouldn’t have been there in postinst.
* <project>/configure_db default value is now always false
* db_sync and pkgos_dbc_postinst are now only done if <project>/configure_db is set to true.
– Rebuilt all packages in Juno and Kilo with the above changes.

Tuesday 13:
– Opened unblock bugs for the release team to unblock all fixed packages.
– Made more tests in Juno and Kilo to make sure the fixed bugs in Icehouse are fixed there too.
– Fixed numerous issues in Trove (missing trove-conductor.conf, wrong trove-api init file, etc.). More work will be needed for it for both Icehouse and newer releases.

Wednesday 14:
– Did a doc meeting about debconf. Some doc contributors still want to kill the debconf / debian manual, and I have to not agree.
– Made a new patch to better document the keystone install procedure:
– Did some bug triaging in the doc about Debian.
– Uploaded new versions of core packages to Experimental (eg: Juno) built against openstack-pkg-tools >= 22~, and some fixes forward ported from Icehouse: Keystone, Ceilometer, Cinder, Glance, Heat, Ironic, Murano, Neutron, Nova, Saraha and Murano-agent. All where rebuilt in Juno (Wheezy + Trusty) and Kilo (Jessie only) on my Jenkins.

Thuesday 15:
– Succesfully booted a live-build Debian live image containing mcollective and nailgun-agent as a Debian replacement for the hardware discovery / boostrap image of Fuel. Now, I need to find a way to use just a kernel + initramfs

Friday 16 to Tuesday 20:
– Worked on the packaging CI.

Wednesday 21:
– Fixed https://bugs.debian.org/775636 (Horizon failed to build due to a Moscow timezone change and wrong test). Uploaded to Sid, asked for unblock.
– Fixed https://bugs.debian.org/775926: CVE-2015-1195: Glance still allows users to download and delete any file in glance-api server (applied upstream patch). Uploaded to Sid, asked for unblock. Uploaded Juno version to Experimental.
– Uploaded openstack-trove with the remaining fixes, asked release team for unblock.
– Uploaded python-glanceclient 0.15.0 (Juno) to Experimental because it fixes an issue with HTTPS. Added to it a patch from James Page not yet merged, which fixes unit test with Python 2.7.9 (7 failures otherwise).
– Uploaded python-xstatic-d3 as it can’t be installed anymore in Sid after a new version of d3 was uploaded.

Thursday 22:
– Uploaded python-xstatic-smart-table and libjs-angularjs-smart-table to Sid (new packages, now in NEW queue).

Friday 23:
– Ask for the removal of the below list of packages from Jessie:
python-xstatic
python-xstatic-angular
python-xstatic-angular-cookies
python-xstatic-angular-mock
python-xstatic-bootstrap-datepicker
python-xstatic-bootstrap-scss
python-xstatic-d3
python-xstatic-font-awesome
python-xstatic-hogan
python-xstatic-jasmine
python-xstatic-jquery
python-xstatic-jquery-migrate
python-xstatic-jquery-ui
python-xstatic-jquery.bootstrap.wizard
python-xstatic-jquery.quicksearch
python-xstatic-jquery.tablesorter
python-xstatic-jsencrypt
python-xstatic-qunit
python-xstatic-rickshaw
python-xstatic-spin
libjs-jsencrypt
libjs-spin.js
libjs-twitter-bootstrap-datepicker
libjs-twitter-bootstrap-wizard

They are used only in OpenStack Horizon starting on 2014.2 (aka Juno), and Jessie is shipped with Icehouse, so it’s IMO best to not carry the burden of maintaining these packages for the life of Jessie.

Monday 26:
– Enhanced and review requested changes for https://review.openstack.org/147296 (ie: Keystone install with more details about what the package does).
– Finished testing network on the CI install. Now need to automate all.

Tuesday 27:
– Closed all bugs on the rabbitmq-server package (2 correction, one bug triage).
– Uploaded a fix for the missing conntrack dependency in neutron-l3-agent.
– Restarted working on CI setup of Juno after success with manual install in a Xen domU.
– Uploaded fix to make sheepdog build reproducible (patch from the Debian BTS).

Thursday 28:
– Fixed and uploaded to Sid openstack-debian-images 2 bugs reported by Steve McIntire. Official Debian images for OpenStack are now available at:
http://cdimage.debian.org/cdimage/openstack/ \o/
Note that this is the weekly build of testing. We wont get Debian Stable images before Jessie is out.
– Documented the new image thing here: http://docs.openstack.org/image-guide/content/ch_obtaining_images.html#debian-images as a new patch: https://review.openstack.org/#/c/151015/
– Fixed my patch for keystone debconf doc at: https://review.openstack.org/#/c/147296/

Wednesday 29:
– Continued working on packaging CI

Thursday 30:
– Fixed CVE on Neutron (Juno): L3 agent denial of service with radvd 2.0+
– Fixed CVE on Glance (Icehouse + Juno): Glance user storage quota bypass. Asked release team for unblock.
– Fixed the image-guide patch after review (ie: https://review.openstack.org/151015)

OpenStack debian image available from cdimage.debian.org

About a year and a half after I started writing the openstack-debian-images package, I’m very happy to announce to everyone that, thanks to Steve McIntyre’s help, the official OpenStack Debian image is now generated at the same time as the official Debian CD ISO images. If you are a cloud user, if you use OpenStack on a private cloud, or if you are a public cloud operator, then you may want to download the weekly build of the OpenStack image from here:

http://cdimage.debian.org/cdimage/openstack/testing/

Note that for the moment, there’s only the amd64 arch available, but I don’t think this is a problem: so far, I haven’t found any public cloud provider offering anything else than Intel 64 bits arch. Maybe this will change over the course of this year, and we will need arm64, but this can be added later on.

Now, for later plans: I still have 2 bugs to fix on the openstack-debian-images package (the default 1GB size is now just a bit too small for Jessie, and the script exits with zero in case of error), but nothing that prevents its use right now. I don’t think it will be a problem for the release team to accept these small changes before Jessie is out.

When generating the image, Steve also wants to generate a sources.tar.gz containing all the source packages that we include on the image. He already has the script (which is used as a hook script when running the build-openstack-debian-image script), and I am planning to add it as a documentation in /usr/share/doc/openstack-debian-images.

Last, probably it would be a good idea to install grub-xen, just as Ian Campbell suggested to make it possible for this image to run in AWS or other Xen based clouds. I would need to be able to test this though. If you can contribute with this kind of test, please get in touch.

Feel free to play with all of this, and customize your Jessie images if you need to. The script is (on purpose) very small (around 400 lines of shell script) and easy to understand (no function, it’s mostly linear from top to bottom of the file), so it is also very easy to hack, plus it has a convenient hook script facility where you can do all sorts of things (copying files, apt-get install stuff, running things in the chroot, etc.).

Again, thanks so much to Steve for working on using the script during the CD builds. This feels me with joy that Debian finally has official images for OpenStack.

Supporting 3 init systems in OpenStack packages

tl;dr: Providing support for all 3 init systems (sysv-rc, Upstart and systemd) isn’t hard, and generating the init scripts / Upstart job / systemd using a template system is a lot easier than I previously thought.

As always, when writing this kind of blog post, I do expect that others will not like what I did. But that’s the point: give me your opinion in a constructive way (please be polite even if you don’t like what you see… I had too many times had to read harsh comments), and I’ll implement your ideas if I find it nice.

History of the implementation: how we came to the idea

I had no plan to do this. I don’t believe what I wrote can be generalized to all of the Debian archive. It’s just that I started doing things, and it made sense when I did it. Let me explain how it happened.

Since it’s clear that many, and especially the most advanced one, may have an opinion about which init system they prefer, and because I also support Ubuntu (at least Trusty), I though it was a good idea to support all the “main” init system: sysv-rc, Upstart and systemd. Though I have counted (for the sake of being exact in this blog) : OpenStack in Debian contains currently 64 init scripts to run daemons in total. That’s quite a lot. A way too much to just write them, all by hand. Though that’s what I was doing for the last years… until this the end of this last summer!

So, doing all by hand, I first started implementing Upstart. Its support was there only when building in Ubuntu (which isn’t the correct thing to do, this is now fixed, read further…). Then we thought about adding support for systemd. Gustavo Panizzo, one of the contributors in the OpenStack packages, started implementing it in Keystone (the auth server for OpenStack) for the Juno release which was released this October. He did that last summer, early enough so we didn’t expect anyone to use the Juno branch Keystone. After some experiments, we had kind of working. What he did was invoking “/etc/init.d/keystone start-systemd”, which was still using start-stop-daemon. Yes, that’s not perfect, and it’s better to use systemd foreground process handling, but at least, we had a unique place where to write the startup scripts, where we check the /etc/default for the logging configuration, configure the log file, and so on.

Then around in october, I took a step backward to see the whole picture with sysv-rc scripts, and saw the mess, with all the tiny, small difference between them. It became clear that I had to do something to make sure they were all the same, with the support for the same things (like which log system to use, where to store the PID, create /var/lib/project, /var/run/project and so on…).

Last, on this month of December, I was able to fix the remaining issues for systemd support, thanks to the awesome contribution of Mikael Cluseau on the Alioth OpenStack packaging list. Now, the systemd unit file is still invoking the init script, but it’s not using start-stop-daemon anymore, no PID file involved, and daemons are used as systemd foreground processes. Finally, daemons service files are also activated on installation (they were not previously).

Implementation

So I took the simplistic approach to use always the same template for the sysv-rc switch/case, and the start and stop functions, happening it at the end of all debian/*.init.in scripts. I started to try to reduce the number of variables, and I was surprised of the result: only a very small part of the init scripts need to change from daemon to daemon. For example, for nova-api, here’s the init script (LSB header stripped-out):

DESC="OpenStack Compute API"
PROJECT_NAME=nova
NAME=${PROJECT_NAME}-api

That is it: only 3 lines, defining only the name of the daemon, the name of the project it attaches (eg: nova, cinder, etc.), and a long description. There’s of course much more complicated init scripts (see the one for neutron-server in the Debian archive for example), but the vast majority only needs the above.

Here’s the sysv-rc init script template that I currently use:

#!/bin/sh
# The content after this line comes from openstack-pkg-tools
# and has been automatically added to a .init.in script, which
# contains only the descriptive part for the daemon. Everything
# else is standardized as a single unique script.

# Author: Thomas Goirand <zigo@debian.org>

# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin

if [ -z "${DAEMON}" ] ; then
	DAEMON=/usr/bin/${NAME}
fi
PIDFILE=/var/run/${PROJECT_NAME}/${NAME}.pid
if [ -z "${SCRIPTNAME}" ] ; then
	SCRIPTNAME=/etc/init.d/${NAME}
fi
if [ -z "${SYSTEM_USER}" ] ; then
	SYSTEM_USER=${PROJECT_NAME}
fi
if [ -z "${SYSTEM_USER}" ] ; then
	SYSTEM_GROUP=${PROJECT_NAME}
fi
if [ "${SYSTEM_USER}" != "root" ] ; then
	STARTDAEMON_CHUID="--chuid ${SYSTEM_USER}:${SYSTEM_GROUP}"
fi
if [ -z "${CONFIG_FILE}" ] ; then
	CONFIG_FILE=/etc/${PROJECT_NAME}/${PROJECT_NAME}.conf
fi
LOGFILE=/var/log/${PROJECT_NAME}/${NAME}.log
if [ -z "${NO_OPENSTACK_CONFIG_FILE_DAEMON_ARG}" ] ; then
	DAEMON_ARGS="${DAEMON_ARGS} --config-file=${CONFIG_FILE}"
fi

# Exit if the package is not installed
[ -x $DAEMON ] || exit 0

# If ran as root, create /var/lock/X, /var/run/X, /var/lib/X and /var/log/X as needed
if [ "x$USER" = "xroot" ] ; then
	for i in lock run log lib ; do
		mkdir -p /var/$i/${PROJECT_NAME}
		chown ${SYSTEM_USER} /var/$i/${PROJECT_NAME}
	done
fi

# This defines init_is_upstart which we use later on (+ more...)
. /lib/lsb/init-functions

# Manage log options: logfile and/or syslog, depending on user's choosing
[ -r /etc/default/openstack ] && . /etc/default/openstack
[ -r /etc/default/$NAME ] && . /etc/default/$NAME
[ "x$USE_SYSLOG" = "xyes" ] && DAEMON_ARGS="$DAEMON_ARGS --use-syslog"
[ "x$USE_LOGFILE" != "xno" ] && DAEMON_ARGS="$DAEMON_ARGS --log-file=$LOGFILE"

do_start() {
	start-stop-daemon --start --quiet --background ${STARTDAEMON_CHUID} --make-pidfile --pidfile ${PIDFILE} --chdir /var/lib/${PROJECT_NAME} --startas $DAEMON \
			--test > /dev/null || return 1
	start-stop-daemon --start --quiet --background ${STARTDAEMON_CHUID} --make-pidfile --pidfile ${PIDFILE} --chdir /var/lib/${PROJECT_NAME} --startas $DAEMON \
			-- $DAEMON_ARGS || return 2
}

do_stop() {
	start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE
	RETVAL=$?
	rm -f $PIDFILE
	return "$RETVAL"
}

do_systemd_start() {
	exec $DAEMON $DAEMON_ARGS
}

case "$1" in
start)
	init_is_upstart > /dev/null 2>&1 && exit 1
	log_daemon_msg "Starting $DESC" "$NAME"
	do_start
	case $? in
		0|1) log_end_msg 0 ;;
		2) log_end_msg 1 ;;
	esac
;;
stop)
	init_is_upstart > /dev/null 2>&1 && exit 0
	log_daemon_msg "Stopping $DESC" "$NAME"
	do_stop
	case $? in
		0|1) log_end_msg 0 ;;
		2) log_end_msg 1 ;;
	esac
;;
status)
	status_of_proc "$DAEMON" "$NAME" && exit 0 || exit $?
;;
systemd-start)
	do_systemd_start
;;  
restart|force-reload)
	init_is_upstart > /dev/null 2>&1 && exit 1
	log_daemon_msg "Restarting $DESC" "$NAME"
	do_stop
	case $? in
	0|1)
		do_start
		case $? in
			0) log_end_msg 0 ;;
			1) log_end_msg 1 ;; # Old process is still running
			*) log_end_msg 1 ;; # Failed to start
		esac
	;;
	*) log_end_msg 1 ;; # Failed to stop
	esac
;;
*)
	echo "Usage: $SCRIPTNAME {start|stop|status|restart|force-reload|systemd-start}" >&2
	exit 3
;;
esac

exit 0

Nothing particularly fancy here… You’ll noticed that it’s really OpenStack centric (see the LOGFILE and CONFIGFILE things…). You may have also noticed the call to “init_is_upstart” which is needed for upstart support. I’m not sure if it’s at the correct place in the init script. Should I put that on top of the script? Was I right with the exit values for it? Please send me your comments…

Then I thought about generalizing all of this. Because not only the sysv-rc scripts needed to be squared-up, but also Upstart. The approach here was to source the sysv-rc script in debian/*.init.in, and then generate the Upstart job accordingly, using the above 3 variables (or more as needed). Here, the fun is that, instead of taking the approach of calculating everything at runtime with the sysv-rc, for Upstart jobs, many things are calculated at build time. For each debian/*.init.in script that the debian/rules finds, pkgos-gen-upstart-job is called. Here’s pkgos-gen-upstart-job:

#!/bin/sh

INIT_TEMPLATE=${1}
UPSTART_FILE=`echo ${INIT_TEMPLATE} | sed 's/.init.in/.upstart/'`

# Get the variables defined in the init template
. ${INIT_TEMPLATE}

## Find out what should go in After=
#SHOULD_START=`cat ${INIT_TEMPLATE} | grep "# Should-Start:" | sed 's/# Should-Start://'`
#
#if [ -n "${SHOULD_START}" ] ; then
#	AFTER="After="
#	for i in ${SHOULD_START} ; do
#		AFTER="${AFTER}${i}.service "
#	done
#fi

if [ -z "${DAEMON}" ] ; then
        DAEMON=/usr/bin/${NAME}
fi
PIDFILE=/var/run/${PROJECT_NAME}/${NAME}.pid
if [ -z "${SCRIPTNAME}" ] ; then
	SCRIPTNAME=/etc/init.d/${NAME}
fi
if [ -z "${SYSTEM_USER}" ] ; then
	SYSTEM_USER=${PROJECT_NAME}
fi
if [ -z "${SYSTEM_GROUP}" ] ; then
	SYSTEM_GROUP=${PROJECT_NAME}
fi
if [ "${SYSTEM_USER}" != "root" ] ; then
	STARTDAEMON_CHUID="--chuid ${SYSTEM_USER}:${SYSTEM_GROUP}"
fi
if [ -z "${CONFIG_FILE}" ] ; then
	CONFIG_FILE=/etc/${PROJECT_NAME}/${PROJECT_NAME}.conf
fi
LOGFILE=/var/log/${PROJECT_NAME}/${NAME}.log
DAEMON_ARGS="${DAEMON_ARGS} --config-file=${CONFIG_FILE}"

echo "description \"${DESC}\"
author \"Thomas Goirand <zigo@debian.org>\"

start on runlevel [2345]
stop on runlevel [!2345]

chdir /var/run

pre-start script
	for i in lock run log lib ; do
		mkdir -p /var/\$i/${PROJECT_NAME}
		chown ${SYSTEM_USER} /var/\$i/${PROJECT_NAME}
	done
end script

script
	[ -x \"${DAEMON}\" ] || exit 0
	DAEMON_ARGS=\"${DAEMON_ARGS}\"
	[ -r /etc/default/openstack ] && . /etc/default/openstack
	[ -r /etc/default/\$UPSTART_JOB ] && . /etc/default/\$UPSTART_JOB
	[ \"x\$USE_SYSLOG\" = \"xyes\" ] && DAEMON_ARGS=\"\$DAEMON_ARGS --use-syslog\"
	[ \"x\$USE_LOGFILE\" != \"xno\" ] && DAEMON_ARGS=\"\$DAEMON_ARGS --log-file=${LOGFILE}\"

	exec start-stop-daemon --start --chdir /var/lib/${PROJECT_NAME} \\
		${STARTDAEMON_CHUID} --make-pidfile --pidfile ${PIDFILE} \\
		--exec ${DAEMON} -- --config-file=${CONFIG_FILE} \${DAEMON_ARGS}
end script
" >${UPSTART_FILE}

The only thing which I don’t know how to do, is how to implement the Should-Start / Should-Stop in an Upstart job. Can anyone shoot me a mail and tell me the solution?

Then, I wanted to add support for systemd. Here, we cheated, since we only just called the sysv-rc script from the systemd unit, however, the systemd-start target uses exec, so the process stays in the foreground. It’s also much smaller than the Upstart thing. However, here, I could implement the “After” thing, corresponding to the Should-Start:

#!/bin/sh

INIT_TEMPLATE=${1}
SERVICE_FILE=`echo ${INIT_TEMPLATE} | sed 's/.init.in/.service/'`

# Get the variables defined in the init template
. ${INIT_TEMPLATE}

if [ -z "${SCRIPTNAME}" ] ; then
	SCRIPTNAME=/etc/init.d/${NAME}
fi
if [ -z "${SYSTEM_USER}" ] ; then
	SYSTEM_USER=${PROJECT_NAME}
fi
if [ -z "${SYSTEM_GROUP}" ] ; then
	SYSTEM_GROUP=${PROJECT_NAME}
fi

# Find out what should go in After=
SHOULD_START=`cat ${INIT_TEMPLATE} | grep "# Should-Start:" | sed 's/# Should-Start://'`

if [ -n "${SHOULD_START}" ] ; then
	AFTER="After="
	for i in ${SHOULD_START} ; do
		AFTER="${AFTER}${i}.service "
	done
fi

echo "[Unit]
Description=${DESC}
$AFTER

[Service]
User=${SYSTEM_USER}
Group=${SYSTEM_GROUP}
WorkingDirectory=/var/lib/${PROJECT_NAME}
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p /var/lock/${PROJECT_NAME} /var/log/${PROJECT_NAME} /var/lib/${PROJECT_NAME}
ExecStartPre=/bin/chown ${SYSTEM_USER}:${SYSTEM_GROUP} /var/lock/${PROJECT_NAME} /var/log/${PROJECT_NAME} /var/lib/${PROJECT_NAME}
ExecStart=${SCRIPTNAME} systemd-start
Restart=on-failure

[Install]
WantedBy=multi-user.target
" >${SERVICE_FILE}

As you can see, it’s calling /etc/init.d/${SCRIPTNAME} sytemd-start, which isn’t great. I’d be happy to have comments from systemd user / maintainers on how to fix it to make it better.

Integrating in debian/rules

To integrate with the Debian package build system, we only need had to write this:

override_dh_installinit:
	# Create the init scripts from the template
	for i in `ls -1 debian/*.init.in` ; do \
		MYINIT=`echo $$i | sed s/.init.in//` ; \
		cp $$i $$MYINIT.init ; \
		cat /usr/share/openstack-pkg-tools/init-script-template >>$$MYINIT.init ; \
		pkgos-gen-systemd-unit $$i ; \
	done
	# If there's an upstart.in file, use that one instead of the generated one
	for i in `ls -1 debian/*.upstart.in` ; do \
		MYPKG=`echo $$i | sed s/.upstart.in//` ; \
		cp $$MYPKG.upstart.in $$MYPKG.upstart ; \
	done
	# Generate the upstart job if there's no already existing .upstart.in
	for i in `ls debian/*.init.in` ; do \
		MYINIT=`echo $$i | sed s/.init.in/.upstart.in/` ; \
		if ! [ -e $$MYINIT ] ; then \
			pkgos-gen-upstart-job $$i ; \
		fi \
	done
	dh_installinit --error-handler=true
	# Generate the systemd unit file
	# Note: because dh_systemd_enable is called by the
	# dh sequencer *before* dh_installinit, we have
	# to process it manually.
	for i in `ls debian/*.init.in` ; do \
		pkgos-gen-systemd-unit $$i ; \
		MYSERVICE=`echo $$i | sed 's/debian\///'` ; \
		MYSERVICE=`echo $$MYSERVICE | sed 's/.init.in/.service/'` ; \
		dh_systemd_enable $$MYSERVICE ; \
	done

As you can see, it’s possible to use a debian/*.upstart.in and not use the templating system, in the more complicated case (I needed it mostly for neutron-server and neutron-plugin-openvswitch-agent).

Conclusion

I do not pretend that what I wrote in the openstack-pkg-tools is the ultimate solution. But I’m convince that it answers our own need as the OpenStack maintainers in Debian. There’s a lot of room for improvements (like implementing the Should-Start in Upstart jobs, or stop calling the sysv-rc script in the systemd units), but that this is a very good move that we did to use templates and generated scripts, as the init scripts are a way more easy to maintain now, in a much more unified way. As I’m not completely satisfied for the systemd and Upstart implementation, I’m sure that there’s already a huge improvements on the sysv-rc script maintainability.

Last and again: please send your comments and help improving the above! :)

Rotten tomatoes

There’s many ways to interpret the last GR. The way I see it is how Joey hoped Debian was: the outcome of the poll shows that we don’t want to do technical decisions by voting. At the beginning of this GR, I was supportive of it, and though it was a good thing to enforce the rule that we care for non-systemd setups. Though I have slowly changed my mind. I still think it was a good idea to see what the community thought after a so long debate. I now think that this final outcome is awesome and couldn’t have been better. Science (and computer science) has never been about voting, otherwise the earth would be flat, without drifting continents.

So my hope is that the Debian project as a whole, will allow itself to do mistakes, iterative trials, errors, and go back on any technical decision if they don’t make sense anymore. When being asked something, it’s ok to reply: “I don’t know”, and it should be ok for the Debian project to have this alternative as one of the possible answers. I’m convince that refusing to take a drastic choice in this point in time was exactly what we needed to do. And my hope is that Joey comes back after he realizes that we’ve all understood and embarrassed his position that science cannot be governed by polls.

For Stretch, I’m sure there’s going to be a lot of new alternatives. Maybe uselessd, eudev and others. Maybe I’ll have a bit of time to work on OpenRC Debian integration myself (hum… I’m dreaming here…). Maybe something else. Let’s just wait. We have more than 300 bugs to fix before Jessie can be released. Let’s happilly work on that together, and forget about the init systems for a while…

P.S: Just to be on the safe side: the rotten tomatoes image was not about criticizing the persons who started the poll, who I respect a lot, especially Ian, who I am convinced is trying to do his best for Debian (hug).

OpenStack packaging activity: October 2014

Wednesday 1:
– Uploaded python-xstatic-jquery removing the .pth file from package.
– Uploaded python-taskflow 0.4 to experimental, needed by Cinder Juno RC1
– Uploaded Cinder Juno RC1 to experimental

Thuesday 2:
– Finally understood that the issue with murano-dashboard was that it doesn’t build without django-nose >= 1.2. Opened new patch at: https://review.openstack.org/125565
– Uploaded murano-dashboard to Experimental, now using django-nose from wheezy-backports in my jenkins setup, so murano-dashboard can be built for Wheezy.
– Uploaded python-oslotest 1.1.0.0 (really is upstream 1.1.0)
– Uploaded python-oslo.serialization 1.0.0-1 (needed by Ceilometer Juno RC1)
– Uploaded Ceilometer Juno RC1
– Uploaded Heat Juno RC1
– Uploaded oslo.rootwrap 1.3.0.0
– Uploaded oslo.db 1.0.2 (bugfix release)
– Wrote a new system in openstack-pkg-tools to generate init scripts and. service files from a template, so we don’t have to write N times the same thing.

Friday 3:
– Reworked openstack-pkg-tools to generate automatically sysv-rc init scripts, upstart jobs and systemd unit files, making the system more unified and consistent.
– Applied the new system to all packages in Juno.
– Uploaded Keystone 2014.1.3-1 to Sid
– Uploaded Nova 2014.1.3-1 to Sid
– Uploaded Glance 2014.1.3-1 to Sid
– Uploaded Neutron 2014.1.3-1 to Sid
– Uploaded Horizon 2014.1.3-1 to Sid
– Uploaded Cinder 2014.1.3-1 to Sid
– Uploaded Trove 2014.1.3-1 to Sid
– Uploaded Ceilometer 2014.1.3-1 to Sid

Saturday 4:
– Uploaded Horizon Juno RC1 to Experimental
– Uploaded oslotest 1.1.0.0 to Experimental
– Uploaded Ironic Juno RC1 to Experimental
– Uploaded Designate Juno RC1 to Experimental
– Uploaded Nova Juno RC1 to Experimental
– Uploaded Neutron Juno RC1 to Experimental
– Uploaded openstack-meta-packages 0.10 to Sid
– Uploaded openstack-pkg-tools 13 to Experimental
– Uploaded murano-agent Juno RC1 to Experimental

Sunday 5:
– Uploaded Sahara Juno RC1 to Experimental (it’s been approved by FTP masters)
– Uploaded Murano Juno RC1 to Experimental (it’s been approved by FTP masters)
– Fixed all debian/watch file to understand ~b and ~rc releases (fixed applied on both Icehouse and Juno branches, though no upload yet, I’ll wait until uploads are needed to have this in the archive…).
– Uploaded Trove Juno RC1 to Experimental
– Uploaded Sahara Juno RC1 to Experimental

With this last upload, everything of Juno RC1 is in Debian Experimental! \o/

Monday 6:
– Uploaded some fixes for Nova 2014.1.3-2 in Sid:
* Removed contrib/boto_v6/* in debian/copyright, replaced bin/nova-manage by nova/cmd/{baremetal_,}manage.py.
* Mangling upstream rc and beta versions in watch file.
* Added 9990_update_german_programm_messages.patch, thanks to Helge Kreutzmann <debian@helgefjell.de>.
* Fixed correct de.po (Closes: #763682).
* Added nl.po initial Debconf translation, thanks to Frans Spiesschaert <Frans.Spiesschaert@yucom.be> (Closes: #764125).
* Standards-Version is now 3.9.6 (no change).
Upstreamed german translation of po file: https://review.openstack.org/126212
– Uploaded Designate 2014.1-12 to Sid, added new de.po also to the Juno branch on alioth (but didn’t upload the fix yet).
– Uploaded sphinxcontrib-httpdomain new upstream 1.3.0 release, added Python 3.x support to the package, and transitionning to the correct namespaced python-sphinxcontrib.httpdomain package name.
– Spent most of the day fixing python-xstatic issues:
o uploaded libjs-twitter-bootstrap-datepicker 1.3.1
o uploaded python-xstatic-bootstrap-datepicker requiring this libjs package
o fixed python-xstatic-jquery-ui package
Now Horizon Juno RC1 builds well, and can be installed again. \o/

Tuesday 7:
– Backported python-libvirt 1.2.8 in Wheezy (for Nova Juno support…)
– Uploaded Ceilometer Juno RC1 with ceilometer-agent-ipmi added (the package will therefore go through the NEW queue).
– Uploaded python-requestbuilder 0.2.2-1, needed by the maintainers of euca2ools.
– Ported the unified generated init system scripts to Icehouse packages.
– Uploaded to Sid updates for: openstack-pkg-tools, ceilometer, cinder, glance, keystone, cinder, nova.

Wednesday 8:
– Uploaded openstack-pkg-tools 16 to Sid
– Uploaded murano-dashboard (with upstream fix to remove font-awesome, which was the reason for FTP master’s rejection)
– Uploaded ceilometer Juno RC1 with new IPMI agent package (needed for Ironic support).
– Uploaded heat 2014.1.3 which I forgot.
– Tested https://review.openstack.org/#/c/126777/ which solves the bug I sent to launchpad and approved the patch.
– Uploaded python-requestbuilder 0.2.3

Thesday 9:
– Worked on fixing Neutron Alembic migration with SQLite3.
– Uploade Neutron 2014.2~rc1-3 with a fix for a patch that was destroying dhcp.py. This still doesn’t include the Alembic migration fixes, which are still a WIP.

Firday 10:
– Finished fixing Neutron SQLite 3 Alembic migrations.
– Uploaded neutron 2014.2~rc1-3 with the fixes.
– Fixed Ceilometer wrong generation of sample config file, using upstream patch (after discussing with Julien Danjou so he wrote it).
– Uploaded Ceilometer 2014.2~rc1-4 with the fix
– Checked that all packages can be installed in non-interactive mode. This works well now! \o/

Saturday 11:
– Uploaded new version of python-xstatic-angular-cookies (ie: 1.2.24.1-2) which allows a higher version of libjs-angularjs (otherwise the package is not installable in Sid/Jessie since last version of angularjs is uploaded).

Sunday 12:
– Uploaded factory-boy fix for FTBFS
– Uploaded python-django-appconf FTBFS
– Uploaded Horizon Juno RC2
– Uploaded Heat Juno RC3
– Uploaded Trove Juno RC2
– Uploaded Glance Juno RC2
– Uploaded Sahara Juno RC2
– Uploaded Nova Juno RC2
– Uploaded Neutron Juno RC2
– Uploaded Cinder Juno RC2
– Uploaded murano-dashboard Juno RC2

Monday 13:
– Uploaded python-heatclient 0.2.12-1 to Experimental
– Uploaded python-yaql with RC bugfix to Sid (missing dep on python3-ply).

Thuesday 14:
– Fixed arping newly added dependency in Neutron
– Started testing install of all of openstack Juno at once

Wednesday 15:
– Fixed missing configuration files in Ceilometer (ceilometer-api couldn’t start)
– Upgraded to Ceilometer Juno RC3.
– Backported python-setuptools, as keystone and others are broken due to the namespace of modules not working correctly with the old version of python-pkg-resources. With the new one, everything is back in order.

Thesday 16:
Uploaded to Debian Experimental the final release of Juno (ie: 2014.2) for:
– Sahara
– Nova
– Ceilometer
– Cinder
– Heat
– Neutron
– Glance
– Keystone
– Horizon (with fix for Django 1.7 in the wsgi file)
Uploaded to Sid:
– Swift 2.2.0
– Horizon 2014.1.3-3 with fix for Django 1.7 in the wsgi file that was crashing apache.

OpenStack Juno packages are out!!! (ready the day of the upstream release…)

Friday 17:
– Investigated Trove RC bug #765348, couldn’t reproduce, and therefore closed it.
– Uploaded Ironic Juno final to Experimental
– Uploaded Designate Juno final to Experimental
– Uploaded a fix for python-jingo which failed to build with Django 1.7. Sent pull request upstream: https://github.com/jbalogh/jingo/pull/63
– Uploaded CVE-2014-7230 & CVE-2014-7231 fixes for both Cinder and Nova in Debian Sid, as per OSSA 2014-036 patches. No need to upload a fix for Trove, as 2014.1.3 already has the fixes.

Saturday 18:
– Started building Trusty packages
– Fixed oslo-config so that it never depends on python3-argparse, which doesn’t exist (uploaded to Experimental)
– Uploaded python-django-pyscss 1.0.3-2 with python-simplejson now as build-depends (it failed to build in my Trusty jenkins without it).
– Uploaded a fix for stevedore and oslo-config to not depends on python3-argparse in Ubuntu (added debian/py3dist-overrides)

Sunday 19:
– Uploaded python-taskflow with ordereddict in debian/pydist-overrides.
– Backported JS packages for Horizon and libvirt for Trusty (from Sid). My new Jenkin server is now producing a full set of Juno packages for Ubuntu trusty. And of course, it’s updated on each git push, just like for the Wheezy backports.

Monday 20:
– Added FORCE_COULEUR=1 when running tests in python-couleur, so that it doesn’t fail when running with git-buildpackage. Uploaded result in Sid.
– Fixed python-mockito so that it never downloads distribute or nose on its clean target, which was annoying when running git-buildpackage. Uploaded to Sid.
– Started to work again on automatic package deployment using openstack-deploy, from the openstack-meta-packages source package.

Thuesday 21, Wednesday 22:
– Worked on testing packages, did couples of minor fixes, reworked some of the default configuration files to match the install-guide, move configuration directive to the correct new section in nova.conf, etc.

Thursday 23:
– Patch the Neutron chapter in the install-guide to take into account the changes done on Thuesday 21, Wednesday 22, and simplify the install procedure in Debian. https://review.openstack.org/#/c/130501/

Friday 24:
– Busy packing my stuff for moving to France… Not much packaging work, except more auto-deploy stuff and some tests.

Saturday 25:
– Uploaded Nova, Neutron, Cinder and Horizon Icehouse in Sid, including some debconf translation updates, beating the Jessie freeze deadline in 10 days.
– Fixed and uploaded openstack-debian-images in Sid: the –login option wasn’t modifying the default sudoers file, which always contained “debian”, instead of the custom login.

Sunday 26:
– Traveled to Moscow

Monday 27 & Tuesday 28:
– Fixed some murano & murano-dashboard stuff, thanks to the help of some murano team members in Moscow office. Uploaded fixes for murano & murano-dashboard. Tested that murano-dashboard works well, and now it does! :)
– Uploaded version dependency fixes for python-xstatic-angular-cookies and python-xstatic-d3 which couldn’t be installed in Sid/Jessie because of libjs-* updates.

Wednesday 29:
– Meeting with Saratov team
– Updated sahara endpoints, but didn’t upload the package yet to Debian.

Thursday 30:
– Uploaded ruby-raemon needed for Astute (part of Fuel web).
– Packaged ruby-symboltable (not uploaded yet).

Friday 31:
– Wrote unit test runner for python-webpy (the current package doesn’t have unit test runs).
– Uploaded python-dbutils (needed by python-web.py unit tests) to Sid: now in NEW queue
– Uploaded python-nose-parametrized & python-nose-timer to Sid: now in NEW queue
– Uploaded sahara -2 fixing the API endpoint registration URL and service name.
– Uploaded python-sphinxcontrib.plantuml to Sid: : now in NEW queue

 

Working with Mirantis from now on

During 2 years, my packaging efforts were sponsored by eNovance. It has been great to work with them, and I would like to thank them for what they did.

However (for a number of reasons which I don’t really think is appropriate to write in this blog), I decided to join Mirantis. This is a formal public announce of it: I will be working from Mirantis office in Grenoble just right after the Paris OpenStack summit of this first week of November.

OpenStack Juno is out, Debian (and Ubuntu Trusty ports) packages ready

This is just a quick announce: Debian packages for Juno are out. In fact, they were ready the day of the release, on the 16th of October. I uploaded it all (to Experimental) the same day, literally a few hours after the final released was git tagged. But I had no time to announce it.

This week-end, I took the time to do an Ubuntu Trusty port, which I also publish (it’s just a mater of rebuilding all, and it should work out of the box). Here are the backports repositories. For Wheezy:

deb http://archive.gplhost.com/debian juno-backports main

deb http://archive.gplhost.com/debian juno main

For trusty:

deb http://archive.gplhost.com/debian trusty-juno-backports main

But of course, everything is also available directly in Debian. Since Sid/Jessie contains OpenStack Icehouse (which has more chance to receive long enough security support), and it will be like this until Jessie is released. So I have uploaded all of Juno into Debian Experimental. This shows on the OpenStack qa page (you may also notice that the team is nearly reaching 200 packages… though am planning to off-load some of that to the Python module team, when the migration to Git will be finished). On the QA page, you may also see that I uploaded all of the last Icehouse point release to Sid, and that all packages migrated to Jessie. There’s only a few minor issues with some Python modules which I fixed, that haven’t migrated to Jessie yet.

I can already tell that all packages can be installed without an issue, and that I know Horizon at least works as expected. But I didn’t have time to test it all just yet. I’m currently working on doing even more installation automation at the package level (by providing some OVS bridging init script and such, to make it more easy to run Tempest functional testing). I’ll post more about this when it’s ready.

OpenStack packaging activity: September 2014

I decided I’d post this monthly. It may be a bit boring, sorry, but I think it’s a nice thing to have this public. The log starts on the 6th, because on the 4th I was back from Debconf (after a day in San Francisco, plus 20 hours of traveling and 15 hours of time gap). It is to be noted that every time something is uploaded in Debian for Icehouse (in Sid), or for Juno (in Experimental), there’s also a corresponding backport produced for Wheezy.

 

Saturday 6th & Sunday 7th:
– packaged libjs-twitter-bootstrap-wizard (in new queue)
– Uploaded python-pint after reviewing the debian/copyright
– Worked on updating python-eventlet in Experimental, and adding Python3 support. It seems Python3 support isn’t ready yet, so I will probably remove that feature from the package update.
– Tried to apply the Django 1.7 patches for python-django-bootstrap-form. They didn’t work, but Raphael came back on Monday morning with new versions
of the patches, which should be good this time.
– Helped the DSA (Debian System Administrators) with the Debian OpenStack cloud. It’s looking good and working now (note: I helped them during Debconf 14).
– Started a page about adding more tasksel tasks: https://wiki.debian.org/tasksel/MoreTasks. It’s looking like Joey Hess is adding new tasks by default in Tasksel, with “OpenStack compute node” and “OpenStack proxy node”. It will be nice to have them in the default Debian installer! :)
– Packaged and uploaded python-dib-utils, now in NEW queue.

Monday 8th:
– Uploaded fixed python-django-bootstrap-form with patch for Django 1.7.
– Packaged and uploaded python-pysaml2.
– Finilized and uploaded python-jingo which is needed for python-django-compressor unit tests
– Finalized and uploaded python-coffin which is needed for python-django-compressor unit tests
– Worked on running the unit tests for python-django-compressor, as I needed to know if it could work with Django 1.7. It was hard to find the correct way to run the unit tests, but finally, they all passed. I will add the unit tests once coffin and jingo will be accepted in Sid.
– Applied patches in the Debian BTS for python-django-openstack-auth and Django 1.7. Uploaded the fixed package.
– Fixed python-django-pyscss compat with Django 1.7, uploaded the result.
– Updated keystone to Juno b3.
– Built Wheezy backports of some JS libs needed for Horizon in Juno, which I already uploaded to Sid last summer:
o libjs-twitter-bootstrap-datepicker
o libjs-jquery.quicksearch
o libjs-spin.js
– Upstreamed the Django 1.7 patch for python-django-openstack-auth:
https://review.openstack.org/119972

Tuesday 9:
– Updated and uploaded Swift 2.1.0. Added swift-object-expirer package to it, together with init script.

Wednesday 10:
Basically, cleaned the Debian BTS of almost all issues today… :P
– Added it.po update to nova (Closes: #758305).
– Backported libvirt 1.2.7 to Wheezy, to be able to close this bug: https://bugs.debian.org/757548 (eg: changed dependency from libvirt-bin to libvirt-daemon-system)
– Uploaded the fixed nova package using libvirt-daemon-system
– Upgraded python-trollius to 1.0.1
– Fixed tuskar-ui to work with Django 1.7. Disabled pep8 tests during build. Added build-conflicts: python-unittest2.
– Fixed python-django-compressor for Django 1.7, and now running unit tests with it, after python-coffin and python-jingo got approved in Sid by FTP masters.
– Fixed python-xstatic wrong upstream URLs.
– Added it.po debconf translation to Designate.
– Added de.po debconf translation to Tuskar.
– Fixed copyright holders in python-xstatic-rickshaw
– Added python-passlib as dependency for python-cinder.

Remaining 3 issues in the BTS: ceilometer FTBFS, Horizon unit test with Django 1.7, Designate fail to install. All of the 3 are harder to fix, and I may try to do so later this week.

Thursday 11:
– Fixed python-xstatic-angular and python-xstatic-angular-mock to deal with the new libjs-angularjs version (closes 2 Debian RC bugs: uninstallable).
– Fixed ceilometer FTBFS (Closes rc bug)

Friday 12:
– Fixed wrong copyright file for libjs-twitter-bootstrap-wizard after the FTP masters told me, and reuploaded to Sid.
– Reuploaded wrong upload of ceilometer (wrong hash for orig.tar.xz)
– Packaged and uploaded python-xstatic-bootstrap-scss
– Packaged and uploaded python-xstatic-font-awesome
– Packaged and uploaded ntpstat

Monday 15:
– packaged and uploaded python-xstatic-jquery.bootstrap.wizard
– Fixed python-xstatic-angular-cookies to use new libjs-angularjs version (fixed version dependencies)
– Fixed Ceilometer FTBFS (Closes: #759967)
– Backported all python-xtatic packages to Wheezy, including all dependencies. This includes backporting of a bunch of packages from nodejs which were needed as build-dependencies (around 70 packages…). Filed about 5 or 6 release critical bugs as some nodejs packages were not buildable as-is.
– Fixed some too restrictive python-xstatic-angular* dependencies on the libjs-angularjs (the libjs-angularjs increased version).

Tuesday 16:
– Uploaded updates to Experimental:
o python-eventlet 0.15.2 (this one took a long time as it needed maintenance)
o oslo-config
o python-oslo.i18n
– Uploaded to Sid:
o python-diskimage-builder 0.1.30-1
o python-django-pyscss 1.0.2-1
– Fixed horizon libapache-mode-wsgi to be a dependency of openstack-dashboard-apache and not just openstack-dashboard (in both Icehouse & Juno).
– Removed the last failing Django 1.7 unit test from Horizon. It doesn’t seem relevant anyway.
– Backported python-netaddr 0.7.12 to Wheezy (needed by oslo-config).
– Started working on oslo.rootwrap, though it failed to build in Wheezy with a unit test failure.

Wednesday 17:
– To experimental:
o Uploaded oslo.rootwrap 1.3.0.0~a1. It needed a build-depends on iproute2 because of a new test.
o Uploaded python-oslo.utils 0.3.0
o Uploaded python-oslo.vmware 0.6.0, fixed sphinx-build conf.py and filed a bug about it: https://bugs.launchpad.net/oslo.vmware/+bug/1370370 plus emailed the commiter of the issue (which appeared 2 weeks ago).
o Uploaded python-pycadf 0.6.0
o Uploaded python-pyghmi 0.6.17
o Uploaded python-oslotest 1.1.0.0~a2, including patch for Wheezy, which I also submited upstream: https://review.openstack.org/122171/
o Uploaded glanceclient 0.14.0, added a patch to not use the embedded version of urllib3 in requests: https://review.openstack.org/122184
– To Sid:
o Uploaded python-zake_0.1.6-1

Thesday 18:
– Backported zeromq3-4.0.4+dfs, pyzmq-14.3.1, pyasn1-0.1.7, python-pyasn1-modules-0.0.5
– Uploaded keystoneclient 0.10.1, fixed the saml2 unit tests which were broken using testtools >= 0.9.39. Filed bug, and warned code author: https://bugs.launchpad.net/python-keystoneclient/+bug/1371085
– Uploade swiftclient 2.3.0 to experimental.
– Uploaded ironicclient 0.2.1 to experimental.
– Uploaded saharaclient, filed bug with saharaclient expecting an up and running keystone server: https://bugs.launchpad.net/python-saharaclient/+bug/1371177

Friday 19:
– Uploaded keystone Juno b3, filed but about unit tests downloading with git, while no network access should be performed during package build (forbidden by
Debian policy)
– Uploaded python-oslo.db 1.0.0 which I forgot in the dependency list, and which was needed for Neutron.
– Uploaded nova 2014.2~b3-1 (added a new nova-serialproxy service daemon to the nova-consoleproxy)

Saturday 20:
– Uploaded Neutron Juno b3.
– Uploaded python-retrying 1.2.3 (was missing from depends upload)
– Uploaded Glance Juno b3.
– Uploaded Cinder Juno b3.
– Fixed python-xstatic-angular-mock which had a .pth packaged, as well as the data folder (uploaded debian release -3).
– Fixed missing depends and build-conflicts in python-xstatic-jquery.

Sunday 21:
– Dropped python-pil & python-django-discover-runner from runtime Depends: of python-django-pyscss, as it’s only needed for tests. It also created a conflicts, because python-django-discover-runner depends on python-unittest2 and horizon build-conflicts with it.
– Forward-ported the Django 1.7 patches for Horizon. Opened new patch: https://review.openstack.org/122992 (since the old fix has gone away after a refactor of the unit test).
– Uploaded Horizon Juno b3.
– Applied https://review.openstack.org/#/c/122768/ to the keystone package, so that it doesn’t do “git clone” of the keystoneclient during build.
– Uploaded oslo.messaging 1.4.0.0 (which really is 1.4.0) to experimental
– Uploaded oslo.messaging 1.4.0.0+really+1.3.1-1 to fix the issue in Sid/Jessie after the wrong upload (due to Zul wrong tagging of Keystone in the 2014.1.2 point release).

Monday 22:
– Uploaded ironic 2014.2~b3-1 to experimental
– Uploaded heat 2014.2~b3-1 (with some fixes for sphinx doc build)
– Uploaded ceilometer 2014.2~b3-1 to experimental
– Uploaded openstack-doc-tools 0.19-1 to experimental
– Uploaded openstack-trove 2014.2~b3-1 to experimental

Tuesday 23:
– Uploaded python-neutronclient with fixed version number for cliff and six. This missing requirement for cliff version produced an error in Trove, which I don’t want to happen again.
– Added fix for unit tests in Trove: https://review.openstack.org/#/c/123450/1,publish
– Uploaded oslo.messaging 1.4.1 in Experimental, fixing the version conflicts with the one in Sid/Jessie. Thanks to Doug Hellman for doing the tagging. I will need to upload new versions of the following packages with the >= 1.4.1 depends:
> – ceilometer
> – ironic
> – keystone
> – neutron
> – nova
> – oslo-config
> – oslo.rootwrap
> – oslo.i18n
> – python-pycadf
See http://lists.openstack.org/pipermail/openstack-dev/2014-September/046795.html for more explanation about the mess I’m repairing…
– Uploaded designate Juno b3.

Wednesday 24:
– Uploaded oslosphinx 2.2.0.0
– Uploaded update to django-openstack-auth (new last minute requirement for Horizon).
– Uploaded final oslo-config package version 1.4.0.0 (really is 1.4.0)
– Packaged and uploaded Sahara. This needs some tests by someone else as I don’t even know how it works.

Thuesday 25:
– Uploaded python-keystonemiddleware 1.0.0-3, fixing CVE-2014-7144] TLS cert verification option not honoured in paste configs. https://bugs.debian.org/762748
– Packaged and uploaded python-yaql, sent pull request for fixing print statements into Python3 compatible print function calls: https://github.com/ativelkov/yaql/pull/15
– Packaged and uploaded python-muranoclient.
– Started the packaging of Murano (not finished yet).
– Uploaded python-keystoneclient 0.10.1-2 with the CVE-2014-7144 fix to Sid, with urgency=high. Uploaded 0.11.1-1 to Experimental.
– Uploaded python-keystonemiddleware fix for CVE-2014-7144.
– Uploaded openstack-trove 2014.2~b3-3 with last unit test fix from https://review.openstack.org/#/c/123450/

Friday 26:
– Uploaded a fix for murano-agent, which makes it run as root.
– Finished the packaging of Murano
– Started packaging murano-dashboard, sent this patch to fix the wrong usage of the /usr/bin/coverage command: https://review.openstack.org/124444
– Fixed wrong BASE_DIR in python-xstatic-angular and python-xstatic-angular-mock

Saturday 27:
– uploaded python-xstatic-boostrap-scss which I forgot to upload… :(
– uploaded python-pyscss 1.2.1

Sunday 28:
– After a long investigation, I found out that the issue when installing the openstack-dasboard package was due to a wrong patch I did for Python 3.2 in Wheezy in python-pyscss. Corrected the patch from version 1.2.1-1, and uploaded version 1.2.1-2, the dashboard now installs correctly. \o/
– Did a new version of an Horizon patch at https://review.openstack.org/122992/ to address Django 1.7 compat.

Monday 29:
– Uploaded new version of python-pyscss fixing the last issue with Python 3 (there was a release critical bug on it).
– Uploaded fixup for python-django-openstack-auth fail to build in the Sid version, which was broken since the last upload of keystoneclient (which makes some of its API now as private).
– Uploaded python-glance-store 0.1.8, including Ubuntu patch to fix unit tests.
– Reviewed the packaging of python-strict-rfc3339 (see https://bugs.debian.org761152).
– Uploaded Sheepdog with fix in the init script to start after corosync (Closes: #759216).
– Uploaded pt_BR.po Brazilian Portuguese debconf templates translation for nova Icehouse in Sid (only commited it in Git for Juno).
– Same for Glance.

Tuesday 30:
– Added Python3 support in python-django-appconf, uploaded to Sid
– Upgraded to python-django-pyscss 1.0.3, and fixed broken unit tests with this new release under Django 1.7. Created pull request: https://github.com/fusionbox/django-pyscss/pull/22
– Fixed designate requirements.txt in Sid (Icehouse) to allow SQLA 0.9.x. Uploaded resulting package to Sid.
– Uploaded new Debian fix for python-tooz: kills memcached only if the package scripts started it (plus cleans .testrepository on clean).
– Uploaded initial release of murano
– Uploaded python-retrying with patch from Ubuntu to remove embedded copy of six.py code.
– Uploaded python-oslo.i18n 1.0.0 to experimental (same as before, just bump of version #)
– Uploaded python-oslo.utils 1.0.0 to experimental (same as before, just bump of version #)
– Uploaded Keystone Juno RC1
– Uploaded Glance Juno RC1

Backporting libjs-angularjs and libjs-d3 to Wheezy

If you didn’t notice, Javascript isn’t as simple as it used to be… Want to backport the 2 simple javascript libs? No problem. You then “just” need to backport a bunch of other packages which are build-dependencies… (and file #761670, #761672, and #761674 on the way when rebuilding…). Here’s the short list:

gyp
node-abbrev
node-ansi
node-ansi-color-table
node-archy
node-async
node-block-stream
node-combined-stream
node-contextify
node-cookie-jar
node-cssom
node-delayed-stream
node-diff
node-eyes
node-forever-agent
node-form-data
node-fstream
node-fstream-ignore
node-github-url-from-git
node-glob
node-graceful-fs
node-gyp
node-htmlparser
node-inherits
node-ini
node-jake
node-jsdom
node-json-stringify-safe
node-lockfile
node-lru-cache
node-marked
node-mime
node-minimatch
node-mkdirp
node-mute-stream
node-node-uuid
node-nopt
node-normalize-package-data
node-npmlog
node-once
node-optimist
node-osenv
node-qs
node-queue-async
node-read
node-read-package-json
node-request
node-retry
node-rimraf
node-semver
node-sha
node-sigmund
node-slide
node-smash
node-tar
node-tunnel-agent
node-uglify
node-underscore
node-utilities
node-vows
node-whic
node-which
node-wordwrap
nodejs
npm
ruby-ronn

Yes, that’s 66 packages above… And of course, backporting some ruby stuff makes sense… :)

Debconf 14 activity

Before I start a short listing of (some of) the stuff I did during Debconf 14, I’d like to say how much I enjoyed everyone there. You guys (all of you, really!) are just awesome, and it’s always a real pleasure to see you all, each time.

Anyway, here’s a bits of the stuff I did.

1/ packaging of Google Cloud Engine client tools.

Thanks to the presence of Eric and Jimmy, I was able to finish the work I started at Debconf 13 last year. All python modules are packaged and uploaded. Only the final client (the “gcloud” command line utility) isn’t uploaded, even though it’s already packaged. The reason is that this client downloads “stuff” from internet, so I need to get the full, bundled, version of it, to avoid this. Eric gave me the link, I just didn’t have time to finish it yet. Though the (unfinished) package is already in the Git in Alioth.

2/ Tasksel talks

We discussed improvements in Tasksel both during the conference, and later (in front of beers…). I was able to add a custom task on a modified version of the Tasksel package for my own use. I volunteered myself for adding a “more task” option in Tasksel for Jessie+1 because I really would like to see this feature, and nobody raised hand, but honestly, I have no idea how to do it, and therefore, I’m not sure I’ll be able to do so. We’ll see… Anyway, before this happen, we must make sure that we know what kind of tasks we want in this “more tasks” screen, otherwise it’d be useless work for nothing. Therefore, I have setup a wiki page. Please edit the page and drop your ideas there. I’ve already added entries for desktops and Debian blends, but I’m sure there’s more that we could add.

3/ Custom Debian CD

I started experimentation on building my own Debian Wheezy CD image (well, DVD, since the resulting image is nearly 2GB). This was fun, but I am still having the issue that the installer fails to install Dash, so the CD is still unusable. I’ll try to debug it. Oh…  I nearly forgot… “of course”, the ISO image aims at including all OpenStack Icehouce packages backported to Wheezy, and the goal was to include the above custom Tasksel task, with an “OpenStack proxy node” task, and a “OpenStack compute” task. Let’s hope I can figure out what the issue is, and finally release it.

4/ OpenStack talk

Nothing special to say, just watch the video. I hope my talk was interesting enough. Of course, after watching myself, I hate everything I see, and would like to correct so many mistakes, but that’s the usual, I guess.

5/ Some RC fixing

Thanks to the nice work of our DPL rebuilding all the archive, I had to fix a couple of FTBFS issues on my own packages. 3 of them have been easy to fix (2 missing build-dependencies which I missed because my automated build environment has them by default, and a unit test failure), I still don’t understand what’s going on with Ceilometer. I also NMU-ed transmission (switching from 2.82 to 2.84, as upstream had the bugfix, and current maintainer was not responsive) which was the last blocker for the miniupnpc transition to Jessie. After the 5 days delay of the upload, it went in Sid, then migrated to Jessie, together with the miniupnpc library. I also fixed a trivial RC bug with python3-webob.

6/ Python team meeting

It was nice to see everyone, and hopefully, we’ll soon implement what we discussed. I hope to start migrating some of my OpenStack dependencies to the team once we move to Git (though please don’t expect this to happen before the Juno release, which keeps me very busy these days).

There’s probably more stuff which I did during Debconf 14 (hacking or otherwise), but either it’s not worth sharing, or I can’t remember… :)