Zigo's blog

Just a quick reply to Rhonda about Moby. You can’t introduce him without telling about Go, which is the title who made him famous, very early in the age of electronic music (November 1990, according to wikipedia). Many attempted to remix this song (and Moby himself), but nothing’s as good as the original version.

OpenStack Mitaka beta 1 Debian packages available

I didn’t find the time to announce it until today, though I have finished last Friday to package Mitaka Beta 1. It is available, as usual, on the Jenkins server Debian repository:

deb http://mitaka-jessie.pkgs.mirantis.com/debian jessie-mitaka-backports main
deb-src http://mitaka-jessie.pkgs.mirantis.com/debian jessie-mitaka-backports main
deb http://mitaka-jessie.pkgs.mirantis.com/debian jessie-mitaka-backports-nochange main
deb-src http://mitaka-jessie.pkgs.mirantis.com/debian jessie-mitaka-backports-nochange main

Not all of the updated packages avialable above has been uploaded to Debian Experimental, mostly those needing to pass the FTP master NEW queue did. I will upload the rest as I find enough time to do so, which unfortunately may not happen before Mitaka b2 (which will be in the middle of January).

OpenStack Liberty uploaded to jessie-backports

Also, as python-repoze.who 2.x finally could migrate to Debian testing (after filed to be removed dependencies got removed by the FTP masters), python-pysaml2 3.0, and then Keystone also did. So this week-end, all of OpenStack Liberty reached testing. So today, I could finally upload all of OpenStack liberty in the official jessie-backports repository. This is 165 packages that I uploaded, out of which 135 are going through the backports NEW queue. I’m sorry to give that much work to the FTP masters, but most OpenStack users do want to use the latest release of OpenStack on top of the latest stable distributions. So this upload really is what OpenStack Debian user will prefer (until we have PPA^Wbikesheds for Debian).

There’s a lot of things I’d like to blog about. The last version of OpenStack, the OpenStack Liberty design summit, Kilo in the official jessie-backports repositories, etc. Maybe the most interesting part of this blog post is the last bit at the end, about a major change in the packaging workflow for OpenStack in Debian. Please read on…

OpenStack release names reminder
Just a reminder to make it easier for the average Debian reader who may know Debian well, but not OpenStack.

OpenStack 2014.1, is Icehouse, and is the version in Jessie. 2014.2 is Juno and was released right before the freeze of Jessie. 2015.1.0 is what has been released just right after jessie, on the 30th of April. Liberty, which probably will be called 12 (as this will be the 12th release of OpenStack), and not 2015.2 (this has been discussed in Vancouver), will be released in about 5 months form now.

The last summit, in Vancouver, BC, Canada, was the Liberty summit, as the OpenStack conventions are always named after the next release (since we are discussing what we will be doing during the next development cycle).

OpenStack 2015.1.0, aka Kilo, release in Debian
5 days after the release of Jessie, OpenStack 2015.1.0, aka Kilo, was released. Since I couldn’t upload to unstable during the freeze, I was holding a lot of packages, and when I did upload them, there was about 20 packages of mine in the FTP master’s NEW queue. Though, since the DSA want to use OpenStack for the Debian infrastructure, the 20 packages were fast track into Sid, thanks to the work of Paultag (thanks man!).

OpenStack Kilo in the official Jessie backports
Previously, I was only uploading OpenStack packages to Debian unstable, and maintaining a non-official Debian repositories for backports to Debian stable. However, for multiple reasons, this wasn’t satisfying.

Then, after packages migrated to Stretch, I started to upload to Debian backports. And right before the summit, almost everything went in. Only python-pysaml2 was missing (as I discovered too late that version 2.0.0 breaks Keystone which needs version 2.4.0). In fact, the last bits of the Kilo release reached jessie-backports in the middle of the OpenStack Liberty summit.

Removal of the Debian install-guide from the official site
As there was not enough efforts working on the documentation, unfortunately, the link to the Debian install-guide has been removed from docs.openstack.org. IMO, this is mostly due to a bad communication between myself and the doc team, and also because one person who promised to work on the Debian side of the install-guide failed to warn everyone that he finally couldn’t (as his managers assigned him to something else).

I hope this will soon be reverted. During the Vancouver summit, I had the opportunity to discuss with the doc team about re-inclusion of the Debian install-guide. Unfortunately, as they are moving away from the XML source format to a more standard RST-based system, the current documentation is frozen, so it seems more realistic to hold on until all of the install-guide is switched to RST.

OpenStack Debian image listed on apps.openstack.org
There’s a new area on the openstack.org where images and apps for OpenStack are listed. Under the “glance image” tab, you will see that both the Jessie and the weekly testing image are listed. There’s also a nice, easily identifiable Debian logo to link to these images.

Also, as there are trademark problems with the Ubuntu images which makes them harder to redistribute, the Murano project (which is shipping a system to automatically install apps that to installed within a few clicks on an OpenStack cloud) decided to switch to Debian for their base image.

Debian listed in the OpenStack market place
On the openstack.org site, there’s a section called Marketplace. In there, vendors supporting OpenStack are listed. To get there, a vendor needs to 1/ have a defined set of OpenStack project supported by the distribution (Debian already has a way more than the required set), 2/ sign some kind of agreement with the OpenStack foundation, and 3/ pay some sponsoring money. During the summit, I discussed this with Jonathan Bryce, from the OpenStack foundation, and he agreed that Debian would not have to pay for this (since we aren’t a big company with big money).

I have put Jonathan and Neil (our Debian Project Leader) in touch so that signing the document may happen, though since we were all busy with the summit, I do not expect Jonathan to send the documents right away. Hopefully, this will be fixed before the end of this month of May 2015.

Debian (and Ubuntu) packages collaboratively maintained upstream
Since about forever (forever is 5 years in the OpenStack world…), I pushed for more collaboration on OpenStack packaging between Debian package maintainers and Canonical. However, for some reasons which I do not wish to expand on in this blog post, it has been socially hard to do so. Also, Canonical always used BZR, which wasn’t to the tastes of everyone.

But during the Liberty summit, some very good things happened. First of all, Launchpad is now able to support Git (it’s been a few weeks it does in fact). Even though it will take a bit of time before the Canonical server team switches to it, we can consider that this problem is already out of the way.

Then it looks like Canonical are now more open than before for collaboration with Debian on the OpenStack packaging. Note that we actually did some work together already, but now we both would like a full alignment of *all* of our packages.

I have discussed this with James Page, who is the head of Canonical’s server team. We will first start to do so on the dependencies: this includes all of the python-*client libraries, but also all of python-oslo.* (the Oslo libs are use by all of the projects and are kind of unifying the project), plus all the third party dependencies the project relies on. James already pushed new versions of some Oslo libraries to Experimental (in order to not overwrite Kilo), which are adding transition packages needed for Ubuntu. We wont need those in Debian, but we want to welcome them to keep the same source packages.

We will then later try to merge the core projects if we can. Unfortunately, since the packaging of the core projects (ie: Nova, Neutron, Cinder, Glance, etc.) was forked, merging probably will be a bit painful. We will have to make some decisions on how this happen. I am however confident that it will be done during the Liberty release cycle.

Move of the packaging to upstream Gerrit
A few weeks after the summit, I wrote a proposal to upstream OpenStack dev list, with as subject: “Adding packaging as an OpenStack project”. What it means is that I have proposed to have Debian/Ubuntu packaging to happen in upstream infrastructure, using Gerrit, and building packages using upstream cloud. We will add all the tests we can, like building with unit tests, lintian, piuparts, adequate, but also maybe a full installation of the packages with functional tests.

My proposal is here:
http://lists.openstack.org/pipermail/openstack-dev/2015-May/064848.html

As everything, this translates into a Gerrit review process:
https://review.openstack.org/#/c/185187/

As you can read in the above thread, Fedora/RDO people, which have used a Gerrit work-flow for a long time already, also would like to join. But it looks like we’ll be doing 2 teams: one for RPMs and one for debs.

The proposal is currently under review by the OpenStack technical committee, which will accept (or not) if the packaging project can be fully considered as an OpenStack project. I expect a final answer next Tuesday. Note that if they deny, we can still use the stackforge namespace instead, their decision is just about the TC blessing the project as being OpenStack or not.

What’s very nice about this, is that not only we will have a better collaboration between Debian & Ubuntu, better automated testing and Q/A, this also opens contributions to potentially anyone. Especially, we welcome operation people, those who are doing actual big deployments.

Sure, it was possible before, but I often had the feedback that many were scared to break anything when trying to contribute. Thanks to the CI/CD form upstream infra, and the Gerrit peer review process, it wont be a problem anymore. So we do expect operation people to contribute more. I will also push more “upstream packaging” within Mirantis, so that MOS (Mirantis OpenStack) aligns fully with Debian & Ubuntu as well.

Another good thing, is that it will be easier for the puppet team to support Debian (they historically were more Ubuntu oriented), and it’s going to be super easy for them to request for packaging fixes. I hope we will be able to work hand-to-hand with them, adding puppet deployment checks in the packaging repo, and packaged deployments within the puppet Gerrit review process.

November 2014:
Sunday 2nd:
– Travel from Moscow to Paris

Monday 3rd to Sunday 8th:
– Summit in Paris

Monday 10th:
– Uploaded python-rudolf to Sid (needed by Fuel)
– Uploaded python-invoke and python-invocations (needed to run fabric’s unit tests)
– Uploaded python-requests-kerberos/0.5-2 fixing CVE-2014-8650: failure to handle mutual authentication. Asked the release team for unblock.
– Uploaded openstack-pkg-tools version 19 fixing startup with systemd in Jessie (added RuntimeDirectory directive). Asked the release team for unblock.
– Opened ticket to remove TripleO, Tuskar and Ironic packages from Jessie. I don’t consider them ready for a Debian stable release, and there’s no long term support from upstream.
– Fixed Designate Juno dbsync process which prevented it from being installed.
– Fixed Ironic Juno unowned files after purge (policy 6.8, 10.8): /var/lib/ironic/{cache, ironicdb} (eg: purging these folders on purge)

Thuesday 11:
– Fixed nova-api “CVE-2014-3708: Nova network DoS through API filtering” in both the Juno and Icehouse release. Asked the release team to unblock the Icehouse version for Jessie. See: https://bugs.debian.org/769163
– Uploaded Cinder with Duch debconf translation fix and pt.po
– Uploaded python-django-pyscss with upstream patch for Django 1.7 support instead of the Debian one that I wrote 2 months ago. Asked the release team to unblock which they did.

Wednesday 12:
– Uploaded fix for horizon (see #769101) unowned files after purge (policy 6.8, 10.8). Now purging /usr/share/openstack-dashboard/openstack_dashboard on purge.
– Uploaded Ironic with Duch translations of debconf
– Uploaded Designate with Duch translations of Debconf screens
– Uploaded openstack-trove with Duch translations of Debconf screens
– Uploaded Tuskar with Duch translations of Debconf screens
– Updated python-oslotest in Experimental to version 1.2.0

Thursday 13:
– Uploaded new packages: python-oslo.middleware and python-oslo.concurrency.
– Opened a new packaging branch for Nova Kilo, and updated (build-)depends.
– Uploaded fix for Icehouse Cinder: “delete volume failed due to unicode problems”, and asked for unblock.
– Uploaded new package: python-pygit2 and python-xmlbuilder, needed for fuel-agent-ci.
– Uploaded sheepdog with Duch debconf translation.
– Uploaded python-daemonize to Sid (in FTP master NEW queue).
– Re-uploaded python-invoke after FTP master rejection (missing copyright information)

Friday 14:
– Uploaded liberasurecode & python-pyeclib to Sid, now in the FTP masters NEW queue waiting for approval. This will soon be needed by Swift.

Monday 17:
– Worked on the Cobbler packaging (all day long…)

Tuesday 18:
– Worked on backporting all of Fuel packages to Wheezy. Done with fuelclient already.
– Uploaded ruby-cstruct and ruby-rethtool to Sid (needed by nailgun-agent)

Wednesday 19:
– Uploaded pyeclib again, with fixes for the build-depends. Package is still in the NEW queue anyway.
– Built a Debian-based bootstrap hardware discovery image for Fuel, and … it seems that it works already (to be checked…)! \o/
To be added as packages in the ISO:
* nailgun-mcagents
* nailgun-net-check
* fuel-agent
* python-tasklib

Thursday 20:
– Uploaded python-tasklib to Sid (now in NEW queue…)
– Continued working on the discovery bootstrap ISO

Friday 21:
– Documented Sahara procedure in Debian in the official install-guide: https://review.openstack.org/136237
– Fixed oslo.messaging so it doesn’t use PROTOCOL_SSLv3 because its support has been removed from Debian (due to possible protocol downgrade attacks): https://review.openstack.org/136278 and uploaded fixed packages for Sid and Experimental.
– Uploaded fixed Neutron packages for CVE-2014-7821 in both Sid and Experimental (eg: Icehouse and Juno)

Monday 24:
– Uploaded new package: python-os-client-config (in NEW queue)
– Installed new Xen server to be used as my new Jenkins build machine
– Moved the juno-wheezy VM to it
– Finished to package python-pymysql and uploaded to Sid. It’s now running all unit tests successfully! \o/

Tuesday 25:
– Uploaded fix for openstack-debian-images to add the -o compat=1.0 option when building an image with Qemu > 1.0. Opened bug to the release team to have it unblocked.
– Continued working on unit tests for fuel-nailgun.

Wednesday 26:
– Uploaded python-os-net-config to Sid (new package)
– Worked briefly on python-cassandra-driver. It needs cassandra to be in, which is a LOT of work.
– Found a (not useable) hack to run nailgun unit tests. It works, however, it doesn’t seem like fuel-nailgun is designed to be able to use unix socket for the postgres connection in its unit tests.
– Uploaded python-pykmip to Sid (new package)
– Updated the Debian wheezy backport repository for libvirt to version 1.2.9 from official wheezy-backports. Removed policykit-1 and libusb from there too, as it broke stuff to use a backported version (X and usb were not useable on my Wheezy laptop when using it…).

Thursday 27 & Friday 28:
– Uploaded new Javascript packages or dependencies for Fuel: libjs-autonumeric, libjs-backbone-deep-model, libjs-backbone.stickit, libjs-cocktail, libjs-i18next, libjs-require-css, libjs-requirejs, libjs-requirejs-text

Sunday 30:
– Uploaded debian/copyright fixes for libjs-backbone-deep-model, libjs-backbone.stickit and libjs-cocktail after the packages were accepted by the FTP masters and they gave remarks about copyright.

DECEMBER 2014

Monday 01:
– Uploaded new Debian image to MOX, after I unerstood the issue was about the architecture field that I was wrongly filling. I’ll be able to use that for Tempest checking on my dev account.

Tuesday 02:
– Uploaded python-q-text-as-data to Sid (new awesome package!)
– Uploaded Horizon with some triggers mechanisms to start the compress when one of its JS depends is updated. That’s very important for security!
– Uploaded a fixed version of heat-cfntools to Sid (it was missing the /usr/lib/python* folder). Asked the release team for an unblock so it can reach Jessie.
– Fixed unit tests in fuel-nailgun, thanks to a patch from Sebastian Kalinowski. Now all unit tests are passing but one (for which I opened a launchpad bug: tests are trying to write in /var/log/nailgun, which is impossible at package build time).

Wednesday 03:
– Uploaded fixed version of ruby-rethtool after FTP master’s rejection and upstream correction of licensing files.
– Uploaded fixed version of libjs-require-css after FTP master’s rejection
– Fixed (in Git only) python-sysv-ipc missing build-depends on dh-python as per bug opened by James Page (this is not so important, but I did it still).
– Continued working on the tempest-ci scripts.
– Added to the image-guide docs about openstack-debian-images: https://review.openstack.org/#/c/138743/

Thursday 04:
– Uploaded new package: python-proliantutils. Send patch to upstream about an issue in indentation (mix-up with space and tabs) which made the package uninstallable with Python 3.4.

Friday 05:
– Worked on the package CI.

Monday 07:
– Worked on the package CI. All works now, up to all of the Tempest tests for Keystone. Now need to fix the neutron config.

Thuesday 08:
– Continued working on the CI.

Wednesday 09:
– Uploaded fix for FTBFS of python-tasklib (Closes: #772606)
– Uploaded fix for libjerasure-deb missing dependency on libgf-complete-dev, package already unblocked and will migrate to Jessie.
– Uploaded fix for Designate Juno fail to upgrade from Icehouse: this was due to the database_connection directive renamed to connection =.
– Uploaded fix for Designate purge in Sid (Icehouse release of Designate).
– Commited to git updates of the German debconf translation in both Icehouse and Juno.
– Updated nova to use libvirtd as init script dependency instead of libvirt-bin (this was renamed in the libvirt-daemon-system package).
– Do not touch the db connection directive if user didn’t ask for db handling by the package.

Thursday 10 to Saturday 13:
– Finally understood the issues with systemd service files not being activated by default. Fixed openstack-pkg-tools, and uploaded version 20 to Sid, after the release team accepted the changes.

Sunday 14:
– Uploaded Juno 2014.2.1 to Experimental: ceilometer, cinder, glance, python-glance-store, heat, horizon, keystone

Monday 15:
– Finished uploading Juno 2014.2.1 to Experimental: Nova, Neutron, Sahara

Tuesday 16:
– Added crontab to flush tokens in Icehouse Keystone
– Some more CI work

Wednesday 17:
– Uploaded keystone with systemd fix and crontab to flush the token table in Sid (eg: Icehouse).
– Uploaded nova Icehouse with a bunch of fixes in Sid.

Thursday 18:
– Updated some issues in Nova Icehouse (Sid/Jessie)

Friday 19:
– Started building a new Jenkins instance for building Kilo packages

Monday 22:
– Finished building the new Jenkins instance for building Kilo packages, and rebuilt every packages there, using Jessie as a base.

Tuesday 23:
– Updated version for the following packages: oslo.utils, oslo.middleware, stevedore, oslo.concurency, pecan, oslo.concurrency, python-oslo.vmware, python-glance-store
– Built so far: Ceilometer, Keystone, python-glanceclient, cinder, glance

Wednesday 24:
– Continued packaging Kilo beta 1. Updated: nova, designate, neutron
– Uploaded python-tempest-lib to Debian Unstable (new package)

Wednesday 31:
– Continued packaging Kilo beta 1. Updated: heat

JANUARY 2015

Thursday 01:
– Continued packaging Kilo beta 1. Updated: ironic, openstack-trove, openstack-doc-tools, ceilometer

Friday 02:
– Finished packaging Kilo beta 1. Updated: Sahara, Murano, Murano-dashboard, Murano-agent

Sunday 04:
– Started testing Kilo beta 1. Fixed a few issues on default configuration for Ceilometer and Glance.

Monday 05:
– Fixed openstack-pkg-tools which failed to create PID files at boot time, Uploaded to Sid, asked the release team for unblock.
– Uploaded ceilometer & cinder to Sid, rebuilt against openstack-pkg-tools 21.
– Did more testing of Kilo beta 1, fixed a few more minor issues.

Tuesday 06:
– Uploaded glance, neutron, nova, designate, keystone, heat, trove to Sid, so that all sysv-rc init scripts are fixed with the new openstack-pkg-tools 21. Designate, heat, keystone and trove contains other minor fixes reported to the Debian BTS.

Wednesday 07:
– Asked the Debian release team (open bugs with debdiff as attachment) for unblocks of glance, neutron, nova, designate, keystone, heat, trove so they migrate to Jessie.
– Fixed a few minor issues tracked in the Debian BTS on various packages.

Thesday 08:
– James Page from Canonical informed me that they are now using openstack-pkg-tools for maintaining their daemons in Nova, Cinder and Keystone in Ubuntu. That’s an awesome news : more QA for both platforms.
– James Page found out that dh_installinit *must* be called *after* the call of dh_systemd_enable, otherwise, daemons aren’t started automatically at the first install of packages, as the unmask of systemd happens after the invoke-rc.d.

Friday 09:
– Did some QA checks on the latest upload. Fixed Heat which broke because using the wrong template name (glance instead of heat).

Monday 12:
– Started re-running the automated openstack-deploy scrip in Icehouse, Juno and Kilo. Found out the issue in Keystone wasn’t fixed in Juno (but was fixed in other releases), and fixed it.
– Removed the use of ssl.PROTOCOL_SSLv3 from heat (removed form Debian). Uploaded the fixed package to Sid.
– All of openstack-deploy (debian/kilo branch) now works and succesfully installs OpenStack again.

If dh_installinit is called before, we have:

# Automatically added by dh_installinit
if [ -x "/etc/init.d/keystone" ]; then
update-rc.d keystone defaults >/dev/null
fi
if [ -x "/etc/init.d/keystone" ] || [ -e "/etc/init/keystone.conf" ]; then
invoke-rc.d keystone start || true
fi
# End automatically added section
# Automatically added by dh_systemd_enable
# This will only remove masks created by d-s-h on package removal.
deb-systemd-helper unmask keystone.service >/dev/null || true

# was-enabled defaults to true, so new installations run enable.
if deb-systemd-helper --quiet was-enabled keystone.service; then
# Enables the unit on first installation, creates new
# symlinks on upgrades if the unit file has changed.
deb-systemd-helper enable keystone.service >/dev/null || true
else
# Update the statefile to add new symlinks (if any), which need to be
# cleaned up on purge. Also remove old symlinks.
deb-systemd-helper update-state keystone.service >/dev/null || true
fi
# End automatically added section

If it’s called after dh_systemd_enable, we have:

# Automatically added by dh_systemd_enable
# This will only remove masks created by d-s-h on package removal.
deb-systemd-helper unmask keystone.service >/dev/null || true

# was-enabled defaults to true, so new installations run enable.
if deb-systemd-helper --quiet was-enabled keystone.service; then
# Enables the unit on first installation, creates new
# symlinks on upgrades if the unit file has changed.
deb-systemd-helper enable keystone.service >/dev/null || true
else
# Update the statefile to add new symlinks (if any), which need to be
# cleaned up on purge. Also remove old symlinks.
deb-systemd-helper update-state keystone.service >/dev/null || true
fi
# End automatically added section
# Automatically added by dh_installinit
if [ -x "/etc/init.d/keystone" ]; then
update-rc.d keystone defaults >/dev/null
fi
if [ -x "/etc/init.d/keystone" ] || [ -e "/etc/init/keystone.conf" ]; then
invoke-rc.d keystone start || true
fi
# End automatically added section

As a consequence, I have to re-upload version 22 of openstack-pkg-tools and also re-upload all OpenStack core packages to Debian Sid.

– Fixed a number of issues like:
* dbc_upgrade = true check which shouldn’t have been there in postinst.
* <project>/configure_db default value is now always false
* db_sync and pkgos_dbc_postinst are now only done if <project>/configure_db is set to true.
– Rebuilt all packages in Juno and Kilo with the above changes.

Tuesday 13:
– Opened unblock bugs for the release team to unblock all fixed packages.
– Made more tests in Juno and Kilo to make sure the fixed bugs in Icehouse are fixed there too.
– Fixed numerous issues in Trove (missing trove-conductor.conf, wrong trove-api init file, etc.). More work will be needed for it for both Icehouse and newer releases.

Wednesday 14:
– Did a doc meeting about debconf. Some doc contributors still want to kill the debconf / debian manual, and I have to not agree.
– Made a new patch to better document the keystone install procedure:
– Did some bug triaging in the doc about Debian.
– Uploaded new versions of core packages to Experimental (eg: Juno) built against openstack-pkg-tools >= 22~, and some fixes forward ported from Icehouse: Keystone, Ceilometer, Cinder, Glance, Heat, Ironic, Murano, Neutron, Nova, Saraha and Murano-agent. All where rebuilt in Juno (Wheezy + Trusty) and Kilo (Jessie only) on my Jenkins.

Thuesday 15:
– Succesfully booted a live-build Debian live image containing mcollective and nailgun-agent as a Debian replacement for the hardware discovery / boostrap image of Fuel. Now, I need to find a way to use just a kernel + initramfs

Friday 16 to Tuesday 20:
– Worked on the packaging CI.

Wednesday 21:
– Fixed https://bugs.debian.org/775636 (Horizon failed to build due to a Moscow timezone change and wrong test). Uploaded to Sid, asked for unblock.
– Fixed https://bugs.debian.org/775926: CVE-2015-1195: Glance still allows users to download and delete any file in glance-api server (applied upstream patch). Uploaded to Sid, asked for unblock. Uploaded Juno version to Experimental.
– Uploaded openstack-trove with the remaining fixes, asked release team for unblock.
– Uploaded python-glanceclient 0.15.0 (Juno) to Experimental because it fixes an issue with HTTPS. Added to it a patch from James Page not yet merged, which fixes unit test with Python 2.7.9 (7 failures otherwise).
– Uploaded python-xstatic-d3 as it can’t be installed anymore in Sid after a new version of d3 was uploaded.

Thursday 22:
– Uploaded python-xstatic-smart-table and libjs-angularjs-smart-table to Sid (new packages, now in NEW queue).

Friday 23:
– Ask for the removal of the below list of packages from Jessie:
python-xstatic
python-xstatic-angular
python-xstatic-angular-cookies
python-xstatic-angular-mock
python-xstatic-bootstrap-datepicker
python-xstatic-bootstrap-scss
python-xstatic-d3
python-xstatic-font-awesome
python-xstatic-hogan
python-xstatic-jasmine
python-xstatic-jquery
python-xstatic-jquery-migrate
python-xstatic-jquery-ui
python-xstatic-jquery.bootstrap.wizard
python-xstatic-jquery.quicksearch
python-xstatic-jquery.tablesorter
python-xstatic-jsencrypt
python-xstatic-qunit
python-xstatic-rickshaw
python-xstatic-spin
libjs-jsencrypt
libjs-spin.js
libjs-twitter-bootstrap-datepicker
libjs-twitter-bootstrap-wizard

They are used only in OpenStack Horizon starting on 2014.2 (aka Juno), and Jessie is shipped with Icehouse, so it’s IMO best to not carry the burden of maintaining these packages for the life of Jessie.

Monday 26:
– Enhanced and review requested changes for https://review.openstack.org/147296 (ie: Keystone install with more details about what the package does).
– Finished testing network on the CI install. Now need to automate all.

Tuesday 27:
– Closed all bugs on the rabbitmq-server package (2 correction, one bug triage).
– Uploaded a fix for the missing conntrack dependency in neutron-l3-agent.
– Restarted working on CI setup of Juno after success with manual install in a Xen domU.
– Uploaded fix to make sheepdog build reproducible (patch from the Debian BTS).

Thursday 28:
– Fixed and uploaded to Sid openstack-debian-images 2 bugs reported by Steve McIntire. Official Debian images for OpenStack are now available at:
http://cdimage.debian.org/cdimage/openstack/ \o/
Note that this is the weekly build of testing. We wont get Debian Stable images before Jessie is out.
– Documented the new image thing here: http://docs.openstack.org/image-guide/content/ch_obtaining_images.html#debian-images as a new patch: https://review.openstack.org/#/c/151015/
– Fixed my patch for keystone debconf doc at: https://review.openstack.org/#/c/147296/

Wednesday 29:
– Continued working on packaging CI

Thursday 30:
– Fixed CVE on Neutron (Juno): L3 agent denial of service with radvd 2.0+
– Fixed CVE on Glance (Icehouse + Juno): Glance user storage quota bypass. Asked release team for unblock.
– Fixed the image-guide patch after review (ie: https://review.openstack.org/151015)

tl;dr: Providing support for all 3 init systems (sysv-rc, Upstart and systemd) isn’t hard, and generating the init scripts / Upstart job / systemd using a template system is a lot easier than I previously thought.

As always, when writing this kind of blog post, I do expect that others will not like what I did. But that’s the point: give me your opinion in a constructive way (please be polite even if you don’t like what you see… I had too many times had to read harsh comments), and I’ll implement your ideas if I find it nice.

History of the implementation: how we came to the idea

I had no plan to do this. I don’t believe what I wrote can be generalized to all of the Debian archive. It’s just that I started doing things, and it made sense when I did it. Let me explain how it happened.

Since it’s clear that many, and especially the most advanced one, may have an opinion about which init system they prefer, and because I also support Ubuntu (at least Trusty), I though it was a good idea to support all the “main” init system: sysv-rc, Upstart and systemd. Though I have counted (for the sake of being exact in this blog) : OpenStack in Debian contains currently 64 init scripts to run daemons in total. That’s quite a lot. A way too much to just write them, all by hand. Though that’s what I was doing for the last years… until this the end of this last summer!

So, doing all by hand, I first started implementing Upstart. Its support was there only when building in Ubuntu (which isn’t the correct thing to do, this is now fixed, read further…). Then we thought about adding support for systemd. Gustavo Panizzo, one of the contributors in the OpenStack packages, started implementing it in Keystone (the auth server for OpenStack) for the Juno release which was released this October. He did that last summer, early enough so we didn’t expect anyone to use the Juno branch Keystone. After some experiments, we had kind of working. What he did was invoking “/etc/init.d/keystone start-systemd”, which was still using start-stop-daemon. Yes, that’s not perfect, and it’s better to use systemd foreground process handling, but at least, we had a unique place where to write the startup scripts, where we check the /etc/default for the logging configuration, configure the log file, and so on.

Then around in october, I took a step backward to see the whole picture with sysv-rc scripts, and saw the mess, with all the tiny, small difference between them. It became clear that I had to do something to make sure they were all the same, with the support for the same things (like which log system to use, where to store the PID, create /var/lib/project, /var/run/project and so on…).

Last, on this month of December, I was able to fix the remaining issues for systemd support, thanks to the awesome contribution of Mikael Cluseau on the Alioth OpenStack packaging list. Now, the systemd unit file is still invoking the init script, but it’s not using start-stop-daemon anymore, no PID file involved, and daemons are used as systemd foreground processes. Finally, daemons service files are also activated on installation (they were not previously).

Implementation

So I took the simplistic approach to use always the same template for the sysv-rc switch/case, and the start and stop functions, happening it at the end of all debian/*.init.in scripts. I started to try to reduce the number of variables, and I was surprised of the result: only a very small part of the init scripts need to change from daemon to daemon. For example, for nova-api, here’s the init script (LSB header stripped-out):

DESC="OpenStack Compute API"
PROJECT_NAME=nova
NAME=${PROJECT_NAME}-api

That is it: only 3 lines, defining only the name of the daemon, the name of the project it attaches (eg: nova, cinder, etc.), and a long description. There’s of course much more complicated init scripts (see the one for neutron-server in the Debian archive for example), but the vast majority only needs the above.

Here’s the sysv-rc init script template that I currently use:

#!/bin/sh
# The content after this line comes from openstack-pkg-tools
# and has been automatically added to a .init.in script, which
# contains only the descriptive part for the daemon. Everything
# else is standardized as a single unique script.

# Author: Thomas Goirand <zigo@debian.org>

# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin

if [ -z "${DAEMON}" ] ; then
	DAEMON=/usr/bin/${NAME}
fi
PIDFILE=/var/run/${PROJECT_NAME}/${NAME}.pid
if [ -z "${SCRIPTNAME}" ] ; then
	SCRIPTNAME=/etc/init.d/${NAME}
fi
if [ -z "${SYSTEM_USER}" ] ; then
	SYSTEM_USER=${PROJECT_NAME}
fi
if [ -z "${SYSTEM_USER}" ] ; then
	SYSTEM_GROUP=${PROJECT_NAME}
fi
if [ "${SYSTEM_USER}" != "root" ] ; then
	STARTDAEMON_CHUID="--chuid ${SYSTEM_USER}:${SYSTEM_GROUP}"
fi
if [ -z "${CONFIG_FILE}" ] ; then
	CONFIG_FILE=/etc/${PROJECT_NAME}/${PROJECT_NAME}.conf
fi
LOGFILE=/var/log/${PROJECT_NAME}/${NAME}.log
if [ -z "${NO_OPENSTACK_CONFIG_FILE_DAEMON_ARG}" ] ; then
	DAEMON_ARGS="${DAEMON_ARGS} --config-file=${CONFIG_FILE}"
fi

# Exit if the package is not installed
[ -x $DAEMON ] || exit 0

# If ran as root, create /var/lock/X, /var/run/X, /var/lib/X and /var/log/X as needed
if [ "x$USER" = "xroot" ] ; then
	for i in lock run log lib ; do
		mkdir -p /var/$i/${PROJECT_NAME}
		chown ${SYSTEM_USER} /var/$i/${PROJECT_NAME}
	done
fi

# This defines init_is_upstart which we use later on (+ more...)
. /lib/lsb/init-functions

# Manage log options: logfile and/or syslog, depending on user's choosing
[ -r /etc/default/openstack ] && . /etc/default/openstack
[ -r /etc/default/$NAME ] && . /etc/default/$NAME
[ "x$USE_SYSLOG" = "xyes" ] && DAEMON_ARGS="$DAEMON_ARGS --use-syslog"
[ "x$USE_LOGFILE" != "xno" ] && DAEMON_ARGS="$DAEMON_ARGS --log-file=$LOGFILE"

do_start() {
	start-stop-daemon --start --quiet --background ${STARTDAEMON_CHUID} --make-pidfile --pidfile ${PIDFILE} --chdir /var/lib/${PROJECT_NAME} --startas $DAEMON \
			--test > /dev/null || return 1
	start-stop-daemon --start --quiet --background ${STARTDAEMON_CHUID} --make-pidfile --pidfile ${PIDFILE} --chdir /var/lib/${PROJECT_NAME} --startas $DAEMON \
			-- $DAEMON_ARGS || return 2
}

do_stop() {
	start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE
	RETVAL=$?
	rm -f $PIDFILE
	return "$RETVAL"
}

do_systemd_start() {
	exec $DAEMON $DAEMON_ARGS
}

case "$1" in
start)
	init_is_upstart > /dev/null 2>&1 && exit 1
	log_daemon_msg "Starting $DESC" "$NAME"
	do_start
	case $? in
		0|1) log_end_msg 0 ;;
		2) log_end_msg 1 ;;
	esac
;;
stop)
	init_is_upstart > /dev/null 2>&1 && exit 0
	log_daemon_msg "Stopping $DESC" "$NAME"
	do_stop
	case $? in
		0|1) log_end_msg 0 ;;
		2) log_end_msg 1 ;;
	esac
;;
status)
	status_of_proc "$DAEMON" "$NAME" && exit 0 || exit $?
;;
systemd-start)
	do_systemd_start
;;  
restart|force-reload)
	init_is_upstart > /dev/null 2>&1 && exit 1
	log_daemon_msg "Restarting $DESC" "$NAME"
	do_stop
	case $? in
	0|1)
		do_start
		case $? in
			0) log_end_msg 0 ;;
			1) log_end_msg 1 ;; # Old process is still running
			*) log_end_msg 1 ;; # Failed to start
		esac
	;;
	*) log_end_msg 1 ;; # Failed to stop
	esac
;;
*)
	echo "Usage: $SCRIPTNAME {start|stop|status|restart|force-reload|systemd-start}" >&2
	exit 3
;;
esac

exit 0

Nothing particularly fancy here… You’ll noticed that it’s really OpenStack centric (see the LOGFILE and CONFIGFILE things…). You may have also noticed the call to “init_is_upstart” which is needed for upstart support. I’m not sure if it’s at the correct place in the init script. Should I put that on top of the script? Was I right with the exit values for it? Please send me your comments…

Then I thought about generalizing all of this. Because not only the sysv-rc scripts needed to be squared-up, but also Upstart. The approach here was to source the sysv-rc script in debian/*.init.in, and then generate the Upstart job accordingly, using the above 3 variables (or more as needed). Here, the fun is that, instead of taking the approach of calculating everything at runtime with the sysv-rc, for Upstart jobs, many things are calculated at build time. For each debian/*.init.in script that the debian/rules finds, pkgos-gen-upstart-job is called. Here’s pkgos-gen-upstart-job:

#!/bin/sh

INIT_TEMPLATE=${1}
UPSTART_FILE=`echo ${INIT_TEMPLATE} | sed 's/.init.in/.upstart/'`

# Get the variables defined in the init template
. ${INIT_TEMPLATE}

## Find out what should go in After=
#SHOULD_START=`cat ${INIT_TEMPLATE} | grep "# Should-Start:" | sed 's/# Should-Start://'`
#
#if [ -n "${SHOULD_START}" ] ; then
#	AFTER="After="
#	for i in ${SHOULD_START} ; do
#		AFTER="${AFTER}${i}.service "
#	done
#fi

if [ -z "${DAEMON}" ] ; then
        DAEMON=/usr/bin/${NAME}
fi
PIDFILE=/var/run/${PROJECT_NAME}/${NAME}.pid
if [ -z "${SCRIPTNAME}" ] ; then
	SCRIPTNAME=/etc/init.d/${NAME}
fi
if [ -z "${SYSTEM_USER}" ] ; then
	SYSTEM_USER=${PROJECT_NAME}
fi
if [ -z "${SYSTEM_GROUP}" ] ; then
	SYSTEM_GROUP=${PROJECT_NAME}
fi
if [ "${SYSTEM_USER}" != "root" ] ; then
	STARTDAEMON_CHUID="--chuid ${SYSTEM_USER}:${SYSTEM_GROUP}"
fi
if [ -z "${CONFIG_FILE}" ] ; then
	CONFIG_FILE=/etc/${PROJECT_NAME}/${PROJECT_NAME}.conf
fi
LOGFILE=/var/log/${PROJECT_NAME}/${NAME}.log
DAEMON_ARGS="${DAEMON_ARGS} --config-file=${CONFIG_FILE}"

echo "description \"${DESC}\"
author \"Thomas Goirand <zigo@debian.org>\"

start on runlevel [2345]
stop on runlevel [!2345]

chdir /var/run

pre-start script
	for i in lock run log lib ; do
		mkdir -p /var/\$i/${PROJECT_NAME}
		chown ${SYSTEM_USER} /var/\$i/${PROJECT_NAME}
	done
end script

script
	[ -x \"${DAEMON}\" ] || exit 0
	DAEMON_ARGS=\"${DAEMON_ARGS}\"
	[ -r /etc/default/openstack ] && . /etc/default/openstack
	[ -r /etc/default/\$UPSTART_JOB ] && . /etc/default/\$UPSTART_JOB
	[ \"x\$USE_SYSLOG\" = \"xyes\" ] && DAEMON_ARGS=\"\$DAEMON_ARGS --use-syslog\"
	[ \"x\$USE_LOGFILE\" != \"xno\" ] && DAEMON_ARGS=\"\$DAEMON_ARGS --log-file=${LOGFILE}\"

	exec start-stop-daemon --start --chdir /var/lib/${PROJECT_NAME} \\
		${STARTDAEMON_CHUID} --make-pidfile --pidfile ${PIDFILE} \\
		--exec ${DAEMON} -- --config-file=${CONFIG_FILE} \${DAEMON_ARGS}
end script
" >${UPSTART_FILE}

The only thing which I don’t know how to do, is how to implement the Should-Start / Should-Stop in an Upstart job. Can anyone shoot me a mail and tell me the solution?

Then, I wanted to add support for systemd. Here, we cheated, since we only just called the sysv-rc script from the systemd unit, however, the systemd-start target uses exec, so the process stays in the foreground. It’s also much smaller than the Upstart thing. However, here, I could implement the “After” thing, corresponding to the Should-Start:

#!/bin/sh

INIT_TEMPLATE=${1}
SERVICE_FILE=`echo ${INIT_TEMPLATE} | sed 's/.init.in/.service/'`

# Get the variables defined in the init template
. ${INIT_TEMPLATE}

if [ -z "${SCRIPTNAME}" ] ; then
	SCRIPTNAME=/etc/init.d/${NAME}
fi
if [ -z "${SYSTEM_USER}" ] ; then
	SYSTEM_USER=${PROJECT_NAME}
fi
if [ -z "${SYSTEM_GROUP}" ] ; then
	SYSTEM_GROUP=${PROJECT_NAME}
fi

# Find out what should go in After=
SHOULD_START=`cat ${INIT_TEMPLATE} | grep "# Should-Start:" | sed 's/# Should-Start://'`

if [ -n "${SHOULD_START}" ] ; then
	AFTER="After="
	for i in ${SHOULD_START} ; do
		AFTER="${AFTER}${i}.service "
	done
fi

echo "[Unit]
Description=${DESC}
$AFTER

[Service]
User=${SYSTEM_USER}
Group=${SYSTEM_GROUP}
WorkingDirectory=/var/lib/${PROJECT_NAME}
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p /var/lock/${PROJECT_NAME} /var/log/${PROJECT_NAME} /var/lib/${PROJECT_NAME}
ExecStartPre=/bin/chown ${SYSTEM_USER}:${SYSTEM_GROUP} /var/lock/${PROJECT_NAME} /var/log/${PROJECT_NAME} /var/lib/${PROJECT_NAME}
ExecStart=${SCRIPTNAME} systemd-start
Restart=on-failure

[Install]
WantedBy=multi-user.target
" >${SERVICE_FILE}

As you can see, it’s calling /etc/init.d/${SCRIPTNAME} sytemd-start, which isn’t great. I’d be happy to have comments from systemd user / maintainers on how to fix it to make it better.

Integrating in debian/rules

To integrate with the Debian package build system, we only need had to write this:

override_dh_installinit:
	# Create the init scripts from the template
	for i in `ls -1 debian/*.init.in` ; do \
		MYINIT=`echo $$i | sed s/.init.in//` ; \
		cp $$i $$MYINIT.init ; \
		cat /usr/share/openstack-pkg-tools/init-script-template >>$$MYINIT.init ; \
		pkgos-gen-systemd-unit $$i ; \
	done
	# If there's an upstart.in file, use that one instead of the generated one
	for i in `ls -1 debian/*.upstart.in` ; do \
		MYPKG=`echo $$i | sed s/.upstart.in//` ; \
		cp $$MYPKG.upstart.in $$MYPKG.upstart ; \
	done
	# Generate the upstart job if there's no already existing .upstart.in
	for i in `ls debian/*.init.in` ; do \
		MYINIT=`echo $$i | sed s/.init.in/.upstart.in/` ; \
		if ! [ -e $$MYINIT ] ; then \
			pkgos-gen-upstart-job $$i ; \
		fi \
	done
	dh_installinit --error-handler=true
	# Generate the systemd unit file
	# Note: because dh_systemd_enable is called by the
	# dh sequencer *before* dh_installinit, we have
	# to process it manually.
	for i in `ls debian/*.init.in` ; do \
		pkgos-gen-systemd-unit $$i ; \
		MYSERVICE=`echo $$i | sed 's/debian\///'` ; \
		MYSERVICE=`echo $$MYSERVICE | sed 's/.init.in/.service/'` ; \
		dh_systemd_enable $$MYSERVICE ; \
	done

As you can see, it’s possible to use a debian/*.upstart.in and not use the templating system, in the more complicated case (I needed it mostly for neutron-server and neutron-plugin-openvswitch-agent).

Conclusion

I do not pretend that what I wrote in the openstack-pkg-tools is the ultimate solution. But I’m convince that it answers our own need as the OpenStack maintainers in Debian. There’s a lot of room for improvements (like implementing the Should-Start in Upstart jobs, or stop calling the sysv-rc script in the systemd units), but that this is a very good move that we did to use templates and generated scripts, as the init scripts are a way more easy to maintain now, in a much more unified way. As I’m not completely satisfied for the systemd and Upstart implementation, I’m sure that there’s already a huge improvements on the sysv-rc script maintainability.

Last and again: please send your comments and help improving the above! :)