Automatic backport script

Since I have to do a lot of backports for the OpenStack packages in Debian Wheezy, I got tired of doing them by hand. Therefore, I have created a script to automate the task. I’m not particularly proud (or ashamed) of that script, but I just want to share it. Probably some fellow readers will happily provide me with enhancements and ideas.

Note that the use of mk-build-deps implies that the “equivs” package has to be installed. What I do is run this “build-backport” script within a cowbuilder, to make sure I always have a clean backport environment.

#!/bin/sh

set -e
set -x

PKG_NAME=${1}
BPO_DISTRO=wheezy-backports
BPO_POSTFIX=bpo70+1
REPO_ROOT=/home/ftp
REPO_DEST=icehouse-backports

if [ `whoami` = "jenkins" ] ; then
        BUILD_ROOT=/var/lib/jenkins/jobs/openstack-pkg-tools/builds/${BUILD_NUMBER}
else
        BUILD_ROOT=~/sources
fi

# Get info from packages.debian.org
PKG_INFO_FILE=`mktemp -t pkg_info_file.XXXXXX`
wget --no-check-certificate -O ${PKG_INFO_FILE} http://packages.debian.org/sid/${PKG_NAME}
DEB_VERSION=`rmadison ${PKG_NAME} | grep sid | awk '{print $3}'`
UPSTREAM_VERSION=`echo ${DEB_VERSION} | cut -d'-' -f1  | cut -d":" -f2`
DSC_URL=`cat ${PKG_INFO_FILE} | grep dsc | cut -d'"' -f2`
rm ${PKG_INFO_FILE}

# Prepare build folder and go in it
MY_CWD=`pwd`
rm -rf ${BUILD_ROOT}/$PKG_NAME
mkdir -p ${BUILD_ROOT}/$PKG_NAME
cd ${BUILD_ROOT}/$PKG_NAME

# Download the .dsc and extract it
dget -d -u ${DSC_URL}
PKG_SRC_NAME=`ls *.dsc | cut -d_ -f1`
PKG_NAME_FIRST_CHAR=`echo ${PKG_SRC_NAME} | awk '{print substr($0,1,1)}'`
dpkg-source -x *.dsc

echo "Now running mk-build-deps -i ${PKG_SRC_NAME}-${UPSTREAM_VERSION}/debian/control"
mk-build-deps -i ${PKG_SRC_NAME}-${UPSTREAM_VERSION}/debian/control

# Build the package as backport using cowbuilder
cd ${PKG_SRC_NAME}-${UPSTREAM_VERSION}
dch --bpo -m "Backported for ${BPO_DISTRO}."
dpkg-buildpackage
cd ..  
PKG_FINAL_DEST=${REPO_ROOT}/debian/pool/${REPO_DEST}/main/${PKG_NAME_FIRST_CHAR}/${PKG_SRC_NAME}
ssh archive.gplhost.com "mkdir -p ${PKG_FINAL_DEST}"
scp *.orig.tar.* *${DEB_VERSION}~${BPO_POSTFIX}* archive.gplhost.com:${PKG_FINAL_DEST}

WordPress auto-updates stupidity

Out of laziness, like many, I use WordPress for this blog. I did try others, but was disappointed (after my blog got hacked a few times), so I just use that.

WordPress has a long history of security issues. So upstream decided to preform automatic updates. This would have been a good thing if … automatic update didn’t completely mess my blog each and every single time.

On my hosting system, PHP scripts have to be chmod +x to be executed. Otherwise, there’s a error, and Apache wont execute the PHP script. The same way, an error will happen if a directory is world writable (eg: chmod 777). This is in order to prevent some of the most common hacks: a hacker finds a way to upload a PHP script (often via a “feature” of the hosted software), and then uses the uploaded script to do nasty things (like installing phishing sites, send spam, you name it…). Checking on these basic unix rights prevents uploaded scripts to be executed, and it’s normally a way harder for hackers to find a way to chmod the PHP scripts than it is to just upload it.

Unfortunately, WordPress, on each upgrade, is resetting these unix rights. Someone got to explain to me the reason why it absolutely needs to chmod 777 the hosted folders, and why it wouldn’t keep the chmod +x on the php scripts. Direct result? My blog often gets completely broken by these automated updates. And I didn’t find a way to disable them (if someone knows, please send me a quick email).

I have reported the bug upstream: https://core.trac.wordpress.org/ticket/27568

OpenStack 2013.2.2 uploaded

This is the 2nd point release of OpenStack Havana (this is the name of the current stable release of OpenStack). It was out on Thursday (US time), and I uploaded it on Friday (Chinese time). Unfortunately, I realized that the latest python-keystoneclient didn’t support the –token and –endpoint command line options, effectively breaking Keystone itself, and all automated endpoint registration in all core packages. So after fixing Keystone and openstack-pkg-tools, I had to re-upload a 2nd Debian release.

However, after this glitch, the packages passed successfully our CI testing suite: my friends at eNovance and I run the tempest functional test suite each time there’s this kind of major update, so that we can validate the packages are working as expected. And all went well after the new keystone command line options were fixed in the postinst scripts.

As a side note, I fail to see consistency on deprecating these interfaces. While we still have the old “glance index” thing in the Glance client (this is really old), keystoneclient just broke backward compatibility, when keystone is quite recent. So, it looks to me that it wasn’t a great idea to just remove –token and –endpoint this way (though I can understand the need of consistency). Hopefully, this will all be fixed anyway, when we move to the unified python-openstackclient command lines instead of per-project clients.

I realized today that the pkg-openstack team now maintains more than 100 packages. This of course includes general purpose Python modules, which probably will move to the Python module team if it one day supports Git (life is too short: I do not wish to learn such a deprecated technology as SVN). Though this is still a lot of packages, and some more are coming: as per the OpenStack TODO list wiki page, Ironic (a replacement for nova-baremetal), TripleO (OpenStack on OpenStack), Tuskar (contols where things are deployed with TripleO) and Designate (DNS as a Service) are already packaged, and I am waiting for the Icehouse (this is the name of the next stable release) beta3 to be out to upload them to Experimental. This is scheduled for the 6th of March. I believe I am currently nearly up-to-date with the current global-requirements.txt (eg: python dependencies of OpenStack), pending a few Python module upgrades, so hopefully, packaging OpenStack Icehouse beta3 will smooth.

Removal of XCP / Xen-API from Jessie and Sid

This is sad, but no choice, due to the lack of upstream support: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=738322

I just hope xenserver-core will work soon in Debian, so we can fix this unfortunate situation.

OpenRC now in Experimental

I thought it would have been smooth, though it wasn’t. OpenRC shipped /sbin/rc, which conflicted with the “rc” shell (an implementation of the AT&T Plan 9 shell), and /sbin/runscript was conflicting with minicom. With the help of upstream authors, /sbin/rc was renamed /sbin/openrc, and /sbin/runscript was renamed /sbin/openrc-run.

However, the main goal is reached: after last summer Google Summer of Code project, and a bit of rework for the ruff edges, OpenRC made it to Debian.

So, if you wish to try OpenRC, which is a direct replacement for sysv-rc, just add the Debian Experimental repository to your sources.list, and do “apt-get install openrc”. The only issue will be the first reboot, though that should be fine if one manually shuts down every running daemon, and then type what the postinst suggests as command echoed on the screen. Suggestions on how to improve this is welcome. I warmly also welcome more general feedback.

I’d like to publicly thank Patrick Lauer, Benda (李明宇),  WIlliam Hubbs, Alexander Vershilov (who are all OpenRC upstreams), Bill Wang who was the GSoC studdent working on OpenRC, Roger Leigh who is the current sysv-init/sysv-rc maintainer, for their help and support when porting OpenRC to Debian. Without them, it wouldn’t have been possible.

Symplifying sysv-rc init.d scripts

Peter Reinholdtsen posted an article on planet.debian.org about symplifying init scripts. I don’t think that’s a good idea. I tried to do that, and then scrapped all my code, because I’ve found OpenRC that was doing what I wrote, in a much better way. We shouldn’t reivent the wheel, OpenRC is there already!

OpenRC runscript example for rsyslog

The point of switching to a new init system is to get rid of huge init scripts. Well, here’s an example OpenRC runscript, rewriting /etc/init.d/rsyslog:

#!/sbin/runscript

command=/usr/sbin/rsyslogd
name="enhanced syslogd"

depend()
{
        provide rsyslogd syslog
        need $remote_fs $time
}
Pretty minimalistic to me... For the record, the original sysv-rc script was 126 lines, and the above is just 10 lines.

OpenRC running on Debian / kFreeBSD

I have always pretended that OpenRC would be very easy to port to Debian GNU/kFreeBSD. Well, now I stop pretending:

http://youtu.be/zoNoi8BgQjs

This was done within a few hours working with upstream.

Now, next-up: Debian GNU/Hurd. Hoping that porters will volunteer to do the work.

OpenStack Havana 2013.2 Debian packages available

OpenStack “upstream” was released today. Thanks to the release team and a big up to TTX for his work.

By the time you read this, probably all of my uploads have reached your local Debian mirror.

Please try Havana using either Sid from any Debian mirror, or using the Wheezy backports available here:

deb http://havana.pkgs.enovance.com/debian havana main
deb http://archive.gplhost.com/debian havana-backports main

Yes, you will need *both* repositories. This is unofficial, though these are the exact same packages as in Sid, just rebuilt for Wheezy.

On the package side, here’s what is new:

– All packages who needs it can now be configured through debconf for the RabbitMQ settings. This is on top of what was already available for Grizzly, which is automated configuration for: keystone auth token, the database, the API endpoints and much more. (remember: this is fully optional, you can always use the non-interactive mode…)

– All Quantum pluggin packages have been removed, and now everything is self-contained in the neutron-common package. The selection of which plugin to use is done directly using the core_plugin= directive in /etc/neutron/neutron.conf. This will also control the init.d script of neutron-server, so that it loads the corresponding ini file in /etc/neutron/plugins. The plugin selection is done through Debconf, so that users don’t have to write the full path of the plugin class, which is (for most) very cryptic (am I the only one who thinks that writing neutron.plugins.openvswitch.ovs_neutron_plugin.OVSNeutronPluginV2 in a configuration file is user friendly?).

– All of the package descriptions and Debconf templates have been reviewed by the Debian internationalization team, and most strings are translated in Czech, Danish, French, Italian, Japanese and Russian (sometimes even more) for almost all packages (thanks everyone!).

I’d like to publicly thanks eNovance for sponsoring my packaging work, and Mehdi Abaakouk for his work on our CI with the tempest tests.

Happy Havana release testing,
Please report bugs through the Debian BTS.

Jenkins remote build trigger (eg: from git push) tokens

After upgrading my Sid virtual machine hosting my Jenkins, build after git push stopped working. This is because version 1.503 and above require an auth token for triggering the build. Since it took me some time to search the web, I’ve decided to blog about it to save time to other Jenkins users.

Under each project configuration screen, in the “Build Triggers” section, tick the “Trigger builds remotely (e.g., from scripts)” option. Then enter a random token (I used a password generator for it). Then in your post-receive hook, use what’s below:

wget -q --no-check-certificate https://<jenkins-url>/job/heat/build?token=<your-token> -O -