Backporting libjs-angularjs and libjs-d3 to Wheezy

If you didn’t notice, Javascript isn’t as simple as it used to be… Want to backport the 2 simple javascript libs? No problem. You then “just” need to backport a bunch of other packages which are build-dependencies… (and file #761670, #761672, and #761674 on the way when rebuilding…). Here’s the short list:

gyp
node-abbrev
node-ansi
node-ansi-color-table
node-archy
node-async
node-block-stream
node-combined-stream
node-contextify
node-cookie-jar
node-cssom
node-delayed-stream
node-diff
node-eyes
node-forever-agent
node-form-data
node-fstream
node-fstream-ignore
node-github-url-from-git
node-glob
node-graceful-fs
node-gyp
node-htmlparser
node-inherits
node-ini
node-jake
node-jsdom
node-json-stringify-safe
node-lockfile
node-lru-cache
node-marked
node-mime
node-minimatch
node-mkdirp
node-mute-stream
node-node-uuid
node-nopt
node-normalize-package-data
node-npmlog
node-once
node-optimist
node-osenv
node-qs
node-queue-async
node-read
node-read-package-json
node-request
node-retry
node-rimraf
node-semver
node-sha
node-sigmund
node-slide
node-smash
node-tar
node-tunnel-agent
node-uglify
node-underscore
node-utilities
node-vows
node-whic
node-which
node-wordwrap
nodejs
npm
ruby-ronn

Yes, that’s 66 packages above… And of course, backporting some ruby stuff makes sense… :)

Debconf 14 activity

Before I start a short listing of (some of) the stuff I did during Debconf 14, I’d like to say how much I enjoyed everyone there. You guys (all of you, really!) are just awesome, and it’s always a real pleasure to see you all, each time.

Anyway, here’s a bits of the stuff I did.

1/ packaging of Google Cloud Engine client tools.

Thanks to the presence of Eric and Jimmy, I was able to finish the work I started at Debconf 13 last year. All python modules are packaged and uploaded. Only the final client (the “gcloud” command line utility) isn’t uploaded, even though it’s already packaged. The reason is that this client downloads “stuff” from internet, so I need to get the full, bundled, version of it, to avoid this. Eric gave me the link, I just didn’t have time to finish it yet. Though the (unfinished) package is already in the Git in Alioth.

2/ Tasksel talks

We discussed improvements in Tasksel both during the conference, and later (in front of beers…). I was able to add a custom task on a modified version of the Tasksel package for my own use. I volunteered myself for adding a “more task” option in Tasksel for Jessie+1 because I really would like to see this feature, and nobody raised hand, but honestly, I have no idea how to do it, and therefore, I’m not sure I’ll be able to do so. We’ll see… Anyway, before this happen, we must make sure that we know what kind of tasks we want in this “more tasks” screen, otherwise it’d be useless work for nothing. Therefore, I have setup a wiki page. Please edit the page and drop your ideas there. I’ve already added entries for desktops and Debian blends, but I’m sure there’s more that we could add.

3/ Custom Debian CD

I started experimentation on building my own Debian Wheezy CD image (well, DVD, since the resulting image is nearly 2GB). This was fun, but I am still having the issue that the installer fails to install Dash, so the CD is still unusable. I’ll try to debug it. Oh…  I nearly forgot… “of course”, the ISO image aims at including all OpenStack Icehouce packages backported to Wheezy, and the goal was to include the above custom Tasksel task, with an “OpenStack proxy node” task, and a “OpenStack compute” task. Let’s hope I can figure out what the issue is, and finally release it.

4/ OpenStack talk

Nothing special to say, just watch the video. I hope my talk was interesting enough. Of course, after watching myself, I hate everything I see, and would like to correct so many mistakes, but that’s the usual, I guess.

5/ Some RC fixing

Thanks to the nice work of our DPL rebuilding all the archive, I had to fix a couple of FTBFS issues on my own packages. 3 of them have been easy to fix (2 missing build-dependencies which I missed because my automated build environment has them by default, and a unit test failure), I still don’t understand what’s going on with Ceilometer. I also NMU-ed transmission (switching from 2.82 to 2.84, as upstream had the bugfix, and current maintainer was not responsive) which was the last blocker for the miniupnpc transition to Jessie. After the 5 days delay of the upload, it went in Sid, then migrated to Jessie, together with the miniupnpc library. I also fixed a trivial RC bug with python3-webob.

6/ Python team meeting

It was nice to see everyone, and hopefully, we’ll soon implement what we discussed. I hope to start migrating some of my OpenStack dependencies to the team once we move to Git (though please don’t expect this to happen before the Juno release, which keeps me very busy these days).

There’s probably more stuff which I did during Debconf 14 (hacking or otherwise), but either it’s not worth sharing, or I can’t remember… :)

sysvinit not sending output to all consoles

I spent many, many hours trying to understand why I couldn’t have both “nova console-log” showing me the output of the log, AND have the OpenStack dashboard (eg: horizon) console to work at the same time. Normally, this is done very easily, by passing multiple times the console= parameter to the Linux kernel as follow:

console=tty0 console=ttyS0,115200

But it never worked. Always, it’s the last console= thing that was taken into account by sysvinit (or, shall I say, bootlogd). Spending hours trying to figure out what would be the correct kernel command to pass didn’t help. Then this week-end, by the magic pure chance of being subscribed to the sysvinit bug reports, I have finally found out. We’ve had this bug in Debian for more than 10 years:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=181756

And it has the patch! It just feels so lame that the issue has been pending since 2003, and with a patch since 2006, and nobody even tried to have it enter Debian. I tried the Wheezy patch in the above bug report, and then tadaaaaaa! I finally had both the “nova console-log” (eg: ttyS0) console output, and the interactive tty0 to work on my Debian cloud image. I have produced a fixed version of the sysvinit package for Wheezy, if anyone wants to try it:

http://archive.gplhost.com/debian/pool/juno-backports/main/s/sysvinit/

This doesn’t only affect only the cloud images use case. Let’s say you have a server. If it’s a modern server, probably you have IPMI 2.0 on it. While having access through the integrated KVM over IP may be nice, seeing the boot process through the serial console redirection is often a lot more snappy than the (often VNC based) video output, plus it wouldn’t require Java. Too often, Java a requirement for these nasty IPMI web interface (that’s the case for at least: Dell DRAC, Supermicro IMPI, and HP iLO). Well, it should now be possible to just use ipmitools to debug the server boot process or to go fix stuff in the single user interactive shell, AND keep the “normal” video output! :)

But keeping this fix private doesn’t help much. I would really love to get this fixed within Debian. So I have sent the patch (which needed a very small rebase) in the Git repository of sysvinit (see http://deb.li/3YxUD). I of course tested it in Sid too. Though I tested only under a Xen virtual machine, I see no reason why it would work there and not elsewhere. That being said, I would welcome more testing, given the high profile of sysvinit (everyone uses/needs it, and I wouldn’t like to carry alone the unhappiness of having no boot log). Please do test it from the sysvinit git, before it’s even uploaded to Sid. Also, these days, sysvinit gets often uploaded to Experimental first. It will probably also be the case for version 2.88dsf-56.

If it works well and nobody complains about the patch, maybe it’d be worth adding it to Wheezy as well (though that decision is of course left to the release team once the fix reaches Jessie).

How can Lintian stop being annoying

As I care to have all the Lintian output and warnings, I have this in my ~/.lintianrc:

info=yes
display-info=yes
display-experimental=yes
pedantic=yes
show-overrides=yes
color=auto

So, by default, my Lintian setup displays all pedantic warnings. However, it’s been annoying me with the debian-watch-may-check-gpg-signature, which I don’t really care about since 1/ there’s never such a PGP stuff upstream, and 2/ I mostly use Git tags from upstream, in which I do check signatures whenever possible (and mostly, if I have upstream’s key in my keyring after a key signing party, which happened a few times). The solution? Well, very easy:

sudo sed -i 's/Disable-Tags:/Disable-Tags: debian-watch-may-check-gpg-signature,/' /usr/share/lintian/profiles/debian/main.profile

There’s nothing more to it! Of course, you can also create your own profile, with the added benefit that your changes wont be overwritten on next upgrade. But that’s overkill if you’re alone in the system, and anyway it’s damned easy to do the changes again, plus I often reinstall this computer from scratch. With the above, you can of course disable the display of any tag you wish.

OpenStack Icehouse bugs all cleaned-up

I’ve done some clean-up in the Debian BTS. The result can be seen in the QA graphs:

The last remaining 6 bugs are only affecting OpenStack Essex (which is what Wheezy ships, and which unfortunately I have not enough time to support properly), and the last one is waiting for FTP masters approval after I added Python 3 support to oslo-config (OpenStack is slowly moving to Python3, and I’ve tried to add support to Python3 in the packages as well each time it was possible). Also, the Icehouse packages have passed the tempest tests (a set of functional tests) which we run in our continuous integration system.

As I was doing some triaging, it is possible that some bugs have been closed a bit quickly (like for example the Ceilometer FTBFS which I couldn’t reproduce, even with sbuild and cowbuilder), so it is well possible that some more QA work will be needed later on. I’m also expecting a new set of patches for supporting Ceph in Nova. I’m sure there’s issues which we will discover later on, however, it is nice to have this result right after the first Icehouse release of OpenStack.

Next up: testing new components that I uploaded for this release: Trove (DB as a Service), Designate (DNS as a Service), Ironic (Metal as a Service, or cloud computing on bare-metal), and TripleO (OpenStack on OpenStack). I unfortunately know already that TripleO and Tuskar wont really work yet, and that it needs some patches to be sent upstream for it to support Debian correctly (let’s work it out for Juno!). So please consider it as a technology preview only. Though Trove, Designate and Ironic are supposed to be in good enough shape, I didn’t have the chance to test them more than just installing the packages and checking daemons are running and connected to the various components of OpenStack (eg: database, keystone and RabbitMQ). Please do test them and report bugs in the BTS.

I’d like to thanks Gustavo Panizzo & Thomas Bechtold who contributed to this release, and also the folks at eNovance (Emilien, Cyril, Seb…) who provided precious help with the CI and package quality. See you in the Juno Atlanta design summit next week (as hopefully, my plane ticket issue will be soon solved).

OpenStack 2014.1, aka Icehouse, is out

The new version of OpenStack is out, and I have just finished uploading it all into Debian Sid. With a total of 38 packages that I uploaded yesterday (which was exhausting!), most, if not all, were only moving from Experimental to Sid with only tiny updates, and this represents the achievement of 6 months of packaging work. The new feature list is impressive, and I would like to highlight some part of it:

  • New Ironic bare metal service.
  • New Designate DNS as a Service project.
  • Trove (DB as a Service) graduated from incubation and should work well now.
  • TripleO (OpenStack On OpenStack) is now fully in Debian, together with Tuskar and Tuskar-UI.
  • OpenStack now has VXLAN support through the new version of OVS and kernel >= 3.13. This solves the scalability issues with GRE tunnels.

For the moment, I haven’t packaged Sahara (eg: Hadoop as a service), but it might come later as a customer of us might require it.

There’s a lot less unit tests issues in the packages I uploaded to Sid: all SQLAlchemy issues have been dealt with. I wasn’t confident with the Havana release that Sid / Testing would be a good environment for OpenStack, but this time with Icehouse, I think it should be much better. Please test this brand new release and report issues on the BTS. As always, the packages are available also as Wheezy backports through the usual channels (see the official install guide).

Automatic backport script

Since I have to do a lot of backports for the OpenStack packages in Debian Wheezy, I got tired of doing them by hand. Therefore, I have created a script to automate the task. I’m not particularly proud (or ashamed) of that script, but I just want to share it. Probably some fellow readers will happily provide me with enhancements and ideas.

Note that the use of mk-build-deps implies that the “equivs” package has to be installed. What I do is run this “build-backport” script within a cowbuilder, to make sure I always have a clean backport environment.

#!/bin/sh

set -e
set -x

PKG_NAME=${1}
BPO_DISTRO=wheezy-backports
BPO_POSTFIX=bpo70+1
REPO_ROOT=/home/ftp
REPO_DEST=icehouse-backports

if [ `whoami` = "jenkins" ] ; then
        BUILD_ROOT=/var/lib/jenkins/jobs/openstack-pkg-tools/builds/${BUILD_NUMBER}
else
        BUILD_ROOT=~/sources
fi

# Get info from packages.debian.org
PKG_INFO_FILE=`mktemp -t pkg_info_file.XXXXXX`
wget --no-check-certificate -O ${PKG_INFO_FILE} http://packages.debian.org/sid/${PKG_NAME}
DEB_VERSION=`rmadison ${PKG_NAME} | grep sid | awk '{print $3}'`
UPSTREAM_VERSION=`echo ${DEB_VERSION} | cut -d'-' -f1  | cut -d":" -f2`
DSC_URL=`cat ${PKG_INFO_FILE} | grep dsc | cut -d'"' -f2`
rm ${PKG_INFO_FILE}

# Prepare build folder and go in it
MY_CWD=`pwd`
rm -rf ${BUILD_ROOT}/$PKG_NAME
mkdir -p ${BUILD_ROOT}/$PKG_NAME
cd ${BUILD_ROOT}/$PKG_NAME

# Download the .dsc and extract it
dget -d -u ${DSC_URL}
PKG_SRC_NAME=`ls *.dsc | cut -d_ -f1`
PKG_NAME_FIRST_CHAR=`echo ${PKG_SRC_NAME} | awk '{print substr($0,1,1)}'`
dpkg-source -x *.dsc

echo "Now running mk-build-deps -i ${PKG_SRC_NAME}-${UPSTREAM_VERSION}/debian/control"
mk-build-deps -i ${PKG_SRC_NAME}-${UPSTREAM_VERSION}/debian/control

# Build the package as backport using cowbuilder
cd ${PKG_SRC_NAME}-${UPSTREAM_VERSION}
dch --bpo -m "Backported for ${BPO_DISTRO}."
dpkg-buildpackage
cd ..  
PKG_FINAL_DEST=${REPO_ROOT}/debian/pool/${REPO_DEST}/main/${PKG_NAME_FIRST_CHAR}/${PKG_SRC_NAME}
ssh archive.gplhost.com "mkdir -p ${PKG_FINAL_DEST}"
scp *.orig.tar.* *${DEB_VERSION}~${BPO_POSTFIX}* archive.gplhost.com:${PKG_FINAL_DEST}

WordPress auto-updates stupidity

Out of laziness, like many, I use WordPress for this blog. I did try others, but was disappointed (after my blog got hacked a few times), so I just use that.

WordPress has a long history of security issues. So upstream decided to preform automatic updates. This would have been a good thing if … automatic update didn’t completely mess my blog each and every single time.

On my hosting system, PHP scripts have to be chmod +x to be executed. Otherwise, there’s a error, and Apache wont execute the PHP script. The same way, an error will happen if a directory is world writable (eg: chmod 777). This is in order to prevent some of the most common hacks: a hacker finds a way to upload a PHP script (often via a “feature” of the hosted software), and then uses the uploaded script to do nasty things (like installing phishing sites, send spam, you name it…). Checking on these basic unix rights prevents uploaded scripts to be executed, and it’s normally a way harder for hackers to find a way to chmod the PHP scripts than it is to just upload it.

Unfortunately, WordPress, on each upgrade, is resetting these unix rights. Someone got to explain to me the reason why it absolutely needs to chmod 777 the hosted folders, and why it wouldn’t keep the chmod +x on the php scripts. Direct result? My blog often gets completely broken by these automated updates. And I didn’t find a way to disable them (if someone knows, please send me a quick email).

I have reported the bug upstream: https://core.trac.wordpress.org/ticket/27568

OpenStack 2013.2.2 uploaded

This is the 2nd point release of OpenStack Havana (this is the name of the current stable release of OpenStack). It was out on Thursday (US time), and I uploaded it on Friday (Chinese time). Unfortunately, I realized that the latest python-keystoneclient didn’t support the –token and –endpoint command line options, effectively breaking Keystone itself, and all automated endpoint registration in all core packages. So after fixing Keystone and openstack-pkg-tools, I had to re-upload a 2nd Debian release.

However, after this glitch, the packages passed successfully our CI testing suite: my friends at eNovance and I run the tempest functional test suite each time there’s this kind of major update, so that we can validate the packages are working as expected. And all went well after the new keystone command line options were fixed in the postinst scripts.

As a side note, I fail to see consistency on deprecating these interfaces. While we still have the old “glance index” thing in the Glance client (this is really old), keystoneclient just broke backward compatibility, when keystone is quite recent. So, it looks to me that it wasn’t a great idea to just remove –token and –endpoint this way (though I can understand the need of consistency). Hopefully, this will all be fixed anyway, when we move to the unified python-openstackclient command lines instead of per-project clients.

I realized today that the pkg-openstack team now maintains more than 100 packages. This of course includes general purpose Python modules, which probably will move to the Python module team if it one day supports Git (life is too short: I do not wish to learn such a deprecated technology as SVN). Though this is still a lot of packages, and some more are coming: as per the OpenStack TODO list wiki page, Ironic (a replacement for nova-baremetal), TripleO (OpenStack on OpenStack), Tuskar (contols where things are deployed with TripleO) and Designate (DNS as a Service) are already packaged, and I am waiting for the Icehouse (this is the name of the next stable release) beta3 to be out to upload them to Experimental. This is scheduled for the 6th of March. I believe I am currently nearly up-to-date with the current global-requirements.txt (eg: python dependencies of OpenStack), pending a few Python module upgrades, so hopefully, packaging OpenStack Icehouse beta3 will smooth.

Removal of XCP / Xen-API from Jessie and Sid

This is sad, but no choice, due to the lack of upstream support: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=738322

I just hope xenserver-core will work soon in Debian, so we can fix this unfortunate situation.