Openstack Grizzly (2013.1~g3) available

This post is just a status update for the Openstack packaging, after the next version froze last week.

The Openstack bi-annual summit of next April will take place this year in Portland, Oregon, and if everything goes as plan, Grizzly will be released just before the summit. Grizzly will be out a bit before the next Ubuntu in April, as releases are following the one of Ubuntu. Openstack uses town names for it’s release names: Austin, Bextar, Cactus (2011.1), Diablo (2011.2), Essex (2012.1), Folsom (2012.2), Grizzly (2013.1).

I started to seriously work on the Openstack packaging in October, and never stopped working on its packaging until now. Slowly, but surely, preparing all the packages and their dependency python modules. One package at a time, working all this every day. Folsom works now pretty well and can be used in production, and I maintain it for security (along with Essex, which is in Wheezy).

Then Grizzly was frozen last week, on the 22nd of February, with the “G3” release. As I already worked on packaging the “G2” release in January, managing the packaging of “G3” was fast. On late Sunday, I had a repository with Grizzly built, and its corresponding python (build-)dependencies. But while just building your own repository is easy, having all the dependencies in Debian is a lot more work.

As of today, if I include all the python modules, I have (at least) touch around 50 packages in total when working on Openstack. Many of them were simply built from scratch. The only python dependency that needs an upgrade in Experimental, so that all dependencies are satisfied, is a new version of pep8. The rest of is new python modules that were not in Debian, and which are currently waiting in the NEW queue for ftp masters approval: python-pecan, python-tablib, python-wsme and websockify. Some of these python modules have been waiting there for a long time, like python-pecan (it’s been waiting in the NEW queue for more than 35 days now), some like websockify and python-wsme have been uploaded only this week. I really hope it can be possible to have all of Grizzly in Debian before the next Openstack summit (this depends mainly on the ftp-masters).

Note that I do not intend to apply security patches in Grizzly until it is released as the new Openstack stable. So use my private Grizzly repository as your own risk. I intend to fix this by working on some constant integration to have nightly builds, like many people are doing with Openstack. If you want to try it out:

deb http://archive.gplhost.com/debian grizzly main

Openstack Folsom fully uploaded to Experimental

It’s been a long time I wanted to blog about my recent work on the Openstack packaging. I finally can find a bit of time to do it.

For those who don’t know yet Openstack, it’s a fairly recent (less than 3 years) cloud computing suite, which is becoming quite huge. If you plan on deploying a private cloud, you should definitively have a look into it.

When the Openstack project started, I immediately was interested. I started packaging the Cactus release (that is, the 3rd version of Openstack), created the Alioth group (at the exact moment when Alioth was migrated to new hardware which added some fun … so lucky I was!), and began to work on the Debian version (Openstack used to be an Ubuntu only project). After some success in integrating some Debian specific patches into the Ubuntu packages, I left it a bit aside, and skipped completely the Diablo release (which was never uploaded in Debian). Then, I worked a bit on the Essex release before the freeze of Wheezy last June, together with other DDs (big up to Julien, Loic, Ghe, Sileth!).

I (re-)started serious packaging work for Openstack just right after Folsom (eg: version 2012.2) was released early last October. I literally worked days and nights on it, in order to provide more automation so that it could become more easy to install. Indeed, it used to be very complicated and painful, with lots of manual tasks to perform on the shell. It’s hopefully a lot easier now, with most of the manual boring shell work replaced by debconf and scripts. But I’m sure there’s more that can be done still.

After 4 months of effort, I finally pressed the red button and uploaded everything at once to Debian Experimental – since Openstack Essex (version 2012.1) is in Wheezy, I can’t of course upload to SID unless Wheezy is out -. This represent 32 sources packages in total (some of them are already uploaded and approved by the FTP masters: sorry to give you so much more work guys…), and 104 binary packages (and counting…). So this isn’t exactly small… And I already have some fixes for what’s currently waiting in the NEW queue: CVE fixes, missing build-depends which we’ve found using the Jenkins server of eNovance who sponsors the packaging work, etc. I probably will post again here to announce when Folsom is completely approved by the FTP masters and reaches Debian experimental. In the mean while, it’s available on GPLHost non-official repositories (see the howto linked bellow).

If you would like to test the latest Openstack release (called Folsom, or 2012.2, if you are following well…), you can read the – quite verbose – howto I wrote here:

https://wiki.debian.org/OpenStackHowto/Folsom

If you do test it (don’t be afraid, it’s not that hard…), I will be very happy to hear from you, and receive any feedback / critics you may have. Note that I would strongly recommend to use the Folsom release, rather than what is currently in Wheezy, for many reasons which would be too long to list in this blog post (let me still drop a few reasons here: less bugs, more automations and it’s more easy to install).

Now, Openstack is a constantly evolving software, with so many companies and developers involved. I don’t think my work on this beast will ever be finished. But it doesn’t mater, as it’s been quite some fun, and I will enjoy to do more.

The next release of Openstack is schedule for next April (they release at the same time as Ubuntu). I hope to be able to join the Portland summit, and see all the persons I have worked with over IRC and mailing list. This time, I hope to be able to release Openstack packages at the same time as the upstream source code is released (Debian packages have historically been lagging a few months behind). The pre-version is by the way already in Alioth.

Last, a (private) message for our famous cheese-master: please wait until everything leaves the NEW queue before bugging everyone with translations. We probably should have a serious talk about how to make it less redundant, easier to translate, and probably find a way to avoid duplication of messages across all packages.

MiniUPnP now fully in Debian

IT’S FINALLY IN!

With this last upload of the MiniUPnP daemon 1.7-3, I’ve reached a first milestone: all of the MiniUPnP libraries, daemons and clients are finally in Debian, in a shape which I am not ashamed anymore.

What is UPnP for?

Back in the days (around Linux 2.0.3x), the only way to share a single Internet connection was to run a Linux box as your gateway. These days are far gone, and nearly every home connection is done through a mini home router, which also often provides a small switch and a WiFi access point. Every computer on the LAN gets its connectivity through the router thanks to NAT, and to be more accurate, masquerading. That works very well for outbound connections (browse the web, send a mail, etc.), but it is then a lot harder that something connects to a “server” located in the LAN. If you run a permanent server with a fixed LAN IP address, that’s fine, you just configure your router to do some port forwarding, so that any inbound connection will be forwarded to the computer of your choice in the LAN. But if one application needs to listen on the public IP address just for a while (for example, IRC using DCC connection), so that “something” can connect to it, it becomes a way more tricky. That’s when UPnP comes into play. UPnP stands for Universal Plug and Play. It’s a protocol so that any application on the LAN can open ports on the public IP address of the gateway. Note that an UPnP router is also often referenced as UPnP IGD (Internet Gateway Device).

Why using MiniUPnP, don’t we have already linux-igd?

Before my packaging work, the only UPnP server that was available in Debian was linux-igd, which has a dependency on libupnpX. That library is quite big for what it does (about 500KiB, on top of which you have to install the 200KiB daemon itself), while MiniUPnP is designed to be lightweight (only 158K, with dependencies on iptables, iproute, uuid-runtime and libc6 only). That seems ridiculous, but when you have only 8MiB of flash in your old router, saving few hundreds KiB is really important, and here, we’re saving more than 500 KiB, with a full implementation of the UPnP protocol, and the Microsoft NATPmP implemented as well (yes, Microsoft decided that one protocol wasn’t enough, and that they should implement their own … (sic!)).

Who is using MiniUPNP?

Probably, you already have the libminiupnpc installed in your SID/wheezy desktop, because the transmission bittorent client which is installed with Gnome by default, depends on it, and minissdpd might be in your desktop too, if you installed the Recommends: packages as well. Which is why these are scoring quite high already in popcon. Previously, transmission was embedding its own version of the MiniUPnP client library, so it’s nice to have it as a standalone shared library now.

Also, because of its very light weight, and the fact that UPnP is often needed for gaming (both XBOX 360 and PSP need, and can use, the MiniUPnP daemon that runs on your IGD), many ISP who ship a router to their customer, install MiniUPnP in it. For example, at least, the 2nd and 3rd largest Internet access provider in France (Free Telecom and SFR) are shipping MiniUPnPd in their “box” (one of them using OpenWRT on their router). So probably as well, you are also using MiniUPnP daemon without knowing it.

But isn’t UPnP unsafe?

To this question, I’d answer that connecting to Internet is unsafe. It isn’t less safe than connecting to Internet on a WiFi hotspot for example. So adding UPnP support on your gateway doesn’t make it less safe, especially since MiniUPnP has some features to help with security (like an ACL with IP and port lists, or a secure mode were the router will only open ports to the client requesting it). Also, we are running a distribution where dependencies are easy to check, so it’s easy for you to check if a client uses UPnP or not, and decide to trust it (or not). I personally have no issue trusting transmission, x-chat or warzone2010 (more exactly: I don’t trust that incoming connectivity will make it less safe for these applications, there are so many other things that could go wrong).

MiniUPnP consists of what?

MiniUPnP is a set of 4 source packages to handle all of your UPnP needs:

– The client library, libminiupnpc (and the corresponding -dev package), with its sidekick miniupnpc client program (you can use it to check the public IP of your gateway, or manually open ports).

– The UPnP IGD daemon, miniupnpd, which just reached Debian Experimental. The latest version, 1.7-3, also includes debconf configuration so that you can choose the NIC and IP address to bind on. If you installed previous version, I strongly recommend upgrading to fix previous problems.

– The libnatpmp client library (and the -dev package) for implementing the Microsoft protocol as a client (it also has a client package binary for your tests).

– The minissdpd daemon to keep memory of all UPnP devices that announced themselves, and speed-up using the UPnP protocol (used by the MiniUPnP client library).

A long process to have MiniUPnP in Debian

All these 4 pieces of the puzzle were not as easy as it seems to be packaged in Debian. Especially, the IGD daemon, MiniUPnPd, used to depend on some header files that were not packaged in Debian. This was a decision of the iptables maintainers, which even after a long discussion, refused to add the necessary headers because it wasn’t supposed to be a kernel API. While this might really be truth, the issue was that there was no public API available in the Linux kernel at all, and other distributions (Fedora, Gentoo, etc.) really did supply these needed files. As a consequence, during a very long time (years… literally), shamefully, it was impossible to build the MiniUPnP daemon in Debian, while it was no problem on other distributions. Note that I do not want to start a troll discussion here, I’m just stating facts, both parties have very strong and respectable arguments. Anyway, this needed API was finally embedded in MiniUPnPd itself thanks to a contributor to the project, and now it builds on both the Squeeze and the Wheezy (that is Linux 3.2) kernel without any issue.

Call for testing

Unfortunately, I don’t have a Linux box as my Internet gateway. So I did the tests that I could: I installed MiniUPnPd on one computer on my LAN, and checked that I could open ports from another computer. So it seems to work, but I had no way to really test it “live”, with some “real” clients. So if you have an Debian SID box as your Internet gateway, feedback would be highly appreciated.

What is remaining to do?

A transition from libminiupnpc5 to libminiupnpc8 has to happen. Of course, this is too late for Wheezy, so this will happen right after the release. There’s not so many reverse dependencies, so it should be ok, but I (yet) have zero experience with transitions. BTW, I had hard time to find out what this transition process was, and maybe some more effort for documenting it would help others (I’m thinking about adding some text in the Debian wiki for that). Also, I have called for help on debian-devel, because I’m currently the only one maintaining all these 4 source packages, and I’m trying to have all my packages checked by others, and team maintained. So if you want to give a hand, or just declare yourself as a backup if I’m busy, please raise your hand! Upstream is a friend of mine, and is very reactive to any requests (it very rarely took more than 48 hours to get a reply).

Enjoy MiniUPnP and report bugs!

Again, feedback would be appreciated. :)

Debian 19th birthday at SHLUG Hacking Thursday

We were nearly 20 people from the Shanghai Linux User Group this evening, at JA cafe (near JinAn temple), celebrating Debian’s 19th birthday. As the only DD present at that event (Li Daobing was busy somewhere else), I was honored by my friends to be the one cutting the cake. Above the (very nice) cake before sharing it, below, a few SHLUG members present that evening.

New release of MLMMJ (version 1.2.18.0) uploaded in Debian

MLMMJ stands for Mailing List Manager Made Joyful. To me, it’s the best mailing list manager available. Not only its written in C (no slow interpreted language here…), but also it is easy and very convenient to setup. No hack is needed to make it run on a multi-domain environment, and no need to run a special subdomain for your lists (yes, I’m thinking about you, silly Mailman…). If you didn’t know, MLMMJ is used to handle relatively high traffic lists: all mailing lists for Gentoo are using MLMMJ for example.

MLMMJ 1.2.18.0 has just been release upstream, few days ago (on the 29 of may). It’s with a great pleasure that I packaged (and uploaded to SID) the latest version, and wrote the changelog, where I wrote that 6 Debian specific patches (out of 8) could be removed! One of the funny changes are the renaming of the “mlmmj-recieve” into “mlmmj-receive” as it always should have been (with a symlink to keep backward compatibility), and the removal of the .sh extension to the mlmmj-make-ml (which was patched in Debian to be kept policy compliant: that’s one of the removed patches…).

Even if I did a few quick functional tests before uploading, I’d be happy to get some feedback before Wheezy is out, so please test and eventually bug-report!

Unit tests for PHP: PHPUnit

PHPUnit, according to its PTS, has been in Debian since 2009. But it was orphaned, and nobody took care of it for a while. That is until few weeks ago, Luis Uribe started to work on it. Since he isn’t a DD, and that I take care of, I believe, about half of the all the PHP PEAR packages in Debian, I started working with him on re-doing the work of packaging PHPUnit for Debian. Previously, the old version was quite wrong, with missing dependencies, and not really working, what a shame…

PHPUnit 3.6 has been in Debian SID for 3 days now. And it’s working well. I’m now adding runs of unit testings from upstream packages at build time (of course, only if DEB_BUILD_OPTIONS doesn’t contains “nocheck”, as per the policy). All together, it’s been quite some fun to hack this, and I’m quite happy of the results, even though there’s still a lot of work remaining.

PHP itself is running unit tests at build time. And not just a few: more than 11000 of them. The only “small” issue, is that they are totally outdated. Over the time, the vardump() function has evolved, and doesn’t print things the same way. One may say that it prints things in a nicer way, but as a result, many of the tests that were suppose to work, actually fails because of these differences in vardump() over time.

So I started working on fixing some of the tests. It’s most of the time very easy to fix, but it takes a long time to fix all these small unit tests. So far, I’ve been able to fix most of what’s in the tests/, Zend/tests (more than 161 tests fixed), and the beginning of ext/*/tests: for the moment: bcmath, bz2, calendar, curl, date, dba, dom, ereg, and a part of exif, which for the moment represent 214 fixed tests, and I’ll try to do more fixes when I have time. I hope to send the patches upstream soon. The final goal is of course to have the build of PHP to fail if unit tests are failing as well. For the moment, unit tests do run at build time, but the build don’t care of the results, which I think is stupid (it’s wasting CPU cycles for no reasons, IMO).

 

If you maintain some PEAR packages, I would welcome you to first join the PKG-PEAR team on Alioth, and team maintain your packages, switch over to pkg-php-tools and dh 8 auto-sequencer, and to follow the PKG-PHP-PEAR packaging guide lines so that we have consistency in the archive. And of course, run unit testings, by doing a build-depends on phpunit (>= 3.6). Note that unit testings in PEAR packages are tracked on the Debian wiki. This post is also a call for help, because I feel quite alone doing this work of packaging PEAR packages, which many PHP applications depend on (think about roundcube, horde, extplorer, and many more). Other teams are doing very well, like the Perl team, and there’s no reason why Debian wouldn’t maintain PHP libraries as well as the ones for Perl.

Uploading a new Git repo to Alioth

Over the years, I’ve always uploaded my local git repo (git clone –bare myrep repo.git) as a tarball, which I then uncompress. But I’ve been tired of doing that by hand, so I wrote a tiny shell script for it. Nothing fancy. You just do:

alioth-new-git openstack

then it uploads your current repo to Alioth under /git/openstack, for example. It’s a really stupid script, please don’t point me to gbp-create-remote-repo, I know it exists, but I prefer my own tiny thing. Here’s the (lame) script:

#!/bin/sh

set -e

CWD=`pwd`
PKG_NAME=`basename ${CWD}`

usage () {
echo "$0 creates a new Git repository out of your current working tree on alioth."
echo "You must be at the root of that local git repository"
echo "$0 will use the current folder as the name for the Alioth git repository."
echo ""
echo "Usage: $0 <destination-project-path-on-alioth>"
echo "example: $0 openstack will create a /git/openstack/${PKG_NAME}.git repository"
echo "note that you will need to have write access on the destination project,"
echo "which means you must be a member of that said project on Alioth."
echo ""
echo "Please send patch/comments to: Thomas Goirand <zigo@debian.org>"
exit 1
}

if [ $# != 1 ] ; then
usage
fi

DEST_PROJECT=$1

# Create the tarball and upload it to Alioth
cd ..
echo "===> Cloning ${PKG_NAME} as bare: ${PKG_NAME}.git"
git clone --bare ${PKG_NAME} ${PKG_NAME}.git
echo "===> Building tarball: ${PKG_NAME}.git.tar.gz"
tar -czf ${PKG_NAME}.git.tar.gz ${PKG_NAME}.git
echo "===> Uploading ${PKG_NAME}.git.tar.gz to vasks.debian.org"
scp ${PKG_NAME}.git.tar.gz vasks.debian.org:

# Uncompress it on Alioth, fix perms and hook
# note that the below block should be put back in a single
# line, but has been broken into multiple lines for readability
# on this blog
ssh vasks.debian.org "cd /git/${DEST_PROJECT} &&
echo '===> Uncompressing ${PKG_NAME}.git.tar.gz in /git/${DEST_PROJECT}' &&
tar -xzf ~/${PKG_NAME}.git.tar.gz &&
echo '===> Activating update-server-info hook' &&
mv ${PKG_NAME}.git/hooks/post-update.sample ${PKG_NAME}.git/hooks/post-update &&
cd /git/${DEST_PROJECT}/${PKG_NAME}.git &&
git --bare update-server-info &&
echo '===> Deleting tarball on alioth' &&
rm ~/${PKG_NAME}.git.tar.gz &&
echo '===> Fxing g+w unix permissions' &&
find /git/${DEST_PROJECT}/${PKG_NAME}.git -exec chmod g+w {} \\;"

# Clean localy created files
echo "===> Cleaning local bare copy and tarball"
rm ${PKG_NAME}.git.tar.gz
rm -rf ${PKG_NAME}.git