Aborting export of VMs with Ovirt 3.1

I was going to upgrade Ovirt from 3.1 to 3.2, and in prepration, I started to export VMs. I thought that I had enough space on export -domain, but it seems that the thinly provisioned disks will use all the space they've been assigned. So I needed to abort exporting.

Exporting locks the images, so I was a little bit afraid that aborting export with good old 'kill' would leave them into that state. After finding http://www.mail-archive.com/users@ovirt.org/msg08588.html, I decided to bit the bullet and just kill the processes. Ovirt actually worked very well, and all images became unlocked automatically. Hooray for proper error handling.

Try at your own risk, though.

trivial-rebase.py in Gerrit 2.6, fixing "trivial_rebase.py: error: Incomplete arguments"

Trivial rebase detection had some changes when Gerrit 2.6.0 was released, and my previous patchsets-created hook didn't work anymore. In logs, there was

[2013-06-25 12:05:33,916] INFO com.google.gerrit.common.ChangeHookRunner : hook[patchset-created] output: trivial_rebase.py: error: Incomplete arguments

To fix this, I modified my patchset-created hook to following (original came from some discussion list, and I was extremely stupid and didn't write down where I got it :( ):

#!/usr/bin/perl

use strict;
use warnings;
use Getopt::Long;

my $gerritHome="/home/gerrit2";
my $gerritRoot=$gerritHome."/review_site";
my $hookDir=$gerritHome."/scripts";

my $change;
my $project;
my $commit;
my $patchset;
my $changeurl;
# Not really needed, but without these error_log will have entries like
# INFO com.google.gerrit.common.ChangeHookRunner : hook[patchset-created] output: Unknown option:
my $branch;
my $uploader;
my $isdraft;

my $result = GetOptions("change=s" => \$change,
"project=s" => \$project,
"branch=s" => \$branch,
"commit=s" => \$commit,
"patchset=s" => \$patchset,
"change-url=s" => \$changeurl,
"is-draft=s" => \$isdraft,
"uploader=s" => \$uploader);

system($hookDir."/trivial_rebase.py",
"--project", $project,
"--commit", $commit,
"--patchset", $patchset,
"--change-url", $changeurl,
"--private-key-path", $gerritRoot."/etc/ssh_host_dsa_key");

To use this, trivial_rebase.py must be in /home/gerrit2/scripts/trivial_rebase.py and it must be executable by gerrit2 user.

Git subtrees and contributing upstream

I needed Puppetlabs Apache -module, so I added it into my vagrant training repository with following command

git-subtree add --squash --prefix=puppet/modules/apache https://github.com/puppetlabs/puppetlabs-apache.git master

As I started to use this module, I spotted a minor commenting error: one class parameter was removed, but it was still
used in comments. This seemed to be a good place to train to contrinute into upstream repository using github workflow.

First I cloned the puppetlabs/puppetlabs-apache in github, and then I tried to pull it as subtree

git subtree pull --prefix=puppet/modules/apache --squash https://github.com/jyrkiput/puppetlabs-apache.git

Didn't actually do anything, as puppetlabs/puppetlabs-apache.git and jyrkiput/puppetlabs-apache.git were both at same commit. So I'm not sure if this was actually needed.

I did just a minor fix, git add -u and git commit, then

git subtree push --prefix=puppet/modules/apache git@github.com:jyrkiput/puppetlabs-apache.git #21262

Some counting happened, and then the commit was transferred to github. Branch name #21262 is ticket number from Puppet Labs issue tracker.

Then I made a pull request to puppetlabs-apache, which got merged in few days. When I got a message telling me that the pull request had been merged, I wanted to see how I could pull changes back. So I tried

git-subtree pull --prefix=puppet/modules/apache https://github.com/puppetlabs/puppetlabs-apache.git master

But this caused a lot of conflict. But adding --squash removed those.

git-subtree pull --squash --prefix=puppet/modules/apache https://github.com/puppetlabs/puppetlabs-apache.git master

I'm don't know if usage of --squash was a mistake, or should I have gotten the whole history? Anyway, I managed to go contribute into upstream project using subtrees.

Converting Git submodules to subtrees

While learning to use Vagrant, I've been using git submodules. But git subtrees seems to be a lot better choice, so I wanted to start using them.

So, in my repository, which can be found from https://github.com/jyrkiput/vagrant-jenkins-gerrit, I had following .gitmodules:

[submodule "puppet/modules/jenkins"]
path = puppet/modules/jenkins
url = https://github.com/rtyler/puppet-jenkins.git
[submodule "puppet/modules/java"]
path = puppet/modules/java
url = https://github.com/puppetlabs/puppetlabs-java.git
[submodule "puppet/modules/stdlib"]
path = puppet/modules/stdlib
url = https://github.com/puppetlabs/puppetlabs-stdlib.git
[submodule "puppet/modules/firewall"]
path = puppet/modules/firewall
url = https://github.com/puppetlabs/puppetlabs-firewall.git
[submodule "puppet/modules/apt"]
path = puppet/modules/apt
url = https://github.com/puppetlabs/puppetlabs-apt.git

And with Git 1.8.3.1 installed from with Homebrew (http://brew.sh/).

Unregister all submodules with git submodule deinit

git submodule deinit .

This removes submodule configurations, so each module must be removed. So for each module

git rm puppet/modules/[modulename]

ie.

git rm puppet/modules/jenkins

Commit your changes

git commit -m "Removed submodules"

Add modules as subtrees, so for each module

git-subtree add --prefix=path/to/subtree --squash repository master

ie.

git-subtree add --prefix=puppet/modules/jenkins --squash https://github.com/jenkinsci/puppet-jenkins master

git-subtree will make a commit when a subtree is added, so these are now in your repository

Finally remove .gitmodules for cleaning up things.

The man page of git subtree was really helpful when finding out what to do, and it contains a lot more information.

Pecking order and crane flogs.

I just read a interesting article about software teams and pecking order.

Even though it is said in the article that software teams aren't like sport teams or flocks, I want to share a view from one ice-hockey coach, Hannu Aravirta. He described his management style to be flying in V-formation, like flock of cranes. The person who is in the front of the formation is changed regularly to give breathing time for everyone. Even though in ice-hockey team (in this case, national team of Finland) there is a formal ranking given by the organization, everyone will take turns in the front of the flock. And everyone must lead the flock in the same direction.

In real world, the rotation is not random, though. The stronger birds might be in the front for a longer time (I'm not sure if this is true). This happens also in the context of software development. The flock, or team, will face variety of problems and issues. Some of those are technical, some are business related, and so on. So it should be the person who has the best capabilities to resolve the current issue who is in front of the flock, giving the direction where to go.

Like all analogies, this isn't complete. It gives an impression that the current leader of the flock will make all decisions alone, which is not the case.

Monitoring, part I: What and why, with example

Originally published at Sysart DevBlog, please comment there.

One important part of administrating servers is monitoring. Monitoring tells you when something goes wrong and why it went wrong, preferably before users notice the failure.

There's two somewhat distinct parts in monitoring, alerts and statistics. At higher level, alerts will notify operations staff when something is wrong, for example server stops responding. Then statistics gives you some ideas what went wrong.

We are currently using Icinga as a alerting tool and Munin as a statistics gatherer. Installation and configuration of these tools is little bit complicated, but by following documentation it is possible to have pretty decent monitoring system running in few hours.

Here's an actual case where monitoring proved to be essential. We had a situation where one server would stop responding after running few weeks and only way to resolve this was restarting server software. To make things worse, this particular system wasn't monitored at all and was running some business critical services. So first step was activating Icinga on that server so we could get alerts when the software stopped responding and after that, Munin was installed with pretty basic configuration.

After few weeks, following graphs were visible:

These graphs show information about network connections which were active. As it can clearly be seen from the "Netstat - by month", we had a pretty serious resource leak. We added a new alert to Icinga, which monitored the active connection count and sent an warning early enough so we could restart that server before resources were exhausted. The time window for restarting was pretty easy to deduct, because we could use graphs from Munin to see how fast the connection rate was going up and the upper limit for connections.

The actual reason for this is still somewhat a mystery, but it was tracked into one specific proxy server that one of our client used: it just didn't close connections properly. So we added an timeout for idle connections, which should have been there from the beginning. And currently monthly graph shows this:

Just try to guess when the fix was deployed.

Few quick observations from VR's new online shop

The Finnish railway company VR launched their new online shop for tickets, and it didn't go too well. As a part of lazy sunday morning, I did a quick sweep through the process of purchasing ticket using Google Chromes developer tools. I didn't actually finish the process, though.

First thing I noticed was tremendous amount of requests that the initial load generates, as it was more than 50. For comparison, loading the front page of google and executing a search generates around 20 requests. A lot of these requests were different css -files and small images. At least those images would have been easy to combine into css -sprites, and some of those css files could have been combined, too.

Second simple thing missing is compression of responses. Again, simple explanation.

Third low hanging fruit would be usage of CDN for JQuery. There might be one problem, as they are using pretty old version of JQuery (I spotted 1.4.2 in one place and 1.3.2 in another, release date of previous was February 19th, 2010) and it might not be available on public CDNs (I didn't check this, lazy sunday as it is).

The backend seems to be implemented in Java and Apache Struts. It is hard to say anything else about the backend without using a lot more time.

Launching a web service as big as this is always a challenge, and I don't know if implementing these small things would have prevented the problems. It seems that most of the page loading time is spent at the backend, so the improvement on the frontend might not have any impact at all. If the application is partitioned into front- and backend, that is.

By the way, at the moment, the online shop is in maintenance mode.