Pecking order and crane flogs.

I just read a interesting article about software teams and pecking order.

Even though it is said in the article that software teams aren't like sport teams or flocks, I want to share a view from one ice-hockey coach, Hannu Aravirta. He described his management style to be flying in V-formation, like flock of cranes. The person who is in the front of the formation is changed regularly to give breathing time for everyone. Even though in ice-hockey team (in this case, national team of Finland) there is a formal ranking given by the organization, everyone will take turns in the front of the flock. And everyone must lead the flock in the same direction.

In real world, the rotation is not random, though. The stronger birds might be in the front for a longer time (I'm not sure if this is true). This happens also in the context of software development. The flock, or team, will face variety of problems and issues. Some of those are technical, some are business related, and so on. So it should be the person who has the best capabilities to resolve the current issue who is in front of the flock, giving the direction where to go.

Like all analogies, this isn't complete. It gives an impression that the current leader of the flock will make all decisions alone, which is not the case.

Monitoring, part I: What and why, with example

Originally published at Sysart DevBlog, please comment there.

One important part of administrating servers is monitoring. Monitoring tells you when something goes wrong and why it went wrong, preferably before users notice the failure.

There's two somewhat distinct parts in monitoring, alerts and statistics. At higher level, alerts will notify operations staff when something is wrong, for example server stops responding. Then statistics gives you some ideas what went wrong.

We are currently using Icinga as a alerting tool and Munin as a statistics gatherer. Installation and configuration of these tools is little bit complicated, but by following documentation it is possible to have pretty decent monitoring system running in few hours.

Here's an actual case where monitoring proved to be essential. We had a situation where one server would stop responding after running few weeks and only way to resolve this was restarting server software. To make things worse, this particular system wasn't monitored at all and was running some business critical services. So first step was activating Icinga on that server so we could get alerts when the software stopped responding and after that, Munin was installed with pretty basic configuration.

After few weeks, following graphs were visible:

These graphs show information about network connections which were active. As it can clearly be seen from the "Netstat - by month", we had a pretty serious resource leak. We added a new alert to Icinga, which monitored the active connection count and sent an warning early enough so we could restart that server before resources were exhausted. The time window for restarting was pretty easy to deduct, because we could use graphs from Munin to see how fast the connection rate was going up and the upper limit for connections.

The actual reason for this is still somewhat a mystery, but it was tracked into one specific proxy server that one of our client used: it just didn't close connections properly. So we added an timeout for idle connections, which should have been there from the beginning. And currently monthly graph shows this:

Just try to guess when the fix was deployed.

Few quick observations from VR's new online shop

The Finnish railway company VR launched their new online shop for tickets, and it didn't go too well. As a part of lazy sunday morning, I did a quick sweep through the process of purchasing ticket using Google Chromes developer tools. I didn't actually finish the process, though.

First thing I noticed was tremendous amount of requests that the initial load generates, as it was more than 50. For comparison, loading the front page of google and executing a search generates around 20 requests. A lot of these requests were different css -files and small images. At least those images would have been easy to combine into css -sprites, and some of those css files could have been combined, too.

Second simple thing missing is compression of responses. Again, simple explanation.

Third low hanging fruit would be usage of CDN for JQuery. There might be one problem, as they are using pretty old version of JQuery (I spotted 1.4.2 in one place and 1.3.2 in another, release date of previous was February 19th, 2010) and it might not be available on public CDNs (I didn't check this, lazy sunday as it is).

The backend seems to be implemented in Java and Apache Struts. It is hard to say anything else about the backend without using a lot more time.

Launching a web service as big as this is always a challenge, and I don't know if implementing these small things would have prevented the problems. It seems that most of the page loading time is spent at the backend, so the improvement on the frontend might not have any impact at all. If the application is partitioned into front- and backend, that is.

By the way, at the moment, the online shop is in maintenance mode.

8th Annual VTT SOFTWARE ENGINEERING SEMINAR - New Thinking - Better Software Engineering - Great Experience

Originally published at http://www.sysart.fi/news/64/58/8th-Annual-VTT-software-engineer-seminar/d,devblogi, please comment there.

Lean, transparency, global software development.... in the cloud.

VTT and leading companies challenge you to learn and discuss about productive ways of doing things in changing environment. Topics include innovation, security, Cloud technologies, lean approach towards efficiency and other software development methods and tools.

The general theme seemed to be transparency and openness in development and innovation. In the keynote, Elektrobits Jari Partanen talked a lot about transparency in the context of process improvement, saying that transparency is key enabler to identify what are the opportunities for improvement for lean and agile approaches - "Transparency attracts". He also emphasized that having a wiki is not enough for information sharing, there must be some activities like workshops and so on.

Another good point in the opening keynote, at least from my point of view, was the comment about how Elektrobit doesn't want to stress word Lean, they wan't to emphasize their "Way of Working", because it more attractive to all of the organizational entities. It is true that Lean, just like agile, is being considered as a thing that development does. But this leads easily into situation where process optimization is on the local level, when you should optimize the whole.

Transparency theme continued in the sessions which talked about innovation. Innovation should be open and transparent on all organizational levels and it should be customer oriented, but usually there isn't any communication channels between different organization levels and then the innovations from customers tend to disappear. So there should be an open, social system supporting innovation. Even though social is hype at the moment, in this context it is interesting. As lean methods say, the improvement ideas should come from the actual workers, and here they mean the support persons, developers and marketing people who will see what the customer wants and how the customer uses the product.

Rest of the sessions were about lean, distributed development, security and user experience. Some of these suffered a little bit from hype, as there was few times when the presenter said when talking about some basic software develepoment, almost as an afterthought, "in the cloud". That term is seriously overused and overloaded. It is funny when some people use cloud to describe how application can interact with each other and form mashups, and some people talk about technical characteristics, like provisioning, scaling, availability etc. But that seems to be how it works.

One session caught my interest just because it had a great name, "Flow-based software development on epic level". But the session was somewhat a disappointment, the main idea can be said in one sentence, "Global development queue of prioritized cross-product epics are assigned to teams eliminating product silos". The problem with this is the lack of domain knowledge; when every team works with every product, no one understands the domains. It might work if all products are on same domain, like in F-Secure, where the presenter, Arto Saari, was working, but I have my doubts. And F-Secure hasn't implemented this at full scale, yet.

Few one liners from different sessions:

• "EB doesn't want to stress word Lean they want to emphasize their "Way of working" - it is more and more attracting all the organizational entities" - Jari Partanen, Elektrobit
• "Transparency is key enabler to identify what are the opportunities for improvement for lean and agile approaches - "Transparency attracts"" -Jari Partanen, Elektrobut "
• Success is increasingly determined by the ability to embrace the potential of innovation ecosystems" - Timo Koivumäki, VTT

Linux - the Ultimate Agile Project?

In a recent article at Network World, the executive director of Linux Foundation, Jim Zemlin,says "that Linux is a success because the people building it don't plan." There's also a mention about "Linus Torvalds told attendees at LinuxCon in August that he can't predict what's next and doesn't think of himself as a visionary kind of guy. He is focused on the changes to the next release and perhaps the one after that."

That sound pretty much like a iterative development cycle.

Rationality and Social acceptance

I've just listened an interview of Bob Lutz, who wrote "Car Guys vs. Bean Counters: The Battle for the Soul of American Business" (haven't got that book, though). One point caught my attention. Lutz said something like "Customers always tell you the rational, socially acceptable reasons when doing customer research with interviews". In the context of car manufacturing, this means that most wanted features in cars are reliability and safety. But when the buying decision is made, it is the irrational reasons that matter.

So you could say that customers doesn't know what they want, and most of customer research by interviews or requirement gathering might be waste. Customers and end users just say what they think is the norm in the domain, they don't think what they really need. They only give the rational and socially acceptable requirements. So what is needed is deeper research on what users really need, interviews or planning meetings aren't enough, and might actually be harmful. 

Ignorance: The ultimate expense

James M. Morgan, Jefrey K. Liker, The Toyota Product Development System, p.213:

Many companies view paying for employees to learn as an unnecessary cost. Anything that is not tactical or does not produce immediate, measurable results is "fluff, and those who endorse opposing view are not considered serious-minded, action-oriented business people. But the absence of deep technical understanding drives a "more is better" philosophy, which leads to more elaborate gauges, more reviews, more audits, more inspections, and more checkpoints (because it is always "safe" to check and check again).

And after few lines where they talk about different quality initiatives like ISO, Six Sigma:

On the surface, these quality initiatives demonstrate a "commitment to quality" and make people feel that they have accomplished something. In truth, however, they fail miserably in achieving those things that guide an intelligent and balanced approach to fostering quality in a PD process: the time, dedication, and hard work required for a deep technical understanding of that process.

First, I'd like to add documentation into those "more is better" things. Same goes with different certifications and process models.

All of those things are used when those who doesn't have deep technical understanding try to control development process. They try to enforce some "best practices" they've seen in some publications without understanding them correctly and just think that "we must do this because everyone does this". There's at least two problems with this:

1. If you only follow best practices, you're only doing what those who developed did few years a go.
2. "Best practice" might not be the best in your context.

Maybe the root cause for all of this is that we tend to be lazy and try to take shortcuts when building our own ways of working. But there is no shortcuts. Yes, you can and should follow what others are doing. But you should find out why and in what context they are doing those things.