Thursday, 5 January 2012

Project Management: Script everything; Test EVERYTHING; Graph everything

Last year I finished a two year project. I would love to tell, (brag) all about it but when I sign an NDA I mean it. The customer is a huge fan of opensource, and encouraged me to share as much as possible about my experience gained from the project, (while keeping an eye on the NDA.) I sent this entry to the client for approval and Q.E.D. they did.

So you have a project. You want to use reliable software, but spend no money on it and get the job done ahead of schedule. You are starting compleatly from scratch, with no support, (are you insane?)
I don't know what works for you, but my advice is as follows:

use dotProject

That is all I really wanted to share. If you are inside of a company, being nurtured with an existing support infrastructure, or if on day one you have an idea, a small laptop and no-money, my advice is the same. If you want a more wordy explanation for when starting with nothing, then read on, (you can delete the parts that you already have.)

Grab a copy of Linux, (I used Stackware and CentOS, but debian/ubuntu also works) and stick it on any computer, (I started out with a six year-old laptop and one aging 1U server - and I did not need the server for the first year!)
 When I did need a server I looked at co-location options, (I build my own server in 1998 and hosted it on my office network until 2005, so that was what I was used to.) The result of the search was, as is often the case, when you go looking for something, but keep an open mind; I found that I rather liked the VPS from Bytemark, (but if you have a dedicated server you can create a virtual machine using KVM.)

Install dotProjectRT: Request TrackerNotice, apache, exim, dovecot and munin to monitor the server. (This took me most of the first morning, but I was new to dotProject then.) Create, on the virtual machine, a new account for each of the developers and give them sudo access to exactly what they need. I like to add SELinux to the mix, lock down access with iptables, and create a pocket certificate-authority so that everything can be signed and where possible encrypted. (E.g. the ejabberd instance. The team used pidgin / finch / Adium to communicate more often than email.)

With dotProject you don't even need RT, (there is a simple ticketing system as a plug-in,) but when your project goes live you should have transitioned into RT for customer facing response. I use Notice as the glue that holds everything together, (so it is written in Perl.) Notice controls apache, exim, dovecot , NSD3, (and anything else like that) and manages all of my data: Which people are in each team; Project assets; CRM; reports; billing; general system administration, (though again, dotProject can manage all of the data for a project. The reason for Notice is "everything else".)

What did I learn from managing this project?

Script everything

# This project used
- 7zip
- bash
- C
- C++
- glusterFS
- javascript: [DOM, jQuery, JSON]
- mysql: master-master replication
- Perl: [DBIx::Class, Template::Toolkit, CPAN]
- svn: then later git
- windows-installer: NSIS
- wine
- x509
- other: probably, but this is all that comes to mind right now
# while consuming .ogg, C8H10N4O2, noodles and C2H5OH

So it was vital to be able to build the binaries with a single command. I looked at Apache Ant but in the end I was won over by Daniel Robbins (of Gentoo fame), and chose bash. This meant that at any moment it was possible to checkout a copy of the code and build it with a single command, (that involved extracting files from archives, applying changes, compressing, compiling more compressing and finally building the installer for each platform.)

Test EVERYTHING: not only the Chuck, but the bit and the whole drill!
(In joke - either you know it or you don't - its called a learning curve)

Thanks to the book, Perl Testing: A Developer's Notebook I fell in love with testing, (I don't expect everyone to become a testing freak - just their code.) With the cross-pollination of python, (use phpcs to see how the indentation gestapo have already spread there,) and what seems like the inevitable convergence of scripting languages directly analogous to car design, I expect that perl style testing will become as ubiquitous as a 5 star Euro NCAP rating before very long. (Talking of which, if there was an Internet Software Security and Quality Assessment Programme, like crossed with, then both quality and security would probably go up, but that is a talk for another day.) If you have not already worked it out, "Script everything" still applies to testing.

Graph everything: If it moves - graph it
(I think I read this quote in an entry by either Paul Graham, (RTML) or Brad Fitzpatrick, (livejournal) but I could be wrong)

For many years I had used cacti, (MRTG) and loved it. It was sometimes tricky to install, and learning how to do custom pages took me longer than I expected, (I'm not saying that it is hard, just that it too me too long.) Then I found Munin.

It was like the sun coming out from behind the clouds. I'm not saying that munin is better than cacti, I'm saying that munin does it for me; I fancy munin and I don't care who knows.
It just fit my needs exactly. I could easily write a new plugin, (if it did not already exist), and I can centralise multiple servers into one report, I can do it all from a bash prompt, (did I mention script everything?) Using ipmi I can even check the air-con in the data centre.

In conclusion, opensource software has matured to a level where it is practical to use it in a commercial environment. Just as any good version control system can contain its own development, any project can now be managed with free software, (and no amount of FUD or snake-oil-talk of ROI or TCO is going to save the proprietary dinosaurs now.)

1 comment:

  1. This is a very well-written blog. I read and liked the blog and have also bookmarked you. All the best for future endeavors.
    sap ps online training
    sap hana development online training


About this blog

Sort of a test blog... until it isn't