Monday, 30 January 2012

Numberphile

If you are like me then you will like http://www.youtube.com/user/Vihart (squee! ♥)

and you will have seen Numberphile... which so far has not been as good as The Singing Banana, until today! (So I declare!)

For me today's episode of numberphile is as cool as I had been hoping they would be.
( I hope that they have just got into their stride.)

The previous episodes have been, for me, between so-so and hmmm, (so worth seeing, but not worth a thumbs up [0].)

If I had to do an episode then mine would be 15 and I would talk about black-hole numbers [1].

I wonder if they would let me pick 1234, but then talk about 1,2,3,4 and trying to create every number from [1..100] just using each of those digits once. i.e.

1+2+3+4   =10 (99 left to find),
(1+2)(3+4)=14  (98) left to find.

Many years ago my friend Shevek wrote, (in 3194 bytes of perl) a program that searched for all possibilities. Before that my friend Nico and I had found most of them by hand.

(If you get stuck then you can ask me for help.)

[0] It would be nice to have some granularity. Did I like the production; Did I like the relevance to what I was looking for? (Come on 'The Internet' not all of us are undergrads ;-)

[1] Recurse through { The sum of the divisors of a natural number } => 15

Sunday, 29 January 2012

Monkey httpd

This weekend I tried monkey httpd. The short version: It rocks.
I had a little trouble getting it to server CGI (.php .pl .cgi .py .rb .bash) because the server that came with it, (written in python - bless their cotton socks) did not work for me.

If you want to use the script that I hacked together then feel free:

https://github.com/alexxroche/AIF/blob/master/scripts/perl/monkey-palm_0.1.pl

The big gotcha for me was "\r\n\r\n" rather than "\n\n". But once I tracked that down it was, (much like that reset of monkey-project) just as I expected, with everything where it should be and even painted the right colours!

I'm off to hack the auth plugin to make it read .htaccess and then I'm going to "uninstall", (yes, remove) apache.

(Then I'm going to port my little perl Net::Server script to native c.)

Friday, 27 January 2012

Buy my stuff

For those of you that would like to own an original design based on one of the many great quotes in my blog, (or my life): Now you can.

Munin about logarithmically

As you all know, I like to graph, (OK, being honest - I need to graph.) So when I create a nifty new plug-in to monitor the charge in my laptop battery, (acpi_volt) I was feeling rather smug. It uses

 acpi -i

to find out two pieces of information. The first is how charged the battery is. (Somewhere between 0 and 100 percent.) The second, (and this is the science bit) is the design capacity in mAh, and the mAh at last full capacity. This is cool. We get two lines, one showing how much charge is left in the battery and another showing how much milk is left in the bottle. (I know that my laptop battery should not suffer from any memory effect or milk-bottle effect, but it does. I don't know what chemical fingers are responsible, but over the past three years the length of time that I can run it off the battery has dropped from about 90 minutes to about 20.)

So I click on the graph to see the day,week,month,year view and *gasp* no month graph.
I console myself that sometimes it takes a while for munin to have enough data in the rrd files to bother creating the month and year graphs. So I go to bed. The next day it has managed to create a year graph, but still no month graph.

I smile. I have been a sysadmin.
- this I know how to solve.

I start with the usual suspects, (file/dir permissions in /var/lib/munin and other munin locations specified in /etc/munin/munin.conf )

Nothing. I check that I can collect the data remotely, and that the acpi_volt.rrd files have valid data in them, (they do.)

hmm - I have never got this far with a munin problem. I guess we will have to go deeper.
su -s /bin/bash munin
munin@laptop:~$ /usr/share/munin/munin-graph --service acpi_volt --nolazy --list-images --month --noyear --noday --noweek

produces the expected output
/var/www/munin/localdomain/localhost.localdomain-acpi_volt-month.png

but it lies!
ls -la ls -la /var/www/munin/localdomain/localhost.localdomain-acpi_volt-month.png
ls: cannot access /var/www/munin/localdomain/localhost.localdomain-acpi_volt-month.png: No such file or directory

ok. how about adding --debug to the end of that
DEBUG: Expanding specials "volt0","volt1","".
DEBUG: Checking field lengths "volt0","volt1","".
DEBUG: Treating fields "volt0","volt1","".
DEBUG: Processing field "volt0".
DEBUG: single_value: Checking field "ARRAY(0x8a0ea38)".
DEBUG: Drawing field "volt0".
DEBUG: Processing field "volt1".
DEBUG: Drawing field "volt1".


rrdtool "graph" "--font"
"LEGEND:7:monospace"
"--font"
"UNIT:7:/usr/share/munin/VeraMono.ttf"
"--font"
"AXIS:7:/usr/share/munin/VeraMono.ttf"
"/var/www/munin/localdomain/localhost.localdomain-acpi_volt-month.png"
"--title"
"Voltages - by month"
"--start"
"-33d"
"--base"
"1000"
"--logarithmic"
"--vertical-label"
"Volt"
"--height"
"175"
"--width"
"400"
"--imgformat"
"PNG"
"DEF:gvolt0=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt0-g.rrd:42:AVERAGE"
"DEF:ivolt0=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt0-g.rrd:42:MIN"
"DEF:avolt0=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt0-g.rrd:42:MAX"
"CDEF:cvolt0=gvolt0"
"COMMENT: "
"COMMENT: Cur\:"
"COMMENT:Min\:"
"COMMENT:Avg\:"
"COMMENT:Max\: \j"
"LINE1.6:gvolt0#22ff22:Full "
"GPRINT:cvolt0:LAST:%6.2lf%s"
"GPRINT:ivolt0:MIN:%6.2lf%s"
"GPRINT:gvolt0:AVERAGE:%6.2lf%s"
"GPRINT:avolt0:MAX:%6.2lf%s\j"

"DEF:gvolt1=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt1-g.rrd:42:AVERAGE"
"DEF:ivolt1=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt1-g.rrd:42:MIN"
"DEF:avolt1=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt1-g.rrd:42:MAX"
"CDEF:cvolt1=gvolt1"
"LINE1.6:gvolt1#0022ff:capacity "
"GPRINT:cvolt1:LAST:%6.2lf%s"
"GPRINT:ivolt1:MIN:%6.2lf%s"
"GPRINT:gvolt1:AVERAGE:%6.2lf%s"
"GPRINT:avolt1:MAX:%6.2lf%s\j"
"COMMENT:Last update\: Fri Jan 27 16\:45\:03 2012\r"
"--end"
"1327672800"
hmm, that all looks ok. Lets take a closer look at the data:

rrdtool info /var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt0-g.rrd
rrdtool info /var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt1-g.rrd
rrd_version = "0003"
step = 300
last_update = 1327685404
ds[42].type = "GAUGE"
ds[42].minimal_heartbeat = 600
ds[42].min = NaN
ds[42].max = NaN
ds[42].last_ds = "100"
ds[42].value = 4.0290790000e+02
ds[42].unknown_sec = 0
rra[0].cf = "AVERAGE"
rra[0].rows = 576
rra[0].cur_row = 88
rra[0].pdp_per_row = 1
rra[0].xff = 5.0000000000e-01
rra[0].cdp_prep[0].value = NaN
rra[0].cdp_prep[0].unknown_datapoints = 0
rra[1].cf = "MIN"
rra[1].rows = 576
rra[1].cur_row = 197
rra[1].pdp_per_row = 1
rra[1].xff = 5.0000000000e-01
rra[1].cdp_prep[0].value = NaN
rra[1].cdp_prep[0].unknown_datapoints = 0
rra[2].cf = "MAX"
rra[2].rows = 576
rra[2].cur_row = 509
rra[2].pdp_per_row = 1
rra[2].xff = 5.0000000000e-01
rra[2].cdp_prep[0].value = NaN
rra[2].cdp_prep[0].unknown_datapoints = 0
...
yup that all looks good, (to me). So lets graph by hand:

rrdtool graphv /var/www/munin/localdomain/localhost.localdomain-acpi_volt-month.png \
-W "Alexx Roche" --end now --start end-33d \
DEF:gvolt0=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt0-g.rrd:42:AVERAGE \
DEF:ivolt0=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt0-g.rrd:42:MIN \
DEF:avolt0=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt0-g.rrd:42:MAX \
CDEF:cvolt0=gvolt0 \
DEF:gvolt1=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt1-g.rrd:42:AVERAGE \
DEF:ivolt1=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt1-g.rrd:42:MIN \
DEF:avolt1=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt1-g.rrd:42:MAX \
LINE1.6:gvolt0#22ff22:Full LINE1.6:gvolt1#0022ff:capacity

yup, that works, (but does not look pretty.) So lets keep adding configurations.


rrdtool graphv /var/www/munin/localdomain/localhost.localdomain-acpi_volt-month.png \
--title "Voltages - by month" \
--end now --start end-33d \
DEF:gvolt0=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt0-g.rrd:42:AVERAGE \
DEF:ivolt0=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt0-g.rrd:42:MIN \
DEF:avolt0=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt0-g.rrd:42:MAX \
CDEF:cvolt0=gvolt0 \
DEF:gvolt1=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt1-g.rrd:42:AVERAGE \
DEF:ivolt1=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt1-g.rrd:42:MIN \
DEF:avolt1=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt1-g.rrd:42:MAX \
CDEF:cvolt1=gvolt1 \
--font LEGEND:7:monospace \
--font UNIT:7:/usr/share/munin/VeraMono.ttf \
--font AXIS:7:/usr/share/munin/VeraMono.ttf \
--base 1000 \
--vertical-label "Volt" \
--height 175 \
--width 400 \
--imgformat PNG \
COMMENT:" " \
COMMENT:" Cur\:" \
COMMENT:"Min\:" \
COMMENT:"Avg\:" \
COMMENT:"Max\: \j" \
LINE1.6:gvolt0#22ff22:Full  \
GPRINT:cvolt0:LAST:%6.2lf%s \
GPRINT:ivolt0:MIN:%6.2lf%s \
GPRINT:gvolt0:AVERAGE:%6.2lf%s \
GPRINT:"avolt0:MAX:%6.2lf%s\j" \
LINE1.6:gvolt1#0022ff:capacity  \
GPRINT:cvolt1:LAST:%6.2lf%s \
GPRINT:ivolt1:MIN:%6.2lf%s \
GPRINT:gvolt1:AVERAGE:%6.2lf%s \
GPRINT:"avolt1:MAX:%6.2lf%s\j" \
"COMMENT:Last update (by LxR)\: Fri Jan 27 16\:45\:03 2012\r"

ah ha! --logarithmic seems to break it. Just for the monthly graph? (Some may ask why I was graphic logarithmically rational numbers between 0..100 but that is not the point.)
 I do not see why munin would not alert or at least log, (somewhere in /var/log/munin/) something to mention that the graphing did not happen, (and even more oddly, for just one of the four graphs.)

So I changed acpi_volt to graph_args => '-1 0' and now I have all four... but I still do not know why rrdtool refused to graph. (Part of me want to keep the .rrd so that others can test it, but I'm not sure how much help that would be.)

Thursday, 26 January 2012

Grandma Nazi

I am not a grammar Nazi. That would be like Stephen Fry applying to join the SS in 1941's Germany.

Everyone is somewhere on the line from zero to Himmler when it come to various typographic and logical errors. The one I want to mention, (though I hear it more often than read it) is:

"Very unique"

Please stop trying to define degrees of uniqueness; Either something is unique or it isn't.

Sunday, 15 January 2012

Match-making Age Ranges

Many years ago I was told that, "Half your age plus seven years is the girlfriend limit."

This immediately seemed both sexist and mathematically incompetent, but just as "Feed a cold, starve a fever" isn't medical advice, (it is reassurance for young mothers who are panicking that their feverish child isn't eating,) it is just a guide-line, ( or in that case -  a throw-away anecdote at some debutante party in London.)

What about those looking to date someone older? This guide can easily be reversed as, "Your age minus seven, doubled."

Mathematically, (where x is your age),

        lower_limit = (x/2)+7
     upper_limit = 2(x-7)

Some time later I was looking for mathematically significant differences between two numbers. This is an area of statistics where arguments are had over tea.

I found, on one rather well written site: (I did not bookmark it, sorry.)

      ( ( x - y ) / 2 ) ^2  

which might not look like much but I rather liked it. (Here y is "their age" for lower limit, and ^2 means "to the power of two" i.e. squared.)


It took a while, but eventually I made the connection between these two and the obvious next move was to graph them. I had already found, with a quick script, the intersection at 34 and 54 by hand, (once I noticed that the output of the two functions was not parallel, I had to search,) but somehow I had missed the one at 49, (see! this is why graphs are cool.)


(I was going to add a vertical line marking the age of consent, but that is a topic for another post.)

I have added some sudo-random labels on the graph, using gimp, so that I could refer to particular areas more easily, (see I'm thinking of you.)

The first thing that is worth noting is that the cougar limit, [yes, I typoed in the graph] and the sugar limit are not both straight lines. This is not due to some form of ageism or sexism, but is due to the use of an odd number in "half your age plus". If it was half your age plus six then they would both be straight.

We can see stepping in the two stats lines, (marked mathematically lower and upper). So the question there becomes, why are they stepping at 18, 24, 32, 40, 50, 60. [ I know that I have three errors in the graph: msl{18,32,50} should all be +1 ].
Are these square numbers? Is there something about their factors? How are they related?

The thing I like here is that when you take the output of the two simple equations that is nearest to x, (your age) you end up with a rectification of this old match-makers guide.
If you are a parent then you can explain to your daughter why, when she is sixteen, a twenty year-old boyfriend should only happen in porno-land, (and other fictitious realms.)

So who ends up in the perpetuity zone? If you imagine having a crush on an older film star, (say forty years-old) when you were younger, (e.g. twenty). By the time you are fifty they would be seventy. If you happened to be compatible when you meant thirty years later, they would be in the perpetuity zone, and in this case I could see people making exceptions, (humans being social creatures.)

Despite my choice of labels, I would recommend that everyone stay within the cradle-snatching:Tapping the Grave areas.



Technical details
Data generated using a bash script and bc.
Graph generated using cairo, Chart::Clicker and a perl script.

( The functionality of the bash script could be extended to calculate how long you would have to wait until someone entered the perpetuity zone. )

Saturday, 14 January 2012

Email Archive; How do you do yours?

If you are anything like me, you have an ever growing pile of email. It sits on a server that does have IMAP4 access and you might even check it using a client like Thunderbird, but for the most part you just use the webmail interface. Since Roundcube and gmail the email client has rather had its day... or has it.

Why not install and fire up a copy of Thunderbird, (it is free). Well I'll tell you why not, (or rather why.) You can use the client to make a backup of all the messages that you received and sent. (If Google goes bust for trying to bind G+ into your searches, you will be thankful to have a backup! Remember: Nothing is forever.)
If you got a new computer in the last six months you will have oodles of disk space; Why not use some of it for something useful?

Remember than IMAP just "looks" at the messages on the server. You have to copy them into "Local Folders". Then you can safely go online and delete any of your old messages. Please make sure that you have a local copy, (heck, with the price of consumer disk-space make two!) before you delete any messages.

If this blog were a film, and you had been reading the subtitles, you will know that I'm a fan of fire-and-forget automation using scripting. So how do I do it?

If you are not using a computer that has UNIX or Linux installed then this may be as far as you need to go today, (if ~/backups/" is gibberish to you then you need to find a good  UNIX / Linux Terminal Tutorial.)

In 2005 I used http://freshmeat.net/projects/imapsync (now called http://freecode.com/projects/imapsync). That was a great project, but now it is a great product.. so it is no longer free (though https://github.com/imapsync/imapsync seems free.) So you look for something else and find mbsync http://isync.sourceforge.net/mbsync.html - maybe a little harder to understand at first, so here is how I use it, (with a cut-n-paste example config):


Install mbsync:

aptitude install isync

(if you are on debian/Ubuntu)

cron + wrapper_script + mbsync (I used to use imapsync, as I mentioned).

For this example we are going to backup two gmail accounts into ~/backups/gmail/

Configure

Get certificates

We want an encrypted connection, (remember that (since 2008) SSL/TLS will only protect you against casual network snooping!) so we willl have to collect the remote servers certificates


     mkdir -p ~/backups/gmail/; 
  openssl s_client -connect imap.gmail.com:993 -showcerts > ~/backups/gmail/certs.pem </dev/null

Then we have to write a config file called .mbsync and put it in the root of our home directory:

~/.mbsync

#example
MaildirStore local

# where do you want to keep the messages?
Path ~/backups/gmail/

# I think of this as the details for one remote account
IMAPStore just.a.label
Host imap.gmail.com

User test-example@gmail.com
Pass notVERYsecure

UseIMAPS yes
CertificateFile /usr/share/purple/ca-certs/thoughtcrime_CA.pem
CertificateFile ~/backups/gmail/certs.pem

# You can have the details for as many accounts as you like

IMAPStore work.email
Host imap.gmail.com

User test.work@gmail.com
Pass 4l50notVERYsecure
UseIMAPS yes
CertificateFile /usr/share/purple/ca-certs/Thawte_Premium_Server_CA.pem
CertificateFile ~/backups/gmail/certs.pem

# Now the "backup instructions"


Channel my.email
Master ":just.a.label:[Gmail]/All Mail"
Slave :local:test-example

Sync PullNew
Create Slave
SyncState *

# The "Channel" is used when invoking mbsync to tell it which Channel to "watch"
# Master tells it which IMAPStore to look at, (in this case look for the one called "just.a.label"
# Slave is where to put it, (this _can_ be a remote IMAP server! Cool for migrations)
# the last three are more example settings. Check the man for more.

# A Channel can pull from multiple accounts or server at the same time!

Channel both                                      # 
Master ":just.a.label:[Gmail]/All Mail"  # from
Master ":work.email:[Gmail]/All Mail"  # from
Slave :local:combined                        # to
Sync PullNew                                    # how
Create Slave                                      # do
SyncState *                                       # what

#End .mbsync config file



Add a crontab entry

58 */8 * * *  echo 'yes'|mbsync -q both 2>/dev/null 
#I know I should listen for errors but it warns about certs

The "echo 'yes'" hacks us past an ssl warning. There is probably a better way to solve this problem, but this worked for me.

But what of the wrapper_script that you mentioned?

Well by calling a wrapper script, (in my case perl) it is trivial to change the path based on the date:

s%^Path ~/backups/gmail/%Path ~/backups/gmail/`echo -n \$(date +%Y)`/%

that way each year gets a separate backup. I do end up with some duplication while I delete the old messages on the server, but duplicates are better than lost data, (and see fdupes.) 

Recovery

There isn't much point in having a backup if you can't use it. I did used to just use grep to search but I found it faster to index ~/.backup/gmail using Thunderbird and then search for messages within that.. which brings me full circle back to Thunderbird. Thank you Mozilla. Keep up the good work.

[update] I've moved to balsa as a client and that can read the Maildir++ format without having to add an IMAP4 server. I tried claws but they no longer support Maildir.

Friday, 13 January 2012

The Year Without Winter

It is Friday the Thirteen, January 2012. I am looking out of the window and the sun is shining. It is 4 degrees Celsius, and I'm wounding where the delivery of winter is going to happen. 1816 may have been "the year without summer" but the end of 2011 is the year without winter. This may be 1947 all over again, but so far Autumn had, for me, a lower average temperature than winter. (I find myself, today, in France and to help the rest of this entry make sense I'm hangin' round Latitude 47 North.)

[ Now that I try to talk about winter and pin it into a year, as we can with a summer it seems that the New year should be moved to be located between winter and spring. Having it in the depth of winter, (which winter where? Did you see one? ) to celebrate the "returning of the light" seems a little antiquated. It may be a dull time in the farming world and an excuse for a party to help us drudge through the long dark nights is probably a good thing, but why should our calendar dictate when to party - take control people! You write on your calendar, don't let your calendar write your life. ]

So am I talking about winter 2011 or winter 2012? As November..December 2011 felt more like spring, in this case I'm talking about winter 2012.

Since the early eighties, (1983 to be exact) I've been noticing the winter seems to recede, (like my hair). I remember thinking, (and reading my diary from that time) that winter no longer included November. By the nineties December was leaving the party, and now January seems to be defecting to the Autumn camp, (or the mid-winter-spring as many of my plants seem to think.)

As October 2011 became November 2011 I was starting to get worried about seeds. Let me explain: Each year I like to have at least one basil plant on my window-sill. They are very easy to look after - I just water mine a little every other day, or when the soil start to turn a shade lighter brown. ( They can drip oils, so make sure, if you have them in pots, that they don't drip on anything that matters.)  I usually stop picking the leaves as I start to think about Christmas presents. A few weeks later they produce little white flowers. I brush them with a small downy feather to transfer pollen, and then I collect the small dark brown seeds, ready to plant in the spring. So with no seeds there are no plants. With no flowers there are no seeds... so I was getting worried. I checked some other plants in the garden and noticed that the lemon-balm, (mine spreads like a weed) didn't have a single flower... even today, (hence the entry.)

My avocado plant has, over the last few weeks, put on a growth spurt as if it already received its invitation to spring.

So it feels fishy, like something is up. I have not been making detailed enough measurements to have any certainty beyond that. My feeling about the weather is probably some form of confirmation bias, (I hope it is), or just because I finally managed to, (in 2011) get round to reading The Body Electric: Electromagnetism and the Foundation of Life, (ISBN 0-688-06971-1).
( I don't intend to review that here, but I did find it very interesting. I tried to maintain a suitable level of scepticism for a man investigating something that was outside of his field, but if his passion for research was anything like as compelling as his writing then it would mean that his voice was more valid than my internal detractor.) 

I hope that the lack of seeds is because of something that I did, (or am doing) and that 2012 is not going to see a run on the seed-banks to rival the 2011, (oh you know where this analogy is going.)

Setting my paranoia aside for a moment, what would the lack of a winter mean for a temperate European climate? The first thing that comes to mind is, an excess of insects surviving to spring, (well for some the first thing would be a lack of good skiing, but I'm trying to think more generally.)
 More insects would be good for those that predate upon them, but could devastate crops and aid in the spread of disease, (the mosquitoes will focus on malaria, the others can help spread the other nasties.) On a more positive note, the increase in temperature may reduce the general susceptibility of the population to contracting influenza.

I'm going to get back to work now. I hope that the seeds that I have left over from 2011 will germinate this spring.


Tuesday, 10 January 2012

SHA-1 countdown...

If you are into security then you will probably have already found, (or know) Valerie Aurora, and if you are not then this might not interest you. I was reading some of her stuff and loved this, (so much that I stole^wadapted it.) So, thank you Valerie for:


Life cycles of popular cryptographic hashes (the "Breakout" chart)
Function\Year90919293949596 979899000102030405 06070809
Snefru
MD4
MD5
MD2
RIPEMD
HAVAL-128
SHA-0
SHA-1
RIPEMD-128 [1]
RIPEMD-160
SHA-2 family
KeyUnbrokenWeakenedBroken
[1] Note that 128-bit hashes are at best 2^64 complexity to break;
using a 128-bit hash is irresponsible based on sheer digest length.


So this post may be more of a bookmark than information. Actually I found this wonderful representation  through her blog.valerieaurora.org. I'm sure that there are others that would like to see an updated page with the addition of all of the entries from The Hash Function Lounge.

I remember when MD5 fell and we created our own x509 root certificate. I'm still waiting, like a child waiting for Christmas morning, for SHA-1 to publically fall.

So how many of you have noticed that in 1996 RIPEMD was being attacked and RIPEMD-160 was being created. This seems logical as the attack and development were neither secret. (What is he on about? I'll tell you ;-)

So let us look at the year that SHA-0 was publicly attacked... and the year that its replacement came out... 1995. Four years is a long time in the cryptographic hashes world, (MD4 and HAVAL-128 went from Unbroken to Broken in that time!)

Now I'm not a conspiracy nut (obviously) so I'm not suggesting that the NSA already knew about the vulnerability and did not announce it. The question is... when will the NSA files leak^wdeclasification reveal the time and date that they broke each hash function. (In the UK it is about 30 years for most files, which seems reasonable, but who knows for those over at Fort Meade.)

That reminds me - I must check on my bees.






Thursday, 5 January 2012

Project Management: Script everything; Test EVERYTHING; Graph everything

Last year I finished a two year project. I would love to tell, (brag) all about it but when I sign an NDA I mean it. The customer is a huge fan of opensource, and encouraged me to share as much as possible about my experience gained from the project, (while keeping an eye on the NDA.) I sent this entry to the client for approval and Q.E.D. they did.

So you have a project. You want to use reliable software, but spend no money on it and get the job done ahead of schedule. You are starting compleatly from scratch, with no support, (are you insane?)
I don't know what works for you, but my advice is as follows:

use dotProject

That is all I really wanted to share. If you are inside of a company, being nurtured with an existing support infrastructure, or if on day one you have an idea, a small laptop and no-money, my advice is the same. If you want a more wordy explanation for when starting with nothing, then read on, (you can delete the parts that you already have.)

Grab a copy of Linux, (I used Stackware and CentOS, but debian/ubuntu also works) and stick it on any computer, (I started out with a six year-old laptop and one aging 1U server - and I did not need the server for the first year!)
 When I did need a server I looked at co-location options, (I build my own server in 1998 and hosted it on my office network until 2005, so that was what I was used to.) The result of the search was, as is often the case, when you go looking for something, but keep an open mind; I found that I rather liked the VPS from Bytemark, (but if you have a dedicated server you can create a virtual machine using KVM.)

Install dotProjectRT: Request TrackerNotice, apache, exim, dovecot and munin to monitor the server. (This took me most of the first morning, but I was new to dotProject then.) Create, on the virtual machine, a new account for each of the developers and give them sudo access to exactly what they need. I like to add SELinux to the mix, lock down access with iptables, and create a pocket certificate-authority so that everything can be signed and where possible encrypted. (E.g. the ejabberd instance. The team used pidgin / finch / Adium to communicate more often than email.)

With dotProject you don't even need RT, (there is a simple ticketing system as a plug-in,) but when your project goes live you should have transitioned into RT for customer facing response. I use Notice as the glue that holds everything together, (so it is written in Perl.) Notice controls apache, exim, dovecot , NSD3, (and anything else like that) and manages all of my data: Which people are in each team; Project assets; CRM; reports; billing; general system administration, (though again, dotProject can manage all of the data for a project. The reason for Notice is "everything else".)


What did I learn from managing this project?

Script everything


# This project used
---
- 7zip
- bash
- C
- C++
- glusterFS
- HTML: CSS
- javascript: [DOM, jQuery, JSON]
Keepalived
- mysql: master-master replication
- Perl: [DBIx::Class, Template::Toolkit, CPAN]
- svn: then later git
- windows-installer: NSIS
- wine
- x509
- XML
- XMPP
- other: probably, but this is all that comes to mind right now
# while consuming .ogg, C8H10N4O2, noodles and C2H5OH

So it was vital to be able to build the binaries with a single command. I looked at Apache Ant but in the end I was won over by Daniel Robbins (of Gentoo fame), and chose bash. This meant that at any moment it was possible to checkout a copy of the code and build it with a single command, (that involved extracting files from archives, applying changes, compressing, compiling more compressing and finally building the installer for each platform.)

Test EVERYTHING: not only the Chuck, but the bit and the whole drill!
(In joke - either you know it or you don't - its called a learning curve)

Thanks to the book, Perl Testing: A Developer's Notebook I fell in love with testing, (I don't expect everyone to become a testing freak - just their code.) With the cross-pollination of python, (use phpcs to see how the indentation gestapo have already spread there,) and what seems like the inevitable convergence of scripting languages directly analogous to car design, I expect that perl style testing will become as ubiquitous as a 5 star Euro NCAP rating before very long. (Talking of which, if there was an Internet Software Security and Quality Assessment Programme, like html5test.com crossed with alternativeto.net, then both quality and security would probably go up, but that is a talk for another day.) If you have not already worked it out, "Script everything" still applies to testing.


Graph everything: If it moves - graph it
(I think I read this quote in an entry by either Paul Graham, (RTML) or Brad Fitzpatrick, (livejournal) but I could be wrong)

For many years I had used cacti, (MRTG) and loved it. It was sometimes tricky to install, and learning how to do custom pages took me longer than I expected, (I'm not saying that it is hard, just that it too me too long.) Then I found Munin.

It was like the sun coming out from behind the clouds. I'm not saying that munin is better than cacti, I'm saying that munin does it for me; I fancy munin and I don't care who knows.
It just fit my needs exactly. I could easily write a new plugin, (if it did not already exist), and I can centralise multiple servers into one report, I can do it all from a bash prompt, (did I mention script everything?) Using ipmi I can even check the air-con in the data centre.

In conclusion, opensource software has matured to a level where it is practical to use it in a commercial environment. Just as any good version control system can contain its own development, any project can now be managed with free software, (and no amount of FUD or snake-oil-talk of ROI or TCO is going to save the proprietary dinosaurs now.)

Notice this?

If you are looking for a working example of CGI::Application combining the plug-ins:

CGI::Application::Plugin::ConfigAuto
CGI::Application::Plugin::DBH
CGI::Application::Plugin::Session
CGI::Application::Plugin::Authentication
CGI::Application::Plugin::Redirect
CGI::Application::Plugin::DBIC::Schema
CGI::Application::Plugin::Forward
CGI::Application::Plugin::TT

in a working  MVC (DBIC,TT,CAS) framework, then Notice should help you. I use it as a Customer Resource and Account Manager; Asset management, but you can adapt it to your needs. Notice is the  distillation of almost twenty years of Internet related solutions. Being a general outline of an idea to, "solve all of the Internet problems" it has a very expansive remit.

Although this example, on github, is from an earlier development branch, it certainly would have saved me some time if even this incarnation of Notice had already existed, when I was looking through the many good Perl frameworks for the one that met my needs.



(This incarnation of Notice is inspired by CGI::Application::Structured and a direct port of my previous, all-my-own-code-from-the-ground-up, version of of Notice.)

I hope that someone will find it useful. Being the outline of the Notice modular system, (that is using the CGI::Application framework), it is easy to create a custom module that stores your data in any way that you want, (or you can strip it down to the bones and build your own animal.)

Notice is interesting for at least two reasons, (possibly more).

First it is an example of a light MVC framework using DBIx::Class and Template::Toolkit. I started out by looking at the largest most popular MVC framework in CPAN; it is great, but I needed something lighter, so I looking for the lightest, (CGI::). ( My previous version of Notice had started out using CGI_light, but eventually I stripped out all dependencies except DBI.)

I found a collection of tutorials, but never quite the full combination (of Authentication with dbic and TT) that I needed, (actually I wanted L18n as well but I did not have time when I was creating this third incarnation for Notice. xkcd.com/974/)

Second I was spending a weekend building a domain registry, (it really isn't that hard) and needed a whois server; I could find an almost endless collection of code to talk to/with a whois server, but not actually a whois daemon. So in the S of Model, View, Control, Sundries is a little RFC3912 compliant whois daemon. If you create a CDB file with a key of awesome.alexx.net and a value of "Why YOU are my friend, you are!" this daemon will serve it up for you. ( It is trivial to have the whois server pull live data from the database - your milage may vary with the level and volume of DDoS or /. directed at your server.)

The unsuspecting world should know that this version is NOT an example of perfect code. Feel free to improve it, laugh at it, (I do) or make a version with the features that you need, e.g. L18n. I'm extending the assets section to have hive-records for for the beehives that I now have listed in my  assets table.

If you need an example of the features mentioned at the start of this post, (or a whois server), this may save you time. Also, there are many modules that have already been developed, that have not been included in this code. I will include a new README, (and possibly a blog entry) explaining the modules and how each instance of Notice can interact with all of the others, (or you can email me with questions about Notice.)

[I have published the whois server separately in CPAN, but with this you can automate the rebuilding, (and distribution) of the whois database from a central management system. I have not yet got permission to include all of the custom code that did the DNSSEC. That sat on a computer with no external access; pulled each zonefile; signed them; pushed them out to each DNS server. ]

This is not another framework; It is an example of CGI::Application. If you ask why?:
Catalyst seems like the daddy-bear of frame-works, (though Mojolicious will be 'The Daddy' for some) and all the others seemed like mummy-bear, so Notice is my baby-bear;

I hope that someone else will think that it is 'just right'.

Monday, 2 January 2012

Linux loves FujiFilm FinePix Z35

"Why FujiFilm no love linux?" was one of the questions that I found while researching how to get a digital camera to work with a client's Linux desktop computer. Mostly for my own reference I have created some notes but if you have to get these two things to dance it will help you, (unless FujiFilm get their act together with the Linux communities by then.) I know that I am biased, but it does see strange that a corporation would fail to even spend the time to create one page explaining how to get their product to work with Linux with the other two desktop operating systems losing market share.

About this blog

Sort of a test blog... until it isn't