Numberphile

If you are like me then you will like http://www.youtube.com/user/Vihart (squee! ♥)

and you will have seen Numberphile... which so far has not been as good as The Singing Banana, until today! (So I declare!)

For me today's episode of numberphile is as cool as I had been hoping they would be.
( I hope that they have just got into their stride.)

The previous episodes have been, for me, between so-so and hmmm, (so worth seeing, but not worth a thumbs up [0].)

If I had to do an episode then mine would be 15 and I would talk about black-hole numbers [1].

I wonder if they would let me pick 1234, but then talk about 1,2,3,4 and trying to create every number from [1..100] just using each of those digits once. i.e.

1+2+3+4   =10 (99 left to find),
(1+2)(3+4)=14  (98) left to find.

Many years ago my friend Shevek wrote, (in 3194 bytes of perl) a program that searched for all possibilities. Before that my friend Nico and I had found most of them by hand.

(If you get stuck then you can ask me for help.)

[0] It would be nice to have some granularity. Did I like the production; Did I like the relevance to what I was looking for? (Come on 'The Internet' not all of us are undergrads ;-)

[1] Recurse through { The sum of the divisors of a natural number } => 15

Monkey httpd

This weekend I tried monkey httpd. The short version: It rocks.
I had a little trouble getting it to server CGI (.php .pl .cgi .py .rb .bash) because the server that came with it, (written in python - bless their cotton socks) did not work for me.

If you want to use the script that I hacked together then feel free:

https://github.com/alexxroche/AIF/blob/master/scripts/perl/monkey-palm_0.1.pl

The big gotcha for me was "\r\n\r\n" rather than "\n\n". But once I tracked that down it was, (much like that reset of monkey-project) just as I expected, with everything where it should be and even painted the right colours!

I'm off to hack the auth plugin to make it read .htaccess and then I'm going to "uninstall", (yes, remove) apache.

(Then I'm going to port my little perl Net::Server script to native c.)

Buy my stuff

For those of you that would like to own an original design based on one of the many great quotes in my blog, (or my life): Now you can.

Munin about logarithmically

As you all know, I like to graph, (OK, being honest - I need to graph.) So when I create a nifty new plug-in to monitor the charge in my laptop battery, (acpi_volt) I was feeling rather smug. It uses

 acpi -i

to find out two pieces of information. The first is how charged the batter is. (Somewhere between 0 and 100 percent.) The second, (and this is the science bit) is the design capacity in mAh, and the mAh at last full capacity. This is cool. We get two lines, one showing how much charge is left in the battery and another showing how much milk is left in the bottle. (I know that my laptop battery should not suffer from any memory effect or milk-bottle effect, but it does. I don't know what chemical fingers are responsible, but over the past three years the length of time that I can run it off the battery has dropped from about 90 minutes to about 20.)

So I click on the graph to see the day,week,month,year view and *gasp* no month graph.
I console myself that sometimes it takes a while for munin to have enough data in the rrd files to bother creating the month and year graphs. So I go to bed. The next day it has managed to create a year graph, but still no month graph.

I smile. I have been a sysadmin.

- this I know how to solve.

I start with the usual suspects, (file/dir permissions in /var/lib/munin and other munin locations specified in /etc/munin/munin.conf )

Nothing. I check that I can collect the data remotely, and that the acpi_volt.rrd files have valid data in them, (they do.)

hmm - I have never got this far with a munin problem. I guess we will have to go deeper.

su -s /bin/bash munin
munin@laptop:~$ /usr/share/munin/munin-graph --service acpi_volt --nolazy --list-images --month --noyear --noday --noweek

produces the expected output
/var/www/munin/localdomain/localhost.localdomain-acpi_volt-month.png

but it lies!

ls -la ls -la /var/www/munin/localdomain/localhost.localdomain-acpi_volt-month.png

ls: cannot access /var/www/munin/localdomain/localhost.localdomain-acpi_volt-month.png: No such file or directory

ok. how about adding --debug to the end of that

DEBUG: Expanding specials "volt0","volt1","".
DEBUG: Checking field lengths "volt0","volt1","".
DEBUG: Treating fields "volt0","volt1","".
DEBUG: Processing field "volt0".
DEBUG: single_value: Checking field "ARRAY(0x8a0ea38)".
DEBUG: Drawing field "volt0".
DEBUG: Processing field "volt1".
DEBUG: Drawing field "volt1".


rrdtool "graph" "--font"
"LEGEND:7:monospace"
"--font"
"UNIT:7:/usr/share/munin/VeraMono.ttf"
"--font"
"AXIS:7:/usr/share/munin/VeraMono.ttf"
"/var/www/munin/localdomain/localhost.localdomain-acpi_volt-month.png"
"--title"
"Voltages - by month"
"--start"
"-33d"
"--base"
"1000"
"--logarithmic"
"--vertical-label"
"Volt"
"--height"
"175"
"--width"
"400"
"--imgformat"
"PNG"
"DEF:gvolt0=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt0-g.rrd:42:AVERAGE"
"DEF:ivolt0=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt0-g.rrd:42:MIN"
"DEF:avolt0=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt0-g.rrd:42:MAX"
"CDEF:cvolt0=gvolt0"
"COMMENT: "
"COMMENT: Cur\:"
"COMMENT:Min\:"
"COMMENT:Avg\:"
"COMMENT:Max\: \j"
"LINE1.6:gvolt0#22ff22:Full "
"GPRINT:cvolt0:LAST:%6.2lf%s"
"GPRINT:ivolt0:MIN:%6.2lf%s"
"GPRINT:gvolt0:AVERAGE:%6.2lf%s"
"GPRINT:avolt0:MAX:%6.2lf%s\j"

"DEF:gvolt1=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt1-g.rrd:42:AVERAGE"
"DEF:ivolt1=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt1-g.rrd:42:MIN"
"DEF:avolt1=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt1-g.rrd:42:MAX"
"CDEF:cvolt1=gvolt1"
"LINE1.6:gvolt1#0022ff:capacity "
"GPRINT:cvolt1:LAST:%6.2lf%s"
"GPRINT:ivolt1:MIN:%6.2lf%s"
"GPRINT:gvolt1:AVERAGE:%6.2lf%s"
"GPRINT:avolt1:MAX:%6.2lf%s\j"
"COMMENT:Last update\: Fri Jan 27 16\:45\:03 2012\r"
"--end"
"1327672800"

hmm, that all looks ok. Lets take a closer look at the data:

rrdtool info /var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt0-g.rrd
rrdtool info /var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt1-g.rrd

rrd_version = "0003"
step = 300
last_update = 1327685404
ds[42].type = "GAUGE"
ds[42].minimal_heartbeat = 600
ds[42].min = NaN
ds[42].max = NaN
ds[42].last_ds = "100"
ds[42].value = 4.0290790000e+02
ds[42].unknown_sec = 0
rra[0].cf = "AVERAGE"
rra[0].rows = 576
rra[0].cur_row = 88
rra[0].pdp_per_row = 1
rra[0].xff = 5.0000000000e-01
rra[0].cdp_prep[0].value = NaN
rra[0].cdp_prep[0].unknown_datapoints = 0
rra[1].cf = "MIN"
rra[1].rows = 576
rra[1].cur_row = 197
rra[1].pdp_per_row = 1
rra[1].xff = 5.0000000000e-01
rra[1].cdp_prep[0].value = NaN
rra[1].cdp_prep[0].unknown_datapoints = 0
rra[2].cf = "MAX"
rra[2].rows = 576
rra[2].cur_row = 509
rra[2].pdp_per_row = 1
rra[2].xff = 5.0000000000e-01
rra[2].cdp_prep[0].value = NaN
rra[2].cdp_prep[0].unknown_datapoints = 0
...

yup that all looks good, (to me). So lets graph by hand:

rrdtool graphv /var/www/munin/localdomain/localhost.localdomain-acpi_volt-month.png \
-W "Alexx Roche" --end now --start end-33d \
DEF:gvolt0=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt0-g.rrd:42:AVERAGE \
DEF:ivolt0=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt0-g.rrd:42:MIN \
DEF:avolt0=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt0-g.rrd:42:MAX \
CDEF:cvolt0=gvolt0 \
DEF:gvolt1=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt1-g.rrd:42:AVERAGE \
DEF:ivolt1=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt1-g.rrd:42:MIN \
DEF:avolt1=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt1-g.rrd:42:MAX \
LINE1.6:gvolt0#22ff22:Full LINE1.6:gvolt1#0022ff:capacity

yup, that works, (but does not look pretty.) So lets keep adding configurations.


rrdtool graphv /var/www/munin/localdomain/localhost.localdomain-acpi_volt-month.png \
--title "Voltages - by month" \
--end now --start end-33d \
DEF:gvolt0=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt0-g.rrd:42:AVERAGE \
DEF:ivolt0=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt0-g.rrd:42:MIN \
DEF:avolt0=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt0-g.rrd:42:MAX \
CDEF:cvolt0=gvolt0 \
DEF:gvolt1=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt1-g.rrd:42:AVERAGE \
DEF:ivolt1=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt1-g.rrd:42:MIN \
DEF:avolt1=/var/lib/munin/localdomain/localhost.localdomain-acpi_volt-volt1-g.rrd:42:MAX \
CDEF:cvolt1=gvolt1 \
--font LEGEND:7:monospace \
--font UNIT:7:/usr/share/munin/VeraMono.ttf \
--font AXIS:7:/usr/share/munin/VeraMono.ttf \
--base 1000 \
--vertical-label "Volt" \
--height 175 \
--width 400 \
--imgformat PNG \
COMMENT:" " \
COMMENT:" Cur\:" \
COMMENT:"Min\:" \
COMMENT:"Avg\:" \
COMMENT:"Max\: \j" \
LINE1.6:gvolt0#22ff22:Full  \
GPRINT:cvolt0:LAST:%6.2lf%s \
GPRINT:ivolt0:MIN:%6.2lf%s \
GPRINT:gvolt0:AVERAGE:%6.2lf%s \
GPRINT:"avolt0:MAX:%6.2lf%s\j" \
LINE1.6:gvolt1#0022ff:capacity  \
GPRINT:cvolt1:LAST:%6.2lf%s \
GPRINT:ivolt1:MIN:%6.2lf%s \
GPRINT:gvolt1:AVERAGE:%6.2lf%s \
GPRINT:"avolt1:MAX:%6.2lf%s\j" \
"COMMENT:Last update (by LxR)\: Fri Jan 27 16\:45\:03 2012\r"

ah ha! --logarithmic seems to break it. Just for the monthly graph? (Some may ask why I was graphic logarithmically rational numbers between 0..100 but that is not the point.)
 I do not see why munin would not alert or at least log, (somewhere in /var/log/munin/) something to mention that the graphing did not happen, (and even more oddly, for just one of the four graphs.)

So I changed acpi_volt to graph_args => '-1 0' and now I have all four... but I still do not know why rrdtool refused to graph. (Part of me want to keep the .rrd so that others can test it, but I'm not sure how much help that would be.)

Grandma Nazi

I am not a grammar Nazi. That would be like Stephen Fry applying to join the SS in 1941's Germany.

Everyone is somewhere on the line from zero to Himmler when it come to various typographic and logical errors. The one I want to mention, (though I hear it more often than read it) is:

"Very unique"

Please stop trying to define degrees of uniqueness; Either something is unique or it isn't.

Match-making Age Ranges

Many years ago I was told that, "Half your age plus seven years is the girlfriend limit."

This immediately seemed both sexist and mathematically incompetent, but just as "Feed a cold, starve a fever" isn't medical advice, (it is reassurance for young mothers who are panicking that their feverish child isn't eating,) it is just a guide-line, ( or in that case -  a throw-away anecdote at some debutante party in London.)

What about those looking to date someone older? This guide can easily be reversed as, "Your age minus seven, doubled."

Mathematically, (where x is your age),

        lower_limit = (x/2)+7
     upper_limit = 2(x-7)

Some time later I was looking for mathematically significant differences between two numbers. This is an area of statistics where arguments are had over tea.

I found, on one rather well written site: (I did not bookmark it, sorry.)

      ( ( x - y ) / 2 ) ^2  

which might not look like much but I rather liked it. (Here y is "their age" for lower limit, and ^2 means "to the power of two" i.e. squared.)


It took a while, but eventually I made the connection between these two and the obvious next move was to graph them. I had already found, with a quick script, the intersection at 34 and 54 by hand, (once I noticed that the output of the two functions was not parallel, I had to search,) but somehow I had missed the one at 49, (see! this is why graphs are cool.)

(I was going to add a vertical line marking the age of consent, but that is a topic for another post.)

I have added some sudo-random labels on the graph, using gimp, so that I could refer to particular areas more easily, (see I'm thinking of you.)

The first thing that is worth noting is that the cougar limit, [yes, I typoed in the graph] and the sugar limit are not both straight lines. This is not due to some form of ageism or sexism, but is due to the use of an odd number in "half your age plus". If it was half your age plus six then they would both be straight.

We can see stepping in the two stats lines, (marked mathematically lower and upper). So the question there becomes, why are they stepping at 18, 24, 32, 40, 50, 60. [ I know that I have three errors in the graph: msl{18,32,50} should all be +1 ].
Are these square numbers? Is there something about their factors? How are they related?

The thing I like here is that when you take the output of the two simple equations that is nearest to x, (your age) you end up with a rectification of this old match-makers guide.
If you are a parent then you can explain to your daughter why, when she is sixteen, a twenty year-old boyfriend should only happen in porno-land, (and other fictitious realms.)

So who ends up in the perpetuity zone? If you imagine having a crush on an older film star, (say forty years-old) when you were younger, (e.g. twenty). By the time you are fifty they would be seventy. If you happened to be compatible when you meant thirty years later, they would be in the perpetuity zone, and in this case I could see people making exceptions, (humans being social creatures.)

Despite my choice of labels, I would recommend that everyone stay within the cradle-snatching:Tapping the Grave areas.



Technical details
Data generated using a bash script and bc.
Graph generated using cairo, Chart::Clicker and a perl script.

( The functionality of the bash script could be extended to calculate how long you would have to wait until someone entered the perpetuity zone. )

Email Archive; How do you do yours?

If you are anything like me, you have an ever growing pile of email. It sits on a server that does have IMAP4 access and you might even check it using a client like Thunderbird, but for the most part you just use the webmail interface. Since Roundcube and gmail the email client has rather had its day... or has it.

Why not install and fire up a copy of Thunderbird, (it is free). Well I'll tell you why not, (or rather why.) You can use the client to make a backup of all the messages that you received and sent. (If Google goes bust for trying to bind G+ into your searches, you will be thankful to have a backup! Remember: Nothing is forever.)
If you got a new computer in the last six months you will have oodles of disk space; Why not use some of it for something useful?

Remember than IMAP just "looks" at the messages on the server. You have to copy them into "Local Folders". Then you can safely go online and delete any of your old messages. Please make sure that you have a local copy, (heck, with the price of consumer disk-space make two!) before you delete any messages.

If this blog were a film, and you had been reading the subtitles, you will know that I'm a fan of fire-and-forget automation using scripting. So how do I do it?

If you are not using a computer that has UNIX or Linux installed then this may be as far as you need to go today, (if ~/backups/" is gibberish to you then you need to find a good  UNIX / Linux Terminal Tutorial.)

In 2005 I used http://freshmeat.net/projects/imapsync (now called http://freecode.com/projects/imapsync). That was a great project, but now it is a great product.. so it is no longer free (though https://github.com/imapsync/imapsync seems free.) So you look for something else and find mbsync http://isync.sourceforge.net/mbsync.html - maybe a little harder to understand at first, so here is how I use it, (with a cut-n-paste example config):

cron + wrapper_script + mbsync (I used to use imapsync, as I mentioned).

For this example we are going to backup two gmail accounts into ~/backups/gmail/

Configure


Get certificates

We want an encrypted connection, (remember that (since 2008) SSL/TLS will only protect you against casual network snooping!) so we willl have to collect the remote servers certificates

  openssl s_client -connect imap.gmail.com:993 -showcerts

Write the output into a ~/backups/gmail/certs/pem

Then we have to write a config file called .mbsync and put it in the root of our home directory:

~/.mbsync

#example
MaildirStore local

# where do you want to keep the messages?
Path ~/backups/gmail/

# I think of this as the details for one remote account
IMAPStore just.a.label
Host imap.gmail.com

User test-example@gmail.com
Pass notVERYsecure

UseIMAPS yes
CertificateFile /usr/share/purple/ca-certs/thoughtcrime_CA.pem
CertificateFile ~/backups/gmail/certs.pem

# You can have the details for as many accounts as you like

IMAPStore work.email

Host imap.gmail.com

User test.work@gmail.com

Pass 4l50notVERYsecure

UseIMAPS yes

CertificateFile /usr/share/purple/ca-certs/Thawte_Premium_Server_CA.pem

CertificateFile ~/backups/gmail/certs.pem

# Now the "backup instructions"

Channel my.email
Master ":just.a.label:[Gmail]/All Mail"
Slave :local:test-example

Sync PullNew
Create Slave
SyncState *

# The "Channel" is used when invoking mbsync to tell it which Channel to "watch"

# Master tells it which IMAPStore to look at, (in this case look for the one called "just.a.label"

# Slave is where to put it, (this _can_ be a remote IMAP server! Cool for migrations)

# the last three are more example settings. Check the man for more.

# A Channel can pull from multiple accounts or server at the same time!

Channel both                                      # 

Master ":just.a.label:[Gmail]/All Mail"  # from

Master ":work.email:[Gmail]/All Mail"  # from

Slave :local:combined                        # to

Sync PullNew                                    # how

Create Slave                                      # do

SyncState *                                       # what

#End .mbsync config file

Add a crontab entry

58 */8 * * *  echo 'yes'|mbsync -q both 2>/dev/null 
#I know I should listen for errors but it warns about certs


The "echo 'yes'" hacks us past an ssl warning. There is probably a better way to solve this problem, but this worked for me.

But what of the wrapper_script that you mentioned?

Well by calling a wrapper script, (in my case perl) it is trivial to change the path based on the date:

s%^Path ~/backups/gmail/%Path ~/backups/gmail/`echo -n \$(date +%Y)`/%

that way each year get a separate backup. I do end up with some duplication while I delete the old messages on the server, but duplicates are better than lost data. 

Recovery

There isn't much point in having a backup if you can't use it. I did used to just use grep to search but I found it faster to index ~/.backup/gmail using Thunderbird and then search for messages within that.. which brings me full circle back to Thunderbird. Thank you Mozilla. Keep up the good work.