Data Visualisation Rules

Data-Driven Journalism recently posted a survival guide for data visualisation. I found it to be almost* comprehensive and decided to put it to the test on the following image.

The Monkey Cage, which has recently moved to the Washington Post asked whether it was the worst chart on record. I think there are a few points worth defending it on - and some points that aren't even mentioned. So here is the survival guide on charts with my comments in bold.

The Effect Of Being Observed

Two of my favourite Twitter accounts had this conversation on Saturday evening.

@thisisanfield does it? Couldn’t it also mean that his injury is more severe than first thought? Hopefully it’s not & he’ll be fit! #InForm
— AnfieldIndex (@AnfieldIndex) September 7, 2013

I had been looking out for an example like this after reading that Bayesian probability means that two people can update their beliefs differently on the same information.

When This Is Anfield is correct is when we can observe the length of an injury. If a player is expected to be out for two weeks but then does not play in two weeks time, it is far more likely that the manager was being cautious than that he is now out for two months.

I have set up a fiddle http://pythonfiddle.com/modelling-injuries-bayesically for the case where Anfield Index is correct - the case where we have not observed the expected length of the injury. I have taken the following injury lengths in weeks [1,2,3,4,6,8,13,26,52] and given them probabilities of 40%^(array index + 1) - except for 52 which gets whatever is left over.

So what do we see?

Unproductive Productivity

Well the NFL is back and it got me thinking about this tweet:

#Bears Most 100+ yds receiving since 2006 and record in those games pic.twitter.com/pGaAIYtXMO
— Scott Lindholm (@ScottLindholm) September 4, 2013

I am interested in things like this because of records like Liverpool not losing when Ian Rush scored and similar for Man U when Van Nistelrooy scored. I always thought these records reflected poorly on the player. Basically the rest of the team was so good that if they got a goal from you they were really unlikely to lose.

What caught my eye about the above list was how similar the players records were to what I thought there team record would be in those seasons. The two exceptions were Roddy White and Marques Colston but it turns out Atlanta have won 67 games in that span and  New Orleans 56 (they followed that their shock 11-5 to win the 2006 NFC South with 7-9 and 8-8 seasons and finished 13-3 the year they started 13-0 and of course you have the bounty season).

What other individual stats correlate poorly with team stats?

Not Prospect Theory

When reading the chapter on Prospect Theory in "Thinking, Fast and Slow" one of the things that struck me was that if you use the reference point for gains and losses as the point you end up at then you would have a pretty good model. For example if you have a hundred and lose 20 your loss is 20/80 but if you win 20 your gain is 20/120.

The formulas for this is x/(x+worth): Plots are done in Wolfram Alpha.

Not sure if this is important yet but here is the derivative:

As opposed to Prospect Theory which looks like this:

One thing I didn't like about Prospect Theory is the steep curve for a loss this does not explain why people are so ready to buy insurance. If you use the above model if you could lose 90% of your value then -90/(-90+100) gives -9 and if you have to pay 1% then -1/(-1+100) gives -1/99 so you would do that for a 1/891 chance. Whereas in PT the steep curve should mean people are against paying insurance.

Someone on  Hacker News submitted the link to the wikipedia article on the St. Petersburg Paradise and so I set up a python fiddle to mess about with this: http://pythonfiddle.com/st-petersburgh-paradox

The game as explained on Wikipedia is:

A casino offers a game of chance for a single player in which a fair coin is tossed at each stage. The pot starts at 1 dollar and is doubled every time a head appears. The first time a tail appears, the game ends and the player wins whatever is in the pot. Thus the player wins 1 dollar if a tail appears on the first toss, 2 dollars if a head appears on the first toss and a tail on the second, 4 dollars if a head appears on the first two tosses and a tail on the third, 8 dollars if a head appears on the first three tosses and a tail on the fourth, and so on. In short, the player wins 2^k−1 dollars if the coin is tossed k times until the first tail appears.

The paradox is that no rational person would pay a substantial amount to play yet it has an infinite expected payout. How does my model resolve this?

Not The Mean Value Theorem

The following question was mentioned in the comments on Hacker News the other day.

If the side of a cube comes from a uniform distribution between 1 and 5 inches, what's the expected volume of the cube ?

The answer is to get the integral of x^3 at 5 minus the integral at one divided by 5-1. The integral is (x^4)/4 so we get 1/4(625/4-1/4) = 39.

But what if we are dealing with a cuboid and the sides are independent random variable. Well its E(length) * E(breadth) * E(width) or 3*3*3. But I wasn't sure if I was correct so I went to one of the newer tools in my mathematical toolbox - that's right I went triple integral.

It's the same as the integral above except we need to remember to divide by 64 because we are working against three axes.

Finally we divide by 64 to get 27. Basically each integral evaluates to 12 (area under the curve) and is divided by four to get the mean.

Yay maths works!

Stepping Stone of Thunder

This hand is from the recent European Open Championship pairs championship. The hand needs to be viewed as a 3NT hand by West (sorry about that but the table closed down while I was still analysing it.) The play potentially involves two spectacular plays:

1. Days of Thunder: coined by Eric Rodwell it involves switching to an opponents long suit to hamper communications
2. Stepping-Stone Squeze: first analysed by Terence Reese in his classic "The Expert Game" and involves overcoming a blockage by throwing in a n opponent

North opens diamonds and West has a pretty obvious duck after the A and J - fair play to south if this is from AKJ. Now we see play one. If South can imagine a club blockage then switching to a spade creates a blockage in that suit.

This is not the end of the story if West can put South with 4+ clubs with the AH. He runs six spades and has now played 8 tricks. South must come down to either 4 clubs and the AH or 3 clubs and AQ hearts. In the latter scenario the clubs will run but in the former West can unblock the clubs and exit in hearts - thus using South as a stepping stone to the clubs.

Champagne-Bottle Sizes and Acronym

I was recently skeptical when a friend used the term  Jeroboam in a game of eye-spy recently. I had never considered the size of champagne and bottles and to use a bit of foreshadowing the word really sounded like a splendid belching noise. Anyway a bottle of champagne is 75cl and is the standard measuring unit we will use. 2 bottle is a magnum and I would imagine most people, like me, would have hear about this far.

Next at 4 bottles we have Jeroboam and at 6 bottles the similar sounding Rehoboam. The latter was a son of Solomon and thus a grandson of David. For some reason the 10 northern tribes were not happy with Rehoboam's ascension and invited Jeroboam to become King of a new state and is thus the first King of Isreal. Rehoboam is the first King of Judah and the last King of the United Kingdom of Isreal.

At 8 bottles we have Methusaleh, who may have been the oldest person to live at 969 years* - his Grandson Noah also lived to 950 years. Methusaleh died 7 days before the great flood. Apparently he is a direct descendant of Adam but then isn't everyone - interestingly it was Adam's third son Seth who got the whole populating the world ball rolling.

* although the Guinness Book of World Record does not recognise him.