Guest Slot – Rod Crosby on the Kalman Filter
Friday, December 29th, 2006
Wouldn’t it be nice…..If we could get from here…..
…to here?
Well, perhaps we can…. I just DID!! The second graph was created using only the few datapoints from the first graph, and a clever box-of-tricks called the Kalman Filter.
What on earth is a Kalman Filter?
Believe it or not, it is a statistical routine that originated in the 1960s in the fields of engineering and signal processing. The Kalman Filter is mathematically proven to be the optimal way of combining noisy data, and is used in many real-word applications – for example, radar and the inertial guidance of rockets. It was a Kalman Filter that took the Apollo spacecraft to the Moon and back.
In 1999, two Yale politics professors, Donald Green and Alan Gerber, hit upon the idea of using the filter to remove “noise” from opinion polls.
They have very kindly made their number-cruncher available online for anyone to use – it’s called SampleMiser
In February 2006, Gallup began including Samplemiser results in its Presidential Approval rating, and in November amateur pollsters claimed success with the technique to precisely forecast the US Senate race.
Samplemiser does an outstanding job not only in giving the best estimate of true support, but in also identifying with amazing precision those points in time where opinion turns…
So I thought I’d put it to the test using basic poll-data for the Tories. The really clever thing Samplemiser did was to take 110 irregularly-spaced polls since May 2005, and create 580 new polls – apparently out of thin air – one for every single day. In other words it enables day-to-day tracking of Tory support…
What tale does the Samplemiser tell?
Out of a blizzard of dots, on 29th September 2005, Samplemiser identified a reversal and sharp upward trend in Tory fortunes. On that day, the TV news was plastered with images of Walter Wolfgang being manhandled out of the Labour Conference, giving the party appalling publicity as perceived “bullies.”
By November the Tories were faltering, but a turn in their favour occurred when the media identified David Cameron as favourite to be the next leader. With uncanny accuracy, on the very day Cameron was elected, 6th December 2005 – Samplemiser projected a 1.5% leap in support, and the Tories soon rose like a rocket, putting on 4 percentage points in a month.
By the first week of 2006 though, there were signs of a levelling-off and slight fall-back at 37%. But the sudden defenestration of Charles Kennedy on 7th January gave Cameron a further boost, with the Tories cresting at 39% on 3rd February. However, this was the same day the leaderless LibDems pulled-off a sensation at Dunfermline, and by the very next day Samplemiser had the Tories falling sharply…
Cameron’s renewed espousal of Green issues during February appeared to pay dividends, but Samplemiser noted the abrupt end of this recovery on 4th March, with the election of Ming Campbell as LibDem leader. The Tories achieved another brief upward spike on 17th March, after 75 Labour MPs rebelled on the Education Bill.
On the 18th April 2006, the “Dave the Chameleon” Political Broadcast was aired for the first time. It proved a highly potent message, as the Tories went into free-fall on the 19th, dropping almost 3 points in 3 days, according to Samplemiser….
If our little timeline had ended there, David Cameron might be just a historical footnote by now. His short honeymoon was certainly over, the LibDems had miraculously revived, and the Tories now stood at below 33% – less than they had achieved at the General Election. Labour had appeared to land a knock-out punch with its scathing party political broadcast, and another bout of Tory in-fighting looked inevitable.
FATE, unbeknown however, was riding to the Tories’ rescue – in the ungainly, ludicrous form of John Prescott….
News of the his affair with Tracy Temple broke on 26th April, and by the 28th Samplemiser had the Tories surging by 1.5% – the start of one of the most rapid political sea-changes in recent history. Over the next 5 weeks the Tories would put on nearly 7 percentage points, peaking at just over 39% by the end of May. Further spikes in June and July were recorded, apparently related to the Prescott/Anschutz story, but interestingly the Tories seemed unable to penetrate a glass-ceiling at just over 39%.
There were fewer polls than usual in July and August. The Levy arrest didn’t register, as it fell inside an unusual poll-free fortnight, but Cameron’s “hug-a-hoodie” speech may have caused a dip in Tory support. The terror alert in mid-August could have actually boosted the Tories, but the apparent breaking of the plot propelled the Conservatives on to the downward path from the 24th August, the day after the suspects appeared in court.
The next sharp upturn in Conservative support identified by Samplemiser occurred on 7th September, the very day Tony Blair finally announced his forthcoming departure. This boost proved short-lived – less than a week – and the Tories resumed a steady decline.
On 20th October, Samplemiser recorded another surge in Tory support – in the week that Blunkett’s sensational memoirs were published, and Sir Richard Dannatt made his outspoken criticisms on Iraq. However, on 3rd November it was reported that David Cameron had returned to the “hoodie” theme. His new comments were widely ridiculed as “Love a lout.” With seeming laser-beam accuracy, Samplemiser showed Tory support plummeting, falling by over 2% from 5th November.
Since then, the Tories have hovered around the 36% mark. We have had a remarkable view of the rollercoaster ride in Tory fortunes that the poll snowstorm conceals, and only time will tell in which direction they head off next.
But one thing seems clear – to keep firm track of them we need the help of the amazing Samplemiser….
Rod Crosby has been a Lib Dem activist but says “I have never voted Labour in my life, but if by some chance there was a danger of David Cameron becoming Prime Minister, I would not hesitate to do so.“
-
Wouldn’t it be nice…..If we could get from here…..
-
…to here?
Well, perhaps we can…. I just DID!! The second graph was created using only the few datapoints from the first graph, and a clever box-of-tricks called the Kalman Filter.
What on earth is a Kalman Filter?
Believe it or not, it is a statistical routine that originated in the 1960s in the fields of engineering and signal processing. The Kalman Filter is mathematically proven to be the optimal way of combining noisy data, and is used in many real-word applications – for example, radar and the inertial guidance of rockets. It was a Kalman Filter that took the Apollo spacecraft to the Moon and back.
In 1999, two Yale politics professors, Donald Green and Alan Gerber, hit upon the idea of using the filter to remove “noise” from opinion polls.
They have very kindly made their number-cruncher available online for anyone to use – it’s called SampleMiser
In February 2006, Gallup began including Samplemiser results in its Presidential Approval rating, and in November amateur pollsters claimed success with the technique to precisely forecast the US Senate race.
Samplemiser does an outstanding job not only in giving the best estimate of true support, but in also identifying with amazing precision those points in time where opinion turns…
So I thought I’d put it to the test using basic poll-data for the Tories. The really clever thing Samplemiser did was to take 110 irregularly-spaced polls since May 2005, and create 580 new polls – apparently out of thin air – one for every single day. In other words it enables day-to-day tracking of Tory support…
What tale does the Samplemiser tell?
Out of a blizzard of dots, on 29th September 2005, Samplemiser identified a reversal and sharp upward trend in Tory fortunes. On that day, the TV news was plastered with images of Walter Wolfgang being manhandled out of the Labour Conference, giving the party appalling publicity as perceived “bullies.”
By November the Tories were faltering, but a turn in their favour occurred when the media identified David Cameron as favourite to be the next leader. With uncanny accuracy, on the very day Cameron was elected, 6th December 2005 – Samplemiser projected a 1.5% leap in support, and the Tories soon rose like a rocket, putting on 4 percentage points in a month.
By the first week of 2006 though, there were signs of a levelling-off and slight fall-back at 37%. But the sudden defenestration of Charles Kennedy on 7th January gave Cameron a further boost, with the Tories cresting at 39% on 3rd February. However, this was the same day the leaderless LibDems pulled-off a sensation at Dunfermline, and by the very next day Samplemiser had the Tories falling sharply…
Cameron’s renewed espousal of Green issues during February appeared to pay dividends, but Samplemiser noted the abrupt end of this recovery on 4th March, with the election of Ming Campbell as LibDem leader. The Tories achieved another brief upward spike on 17th March, after 75 Labour MPs rebelled on the Education Bill.
On the 18th April 2006, the “Dave the Chameleon” Political Broadcast was aired for the first time. It proved a highly potent message, as the Tories went into free-fall on the 19th, dropping almost 3 points in 3 days, according to Samplemiser….
If our little timeline had ended there, David Cameron might be just a historical footnote by now. His short honeymoon was certainly over, the LibDems had miraculously revived, and the Tories now stood at below 33% – less than they had achieved at the General Election. Labour had appeared to land a knock-out punch with its scathing party political broadcast, and another bout of Tory in-fighting looked inevitable.
FATE, unbeknown however, was riding to the Tories’ rescue – in the ungainly, ludicrous form of John Prescott….
News of the his affair with Tracy Temple broke on 26th April, and by the 28th Samplemiser had the Tories surging by 1.5% – the start of one of the most rapid political sea-changes in recent history. Over the next 5 weeks the Tories would put on nearly 7 percentage points, peaking at just over 39% by the end of May. Further spikes in June and July were recorded, apparently related to the Prescott/Anschutz story, but interestingly the Tories seemed unable to penetrate a glass-ceiling at just over 39%.
There were fewer polls than usual in July and August. The Levy arrest didn’t register, as it fell inside an unusual poll-free fortnight, but Cameron’s “hug-a-hoodie” speech may have caused a dip in Tory support. The terror alert in mid-August could have actually boosted the Tories, but the apparent breaking of the plot propelled the Conservatives on to the downward path from the 24th August, the day after the suspects appeared in court.
The next sharp upturn in Conservative support identified by Samplemiser occurred on 7th September, the very day Tony Blair finally announced his forthcoming departure. This boost proved short-lived – less than a week – and the Tories resumed a steady decline.
On 20th October, Samplemiser recorded another surge in Tory support – in the week that Blunkett’s sensational memoirs were published, and Sir Richard Dannatt made his outspoken criticisms on Iraq. However, on 3rd November it was reported that David Cameron had returned to the “hoodie” theme. His new comments were widely ridiculed as “Love a lout.” With seeming laser-beam accuracy, Samplemiser showed Tory support plummeting, falling by over 2% from 5th November.
Since then, the Tories have hovered around the 36% mark. We have had a remarkable view of the rollercoaster ride in Tory fortunes that the poll snowstorm conceals, and only time will tell in which direction they head off next.
But one thing seems clear – to keep firm track of them we need the help of the amazing Samplemiser….
Rod Crosby has been a Lib Dem activist but says “I have never voted Labour in my life, but if by some chance there was a danger of David Cameron becoming Prime Minister, I would not hesitate to do so.“
