- eCampus News - https://www.ecampusnews.com -

How big data lets us see a little further into the unknown

It is Monday, and it is raining again in the south of France. But it was sunny yesterday. And it was also dry last Wednesday, although it then rained almost continuously from Thursday through to Saturday, the Financial Times reports.

A small consolation is that these intervals of storm and sun have been accurately predicted several days in advance. That is why I am writing this column today rather than yesterday. The quality of weather forecasting has improved considerably.

The BBC has re-released the worst weather forecast in its history. In 1987 Michael Fish went on television to reassure viewers rumours of an imminent hurricane were unfounded [1]. A few hours later the most severe winds in decades lifted roofs and felled trees all over Britain.

But such a blunder is much less likely now. Short-term weather forecasting is one of the triumphs, perhaps the greatest triumph, of big data [2] – the opportunity supercomputers provide to process data sets of unbelievable size and complexity. I understand that the latest machines can handle an exabyte of data, which is about 20m times the capacity of my Apple Mac.

… It is still true, however, that accuracy declines rapidly as you look further ahead. There is a clear contrast between the ability of weather forecasters to give us a reasonably accurate description of today and tomorrow; and their continued inability to make good longer-term forecasts. The exceptional weather conditions [3] of this winter were not anticipated.

Read more [4]