Smashing the Earth Observation Data Silos

Artist's rendition of a satellite - paulfleet/123RF Stock

Artist’s rendition of a satellite – paulfleet/123RF Stock

Earth observation (EO) is an all-encompassing term for monitoring our world, however as soon as you start examining the topography of the field in detail you’ll find all sorts of mountains, valleys and oceans. An illustration of the different stands can be seen if you consider the subject areas such as hydrography, geology, surveying and remote sensing, or think about areas of interest like the land and the marine specialists, and finally think about sensors specialists for LIDAR, optical or hyperspectral imaging. Historically a lot of these groupings have tended to work in relative isolation with a limited amount of interaction between them, which has created a lot of EO data, and knowledge, silos. However as satellite technology has developed, the quantity of EO data available has increased exponentially; for example, Landsat is currently collecting fourteen times as many images each day than it was in the 1980’s. Whilst many datasets have been collected, few have been brought together. This is due to both computing power required to manage large datasets and the difficulties of cross-calibrating sensors with different errors and uncertainties. Cloud computing has broken through most of the data processing obstacles, giving the potential for many more people to get involved in data manipulation, modelling and visualisation. The next challenge is to smash open these data silos, and provide access to historical archives, and new collections, to both the scientific community, and anyone else who is interested. Joining together the different strands of data and knowledge will promote innovation and help us significantly develop our understanding of the planet. Individual space agencies are working on this through making new data freely available and by analysing their own historical archives and then reprocessing them to improve consistency. Some examples include:

Progress is being made, but there are still limitations as often this only represents the bringing together of data from a single mission; a product set or thematic group. There is a need to be bolder and to amalgamate much wider datasets. Last week, Taiwan demonstrated how this could be achieved by presenting their petascale database for assessing climatic conditions, which has brought together data from the atmosphere, hydrology, ocean currents, tectonics and space. The Earth Science Observation Knowledge base holds ten and half million records and gives scientists near real time access to data. EO has a vast array of valuable data and is collecting more every day. We’re starting to smash the data silos, but we need to do more to achieve the next step change in understanding how our world works.