Making Big Data Bite-size

Earth Observation satellites beam enormous amounts of data to us about our planet every day. To manage and use this Big Data efficiently, scientists and technologists around the world need a way to ‘bite-size’ it. SERVIR is considering one such method – data cubes – to help its hubs and end users in developing countries access, organize, and use the data.

The SERVIR Science Coordination Office invited data cube experts to visit and hold an onsite workshop so SERVIR scientists and Geospatial Information Technology (GIT) staff could become familiar with the method and assess its potential usefulness for SERVIR-Eastern & Southern Africa, SERVIR-West Africa, SERVIR-Himalaya, and SERVIR-Mekong. These hubs help end users in their regions use Earth Observation data for decision making in water resources management, natural resources management, agriculture, and more.

By stacking massive amounts of Earth Observation satellite data from multiple time periods into a data cube, users can quickly perform complex analyses for their area of interest.

“You might not care about the other ¾ of the scene; we find the pixels you care about, saving you tremendous amounts of time” explains NASA Langley Research Center’s Brian Killough, who led the training. “And because we use pre-processed data, once those pixels are in the data cube, they’re ready to be analyzed – with confidence.”

Killough heads up the international Committee on Earth Observation Satellites (CEOS) Systems Engineering Office. CEOS has been working with Colombia’s Institute of Hydrology, Meteorology and Environmental Studies (IDEAM) and the University of the Andes in Bogotá, helping them build and use data cubes of Landsat data to support deforestation monitoring and planning.

“They told us that they used to spend 80% of their time just getting and processing data to use. With data cube, we were able to give them a 2-terabyte drive with Landsat data for a portion of their country, for the year 2000 forward to now. It took us less than a week to order the processed data and create the cube. IDEAM staff members said it would have taken them a full year to find and download that data and have it processed into analysis-ready format.”

Now IDEAM personnel are using data cubes on their own to get similar data for the rest of the country.

“Data cube opens up a whole new set of science to non-technical users in developing countries,” says Killough.

“We’re very excited to strengthen our capacity to create and implement data cubes, in particular to support potential needs in the regions with whom we work around the globe,” says Dan Irwin, NASA’s SERVIR Global Program Manager.

Notes:

• Data cubes were developed by Geoscience Australia in partnership with the Commonwealth Scientific and Industrial Research Organization (CSIRO), which is the federal government agency for scientific research in Australia, and the National Computational Infrastructure at the Australian National University.
• The CEOS data cube effort is intentionally using Open Source technology and freely available data, to make it possible for other governments to easily adapt and implement their own versions of the Data Cube.
• The Data Cube consortium is now actively working to build a global community that further develops the open-source codebase as a resource for the benefit of the broader international community. Through partnerships with CEOS, the technology has already been adapted and is being implemented in Cameroon, Colombia, and Kenya.