New B2B commerce study
Download now
Supporting ESA with innovative solutions for data processing
Optimized data processing for faster data product generation
The European Space Agency (ESA) is an intergovernmental organization dedicated to space exploration. In collaboration with scientific partners, ESA faced the challenge of efficiently processing large volumes of Earth observation data to extract valuable scientific insights. With the CAWA project, aimed at enhancing atmospheric data products, Cloudflight was brought on board to provide technical expertise and innovative solutions for data processing.
The challenge
Optimize complex data processing algorithms
ESA and its scientific partners faced challenges when optimizing complex data processing algorithms essential for analyzing large-scale Earth observation data. The challenge was to streamline these processes while maintaining the scientific accuracy and value of the data. Additionally, the existing legacy code, developed in Fortran, was not optimized for modern data processing requirements, posing significant hurdles in terms of parallelization, automation, and scalability. These technical limitations had to be addressed while ensuring seamless integration with ESA’s existing workflows.
The solution
Modernized data processing pipeline
The approach centered on developing a tailored solution to modernize the data processing pipeline, leveraging cutting-edge cloud computing and data engineering techniques. By optimizing the existing algorithms, parallelizing workflows, and automating data handling processes, the goal was to significantly reduce processing time while maintaining scientific rigor. This solution aimed to enhance the usability of Earth observation data, transforming raw data into valuable insights for the scientific community, and preparing it for future space missions.
The value
Efficient data processing framework
In the end, ESA benefited from an efficient data processing framework, utilizing advanced technologies such as Python, DASK, and Kubernetes. The solution optimized the existing Fortran-based algorithms by implementing parallelization and automation, which significantly reduced the time required to process large datasets. Additionally, a multi-tile approach for data partitioning was introduced, enabling scalable processing of Earth observation data.
Our partner
European Space Agency (ESA)
The European Space Agency (ESA) is Europe’s gateway to space. Its mission is to shape the development of Europe’s space capability and ensure that investment in space continues to deliver benefits to the citizens of Europe and the world.