New B2B commerce study
Download now
Solving earthly projects with space information
Solving problems on earth by harnessing space – this is Spire’s mission, and they achieve it with more than 120 multi-purpose satellites as well as an extensive network of ground stations. In their pursuit of making the planet a better place, they collect data in four key focus areas: Aviation, Weather, Maritime, and Earth Intelligence.
Cloudflight supports Spire in scaling their data platform team, with a particular focus on co-creating an aviation product based on processing Automatic Dependent Surveillance–Broadcast (ADS-B) data. ADS-B uses GPS satellites and other navigation tools to help determine an aircraft’s location at any given time. ADS-B provides many benefits to flight services operators such as airlines and airports on the one hand, to data aggregators such as Spire Aviation on the other hand that improve the efficiency of flight and support the collection of data in order to prevent unforeseen events from happening.
With our help, Spire Aviation can not only handle large amounts of data but also develop new features and customer-oriented products with low latency.
The challenge
Seamlessly integrating a new team into an existing one and handling Big Data
The main challenge of the project? Getting more work power. Here, however, the hurdle comes two-fold. On one side, a suitable collaboration partner must have the right know-how not to impair the project’s timeline and scope. On the other side, a company cannot afford to have a partner whose expertise ends up turning upside down the established working mode.
What we provided was extensive expertise, joining Spire’s team, and adapting to the status quo – the result was small yet major steps in the right direction.
Next to the team setup there was an additional technical challenge: The large amount of single data points that were gathered over the last couple of years and continously add to the system – 200 billion data points needed to be processed, stored and in the end delivered to customers via a comprehensive and easy to use API.
What makes this amount of data interesting is the requirement to still retrieve a certain excerpt of the data within a tolerable response time. In addition, we needed to reprocess the entire data every once in a while when additional patterns needed to be identified or the details of the extraction logic were updated. Making this process robust also under massive parallelization comes with some challenges.
Digression: What is ADS-B tracking?
• A form of surveillance technology, allowing air-to-ground as well as air-to-air communications
• Provides real-time information on aircraft position and parameters: twice per second whereas regular radar technology needs 4 seconds to update an aircraft’s position
• Enables real-time decisions of air traffic controllers (ATCs)
Why go for ADS-B tracking?
The Solution
Tracking planes from above
Cloudflights team is supporting Spire in the creation of an ADS-B-based product that uses aircraft position and flight schedule to create a comprehensive events-and-flights API.
Instead of capturing ADS-B data mostly through terrestrial receivers, as it happens usually with conventional providers, the aircraft additionally broadcast their messages also in upward direction, allowing Spire’s satellite constellation (the largest multi-use nano satellite constellation) to pick them up.
This provides detailed information about all current airborne planes even when it comes to coverage over ocean, deserts and other remote regions – a game-changer in predictive maintenance and fuel optimization. For example, planning of flight routes within a suitable landing window as well as optimizing to burn the minimum amount of jet fuel as possible.
Data storage and processing has been done using storage and compute clusters on demand on AWS. This allows for very high parallelism during (re-)processings of the hundreds of TB of data while saving costs at other times. Tools and frameworks to process the data include the cloud big data platform AWS EMR (HBase, Flink, other data warehouse solutions), Kafka as well as ElasticSearch. Cloud resources are managed via Infrastructure as Code and are therefore integrated and connected to the continuous development of the whole solution.
Spire
While information about the earth and its environment has traditionally been collected from ground networks and satellite imagery, Spire set out to do something different: create a vast network of affordable nanosatellites that orbit just above Earth’s atmosphere, collecting rich, granular data from the ultimate vantage point. Spire Aviation empowers businesses and mission teams to develop new solutions and draw critical insights using space-based global flight traffic data.
