Descartes Labs Achieves #40 in TOP500 with Cloud-based Supercomputing Demonstration Powered by AWS, Signaling New Era for Geospatial Data Analysis at Scale
Descartes Labs uses Amazon Web Services, Inc. (AWS) to run a 9.95 petaflops High-Performance LINPACK (HPL) benchmark, placing it #40 in the June 2021 TOP500 ranking. The company improves on its previous AWS-based 2019 TOP500 submission (1.926 petaflops, #136) by 417% in HPL performance and 96 ranking spots in only two years.
Jun 28, 2021 · 4 min read
The volume of global sensor data processed by Descartes Labs requires cutting edge cloud technology capable of running High Performance Computing (HPC) and Machine Learning (ML) workflows at scale. This allows the Descartes Labs engineering teams to compute, analyze, and store near real-time data in order to deliver mission critical business analytics at a global scale to customers in the agriculture, mining, defense and intelligence, and consumer packaged goods (CPG) industries. This capability is becoming increasingly important for applications that rely on a combination of geospatial analytics on a global level, such as environmental monitoring, severe weather tracking and forecasting, multimodal data fusion, and climate change analysis. An example of this mission are the six petabytes of Sentinel-1 radar data recently processed by Descartes Labs to help track deforestation at continental scale, understand where wildfire risk is highest, and monitor regenerative agriculture practices.
In collaboration with the AWS HPC team, Descartes was able to spin up the resources needed for its TOP500 run using general purpose, publicly available Amazon EC2 instances across C5/M5/R5 families. Thanks to the scale and elasticity of the AWS Cloud, Descartes Labs was able to scale up infrastructure for this test without affecting its production workflows. Additionally, due to the elasticity of Amazon EC2’s compute resources, Descartes was also able to test out different cluster sizes with multiple HPL runs spanning 1 petaflops to 8.5 petaflops, leading up to the final 9.95 petaflops result.
“The fact that Descartes Labs was able to use Amazon EC2 to build a 9.95 petaflops supercomputer ranking #40 on the TOP500 list is a truly impressive accomplishment,” said Dave Brown, Vice President, Amazon EC2, AWS. “AWS is proud to support Descartes Labs in their mission to develop planetary scale geospatial insights for their customers. We expect the demand for instant data and insights to grow into the future, and this is a powerful demonstration of the value that AWS can bring to customers and partners on any scale.”
The LINPACK benchmark used in this supercomputing demonstration helps to approximate how fast a computer will perform when solving "tightly coupled" real world-problems, such as weather forecasting. By building systems that can run the LINPACK benchmark at high performance, Descartes Labs can use the same systems to model the weather quickly, accurately, and over multiple locations simultaneously to enhance agricultural yield forecasts.
Such cutting-edge science has been instrumental in Descartes Labs' collaboration with customers like CJ OliveNetworks, an organization that recently migrated all of CJ Group Companies’ IT systems to AWS. The company leveraged Descartes Labs technology to carry out a purchasing innovation project to predict sugar input prices for CJ CheilJedang, which imports 600,000 tons of raw sugar per year. CJ has achieved successful business innovation from this project through satellite imaging technology, climate, and vegetation data analysis.
Says CEO of OliveNetworks, and Group Chief Digital Officer In hyok Cha, "I wish to congratulate both Descartes Labs and AWS on achieving such a large scale computation to support the infrastructure work behind planetary sustainability initiatives, and also wish to continue to collaborate with the two companies to develop more innovative earth observation use cases and business applications."
About Descartes Labs
View Recent Blogs
Sep 21, 2022 · 4 min read
This is Part 5 of a technical series focusing on Descartes Labs’ global SAR processing...