The weather forecast generally influences daily decisions like what to wear in the morning or a baseball game for the kids this weekend. Mostly, all types of businesses need this information, such as commodity traders for reporting their trading positions or transport companies to prepare for potential shipping and logistics disruptions.
As of now, it takes an average of 100 minutes to process data for weather prediction. This is a lot of time that can prove detrimental for an adequate response. To speed this time up, Maxar Technologies Inc. embarked on a project and partnered with National Oceanic and Atmospheric Administration (NOAA) and Amazon Web Services. They have developed a high-performance computing solution that can generate numerical weather forecasts in 53 minutes. It almost brings down the prediction time to half. Maxar recently received a “Best High-Performance Computing Solution” award from AWS Public Sector in recognition of this accomplishment.
“When we started running the model in a small-scale HPC environment, we discovered some challenges in matching the output with what NOAA was providing through its public access points. We figured if we couldn’t match NOAA’s output using the same configuration of their FV3GFS model, then we had something awry in our system,” said Travis Hartman, Director of Analytics Products division at Maxar.
He further stated, “Thanks to some late nights, coffee and a ton of passion and dedication from the Maxar and AWS team, we combed through the FV3GFS source code and the various supporting libraries, as well as system architecture configs, to get the output to match. From there, it was all about scaling, developing the Continuous Integration & Continuous Delivery (CI/CD) workflows, establishing the post-processing and designing the subsequent model configurations that would allow us to not just make a faster forecast quicker (i.e., more cost efficient and less capital intense) but also make better forecasts!”
Automated Workflow within AWS
Maxar has four active satellites orbiting the Earth that collect images of 3 million square kilometres per day. The company also designed a cloud HPC cluster with 234 Amazon EC2 instances, in tandem with NOAA and the National Weather Service to provide forecasts for protecting life and property in the country.
By combining teams of data scientists and engineers, Maxar and AWS were able to create automated workflows, which upgrades the entire set of AWS tools and services, as well as allows Maxar physicists to dig into the Earth Science side of NWP.
Hartman further stated, “This setup allows us to reach a similar forecast to what NOAA’s supercomputer produces – but we get the results a lot faster. Depending on the configuration (again – flexibility thanks to the AWS Cloud) and the various customer requirements, we can cut the computational time in half for the model solution and we have our sights on likely finding another 50% reduction in computation time relative to the NOAA setup. Additionally, the cost to develop and operate this system is significantly below the $500+ million, 10-year program deal NOAA inked a few years ago, as well as the additional tens of millions they’ve allocated in subsequent funding for their fleet of supercomputers. Admittedly, that’s not necessarily a fair comparison given their machines are humming along doing all sorts of computations beyond just the FV3GFS, but we also didn’t need an act of Congress to go from “zero” to “full-production,” and we did it in a matter of weeks (again – kudos to the AWS support team and the Maxar data scientists and engineers who rocked this)!”
Achieving such an astonishing feat in the area of weather prediction is no easy thing. Weather prediction technology can benefit a lot from the technological advancements offered by Cloud computing. We hope there are more such collaborations in the future, they lead to advancements in weather predictions and help agencies better prepare for weather-related emergencies in advance.
Please support our mission of simplifying the complex cloud ecosystem. We exist to serve the new generation of Geeks with information that shapes the cloud ecosystem. We help our audience get informed and engaged with every seismic change around the world of Cloud. The changes will help determine the future of technology in the next decade.