Boosting HPC capacity with net-zero emissions at Shell’s data center

Boosting HPC Capacity with Lower Emissions at Shell’s Houston Data Center

Data Centers connect supply chains, enable collaboration and innovation, maintain information security, and increasingly keep society and the economy running.

Meanwhile the use of data-intensive technologies including AI, machine learning and the Internet of Things are spurring exponential growth in demand for server space and placing ever greater demands on modern data centers.
As both an energy user and provider, Shell plc (Shell Group) is facing up to these challenges firsthand. In managing its own data centers, its information technology organization recognizes performance must be balanced against cost and sustainability objectives. Most recently for its high-performance computing (HPC) cluster within its Houston data center, Shell Information Technology International (Shell IT) found Penguin Computing Altus servers, powered by AMD EPYC processors combined with immersion cooling technology were an essential piece of the puzzle to help optimize performance relative to cost, while revolutionizing system efficiency.

Satisfying an increasing need for computational power

“The Shell Group has set a target to become a net-zero emissions energy business by 2050,” says Sjors van de Rijt, Global Head of Sustainability Partnerships, Technology & Energy, Shell Energy. “We play several roles on this journey. Our activities range from that of an energy producer and provider, to energy user ourselves, as well as partner for change. And it’s in support of all these activities that our HPC team in Shell IT is helping contribute to that goal. Delivering computational facilities for more efficient exploration and production of oil and natural gas is an important part of what Shell IT delivers. But our computing needs also increasingly support a range of digital initiatives, for example, innovative electric vehicle (EV) charging solutions that can connect communities in safer, cleaner and smarter ways.”

Digital solutions are a crucial part of the Shell Group’s business and a fundamental enabler of both its own and its customers’ sustainability goals. “Technology is unlocking new possibilities for clean energy systems, helping to optimize existing operations and enabling emissions to be tracked and reported more accurately,” says van de Rijt. "But digitalization also means increasing data and workloads, which require more energy, and impacts on system performance, cost, and carbon footprint are what my team within Shell Energy are working together with customers to help address.”

“The Shell Group is a very large consumer of data, data storage, and analytics, which then goes into machine learning and AI,” says David Baldwin, High Performance Computing Manager, Shell IT. “We are a user and a buyer of cloud platforms, on-premises services, and digital twins. Through Shell Energy we also market energy and efficiency solutions to the tech industry. We sit in all parts of that value chain.”

“We use a wide range of applications,” says Baldwin. “This includes upstream applications, such as seismic processing, looking for oil and gas reserves, and carbon capture targets. Shell IT also supports everything from computational chemistry, fluid dynamics and reservoir engineering, to smart grids, as well as AI and machine learning. Across all these applications, we fine tune the systems to optimize performance and cost effectiveness. We are implementing smart controls, virtualization software, and artificial intelligence (AI) to help intelligently manage workloads, distributing computing tasks more evenly and maximizing utilization rates.”

“The company has decades of accumulated data. Processing this data poses significant challenges around power consumption, particularly as processor chip wattage continues to increase,” continues Baldwin.

Shell’s Houston data center already draws from 100% renewable power supplied by Shell Energy North America. The challenge for Shell IT’s HPC team is how to drive down its Power Usage Effectiveness (PUE) ratio while boosting performance. This motivated the upgrade to Penguin Computing Altus servers, powered by AMD EPYC processors paired with immersion cooling technology. “Penguin has 25 years of experience building and deploying large HPC clusters that run some of the world’s most demanding workloads,” says Phil Pokorny, Chief Technology Officer, Penguin Solutions. “Our technology partnerships allow us to be at the forefront of integrating new and emerging technologies, like immersion cooling.”

Solving the data center cooling problem

Shell IT’s relationship with AMD EPYC processors goes back six years. “We changed over because most of our applications are memory bandwidth limited, and AMD EPYC chips had the advantage in this area,” says Baldwin. During this period, Shell IT began working with Penguin Solutions as their technology partner. “We get early access to new technologies with Penguin.”

Shell IT worked with Penguin to find a way to keep the increasingly power-hungry processors cool. “We aim to get very high memory bandwidth from our CPUs,” says Baldwin. “We're always picking either top bin processors or just one down. In the past six years alone, we've gone from 140-watt chips to 360-watt chips. This has taken us to the point where air cooling isn't sufficient. We implemented direct-to-chip liquid cooling, but we felt we needed to take that next step to immersion cooling. We continue to push the bounds of what’s possible in terms of performance, efficiency, and sustainability.”

“When we run direct-to-chip liquid cooling, this covers around two-thirds of the heat in the server, which is primarily the CPU,” says Baldwin. “You still have the other third you're trying to air cool, which is inefficient. Memory and networks are getting hungry for power as we increase the bandwidth. Then there are the storage devices as well. With immersion cooling, you submerge everything in a non-conducting (dielectric) fluid that absorbs heat more than a thousand times better than air, so you're dramatically improving your ability to deal with all the heat being produced.”

Shell IT has installed 864 dual-socket systems in its new Houston data center, using 96-core 4th Gen AMD EPYC 9654 CPUs, for a total of 1,728 processors and 165,888 cores. “We see a step change from a 4th Gen AMD chip over 3rd Gen,” says Baldwin. “They can do a lot more. We have more cores and more memory. We have managed to make our money go further.” But this has accentuated the need for effective cooling. “As we've gone to more cores and higher wattage on the chips, our data center rack consumption has risen. We run air cool racks at roughly 30kW. We can run the direct-to-chip liquid cooling at about 60kW per rack, and the new immersion-cooled machines run at about 100kW per rack.”

Showcasing the performance benefits of immersion

“Around two years ago, we first deployed immersion cooling for our high-performance computing cluster at a Shell data center in Amsterdam,” says Baldwin. “Both use Shell immersion cooling fluids, but our latest installation in Houston is considerably bigger, and also benefits from renewable power and smart energy management solutions supplied by Shell Energy.” Shell Lubricants is a pioneer in the new frontier of immersion cooling fluids for data centers, and together with Shell Energy, is utilizing these deployments to demonstrate fully integrated, optimized, and scalable solutions suitable for enterprise-level HPC. “At Shell, our businesses are facing many of the very same challenges with our own data hosting and energy needs as our customers,” says van de Rijt. “This places us in a unique position to help leading companies in the technology industry and beyond through first-hand knowledge of products and services that we believe will be instrumental to meeting their business and sustainability goals.”

Although the overall power per rack has increased, the core density of the AMD EPYC processors means it’s a much more efficient solution than air cooling by having a data center footprint that must be spread out spatially to achieve the same performance. “The bigger the footprint becomes, the harder it is to manage the air in the data center,” says Baldwin. “You end up with hotspots. There’s also networking challenges. If you space all your computers out, you need more fiber-optic cabling, which is extremely expensive. Immersion cooling allows us to consolidate more nodes into a smaller power density footprint.”

“As a data center operator, we believe immersion cooling is the future,” says Baldwin. “Penguin Solutions has been instrumental in helping us bring the right technologies together to deliver on Shell IT’s business and sustainability goals along with AMD. The price, performance, and memory bandwidth of AMD EPYC CPUs are perfect for our needs. They are world class.”