MPAS - Beta

 

The Model for Prediction Across Scales (MPAS) is a collaborative project for developing atmosphere, ocean, and other earth-system simulation components for use in climate, regional climate, and weather studies. The primary development partners are the climate modeling group at Los Alamos National Laboratory (COSIM) and the National Center for Atmospheric Research. Both primary partners are responsible for the MPAS framework, operators, and tools common to the applications; LANL has primary responsibility for the ocean model, and NCAR has primary responsibility for the atmospheric model.

The MPAS framework facilitates the rapid development and prototyping of models by providing infrastructure typically required by model developers, including high-level data types, communication routines, and I/O routines. By using MPAS, developers can leverage pre-existing code and focus more on development of their model.

More information can be found here: https://mpas-dev.github.io/

 

Single-Node Server Requirements

---

CPUs / GPUs	Operating Systems	ROCm™ Driver	Container Runtimes
X86_64 CPU(s) AMD Instinct MI100 GPU(s) AMD Instinct MI200 GPU(s)	Cray/HPE	ROCM 5.0.2	N/A

 

Licensing Information

---

Your use of this application is subject to the terms of the BSD-3 License (https://opensource.org/licenses/BSD-3-Clause) as set forth below. By accessing and using this application, you are agreeing to fully comply with the terms of this license.  If you do not agree to the terms of this license, do not access or use this application.

Copyright © 2023 Advanced Micro Devices, Inc. (AMD)

Copyright © 2013, Los Alamos National Security, LLC (LANS) (Ocean: LA-CC-13-047; Land Ice: LA-CC-13-117) and the University Corporation for Atmospheric Research (UCAR)
All rights reserved.

LANS is the operator of the Los Alamos National Laboratory under Contract No. DE-AC52-06NA25396 with the U.S. Department of Energy. UCAR manages the National Center for Atmospheric Research under Cooperative Agreement ATM-0753581 with the National Science Foundation. The U.S. Government has rights to use, reproduce, and distribute this software. NO WARRANTY, EXPRESS OR IMPLIED IS OFFERED BY LANS, UCAR OR THE GOVERNMENT AND NONE OF THEM ASSUME ANY LIABILITY FOR THE USE OF THIS SOFTWARE. If software is modified to produce derivative works, such modified software should be clearly marked, so as not to confuse it with the version available from LANS and UCAR.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

 

Building Application

---

Step 0) Prerequisites:

AMD GPU offloading is enabled only through the Cray ftn compiler (CCE 14) at this time. Appropriate modules must be loaded (e.g., rocm/5.0.2 and craype-accel-amd-gfx90a).

The following libraries must be installed:
- NETCDF
- PNETCDF
- ParallelIO (https://github.com/NCAR/ParallelIO )

NETCDF and/or PNETCDF should already be provided on most HPE systems. If not, the user is responsible for providing standard builds with CC=cc and FC=ftn.

 

To build ParallelIO, follow these instructions:

git clone https://github.com/NCAR/ParallelIO
cd ParallelIO && mkdir build && cd build
cmake .. CC=cc FC=ftn \
-DNetCDF_C_PATH=/path/to/netcdf \
-DNetCDF_Fortran_PATH=/path/to/netcdf \
-DPnetCDF_PATH=/path/to/pnetcdf \
-DCMAKE_INSTALL_PREFIX=… \
-DCMAKE_Fortran_FLAGS=-ef
make -j8 install

Minimum 2 GPUs (or 1 MI250/Mi250X GPU) required for this application. GPU-aware MPI is also required.

Step 1) Download MPAS-Model source code

Obtain the MPAS-Model source code using the following commands:

git clone https://github.com/jychang48/MPAS-Model.git
cd MPAS-Model
git checkout atmosphere/develop-openacc-AMD-ftnpass

Step 2) Download JW Baroclinc Wave benchmark:

Download and extract the tar file from here: http://www2.mmm.ucar.edu/projects/mpas/test_cases/v7.0/jw_baroclinic_wave.tar.gz

Step 3) Environment Variables:
Following environmental variables must be set for building MPAS:

NETCDF=/path/to/netcdf
PNETCDF=/path/to/pnetcdf
PIO=/path/to/ParallelIO
USE_PIO2=true
OPENMP_AMD_OFFLOAD=true

Step 4) Compiling init_atmosphere_model:
Execute the following make command inside MPAS-Model:

cd /path/to/MPAS-Model
make CORE=init_atmosphere ftn-offload
cp init_atmosphere_model /path/to/jw_baroclinic_wave

This will create a new binary called init_atmosphere_model. Copy this binary into the benchmark directory:

cp init_atmosphere_model /path/to/jw_baroclinic_wave

Step 5) Compiling atmosphere_model:
Execute the following make command inside MPAS-Model:

make CORE=init_atmosphere clean
make CORE=atmosphere ftn-offload

This will create a new binary called atmosphere_model. Copy this binary into the benchmark directory:

cp atmosphere_model /path/to/jw_baroclinic_wave

Step 6) Preprocessing with init_atmosphere_model:
Execute the following make command inside /path/to/jw_baroclinic:

srun -n 1 ./init_atmosphere_model

This will create a new filed called x1.40962.init.nc and only needs to be created once. 

 

Running Application

---

2 GPUs/GCDs:
Set the following runtime environment variables:

AMD_DIRECT_DISPATCH=1
ROC_ACTIVE_WAIT_TIMEOUT=0
MPAS_DYNAMICS_RANKS_PER_NODE=2
MPAS_RADIATION_RANKS_PER_NODE=2
MPICH_GPU_SUPPORT_ENABLED=1

Execute the following make command inside /path/to/jw_baroclinic_wave:

srun -n 4 ./atmosphere_model

4 GPUs/GCDs:
Set the following runtime environment variables:

AMD_DIRECT_DISPATCH=1
ROC_ACTIVE_WAIT_TIMEOUT=0
MPAS_DYNAMICS_RANKS_PER_NODE=4
MPAS_RADIATION_RANKS_PER_NODE=2
MPICH_GPU_SUPPORT_ENABLED=1

Execute the following make command inside /path/to/jw_baroclinic_wave:

srun -n 6 ./atmosphere_model

8 GPUs/GCDs:
Set the following runtime environment variables:

AMD_DIRECT_DISPATCH=1
ROC_ACTIVE_WAIT_TIMEOUT=0
MPAS_DYNAMICS_RANKS_PER_NODE=8
MPAS_RADIATION_RANKS_PER_NODE=2
MPICH_GPU_SUPPORT_ENABLED=1 

Execute the following make command inside /path/to/jw_baroclinic_wave:

srun -n 10 ./atmosphere_model

 

Troubleshooting

---

If run successfully, you should have a log file called log.atmosphere.role01.0000.out.

For more information on MPAS, including visualization, please visit the official website: https://mpas-dev.github.io/

 

Known Issues

---

This AMD GPU build recipe is only applicable to HPE/Cray systems with CCE 14.

 

Disclaimer

---

The information contained herein is for informational purposes only, and is subject to change without notice. In addition, any stated support is planned and is also subject to change. While every precaution has been taken in the preparation of this document, it may contain technical inaccuracies, omissions and typographical errors, and AMD is under no obligation to update or otherwise correct this information. Advanced Micro Devices, Inc. makes no representations or warranties with respect to the accuracy or completeness of the contents of this document, and assumes no liability of any kind, including the implied warranties of noninfringement, merchantability or fitness for particular purposes, with respect to the operation or use of AMD hardware, software or other products described herein. No license, including implied or arising by estoppel, to any intellectual property rights is granted by this document. Terms and limitations applicable to the purchase or use of AMD’s products are as set forth in a signed agreement between the parties or in AMD's Standard Terms and Conditions of Sale.

 

Notices and Attribution

---

© 2023 Advanced Micro Devices, Inc. All rights reserved. AMD, the AMD Arrow logo, Instinct, Radeon Instinct, ROCm, and combinations thereof are trademarks of Advanced Micro Devices, Inc.

Docker and the Docker logo are trademarks or registered trademarks of Docker, Inc. in the United States and/or other countries. Docker, Inc. and other parties may also have trademark rights in other terms used herein. Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.

All other trademarks and copyrights are property of their respective owners and are only mentioned for informative purposes.