Trilinos

Trilinos natively supports Serial, OpenMP, CUDA (NVidia), and HIP (AMD) layers for CPUs and associated accelerators. To run the benchmark within this container, a supported AMD GPU must be available. To build against the Trilinos installed in this container, a valid GPU is not required, however it will be required to run the resulting binary. The server must also have a Docker Engine installed to run the container.

Single-Node Server Requirements

---

CPUs	GPUs	Operating Systems	ROCm™ Driver	Container Runtimes
X86_64 CPU(s)	AMD Instinct MI200 GPU(s) AMD Instinct MI100 GPU(s)	Ubuntu 20.04 Red Hat 8	ROCm v5.x compatibility	Docker Engine Singularity

Please visit the Docker Engine install web site at https://docs.docker.com/engine/install/ to install the latest Docker Engine for the operating system installed on the server.

Please visit the Singularity Documentation site at https://sylabs.io/docs/ for the latest Singularity install documentation.

Please visit Introduction to AMD ROCm Installation Guide for Linux for information on the ROCm driver installation and compatibility with AMD GPUs.

 

Running Containers

---

The benchmark provided in this container is an algebraic multigrid (AMG) preconditioned conjugant gradient (CG) solve applied to a 27 point structured finite-difference stencil. By default it operates on a 256x256x256 grid of points using 10 CG solver iterations with a Chebyshev smoother. Output timing includes the fastest AMG preconditioner setup time, CG solve time, and the total benchmark time which is the sum of the setup and solve times. This benchmark is included as a test of the linear solver components of Trilinos for the purposes of testing linear solver operations (e.g. SpMV, SpGEMM) as well as scaling performance on AMD hardware. The benchmark can be found in the /benchmark directory, and is not included as a component of Trilinos itself.

Using Docker 

Interactive

To run the container interactively run the following command, substituting the image name from the Pull Command section above if newer:

docker run --device=/dev/kfd --device=/dev/dri --security-opt seccomp=unconfined --cap-add CAP_SYS_PTRACE --shm-size=8g -e HIP_VISIBLE_DEVICES=0 -it amdih/trilinos:1.7 /bin/bash

At this point you can move to the /benchmark directory, and run the benchmark directly using the command:

cd /benchmark
mpirun -n 1 ./run_benchmark --mx=1 --my=1 --mz=1 --kokkos-num-devices=1

To run the benchmark on multiple GPUs (e.g. 4) launch the docker container and define the GPUs to target using the variable

HIP_VISIBLE_DEVICES:
docker run --device=/dev/kfd --device=/dev/dri --security-opt seccomp=unconfined --cap-add CAP_SYS_PTRACE --shm-size=8g -e HIP_VISIBLE_DEVICES=0,1,2,3 -it amdih/trilinos:1.7 /bin/bash

Then run the benchmark, but be sure to set the partitioning of the mesh along x, y, and z (using the flags --mx, --my, and --mz) and tell Kokkos to assign the GPUs to the different MPI ranks in a round-robin fashion (using --kokkos-num-devices):
cd /benchmark
mpirun -n 4 ./run_benchmark --mx=2 --my=2 --mz=1 --kokkos-num-devices=4

Non-Interactive 

To run the benchmark for 1, 2, 4, and 8 GPUs, use the commands documented in the following sections.

1 GPU

docker run --device=/dev/kfd --device=/dev/dri --security-opt seccomp=unconfined --cap-add CAP_SYS_PTRACE --shm-size=8g -e HIP_VISIBLE_DEVICES=0 amdih/trilinos:1.7 mpirun -n 1 /benchmark/run_benchmark --mx=1 --my=1 --mz=1 --kokkos-num-devices=1

2 GPUs

docker run --device=/dev/kfd --device=/dev/dri --security-opt seccomp=unconfined --cap-add CAP_SYS_PTRACE --shm-size=8g -e HIP_VISIBLE_DEVICES=0,1 amdih/trilinos:1.7 mpirun -n 2 /benchmark/run_benchmark --mx=2 --my=1 --mz=1 --kokkos-num-devices=2

4 GPUs

docker run --device=/dev/kfd --device=/dev/dri --security-opt seccomp=unconfined --cap-add CAP_SYS_PTRACE --shm-size=8g -e HIP_VISIBLE_DEVICES=0,1,2,3 amdih/trilinos:1.7 mpirun -n 4 /benchmark/run_benchmark --mx=2 --my=2 --mz=1 --kokkos-num-devices=4

8 GPUs

docker run --device=/dev/kfd --device=/dev/dri --security-opt seccomp=unconfined --cap-add CAP_SYS_PTRACE --shm-size=8g -e HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 amdih/trilinos:1.7 mpirun -n 8 /benchmark/run_benchmark --mx=2 --my=2 --mz=2 --kokkos-num-devices=8

Using Singularity

This section assumes that an up-to-date version of Singularity is installed on your system and properly configured for your system. Please consult with your system administrator or view official Singularity documentation.

Pull and convert docker image to singularity image format, substituting the image name from the Pull Command section above if newer:

singularity pull trilinos.sif docker://amdih/trilinos:1.7

You can then use examples from the preceding section to use the image. For example, to run the benchmark problem, launch a container in interactive mode:

singularity run --pwd /benchmark --writable-tmpfs Trilinos.sif /bin/bash

From within the singularity container or replace /bin/bash with the following commands to execute the benchmarks:

1 GPU

mpirun -n 1 /benchmark/run_benchmark --mx=1 --my=1 --mz=1 --kokkos-num-devices=1

2 GPUs

mpirun -n 2 /benchmark/run_benchmark --mx=2 --my=1 --mz=1 --kokkos-num-devices=2

4 GPUs

mpirun -n 4 /benchmark/run_benchmark --mx=2 --my=2 --mz=1 --kokkos-num-devices=4

8 GPUs

mpirun -n 8 /benchmark/run_benchmark --mx=2 --my=2 --mz=2 --kokkos-num-devices=8

 

Licensing Information

---

Your use of this application is subject to the terms of the applicable component-level license identified below. To the extent any subcomponent in this container requires an offer for corresponding source code, AMD hereby makes such an offer for corresponding source code form, which will be made available upon request. By accessing and using this application, you are agreeing to fully comply with the terms of this license. If you do not agree to the terms of this license, do not access or use this application. 

The application is provided in a container image format that includes the following separate and independent components: Ubuntu (License: Creative Commons CC-BY-SA version 3.0 UK licence), CMAKE (License: BSD 3), OpenMPI (License: BSD 3-Clause), OpenUCX (License: BSD 3-Clause), ROCm (License: Custom/MIT/Apache V2.0/UIUC OSL), Trilinos (License: BSD 3-Clause, LGPL). Additional third-party content in this container may be subject to additional licenses and restrictions. The components are licensed to you directly by the party that owns the content pursuant to the license terms included with such content and is not licensed to you by AMD. ALL THIRD-PARTY CONTENT IS MADE AVAILABLE BY AMD “AS IS” WITHOUT A WARRANTY OF ANY KIND. USE OF THE CONTAINER IS DONE AT YOUR SOLE DISCRETION AND UNDER NO CIRCUMSTANCES WILL AMD BE LIABLE TO YOU FOR ANY THIRD-PARTY CONTENT. YOU ASSUME ALL RISK AND ARE SOLELY RESPONSIBLE FOR ANY DAMAGES THAT MAY ARISE FROM YOUR USE OF THE CONTAINER. 

See Trilinos Licenses for more information.

 

Disclaimer

---

The information contained herein is for informational purposes only, and is subject to change without notice. In addition, any stated support is planned and is also subject to change. While every precaution has been taken in the preparation of this document, it may contain technical inaccuracies, omissions and typographical errors, and AMD is under no obligation to update or otherwise correct this information. Advanced Micro Devices, Inc. makes no representations or warranties with respect to the accuracy or completeness of the contents of this document, and assumes no liability of any kind, including the implied warranties of noninfringement, merchantability or fitness for particular purposes, with respect to the operation or use of AMD hardware, software or other products described herein. No license, including implied or arising by estoppel, to any intellectual property rights is granted by this document. Terms and limitations applicable to the purchase or use of AMD’s products are as set forth in a signed agreement between the parties or in AMD's Standard Terms and Conditions of Sale.

 

Notices and Attribution

---

© 2023 Advanced Micro Devices, Inc. All rights reserved. AMD, the AMD Arrow logo, Instinct, Radeon Instinct, ROCm, and combinations thereof are trademarks of Advanced Micro Devices, Inc.  

Docker and the Docker logo are trademarks or registered trademarks of Docker, Inc. in the United States and/or other countries. Docker, Inc. and other parties may also have trademark rights in other terms used herein.  Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.    

All other trademarks and copyrights are property of their respective owners and are only mentioned for informative purposes.