HPC Application Performance of NVIDIA Data Center GPUs

Some of the world’s most pressing scientific and engineering problems can only be solved using modern HPC data centers. Breakthrough performance can be achieved with dramatically fewer servers, less power consumption, and reduced network overhead, resulting in a total cost savings of 5X to 10X with NVIDIA A100, V100 and T4 GPUs.

The node replacement factor is the number of CPU-only servers that can be replaced by a single GPU-accelerated server (NRF). NRF is calculated by comparing the performance of an application on up to eight CPU-only servers. The NRF is then calculated at a larger scale using linear scaling. Application-specific factors influence NRF.

AMBER

Molecular Dynamics Suite of programs to simulate molecular dynamics on biomolecule

VERSION 20.12-AT_21.12

ACCELERATED FEATURES PMEMD Explicit Solvent and GB Implicit Solvent

SCALABILITY Multi-GPU and Single Node

MORE INFORMATION  http://ambermd.org/GPUSupport.ph

Application	Metric	Test Modules	Bigger is better	Dual Cascade Lake 6240 (CPU-Only)	1x A100 SXM 80GB	2x A100 SXM 80GB	4x A100 SXM 80GB	8x A100 SXM 80GB	1x A100 PCIe 80GB	2x A100 PCIe 80GB	4x A100 PCIe 80GB	8x A100 PCIe 80GB
AMBER [PME-Cellulose_NPT_4fs]	ns/day	DC-Cellulose_NPT	yes	4.34	152	303	606	1,212	154	307	614	1,228
AMBER [PME-Cellulose_NPT_4fs]	NRF	DC-Cellulose_NPT	yes	1x	35x	70x	140x	279x	35x	71x	141x	283x
AMBER [PME-Cellulose_NVE_4fs]	ns/day	DC-Cellulose_NVE	yes	4.37	164	328	656	1,313	165	331	662	1,324
AMBER [PME-Cellulose_NVE_4fs]	NRF	DC-Cellulose_NVE	yes	1x	38x	75x	150x	300x	38x	76x	151x	303x
AMBER [PME-FactorIX_NPT_4fs]	ns/day	DC-FactorIX_NPT	yes	22.24	534	1,067	2,134	4,268	517	1,033	2,066	4,133
AMBER [PME-FactorIX_NPT_4fs]	NRF	DC-FactorIX_NPT	yes	1x	24x	48x	96x	192x	23x	46x	93x	186x
AMBER [PME-FactorIX_NVE_4fs]	ns/day	DC-FactorIX_NVE	yes	22.74	570	1,140	2,279	4,558	590	1,180	2,360	4,720
AMBER [PME-FactorIX_NVE_4fs]	NRF	DC-FactorIX_NVE	yes	1x	25x	50x	100x	200x	26x	52x	104x	208x
AMBER [PME-JAC_NPT_4fs]	ns/day	DC-JAC_NPT	yes	93.9	1,178	2,356	4,712	9,424	1,263	2,527	5,053	10,106
AMBER [PME-JAC_NPT_4fs]	NRF	DC-JAC_NPT	yes	1x	13x	25x	50x	100x	13x	27x	54x	108x
AMBER [PME-JAC_NVE_4fs]	ns/day	DC-JAC_NVE	yes	95.41	1,242	2,484	4,967	9,934	1,256	2,512	5,024	10,048
AMBER [PME-JAC_NVE_4fs]	NRF	DC-JAC_NVE	yes	1x	13x	26x	52x	104x	13x	26x	53x	105x
AMBER [PME-STMV_NPT_4fs]	ns/day	DC-STMV_NPT	yes	1.43	55	110	221	441	55	110	220	440
AMBER [PME-STMV_NPT_4fs]	NRF	DC-STMV_NPT	yes	1x	39x	77x	154x	308x	38x	77x	154x	307x
AMBER [FEP-GTI_Complex 1fs]	ns/day	FEP-GTI_Complex	yes	10.97	145	289	578	1,157	151	302	603	1,206
AMBER [FEP-GTI_Complex 1fs]	NRF	FEP-GTI_Complex	yes	1x	13x	26x	53x	105x	14x	27x	55x	110x

CHROMA

Physics Lattice Quantum Chromodynamics (LQCD)

VERSION V 2021.05

ACCELERATED FEATURES Wilson-clover fermions, Krylov solvers, Domain-decomposition

SCALABILITY Multi-GPU and Multi-Node

MORE INFORMATION http://jeffersonlab.github.io/chroma/

https://ngc.nvidia.com/catalog/containers/hpc:chroma

Application	Metric	Test Modules	Bigger is better	Dual Cascade Lake 6240 (CPU-Only)	1x A100 SXM 80GB	2x A100 SXM 80GB	4x A100 SXM 80GB	8x A100 SXM 80GB	1x A100 PCIe 80GB	2x A100 PCIe 80GB	4x A100 PCIe 80GB	8x A100 PCIe 80GB
Chroma	Total Time (Sec)	szscl21_24_128	no	1,115	36	20	11	7	44	25	13	9
Chroma	NRF	szscl21_24_128	yes	1x	32x	55x	99x	163x	26x	46x	84x	129x

FUN3D

Engineering A suite of tools actively developed at NASA for Aeronautics and Space Technology by modeling fluid flow

VERSION 13.7 (update 1)

ACCELERATED FEATURES Full range of Mach number regimes for the Reynolds-averaged Navier Stokes (RANS) formulation

SCALABILITY Multi-GPU and Single-Node

MORE INFORMATION https://fun3d.larc.nasa.gov

Application	Metric	Test Modules	Bigger is better	Dual Cascade Lake 6240 (CPU-Only)	1x A100 SXM 80GB	2x A100 SXM 80GB	4x A100 SXM 80GB	8x A100 SXM 80GB	1x A100 PCIe 80GB	2x A100 PCIe 80GB	4x A100 PCIe 80GB	8x A100 PCIe 80GB
FUN3D	Loop Time (Sec)	dpw_wbt0_crs-3.6Mn_5	no	524	52	28	16	11	54	28	16	12
FUN3D	NRF	dpw_wbt0_crs-3.6Mn_5	yes	1x	13x	24x	41x	58x	12x	23x	42x	53x

GROMACS

Molecular Dynamics Simulation of biochemical molecules with complicated bond interactions

VERSION 2022

ACCELERATED FEATURES Implicit (5x), Explicit (2x) Solvent

SCALABILITY Multi-GPU, Single Node

MORE INFORMATION http://www.gromacs.org https://ngc.nvidia.com/catalog/containers/hpc:gromacs

Application	Metric	Test Modules	Bigger is better	Dual Cascade Lake 6240 (CPU-Only)	1x A100 SXM 80GB	2x A100 SXM 80GB	4x A100 SXM 80GB	8x A100 SXM 80GB	1x A100 PCIe 80GB	2x A100 PCIe 80GB	4x A100 PCIe 80GB	8x A100 PCIe 80GB
GROMACS [ADH Dodec]	ns/day	ADH Dodec	yes	58	331	-	598	-	324	370	548	-
GROMACS [ADH Dodec]	NRF	ADH Dodec	yes	1x	8x	-	14x	-	7x	8x	13x	-
GROMACS [Cellulose]	ns/day	Cellulose	yes	17	97	145	250	291	95	134	169	204
GROMACS [Cellulose]	NRF	Cellulose	yes	1x	9x	13x	22x	26x	8x	12x	15x	18x
GROMACS [STMV]	ns/day	STMV	yes	4	23	40	59	114	22	39	57	80
GROMACS [STMV]	NRF	STMV	yes	1x	6x	10x	15x	30x	6x	10x	15x	21x

GTC

Physics GTC is used for Gyrokinetic Particle Simulation of Turbulent Transport in Burning Plasmas

VERSION V 4.5 Updated

ACCELERATED FEATURES Push, shift, and collision

SCALABILITY Multi-GPU and Multi-Node

MORE INFORMATION

Application	Metric	Test Modules	Bigger is better	Dual Cascade Lake 6240 (CPU-Only)	1x A100 SXM 80GB	2x A100 SXM 80GB	4x A100 SXM 80GB	8x A100 SXM 80GB	1x A100 PCIe 80GB	2x A100 PCIe 80GB	4x A100 PCIe 80GB	8x A100 PCIe 80GB
GTC	Mpush/Sec	moi#proc.in	yes	35	489	923	1,803	3,566	480	893	1,748	3,436
GTC	NRF	moi#proc.in	yes	1x	14x	27x	53x	104x	14x	26x	51x	100x

ICON

Weather and Climate A global unified atmosphere model for numerical weather prediction and climate modeling research

VERSION 2.6.5_RC

ACCELERATED FEATURES Full model of dynamics and physics

SCALABILITY Multi-GPU and Multi-Node

MORE INFORMATION https://code.mpimet.mpg.de/projects/iconpublic

Application	Metric	Test Modules	Bigger is better	Dual Cascade Lake 6240 (CPU-Only)	1x A100 SXM 80GB	2x A100 SXM 80GB	4x A100 SXM 80GB	8x A100 SXM 80GB	1x A100 PCIe 80GB	2x A100 PCIe 80GB	4x A100 PCIe 80GB
ICON [SLAM 191 - 160KM - no radiation]	Integrate_nh (sec)	SLAM 191 levels 160 km resolution without radiation	no	2,292	326	242	177	157	331	245	200
ICON [SLAM 191 - 160KM - no radiation]	NRF	SLAM 191 levels 160 km resolution without radiation	yes	1x	7x	9x	13x	15x	7x	9x	11x
ICON [QUBICC 160 km resolution]	Integrate_nh (sec)	SLAM 191 levels 160 km resolution with radiation	no	2,304	302	215	155	139	294	209	155
ICON [QUBICC 160 km resolution]	NRF	SLAM 191 levels 160 km resolution with radiation	yes	1x	8x	11x	15x	17x	8x	11x	15x

LAMMPS

Molecular Dynamics Classical molecular dynamics package

VERSION patch_4May2022

ACCELERATED FEATURES Lennard-Jones, Gay-Berne, Tersoff, many more potentials

SCALABILITY Multi-GPU and Multi-Node

MORE INFORMATION http://lammps.sandia.gov/index.html https://ngc.nvidia.com/catalog/containers/hpc:lammps

Application	Metric	Test Modules	Bigger is better	Dual Cascade Lake 6240 (CPU-Only)	1x A100 SXM 80GB	2x A100 SXM 80GB	4x A100 SXM 80GB	8x A100 SXM 80GB	1x A100 PCIe 80GB	2x A100 PCIe 80GB	4x A100 PCIe 80GB	8x A100 PCIe 80GB
LAMMPS [LJ 2.5]	ATOM-Time Steps/s	LJ 2.5	yes	1.11E+08	5.57E+08	1.07E+09	2.00E+09	3.64E+09	5.44E+08	1.03E+09	1.86E+09	-
LAMMPS [LJ 2.5]	NRF	LJ 2.5	yes	1x	5x	10x	18x	34x	5x	10x	17x	-
LAMMPS [EAM]	ATOM-Time Steps/s	EAM	yes	5.04E+07	2.77E+08	4.89E+08	8.76E+08	1.53E+09	2.76E+08	4.84E+08	8.19E+08	-
LAMMPS [EAM]	NRF	EAM	yes	1x	6x	10x	17x	30x	6x	10x	16x	-
LAMMPS [ReaxFF/C]	ATOM-Time Steps/s	ReaxFF/C	yes	4.23E+05	4.32E+06	7.98E+06	1.46E+07	2.34E+07	4.29E+06	7.92E+06	1.37E+07	1.76E+07
LAMMPS [ReaxFF/C]	NRF	ReaxFF/C	yes	1x	17x	32x	59x	94x	17x	32x	55x	71x
LAMMPS [SNAP]	ATOM-Time Steps/s	SNAP	yes	1.09E+05	2.11E+06	4.19E+06	8.23E+06	1.57E+07	2.07E+06	4.10E+06	8.21E+06	1.57E+07
LAMMPS [SNAP]	NRF	SNAP	yes	1x	19x	38x	75x	143x	19x	37x	75x	143x
LAMMPS [Tersoff]	ATOM-Time Steps/s	Tersoff	yes	2.80E+07	4.90E+08	8.98E+08	1.60E+09	2.86E+09	4.76E+08	8.05E+08	1.30E+09	-
LAMMPS [Tersoff]	NRF	Tersoff	yes	1x	17x	32x	57x	102x	17x	29x	46x	-

MILC

Physics Lattice Quantum Chromodynamics (LQCD) codes simulate how elemental particles are formed and bound by the “strong force” to create larger particles like protons and neutrons

VERSION develop_3971e182

ACCELERATED FEATURES Staggered fermions, Krylov solvers, Gauge-link fattening

SCALABILITY Multi-GPU and Multi-Node

MORE INFORMATION https://ngc.nvidia.com/catalog/containers/hpc:milc

Application	Metric	Test Modules	Bigger is better	Dual Cascade Lake 6240 (CPU-Only)	1x A100 SXM 80GB	2x A100 SXM 80GB	4x A100 SXM 80GB	8x A100 SXM 80GB	1x A100 PCIe 80GB	2x A100 PCIe 80GB	4x A100 PCIe 80GB	8x A100 PCIe 80GB
MILC	Total Time (Sec)	Apex Medium	no	67,366	2,291	1,342	703	399	2,357	1,369	676	664
MILC	NRF	Apex Medium	yes	1x	32x	55x	105x	186x	31x	54x	110x	112x

NAMD

Molecular Dynamics Designed for high-performance simulation of large molecular systems

VERSION GPU, AMD CPU V 3.0a11 ; Intel CPU V 2.15a AVX512

ACCELERATED FEATURES Full electrostatics with PME and most simulation features

SCALABILITY Up to 100M atom capable, multi-GPU, single node

MORE INFORMATION http://www.ks.uiuc.edu/Research/namd/ https://ngc.nvidia.com/catalog/containers/hpc:namd

Application	Metric	Test Modules	Bigger is better	Dual Cascade Lake 6240 (CPU-Only)	1x A100 SXM 80GB	2x A100 SXM 80GB	4x A100 SXM 80GB	8x A100 SXM 80GB	1x A100 PCIe 80GB	2x A100 PCIe 80GB	4x A100 PCIe 80GB	8x A100 PCIe 80GB
NAMD [apoa1_npt_cuda]	Ave ns/day	apoa1_npt_cuda	yes	19.12	134	270	542	1,072	140	277	556	1,112
NAMD [apoa1_npt_cuda]	NRF	apoa1_npt_cuda	yes	1x	7x	14x	28x	56x	7x	14x	29x	58x
NAMD [apoa1_nptsr_cuda]	Ave ns/day	apoa1_nptsr_cuda	yes	19.63	141	282	565	1,123	145	290	576	1,161
NAMD [apoa1_nptsr_cuda]	NRF	apoa1_nptsr_cuda	yes	1x	7x	14x	29x	57x	7x	15x	29x	59x
NAMD [apoa1_nve_cuda]	Ave ns/day	apoa1_nve_cuda	yes	20.64	185	363	727	1,469	189	378	756	1,506
NAMD [apoa1_nve_cuda]	NRF	apoa1_nve_cuda	yes	1x	9x	18x	35x	71x	9x	18x	37x	73x
NAMD [stmv_npt_cuda]	Ave ns/day	stmv_npt_cuda	yes	1.78	14	27	54	108	14	27	54	108
NAMD [stmv_npt_cuda]	NRF	stmv_npt_cuda	yes	1x	8x	15x	30x	61x	8x	15x	30x	61x
NAMD [stmv_nptsr_cuda]	Ave ns/day	stmv_nptsr_cuda	yes	1.8	14	28	55	111	14	28	55	111
NAMD [stmv_nptsr_cuda]	NRF	stmv_nptsr_cuda	yes	1x	8x	15x	31x	62x	8x	16x	31x	62x
NAMD [stmv_nve_cuda]	Ave ns/day	stmv_nve_cuda	yes	1.95	16	32	64	129	16	32	64	128
NAMD [stmv_nve_cuda]	NRF	stmv_nve_cuda	yes	1x	8x	17x	33x	66x	8x	16x	33x	65x

QUANTUM ESPRESSO

Material Science (Quantum Chemistry) An Open-source suite of computer codes for electronic structure calculations and materials modeling at the nanoscale

VERSION V6.7 CPU; V7.0 GPU

ACCELERATED FEATURES linear algebra (matrix multiply), explicit computational kernels, 3D FFTs

SCALABILITY Multi-GPU and Multi-Node

MORE INFORMATION http://www.quantum-espresso.org

Application	Metric	Test Modules	Bigger is better	Dual Cascade Lake 6240 (CPU-Only)	1x A100 SXM 80GB	2x A100 SXM 80GB	4x A100 SXM 80GB	8x A100 SXM 80GB	1x A100 PCIe 80GB	2x A100 PCIe 80GB	4x A100 PCIe 80GB	8x A100 PCIe 80GB
Quantum Espressso	Total Wall Time	AUSURF112-jR	no	718	118	77	53	42	120	75	53	47
Quantum Espressso	NRF	AUSURF112-jR	yes	1x	7x	10x	15x	19x	7x	11x	15x	17x

RELION

Microscopy Stand-alone computer program that employs an empirical Bayesian approach to refinement of (multiple) 3D reconstructions or 2D class averages in electron cryo-microscopy (cryo-EM)

VERSION 3.1.3

ACCELERATED FEATURES Reduced memory requirements; high-resolution cryo-EM structure determination in a matter of day on a single workstation

SCALABILITY Multi-GPU and Single Node

MORE INFORMATION https://www2.mrc-lmb.cam.ac.uk/relion/index.php/Download_%26_install

https://ngc.nvidia.com/catalog/containers/hpc:relion

Application	Metric	Test Modules	Bigger is better	Dual Cascade Lake 6240 (CPU-Only)	1x A100 SXM 80GB	2x A100 SXM 80GB	4x A100 SXM 80GB	8x A100 SXM 80GB	1x A100 PCIe 80GB	2x A100 PCIe 80GB	4x A100 PCIe 80GB	8x A100 PCIe 80GB
Relion [Plasmodium Ribosome]	Total Wall Clock (Sec)	MB numbers Plasmodium Ribosime on Relion-3.0	no	12,742	3,110	1,781	1,458	1,313	3,401	1,994	1,838	-
Relion [Plasmodium Ribosome]	NRF	MB numbers Plasmodium Ribosime on Relion-3.0	yes	1x	4x	7x	9x	10x	4x	6x	7x	-
Relion [Plasmodium Ribosome 2D]	Total Wall Clock (Sec)	Plasmodium Ribosome (2D)	no	90,108	22,960	12,416	8,587	6,299	25,275	13,414	9,583	7,266
Relion [Plasmodium Ribosome 2D]	NRF	Plasmodium Ribosome (2D)	yes	1x	4x	7x	10x	14x	4x	7x	9x	12x

RTM

Geoscience Reverse time migration (RTM) modeling is a critical component in the seismic processing workflow of oil and gas exploration

VERSION nvidia_2021_05

ACCELERATED FEATURES Batch algorithm

SCALABILITY Multi-GPU and Multi-Node

MORE INFORMATION http://www.tsunamidevelopment.com/assets/rtm.pdf

Application	Metric	Test Modules	Bigger is better	Dual Cascade Lake 6240 (CPU-Only)	1x A100 SXM 80GB	2x A100 SXM 80GB	4x A100 SXM 80GB	8x A100 SXM 80GB	1x A100 PCIe 80GB	2x A100 PCIe 80GB	4x A100 PCIe 80GB	8x A100 PCIe 80GB
RTM [Isotropic Radius 4]	Mcells/s	Isotropic Radius 4	yes	11,318	89,414	178,194	356,418	712,731	85,271	170,060	339,993	679,883
RTM [Isotropic Radius 4]	NRF	Isotropic Radius 4	yes	1x	8x	16x	31x	63x	8x	15x	30x	60x
RTM [TTI Radius 8 1-pass]	Mcells/s	TTI Radius 8 1-pass	yes	3,773	12,828	25,462	50,863	100,871	12,864	25,688	51,313	102,502
RTM [TTI Radius 8 1-pass]	NRF	TTI Radius 8 1-pass	yes	1x	3x	7x	13x	27x	3x	7x	14x	27x
RTM [TTI RX 2Pass mgpu]	Mcells/s	TTI RX 2Pass mgpu	yes	3,773	13,910	27,437	54,323	107,533	13,589	27,049	53,755	107,254
RTM [TTI RX 2Pass mgpu]	NRF	TTI RX 2Pass mgpu	yes	1x	4x	7x	14x	29x	4x	7x	14x	28x

SPECFEM3D

Geoscience Simulates Seismic wave propagation

VERSION devel_44e098a3

ACCELERATED FEATURES OpenCL and CUDA hardware accelerators, based on an automatic source-to-source transformation library

SCALABILITY Multi-GPU and Single-Node

MORE INFORMATION https://geodynamics.org/cig/software/specfem3d/

Application	Metric	Test Modules	Bigger is better	Dual Cascade Lake 6240 (CPU-Only)	1x A100 SXM 80GB	2x A100 SXM 80GB	4x A100 SXM 80GB	8x A100 SXM 80GB	1x A100 PCIe 80GB	2x A100 PCIe 80GB	4x A100 PCIe 80GB	8x A100 PCIe 80GB
SPECFEM3D	Total Time (Sec)	four_material_simple_model	no	1,920	76	40	22	14	77	40	21	16
SPECFEM3D	NRF	four_material_simple_model	yes	1x	29x	54x	101x	155x	28x	55x	102x	135x