BlueGene

1. Installing GROMACS 4.5.x
2. Installing and Configuring Gromacs 4.0.x on IBM BlueGene\P and BlueGene\L
1. 2.1. 1) Getting the package.
2. 2.2. 2) Prerequisites
3. 2.3. 3) Configuring and Building the package
3. Installing Gromacs 3.x

1. Installing GROMACS 4.5.x
2. Installing and Configuring Gromacs 4.0.x on IBM BlueGene\P and BlueGene\L
1. 2.1. 1) Getting the package.
2. 2.2. 2) Prerequisites
3. 2.3. 3) Configuring and Building the package
3. Installing Gromacs 3.x

Installing GROMACS 4.5.x

autotools, as described for 4.0.x can be used, but do not use --enable-fortran.

In all cases you may find it necessary to disable dlopen and/or XML support.

Alternatively cmake can be used (requires cmake >=2.8.4, read the cmake installation guide for general advice). To build GROMACS 4.5.6 (when it is released) for use on the back end, use:

ccmake .. \
  -DFFTW3F_INCLUDE_DIR:PATH=/your/path/to/fftw3/include \
  -DFFTW3F_LIBRARIES:FILEPATH=/your/path/to/fftw3/lib/libfftw3f.a \
  -DCMAKE_TOOLCHAIN_FILE=../cmake/Toolchain-BlueGeneP.cmake \
  -DBUILD_SHARED_LIBS=OFF \
  -DGMX_XML=OFF \
  -DCMAKE_C_FLAGS="-O3 -qarch=450d -qtune=450" \
  -DCMAKE_INSTALL_PREFIX=/path/to/install/location

The warning about unused CMAKE_TOOLCHAIN_FILE at the end is harmless. You can suppress it on the ccmake command line with --no-warn-unused-cli.

For GROMACS 4.5 to 4.5.5 for use on the back end, use

ccmake .. \
  -DFFTW3F_INCLUDE_DIR:PATH=/path/to/fftw3/include \
  -DFFTW3F_LIBRARIES:FILEPATH=/path/to/fftw3/lib/libfftw3f.a \
  -DCMAKE_SYSTEM_NAME=BlueGeneP-static \
  -DBUILD_SHARED_LIBS=OFF \
  -DGMX_XML=OFF \
  -DCMAKE_C_COMPILER=mpixlc_r \
  -DGMX_ACCELERATION=BlueGene \
  -DCMAKE_C_FLAGS="-O3 -qarch=450d -qtune=450" \
  -DCMAKE_INSTALL_PREFIX=/path/to/install/location

For BlueGene/L, the standard CMake toolchain has been packaged with GROMACS, so you can use

ccmake .. \
 -D CMAKE_TOOLCHAIN_FILE=../cmake/Toolchain-BlueGeneL-xlc.cmake \
 -D FFTW3F_INCLUDE_DIR:PATH=/your/path/to/fftw3/include \
 -D FFTW3F_LIBRARIES:FILEPATH=/your/path/to/fftw3/lib/libfftw3f.a

Installing and Configuring Gromacs 4.0.x on IBM BlueGene\P and BlueGene\L

1) Getting the package.

Download the latest CVS source tree using

cvs -d :pserver:anoncvs@cvs.gromacs.org:/home/gmx/cvs co gmx

Go to the ./gmx directory and type ./bootstrap

Alternatively, download the latest 4.0.x release package, untar it and change to the gromacs-4.0 directory.

2) Prerequisites

i) The MPI Library in driver R1V2M0 on BlueGene/P has a known defect reported as PMR 48603,033,724 for which a specific fix must be requested from IBM. Without the fix the R1V2M0 driver will generate a segmentation fault early in the program startup inside MPI_Scatterv(). Later drivers should have the fix included.

ii) You will need the FFTW 3.1.X package or higher configured and installed for both the BlueGene system itself and the frontend node. Note that as for non-BlueGene GROMACS installations, you do need to match the precision with which FFTW and GROMACS have been configured, single vs double. Double precision will be much slower on at least BlueGene/L, but might be wanted for some applications. Alternatively, Stephan Kral's version of FFTWGEL-2.1.5 can also be used for improved FFT performance over FFTW 3.1 in many cases. mabraham notes that this seems unlikely to show gains, since Kral's website claims the SIMD-vectorizing code generator has been incorporated in FFTW, and FFTW's website claims they have such a generator. With -O2-compiled versions of FFTW 3.1.3 and Kral's FFTWGEL on BlueGene/L, he observed similar average performance between the two on the FFTW benchmarks. With -O5-compiled versions of the two, he observed 5-10% faster GROMACS simulation times for FFTW 3.1.3. Still, it can't hurt to test them both out and compare.

3) Configuring and Building the package

i) You first need to configure/build GROMACS for the BlueGene frontend system. a) Inside the ./gmx directory make a directory named frontend or similar and change into that directory b) Execute the configure script using the following recommended set of options

../configure --prefix=<path-to-your-gromacs-install-dir> \
              --enable-ppc-sqrt \
              --disable-ppc-altivec \
              --enable-fortran \
              --with-fft=fftw3 \
              --without-x \
              CC="xlc_r -q64" CFLAGS="-O3 -qarch=auto -qtune=auto" \
              CXX="xlC_r -q64" CXXFLAGS="-O3 -qarch=auto -qtune=auto" \
              CPPFLAGS="-I/home/admin/ibm021/include -I/home/admin/ibm021/fftw-3.1.2/include" \
              F77="xlf_r -q64" FFLAGS="-O3 -qnoprefetch -qarch=auto -qtune=auto" \
              LDFLAGS="-L/home/admin/ibm021/lib -L/home/admin/ibm021/fftw-3.1.2/lib" \
              LIBS="-lmass"

You may need to adjust the include paths for C-headers and libaries to your system installation in the CPPFLAGS and LDFLAGS variables.

Note, that you need a 64-bit compiled version of FFTW-3.1 or higher for LINUX on Power.

c) Type make all

d) Type make install

ii) You now need to configure/build the mdrun binary for the actual BlueGene system.

a) Inside the ./gmx directory make a directory named bgp or similar and change into that directory.

b) Execute the configure using the following recommended set of options

../configure --prefix=<your-gromacs-install-dir> \
                   --host=ppc \
                   --build=ppc64 \
                   --enable-ppc-sqrt=1 \
                   --enable-bluegene \
                   --enable-fortran \
                   --enable-mpi \
                   --with-fft=fftw3 \
                   --without-x \
                   --program-suffix=_mpi_bg \
                   CC="mpixlc_r" CFLAGS="-O3 -qarch=450d -qtune=450" \
                   MPICC="mpixlc_r" \
                   CXX="mpixlC_r" CXXFLAGS="-O3 -qarch=450d -qtune=450" \
                   CPPFLAGS="-I/home/admin/ibm021/bg/include -I/home/admin/ibm021/bg/fftw-3.1.2-openmp/include" \
                   F77="mpixlf_r" FFLAGS="-O3 -qarch=auto -qtune=auto" \
                   LDFLAGS="-L/home/admin/ibm021/bg/lib -L/home/admin/ibm021/bg/fftw-3.1.2-openmp/lib" \
                   LIBS="-lmass"

Again, you may need to adjust the include paths for C-headers and libaries to your system installation.

On a BlueGene/L type system you may not have the mpixlc_r MPI wrappers of the compilers and you may need to use the regular XL cross compilers directly and add the BlueGene system libraries.

mabraham finds that on BlueGene/L the following works well to compile for the compute nodes:

#!/bin/bash
FFTW=/bgl/local/pkg/fftw/3.1.2-single
MPI=/bgl/BlueLight/ppcfloor/bglsys
export CC="blrts_xlc"
export MPICC=$CC
export CC_FOR_BUILD="xlc"
export F77="blrts_xlf"
BG_INCLUDE="-I$FFTW/include -I$MPI/include"

GENERICFLAGS="-O3 -qbgl -qarch=440d -qcache=level=1:type=i:size=32:line=32:assoc=64:cost=8 \
 -qcache=level=1:type=d:size=32:line=32:assoc=64:cost=8 \
-qcache=level=2:type=c:size=4096:line=128:assoc=8:cost=40 -qnoautoconfig -qfloat=norngchk -qhot $BG_INCLUDE"

export CFLAGS="$GENERICFLAGS"
export FFLAGS="$GENERICFLAGS -qspillsize=4096"

MPILIBS="-L$MPI/lib -lmpich.rts -lmsglayer.rts -ldevices.rts -lrts.rts -ldevices.rts -lrts.rts"
FFTLIBS="-L$FFTW/lib"
export LIBS="-L$XLCPATH/lib $FFTLIBS -lmassv -lxlopt -lnss_files -lnss_dns -lresolv -lc -lnss_files \
 -lnss_dns -lresolv $MPILIBS"
 
../configure --prefix $HOME/progs --without-x --enable-ppc-sqrt --config-cache --program-suffix=_mpi_bg \
 --with-fft=fftw3 --enable-bluegene --host=powerpc --build=ppc64 --enable-mpi --enable-fortran

make mdrun
make install-mdrun

Again, one will need to tweak directories for FFTW libraries, etc. There may well be value in using -qipa=level=2 (or just -O4 for the works) for the non-kernel code.

Additional hints on compiler options:

You may also want to try to configure the compiler options as -O3 -qalias=ansi -qlibansi -qhot -qipa=level=2 as this is known to improve the performance of several sections of the code, except the short-range force computation kernels, which typically consume most of the CPU time. These kernels will run faster with plain -O3. Currently GROMACS configure scripts do not allow for a separate set of compiler flags just for the force kernels and you would have to modify the Makefile in ./src/gmxlib/nonbonded/nb_kernel_bluegene manually after the configure process to remove additional higher level compiler flags (in particula -qhot and -qipa) from the set of compiler options.

You may also want to build with --with-fft=fftw2 if you have Stephan Kral's "double hummer" optimized version of FFTWGEL-2.1.5 installed on your system for improved performance of the FFT kernels during PME. mabraham suggests otherwise, see above.

c) Type make mdrun. You do not need the utilities compiled for the BlueGene.

d) Type make install-mdrun. This will install mdrun_mpi_bg as the executable you should later call with mpirun.

Installing Gromacs 3.x

After configure (see advice below) when using make or make mdrun, GROMACS runs a C program that generates source files for the nonbonded routines (also known as kernels). This may be run on the back-end compute node, but the more elegant solution is to compile just this program for the front-end (see below). Thus make will fail at this point, with a message something like

./mknb   -software_invsqrt
>>> Gromacs nonbonded kernel generator (-h for help)
>>> Generating single precision functions in C.
>>> Using Gromacs software version of 1/sqrt(x).
Error: Cannot open nb_kernel010_c.c for writing.

A work-around was reported by Fiona Reid in November 2007 on the GROMACS mailing list. Manually running on the back-end the mknb command per the error message above allows a subsequent make process to pass this point. With Gromacs 3.3.3 CC_FOR_BUILD in src/gmxlib/nonbonded/nb_kernel/Makefile may be set to a compiler that produces code for the front-end, e.g. gcc. In this case only mknb is compiled in front-end code while the rest of the code will be compiled in back-end code.

Then in the final link stage there may be problems with undefined symbols beginning _nss_*. This can be worked around by defining LDFLAGS environment variables. This can be most easily done on the configure command line, i.e.

./configure LDFLAGS="-lnss_files -lnss_dns -lresolv -lc -lnss_files -lnss_dns -lresolv"

(in addition to your normal configure flags).

Attach file