This is an old revision of the document!
Model Description
These scalability tests were run using CitcomS 3.2.0 with default configuration. The mesh for these tests is a regional cap with 129x129x129 nodes. Total velocity unknowns is 129^3 x 3 = 6.4 million. The model is run for 11 time steps. The result reported is the total wall clock time. Each node on this cluster has 2 Xeon 5680 series 3.33GHz hex-core processors with a 12MB unified L3 cache and 24GB RAM, for a total of 12 cores per node. The interconnect is QDR InfiniBand.
Partition | Total Procs | Wall Time (sec) | Speedup | Scalability |
---|---|---|---|---|
1x1x1 | 1 | 47217 | 1.000 | 1.000 |
1x1x2 | 2 | 25466 | 1.854 | 0.927 |
1x1x4 | 4 | 14645 | 3.224 | 0.806 |
2x2x1 | 4 | 14438 | 3.270 | 0.818 |
2x2x2 | 8 | 8980 | 5.258 | 0.657 |
2x2x4 | 16 | 4432 | 10.654 | 0.666 |
4x4x1 | 16 | 5367 | 8.798 | 0.550 |
4x4x2 | 32 | 2460 | 19.194 | 0.600 |
4x4x4 | 64 | 1346 | 35.079 | 0.548 |
8x8x2 | 128 | 583 | 80.990 | 0.633 |
8x8x4 | 256 | 337 | 140.110 | 0.547 |
The input file is available here. It is currently configured for 1x1x1 processors, to do different processor divisions you must change the nprocx, nprocy, and nprocz parameters. You must create a folder named “scratch” in the working directory for the output files. The input file uses the non-Python version of CitcomS, located at CitcomS-3.2.0/bin/CitcomSRegional.
Q: What are the dependencies of CitcomS-2.2 or later?
CitcomS-2.2 or later depends on the MPI library. In addition, if you like to use the Pyre framework (which provides an ease-of-use environment), you will need Python 2.3 or newer. (Note that if your machine is 64-bit, you will need Python 2.4.)
Q: I want to use CitcomS in the old way (pure C code, no Python). How can I do that?
You can configure CitcomS to only use C code by running:
./configure --without-pyre
before compiling the code.
Q: What does it mean when “configure” reports this error message
OverflowError: signed integer is greater than maximum
Are you on a 64-bit machine? If so, you will need to upgrade your Python to v2.4 or newer. Python 2.3 or earlier is not 64-bit safe.
Q: The gcc/python/mpi on my system is too old, but I don't have the privilege to update the software. How can I install gcc/python/mpi under my home directory?
1 Get the tarball of gcc/python/mpi and untar it
2 cd into the untarred directory
3 Run the following to install the package under the directory $HOME/opt
./configure --prefix=$HOME/opt && make && make install
4 Add $HOME/opt/bin
into your $PATH
and (maybe) $HOME/opt/lib
into your $LD_PATH_LIBRARY
.
Q: I want a specific temperature boundary condition. Where to modify?
Look at lib/ directory, the function full_temperature_boundary_conditions() in Full_boundary_conditions.c or regional_temperature_boundary_conditions in Regional_boundary_conditions.c is the function you need to modify.
Q: I want a specific velocity boundary condition. Where to modify?
Look at lib/ directory, the function full_velocity_boundary_conditions() in Full_boundary_conditions.c or regional_velocity_boundary_conditions() in Regional_boundary_conditions.c.
Q: I want a specific rheology law. Where to modify?
Look at lib/ directory, the function get_system_viscosity() in Viscosity_structures.c.
Q: I want a new equation of state or a new law for heat expansivity, heat capacity, or thermal conductivity. Where to modify?
Look at lib/ directory, the function () in Material_properties.c.
CitcomS Tutorial PDF slides of talk given by Eh Tan, CIG Training Session, EarthScope 2009, Boise, ID (May 12, 2009), on large scale models
Modified Legendre Functions A technical note by Shijie Zhong.