This is an old revision of the document!
The following instructions assume that you have installed CitcomS and familiarized yourself with how you will run the package based on your computer configuration, as detailed in the CitcomS User Manual (Chapter 2 provides installation help and Chapter 3 details how to run CitcomS on various platforms).
These cookbook examples are distributed with the package under the examples directory. However, you might need to edit these example scripts slightly to launch the job on your cluster (see Section 3.4 of the CitcomS User Manual).
More tutorials are found in Chapter 6 of the CitcomS User Manual
This example solves for thermal convection within a full spherical shell domain.
This example solves for thermal convection within a full spherical shell domain. The full spherical version of CitcomS.py is designed to run on a cluster that decomposes the spherical shell into 12 equal “caps” and then distributes the calculation for caps onto separate processors. To run CitcomS.py with the full solver parameter set, it is recommended that you have a minimum of 12 processors available on your cluster.
You will use cookbook1.cfg. The first set of parameters specifies that you are going to use the full spherical version of the solver. The default is to use the regional spherical version, so this must be set:
solver = full
The second set of parameters specifies the number of time steps (101), how often full outputs of the computation are created (25), and the prefix of output filenames (cookbook1):
steps = 101 monitoringFrequency = 25 datafile = cookbook1
The solver.ic facility controls the temperature field for the initial conditions. The last set of parameters includes the number of perturbations to the initial temperature (1), the number of nodal lines of the perturbation in the longitudinal direction, e.g., the spherical harmonic order (3), the number of nodal lines in the latitudinal direction, e.g., the spherical harmonic degree (2), which layer the pertubation is on (5), and the amplitude of the perturbation (0.05). Note that although the number of perturbations is assigned here as num_perturbations=1, that is actually the default value:
num_perturbations = 1 perturbl = 3 perturbm = 2 perturblayer = 5 perturbmag = 0.05
This example is executed by typing '$ citcoms cookbook1.cfg'
Example: Global Model, cookbook1.cfg:
[CitcomS] solver = full steps = 101 ; number of time steps [CitcomS.controller] monitoringFrequency = 25 ; how often outputs are created [CitcomS.solver] datafile = cookbook1 ; prefix of output filenames [CitcomS.solver.ic] num_perturbations = 1 perturbl = 3 perturbm = 2 perturblayer = 5 perturbmag = 0.05
Once you have run the model, you can visualize the results using OpenDX, described in the previous chapter. When you invoke “Edit Visual Program,” select visFull.net.
You have generated a simple example of the full CitcomS.py model, with minimal parameter alterations. With a default Rayleigh number of 105 and perturbation on the initial temperature field which has a degree-3 and an order-2 pattern, after 100 time steps, the convection pattern remains relatively steady.
As a side note, it is not required that 12 processors, with one spherical cap per processor, be used. As an end-member possibility, for example, 12 different jobs could be run on a single computer (n001 in this example) by invoking:
$ citcoms cookbook1.cfg --launcher.nodegen="n%03d"\ --launcher.nodelist=[1,1,1,1,1,1,1,1,1,1,1,1]
This is not particularly efficient, but it does illustrate the flexibility of both mpi and Pyre.
This example solves for thermal convection with velocity boundary conditions imposed on the top surface within a given region of a sphere. This requires using the regional version of CitcomS.py.
You will use cookbook2.cfg. The parameters specify the number of time steps (61), the prefix of the output filenames (cookbook2), and how often outputs will be created (30):
steps = 61 monitoringFrequency = 30 datafile = cookbook2
The solver.mesher facility has several properties involved in the generation of the computational mesh. Continue to use the default values for the physical portion of the domain in which you are interested. However, try modifying the layout of the mesh as shown:
nprocx = 2 nprocy = 2 nodex = 17 nodey = 17 nodez = 9
The solver.bc facility allows you to impose a uniform velocity across the top surface. You have a velocity which is purely in the colatitude direction with a non-dimensional velocity of 100 (see Equation 1.6 in the CitcomS User Manual for how to dimensionalize it):
topvbc = 1 topvbxval = 100
In addition, the initial temperature field has a linear conductive profile. The amplitude of initial temperature perturbation is set to zero using the solver.ic facility
num_perturbations = 1 perturbmag = 0.0
Example: Velocity Boundary Conditions, cookbook2.cfg:
[CitcomS] steps = 61 ; number of time steps [CitcomS.controller] monitoringFrequency = 30 ; how often outputs are created [CitcomS.solver] datafile = cookbook2 ; prefix of output filenames # Modify the layout of the mesh. [CitcomS.solver.mesher] nprocx = 2 nprocy = 2 nodex = 17 nodey = 17 nodez = 9 # Impose a uniform velocity across the top surface. [CitcomS.solver.bc] topvbc = 1 topvbxval = 100 # Modify the layout of the mesh. [CitcomS.solver.mesher] nprocx = 2 nprocy = 2 nodex = 17 nodey = 17 nodez = 9 # Impose a uniform velocity across the top surface. [CitcomS.solver.bc] topvbc = 1 topvbxval = 100 topvbyval = 0 # In addition, set the initial temperature perturbation to zero. [CitcomS.solver.ic] num_perturbations = 1 perturbmag = 0.0
Using OpenDX to visualize the results (Figure 3 below), this model allows you to create a plate-driven convection in which there is a thermal upwelling on one wall, a thermal downwelling on another, and uniform horizontal velocity across the top. The downwelling is not exactly subduction because the default boundary conditions are close to zero shear stress on the boundaries. This means that there is a symmetrical downwelling in a vertical domain on the other side.
More tutorials are found in Chapter 6 of the CitcomS User Manual
The temperature cross section and a composition isosurface are shown. A dense chemical layer is at the base of the mantle initially. As the layer heats up and convection develops, the layer gets entrained into the ambient mantle. (original)
CitcomS: Evolution of Plume Conduit in the Coupled Model
The containing solver (not shown) is a global model, with plate motion imposed on the top surface. The embedded Solver (shown in the animation) is a high-resolution regional model, with boundary conditions retrieved from the containing solver. The black line is the past hotspot location. The red segment is the assumed melt conduit, starting at 160 km depth. The velocity vector is in yellow. The temperature BC at the CMB is shown in color. The plume is visualized as an iso-surface (T=0.08). The numbers of grid points of both meshes are reduced for visualization purposes.
Model Description
These scalability tests were run using CitcomS 3.2.0 with default configuration. The mesh for these tests is a regional cap with 129x129x129 nodes. Total velocity unknowns is 129^3 x 3 = 6.4 million. The model is run for 11 time steps. The result reported is the total wall clock time. Each node on this cluster has 2 Xeon 5680 series 3.33GHz hex-core processors with a 12MB unified L3 cache and 24GB RAM, for a total of 12 cores per node. The interconnect is QDR InfiniBand.
Partition | Total Procs | Wall Time (sec) | Speedup | Scalability |
---|---|---|---|---|
1x1x1 | 1 | 47217 | 1.000 | 1.000 |
1x1x2 | 2 | 25466 | 1.854 | 0.927 |
1x1x4 | 4 | 14645 | 3.224 | 0.806 |
2x2x1 | 4 | 14438 | 3.270 | 0.818 |
2x2x2 | 8 | 8980 | 5.258 | 0.657 |
2x2x4 | 16 | 4432 | 10.654 | 0.666 |
4x4x1 | 16 | 5367 | 8.798 | 0.550 |
4x4x2 | 32 | 2460 | 19.194 | 0.600 |
4x4x4 | 64 | 1346 | 35.079 | 0.548 |
8x8x2 | 128 | 583 | 80.990 | 0.633 |
8x8x4 | 256 | 337 | 140.110 | 0.547 |
The input file is available here. It is currently configured for 1x1x1 processors, to do different processor divisions you must change the nprocx, nprocy, and nprocz parameters. You must create a folder named “scratch” in the working directory for the output files. The input file uses the non-Python version of CitcomS, located at CitcomS-3.2.0/bin/CitcomSRegional.
Q: What are the dependencies of CitcomS-2.2 or later?
CitcomS-2.2 or later depends on the MPI library. In addition, if you like to use the Pyre framework (which provides an ease-of-use environment), you will need Python 2.3 or newer. (Note that if your machine is 64-bit, you will need Python 2.4.)
Q: I want to use CitcomS in the old way (pure C code, no Python). How can I do that?
You can configure CitcomS to only use C code by running:
./configure --without-pyre
before compiling the code.
Q: What does it mean when “configure” reports this error message
OverflowError: signed integer is greater than maximum
Are you on a 64-bit machine? If so, you will need to upgrade your Python to v2.4 or newer. Python 2.3 or earlier is not 64-bit safe.
Q: The gcc/python/mpi on my system is too old, but I don't have the privilege to update the software. How can I install gcc/python/mpi under my home directory?
1 Get the tarball of gcc/python/mpi and untar it
2 cd into the untarred directory
3 Run the following to install the package under the directory $HOME/opt
./configure --prefix=$HOME/opt && make && make install
4 Add $HOME/opt/bin
into your $PATH
and (maybe) $HOME/opt/lib
into your $LD_PATH_LIBRARY
.
Q: I want a specific temperature boundary condition. Where to modify?
Look at lib/ directory, the function full_temperature_boundary_conditions() in Full_boundary_conditions.c or regional_temperature_boundary_conditions in Regional_boundary_conditions.c is the function you need to modify.
Q: I want a specific velocity boundary condition. Where to modify?
Look at lib/ directory, the function full_velocity_boundary_conditions() in Full_boundary_conditions.c or regional_velocity_boundary_conditions() in Regional_boundary_conditions.c.
Q: I want a specific rheology law. Where to modify?
Look at lib/ directory, the function get_system_viscosity() in Viscosity_structures.c.
Q: I want a new equation of state or a new law for heat expansivity, heat capacity, or thermal conductivity. Where to modify?
Look at lib/ directory, the function () in Material_properties.c.
CitcomS Tutorial PDF slides of talk given by Eh Tan, CIG Training Session, EarthScope 2009, Boise, ID (May 12, 2009), on large scale models
Modified Legendre Functions A technical note by Shijie Zhong.