====== PyLith Hints, Tips, and Tricks ====== ==== cig-short email archive ==== To search the cig-short email archive, do a Google search with site:http://www.geodynamics.org/pipermail/cig-short/ MY_SEARCH_STRING ==== Installation ==== // Issues related to installing PyLith // === OS X binary === == Errors on starting PyLith == * Error: 'import site' failed; use -v for traceback --LOTS OF OUTPUT-- TypeError: stat() argument 1 must be encoded string without NULL bytes, not str We have seen this error on Darwin system running OS X 10.5 and later. Files downloaded from the web are marked with an extra attribute that can prevent python from starting up properly. The best solution in these cases is to download PyLith using command line tools: mkdir pylith cd pylith curl -O http://www.geodynamics.org/cig/software/pylith/pylith-1.6.2-darwin-10.6.8.tgz tar -zxf pylith-1.6.2-darwin-10.6.8.tgz === Windows binary === == Error on starting PyLith == * Error: ImportError: numpy.core.multiarray failed to import Traceback (most recent call last): File "/usr/bin/pylith", line 37, in from pylith.apps.PyLithApp import PyLithApp File "/usr/lib/python2.6/site-packages/pylith/apps/PyLithApp.py", line 26, in class PyLithApp(PetscApplication): File "/usr/lib/python2.6/site-packages/pylith/apps/PyLithApp.py", line 33, in PyLithApp class Inventory(PetscApplication.Inventory): File "/usr/lib/python2.6/site-packages/pylith/apps/PyLithApp.py", line 51, in Inventory from pylith.topology.MeshImporter import MeshImporter File "/usr/lib/python2.6/site-packages/pylith/topology/MeshImporter.py", line 28, in class MeshImporter(MeshGenerator): File "/usr/lib/python2.6/site-packages/pylith/topology/MeshImporter.py", line 37, in MeshImporter class Inventory(MeshGenerator.Inventory): File "/usr/lib/python2.6/site-packages/pylith/topology/MeshImporter.py", line 58, in Inventory from pylith.meshio.MeshIOAscii import MeshIOAscii File "/usr/lib/python2.6/site-packages/pylith/meshio/MeshIOAscii.py", line 26, in from MeshIOObj import MeshIOObj File "/usr/lib/python2.6/site-packages/pylith/meshio/MeshIOObj.py", line 26, in from meshio import MeshIO as ModuleMeshIO File "/usr/lib/python2.6/site-packages/pylith/meshio/meshio.py", line 25, in _meshio = swig_import_helper() File "/usr/lib/python2.6/site-packages/pylith/meshio/meshio.py", line 21, in swig_import_helper _mod = imp.load_module('_meshio', fp, pathname, description) ImportError: numpy.core.multiarray failed to import This error arises from having another version of Python installed that interferes with the Python included with PyLith. The solution is to set your environment variables so that the shell doesn't see the existing Python when you run PyLith. * Double click on the PyLith icon to get the cygwin terminal window for Pylith. * Run "env >& env.log" and examine the env.log file for environment variables related to Python. * Remove all environment variables related to Python and set the PATH variable to the absolute minimum. For example: unset PYTHON unset PYTHON26 unset PYTHON27 unset PYTHONPATH PATH=/usr/bin:/bin:/lib:/lib/lapack * You should now be able to use PyLith. In general, you will need to perform steps 1 and 3 (but not 2) every time you run PyLith. To add these commands to the PyLith startup script, add the commands to the bottom of the pylithrc file included in the PyLith distribution. Usually this file is in Program Files (x86)/PyLith. This shell script is run every time PyLith starts up. === PyLith installer utility === See the INSTALL file included with the PyLith installer utility for directions and example configuration parameters. Troubleshooting tips are also included at the end of the INSTALL file. * gcc 4.7 * We have encountered segmentation faults building netcdf with gcc 4.7 on OS X 10.8 (Mountain Lion). Running "make check" in nctest generates a segmentation fault. * clang * We have encountered problems building PETSc w/Sieve support using clang due to C/C++ issues. ---- ==== Meshing ==== // Issues related to generating a mesh to use as input for PyLith. // === Best Practices === * Always check the quality of your mesh BEFORE running a PyLith simulation. The ideal aspect ratio is 1.0. * CUBIT: Aspect ratios should be less than about 4.0 in quasi-static simulations and less than about 2.0 in dynamic simulations. Use the "Quality" command to show a summary of the element quality. See the "Mesh Quality Assessment" section of the CUBIT manual for more information about the various quality checks. We find the aspect ratio and condition number checks the most useful. We generally use condition number smoothing to improve the mesh quality. * LaGriT: Aspect ratios should be greater than about 0.2 in quasi-static simulations and greater than about 0.4 in dynamic simulations. Use the command "quality" to show a summary of the element quality. * Always check your nodesets/psets to make sure they match the desired boundaries, etc. Nodesets/pets for Dirichlet boundary conditions cannot overlap if they set the same components (i.e., x-component, y-component, z-component), but they can overlap if they set difference components. === CUBIT === * Nonplanar fault geometry See ''examples/2d/subduction'' and Sessions III and IV of the 2011 Crustal Deformation Modeling tutorial. * Varying discretization size with distance from the fault See ''examples/2d/subduction'' and ''examples/meshing/cubit_cellsize''. === LaGriT === * Identifying faults using surface meshes Using surface meshes to identify the fault surface is a development feature that is fragile and untested. ** Use with extreme caution. ** Identifying faults using psets is a much more thoroughly tested feature. ---- ==== General ==== // General issues related to running PyLith // === Nondimensionalization === It is ** VERY IMPORTANT ** to make sure that the scales used in the nondimensionalization are appropriate for your problem. PyLith can solve problems across an extremely wide range of spatial and temporal scales if the appropriate scales are used in the nondimensionalization. Due to roundoff errors and convergence tolerances in the iterative solvers, PyLith relies on reasonable scales in the solution in constructing the friction criterion and preconditioning the system. Failure to set appropriate scales in the nondimensionalization will cause the solution to be garbage. * Quasi-static problems Default values: relaxation_time = 1.0*year length_scale = 1.0*km pressure_scale = 3.0e+10*Pa Recommended values: relaxation_time = TIME_STEP length_scale = DISCRETIZATION_SIZE or DISPLACEMENT_MAGNITUDE pressure_scale = SHEAR_MODULUS * Dynamic problems Default values: shear_wave_speed = 3.0*km/s density = 3000.0*kg/m**3 wave_period = 1.0*s Recommended values: shear_wave_speed = MINIMUM_SHEAR_WAVE_SPEED density = DENSITY wave_period = MINIMUM_WAVE_PERIOD === Fault output === Slip and traction vectors are output in the fault coordinate system (along strike, up-dip, and opening). The direction of the slip vector corresponds to the direction of motion on the "negative" side of the fault, which is defined by the origin of the fault normal vector. To convert to the global coordinate system, request that the fault orientation be included in the fault output via: vertex_info_fields = [strike_dir,dip_dir,normal_dir] With this information it is easy to rotate the slip or traction vector from the fault coordinate system to the global coordinate system. This is usually done in a Python script with HDF5 output or within ParaView using the calculator. The expression for the slip in global coordinates is: (slip_X*strike_dir_X+slip_Y*dip_dir_X)*iHat+(slip_X*strike_dir_Y+slip_Y*dip_dir_Y)*jHat+(slip_X*strike_dir_Z+slip_Y*dip_dir_Z)*kHat === Spontaneous (Dynamic) Rupture in Quasistatic Simulations === Use of the FaultCohesiveDyn object for spontaneous (dynamic) rupture in quasistatic simulations requires careful selection of solver parameters. See [[software:pylith:cdm2013|Session V]] of the 2013 Crustal Deformation Modeling tutorial for a detailed discussion. ---- ==== Errors running PyLith ==== // Errors when running PyLith // === Spatialdata === * Error: RuntimeError: Error occurred while reading spatial database file 'FILENAME'. I/O error while reading SimpleDB data. Make sure the ''num-locs'' values in the header matches the number of lines of data and that the last line of data includes an end-of-line character. ---- ==== Running on a Cluster ==== // Issues related to running PyLith on a cluster or other parallel computer. // === OpenMPI and Infiniband === * Segmentation faults when using OpenMPI with Infiniband PETSC ERROR: ------------------------------------------------------------------------ PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[14]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run PETSC ERROR: to get more information on the crash. PETSC ERROR: --------------------- Error Message ------------------------------------ PETSC ERROR: Signal received! PETSC ERROR: ------------------------------------------------------------------------ PETSC ERROR: Petsc Development HG revision: 78eda070d9530a3e6c403cf54d9873c76e711d49 HG Date: Wed Oct 24 00:04:09 2012 -0400 PETSC ERROR: See docs/changes/index.html for recent updates. PETSC ERROR: See docs/faq.html for hints about trouble shooting. PETSC ERROR: See docs/index.html for manual pages. PETSC ERROR: ------------------------------------------------------------------------ PETSC ERROR: /home/brad/pylith-1.8.0/bin/mpinemesis on a arch-linu named des-compute11.des by brad Tue Nov 13 10:44:06 2012 PETSC ERROR: Libraries linked from /home/brad/pylith-1.8.0/lib PETSC ERROR: Configure run at Wed Nov 7 16:42:26 2012 PETSC ERROR: Configure options --prefix=/home/brad/pylith-1.8.0 --with-c2html=0 --with-x=0 --with-clanguage=C++ --with-mpicompilers=1 --with-debugging=0 --with-shared-libraries=1 --with-sieve=1 --download-boost=1 --download-chaco=1 --download-ml=1 --download-f-blas-lapack=1 --with-hdf5=1 --with-hdf5-include=/home/brad/pylith-1.8.0/include --with-hdf5-lib=/home/brad/pylith-1.8.0/lib/libhdf5.dylib --LIBS=-lz CPPFLAGS="-I/home/brad/pylith-1.8.0/include " LDFLAGS="-L/home/brad/pylith-1.8.0/lib " CFLAGS="-g -O2" CXXFLAGS="-g -O2 -DMPICH_IGNORE_CXX_SEEK" FCFLAGS="-g -O2" PETSC_DIR=/home/brad/build/pylith_installer/petsc-dev PETSC ERROR: ------------------------------------------------------------------------ PETSC ERROR: User provided function() line 0 in unknown directory unknown file This appears to be associated with how OpenMPI interprets calls to fork() when PyLith starts up. Set your environment (these can also be set on the command line like other OpenMPI parameters) to turn off Infiniband support for fork so that a normal fork call is made: export OMPI_MCA_mpi_warn_on_fork=0 export OMPI_MCA_btl_openib_want_fork_support=0 * Turn on processor and memory affinity by using the ''--bind-to-core'' command line argument for mpirun. === Submitting to batch systems === == PBS/Torque == * pylithapp.cfg: [pylithapp] scheduler = pbs [pylithapp.pbs] shell = /bin/bash qsub-options = -V -m bea -M johndoe@university.edu [pylithapp.launcher] command = mpirun -np ${nodes} -machinefile ${PBS_NODEFILE} Command line arguments: −−nodes=NUMPROCS --scheduler.ppn=N --job.name=NAME --job.stdout=LOG_FILE # NPROCS = total number of processes # N = number of processes per compute node # NAME = name of job in queue # LOG_FILE = name of file where stdout will be written == Sun Grid Engine == * pylithapp.cfg: [pylithapp] scheduler = sge [pylithapp.pbs] shell = /bin/bash pe-name = orte qsub-options = -V -m bea -M johndoe@university.edu -j y [pylithapp.launcher] command = mpirun -np ${nodes} # Use the options below if not using the OpenMPI ORTE Parallel Environment #command = mpirun -np ${nodes}-machinefile ${PE_HOSTFILE} -n ${NSLOTS} Command line arguments: −−nodes=NPROCS --job.name=NAME --job.stdout=LOG_FILE # NPROCS = total number of processes # NAME = name of job in queue # LOG_FILE = name of file where stdout will be written === HDF5 and parallel I/O === The PyLith HDF5 data writers (DataWriterHDF5Mesh, etc) use HDF5 parallel I/O to write files in parallel. As noted in the PyLith manual, this is not nearly as robust as the HDF5Ext data writers (DataWriterHDF5ExtMesh, etc) that write raw binary files using MPI I/O accompanied by an HDF5 metadata file written. If you experience errors when running on multiple compute nodes where jobs mysteriously get hung up with or without HDF5 error messages, switching from the DataWriterHDF5 data writers to the DataWriterHDF5Ext data writers may fix the problem (if HDF5 parallel I/O is the source of the problem). This will produce one raw binary file per HDF5 dataset, so it means lots more files that must be kept together. ----