Installing and running LPJmL C Version 4.0.001

Installation for Unix-like systems

This text describes how to install and run LPJmL on your system. The code has been tested for AIX, Linux, Mac OS X, and Windows with cygwin and Microsoft C compiler (sequential version without MPI). Cygwin is a Linux-like environment for Windows (http://www.cygwin.com) On Mac OS the xcode suite must be installed in order to have the gcc compiler.

create lpj directory:

mkdir lpjml
Go to lpj directory:

cd lpjml
untar source and data files

gzip -cd lpjml-4.0.001.tar.gz|tar -xf -
Directory tree after extracting the tar files:

--lpjml
|
+-config        : OS- and compiler-specific Makefiles
|
+-bin           : Directory for executables and shell scripts
|
+-R             : Scripts for R statistical package
|
+-man           : Manual pages root directory
| |
| +-man1        : Manual pages for programs and scripts
| |
| +-man3        : Manual pages for functions
| |
| +-man5        : Manual pages for file formats
|
+--html         : Documentation and man pages in HTML format
|
+--doc          : Documentation
|
+--par          : Parameter files for PFTs and soils
|
+--input        : input data set for climate
|
+--output       : output data (created by make test)
|
+--restart      : restart data (created by make test)
|
+--lib          : library files (created by make)
|
+--include      : include files
|
+--src          : source tree
|
+--numeric   : source for numerical routines
|
+--tools     : source for tools
|
+--pnet      : source for library for distributed networks
|
+--climate   : climate code 
|
+--lpj       : lpj functions
|
+--soil      : soil code
|
+--grass     : grass PFT code
|
+--tree      : tree PFT code
|
+--crop      : crop PFT code
|
+--landuse   : land use code
|
+--spitfire  : spitfire fire code
|
+--reservoir : reservoir code
|
+--socket    : socket communication library
|
+--image     : Coupler to IMAGE model
|
+--netcdf    : NetCDF output functions
|
+--utils     : utility programs
Configure for your operating system

On the HLRS2015 cluster at PIK you have to set the lpjml module for compilation and linking:
module load lpjml
% ./configure.sh
Configuring LPJmL 3.5.003...
Operating system is Linux
Intel MPI found
Create executables with make all
Put . $PWD/bin/lpj_paths.sh in your $HOME/.profile
The configure script creates scripts lpj_paths.sh/lpj_paths.csh for the bash/csh shell that sets all necessary environment variables. They are set by invoking
. bin/lpj_paths.sh
from the LPJmL root directory. It is recommended to put this statement into your ~./.profile. If configure script exits with message "Unsupported operating system", Makefile.$osname is created from Makefile.gcc and probably has to be modified for your operating system/compiler. If the configure script finds a MPI environment a parallel version of lpjml is built. This is done by checking for mpicc/mpiicc in the search path for commands. The configure script creates a copy of the following OS-specific makefiles from directory config:

MakefileDescription
Makefile.aix IBM AIX settings (xlc compiler)
Makefile.aix_mpi IBM AIX and MPI environment
Makefile.gcc GNU C-compiler settings
Makefile.intel Intel C-compiler settings
Makefile.intel_mpi Intel C-compiler and Intel MPI settings
Makefile.cluster2015 Intel C-compiler and Intel MPI on cluster2015 at PIK
Makefile.mpich GNU C-Compiler and MPI Chameleon settings
Makefile.win32 Windows settings (used by configure.bat)

Compilation flags

Compilation of lpjml is customized by definition of macros in the LPJFLAGS section of Makefile.inc:
LPJFLAGS= -Dflag1 ...
FlagDescription
DAILY_ESTABLISHMENT enable daily establishment
DEBUG diagnostic output is generated for debugging purposes
DOUBLE_HARVEST adding correct sequencing of harvest events
IMAGE Coupler to IMAGE model enabled via socket library
LINEAR_DECAY use linearized functions for litter decay
SAFE code is compiled with additional checks
STORECLIMATE Store climate input data in memory for spin up phase
USE_MPI compile parallel version of LPJmL
USE_NETCDF enable NetCDF input/output
USE_NETCDF4 enable NetCDF version 4 input/output
USE_RAND48 use drand48() random number generator
USE_UDUNITS enable unit conversion in NetCDF files
WITH_FPE floating point exceptions are enabled for debugging purposes
Create executables:

make    
One executable in directory bin is created:

lpjml - LPJmL simulation code

Utility programs are compiled by

make utils
The following programs are created in the bin directory:

ProgramDescription
lpjliveview Live view on screen of simulation output. lpjml must be started with '-output socket=hostname' option.
cft2clm convert binary land use data files to clm data files for LPJmL
lpjcat concatenates LPJmL restart files.
lpjcheck checks syntax of LPJmL configuration files.
cru2clm converts CRU data into file format suitable for LPJmL.
printheader prints file header of LPJmL input data files.
lpjprint prints content of restart file to stdout.
txt2clm converts CRU data files into LPJmL climate data files. CRU data files have to be in the format specified here.
An installation directory can be defined during configuration:
./configure.sh -prefix /data/biosx/lpjml-3.5.003
Then the binaries can be copied in the installation directory by invoking

make install
The necessary parameter and include files will be copied, too. Create output and restart directory:

make test
On the compute cluster at PIK it is better to create symbolic links of the input, output, and restart directory to the parallel GPFS filesystem /scratch. This improves performance in particular for the parallel code.

% mkdir -p /p/tmp/$USER/lpj/restart
% mkdir -p /p/tmp/$USER/lpj/output
% ln -sf /p/tmp/$USER/lpj/restart restart
% ln -sf /p/tmp/$USER/lpj/output output

Man pages

Manual pages for the man command are located in the $LPJROOT/man directory. The MANPAPTH variable is set by the lpj_paths.sh script. HTML versions of the manual pages are located here. Invoking

apropos lpj
yields list of man pages related to LPJmL.

Running the program

Sequential version can be started by invoking:

./bin/lpjml 
By default the configuration file lpjml.conf is read. The contents of this file can be viewed here. If you run lpjml outside the root directory of LPJmL, the following environment variable should be set (done by lpj_paths.sh):

export LPJROOT=$HOME/lpjml
Then all includes are found.

Environment variables

The following environment variables are used by LPJmL:

Environment variableDescription
LPJCONFIG default LPJmL configuration filename
LPJPREP defines preprocessor command for LPJmL configuration file, default is cpp -P
LPJROOT defines the root directory for LPJmL. This directory is added to the include directories of the preprocessor. Is usually set by lpj_paths.sh.
LPJIMAGE sets host where IMAGE model is running
LPJWAITIMAGE sets time to wait for connection to IMAGE model
LPJINPATH Path append to the input filenames. Only done for filenames without absolute path.
LPJRESTARTPATH Path append to the restart filenames. Only done for filenames without absolute path.
LPJOPTIONS preprocessor runtime options for LPJmL
LPJOUTPATH Path appended to the output filenames. Only done for filenames without absolute path.
LPOUTPUT Default method for generating output files. Valid values are write, mpi2, gather, and socket. Method mpi2 and gather are used by the MPI version only.

Runtime options of LPJmL

The following runtime options are defined for lpjml:

OptionDescription
-Iincludepath Adds include path for LPJmL configuration file
-Dmacro[=value] Defines macro
-h print usage of lpjml
-v print compiler used and LPJmL flags set
-param print LPJmL parameter for soils and PFTs
-pp cmd set preprocessor program to cmd. Default is cpp -P.
-fpe enable floating point exceptions
-image host set host where IMAGE model is running. Default is 'localhost'. Option is only availabel for the IMAGE version.
-wait time set time to wait for connection to IMAGE model measured in sec. Default is 10 sec.
-output method Method for generating output files. Valid values for are write, mpi2, gather, and socket. Methods mpi2 and gather are only available for the MPI version
-inpath dir input directory path
-outpath dir output directory path
-restartpath dir restart directory path
The first two options will be sent to the preprocessor. The shell script lpjdistribute can be used to create *.conf files in order to divide the simulation into pieces for the sequential version:

lpjdistribute <number of pieces> [<number of grid cells>]
For the parallel version LoadLeveler files for Linux and AIX are provided. A job can be submitted by invoking the lpjsubmit script:

lpjsubmit ntasks LPJmL_args...
Depending on your MPI version installed the program can be started interactively:

% mpirun -np 256 $LPJROOT/bin/lpjml
******************************************************************************
****        _     ____     _           _       _  _          ___          ****
****       | |   |  _ \   | |_ __ ___ | |     | || |        / _ \         ****
****       | |   | |_) |  | | '_ ` _ \| |     | || |_      | | | |        ****
****       | |___|  __/ |_| | | | | | | |___  |__   _|  _  | |_| |        ****
****       |_____|_|   \___/|_| |_| |_|_____|    |_|   (_)  \___/         ****
****                                                                      ****
****              lpjml C Version 4.0.001-3352 (Aug  1 2017)              ****
****    Dynamic global vegetation model with natural and managed land     ****
****                     Compiled for Linux with MPI                      ****
****       (C) Potsdam Institute for Climate Impact Research (PIK),       ****
****                          see COPYRIGHT file                          ****
****              Authors, and contributors see AUTHORS file              ****
**** This version of LPJmL is licensed under GNU AGPL Version 3 or later  ****
****        See LICENSE file or go to http://www.gnu.org/licenses/        ****
****                 or invoke lpjml -l to print license                  ****
****          Contact: https://gitlab.pik-potsdam.de/lpjml/lpjml          ****
****                                                                      ****
******************************************************************************

Running for user bloh on login01 at Tue Aug  1 15:33:19 2017
Reading configuration from 'lpjml.conf' with options '-DFROM_RESTART'.
==============================================================================
Simulation "LPJmL Run" running on 256 tasks
Simulation with random precipitation, fire, permafrost,
land use, potential irrigation.
Working directory: /home/bloh/lpjtrunk
Starting from restart file 'restart/restart_1900_nv_stdfire.lpj'.
Input files:
Variable   Fmt Varname  Filename
---------- --- -------- ------------------------------------------------------
soil       raw          /data/biosx/LPJ/cru06/soil.bin
coord      clm          /data/biosx/LPJ/cru06/grid.bin
drainclass raw          /data/biosx/LPJ/cru06/drainclass.bin
temp       clm          /data/biosx/LPJ/cru06/tmp.clm
prec       clm          /data/biosx/LPJ/cru06/pre.clm
pintensity clm          /data/biosx/LPJ/cru06/pintensity.clm
cloud      clm          /data/biosx/LPJ/cru06/cld.clm
co2        txt          /data/biosx/LPJ/cru06/co2_1901-2007.dat
countries  clm          /data/biosx/LPJ/cru06/cow_mg_2006.bin
landuse    clm          /data/biosx/LPJ/input_new/cft1700_2005_16cfts_SR.bin
wetdays    clm          /data/biosx/LPJ/cru06/wet.clm
---------- --- -------- ------------------------------------------------------
Writing restart file 'restart/restart_1900_crop_stdfire.lpj' after year 1900.
Number of output files:       21
Byte order in output files:   little endian
Output method: gathered
Variable    Fmt Unit         Type   Filename
----------- --- ------------ ------ ------------------------------------------
grid        raw       degree  short output/grid.bin
fpc         cdf            -  float output/fpc.nc
mnpp        cdf  gC/m2/month  float output/mnpp.nc
mgpp        cdf  gC/m2/month  float output/mgpp.nc
mrh         cdf  gC/m2/month  float output/mrh.nc
mfpar       cdf            -  float output/mfpar.nc
mtransp     cdf     mm/month  float output/mtransp.nc
mrunoff     cdf     mm/month  float output/mrunoff.nc
mevap       cdf     mm/month  float output/mevap.nc
minterc     cdf     mm/month  float output/minterc.nc
mswc1       cdf           mm  float output/mswc1.nc
mswc2       cdf           mm  float output/mswc2.nc
firec       cdf     gC/m2/yr  float output/firec.nc
firef       cdf         1/yr  float output/firef.nc
vegc        cdf        gC/m2  float output/vegc.nc
soilc       cdf       kgC/m2  float output/soilc.nc
litc        cdf        gC/m2  float output/litc.nc
flux_estab  cdf     gC/m2/yr  float output/flux_estab.nc
harvest     cdf     gC/m2/yr  float output/flux_harvest.nc
pft_harvest cdf     gC/m2/yr  float output/pft_harvest.pft.nc
cftfrac     cdf            -  float output/cftfrac.nc
----------- --- ------------ ------ ------------------------------------------
Spinup years:                   390
Cycle length during spinup:      30
First year:                    1901
Last year:                     2006
Number of grid cells:         67420
==============================================================================
Invalid soil code in 8221 cells.
Simulation begins...

  Carbon flux (GtC)                        Water (km3)
--------------------------------------- ----------------------------------
Spinup NEP     estab   fire    harvest total   transp     evap    interc  wd
------ ------- ------- ------- ------- ------- ---------- ------- ------- -------
1511   1.501   0.035   2.880   0.941  -2.285    38904.6 11267.0  6164.6    42.8
1512   4.581   0.039   2.854   0.882   0.883    40345.6  9377.3  6026.2    19.6
1513   8.343   0.034   2.858   0.928   4.591    42931.2  8257.5  6324.6    19.5
1895   5.057   0.092   3.005   3.241  -1.097    41048.9  9708.0  6208.7   402.8
1896   4.440   0.093   3.108   3.264  -1.840    40719.4  9421.6  6157.5   408.8
1897   5.420   0.093   2.953   3.385  -0.825    41137.3  9570.0  6379.0   421.2
1898   4.767   0.094   3.099   3.415  -1.653    40487.9  9437.5  6225.7   427.8
1899   4.396   0.095   3.195   3.247  -1.951    40203.3  9265.7  6165.1   442.0
1900   4.745   0.095   3.087   3.385  -1.632    40936.6  9858.3  6073.5   442.1
...
2004  14.578   0.166   2.684  10.105   1.955    39398.7 12500.4  6838.8  2338.6
2005  12.729   0.166   2.906   9.744   0.245    38922.7 12371.3  6835.9  2348.9
2006  14.443   0.166   2.652   9.823   2.134    39089.3 12688.4  6930.5  2325.7
Simulation ended.
lpjml successfully terminated, 59199 grid cells processed.
Wall clock time:	2361 sec, 8e-05 sec/cell/year.
Total wall clock time:	2478 sec

Error codes

If lpjml fails an error message is displayed in the following format

ERRORxxx: message
where xxx is the error code returned. The following error codes are defined:

/
Error code Description Error type
1 Error reading configuration External
2 Error initializing input data External
3 Error initializing grid External
4 Invalid carbon balance Internal
5 Invalid water balance Internal
6 Negative discharge Internal
7 Negative fire probability Internal
8 Negative soil moisture Internal
9 Error allocating memory External
10 Negative stand fraction Internal
11 Stand fraction sum error Internal
12 List is empty in dellistitem Internal
13 Index out of range in dellistitem Internal
15 Invalid year in getco2() External
16 Crop fraction >1 Internal
17 No natural stand for deforest Internal
18 Wrong cultivation type Internal
19 Floating point error Internal
21 PFT list is not empty in setaside Internal
22 Negative establishment rate Internal
23 Output channel is broken in socket connection External
24 Sending data to IMAGE model failed External
25 Opening connection to IMAGE model failed External
26 Not enough setaside stand created Internal
27 Forest left after deforestation Internal
28 Outflow reservoir error Internal
29 Permafrost error Internal
30 Global waterbalance error Internal
31 Cannot store climate data External
32 NO FMS coupler External
33 Cannot initialize soil temperature External
External errors are caused by invalid or missing input files while internal errors are caused by problems inside the LPJmL code. The latter will cause a core dump and have to be fixed by program changes. Some errors will only be generated if -DSAFE is set in the compile options of Makefile.inc. The options set at compile time can be obtained by invoking
% lpjml -v
lpjml C Version 4.0.001-3352 (Aug  1 2017)
Operating system: Linux, little endian
Compiler:         Intel C version 15.00, 64 bit
Build number:     1320
Compile flags:    "-DUSE_RAND48 -DUSE_MPI -DSAFE -DWITH_FPE -DSTORECLIMATE -DUSE_NETCDF -DUSE_UDUNITS "
It is recommended to compile the code without optimization and inlining making the inspection of the core file easier. Configuring in $LPJROOT with
./configure.sh -debug
make clean
make
will do the job. If no core file is generated set the user limit for core files:
% ulimit -c
0
% ulimit -c unlimited
Then the core can be analysed with the gdb debugger:
gdb $LPJROOT/bin/lpjml core
or by invoking the backtrace script:
backtrace

Building with Visual C++ 2005

How to get the software

Microsoft free version of the Visual Studio allows you to download the program, including a debugger. You can get it at https://www.visualstudio.com/downloads and choose Visual Studio Community. The link might only work for the newest version 2017, since that is available now. You should still be able to download the 2015 version at https://www.microsoft.com/de-DE/download/details.aspx?id=48146 or https://www.visualstudio.com/vs/older-downloads/. If you are working with windows systems anyway, this is a nice and handy developer environment. The main difference to alternatives (eclipse, text editors, etc.) is that VS uses no makefiles. Instead, you'll have to configure the project by hand. However, you still may use the makefiles supplied with the code in windows systems, if you have VS C++ installed.

SVN management

The plugin VisualSVN (download here) allows for smoothly integrating the SVN functionality of Tortoise in Visual Studio and really makes things nice there. However, it seems that PIK computers are member of an Active Directory domain, so buying a license is needed... :-(

Setting up a VS project

lick near the icon in the upper left to start a new project a new project on the downward arrow and choose 'start project from existing code', which starts a project wizard:
In case you are not able to create a project (error message: "Exception from HRESULT: 0x80041FE2"), go to the Windows control panel → uninstall a program → VS2015 → 'right click': change → MODIFY → drop down menu: Programming Languages → Visual C++ → check the box: 'Windows XP Support for C++' → Update That should solve the problem and you can now create a project from existing code. After the project has been created, you need to exclude all .c files from the project that are not essential to the code. This can be done in the Solution Explorer (upper left hand corner). Finally, you need to specify the project properties. For this, right-click on the project in the Solution Explorer.

Done. Now right-click on the project in the Solution Explorer and select rebuild or build.
Last change: 15 November 2017 by Werner von Bloh, PIK Potsdam