# Installing and running LPJmL C Version 4.0.001

## Installation for Unix-like systems

This text describes how to install and run LPJmL on your system. The code has been tested for AIX, Linux, Mac OS X, and Windows with cygwin and Microsoft C compiler (sequential version without MPI). Cygwin is a Linux-like environment for Windows (http://www.cygwin.com) On Mac OS the xcode suite must be installed in order to have the gcc compiler. create lpj directory:

    mkdir lpjml

Go to lpj directory:

    cd lpjml

untar source and data files

    gzip -cd lpjml-4.0.001.tar.gz|tar -xf -

Directory tree after extracting the tar files:

    --lpjml
    |
    +-config        : OS- and compiler-specific Makefiles
    |
    +-bin           : Directory for executables and shell scripts
    |
    +-R             : Scripts for R statistical package
    |
    +-man           : Manual pages root directory
    | |
    | +-man1        : Manual pages for programs and scripts
    | |
    | +-man3        : Manual pages for functions
    | |
    | +-man5        : Manual pages for file formats
    |
    +--html         : Documentation and man pages in HTML format
    |
    +--doc          : Documentation
    |
    +--par          : Parameter files for PFTs and soils
    |
    +--input        : input data set for climate
    |
    +--output       : output data (created by make test)
    |
    +--restart      : restart data (created by make test)
    |
    +--lib          : library files (created by make)
    |
    +--include      : include files
    |
    +--src          : source tree
    |
    +--numeric   : source for numerical routines
    |
    +--tools     : source for tools
    |
    +--pnet      : source for library for distributed networks
    |
    +--climate   : climate code
    |
    +--lpj       : lpj functions
    |
    +--soil      : soil code
    |
    +--grass     : grass PFT code
    |
    +--tree      : tree PFT code
    |
    +--crop      : crop PFT code
    |
    +--landuse   : land use code
    |
    +--spitfire  : spitfire fire code
    |
    +--reservoir : reservoir code
    |
    +--socket    : socket communication library
    |
    +--image     : Coupler to IMAGE model
    |
    +--netcdf    : NetCDF output functions
    |
    +--utils     : utility programs

Configure for your operating system On the HLRS2015 cluster at PIK you have to set the lpjml module for compilation and linking:

    module load lpjml

    % ./configure.sh
    Configuring LPJmL 3.5.003...
    Operating system is Linux
    Intel MPI found
    Create executables with make all
    Put . $PWD/bin/lpj_paths.sh in your $HOME/.profile

The configure script creates scripts `lpj_paths.sh`/`lpj_paths.csh` for the bash/csh shell that sets all necessary environment variables. They are set by invoking

    . bin/lpj_paths.sh

from the LPJmL root directory. It is recommended to put this statement into your `~./.profile`. If configure script exits with message "Unsupported operating system", `Makefile.$osname` is created from `Makefile.gcc` and probably has to be modified for your operating system/compiler. If the configure script finds a MPI environment a parallel version of `lpjml` is built. This is done by checking for `mpicc`/`mpiicc` in the search path for commands. The configure script creates a copy of the following OS-specific makefiles from directory `config`:

Makefile | Description |
--- | --- | ---
`Makefile.aix` | IBM AIX settings (xlc compiler)
`Makefile.aix_mpi` | IBM AIX and MPI environment
`Makefile.gcc` | GNU C-compiler settings
`Makefile.intel` | Intel C-compiler settings
`Makefile.intel_mpi` | Intel C-compiler and Intel MPI settings
`Makefile.cluster2015` | Intel C-compiler and Intel MPI on cluster2015 at PIK
`Makefile.mpich` | GNU C-Compiler and MPI Chameleon settings
`Makefile.win32` | Windows settings (used by `configure.bat`)

### Compilation flags

Compilation of `lpjml` is customized by definition of macros in the `LPJFLAGS` section of `Makefile.inc`:

    LPJFLAGS= -Dflag1 ...

Flag | Description |
--- | --- | ---
DAILY_ESTABLISHMENT | enable daily establishment
DEBUG | diagnostic output is generated for debugging purposes
DOUBLE_HARVEST | adding correct sequencing of harvest events
IMAGE | Coupler to IMAGE model enabled via socket library
LINEAR_DECAY | use linearized functions for litter decay
SAFE | code is compiled with additional checks
STORECLIMATE | Store climate input data in memory for spin up phase
USE_MPI | compile parallel version of LPJmL
USE_NETCDF | enable NetCDF input/output
USE_NETCDF4 | enable NetCDF version 4 input/output
USE_RAND48 | use `drand48()` random number generator
USE_UDUNITS | enable unit conversion in NetCDF files
WITH_FPE | floating point exceptions are enabled for debugging purposes

Create executables:

    make

One executable in directory `bin` is created: `lpjml` - LPJmL simulation code Utility programs are compiled by

    make utils

The following programs are created in the `bin` directory:

Program | Description |
--- | --- | ---
`lpjliveview` | Live view on screen of simulation output. lpjml must be started with '-output socket=hostname' option.
`cft2clm` | convert binary land use data files to clm data files for LPJmL
`lpjcat` | concatenates LPJmL restart files.
`lpjcheck` | checks syntax of LPJmL configuration files.
`cru2clm` | converts CRU data into file format suitable for LPJmL.
`printheader` | prints file header of LPJmL input data files.
`lpjprint` | prints content of restart file to stdout.
`txt2clm` | converts CRU data files into LPJmL climate data files. CRU data files have to be in the format specified here.

An installation directory can be defined during configuration:

    ./configure.sh -prefix /data/biosx/lpjml-3.5.003

Then the binaries can be copied in the installation directory by invoking

    make install

The necessary parameter and include files will be copied, too. Create output and restart directory:

    make test

On the compute cluster at PIK it is better to create symbolic links of the input, output, and restart directory to the parallel GPFS filesystem `/scratch`. This improves performance in particular for the parallel code.

    % mkdir -p /p/tmp/$USER/lpj/restart
    % mkdir -p /p/tmp/$USER/lpj/output
    % ln -sf /p/tmp/$USER/lpj/restart restart
    % ln -sf /p/tmp/$USER/lpj/output output

### Man pages

Manual pages for the `man` command are located in the `$LPJROOT/man` directory. The MANPAPTH variable is set by the `lpj_paths.sh` script. HTML versions of the manual pages are located here. Invoking

    apropos lpj

yields list of man pages related to LPJmL.

## Running the program

Sequential version can be started by invoking:

    ./bin/lpjml

By default the configuration file `lpjml.conf` is read. The contents of this file can be viewed here. If you run `lpjml` outside the root directory of LPJmL, the following environment variable should be set (done by `lpj_paths.sh`):

    export LPJROOT=$HOME/lpjml

Then all includes are found.

### Environment variables

The following environment variables are used by LPJmL:

Environment variable | Description |
--- | --- | ---
LPJCONFIG | default LPJmL configuration filename
LPJPREP | defines preprocessor command for LPJmL configuration file, default is `cpp -P`
LPJROOT | defines the root directory for LPJmL. This directory is added to the include directories of the preprocessor. Is usually set by `lpj_paths.sh`.
LPJIMAGE | sets host where IMAGE model is running
LPJWAITIMAGE | sets time to wait for connection to IMAGE model
LPJINPATH | Path append to the input filenames. Only done for filenames without absolute path.
LPJRESTARTPATH | Path append to the restart filenames. Only done for filenames without absolute path.
LPJOPTIONS | preprocessor runtime options for LPJmL
LPJOUTPATH | Path appended to the output filenames. Only done for filenames without absolute path.
LPOUTPUT | Default method for generating output files. Valid values are write, mpi2, gather, and socket. Method mpi2 and gather are used by the MPI version only.

### Runtime options of LPJmL

The following runtime options are defined for `lpjml`:

Option | Description |
--- | --- | ---
-Iincludepath | Adds include path for LPJmL configuration file
-Dmacro[=value] | Defines macro
-h | print usage of lpjml
-v | print compiler used and LPJmL flags set
-param | print LPJmL parameter for soils and PFTs
-pp cmd | set preprocessor program to cmd. Default is cpp -P.
-fpe | enable floating point exceptions
-image host | set host where IMAGE model is running. Default is 'localhost'. Option is only availabel for the IMAGE version.
-wait time | set time to wait for connection to IMAGE model measured in sec. Default is 10 sec.
-output method | Method for generating output files. Valid values for are write, mpi2, gather, and socket. Methods mpi2 and gather are only available for the MPI version
-inpath dir | input directory path
-outpath dir | output directory path
-restartpath dir | restart directory path

The first two options will be sent to the preprocessor. The shell script `lpjdistribute` can be used to create `*.conf` files in order to divide the simulation into pieces for the sequential version:

    lpjdistribute <number of pieces> [<number of grid cells>]

For the parallel version LoadLeveler files for Linux and AIX are provided. A job can be submitted by invoking the `lpjsubmit` script:

    lpjsubmit ntasks LPJmL_args...

Depending on your MPI version installed the program can be started interactively:

    % mpirun -np 256 $LPJROOT/bin/lpjml
    ******************************************************************************
    ****        _     ____     _           _       _  _          ___          ****
    ****       | |   |  _ \   | |_ __ ___ | |     | || |        / _ \         ****
    ****       | |   | |_) |  | | '_ ` _ \| |     | || |_      | | | |        ****
    ****       | |___|  __/ |_| | | | | | | |___  |__   _|  _  | |_| |        ****
    ****       |_____|_|   \___/|_| |_| |_|_____|    |_|   (_)  \___/         ****
    ****                                                                      ****
    ****              lpjml C Version 4.0.001-3352 (Aug  1 2017)              ****
    ****    Dynamic global vegetation model with natural and managed land     ****
    ****                     Compiled for Linux with MPI                      ****
    ****       (C) Potsdam Institute for Climate Impact Research (PIK),       ****
    ****                          see COPYRIGHT file                          ****
    ****              Authors, and contributors see AUTHORS file              ****
    **** This version of LPJmL is licensed under GNU AGPL Version 3 or later  ****
    ****        See LICENSE file or go to http://www.gnu.org/licenses/        ****
    ****                 or invoke lpjml -l to print license                  ****
    ****          Contact: https://gitlab.pik-potsdam.de/lpjml/lpjml          ****
    ****                                                                      ****
    ******************************************************************************

    Running for user bloh on login01 at Tue Aug  1 15:33:19 2017
    Reading configuration from 'lpjml.conf' with options '-DFROM_RESTART'.
    ==============================================================================
    Simulation "LPJmL Run" running on 256 tasks
    Simulation with random precipitation, fire, permafrost,
    land use, potential irrigation.
    Working directory: /home/bloh/lpjtrunk
    Starting from restart file 'restart/restart_1900_nv_stdfire.lpj'.
    Input files:
    Variable   Fmt Varname  Filename
    ---------- --- -------- ------------------------------------------------------
    soil       raw          /data/biosx/LPJ/cru06/soil.bin
    coord      clm          /data/biosx/LPJ/cru06/grid.bin
    drainclass raw          /data/biosx/LPJ/cru06/drainclass.bin
    temp       clm          /data/biosx/LPJ/cru06/tmp.clm
    prec       clm          /data/biosx/LPJ/cru06/pre.clm
    pintensity clm          /data/biosx/LPJ/cru06/pintensity.clm
    cloud      clm          /data/biosx/LPJ/cru06/cld.clm
    co2        txt          /data/biosx/LPJ/cru06/co2_1901-2007.dat
    countries  clm          /data/biosx/LPJ/cru06/cow_mg_2006.bin
    landuse    clm          /data/biosx/LPJ/input_new/cft1700_2005_16cfts_SR.bin
    wetdays    clm          /data/biosx/LPJ/cru06/wet.clm
    ---------- --- -------- ------------------------------------------------------
    Writing restart file 'restart/restart_1900_crop_stdfire.lpj' after year 1900.
    Number of output files:       21
    Byte order in output files:   little endian
    Output method: gathered
    Variable    Fmt Unit         Type   Filename
    ----------- --- ------------ ------ ------------------------------------------
    grid        raw       degree  short output/grid.bin
    fpc         cdf            -  float output/fpc.nc
    mnpp        cdf  gC/m2/month  float output/mnpp.nc
    mgpp        cdf  gC/m2/month  float output/mgpp.nc
    mrh         cdf  gC/m2/month  float output/mrh.nc
    mfpar       cdf            -  float output/mfpar.nc
    mtransp     cdf     mm/month  float output/mtransp.nc
    mrunoff     cdf     mm/month  float output/mrunoff.nc
    mevap       cdf     mm/month  float output/mevap.nc
    minterc     cdf     mm/month  float output/minterc.nc
    mswc1       cdf           mm  float output/mswc1.nc
    mswc2       cdf           mm  float output/mswc2.nc
    firec       cdf     gC/m2/yr  float output/firec.nc
    firef       cdf         1/yr  float output/firef.nc
    vegc        cdf        gC/m2  float output/vegc.nc
    soilc       cdf       kgC/m2  float output/soilc.nc
    litc        cdf        gC/m2  float output/litc.nc
    flux_estab  cdf     gC/m2/yr  float output/flux_estab.nc
    harvest     cdf     gC/m2/yr  float output/flux_harvest.nc
    pft_harvest cdf     gC/m2/yr  float output/pft_harvest.pft.nc
    cftfrac     cdf            -  float output/cftfrac.nc
    ----------- --- ------------ ------ ------------------------------------------
    Spinup years:                   390
    Cycle length during spinup:      30
    First year:                    1901
    Last year:                     2006
    Number of grid cells:         67420
    ==============================================================================
    Invalid soil code in 8221 cells.
    Simulation begins...

      Carbon flux (GtC)                        Water (km3)
    --------------------------------------- ----------------------------------
    Spinup NEP     estab   fire    harvest total   transp     evap    interc  wd
    ------ ------- ------- ------- ------- ------- ---------- ------- ------- -------
    1511   1.501   0.035   2.880   0.941  -2.285    38904.6 11267.0  6164.6    42.8
    1512   4.581   0.039   2.854   0.882   0.883    40345.6  9377.3  6026.2    19.6
    1513   8.343   0.034   2.858   0.928   4.591    42931.2  8257.5  6324.6    19.5
    1895   5.057   0.092   3.005   3.241  -1.097    41048.9  9708.0  6208.7   402.8
    1896   4.440   0.093   3.108   3.264  -1.840    40719.4  9421.6  6157.5   408.8
    1897   5.420   0.093   2.953   3.385  -0.825    41137.3  9570.0  6379.0   421.2
    1898   4.767   0.094   3.099   3.415  -1.653    40487.9  9437.5  6225.7   427.8
    1899   4.396   0.095   3.195   3.247  -1.951    40203.3  9265.7  6165.1   442.0
    1900   4.745   0.095   3.087   3.385  -1.632    40936.6  9858.3  6073.5   442.1
    ...
    2004  14.578   0.166   2.684  10.105   1.955    39398.7 12500.4  6838.8  2338.6
    2005  12.729   0.166   2.906   9.744   0.245    38922.7 12371.3  6835.9  2348.9
    2006  14.443   0.166   2.652   9.823   2.134    39089.3 12688.4  6930.5  2325.7
    Simulation ended.
    lpjml successfully terminated, 59199 grid cells processed.
    Wall clock time:	2361 sec, 8e-05 sec/cell/year.
    Total wall clock time:	2478 sec

### Error codes

If `lpjml` fails an error message is displayed in the following format

    ERRORxxx: message

where `xxx` is the error code returned. The following error codes are defined:

Error code | Description | Error type |
--- | --- | --- | ---
1 | Error reading configuration | External
2 | Error initializing input data | External
3 | Error initializing grid | External
4 | Invalid carbon balance | Internal
5 | Invalid water balance | Internal
6 | Negative discharge | Internal
7 | Negative fire probability | Internal
8 | Negative soil moisture | Internal
9 | Error allocating memory | External
10 | Negative stand fraction | Internal
11 | Stand fraction sum error | Internal
12 | List is empty in `dellistitem` | Internal
13 | Index out of range in `dellistitem` | Internal
15 | Invalid year in `getco2()` | External
16 | Crop fraction >1 | Internal
17 | No natural stand for deforest | Internal
18 | Wrong cultivation type | Internal
19 | Floating point error | Internal
21 | PFT list is not empty in `setaside` | Internal
22 | Negative establishment rate | Internal
23 | Output channel is broken in socket connection | External
24 | Sending data to IMAGE model failed | External
25 | Opening connection to IMAGE model failed | External
26 | Not enough setaside stand created | Internal
27 | Forest left after deforestation | Internal
28 | Outflow reservoir error | Internal
29 | Permafrost error | Internal
30 | Global waterbalance error | Internal
31 | Cannot store climate data | External
32 | NO FMS coupler | External
33 | Cannot initialize soil temperature | External
/

External errors are caused by invalid or missing input files while internal errors are caused by problems inside the LPJmL code. The latter will cause a core dump and have to be fixed by program changes. Some errors will only be generated if -DSAFE is set in the compile options of `Makefile.inc`. The options set at compile time can be obtained by invoking

    % lpjml -v
    lpjml C Version 4.0.001-3352 (Aug  1 2017)
    Operating system: Linux, little endian
    Compiler:         Intel C version 15.00, 64 bit
    Build number:     1320
    Compile flags:    "-DUSE_RAND48 -DUSE_MPI -DSAFE -DWITH_FPE -DSTORECLIMATE -DUSE_NETCDF -DUSE_UDUNITS "

It is recommended to compile the code without optimization and inlining making the inspection of the core file easier. Configuring in $LPJROOT with

    ./configure.sh -debug
    make clean
    make

will do the job. If no core file is generated set the user limit for core files:

    % ulimit -c
    0
    % ulimit -c unlimited

Then the core can be analysed with the `gdb` debugger:

    gdb $LPJROOT/bin/lpjml core

or by invoking the `backtrace` script:

    backtrace

### Building with Visual C++ 2005

#### How to get the software

Microsoft free version of the Visual Studio allows you to download the program, including a debugger. You can get it at https://www.visualstudio.com/downloads and choose Visual Studio Community. The link might only work for the newest version 2017, since that is available now. You should still be able to download the 2015 version at https://www.microsoft.com/de-DE/download/details.aspx?id=48146 or https://www.visualstudio.com/vs/older-downloads/. If you are working with windows systems anyway, this is a nice and handy developer environment. The main difference to alternatives (eclipse, text editors, etc.) is that VS uses no makefiles. Instead, you'll have to configure the project by hand. However, you still may use the makefiles supplied with the code in windows systems, if you have VS C++ installed.

#### SVN management

The plugin VisualSVN (download here) allows for smoothly integrating the SVN functionality of Tortoise in Visual Studio and really makes things nice there. However, it seems that PIK computers are member of an Active Directory domain, so buying a license is needed... :-(

#### Setting up a VS project

lick near the icon in the upper left to start a new project a new project on the downward arrow and choose 'start project from existing code', which starts a project wizard:

- What type of project: Visual C++
- general properties

- name your project
- enter file location
- check "add files to project from these folders"
- check "add subfolders"
- check "show all files in Solution Explorer"

- project type

- check "Use Visual Studio"
- project type: "console application project"

- click finish, the rest can be specified later

In case you are not able to create a project (error message: "Exception from HRESULT: 0x80041FE2"), go to the Windows control panel → uninstall a program → VS2015 → 'right click': change → MODIFY → drop down menu: Programming Languages → Visual C++ → check the box: 'Windows XP Support for C++' → Update That should solve the problem and you can now create a project from existing code. After the project has been created, you need to exclude all .c files from the project that are not essential to the code. This can be done in the Solution Explorer (upper left hand corner).

- exclude all files from the project that are not LPJ-essential (right click → exclude from project). This comprises in directory `src`: the file `nooutput_gwb.c` in directory `lpj`, `utils`, and `lpj_climber4.c`.

Finally, you need to specify the project properties. For this, right-click on the project in the Solution Explorer.

- configuration properties:

- General:

- character set → use Multi-Byte Character Set
- IMPORTANT: if you want to run the executable also on other computers you need to statically link the libraries to the exe file. For this, choose "Use MFC in static library" for the "Use of MFC" option. Otherwise, this can remain with the "Use Standard Windows Libraries"

- Debugging:

- Command arguments `lpjml.conf`
- here, you can specify your working directory

- C/C++

- General:

- Additional include directories: add the `..\include` directory here
- Debug Information Format: C7 Compatible (`/Z7`)

- Optimization

- disabled (or whatever you want)

- Preprocessor

- preprocessor definitions: basically, here all compiler Flags from the `src/Makefile` can be specified, but without the "-D" prefix.

- Essential preprocessor definitions are: `WIN32; _DEBUG; _CONSOLE; SAFE; _USE_MATH_DEFINES; _CRT_SECURE_NO_DEPRECATE;`
- optional: `IMAGE`

- Code Generation

- Enable Minimal Rebuild → No
- Basic Runtime Checks → Default

- Advanced

- compile as → compile as C code (`/TC`)

- Linker

- General

- under 'Enable Incremental Linking' → 'NO (/INCREMENTAL:NO)'

- Input

- Additional Dependencies → `advapi32.lib ws2_32.lib`

Done. Now right-click on the project in the Solution Explorer and select rebuild or build.