MeteoIODoc  MeteoIODoc-2.6.0
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
Processing overview

The pre-processing infrastructure is described in ProcessingBlock (for its API). The goal of this page is to give an overview of the available filters and processing elements and their usage.

Modes of operation

It should be noted that filters often have two modes of operations: soft or hard. In soft mode, all value that is rejected is replaced by the filter parameter's value. This means that for a soft min filter set at 0.0, all values less than 0.0 will be replaced by 0.0. In hard mode, all rejected values are replaced by nodata.

Several filter take arguments describing a processing window (for example, FilterStdDev). In such a case, two values are given:

  • the minimum time span of the window
  • the minimum number of points in the window

The ProcessingBlock will walk through the data, starting at the current point and adding points to the processing window. When both of these criterias are met, the window is accepted. This means that a window defined as "6 21600" is a window that contains 6 points minimum AND spans at least 21600 seconds.

Filtering section

The filters are specified for each parameter in the [Filters] section. This section contains a list of the various meteo parameters (see MeteoData) with their associated choice of filtering algorithms and optional parameters.The filters are applied serialy, in the order they are given in. An example of such section is given below:

[Filters]
TA::filter1 = min_max
TA::arg1 = 230 330
RH::filter1 = min_max
RH::arg1 = -0.2 1.2
RH::filter2 = min_max
RH::arg2 = soft 0.0 1.0
PSUM::filter1 = min
PSUM::arg1 = -0.1
PSUM::filter2 = min
PSUM::arg2 = soft 0.

Available processing elements

New filters can easily be developed. The filters that are currently available are the following:

  • NONE: this does nothing (this is useful in an IMPORT to overwritte previous filters);
  • MIN: minimum check filter, see FilterMin
  • MAX: maximum check filter, see FilterMax
  • MIN_MAX: range check filter, see FilterMinMax
  • RATE: rate of change filter, see FilterRate
  • STD_DEV: reject data outside mean +/- k*stddev, see FilterStdDev
  • MAD: median absolute deviation, see FilterMAD
  • TUKEY: Tukey53H spike detection, based on median, see FilterTukey
  • UNHEATED_RAINGAUGE: detection of snow melting in a rain gauge, see FilterUnheatedPSUM
  • NO_CHANGE: reject data that changes too little (low variance), see FilterNoChange
  • TIME_CONSISTENCY: reject data that changes too much , see FilterTimeconsistency
  • DETECT_GRASS: detection of grass growing under the snow height sensor, see FilterDeGrass
  • POTENTIALSW: ensuring physically realistic incoming short wave radiation, see FilterPotentialSW

Some data transformations are also supported besides filtering, both very basic and generic data transformations:

  • SUPPR: delete all or some data, see FilterSuppr
  • ADD: adds a given offset to the data, see ProcAdd
  • MULT: multiply the data by a given factor, see ProcMult
  • NOISE: add to (or multiply by) randomly distributed noise, see ProcNoise

As well as more specific data transformations:

  • EXP_SMOOTHING: exponential smoothing of data, see ProcExpSmoothing
  • WMA_SMOOTHING: weighted moving average smoothing of data, see ProcWMASmoothing
  • IIR: Low Pass or High Pass critically damped filter, see ProcIIR
  • AGGREGATE: various data aggregation algorithms, see ProcAggregate
  • UNDERCATCH_WMO: WMO rain gauge correction for undercatch, using various correction models, see ProcUndercatch_WMO
  • UNDERCATCH_FORLAND: Forland1996 rain gauge correction for solid and liquid undercatch, using various correction models, see ProcUndercatch_Forland
  • UNDERCATCH_HAMON: Hamon1973 rain gauge correction for undercatch, see ProcUndercatch_Hamon
  • UNVENTILATED_T: unventilated temperature sensor correction, see ProcUnventilatedT
  • PSUM_DISTRIBUTE: distribute accumulated precipitation over preceeding timesteps, see ProcPSUMDistribute
  • SHADE: apply a shading mask to the Incoming or Reflected Short Wave Radiation, see ProcShade