TABCMP_V

Latest: TABCMP_V v2.01 (04JUN20)
22MAR24
Contents

Introduction

Part of the RFM handling of
Look-Up Tables.

TABCMP_V is a Fortran90 program to compress RFM-generated look-up tables of pre-tabulated absorption coefficient (.tab files, created using the TAB option) in the spectral domain, essentially by removing points where the absorption coefficient varies smoothly enough with wavenumber that it can be accurately reconstructed by interpolation from the adjacent points.

This typically provides a factor 10 compression.

Further compression in the other tabulation axes can then be obtained by running the TABCMP_X program.

General Comments

  1. This program can only be run on .tab files with a regular spectral grid &mdashl it cannot be re-run using its output as input.
  2. The program can be run on either ASCII (text) or binary versions of the .tab file, and output can be in either format (see note on Binary files).
  3. The program asks for a convolution width, which should approximately match the spectral resolution of the measurements you are simulating. If in doubt, go for a smaller width, or no convolution at all (type <CR> in response to the prompt), however this will result in less compression.
  4. The program also asks for an Accuracy Criterion. This approximates the maximum interpolation error in transmittance, or as a fraction of radiance, resulting from the compression. If in doubt, go for a smaller number (but again, resulting in less compression).
  5. You are advised to compare RFM runs for a few selected cases using both uncompressed and compressed .tab files (just a matter of changing the filename(s) in the *LUT section of the driver file) to verify that the additional error from the compression is acceptable.

Compression Procedure

A
.tab file contains a tabulation of absorption coefficient k(ν,p,T,q) [m2/kmol] of a single species, where ν,p,T,q are axes of wavenumber, pressure, temperature and volume mixing ratio (VMR) scale factor (the last two are optional), as well embedded profiles of temperature and VMR interpolated to the pressure axis.

By default, the RFM generates .tab files on a regularly spaced spectral axis. As with the line-by-line calculations, a fine grid (typically 0.001 cm-1 for the mid-infrared for nadir viewing, 0.0005 cm-1 for limb-viewing) is required to capture the sharpest details present in the atmospheric spectrum, even if subsequently convolved to a much lower instrument resolution. However, between the spectral lines, large regions may be adequately represented by an interpolation of k (In practice, the RFM performs a linear interpolation of ln(k) for LUTs). Also, if it is known that the spectra will be subsequently convolved to a lower resolution, spectrally-confined interpolation errors of different sign may cancel out in the convolution.

Using the p,T,q tabulation axis intervals to define 'Cells' (see Accuracy Criteria), the user then sets the accuracy Δτ within which the transmittance of every cell has to match the original transmittance as spectral points k are removed and replaced by an interpolated value k':

| exp(-k'lu) - exp(-klu) | ≤ Δτ

This limit has to be satisfied for every spectral point νl for every cell (equivalent to (pi, Tj, qk) axis points). However, the user can instead specify that it only has to be met after convolution with a triangular kernel Ψ(ν) of FWHM Δν (i.e., triangle with base 2Δν)

| l Ψ(νl) [ exp(-k'lu) - exp(-klu)] | ≤ Δτ

where the summation includes all fine grid points within ± Δν and is evaluated at a spacing of Δν along the spectral axis. It is suggested that Δν is chosen to match the resolution of the modelled instrument.

The procedure is to evaluate the maximum error over all cells if each spectral grid point were replaced by an interpolated value, and the point with the smallest maximum error is removed. The procedure is repeated until no more points can be removed without exceeding the specified interpolation error.

Installing tabcmp_v

Download the source code: [tabcmp_v.f90] and the auxiliary modules [tabaux.f90]

Then compile with any FORTRAN compiler, eg

gfortran tabaux.f90 tabcmp_v.f90 -o tabcmp_v
(although, if using binary .tab files, use the same compiler as for the RFM which created the original files)

Running tabcmp_v

To run the program, simply type tabcmp_v and respond to the prompts.

A typical run might be (user responses in bold)

R-TABCMP_V: Running TABCMP_V v2.0 Input file: tab_co.asc Convolved resolution [cm-1]: 0.5 Max absorption error (eg 0.001): 0.001 Output file: tab_co.asc_cmp I-TABCMP_V: Reading in LUT ... I-TABCMP_V: Starting to remove points. Initial No.= 250001 250000 2000.0500 0.0000E+00 100.0% 240000 2028.1920 0.0000E+00 96.0% ... 20000 2127.7490 4.3019E-06 8.0% 10000 2153.8520 2.0206E-05 4.0% 1634 Err.lim.= 1.0034E-03 I-TABCMP_V: Summary: Orig Npts= 250001 New Npts= 1634 Red.Factor= 0.7% I-TABCMP_V: Writing new .tab file ... R-TABCMP_V: Successful completion

The program took a few hours since this is both a large number of initial points and a broad convolution. However the new .tab file is only 0.7% of the size of the original file.

Notes

  1. The first line tells you which version (2.0) of the program is being executed.
  2. The second line asks you for the uncompressed .tab file, i.e. the file originally generated by the RFM with the TAB option
  3. The third line asks for the convolution width, i.e. the FWHM of the triangular convolution function to be applied to the transmittance spectra. Typing <CR> will result in no convolution.
  4. The fourth line asks for the maximum error in transmittance for each cell. A value 0.001 is suggested which approximately equates to a 0.1% maximum difference in spectra (transmittance, absorption or radiance) from the applied spectral compression.
  5. The fifth line prompts you for the name of the compressed, output .tab file. If this filename contains the string .asc it will be an ASCII (text) file, if it contains .bin it will be a binary file, otherwise it will match the type of the input file.
  6. The sixth line tells you that it is starting to read in the uncompressed .tab file. If it is a large file this can take a few seconds.
  7. The following lines indicate progress. The first such message tells you the number of spectral points in the original file (in this case covering the range 2000–2250 cm-1 at 0.001 cm-1 resolution).
    Then a series of progress messages as every 10 000 pts are removed, listing also the wavenumber of the removed point, the maximum interpolation error (this number gradually increases until the max errror, 0.001 in this case, is reached), and the percentage of original points still retained.
  8. The processing finishes with a summary of the number of points retained
  9. The new .tab file is written out (usually a quick process since it is much reduced)
  10. The program should finish with the message stating successful completion.

Subset Processing

An uncompressed LUT may be too large to be accommodated in memory so if the program fails to execute the suggestion is to run an additional program to subset the file into smaller spectral ranges and then merge them again
tabmrg
To merge/extend/truncate files in spectral domain

Version History

v2.01 (04JUN20)
Move subroutines and functions into tabaux.f90. Changes compilation but results should be identical.
v2.00 (18APR19)
Rewritten in F90. Also process entire file rather than sub-dividing.
v1.21 (05AUG15)
Change '$' in WRITE statements to avoid gfortran warnings
v1.2 (24FEB15)
Improve auto-detection of input file-type
v1.1 (29DEC14)
Allow for extra format identifier record in .tab file header and for tabulated ln(k) rather than tabulated k.
v1.00 (04NOV14)
Original code