Searching for new microwindows, either from scratch or following on after incorporating an existing microwindow database, is a two-stage process
- Survey: in which each measurement (wavenumber,altitude) is tested individually for the improvement it would introduce to the retrieval (effectively treated as single-point microwindows). This creates a (wavenumber,altitude) map of `usefulness' of individual points.
- Growth: starting with a measurement in a region of high density of individually useful points, microwindows are grown `outwards' by adding adjacent measurements until no further improvement is obtained. This is repeated for a number of trial microwindows different starting points, and the best grown microwindow is retained.
A related use of the program is to generate different `occupation matrices', optimal subsets of microwindows selected for a particular retrieval ranges and/or situations when measurements are missing in particular bands or tangent altitudes.
2. Compiling MWMAKE
The source code consists of one program module mwmake.for plus a number of subroutines *.for and include modules *.inc. These should all be placed in a new directory.Then the module namfil.for should be edited so that the string constant PTBDIR (initially set to '/home/crun/eodg/mipas/spectra/') points to the top level directory that will contain the input spectra for the various atmospheres. The length of the character constant (C*30 initially) also has to be adjusted according to the new directory name (max of C*54).
NB: The subroutine dattim.for calls a FORTRAN 90 function DATE_AND_TIME in order to return the current date and time, written to some output files. This is not implemented for Sun systems, so a replacement subroutine date_and_time.for is supplied ( see Additional Software) and should be included in the directory before compilation.
The code can then simply be compiled as an executable mwmake using
f77 *.for -o mwmake
3. Input Spectra
MWMAKE operates by analysing sets of precomputed spectra representing Jacobians for each retrieved species (including continuum) and various Error Sources represented as 1-sigma perturbation spectra. These are stored as binary files, one file containing the Jacobian or Error spectrum for one MIPAS band for one tangent altitude for one atmosphere.Jacobian spectra have names including "jac", e.g.
ptb_d06000.bin_day_jac_h2o
where d is the band, 06000
is the tangent height
(6km), day is the atmosphere,
h2o is the retrieved species.
Error spectra have names excluding "jac", e.g
ptb_d06000.bin_day_var_h2o
where var_h2o is the error source
(variability of H2O, in this case),
Spectra exist for all 17 altitudes 6-68km represented in the nominal MIPAS scan pattern, but not all bands (eg there are no CH4 Jacobian or error spectra for the A band since there are no CH4 spectral features in the A band). The full list of spectra is given in the Appendix.
All these spectra should be placed in a subdirectories of the
top level directory (as defined by PTBDIR,
see section on Compiling MWMAKE).
./day/jac
day atmosphere, Jacobian spectra (anything with "jac" in the
filename)
./day/err
day atmosphere, error spectra (anything without "jac" in the
filename)
similarly for the other atmospheres ngt,
sum, win and equ
The spectra are generated as ASCII files (using the RFM), but, in order to save space and increase access speeds, stored as binary files (see binasc.f conversion program).
If it is necessary to access the spectra as ASCII files, just edit the filenames in namfil.for. The subroutine which actually opens the files (getspc.for) already checks for a string ".bin" in the filename and treats the file as binary/ASCII appropriately.
Back to Contents
4. Terminal Inputs
On execution, MWMAKE
will prompt the user for information defining the task
to be performed, most of which have reasonable defaults (ie the user can type
a carriage return, indicated as <CR>).
Input can either be via a terminal, or using a driver file.
The driver file option can be invoked either by specifying the file name
in response to the first prompt, or directly using the Unix construction,
e.g.
mwmake < mwmake.drv
The terminal dialogue/prompts are described in sequence below, together with possible user responses and defaults.
- R-MWMAKE: Running MWMAKE v.01MAR02 (or later version)
-
Info only: identifies the version of the code being executed. This version
ID is also written to all output files.
- Driver file [<CR>=terminal input]:
- Either the name of a driver file which contains all subsequent responses, or <CR> signifying inputs via terminal.
- Example Responses
- mwmake_ch4.drv
- Name of a driver file containing subsequent user-responses
- <CR> (Default - set in mwmake.for)
- No driver file - all subsequent inputs via terminal
- Target species:
- Parameter(s) to be retrieved, eg:
- ch4, h2o, hno3, n2o, no2, o3 and/or pt (upper/lower case not significant, at least one space or comma between species to be retrieved simultaneously).
- pt retrieves both pressure and temperature profiles jointly, assuming assuming a priori pointing knowledge (random 1.5% pressure between adjacent sweeps). ctm (continuum) can also be entered as a target parameter: a continuum profile is always retrieved for each microwindow but if specified as a target species then this is also incorporated in the Figure of Merit calculation.
- Example Responses
- CH4 N2O
- Methane and nitrous oxide joint-retrieval microwindows
- NB: There is no default option
- [prompt only if joint retrieval specified in previous response, or retrieved molecule > 4 characters in length]
- Retrieval code (up to C*4):
- Microwindows have labels of the form RRRRnnnn where nnnn is a number (padded with leading zeros). For single target species with molecule names 4 characters or less (including 'PT'), RRRR is set automatically to be the the retrieved species padded with trailing underscores. For other cases the user has to supply the 4-character code (case sensitive).
- Example Responses
- pTwv
- Pressure, temperature and Water Vapour joint retrieval (ie generate MWs with labels like pTwv0001)
- NB: There is no default option
- Atmosphere: [day<CR>,ngt,sum,win,equ,glo,glw]:
- The three-letter code selecting atmosphere(s) to be
used for evaluating the Figure of Merit. First five are as in
input spectra, additional options are
- glo Global average of 5 atmospheres
- glw Global average plus extra weight for polar winter
- Example Responses
- glw
- Use Polar winter-weighted global set
- <CR> (Default - set in mwmreq.for)
- Use mid-latitude day-time only
- VMR Accuracy requirement? [<CR>=25.0%]:
(if molecule selected as the (first) target species),
or
Temp Accuracy requirement? [<CR>=3.0K]: (if pT selected as the (first) target species.) - This sets up a constant value for the 'target accuracy profile', although how it is used is defined by other parameters. In the case of joint retrievals, it is defined for the first species specified in the Target species section.
- Example Responses
- 15
- Target profile set to 15% (assuming VMR) or 15K (assuming pT)
- <CR> (Default - set in mwmreq.for)
- Target profile set to 25% for VMR retrievals, 3K for temperature retrievals.
- Waveno.range or bands [<CR>=all]:
-
The lower, upper wavenumber limits [cm-1]
to be used for microwindow selection, or,
alternatively, a list of MIPAS bands:
- A 685-970 cm-1
- AB 1020-1170 cm-1
- B 1215-1500 cm-1
- C 1570-1750 cm-1
- D 1820-2410 cm-1
-
(v14MAY02 onwards) if a wavenumber range is specified, it is assumed
that this only applies to the search range for new microwindows.
However, if bands are specified it is assumed that this applies to
new microwindows and to included microwindows
(in earlier versions a specified wavenumber range also applied to both)
- Note: No points are selected within 0.2 cm-1 of the MIPAS band edges (to allow for AILS convolution)
- Example Responses
- 1200 1300
- Search range 1200-1300 cm-1, although this will actually only only cover 1215-1300cm-1 since 1200 lies between the AB and B bands, and the B band nominally covers 1215-1500cm-1.
- a d b (case-insensitive)
- Search MIPAS bands A, B and D only and, if existing microwindows are to be included, include only MWs within these bands.
- <CR> (Default - set in reqbnd.for)
- Search entire MIPAS range (685-2410 cm-1, excluding gaps between filters, which are not covered by the input spectra)
- I-REQBND: Searching Wavenumber range 685.200 2409.800 (for example)
- Info only: verification of the lower and upper wavenumber limits to be searched.
- Note: the above response would also be obtained if the user had typed a d b in response to the previous prompt - this message only lists the extreme lower and upper wavenumbers and not the details of bands to be skipped in between these limits.
- Merit(0:7), CPU Cost(0:9) and Mask(0:4) Fns? [<CR>= 0 0 0]:
- Integer parameters defining Merit function, CPU Cost function and MW growth algorithm to be used.
- These 3 digits are concatenated to form the `identifier' (ijk) for this particular run of MWMAKE (although this will be overridden if missing bands/altitudes are selected)
- Example Responses
- 4 1 2
- Use Merit Fn#4, CPU Cost Fn#1 and MW Growth Algorithm#2 Output filenames will contain the string '412'.
- <CR> (Default - set in mwmreq.for)
- Use Merit Fn#0, CPU Cost Fn#0 and MW Growth Algorithm#0. Output filenames will contain the string '000'.
- List of retrieval altitudes [km] [<CR>=6,9,..68]:
- Tangent altitudes for both measurements and retrieval (arbitrary order, but separated by at least 0.5km)
- Example Responses
- List of missing altitudes [km] [<CR>=none]
- Any tangent altitudes missing from the measurements (but not the retrieval) - used for constructing Occupation Matrices for missing-data scenarios. These altitudes have to be a subset of the altitude list supplied previously, or an error message will result. This overrides the usual identifier used in output filenames.
- Example Responses:
- 9 11
- missing measurements at 9 and 12 km
- <CR> (Default - set in reqalt.for)
- retrieve at 8, 11, ... 53km (16 tangent heights), with spectra available at all altitudes.
- [prompt only if missing altitude(s) have been specified in previous response]
- Missing Band (A,AB,B,C or D)? [<CR>=all]:
- For the missing altitude(s), the spectra may either be missing for all bands or for just one band.
- Example Responses
- ab (case insensitive)
- At previously specified missing altitudes, only AB band missing (A, B, C and D band spectra all available).
- <CR> (Default set in reqalt.for)
- All bands missing at missing altitude(s).
- Termination criteria Npt,NMW,DMrt,ReqF? [<CR>=5000 10 0.0 0.0]:
- The microwindow selection will finish when any of these criteria are met.
- See section on Termination Criteria for explanations.
- Example Responses
- 99999 10 0.1 1.0
- Terminate when either 10 microwindows have been found, or FOM no longer increases by at least 0.1, or when the requirement profile has been met at all altitudes
- <CR> (Default - set in reqend.for)
- Terminate when either 5000 points or 10 microwindows have been used, or when FOM no longer increases, and ignore requirement profile as a termination criterion.
- Minimum ctm info/level [<CR>=0 bits]:
- The minimum continuum information content required for each altitude in each microwindow. 1 `bit' corresponds to a factor 2 reduction from the a priori uncertainty (set at +/- 0.1/km extinction).
- Example Responses
- 5
- 5 bits, as currently used for MIPAS microwindows, equivalent to extinction retrieval to accuracy of +/-0.003/km
- <CR> (Default - set in mwmreq.for)
- No continuum requirement (allows opaque tangent paths)
- I-MWMREQ: Min Ctm = A Pr.Cov* 9.7656250E-04 (for example)
- Info only: verification of the fraction of the a priori covariance that will be used as a threshold value for the continuum retrieval. Since covariance is the square of the uncertainty, this reduces by a factor 4 for every `bit' selected, so for 5 bits, 9.8E-04 = 1/1024.
- Min,Max Trials for each new MW? [<CR>= 30 99]:
-
The minimum and maximum number of trial MWs that will be grown before a
new MW selected (see Growth).
After the minimum number of trials, the best of any acceptable microwindow
found so far will be selected. If no acceptable MWs have been found,
selection continues until one is found, or the maximum number
of trials is reached at which point the selection ceases with a message
I-MWMAKE: No more MWs found
Increasing the minimum number of trials increases the likelihood of a better MW being found at each stage, but also increases the program time. 99 is the maximum that fits the 2-digit representation of trial# in the diagnostics file. - Example Responses
- 10 10
- Fix 10 trials for each new microwindow.
- <CR> (Default - set in mwmake.for)
- min 30 max 99 trials for each new microwindow
- Exclude database
- Name of a microwindow database file (IMK-format) containing a list of microwindows whose spectral ranges will be excluded to prevent new microwindows having spectral overlaps.
- Example Responses
- MW_CH4_103.DAT
- Exclude spectral grid points covered by any microwindow within database MW_CH4_103.DAT
- <CR> (Default - set in mwmprv.for)
- No excluded spectral points
- Include database
- Name of an microwindow database file (IMK-format)
containing a list of microwindows to be included in the retrieval prior to
the addition of new microwindows.
Microwindows from this file will be skipped if any of the following
apply:
- MW not completely within selected wavenumber range; or
- Any wavenumber grid pt within the MW already flagged to be `excluded' (eg if already included in another MW); or
- MW altitude range does not overlap the required range; or
- MW altitude range only overlaps at 'missing' altitudes; or
- No 'T' masks in any included MW altitudes
- Example Responses
- MW_CH4_103.DAT
- Include microwindows within database MW_CH4_103.DAT
- <CR> (Default - set in mwmprv.for)
- No previous microwindows used.
-
[Prompt only if an `included database' is specified in the previous response.
There may a slight pause at this
point as the entire file is scanned to determine the highest microwindow#
currently used]
- Sort database (Y/N)? [<CR>=N]:
- An 'included' database may be 'sorted', in which case microwindows from that database will be selected in sequence according to improvement in the figure of merit. If the database is not sorted, the microwindows are included in the sequence in which they occur in the file.
- Note: The previously specified termination criteria generally apply during the inclusion of predefined MWs as well as when creating new MWs. However, if predefined MWs are to be included without sorting, the 'DMrt' criterion is temporarily suspended since the MWs in the file are assume to occur in a random sequence (and therefore some may temporarily decrease the figure of merit depending on the order in which they are incorporated).
- Example Responses
- y (case insensitive)
- Sort microwindows in `included database'
- <CR> (Default - set in mwmprv.for)
- Use microwindows in `included database' in the order in which they occur
- New MW#i from i=? (0=none) [<CR>=0001]:
['0001' in the prompt message will be replaced by N+1 if an `included database' containing microwindows up to number #N has been specified] - Microwindows are identified by labels such as 'CH4_0015' and the user can specify the starting number for the sequence of new microwindows to be created.
- Note: new microwindows will not be created if the termination criteria are met while the `include database' is being read.
- Example Responses
- 0
- Create no new microwindows (i.e. simply evaluate and/or sort microwindows supplied via an `include database')
- 21
- Create new microwindows starting from #0021
- <CR> (Default - set in mwmprv.for)
- Create new microwindows starting from number given in prompt message
5. Example Driver File
Below is an example of a driver file for MWMAKE. Record starting with '!' signify a comment. The first record *must* be a comment, but after that they are optional and may be inserted anywhere (here, they are used to clarify the sequence of prompts requiring each response). Note that responses given to `List of missing altitudes' and `Include Database' determine what the next prompt will be.! MWMAKE Driver File ! Target species: ch4 ! Atmosphere: [day,ngt,sum,win,equ,glo,glw]: glw ! Temp Accuracy requirement? [ =3.0K]: ! Waveno.range or bands [ =all]: ab c d ! Merit(0:7), CPU Cost(0:9) and Mask(0:3) Fns? [ = 0 0 0]: 0 0 3 ! List of retrieval altitudes [km] [ =6,9,..68]: ! List of missing altitudes [km] [ =none]: ! NB: If any missing altitude specified, next entry should be response to ! prompt for missing bands ! Termination criteria Npt,NMW,DMrt,ReqF? [ = 5000 10 0.0 0.0]: ! Minimum ctm info/level [ =0 bits]: 5 ! Min,Max Trials for each new MW? [ = 30 99]: ! Exclude database: ! Include database: MW_CH4_103.DAT ! sort (only applies if an Include Database has been specified previously) N ! New MW#i from i=? (0=none) [ =0001]: 26 ! End
6. Output Files
A run of MWMAKE will produce the following output files. rtv is the retrieved species (lower case). If no missing altitudes are selected, then ijk are the selected Merit/CPU/Growth parameters. If missing altitudes are selected, then i indicates the missing band (i=0 all bands, i=1 A band, i=2 AB band etc) and jk becomes the (highest) missing altitude [km] (e.g. jk=42 if both 39 and 42 km specified as missing).- rtv_ijk.dia
- Diagnostics file. Opened at start of program. Regularly updated.
- rtv_ijk.err
- Error analysis. Opened at start of program.
Updated after each microwindow.
- rtv_ijk.lst
- Microwindow listing. Simplified list of microwindows and absorbers for each. Opened at start of program. Updated after each microwindow.
- rtv_ijk.tmp
- Temporary file for creating microwindow file. Opened at start of program. Rewritten prior to production of each microwindow file, and can be deleted upon completion.
- RRRRnnnn.imk
- Microwindow file (for new microwindows only), RRRR is the upper-case version of rtv padded to 4-characters with underscores nnnn is the microwindow number. Created as each new microwindow is found.
- OM_RRRRijk.DAT
- Occupation matrix. Opened at start of program. Data written at end of program.
Diagnostics File
Diagnostics of the MW selection/creation process. Types of record:- Diagnostics Header
- 2 records at start of each file
giving general info on this particular run, e.g.
! Microwindow diag. 1020.200:2409.800 MRT=1 CPU=0 MSK=3 glw MWMAKE v.01MAR02 ! Sweep range= 6.0:68.0 Missing band=none Missing alt=none
- Description
- 1020.200:2409.800
- Lower and Upper Wavenumber limits (cm-1) of search range (i.e., ab, b, c and d bands here)
- MRT=1 CPU=0 MSK=3
- Selected Merit Fn#1, CPU Cost Fn#0, MW Growth Algorithm#3.
- glw
- atmosphere(s) used for selection
- MWMAKE v.01MAR02
- Version of MWMAKE used
- Sweep range= 6.0:68.0
- Spectra cover tangent altitudes from 6-68 km
- Missing band=none
- No missing bands
- Missing alt=none
- No missing altitudes i.e., all 17 spectra from 6 to 68 km are available.
- Microwindow Diagnostics
- 6 records for each microwindow included or created, e.g.
CH4_0022 1607.750 1610.750 15 60 1573 918 2308 1225 19.802 4.405 12.036 34.4 38.2 41.0 35.5 32.3 32.0 36.0 42.7 41.9 40.6 45.1 49.6 48.3 59.8 86.7100.0 44.6 41.9 43.1 26.4 19.4 16.1 14.7 15.8 12.3 14.4 20.8 14.9 12.0 20.2 80.4 0.0 56.3 56.7 59.5 44.2 37.7 35.8 38.9 45.6 43.7 43.1 49.6 51.7 49.8 63.2118.2100.0
- Description
- CH4_0022
- Microwindow label (RRRRnnnn format).
- 1607.750 1610.750
- Lower, Upper wavenumber boundaries for microwindow (3cm-1 is the maximum width, =121 spectral grid points @ 0.025cm-1 spacing)
- 15 60
- Lower, Upper altitude boundaries for microwindow (=13 altitudes)
- 1573 918
- 1573 points in this microwindow (=121 spectral grid * 13 altitude grid) of which 918 points are masked `FALSE'
- 2308 1225
- Cumulative totals for number of points and number of F masks so far for this and any preceding microwindows.
- 19.802
- Cumulative Shannon information content for the retrieval (i.e. evaluated using Merit Fn#4, irrespective of merit function selected for this run.)
- 4.405
- Figure of Merit increment for this microwindow, evaluated according to the selected Merit Fn (#1 in this case)
- 12.036
- Cumulative Figure of Merit for the current selection.
- 34.4 38.2 41.0 ...
- Random Error component of current retrieval for lowest 16 altitudes (i.e. 34.4% uncertainty at 6km, 38.2% at 9km, etc)
- 44.6 41.9 43.1 ...
- Systematic Error component at same altitudes
- 56.3 56.7 59.5 ...
- Total Error component at same altitudes (= Sqrt (Random^2 + Systematic^2) )
- Trial Diagnostics
- One record generated for each trial microwindow, e.g.
Trial# 5 Wno=1305.500[1305.750]1306.150 Alt=52[60]68 Npt= 81 Mer= 2.544*
- Description
- Trial# 5
- 5th trial microwindow to be grown
- Wno=1305.500[1305.750]1306.150
- Lower boundary, [Starting point], Upper boundary in the wavenumber domain.
- Alt=52[60]68
- Lower boundary, [Starting point], Upper boundary in the tangent height domain
- Npt=81
- Number of points in the microwindow.
- Mer= 2.544
- Increment in the Figure of Merit if this microwindow is used.
- An additional character may be present at the end of the record:
- * = best microwindow so far (as in example)
- - = MW excluded since it fails to meet the minimum spectral width requirement of 2 grid points
- c = MW excluded since it fails to meet the minimum continuum information requirement
- Mask Optimisation Diagnostics
- 2 Records summarising start and end of mask optimisation
(only if MW Growth Algoritms #2 or #3 selected), e.g.
I-OPTMSK: Iter# 0 Merit: 1.642324 NMask: 500 I-OPTMSK: Iter# 5 Merit: 1.655388 NMask: 495
- Description
- Iter# 0
- Iteration number, 0 being the state prior to any toggling. The logical status of one mask is changed each iteration, so in this case there were 5 mask changes. It is possible for the status of a mask is changed from F to T in one iteration and then back to F in a subsequent iteration, although not the next iteration).
- Merit: 1.642324
- Value of the Figure of Merit increment upon completion of iteration.
- NMask: 500
- Number of False masks in microwindow. In this example, the toggling set 5 points from F to T. Since there were 5 iterations it follows that no points were set from T to F.
- Termination Diagnostics
- Record indicating the reason why the selection terminated, e.g.
I-MWMWRT: Terminated due to No.MWs = limit ( 5)
Error Analysis File
The retrieval error and its components after each MW is incorporated. Types of record:- Error Analysis Header
- 2 records at start of file, containing same information as
Diagnostics Header, e.g.
! MWMAKE error analysis 1020.200:2409.800 MRT=1 CPU=0 MSK=3 glw MWMAKE v.01MAR02 ! Sweep range= 6.0:68.0 Missing band=none Missing alt=none
- Array size information
- 1 record containing the number of retrieved target parameters and
number of profile levels, e.g.
3 17 - Description
- 3
- Number of target parameters (pT counts as two)
- 17
- Number of profile levels
- List of target parameters
- 1 record containing the list of retrieved species, e.g.
tem pre h2o
- List of retrieval altitudes
- 1 or more records containing list of retrieval
altitudes, 16 altitudes per record, e.g.
6.0 9.0 12.0 15.0 18.0 21.0 24.0 27.0 30.0 33.0 36.0 39.0 42.0 47.0 52.0 60.0 68.0
- Error Profiles
- 2 or more records per error profile.
The first 3 sets of records are the Random, Systematic and Total Error
profiles, as in the Microwindow Diagnostics but with the
first record giving the title of each and subsequent record(s) containing
error values for all retrieval altitudes (not just the lowest 16).
For joint retrievals the first row(s) are for the first listed retrieval
parameter, followed by the next parameters starting on a new row etc.
Then sets of records for each significant systematic error component, e.g.
hitran 6.5 13.4 5.7 4.7 3.2 4.6 3.4 1.8 1.2 1.2 1.4 1.2 1.2 2.2 2.5 0.0 0.0
- Description
- hitran
- Error source (see Spectra for full list).
- 6.5 13.4 5.7 ...
- HITRAN database uncertainties contribute 6.5% error at the lowest profile level, 13.4% at the next level etc. Errors in a temperature retrieval are written to 2 decimal places and expressed in K.
Microwindow Listing
List of microwindows in selection/creation sequence.- Microwindow Listing Header
- 2 records at the start of file, containing same information as the
Diagnostics Header, e.g.
! Microwindow list 1020.200:2409.800 MRT=1 CPU=0 MSK=3 glw MWMAKE v.01MAR02 ! Sweep range= 6.0:68.0 Missing band=none Missing alt=none
- Microwindow Information
- One record for each microwindow, fully
defining microwindow apart from mask information, e.g.
3 CH4_0022 1607.750 1610.750 15 60 6 1 3 4 10 8 11
- Description
- 3
- Number of this microwindow in selection/creation sequence
- CH4_0022
- Label, in RRRRnnnn format.
- 1607.750 1610.750
- Lower and upper wavenumber [cm-1] boundaries
- 15 60
- Lower and upper tangent altitude [km] boundaries
- 6 1 3 4 10 8 11
- List of HITRAN/RFM codes for each absorber. Target species first (6=CH4, with 2 = CO2 for temperature microwindows), but order of remainder is not significant.
Occupation Matrix
This file contains a summary of the microwindows and altitudes used for a particular selection. The format of occupation matrices is defined in [2], and the following Field# and Descriptions are taken from that document. Note that '#' indicates an optional comment record, but those generated by MWMAKE are listed below as they sometimes contain additional information not specified in the file format. A typical file is constructed as follows- Field#0 Common File Header
- General header for all ESA files, containing the generation time
and a comment describing the version. For files generated by
MWMAKE the comment
contains the same information as the first record of the
Diagnostics Header), e.g.
27-MAR-2001 16:06:13.046000 # Oxf Occupation matrix 1020.200:2409.800 MRT=1 CPU=0 MSK=3 glw MWMAKE v.01MAR02
- Field#1 Label of Occupation Matrix
- OM Labels are of the form OM_RRRRijk
(where OM_RRRRijk.DAT is the file name)
# OM label: OM_CH4_023
- Notes
- RRRR is the retrieved species, upper case, padded to 4 characters with trailing underscores (`_').
- ijk will either be the selection criteria, or will represent the specified missing bands and altitudes. E.g, '023' could either indicate Merit Fn#0, CPU Fn#2 and Growth Algorithm#3, or that this OM was selected for all bands missing ('0') at 23 km ('23') altitude.
- [Comment] Figure of Merit
- Figure of Merit
for this particular selection, used to compare different OMs. Note that this
is inserted here as a comment as it is not part of the defined file format, e.g.
# Figure of Merit= 56.00363
- Field#2 Number of Altitudes (Rows) in this Matrix
- Number of tangent altitudes for spectra, e.g,
# No of altitudes: 16
- Field#3 Altitude Bands within which this Occupation Matrix is Valid
- Lower and Upper altitudes for each spectrum, assuming 1.5 km tolerance.
For spectra nominally at 8 - 53km, this gives 16 records (in descending altitude)
# Altitude bands: 51.50 54.50 48.50 51.50 .... 6.50 9.50 - Field#4 Number of Microwindows (columns) in this Occupation Matrix
- Number of MWs selected, e.g.,
# Number of MWs: 10
- Field#5 Labels of Microwindows
- List of MW Labels, 7 per record, sorted by increasing wavenumber e.g.,
CH4_0006 CH4_0004 CH4_0007 CH4_0001 CH4_0008 CH4_0009 CH4_0005 CH4_0002 CH4_0003 CH4_0010
- Field#6 Occupation Matrix
- The occupation matrix itself, one column per microwindow in sequence
shown by preceding list of labels,
one row per tangent altitude (descending order)
as listed in the altitude bands. A figure
'1' indicates MW/Altitude is used, '0'
indicates that it is not,
e.g.
# Occupation matrix: #B B B B B B B B B B 1 1 0 1 0 1 0 1 1 0 1 1 0 1 0 1 1 1 1 0 ... 1 1 1 1 1 1 1 1 0 1
- Notes
- The second comment line (#B B ...) indicates the MIPAS band containing each microwindow (all B-band microwindows in this case).
- There should be at least one '1' in each column, indicating that each microwindow uses data from at least one tangent height
- If any sweeps (and bands) are designated as missing, there will be '0' for all entries on the corresponding row (and band)
- If any rows of the OM are entirely zero, the retrieval level will be set to zero for that level as well.
- Field#7 Logical Retrieval Vector
- Set of flags indicating whether a retrieval is attempted ('1')
or not ('0') at each sweep altitude, e.g., if the retrieval is
not attempted at the highest altitude,
# Retrieval vector: 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
- Notes
- If the retrieval does not meet the requirement accuracy at a particular altitude, the retrieval vector is set to 0
- If a retrieval vector is set to 0 at either end of the profile, all corresponding tangent altitude row elements of the occupation matrix also set to 0. For pT retrievals this also applies if an internal retrieval vector element is set to 0.
- Field#8 Number of Sweeps for which Parameters will be Fitted
- No of 1's in the retrieval vector, e.g.,
# No. sweeps for which profile fitted 16
7. Figure of Merit
The Figure of Merit H, measured in `bits', represents some form of the information content of a set of measurements. It is defined as:This value is determined within the function fmerit.for
Merit Functions
MWMAKE uses a merit function with the general form
- Weighting Factors
- 0 or 4: Ft=1 Fs=0 Fr=0 (select according to total error only)
- 1 or 5: Ft=1/2 Fs=1 Fr=0 (bias towards low sys errors)
- 2 or 6: Ft=1 Fs=0 Fr=1 (bias towards improvements at levels where requirement not met)
- 3 or 7: Ft=1/3 Fs=2/3 Fr=1 (bias towards low sys errors & where req.not met)
- Determinant Operator
- 0-3 D based only on product of diagonals
- 4-7 D based on determinant of full covariance matrix
CPU Cost Function
This represents a modification to the Figure of Merit expressing the penalty associated with CPU cost, i.e., how much improvement in the retrieval can be traded for faster processing time.The CPU time T required to perform a retrieval is (somewhat arbitrarily) represented by
The user-selected value of the CPU cost-function j (an integer in the range 0:9) determines how this time T is converted to CPU Cost C according to
For existing microwindows which have associated SVD-decomposed look-up tables (so that the total number of singular values, Nsv, is written in the database) there are additional CPU-cost functions which may be used to sort a database. These are used for values of j = A, B, C ... F (i.e. 10-15 in hexadecimal notation). In these cases, the CPU cost is modified so that Npt is now scaled by Nsv/100:
8. Growth Algorithms
A microwindow size is defined by its dimensions w*a (w = number of spectral grid points, a = number of altitude grid points), with w = a = 1 initially. It is also possible to have individual measurements excluded from within the microwindow by setting a `False' mask but the CPU cost estimates are based only on N = w*a (this is because it is not practical to exclude masked points from the forward model calculations, which is the time-consuming part, they are only removed in the matrix operations associated with updating the retrieval estimate).There are two methods for growing microwindows
- Edgewise: increase dimension w or a by one so as to include all points along whichever boundary gives the most improvement. This scheme also tests increments of 2 (i.e., adding two rows or two columns at a time) to allow for more expansion possibilites.
- Pointwise: add single points around perimeter according to which gives the
most improvement, and once no further points are found increment w
and/or a to represent the new boundaries and repeat.
Apart from lack of improvement, growth can also be truncated by specified altitude/wavenumber boundaries, or by a maximum width requirement of 3cm-1 (w=121 spectral points @ 0.025cm-1 resolution).
Note that the Edgewise algorithm will only produce contiguous microwindows (all points set `True', meaning `use all points withing boundaries'), while the Pointwise algorithm will generally produce perforated algorithms by choosing to omit certain points (set `False').
For microwindows grown pointwise, there are two further steps which can be applied to optimise the mask (applied to the finally selected microwindow only:
- Mask optimisation
- `Toggle' each point within the microwindow (i.e. set `True' to `False' or vice-versa), save the new status of whichever point gives the most improvement and continue until toggling each point reduces the quality of the retrieval.
- Mask uniformity
- An additional penalty can be imposed to discourage fragmented mask patterns, i.e., encourage solitary `True' points surrounded by `False' points to be set `False' and vice-versa. A larger penalty is applied for non-uniformity in the altitude domain than in the spectral domain.
(v14MAY02 onwards) Remask: for included microwindows, there is a further option which is to reapply mask optimisation plus uniformity. This creates new microwindows but constrained to have the same altitude/wavenumber boundaries as the old microwindow. Any subsequent microwindows which are grown are then generated pointwise, also applying mask optimisation plus mask uniformity.
The particular growth algorithm used is determined by a user-supplied Mask Function, a parameter in the range 0-3.
- 0: Edgewise (i.e. contiguous microwindows only)
- 1: Pointwise
- 2: Pointwise plus mask optimisation
- 3: Pointwise plus mask optimisation plus mask uniformity
- 4: Remask included MWs, then as #3 for any additional microwindows
9. Termination Criteria
Microwindow selection, whether by growing new microwindows or using an existing database, will terminate when any one of four criteria are met.- Total Number of Points Npt
- The total number of points selected (ie including those which are "mask ed") exceeds the specified number.
- Disabled by setting a large number, eg 999999.
- Total Number of Microwindows NMW
- The total number of microwindows found/used.
- Disabled by setting a large number, eg 9999.
- Minimum increments in FOM DMrt No microwindow can be found that increases the FOM by at least this amount
- Note: this test is switched off while an existing set of microwindows is `included' without being sorted since there is no guarantee that the microwindows are stored in the database in a sequence that will give a monotonically increasing FOM, nor that the FOM used to create the database is the same as currently being used.
- Disabled by setting a large negative number, eg -1.0E6 (although you will almost certainly want to retain a minimum value of 0.0)
- Requirement accuracy factor ReqF
- Factor * Requirement accuracy which has to be met at all altitudes. Eg, if the requirement profile is 25%, and a figure 0.6 is supplied for this parameter, the program will terminate when the total retrieval error is 15% or less at all levels.
- Note that this factor allows different values of the requirement accuracy to be specified for the Merit Function and to terminate the selection.
- Disabled by setting a value 0 (since zero error cannot be achieved).
10. Microwindow Database File
The ESA format for a microwindow database is defined in [1] and [2]. When MWMAKE creates new microwindows it writes each one as its own database file with a name RRRRnnnn.imk where RRRRnnnn is the Microwindow Label, eg O3__0002. This is the only file in which mask information is stored. For microwindows without masks (i.e. generated using Mask Fn#0), the microwindow is fully specified, and probably more conveniently read, from the Microwindow Listing.When sets of microwindows are to be `excluded' or `included', these have to be contained in a single database file (use imkmrg.for).
11. Appendix: Full List of Input Spectra
This is the list of sample filenames of spectra avaialable for use with MWMAKE. Spectra exist for all 5 atmospheres, for all 5 bands except where otherwise stated, and for all 17 altitudes in the nominal scan pattern:- 6km, 9km, ... 42 km @ 3km intervals, 47, 52, 60, and 68km.
Jacobians
Jacobian spectra represent the difference between the nominal radiance and the radiance if the target quantity is perturbed at the tangent level. Perturbation sizes are 1K for temperature, 1% for VMR or pressure, 1e-4/km extinction for continuum.- ptb_b06000.bin_jac_day_ch4 (AB, B, C, D)
- ptb_a06000.bin_jac_day_h2o
- ptb_a06000.bin_jac_day_hno3 (A, AB, B, C)
- ptb_a06000.bin_jac_day_n2o
- ptb_a06000.bin_jac_day_no2 (A, B, C)
- ptb_a06000.bin_jac_day_o3
- ptb_a06000.bin_jac_day_pre
- ptb_a06000.bin_jac_day_tem
- ptb_a06000.bin_jac_day_ctm
Composition Uncertainties
These represent the difference between spectra calculated for the nominal atmosphere and an atmosphere where the constitutent profile is perturbed by its estimated climatological variability (+1 sigma).- ptb_a06000.bin_day_var_c2h2 (A)
- ptb_a06000.bin_day_var_c2h6 (A)
- ptb_a06000.bin_day_var_ccl4 (A)
- ptb_b06000.bin_day_var_ch4 (AB, B, C, D)
- ptb_a06000.bin_day_var_clo (A)
- ptb_a06000.bin_day_var_clono2 (A, B, C)
- ptb_a06000.bin_day_var_co (D)
- ptb_a06000.bin_day_var_co2 (A, AB, B, D)
- ptb_a06000.bin_day_var_cof2 (A, B, D)
- ptb_a06000.bin_day_var_f11 (A, AB)
- ptb_a06000.bin_day_var_f12 (A, AB)
- ptb_b06000.bin_day_var_f14 (B)
- ptb_a06000.bin_day_var_f22 (A, AB, B)
- ptb_a06000.bin_day_var_h2o
- ptb_a06000.bin_day_var_hcn (A, B)
- ptb_a06000.bin_day_var_hno3 (A, AB, B, C)
- ptb_a06000.bin_day_var_hno4 (A)
- ptb_b06000.bin_day_var_hocl (B)
- ptb_a06000.bin_day_var_n2o
- ptb_a06000.bin_day_var_n2o5 (A, B, C)
- ptb_a06000.bin_day_var_nh3 (A, AB, C, D)
- ptb_c06000.bin_day_var_no (C, D)
- ptb_a06000.bin_day_var_no2 (A, B, C)
- ptb_a06000.bin_day_var_o3
- ptb_a06000.bin_day_var_ocs (A, D)
- ptb_a06000.bin_day_var_sf6 (A)
- ptb_a06000.bin_day_var_so2 (AB, B)
Other Error Sources
- ptb_a06000.bin_day_co2mix (A, D) Ignoring CO2 line-mixing
- ptb_a06000.bin_day_ctmerr Uncertainty in CO2, H2O, O2, N2 continuum absorption (25%)
- ptb_a06000.bin_day_gain Uncertainty in radiometric gain calibration (1%, i.e. these are 1% of the nominal radiance spectrum)
- ptb_a06000.bin_day_gra Ignoring horizontal gradients (temperature gradient of 1K/100km)
- ptb_a06000.bin_day_hitran Uncertainties in HITRAN database (not including uncertainties in absolute line strength)
- ptb_a06000.bin_day_ils Uncertainties in ILS
- ptb_a06000.bin_day_nonlte Ignoring non-LTE effects
- ptb_a06000.bin_day_shift Uncertainty in spectral calibration (0.001cm-1)
Errors due to Gain, ILS and HITRAN uncertainties are assumed uncorrelated between microwindows. Errors due to temperature and pointing are assumed uncorrelated between tangent altitudes.
12. Additional Software
- rfmrd.pro
- IDL procedure to read binary/ASCII spectra
- binasc.f
- Fortran program to convert spectra binary-ASCII or ASCII-binary
- imkmrg.f
- Fortran program to merge RRRRnnnn.imk IMK-format database files produced for each MW into a single database file suitable for input as excluded or included.
- date_and_time.for
- Fortran subroutine for Sun systems replacing FORTRAN 90 function.
13. References
-
The Revised Microwindow Database Format
Tech Note PO-TN-IMK-GS-0002 (April, 1999) - ASCII Input Data Interface Control Document (DD53) PO-IF-DOG-GS-0002 (April, 1999)
- Dudhia, A., V. L. Jay and C. D. Rodgers Microwindow Selection for High-Spectral-Resolution Sounders App. Optics, 41, 3665, 2002
Atmospheric, Oceanic and Planetary Physics Home Page