L2 User Manual

05AUG24 New HIROS L1B format and Python code

The current example consists of a sequence of 4 MORSE runs.

L2 Sequence
Stage	Inputs	Driver File	Outputs	Description
1	hsdi.l1c	morse_hsdi_pt.drv	hsdi_pt.rtv	HSDI pT-pointing retrieval
			hsdi_pt.orb
2	hsdi.l1c	morse_hsdi_aer01.drv	hsdi_aer01.rtv	HSDI_01 (802nm) aerosol extinction
	hsdi_pt.orb
3	hiros.l1c	morse_hiros_pt.drv	hiros_pt.rtv	HIROS pT-pointing-H2O retrieval
	hsdi_pt.orb		hiros_pt.orb
4	hiros.l1c	morse_hiros_ch4.drv	hiros_ch4.rtv	HIROS CH4 retrieval, with Averaging Kernel
	hiros_pt.orb		hiros_ch4.avk

(Note that stages 2,3 could be performed in parallel/reverse order).

Pre-Requisites

• Generic FORTRAN compiler, eg GNU Fortran
• Either: IDL (Interactive Data Language), from NV5,
  Or: Python, from python.org
• MORSE source code, from [morse_v31JUL24.tar.gz]
• HITRAN data in binary form, see [HITRAN Data]
• Other L2 modules and files: l2.tar.gz

1. Unpack and compile the MORSE program, see the [MORSE Web-page] for instructions.
2. Unpack the L2 tar file, eg with linux

   tar -xzf l2.tar.gz

   which will expand the contents into a sub-directory (ie folder) ./l2 containing files and software.

L2 Pre-processing IDL Modules
Type	HSDI	HIROS	Description
Processing	l1c_hsdi.pro	l1c_hiros.pro	Top level L1C Pre-processors
Auxiliary	read_l1b_hsdi.pro	read_l1b_hiros.pro	Functions to read L1B files
Auxiliary	ymd.pro		Convert Julian Day# to YYYYMMDD
Auxiliary	hms.pro		Convert Milliseconds to HHMMSS
L2 Post-processing IDL Modules
Type	HSDI	HIROS	Description
Processing	make_l2.pro		Convert MORSE .rtv output to netCDF
Auxiliary	read_rtv.pro		Function to read MORSE .rtv or .orb file
Auxiliary	read_avk.pro		Function to read MORSE .avk file
Auxiliary	read_l2.pro		Function to read MORSE L2 netCDF file

Python Modules

L2 Pre-processing IDL Modules
Type	HSDI	HIROS	Description
Processing	l1c_hsdi.py	l1c_hiros.py	Top level L1C Pre-processors
Auxiliary	11b_hsdi_class.py	l1b_hiros_class.py	Data and Methods for instrument L1B file
L2 Post-processing IDL Modules
Type	HSDI	HIROS	Description
Processing	make_l2.py		Convert MORSE .rtv output to netCDF
Auxiliary	read_rtv.py		Function to read MORSE .rtv or .orb file
Auxiliary	l2_class.py		Data and Methods for L2 netCDF file

Files

Level 2 Processing Files
Type	HSDI	HIROS	Description
Configuration	morse_hsdi_pt.drv [FMT]	morse_hiros_pt.drv [FMT]	MORSE Driver Files
	morse_hsdi_aer01.drv [FMT]	morse_hiros_ch4.drv [FMT]
Input	[FMT]	hiros.l1b [FMT]	L1B files (examples from Instrument Bypass)
Auxiliary	hgt_std.atm [FMT]		MORSE internal altitude grid
Auxiliary	equ.atm [FMT]		Climatology profile
Auxiliary	aerosol.atm [FMT]		Aerosol extinction profile
Auxiliary		hiros_mw.lst [FMT]	HIROS microwindow list
Auxiliary		hiros.fov [FMT]	Field-of-View data
Auxiliary	(eg) HSDI_06.grd [FMT]		Irregular spectral grid
Auxiliary	(eg) HSDI_01.srf [FMT]	hiros.ils [FMT]	Spectral convolution data
Auxiliary	hitran_2016.bin [FMT]		HITRAN binary file (not supplied)
Auxiliary	(eg) tab_HSDI_06h2o.asc [FMT]		Look-Up Table of absorption coefficients
Intermediate	hsdi.l1c [FMT]	hiros.l1c [FMT]	L1C Pre-Processor files
Intermediate	hsdi_pt.orb [FMT]	hiros_pt.orb [FMT]	MORSE profile outputs
Diagnostic	morse_hsdi_pt.log [FMT]	morse_hiros_pt.log [FMT]	MORSE log files
Diagnostic	morse_hsdi_aer01.log [FMT]	morse_hiros_ch4.log [FMT]	MORSE log files
Intermediate	hsdi_pt.rtv [FMT]	hiros_pt.rtv [FMT]	MORSE state vector outputs
Intermediate	hsdi_aer01.rtv [FMT]	hiros_ch4.rtv [FMT]	MORSE state vector outputs
Output		hiros_ch4.avk [FMT]	CH4 Averaging Kernel
Output	hsdi_pt.nc [FMT]	hiros_pt.nc [FMT]	L2 outputs
Output	hsdi_aer01.nc [FMT]	hiros_ch4.nc [FMT]	L2 outputs

Configuration

L1C Pre-Processor

The L1C Pre-processors (separate IDL/Python procedures for HSDI and HIROS) are entirely controlled by arguments passed at execution, primarily specifying input (L1B) and output (L1C) filenames, but other options are also available.

L1C Pre Processor Arguments
Argument	Type	Value	Description
L1B_FILE	Positional	String	Name of L1B Input file
L1C_FILE	Positional	String	Name of L1C Output file
ALTMIN	Keyword	Float	Minimum Altitude [km]
ALTMAX	Keyword	Float	Maximum Altitude [km]
CHNLST	Keyword	StrArr	List of selected HSDI channels

Notes

1. IDL/Python procedures can take Positional arguments, which are always mandatory and defined by their position in the argument list (L1B_FILE and L1C_FILE here), and/or Keyword arguments (eg ALTMIN=10.0) which can appear in any order and may or may not be required (here, all Keyword arguments are optional).
2. The default behaviour for L1C is to pass all measurements from the L1B file into the L1C file (with selection of channels and/or altitude range specified in the subsequent MORSE driver file); but it is also possible to restrict the tangent altitude range at the L1C stage and (for HSDI) the set of channels, eg to reduced the L1C file size.
3. For HSDI, ALTMIN and ALTMAX refer to the tangent altitudes of the individual pixels rather than the boresight
4. CHNLST is an array containing a list of required HSDI channel labels, eg. ['HSDI_06', 'HSDI_07']. If absent, all channels are passed.

MORSE

The L2 program, MORSE, is entirely controlled by a Driver File, which is opened in the local directory with the name morse.drv. For this example, four different driver files are supplied, one for each stage in the sequence:

1. morse_hsdi_pt.drv
2. morse_hsdi_aer01.drv
3. morse_hiros_pt.drv
4. morse_hiros_ch4.drv

A detailed description of the format of these driver files can be found on the [MORSE web-page] but for the E2E sample runs used here, a summary of the configuration parameters are as follows.

MORSE Driver File
Section	Field	Description
Mandatory sections, in this order
*HDR	[Text]	Up to 79 characters of descriptive text
*FLG	CTM	Use molecular continua (eg H2O)
	INF	Include Information Content diagnostics in .rtv file output
	RES	Include vertical resolution scale factors in .rtv file output
	MOV	Allow for moving tangent point during scan (HIROS only)
	REX	Use Rayleigh extinction (HSDI only)
*L1C	hsdi.l1c	Name of L1C input file (HSDI)
	hiros.l1c	Name of L1C input file (HIROS)
*ALT	hgt_std.atm	Internal altitude grid
*RTV	11:36	Indices of internal altitude grid for retrieved products (here 10–50 km)
	TEM	Retrieve Temperature profile
	REFPRE	Retrieve Reference Pressure
	POI	Retrieve Pointing correction profile
	H2O, CH4	Retrieve H2O, CH4 profiles (HIROS)
	Aerosol	Retrieve Aerosol Extinction profile (HSDI)
*MIC	hiros_mw.lst	File containing list of microwindows and absorbers (HIROS)
	eg HSDI_06	Filter to be used (HSDI)
	eg o2	Absorber to be included with filter (HSDI)
	altmin=10	(optional) mininum altitude cut-off for filter
	altmax=50	(optional) maximum altitude cut-off for filter
*ATM	equ.atm	Climatological atmospheric profile
	aerosol.atm	Climatological aerosol extinction profile (HSDI)
	hsdi_pt.orb	Input pT-pointing profile from HSDI retrieval
	hiros_pt.orb	Input pT-pointing profile from HIROS retrieval
*SRF	./srf/#MIC.srf	Spectral Response Function files (#MIC is token for filter labels from *MIC section) (HSDI)
*ILS	hiros.ils	Instrument Line Shape file (HIROS)
*FOV	*	δ-fn field-of-view (HSDI)
	hiros.fov	Field-of-view file (HIROS)
Optional sections, in any order
*HIT	hitran_2016.bin	HITRAN binary file
*CNV	CHILIM=1.0	Convergence when Chi-sq ≤ 1
*OUT	eg RTVFIL=hsdi_pt.rtv	Output product types (RTV,ORB,AVK) and filenames
*FIN	0.005	Spectral resolution [cm-1] for forward model calculations (HSDI)
*LUT	./lut/tab_#MIC#GAS.asc	LUT filename template (HSDI)
*GRD	./grd/#MIC.grd	Irreg. grid filename template (HSDI)
*END		End of driver file marker

Most of these parameters are expected to be static, ie not changed between different runs, so the same driver files can be re-used. Even where filenames have to be changed, this can also be performed externally to the driver file via symbolic links. However for certain tests, eg changing the retrieval grid or convergence criteria, a different driver file will be required.

L2 Post-Processor

The post-processor, make_l2, takes just two mandatory arguments: input and output filenames.

make_l2 Postprocessor Arguments
Argument	Type	Value	Description
RTV_FILE	Positional	String	Name of MORSE .rtv Input file
NC_FILE	Positional	String	Name of L2 netCDF Output file

1. Start IDL
   >idl
   After various notices regarding licenses etc this should produce the IDL command prompt 'IDL>'

2. Either run the IDL procedure l1c_hsdi.pro or l1c_hiros.pro, supplying the name of an existing L1B (input) file and specifying the name of the L1C (output) file to be generated.

   These procedures use other IDL modules (in the ./l2 directory) which should be picked up automatically if present in the same directory.

   Simple version for HIROS:

   IDL> l1c_hiros, 'l1b_hiros.nc', 'hiros.l1c'

   More complicated version for HSDI using optional keyword arguments

   IDL> l1c_hsdi, 'l1b_hsdi.nc', 'hsdi.l1c', $ IDL> CHNLST=['HSDI_06','HSDI_07'], ALTMIN=5

   Note the use of the '$' character to continue the list of L1C_HSDI arguments to a second line. In this case, only HSDI channels 06 and 07 are passed to the L1C output file, and only pixel altitudes 5km and above. IDL allows keyword arguments to be supplied in any order, and keywords can be upper or lower case.

   The above should only take a second or two, and recreate files hsdi.1lc, hiros.l1c.

   The Python functions can be run interactively in a similar way, or directly from the terminal (see edited-out examples in the last line of the code).

   python3 hsdi_l1c.py

3. Copy the relevant MORSE driver file to morse.drv, eg
   > cp morse_hsdi_pt.drv morse.drv

4. Run MORSE, for example if the executable is in directory ./morse_source/
   > ./morse_source/morse R-MORSE: Running MORSE v16SEP22 I-MORSE: Processing Pixel# 1 of 1 [...] R-MORSE: Successful completion

   These take of the order of 10s of minutes to run.

   The messages starting 'R-MORSE: ...' and 'I-MORSE: ...' are printed to the terminal by the program. Any fatal error will produce a message starting 'F-MORSE: ...' with some text describing the error, and the file morse.log can also be inspected to determine the cause.

   Depending on the driver file, this should recreate files *.rtv, *.orb, and/or *.avk.

5. The log file morse.log should then be renamed if you want to save it, since it will be overwritten the next time MORSE is run.

6. Start IDL and run the post-processor for each .rtv file
   IDL> make_l2, 'hsdi_pt.rtv', 'l2_hsdi_pt.nc' IDL> make_l2, 'hsdi_aer01.rtv', 'l2_hsdi_aer01.nc' IDL> make_l2, 'hiros_pt.rtv', 'l2_hiros_pt.nc' IDL> make_l2, 'hiros_ch4.rtv', 'l2_hiros_ch4.nc'
   (or similarly with the Python version)