Oxford MIPAS meeting#44
25 Nov 03

Dear colleagues,
An important reconfiguration of the MIPAS processor has been implemented on 4 November 2003. Relevant information for validation is provided below, anticipating the detailed information in the upcoming MIPAS Monthly Reports. I apologise for the length of the message, but recommend that you take your time to read, as the upgrade will definitely affect your validation results.

MIPAS Processor Reconfiguration on 4 November 2003

1. Background
   The MIPAS instrument has been in orbits since the launch of its Envisat platform on 1 March 2002. Its configuration has been stable since the pre-amplifier gain change on July 2002.

   The analyses performed during the Commissioning Phase have resulted in modifications to the processor and to its configuration. The processor update to version 4.59 has taken place on 23 July 2003. This message describes the subsequent update of the configuration of the data processor, performed on 4 November 2003.

   The combination of updated processor and updated configuration has been a prerequisite for:

   • the release of MIPAS products to all users outside the Calibration/Validation teams,
   • the reprocessing of the MIPAS Validation Data Set (for the upcoming Validation Workshop in May 2004),
   • the reprocessing of the full MIPAS mission.
   An important issue in the configuration update is the resulting large difference between the configuration of the Off-Line (OFL) and the Near-Real-Time (NRT) processors. From this update onwards, the use of OFL products is to be preferred for any analysis that is not limited to short delivery timescales. Also the MIPAS Validation Data Set and the full mission history will both be generated soon using the OFL processor configuration.

   The data from the reconfigured NRT processor are immediately available (from orbit 8760 onwards) to the Cal/Val teams on the usual FTP server. In due time, practical information will be provided to all Cal/Val teams (ACVT and MIPAS QWG) for access to the OFL products via FTP.

   Access to the products from the OFL chain will be activated from orbit 8587 onwards in forward processing (aproximately 2 weeks from now). OFL Validation Data Set reprocessing will be started after completion of the calibration-history consolidation for the Commissioning Phase. Although the new configuration achieves one of the prerequisites for MIPAS product release (NRT and OFL) to non-Cal/Val users, this latter subject is out of the scope of this message, and these users will be informed via appropriate channels.

2. Level 1B processing chain reconfiguration
   As a result of Commissioning-Phase findings and subsequent Quality Working Group investigations, important modifications have been applied to the configuration of the MIPAS level 1B processor.
   1. Update of the instrument characterisation file (MIP_CA1_AX) This auxiliary file has been updated with improved non-linearity characterisation. The original configuration applied identical non-linearity characterisation parameter values to data from forward and reverse interferometer sweeps. This has proven not to be adequate, as oscillations have resulted in the retrieved radiances, propagating into concentration oscillations in the trace gas retrieval. A correction to the reverse coefficients has been derived from the known difference in min/max ADC signal for the two sweep directions and implemented into the new MIP_CA1_AX file. In the mean time an improved non-linearity correction algorithm is being implemented, and will be applied to future half-yearly non-linearity characterisation measurements.
   2. Update of the Level 1 microwindow dictionary (MIP_MW1_AX) Originally the processor had been configured to perform its spectral calibration (once per 4 vertical scans), at an altitude of 47 km. The high noise levels present at this altitude have adversely affected the quality of the spectral calibration. In addition, one microwindow (D_H2Ob at 1840.8049 cm-1) has shown deviations from all other microwindows. To avoid these problems, in the updated configuration the spectral calibration microwindow altitude has been lowered to 32 km, and the inconsistent microwindow has been removed. Microwindows for ILS retrieval remain at 47 km until ILS optimisation has been completed.
   3. Update of the L1 Processing Parameters file (MIP_PS1_AX) As both ILS retrieval and Spectral Calibration are affected by high noise levels, the processor has been reconfigured to perform a higher number of co-additions. The number of co-additions has been increased to 4 for Spectral Calibration, and 10 for ILS retrieval.
   4. Frequently updated auxiliary files In addition to the configuration files discussed before, four auxiliary files are updated frequently:
      • The gain and offset calibration ADFs (MIP_CG1_AX and MIP_CO1_AX) are updated weekly to compensate for contamination effects. All historic ADF files will be regenerated as the updated configuration allows to improve the Gain and Offset characterisation.
      • The spectral calibration ADF (MIP_SP1_AX) will be updated on a weekly basis, prior to the gain calibration, as analysis has demonstrated this to result in improved gain calibration, and improvements for processing of the initial scans of each product. The spectral calibration history will be reconstructed prior to reconstruction of the gain history.
      • The line-of-sight calibration ADF (MIP_CL1_AX) will be updated frequently to correct for deviations in platform attitude. The calibration history for this file has already been reconstructed for historic deviations in pitch. It is to be noted that recent NRT data until 12 November (and potentially early OFL data) will be affected by inadequate LOS calibration.

3. Level 2 Processing reconfiguration
   As a result of Commissioning-Phase findings and subsequent Quality Working Group investigations, important modifications have been applied to the configuration of the MIPAS level-2 processor. Most of these changes have only been applied to the Off-line (OFL) processing chain as the increase in calculation resources that they require are not compatible with processing in Near-Real Time. Only a much smaller subset of modifications has been applied to the Near-Real-Time chain.
   1. Configuration differences between NRT and OFL chains
      MIPAS Near-Real-Time data generation is performed at the processing centres PDHS-K (Kiruna) and PDHS-E (ESRIN, after downlink to the Svallbard ground station). At these centres, predicted orbit information is used for level 1B processing, and meteorological predictions are used to initialise the level 2 processor. For level 2 NRT processing, the level 1 input product is split in 8 segments, which are processed separately and later concatenated. Note that for each of these segments external initialisation information is used for the initial vertical scans. The NRT processing chain benefits from all level 1B configuration changes, but for level 2 only from the MIP_PS2_AX level 2 configuration change.
      The off-line chain consists of consolidation of level 0 products and generation of consolidated level 1B products using restituted orbit information at the LRAC. Off-line level 2 product generation is performed at the D-PAC using meteorological analyses for initialisation and products are processed in their entirety on a single machine, meaning that almost all vertical scans profit from initialisation information generated by preceding scans from the same orbit. All additional differences between the NRT and OFL chains are contained in the auxiliary data files discussed below.
   2. NRT L2 processing parameters file (MIP_PS2_AX)
      This auxiliary file has been updated to provide an initial correction for an interface error between level 1 and level 2 processing. This ADF and the level 2 processor expect the ILS characterisation parameters 'linear shear' and 'linear shear variance' as a vector and scalar respectively, whereas the ILS characterisation actually generates a linear-shear scalar, and a linear-shear variance vector. In all MIPAS data processed prior to the auxiliary file update, the linear shear value has been inadvertently passed to the level 2 processor as linear-shear variance, and vice versa. The current temporary ADF update now provides the processor with a vector of identical shear scalars, and a scalar containing the average of the actual shear-variance vector. This temporary solution already removes most of the ILS deviations found during the Commissioning Phase, but will obviously be superseded by a level 2 implementation correction at the next processor upgrade.
   3. OFL L2 processing parameters file (MIP_PS2_AX)
      This auxiliary file includes the same update as its NRT variant, but in addition:
      • Retrieval altitude ranges have been extended to the range 6 to 68 km (precise ranges vary per species).
      • Extension of all associated look-up tables to ensure full coverage of altitude range (e.g. number of simulated geometries increased from 25 to 30)
      • Stricter convergence criteria have been set for the iteration procedure
   4. OFL Microwindows and Occupation Matrices (MIP_MW2_AX, MIP_OM2_AX) Look-up tables in these files have been recalculated and expanded for consistency with the MIP_PS2_AX update (range extension, convergence criteria), including the PT error propagation matrix. The lookup table expansion includes the cloud-detection microwindow table in the MW2 file. For the expansion to lower altitudes, the values for 11 km have been copied to the fields for lower altitudes as a first approximation. This first approximation leads to inadvertent cloud flagging in conditions of high water vapour concentration at low altitudes, and improved settings are being derived for the future MW2 updates.