• Introduction
• Outline
• Setting up the DLM code
• The main procedure

During the development of various pieces of IDL code (such as the Mie scattering routines found here) I have had to come to terms with IDL Dynamically Loaded Modules (DLMs). DLMs make use of IDL's own Application Programming Interface API (i.e. the C type definitions, functions and macros which IDL uses to call it's own internal routines) to allow code written in some other programming language to be called from within IDL exactly as if it were a native IDL routine.

So why use DLMs?

The main reason you'd want to call external routines from IDL is speed. A well written piece of C or FORTRAN will pretty much always be a lot faster than the equivalent IDL code. Of course, IDL is a lot more convenient than any low level language for data analysis type tasks: DLMs give you the best of both worlds.

There are different ways of utilising external routines from IDL, in particular CALL_EXTERNAL and LINKIMAGE. I don't have any experience of LINKIMAGE, but I know from personal experience that things can go horribly wrong with CALL_EXTERNAL calls (with the values being based back to IDL occasionally becoming corrupted). DLMs have the advantage of working directly with the internals of IDL, which means they're a breeze to use (once they're written).

IDL is written in C, so IDL DLMs must also be written in C. However code written in other languages can also be used by writing a simple C wrapper function which calls routine in question. This is the approach I have taken with my DLM development, since I have needed to call FORTRAN routines. This page will use the example of the mie_dlm_single routine to illustrate what is involved in programming your own DLMs.

A working DLM consists of two or three components:

• A ASCII module description file (name.dlm) which defines the DLM
• A shared object file (.so on Unix or .dll on MS Windows) which contains the routine itself
• On Windows only, a linker definition file.

The details of the module description file can be found on page 453 of the EDG. The contents of the .dlm file for mie_dlm_single are:
MODULE mie_single
DESCRIPTION Mie code for single particle scattering
VERSION 1.0
SOURCE Gareth Thomas (AOPP)
BUILD_DATE March 2005
PROCEDURE MIE_DLM_SINGLE 2 9 KEYWORDS
So, essentially, the file contains some helpful info on the routine and, on the last line, defines it so IDL knows what to expect: ie. MIE_DLM_SINGLE is defined as a procedure which takes between 2 and 9 arguments plus some keywords. A single DLM can supply more than one procedure or function, and each of these should have a corresponding line in the .dlm file. There are other options which can be included in a PROCEDURE of FUNCTION line, but you almost certainly won't need them (and they're listed in the EDG).

Now, the code itself...

If you look at the mie_dlm_single source code (download) you will see there is a C source file and associated header (mie_dlm_single.c and mie_dlm_single.h) as well as a FORTRAN source file (mieint.f). mieint.f contains a completely standard FORTRAN 77 subroutine, while the C code provides an interface between this and IDL. Now, look at the mie_dlm_single.c file and it's header.

The first part of the code sets up the elements necessary for all DLMs. Firstly we include the IDL API header file "export.h" (which can be found in the "external" subdirectory of the IDL directory). We then define a IDL_SYSFUN_DEF2 structure (assuming the IDL version is 5.3 or newer, otherwise the now obsolete IDL_SYSFUN_DEF structure is defined). This structure is what is used to pass information about the routine to IDL and contains the following elements:

• IDL_SYSRTN_GENERIC funct_addr: the address of the function implementing the routine, in this case (IDL_FUN_RET) mie_dlm_single
• char * name: The name by which the routine is called from with IDL, in this case "MIE_DLM_SINGLE". Note that the name should be in capitals and if the routine is an object method, it should be fully qualified (e.g. "CLASS::METHOD").
• unsigned short arg_min: The minimum number of arguments the routine can be called with, in this case 2.
• unsigned short arg_max: The maximum number of arguments the routine can be called with, in this case 9. If you don't want an upper bound on the number of arguments you can used the predefined value IDL_MAXPARAMS.
• int flags: This is a bit mask by which the following options can be set:
  • IDL_SYSFUN_DEF_F_OBSOLETE: IDL will issue a warning message and set !WARN.OBS_ROUTINE when the routine is called.
  • IDL_SYSFUN_DEF_F_KEYWORDS: The routine accepts keywords.
  • IDL_SYSFUN_DEF_F_METHOD: The routine is an object method.
  In order to specify multiple options, use the bitwise OR operator to combine them. In this case we specify the IDL_SYSFUN_DEF_F_KEYWORDS value.
• void * extra: reserved for RSI use. Set it to "0".

Next there is a function called IDL_Load, which accepts no input parameters and returns an integer. Every DLM must have this the function, which is called when the DLM is first loaded. This function does two things: it calls the IDL_SysRtnAdd function (or the IDL_AddSystemRoutine in IDL < 5.3) and then registers a exit handler, which will perform any final tidying up when IDL shuts down, using the function IDL_ExitRegister. The format of the IDL_SysRtnAdd call is pretty straight forward. The first argument is the function or proceedure structure defined earlier (mie_single_procedures), followed by either TRUE (indicating that the routine is a function) or FALSE (indicating a procedure) and then the macro ARLEN is called (which is defined in mie_dlm_single.h). The IDL_Load function will return IDL_TRUE (= TRUE) if successful, or IDL_FALSE (= FALSE) if it fails.

The final piece of setup is to define the exit handler, which in this case does nothing.

Once the IDL_Load and exit handler functions have been defined we can move onto the piece of code that is actually run each time the DLM is called. In this case it is declared as "extern void IDL_CDECL mie_dlm_single" - as are all IDL DLM procedures. IDL DLM functions are always "extern IDL_VPTR IDL_CDECL". IDL uses the arguments argc, argv and argk to pass arguments and keywords to external routines. argc is then number of arguments being passed, argv is an array of IDL defined pointers (type IDL_VPTR), with one element per argument, and argk contains any keywords being passed.

The mie_dlm_single basically does the following:

1. It checks to see if the keyword DQV exists and loads it into a local variable if it does.
2. It checks the input parameters passed from IDL to ensure that they are of the expected type.
3. Variables of the right type to be passed to the FORTRAN mie scattering sub-routine, mieint, are then defined.
4. The mieint sub-routine is then called and if a particular error occurs the DLM exits gracefully with an error message.
5. The output of mieint is then copied into the argv array, based on the number of parameters the DLM was called with from within IDL. If a given parameter isn't required the corresponding result array is deallocated.
6. Finally, the remaining temporary arrays created during the procedure are deallocated.

The fact that this procedure uses arrays as input and output parameters as well as making use of keywords adds a few complicating factors to the code, as well as revealing some pitfalls for the unwary. We'll deal with the keyword first.

In dealing with the keyword we need to define a new IDL_VPTR array (outargv in this case) which will point to the keyword values. This array needs to have enough elements to hold all of the possible keywords which can be passed to the routine, i.e. 1 in this case. We also define a local variable to hold the values passed by the keyword, Dqvdata. Next we define the IDL_KW_ARR_DESC structure, which we name Dqvdesc. This structure contains the name of the variable which will contain the keyword data, followed by the minimum and maximum number of elements to allow in this keyword. The final element of the structure is always set to 0, but will contain the actual number of elements passed when the code is executed.

Next we have the "keyword" array, "static IDL_KW_PAR kw_pars[]". If a routine acepts multiple keywords this array would contain one entry for each keyword in alphabetical order - in our case we only have one keyword to deal with. The first element of kw_pars is the IDL_KW_FAST_SCAN flag - this ensures that the keyword search is as efficient as possible, which isn't really necessary in our case, but doesn't do any harm. Next comes the keyword descriptions, which contain the following elements:

• The name of the keyword (in capital letters): "DQV"
• The data type of the keyword: IDL_TYP_DOUBLE
• A bit mask which activates this keyword in this routine (in conjunction with IDL_KWGetParams below) - this allows the same IDL_KW_PAR array to be used by multiple routines: 1
• Next comes a bit mask set using predefined flags. In this case the IDL_KW_ARRAY flag indicates that the keyword must be an array
• Next comes a pointer to a C integer dqvexists - if the keyword is specified when the DLM is called, this variable will be set to TRUE.
• The final entry is the IDL_KW_ARR_DESC we defined above.

Next the IDL_KWCleanup function is called with the parameter IDL_KW_MARK. This, paired with a second call to IDL_KWCleanup at the end of the mie_dlm_single function ensures all temporary arrays created by the keyword are deallocated before the routine returns. This is actually only strictly necessary if you are passing a variable of type IDL_STRING, or if you are using the keyword to pass information back to IDL... but once again, it doesn't hurt.

The final line to with detecting and reading a keyword is the call to the IDL_KWGetParams function. This is the thing which actually sorts out the keywords and puts the information in the right local variables. The first 3 arguments are the same as those passed to the main mie_dlm_single function. Next comes the IDL_KW_PAR array and the IDL_VPTR array outargv. The final argument is a mask which is ANDed with the mask in the IDL_KW_PAR array, to define which keywords are active in this routine.

Once the keyword has been dealt with, we move onto checking the type of the other arguments passed to the DLM by the user - if they're not what's expected we return to IDL with a error message. See the EDG for the full set of these macros.

Finally, we can actually started manipulating the arguments passed to the routine!
Local variables are populated from the argv array of IDL_VPTR - see the EDG for a full definition of the elements of a IDL_VPTR. When the argument is an array or vector to be passed back to IDL (i.e. can be passed to the routine undefined) IDL_MakeTempArray or IDL_MakeTempVector functions are used. The first argument for both functions specifies the IDL type of the array. In the case of IDL_MakeTempVector, the next argument is the number of elements in the vector; in the case of IDL_MakeTempArray it is the number of dimensions followed by an array with one element for each dimension of the resulting array. We then have a bit-mask set by flags - here we use the IDL_ARR_INI_ZERO flag, which should be pretty self explanatory. Finally we have the address of the variable which will be passed back to IDL, which is (of course) a IDL_VPTR. These functions return pointers, which we can use to populate with the data to be returned to IDL.

Aside for the test to see if we have the DQV keyword has been set, the next interesting thing is the call to the FORTRAN subroutine, mieint. There are a couple of important points here:

1. When a FORTRAN routine is called from a C based programme a "_" must be appended to the name - something to do with the way FORTRAN compilers store the names of subroutines.
2. ALL variables must be passed as pointers.
3. You have to be very careful that the types of your variables correspond in the C and FORTRAN code. Here is a rough guide:
   • IDL_TYP_DOUBLE = C double = FORTRAN Real * 8
   • C IDL_LONG = FORTRAN Integer * 4
   • C IDL_DCOMPLEX = FORTRAN Complex * 16
   If you are trying to call FORTRAN from C and you're getting segmentation faults/address errors (or whatever you want to call them) check for a type mismatch first!

Once the FORTRAN has done all the hard computational work, we need to pass the output arguments back to IDL and clean up. Since we don't know how many of the arguments the user will specify ahead of time this is a little untidy. Essentially a check is done on the number of arguments passed to the DLM and based on this the output data arrays are either copied to the corresponding element of argv using the IDL_VarCopy function, or are deallocated with the IDL_DELTMP macro.

Finally the IDL_KWCleanup function is called again, this time with the IDL_KW_CLEAN flag and the IDL_DELTMP macro is used to deallocate any remaining temporary arrays. NOTE: Not using IDL_DELTMP on all the temporary arrays which are not copied to an output variable (IDL_VarCopy deletes the temporary variable itself) will not stop you code from working, but you will get an annoying warning message every time the DLM is called:
Temporary variables are still checked out - cleaning up...
So, if you see this message and you're using a DLM, you've not cleaned up your variables properly.

And that's pretty much it. It's all a bit involved, but if you will be calling some piece of computationally expensive code repeditively in IDL, the speed gain will more than make up for the effort of creating a DLM. Have fun!

Maintained by Gareth Thomas