Creating IDL DLMs


During the development of various pieces of IDL code (such as the Mie scattering routines found here) I have had to come to terms with IDL Dynamically Loaded Modules (DLMs). DLMs make use of IDL's own Application Programming Interface API (i.e. the C type definitions, functions and macros which IDL uses to call it's own internal routines) to allow code written in some other programming language to be called from within IDL exactly as if it were a native IDL routine.

So why use DLMs?

The main reason you'd want to call external routines from IDL is speed. A well written piece of C or FORTRAN will pretty much always be a lot faster than the equivalent IDL code. Of course, IDL is a lot more convenient than any low level language for data analysis type tasks: DLMs give you the best of both worlds.

There are different ways of utilising external routines from IDL, in particular CALL_EXTERNAL and LINKIMAGE. I don't have any experience of LINKIMAGE, but I know from personal experience that things can go horribly wrong with CALL_EXTERNAL calls (with the values being based back to IDL occasionally becoming corrupted). DLMs have the advantage of working directly with the internals of IDL, which means they're a breeze to use (once they're written).

IDL is written in C, so IDL DLMs must also be written in C. However code written in other languages can also be used by writing a simple C wrapper function which calls routine in question. This is the approach I have taken with my DLM development, since I have needed to call FORTRAN routines. This page will use the example of the mie_dlm_single routine to illustrate what is involved in programming your own DLMs.

Throughout this document I will be referring to the IDL External Development Guide (EDG) which is included in the IDL online documentation system as a PDF file. In addition Ronn Kling's book "IDL and C, Using DLM’s to extend your IDL code" has been of great help to me in getting DLMs working - if you're going to under take DLM development you should definitely consider buying this book!


A working DLM consists of two or three components:

Since I don't use IDL under Windows if I can help it, I am not going to go into the differences between DLMs on Unix and Windows... especially since I don't know myself.

Setting up the DLM code

The details of the module description file can be found on page 453 of the EDG. The contents of the .dlm file for mie_dlm_single are:
MODULE mie_single
DESCRIPTION Mie code for single particle scattering
SOURCE Gareth Thomas (AOPP)
BUILD_DATE March 2005
So, essentially, the file contains some helpful info on the routine and, on the last line, defines it so IDL knows what to expect: ie. MIE_DLM_SINGLE is defined as a procedure which takes between 2 and 9 arguments plus some keywords. A single DLM can supply more than one procedure or function, and each of these should have a corresponding line in the .dlm file. There are other options which can be included in a PROCEDURE of FUNCTION line, but you almost certainly won't need them (and they're listed in the EDG).

Now, the code itself...

If you look at the mie_dlm_single source code (download) you will see there is a C source file and associated header (mie_dlm_single.c and mie_dlm_single.h) as well as a FORTRAN source file (mieint.f). mieint.f contains a completely standard FORTRAN 77 subroutine, while the C code provides an interface between this and IDL. Now, look at the mie_dlm_single.c file and it's header.

The first part of the code sets up the elements necessary for all DLMs. Firstly we include the IDL API header file "export.h" (which can be found in the "external" subdirectory of the IDL directory). We then define a IDL_SYSFUN_DEF2 structure (assuming the IDL version is 5.3 or newer, otherwise the now obsolete IDL_SYSFUN_DEF structure is defined). This structure is what is used to pass information about the routine to IDL and contains the following elements:

Note that the definition of IDL_SYSFUN_DEF (in the case of IDL older than 5.3) is the same as this, except that it doesn't include the extra parameter. Note also that the definition here has to match the definition in the DLM file, or you'll get this error when you try to run the DLM:
Re-definition of routine ROUTINENAME is incompatible with original.
% Dynamically loadable module failed to load: DLMNAME.
% Execution halted at: $MAIN$

Next there is a function called IDL_Load, which accepts no input parameters and returns an integer. Every DLM must have this the function, which is called when the DLM is first loaded. This function does two things: it calls the IDL_SysRtnAdd function (or the IDL_AddSystemRoutine in IDL < 5.3) and then registers a exit handler, which will perform any final tidying up when IDL shuts down, using the function IDL_ExitRegister. The format of the IDL_SysRtnAdd call is pretty straight forward. The first argument is the function or proceedure structure defined earlier (mie_single_procedures), followed by either TRUE (indicating that the routine is a function) or FALSE (indicating a procedure) and then the macro ARLEN is called (which is defined in mie_dlm_single.h). The IDL_Load function will return IDL_TRUE (= TRUE) if successful, or IDL_FALSE (= FALSE) if it fails.

The final piece of setup is to define the exit handler, which in this case does nothing.

The main procedure

Once the IDL_Load and exit handler functions have been defined we can move onto the piece of code that is actually run each time the DLM is called. In this case it is declared as "extern void IDL_CDECL mie_dlm_single" - as are all IDL DLM procedures. IDL DLM functions are always "extern IDL_VPTR IDL_CDECL". IDL uses the arguments argc, argv and argk to pass arguments and keywords to external routines. argc is then number of arguments being passed, argv is an array of IDL defined pointers (type IDL_VPTR), with one element per argument, and argk contains any keywords being passed.

The mie_dlm_single basically does the following:

  1. It checks to see if the keyword DQV exists and loads it into a local variable if it does.
  2. It checks the input parameters passed from IDL to ensure that they are of the expected type.
  3. Variables of the right type to be passed to the FORTRAN mie scattering sub-routine, mieint, are then defined.
  4. The mieint sub-routine is then called and if a particular error occurs the DLM exits gracefully with an error message.
  5. The output of mieint is then copied into the argv array, based on the number of parameters the DLM was called with from within IDL. If a given parameter isn't required the corresponding result array is deallocated.
  6. Finally, the remaining temporary arrays created during the procedure are deallocated.

The fact that this procedure uses arrays as input and output parameters as well as making use of keywords adds a few complicating factors to the code, as well as revealing some pitfalls for the unwary. We'll deal with the keyword first.

In dealing with the keyword we need to define a new IDL_VPTR array (outargv in this case) which will point to the keyword values. This array needs to have enough elements to hold all of the possible keywords which can be passed to the routine, i.e. 1 in this case. We also define a local variable to hold the values passed by the keyword, Dqvdata. Next we define the IDL_KW_ARR_DESC structure, which we name Dqvdesc. This structure contains the name of the variable which will contain the keyword data, followed by the minimum and maximum number of elements to allow in this keyword. The final element of the structure is always set to 0, but will contain the actual number of elements passed when the code is executed.

Next we have the "keyword" array, "static IDL_KW_PAR kw_pars[]". If a routine acepts multiple keywords this array would contain one entry for each keyword in alphabetical order - in our case we only have one keyword to deal with. The first element of kw_pars is the IDL_KW_FAST_SCAN flag - this ensures that the keyword search is as efficient as possible, which isn't really necessary in our case, but doesn't do any harm. Next comes the keyword descriptions, which contain the following elements:

The final element of of an IDL_KW_PAR array must always be {NULL}. There are, of course, many variations on this theme for generating the elements of the IDL_KW_PAR array depending on the type and purpose of the keywords: See "IDL and C" for a fairly comprehensive list.

Next the IDL_KWCleanup function is called with the parameter IDL_KW_MARK. This, paired with a second call to IDL_KWCleanup at the end of the mie_dlm_single function ensures all temporary arrays created by the keyword are deallocated before the routine returns. This is actually only strictly necessary if you are passing a variable of type IDL_STRING, or if you are using the keyword to pass information back to IDL... but once again, it doesn't hurt.

The final line to with detecting and reading a keyword is the call to the IDL_KWGetParams function. This is the thing which actually sorts out the keywords and puts the information in the right local variables. The first 3 arguments are the same as those passed to the main mie_dlm_single function. Next comes the IDL_KW_PAR array and the IDL_VPTR array outargv. The final argument is a mask which is ANDed with the mask in the IDL_KW_PAR array, to define which keywords are active in this routine.

Once the keyword has been dealt with, we move onto checking the type of the other arguments passed to the DLM by the user - if they're not what's expected we return to IDL with a error message. See the EDG for the full set of these macros.

Finally, we can actually started manipulating the arguments passed to the routine!
Local variables are populated from the argv array of IDL_VPTR - see the EDG for a full definition of the elements of a IDL_VPTR. When the argument is an array or vector to be passed back to IDL (i.e. can be passed to the routine undefined) IDL_MakeTempArray or IDL_MakeTempVector functions are used. The first argument for both functions specifies the IDL type of the array. In the case of IDL_MakeTempVector, the next argument is the number of elements in the vector; in the case of IDL_MakeTempArray it is the number of dimensions followed by an array with one element for each dimension of the resulting array. We then have a bit-mask set by flags - here we use the IDL_ARR_INI_ZERO flag, which should be pretty self explanatory. Finally we have the address of the variable which will be passed back to IDL, which is (of course) a IDL_VPTR. These functions return pointers, which we can use to populate with the data to be returned to IDL.

Aside for the test to see if we have the DQV keyword has been set, the next interesting thing is the call to the FORTRAN subroutine, mieint. There are a couple of important points here:

  1. When a FORTRAN routine is called from a C based programme a "_" must be appended to the name - something to do with the way FORTRAN compilers store the names of subroutines.
  2. ALL variables must be passed as pointers.
  3. You have to be very careful that the types of your variables correspond in the C and FORTRAN code. Here is a rough guide: If you are trying to call FORTRAN from C and you're getting segmentation faults/address errors (or whatever you want to call them) check for a type mismatch first!

Once the FORTRAN has done all the hard computational work, we need to pass the output arguments back to IDL and clean up. Since we don't know how many of the arguments the user will specify ahead of time this is a little untidy. Essentially a check is done on the number of arguments passed to the DLM and based on this the output data arrays are either copied to the corresponding element of argv using the IDL_VarCopy function, or are deallocated with the IDL_DELTMP macro.

Finally the IDL_KWCleanup function is called again, this time with the IDL_KW_CLEAN flag and the IDL_DELTMP macro is used to deallocate any remaining temporary arrays. NOTE: Not using IDL_DELTMP on all the temporary arrays which are not copied to an output variable (IDL_VarCopy deletes the temporary variable itself) will not stop you code from working, but you will get an annoying warning message every time the DLM is called:
Temporary variables are still checked out - cleaning up...
So, if you see this message and you're using a DLM, you've not cleaned up your variables properly.

And that's pretty much it. It's all a bit involved, but if you will be calling some piece of computationally expensive code repeditively in IDL, the speed gain will more than make up for the effort of creating a DLM. Have fun!

Maintained by Gareth Thomas