Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

Home

Ross Smith

The Biozentrum Micrograph Data Processing Program

Introductory Information

Description:

MDPP is a fully-featured general purpose image processing package originally written to support research in structural biology requiring electron microscopy and image processing. It has focused on the analysis of images using Fourier techniques, particularly periodic images, but has extensive tools for other processing options (e.g. DNA sequencing, point-counting, image quantitation and display). Three-dimensional reconstruction methods are supported, including iterative deconvolution schemes for light micrographs. Color images are supported with both color pallet and RGB options.

The package is easy to install and use, even by the novice user. Extensive documentation is included and is accessible from within the program as on-line HELP and also as HTML documentation. Care has been taken to support interfaces to other image processing packages, e.g. Macintosh applications via the TIFF image format, and other EM-targeted image processing packages such as Spider, Semper, PIC and the MRC package. The basic package is command-line driven, but the user can choose to write sophisticated command scripts and use a menu-driven full-screen or MOTIF interface to call them. Foreground and background operation is supported.

Operating Systems: The current version of MDPP runs on UNIX-type systems such as MacOSX and Linux in a 64-bit environment.  OpenVMS on Itanium and Alpha (and VAX should still work) v7.1 and above should work, but VMS is no longer supported. 

Requirements: MDPP is a "Terminal Application"; it runs from the command-line in a terminal window. Hosts: MacOSX, Linux, HP VAX, AXP and IA64 with OpenVMS. Displays: Any X-Windows/R6 Server (Motif for full functionality under UNIX). On a Mac, start the X11 application and run MDPP from inside an xterm window for best results. Compilers: FORTRAN V7.0 or later is required for OpenVMS systems; Absoft v9.2 or later or Intel 10.1 or later for Mac/Linux.

Distribution: MDPP has been made into a SourceForge project.  

Availability: MDPP is available from the SourceForge project web site. Other sources may become available in the future.

Getting Started: The MDPP code and pre-linked executables for MacOSX and Linux can be downloaded from the "git" archive on SourceForge. Consult the installation information on the "Installation" tab on this page.

Support: The development of this software has been supported in part by the NSF through grant 9410750 to P.R. Smith and through the generous provision of hardware resources by Hewlett-Packard Corp. and Intel Corp, for which we are very grateful.


MDPP Installation & Startup Guide
Basic Installation Information

The primary distribution method for MDPP is through SourceForge. Unsupported versions of MDPP can be downloaded as a set of "tarballs" containing both VMS and UNIX distributions. Support is provided for MacOSX and Linux on 64-bit Intel processors. The distribution contains the executable tasks, and the sources (FORTRAN & C) and the command files to permit the end user to modify MDPP to incorporate alternative output display devices and add new commands to the basic set provided.

All older distributions are now obsolete.

Requirements: MacOSX

FORTRAN is a licensed product for most UNIX platforms. For the Mac under OSX we have used the Intel compiler (use v10.1 or newer) which you will need to buy should you want to modify the code: MDPP will not compile with g77 and I doubt that there is any rationale to try to make it do so.

MacOSX has a full BSD UNIX core. Since the Mac does not use X11 as its display you need to start the X server before you try to run MDPP and display an image: the icon is located by default in the "/Applications/Utilities" directory. When the X server starts it opens an "xterm" window. While it is not necessary, it makes things easier to run MDPP from the xterm rather than another terminal window. I have modified the definition for the xterm in the X11 "Customize Application Menu" to be
xterm -rightbar -sb +sk -sl 5000
This gives you a useful window that can be scrolled, which the default window can't. But with these modest actions MDPP behaves identically to other UNIXes and delivers excellent performance. Note that the MDPP image distributed on the web is built for the latest version of MacOSX on the Intel platform. The code can be recompiled using the Absoft compilers for the G4 and G5 boxes, and should run well.

Requirements: Linux

Linux installation is trivial and essentially identical to the MacOSX installation.  The version included in the distribution was compiled with the Intel compiler and may need access to the FORTRAN shared libraries provided with that compiler to run (which means you'd have to buy a copy).  The Linux version should run identically to the MacOSX version.  Let me know if you try to use it.

Requirements: VMS

The VMS libraries contain code compiled using the new FORTRAN compiler. This compiler is only supported on VMS versions 5.4 and above. To use MDPP linked using the libraries you must have the new Math RunTime library that ships with the V6.0 compiler installed (not the compiler itself: you may not have a license for it). To use the MDPP on lower versions of VMS will require full re-compilation of all sources.

Requirements: Other UNIX

I don't have access to other UNIX boxes, so I can't comment directly on the difficulty of getting MDPP running.  Tru64 UNIX is not  a problem since the sources already contain modifications to support it.  Unfortunately that OS is now obsolete. I'd be surprised if a SUN posed a problem, since the starting point for the OSX version was an obsolete SUN port of MDPP and Macs have the same floating point and "endian" formats. If you want to try a port, contact me to be sure that the version of the source on-line is indeed the most up-to-date.

Requirements: Running MDPP on a remote machine

You can, of course, telnet or ssh into another machine (e.g. mdpp-server.uni.edu) and run MDPP using the machine you are logged in to (e.g a laptop) as an X server to view the images. If you do this you need to do two things:

(1) You must authorize the remote machine to put X-windows on the local machine (your laptop) using the xhost command on your laptop as follows:
my-laptop$ xhost mdpp-server.uni.edu

(2) Before you start MDPP on the remote machine you must direct the X11 windows on that machine to your laptop:
mdpp-server$ setenv DISPLAY my-laptop.uni.edu:0.0

Now you can start MDPP and when you display an image it will appear on your laptop.

The easiest way to run MDPP is from an "xterm" window. If you are running remotely, ssh to the remote machine from an xterm, which guarantees that X11 is awake and functioning when you need it to field incoming images.

The first task: Downloading the distribution

Using SourceForge:

The recommended method is to use "git" to create a clone of the MDPP distribution. Alternatively, the web site can be used to download selected components of the distribution.

Using the tarballs:

The "obsolete" material can be obtained by downloading and expanding the tarballs.

The distribution files should be unloaded using the tar utility, with the command '$VMSTAR -X' (VMS:decompress first) or 'gtar -xz' (UNIX). Issue this command from within the directory where you want to root the MDPP directory tree.

Defining Required Components and Decompressing the Archives

The distribution components will create a small directory-trees that may or may not contain compressed files. These files are of two types: BACKUP (.BCK) savesets and tar archives (.tar). If you are running UNIX only, you can safely delete any compressed BACKUP savesets. VMS users should not delete the tar archive since common files are stored in them: wait until you have identified all the pieces. Decompression of archives has been done with LZDCMP (for files like *.BCK-Z: VMS) and GZIP (for files like *.tar-gz). Yes we plan to elimiate the use of LZDCMP compression: gzip is better and is cross-platform. On UNIX systems rename the files to standard form: e.g. file.tar-gz to file.gtz before attempting to untar with "gtar". Once decompressed, use BACKUP or tar/VMSTAR to restore the directories.

Continuing the Installation

The next task is gaining access to the on-line documentation. Start a WWW browser (e.g. Safari, Netscape, IE ..) and open the Installation page. If you are opening it locally, use the file
MPP:[HTMLDOC]INSTALLATION.HTML for VMS or
$MPP/htmldoc/installation.html for UNIX.
Note that this requires MPP to be defined to point to the location of the root of the MDPP files.

Reminder on Customization

The separate components of MDPP are bulky: delete pieces you don't want. Use TAILOR.COM to eliminate pieces you do not need (for VMS), or remove the directories that are not needed with a command like: 'rm -R ogen' (UNIX).

Code Base

At this point MDPP has a single integrated code base that contains merged VMS and UNIX code. If the VMS or UNIX command files are used to re-compile the code a working version of MDPP should be built. This is achieved by compile-time (#ifdef) statements inside each routine that needs different code for VMS vs. UNIX. Obviously, if you want to modify the code and have it re-compile on all systems you will need to insert appropriate conditional compilation directives and not break the ones that are there.

Sources

The distribution contains MDPP sources that can be used to re-build MDPP or as a source of sample code for extending other systems.  Downloading the "MDPP Sources" distribution should provide all the source we wrote.  Also provided are the distributions for libjpeg and fftw-3.1.2  For UNIX we recommend that these NOT be used,  Rather, you should download the latest versions of these packages from the net and compile and install them in an area pointed to by the logical MDP_UTIL_LIB  If you are using MacOSX the best way to get these is to use the fink package management system to install them, set MDP_UTIL_LIB to point to /sw/lib  and then link MDPP.  On Linux you may need to define your own provide area for MDP_UTIL_LIB, download the distributions and use the "--prefix=" argument for ./configure to point to your area.  For VMS things are a little more complicated.  We provide VMS-customized versions of the jpeg and fftw distributions (identical to the UNIX versions, but not necessarily the absolute newest) with the extra files needed to create the libraries with MMS.  If you're working with VMS download the vms-utils.bck-gz package, gunzip it and restore the backup saveset: you will not need the other distributions.



Step-by-Step Installation of the Software (VMS)

The programs should be loaded into the VMS in the following way:-

  1. IF AND ONLY IF you have a previous version of MDPP then: Rename the root directory [MDPP] of the old version to a new directory, e.g. [MDPPOLD]. The command will look like:
    REN DISK:[000000]MDPP.DIR DISK:[000000]MDPPOLD.DIR

    where "DISK:" is the name of your disk volume. You may need help from the System Manager to do this. This old directory can be deleted when the new version of MDP is running (remember to save the old [MDPP.LOCAL] area if you have made your own local changes to MDP).

  2. Create a new directory using the command:
    $ cre/dir [.mdpp]
    Set your current directory to this new directory and copy each of the distribution components into it. Use GUNZIP to decompress the ".gtz" files then use VMSTAR to unload the contents.

  3. The file "DISK:[MDPP]AA_README.1ST" is obsolete, but you may want to read it either directly on line, or print the file AA_README_1ST.PS (PostScript). Most of the information that used to be in that file is now here. But its still a good read!

  4. Use LZDCMP and GZIP to decompress the files in the directory tree VMSTAR generates for you.

  5. Use BACKUP and VMSTAR to re-create the directory-trees in each of the sub-directories.

  6. Edit and then execute the file MDPP.COM in the [GENERAL] directory. This sets up the pointers to the directories and the symbols to access utility programs, etc. This is not an essential step at this point, but it may significantly simplify getting started. If you make a mistake you can alter MDPP.COM and then re-execute it. The key step at this point is to create the logical MPP: This is essential so as to perform the next step. You do need to customize the MDPP.COM file for your site. This file defines SYMBOLS and LOGICALS for various devices to allow printing of pictures, etc. The file MDPP.COM should contain sufficient information to permit the correct changes to be made. Remember to set the PL0: definition correctly if you want to use ReGIS plotting.

  7. Use the command TAILOR.COM to eliminate MDPP components not needed at your site: e.g. Alpha, Ultrix or VAX/VMS code may not be required.

  8. The BOOKREADER documentation (optional).
    (**WARNING** The BOOKREADER docs are obsolete and are nolonger updated or maintained. Please ensure that you can use the WWW-distributed documentation for future releases.) These can be installed using the command file MPP:[ON-LINE]MDPP-BOOK.COM. This command file adds the MDPP documentation to the system-wide documentation on the VMS machine. To install these on an on-going basis, execute the command procedure above in the SYSTARTUP file: ask the system manager for help. [WARNING: The logical MPP must be defined BEFORE executing the procedure or it will fail and may prematurely terminate the system startup job.]

  9. NOTE that you will probably need to relink the MDPP executable task. However, BEFORE re-linking read the section on customizing MDPP to your site. Use the command file CONFIG.COM found in the directory [GENERAL] to create the command file that you will use to re-link.


Step-by-Step Installation of the Software (UNIX)

The programs should be loaded in the following way:-

  1. IF AND ONLY IF you have a previous version of MDPP then: Rename the root directory mdpp-unix of the old version to a new directory, e.g.
    mdpp-unix-old. The command will look like:
    mv mdpp-unix mdpp-unix-old

    You may need root privs to do this. This old directory can be deleted when the new version of MDP is running (remember to save any customized files if you have made your own local changes to MDP).

  2. Create a new directory using the command:
    $ mkdir ./mdpp

  3. Copy the distribution components into the new directory using the command found at the top of the "code" section of the SourceForge project. MDPP is quite large. Therefore do not create it in a partition with limited space (as /usr often is).

  4. The file "AA_README.1ST" is obsolete, but you might want to give it a quick read either directly on line, or print the file AA_README_1ST.PS (PostScript).

  5. Edit and then 'source' the file ./gen/example-xxx.login This script should be copied by each prospective user of the MDPP and merged with their current .login file. The script sets up the pointers to the directories and the symbols to access utility programs, etc. The key step at this point is to define the environmental MPP to point to the location of the MDPP directory tree. This is essential so that new users can get a copy of the file that has this basic pointer correct. They can then make their own further customizations.

  6. Gain access to the on-line documentation set. Use your WWW browser to open and read the documentation.

  7. NOTE that you will probably need to relink the MDPP executable task. The FORTRAN compiler may be necessary for you to do this.

Customizing the VMS version.

The basic customization you need to do for MDPP is to correctly identify the display type you have available. Before re-linking MDPP run the command file CONFIG.COM which will create the file MDPPB.COM which you should use to rebuild MDPP. The file MDPPBLD.COM file in the [GENERAL] directory is the old build file and should not be used. The default driver is the DECWindows (X11) driver and the only one that will be supported in the future. (Note that drivers for the LEX, AED, GOULD, VWS and METHEUS display systems were written at various times. But all this stuff is now too old, obsolete and anyway probably won't work for anyone any more.)

Users who wish to use large arrays may experience difficulties due to having the wrong limits set for common login parameters. For some of the processing the default limits will be inadequate, which will cause MDPP to crash. We have PGFLQUOTA=100000, BYTLM=64000, FILLM=100, BIOLM=150, DIOLM=150, ASTLM=250, TQELM=10 and ENQLM=2000 for our heaviest user (using AXP, on a VAX smaller values may suffice and on IA64 larger). A large value for PGFLQUOTA is needed to re-link MDPP be cause of its humongous size. Also, the values of the working set sizes may need adjustment to avoid excessive page faulting. We are currently using WSQUO=4000, WSDEF=2000 and WSEXTENT=16384. Very large values for WSxxx parameters are dangerous since they may lock the system: do not exceed the value in the system parameter WSMAX. Discuss the matter with the System Manager who will need to re-set these for you using the AUTHORIZE utility in the event you have a problem. Obviously, the system parameters need to be adequate to allow these values. On the 'good' side, the current releases of OpenVMS come with defaults that appear to be adequate.

You may also need to customize the MDPP.COM file for your site. This file defines SYMBOLS and LOGICALS for various devices to allow printing of pictures, etc. The file MDPP.COM should contain sufficient information to permit the correct changes to be made. Remember to set the PL0: definition correctly if you want to use ReGIS plotting.

The linked version of MDPP in this distribution was built under VMS with Motif V1.2 installed. This means that it will not run if you have a lower version of Motif: if you have a problem, upgrade.

Customizing the UNIX version.

There are really no features that can be customized on the UNIX version right now. What you have is what you get.

Customizing Features.

Including your own commands. You may also need to customize MDPP for your own site to include special routines you need for image output, and also to add new commands.

In the VMS version, to install your own routines, simply install your own local [.LOCAL] directory with its own versions of these routines compiled and loaded into the library file. Then edit the file MDPP.COM and alter the assignment statement which defines "LOCAL_LIB:" to point to the file which you have used to put the new routines into (if different from the one provided). Note that you need to protect yourself and put this out of the way so that this new local area is neither deleted or renamed when you get a new distribution of MDPP. MDPP will rebuild with your routines in place when "MDPPBLD.COM" is executed.

In the UNIX version, to install your own routines simply edit and load the new or modified routines into the library (mdpp.ls or mdppo.ls). Remember to KEEP a copy of the unmodified libraries safely somewhere before messing with them. Re-link using the command provided.

If you need to add a new command you will need to follow a more complicated procedure. This is outlined in the file "[MDPP.DOC]NEW_CMD.DOC" which you should read before attempting to add your own command. Note that this procedure is automated to a degree for VMS, but not for UNIX as yet.

Customizing for Individual Users: The file MDPP.DFL

Basic customization of MDPP for each user at your site can be achieved by means of the file MDPP.DFL, which each user places in his/her directory. This file allows you optionally to define how you wish to obtain display output, and the characteristics of the display device.

The user give MDPP access to this file by defining the logical MDP_USRDFL to point to the file which provides the appropriate default definitions. If he does not have special requirements then he can omit this definition, and MDP will use the MDP_SYSDFL definition to get system-wide definitions.

The MDPP mode switch allows the user to switch between keyboard (0), display screen oriented (1) and terminal screen MENU driven (2) input modes.

In keyboard mode all commands come from the keyboard, display screen options are available, but are not selected automatically, the user must enter the appropriate command to activate screen input.

In display screen oriented mode the input is from the keyboard but the screen options of each command are selected automatically, if there are any. This option has not been fully implemented, and should not be used.

In menu driven mode MDPP is driven entirely from user-defined menu screens. These menus are user programmable, as are their functions, via the PROCEDURE driver. If you wish to use MODE=2 (MENU) you should be aware that you have to write your own menu file and the procedures called by it. If the user wants to use MODE=2, then he must also define the logical MDP_MENU, which must point to his menu file. There is no system-wide default for MDP_MENU, since menus are expected to be user-specific. The user may also wish to write his own MENU help files. The logical MDP_MENU_HELP, for which there is no system-wide default, needs to be defined in order that such help be available. The user's individual definitions should be placed in his LOGIN.COM file, after the call to MDPP.COM.

X-Server Font Problems: The MDPP attempts to get fonts for the display of cursor locations, etc., using a wild-card font request. Some servers (the box driving the screen that you look at) respond with a choice of font that is unappealing or just plain wrong (like: shows odd symbols rather than nice characters). If this happens to you, there are two ways to fix it.

System wide: If you can identify a particular font that looks good on all the servers you need to run MDPP on, then you can define it in MDPP.COM with a statement like:
$ DEF MDP_DECW_FONT -
$_ "-MISC-FIXED-MEDIUM-R-NORMAL--13-120-75-75-C-80-ISO8859-1"

This line is already in the MDPP.COM file. Just un-comment it and edit it, if ncessary, to point to the font you require.

For each user individually: If you cannot get consensus on a good font that everyone can use, then each user can define the MDP_DECW_FONT logical in their own LOGIN.COM file, AFTER MDPP.COM has been executed.

The Utilities: MDPP.COM (VMS) contains a call to the file UTILITIES.COM that sets up several useful utility programs. Check this file to see if these overlap with routines you are already using. If they do, you may wish to edit this file and update the information in it. The utilities are not a part of MDPP and are not supported as a part of it. However, we do try to ensure that they are kept current and will update any that we can find in a straight-forward manner. Let us know if any utilities are busted and we'll try to help.

HELP: The manual for the system is the same for both the VMS and Ultrix versions of the system. There is now an extensive set of 'HELP' files located in the directory [MDPP.DOC] which is pointed to by the logical name MDPDOC:. These files can be printed out and constitute the manual for the system. MDPP accesses the HELP files using its own internal reading system to provide on-line help. Obtain on-line help by typing "HELP <command_name>" (e.g. "HELP VF") from within MDPP in command mode, or a "?" at any prompt for input. Note that a system level HELP entry for VMS can be obtained by loading the file MDPP.HLP into the system help library file using the command "LIBR/HELP SYS$HELP:HELPLIB MDPP". We have not tried to make a "man" file for UNIX.

On-line HELP via BOOKREADER has not been maintained. Avoid it.

Note also that the plotting program subroutine library its own help library which can be accessed using the appropriate system HELP command.

MDPP Directory Structure

The following is a sketch of the (minimal) directory structure of the VMS/UNIX version of MDPP.


MDPP | +--- GENERAL The VMS/UNIX programs/executables. | +--- LIB The VMS/UNIX Libraries and some .COM files | |---- RGB The color tables | |---- MENU Some examples of user menus | +---- PROC Some examples of procedures | +--- DRIVERS The Driver libraries | +---- UNIX The X11 driver (VMS and UNIX) | +--- DOC The text docs +--- HTMLDOC The docs in HTML | +--- EXAMPLES Some examples. +---- HELICES | +------- STRAIGHTEN +---- MOVIE +---- PLANE_LAYER_FILTERING +---- PLANE_LAYER_RECONSTRUCTION

Differences between VMS and UNIX

Note that the behavior of MDP under UNIX is essentially identical to that under VMS. The interrupt control characters behave in the same way, except that they follow the UNIX model, not the VMS model. In particular under Ultrix, <Cntl-\> means 'Cancel' and <Cntl-C> means 'Quit', and <Cntl-D> means <EOF> (equivalent to Cntl-C,Y and Z, respectively).

Obviously there will be small differences between the VMS and UNIX versions. Mostly, these are listed in the docs. Please look at the docs for RD, SV and IO for the file processing differences.

Getting Started with MDPP

Once you have installed the MDPP you need to figure out how to use it. The MDPP can be started on the VAX by typing 'MDPT', if the approproiate task has been installed. The program will begin by printing a series of messages to you which you answer as follows:


.IS OUTPUT FOR A CRT (1=NO):> x
.ENTER NAME OF OUTPUT FILE (A20):> out/O/I/S
.ENTER NAME OF INPUT FILE (A20):> inp

If the output device is not a CRT then enter '1'; otherwise enter a <cr>. The output file name is entered in response to the next prompt; the default is 'SYS$OUTPUT:'. Output processing is controlled by the switches '/O', '/I' and '/S'. Using the '/I[NPUT_LOG]' switch causes an input log to be generated containing all commands and data entered to MDP. This logfile, called 'INPUT.LOG', should be able to be re-submitted to MDP to re-create the current session. Note that it is only useful to use this switch if you are logging a terminal session. The '/O[UTPUT_LOG]' switch controls whether an output log, called 'OUTPUT.LOG', is to be generated. There is no point in requesting an output log if a general output file is being created. If a general output file is request then the '/S[POOL]' switch controls the spooling of this output file. If the file is to be spooled then append '/S' to the file name (e.g. 'MYFILE.OUT/S'). If just '/S' is entered then a default filename is assigned (MDPPSPL.DMP) and the output is spooled at the end of the run and the file deleted. The input file is to be entered; the default is 'SYS$INPUT:'. If an input file name is entered input will come from that file. The default extension is '.INP'. If an input file is entered then MDPP will ask if MACRO processing is to be done (see the write-up for the MACRO and LOOP processors). The prompt is:-

.ENTER '1' FOR MACRO PROCESSING: m

Entering '1' will cause the MACRO processor to be invoked and an ex panded input file will be generated.

.FILES OPENED SUCCESSFULLY

indicates that processing, if requested, is complete and all further input will be from the designated input file. For the beginer, this will be the terminal.

This tedious input can be simplified considerably and the input all placed on a single line in a comma-separated list, as follows:-

$ MDPT x,out,inp,sw (for VMS)
mcgcr0-14> mdp x,out,inp,sw (for UNIX)

As before, 'x' is either "1" for the 'dumb terminal' mode or blank for the SMG screen managed output., and 'sw' is either "1" to activate the MACRO processor (only if you have an input file please) or blank to skip MACRO processing.

The program will now clear the screen and be ready to start work. The actual response will depend on the manner that it has been started.

The ":>" or "Command:>" prompts indicate that the program is ready to receive input from the terminal and the user, by now in a frenzy of excitment, will comply.

Use Of The MDPP

Prompts (e.g. ':>') indicate that the system expects a command. These commands consist of a two letter command name usually reminiscent of the procedure one wants to perform (e.g. 'rd' means read, 'sv' means save, etc.). The table in the MENU shows which commands are available. The typing of a command not part of the system leads to an error message; three incorrect entries one after the other causes the program to terminate.

Commands are read in the FORTRAN format (A2,2I2). Spaces are therefore significant. The 2I2 allows two numbers to be read in which provide extra information for the commands: the user is advised to respect the formating as failure to do so will lead to disaster.

Description of the Interactive Commands

The descriptions of the commands are organized in help files and in the on-line documentation. The purpose of the command is given and the general description of it is included. Text in small letters is typed on the console; capital letters indicate the computer response. Descriptions and explanations are interposed when necessary. Examples are sometimes included as is a "Remarks" section containing further information about the command of which the user should be aware.

In some of the simpler subroutines errors are not explicitly indicated by an error message but the expected output is suppressed (very rare in this version of MDPP). All commands should terminate with a confirmatory message: if this message is missing and the console simply types the prompt asking for the next command, something has probably gone wrong. It's up to the bright intelligent user to spot his silly mistake.

Basic Program Structure & Philosophy

The fundamental problem in system design from a practical programming point of view is data organization and storage. All these programs are organized around the choice which was made, for better or worse, as to the 'best' way of doing this job way back in 1973 when work was begun, and then again in the 1980's when the program was ported to PDPs and the VAX.

Data involved in processing steps is read from a permanent disk file into a disk or a core buffer area depending on its size, the availability of core in the program and the nature of the processing to be done. Once the data are in the program buffer (the so-called current file) the user can perform operations on these data using the MDPP's commands. At the end of each step (command) in the analysis the data is left stored in the buffer. When the user is finished, the modified data can be written back out to disk. Output is very flexible: the user can write out data after each and every step, after a small series of steps, at the end of the entire chain of processing operations, or not at all. (Note that this is different from some other packages which return the data to disk after each step. Some other packages retain the original data used at each step: MDPP does not, unless the data has been saved by the user explicitly). In the MDPP data read from a disk file is transformed by each command and then returned to the buffer, ready for the next step.

(Generally speaking it is not necessary for the user to pay particular attention to the particular type of buffer being used in a calculation. There are exceptions however when the arrays being used are 'stacks' of pictures or data, and when data arrays are in core. This is sometimes important because some of the routines will work only on data held in core: usually however, these are output processes or those designed for use on processed data.)

To reiterate, analysis of the data is performed by a series of subroutines which are called in the order required by the user, the main program acting as a 'shunting yard' passing the data between each of the subroutines which actually do the work. The order in which the subroutines are called is controlled by the user who generates a so-called 'command list'. The command list consists of a series of two-letter commands, which specify the operation (subroutine) the user wishes to utilize, followed by a series of data lines, should these be needed.

The computational work done by each subroutine is called a job-step. Each job-step is, up to a point, independent of those steps preceding it and those which follow. The programs have been written to permit a reasonable amount of flexibility in organizing the analysis but the user should bear in mind that the order is not really variable if sensible results are to be obtained.

Memory Management

Memory is allocated on the fly as it is needed for the main arrays and buffer arrays. MDPP starts with a relatively small core buffer and this is expanded as required when arrays are read and manipulated. Only when expanding virtual memory for the application fails is the disk buffer used. The problem is that disk array management has not been written for some of the newer routines, so if a virtual memory allocation request fails some commands will not work.

Data Storage

As noted, data is read from disk with the 'RD' command and saved to disk with 'SV'.  MDPP's default file format is the so-called BMD (Binary Micrograph Data) format which is generally optimized for read/write speed and to carry the metadata needed properly to identify it.  This format is not used by other packages so MDPP allows files to be read and written in MRC/CCP4, TIFF and JPEG formats.  e.g.  "RD actin-map.mrc" will cause the file "actin-map" to be read assuming that it is indeed in MRC format.  If the file name extension is omitted BMD is assumed and if such a file does not exist a search is done for files with extensions: .MAP, .MRC, .TIF and .JPG.  Files from other packages (e.g. Spider, PIC) can be read and written using the "XX" (external exchange) command.

Image Visualization

NOTE: Image output (via 'DV') is directed to the X-Windows display device (usually your desktop main screen). Other output methods, such as 'DP', are now esentially obsolete and may no longer work without customization and programming I don't plan to do...

That said, however, image output can be obtained using the commands 'DP' (DisPlay) and 'DV' (Display_Visual). In the version of MDPP distributed here DP sends output to a FACIT 4542 or 4544 color printer. Note that there are routines to generate output for a SIXEL (DEC) printer or terminal. (Current options "DP 3" to the terminal VT330/340, "DP 4" to the SIXEL printer e.g. (LA100).

Output from 'DV' is directed to the terminal, or to a display device. The command causes a file to be output to the device designated by the line 'Display type' in the file 'MDPP.DFL' (see below for a description of this file). At NYU we set this to '1' which designates a video display device. If you have a video display you must have (or write) a driver for it which interfaces MDPP with the device. (See above.)

'DV 3' puts up a very crude picture on the screen of any terminal. 'DV' also uses ReGIS to plot images on the screen of a VT125 or VT240 series terminal; 'DV 1' is quite quick; 'DV 2' is slow, but the quality is surprisingly good (these two programs were provided by Dr. Michael Rademacher in Albany, NY).

Interacting with the Display

Note that your interaction with the MDPP display is different from the interaction with a Macintosh, for example, since the images are not part of an integrated UI. To use the tools that work with the displayed image you need to start a program which uses displayed image data, like the 'VD' command (e.g. 'VD 8' which can be used to inspect both real & Fourier images). Once the program has been started, the cursor changes to the MDP cross-hair cursor within the image window, and the mouse keys can be used to control the program. If a program is not running, the workstation cursor is the same both inside and outside the MDP image window and mouse keys have no effect, as far as MDP is concerned. For user convenience (and contrary to the Motif style-guide!) the cursor is moved into the image window center when a program is started.

Many programs (but not all) which require the cursor position to be marked inside the image window allow you to double-click inside the window (or hit <Cntl-E>, VAX only). This causes a sub-window to be created with an enlargement of the area within it. This is to allow the user to position the cursor to within a sample point in the image: this is generally impossible to do on the main display since the MDP images of large areas are displayed at a resolution too low to permit cursor positioning at the single pixel level. This sub-window automatically extinguishes itself if a mouse key is clicked, or the cursor is moved outside the displayed area. A subwindow is not displayed if the magnification of the original image is already sufficient to allow single-pixel positioning.

The Mouse Button Order

A significant issue with the usability of the package is the disposition of mouse keys. On the Lexidata (from our standpoint, the 'father' of all displays, existing well before X-Windows or Aqua) the key order was right-to-left, with the RH key being the 'busy' key that got most of the work. On DECwindows the opposite is true: the LH key is the key you use most for your work on the workstation. When MDPP was ported to DECwindows we switched the key order to make the LH key the 'busy' key, but for all sorts of bad reasons the messages on the screen which told the user which button to use were not changed. In the V93.200 version of MDPP the 'RH-busy' order has been restored so that the mouse-key order corresponds to the messages. Since most long(er) time users of MDPP are now accustomed to the LH order there is a new logical called 'MDP_REVERSE_MOUSE_BUTTONS', which if defined to 'TRUE' reverses the button order and gives them back the LH order they have been using for the last few years. Note that the key to be double-clicked to POP windows remains the LH button which is not a good choice, but thats the way it is for now, at least.

Plotting

Plotting routines have been provided and are used by MDPP for displaying images via contour plots, etc. The best method available at present is to use the GKS option in the 'PL' command. This permits both full screen and hard copy (on a LA100 type printer) plotter output. Read the documentation. Other options include the generation of contour files suitable for re-input, or transport to MOVIE-BYU. The generation of contours on an image on the current display device is a powerful method for displaying graphical output such as contours. Alternatively, plotting can be done using a ReGIS plotting device designated 'PL0:', which the user must assign. The plotting device is initially assigned to 'NL:', the null device, which is satisfactory for GKS use, but which you will need to re-assign if you need pure ReGIS plotting. If you have no plotting device at all (i.e. no VT240, VT125, TEK4010....) then select ReGIS plotting in programs which require plotting output; the plotter output will be dumped in the null device and cause no problem. Additionally, plotter output can be obtained for the FACIT printer by re-linking MDP with the appropriate library (available for V86.100 or later). This is a fairly clumsy method to obtain hard copy graphics, but due to the relative scarcity of these printers, an alternative seems inappropriate at this time. If you have a ReGIS hard copy device, then that can be used without modification, of course. Using dump_graphics from the VT125 screen, is another option.

Communications

The communications utility KERMIT has been provided. KERMIT is a terminal emulation package which permits 'error-free' data transmission. In addition we have provided three programs which act as encoders/decoders to permit binary (e.g. .EXE, .OLB, etc.) files to be transfered by KERMIT and/or the MAIL programs. These are CODE, SEND_FILE and MFTU. CODE is inefficient, but is available at many sites which support PMDF, SEND_FILE is good for text file transmission, and MFTU is to be preferred for binary (although all file types can be used as input to all the programs). Documentation has been provided in the [DOC] directory. You may need these programs to update MDPP remotely from NYU. Note that file compression/ decompression can be performed using LZCMP and LZDCM programs, considerable reductions in size can be achieved in some cases, which is useful for data transmission and for long-term storage of images on line: these two programs are compatible with UNIX also. TAR2VMS is a program to read TAR tapes from UNIX.

Defaults (or: What am I going to do NOW!!?)

The MDPP is a complex program. The documentation is idiosyncratic and there are going to be many times when figuring out what to do will be well-nigh impossible. As an aid to the confused (and to make life easier for the cognoscenti) MDPP tries to have acceptable defaults available for all the inputs requested by the various commands. The correct thing to do when faced with a confusing situation, is to just hit <cr> and see what MDPP thinks the right answers are. Now obviously this is not going to work all the time. So if you are confused and you have not saved the last step, back off from the command by hitting <cntl-Z> (VMS) or <cntl-D> (UNIX) to exit to the command prompt, save the file, and then restart the command. Now you can play: if you get the input arguments wrong just re-read the file and try again. If the file is saved from the previous step, there is really nothing that can get seriously screwed up..

Getting HELP

You can get help at any time input is requested by typing '?' and then <cr>. Help for a given command is obtained by typing 'HELP XX' at the command prompt, where 'XX' is the command of interest.


Using the Examples

The outline here for the usage of the MDPP is undoubtedly less adequate for a user that needs to get started than it might be. Unfortunately, there is no good way to communicate the 20-odd years that have gone into program development and the thinking (and sometimes the lack of it) that has influenced the growth of this package.

For the most part, MDPP deals best with the problems it was designed to tackle, EM images and structural biology. Some of the tools here are unique and have never been successfully reproduced on other integrated packages (the helix stuff is an example, shadowed particle processing is another). We are very happy with the 2-D filtering tools as well and the 3-D reconstructions from tilted views of 2-D crystals.

However, if you are looking for a package to do general image processing of color images, or morphing, etc. MDPP is not the tool for you. There are a few tools, but sophisticated stuff isn't there, and never will be, probably. If you want to slice and dice 3-D images at high speed, you need a SGI and something like AVS as your IP package, not MDPP. My guess is that the tools for analyzing radiological data are probably all there, but we do not have the experience or expertise to recommend MDPP in this application.

Getting started is tough. But the only way to come to grips with the programs is to use them. Play around! Read in one of the example files (e.g. DONNA.BMD, or MANDRIL.BMD), get maximums and minimums (MM), generate noise images (GN), display them on the workstation (DV), slice their values (SL), add noise to DONNA (SU), smooth the result (SO), compute a total image power (TP), renormalize (NR), draw some contours (PL), warp the image (WA), etc, etc... Look thought the "Menu of commands" and try some things you like the look of (maybe they will work [our fault], maybe not [your fault] :-). But don't get scared off and if you have a major problem, feel free to call or E-mail for help.

Finally, when you have built up a bit of confidence, look at the examples in the area MPP:[EXAMPLES]. This are contains complete sets of data and command lists to allow you to see the MDPP in action for several of the major problem areas where the MDPP works well. Don't screw up the files in that area!   Rather, copy them over to your own space and play with them. Change the command lists, use the commands interactively, fiddle. Report the bugs if you find any! Don't accept stuff you do not understand, and don't accept anomalies in the processing (that is how we have found some key errors).

Finally, good luck! We have fun making MDPP (for the most part that is) and think it is a really great product. Your use of it can make it better. Keep in touch!

The Mandatory Disclaimer In tiny print


Users of this software package, which includes MDPP and the utilities, accept
full responsibility for its possession and use. Users understand that this is
an unsupported package provided at no charge consisting of experimental
software intended for research purposes only. The package is provided as-is
without any warranty of any kind expressed or implied as to its fitness to
perform any function or meet any specification. No warranty is made that the
contents of this package or the use of this package will not infringe on any
patent, copyright, trade secret or trade mark of any third person. None of
the contributing authors, nor the institutions where they currently work or
have worked in the past, nor HEWLETT-PACKARD CORPORATION (HP)
accept any liability for the contents of the package, the use of this package
or any fragment of it, or the consequences of using the package or any of the
utility programs. A user of this package accepts full legal liability for
the possession of the package, its components and any and all use thereof. A
user accepts all costs of legal defense for any and all legal actions which
may arise from use of the package. A request for distribution components of the
package, or their use, is understood to be construed that user undertakes to
defend the authors and their institutions, at the users own cost, in any
legal actions which may arise from the possession or use of the package by
that user. Use of this package in conjunction with medical patient care
is not recommended
. Use of this package in a clinical environment is
the sole responsibility of the Chief of Service of the relevant clinical
department who must establish on his own authority that the package is free
from errors which could affect any aspect of patient care. It is recommended
that patient "informed consent" be obtained from all individuals whose
clinical data is to be analyzed with this package, appraising them of these
conditions of use. The package is believed to be free from extraneous code
which would constitute a 'Virus', 'Worm', 'Trojan Horse' or similar code
which would damage a computer or its files: however, no specific checks have
been made for such extraneous code and no guarantees can be made that such
code has not been inserted, unknown to the authors. Therefore, the authors
and their institutions accept no responsibility for damage to data or
equipment, loss of use of any system, disruption of programmers time or any
other disaster or difficulty arising from the use of MDPP or any of the
utility programs.