= p3d README =
Version information:
This is the release version 2.6.3 of p3d, "Discovery".
$Id: README 5214 2019-09-05 08:30:54Z christersandin $
= Table of Contents =
1) Introduction
o 1.1 About p3d
o 1.2 Installing p3d
o 1.3 Use p3d with the IDL Virtual Machine, without a license
o 1.4 Use p3d from the IDL command line, with an IDL license
o 1.5 Additional details about the auxiliary software
2) Contact
o 2.1 Funding acknowledgments
o 2.2 Collaborators
o 2.3 Reporting Bugs
o 2.4 Feature Requests
3) Software Platform
o 3.1 Coding Philosophy
4) Supported Instruments
5) Recompiling the binary meta-files for the IDL Virtual Machine
6) IDL Terms of use
= 1) Introduction =
Section 1.1 contains a very brief introduction to p3d while the
installation procedure is discussed in Section 1.2. Sections 1.3 and
1.4 describe how p3d is used from outside of IDL with the IDL Virtual
Machine or from the IDL command line using a license, respectively.
Section 1.5 discusses details of the required auxiliary software for
users with a specific interest.
== 1.1) About p3d ==
p3d is a data-reduction tool for fiber-fed integral-field
spectrographs. Please see the project web site for more details:
https://p3d.sourceforge.io.
== 1.2) Installing and using p3d ==
The number of required steps to install p3d is very small for basic
use. The only requirement besides p3d itself is a working version of
Python (version >= 2.7) and IDL (version >= 6.2). In case that IDL is
unavailable on the system, there is the option to ask us for a
p3d-specific IDL distribution that works on all platform supported by
IDL; IDL is assumed unavailable when the environment variable IDL_DIR
is unset. To get a download link to the p3d-specific IDL distribution,
please send an e-mail to the address given in the contact section.
The minimal installation consists of retrieving p3d, IDL, as well as
the additional (auxiliary) software that p3d uses. p3d can download
the additional software by itself when it is launched from outside
IDL (with one exception, see Section 1.5); otherwise, it is only
necessary to launch the software download tool p3d_acquire_xtools.py
in the p3d vm directory. There are two approaches to retrieving a copy
of IDL, see sections i and ii. Section iii contains information on how
to acquire p3d itself (including this file).
Attention! While p3d can be run using IDL versions >= 6.2, there seems
to be a bug with IDL versions < 8.4 that appears when subtracting
background regions in the spectrum viewer. Therefore, consider using
IDL version 8.4 or higher.
=== i) Use the p3d-specific IDL distribution ===
Due to licensing issues of IDL, we cannot put the p3d IDL distribution
on the internet for free download, but just send us an e-mail and
receive a download link in return. The p3d IDL distribution is about
270MB large and works on all platforms supported by IDL version 8.7.0;
this distribution is only meant to be used with the pre-compiled
binary save files of p3d (regardless of if an IDL license is available
or not) and not from the IDL command prompt. The reason is that it
does not contain all of IDL, but only the parts that p3d use.
The distribution is a zip archive file that must be placed so as to
have the idlrteX.X.X directory in the p3d contrib directory (see
section iii below). The environment variable IDL_DIR, which
configuration is described below, is set automatically to the correct
path when it is unset and the p3d-specific distribution is identified.
=== ii) Install a full copy of IDL ===
A complete version of IDL can otherwise be downloaded from the
following web page (it is necessary to register to be able to download
the software):
https://www.harrisgeospatial.com/My-Account/Downloads
See the relevant IDL documents on how to install IDL (i.e. extract or
copy the files to a proper location on your computer) and on how to
acquire and setup an IDL license (if one is required). After IDL is
installed, it is necessary to setup the system environment to tell it
where IDL is found; this is unnecessary if the p3d-specific
distribution is used (see i) or if your system administrator(s) has
already done this. The environment variable IDL_DIR must be set to the
name of the directory where IDL is installed. Check if it is already
set by entering the following command (on *NIX-type platforms):
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
printenv IDL_DIR
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
If the output of the command points to the path of the IDL version to
use, there is no need to set the path anew.
If IDL_DIR is unset and the p3d-specific distribution is not used, or
if there is a wish to use a personal downloaded copy of IDL, the
following line can, depending on the used system shell, be added to
the bash terminal setup file in the home directory (~/.bashrc):
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
export IDL_DIR=/path/to/IDL/
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
or if the csh is used (~/.cshrc):
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
setenv IDL_DIR /path/to/IDL/
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
where the tilde character (~) denotes the user home directory.
It is possible to use the p3d-specific IDL distribution instead of an
installed version of IDL (if the files are placed in the required
directory) by simply unsetting the IDL_DIR environment variable as
follows (on *NIX-type platforms):
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
unset IDL_DIR
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Attention! If p3d is to be used on a Windows-based platform, the
IDL_DIR environment variable is defined using Windows GUI tools. The
exact procedure to set this variable varies depending on the used
version of Windows, see below. The variable should also not in this
case be set when using the p3d-specific IDL distribution.
Check that the IDL executable is available by entering the following
command (this step is not required on Windows-based platforms):
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
which idl
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
If there is no output, a symbolic link could be created that points at
the IDL binary; this can be done using the following example (assuming
there is a bin/ directory in the home directory and that ~/bin exists
in the PATH environment variable:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
cd ~/bin
ln -s ${IDL_DIR}/bin/idl
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
When "idl" is typed at the prompt IDL should start. If there is no IDL
license available, there will be a lot of information about this
fact - but there is no need for concern about these particular
messages if the intention is to run p3d using the IDL Virtual Machine.
=== iii) Get the p3d software ===
Either use the p3d development version through the subversion tool,
or download one of the prepared releases from the program files area
(https://sourceforge.net/projects/p3d/files/). The former approach is
preferred when new functionality is required, which was not yet
included in a release version; although, there is a slight chance that
the development code contains new bugs (that are corrected as soon as
they are found and time permits, or when we are told of their
existence).
To download p3d from the subversion repository at sourceforge.net,
open up a terminal and change the directory to where the software is
to be placed. Thereafter, enter (or paste) the following command:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
svn checkout svn://svn.code.sf.net/p/p3d/code/trunk p3d
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
whereby all p3d-related files are placed in the directory named p3d
(any directory name can be used). It is also possible to use the http
protocol instead of svn by replacing "svn:" with "http:".
To update p3d when using the development version, change to the p3d
directory in a terminal (or whatever name is used for this directory)
and type the command 'svn update'. For any additional functionality of
subversion, including a complete and very extensive manual, please
consult its web page: http://subversion.apache.org/.
The required auxiliary programs of p3d are downloaded automatically to
the p3d contrib and src directories if they are not detected when a
p3d tool is launched from the shell, with one exception being the
Voronoi binning tool of Cappellari & Copin (2003). The files of the
following sources are required for full functionality (the last two
are optional):
o Astro-lib (http://idlastro.gsfc.nasa.gov/homepage.html)
o All files (because it is easier to download all files).
o Mpfit (http://purl.com/net/mpfit)
o mpfit.pro
o cmset_op.pro
o cmpfit-1.3a.tar.gz
o Voronoi bin tool (http://www-astro.physics.ox.ac.uk/~mxc/software/)
o voronoi_2d_binning.pro
o SIMBAD astropy / astroquery (https://astroquery.readthedocs.io)
When the intention is to only use p3d from the IDL command line (see
below), it is necessary to download the required auxiliary programs
separately using the tool p3d_acquire_xtools.py; the same tool is
called by the p3d Python tool and the p3d bash scripts when the files
are non-existent:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
p3d_acquire_xtools.py
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
The files are placed in contrib/astro-lib/ (for the files of
Astro-lib) and in contrib/ (mpfit.pro and cmset_op.pro). The archive
file cmpfit-1.3a.tar.gz is extracted in src/, where it creates the
sub-directory cmpfit-1.3a.
Owing to licensing issues, the Voronoi bin tool voronoi_2d_binning is
not downloaded automatically, but must be downloaded separately,
putting the required file (voronoi_2d_binning.pro) in the p3d contrib
directory (all tools will still work without this routine, but the
related parts are deactivated).
Additionally, the Python astroquery package is not installed by p3d.
To use the target-selection functionality of the p3d spectrum viewer,
you need to install astroquery yourself, either on your computer or in
a virtual environment that you launch p3d from. So, if you cannot
install new Python packages on your computer, have a look at the tool
'virtualenv'.
Note! It is also possible to download all parts and releases of p3d to
the same directory by leaving out the "/trunk" part in the command
above. Few users will want to do this. (The directory is in this case
about 1400MB at around revision 3800.)
Note! Due to updated software on the sourceforge.net side, before
2012-12-07, the repository address was different. The old and the new
repositories are incompatible, which is why it is necessary to
replace older repositories with one using the address above, to allow
any software updates.
Attention! When p3d starts, it places the directories it uses at the
front of the IDL path, in the correct order. If other IDL tools
depend on p3d, it is necessary to set the environment variable
IDL_PATH or the IDL !path variable in the IDL startup file, adding
the p3d directory at the required position in the used
configuration. Doing this, p3d adds the additional directories at
the location of that string. (There will be confusion if p3d
versions are mixed up in IDL_PATH or !path.) The way p3d works, it
uses the parent directory of the called p3d tool (using the IDL
Virtual Machine or from the IDL command line) as the path to
p3d. Likewise, p3d sets up the preferences (environment) variable
IDL_DLM_PATH to include the src directory in the p3d path.
Attention! If the intention is to use p3d on a Windows-based platform,
these environment variables are defined using Windows GUI
tools. The exact procedure varies depending on which version of
Windows is used.
Windows XP: Control Panel --> System -->
Advanced system settings --> Environment Variables
Windows Vista: Start --> Computer --> Right Click --> Properties -->
Advanced System Settings -> Advanced -> Environment Variables
Windows 7: Start --> right click Computer --> Properties -->
Advanced System Settings --> Advanced tab -->
Environment Variables
Windows 8: Start --> All Apps --> Control panel --> System -->
Advanced System Settings --> Advanced --> Environment variables
Windows 10: Window Key + X --> System -->
Advanced System Settings --> Environment Variables...
Attention! Set the environment variable ${p3d_font} to the name of an
existing font (see the above example) if you want to use some
other font than the IDL default font; this might not always be a
good idea, but the option is available nevertheless. It is important
to select a font that exists on the system, check which ones are
installed using, e.g., the 'xlsfonts' command. It is, additionally
important to check that the used font is not too large; because, if
it is, some widgets might not fit inside the respective tool (if
this point seems vague, try a very large font, launch p3d or p3d_sv
and look at the resulting windows). Due to restrictions in how IDL
parses command-line arguments, the font string must not contain any
white space!
==== a) Use the Virtual Machine ====
The intention is that p3d is used with the IDL Virtual Machine. p3d
configures itself when its main tools are launched; the use is more
comfortable when the p3d vm directory is added to the system PATH.
Additionally, it might be helpful to set the environment variable
p3d_data_path to the name of the (base) directory where the raw data
are kept. Depending on the system shell, the following lines could be
added to a suitable place in the ~/.bashrc file (if using bash; or
another "Bourne Shell"-type environment):
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
export PATH=$PATH:/path/to/p3d/vm # Useful
export p3d_data_path=/path/to/IFU/data/ # Optional, but useful
export p3d_font=lucidasans-14 # Optional, experts only
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
or ~/.cshrc (csh):
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
setenv PATH ${PATH}:/path/to/p3d/vm # Useful
setenv p3d_data_path /path/to/IFU/data/ # Optional, but useful
setenv p3d_font lucidasans-14 # Optional, experts only
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
==== b) Use p3d from the IDL command line ====
It is convenient to add the p3d directory to the IDL path to avoid
having to enter the full path to the tool each time it is used. There
are two ways to do this. Either set the environment variable IDL_PATH
or the environment variable IDL_STARTUP. Avoid using both approaches,
because there will be confusion and possible problems if the used p3d
path changes and the change is not made in both places.
The first approach requires IDL_PATH to be set to include the p3d
path. Put the p3d path at the front, unless an IDL program is used
that itself uses p3d. Add the following line to ~/.bashrc (ensure that
the string ":<IDL_DEFAULT>" is added at the end) using a
"Bourne Shell"-type environment:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
export IDL_PATH="/path/to/p3d:<IDL_DEFAULT> # Modify for your system
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Instead add the following line to ~/.cshrc in a "C Shell"-type
environment:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
setenv IDL_PATH "/path/to/p3d:<IDL_DEFAULT> # Modify for your system
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
The second approach is to set the environment variable IDL_STARTUP to
the name of an IDL program that defines the path. The following lines
can be used to do this (the filename used here "IDL_STARTUP.PRO" can
of course be chosen freely). For a "Bourne Shell"-type environment,
add the following line to ~/.bashrc:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
export IDL_STARTUP=/path/to/IDL_STARTUP.PRO # Modify for your system
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
For a "C Shell"-type environment, add this line to the file ~/.cshrc
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
setenv IDL_STARTUP /path/to/IDL_STARTUP.PRO # Modify for your system
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
As in the first approach, the p3d path is added at the beginning of
the string unless an IDL program is used that itself uses p3d; in the
latter case, the p3d path should come after the other IDL program.
Add the following line to the file IDL_STARTUP.PRO:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
!path = '/path/to/p3d:' + !path
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Once the p3d tools are used, the IDL path is modified to fit p3d. If
it is necessary to call one of the p3d modules in the p3d routines
directory, it is recommended to first launch p3d with the SETUPONLY
keyword, which only configures the IDL !path and then exits:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
p3d, /setuponly
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
== 1.3) Use p3d with the IDL Virtual Machine, without a license ==
A Python tool as well as multiple bash scripts are provided to run
p3d from the shell, outside of IDL, using the IDL Virtual Machine on
all *NIX-type platforms as well as in Windows. To start the p3d main
GUI using the p3d queue system, enter the following in a terminal (if
p3d/vm was added to the system path):
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
p3d_dispatch.py [-s] <tool> <options>
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
or, if p3d/vm was not added to the system path (this is just an
example, the full path could be used instead of setting p3d_path):
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
${p3d_path}/vm/p3d_dispatch.py [-s] <tool> <options>
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Add the optional flag --noqueue (or its equivalent -s) before the tool
argument if you want to avoid using the p3d queue system, which might
require some effort to use efficiently as well as understand. The
purpose of the queue system is to click the IDL 'click-me' popup
window fewer times when no IDL license is available; this is
facilitated by the use of p3d server sessions that run in the
background where each session picks one job at a time from a job queue
managed by a p3d Python queue handler (see below).
The tool can be selected as one of the following 15 strings: p3d,
p3d_sv, p3d_cmbias, p3d_ctrace, p3d_cflatf, p3d_cobjex,
p3d_cvimos_combine, p3d_cr, p3d_darc, p3d_rss2cube, p3d_cexposure,
p3d_fluxsens, p3d_fluxcal, p3d_ifsfit, p3d_ifsfit_mosaic, p3d_csim
(more will be added). It is also possible to use any of the older
bash-based scripts (which all call p3d_dispatch.py).
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
p3d_vm.sh [-s] [-rt] <options>
p3d_cmbias_vm.sh [-s] [-rt] <options>
p3d_ctrace_vm.sh [-s] [-rt] <options>
p3d_cdmask_vm.sh [-s] [-rt] <options>
p3d_cflatf_vm.sh [-s] [-rt] <options>
p3d_cobjex_vm.sh [-s] [-rt] <options>
p3d_cvimos_combine_vm.sh [-s] [-rt] <options>
p3d_rss_vm.sh [-s] [-rt] <options>
p3d_cr_vm.sh [-s] [-rt] <options>
p3d_darc_vm.sh [-s] [-rt] <options>
p3d_rss2cube_vm.sh [-s] [-rt] <options>
p3d_cexposure_vm.sh [-s] [-rt] <options>
p3d_fluxsens_vm.sh [-s] [-rt] <options>
p3d_fluxcal_vm.sh [-s] [-rt] <options>
p3d_ifsfit_vm.sh [-s] [-rt] <options>
p3d_ifsfit_mosaic_vm.sh [-s] [-rt] <options>
p3d_csim_vm.sh [-s] [-rt] <options>
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
See the respective tool for detailed information regarding available
options:
p3d :: (and p3d_tool_gui) for the p3d GUI.
p3d_cmbias :: to create a master-bias image.
p3d_ctrace :: to create a trace-mask image.
p3d_cdmask :: to create a dispersion-mask image.
p3d_cflatf :: to create a flat-field image.
p3d_cobjex :: to extract science-object spectra.
p3d_fluxsens :: to create a sensitivity function from a summed
standard-star spectrum. The sensitivity function
is then used to flux calibrate the data. A
sensitivity function can also be created using the
p3d spectrum viewer (p3d_sv).
p3d_fluxcal :: to flux calibrate extracted data using the
sensitivity function created with p3d_fluxsens or
p3d_sv (or even IRAF).
p3d_cvimos_combine :: to combine extracted VIMOS spectra of the
four separate quadrants into one spectrum image.
p3d_sv :: to view and analyze extracted spectra as well as
create a sensitivity function for flux
calibration. Accepts both RSS images created using
p3d and data cubes of any origin.
p3d_cr :: to clean single images of cosmic-ray hits.
p3d_darc :: to correct data for effects of differential
atmospheric refraction; only works with
instruments that use square-shaped elements,
currently.
p3d_rss2cube :: to convert the resulting RSS-formatted images
into three-dimensional cubes; only works with
instruments that use square-shaped elements,
currently.
p3d_cexposure :: to combine multiple extracted images; can
select files that used different exposure times to
replace data in extracted spectrum elements that
involved saturated regions in the raw data.
p3d_ifsfit :: to fit any number of emission and absorption lines
to the spectra, in countless combinations.
p3d_ifsfit_mosaic :: to create publication-ready plots of the
outcome of p3d_ifsfit.
p3d_csim :: to create simulated raw data, which can be used to
test instrument properties.
Note! The options are specified in the usual IDL way, almost, but with
a space between the options (instead of a comma, as is the case when
p3d is used from the IDL command line). Additionally, array values
are specified without brackets and without white space between the
elements. For example, some arguments and keywords accept multiple
files. Such files are separated with a comma, but without any white
space; for example, use "file1.fits.gz,file2.fits.gz range=200,2200"
in a system shell and
"['file1.fits.gz', 'file2.fits.gz'], range=[200, 2200]" on the IDL
command line.
Note! The p3d Python tool as well as all shell scripts can be called
with the -rt option. An IDL license is required in this case. This
behavior could be useful if one wants to avoid having to click in the
splash window that always appears each time the IDL VM is launched.
Although, with the p3d queue server system, the need to click on
those splash screens is delimited to when the p3d queue system is
first launched.
Tutorials on how to use p3d in the data reduction and visualization
are available in the doc directory. Here are two examples on how to
use the tools from the system shell to get started:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
p3d_dispatch.py p3d /cwd /ppak4k logfile=dred.log
opath=output_directory
p3d_dispatch.py [--noqueue] p3d_cobjex \
file1.tar.gz,file2.tar.gz \
opath=$opath masterbias=${opath}/masterbias.fits.gz \
tracemask=${opath}/tracemask.fits.gz \
flatfield=${opath}/flatfield.fits.gz \
userparfile=${opath}/user_p3d.par logfile=${opath}/dred.log \
loglevel=2 /verbose /quiet
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
The p3d IDL server allows any number of jobs to be submitted to the
p3d queue. The p3d queue handler feeds the jobs to the p3d servers,
which are launched automatically and continue to run until the queue
is shut down. Each p3d server uses one thread (core) on the computer;
the server is unaware of how many other threads there are and how many
threads it can use by itself. To make efficient use of system
resources, it is therefore necessary to specify how many threads
should be used when a job is submitted. When all jobs are done, the
queue is shut down using the following command:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
p3d_queue_stop.py
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
If the queue should stop working for some strange reason, it is
possible to kill all IDL sessions launched by the queue handler with
the following command:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
p3d_queue_kill.py
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Note! The p3d_dispatch.py program checks if the required IDL binary
meta files (e.g. p3d_*.sav) are present in the vm directory. IDL
launches using these files.
Note! The Python tool p3d_dispatch.py as well as all bash scripts
p3d*_vm.sh require that Python is installed on your computer, any
version >= 2.7 will do. Also note that, if the command /usr/bin/env
does not exist, it is necessary to launch this (as well as the other)
python program(s) preceeded by the name of the Python command
(with the full path):
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
python p3d_dispatch.py
python p3d_acquire_xtools.py
python p3d_tool_makevm.py
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
If there is a need to use the bash scripts instead of the Python
program, it is necessary to have bash installed as /bin/bash on the
computer (this does not apply to Windows computers). Please, edit the
first line in the scripts accordingly if this is not the case (i.e.
the first line stating "#!/bin/bash"), or execute them as in the
following examples:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
bash p3d_vm.sh
bash p3d_cmbias_vm.sh
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Note! With an IDL license, it is possible to recompile the p3d binary
files used with the IDL Virtual Machine. For this purpose, it is
strongly advised to use the provided Python tool p3d_tool_makevm.py
(see section 5.0).
Note! The debug option (/DEBUG) is unavailable when using p3d from the
system shell (also when the -rt option is used). However, it is
still possible to select not-yet-fully supported instruments from
the command line, which can otherwise only be selected with the
instrument-selection GUI when /DEBUG is set.
== 1.4) Use p3d from the IDL command line, with an IDL license ==
After performing the installation as decribed in Section 1.2, just
enter the name of the tool to use on the command line, with the
necessary arguments and keywords. For example, to launch the p3d
data-reduction GUI, enter:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
p3d
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Available options are found in the source file, which can be browsed
using any available text editor or, e.g., the commands more or less),
but they can also be shown using the HELP keyword:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
p3d, /help
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
The help option is available with all tools and routines of p3d.
Note! The exact order of entries in the IDL path is important.
Note! Do not include the p3d vm directory in the IDL !path variable!
If this is done, there will be a confusion in the path if any of the
program routines is changed. So, even though a p3d routine was
changed, IDL might choose to use the pre-compiled version of the
same routine in the binary meta file p3d_tool_server.sav.
Consequently, if the vm directory is kept in the IDL !path, it is
necessary to recompile all routiness after any change is made.
Avoid this hassle by not having the vm directory in !path.
=== 1.5) Additional details about the auxiliary software ===
Some routines or parts of routines are, in addition to their IDL
implementation, also provided in C. The reason is that compiled C code
in computationally intensive cases (involving nested loops) run faster
than the corresponding IDL code; often drastically faster.
Additionally, in C routines it is possible to make use of more of the
computing power on a multi-processor computer; through OpenMP
(http://www.openmp.org).
Here are some of the tasks available in both C and IDL code:
o The calculation of cross-dispersion line profiles.
o The calculation of a median-filtered science image, in regions
where there are no spectra, to subtract a scattered-light, or
a dark-current, component.
o The spectrum extraction, using regular tophat extraction and
optimal extraction with the multi-profile decvonvolution method
and the band-diagonal option.
o The conversion from pixel space to spatial-element space when
selecting a region of interest in the spectrum viewer.
o The calculation of pixel-based cross-dispersion profile images
used in the optimal extraction.
o The combination of multiple images using the min/max filter,
and the "slowimcombine" mode.
o Cosmic-ray cleaning of single images.
o The resampling during wavelength calibration from pixel-based
coordinates to wavelength.
These particular routines were implemented in C because their IDL
implementations are more-to-extremely time-consuming.
The C routines are compiled automatically by p3d whenever they are
required. With the exception of the spectrum viewer p3d_sv, all
available threads on a multi-processor computer are used by default,
use the NTHREADS and CRNTHREADS keywords to change this number. The
spectrum viewer uses one thread by default. To use the compiled C
routines, ensure a working C compiler is available. If the compilation
fails, for any reason, p3d falls back to use the corresponding IDL
version. Use the keyword CDEBUG to make the compilation process more
verbose. It is also possible to completely deactivate the C modules by
setting the keyword NOC or the user parameter 'dontusecroutines' to
'yes'. The C-based cosmic-ray cleaning routine can be deactivated
separately by setting the NOCRC keyword (NOC with p3d_cr) or the
'nocrc' user parameter.
Note! If one or several of the C routines in are updated after the
shared library is built, it is necessary to either set the rebuild
library parameter ('rebuildlib yes') or to remove the library file
in src/p3d_clib[.so,.dll]. A new library file is then compiled from
the source files.
Note! The C routines are parallelized using OpenMP. Please try a more
recent compiler in case messages are shown indicating that
OpenMP-related routines are not found.
Note! The native compiler on MacOSX platforms clang does not include
support for OpenMP (it seems). Therefore, p3d uses the gcc compiler,
which needs to be installed on the system; one way to install gcc
is to use the tool 'Homebrew'. Have a look at the following web page
on how to install this tool:
https://brew.sh
Should you choose to install, for example, gcc version 7 using the
formula gcc@7, the resulting compiler binary is named gcc-7. The
next step is to set the environment variable P3D_GCC_FULL_PATH to
the full path of the used gcc compiler, for example:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
export P3D_GCC_FULL_PATH=/usr/local/bin/gcc-8
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Note! The C routines currently compile on Linux and MacOSX platforms.
Please contact us if you need support for Windows. In principle, it
should only be a matter of launching idl and p3d from the Microsoft
Visual Studio command prompt (because, only that way is the compiler
environment setup properly). It will also be a matter of checking
that the compiler options set in "p3d_misc_get_callext_exlib" are
the correct ones. Any input on this issue is well received.
Warning! If p3d is used on a platform while switching between the
32 bit and the 64 bit IDL environments, please remember to delete
the shared-library files between the switches, otherwise the
shared library will be of an incompatible architecture, which means
big problems. p3d can use its scripts with the 32 bit environment on
64 bit machines if the IDL_OPTS environment variable is set to
"-32". However, it is not recommended to do this switching between
32 and 64 bits, as the outcome is system-dependent and therefore
generally unpredictable.
= 2) Contact =
All collaborators, unless noted otherwise, have the following
affiliation:
Leibniz Institute for Astrophysics Potsdam (AIP)
Instrumentation
An der Sternwarte 16
D-14482 Potsdam
GERMANY
Tel: +49-(0)331-74990
http://www.aip.de
The p3d-project web site is:
https://p3d.sourceforge.io
where the software can be downloaded, the documentation viewed, and
much more. Please send an e-mail to the following address, should you
want to use the p3d-specific IDL distribution:
olestreicher or p2w@users.sourceforge.net
Documentation is available in the individual routines - the same text
is also available on the Documentation page at the project web site;
this documentation is updated whenever there is a release. Detailed
descriptions of the algorithms are provided in the (freely available)
Astronomy & Astrophysics paper about p3d:
Sandin et al. 2010, A&A, 515, 35
http://www.aanda.org/articles/aa/pdf/2010/07/aa14022-10.pdf
And also see the Messenger and SPIE papers (download links are
available at the p3d web site):
Sandin et al. 2011, The Messenger, 144, 13-16
"p3d ~ A Data Reduction Tool for the Integral-field Modes of VIMOS
and FLAMES"
Sandin et al. 2012, in: "Software and Cyberinfrastructure for
Astronomy II", Radziwill N. M., Chiozzi G., eds, Proc. SPIE,
8451, 84510F-1~16
"Automated and generalized integral-field spectroscopy data
reduction using p3d"
== 2.1) Funding acknowledgments ==
This project has been made possible through funds from the following
sources:
BMBF: 03Z2A51, 05A08BA1
== 2.2) Collaborators ==
=== All programming contributors of p3d ===
Christer Sandin :: Project leader and programmer, until 2016-07-22.
If you have comments of bugs or feature
requests, which are directly related to p3d, and
that you cannot add to the tracker at the
project web site (see below for instructions),
please send them to the e-mail address given in
the Contact section.
Fachreddin Tabataba-Vakili :: Has ported the multi-profile
deconvolution spectrum extraction routine from
IDL (p3d_extract_optimal_mpd.pro) to
automatically compiled C (p3d_cmpd.c), which is
also parallelized with OpenMP. He has also made
the main work on the inclusion of interactive
flux calibration (using a graphical user
interface) in p3d. He has written a tool that
generates IDL runtime distributions, and the
framework tools around the p3d IDL server. He is
finally working on the conversion of various
slow routines to compiled and parallelized C.
Peter Weilbacher :: Has ported the slowest routine of p3d, the
cross-dispersion line profile fitting routine,
to C, which is also parallelized using OpenMP.
Has also included an option to calibrate the
dispersion mask against sky emission lines. He
also reviews the routine documentation.
Ole Streicher :: Has provided p3d with a first version of the
tutorials at the project web site. Has also
written various tools that automatically
translate the routine documentation into xhtml
documents, which can be viewed at the project
documentation web page:
https://p3d.sourceforge.io/index.php?page=doc
=== Additional acknowledgments ===
Please see the p3d project web site for further acknowledgments.
== 2.3) Reporting Bugs ==
To report any kind of bugs we urge you to use the bug tracker
available at the p3d-project web site (Develop/Tracker/Bugs); this
way the bugs will be seen by all users and programmers, and won't be
lost in an e-mail program.
At the bug-tracker page you can search previously reported bugs, to
see if your bug has already been reported. Or you can open your own
bug description (by clicking on "Add new"). Doing this, please ensure
that you use the latest version on the subversion trunk of p3d (or
specify the revision or version of the release you use). Thereafter,
specify a category for the bug, and provide a short summary of the
problem, and most importantly, describe the problem with enough detail
that somebody who does not use your setup and your data files is able
to follow your arguments. Bug reports without a useful description
will be assigned a low priority, or even be deleted. If you think that
it would help us, attach all relevant data files to your report
(please compress fits-files using tar/gzip/bzip2/xz, zip, or use the
compress-option of p3d to save space). Additionally, please specify a
reasonable priority to your bug. While our intention is to correct
serious bugs as soon as possible ~ these are the ones that prevent p3d
from producing spectra ~ GUI-related issues are assigned a lower
priority.
Note! When reporting a bug, turn on the debugging mode (/DEBUG) of p3d
and paste the resulting output from the IDL terminal/window (once
the bug has been reached) into your bug description. The reason we
ask you to do this is that the information returned by the error
handler, when the DEBUG keyword is unset, only provides the name of
the file where the problem occurred, and an IDL-specific number.
Such sparse information is nearly always insufficient to interpret.
== 2.4) Feature requests ==
If p3d is missing some functionality that you would very much like to
have included, you can make a feature request using the tracker on the
p3d project web site (Develop/Tracker/Feature request). Following the
description on how to add a new bug description we ask you, also in
this case, to be as detailed as possible. Please understand that we
are a team with limited resources. If your request concerns a huge
modification, which requires many days, or even months, to code, it
might not happen at all. On the other hand, we are highly interested
in taking on project members who are interested in contributing to p3d
with their own time and coding. Please drop us a note if you think
this could apply to you.
Alternatively, send an e-mail with your feature request to the e-mail
address provided in Section 2.
= 3) Software Platform =
p3d is nearly completely based on the language of the (proprietary)
Interactive Data Language (IDL; cf. http://www.exelisvis.com), and
cannot be used if IDL is unavailable ~ the easiest way to get IDL is
to ask us for the p3d-specific IDL distribution, see Section 1.2.
Although, it is necessary to have an IDL license to modify the code,
p3d can be run without an IDL-license using the supplied binary
meta-files (vm/p3d*.sav). In this case, using the, so-called, IDL
Virtual Machine; p3d has been implemented in such a way that it can be
used with full functionality without a license.
Since p3d is based on IDL it can be used on any platform supported by
IDL. However, the installation procedure differs slightly for an
installation on a Windows-based platform compared to on the other
supported platforms; the required environment variables need to be
setup differently (see above) and the compilation of the C routines
was not yet configured [please inform us if you are willing to help us
finish this task].
== 3.1) Coding Philosophy ==
p3d is a large and growing piece of software. To keep it manageable,
we have applied rules of coding practice; use of this practice has so
far made debugging easy as bugs are located quickly.
These are the rules, which we would ask any project programming member
to follow:
o All files must have filenames which conform to: p3d_task_name.pro
o Code layout:
+ All files must have a fully explained header.
+ All routines use the options: compile_opt hidden, IDL2
+ The maximum number of character columns per line is 79.
+ All blocks are indented by 2 characters.
+ Comment lines begin with ';;'
+ Integers are with few exceptions specified as long with L, e.g.
'0L'. Exceptions: VERBOSE and ERROR variables use integers.
+ All commands and statements are written with small characters.
o Common blocks are only allowed for variables which obviously
will never ever change, i.e. pretty much never.
o An error handler is setup for every routine, unless the user
keyword DEBUG is set.
o Logging comment statements must always use 'p3d_misc_logger',
adopting the different levels of verbosity.
o Input parameters should be checked for consistency in terms
of type, size, value and range, as far as possible.
o All command keywords are for clarity specified in full, i.e.
no abbreviations (for example, /xstyle not /xst).
o Text widgets must include error checking to ensure that entered
values are of the right type and range.
o All widgets in GUI tools with a status line must include the
UNAME field that specifies what the widget in questions does.
Issues related to the subversion repository of p3d:
o No file on the trunk should, as far as possible, be committed
before the modifications are confirmed to work as intended.
o All commits to the repository should include a description of the
modification (the longer the better, usually). An example is:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
svn ci --message 'Correct/modify this and that.' <file>
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
o Larger modifications should use a branch, and not the trunk.
o The idea is to copy the trunk to branch/ in preparations for a
release. Then we only make bug fixes on that branch before we
copy the final release version to tags/.
= 4) Supported Instruments =
We have developed p3d to be a general tool. General in the sense that
it can be used with any integral field unit (IFU) of fiber-fed
integral-field spectrographs. Additionally, the p3d spectrum viewer
can be used with a data cube of any origin, such as MUSE.
Before p3d can be used with any instrument, it must be properly setup.
p3d has been setup, tested, and found to work well with the following
instruments:
PMAS :: PPAK and LARR, for both the 2kx4k, and the newer 4kx4k
CCD.
VIRUS-P :: For fiber bundles 1 and 2.
SPIRAL :: For both the red and the blue arm.
VIMOS :: The MR, HR-Blue, HR-Orange, and HR-Red modes are
supported, for both the older configuration as well as
the refurbished instrument (2010.08- and 2012.03.15-).
There are no plans to include any support for the
low-resolution modes (LR).
FLAMES :: The ARGUS IFU and the two mini-IFU modes.
GMOS-S/N :: The red and the blue slit modes.
MPFS :: For the newer 3kx2k CCD.
VIRUS-W :: Both the low-resolution and the high-resolution modes.
INTEGRAL :: Both the older single CCD and the newer double CCD
setups are implemented.
ERA2 :: A life-sciences instrument at the AIP.
ERASMUS-F :: A test bench at the AIP.
MRS1 (MRS2) :: A life-sciences instrument at the AIP.
R10E-Atik :: A life-sciences instrument at the AIP.
PRAXIS :: An experimental setup using a Fiber Bragg Grating in
the near infrared.
SAMI :: Both the red and the blue arms are basically supported.
(IMACS :: For all (known) modes - under development)
See the p3d-project web page for more detailed information.
Specifically, have a look at the documentation WIKI for
instrument-specific recommendations for your data-reduction work:
https://sourceforge.net/p/p3d/wiki/Home/
Note! Cosmic-ray cleaning of single images and both approaches of
optimal extraction are supported on all instruments.
= 5) Recompiling the binary meta-files for use with the IDL VM =
The simplest way to (re-)compile the binary meta-files, which are used
with the shell scripts (and the IDL Virtual Machine), is to use the
provided script in p3d_path/vm/. An IDL license is required to
recompile p3d! The following steps could be used from a terminal:
1. :: Set the p3d root directory and the IDL_DIR environment
variables. In a "Bourne Shell"-type environment (e.g. bash), type:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
export p3d_path=/path/to/p3d/ # Optional
export IDL_DIR=/path/to/IDL/ # IDL version >=6.2 is required!
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
In a "C Shell"-type environment (e.g. csh), type:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
setenv p3d_path /path/to/p3d/ # Optional
setenv IDL_DIR /path/to/IDL/ # IDL version >=6.2 is required!
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
2. :: Download the required auxiliary files (unless this was already
done); this could, most conveniently, be done using the provided
Python tool:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
p3d_acquire_xtools.py
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Note! The binary meta-file generation script requires the additional
files to be present in p3d_path/contrib/astro-lib (with all
.pro-files in p3d_path/contrib/astro-lib/pro/, and
subdirectories) and in p3d_path/contrib/. It may sometimes
happen that a routine has been moved to another directory. In this
case, it will be necessary to modify the path of p3d - if the
development version of p3d is used, this might already have been
solved in a more recent revision.
3. :: Generate the binary meta-file in vm/p3d_*.sav; this is most
conveniently done using the provided Python tool:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
p3d_tool_makevm.py
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Note! If there is any problem in the binary meta-file generation
process, please check the output of the scripts to find out what
the problem is.
= 6) IDL Terms of use =
The following terms of use apply when using IDL; the text is pasted
from the web page of exelisvis:
IMPORTANT: READ CAREFULLY
Licensee acknowledges that it is not a citizen, national, or resident
of, and is not under the control of the government of Cuba, Iran,
North Korea, Sudan or Syria, and that Licensee will not download or
otherwise export or re-export the Software Product and any related
technical data directly or indirectly to the above mentioned countries
nor to citizens, nationals, or residents of those countries. Licensee
will comply with all applicable export, re-export and foreign policy
controls and restrictions imposed by the United States and will take
the necessary actions and precautions to ensure that it complies with
all such laws or regulations.
I am not located in and that I am not a national of Cuba, Iran,
North Korea, Syria, or Sudan, and I agree not to transfer the Software
Product to any such locations or individuals;
I am not listed or affiliated with anyone on any of the following
lists:
Denied Persons List - U.S. Department of Commerce
Unverified List - U.S. Department of Commerce
Entity List - Supplement No. 4 to part 744 of the Export
Administration Regulations
Specially Designated Nationals - U.S. Treasury
I also agree to comply with all U.S. and international laws (including
the obtaining of any required import license or authorization), and I
understand and agree that Exelis Visual Information Solutions is not
responsible for my failure to comply with any such laws.
You hereby certify that you are authorized to act on behalf of your
company.
I Agree
Cancel
View our Export Compliance document for information regarding the
export of the Software Product. We recommend you seek legal counsel if
you have any questions regarding the legality of your use of the
Software Product.
Download Latest Version
p3d-2.10.zip (48.3 MB)
