= p3d README =
Version information:
This is the release version 2.2.6 of p3d, "Serenity".
$Id: README 3221 2014-09-22 14:33:15Z christersandin $
= Table of Contents =
1.0) Introduction
o 1.1 About p3d
o 1.2 Installing p3d
2.0) Contact
o 2.1 Funding acknowledgments
o 2.2 Collaborators
o 2.3 Reporting Bugs
o 2.4 Feature Requests
3.0) Software Platform
o 3.1 Coding Philosophy
4.0) Supported Instruments
5.0) Recompiling the binary meta-file for the IDL Virtual Machine
= 1.0) Introduction =
== 1.1) About p3d ==
p3d is a data-reduction tool for fiber-fed integral-field
spectrographs. Please see the project web site for more details:
http://p3d.sourceforge.net.
== 1.2) Installing p3d ==
Installing p3d requires a few steps depending on if you only intend to
use it without an IDL license through the IDL Virtual Machine (see
3.0), or if you want to use it and develop the code with an available
IDL license.
In any case you need to have IDL (version>=6.2) and python
(2.6<=version<3.0) installed on your system. IDL can be downloaded
from the following web page (you have to register in order to be able
to download the software):
http://www.exelisvis.com/Downloads/ProductDownloads.aspx
See the relevant IDL documents on how to install IDL (i.e. expand or
copy the files to a proper location on your computer), and on how to
acquire and setup an IDL license (if you require one).
After you have installed IDL you might have to setup your environment
to tell the system where it can find IDL (unless your system
administrator has done this for you). The environment variable
IDL_DIR must be set to the name of the directory where your IDL
installation resides. Check if it is already set by entering the
following command (on *NIX-type platforms):
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
printenv IDL_DIR
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
If the output of the command points to the path of the IDL version you
want to use then there is no need to set the path again.
If IDL_DIR is not set, or if you have downloaded your own copy of IDL,
you can, depending on your system shell, add the following line to the
bash terminal setup file in your home directory (~/.bashrc):
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
export IDL_DIR=/path/to/IDL/
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
or if you use the csh (~/.cshrc):
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
setenv IDL_DIR /path/to/IDL/
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
where the tilde character (~) denotes your home directory.
Attention! If you want to use p3d on a Windows-based platform the
IDL_DIR environment variable is defined using Windows GUI tools. The
exact procedure varies depending on which version of Windows you are
using.
Also, check that the idl executable is available to you by entering
(this step is not required on Windows-based platforms):
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
which idl
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
If there is no output then you should create a symbolic link that
points at the IDL binary. You could do this using the following
example (assuming that you have a bin/ directory in your home
directory and that ~/bin exists in your path environment variable,
and that IDL resides in /opt/itt/idl):
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
cd ~/bin
ln -s /opt/itt/idl/bin/idl
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
If you then type "idl" at the prompt IDL should start. If you do not
have any IDL license you will get a lot of information about this fact
- but there is no need for concern about these messages if you intend
to run p3d using the IDL Virtual Machine.
=== i) Acquiring the p3d software ===
You can either download one of the prepared releases of p3d from the
program files area (https://sourceforge.net/projects/p3d/files/), or
you can download the latest bleeding edge development version using
subversion. The latter approach is preferred if you want new
functionality, which has not yet been included in the released
versions. Although, note that there is no guarantee that the code of
subversion is free of bugs (it is being corrected as soon as our time
permits, and as soon as we find the bugs).
To download p3d from the subversion repository at sourceforge.net,
open a terminal and change the directory to where you would like to
have the software (sub-directory). Thereafter, type the following
command:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
svn checkout svn://svn.code.sf.net/p/p3d/code/trunk p3d
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
whereby all p3d-related files are placed in the directory 'p3d/' (you
can of course use any name you wish for this directory). It is also
possible to use the http protocol instead of svn by replacing "svn:"
with "http:".
To update p3d when you use the development version, you can at any
time use your favorite terminal and change directory to p3d/ (or
whatever name you have used on this directory) and type 'svn update'.
For any additional functionality of subversion, including a complete
and very extensive manual, please have a look at its web page:
http://subversion.apache.org/.
Note! Due to updated software on the sourceforge.net side, before
7.12.2012, the repository address was different. The old and the new
repositories are incompatible, which is why it is necessary to
replace old repositories with one using the address above, to allow
any software updates.
Note! It is also possible to download all releases of p3d to the same
directory by leaving out the "/trunk" part in the command above. Few
users will want to do this.
You most likely want to use p3d with the IDL Virtual Machine (there is
no reason not to, unless you want to help us with debugging), and then
it is necessary to set the environment variable "p3d_path" to the name
of the directory where you have placed the p3d distribution.
Additionally, you might want to set the environment variable
"p3d_data_path" to the name of the (base) directory where you keep the
raw data that you will work with. You may also want to add the
${p3d_path}/vm/ directory to your environment path variable so that
you do not have to enter the full path of p3d every time you want to
run it. Depending on your system shell you can add the following lines
to a suitable place in your ~/.bashrc file (if you use bash; or a
"Bourne Shell"-type environment):
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
export p3d_path=/path/to/p3d/ # Modify to fit your system
export p3d_data_path=/path/to/IFU/data/ # Optional
export p3d_font=lucidasans-14 # Optional (see below)
export PATH=$PATH:${p3d_path}/vm # Optional
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
or ~/.cshrc (csh):
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
setenv p3d_path /path/to/p3d/ # Modify to fit your system
setenv p3d_data_path /path/to/IFU/data/ # Optional
setenv p3d_font lucidasans-14 # Optional (see below)
setenv PATH ${PATH}:${p3d_path}/vm # Optional
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Note! It is strictly only necessary to set the environment variable
p3d_path to the directory where p3d resides -if- you choose to run
p3d using the IDL Virtual Machine. However, if you would also like
to use some of the auxiliary scripts of p3d (such as
${p3d_path}/vm/p3d_acquire_xtools.py, cf. below), then it is still
necessary to set p3d_path (to tell the scripts where they can find
the necessary files).
Attention! If you want to use p3d on a Windows-based platform the
p3d_path environment variable is defined using Windows GUI tools.
The exact procedure varies depending on which version of Windows you
are using.
Windows XP: Control Panel --> System --> Advanced system settings
--> Environment Variables
Attention! Simply set the environment variable ${p3d_font} to the name
of an existing font (see the above example), if you want to use some
other font than the IDL default font. It is important to select a
font that exists on your system, check which ones are installed
using, e.g., the 'xlsfonts' command. You should, additionally make
sure that you do not use a too large font; because, if you do some
widgets might not fit inside the respective tool (if you do not
understand this point, try a very large font, start p3d and look at
the resulting windows). Due to restrictions in how IDL parses
command-line arguments, the font string must not contain any white
space!
=== ii) Using the IDL Virtual Machine ===
A (python) script is provided to run p3d from the shell, outside of
IDL, using the IDL Virtual Machine on all *NIX-type platforms, as well
as Windows. Although, before you can use the script you have to
prepare your system in a few steps. The environment variables IDL_DIR
and p3d_path must be set; according to the instructions above. And the
required mpfit C routines (cmpfit) need to be placed in the
${p3d_path}/contrib directory; see section iv) on how to acquire them.
To start the p3d main GUI you enter the following in a terminal (if
you added ${p3d_path}/vm to your system path):
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
p3d_dispatch.py <tool> <options>
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
or, if you did not add ${p3d_path}/vm to your path:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
${p3d_path}/vm/p3d_dispatch.py <tool> <options>
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
The tool can be selected as one of the following: p3d, p3d_rss,
p3d_cmbias, p3d_ctrace, p3d_cflatf, p3d_cobjex, p3d_cvimos_combine,
p3d_cr, p3d_darc, p3d_rss2cube, p3d_cexposure, p3d_fluxsens, and
p3d_fluxcal (more will be added). It is also possible, but not
recommended, to use any of the older bash-based scripts (which all
call p3d_dispatch.py). These are available for compatibility with
already written reduction scripts.
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
${p3d_path}/vm/p3d_vm.sh [-rt] <options>
${p3d_path}/vm/p3d_cmbias_vm.sh [-rt] <options>
${p3d_path}/vm/p3d_ctrace_vm.sh [-rt] <options>
${p3d_path}/vm/p3d_cdmask_vm.sh [-rt] <options>
${p3d_path}/vm/p3d_cflatf_vm.sh [-rt] <options>
${p3d_path}/vm/p3d_cobjex_vm.sh [-rt] <options>
${p3d_path}/vm/p3d_cvimos_combine_vm.sh [-rt] <options>
${p3d_path}/vm/p3d_rss_vm.sh [-rt] <options>
${p3d_path}/vm/p3d_cr_vm.sh [-rt] <options>
${p3d_path}/vm/p3d_darc_vm.sh [-rt] <options>
${p3d_path}/vm/p3d_rss2cube_vm.sh [-rt] <options>
${p3d_path}/vm/p3d_cexposure_vm.sh [-rt] <options>
${p3d_path}/vm/p3d_fluxsens_vm.sh [-rt] <options>
${p3d_path}/vm/p3d_fluxcal_vm.sh [-rt] <options>
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
See the respective routine for detailed information regarding the
available options:
p3d.pro :: (and p3d_gui.pro; except for the DISPLIB and BOX
parameters) for the p3d GUI.
p3d_cmbias.pro :: to create a master-bias image.
p3d_ctrace.pro :: to create a trace-mask image.
p3d_cdmask.pro :: to create a dispersion-mask image.
p3d_cflatf.pro :: to create a flat-field image.
p3d_cobjex.pro :: to extract science-object spectra.
p3d_fluxsens.pro :: to create a sensitivity function from a summed
standard-star spectrum. That sensitivity function
is then used to flux calibrate the data.
p3d_fluxcal.pro :: to flux calibrate extracted data using the
sensitivity function that is created with
p3d_fluxsens.
p3d_cvimos_combine.pro :: to combine the extracted VIMOS spectra of
the four separate quadrants into one spectrum
image.
p3d_rss.pro :: to view the extracted spectra.
p3d_cr.pro :: to clean single images of cosmic-ray hits.
p3d_darc.pro :: to correct the data for effects of differential
atmospheric refraction; only works with
instruments that use square-shaped elements
currently.
p3d_rss2cube.pro :: to convert the resulting RSS-formatted images
into a three-dimensional cube; only works with
instruments that use square-shaped elements
currently.
p3d_cexposure.pro :: to combine multiple extracted images; can
select files that used different exposure times to
replace data in extracted spectrum elements that
involved saturated regions in the raw data.
Note! The options are specified in the usual IDL way, but with a space
between the options (instead of a comma, as is the case when p3d is
used from the IDL command line).
Note! Some arguments and keywords accept multiple files. Such files
are separated with a comma.
Note! All of the shell scripts can be called with the -rt option. In
this case an IDL license is required. This behavior could be useful
if you want to avoid having to click in the splash window that
always appears each time the IDL VM is launched. Although, with the
p3d queue server system the need to click on those splash screens is
delimited to when the p3d queue system is first launched.
Here are two examples to get you started:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
${p3d_path}/vm/p3d_dispatch.py p3d /cwd /ppak4k logfile=dred.log
opath=output_directory
${p3d_path}/vm/p3d_dispatch.py p3d_cobjex \
file1.tar.gz,file2.tar.gz \
opath=$opath masterbias=${opath}/masterbias.fits.gz \
tracemask=${opath}/tracemask.fits.gz \
flatfield=${opath}/flatfield.fits.gz \
userparfile=${opath}/user_p3d.par logfile=${opath}/dred.log \
loglevel=2 /verbose /quiet
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
The p3d IDL server allows you to submit any number of jobs to the p3d
queue. The p3d queue handler - which is launched automatically -
feeds the jobs to the running p3d servers, and continues until all
jobs are done.
Note! The p3d_dispatch.py program checks if the required IDL binary
meta-file (p3d_tool_server.sav) is present in the directory
${p3d_path}/vm/. IDL is launched with that file if it exists.
Note! The program ${p3d_path}/vm/p3d_dispatch.py requires that python
is installed on your computer, versions >=2.6 and <3.0 will do. Also
note that, if the command /usr/bin/env does not exist, it is
necessary to launch this (as well as the other) python program(s)
preceeded by the name of the python command (with the full path):
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
python ${p3d_path}/vm/p3d_dispatch.py
python ${p3d_path}/vm/p3d_acquire_xtools.py
python ${p3d_path}/vm/p3d_tool_makevm.py
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
If you want to use the bash scripts instead, it is necessary to have
bash installed as /bin/bash on your computer. Please, edit the first
line in the scripts accordingly if this is not the case (i.e. the
first line stating #!/bin/bash), or execute them as in the following
examples:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
bash ${p3d_path}/vm/p3d_vm.sh
bash ${p3d_path}/vm/p3d_cmbias_vm.sh
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Note! If you have an IDL license, and want to recompile the binary
files for use with, or without, the IDL Virtual Machine, it is
strongly recommended to use the provided bash script
${p3d_path}/vm/p3d_tool_makevm.py (see below for further
instructions, section 5.0).
Note! The debug option (/DEBUG) is unavailable when using the IDL VM
(also when the -rt option is used). However, it is still possible to
select not-yet-fully supported instruments from the command line -
that can otherwise only be selected with the instrument-selection
GUI - when /DEBUG is set.
=== iii) Using an IDL license ===
In order to run p3d using an IDL license it is necessary to tell IDL
where it can find all the required source files. In addition to all
files of the p3d distribution, the files of the following sources are
required:
o Astro-lib (http://idlastro.gsfc.nasa.gov/homepage.html)
o All files (because it is easier to download all files).
o Mpfit (http://purl.com/net/mpfit)
o mpfit.pro
o cmpfit-1.2.tar.gz
These files can, with the exception of the cmpfit tarball, be put
anywhere you like. If you want to use the provided scripts when
generating a binary meta-file for use with the IDL VM shell scripts
(if you made any changes to the code), it is recommended to download
these files to ${p3d_path}/contrib/astro-lib/ (for the files of
Astro-lib) and to ${p3d_path}/contrib/ (mpfit.pro). The file
cmpfit-1.2.tar.gz must be unpacked in ${p3d_path)/src/, there creating
the subdirectory cmpfit-1.2. You can download all these files
automatically (to these directories), if you use the provided python
script:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
${p3d_path}/vm/p3d_acquire_xtools.py
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
When all files are downloaded, or if they are already present on the
system, it is necessary to setup IDL so that it finds them. The most
straightforward way is to set the environment variable IDL_STARTUP
to the name of an IDL program where the necessary information is
defined. The following lines can be used to do this (the filename used
here "IDL_STARTUP.PRO" can of course be chosen freely):
For a "Bourne Shell"-type environment, add these lines to the file
~/.bashrc (please modify the paths to fit your system):
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
export IDL_STARTUP=/path/to/IDL_STARTUP.PRO # Modify for your system
export p3d_data_path=/path/to/IFU/data/ # Optional
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
For a "C Shell"-type environment, add these lines to the file ~/.cshrc
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
setenv IDL_STARTUP /path/to/IDL_STARTUP.PRO # Modify for your system
setenv p3d_data_path /path/to/IFU/data/ # Optional
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Thereafter, add the following lines to IDL_STARTUP.PRO:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
astrolib_path='/path/to/astrolib/root/dir' ; Modify for your system
markwardt_path='/path/to/markwardt/dir' ; Modify for your system
p3d_path='/path/to/p3d' ; Modify for your system
!path=astrolib_path+'/pro/jhuapl:'+!path
!path=markwardt_path+':'+astrolib_path+'/pro:'+!path
!path=p3d_path+':'+p3d_path+'/routines:'+!path
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
If you chose to put the additionally required files under
${p3d_path}/contrib, as we recommend you to do, you could instead add
the following lines to IDL_STARTUP.PRO:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
p3d_path='/path/to/p3d' ; Modify for your system
!path=p3d_path+'/contrib/astro-lib/pro/jhuapl:'+!path
!path=p3d_path+'/contrib:'+p3d_path+'/contrib/astro-lib/pro:'+!path
!path=p3d_path+':'+p3d_path+'/routines:'+!path
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
p3d can now be started from the IDL command line using the command
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
p3d
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
If you would like to know the available options to the routine
p3d.pro, you could just have a look in the program file (using your
favorite text editor, or e.g. more or less), or type:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
p3d,/help
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
The /help option is available with all routines of p3d.
Note! The exact order of entries in the IDL path is important. Also,
observe that no extra directory separator ('/') should be used in
the final position of the path entries (i.e., '/path/to/p3d' not
'/path/to/p3d/').
Note! Do not include the ${p3d_path}/vm/ directory in the IDL !path
variable! If you do this then there will be a confusion in the path
if you modify any of the program routines. So, even though you have
made changes to a p3d routine, IDL might choose to use the
pre-compiled routines in the binary files ${p3d_path}/vm/p3d*.sav.
If you do not remove vm/ from the path you have to recompile all the
routines when you make any changes. Avoid these problems by -not-
having ${p3d_path}/vm/ in !path.
Note! Starting with revision 516 the astro-lib sub-directory "jhuapl"
must be added to the IDL path; because, the routine REPCHR.PRO has
moved to that directory. If the astro-lib routines are updated then
it is necessary to make this change. Update 29.5.2013: REPCHR.PRO
was moved back to its original directory on 16.5.2013; this is
accounted for in revisions >=2770.
=== iv) Additional software that allows p3d to run faster ===
Some of the routines are, in addition to their IDL implementation,
also provided in C. The reason is that compiled C code in
computationally intensive cases (involving nested loops) run faster
than the corresponding IDL code. Additionally, in C routines it is
possible to make use of more of the computing power on a
multi-processor computer; through OpenMP (http://www.openmp.org).
Currently, the following tasks are available in both C and IDL code:
o The calculation of cross-dispersion line profiles.
o The calculation of a median-filtered science image, in regions
where there are no spectra, to subtract a scattered-light, or
a dark-current, component.
o The spectrum extraction, using optimal extraction with the
multi-profile decvonvolution method and the band-diagonal option.
o The conversion from pixel space to spatial-element space when
selecting a region of interest in the spectrum viewer.
o The calculation of pixel-based cross-dispersion profile images
that are used in the optimal extraction.
o The combination of multiple images using the min/max filter,
and the "slowimcombine" mode.
o Cosmic-ray cleaning of single images.
o The resampling during wavelength calibration from pixel-based
coordinates to wavelength.
These particular routines were selected because their IDL
implementations are more-to-extremely time-consuming.
The C routines are compiled automatically by p3d whenever they are
required. All available threads on a multi-processor computer are used
by default, use the NTHREADS and CRNTHREADS keywords to change this
number. To use the compiled C routines, make sure that you have a
working compiler, and the necessary source code in ${p3d_path}/src/.
Only cmpfit has to be downloaded separately, outside of p3d,
currently. If the compilation fails, for any reason, p3d falls back to
use the corresponding IDL version. Use the /DEBUG and the VERBOSE
flags to make the compilation process more verbose. It is also
possible to completely deactivate the C routines by setting the user
parameter 'dontusecroutines' to 'yes'. The C-based cosmic-ray cleaning
routine can be deactivated separately by setting the NOCRC keyword
(NOC with p3d_cr) or the 'nocrc' user parameter.
Note! If one or several of the C routines in ${p3d_path} are updated
after the shared library is built, it is necessary to either set the
rebuild library parameter ('rebuildlib yes') or to remove the
library file ${p3d_path}/src/p3d_clib[.so,.dll]. p3d then compiles
a new library file from the new source files.
Note! The C routines are parallelized using OpenMP. In the case that
you get messages indicating that OpenMP-related routines are not
found, please try a more recent compiler.
Note! The C routines are currently only compiled on Linux and MacOSX
platforms. Please contact us if you need support on one of the other
platforms; i.e. Windows and Solaris. In principle it is only a
question of adding an appropriate case-statement block in the
file "p3d_misc_get_callext_exlib.pro", but it can get complicated
if you are using a non-standard compiler. Any input on this issue
is well received.
Warning! If you run IDL on a platform where you switch between the
32 bit and the 64 bit IDL environments, please remember to delete
the shared-library files between the switches, otherwise the
shared library will be of an incompatible architecture. p3d can
use its scripts with the 32 bit environment on 64 bit machines if
the IDL_OPTS environment variable is set to "-32". However, it is
not recommended to do this switching between 32 and 64 bits, as
the outcome is system-dependent and therefore generally
unpredictable.
= 2.0) Contact =
All collaborators, unless noted otherwise, have the following
affiliation:
Leibniz Institute for Astrophysics Potsdam (AIP)
Instrumentation
An der Sternwarte 16
D-14482 Potsdam
GERMANY
Tel: +49-(0)331-74990
http://www.aip.de
The p3d-project web site is:
http://p3d.sourceforge.net
where you can download the software, view the documentation, and much
more.
Documentation is available in the individual routines - the same text
is also available on the Documentation page at the project web site;
this documentation is updated whenever there is a release. Detailed
descriptions of the algorithms are provided in the (freely available)
Astronomy & Astrophysics paper about p3d:
Sandin et al. 2010, A&A, 515, 35
http://www.aanda.org/articles/aa/pdf/2010/07/aa14022-10.pdf
== 2.1) Funding acknowledgments ==
This project has been made possible through funds from the following
sources:
BMBF: 03Z2A51, 05A08BA1
== 2.2) Collaborators ==
=== All programming contributors of p3d ===
Christer Sandin :: Project leader and programmer.
If you have comments of bugs or feature
requests, which are directly related to p3d, and
that you cannot add to the tracker at the
project web site (see below for instructions)
then please send them to:
Christer Sandin <CSandin@aip.de>
Fachreddin Tabataba-Vakili :: Has ported the multi-profile
deconvolution spectrum extraction routine from
IDL (p3d_extract_optimal_mpd.pro) to
automatically compiled C (p3d_cmpd.c), which is
also parallelized with OpenMP. He has also made
the main work on the inclusion of interactive
flux calibration (using a graphical user
interface) in p3d. He has written a tool that
generates IDL runtime distributions, and the
framework tools around the p3d IDL server. He is
finally working on the conversion of various
slow routines to compiled and parallelized C.
Peter Weilbacher :: Has ported the slowest routine of p3d, the
cross-dispersion line profile fitting routine,
to C, which is also parallelized using OpenMP.
Has also included an option to calibrate the
dispersion mask against sky emission lines. He
also reviews the routine documentation.
Ole Streicher :: Has provided p3d with a first version of the
tutorials at the project web site. Has also
written various tools that automatically
translate the routine documentation into xhtml
documents, which can be viewed at the project
documentation web page:
http://p3d.sourceforge.net/index.php?page=doc
=== Additional acknowledgments ===
Please see the p3d project web site for further acknowledgments.
== 2.3) Reporting Bugs ==
In order to report any kind of bug we urge you to use the bug tracker
that is available at the p3d-project web site (Develop/Tracker/Bugs).
This way your bug will be seen by all users and programmers - and it
will not be lost in an e-mail program.
At the bug-tracker page you can search previously reported bugs, to
see if your bug has already been reported. Or you can open your own
bug description (by clicking on "Add new"). Doing this, please make
sure that you are using the latest version on the subversion trunk of
p3d (or specify the revision or version of the release you are using).
Thereafter specify a category for the bug, and provide a short summary
of the problem, and most importantly, describe the problem with enough
detail that somebody who does not use your setup and your data files
is able to follow your arguments. Bug reports without a useful
description will be assigned a low priority, or even be deleted. If
you think that it would help us, attach all the relevant data files to
your report (please compress fits-files using tar/gzip/bzip2, zip, or
use the compress-option of p3d to save space). Additionally, please
specify a reasonable priority to your bug. While our intention is to
correct serious bugs, which prevent p3d from producing spectra, as
soon as possible, we will assign GUI-related issues a lower priority.
Note! When reporting a bug, turn on the debugging mode (/DEBUG) of p3d
and paste the resulting output from the IDL terminal/window (once
the bug has been reached) into your bug description. The reason we
ask you to do this is that the information returned by the error
handler, when the debug keyword is unset, only provides the name of
the file where the problem occurred, and an IDL-specific number.
Such sparse information is nearly always impossible to interpret.
== 2.4) Feature requests ==
If p3d is missing some functionality that you would very much like to
have included, then you can make a feature request using the tracker
on the p3d project web site (Develop/Tracker/Feature request).
Following the description on how to add a new bug description we ask
you, also in this case, to be as detailed as possible. Please
understand that we are a team with limited resources. If your request
concerns a huge modification, which requires many days, or even
months, to code it might not happen at all. On the other hand, we are
highly interested in taking on project members who are interested in
contributing to p3d with their own time and coding. Please drop us a
note if you think this could apply to you.
= 3.0) Software Platform =
p3d is nearly completely based on the language of the (proprietary)
Interactive Data Language (IDL; cf. http://www.exelisvis.com), and
cannot be used if IDL is not installed on the system. Although, it is
necessary to have an IDL license in order to modify the code, p3d can
be run without an IDL-license using the supplied binary meta-files
(${p3d_path}/vm/p3d*.sav'). In this case using the, so-called,
IDL Virtual Machine; p3d has been implemented in such a way that it
can be used with full functionality without a license.
Since p3d is based on IDL it can be used on any platform that is
supported by IDL. Note, however, that the installation procedure
differs for an installation on a Windows-based platform compared to
that of the other supported platforms (the Windows version of the
installation instructions is not yet finished [please tell us if you
would be willing to help us out on this issue and write one]).
== 3.1) Coding Philosophy ==
p3d is a large, and growing, piece of software. In order to keep it
manageable we have applied rules of coding practice; use of this
practice has so far made debugging easy as bugs are located quickly.
These are the rules, which we would ask any project programming member
to follow:
o All files must have filenames which conform to: p3d_task_name.pro
o Code layout:
+ All files must have a fully explained header.
+ All routines use the options: compile_opt hidden,IDL2
+ The maximum number of character columns per line is 79.
+ All blocks are indented by 2 characters.
+ Comment lines begin with ';;'
+ Integers are always specified as long with L, e.g. '0L'.
Exceptions: VERBOSE and ERROR variables use integers.
+ All commands and statements are written with small characters.
o Common blocks are only allowed for variables which obviously
will never ever change, i.e. pretty much never.
o An error handler is setup for every routine, unless the user
keyword DEBUG is set.
o Logging comment statements must always use 'p3d_misc_logger'.
o Input parameters should be checked for consistency in terms
of type, size, value and range, as far as possible.
o All command keywords are for clarity specified in full, i.e.
no abbreviations (for example, /XSTYLE not /XST).
o Text widgets must include error checking to ensure that the
entered values are of the right type and range.
o All widgets in GUI tools with a status line must include the
UNAME field that specifies what the widget in questions does.
Issues related to the subversion repository of p3d:
o No file on the trunk should, as far as possible, be committed
before the modifications are confirmed to work as intended.
o All commits to the repository should include a description of the
modification (the longer the better, usually). An example is:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
svn ci --message 'Correct/modify this and that.' <file>
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
o Larger modifications should use a branch, and not the trunk.
o The idea is to copy the trunk to branch/ in preparations for a
release. Then we only make bug fixes on that branch before we
copy the final release version to tags/.
= 4.0) Supported Instruments =
We have developed p3d to be a general tool. General in the sense that
it can be used with any integral field unit (IFU) of fiber-fed
integral-field spectrographs. However, before p3d can be used with any
instrument it must be properly setup. p3d has been setup, tested, and
found to work well with the following instruments:
PMAS :: PPAK and LARR, for both the 2kx4k, and the newer 4kx4k
CCD.
VIRUS-P :: For fiber bundles 1 and 2.
SPIRAL :: For both the red and the blue arm.
VIMOS :: The MR, HR-Blue, HR-Orange, and HR-Red modes are
supported, for both the older configuration as well as
the refurbished instrument (2010.08-). There are no plans
to include support for the low-resolution modes (LR).
FLAMES :: The ARGUS IFU, and the two mini-IFU modes.
GMOS-S/N :: The red and the blue slit modes.
MPFS :: For the newer 3kx2k CCD.
VIRUS-W :: Both the low-resolution and the high-resolution modes.
INTEGRAL :: Both the older single CCD and the newer double CCD setups
are implemented.
ERA2 :: A life-sciences instrument at AIP.
SAMI :: Both the red and the blue arms are basically supported,
but information regarding fiber positions is missing.
See the p3d-project web page for more detailed information.
Specifically, have a look at the documentation WIKI for
instrument-specific recommendations for your data-reduction work:
http://sourceforge.net/apps/mediawiki/p3d/index.php
Note! Cosmic-ray cleaning of single images and both approaches of
optimal extraction are supported on all instruments.
= 5.0) Recompiling the binary meta-files for use with the IDL VM =
The simplest way to (re-)compile the binary meta-files, which are used
with the shell scripts (and the IDL Virtual Machine), is to use the
provided script in ${p3d_path}/vm/. In order to recompile p3d you must
have an IDL license! You could use the following steps from a
terminal:
1. :: Set the p3d root directory and the IDL_DIR environment
variables. In a "Bourne Shell"-type environment (e.g. bash) type:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
export p3d_path=/path/to/p3d/
export IDL_DIR=/path/to/IDL/ # IDL version >=6.2 is required!
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
In a "C Shell"-type environment (e.g. csh) type:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
setenv p3d_path /path/to/p3d/
setenv IDL_DIR /path/to/IDL/ # IDL version >=6.2 is required!
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
2. :: Download the required auxiliary files. You could, most
conveniently, use the provided python script:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
${p3d_path}/vm/p3d_acquire_xtools.py
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Note! The binary meta-file generation script requires the additional
files to be present in ${p3d_path}/contrib/astro-lib (with all
.pro-files in ${p3d_path}/contrib/astro-lib/pro/, and
subdirectories) and in ${p3d_path}/contrib/. It may sometimes
happen that a routine has been moved to another directory. In this
case it will be necessary to modify the path of p3d - if you use
the development version of p3d this might already have been solved
in a more recent revision.
3. :: Generate the binary meta-file
${p3d_path}/vm/p3d_tool_server.sav. You could, most conveniently,
use the provided python script:
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
${p3d_path}/vm/p3d_tool_makevm.py
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Note! If there is any problem in the binary meta-file generation
process, please check the output of the scripts to find out what
the problem is.
Download Latest Version
p3d-2.10.zip (48.3 MB)
