Menu
▾
▴
netcdftools-commit
|
From: <rob...@us...> - 2010-06-18 01:28:54
|
Revision: 8
http://netcdftools.svn.sourceforge.net/netcdftools/?rev=8&view=rev
Author: robertbridle
Date: 2010-06-18 01:28:48 +0000 (Fri, 18 Jun 2010)
Log Message:
-----------
ANDSWRON-618 - Change URL from generated site to point to SourceForge's download page.
Modified Paths:
--------------
trunk/pom.xml
trunk/src/site/apt/index.apt
Modified: trunk/pom.xml
===================================================================
--- trunk/pom.xml 2010-06-17 07:34:10 UTC (rev 7)
+++ trunk/pom.xml 2010-06-18 01:28:48 UTC (rev 8)
@@ -169,8 +169,10 @@
</execution>
</executions>
</plugin>
-
+
+ <!-- WE NO LONGER COPY THESE TO THE GENERATED SITE, WE NOW POINT THE SITE TO THE SOURCEFORGE DOWNLOAD URL -->
<!-- Include the created artifacts in the site -->
+ <!--
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId>
@@ -197,8 +199,9 @@
</configuration>
</execution>
</executions>
- </plugin>
-
+ </plugin>
+ -->
+
</plugins>
</build>
Modified: trunk/src/site/apt/index.apt
===================================================================
--- trunk/src/site/apt/index.apt 2010-06-17 07:34:10 UTC (rev 7)
+++ trunk/src/site/apt/index.apt 2010-06-18 01:28:48 UTC (rev 8)
@@ -24,7 +24,7 @@
Download
- You can download the tools set from the {{{./downloads}downloads folder}}.
+ You can download the tools set from the {{{https://sourceforge.net/projects/netcdftools/files/}downloads folder}}.
For each version there will be at least three files:
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <jde...@us...> - 2010-06-22 03:09:00
|
Revision: 17
http://netcdftools.svn.sourceforge.net/netcdftools/?rev=17&view=rev
Author: jdempsey
Date: 2010-06-22 02:24:18 +0000 (Tue, 22 Jun 2010)
Log Message:
-----------
ANDSWRON-657 - Allow defineAttributes to manipulate multiple files
Modified Paths:
--------------
trunk/USAGE.txt
trunk/pom.xml
trunk/src/main/java/au/csiro/netcdf/NcDefineAttributes.java
trunk/src/main/java/au/csiro/netcdf/util/Util.java
Modified: trunk/USAGE.txt
===================================================================
--- trunk/USAGE.txt 2010-06-21 06:41:03 UTC (rev 16)
+++ trunk/USAGE.txt 2010-06-22 02:24:18 UTC (rev 17)
@@ -49,21 +49,27 @@
ncdefineAtt
Define a global attribute in a netCDF file.
--o,--outputFileName <file> 1: the filename of the netCDF file to be created.
--i,--inputFileName <file> 2. the filename of a text file containing
- attributes to be loaded. OPTIONAL, ensure that text
- containing '=' or ',' characters are delimited by a
- backslash.
--s,--standardInput 3: OPTIONAL, read attributes from Stdin, ensure
- that text containing '=' or ',' characters are
- delimited by a backslash.
--a,--globalAttributes <text> 4: a comma separated list of attribute-value pairs,
- OPTIONAL, e.g. "netCDF
- Java=4.0,Conventions=CF-1.0", ensure that text
- containing '=' or ',' characters are delimited by a
- backslash.
--l,--largeFileSupport 5: OPTIONAL, set if more than 2 GB of data will
- need to be stored in this file.
+-o,--outputFileName <file> 1: the filename of the netCDF file to be
+ created.
+-i,--inputFileName <file> 2. the filename of a text file containing
+ attributes to be loaded. OPTIONAL, ensure that
+ text containing '=' or ',' characters are
+ delimited by a backslash.
+-s,--standardInput 3: OPTIONAL, read attributes from Stdin, ensure
+ that text containing '=' or ',' characters are
+ delimited by a backslash.
+-a,--globalAttributes <text> 4: a comma separated list of attribute-value
+ pairs, OPTIONAL, e.g. "netCDF
+ Java=4.0,Conventions=CF-1.0", ensure that text
+ containing '=' or ',' characters are delimited
+ by a backslash.
+-l,--largeFileSupport 5: OPTIONAL, set if more than 2 GB of data will
+ need to be stored in this file.
+-p,--pattern <filename pattern> 6: OPTIONAL, a pattern to match multiple
+ existing files, where multiple files should be
+ processed. If this option is present the
+ outptuFilename is expected to be a direcotry
+ containing the files to be processed.
Example: ncdefineatt -outputFileName ABC.nc -globalAttributes
"netCDF-Java=4.0,Conventions=value with a space"
@@ -85,7 +91,7 @@
delimited by a backslash.
-v,--variableName <text> 4: the name to be given to the variable.
-t,--variableDataType <text> 5: the data type of the variable, e.g. [int,
- byte, char, short, double, float]
+ double, char, float, short, byte]
-a,--variableAttributes <text> 6: a comma separated list of attribute-value
pairs, OPTIONAL, e.g. "units=mm", ensure that
text containing '=' or ',' characters are
Modified: trunk/pom.xml
===================================================================
--- trunk/pom.xml 2010-06-21 06:41:03 UTC (rev 16)
+++ trunk/pom.xml 2010-06-22 02:24:18 UTC (rev 17)
@@ -349,6 +349,11 @@
<artifactId>commons-cli</artifactId>
<version>1.2</version>
</dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ <version>1.4</version>
+ </dependency>
</dependencies>
<reporting>
<plugins>
Modified: trunk/src/main/java/au/csiro/netcdf/NcDefineAttributes.java
===================================================================
--- trunk/src/main/java/au/csiro/netcdf/NcDefineAttributes.java 2010-06-21 06:41:03 UTC (rev 16)
+++ trunk/src/main/java/au/csiro/netcdf/NcDefineAttributes.java 2010-06-22 02:24:18 UTC (rev 17)
@@ -87,6 +87,11 @@
public static final String IS_LARGE_FILE = "largeFileSupport";
/**
+ * The pattern for accepted file names where multiple files are accepted.
+ */
+ public static final String FILENAME_PATTERN = "pattern";
+
+ /**
* Constant that defines the logger to be used.
*/
private static final Logger LOG = Logger.getLogger(NcDefineAttributes.class.getName());
@@ -124,6 +129,8 @@
String globalAttributesArg = (parsedCommandLine.hasOption(GLOBAL_ATTRIBUTES)) ? parsedCommandLine
.getOptionValue(GLOBAL_ATTRIBUTES) : "";
boolean isLargeFileSupport = parsedCommandLine.hasOption(IS_LARGE_FILE);
+ String pattern = (parsedCommandLine.hasOption(FILENAME_PATTERN)) ? parsedCommandLine
+ .getOptionValue(FILENAME_PATTERN) : "";
// check whether large file support is needed
if (Util.fileExists(outputFilenameArg)
@@ -134,6 +141,8 @@
+ " flag.");
}
+ List<String> targetFileNames = Util.getListOfTargetFiles(outputFilenameArg, pattern);
+
// check that if an input file is specified then it actually exists.
if (!inputFilenameArg.isEmpty())
{
@@ -165,7 +174,11 @@
{
globalAttributes.addAll(NetCDFUtils.readAttributesFromStream(System.in));
}
- this.execute(outputFilenameArg, globalAttributes, isLargeFileSupport);
+
+ for (String outputFileName : targetFileNames)
+ {
+ this.execute(outputFileName, globalAttributes, isLargeFileSupport);
+ }
}
/**
@@ -348,6 +361,12 @@
"5: OPTIONAL, set if more than 2 GB of data will need to be stored in this file.").isRequired(false)
.withLongOpt(IS_LARGE_FILE).create("l");
+ Option pattern = OptionBuilder.withArgName("filename pattern").hasArg().withDescription(
+ "6: OPTIONAL, a pattern to match multiple existing files, where multiple files should be "
+ + "processed. If this option is present the outptuFilename is expected to be a direcotry "
+ + "containing the files to be processed.").isRequired(false).withLongOpt(FILENAME_PATTERN)
+ .create("p");
+
Options options = new Options();
options.addOption(outputFileName);
@@ -355,6 +374,7 @@
options.addOption(standardInput);
options.addOption(globalAttributes);
options.addOption(largeFileSupport);
+ options.addOption(pattern);
return options;
}
@@ -392,9 +412,11 @@
String globalAttributesArg = (parsedCommandLine.hasOption(GLOBAL_ATTRIBUTES)) ? parsedCommandLine
.getOptionValue(GLOBAL_ATTRIBUTES) : "";
boolean isLargeFileSupport = parsedCommandLine.hasOption(IS_LARGE_FILE);
+ String pattern = (parsedCommandLine.hasOption(FILENAME_PATTERN)) ? parsedCommandLine
+ .getOptionValue(FILENAME_PATTERN) : "";
// check whether large file support is needed
- if (Util.fileExists(outputFilenameArg)
+ if (pattern.length() == 0 && Util.fileExists(outputFilenameArg)
&& Util.getExistingFile(outputFilenameArg).length() >= MAX_32BIT_OFFSET_FILE_SIZE
&& !isLargeFileSupport)
{
@@ -402,6 +424,22 @@
+ " flag.");
}
+ // Check that if a pattern is present that the output file is a directory
+ if (pattern.length() > 0)
+ {
+ if (!Util.fileExists(outputFilenameArg))
+ {
+ throw new IllegalArgumentException("When a pattern is specified " +
+ "the output file must be an existing directory.");
+ }
+ File file = new File(outputFilenameArg);
+ if (!file.isDirectory())
+ {
+ throw new IllegalArgumentException("When a pattern is specified "
+ + "the output file must be an existing directory.");
+ }
+ }
+
// check that if an input file is specified then it actually exists.
if (!inputFilenameArg.isEmpty())
{
Modified: trunk/src/main/java/au/csiro/netcdf/util/Util.java
===================================================================
--- trunk/src/main/java/au/csiro/netcdf/util/Util.java 2010-06-21 06:41:03 UTC (rev 16)
+++ trunk/src/main/java/au/csiro/netcdf/util/Util.java 2010-06-22 02:24:18 UTC (rev 17)
@@ -17,15 +17,19 @@
package au.csiro.netcdf.util;
import java.io.File;
+import java.io.FileFilter;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
+import java.util.Collections;
import java.util.LinkedList;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
+import org.apache.commons.io.filefilter.WildcardFileFilter;
+
/**
* Util - utility class with various helpful tools...
*
@@ -202,4 +206,40 @@
splitted.add(text.substring(last_match));
return splitted.toArray(new String[splitted.size()]);
}
+
+ /**
+ * Retrieve a list of the absolute path names of the files in a folder that match a pattern. The pattern uses the
+ * characters '?' and '*' to represent a single or multiple wildcard characters. This is the same as often found on
+ * Dos/Unix command lines.
+ *
+ * @param foldername
+ * The name of the folder to be searched or the single file if no pattern
+ * @param pattern
+ * The pattern to be matched against. The wildcard characters '?' and '*' are supported.
+ * @return A list of the absolute names of the matching files
+ */
+ public static List<String> getListOfTargetFiles(String foldername, String pattern)
+ {
+ List<String> fileNameList = new ArrayList<String>();
+ if (pattern == null || pattern.length() == 0)
+ {
+ fileNameList.add(foldername);
+ return fileNameList;
+ }
+
+ File dir = new File(foldername);
+ if (!dir.exists() || !dir.isDirectory() || !dir.canRead())
+ {
+ return Collections.emptyList();
+ }
+
+ FileFilter patternFilter = new WildcardFileFilter(pattern);
+ File[] files = dir.listFiles(patternFilter);
+ for (int i = 0; i < files.length; i++)
+ {
+ fileNameList.add(files[i].getAbsolutePath());
+ }
+
+ return fileNameList;
+ }
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <rit...@us...> - 2010-06-22 06:52:46
|
Revision: 18
http://netcdftools.svn.sourceforge.net/netcdftools/?rev=18&view=rev
Author: ritacsiro
Date: 2010-06-22 06:52:39 +0000 (Tue, 22 Jun 2010)
Log Message:
-----------
ANDSWRON-678 - Add variable parameter to defineAttributes.
Modified Paths:
--------------
trunk/USAGE.txt
trunk/src/main/java/au/csiro/netcdf/NcDefineAttributes.java
Modified: trunk/USAGE.txt
===================================================================
--- trunk/USAGE.txt 2010-06-22 02:24:18 UTC (rev 17)
+++ trunk/USAGE.txt 2010-06-22 06:52:39 UTC (rev 18)
@@ -49,6 +49,8 @@
ncdefineAtt
Define a global attribute in a netCDF file.
+If variable is not specified, then a global attribute is defined.
+If the value of the attribute is null, then the attribute will be deleted.
-o,--outputFileName <file> 1: the filename of the netCDF file to be
created.
-i,--inputFileName <file> 2. the filename of a text file containing
@@ -70,11 +72,13 @@
processed. If this option is present the
outptuFilename is expected to be a direcotry
containing the files to be processed.
+-v,--variable <text> 7: OPTIONAL, the variable to assign attribute(s)
+ to.
-Example: ncdefineatt -outputFileName ABC.nc -globalAttributes
-"netCDF-Java=4.0,Conventions=value with a space"
-Will add the two listed global attributes to the file ABC.nc. The file will be
-created if it doesn't already exist.
+Example: ncdefineatt -outputFileName ABC.nc -attributes
+"netCDF-Java=4.0,Conventions=value with a space,toDelete=null"
+Will add the two and delete one listed global attributes to the file ABC.nc. The
+file will be created if it doesn't already exist.
-------------------------------------
Modified: trunk/src/main/java/au/csiro/netcdf/NcDefineAttributes.java
===================================================================
--- trunk/src/main/java/au/csiro/netcdf/NcDefineAttributes.java 2010-06-22 02:24:18 UTC (rev 17)
+++ trunk/src/main/java/au/csiro/netcdf/NcDefineAttributes.java 2010-06-22 06:52:39 UTC (rev 18)
@@ -38,13 +38,14 @@
import ucar.nc2.Attribute;
import ucar.nc2.NetcdfFileWriteable;
+import ucar.nc2.Variable;
import au.csiro.netcdf.cli.Command;
import au.csiro.netcdf.cli.CommandLineOptionsComparator;
import au.csiro.netcdf.util.NetCDFUtils;
import au.csiro.netcdf.util.Util;
/**
- * The <strong>ncdefineAtt</strong> command defines global {@link Attribute}s in a netCDF file.
+ * The <strong>ncdefineAtt</strong> command defines global or variable {@link Attribute}s in a netCDF file.
* <p>
* Copyright 2010, CSIRO Australia
* All rights reserved.
@@ -76,11 +77,16 @@
private static final String STANDARD_INPUT = "standardInput";
/**
- * The name of the command line option used for specifying the global attributes of a netCDF file.
+ * The name of the command line option used for specifying the attributes of a netCDF file.
*/
- public static final String GLOBAL_ATTRIBUTES = "globalAttributes";
+ public static final String ATTRIBUTES = "attributes";
/**
+ * The name of the command line option used for specifying the name of the variable receiving the attributes.
+ */
+ public static final String VARIABLE_NAME = "variable";
+
+ /**
* Whether the netCDF file should be written with large file support, that is, 64-bit addressing for files greater
* than 2 GB.
*/
@@ -126,11 +132,13 @@
String inputFilenameArg = (parsedCommandLine.hasOption(INPUT_FILE)) ? parsedCommandLine
.getOptionValue(INPUT_FILE) : "";
boolean isStandardInput = parsedCommandLine.hasOption(STANDARD_INPUT);
- String globalAttributesArg = (parsedCommandLine.hasOption(GLOBAL_ATTRIBUTES)) ? parsedCommandLine
- .getOptionValue(GLOBAL_ATTRIBUTES) : "";
+ String attributesArg = (parsedCommandLine.hasOption(ATTRIBUTES)) ? parsedCommandLine
+ .getOptionValue(ATTRIBUTES) : "";
boolean isLargeFileSupport = parsedCommandLine.hasOption(IS_LARGE_FILE);
String pattern = (parsedCommandLine.hasOption(FILENAME_PATTERN)) ? parsedCommandLine
.getOptionValue(FILENAME_PATTERN) : "";
+ String variableNameArg = (parsedCommandLine.hasOption(VARIABLE_NAME)) ? parsedCommandLine
+ .getOptionValue(VARIABLE_NAME) : "";
// check whether large file support is needed
if (Util.fileExists(outputFilenameArg)
@@ -149,35 +157,35 @@
Util.getExistingFile(inputFilenameArg);
}
- // try getting global attributes from the command line
- List<Attribute> globalAttributes = new ArrayList<Attribute>(); // this is non-mandatory option, its default
+ // try getting attributes from the command line
+ List<Attribute> attributes = new ArrayList<Attribute>(); // this is non-mandatory option, its default
// value is an empty list.
- if(!globalAttributesArg.isEmpty())
+ if(!attributesArg.isEmpty())
{
try
{
- globalAttributes = NetCDFUtils.mapStringToAttributeValueList(globalAttributesArg);
+ attributes = NetCDFUtils.mapStringToAttributeValueList(attributesArg);
}
catch (IllegalArgumentException iae)
{
- throw new IllegalArgumentException(NcDefineAttributes.GLOBAL_ATTRIBUTES
- + " value is not a comma separated String of attribute-value pairs: " + globalAttributesArg);
+ throw new IllegalArgumentException(NcDefineAttributes.ATTRIBUTES
+ + " value is not a comma separated String of attribute-value pairs: " + attributesArg);
}
}
// try getting variable's attributes from a file or stdin.
if (!inputFilenameArg.isEmpty())
{
- globalAttributes.addAll(NetCDFUtils.readAttributesFromStream(new FileInputStream(inputFilenameArg)));
+ attributes.addAll(NetCDFUtils.readAttributesFromStream(new FileInputStream(inputFilenameArg)));
}
if(isStandardInput)
{
- globalAttributes.addAll(NetCDFUtils.readAttributesFromStream(System.in));
+ attributes.addAll(NetCDFUtils.readAttributesFromStream(System.in));
}
for (String outputFileName : targetFileNames)
{
- this.execute(outputFileName, globalAttributes, isLargeFileSupport);
+ this.execute(outputFileName, attributes, isLargeFileSupport, variableNameArg);
}
}
@@ -186,7 +194,7 @@
*
* @param outputFilename
* the netCDF file in which to define a dimension.
- * @param globaAttributes
+ * @param attributes
* a attributes of the file.
* @param isLargeFileSupport
* whether the netCDF file should be written with large file support, i.e. 64-bit addressing for files
@@ -196,8 +204,31 @@
* @throws SecurityException
* thrown if a security manager exists and it prevents the netCDF file from being created.
*/
- public void execute(String outputFilename, List<Attribute> globaAttributes, boolean isLargeFileSupport) throws IOException, SecurityException
+ public void execute(String outputFilename, List<Attribute> attributes, boolean isLargeFileSupport) throws IOException, SecurityException
{
+ this.execute(outputFilename, attributes, isLargeFileSupport, "");
+ }
+
+ /**
+ * Allows the command to be run programmatically, instead of from a command line.
+ *
+ * @param outputFilename
+ * the netCDF file in which to define a dimension.
+ * @param attributes
+ * a attributes of the file.
+ * @param isLargeFileSupport
+ * whether the netCDF file should be written with large file support, i.e. 64-bit addressing for files
+ * greater than 2 GB.
+ * @param variableName
+ * the variable name to assign attributes to
+ * @throws IOException
+ * thrown if netCDF can to be written to or read from.
+ * @throws SecurityException
+ * thrown if a security manager exists and it prevents the netCDF file from being created.
+ */
+ public void execute(String outputFilename, List<Attribute> attributes,
+ boolean isLargeFileSupport, String variableName) throws IOException, SecurityException
+ {
// the netcdf file to be written.
NetcdfFileWriteable ncfile = null;
@@ -223,12 +254,49 @@
ncfile.setRedefineMode(true);
ncfile.setLargeFile(isLargeFileSupport);
- // add attributes
- for(Attribute attribute : globaAttributes)
+ // add attributes to file
+ if (variableName.isEmpty())
{
- ncfile.addAttribute(null, attribute);
+ for(Attribute attribute : attributes)
+ {
+ // remove Attribute if the value is "null"
+ if ("null".equalsIgnoreCase(attribute.getStringValue()))
+ {
+ ncfile.deleteGlobalAttribute(attribute.getName());
+ }
+ else
+ {
+ ncfile.addAttribute(null, attribute);
+ }
+ }
}
+ // add attributes to variable
+ else
+ {
+ Variable variable = ncfile.findVariable(variableName);
+ if (variable == null)
+ {
+ throw new IllegalArgumentException("The variable " +
+ variableName + " doesn't exist in file " + outputFilename + ".");
+ }
+ else
+ {
+ for(Attribute attribute : attributes)
+ {
+ // remove Attribute if the value is "null"
+ if ("null".equalsIgnoreCase(attribute.getStringValue()))
+ {
+ variable.removeAttributeIgnoreCase(attribute.getName());
+ }
+ else
+ {
+ variable.addAttribute(attribute);
+ }
+ }
+ }
+ }
+
// close editing header info
ncfile.setRedefineMode(false);
}
@@ -268,7 +336,7 @@
try
{
// add attributes
- for(Attribute attribute : globaAttributes)
+ for(Attribute attribute : attributes)
{
ncfile.addAttribute(null, attribute);
}
@@ -311,11 +379,13 @@
*/
public String toString()
{
- // generate the help/usage statement
- String header = "Define a global attribute in a netCDF file.";
+ // generate the help/usage statement //TODO:
+ String header = "Define an attribute in a netCDF file. "
+ + "\nIf variable is not specified, then a global attribute is defined."
+ + "\nIf the value of the attribute is null, then the attribute will be deleted.";
String footer = "\nExample: ncdefineatt -outputFileName ABC.nc "
- + "-globalAttributes \"netCDF-Java=4.0,Conventions=value with a space\"\n"
- + "Will add the two listed global attributes to the file ABC.nc. "
+ + "-attributes \"netCDF-Java=4.0,Conventions=value with a space,toDelete=null\"\n"
+ + "Will add the two and delete one listed global attributes to the file ABC.nc. "
+ "The file will be created if it doesn't already exist. ";
StringWriter sw = new StringWriter();
HelpFormatter formatter = new HelpFormatter();
@@ -350,12 +420,12 @@
"3: OPTIONAL, read attributes from Stdin, ensure that text containing '=' or ',' characters are delimited by a backslash.").isRequired(false)
.withLongOpt(STANDARD_INPUT).create("s");
- Option globalAttributes = OptionBuilder
+ Option attributes = OptionBuilder
.withArgName("text")
.hasArg()
.withDescription(
"4: a comma separated list of attribute-value pairs, OPTIONAL, e.g. \"netCDF Java=4.0,Conventions=CF-1.0\", ensure that text containing '=' or ',' characters are delimited by a backslash.")
- .isRequired(false).withLongOpt(GLOBAL_ATTRIBUTES).create("a");
+ .isRequired(false).withLongOpt(ATTRIBUTES).create("a");
Option largeFileSupport = OptionBuilder.withDescription(
"5: OPTIONAL, set if more than 2 GB of data will need to be stored in this file.").isRequired(false)
@@ -366,15 +436,22 @@
+ "processed. If this option is present the outptuFilename is expected to be a direcotry "
+ "containing the files to be processed.").isRequired(false).withLongOpt(FILENAME_PATTERN)
.create("p");
+ Option variableName = OptionBuilder
+ .withArgName("text")
+ .hasArg()
+ .withDescription(
+ "7: OPTIONAL, the variable to assign attribute(s) to.") //TODO wording....
+ .isRequired(false).withLongOpt(VARIABLE_NAME).create("v");
Options options = new Options();
options.addOption(outputFileName);
options.addOption(inputFileName);
options.addOption(standardInput);
- options.addOption(globalAttributes);
+ options.addOption(attributes);
options.addOption(largeFileSupport);
options.addOption(pattern);
+ options.addOption(variableName);
return options;
}
@@ -409,8 +486,8 @@
.getOptionValue(OUTPUT_FILE) : "";
String inputFilenameArg = (parsedCommandLine.hasOption(INPUT_FILE)) ? parsedCommandLine
.getOptionValue(INPUT_FILE) : "";
- String globalAttributesArg = (parsedCommandLine.hasOption(GLOBAL_ATTRIBUTES)) ? parsedCommandLine
- .getOptionValue(GLOBAL_ATTRIBUTES) : "";
+ String attributesArg = (parsedCommandLine.hasOption(ATTRIBUTES)) ? parsedCommandLine
+ .getOptionValue(ATTRIBUTES) : "";
boolean isLargeFileSupport = parsedCommandLine.hasOption(IS_LARGE_FILE);
String pattern = (parsedCommandLine.hasOption(FILENAME_PATTERN)) ? parsedCommandLine
.getOptionValue(FILENAME_PATTERN) : "";
@@ -446,18 +523,18 @@
Util.getExistingFile(inputFilenameArg);
}
- // try getting global attributes from the command line.
- if(!globalAttributesArg.isEmpty())
+ // try getting attributes from the command line.
+ if(!attributesArg.isEmpty())
{
try
{
- NetCDFUtils.mapStringToAttributeValueList(globalAttributesArg);
+ NetCDFUtils.mapStringToAttributeValueList(attributesArg);
}
catch (IllegalArgumentException iae)
{
- throw new IllegalArgumentException(NcDefineAttributes.GLOBAL_ATTRIBUTES
+ throw new IllegalArgumentException(NcDefineAttributes.ATTRIBUTES
+ " value is not a comma separated String of attribute-value pairs: "
- + globalAttributesArg);
+ + attributesArg);
}
}
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <rit...@us...> - 2010-06-22 07:16:42
|
Revision: 19
http://netcdftools.svn.sourceforge.net/netcdftools/?rev=19&view=rev
Author: ritacsiro
Date: 2010-06-22 07:16:36 +0000 (Tue, 22 Jun 2010)
Log Message:
-----------
ANDSWRON-678 - Add variable parameter to defineAttributes.
Modified Paths:
--------------
trunk/USAGE.txt
trunk/src/main/java/au/csiro/netcdf/NcDefineAttributes.java
Modified: trunk/USAGE.txt
===================================================================
--- trunk/USAGE.txt 2010-06-22 06:52:39 UTC (rev 18)
+++ trunk/USAGE.txt 2010-06-22 07:16:36 UTC (rev 19)
@@ -47,10 +47,8 @@
-------------------------------------
-ncdefineAtt
-Define a global attribute in a netCDF file.
-If variable is not specified, then a global attribute is defined.
-If the value of the attribute is null, then the attribute will be deleted.
+usage: ncdefineAtt
+Define an attribute in a netCDF file.
-o,--outputFileName <file> 1: the filename of the netCDF file to be
created.
-i,--inputFileName <file> 2. the filename of a text file containing
@@ -60,11 +58,12 @@
-s,--standardInput 3: OPTIONAL, read attributes from Stdin, ensure
that text containing '=' or ',' characters are
delimited by a backslash.
--a,--globalAttributes <text> 4: a comma separated list of attribute-value
+-a,--attributes <text> 4: a comma separated list of attribute-value
pairs, OPTIONAL, e.g. "netCDF
Java=4.0,Conventions=CF-1.0", ensure that text
containing '=' or ',' characters are delimited
- by a backslash.
+ by a backslash. Set value to null to delete
+ existing attribute.
-l,--largeFileSupport 5: OPTIONAL, set if more than 2 GB of data will
need to be stored in this file.
-p,--pattern <filename pattern> 6: OPTIONAL, a pattern to match multiple
@@ -73,7 +72,7 @@
outptuFilename is expected to be a direcotry
containing the files to be processed.
-v,--variable <text> 7: OPTIONAL, the variable to assign attribute(s)
- to.
+ to, set to define variable attribute.
Example: ncdefineatt -outputFileName ABC.nc -attributes
"netCDF-Java=4.0,Conventions=value with a space,toDelete=null"
Modified: trunk/src/main/java/au/csiro/netcdf/NcDefineAttributes.java
===================================================================
--- trunk/src/main/java/au/csiro/netcdf/NcDefineAttributes.java 2010-06-22 06:52:39 UTC (rev 18)
+++ trunk/src/main/java/au/csiro/netcdf/NcDefineAttributes.java 2010-06-22 07:16:36 UTC (rev 19)
@@ -379,10 +379,8 @@
*/
public String toString()
{
- // generate the help/usage statement //TODO:
- String header = "Define an attribute in a netCDF file. "
- + "\nIf variable is not specified, then a global attribute is defined."
- + "\nIf the value of the attribute is null, then the attribute will be deleted.";
+ // generate the help/usage statement
+ String header = "Define an attribute in a netCDF file.";
String footer = "\nExample: ncdefineatt -outputFileName ABC.nc "
+ "-attributes \"netCDF-Java=4.0,Conventions=value with a space,toDelete=null\"\n"
+ "Will add the two and delete one listed global attributes to the file ABC.nc. "
@@ -424,7 +422,9 @@
.withArgName("text")
.hasArg()
.withDescription(
- "4: a comma separated list of attribute-value pairs, OPTIONAL, e.g. \"netCDF Java=4.0,Conventions=CF-1.0\", ensure that text containing '=' or ',' characters are delimited by a backslash.")
+ "4: a comma separated list of attribute-value pairs, OPTIONAL, e.g. \"netCDF Java=4.0,Conventions=CF-1.0\", "
+ + "ensure that text containing '=' or ',' characters are delimited by a backslash. "
+ + "Set value to null to delete existing attribute.")
.isRequired(false).withLongOpt(ATTRIBUTES).create("a");
Option largeFileSupport = OptionBuilder.withDescription(
@@ -440,7 +440,8 @@
.withArgName("text")
.hasArg()
.withDescription(
- "7: OPTIONAL, the variable to assign attribute(s) to.") //TODO wording....
+ "7: OPTIONAL, the variable to assign attribute(s) to, "
+ + "set to define variable attribute.")
.isRequired(false).withLongOpt(VARIABLE_NAME).create("v");
Options options = new Options();
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <jde...@us...> - 2010-06-24 00:29:14
|
Revision: 29
http://netcdftools.svn.sourceforge.net/netcdftools/?rev=29&view=rev
Author: jdempsey
Date: 2010-06-24 00:29:08 +0000 (Thu, 24 Jun 2010)
Log Message:
-----------
ANDSWRON-657 - Allow defineAttributes to manipulate multiple files
- Update to address review feedback and only a single file matching the pattern
Modified Paths:
--------------
trunk/USAGE.txt
trunk/src/main/java/au/csiro/netcdf/NcDefineAttributes.java
trunk/src/main/java/au/csiro/netcdf/util/Util.java
Added Paths:
-----------
trunk/src/test/java/au/csiro/netcdf/util/UtilTest.java
Modified: trunk/USAGE.txt
===================================================================
--- trunk/USAGE.txt 2010-06-23 08:29:49 UTC (rev 28)
+++ trunk/USAGE.txt 2010-06-24 00:29:08 UTC (rev 29)
@@ -1,19 +1,3 @@
-====
- Copyright 2010, CSIRO Australia.
-
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
-====
-
ncextractcsv
Extract a block of data from a CSV file. The data can then be piped into the
ncwritevar command to add the data to a variable in a netCDF file.
@@ -47,7 +31,7 @@
-------------------------------------
-usage: ncdefineAtt
+ncdefineAtt
Define an attribute in a netCDF file.
-o,--outputFileName <file> 1: the filename of the netCDF file to be
created.
@@ -69,10 +53,16 @@
-p,--pattern <filename pattern> 6: OPTIONAL, a pattern to match multiple
existing files, where multiple files should be
processed. If this option is present the
- outptuFilename is expected to be a direcotry
- containing the files to be processed.
+ outputFilename is expected to be a directory
+ containing the files to be processed. The
+ wildcard characters * and ? are supported but
+ may need to be escaped by a \ to avoid
+ processing by the shell. e.g. \*.nc
-v,--variable <text> 7: OPTIONAL, the variable to assign attribute(s)
to, set to define variable attribute.
+-t,--attributeDataType <text> 8: OPTIONAL, the data type of the attribute(s),
+ e.g. [byte, short, String, float, double, int,
+ char]
Example: ncdefineatt -outputFileName ABC.nc -attributes
"netCDF-Java=4.0,Conventions=value with a space,toDelete=null"
@@ -93,8 +83,8 @@
that text containing '=' or ',' characters are
delimited by a backslash.
-v,--variableName <text> 4: the name to be given to the variable.
--t,--variableDataType <text> 5: the data type of the variable, e.g. [int,
- double, char, float, short, byte]
+-t,--variableDataType <text> 5: the data type of the variable, e.g. [byte,
+ short, float, double, int, char]
-a,--variableAttributes <text> 6: a comma separated list of attribute-value
pairs, OPTIONAL, e.g. "units=mm", ensure that
text containing '=' or ',' characters are
Modified: trunk/src/main/java/au/csiro/netcdf/NcDefineAttributes.java
===================================================================
--- trunk/src/main/java/au/csiro/netcdf/NcDefineAttributes.java 2010-06-23 08:29:49 UTC (rev 28)
+++ trunk/src/main/java/au/csiro/netcdf/NcDefineAttributes.java 2010-06-24 00:29:08 UTC (rev 29)
@@ -168,17 +168,21 @@
.getOptionValue(VARIABLE_NAME) : "";
String attributeDataTypeArg = (parsedCommandLine.hasOption(ATTRIBUTE_DATA_TYPE)) ? parsedCommandLine
.getOptionValue(ATTRIBUTE_DATA_TYPE) : "String";
+
+ List<String> targetFileNames = Util.getListOfTargetFiles(outputFilenameArg, pattern);
// check whether large file support is needed
- if (Util.fileExists(outputFilenameArg)
- && Util.getExistingFile(outputFilenameArg).length() >= MAX_32BIT_OFFSET_FILE_SIZE
- && !isLargeFileSupport)
+ for (String outputFileName : targetFileNames)
{
- throw new IllegalArgumentException("The netCDF file will be too large, please use " + IS_LARGE_FILE
- + " flag.");
+ if (Util.fileExists(outputFileName)
+ && Util.getExistingFile(outputFileName).length() >= MAX_32BIT_OFFSET_FILE_SIZE
+ && !isLargeFileSupport)
+ {
+ throw new IllegalArgumentException("The netCDF file " + outputFileName
+ + " will be too large, please use " + IS_LARGE_FILE + " flag.");
+ }
}
- List<String> targetFileNames = Util.getListOfTargetFiles(outputFilenameArg, pattern);
// check that if an input file is specified then it actually exists.
if (!inputFilenameArg.isEmpty())
@@ -217,6 +221,7 @@
for (String outputFileName : targetFileNames)
{
+ LOG.debug("Processing file : " + outputFileName);
this.execute(outputFileName, attributes, isLargeFileSupport, variableNameArg);
}
}
@@ -465,9 +470,10 @@
Option pattern = OptionBuilder.withArgName("filename pattern").hasArg().withDescription(
"6: OPTIONAL, a pattern to match multiple existing files, where multiple files should be "
- + "processed. If this option is present the outptuFilename is expected to be a direcotry "
- + "containing the files to be processed.").isRequired(false).withLongOpt(FILENAME_PATTERN)
- .create("p");
+ + "processed. If this option is present the outputFilename is expected to be a directory "
+ + "containing the files to be processed. The wildcard characters * and ? are supported but "
+ + "may need to be escaped by a \\ to avoid processing by the shell. e.g. \\*.nc ").isRequired(
+ false).withLongOpt(FILENAME_PATTERN).create("p");
Option variableName = OptionBuilder
.withArgName("text")
.hasArg()
Modified: trunk/src/main/java/au/csiro/netcdf/util/Util.java
===================================================================
--- trunk/src/main/java/au/csiro/netcdf/util/Util.java 2010-06-23 08:29:49 UTC (rev 28)
+++ trunk/src/main/java/au/csiro/netcdf/util/Util.java 2010-06-24 00:29:08 UTC (rev 29)
@@ -220,6 +220,7 @@
*/
public static List<String> getListOfTargetFiles(String foldername, String pattern)
{
+ String cleanPat = pattern.replaceAll("\\\\\\*", "*").replaceAll("\\\\\\?", "?");
List<String> fileNameList = new ArrayList<String>();
if (pattern == null || pattern.length() == 0)
{
@@ -233,7 +234,7 @@
return Collections.emptyList();
}
- FileFilter patternFilter = new WildcardFileFilter(pattern);
+ FileFilter patternFilter = new WildcardFileFilter(cleanPat);
File[] files = dir.listFiles(patternFilter);
for (int i = 0; i < files.length; i++)
{
Added: trunk/src/test/java/au/csiro/netcdf/util/UtilTest.java
===================================================================
--- trunk/src/test/java/au/csiro/netcdf/util/UtilTest.java (rev 0)
+++ trunk/src/test/java/au/csiro/netcdf/util/UtilTest.java 2010-06-24 00:29:08 UTC (rev 29)
@@ -0,0 +1,79 @@
+/**
+ * Copyright 2010, CSIRO Australia.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package au.csiro.netcdf.util;
+
+import java.util.List;
+
+import junit.framework.TestCase;
+
+/**
+ * Check the function of the Util class.
+ *
+ * @author James Dempsey on 23/06/2010
+ * @version $Revision$ $Date$
+ */
+public class UtilTest extends TestCase
+{
+
+ /* (non-Javadoc)
+ * @see junit.framework.TestCase#setUp()
+ */
+ protected void setUp() throws Exception
+ {
+ super.setUp();
+ }
+
+ /**
+ * Test method for {@link au.csiro.netcdf.util.Util#getListOfTargetFiles(java.lang.String, java.lang.String)}.
+ */
+ public final void testGetListOfTargetFiles()
+ {
+ List<String> files = Util.getListOfTargetFiles(".", "*.txt");
+ assertNotNull("Files should be a non-empty list", files);
+ assertTrue("Files should be a non-empty list", files.size()>1);
+ boolean hasUsage = false;
+ for (String string : files)
+ {
+ if (string.endsWith("USAGE.txt"))
+ {
+ hasUsage = true;
+ }
+ }
+ assertTrue("Expected to find USAGE.txt in result set.", hasUsage);
+
+ }
+
+ /**
+ * Test method for {@link au.csiro.netcdf.util.Util#getListOfTargetFiles(java.lang.String, java.lang.String)}.
+ */
+ public final void testGetListOfTargetFilesEscaped()
+ {
+ List<String> files = Util.getListOfTargetFiles(".", "\\*.txt");
+ assertNotNull("Files should be a non-empty list", files);
+ assertTrue("Files should be a non-empty list", files.size()>1);
+ boolean hasUsage = false;
+ for (String string : files)
+ {
+ if (string.endsWith("USAGE.txt"))
+ {
+ hasUsage = true;
+ }
+ }
+ assertTrue("Expected to find USAGE.txt in result set.", hasUsage);
+
+ }
+
+}
Property changes on: trunk/src/test/java/au/csiro/netcdf/util/UtilTest.java
___________________________________________________________________
Added: svn:keywords
+ Author Revision Date Id
Added: svn:eol-style
+ native
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <rob...@us...> - 2010-07-08 02:05:49
|
Revision: 67
http://netcdftools.svn.sourceforge.net/netcdftools/?rev=67&view=rev
Author: robertbridle
Date: 2010-07-08 02:05:42 +0000 (Thu, 08 Jul 2010)
Log Message:
-----------
ANDSWRON-703 - Created a program to convert netCDF files (split by latitude) into csv files.
Added Paths:
-----------
trunk/launch/NetCDF2CSVConverter.launch
trunk/src/main/java/au/csiro/netcdf/wron/MdbsyNetCDF2CSVConverter.java
trunk/src/test/java/au/csiro/netcdf/wron/TestMdbsyNetCDF2CSVConverter.java
Added: trunk/launch/NetCDF2CSVConverter.launch
===================================================================
--- trunk/launch/NetCDF2CSVConverter.launch (rev 0)
+++ trunk/launch/NetCDF2CSVConverter.launch 2010-07-08 02:05:42 UTC (rev 67)
@@ -0,0 +1,14 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<launchConfiguration type="org.eclipse.jdt.launching.localJavaApplication">
+<listAttribute key="org.eclipse.debug.core.MAPPED_RESOURCE_PATHS">
+<listEntry value="/netcdf-tools/src/main/java/au/csiro/netcdf/wron/MdbsyNetCDF2CSVConverter.java"/>
+</listAttribute>
+<listAttribute key="org.eclipse.debug.core.MAPPED_RESOURCE_TYPES">
+<listEntry value="1"/>
+</listAttribute>
+<stringAttribute key="org.eclipse.jdt.launching.CLASSPATH_PROVIDER" value="org.maven.ide.eclipse.launchconfig.classpathProvider"/>
+<stringAttribute key="org.eclipse.jdt.launching.MAIN_TYPE" value="au.csiro.netcdf.wron.MdbsyNetCDF2CSVConverter"/>
+<stringAttribute key="org.eclipse.jdt.launching.PROGRAM_ARGUMENTS" value="--inputDir "C:\test\netcdf\\" -outputDir "C:\test\csv\\""/>
+<stringAttribute key="org.eclipse.jdt.launching.PROJECT_ATTR" value="netcdf-tools"/>
+<stringAttribute key="org.eclipse.jdt.launching.SOURCE_PATH_PROVIDER" value="org.maven.ide.eclipse.launchconfig.sourcepathProvider"/>
+</launchConfiguration>
Added: trunk/src/main/java/au/csiro/netcdf/wron/MdbsyNetCDF2CSVConverter.java
===================================================================
--- trunk/src/main/java/au/csiro/netcdf/wron/MdbsyNetCDF2CSVConverter.java (rev 0)
+++ trunk/src/main/java/au/csiro/netcdf/wron/MdbsyNetCDF2CSVConverter.java 2010-07-08 02:05:42 UTC (rev 67)
@@ -0,0 +1,693 @@
+package au.csiro.netcdf.wron;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.RandomAccessFile;
+import java.io.StringWriter;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.GnuParser;
+import org.apache.commons.cli.HelpFormatter;
+import org.apache.commons.cli.MissingOptionException;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.OptionBuilder;
+import org.apache.commons.cli.Options;
+import org.apache.commons.cli.ParseException;
+import org.apache.log4j.Logger;
+
+import ucar.ma2.Array;
+import ucar.ma2.InvalidRangeException;
+import ucar.nc2.NetcdfFile;
+import ucar.nc2.Variable;
+import ucar.nc2.units.DateFormatter;
+import au.csiro.netcdf.cli.CommandLineOptionsComparator;
+import au.csiro.netcdf.util.CSVTokenizer;
+import au.csiro.netcdf.util.Util;
+
+/**
+ * The class is a converter control class to convert Murray Darling Basin Sustainable Yields netCDF files into CSV
+ * files.
+ *
+ * Copyright 2010, CSIRO Australia
+ *
+ * @author Robert Bridle on 07/07/2010
+ * @version $Revision: {revision} $ $Date: 2008-10-31 11:24:49 +1100 (Fri, 31 Oct 2008) $
+ */
+public class MdbsyNetCDF2CSVConverter
+{
+ private static final String EPOC = "1895-01-01";
+
+ /**
+ * Constant that defines the logger to be used.
+ */
+ private static final Logger LOG = Logger
+ .getLogger(MdbsyNetCDF2CSVConverter.class.getName());
+
+ /**
+ * The latitude variable name used in the netCDF files.
+ */
+ private static final String LATITUDE_VARIABLE_NAME = "lat";
+
+ /**
+ * The longitude variable name used in the netCDF files.
+ */
+ private static final String LONGITUDE_VARIABLE_NAME = "long";
+
+ /**
+ * The time variable name used in the netCDF files.
+ */
+ private static final String TIME_VARIABLE_NAME = "time";
+
+ /**
+ * The number of milliseconds in a 24 hours.
+ */
+ private static final Long MILLISECONDS_IN_A_DAY = new Long("86400000"); // milliseconds in a day.
+
+ /**
+ * The date column text that is found in all csv files.
+ */
+ // private static final String DATE_COLUMN_TEXT = writeOutDateColumn();
+
+ /**
+ * The name of the command line option used for specifying the input directory containing the netCDF files.
+ */
+ public static final String INPUT_DIR = "inputDir";
+
+ /**
+ * The name of the command line option used for specifying the directory to create the csv files in.
+ */
+ public static final String OUTPUT_DIR = "outputDir";
+
+ /**
+ * The pattern for accepted file names where multiple files are accepted.
+ */
+ public static final String FILENAME_PATTERN = "pattern";
+
+ /**
+ * The column that the variable name is stored in the csv file.
+ */
+ private static final int VARIABLE_COLUMN = 0;
+
+ /**
+ * The only variables that we are interested in writing out to csv files are those that are in three dimensions
+ * (i.e. rank=3)
+ */
+ private static final int RANK_THREE = 3;
+
+ /**
+ * NetCDf file extension.
+ */
+ private static final Object CSV_FILE_EXTENSION = ".csv";
+
+ /**
+ * CSV file extension.
+ */
+ private static final Object NETCDF_FILE_EXTENSION = ".nc";
+
+ /**
+ * A string containing the date column text to be written out as the first column in all the csv files.
+ */
+ private static String DATE_COLUMN_TEXT = null;
+
+ /**
+ * @param args
+ */
+ @SuppressWarnings("static-access")
+ public static void main(String[] args) throws ParseException, IOException,
+ InvalidRangeException, java.text.ParseException
+ {
+ Options options = new Options();
+ try
+ {
+ Option inputDirectoryName = OptionBuilder
+ .withArgName("dir")
+ .hasArg()
+ .withDescription(
+ "1. the directory path containing the netCDF files to be converted.")
+ .isRequired(true).withLongOpt(INPUT_DIR).create("i");
+
+ Option outputDirectoryName = OptionBuilder
+ .withArgName("dir")
+ .hasArg()
+ .withDescription(
+ "2: the directory path to place the new csv files.")
+ .isRequired(true).withLongOpt(OUTPUT_DIR).create("o");
+
+ Option pattern = OptionBuilder
+ .withArgName("filename pattern")
+ .hasArg()
+ .withDescription(
+ "3: OPTIONAL, a pattern to match multiple existing files, where multiple files should be "
+ + "processed. If this option is present the outputFilename is expected to be a directory "
+ + "containing the files to be processed. The wildcard characters * and ? are supported but "
+ + "may need to be escaped by a \\ to avoid processing by the shell. e.g. \\*.nc ")
+ .isRequired(false).withLongOpt(FILENAME_PATTERN)
+ .create("p");
+
+ options.addOption(inputDirectoryName);
+ options.addOption(outputDirectoryName);
+ options.addOption(pattern);
+
+ // parse the command line arguments
+ CommandLine parsedCommandLine = new GnuParser()
+ .parse(options, args);
+
+ String inputDir = parsedCommandLine.getOptionValue(INPUT_DIR);
+ String outputDir = parsedCommandLine.getOptionValue(OUTPUT_DIR);
+
+ MdbsyNetCDF2CSVConverter converter = new MdbsyNetCDF2CSVConverter();
+
+ long start = System.currentTimeMillis();
+ converter.execute(inputDir, outputDir);
+ long end = System.currentTimeMillis() - start;
+ LOG.warn("Successfully converted all netcdf files to csv files in: " + end + " ms.");
+ }
+ catch (MissingOptionException moe)
+ {
+ LOG.error(moe.getMessage());
+
+ // generate the help/usage statement
+ String header = "Recreate ";
+ String footer = "\nExample: netcdf2csv -inputDirPath \"C:\\input\" -outputDirPath \"C:\\output\" \n"
+ + "Will convert the netCDF files in the input directory to csv file in the output directory.";
+ StringWriter sw = new StringWriter();
+ HelpFormatter formatter = new HelpFormatter();
+ formatter.setOptionComparator(new CommandLineOptionsComparator());
+ formatter.printHelp(new PrintWriter(sw), 80, "-", header, options,
+ 0, 1, footer);
+ System.out.println(sw.toString());
+ }
+ }
+
+ /**
+ * Performs the conversion.
+ *
+ * @param inputDir
+ * the directory containing the Scenario A or C netCDF files.
+ * @param outputDir
+ * the directory in which the csv file will be placed.
+ * @throws InvalidRangeException
+ * thrown if an invalid range is attempted to be read from a netCDf variable.
+ * @throws IOException
+ */
+ public void execute(String inputDir, String outputDir) throws IOException,
+ InvalidRangeException
+ {
+ // perform an initial conversion from netCDF to csv, the csv file will look like this:
+ // Date, 1985-01-01, 1895-01-02, 1985-01-03, ...
+ // APET, 0.1, 0.3, 0.0, ....
+ // rainfall,0.0, 0.4, 0.1, ...
+ writeCSVFiles(inputDir, outputDir);
+
+ // transpose the csv files so the will look like this:
+ // Date, APET, rainfall
+ // 1895-01-01, 0.1, 0.0
+ // 1895-01-02, 0.3, 0.4
+ // 1895-01-03, 0.0, 0.1
+ transposeCSVFiles(outputDir);
+ }
+
+ /**
+ * For each netCDF file in the input directory:
+ * <p>
+ * Create a csv file for every latitude and longitude position.
+ * <p>
+ * Write out the value of a variable at every time interval, at a specific latitude and longitude position.
+ *
+ * @param inputDir
+ * the directory containing the Scenario A or C netCDF files.
+ * @param outputDir
+ * the directory in which the csv file will be placed.
+ * @throws InvalidRangeException
+ * thrown if an invalid range is attempted to be read from a netCDf variable.
+ * @throws IOException
+ */
+ private void writeCSVFiles(String inputDir, String outputDir)
+ throws IOException, InvalidRangeException
+ {
+ File dir = new File(inputDir);
+ File[] files = dir.listFiles();
+ if(files == null)
+ {
+ System.out.println("The input directory does not exist: " + inputDir);
+ LOG.error("The input directory does not exist: " + inputDir);
+ return;
+ }
+
+ for (int fileIndex = 0; fileIndex < files.length; fileIndex++)
+ {
+ if (files[fileIndex].isFile() && NETCDF_FILE_EXTENSION.equals(Util.getFileExtension(files[fileIndex].getName())))
+ {
+ NetcdfFile nc = null;
+ try
+ {
+ nc = NetcdfFile.open(files[fileIndex].getAbsolutePath());
+
+ Array latitudes = getLatitudeValues(nc);
+ Array longitudes = getLongitudeValues(nc);
+
+ LOG.info("latitude coordinate variable size: "
+ + latitudes.getSize());
+ LOG.info("longitude coordinate variable size: "
+ + longitudes.getSize());
+
+ for (int latIndex = 0; latIndex < latitudes.getSize(); latIndex++)
+ {
+ for (int longIndex = 0; longIndex < longitudes
+ .getSize(); longIndex++)
+ {
+ String fileName = createFileNameBasedOnLatLong(nc,
+ latIndex, longIndex);
+ String filePath = outputDir
+ + System.getProperty("file.separator")
+ + fileName + ".csv";
+
+ // if the csv file already exists, then append the next variable.
+ if (Util.fileExists(filePath))
+ {
+ File file = new File(filePath);
+ PrintWriter out = new PrintWriter(
+ new BufferedWriter(new FileWriter(file,
+ true /* append mode */)));
+
+ List<String> variablesAlreadyWrittenOut = getVariablesAlreadyWrittenOut(file);
+ List<String> variablesThatCanBeWrittenOut = getVariablesThatCanBeWrittenOut(nc);
+
+ variablesThatCanBeWrittenOut
+ .removeAll(variablesAlreadyWrittenOut);
+
+ try
+ {
+ writeOutVariable(nc, out,
+ variablesThatCanBeWrittenOut,
+ latIndex, longIndex);
+ }
+ finally
+ {
+ if (out != null)
+ out.close();
+ }
+ }
+ else
+ // if the csv file does not exist, create a new csv file and write all available variables.
+ {
+ PrintWriter out = new PrintWriter(new BufferedWriter(
+ new FileWriter(new File(filePath),
+ false /* append mode */)));
+
+ List<String> variablesThatCanBeWrittenOut = getVariablesThatCanBeWrittenOut(nc);
+
+ try
+ {
+ out.println(writeOutDateColumn(nc));
+
+ writeOutVariable(nc, out,
+ variablesThatCanBeWrittenOut,
+ latIndex, longIndex);
+ }
+ finally
+ {
+ if (out != null)
+ out.close();
+ }
+ }
+ }
+ }
+ }
+ catch (java.io.FileNotFoundException fnfe)
+ {
+ System.out.println("file not found= " + files[fileIndex]);
+ LOG.error(fnfe);
+ }
+ finally
+ {
+ if (nc != null)
+ nc.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Writes out the value of a variable at every time interval, at a specific latitude and longitude position.
+ *
+ * @throws InvalidRangeException
+ * @throws IOException
+ *
+ */
+ private void writeOutVariable(NetcdfFile nc, PrintWriter out,
+ List<String> variablesThatCanBeWrittenOut, int latIndex,
+ int longIndex) throws IOException, InvalidRangeException
+ {
+ for (String variableName : variablesThatCanBeWrittenOut)
+ {
+ Array subsection = getVariableAcrossTime(nc, variableName,
+ latIndex, longIndex);
+ out.println(variableName + ","
+ + subsection.toString().replaceAll(" ", ","));
+ }
+ }
+
+ /**
+ * Transpose the csv files so that columns represent a variable's value over time.
+ *
+ * @param outputDir
+ * the directory in which the csv file will be placed.
+ * @throws IOException
+ */
+ private void transposeCSVFiles(String outputDir) throws IOException
+ {
+ File dir = new File(outputDir);
+ File[] files = dir.listFiles();
+ if(files == null)
+ {
+ System.out.println("The output directory does not exist: " + outputDir);
+ LOG.error("The output directory does not exist: " + outputDir);
+ return;
+ }
+
+ for (int fileIndex = 0; fileIndex < files.length; fileIndex++)
+ {
+ if (files[fileIndex].isFile() && CSV_FILE_EXTENSION.equals(Util.getFileExtension(files[fileIndex].getName())))
+ {
+ String[][] matrix = readLookupFile(files[fileIndex]);
+
+ // write over-the-top of the csv file.
+ PrintWriter out = new PrintWriter(
+ new BufferedWriter(new FileWriter(files[fileIndex],
+ false /* append mode */)));
+
+ // transpose String[][] matrix and write back out to csv
+ String[][] transposed = transpose(matrix);
+
+ try
+ {
+ for (int row = 0; row < transposed.length; row++)
+ {
+ for (int col = 0; col < transposed[0].length; col++)
+ {
+ out.print(transposed[row][col]);
+ if(col < (transposed[0].length-1))
+ {
+ out.print(", ");
+ }
+ }
+ out.println();
+ }
+ }
+ finally
+ {
+ if (out != null)
+ out.close();
+ }
+ }
+ }
+ }
+
+ /**
+ * Returns the names of the variables in the netCDF file that we need to write out for a given latitude x longitude
+ * position.
+ *
+ * @param nc
+ * a netCDF file {@link NetcdfFile}
+ * @return the names of the variables in the netCDF file that we need to write out for a given latitude x longitude
+ * position.
+ */
+ private List<String> getVariablesThatCanBeWrittenOut(NetcdfFile nc)
+ {
+ List<String> variablesThatCanBeWrittenOut = new ArrayList<String>();
+
+ for (Variable variable : nc.getVariables())
+ {
+ if (variable.getRank() == RANK_THREE)
+ {
+ variablesThatCanBeWrittenOut.add(variable.getName().trim());
+ }
+ }
+
+ return variablesThatCanBeWrittenOut;
+ }
+
+ /**
+ * Returns the name of the variables that have already be written out to a csv file.
+ *
+ * @param file
+ * the csv file being written out too.
+ * @return the name of the variables that have already be written out to a csv file.
+ * @throws IOException
+ */
+ private List<String> getVariablesAlreadyWrittenOut(File file)
+ throws IOException
+ {
+ List<String> variablesAlreadyWrittenOut = new ArrayList<String>();
+
+ String[][] matrix = readLookupFile(file);
+ for (int rows = 0; rows < matrix.length; rows++)
+ {
+ variablesAlreadyWrittenOut.add(matrix[rows][VARIABLE_COLUMN].trim());
+ }
+
+ return variablesAlreadyWrittenOut;
+ }
+
+ /**
+ * Creates a file name based on a given latitude and longitude.
+ *
+ * @param nc
+ * a netCDF file {@link NetcdfFile}
+ * @param latIndex
+ * the latitude index.
+ * @param longIndex
+ * the longitude index.
+ * @return a file name based on latitude and longitude position.
+ * @throws InvalidRangeException
+ * @throws IOException
+ */
+ private String createFileNameBasedOnLatLong(NetcdfFile nc, int latIndex,
+ int longIndex) throws IOException, InvalidRangeException
+ {
+ return "region_"
+ + nc.readSection("lat(" + latIndex + ":" + latIndex + ")")
+ .toString().trim()
+ + "_"
+ + nc.readSection("long(" + longIndex + ":" + longIndex + ")")
+ .toString().trim();
+ }
+
+ /**
+ * Parses a csv file into an row-by-column string matrix.
+ *
+ * @param file
+ * a csv file.
+ * @return row-by-column string matrix of the csv file.
+ * @throws IOException
+ */
+ public static String[][] readLookupFile(File file) throws IOException
+ {
+ RandomAccessFile raf = new RandomAccessFile(file, "r");
+ List<String[]> lineList = new ArrayList<String[]>();
+
+ try
+ {
+ String line = raf.readLine();
+ while (line != null)
+ {
+ String[] cols = new CSVTokenizer(line).getAllColumns();
+ lineList.add(cols);
+ line = raf.readLine();
+ }
+ }
+ finally
+ {
+ raf.close();
+ }
+
+ String[][] lineCols = lineList.toArray(new String[][] {});
+ return lineCols;
+ }
+
+ /**
+ * Transpose a matrix, e.g. [a][b] -> [b][a]
+ *
+ * @param values a String[][] matrix.
+ */
+ private String[][] transpose(String[][] values)
+ {
+ if(values.length == 0)
+ {
+ return new String[0][0];
+ }
+
+ String[][] transposed = new String[values[0].length][values.length];
+ for (int row = 0; row < values.length; row++)
+ {
+ for (int col = 0; col < values[0].length; col++)
+ {
+ transposed[col][row] = values[row][col];
+ }
+ }
+ return transposed;
+ }
+
+ /**
+ * Returns the value of a variable at every time interval, at a specific latitude and longitude position.
+ *
+ * @param nc
+ * a netCDF file {@link NetcdfFile}
+ * @param variableName
+ * the variable whose values are to be read.
+ * @param latIndex
+ * the latitude index.
+ * @param longIndex
+ * the longitude index.
+ * @return the value of a variable at every time interval, at a specific latitude and longitude position.
+ * @throws IOException
+ * @throws InvalidRangeException
+ */
+ private Array getVariableAcrossTime(NetcdfFile nc, String variableName,
+ int latIndex, int longIndex) throws IOException,
+ InvalidRangeException
+ {
+ try
+ {
+ return nc.readSection(variableName + "(" + latIndex + ":"
+ + latIndex + "," + longIndex + ":" + longIndex + ",:)");
+ }
+ catch (InvalidRangeException ire)
+ {
+ ire.printStackTrace();
+ LOG.error("Could not read section: " + variableName + "("
+ + latIndex + ":" + latIndex + "," + longIndex + ":"
+ + longIndex + ",:)", ire);
+ }
+ return null;
+ }
+
+ /**
+ * Returns the longitude coordinate variable values.
+ *
+ * @param nc
+ * a netCDF file {@link NetcdfFile}
+ * @return the longitude coordinate variable values.
+ * @throws IOException
+ * @throws IllegalArgumentException
+ */
+ private Array getLongitudeValues(NetcdfFile nc) throws IOException,
+ IllegalArgumentException
+ {
+ for (Variable v : nc.getVariables())
+ {
+ if (v.isCoordinateVariable()
+ && LONGITUDE_VARIABLE_NAME.equals(v.getName()))
+ {
+ return v.read();
+ }
+ }
+ LOG.error("Could not find coordinate variable: "
+ + LONGITUDE_VARIABLE_NAME);
+ throw new IllegalStateException("Could not find coordinate variable: "
+ + LONGITUDE_VARIABLE_NAME);
+ }
+
+ /**
+ * Returns the latitude coordinate variable values.
+ *
+ * @param nc
+ * a netCDF file {@link NetcdfFile}
+ * @return the latitude coordinate variable values.
+ * @throws IOException
+ * @throws IllegalArgumentException
+ */
+ private Array getLatitudeValues(NetcdfFile nc) throws IOException,
+ IllegalArgumentException
+ {
+ for (Variable v : nc.getVariables())
+ {
+ if (v.isCoordinateVariable()
+ && LATITUDE_VARIABLE_NAME.equals(v.getName()))
+ {
+ return v.read();
+ }
+ }
+ LOG.error("Could not find coordinate variable: "
+ + LATITUDE_VARIABLE_NAME);
+ throw new IllegalStateException("Could not find coordinate variable: "
+ + LATITUDE_VARIABLE_NAME);
+ }
+
+ /**
+ * Returns the time variable.
+ *
+ * @param nc
+ * a netCDF file {@link NetcdfFile}
+ * @return the time variable.
+ * @throws IllegalArgumentException
+ */
+ private static Variable getTimeVariable(NetcdfFile nc)
+ throws IllegalArgumentException
+ {
+ for (Variable v : nc.getVariables())
+ {
+ if (v.isCoordinateVariable()
+ && TIME_VARIABLE_NAME.equals(v.getName()))
+ {
+ return v;
+ }
+ }
+ LOG.error("Could not find coordinate variable: " + TIME_VARIABLE_NAME);
+ throw new IllegalStateException("Could not find coordinate variable: "
+ + TIME_VARIABLE_NAME);
+ }
+
+ /**
+ * Converts a "days since" time coordinate variable into a list of formatted dates.
+ *
+ * @param nc
+ * a netCDF file {@link NetcdfFile}
+ * @return a list of formatted dates.
+ */
+ private static String writeOutDateColumn(NetcdfFile nc)
+ {
+ // we only need to create the date column text once, it is the same for all csv files.
+ if (DATE_COLUMN_TEXT == null)
+ {
+ StringBuffer strBuf = new StringBuffer();
+
+ DateFormatter dateFormatter = new DateFormatter();
+ Date epoc;
+ try
+ {
+ epoc = dateFormatter.dateOnlyFormat(EPOC);
+ strBuf.append("Date,");
+
+ long timeVariableSize = getTimeVariable(nc).getSize();
+ for (int i = 0; i < timeVariableSize; i++)
+ {
+ strBuf.append(dateFormatter.toDateOnlyString(new Date(epoc
+ .getTime()
+ + (i * MILLISECONDS_IN_A_DAY))).trim());
+
+ if(i < (timeVariableSize-1))
+ {
+ strBuf.append(",");
+ }
+ }
+
+ DATE_COLUMN_TEXT = strBuf.toString();
+ }
+ catch (java.text.ParseException pe)
+ {
+ pe.printStackTrace();
+ LOG.error("Error parsing date", pe);
+ }
+
+ }
+ return DATE_COLUMN_TEXT;
+ }
+}
Added: trunk/src/test/java/au/csiro/netcdf/wron/TestMdbsyNetCDF2CSVConverter.java
===================================================================
--- trunk/src/test/java/au/csiro/netcdf/wron/TestMdbsyNetCDF2CSVConverter.java (rev 0)
+++ trunk/src/test/java/au/csiro/netcdf/wron/TestMdbsyNetCDF2CSVConverter.java 2010-07-08 02:05:42 UTC (rev 67)
@@ -0,0 +1,227 @@
+package au.csiro.netcdf.wron;
+
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintStream;
+import java.util.ArrayList;
+
+import au.csiro.netcdf.NcWriteVariable;
+import au.csiro.netcdf.util.Util;
+
+import ucar.ma2.DataType;
+import ucar.nc2.Dimension;
+import ucar.nc2.NCdumpW;
+import ucar.nc2.NetcdfFile;
+import ucar.nc2.NetcdfFileWriteable;
+import ucar.nc2.Variable;
+import junit.framework.TestCase;
+
+/**
+ * This class is a unit test suite to verify that the {@link MdbsyNetCDF2CSVConverter} operates correctly.
+ *
+ * Copyright 2010, CSIRO Australia
+ *
+ * @author Robert Bridle on 08/07/2010
+ * @version $Revision: {revision} $ $Date: 2008-10-31 11:24:49 +1100 (Fri, 31 Oct 2008) $
+ */
+public class TestMdbsyNetCDF2CSVConverter extends TestCase
+{
+ /**
+ * The testing value for the latitude coordinate dimension.
+ */
+ private static final String DIM_LATITUDE = "lat";
+ private static final String VAR_LATITUDE = DIM_LATITUDE;
+ private static final String VAR_LATITUDE_VALUE = "-24.25";
+
+ /**
+ * The testing value for the longitude coordinate dimension.
+ */
+ private static final String DIM_LONGITUDE = "long";
+ private static final String VAR_LONGITUDE = DIM_LONGITUDE;
+ private static final String VAR_LONGITUDE_VALUE = "138.10";
+
+ /**
+ * The expected name of the converted csv file.
+ */
+ private static final String CONVERTED_CSV_FILE_NAME = System.getProperty("user.dir") + "\\region_-24.25_138.1.csv";
+
+ /**
+ * The testing value for the time coordinate dimension.
+ */
+ private static final String DIM_TIME = "time";
+ private static final String VAR_TIME = DIM_TIME;
+ private static final String VAR_TIME_VALUES = "0" + System.getProperty("line.separator") + "1"
+ + System.getProperty("line.separator") + "2" + System.getProperty("line.separator") + "3"
+ + System.getProperty("line.separator") + "4" + System.getProperty("line.separator") + "5"
+ + System.getProperty("line.separator") + "6" + System.getProperty("line.separator") + "7"
+ + System.getProperty("line.separator") + "8" + System.getProperty("line.separator") + "9";
+
+ /**
+ * The testing value for a variable defined over three coordinate dimensions (lat, long, time).
+ */
+ private static final String VAR_APET = "APET";
+ private static final String VAR_APET_VALUES = "0.000" + System.getProperty("line.separator") + "0.111"
+ + System.getProperty("line.separator") + "0.222" + System.getProperty("line.separator") + "0.333"
+ + System.getProperty("line.separator") + "0.444" + System.getProperty("line.separator") + "0.555"
+ + System.getProperty("line.separator") + "0.666" + System.getProperty("line.separator") + "0.777"
+ + System.getProperty("line.separator") + "0.888" + System.getProperty("line.separator") + "0.999";
+
+
+ /**
+ * The expected output converted csv structure.
+ */
+ private static String[] EXPECTED_CSV_FILE = { "Date, APET",
+ "1895-01-01, 0.0",
+ "1895-01-02, 0.111",
+ "1895-01-03, 0.222",
+ "1895-01-04, 0.333",
+ "1895-01-05, 0.444",
+ "1895-01-06, 0.555",
+ "1895-01-07, 0.666",
+ "1895-01-08, 0.777",
+ "1895-01-09, 0.888",
+ "1895-01-10, 0.999" };
+
+ /**
+ * The testing value for the dimension sizes.
+ */
+ private static final int LAT_DIM_SIZE = 1;
+ private static final int LONG_DIM_SIZE = 1;
+ private static final int TIME_DIM_SIZE = 10;
+
+ /**
+ * The testing value for the netCDF file to write to.
+ */
+ private static final String NC_FILE_NAME = System.getProperty("user.dir") + "\\ABC.nc";
+
+ /*
+ * (non-Javadoc)
+ *
+ * @see junit.framework.TestCase#setUp()
+ */
+ @Override
+ protected void setUp() throws Exception
+ {
+ super.setUp();
+
+ // create dummy netCDF file with content.
+ createDummyNCFile(NC_FILE_NAME);
+ }
+
+ /*
+ * (non-Javadoc)
+ *
+ * @see junit.framework.TestCase#tearDown()
+ */
+ @Override
+ protected void tearDown() throws Exception
+ {
+ super.tearDown();
+
+ System.setOut(null);
+
+ // delete dummy netCDF file
+ File ncFile = new File(NC_FILE_NAME);
+ if (ncFile.exists())
+ {
+ NetcdfFile netcdfFile = NetcdfFile.open(NC_FILE_NAME);
+ netcdfFile.close();
+ ncFile.delete();
+ }
+
+ // delete converted csv file
+ File csvFile = new File(CONVERTED_CSV_FILE_NAME);
+ if (csvFile.exists())
+ {
+ csvFile.delete();
+ }
+ }
+
+ /**
+ * Test execute a valid NcWriteVariable command using Double datatype
+ */
+ public final void testNetCDF2CSVConversion() throws Exception
+ {
+ MdbsyNetCDF2CSVConverter.main(new String[] { "-i", System.getProperty("user.dir"), "-o",
+ System.getProperty("user.dir") });
+
+ // check that the converted csv file was created.
+ assertTrue("The expected converted csv file does not exist: " + CONVERTED_CSV_FILE_NAME, Util
+ .fileExists(CONVERTED_CSV_FILE_NAME));
+
+ String[][] csvMatrix = MdbsyNetCDF2CSVConverter.readLookupFile(new File(CONVERTED_CSV_FILE_NAME));
+
+ for (int row = 0; row < csvMatrix.length; row++)
+ {
+ StringBuffer line = new StringBuffer();
+ for (int col = 0; col < csvMatrix[0].length; col++)
+ {
+ line.append(csvMatrix[row][col]);
+ if(col < (csvMatrix[0].length-1))
+ {
+ line.append(", ");
+ }
+ }
+
+ assertEquals("The expected csv output is not correct.", EXPECTED_CSV_FILE[row], line.toString());
+ }
+ }
+
+ /**
+ * Makes a dummy netCDF file with known contents.
+ */
+ private void createDummyNCFile(String outputFilename) throws IOException
+ {
+ NetcdfFileWriteable ncFile = null;
+
+ ncFile = NetcdfFileWriteable.createNew(outputFilename, true);
+ try
+ {
+ Dimension dimension = new Dimension(DIM_LATITUDE, LAT_DIM_SIZE, true /* isShared */,
+ false /* isUnlimited */, false /* isVariableLength */);
+ ncFile.addDimension(null, dimension);
+
+ dimension = new Dimension(DIM_LONGITUDE, LONG_DIM_SIZE, true /* isShared */, false /* isUnlimited */, false /* isVariableLength */);
+ ncFile.addDimension(null, dimension);
+
+ dimension = new Dimension(DIM_TIME, TIME_DIM_SIZE, true /* isShared */, false /* isUnlimited */, false /* isVariableLength */);
+ ncFile.addDimension(null, dimension);
+
+ // define coordinate variables.
+ ncFile.addVariable(VAR_LATITUDE, DataType.FLOAT, DIM_LATITUDE);
+ ncFile.addVariable(VAR_LONGITUDE, DataType.FLOAT, DIM_LONGITUDE);
+ ncFile.addVariable(VAR_TIME, DataType.INT, DIM_TIME);
+
+ // variable is defined over three coordinates dimensions.
+ ncFile.addVariable(VAR_APET, DataType.FLOAT, DIM_LATITUDE + " " + DIM_LONGITUDE + " " + DIM_TIME);
+ ncFile.create();
+ }
+ finally
+ {
+ ncFile.close();
+ }
+
+ // populate the coordinate variables with string of int values.
+ NcWriteVariable command = new NcWriteVariable();
+ command.execute(new File(outputFilename), VAR_LATITUDE, "0-" + String.valueOf(LAT_DIM_SIZE - 1),
+ new ByteArrayInputStream(VAR_LATITUDE_VALUE.getBytes("UTF-8")), false);
+ command.execute(new File(outputFilename), VAR_LONGITUDE, "0-" + String.valueOf(LONG_DIM_SIZE - 1),
+ new ByteArrayInputStream(VAR_LONGITUDE_VALUE.getBytes("UTF-8")), false);
+ command.execute(new File(outputFilename), VAR_TIME, "0-" + String.valueOf(TIME_DIM_SIZE - 1),
+ new ByteArrayInputStream(VAR_TIME_VALUES.getBytes("UTF-8")), false);
+
+ try
+ {
+ command.execute(new File(outputFilename), VAR_APET, "0-" + String.valueOf(LAT_DIM_SIZE - 1) + ", 0-"
+ + String.valueOf(LONG_DIM_SIZE - 1) + ", 0-" + String.valueOf(TIME_DIM_SIZE - 1),
+ new ByteArrayInputStream(VAR_APET_VALUES.getBytes("UTF-8")), false);
+ }
+ catch (IllegalArgumentException iae)
+ {
+ iae.printStackTrace();
+ fail("could not create file: " + outputFilename + " to being testing of netCDF 2 CSV converter");
+ }
+ }
+}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <rob...@us...> - 2010-07-13 23:12:56
|
Revision: 74
http://netcdftools.svn.sourceforge.net/netcdftools/?rev=74&view=rev
Author: robertbridle
Date: 2010-07-13 23:12:49 +0000 (Tue, 13 Jul 2010)
Log Message:
-----------
ANDSWRON-665 - Place IPCC standard model names as metadata in the netCDF files.
Modified Paths:
--------------
trunk/launch/ScenarioCConverter-AllData.launch
trunk/launch/ScenarioCConverter.launch
trunk/src/main/java/au/csiro/netcdf/wron/MdbsyScenarioCConverter.java
Modified: trunk/launch/ScenarioCConverter-AllData.launch
===================================================================
--- trunk/launch/ScenarioCConverter-AllData.launch 2010-07-13 06:36:28 UTC (rev 73)
+++ trunk/launch/ScenarioCConverter-AllData.launch 2010-07-13 23:12:49 UTC (rev 74)
@@ -8,7 +8,7 @@
</listAttribute>
<stringAttribute key="org.eclipse.jdt.launching.CLASSPATH_PROVIDER" value="org.maven.ide.eclipse.launchconfig.classpathProvider"/>
<stringAttribute key="org.eclipse.jdt.launching.MAIN_TYPE" value="au.csiro.netcdf.wron.MdbsyScenarioCConverter"/>
-<stringAttribute key="org.eclipse.jdt.launching.PROGRAM_ARGUMENTS" value="--lookupFile "C:\dev\scenC\unq_cells_dem9s2rc.csv_corLat_regions.csv" --inputDirectory "U:\_ScenarioC_RRModellingData\cccma_t47\High_Global_Warming\\" --outputDirectory "C:\dev\scenC\output\\" --metadataFile "C:\dev\scenC\attributes.txt" --scenario "Scenario-C" --model "Model-ccma_t47" --case "Case-High" -r 2 --decade --latitude"/>
+<stringAttribute key="org.eclipse.jdt.launching.PROGRAM_ARGUMENTS" value="--lookupFile "C:\dev\scenC\unq_cells_dem9s2rc.csv_corLat_regions.csv" --inputDirectory "U:\_ScenarioC_RRModellingData\cccma_t47\High_Global_Warming\\" --outputDirectory "C:\dev\scenC\output\\" --metadataFile "C:\dev\scenC\attributes.txt" --scenario "Scenario-C" --model "Model-cccma_t47" --case "Case-High" -r 2 --decade --latitude"/>
<stringAttribute key="org.eclipse.jdt.launching.PROJECT_ATTR" value="netcdf-tools"/>
<stringAttribute key="org.eclipse.jdt.launching.SOURCE_PATH_PROVIDER" value="org.maven.ide.eclipse.launchconfig.sourcepathProvider"/>
<stringAttribute key="org.eclipse.jdt.launching.VM_ARGUMENTS" value="-Xmx1500m -Xms800m"/>
Modified: trunk/launch/ScenarioCConverter.launch
===================================================================
--- trunk/launch/ScenarioCConverter.launch 2010-07-13 06:36:28 UTC (rev 73)
+++ trunk/launch/ScenarioCConverter.launch 2010-07-13 23:12:49 UTC (rev 74)
@@ -8,7 +8,7 @@
</listAttribute>
<stringAttribute key="org.eclipse.jdt.launching.CLASSPATH_PROVIDER" value="org.maven.ide.eclipse.launchconfig.classpathProvider"/>
<stringAttribute key="org.eclipse.jdt.launching.MAIN_TYPE" value="au.csiro.netcdf.wron.MdbsyScenarioCConverter"/>
-<stringAttribute key="org.eclipse.jdt.launching.PROGRAM_ARGUMENTS" value="--lookupFile "C:\dev\scenC\unq_cells_dem9s2rc.csv_corLat_regions.csv" --inputDirectory "U:\_ScenarioC_RRModellingData\cccma_t47\High_Global_Warming\\" --outputDirectory "C:\dev\scenC\output\\" --metadataFile "C:\dev\scenC\attributes.txt" --scenario "Scenario-C" --model "Model-ccma_t47" --case "Case-High" -r 250 --decade --latitude"/>
+<stringAttribute key="org.eclipse.jdt.launching.PROGRAM_ARGUMENTS" value="--lookupFile "C:\dev\scenC\unq_cells_dem9s2rc.csv_corLat_regions.csv" --inputDirectory "U:\_ScenarioC_RRModellingData\cccma_t47\High_Global_Warming\\" --outputDirectory "C:\dev\scenC\output\\" --metadataFile "C:\dev\scenC\attributes.txt" --scenario "Scenario-C" --model "Model-cccma_t47" --case "Case-High" -r 250 --latitude"/>
<stringAttribute key="org.eclipse.jdt.launching.PROJECT_ATTR" value="netcdf-tools"/>
<stringAttribute key="org.eclipse.jdt.launching.SOURCE_PATH_PROVIDER" value="org.maven.ide.eclipse.launchconfig.sourcepathProvider"/>
<stringAttribute key="org.eclipse.jdt.launching.VM_ARGUMENTS" value="-Xmx1500m -Xms800m"/>
Modified: trunk/src/main/java/au/csiro/netcdf/wron/MdbsyScenarioCConverter.java
===================================================================
--- trunk/src/main/java/au/csiro/netcdf/wron/MdbsyScenarioCConverter.java 2010-07-13 06:36:28 UTC (rev 73)
+++ trunk/src/main/java/au/csiro/netcdf/wron/MdbsyScenarioCConverter.java 2010-07-13 23:12:49 UTC (rev 74)
@@ -82,7 +82,7 @@
// netCDF file naming components
private static String COLLECTION = "Collection-MDBSY_Climate";
private static String SCENARIO = "";
- private static String MODEL = "";
+ private static String[] MODEL_NAMES;
private static String CASE = "";
/**
@@ -94,7 +94,7 @@
* The file containing the metadata attributes (if any).
*/
private static String METADATA_FILE = "";
-
+
/**
* The directory containing the Scenario C csv data files.
*/
@@ -114,6 +114,34 @@
* Encoding used to fill variables.
*/
private static final String ENCODING = "UTF-8";
+
+ /**
+ * The 15 climate models used in the MDBSY project.
+ */
+ private static Map<String, String> CLIMATE_MODELS = new HashMap<String, String>();
+ static
+ {
+ CLIMATE_MODELS.put("cccma_t47", "CCCMA T47");
+ CLIMATE_MODELS.put("cccma_t63", "CCCMA T63");
+ CLIMATE_MODELS.put("cnrm", "CNRM");
+ CLIMATE_MODELS.put("csiro", "CSIRO-MK3.0");
+ CLIMATE_MODELS.put("gfdl", "GFDL 2.0");
+ CLIMATE_MODELS.put("giss_aom", "GISS-AOM");
+ CLIMATE_MODELS.put("iap", "IAP");
+ CLIMATE_MODELS.put("inmcm", "INMCM");
+ CLIMATE_MODELS.put("ipsl", "IPSL");
+ CLIMATE_MODELS.put("miroc", "MIROC-M");
+ CLIMATE_MODELS.put("miub", "MIUB");
+ CLIMATE_MODELS.put("mpi", "MPI-ECHAM5");
+ CLIMATE_MODELS.put("mri", "MRI");
+ CLIMATE_MODELS.put("ncar_ccsm", "NCAR-CCSM");
+ CLIMATE_MODELS.put("ncar_pcm", "NCAR-PCM1");
+ }
+ /**
+ * Index of model names in {@link MdbsyScenarioCConverter#MODEL_NAMES}.
+ */
+ private static final int FILE_NAME_MODEL_NAME = 0;
+ private static final int IPCC_MODEL_NAME = 1;
// Variables
private static final String[] variableNames = new String[] { "rainfall", "APET" };
@@ -291,7 +319,7 @@
SCENARIO = (parsedCommandLine.hasOption("scenario")) ? parsedCommandLine.getOptionValue("scenario") : "";
METADATA_FILE = (parsedCommandLine.hasOption("metadataFile")) ? parsedCommandLine
.getOptionValue("metadataFile") : "";
- MODEL = (parsedCommandLine.hasOption("model")) ? parsedCommandLine.getOptionValue("model") : "";
+ MODEL_NAMES = getModelMapping((parsedCommandLine.hasOption("model")) ? parsedCommandLine.getOptionValue("model") : "");
CASE = (parsedCommandLine.hasOption("case")) ? parsedCommandLine.getOptionValue("case") : "";
ROW_CHUNK_SIZE = (parsedCommandLine.hasOption("rows")) ? Integer.valueOf(parsedCommandLine
.getOptionValue("rows")) : ROW_CHUNK_SIZE;
@@ -526,9 +554,11 @@
// Add attributes
DateFormat utcDateTime = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ssZ");
Attribute converter = new Attribute("History", utcDateTime.format(new Date()) + " Converted to netCDF by "
- + "$Id: MdbsyScenarioCConverter.java 7155 2010-06-10 02:51:25Z dem040 $");
+ + "$Id: MdbsyScenarioCConverter.java 7155 2010-06-10 02:51:25Z dem040 $");
+ Attribute ipccClimateModel = new Attribute("IPCC Climate Model Name", MODEL_NAMES[IPCC_MODEL_NAME]);
List<Attribute> globalAttrs = new ArrayList<Attribute>();
globalAttrs.add(converter);
+ globalAttrs.add(ipccClimateModel);
if (METADATA_FILE.length() > 0)
{
globalAttrs.addAll(NetCDFUtils.readAttributesFromStream(new FileInputStream(METADATA_FILE)));
@@ -1259,9 +1289,9 @@
private String generateLatitudeFilename(String latitudeDegree, String variable)
{
String filename = OUTPUT_NETCDF_DIRECTORY + COLLECTION + "." + SCENARIO + ".";
- if (MODEL.length() > 0)
+ if (MODEL_NAMES[FILE_NAME_MODEL_NAME].length() > 0)
{
- filename += MODEL + ".";
+ filename += MODEL_NAMES[FILE_NAME_MODEL_NAME] + ".";
}
if (CASE.length() > 0)
{
@@ -1283,9 +1313,9 @@
private String generateDecadeFilename(String decadeStr, String variable)
{
String filename = OUTPUT_NETCDF_DIRECTORY + COLLECTION + "." + SCENARIO + ".";
- if (MODEL.length() > 0)
+ if (MODEL_NAMES[FILE_NAME_MODEL_NAME].length() > 0)
{
- filename += MODEL + ".";
+ filename += MODEL_NAMES[FILE_NAME_MODEL_NAME] + ".";
}
if (CASE.length() > 0)
{
@@ -1324,5 +1354,34 @@
{
CELLID, LONGITUDE, LATITUDE, ELEVATION, CATCHMENTID, REPORTINGREGIONID
}
+
+ /**
+ * Maps a climate model name used for file naming to an IPCC standard climate model name.
+ *
+ * @param directoryModelName
+ * a climate model name used for file naming.
+ * @return a String[] which forms a mapping from a climate model name used for file naming to an IPCC standard
+ * climate model name.
+ */
+ private static String[] getModelMapping(String directoryModelName)
+ {
+ // strip "Model-" prefix that may be used.
+ int beginIndex = directoryModelName.indexOf("Model-");
+ if (beginIndex != -1)
+ {
+ directoryModelName = directoryModelName.substring(beginIndex + ("Model-1".length() - 1));
+ }
+ // determine the mapping between the climate model used to name the file and the IPCC standard climate
+ // model name stored as metadata.
+ if (CLIMATE_MODELS.containsKey(directoryModelName))
+ {
+ String[] modelNameMapping = new String[2];
+ modelNameMapping[FILE_NAME_MODEL_NAME] = directoryModelName;
+ modelNameMapping[IPCC_MODEL_NAME] = CLIMATE_MODELS.get(directoryModelName);
+ return modelNameMapping;
+ }
+ throw new IllegalArgumentException("Unknown model: " + directoryModelName + ". Valid models are: "
+ + CLIMATE_MODELS.keySet());
+ }
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X