Class cwexport
The export tool translates earth data into external file formats.
Name
cwexport - translates earth data into external file formats.
Synopsis
cwexport [OPTIONS] input output
General options:
-f, --format=TYPE
-h, --help
-H, --header
-m, --match=PATTERN
-M, --missing=VALUE
-v, --verbose
--version
Binary raster options:
-c, --scale=FACTOR/OFFSET
-o, --byteorder=ORDER
-r, --range=MIN/MAX
-s, --size=TYPE
ASCII text options:
-d, --dec=DECIMALS
-D, --delimit=STRING
-n, --nocoords
-R, --reverse
NetCDF options:
-S, --dcs
-C, --cw
GeoTIFF options:
-T, --tiffcomp=TYPE
-F, --tiffsize=TYPE
Description
The export tool translates earth data into external formats as described below. In all cases, 2D data sets are exported in row major order starting from row 0. For example, if the earth data values form the 2D array:
0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15
then values are output in the order:
0 1 2 3 4 5 6 7 ...
In the general case, multiple variables may be exported to the same data file. The use of the --match option may be used to select a specific variable or subset of variables.
Binary raster:
The output is a stream of binary data values — either 8-bit unsigned bytes, 16-bit signed integers, or 32-bit IEEE floating point values. For 8- and 16-bit output, data values may be scaled to integers (essentially a packing scheme to reduce the output file size) using a minimum and maximum or by using a scaling factor and offset. For minimum/maximum scaling, integer data is calculated from data values using the equation:
integer = type_min + type_range*((value - min) / (max - min))
where type_min is 0 for 8-bit and -32768 for 16-bit, and
type_range is 255 for 8-bit and 65535 for 16-bit. For
scaling factor and offset, the following equation is used:
integer = value/factor + offset
In both cases, the results are rounded to the nearest integer and out of range values are assigned the missing value.
ASCII text:
The output is an ASCII text file with latitude, longitude, and data value printed — one data value per line.
ArcGIS binary grid:
The output is a stream of 32-bit IEEE floating point values, ready for input to ArcGIS applications as a binary grid file. A header file may also be created to specify the earth location and other parameters. In the case of the ArcGIS format, only one variable is allowed per binary grid file. If an attempt to export multiple variables is made, only the first variable is actually written.
NetCDF:
The output is either a NetCDF 3 or 4 dataset with CF 1.4 convention metadata. The formatting follows as much as possible the recommendations and examples in the document "Encoding CoastWatch Satellite Data in NetCDF using the CF Metadata Conventions", Peter Hollemans, February 2010. In some cases, the source data for some attributes may not be available, in which case the output NetCDF may need to be modified and extended.
GeoTIFF:
The output is an 8-bit unsigned integer, 16-bit unsigned integer, or 32-bit IEEE floating pointing data TIFF file with GeoTIFF georeference tags. The number of samples per pixel in the TIFF image matches the number of variables exported. The resulting image file is not suitable for display as a regular TIFF image, but rather is meant for import into a GIS package.
Parameters
Main parameters:
- input
- The input data file name.
- output
- The output data file name. Unless the --format
option is used, the output file extension indicates the
desired output format:
- filename.raw = binary
- filename.txt = text
- filename.flt = ArcGIS
- filename.nc = NetCDF 3
- filename.nc4 = NetCDF 4
- filename.tif = GeoTIFF
General options:
- -f, --format=TYPE
- The output format. The current formats are 'bin' for binary raster, 'text' for ASCII text, 'arc' for ArcGIS binary grid, 'netcdf' for NetCDF 3, 'netcdf4' for NetCDF 4, 'geotiff' for GeoTIFF, or 'auto' to detect the format from the output file name. The default is 'auto'.
- -h, --help
- Prints a brief help message.
- -H, --header
- Specifies that a header should be written as well as the output data.
The method of writing the header is different depending on the
output format:
- Binary: The header consists of one byte specifying the number of dimensions followed by a series of 32-bit signed integers specifying the dimension lengths. The header is written on a per-variable basis, before each block of variable values in the output file.
- Text: The header is one line consisting of an integer specifying the number of dimensions followed by a series of integers specifying the dimension lengths. The header is written on a per-variable basis before each set of variable values in the output file.
- ArcGIS: The header is a separate file used by ArcGIS applications to determine the dimensions of the data, the geographic position and resolution, and other parameters. The header file name is created by replacing any '.' followed by an extension in the output file name with '.hdr'
- NetCDF, GeoTIFF: Not applicable, all metadata in NetCDF and GeoTIFF is written into the dataset itself.
- -m, --match=PATTERN
- The variable name matching pattern. If specified, the pattern is used as a regular expression to match variable names. Only variables matching the pattern will be exported. By default, no pattern matching is performed and all variables are exported.
- -M, --missing=VALUE
- The output value for missing or out of range data. The
default missing value is different depending on the output format:
- Binary: The default is 0 for 8-bit unsigned bytes, -32768 for 16-bit signed integers, and the IEEE NaN value for floating point.
- Text: The default is to print 'NaN' for missing values.
- ArcGIS: The missing value is fixed at -3.4e38 and the --missing option is ignored.
- NetCDF: Not applicable, the missing values for variable data are copied from the input data source.
- GeoTIFF: The default is to write (i) zeros for integer data types, or (ii) 32-float IEEE NaN values with bit pattern 0x7fc00000 for 32-bit float data in the TIFF image data.
- -v, --verbose
- Turns verbose mode on. The current status of data conversion is printed periodically. The default is to run quietly.
- --version
- Prints the software version.
Binary raster options:
- -c, --scale=FACTOR/OFFSET
- The data scale factor and offset for integer packing. Data values are scaled to integers using the factor and offset (see the equation in the Description section above, under Binary raster). The default factor is 1 and offset is 0.
- -o, --byteorder=ORDER
- The output byte order. Valid choices are 'host' for the host byte order, 'msb' for most significant byte first, or 'lsb' for least significant byte first. The default is the host byte order.
- -r, --range=MIN/MAX
- The data scaling range for integer packing. Data values are mapped to integers using the minimum and maximum values (see the equation in the Description section above, under Binary raster). There is no default range.
- -s, --size=TYPE
- The binary value size. Valid choices are 'byte' for 8-bit unsigned bytes, 'short' for 16-bit signed integers, or 'float' for 32-bit IEEE floating point values. The default is 32-bit floats.
ASCII text options:
- -d, --dec=DECIMALS
- The number of decimal places for printed geographic coordinate values. The default is 6 decimals.
- -D, --delimit=STRING
- The value delimiter string. By default, values are separated with a single space character.
- -n, --nocoords
- Turns geographic coordinate printing off. By default, each line has the form 'latitude longitude value' but with no coordinates, each line simply contains the data value.
- -R, --reverse
- Specifies that coordinates should be printed in reverse order, 'longitude latitude'. The default is 'latitude longitude'.
NetCDF options:
- -S, --dcs
- Turns on writing of ocean color Data Content Standard metadata. The default is to write only CF-1.4 metadata. DCS metadata is written as a set of global NetCDF attributes with the namespace prefix 'dcs'. Only the minimal set of 15 required attributes are written. Since the NetCDF file will generally contain more than one variable, the required DCS attributes observedProperty and observedPropertyAlgorithm are set to 'Unknown' and must be modified manually.
- -C, --cw
- Turns on writing of CoastWatch HDF-style metadata. The default is to write only CF-1.4 metadata. CoastWatch metadata is written as a set of global- and variable-level NetCDF attributes with the namespace prefix 'cw'. Only a very small subset of the original CoastWatch HDF metadata is written, those attributes that have no CF-1.4 equivalent.
GeoTIFF options:
- -T, --tiffcomp=TYPE
- The TIFF compression algorithm. The valid types are 'none' for no compression, and 'deflate' or 'lzw' for ZIP style compression. The default is to use deflate compression.
- -F, --tiffsize=TYPE
- The TIFF data value size. The valid types are 'byte' for 8-bit unsigned integers, 'ushort' for 16-bit unsigned integers, or 'float' for 32-bit IEEE floating point values. The default is to write 32-bit floats.
Exit status
0 on success, > 0 on failure. Possible causes of errors:
- Invalid command line option
- Invalid input or output file names
- Invalid variable name
- Unrecognized format, size, or byte order
Examples
The following shows the export of AVHRR channel 1 data from a CoastWatch HDF file with to a binary 32-bit IEEE floating point value format, host byte order, no header, in verbose mode:
phollema$ cwexport -v --match 'avhrr_ch1' 2019_250_2241_n19_mr.hdf
2019_250_2241_n19_mr_ch1.raw
[INFO] Creating output 2019_250_2241_n19_mr_ch1.raw
[INFO] Writing avhrr_ch1
The example below shows the export of AVHRR channels 1, 2, and 4 to the same output file from a CoastWatch HDF file using 8-bit unsigned byte format, no header, in verbose mode. Range scaling is used to scale all values between -30 and 30 prior to conversion to byte values in the range 0 to 255. Note that some values may fall outside the range and be clipped, especially albedo values which can range up to 100. The clipped values are assigned the default missing value, which for byte data is 0.
phollema$ cwexport -v --match 'avhrr_ch[124]' --size byte --range -30/30
2019_250_2241_n19_mr.hdf 2019_250_2241_n19_mr_ch124.raw
[INFO] Creating output 2019_250_2241_n19_mr_ch124.raw
[INFO] Writing avhrr_ch1
[INFO] Writing avhrr_ch2
[INFO] Writing avhrr_ch4
The next example shows the export of AVHRR channel 4 data to an ASCII text file from a CoastWatch HDF file in verbose mode. The geographic coordinates are printed in the order (longitude, latitude), and delimited with a comma character. Any missing values are denoted with the value -999. A one line dimension header is prepended to the dataset.
phollema$ cwexport -v --match 'avhrr_ch4' --format text --reverse
--delimit ',' --missing -999 --header 2019_250_2241_n19_mr.hdf
2019_250_2241_n19_mr_ch4.txt
[INFO] Creating output 2019_250_2241_n19_mr_ch4.txt
[INFO] Writing avhrr_ch4
The first few lines of the output file are as follows:
2,1101,1401 -98.243664,31.051575,31.48 -98.230459,31.051575,31.59 -98.217254,31.051575,31.59 -98.204049,31.051575,31.59 -98.190843,31.051575,32.03 -98.177638,31.051575,32.03 -98.164433,31.051575,32.47 -98.151228,31.051575,32.25 -98.138022,31.051575,32.58
The example below shows the export of AVHRR channel 2 data to an ArcGIS binary grid file from a CoastWatch HDF file in verbose mode. The binary grid data is written to a '.flt' file and the header data to a '.hdr' file.
phollema$ cwexport -v --format arc --match 'avhrr_ch2' --header
2019_250_2241_n19_mr.hdf 2019_250_2241_n19_mr_ch4.flt
[INFO] Creating output 2019_250_2241_n19_mr_ch4.flt
[INFO] Writing avhrr_ch2
The header data is written to 2019_250_2241_n19_mr_ch4.hdr as follows:
nrows 1101 ncols 1401 xllcorner -1.0937169680601347E7 yllcorner 1999676.93911416 cellsize 1470.0 nodata_value -3.4E38 byteorder MSBFIRST nbits 32
A final example shows the export of SST and cloud data to a NetCDF dataset:
phollema$ cwexport -v --match '(sst|cloud)' 2010_040_1636_m02_wj.hdf
2010_040_1636_m02_wj.nc
[INFO] Creating output 2010_040_1636_m02_wj.nc
[INFO] Writing cloud
[INFO] Writing sst
Running the NetCDF software ncdump -h command shows the
file contents:
netcdf 2010_040_1636_m02_wj {
dimensions:
time = 1 ;
level = 1 ;
row = 1024 ;
column = 1024 ;
variables:
int coord_ref ;
coord_ref:grid_mapping_name = "mercator" ;
coord_ref:longitude_of_projection_origin = 0. ;
coord_ref:standard_parallel = 0. ;
coord_ref:false_easting = 0. ;
coord_ref:false_northing = 0. ;
coord_ref:semi_major_axis = 6378137. ;
coord_ref:inverse_flattening = 298.257223653 ;
coord_ref:longitude_of_prime_meridian = 0. ;
double x(column) ;
x:standard_name = "projection_x_coordinate" ;
x:units = "m" ;
double y(row) ;
y:standard_name = "projection_y_coordinate" ;
y:units = "m" ;
double lat(row, column) ;
lat:standard_name = "latitude" ;
lat:units = "degrees_north" ;
double lon(row, column) ;
lon:standard_name = "longitude" ;
lon:units = "degrees_east" ;
double time(time) ;
time:standard_name = "time" ;
time:units = "seconds since 1970-01-01 00:00:00 UTC" ;
double level(level) ;
level:standard_name = "height" ;
level:units = "m" ;
level:positive = "up" ;
byte cloud(time, level, row, column) ;
cloud:missing = 0b ;
cloud:valid_range = 0, 255 ;
cloud:coordinates = "lat lon" ;
cloud:cell_methods = "area: mean" ;
cloud:grid_mapping = "coord_ref" ;
short sst(time, level, row, column) ;
sst:scale_factor = 0.01 ;
sst:add_offset = -0. ;
sst:missing = -32768s ;
sst:units = "celsius" ;
sst:coordinates = "lat lon" ;
sst:cell_methods = "area: mean" ;
sst:grid_mapping = "coord_ref" ;
sst:source = "nonlinear_split_window linear_triple_window_modified" ;
// global attributes:
:Conventions = "CF-1.4" ;
:source = "METOP2_AVHRR " ;
:institution = "USDOC/NOAA/NESDIS CoastWatch" ;
:history = "[2010-03-13 09:46:43 IST cwf-3.2.4-pre-build169 phollema] cwexport
-v --match (sst|cloud) 2010_040_1636_m02_wj.hdf 2010_040_1636_m02_wj.nc" ;
}
- Since:
- 3.1.0
- Author:
- Mark Robinson
-
Method Summary
-
Method Details
-
main
Performs the main function.- Parameters:
argv- the list of command line parameters.
-