From 7c51c3a94a560bcdb95c4ff6405f2d4f948e9d02 Mon Sep 17 00:00:00 2001
From: Jeremy Fee <jmfee@usgs.gov>
Date: Wed, 11 May 2016 14:24:49 -0600
Subject: [PATCH] Update documentation examples based on new input/output
arguments
---
docs/algorithms/DeltaF_usage.md | 6 +-
docs/algorithms/SqDist_usage.md | 10 +-
docs/algorithms/XYZ_usage.md | 16 ++-
docs/io.md | 217 +++++++++++++++++++-------------
docs/io/Edge.md | 6 +-
docs/io/Iaga2002.md | 6 +-
docs/metadata.md | 5 +-
docs/usage.md | 38 ++++--
8 files changed, 183 insertions(+), 121 deletions(-)
diff --git a/docs/algorithms/DeltaF_usage.md b/docs/algorithms/DeltaF_usage.md
index beb0b2d02..67b6a9f99 100644
--- a/docs/algorithms/DeltaF_usage.md
+++ b/docs/algorithms/DeltaF_usage.md
@@ -29,9 +29,9 @@ geomag.py \
--outchannels G \
--starttime 2015-11-01T00:00:00Z \
--endtime 2015-11-01T00:10:00Z \
- --input-edge cwbpub.cr.usgs.gov \
- --input-edge-port 2060 \
- --output-iaga-stdout
+ --input edge \
+ --output iaga2002 \
+ --output-stdout
```
diff --git a/docs/algorithms/SqDist_usage.md b/docs/algorithms/SqDist_usage.md
index 78df38eda..07dc7e54d 100644
--- a/docs/algorithms/SqDist_usage.md
+++ b/docs/algorithms/SqDist_usage.md
@@ -13,7 +13,7 @@ This example uses a state file to produce magnetic-h-based Dist, SQ, and SV
channels using the EDGE channel naming convention.
bin/geomag.py \
- --input-edge cwbpub.cr.usgs.gov \
+ --input edge \
--observatory BOU \
--inchannels H E Z F \
--starttime 2016-01-03T00:01:00 \
@@ -25,12 +25,13 @@ channels using the EDGE channel naming convention.
--rename-output-channel H_SV MSV \
--rename-output-channel H_Dist MDT \
--outchannels MDT MSQ MSV \
- --output-iaga-stdout
+ --output iaga2002 \
+ --output-stdout
This example processes just one channel (X).
bin/geomag.py \
- --input-edge cwbpub.cr.usgs.gov \
+ --input edge \
--observatory BOU \
--inchannels X \
--starttime 2016-01-03T00:01:00 \
@@ -41,7 +42,8 @@ This example processes just one channel (X).
--rename-output-channel X_SQ MXQ \
--rename-output-channel X_SV MXV \
--outchannels MXT MXQ MXV \
- --output-iaga-stdout
+ --output iaga \
+ --output-stdout
> Note only one inchannel is specified and the --sqdist-mag option is omitted.
diff --git a/docs/algorithms/XYZ_usage.md b/docs/algorithms/XYZ_usage.md
index d154887f8..bf38c881e 100644
--- a/docs/algorithms/XYZ_usage.md
+++ b/docs/algorithms/XYZ_usage.md
@@ -30,11 +30,17 @@ There are 3 reference frames in this library.
To convert HEZF data in pcdcp files to XYZF for Tucson observatory for all of
March 2013 output to iaga2002 files:
- geomag.py --algorithm xyz --observatory TUC \
- --starttime 2013-03-01T00:00:00Z --endtime 2013-03-31T23:59:00Z \
- --input-pcdcp-url file://data-pcdcp/./%(OBS)s%(year)s%(julian)s.%(i)s \
- --output-iaga-url file://data-iaga/./$(obs)s%(Y)s%(j)s.%(i)s \
- --type variation --interval minute
+ bin/geomag.py \
+ --algorithm xyz \
+ --observatory TUC \
+ --starttime 2013-03-01T00:00:00Z \
+ --endtime 2013-03-31T23:59:00Z \
+ --input pcdcp \
+ --input-url file://data-pcdcp/./{OBS}{date:%Y%j}.{i} \
+ --output iaga2002 \
+ --iaga-url file://data-iaga/./{obs}{date:%Y%j}.{i} \
+ --type variation \
+ --interval minute
### Library Notes
diff --git a/docs/io.md b/docs/io.md
index cd75237ae..7933877dc 100644
--- a/docs/io.md
+++ b/docs/io.md
@@ -1,156 +1,193 @@
IO Formats
==========
-#### EDGE
-`--input-edge HOST PORT`
+## Input
-Specify an IP address or host name where your Edge lives along with a port.
+### Input Format
+`--input {edge, goes, iaga2002, imfv283, pcdcp}`
+Specify input format.
-#### Iaga2002
+`edge`
+ EDGE/Earthworm server.
-`--input-iaga-file FILENAME`
+`iaga2002`
+ IAGA2002 format.
-Specify the name of the file to read from.
+`imfv283`
+ IMFV283 binary format.
-`--input-iaga-magweb`
+`pcdcp`
+ PCDCP format.
-Data will be pulled from geomag.usgs.gov/data/magnetometer if it exists.
-`--input-iaga-stdin`
+### Input Source
+For input format `edge`
-Use redirection on the command line to send your data in.
+`--input-host HOST`
+ (Default `cwbpub.cr.usgs.gov`)
-`--input-iaga-url URL`
+`--input-port PORT`
+ (Default `2060`)
-Use a file pattern to read from multiple IAGA2002 files.
+For input formats `iaga2002`, `imfv283`, `pcdcp`
-`--output-iaga-file FILENAME`
+`--input-file FILE`
+ Read from a specific file.
-Specify the name of the file to write to.
+`--input-stdin`
+ Read from standard input
-`--output-iaga-stdout`
+`--input-url URLTEMPLATE`
+ Read one or more files using a pattern.
-Output will be send directly to the command line.
+`--input-url-interval URLINTERVAL`
+ (Default `86400` seconds)
-`--output-iaga-url URL`
+Interval specifies the amount of data in each url and defaults to 1 day.
-Use a file pattern to write to multiple IAGA2002 files.
-#### IMFV283
+## Output
-`--input-imfv283-file FILENAME`
+### Output Format
-Specify the name of the file to read from.
+`--output {binlog, edge, iaga2002, pcdcp, plot, temperature, vbf}`
-`--input-imfv283-stdin`
+Specify output format.
-Use redirection on the command line to send your data in.
+`binlog`
+ BINLOG format.
-`--input-imfv283-url URL`
+`edge`
+ EDGE server.
-Use a file pattern to read IMFV283 file.
+`iaga2002`
+ IAGA2002 format.
-`--input-imfv283-goes`
+`pcdcp`
+ PCDCP format.
-Reads IMFV283 data from an internet goes server.
+`plot`
+ Interactive matplotlib plot.
+`temperature`
+ Temperature file format.
-#### PCDCP
+`vbf`
+ Volt/Bin format.
-`--input-pcdcp-file FILENAME`
-Specify the name of the file to read from.
+### Output Target
+For output format `edge`
-`--input-pcdcp-stdin`
+`--output-edge-read-port PORT`
+ (Default `2060`)
+ Port used when checking for output gaps to fill.
-Use redirection on the command line to send your data in.
+`--output-edge-forceout`
+ Force miniseed blocks to be written instead of waiting for more data.
-`--input-pcdcp-url URL`
+`--output-edge-tag TAG`
+ (Default `GEOMAG`)
+ Unique identifier used for data being loaded.
-Use a file pattern to read from multiple PCDCP files.
+`--output-host HOST`
+ (Default `cwbpub.cr.usgs.gov`)
-`--output-pcdcp-file FILENAME`
+`--output-port PORT`
+ (Default `2060`)
-Specify the name of the file to write to.
+For output formats `binlog`, `iaga2002`, `pcdcp`, `temperature`, `vbf`
-`--output-pcdcp-stdout`
+`--output-file FILE`
+ Write to a specific file.
-Output will be send directly to the command line.
+`--output-stdin`
+ Write to standard output
-`--output-pcdcp-url`
+`--output-url URLTEMPLATE`
+ Write one or more files using a pattern.
-Use a file pattern to write to multiple PCDCP files.
+ Only "file://" urls are currently supported for output.
+`--output-url-interval URLINTERVAL`
+ (Default `86400` seconds)
-## IO Methods
-Several methods exist for retrieving and storing data.
+## URL Templates
-#### Edge Server
+URLs can be used to fetch or store groups of data using pattern matching. In
+order to use a directory of files on a local machine, just specify "file://"
+at the beginning of the pattern.
-`--input-edge HOST PORT`
+NOTE: Any protocols supported by the systems `libcurl` are also supported
+by this application. Certain protocols like 'sftp' require you to manually
+connect and accept the remove servers key fingerprint before they will work
+in a url template.
-Specify an IP address or host name where your Edge lives along with a port.
+`--input-url URLTEMPLATE`
-#### Single File
+`--output-url URLTEMPLATE`
-```
---input-iaga-file FILENAME
---input-imfv283-file FILENAME
---input-pcdcp-file FILENAME
---output-iaga-file FILENAME
---output-pcdcp-file FILENAME
-```
+Patterns that will be matched with information from the data:
-Specify a single file name for the data to be read from or written to.
+| Key | Purpose |
+| ------------- | -------------------------------------------------------- |
+| __date__ | datetime object, for custom strftime format patterns |
+| __i__ | interval abbreviation (sec, min, hor, etc.) |
+| __interval__ | interval name (second, minute, hour, etc.) |
+| __minute__ | minute of day |
+| __month__ | lower case abbreviated month name |
+| __MONTH__ | upper case abbreviated month name |
+| __obs__ | lowercase observatory 3-letter code |
+| __OBS__ | uppercase observatory 3-letter code |
+| __t__ | type abbreviation (v, q, d, etc.) |
+| __type__ | type name (variation, quasi-definitive, definitive, etc.)|
+| _julian_ | deprecated. julian day formatted as JJJ |
+| _year_ | deprecated. year formatted as YYYY |
+| _ymd_ | deprecated. time formatted as YYYYMMDD |
-#### Multiple Files
-```
---input-iaga-url
---input-imfv283-url
---input-pcdcp-url
---output-iaga-url
---output-pcdcp-url
-```
+These patterns can be used with python string formatting (_recommended_),
+or older string interpolation (for backward compatibility).
-URLs can be used to fetch or store groups of data using pattern matching. In
-order to use a directory of files on a local machine, just specify "file://"
-at the beginning of the pattern.
+See http://strftime.org/ for a list of available date format options.
-Patterns that will be matched with information from the data:
+### IAGA2002 example
+- `file://./{obs}{date:%Y%m%d}{t}{i}.{i}` (string formatting, _recommended_)
+- `file://./%(obs)s%(ymd)s%(t)%(i)s.%(i)s` (string interpolation)
+
+For date=2013-04-02, type=variation, interval=minute, the resulting url is
+
+`file://./bou20130402vmin.min`
+
+
+### PCDCP example:
+- `file://./{OBS}/{date:%Y%j}.{i}` (string formatting, _recommended_)
+- `file://./%(OBS)s/%(Y)s%(j)s.%(i)s` (string interpolation)
+
+For date=2013-04-02, type=variation, interval=minute, the resulting url is
+
+`file://./BOU2013092.min`
-| Pattern | Purpose |
-| ----------------- | -------------------------------------------------------- |
-| __%(i)s__ | interval abbreviation (sec, min, hor, etc.) |
-| __%(interval)s__ | interval name (second, minute, hour, etc.) |
-| __%(julian)s__ | julian day formatted as JJJ |
-| __%(obs)s__ | lowercase observatory 3-letter code |
-| __%(OBS)s__ | uppercase observatory 3-letter code |
-| __%(t)s__ | type abbreviation (v, q, d, etc.) |
-| __%(type)s__ | type name (variation, quasi-definitive, definitive, etc.)|
-| __%(year)s__ | year formatted as YYYY |
-| __%(ymd)s__ | time formatted as YYYYMMDD |
-Typical IAGA2002 files are stored as `file://./%(obs)s%(ymd)s%(t)%(i)s.%(i)s`
+## URL Interval
-Example: bou20130402vmin.min
+`--input-url-interval INTERVAL`
-Typical PCDCP files are stored as `file://./%(OBS)s%(Y)s%(j)s.%(i)s`
+`--output-url-interval INTERVAL`
-Example: BOU2013092.min
-#### Std In and Std Out
+The default URL Interval is `86400` seconds, which results in a separate url
+for each day. The url template is called once for each interval
+(intervals start at unix epoch 1970-01-01T00:00:00Z).
-```
---input-iaga-stdin
---input-imfv283-stdin
---input-pcdcp-stdin
---output-iaga-stdout
---output-pcdcp-stdout
-```
+Examples
+ `600` - 10 minutes per url
+- `3600` - one hour per url
+- `604800` - 7 days per url
-For standard in, pass the data in with redirection.
+When customizing the interval, be sure the URL Template is unique for each
+interval otherwise urls may overlap and lead to undesirable results.
diff --git a/docs/io/Edge.md b/docs/io/Edge.md
index 3238158cc..988873090 100644
--- a/docs/io/Edge.md
+++ b/docs/io/Edge.md
@@ -9,15 +9,15 @@ Read from an Earthworm-style interface
(backslashes added for readability)
<pre>
geomag.py \
- <b>--input-edge cwbpub.cr.usgs.gov</b> \
- <b>--input-edge-port 2060</b> \
+ <b>--input edge</b>
--observatory BOU
--inchannels H E Z F \
--interval minute \
--type variation \
--starttime=2015-11-01T00:00:00Z \
--endtime=2015-11-01T23:59:59Z \
- --output-iaga-stdout
+ --output iaga2002
+ --output-stdout
</pre>
diff --git a/docs/io/Iaga2002.md b/docs/io/Iaga2002.md
index fcea4710b..a73eaf1f5 100644
--- a/docs/io/Iaga2002.md
+++ b/docs/io/Iaga2002.md
@@ -11,9 +11,11 @@ https://www.ngdc.noaa.gov/IAGA/vdat/iagaformat.html
(backslashes added for readability)
```
geomag.py \
- --input-iaga-file /PATH/TO/IAGAFILE
+ --input iaga2002
+ --input-file /PATH/TO/IAGAFILE
--inchannels H D Z F
- --output-iaga-stdout
+ --output iaga2002
+ --output-stdout
```
diff --git a/docs/metadata.md b/docs/metadata.md
index 1a523e73a..d8c964fe3 100644
--- a/docs/metadata.md
+++ b/docs/metadata.md
@@ -17,7 +17,7 @@ Metadata is stored in the `Stats` dictionary of each `Trace` object.
- `channel`
Data channel.
-
+
Examples:
`H`,
`D`,
@@ -56,12 +56,12 @@ Metadata is stored in the `Stats` dictionary of each `Trace` object.
- `data_interval`
Time between data samples.
+ This is usually set, but more useful properties are `delta` and `sampling_rate`
Examples:
`daily`,
`hourly`,
`minute`,
- `monthly`,
`second`
- `data_interval_type`
@@ -85,4 +85,3 @@ Metadata is stored in the `Stats` dictionary of each `Trace` object.
- `comments`
Other comments about this data.
-
diff --git a/docs/usage.md b/docs/usage.md
index ea3ce2ba2..d27cf7d82 100644
--- a/docs/usage.md
+++ b/docs/usage.md
@@ -15,8 +15,10 @@ examples. They include
| __start time__ | YYYY-MM-DDTHH:MM:SSZ |
| __end time__ | YYYY-MM-DDTHH:MM:SSZ |
| __observatory code__ | 3-letter observatory code |
-| __data input source__ | {--input-edge, --input-iaga-file, ...} |
-| __data output source__ | {--output-pcdcp-url, --output-iaga-file, ...}|
+| __data input format __ | 'edge', 'iaga2002', 'pcdcp', ... |
+| __data input source__ | {--input, --input-file, ...} |
+| __data output format__ | 'edge', 'iaga2002', 'pcdcp', ... |
+| __data output source__ | {--output-url, --output-file, ...} |
| __input data channels__ | {[CHANNEL [CHANNEL ...]]} |
| __output data channels__ | {[CHANNEL [CHANNEL ...]]} |
| __data type__ | {variation, quasi-definitive, definitive} |
@@ -30,13 +32,18 @@ To retrieve all raw (**_variation_**) **_H_**, **_E_**, **_Z_** and **_F_**
**_July 1st 2014_** from an **_iaga2002_** formatted file and output
**_H_**, **_E_**, **_Z_** and **_F_** data to a **_pcdcp_** formatted file:
- geomag.py --type variation --inchannels H E Z F --interval minute \
+ geomag.py \
+ --type variation \
+ --inchannels H E Z F \
+ --interval minute \
--observatory BOU \
--starttime 2014-07-01T00:00:00Z \
--endtime 2014-07-01T23:59:00Z \
- --input-iaga-file BOU20140701vmin.min \
+ --input iaga2002 \
+ --input-file BOU20140701vmin.min \
--outchannels H E Z F \
- --output-pcdcp-file BOU2014182.min
+ --output pcdcp \
+ --output-file BOU2014182.min
To retrieve all raw (**_variation_**) **_H_**, **_E_**, **_Z_** and **_F_**
**_minute_** data from Tucson Observatory (**_TUC_**) for the entire month of
@@ -44,12 +51,17 @@ To retrieve all raw (**_variation_**) **_H_**, **_E_**, **_Z_** and **_F_**
directory and output **_H_**, **_E_**, **_Z_** and **_F_** data to a group of
**_iaga2002_** formatted files in a "data-iaga" directory:
- geomag.py --type variation --inchannels H E Z F --interval minute \
+ geomag.py \
+ --type variation \
+ --inchannels H E Z F \
+ --interval minute \
--observatory TUC \
--starttime 2013-03-01T00:00:00Z \
--endtime 2013-03-31T23:59:00Z \
- --input-pcdcp-url file://data-pcdcp/./%(OBS)s%(year)s%(julian)s.%(i)s \
- --output-iaga-url file://data-iaga/./$(obs)s%(Y)s%(j)s.%(i)s \
+ --input pcdcp
+ --input-url file://data-pcdcp/./%(OBS)s%(year)s%(julian)s.%(i)s \
+ --output iaga2002
+ --output-url file://data-iaga/./$(obs)s%(Y)s%(j)s.%(i)s \
--outchannels H E Z F
To retrieve all **_Dst 4 minute_**, and **_Dst 3 minute_** data from **_USGS_**
@@ -57,12 +69,16 @@ for the entire day of **_Oct 1st 2015_** from an **_edge server_** (at
cwbpub.cr.usgs.gov) and output **_Dst 4 minute_**, and **_Dst 3 minute_**
data to an **_iaga2002_** formatted file:
- geomag.py --type variation --inchannels MGD MSD --interval minute \
+ geomag.py \
+ --type variation \
+ --inchannels MGD MSD \
+ --interval minute \
--observatory USGS \
--starttime 2015-10-01T00:00:00Z \
--endtime 2015-10-01T23:59:00Z \
- --input-edge cwbpub.cr.usgs.gov \
- --output-iaga-stdout \
+ --input edge
+ --output iaga2002
+ --output-stdout \
--outchannels MGD MSD
--
GitLab