Skip to content
Snippets Groups Projects
Commit 7ab11030 authored by Jeremy M Fee's avatar Jeremy M Fee
Browse files

Merge pull request #73 from emcwhirter-usgs/usgs/geomag-algorithms#56

Usgs/geomag algorithms#56
parents 543e2697 3e21abb3
No related branches found
No related tags found
No related merge requests found
......@@ -9,6 +9,9 @@ Geomag algorithms includes tools to fetch, process, and output geomag data.
### [Install](./docs/install.md)
First time install. Walk through dependencies and other considerations.
Document: [/docs/install.md](./docs/install.md)
### [Command Line Usage](./docs/usage.md)
Details and examples for proper usage. Get started quickly.
......@@ -17,13 +20,21 @@ Basic usage:
- Use the main script, `geomag.py -h`
- In python scripts, `import geomagio`
Document: [/docs/usgage.md](./docs/usage.md)
### [Python API](./docs/api.md)
API Overview and examples for using the geomag-algorithms project in python.
### [Develop](./docs/develop_install.md)
Document: [/docs/api.md](./docs/api.md)
### [Develop](./docs/develop.md)
Development dependencies discussed here. Project is built with Grunt and Node
and is written primarily in Python 2.7.
Document: [/docs/develop.md](./docs/develop.md)
## Supported Formats
......
......@@ -31,14 +31,12 @@ Begin Developing
### Windows
1. Install the newest release of [Node (http://nodejs.org/download/)][] for
Windows, using the Windows Installer (.msi).
- All of the defaults should be sufficient.
1. Install the newest release of Node
[(http://nodejs.org/download/)](http://nodejs.org/download/) for Windows,
using the Windows Installer (.msi).
1. Close and re-open your terminal so that your new PATH is loaded.
[Node]: http://nodejs.org/download/
### Coding Standards
......@@ -51,27 +49,3 @@ This project adheres to PEP8 standards in most cases:
- line continuations should use two indentations (8 spaces).
- do not use visual indents.
### Add an SSH Key to GitHub
1. `ssh-keygen -t rsa -b 4096` (in terminal)
2. Press **Enter** to accept the default save location.
3. Enter a passphrase that you will remember.
4. `cat ~/.ssh/id_rsa.pub`
Copy the text block that is displayed.
This is your SSH key.
If you're on Window and can't copy the text from the terminal, go
to `C:\Users\[your username here]\.ssh` and open the `id_rsa.pub` file
with notepad.
5. In GitHub, click **Edit Your Profile**.
6. Select **SSH Keys** on the left.
7. Click **Add SSH key**. Give it a meaningful title.
8. Copy your SSH Key into the Key, and click **Add key**.
Installation
============
We recommend using
[virtual environments (http://docs.python-guide.org/en/latest/dev/virtualenvs/)][].
We recommend using virtual environments:
[(http://docs.python-guide.org/en/latest/dev/virtualenvs/)](http://docs.python-guide.org/en/latest/dev/virtualenvs/)
Check out the [Mac](#mac) and [Windows](#windows) details below for more
specific dependency information.
[Mac](#mac) and [Windows](#windows) specific details are below.
1. Install __geomagio__
pip install git+https://github.com/usgs/geomag-algorithms.git
pip install git+https://github.com/usgs/geomag-algorithms.git
1. Use __geomagio__
......@@ -19,25 +18,25 @@ specific dependency information.
The [Command Line Usage](./usage.md) and [Python API](./api.md)
pages have more instructions and examples.
[virtual environments]: http://docs.python-guide.org/en/latest/dev/virtualenvs/
---
### Mac ###
1. Install `python` (2.7.X)
On OS X, we recommend using Homebrew ( http://brew.sh/ )
On OS X, we recommend using Homebrew [http://brew.sh/](http://brew.sh/)
1. Install numpy, obspy requires this be installed separately.
pip install numpy
1. Install `node`, `git` and `python` (2.7.X).
On OS X, we recommend using [Homebrew (http://brew.sh/)][]
brew install node
brew install git
On OS X, we recommend using Homebrew
```
brew install node
brew install git
```
1. Use pip to install `numpy`, `scipy`, `obspy`, and `flake8`
......@@ -54,30 +53,18 @@ pages have more instructions and examples.
1. Close and re-open your terminal so that your new PATH is loaded.
Make sure to navigate back to your `geomag-algorithms` project directory.
[Homebrew]: http://brew.sh/
---
### Windows ###
1. You will need a terminal tool for Windows.
[Git Bash (http://git-scm.com/download/win)][] was used to test
these steps, but [Cygwin (http://cygwin.com/install.html)][] or another
unix-like editor should work too.
- Choosing PATH environment: We recommend the __last option__ here to include
Unix tools, but if you don't understand what that entails then use the
_second option_ which still adds Git to your system PATH.
- Besides that, all of the defaults should be sufficient.
1. You will need a terminal tool for Windows. We recommend Git Bash
[(http://git-scm.com/download/win)](http://git-scm.com/download/win), but
another unix-like editor should work too.
1. Install [Anaconda (http://continuum.io/downloads)][], which includes
`numpy`, `scipy` and `flake8`.
- We recommend using _"Just Me"_ for the install to avoid potential admin
issues.
1. Install Anaconda
[(http://continuum.io/downloads)](http://continuum.io/downloads), which
includes `numpy`, `scipy` and `flake8`.
1. Run `python` to verify that Anaconda's version of Python is being used.
If it isn't, update your PATH so that Conda's HOME is before any other
versions of Python.
[Git Bash]: http://git-scm.com/download/win
[Cygwin]: http://cygwin.com/install.html
[Anaconda]: http://continuum.io/downloads
......@@ -4,105 +4,136 @@ IO Formats
#### EDGE
`--input-edge HOST PORT`
Specify an IP address or host name where your Edge lives along with a port.
#### Iaga2002
`--input-iaga-file FILENAME`
Specify the name of the file to read from.
`--input-iaga-magweb`
Data will be pulled from geomag.usgs.gov/data/magnetometer if it exists.
`--input-iaga-stdin`
Use redirection on the command line to send your data in.
`--input-iaga-url URL`
Use a file pattern to read from multiple IAGA2002 files.
`--output-iaga-file FILENAME`
Specify the name of the file to write to.
`--output-iaga-stdout`
Output will be send directly to the command line.
`--output-iaga-url URL`
Use a file pattern to write to multiple IAGA2002 files.
#### IMFV283
`--input-imfv283-file FILENAME`
Specify the name of the file to read from.
`--input-imfv283-stdin`
Use redirection on the command line to send your data in.
`--input-imfv283-url URL`
Use a file pattern to read IMFV283 file.
`--input-imfv283-goes`
Reads IMFV283 data from an internet goes server.
#### PCDCP
`--input-pcdcp-file FILENAME`
Specify the name of the file to read from.
`--input-pcdcp-stdin`
Use redirection on the command line to send your data in.
`--input-pcdcp-url URL`
Use a file pattern to read from multiple PCDCP files.
`--output-pcdcp-file FILENAME`
Specify the name of the file to write to.
`--output-pcdcp-stdout`
Output will be send directly to the command line.
`--output-pcdcp-url`
Use a file pattern to write to multiple PCDCP files.
## IO Methods
Several methods exist for retrieving and storing data.
#### Edge Server
`--input-edge HOST PORT`
Specify an IP address or host name where your Edge lives along with a port.
#### Single File
`--input-iaga-file FILENAME`
`--input-imfv283-file FILENAME`
`--input-pcdcp-file FILENAME`
`--output-iaga-file FILENAME`
`--output-pcdcp-file FILENAME`
```
--input-iaga-file FILENAME
--input-imfv283-file FILENAME
--input-pcdcp-file FILENAME
--output-iaga-file FILENAME
--output-pcdcp-file FILENAME
```
Specify a single file name for the data to be read from or written to.
#### Multiple Files
`--input-iaga-url`
`--input-imfv283-url`
`--input-pcdcp-url`
`--output-iaga-url`
`--output-pcdcp-url`
```
--input-iaga-url
--input-imfv283-url
--input-pcdcp-url
--output-iaga-url
--output-pcdcp-url
```
URLs can be used to fetch or store groups of data using pattern matching. In
order to use a directory of files on a local machine, just specify "file://"
at the beginning of the pattern.
Patterns that will be matched with information from the data:
- __%(i)s__ : interval abbreviation (sec, min, hor, etc.)
- __%(interval)s__: interval name (second, minute, hour, etc.)
- __%(julian)s__ : julian day formatted as JJJ
- __%(obs)s__ : lowercase observatory 3-letter code
- __%(OBS)s__ : uppercase observatory 3-letter code
- __%(t)s__ : type abbreviation (v, q, d, etc.)
- __%(type)s__ : type name (variation, quasi-definitive, definitive, etc.)
- __%(year)s__ : year formatted as YYYY
- __%(ymd)s__ : time formatted as YYYYMMDD
| Pattern | Purpose |
| ----------------- | -------------------------------------------------------- |
| __%(i)s__ | interval abbreviation (sec, min, hor, etc.) |
| __%(interval)s__ | interval name (second, minute, hour, etc.) |
| __%(julian)s__ | julian day formatted as JJJ |
| __%(obs)s__ | lowercase observatory 3-letter code |
| __%(OBS)s__ | uppercase observatory 3-letter code |
| __%(t)s__ | type abbreviation (v, q, d, etc.) |
| __%(type)s__ | type name (variation, quasi-definitive, definitive, etc.)|
| __%(year)s__ | year formatted as YYYY |
| __%(ymd)s__ | time formatted as YYYYMMDD |
Typical IAGA2002 files are stored as `file://./%(obs)s%(ymd)s%(t)%(i)s.%(i)s`
......@@ -114,10 +145,12 @@ Example: BOU2013092.min
#### Std In and Std Out
`--input-iaga-stdin`
`--input-imfv283-stdin`
`--input-pcdcp-stdin`
`--output-iaga-stdout`
`--output-pcdcp-stdout`
```
--input-iaga-stdin
--input-imfv283-stdin
--input-pcdcp-stdin
--output-iaga-stdout
--output-pcdcp-stdout
```
For standard in, pass the data in with redirection.
......@@ -10,22 +10,25 @@ input parameters.
Some of the key parameters that should be specified are listed here along with
examples. They include
- __start time__ - YYYY-MM-DDTHH:MM:SSZ
- __end time__ - YYYY-MM-DDTHH:MM:SSZ
- __observatory code__ - 3-letter observatory code
- __data input source__ - {--input-edge, --input-iaga-file, ...}
- __data output source__ - {--output-pcdcp-url, --output-iaga-file, ...}
- __input data channels__ - {[CHANNEL [CHANNEL ...]]}
- __output data channels__ - {[CHANNEL [CHANNEL ...]]}
- __data type__ - {variation, quasi-definitive, definitive}
- __interval__ - {minute, second, hour, day}
| Parameter | Format |
| -------------------------- | -------------------------------------------- |
| __start time__ | YYYY-MM-DDTHH:MM:SSZ |
| __end time__ | YYYY-MM-DDTHH:MM:SSZ |
| __observatory code__ | 3-letter observatory code |
| __data input source__ | {--input-edge, --input-iaga-file, ...} |
| __data output source__ | {--output-pcdcp-url, --output-iaga-file, ...}|
| __input data channels__ | {[CHANNEL [CHANNEL ...]]} |
| __output data channels__ | {[CHANNEL [CHANNEL ...]]} |
| __data type__ | {variation, quasi-definitive, definitive} |
| __interval__ | {minute, second, hour, day} |
### Examples ###
To retrieve all _raw_ (variation) _H_, _E_, _Z_ and _F_ _minute_ data from
_Boulder Observatory_ for the entire day of _July 1st 2014_ from an _iaga2002_
formatted file and output _H_, _E_, _Z_ and _F_ data to a _PCDCP_ formatted
file:
To retrieve all raw (**_variation_**) **_H_**, **_E_**, **_Z_** and **_F_**
**_minute_** data from Boulder Observatory (**_BOU_**) for the entire day of
**_July 1st 2014_** from an **_iaga2002_** formatted file and output
**_H_**, **_E_**, **_Z_** and **_F_** data to a **_pcdcp_** formatted file:
geomag.py --type variation --inchannels H E Z F --interval minute \
--observatory BOU \
......@@ -35,10 +38,11 @@ file:
--outchannels H E Z F \
--output-pcdcp-file BOU2014182.min
To retrieve all _raw_ (variation) _H_, _E_, _Z_ and _F_ _minute_ data from
_Tucson Observatory_ for the entire month of _March 2013_ from _pcdcp_
formatted files in a "data-pcdcp" directory and output _H_, _E_, _Z_ and _F_
data to a group of _iaga2002_ formatted files in a "data-iaga" directory:
To retrieve all raw (**_variation_**) **_H_**, **_E_**, **_Z_** and **_F_**
**_minute_** data from Tucson Observatory (**_TUC_**) for the entire month of
**_March 2013_** from a group of **_pcdcp_** formatted files in a "data-pcdcp"
directory and output **_H_**, **_E_**, **_Z_** and **_F_** data to a group of
**_iaga2002_** formatted files in a "data-iaga" directory:
geomag.py --type variation --inchannels H E Z F --interval minute \
--observatory TUC \
......@@ -48,9 +52,10 @@ data to a group of _iaga2002_ formatted files in a "data-iaga" directory:
--output-iaga-url file://data-iaga/./$(obs)s%(Y)s%(j)s.%(i)s \
--outchannels H E Z F
To retrieve all _Dst 4 minute_, and _Dst3 minute_ data from _USGS_ for the
entire day of _Oct 1st 2015_ from _edge server cwbpub.cr.usgs.gov_ output
_Dst 4 minute_, and _Dst 3 minute_ data to an _iaga2002_ formatted file:
To retrieve all **_Dst 4 minute_**, and **_Dst 3 minute_** data from **_USGS_**
for the entire day of **_Oct 1st 2015_** from an **_edge server_** (at
cwbpub.cr.usgs.gov) and output **_Dst 4 minute_**, and **_Dst 3 minute_**
data to an **_iaga2002_** formatted file:
geomag.py --type variation --inchannels MGD MSD --interval minute \
--observatory USGS \
......@@ -69,11 +74,16 @@ There are flags to specify certain algorithms should be run against the data.
#### XYZ ####
`--algorithm xyz`
`--xyz-from {geo, mag, obs, obsd}` (default is `obs`)
`--xyz-to {geo, mag, obs, obsd}` (default is `geo`)
#### [XYZ Usage](./algorithms/XYZ_usage.md) ####
Rotate data from HEZ (obs) or HDZ (mag) to XYZ (geo) and back.
Document: [/algorithms/XYZ_usage.md](./algorithms/XYZ_usage.md)
Extensive explanation of all input and output methods:
[IO Methods](./io.md)
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment