Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
G
geomag-algorithms
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Container Registry
Model registry
Operate
Environments
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
ghsc
National Geomagnetism Program
geomag-algorithms
Commits
7ab11030
Commit
7ab11030
authored
9 years ago
by
Jeremy M Fee
Browse files
Options
Downloads
Plain Diff
Merge pull request
#73
from emcwhirter-usgs/usgs/geomag-algorithms#56
Usgs/geomag algorithms#56
parents
543e2697
3e21abb3
No related branches found
No related tags found
No related merge requests found
Changes
5
Hide whitespace changes
Inline
Side-by-side
Showing
5 changed files
README.md
+12
-1
12 additions, 1 deletion
README.md
docs/develop.md
+3
-29
3 additions, 29 deletions
docs/develop.md
docs/install.md
+16
-29
16 additions, 29 deletions
docs/install.md
docs/io.md
+57
-24
57 additions, 24 deletions
docs/io.md
docs/usage.md
+30
-20
30 additions, 20 deletions
docs/usage.md
with
118 additions
and
103 deletions
README.md
+
12
−
1
View file @
7ab11030
...
...
@@ -9,6 +9,9 @@ Geomag algorithms includes tools to fetch, process, and output geomag data.
### [Install](./docs/install.md)
First time install. Walk through dependencies and other considerations.
Document:
[
/docs/install.md
](
./docs/install.md
)
### [Command Line Usage](./docs/usage.md)
Details and examples for proper usage. Get started quickly.
...
...
@@ -17,13 +20,21 @@ Basic usage:
-
Use the main script,
`geomag.py -h`
-
In python scripts,
`import geomagio`
Document:
[
/docs/usgage.md
](
./docs/usage.md
)
### [Python API](./docs/api.md)
API Overview and examples for using the geomag-algorithms project in python.
### [Develop](./docs/develop_install.md)
Document:
[
/docs/api.md
](
./docs/api.md
)
### [Develop](./docs/develop.md)
Development dependencies discussed here. Project is built with Grunt and Node
and is written primarily in Python 2.7.
Document:
[
/docs/develop.md
](
./docs/develop.md
)
## Supported Formats
...
...
This diff is collapsed.
Click to expand it.
docs/develop.md
+
3
−
29
View file @
7ab11030
...
...
@@ -31,14 +31,12 @@ Begin Developing
### Windows
1.
Install the newest release of
[
Node
(http://nodejs.org/download/)
][]
for
Windows, using the Windows Installer (.msi).
-
All of the defaults should be sufficient
.
1.
Install the newest release of Node
[
(http://nodejs.org/download/)
](
http://nodejs.org/download/
)
for Windows,
using the Windows Installer (.msi)
.
1.
Close and re-open your terminal so that your new PATH is loaded.
[
Node
]:
http://nodejs.org/download/
### Coding Standards
...
...
@@ -51,27 +49,3 @@ This project adheres to PEP8 standards in most cases:
- line continuations should use two indentations (8 spaces).
- do not use visual indents.
### Add an SSH Key to GitHub
1.
`ssh-keygen -t rsa -b 4096`
(in terminal)
2.
Press
**Enter**
to accept the default save location.
3.
Enter a passphrase that you will remember.
4.
`cat ~/.ssh/id_rsa.pub`
Copy the text block that is displayed.
This is your SSH key.
If you're on Window and can't copy the text from the terminal, go
to
`C:\Users\[your username here]\.ssh`
and open the
`id_rsa.pub`
file
with notepad.
5.
In GitHub, click
**Edit Your Profile**
.
6.
Select
**SSH Keys**
on the left.
7.
Click
**Add SSH key**
. Give it a meaningful title.
8.
Copy your SSH Key into the Key, and click
**Add key**
.
This diff is collapsed.
Click to expand it.
docs/install.md
+
16
−
29
View file @
7ab11030
Installation
============
We recommend using
[
virtual environments
(http://docs.python-guide.org/en/latest/dev/virtualenvs/)
]
[]
.
We recommend using
virtual environments:
[
(http://docs.python-guide.org/en/latest/dev/virtualenvs/)
]
(
http://docs.python-guide.org/en/latest/dev/virtualenvs/
)
Check out the
[
Mac
](
#mac
)
and
[
Windows
](
#windows
)
details below for more
specific dependency information.
[
Mac
](
#mac
)
and
[
Windows
](
#windows
)
specific details are below.
1.
Install __geomagio__
pip install git+https://github.com/usgs/geomag-algorithms.git
pip install git+https://github.com/usgs/geomag-algorithms.git
1.
Use __geomagio__
...
...
@@ -19,25 +18,25 @@ specific dependency information.
The
[
Command Line Usage
](
./usage.md
)
and
[
Python API
](
./api.md
)
pages have more instructions and examples.
[
virtual environments
]:
http://docs.python-guide.org/en/latest/dev/virtualenvs/
---
### Mac ###
1. Install `python` (2.7.X)
On OS X, we recommend using Homebrew (
http://brew.sh/
)
On OS X, we recommend using Homebrew
[http://brew.sh/]
(http://brew.sh/)
1. Install numpy, obspy requires this be installed separately.
pip install numpy
1. Install `node`, `git` and `python` (2.7.X).
On OS X, we recommend using [Homebrew (http://brew.sh/)][]
brew install node
brew install git
On OS X, we recommend using Homebrew
```
brew install node
brew install git
```
1. Use pip to install `numpy`, `scipy`, `obspy`, and `flake8`
...
...
@@ -54,30 +53,18 @@ pages have more instructions and examples.
1. Close and re-open your terminal so that your new PATH is loaded.
Make sure to navigate back to your `geomag-algorithms` project directory.
[
Homebrew
]
:
http://brew.sh/
---
### Windows ###
1.
You will need a terminal tool for Windows.
[
Git Bash (http://git-scm.com/download/win)
][]
was used to test
these steps, but
[
Cygwin (http://cygwin.com/install.html)
][]
or another
unix-like editor should work too.
-
Choosing PATH environment: We recommend the __last option__ here to include
Unix tools, but if you don't understand what that entails then use the
_second option_ which still adds Git to your system PATH.
-
Besides that, all of the defaults should be sufficient.
1.
You will need a terminal tool for Windows. We recommend Git Bash
[
(http://git-scm.com/download/win)
](
http://git-scm.com/download/win
)
, but
another unix-like editor should work too.
1.
Install
[
Anaconda (http://continuum.io/downloads)
][]
, which includes
`numpy`
,
`scipy`
and
`flake8`
.
-
We recommend using _"Just Me"_ for the install to avoid potential admin
issues.
1.
Install Anaconda
[
(http://continuum.io/downloads)
](
http://continuum.io/downloads
)
, which
includes
`numpy`
,
`scipy`
and
`flake8`
.
1.
Run
`python`
to verify that Anaconda's version of Python is being used.
If it isn't, update your PATH so that Conda's HOME is before any other
versions of Python.
[
Git Bash
]:
http://git-scm.com/download/win
[
Cygwin
]:
http://cygwin.com/install.html
[
Anaconda
]:
http://continuum.io/downloads
This diff is collapsed.
Click to expand it.
docs/io.md
+
57
−
24
View file @
7ab11030
...
...
@@ -4,105 +4,136 @@ IO Formats
#### EDGE
`--input-edge HOST PORT`
Specify an IP address or host name where your Edge lives along with a port.
#### Iaga2002
`--input-iaga-file FILENAME`
Specify the name of the file to read from.
`--input-iaga-magweb`
Data will be pulled from geomag.usgs.gov/data/magnetometer if it exists.
`--input-iaga-stdin`
Use redirection on the command line to send your data in.
`--input-iaga-url URL`
Use a file pattern to read from multiple IAGA2002 files.
`--output-iaga-file FILENAME`
Specify the name of the file to write to.
`--output-iaga-stdout`
Output will be send directly to the command line.
`--output-iaga-url URL`
Use a file pattern to write to multiple IAGA2002 files.
#### IMFV283
`--input-imfv283-file FILENAME`
Specify the name of the file to read from.
`--input-imfv283-stdin`
Use redirection on the command line to send your data in.
`--input-imfv283-url URL`
Use a file pattern to read IMFV283 file.
`--input-imfv283-goes`
Reads IMFV283 data from an internet goes server.
#### PCDCP
`--input-pcdcp-file FILENAME`
Specify the name of the file to read from.
`--input-pcdcp-stdin`
Use redirection on the command line to send your data in.
`--input-pcdcp-url URL`
Use a file pattern to read from multiple PCDCP files.
`--output-pcdcp-file FILENAME`
Specify the name of the file to write to.
`--output-pcdcp-stdout`
Output will be send directly to the command line.
`--output-pcdcp-url`
Use a file pattern to write to multiple PCDCP files.
## IO Methods
Several methods exist for retrieving and storing data.
#### Edge Server
`--input-edge HOST PORT`
Specify an IP address or host name where your Edge lives along with a port.
#### Single File
`--input-iaga-file FILENAME`
`--input-imfv283-file FILENAME`
`--input-pcdcp-file FILENAME`
`--output-iaga-file FILENAME`
`--output-pcdcp-file FILENAME`
```
--input-iaga-file FILENAME
--input-imfv283-file FILENAME
--input-pcdcp-file FILENAME
--output-iaga-file FILENAME
--output-pcdcp-file FILENAME
```
Specify a single file name for the data to be read from or written to.
#### Multiple Files
`--input-iaga-url`
`--input-imfv283-url`
`--input-pcdcp-url`
`--output-iaga-url`
`--output-pcdcp-url`
```
--input-iaga-url
--input-imfv283-url
--input-pcdcp-url
--output-iaga-url
--output-pcdcp-url
```
URLs can be used to fetch or store groups of data using pattern matching. In
order to use a directory of files on a local machine, just specify "file://"
at the beginning of the pattern.
Patterns that will be matched with information from the data:
-
__%(i)s__ : interval abbreviation (sec, min, hor, etc.)
-
__%(interval)s__: interval name (second, minute, hour, etc.)
-
__%(julian)s__ : julian day formatted as JJJ
-
__%(obs)s__ : lowercase observatory 3-letter code
-
__%(OBS)s__ : uppercase observatory 3-letter code
-
__%(t)s__ : type abbreviation (v, q, d, etc.)
-
__%(type)s__ : type name (variation, quasi-definitive, definitive, etc.)
-
__%(year)s__ : year formatted as YYYY
-
__%(ymd)s__ : time formatted as YYYYMMDD
| Pattern | Purpose |
| ----------------- | -------------------------------------------------------- |
| __%(i)s__ | interval abbreviation (sec, min, hor, etc.) |
| __%(interval)s__ | interval name (second, minute, hour, etc.) |
| __%(julian)s__ | julian day formatted as JJJ |
| __%(obs)s__ | lowercase observatory 3-letter code |
| __%(OBS)s__ | uppercase observatory 3-letter code |
| __%(t)s__ | type abbreviation (v, q, d, etc.) |
| __%(type)s__ | type name (variation, quasi-definitive, definitive, etc.)|
| __%(year)s__ | year formatted as YYYY |
| __%(ymd)s__ | time formatted as YYYYMMDD |
Typical IAGA2002 files are stored as
`file://./%(obs)s%(ymd)s%(t)%(i)s.%(i)s`
...
...
@@ -114,10 +145,12 @@ Example: BOU2013092.min
#### Std In and Std Out
`--input-iaga-stdin`
`--input-imfv283-stdin`
`--input-pcdcp-stdin`
`--output-iaga-stdout`
`--output-pcdcp-stdout`
```
--input-iaga-stdin
--input-imfv283-stdin
--input-pcdcp-stdin
--output-iaga-stdout
--output-pcdcp-stdout
```
For standard in, pass the data in with redirection.
This diff is collapsed.
Click to expand it.
docs/usage.md
+
30
−
20
View file @
7ab11030
...
...
@@ -10,22 +10,25 @@ input parameters.
Some of the key parameters that should be specified are listed here along with
examples. They include
-
__start time__ - YYYY-MM-DDTHH:MM:SSZ
-
__end time__ - YYYY-MM-DDTHH:MM:SSZ
-
__observatory code__ - 3-letter observatory code
-
__data input source__ - {--input-edge, --input-iaga-file, ...}
-
__data output source__ - {--output-pcdcp-url, --output-iaga-file, ...}
-
__input data channels__ - {[CHANNEL [CHANNEL ...]]}
-
__output data channels__ - {[CHANNEL [CHANNEL ...]]}
-
__data type__ - {variation, quasi-definitive, definitive}
-
__interval__ - {minute, second, hour, day}
| Parameter | Format |
| -------------------------- | -------------------------------------------- |
| __start time__ | YYYY-MM-DDTHH:MM:SSZ |
| __end time__ | YYYY-MM-DDTHH:MM:SSZ |
| __observatory code__ | 3-letter observatory code |
| __data input source__ | {--input-edge, --input-iaga-file, ...} |
| __data output source__ | {--output-pcdcp-url, --output-iaga-file, ...}|
| __input data channels__ | {[CHANNEL [CHANNEL ...]]} |
| __output data channels__ | {[CHANNEL [CHANNEL ...]]} |
| __data type__ | {variation, quasi-definitive, definitive} |
| __interval__ | {minute, second, hour, day} |
### Examples ###
To retrieve all
_
raw
_
(variation
) _H_, _E_,
_Z_ and _F_
_minute_ data from
_
Boulder Observatory
_
for the entire day of
_July 1st 2014_ from an _iaga2002_
formatted file and output _H_, _E_, _Z_ and _F_ data to a _PCDCP_ formatted
file:
To retrieve all raw (
**_
variation
_**
)
**_H_**
,
**_E_**
,
**
_Z_
**
and
**
_F_
**
**_minute_**
data from
Boulder Observatory
(
**_BOU_**
)
for the entire day of
**_July 1st 2014_**
from an
**_iaga2002_**
formatted file and output
**_H_**
,
**_E_**
,
**_Z_**
and
**_F_**
data to a
**_pcdcp_**
formatted
file:
geomag.py --type variation --inchannels H E Z F --interval minute \
--observatory BOU \
...
...
@@ -35,10 +38,11 @@ file:
--outchannels H E Z F \
--output-pcdcp-file BOU2014182.min
To retrieve all _raw_ (variation) _H_, _E_, _Z_ and _F_ _minute_ data from
_Tucson Observatory_
for the entire month of _March 2013_ from _pcdcp_
formatted files in a "data-pcdcp" directory and output _H_, _E_, _Z_ and _F_
data to a group of _iaga2002_ formatted files in a "data-iaga" directory:
To retrieve all raw (
**_variation_**
)
**_H_**
,
**_E_**
,
**_Z_**
and
**_F_**
**_minute_**
data from Tucson Observatory (
**_TUC_**
) for the entire month of
**_March 2013_**
from a group of
**_pcdcp_**
formatted files in a "data-pcdcp"
directory and output
**_H_**
,
**_E_**
,
**_Z_**
and
**_F_**
data to a group of
**_iaga2002_**
formatted files in a "data-iaga" directory:
geomag.py --type variation --inchannels H E Z F --interval minute \
--observatory TUC \
...
...
@@ -48,9 +52,10 @@ data to a group of _iaga2002_ formatted files in a "data-iaga" directory:
--output-iaga-url file://data-iaga/./$(obs)s%(Y)s%(j)s.%(i)s \
--outchannels H E Z F
To retrieve all _Dst 4 minute_, and _Dst3 minute_ data from _USGS_ for the
entire day of _Oct 1st 2015_ from _edge server cwbpub.cr.usgs.gov_ output
_Dst 4 minute_
, and _Dst 3 minute_ data to an _iaga2002_ formatted file:
To retrieve all
**_Dst 4 minute_**
, and
**_Dst 3 minute_**
data from
**_USGS_**
for the entire day of
**_Oct 1st 2015_**
from an
**_edge server_**
(at
cwbpub.cr.usgs.gov) and output
**_Dst 4 minute_**
, and
**_Dst 3 minute_**
data to an
**_iaga2002_**
formatted file:
geomag.py --type variation --inchannels MGD MSD --interval minute \
--observatory USGS \
...
...
@@ -69,11 +74,16 @@ There are flags to specify certain algorithms should be run against the data.
#### XYZ ####
`--algorithm xyz`
`--xyz-from {geo, mag, obs, obsd}`
(default is
`obs`
)
`--xyz-to {geo, mag, obs, obsd}`
(default is
`geo`
)
#### [XYZ Usage](./algorithms/XYZ_usage.md) ####
Rotate data from HEZ (obs) or HDZ (mag) to XYZ (geo) and back.
Document:
[
/algorithms/XYZ_usage.md
](
./algorithms/XYZ_usage.md
)
Extensive explanation of all input and output methods:
[
IO Methods
](
./io.md
)
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment