Merge branch 'usgs/geomag-algorithms#64' of...

Merge branch 'usgs/geomag-algorithms#64' of https://github.com/emcwhirter-usgs/geomag-algorithms into emcwhirter-usgs-usgs/geomag-algorithms#64

README.md

@@ -47,4 +47,5 @@ Document: [/docs/develop.md](./docs/develop.md)
 ## Supported Algorithms
 
 - [DeltaF](./docs/algorithms/DeltaF_usage.md)
+- [SqDist](./docs/algorithms/SqDist_usage.md)
 - [XYZ](./docs/algorithms/XYZ_usage.md)

docs/algorithms/SqDist.md

+Geomagnetic Secular Variation, Solar Quiet, and Disturbance
+===========================================================
+
+E. Joshua Rigler <[erigler@usgs.gov](mailto:erigler@usgs.gov)>
+
+
+## Background and Motivation
+
+The magnetic field measured at a given point on Earth’s surface is often
+assumed to be static, but in reality it is constantly changing, and on a
+variety of time scales associated with distinct physical phenomena. These are:
+
+- Secular variation (SV) - slow variations in the geomagnetic field associated
+  with changes in Earth's interior.
+- Solar quiet variation (SQ) - shorter-term periodic variations in the
+  geomagnetic field associated with Earth's rotation beneath quasi-static
+  geospace electric currents that are phase-locked with the sun.
+- Disturbance (DIST) - shorter-term non-periodic variations in the geomagnetic
+  field, typically associated with episodic events like geomagnetic storms and
+  substorms.
+
+SV is fairly easily separated from higher frequency variations using low-order
+polynomials to *detrend* the data. SQ and DIST have similar time scales, and
+are therefore more difficult to separate. Fourier series can be fit to data to
+estimate SQ, which works well in non-real time situations. This approach
+suffers in real time situations for both practical and theoretical reasons
+that we won't discuss in detail here.
+
+
+## Exponential Smoothing
+
+Real time decomposition of geomagnetic time series into SV, SQ, and DIST should
+explicitly acknowledge and address time-causal nature of real time
+observations. To this end, we employ a form of exponential smoothing, with
+"seasonal" adjustments, that updates estimates of SV and SQ based only on past
+observations, weighting older observations less and less as time passes. In
+effect, SV and SQ adapt to changing conditions at a predictable rate that can
+be specified by the user.
+
+In addition, this approach is significantly less computationally expensive than
+traditional Fourier techniques. No Fourier transform of months-to-years-long
+data series is required, and memory requirements are comparably reduced, since
+a description of the state of the system at any given moment is only 1+m, where
+m is the number of data points in an SQ cycle, nominally 1 day.
+
+Finally, exponential smoothing is generally more robust to common issues with
+real time data series; it easily extrapolates SV and SQ across gaps in the
+data; it provides a running estimate of the variance of DIST, which can be used
+to set a threshold for spike detection; and it adjusts SV to accommodate DC
+offsets at rate specified by the user.
+
+
+## Example
+
+Usage and expected output for this algorithm is shown in this
+[Solar Quiet and Disturbance (Holt Winters)](SqDist.ipynb) IPython Notebook
+example.
+
+
+## References
+
+ - Archibald, B.C., and A.B. Koehler (2003), [Normalization of seasonal
+   factors in Winters'
+   methods](http://www.sciencedirect.com/science/article/pii/S0169207001001170),
+   Int. J. of Forecasting, 19(1), 143-148.
+
+ - Bodenham, D., and N. Adams (2013), [Technical Report: Continuous changepoint
+   monitoring of data streams using
+   adaptive estimation](http://wwwf.imperial.ac.uk/~dab10/techreport.pdf), ...
+   (Submitted to Elsevier; also a longer thesis is available from Imperial
+   College London)
+
+ - Byrd, R. H., P. Lu, and J. Nocedal (1995), [A limited memory algorithm for
+   bound constrained
+   optimization](http://epubs.siam.org/doi/abs/10.1137/0916069), SIAM J.
+   Scientific and Stat. Computing, 16(5), 1190-1208.
+
+ - Gardner, E. S. (2006), [Exonential smoothing: The state of the art --
+   Part II](http://www.sciencedirect.com/science/article/pii/S0169207006000392),
+   Int. J. of Forecasting, 22(4), 637-666.
+
+ - Hyndman, R. J., A.B. Koehler, J.K. Ord, and R.D. Snyder (2005), [Prediction
+   intervals for exponential smoothing using two new classes of state space
+   models](http://onlinelibrary.wiley.com/doi/10.1002/for.938/abstract), J.
+   Forecast., 24(1), 17-37.
+
+ - Hyndman, Rob J., and George Athana­sopou­los. "Forecasting: Principles and
+   Practice." Forecasting: Principles and Practice. OTexts: Online,
+   Open-Access Textbooks, May 2012. Web. <https://www.otexts.org/fpp>.
+
+ - Hyndman, R. J., and G. Athanasopoulos (2013), [Forecasting: principles and
+   practice](https://www.otexts.org/fpp), OTexts.

docs/algorithms/SqDist_usage.md

+Solar Quiet and Disturbance Algorithm Usage
+===========================================
+
+The Solar Quiet and Disturbance Algorithm calculates `Solar Variation`,
+`Solar Quiet` and `Disturbance` from raw Geomagnetic data.
+
+`geomag.py --algorithm sqdist`
+
+
+### Example
+
+This example uses a state file to produce magnetic-h-based Dist, SQ, and SV
+channels using the EDGE channel naming convention.
+
+    bin/geomag.py \
+      --input-edge cwbpub.cr.usgs.gov \
+      --observatory BOU \
+      --inchannels H E Z F \
+      --starttime 2016-01-03T00:01:00 \
+      --endtime 2016-01-04T00:00:00 \
+      --algorithm sqdist \
+      --sqdist-mag \
+      --sqdist-statefile=/tmp/sqdist_h_state.json \
+      --rename-output-channel H_SQ MSQ \
+      --rename-output-channel H_SV MSV \
+      --rename-output-channel H_Dist MDT \
+      --outchannels MDT MSQ MSV \
+      --output-iaga-stdout
+
+This example processes just one channel (X).
+
+    bin/geomag.py \
+      --input-edge cwbpub.cr.usgs.gov \
+      --observatory BOU \
+      --inchannels X \
+      --starttime 2016-01-03T00:01:00 \
+      --endtime 2016-01-04T00:00:00 \
+      --algorithm sqdist \
+      --sqdist-statefile=/tmp/sqdist_x_state.json \
+      --rename-output-channel X_Dist MXT \
+      --rename-output-channel X_SQ MXQ \
+      --rename-output-channel X_SV MXV \
+      --outchannels MXT MXQ MXV \
+      --output-iaga-stdout
+
+> Note only one inchannel is specified and the --sqdist-mag option is omitted.
+
+
+### Library Notes
+
+> Note: this library internally represents data gaps as NaN, and factories
+> convert to this where possible.
+
+
+### [Algorithm Theoretical Basis for "Geomag Solar Quiet and Disturbance"](SqDist.md) ###
+Describes the theory behind the Solar Quiet and Disturbance algorithm, as
+well as some implementation issues and solutions.