diff --git a/docs/algorithms/SqDist.md b/docs/algorithms/SqDist.md index 02789286236a54777aac4ec41d38ef122c822d69..1b41f1f47048be7dd171355d526d816199ee1c5a 100644 --- a/docs/algorithms/SqDist.md +++ b/docs/algorithms/SqDist.md @@ -20,22 +20,44 @@ variety of time scales associated with distinct physical phenomena. These are: substorms. SV is fairly easily separated from higher frequency variations using low-order -polynomials to *detrend* the data. SQ and DIST have similar time scales, and -are therefore more difficult to separate. Fourier series can be fit to data to -estimate SQ, which works well in non-real time situations. This approach -suffers in real time situations for both practical and theoretical reasons -that we won't discuss in detail here. +polynomials to *detrend* the data. SQ and DIST have similar time scales, and are +therefore more difficult to separate. Fourier series can be fit to data to +estimate SQ, which works well in non-real time situations. This approach suffers +in real time situations for both practical and theoretical reasons that we won't +discuss in detail here. ## Exponential Smoothing Real time decomposition of geomagnetic time series into SV, SQ, and DIST should explicitly acknowledge and address the time-causal nature of real time -observations. To this end, we employ a discrete form of exponential smoothing, with "seasonal" adjustments, to update estimates of SV and SQ based only on past observations. - -Simple exponential smoothing is a weighted average of the most recent observation and the previous weighted average, where the observation weight is, by definition, between 0 and 1. This weight is often referred to as a "forgetting factor", while its inverse referred to as the memory. More specifically, it represents the average age of the data that informs the current estimate of the average. If the forgetting factor is 0.5, the average age of the data used to estimate the current average is 2 samples; if the forgetting factor is 0.1, the average age is 10 samples; and so forth. If a memory in terms of actual time units is desired, simply define a forgetting factor equal to 1/memory_in_time_units/samples_per_time_unit. For example, if working with a 1-minute resolution time series, and the running average must most reflect the previous 30 days worth of observations, set the forgetting factor equal to 1/30/1440. - -Simple exponential smoothing can be extended to include "seasonal" adjustments. In other words, if there is a repeating cycle superposed on slowly varying baseline (e.g., SQ on top of SV), exponential smoothing can be applied to each element of the set of correction factors. In this case, if a forgetting factor is required to be in units of actual time, we must account for the fact that each correction factor only gets updated once-per-cycle, and multiply by the number of correction factors per cycle. For regular time series, this means samples_per_time_unit, so the forgetting factor for SQ that adapts on a 30-day time scale is simply 1/30. +observations. To this end, we employ a discrete form of exponential smoothing, +with "seasonal" adjustments, to update estimates of SV and SQ based only on past +observations. + +Simple exponential smoothing is a weighted average of the most recent +observation and the previous weighted average, where the observation weight is, +by definition, between 0 and 1. This weight is often referred to as a +"forgetting factor", while its inverse referred to as the memory. More +specifically, it represents the average age of the data that informs the current +estimate of the average. If the forgetting factor is 0.5, the average age of the +data used to estimate the current average is 2 samples; if the forgetting factor +is 0.1, the average age is 10 samples; and so forth. If a memory in terms of +actual time units is desired, simply define a forgetting factor equal to +1/memory_in_time_units/samples_per_time_unit. For example, if working with a +1-minute resolution time series, and the running average must most reflect the +previous 30 days worth of observations, set the forgetting factor equal to +1/30/1440. + +Simple exponential smoothing can be extended to include "seasonal" adjustments. +In other words, if there is a repeating cycle superposed on slowly varying +baseline (e.g., SQ on top of SV), exponential smoothing can be applied to each +element of the set of correction factors. In this case, if a forgetting factor +is required to be in units of actual time, we must account for the fact that +each correction factor only gets updated once-per-cycle, and multiply by the +number of correction factors per cycle. For regular time series, this means +samples_per_time_unit, so the forgetting factor for SQ that adapts on a 30-day +time scale is simply 1/30. In addition to real time data considerations, this approach is significantly less computationally expensive than traditional Fourier techniques. No Fourier @@ -45,17 +67,18 @@ system at any given moment is only 1+m, where m is the number of data points in an SQ cycle, nominally 1 day. Finally, exponential smoothing is generally more robust to common issues with -real time data series; it easily extrapolates SV and SQ across gaps in the -data; it provides a running estimate of the variance of DIST, which can be used -to set a threshold for spike detection; and it adjusts SV to accommodate DC -offsets at rate specified by the user. +real time data series; it easily extrapolates SV and SQ across gaps in the data; +it provides a running estimate of the variance of DIST, which can be used to set +a threshold for spike detection; and it adjusts SV to accommodate DC offsets at +rate specified by the user. ## Example -Usage examples can be found [here](SqDist_usage.md), and a much more detailed -description of this algorithm, and example inputs and outputs, can be found -[here](SqDistValidate.ipynb (a Jupyter/IPython Notebook)). +Usage examples can be found in [SqDist_usage.md](SqDist_usage.md), and a much +more detailed description of this algorithm, and example inputs and outputs, can +be found in [SqDistValidate.ipynb](SqDistValidate.ipynb (a Jupyter/IPython +Notebook)). ## References