Erin (Josh) Rigler authored 6 years ago and Claycomb, Abram Earl committed 6 years ago

The SqDistAlgorithm has had an issue since first being deployed wherein its
SV+SQ terms de-synchronized with the Dist term, which in turn led to frequent
reinitialization in real time operations. Reinitialization requires loading
and processing 90 days worth of data, which nullifies two major advantages of
this recursive algorithm (i.e., small data footprint, low cpu-usage).

This commit adds the ability to optionally trim NaNs off the end of the input
stream prior to calling the 'additive' Holt-Winters exponential smoother. This
ensures that the SV+SQ+Dist outputs are always synchronized, and that the
`next_starttime` element of the state file used to store the state between
calls to SqDistAlgorithm will always coincide with the beginning of the most
recent gap in a real time data stream.

Finally, this change does not address the situation where more than a single
"end gap" within a chunk of input data exists. In such a case, the algorithm
will still reinitialize because the `next_starttime` will not coincide with
the beginning of this gap. If this is undesirable, the calling procedure
should be set up such that the requested input chunks should be the same
length as the interval since the last call to SqDistAlgorithm. This can be
easily done given a recent commit to change the controller's `--realtime`
option.

Erin (Josh) Rigler authored 6 years ago and Claycomb, Abram Earl committed 6 years ago

The `--realtime` option for the Controller/CLI was previously only a
boolean flag that specified a fixed start/end interval depending on
the interval (minute or second). It still does this, but the user now
has the option to pass an integer argument (e.g., `--realtime 300`)
that allows the user to specify the number of seconds before NOW
to consider "realtime". For a clear explanation of how this works,
see: https://stackoverflow.com/a/34652207.

Erin (Josh) Rigler authored 6 years ago and Claycomb, Abram Earl committed 6 years ago

- allow adjusted and provisional strings in --type controller argument
- allow adjusted, provisional, and definitive strings in _get_url() in TimeseriesFactory
- return appropriate abbreviation for adjusted/provisional data type
- translate adjuste/prvisional to A0 location code in EdgeFactory
- only write out DECBAS for variation/reported data types
- translate adjusted/provisional to A0 location code in IMFJSONWriter.py
- in several places I set it up so that 'variation' and 'reported' were synonymous,
  although I am not sure I found every such instance.

Erin (Josh) Rigler authored 6 years ago and Claycomb, Abram Earl committed 6 years ago

Use NumPy's `isnan(var)` function instead of `var is np.nan`, since
the latter doesn't work if var is a 64bit float NaN.

change goes error behavior

changed from specific error to anything containing `FATAL`

Update version to 0.7.1 in package.json

Update version to 0.7.1 in setup.py

Arigdon usgs avg docs

ObsPy/NumPy have a bug related to masked arrays that this small
change works around by forcing gaps to be filled by NaNs, even
during intermediate processing.

- simplified default assignments in AverageAlgorithm.process() as per
  @jmfee-usgs' recommendation
- inserted command-line configuration logic to deal with location
  code

Merge branch 'AvgDocs' of https://github.com/arigdon-usgs/geomag-algorithms into arigdon-usgs-AvgDocs

moved ipynb to .gitignore, updated docs a little, cleaned up and reorganized AvgAlgorithm.py(); got rid of get_trace instead of removing, a new way of setting the latitude corrections with a dictionary (in process so that it works the same for command line vs script performance), set observatories to be more pythonic

created an interactive python notebook for average algorithm. Removed redundant metadata and added better functionality for scaling and observatory checks

Merge streams fix

Changed sampling_rate to delta in __create_trace() so that the start and endtimes are calculated correctly

Fixed some linting errors and made the merge_streams() code cleaner as well as added more unit testing