Erin (Josh) Rigler authored 5 years ago and Claycomb, Abram Earl committed 5 years ago

Significant updates to this demonstration and validation Notebook.

Erin (Josh) Rigler authored 5 years ago and Claycomb, Abram Earl committed 5 years ago

All the relevant information was transferred to AdjustedValidate.ipynb. These are
redundant, and may not even work with the latest versions of Geomag-Algorithms (I
didn't actually test this, but they haven't been executed in years).

Erin (Josh) Rigler authored 5 years ago and Claycomb, Abram Earl committed 5 years ago

Notebooks "change" just by opening them, which is unfortunate. Nothing substantial
actually changed, but I'm committing anyway, just to be safe.

Erin (Josh) Rigler authored 5 years ago and Claycomb, Abram Earl committed 5 years ago

Even on Github, the notebook does not always render the same way it does
on my system. So, I printed the AdjustedValidate.ipynb notbook as a pdf
file. It is not ideal, since links don't work, and some formatting is
still a little off. But it is readable...something kind of important for
"documentation".

Erin (Josh) Rigler authored 5 years ago and Claycomb, Abram Earl committed 5 years ago

- Substantial additions to ATB markdown
- Minor tweaks to code cells
- Address discrepancies between how WebAbsolutes and residual method
  spreadsheets define/process H baselines (doesn't fix bias noted in
  CMO quasi-definitive data)
- Re-ran entire updated validation suite on all observatories

Erin (Josh) Rigler authored 5 years ago and Claycomb, Abram Earl committed 5 years ago

This is the first of several related commits intended to better organize
algorithm docs that grow beyond a simple .md or .ipynb file. I created
an Adjusted folder, and then under that, an images folder. I then moved
a couple files around accordingly.

Erin (Josh) Rigler authored 5 years ago and Claycomb, Abram Earl committed 5 years ago

This notebook currently serves as an Algorithm Theoretical Basis,
validation report, and a not-so-user friendly tool for generating
Adjusted Data transform matrices manually. It needs to be cleaned,
and possibly broken into 2 or even 3 separate notebooks, but I'm
putting it into GitHub as a WIP to get some initial feedback.

moved ipynb to .gitignore, updated docs a little, cleaned up and reorganized AvgAlgorithm.py(); got rid of get_trace instead of removing, a new way of setting the latitude corrections with a dictionary (in process so that it works the same for command line vs script performance), set observatories to be more pythonic

created an interactive python notebook for average algorithm. Removed redundant metadata and added better functionality for scaling and observatory checks

Duff Stewart had several helpful comments regarding the last commit for this PR.
These were addressed, along with some additional minor Notebook clean-up.

- fixed datetime labels on plots so they are readable
- added another example of a constrained estimate that might be useful

Fixed a number of issues with the Adjusted Data affine transformation matrix generator
in the IPython Notebook AdjustedPhase1GenerationTool.ipynb:

- original version mistakenly pointed to geohazards server for baseline web service,
  which requires a password; we only needed to read baselines, not write, and this
  functionality is provided by the geomag server.
- original version choked if only 1 baseline set was available when extracting a
  pier correction from the set
- original version did not filter time stamps quite right; updated version at least
  does not include times not requested, although it may return fewer than expected if
  the timestamp associated with a baseline set is later/earlier than the measurement
  times (this frequently happens due to a bug in the webabsolutes package that has
  been reported)
- generate plots with datetimes
- added non-active cell that constrains the affine transformation matrix to apply a
  uniform scaling factor instead of allowing separate scaling for different vecor
  components
- redid the pseudo definitive data calculations based on average baselines to better
  reflect how MagProc software works.
- inserted/modified markdown and docstrings to better describe what is going on.

In response to Abe Claycomb's technical algorithm review, I attempted to
further clarify how alpha, beta, and gamma work. This includes a link to
a third-party website that does a good job of explaining what the "average
age" of an exponentially smoothed series is. SqDist.md will need to be
modified at least once more when the OFR is complete, so that it can
link to the official USGS documentation in addition to all the supporting
documents and websites.

- improved explanation of exponential smoothing, and how the alpha,
  beta, and gamma parameters work in SqDist.md.
- cleaned up command-line examples in SqDist_usage.md, and incorporated
  a description of the programming interface that was previously in the
  IPython/Jupyter notebook.
- removed programming interface description, and cleaned up markdown
  accordingly; re-ran entire notebook and regenerated graphics.

- (hopefully) a more clear explanation of how --sqdistd-alpha, --sqdist-beta,
  --sqdist-m, and --sqdist-gamma options work;
- actually tested the examples against an updated geomag-algorithms install,
  and it works!