Draft: Overhaul EdgeFactory for Edge data migration
ABANDONED MERGE REQUEST
After offline discussion with the Geomag dev team, a consensus arose that this MR might be too complicated/clever for its own good, and therefore difficult to maintain. Probably the biggest driver of this complexity was the desire to use both the queryserver and earthworm protocols (the former being more efficient, the latter more real time) in a single call. A related consideration was a desire to allow either the raw input server or miniseed server for uploading data to Edge.
I will re-work this into a new and probably much simpler MR (MR !386 (merged)). EdgeFactory and MiniSeedFactory will not be combined, but there will be some items in this MR that should improve both those factories separately (and there were definitely a few bug fixes that shouldn't be forgotten).
This is a substantial merge request required before we can initiate the long-anticipated/promised Edge data migration. It overhauls the EdgeFactory to handle both the "traditional" compressed miniseed, and the uncompressed float32 miniseed returned by ObsRIO data loggers. Some of the changes to EdgeFactory might well have been to the so-called MiniSeedFactory that has been used for a few years now. However, that name was never really appropriate, and will eventually be re-purposed for a file-based miniseed factory.
In short, the main changes are:
- allow user to specify a SNCL mapper
- read integer or float data via CWBQuery ports using obspy's neic client
(CWBQuery cannot read from Edge's realtime RAM buffer, but is more efficient for long historical data retrievals) - write integer or float data via MiniSeedServer ports
(MiniSeedServer cannot update Edge's realtime RAM buffer, but it is more efficient for non-realtime data batch uploads) - retain RawInputServer and Earthworm capability to be able to manipulate data in Edge's realtime RAM buffer
- remove all default write ports, and fall back to a single default read port
(frankly, a user should really know what they are doing before writing to Edge anyway) - update filters.py so real time processing that uses the geomag-filter typer app will work while no longer requiring the MiniSeedFactory
(this was tested extensively using a local edge server that was configured reasonably similarly to the mages)
I offer some detailed pre-comments below to explain and motivate discussion of some of the changes. Any additional comments or critiques are welcomed. This is important enough that I hope to get several sets of eyes on it before merging, so...