New Instrument class and Observatory class edits

original MR: !279 (closed)

Tagging @bgeels, @nshavers and @swilbur to keep them in the loop.

The goal of this merge request is to develop the metadata framework and backend tools for a frontend inventory manager.

Some confusing background information:

The database, or metadata table assumes the structure of Metadata.py with attributes as columns, this is defined in metadata_table.py. The metadata_history_table preserves the history of a metadata object from the metadata_table.py but copies the same structure. So, all of the Metadata.py attribute values (and therefore the indices) can be redefined except for id which serves as the key to the metadata table. This is important because we don't want to create a new metadata object every time something about an observatory or instrument changes but we want to preserve the history of the changes.

If you look at the Metadata.py you will notice the category attribute, the categories are defined here. Currently, we have a class defined for Reading and Observatory but I would like to further define Observatory and write an Instrument class in this MR. Keep in mind that these classes will define what is in the JSON blob within the Metadata class in the metadata attribute (currently defined as a Dict). Here is an example:

Click to expand

{ "id": 35535,
"metadata_id": null,
"created_by": "copy_observatory.py",
"created_time": "2023-06-27T14:54:50.000000Z",
"updated_by": null,
"updated_time": null,
"starttime": null,
"endtime": null,
"network": "NT",
"station": "BSL",
"channel": null,
"location": null,
"category": "observatory",
"priority": 1,
"data_valid": true,
"metadata": {
"id": "BSL",
"name": "Stennis Space Center",
"agency": "USGS",
"latitude": 30.35,
"elevation": 8,
"longitude": 270.365,
"agency_name": "United States Geological Survey (USGS)",
"declination_base": 215772,
"sensor_orientation": "HDZF"
},
"comment": null,
"review_comment": null,
"status": "new"
},\

The database was designed this way to have freedom defining what is within the JSON blobs and also so that we do not use linked tables. However, we can link elements of a single table (or two, counting the history table) through fields in their JSON blobs.

A major topic of discussion between Josh, Brendan and I so far has been what features should be specifically associated with Instruments and Observatories and how to link them. Here are some thoughts I/we have had:

• Instrument class should include offsets, electronics, sensors, dataloggers, and digitizers.
• the Active_Instruments class will serve as a "bucket" for these items that are currently active and will be defined within an Observatory metadata object but also simultaneously linked to an Instrument metadata object (probably via starttime/endtime).
• Observatory class should include marks and piers, however there can be multiple marks and piers at a site and therefore multiple azimuth values. We may choose to differentiate certain mark and pier arrangements with different location codes, meaning we could have two "BOU" observatory metadata objects with their own metadata_history, but they would have different location codes and contain different mark/pier arrangements.
• We may have a single observatory "bucket" similar to the Active_Instruments class that will contain all observatory metadata for a particular observatory. Meaning, we may have one "BOU" metadata object, that contains and links to all other "BOU" metadata objects with different location codes that I described above.
• I have yet to decide if I think the above point is making things too complicated by having buckets within buckets, and I may just want to have all marks and piers under one observatory metadata object instead of with different configurations under different location codes.

requested review from @erigler

assigned to @awernle

changed the description

added 26 commits

• 38311060...f3b2b991 - 23 commits from branch ghsc/geomag:master
• 254f8cb5 - New Instrument class
• a3a4aac5 - Moving Instrument.py to category folder under metadata
• d5736583 - Adding more attributes and GeoJSONs to Observatory class

Compare with previous version

mentioned in merge request !292 (merged)

Alex and I spoke in real time about how to best proceed with development of tools needed to support an "inventory manager". Here is a summary of what we agreed upon:

1. for now, we will continue to store observatory category objects as flat json blobs (that is, no cross-references to pier or mark database objects yet); this maintains compatibility with the front-end (see comment above about geojson)
2. Alex will finish polishing this MR and merge.
3. Alex will work on back-end tools needed to support an "Inventory Manager" application.
4. Once the Inventory Manager is complete, we will work on integrating it with the Operations app so that instrument and observatory sub-categories can be tracked.

The intent is to provide freedom to develop the inventory app as-needed, without worrying too much about the Operations app, which can function well-enough in its current state. Eventually we will want to integrate the two.

added 46 commits

• d5736583...f61547ef - 44 commits from branch ghsc/geomag:master
• 9e75038f - New Instrument class
• 09e41fcd - Moving Instrument.py to category folder under metadata

Compare with previous version

mentioned in issue #111 (closed)

added 1 commit

• 4f115756 - Cleaning up Instrument class with descriptions and deleting unnecessary attributes.

Compare with previous version

added 1 commit

• 5a869296 - Adding Field descriptions to mroe attributes.

Compare with previous version

added 2 commits

• 985cfd6e - New Typer app to upload Offset, Sensor and/or Electronics metadata to database.
• 3c6e3243 - Added copy-instrument command.

Compare with previous version

added 1 commit

• f90a1892 - Adding InstrumentCategory class in the Instrument base class and initializers to child classes.

Compare with previous version