Skip to content

Draft: New flag metadata class

Wernle, Alexandra Nicole requested to merge ghsc/users/awernle/geomag-algorithms:flag_metadata into master

Tagging @bgeels, @nshavers and @swilbur to keep them in the loop.

The purpose of this merge request is to create a Flag class for the Flag metadata category. This class will serve as a guide for identifying problems or events of interest in our magnetic timeseries data (like spikes, offsets, etc.)

I am going to assume you generally understand how the metadata database works based on the description here :!290.

I imagine that flag metadata will be created by: 1) automated algorithms such as a spike detection algorithm that runs on a chron job at the end of the day, 2) frontend interaction like highlighting "field work" times, and 3) during processing. Because this flagging will serve as a major part of our future processing procedures, many nuances will need to be considered.

Some things to think about:

• How exactly will the above 1, 2 and 3 function together in the future and what would be the most efficient approach?

	○ We could lump all flags that occur in a day into a single daily metadata object .
		§ This would look like one single metadata object for a particular day, but with multiple flags within the metadata
		§ This should work for IAGA and PCDCP files that are always split by days.
		§ How well would this work with the frontend? If we want to highlight these flags on plots, would this impact the resolution at which the flags load while scrolling in and out?
		§ This should work for frontend interactions such that if someone is selecting/editing flags for a day, they will only be adding to a single daily metadata object.

• We need to agree on how certain features are defined and if validators should be put into place.
     - in a case like Guam, I may qualify an entire period as "noise", whereas Brendan may want to qualify noise, offsets and spikes. How should this be handled? What about a period where there are many spikes? In this case someone may not want to select individual spikes.

Other considerations:

• Should we consider temperature variations?
• How will this look for data of different frequencies? 
• What might a future processing tools metadata class look like?
	○ it will document processing tools like filters
	○ Raw data + flag metadata + processing order of operations= processed data
	○ Order of operations needs to be documented =>look into stationxml stages as example
• We need to create tests for efficiency.
    • How might older metadata be compressed in the future?
Edited by Wernle, Alexandra Nicole

Merge request reports