Modify TimeseriesUtility.py's split_trace()

The split_trace() function always dropped the last sample. This was seemingly part of a flawed logic designed to prevent traces being written to Edge by the MiniSeedInputClient that spanned midnight. This worked, but only if the original trace actually spanned midnight. For the majority of data that do not span midnight, this dropped a sample every time. The updated logic does what I believe was the original intent (that is, it breaks traces into more manageable chunks) without duplicating or losing any data. This is consistent with obspy's concept of data slices, where a starttime and endtime are always inclusive, but it deviates somewhat from MiniSeed logic, which assumes a starttime, plus a delta, plus a number of samples.

It was necessary to update a couple unit tests for the MiniSeedFactory to work with this new logic, but I am certain that the original test logic was not based on anything but what the TimeseriesUtility.py function(s) returned when they were originally written.

requested review from @awernle

assigned to @erigler

Here's a lightly-edited copy of a Teams discussion that included both @bgeels and @awernle. It is not actually critical to the review of this MR, but it relates to a discrepancy between how obspy creates MiniSeed blocks, and the standard employed by the USGS's Edge software:

[09:12] Rigler, Erin (Josh)

I'm cleaning up some small mistakes that ultimately led to this gaps-on-update issue. Long-story-short, [X] built logic into the MiniSeed writer that prevented blocks that spanned midnight. This logic actually threw away the last sample of every trace, regardless of whether it spanned midnight or not. I've fixed this.

Now I have to update some unit tests, and find myself confused (again) by the MiniSeed standard. What I think I know is that for 64-bit floats, the data returned from ObsRIO packs 56 samples into each 512-byte MiniSeed block (112 for 32-bit floats). I always assumed this was because the first 64 bytes of the MiniSeed block is reserved for various data flags, leaving 448 bytes for actual data.

However, if obspy is used to write the MiniSeed blocks (using trace.write(), as [X]'s code does), it seems to force 114 samples in each 512-byte block. In other words, it's like it assumes a 56-byte header instead of 64.

[09:30] Geels, Brendan R.

I remember finding the same discrepancy between [Edge]'s definition of a miniseed packet and the latest miniseed handbook. In the handbook the 64 byte header actually ends at the 55th byte and you can start placing data after that (in byte 44 you specify where the data starts so theoretically the data start point can move around if you'd like it to). 

I recommend testing this in mage1 before merging to mage2, maybe @bgeels can create a false outage in one of the test systems and we can check if all of the data backfills, but otherwise I approve.

approved this merge request

mentioned in commit 86ee09d8

merged

I will disable the importer for CMT in about 10 minutes then re-enable it after about 2 hours. Does that sound like a sufficient timespan for a test?

Wrong button

resolved all threads