Combining identical SNCL traces with overlapping times result in gaps

If data gets "overwritten" in the Edge database, there will be multiple miniseed (MS) blocks holding similar data, since Edge, by design, never deletes previously written data.

The obspy NEIC client, used by geomag-algorithms' MiniSeedFactory, does not attempt to filter or sort these blocks. It simply pulls every MS block that holds any data falling within the requested time interval and places it into an obspy Stream object as a unique Trace. Finally, it uses obspy's powerful Stream.merge(-1) method to combine these similar Traces.

If there are time overlaps in these Traces, Stream.merge(-1) will leave the offending Traces alone, and return a multi-Trace Stream. Therefore, following the example of previously written I/O factories in geomag-algorithms, the MiniSeedFactory calls Stream.merge() to force the multiple Traces to be combined.

That said, whoever wrote the first geomag-algorithms I/O factory that used this method did not fully understand or appreciate what the default and implicit "0" option did when Stream.merge() was invoked. Any time overlaps are turned into "gaps"! This wreaks havoc with the logic used by geomag-algorithms' update mechanism, and makes it nearly impossible to overwrite data and retrieve these same data.

This was rarely, if ever, an issue until the MiniSeedFactory came along because the older EdgeFactory, which uses the obspy earthworm client, could rely on Edge to never return duplicate copies of data. Rather it always returned the most recently written data, effectively disallowing access to "overwritten" data in Edge.

One possible solution is to invoke Stream.merge(1), which applies obspy's logic for over-writing data. This logic is described in detail in their online docs, which is a good thing because the actual code is difficult to decipher.

HOWEVER, this logic fails us if we want to overwrite an interval that falls within (is "contained by", using obspy's language) a previously written interval. There won't be a gap, but the expectation that data written at a later time will take precedence upon retrieval is violated, and only the larger "containing" interval data will be returned.

After much experimentation with Edge and the NEIC client, I convinced myself that Edge returns MS blocks in the order they were originally written. We can put this to our advantage by introducing a fairly small amount of new code to the MiniSeedFactory to replace where it originally called data.merge() ("data" being the Stream object returned from the NEIC client).

# initialize empty temporary stream
st_tmp = Stream() 
# loop over returned traces in the order they were returned
for tr in data: 
    # add next tr to temporary stream
    st_tmp += tr
    # replace time overlaps with gaps
    st_tmp.merge(0)
    # split on gaps, then add next trace to stream, and finally,
    # replace gaps with most recently written data in tr
    st_tmp = (st_tmp.split() + tr).merge(0)
# point `data` to the newly merged stream and continue processing
data = st_tmp

This should always give precedence to the latest miniseed block written to edge, regardless of its time stamp interval. This is very Edge-specific, and may not be appropriate for data retrieved from other sources, or using different protocols. This should be OK for geomag-algorithms' MiniSeedFactory.

added Discuss label

assigned to @erigler

@bgeels , @awernle : Please read this issue description closely, and let me know if you have any questions. The problem gets a little complicated, but I believe I have come up with a relatively simple solution after considerable thought and experimentation.

Please pay particular attention to the example code snippet, and my justification for proposing it.

Adding @swilbur, just because others are on travel this week. Any comments on this discussion issue will be appreciated though.

Hi Josh, I'm most likely misunderstanding some logic here but this is my one question. In this line "st_tmp = (st_tmp.split() + tr).merge(0)" is the overlap not going to be filled in by another gap? Or by giving a trace will merge.(0) use the trace to fill the gap and not resort to creating a gap based on where the traces differ?

The online docs are a bit confusing, but are you trying to replicate what is being done in this example?

Method = 0 Trace 1: AAAA--AAAAAA (contained trace, missing samples) Trace 2: FF 1 + 2 : AAAAFFAAAAAA

That example did not keep the line indenting in the way I intended. Here is a screenshot

Yes.

The first call to st_tmp.merge(0) forces any overlaps to be returned as gaps. Then I split the resulting stream, which will create multiple traces. Finally, I st_tmp.merge(0) one last time to do exactly what you point out in the screenshot above.

Although, as I type, I now wonder if I don't have a redundant "split" in there. Maybe I can just do: st_tmp = (st_tmp + tr).merge(0). I'll need to test that.

I confirmed that the .split() can be removed from my example code and get identical results.

changed the description

I'm uncomfortable with the fact that we're relying on an undocumented 'feature' of edge here. But at the same time I don't see any other way to do it since the edge blocks lack any kind of 'version' or 'upload date' info.

I think I understand what the code here is doing, but am not super confident. Have you been able to test out this code snippet on some example data with gaps and overlaps?

I did test the code I originally wrote above, but note my response to Spencer's question. I might be able to make this even cleaner, but I need to test.

As for "undocumented features", I agree, but I believe it is something very similar that Dave implemented in the Earthworm client, and which ensures only the latest uploaded data is ever returned. I can check and confirm this with Dave K.

Dave K. confirmed that Edge does indeed return duplicate blocks in the order they were written. I'm not sure it's actually documented, but Dave's explanation makes it seem very deliberate:

The data from the QueryServer is always in reception order as it is written as it is found in the source files unless the source files have been reorganized. The USGS does not reorganize its data but others like CalTech do because it increases performance for research processing).

mentioned in merge request !260 (merged)

closed with commit 6c1bbf3c

closed with commit 204f2fdc

closed with commit c1247b2b

closed with merge request !260 (merged)