FDSNFactory should use Trace.remove_sensitivity(), not Trace.remove_response(), to calibrate data

ObsPy's Trace.remove_response() function is already problematic for various reasons, and to be honest, I'm not even sure I fully understand its purpose when working correctly (is it supposed to add back signal that gets filtered out by the instrument/electronics?).

A simpler approach, and more consistent with Geomag observatory data processing, is to use ObsPy's Trace.remove_sensitivity() function instead. Here is an example snippet:

from obspy.clients import fdsn
from obspy.core import UTCDateTime

endtime = UTCDateTime.now()
starttime = endtime - 1800

client = fdsn.Client()
stream = client.get_waveforms(
    'N4', 'E46A', '40', 'BF?',
    starttime, endtime,
    attach_response=True,
)

for trace in stream:
    # apply total sensitivity
    trace.remove_sensitivity()
    if trace.stats.response.instrument_sensitivity.input_units == "T":
        # convert to nT (nanoteslas) if units are "T" (Teslas)
        trace.data *= 1e9

Note use of the magnetic variometer at site N4-E46A-40. ObsPy's remove_response() function returns junk if used here, but remove_sensitivity() works fine (N4-E46A-40 metadata also specifies physical units of "T" (Teslas), most others specify "nT" (nanoteslas)).

As an added bonus, this approach fixes an issue where all NaNs were returned from the FDSNFactory if very recent data are requested. The reason was/is that remove_response() uses an FFT to "remove" responses in the frequency domain, but the FDSNFactory would pad the timeseries with NaNs (using TimeseriesUtility.pad_and_trim_trace()) in order to include the requested endtime, even if that data were not yet available...numpy's fft library doesn't handle NaNs well. However, with remove_sensitivity(), only actual missing data are returned as NaNs, which is what we expect.

assigned to @swilbur

A couple minor additions to this issue:

• create "default" logic that:
  • uses Trace.remove_response() if the object Trace.stats.response.instrument_sensitivity does not exist¹
  • uses Trace.remove_sensitivity if the object Trace.stats.response.instrument_sensitivity DOES exist
• provide a parameter in FDSNFactory's __init__() to force either of these options if desired

1. According to FDSN's station.xml standard, this should only be possible if the response is an "InstrumentPolynomial", in which case no FFT is taken to "invert" the spectrum.

closed with commit 0792d5a0