Add zmax keyword; remove DIST values exceeding +/- (zmax * sigma)
-
add new keyword zmax=None
to SqDistAlgorithm.__init__() -
add new command line argument --sqdist-zmax
-
modify SqDistAlgorithm.process_one() to replace values in dist
that exceedzmax * sigmahat
with NaNs ifzmax
is not None;
Details
The SqDistAlgorithm.process_one() method produces an output named channel + "_Sigma"
. This is the recursively estimated standard deviation of channel + "_Dist"
. Note that channel + "_Sigma"
is not currently stored in Edge by the realtime data processing pipeline, although perhaps one day it will be, and it might be incorporated into other automatic "bad data" detectors. This will be a different merge request.
For what it's worth, SqDistAlgorithm also has a parameter named zthresh
, which is an input to SqDistAlgorithm.additive() (the Holt-Winters exponential smoother underlying this entire algorithm). This serves as a "z-score" threshold in the error term used to update SQ
and SV
. In other words, if the absolute value of err
exceeds zthresh * sigma
, SqDistAlgorithm.additive() will simply carry the current values of SQ
and SV
forward to the next time step. zthresh
does not cause any outputs to be modified.
In general, zmax
should be greater-than-or-equal-to zthresh
, but it isn't really necessary to enforce this. Both should always be positive.