Inhaltsverzeichnis
Filters
Filters are mathematical operations running on a timeserie as a whole. They can be used as a preprocessing step, applied to the raw measurment data, at the end of the processing as a postprocessing step or „temporarily“ for visualisation in a plot sheet.
There is a default set of filters implemented. The available filters are explained in the following chapters.
Every filter can have attributes. Some attributes are used from most of the filters. For some of these attributes default values are defined:
stepWidth ="1/120" samplingFrequency="120" windowWidth ="12" noisyData ="true" cutOffFrequency ="5" zeroPhase ="true" sectionMinLength ="10" useBinValues ="false"
The filter types and characteristics can be defined in the XML markerset/labelset files which define the data loaded for processing or which parts of the processed data should be written in an output file.
Custom filters can be added by an plugin architecture.
in pre-/postprocessing
The filter-elements, defining the filter charatericstics, can be used at different positions in the hierarchie of a labelset:
<?xml version="1.0" encoding="UTF-8"?> <LabelSet> <Filter name="butterorder2" implName="default" cutOffFrequency="20" windowWidth="" stepWidth="" noisyData="true" samplingFrequency="1211" zeroPhase="true" sectionMinLength="3"/> <Labels fileSuffix="c3d" mimeType="application/c3d-vicon"> <Filter name="butterorder2" implName="default" cutOffFrequency="20" windowWidth="" stepWidth="" noisyData="true" samplingFrequency="1211" zeroPhase="true" sectionMinLength="3"/> <Label name="ABC" comment="" flag="" firstDerivative="" secondDerivative="" thirdDerivative="" differentiations="" optional="" unchangeable="" mathType=""> <Filter name="butterorder2" implName="default" cutOffFrequency="20" windowWidth="" stepWidth="" noisyData="true" samplingFrequency="1211" zeroPhase="true" sectionMinLength="3"/> </Label> </Labels> </LabelSet>
Definitions as the first child of the labelset-element are applied to all timeseries, as a child of an label-element are applied only to the timeserie the label corresponds to.
in the plot-sheet
The filter-elements defining the filter charatericstics, can be used also as children of the label-elements in a plot sheet definition.
<?xml version="1.0" encoding="iso-8859-1" ?> <Sheet name="Bilateral shoulder abdcution"> <Pages> <Page name="Mir 100" rows="4" columns="3"> ... <Diagram name="Height" column="1" row="1" xunit="%" yunit="mm" diagramDescriptionDown="down" diagramDescriptionUp="up" plane="" diagramType="xt" diagramTitle="Height"> <Label name="Height" label="Height"> <Filter name="dft" implName="default"/> </Label> </Diagram>
Comments
- Till now, only in xt()- and xy()-plots timeseries are filterd.
Available Filter implementations
Poly4Diff
General
A differentiation method, implemented by application of a 4-th order interpolation polynom (5 points) based on a sliding window with width 5. Thaś why the minimum trial length is five.
Attributes
Attribute | Default | Description | Required |
---|---|---|---|
stepWidth | 1/120 [s] | The time distance between succeeding frames in the unit seconds. | No |
samplingFrequency | 120 [Hz] | The sampling frequency in the unit Hz as an alternative parameter to stepWidth. | No |
References
- Hildebrand, F.B. (1974). Introduction to Numerical Analysis, 2nd Edition, pp.111
- Kreyszig, Erwin (1983). Advanced Engineering Mathematics, 5th Edition, pp.793
- Yakowitz Sydney and Szidarovsky, Ferenc (1989). An Introduction to Numerical Computations, 2nd Edition, pp.18
NaNs
If the timeserie includes NaNs the differention is done for the parts between the NaNs. If these parts are less than five, the values of this range is set to NaN.
Differentiation with smooth (smoothdiff)
A differentiation method that simultaneously smoothes the data. This method is based on a sliding window technique. The data is approximated within the window with a quadratic polynomial. The coefficients of the quadratic polynomial were then used for calculating the derivative at the window's center (Rand and Stelmach, 2005).
Attributes
Attribute | Default | Description | Required |
---|---|---|---|
samplingFrequency | 120 [Hz] | Scaling coeffiecient (In our case : sampling frequency) | No |
windowWidth | 12 | Number of points in sliding window | No |
noisyData | false | Ways to treat ends - false ⇒ Not so good for noisy waveforms, but good (best) for smooth ones. true ⇒ For noisy waveforms is better than 1; Flattens ends for smooth ones. | No |
References
- The original C++ implementation is from Dr. Yuri Shimansky.
- Thanks for Laetitia Fradet for the idea to use this filter and to have a look in a Matlab implementation of this filter.
Issues
- The last index is set to NaN calculation of the diffential quotient needs the value for the index i+1.
- completly broken in the current version
Chunk Mean (chunkmean)
Calculate the mean of a chunk of a timeserie.
The timeserie length must be a multiple of the windows size. The phase of the result is shifted by (window size) /2 frames to the front. The result timeserie length = (original timeserie length)/(window size). This reduction of the timeserie length makes this filter useless in most of the cases.
Attribute | Default | Description | Required |
---|---|---|---|
windowWidth | 12 | Width of the window in which the mean is determined. | No |
WeightedAverage
Simple Low pass digital filter.
result[i] = (data[i-2] + 3d*data[i-1] + 4d*data[i] + 3d*data[i+1] + data[i+2])/12d
Butterworth and differentiating filter (butterdiff)
This filter determines the derivative based on a 4-th order interpolation polynom (poly4diff) after conversion of 0-values to NaN and a data smoothing by a butterworth filter with second order. This filter is used as default for calculation of derivatives, e.g. configured by the attribute „differentiations“ in the label of input labelset.
Attribute | Default | Unit | Description | Required |
---|---|---|---|---|
stepWidth | 1/120 | s | No | |
samplingFrequency | 120 | Hz | No | |
cutOffFrequency | 5 | Hz | No | |
zeroPhase | true | If set to false, only foward direction filterering is done resulting in phase shifting. | No | |
windowWidth | 5 | Width of the window which is moved over the timeserie. | No | |
deleteBorderFrames | 5 | Sometimes the borders to ranges of NaN are noise. With this attribute a number of frames can be defined which are set to NaN on the border. | No |
References
- Hildebrand, F.B. (1974). Introduction to Numerical Analysis, 2nd Edition, pp.111.
- Kreyszig, Erwin (1983). Advanced Engineering Mathematics, 5th Edition, pp.793.
- Yakowitz Sydney and Szidarovsky, Ferenc (1989). An Introduction to Numerical Computations, 2nd Edition, pp.18.
Butterworth second order (butterorder2)
Zero-phase forward and reverse (optional) butterworth filter of second order.
Attributre | Default | Unit | Description | Required |
---|---|---|---|---|
samplingFrequency | 120 | Hz | No | |
cutOffFrequency | 5 | Hz | No | |
windowWidth | 5 | No | ||
sectionMinLength | 10 | Segments minimal count of frames. If the data contains segments with less frames a warning is given and the data of that segment is not filtered. | No | |
zeroPhase | false | No |
Discrete Fourier Transformation (dft/dft_bin)
Computes 1D Discrete Fourier Transform (DFT). Each DFT output value S[k] is represents the amplitude at frequency 2πkN. The multiplication by the exponential function effectively shifts s[n] down in frequency by 2πkN. The sum over N time samples can then be thought of as applying a decimating lowpass filter. So, effectively, the N outputs of the DFT represent the results of applying a bank of equally-spaced filters across s[n]'s frequency band, thus measuring „how much“ energy is present in various frequency bands in the input signal.
Assuming that a large-enough sample rate was used during the conversion to discrete-time, the result of the DFT will look a lot like the Fourier transform of the original signal s(t).The DFT is only applicable to finite-length signals, so in order to apply it, one must truncate the discrete-time signal s[n] to some finite length N, resulting in a potentially-shorter signal sN[n].
The result array contains the power specturm based on the dft() without the leading DC component for k=0:
spectrum[k]=Math.sqrt(Math.pow(data[k*2],2.0)+Math.pow(data[k*2+1],2.0));
The center frequencies for the bins are available, if the needed attributes are set:
Attribute | Default | Description | Required |
---|---|---|---|
samplingFrequency | Sample rate in [Hz], needed to calculate the center frequencies for the bins. | No | |
useBinValues | false | If set to „true“ than a timeserie with the bin values are determined. This needs to set the samplingFrequency attribute too | No |
Comments
- The result array with the power spectrum has half size as the input array. Thatś why this filter can not be used in the input- or output-labelsets. It is assumed, that this filter is usefull only in the plotsheet functionality. If there are new applications which needs this filter to be defined in input- or output-labelset the needed resampling code can be added easy.
GCVSPL (Woltring filter)
Natural B-spline data smoothing, using the Generalized Cross-Validation and Mean-Squared Prediction Error Criteria of Craven & Wahba (1979).
Attribute | Default | Description | Required |
---|---|---|---|
order | cubic | linear, cubic, quintic, heptic | No |
derivative | 0 | 0= no derivative, 1=first derivative, 2=second derivative, … | No |
cutoffFrequency | auto | cut off frequency in [Hz] or „auto“ for automatic determination | No |
errorVariance | GCV | With the default value „GCV“ the degree of smoothing to minimize the generalized cross validation function is determined. This is asymptotically the same as minimizing the true mean squared error (Craven & Wahba, 1979) or set error variance explicit as double value >= 0 | No |
interpolateGaps | true | If set to true, then gaps will be interpolated | No |
The following parameters are determined and can be printed in the log-file:
CutOffFrequency, if it is autodetected GeneralizedCrossValidationValue (WK(1) ) MeanSquaredResidual (WK(2) ) ResidualSumSquaresDimension (WK(3)) = Estimate of the number of degrees of freedom of the residual sum of squares per dataset, with 0.lt.WK(3).lt.N-M. SmoothingParameter (WK(4)) = Smoothing parameter p, multiplicative with the splines' derivative constraint. TrueMeanSquareError (WK(5) ) = Estimate of the true mean squared error (different formula for |MD| = 3). GaussMarkovErrorVariance (WK(6))
MEMO: GCVSPL software package (C) COPYRIGHT 1985, 1986: H.J. Woltring
For large datasets (N » 0) and negligible boundary artefacts, the behaviour of a natural spline approximates that of a periodic spline. For the latter case, the frequency characteristic in the equidistantly sampled, uniformly weighted case is that of a double, phase-symmetric Butterworth filter, with transfer function H(w) = [1 + (w/wo)^2M]^-1, where w is the frequency, wo = (p*T)^(-0.5/M) the filter's cut-off frequency, p the smoothing parameter, T the sampling interval, and 2M the order of the spline. If T is expressed in seconds, the frequen- cies are expressed in radians/second.
It has been found empirically, that the effective number of estimated spline parameters Np is related to the Butterworth cut-off frequency wo as Np ~ M/2 + KM * wo * N * T, where Np ranges between M and N, and where KM is the integral over x from 0 to infinity of (1 + x^2M)^- divided by PI. For large M, KM approaches 1/PI from above; values for small M are: K1 = 1/2, K2 = 1/V8, K3 = 1/3. This relation has also been found to apply for uniformly weighted data which are sampled slightly anequidistantly, with T taken as the average sampling inter-val. For large Np, the relation with wo * N * T becomes nonlinear.
zero2nan
This filter is used as a workaround for data collection by a vicon system where it can happen that all three components of a position are equal 0 which mean that the data is wrong. The filter detects this case and sets the data point values to NaN.
moverms
defaultemg
bandpass
rectification
Caculates the absolute values. There are no properties.
removeoffset
oneeuro
This is a simple lowpassfilter especially usefull for filtering humon motion data.
To minimize jitter and lag when tracking human motion, the two parameters (minCutOffFrequency and beta) can be set using a simple two-step procedure.
First beta is set to 0 and minCutOffFrequency to a reasonable middle-ground value such as 1 Hz. Then the body part is held steady or moved at a very low speed while minCutOffFrequency is adjusted to remove jitter and preserve an acceptable lag during these slow movements (decreasing minCutOffFrequency reduces jitter but increases lag, minCutOffFrequency must be > 0).
Next, the body part is moved quickly in different directions while beta is increased with a focus on minimizing lag. First find the right order of magnitude to tune beta, which depends on the kind of data you manipulate and their units: do not hesitate to start with values like 0.001 or 0.0001. You can first multiply and divide beta by factor 10 until you notice an effect on latency when moving quickly. Note that parameters minCutOffFrequency and beta have clear conceptual relationships: if high speed lag is a problem, increase beta; if slow speed jitter is a problem, decrease minCutOffFrequency.
Attribute | Default | Unit | Description | Required |
---|---|---|---|---|
samplingFrequency | 120 | Hz | No | |
minCutOffFrequency | 5 | Hz | No | |
beta | 1 | Hz | No | |
dCutOffFrequency | 0 | Hz | No |
References
- Casiez G., Roussel N., et.al. 1EUR Filter (2012). CHI'12, the 30th Conference on Human Factors in Computing Systems
resampling
Resampling with simple linear inper/extrapolation.
Attribute | Default | Unit | Description | Required |
---|---|---|---|---|
inFrequency | 120 | Hz | No | |
outFrequency | Hz | Yes |
If outFrequency > inFrequency the more frames at the end are cut, else the unavailable frames at the end are filled with NaN.
interpolation() - Interpolation
Interpolation NaN/Inf-values with different possible methods.
Attribute | Default | Description |
---|---|---|
method | linear | Possible methods are dividedDifference, linear, loess, neville and spline. |
Method | References |
---|---|
linear | |
spline | |
loess | Locally weighted regression and smoothing scatterplots, W.S. Cleveland, Journal of the American Statistical Association, Vol. 74, No. 368 (Dec., 1979), pp. 829-836. |
neville | Introduction to Numerical Analysis, ISBN 038795452X, chapter 2. |
dividedDifference | Introduction to Numerical Analysis, ISBN 038795452X, chapter 2. or Hildebrand, F. B. Introduction to Numerical Analysis. New York: McGraw-Hill, pp. 43-44 and 62-63, 1956. |
NaN-values at the end of the timeserie can not be interpolated.
ransaclin() - RANSAC linear model (1d)
RANSAC with linear model.
Attribute | Default | Description | Optional |
---|---|---|---|
threshold | 0.04 | Threshold value to determine data points that are fit well by the linear model. | yes |
maxiterations | 1000 | Maximum number of iterations allowed in the algorithm. | yes |
inlierratio | 0.8 | Ratio of inlier number of data points to estimate model parameters from all points. The result is substitution by the current iteration only if at minimum the count of inlierratio of all frames fits to the result. | yes |
minimumnumber | 50 | Number of close data points required to assert that a model fits well to data. This is the number of frames which are randomly selected in a iteration of optimization | yes |
excludethreshold | not used | If this attribute is set to a double value bigger than zero, than fitted data is compared to the raw data and if the difference between both exceeds this threshold value the value is set to NaN else the raw data value is used. | yes |
fit | not used/false | Only used if excludethreshold is set. If set to true and excludethreshold is set, then all positions for which the value is bigger than excludethreshold the value is substituted by the fitted one instead of NaN. | yes |
If this filter is used to filter multi-component data types, e.g. Matrix3d, it can happen that only some components are set to NaN.