Removing Outliers with BufStats

by Ted Moore

Prevent outliers from negatively impacting BufStats' statistical summary.

BufStats can find and remove outliers by using an analysis of the data to set boundaries for each channel in the buffer. Any frames that have a value outside the its channel’s boundaries will not be used to compute the statistics. The strictness of these boundaries is determined by outliersCutoff (see below for how this parameter is used). Removing these outliers before computing the statistics will prevent them from affecting the statistical summary so the output of BufStats is a better representation of the majority of the data.

The boundaries of each channel are computed using the interquartile range (IQR), thereby ensuring that the boundaries are relative to the scale and distribution of values. The lower bound of this range is 25th percentile - (IQR * outliersCutoff). The upper bound of this range is 75th percentile + (IQR * outliersCutoff). The 25th and 75th percentiles are also called “Q1” and “Q3” respectively, short for the 1st Quartile and 3rd Quartile.

The default of -1 bypasses this function, using all the frames in the statistical measurements.

An example

To demonstrate how this works, consider this output of a SpectralShape analysis (columns are frames and rows are channels of the features buffer):

First, BufStats will find the Q1 and Q3. Using these it will calculate the interquartile range (IQR) which is the difference between these two values (Q3 - Q1). Next a “Margin” is calculated as IQR * outliersCutoff (in this example, outliersCutoff = 1.1). Finally, the lower and upper bounds are calculated as a “Margin distance” below Q1 and above Q3. lower bound = Q1 - Margin, upper bound = Q3 + Margin.

Now, using the lower and upper bounds, BufStats checks the original values in the buffer to see if any fall outside this range.

Frames 1, 2, and 3 all have values that fall outside the boundaries. Each of these three frames will be removed from what is used to compute the statistical summary leaving these frames:

Here are the statistics these selected frames produce:

Compared to the statistics if all the frames were included: