Skip to content

approx-distinct-rolling

Input Stack:
w: Window
expr: TimeSeriesExpr
 ⇨ Output Stack:
TimeSeriesExpr
 

Estimate the number of distinct values seen within a trailing window, for example the number of distinct users active over the last hour. Where :approx-distinct estimates each interval on its own, this counts a value once if it was seen anywhere in the window.

For a live event, :approx-distinct answers "how many people are watching right now", :approx-distinct-rolling answers "how many people have watched in the last 30 minutes", and :approx-distinct-cumulative answers "how many people have watched at all".

Parameters

  • expr: A query for a metric published as a distinct count sketch. See :approx-distinct for how to publish it
  • w: Window size, either a duration, such as 30m, or a number of datapoints including the current value, such as 5. See Window Size

Window Size

Use a duration, 30m, to cover a fixed amount of time independent of the step size. The duration is rounded down to the nearest step boundary. A number of datapoints can also be used, but then the amount of time covered by the estimate will change when zooming out to a time frame that causes consolidation.

If the step size is at least as large as the window, then a single interval already covers the window and the result is the same as :approx-distinct.

Behavior

  • At least the per interval count: A value seen in any interval of the window is counted, so the result is at least :approx-distinct for the same interval, and is larger when the values turn over within the window. For small counts, a few hundred or fewer, the estimator can occasionally put the window slightly below the interval
  • Not a rolling max of the estimate: Applying :rolling-max to the output of :approx-distinct gives the highest number seen at one time in the window, which is smaller. Use :approx-distinct-rolling for the number seen across the window
  • The count is approximate: The same accuracy applies as for :approx-distinct, a relative standard error of roughly 13%
  • Grouping: Add (,key,),:by before the operation to get a count per group. As with :approx-distinct, the groups only add up when a distinct value can appear in just one group

Examples

The audience over the last 30 minutes next to the audience at any one time, for the live event in the sample data. The gap between the lines is the turnover within the window:

name,viewers.concurrent,:eq,
:sum,
:dup,
:approx-distinct,
active+now,:legend,
:swap,
30m,:approx-distinct-rolling,
last+30m,:legend

Rolling count broken out by device:

name,viewers.concurrent,:eq,
:sum,
(,device,),:by,
30m,:approx-distinct-rolling

Since: 1.9