Skip to main content

Sliding Window Mechanism

Summary​

The tremor-query language currently supports temporal window processing based on a provided system (wall) clock or data-driven intervals. Currently, however, the only supported windowing style in select statements are tumbling.

There is no mechanism for tumbling windows where there may be multiple simultaneous (overlapping) windows. The sliding window mechanism corrects this.

This RFC addresses these limitations by introducing a sliding window mechamism that can be configured with a number of steps.

Motivation​

The tremor-query language cannot currently define windowing mechanics such as pair-wise comparisons or sliding data-driven windows that captures 'the last seconds worth' of data that update on an event by event basis.

Sliding windows occur frequently in event processing algorithms and their addition to tremor-query is a natural extension to the language.

Guide-level Explanation​

Definition of a sliding window of step size 2:

define sliding window pairs
with
  size = 2,
end;

Application of the pairs window in a select statement:

select {
  "window": window,
  "count": stats::count(),
  "of": win::collect(event),
}
from in[pairs]
into out;

Sliding windows should work with tilt frames:

select {
  "window": window,
  "count": stats::count(),
  "of": win::collect(event),
}
from in[sliding_pairs, tumbling_pairs, tumbling_triples, sliding_quads]
into out;

An interval-based sliding window of duration 1 second:

define sliding window last_second
with
  interval = datetime::with_seconds(1),
end;

The addition of a sliding window mechanism is a fairly cosmetic language change, but a fairly significant change to runtime facililities in window processing semantics and mechanics, grouping mechanism, tilt framing and may involve other enhancements or changes in order to manage memory pressure optimally.

This RFC does not concern itself with implementation specifics.

Reference-level Explanation​

None.

Drawbacks​

Sliding window mechanism uses relatively more memory when compared with tumbling windows, and this should feature in documentation and examples.

Rationale and Alternatives​

As described in the summary.

Prior Art​

Sliding windows are a common feature in CEP/ESP and aggregation systems.

Unresolved Questions​

This RFC does not specify internals or implementation, which is left to the implementor. The motiviating example should be sufficient to drive a suitable implementation.

Future Possibilities​

None known at this time.