Operators

This page introduces the MQL operators, including a syntax summary, usage examples, and a brief description of each operator.

For the most part these operations use their RxJava analogs:

MQL operator RxJava operator
select map
window window
where filter
group by groupBy
order by n/a
limit take

For this reason you may find the ReactiveX documentation useful as well, though it is a goal of MQL to provide you with a layer of abstraction such that you will not need to use or think about ReactiveX when writing queries.

MQL attempts to stay true to the SQL syntax in order to reduce friction for new users writing queries and to allow them to leverage their experiences with SQL.

An essential concept in MQL is the “property,” which can take on one of several forms:

  • property.name.here
  • e["this"]["accesses"]["nested"]["properties"]
  • 15
  • "string literal"

Most of the MQL operators use these forms to refer to properties within the event stream, as well as string and numeric literals.

The last detail of note concerns unbounded vs. discrete streams. ReactiveX Observables that are not expected to close are an unbounded stream and must be bounded/discretized in order for certain operators to function with them. The window operator is useful for partitioning unbounded streams into discrete bounded streams for operators such as aggregate, group by, or order by.

select

Syntax: select property from source

Examples:

  • "select * from servo"
  • "select nf.node from servo"
  • "select e["tags"]["nf.cluster"] from servo"

The select operator allows you to project data into a new object by specifying which properties should be carried forward into the output. Properties will bear the same name as was used to select them. In the case of numbers and string literals their value will also be their name. In the case of nested properties the result will be a top-level object joined with dots.

For example, the following select example…

  • "select "string literal", 45, e["tags"]["cluster"], nf.node from servo"

…would result in an object like:

{
  "string literal": "string literal",
  45: 45,
  "tags.cluster": "mantisagent",
  "nf.node": "i-123456"
}

Warning

Be careful to avoid collisions between the top-level objects with dotted names and the nested objects that result in top-level properties with dotted names.

Aggregates

Syntax: "select aggregate(property) from source"

Examples:

  • "select count(e["node"]) from servo window 60"
  • "select count(e["node"]) from servo window 60 where e["metrics"]["latency"] > 350"
  • "select average(e["metrics"]["latency"]), e["node"] from servo window 10"

Supported Aggregates:

  • Min
  • Max
  • Average
  • Count
  • Sum

Aggregates add analysis capabilities to MQL in order to allow you to answer interesting questions about data in real-time. They can be intermixed with regular properties in order to select properties and compute information about those properties, such as the last example above which computes average latency on a per-node basis in 10 second windows.

Note

Aggregates require that the stream on which they operate be discretized. You can ensure this either by feeding MQL a cold Observable in its context or by using the window operator on an unbounded stream.

You can use the distinct operator on a property to operate only on unique items within the window. This is particularly useful with the count aggregate if you want to count the number of items with some distinct property value in that window. For example:

  • "select count(distinct esn) from stream window 10 1"

from

Syntax: "select something from source"

Example:

  • "select * from servo"

The from clause indicates to MQL from which Observable it should draw data. This requires some explanation, as it bears different meaning in different contexts.

Directly against queryable sources the from clause refers to the source Observable no matter which name is given. The operator is in fact optional, and the name of the source is arbitrary in this context, and the clause will be inserted for you if you omit it.

When you use MQL as a library, the source corresponds with the names in the context map parameter. The second parameter to eval-mql() is a Map<String, Observable<T>> and the from clause will attempt to fetch the Observable from this Map.

window

Syntax: "WINDOW integer"

Examples:

  • "select node, latency from servo window 10"
  • "select MAX(latency) from servo window 60"

The window clause divides an otherwise unbounded stream of data into discrete time-bounded streams. The integer parameter is the number of seconds over which to perform this bounding. For example "select * from observable window 10" will produce 10-second windows of data.

This discretization of streams is important for use with aggregate operations as well as group-by and order-by clauses which cannot be executed on, and will hang on, unbounded streams.

where

Syntax: "select property from source where predicate"

Examples:

  • "select * from servo where node == "i-123456" AND e["metrics"]["latency"] > 350"
  • "select * from servo where (node == "i-123456" AND e["metrics"]["latency"] > 350) OR node == "1-abcdef""
  • "select * from servo where node ==~ /i-123/"

The where clause filters any events out of the stream which do not match a given predictate. Predicates support AND and OR operations. Binary operators supported are =, ==, <>, !=, <, <=, >, >=, ==~. The first two above are both equality, and either of the next two represent not-equal. You can use the last of those operators, ==~, with a regular expression as in: "where property ==~ /regex/" (any Java regular expression will suffice).

group by

Syntax: "GROUP BY property"

Examples:

  • "select node, latency from servo where latency > 300.0 group by node"
  • "select MAX(latency), e["node"] from servo group by node"

group by groups values in the output according to some property. This is particularly useful in conjunction with aggregate operators in which one can compute aggregate values over a group. For example, the following query calculates the maximum latency observed for each node in 60-second windows:

  • "select MAX(latency), e["node"] from servo window 60 group by node"

Note

The group by clause requires that the stream on which it operates be discretized. You can ensure this either by feeding MQL a cold Observable in its context or by using the window operator on an unbounded stream.

order by

Syntax: "ORDER BY property"

Example:

  • "select node, latency from servo group by node order by latency"

The order by operator orders the results in the inner-most Observable by the specified property. For example, the query "select * from observable window 5 group by nf.node order by latency" would produce an Observable of Observables (windows) of Observables (groups). The events within the groups would be ordered by their latency property.

Note

The order by clause requires that the stream on which it operates be discretized. You can ensure this either by feeding MQL a cold Observable in its context or by using the window operator on an unbounded stream.

limit

Syntax: "LIMIT integer"

Examples:

  • "select * from servo limit 10"
  • "select AVERAGE(latency) from servo window 60 limit 10"

The limit operator takes as a parameter a single integer and bounds the number of results to ≤ integer.

Note

Limit does not discretize the stream for earlier operators such as group by, order by, aggregates.