MQL
This page introduces the MQL operators, including a syntax summary, usage examples, and a brief description of each operator.
For the most part these operations use their RxJava analogs:
| MQL operator | RxJava operator |
|---|---|
select |
map |
window |
window |
where |
filter |
group by |
groupBy |
order by |
n/a |
limit |
take |
For this reason you may find the ReactiveX documentation useful as well, though it is a goal of MQL to provide you with a layer of abstraction such that you will not need to use or think about ReactiveX when writing queries.
MQL attempts to stay true to the SQL syntax in order to reduce friction for new users writing queries and to allow them to leverage their experiences with SQL.
An essential concept in MQL is the “property,” which can take on one of several forms:
property.name.heree["this"]["accesses"]["nested"]["properties"]15"string literal"
Most of the MQL operators use these forms to refer to properties within the event stream, as well as string and numeric literals.
The last detail of note concerns unbounded vs. discrete streams. ReactiveX Observables that are
not expected to close are an unbounded stream and must be bounded/discretized in order for certain
operators to function with them. The window operator is useful for partitioning unbounded streams
into discrete bounded streams for operators such as aggregate, group by, or order by.
select¶
Syntax: select property from source
Examples:
"select * from servo""select nf.node from servo""select e["tags"]["nf.cluster"] from servo"
The select operator allows you to project data into a new object by specifying which properties
should be carried forward into the output. Properties will bear the same name as was used to select
them. In the case of numbers and string literals their value will also be their name. In the case of
nested properties the result will be a top-level object joined with dots.
For example, the following select example…
"select "string literal", 45, e["tags"]["cluster"], nf.node from servo"
…would result in an object like:
{
"string literal": "string literal",
45: 45,
"tags.cluster": "mantisagent",
"nf.node": "i-123456"
}
Warning
Be careful to avoid collisions between the top-level objects with dotted names and the nested objects that result in top-level properties with dotted names.
Aggregates¶
Syntax: "select aggregate(property) from source"
Examples:
"select count(e["node"]) from servo window 60""select count(e["node"]) from servo window 60 where e["metrics"]["latency"] > 350""select average(e["metrics"]["latency"]), e["node"] from servo window 10"
Supported Aggregates:
MinMaxAverageCountSum
Aggregates add analysis capabilities to MQL in order to allow you to answer interesting questions about data in real-time. They can be intermixed with regular properties in order to select properties and compute information about those properties, such as the last example above which computes average latency on a per-node basis in 10 second windows.
Note
Aggregates require that the stream on which they operate be discretized. You can ensure this
either by feeding MQL a cold Observable in its context or by using the window operator on an
unbounded stream.
You can use the distinct operator on a property to operate only on unique items within the window.
This is particularly useful with the count aggregate if you want to count the number of items with
some distinct property value in that window. For example:
"select count(distinct esn) from stream window 10 1"
from¶
Syntax: "select something from source"
Example:
"select * from servo"
The from clause indicates to MQL from which Observable it should draw data. This requires some
explanation, as it bears different meaning in different contexts.
Directly against queryable sources the from clause refers to the source Observable no matter which
name is given. The operator is in fact optional, and the name of the source is arbitrary in this
context, and the clause will be inserted for you if you omit it.
When you use MQL as a library, the source corresponds with the names in the context map
parameter. The second parameter to eval-mql() is a Map<String, Observable<T>> and the from
clause will attempt to fetch the Observable from this Map.
window¶
Syntax: "WINDOW integer"
Examples:
"select node, latency from servo window 10""select MAX(latency) from servo window 60"
The window clause divides an otherwise unbounded stream of data into discrete time-bounded
streams. The integer parameter is the number of seconds over which to perform this bounding. For
example "select * from observable window 10" will produce 10-second windows of data.
This discretization of streams is important for use with aggregate operations as well as group-by and order-by clauses which cannot be executed on, and will hang on, unbounded streams.
where¶
Syntax: "select property from source where predicate"
Examples:
"select * from servo where node == "i-123456" AND e["metrics"]["latency"] > 350""select * from servo where (node == "i-123456" AND e["metrics"]["latency"] > 350) OR node == "1-abcdef"""select * from servo where node ==~ /i-123/""select * from servo where e["metrics"]["latency"] != null"select * from servo where e["list_of_requests"][*]["status"] == "success"
The where clause filters any events out of the stream which do not match a given predictate.
Predicates support AND and OR operations. Binary operators supported are =, ==, <>, !=,
<, <=, >, >=, ==~. The first two above are both equality, and either of the next two
represent not-equal. You can use the last of those operators, ==~, with a regular expression as
in: "where property ==~ /regex/" (any Java regular expression will
suffice). To take an event with certain attribute, use e["{{key}}"] != null.
If the event contains a list field, you can use the [*] operator to match objects inside that list.
For example, say the events have a field called list_of_requests and each item in the list has a field
called status. Then, the condition e["list_of_requests"][*]["status"] == "success" will return true
if at least 1 item has status equals success. Further, you can combine multiple conditions on the
list. For example,
e["list_of_requests"][*]["status"] == "success" AND e["list_of_requests"][*]["url"] == "abc"
This condition returns true if at least 1 item has status equals success and url equals abc.
group by¶
Syntax: "GROUP BY property"
Examples:
"select node, latency from servo where latency > 300.0 group by node""select MAX(latency), e["node"] from servo group by node"
group by groups values in the output according to some property. This is particularly useful in
conjunction with aggregate operators in which one can compute aggregate values over a group. For
example, the following query calculates the maximum latency observed for each node in 60-second
windows:
"select MAX(latency), e["node"] from servo window 60 group by node"
Note
The group by clause requires that the stream on which it operates be discretized. You can
ensure this either by feeding MQL a cold Observable in its context or by using the window
operator on an unbounded stream.
order by¶
Syntax: "ORDER BY property"
Example:
"select node, latency from servo group by node order by latency"
The order by operator orders the results in the inner-most Observable by the specified property.
For example, the query "select * from observable window 5 group by nf.node order by latency" would
produce an Observable of Observables (windows) of Observables (groups). The events within the groups
would be ordered by their latency property.
Note
The order by clause requires that the stream on which it operates be discretized. You can
ensure this either by feeding MQL a cold Observable in its context or by using the window
operator on an unbounded stream.
limit¶
Syntax: "LIMIT integer"
Examples:
"select * from servo limit 10""select AVERAGE(latency) from servo window 60 limit 10"
The limit operator takes as a parameter a single integer and bounds the number of results to ≤
integer.
Note
Limit does not discretize the stream for earlier operators such as group by, order by,
aggregates.