Publishing Events to Mantis

The Mantis Publish library allows your application to stream events into Mantis on-demand. Currently on the JVM based library is available in Open Source Mantis.

Mantis Publish takes care of filtering the events you send into Mantis, and it will only transmit them over the network if a downstream consumer that is interested in such events is currently subscribed.

Mantis Publish contains a subscription registry where each client subscription is represented by its corresponding Mantis Query Language (MQL) query. Mantis Publish evaluates all MQL queries from its subscription registry against each event as the event is generated by your application. It then tags events with all of the matching MQL queries and enriches the event with a superset of fields from the matched queries. That is, rather than emitting n events for n matching MQL queries, Mantis Publish will instead emit a single event containing all of the fields requested by the matched queries. This happens directly within your application before any events are sent over the network into Mantis. Further, Mantis Publish only dispatches events if a client is subscribed with a matching query. This means that you can freely produce events without incurring the cost until an active subscription exists.

How to Add the Mantis Publish Library

To add this library to your application, include mantis-publish-netty in your application’s dependencies.

compile 'io.mantisrx:mantis-publish-netty:1.2.+'

If your application is guice enabled you can do the following

compile 'io.mantisrx:mantis-publish-netty-guice:1.2.+'

Note

A spring based module is coming soon.

How to Stream Events Into Mantis

Setting up the client.

To stream events into Mantis

Without dependency injection:

Directly instantiate a MantisEventPublisher in your application code. In order to create a MantisEventPublisher, you will have to inject a few parameters. An example of which parameters are required and how to inject them can be found in here. Next, you must use the MrePublishClientInitializer class and call MrePublishClientInitializer#start to start all the underlying components

With Guice:

Inject the MantisRealtimeEventsPublishModule into your application. In addition to injecting MantisRealtimeEventsPublishModule you will also need to add the ArchaiusModule and the SpectatorModule if not already injected.

    Injector injector = Guice.createInjector(new BasicModule(), new ArchaiusModule(),
                new MantisRealtimeEventsPublishModule(), new SpectatorModule());

Once injected, either manually via constructor or an injection framework such as Guice or Spring, the mantis-publish library is ready for use.

For each event your application wishes to send to Mantis, create a Event object with your desired event fields, and pass that Event to the MantisEventPublisher#publish method. For example:

// Create an `Event` for Mantis Publish using your application event.
final Event event = new Event();
event.set("testKey", "testValue");

// Send your `Event` into Mantis.
// Note: This event will only be dispatched over the network
// if a subscription with a matching MQL query exists.
eventPublisher.publish(event);

Configuring where to send data

We need to configure the location of the Mantis API server for the mantis-publish library to bootstrap

Add the following properties to your application.properties

mantis.publish.discovery.api.hostname=<IP of Mantis API>

# mantis api port
mantis.publish.discovery.api.port=<port for Mantis API>

# This application's name
mantis.publish.app.name=JavaApp

The Runtime Flow of Mantis Publish

Mantis Publish runtime flow consists of three phases: connecting, event processing, and event delivery.

Phase 1: Connecting

Mantis Publish will only stream an event from your application into Mantis if there is a subscriber to that specific application instance with an MQL query that matches the event. Any Mantis Job can connect to these applications. However, it is a best practice to have Source Jobs connect to Mantis Publish applications. This is because Source Jobs provide several conveniences over regular Mantis Jobs, such as multiplexing events and connection management. By leveraging Source Jobs as an intermediary, Mantis Jobs are able to consume events from an external source without having to worry about lower-level details of that external source.

This is possible through job chaining, which Mantis provides by default. When connecting to a Mantis Publish application, downstream Mantis Jobs will send subscription requests with an MQL query via HTTP to a Source Job. The Source Job will store these subscriptions in memory. These subscriptions are then fetched by upstream applications at the edge running the Mantis Publish library. Once the upstream edge Mantis Publish application is aware of the subscription, it will start pushing events downstream into the Source Job.

Note

The Mantis Publish library not only handles subscriptions, but also takes care of discovering Source Job workers so you do not have to worry about Source Job rebalancing/autoscaling. For more information about Source Jobs see Mantis Internals: Mantis Source Jobs.

Creating Stream Subscriptions

Clients such as Mantis Jobs connect to a Mantis Publish application by submitting a subscription represented by an HTTP request. Mantis Publish’s StreamManager maintains these subscriptions in-memory. The StreamManager manages internal resources such as subscriptions, streams, and internal processing queues.

Clients can create subscriptions to different event streams. There are two types:

  1. default streams contain events emitted by applications that use the Mantis Publish library.
  2. log streams contain events which may not be core to the application, such as log or general infrastructure events.

Phase 2: Event Processing

Event processing within Mantis Publish takes place in two steps: event ingestion and event dispatch.

Event Ingestion

Event ingestion begins at the edge, in your application, by invoking EventPublisher#publish which places the event onto an internal queue for dispatching.

Event Dispatch

Events are dispatched by a drainer thread created by the Event Publisher. The drainer will drain events from the internal queue previously populated by EventPublisher#publish, perform some transformations, and finally dispatch events over the network and into Mantis.

Events are transformed by an EventProcessor which processes events one at a time. Transformation includes the following steps:

  1. Masks sensitive fields in the event.
    • Sensitive fields are referenced by a blacklist defined by a configuration (mantis.publish.blacklist).
    • This blacklist is a comma-delimited string of keys you wish to blacklist in your event. The param.password key is included by default.
  2. Evaluates the MQL query of each subscription and builds a list of matching subscriptions.
  3. For each matching subscription, enriches the event with a superset of fields from the MQL query from all the other matching subscriptions (see the following diagram).
  4. Sends this enriched event to all of the subscribers (see Event Delivery below for the details).

Note

More Mantis Publish configuration options can be found here.

RequestEventProcessor includes all RequestEvent fields requested by any subscriber in the results it gives to all subscribers

Phase 3: Event Delivery

Mantis Publish delivers events on-demand. When a client subscribes to a Mantis Job that issues an MQL query, the Event Publisher delivers the event using non-blocking I/O.