Publishing Events into Mantis
The Mantis Publish library allows your application to stream events into Mantis on-demand.
Mantis Publish takes care of filtering the events you send into Mantis, and it will only transmit them over the network if a downstream consumer that is interested in such events is currently subscribed.
Mantis Publish contains a subscription registry where each client subscription is represented by its corresponding Mantis Query Language query. Mantis Publish evaluates all MQL queries from its subscription registry against each event as the event is generated by your application. It then tags events with all of the matching MQL queries and enriches the event with a superset of fields from the matched queries. That is, rather than emitting n events for n matching MQL queries, Mantis Publish will instead emit a single event containing all of the fields requested by the matched queries. This happens directly within your application before any events are sent over the network into Mantis. Further, Mantis Publish only dispatches events if a client is subscribed with a matching query. This means that you can freely produce events without incurring the cost until an active subscription exists.
How to Add the Mantis Publish Library¶
To add this library to your application, include mantis-publish-netty in your application’s dependencies.
If your application is guice enabled you can do the following
A spring based module is coming soon.
How to Stream Events Into Mantis¶
Setting up the client.¶
To stream events into Mantis
Without dependency injection:
Directly instantiate a
MantisEventPublisher in your application code. In order to create a
you will have to inject a few parameters. An example of which parameters are required and how to inject them
can be found in here.
Next, you must use the
MrePublishClientInitializer class and call
MrePublishClientInitializer#start to start all
the underlying components
MantisRealtimeEventsPublishModule into your application.
In addition to injecting
MantisRealtimeEventsPublishModule you will also need to add the
ArchaiusModule and the
SpectatorModule if not already injected.
Injector injector = Guice.createInjector(new BasicModule(), new ArchaiusModule(), new MantisRealtimeEventsPublishModule(), new SpectatorModule());
Once injected, either manually via constructor or an injection framework such as Guice or Spring, the mantis-publish library is ready for use.
For each event your application wishes to send to Mantis, create a
Event object with your desired event fields,
and pass that
Event to the
MantisEventPublisher#publish method. For example:
// Create an `Event` for Mantis Publish using your application event. final Event event = new Event(); event.set("testKey", "testValue"); // Send your `Event` into Mantis. // Note: This event will only be dispatched over the network // if a subscription with a matching MQL query exists. eventPublisher.publish(event);
Configuring where to send data¶
We need to configure the location of the Mantis API server for the mantis-publish library to bootstrap
Add the following properties to your
mantis.publish.discovery.api.hostname=<IP of Mantis API> # mantis api port mantis.publish.discovery.api.port=<port for Mantis API> # This application's name mantis.publish.app.name=JavaApp
The Runtime Flow of Mantis Publish¶
Mantis Publish runtime flow consists of three phases: connecting, event processing, and event delivery.
Phase 1: Connecting¶
Mantis Publish will only stream an event from your application into Mantis if there is a subscriber to that specific application instance with an MQL query that matches the event. Any Mantis Job can connect to these applications. However, it is a best practice to have Source Jobs connect to Mantis Publish applications. This is because Source Jobs provide several conveniences over regular Mantis Jobs, such as multiplexing events and connection management. By leveraging Source Jobs as an intermediary, Mantis Jobs are able to consume events from an external source without having to worry about lower-level details of that external source.
This is possible through job chaining, which Mantis provides by default. When connecting to a Mantis Publish application, downstream Mantis Jobs will send subscription requests with an MQL query via HTTP to a Source Job. The Source Job will store these subscriptions in memory. These subscriptions are then fetched by upstream applications at the edge running the Mantis Publish library. Once the upstream edge Mantis Publish application is aware of the subscription, it will start pushing events downstream into the Source Job.
The Mantis Publish library not only handles subscriptions, but also takes care of discovering Source Job workers so you do not have to worry about Source Job rebalancing/autoscaling. For more information about Source Jobs see Mantis Source Jobs.
Creating Stream Subscriptions¶
Clients such as Mantis Jobs connect to a Mantis Publish application by submitting a subscription represented
by an HTTP request. Mantis Publish’s
StreamManager maintains these subscriptions in-memory. The
StreamManager manages internal resources such as subscriptions, streams, and internal processing
Clients can create subscriptions to different event streams. There are two types:
defaultstreams contain events emitted by applications that use the Mantis Publish library.
logstreams contain events which may not be core to the application, such as log or general infrastructure events.
Phase 2: Event Processing¶
Event processing within Mantis Publish takes place in two steps: event ingestion and event dispatch.
Event ingestion begins at the edge, in your application, by invoking
EventPublisher#publish which places the
event onto an internal queue for dispatching.
Events are dispatched by a drainer thread created by the Event Publisher. The drainer will drain events
from the internal queue previously populated by
EventPublisher#publish, perform some transformations,
and finally dispatch events over the network and into Mantis.
Events are transformed by an
EventProcessor which processes events one at a time.
Transformation includes the following steps:
- Masks sensitive fields in the event.
- Sensitive fields are referenced by a blacklist defined by a configuration
- This blacklist is a comma-delimited string of keys you wish to blacklist in your event. The
param.passwordkey is included by default.
- Sensitive fields are referenced by a blacklist defined by a configuration (
- Evaluates the MQL query of each subscription and builds a list of matching subscriptions.
- For each matching subscription, enriches the event with a superset of fields from the MQL query from all the other matching subscriptions (see the following diagram).
- Sends this enriched event to all of the subscribers (see Event Delivery below for the details).
More Mantis Publish configuration options can be found here.
Phase 3: Event Delivery¶
Mantis Publish delivers events on-demand. When a client subscribes to a Mantis Job that issues an MQL query, the Event Publisher delivers the event using non-blocking I/O.