📥

Continual Receiver

Event delivery is a fundamental piece of any monitoring solution. Continual's Receiver API is where events typically enter the solution and are routed for processing.

We want the mechanics of injecting an event to be dead simple and the API reflects this: it's a simple HTTP POST of event data sent in a common format (e.g. JSON, web form, or plain text).

Concepts

The Receiver API does not dictate a particular message broker technology, but it does use the concepts of topics and event streams.

Topics

Events related to the same basic subject (e.g. bank transactions, SNMP traps, weather updates) belong on the same topic. This gives downstream event processors a single place to subscribe to events about that subject.

Stateless Streams

When working with timestamped data, out-of-order arrival is harmless. The data is generally just pushed to a time-series database and indexed on timestamp, so the data is re-ordered later.

Stateful Streams

Stateful event streams, on the other hand, require order guarantees. Suppose a Smart Home monitoring system reports that a door has opened, and a few seconds later that the door has closed. If downstream systems receive these events out of order, their understanding of the state of the "real world" will be out-of-synch.

There are various strategies for retaining message order for downstream delivery, but many of them struggle to scale out. For the Receiver API, we borrow Apache Kafka's topic partitioning concept and allow event senders to specify an event stream name. The Receiver service guarantees that all events with the same event stream name are delivered to downstream consumers in the order they were received. (For Kafka users: an "event stream name" is essentially equivalent to a "partition key.")

Paths

When sending data to the Receiver service, client systems HTTP POST their data to:

POST /events/{topic}/{eventStreamName}

Events in the payload are pushed to the given topic using the given event stream name. For scenarios where event ordering is not important, senders can use a simpler path:

POST /events/{topic}

Finally, events can be pushed into a default topic by leaving the topic name out of the path:

POST /events

Formats

Internally, the Receiver API delivers events encoded as JSON objects. Inbound events are converted from their payload format into JSON objects as follows:

Payload Content Type	Treatment
`application/json`	The payload may be a JSON array or a single object. If it's an array, each member is considered an individual event. Any array member that's not an object will be converted to an object with a single member, "message", which has the original member value as its value.
`text/plain`	The entire payload is considered a single event. The event object has a single member "message" with the text payload as its value.
`application/x-www-form-urlencoded` or `multipart/form-data`	The form post is converted to an event object in which each <form> field is a member with its respective stringified value. File upload data in a multipart/form-data post is currently ignored.

Other Considerations

Calls to the Receiver API are authenticated, enabling multi-tenant use.
The Receiver service does not require a specific message broker technology. Our codebase includes a Kafka-backed implementation but any topic-oriented broker with message ordering guarantees could work.