Skip to main content

Firehose: configure streams and connectors

For the message format, installation identity and measurement fields, see Payload and schema within this Firehose section.

Firehose sends measurements from your installations to your own receiving system. Open Firehose under Developer when your account has access to this feature.

A pipeline defines which installations supply data, how it is delivered and where it goes. Each pipeline has one destination. Create two pipelines to send data to two different systems.

The Noorderlicht example configuration includes a monitoring stream for Zwolle and a paused archive stream. The instructions below explain how to configure pipelines for HTTP, Kafka, AMQP 0.9, AMQP 1.0 and MQTT. The example addresses are fictional: replace them with your receiving system's details.

Noorderlicht Firehose overview

Choose a connector​

ConnectorUseDetails you need
HTTPSend measurement messages to an API.URL, HTTP method, headers and any credentials.
KafkaPublish measurement messages to a topic.Broker addresses, topic and publishing permissions.
AMQP 0.9Route measurement messages through an exchange to queues.Host, exchange, routing key and a matching queue binding.
AMQP 1.0Send measurement messages to an AMQP 1.0 destination.Broker host, username, password and target address.
MQTTPublish measurement messages for an application to subscribe to.Broker host, topic and any client ID and credentials.

Prepare the receiving system before creating the pipeline. Use that system's connection details, not your Watt login credentials.

AMQP 0.9 is selected by default for a new pipeline. Choose AMQP 1.0 when your receiving system uses that protocol. These are separate protocols: an AMQP 0.9 exchange and routing key are not substitutes for an AMQP 1.0 target address.

Create a pipeline​

  1. Open Developer → Firehose and choose Create pipeline.
  2. Enter a recognisable name, such as Noorderlicht HTTP.
  3. Choose a delivery mode: Live or Durable.
  4. Select the installations whose measurements you want to send.
  5. Choose the connector and complete its destination fields.
  6. Choose Save pipeline.
  7. Check the linked installations, destination and status on the detail page. Then check that your receiving system receives messages.

Use search to find installations. Select visible selects the installations matching your current search. Clear removes the selection. Select at least one installation; the maximum is 10,000. The example selection is Noorderlicht Distributiecentrum — Zwolle.

If a message says that some installations were not linked, open the existing pipeline, choose Edit pipeline and correct the selection. Check the installation table afterwards. Do not create a second pipeline to repair incomplete subscriptions.

Choose the delivery mode​

Live delivers messages while the pipeline is running. Durable stores messages until they are delivered. Choose Durable to accommodate temporary interruptions at the receiver. The Noorderlicht examples use Durable.

To download earlier measurements, open Details and select the required period.

Set Max in flight​

Max in flight controls how many deliveries can run concurrently. The default is 64; use a value from 1 through 512.

Start with the default unless your receiving system requires a lower limit. More concurrent deliveries increase receiver load. When changing this setting, also check backlog and delivery latency. The HTTP example value is 16.

Configure TLS and the broker port​

For Kafka, AMQP 0.9, AMQP 1.0 and MQTT, select Enable TLS for an encrypted connection to the broker. TLS is off by default for a new pipeline. Use the setting required by your broker; configure the username and password separately.

Enter a hostname in Host, optionally followed by :port. Do not include a URL scheme such as amqps:// or mqtts://, a path or credentials. For Kafka, enter broker addresses with their ports in Brokers.

When no port is specified, these default ports apply:

ConnectorTLS offTLS on
AMQP 0.9 and AMQP 1.056725671
MQTT18838883

An explicitly entered port remains unchanged when you turn TLS on or off. Check the port as well: a broker may use a different port for TLS. Copy the correct combination from the broker configuration. For Kafka, use each broker's port; changing TLS does not change those addresses.

HTTP has no separate TLS setting. Use a URL starting with https:// for an encrypted HTTP connection.

HTTP​

Use HTTP to send measurement messages to Noorderlicht's monitoring API.

FieldExampleWhat to enter
URLhttps://noorderlicht.example.com/measurementsThe full receiving address, including the path.
MethodPOSTThe method expected by the API: POST, PUT or PATCH.
User / PasswordEmpty in this exampleCredentials for the receiving API, when required.
Max in flight16The maximum number of concurrent deliveries.
HeadersSee belowOne header per line, in Name: value format.
X-Noorderlicht-Source: watt
Content-Type: application/json

Enter each header once, with a colon between the name and value. If a name is repeated, the last value applies. A line without a valid name and colon is not sent.

Use the headers required by your API. A Content-Type header describes the message format; it does not change the measurement message's contents. Check that the API can process those contents. Keep secret header values out of screenshots and shared documents.

HTTP pipeline with Zwolle selected

Kafka​

Use Kafka to publish measurement messages to noorderlicht.measurements.

FieldExampleSetting
BrokersThe two addresses belowOne broker address per line or comma-separated.
Enable TLSOffSelect when the brokers require TLS. Check their ports as well.
Topicnoorderlicht.measurementsThe topic receiving published messages.
User / PasswordUser: noorderlicht-wattCredentials with publishing permissions for the topic.
Max in flight64The maximum number of concurrent deliveries.
Batching count20Batch count threshold: 1–128. With 0, you set no count threshold.
Batch period (ms)1000Batch time threshold: 1–5000 milliseconds. With 0, you set no time threshold.
Custom topic creationEnabledUse custom creation settings for a new topic.
Topic partitions3Number of partitions at creation: 1–64. With -1, you specify no custom count.
Replication factor3Number of replicas at creation: 1–7. With -1, you specify no custom count.
kafka-1.noorderlicht.example.com:9092
kafka-2.noorderlicht.example.com:9092

Batching​

Batching groups messages. With the example settings, a batch is sent at the count threshold of 20 or the time threshold of 1000 ms. The time threshold prevents a small number of messages from waiting for a full batch. Larger batches can make processing more efficient but add delay.

Topic creation​

Partitions divide the topic; replicas determine how many copies the cluster stores. Use counts appropriate for your cluster's capacity and configuration. The account needs creation permissions to create a topic. These settings apply to a new topic, not to changing an existing one.

Kafka brokers and topic

Kafka batching and topic creation

AMQP 0.9​

Use AMQP 0.9 to route measurement messages through RabbitMQ to a receiving application.

FieldExampleWhat to enter
Hostrabbitmq.noorderlicht.example.com:5672One hostname with a port.
Enable TLSOffThis example uses a connection without TLS on port 5672.
User / PasswordUser: noorderlicht-wattCredentials with publishing permissions for the exchange.
Exchangenoorderlicht.telemetryThe exchange receiving published messages.
Routing keymeasurements.zwolleThe key used by the exchange to route messages to the appropriate queue.
Max in flight64The maximum number of concurrent deliveries.

Ensure the exchange exists and the receiving queue is linked to it with a binding matching the routing key. The exchange receives messages; the application reads them from the queue. Without a matching binding, messages do not reach the intended queue.

AMQP exchange and routing key

AMQP 1.0​

Use AMQP 1.0 to send Zwolle's measurement messages to an AMQP 1.0 broker. Ensure the destination exists, your account can send messages to it and your receiving application can read them.

FieldExampleWhat to enter
Hostbroker.noorderlicht.example.comThe broker hostname, optionally with :port. Do not include a URL scheme, path or credentials.
Enable TLSOnThis example uses TLS. Without a specified port, 5671 is used.
Usernoorderlicht-wattThe required broker username, with permission to send to the target address.
PasswordYour broker passwordRequired when creating the pipeline. Enter it in the separate password field; do not use your Watt password.
Target addressqueue:/noorderlicht.telemetryThe destination address, in the format required by your broker.
Max in flight64A whole number from 1 through 512. The default is 64.

Choose the correct target address​

The target address is not the host. Host determines which broker the pipeline connects to; Target address determines where that broker receives the measurement messages. Use the address configured for your destination:

  • A broker may expect /noorderlicht.telemetry, for example.
  • A broker using queue/topic addresses may expect queue:/noorderlicht.telemetry or topic:/noorderlicht.telemetry.
  • For an Azure Service Bus destination, use the entity name, such as noorderlicht-telemetry.

These formats are not interchangeable. Copy the exact address from your receiving system's configuration. Do not enter an HTTP URL, exchange or routing key here. Enter a path only in Target address, not in Host.

Create the AMQP 1.0 pipeline​

  1. Choose Create pipeline, enter Noorderlicht AMQP 1.0 and select Zwolle.
  2. Choose the required delivery mode and select AMQP 1.0.
  3. Enter Host, User, Password and Target address. Select Enable TLS for this example. Leave Max in flight at 64 unless your receiver requires a lower limit.
  4. Choose Save pipeline.
  5. Check the protocol, connection and target address on the detail page. Then check that the receiving application receives Zwolle's measurement messages.

AMQP 1.0 host and target address

Change the password or protocol​

Stored passwords are not returned. When editing an existing AMQP 1.0 pipeline, leave Password empty to retain the stored password. Enter a new password to replace it. An empty field does not delete the stored password.

When switching from HTTP, Kafka, AMQP 0.9 or MQTT to AMQP 1.0, always enter the new broker details and password. The old connector's password is not reused as an AMQP 1.0 password. Check the target address and receiving permissions before saving. After saving, check the connection and received messages.

MQTT​

Use MQTT to publish measurement messages to a topic. The receiving application must subscribe to that topic.

FieldExampleWhat to enter
Hostmqtt.noorderlicht.example.com:1883One broker host with a port.
Enable TLSOffThis example uses a connection without TLS on port 1883.
Topicnoorderlicht/zwolle/measurementsThe topic for measurement messages.
User / PasswordEmpty in this exampleBroker credentials, when required.
Client IDnoorderlicht-watt-zwolleA unique client identity for this connection.
Max in flight64The maximum number of concurrent deliveries.

Do not use the same client ID for different simultaneous connections. Also check that the receiver subscribes to the correct topic and has permission to read it.

Last will​

A last will is a status message published by the broker after unexpected connection loss. Enable Last will enabled to configure it.

FieldExampleMeaning
Last will topicnoorderlicht/zwolle/statusThe topic receiving the status message.
Last will payloadofflineThe contents of the status message.
QoS10: at most one delivery. 1: at least one delivery; duplicates are possible. This applies to the last will.
Last will retainedEnabledThe broker stores the last status message for new subscribers.

Use a separate status topic so the receiver can distinguish status messages from measurements. A retained message may predate a new subscription session. A normal pause is not the same as unexpected connection loss.

MQTT topic and client ID

Last-will settings

Edit a pipeline​

  1. Open the pipeline and choose Edit pipeline.
  2. Change the name, delivery mode, installation subscriptions or destination. For a broker connector, you can also turn Enable TLS on or off. Check the broker port as well.
  3. Check all connector fields. Stored passwords are not displayed. For an existing AMQP 1.0 pipeline, leave the password field empty to retain the stored password; switching to AMQP 1.0 requires a password.
  4. Choose Save pipeline.
  5. Check the detail page and reception in your own system.

Clear optional fields only when you want to remove those settings, such as headers or username. When switching protocols, check the entire new destination.

For example, change the HTTP path to /measurements/v2. After saving, check that the endpoint contains the new value.

Created HTTP pipeline

Edit the HTTP endpoint

Pause, resume or delete a pipeline​

Choose Pause to pause the pipeline and Continue to resume it. Check its status afterwards. Data already at the receiver remains there.

Choose Delete pipeline to remove a pipeline. Check the name in the confirmation dialog before confirming. Deletion does not remove previously received data from your own system.

Destination and statistics

Paused pipeline

Check delivery and resolve problems​

The detail page distinguishes pipeline status from outgoing connection status. Check both, along with these statistics:

StatisticMeaningWhat to check
Messages per minuteSending rate since you opened the detail page.The first value may be 0. Wait for subsequent updates before drawing conclusions.
Delivery latency (p50)The median delivery latency: half of measured deliveries take no longer.A rising value indicates slower delivery; this is not total processing time in your application.
BacklogMessages awaiting delivery.A persistently growing backlog calls for checking connectivity and receiver capacity.

Statistics refresh approximately every two seconds. Missing statistics are not zero measurements.

ProblemCheck and next step
No messagesCheck that the pipeline is active and the intended installation is linked and enabled. Then check the outgoing connection.
Connection errorRead the error message and check host/URL, port, TLS setting and destination credentials. Copy the TLS setting and port from the broker configuration.
HTTP receives nothingCheck the URL path, method, headers and whether the API accepts the message format.
Kafka receives nothingCheck broker addresses, topic and publishing permissions. For a new topic, also check creation permissions.
AMQP 0.9 queue stays emptyCheck the exchange, routing key and receiving queue's binding.
AMQP 1.0 receives nothingCheck that the broker accepts AMQP 1.0, the host/port and TLS setting are correct and the target address exactly matches the broker configuration. Check send permissions and that the application reads the correct destination.
AMQP 1.0 cannot be savedEnter the host, username and target address. Also enter a password when creating the pipeline or switching protocols. Use a whole number from 1 through 512 for Max in flight.
MQTT subscriber receives nothingCheck broker, topic, read/write permissions and a unique client ID.
Backlog growsCheck errors, latency and receiver load. Change Max in flight only when the receiver can handle the extra concurrency.
Installations are missingEdit the existing pipeline, correct the selection and check the detail table after saving.

After every correction, check status, subsequent statistic updates and the messages actually received. This checks the entire connection, from installation subscriptions to your receiving application.