Firehose: configure streams and connectors
For the message format, installation identity and measurement fields, see Payload and schema within this Firehose section.
Firehose sends measurements from your installations to your own receiving system. Open Firehose under Developer when your account has access to this feature.
A pipeline defines which installations supply data, how it is delivered and where it goes. Each pipeline has one destination. Create two pipelines to send data to two different systems.
The Noorderlicht example configuration includes a monitoring stream for Zwolle and a paused archive stream. The instructions below explain how to configure pipelines for HTTP, Kafka, AMQP 0.9, AMQP 1.0 and MQTT. The example addresses are fictional: replace them with your receiving system's details.

Choose a connector
| Connector | Use | Details you need |
|---|---|---|
| HTTP | Send measurement messages to an API. | URL, HTTP method, headers and any credentials. |
| Kafka | Publish measurement messages to a topic. | Broker addresses, topic and publishing permissions. |
| AMQP 0.9 | Route measurement messages through an exchange to queues. | Host, exchange, routing key and a matching queue binding. |
| AMQP 1.0 | Send measurement messages to an AMQP 1.0 destination. | Broker host, username, password and target address. |
| MQTT | Publish measurement messages for an application to subscribe to. | Broker host, topic and any client ID and credentials. |
Prepare the receiving system before creating the pipeline. Use that system's connection details, not your Watt login credentials.
AMQP 0.9 is selected by default for a new pipeline. Choose AMQP 1.0 when your receiving system uses that protocol. These are separate protocols: an AMQP 0.9 exchange and routing key are not substitutes for an AMQP 1.0 target address.
Create a pipeline
- Open Developer → Firehose and choose Create pipeline.
- Enter a recognisable name, such as
Noorderlicht HTTP. - Choose a delivery mode: Live or Durable.
- Select the installations whose measurements you want to send.
- Choose the connector and complete its destination fields.
- Choose Save pipeline.
- Check the linked installations, destination and status on the detail page. Then check that your receiving system receives messages.
Use search to find installations. Select visible selects the installations matching your current search. Clear removes the selection. Select at least one installation; the maximum is 10,000. The example selection is Noorderlicht Distributiecentrum — Zwolle.
If a message says that some installations were not linked, open the existing pipeline, choose Edit pipeline and correct the selection. Check the installation table afterwards. Do not create a second pipeline to repair incomplete subscriptions.
Choose the delivery mode
Live delivers messages while the pipeline is running. Durable stores messages until they are delivered. Choose Durable to accommodate temporary interruptions at the receiver. The Noorderlicht examples use Durable.
To download earlier measurements, open Details and select the required period.
Set Max in flight
Max in flight controls how many deliveries can run concurrently. The default is 64; use a value from 1 through 512.
Start with the default unless your receiving system requires a lower limit. More concurrent deliveries increase receiver load. When changing this setting, also check backlog and delivery latency. The HTTP example value is 16.
Configure TLS and the broker port
For Kafka, AMQP 0.9, AMQP 1.0 and MQTT, select Enable TLS for an encrypted connection to the broker. TLS is off by default for a new pipeline. Use the setting required by your broker; configure the username and password separately.
Enter a hostname in Host, optionally followed by :port. Do not include a URL scheme such as amqps:// or mqtts://, a path or credentials. For Kafka, enter broker addresses with their ports in Brokers.
When no port is specified, these default ports apply:
| Connector | TLS off | TLS on |
|---|---|---|
| AMQP 0.9 and AMQP 1.0 | 5672 | 5671 |
| MQTT | 1883 | 8883 |
An explicitly entered port remains unchanged when you turn TLS on or off. Check the port as well: a broker may use a different port for TLS. Copy the correct combination from the broker configuration. For Kafka, use each broker's port; changing TLS does not change those addresses.
HTTP has no separate TLS setting. Use a URL starting with https:// for an encrypted HTTP connection.
HTTP
Use HTTP to send measurement messages to Noorderlicht's monitoring API.
| Field | Example | What to enter |
|---|---|---|
| URL | https://noorderlicht.example.com/measurements | The full receiving address, including the path. |
| Method | POST | The method expected by the API: POST, PUT or PATCH. |
| User / Password | Empty in this example | Credentials for the receiving API, when required. |
| Max in flight | 16 | The maximum number of concurrent deliveries. |
| Headers | See below | One header per line, in Name: value format. |
X-Noorderlicht-Source: watt
Content-Type: application/json
Enter each header once, with a colon between the name and value. If a name is repeated, the last value applies. A line without a valid name and colon is not sent.
Use the headers required by your API. A Content-Type header describes the message format; it does not change the measurement message's contents. Check that the API can process those contents. Keep secret header values out of screenshots and shared documents.

Kafka
Use Kafka to publish measurement messages to noorderlicht.measurements.
| Field | Example | Setting |
|---|---|---|
| Brokers | The two addresses below | One broker address per line or comma-separated. |
| Enable TLS | Off | Select when the brokers require TLS. Check their ports as well. |
| Topic | noorderlicht.measurements | The topic receiving published messages. |
| User / Password | User: noorderlicht-watt | Credentials with publishing permissions for the topic. |
| Max in flight | 64 | The maximum number of concurrent deliveries. |
| Batching count | 20 | Batch count threshold: 1–128. With 0, you set no count threshold. |
| Batch period (ms) | 1000 | Batch time threshold: 1–5000 milliseconds. With 0, you set no time threshold. |
| Custom topic creation | Enabled | Use custom creation settings for a new topic. |
| Topic partitions | 3 | Number of partitions at creation: 1–64. With -1, you specify no custom count. |
| Replication factor | 3 | Number of replicas at creation: 1–7. With -1, you specify no custom count. |
kafka-1.noorderlicht.example.com:9092
kafka-2.noorderlicht.example.com:9092
Batching
Batching groups messages. With the example settings, a batch is sent at the count threshold of 20 or the time threshold of 1000 ms. The time threshold prevents a small number of messages from waiting for a full batch. Larger batches can make processing more efficient but add delay.
Topic creation
Partitions divide the topic; replicas determine how many copies the cluster stores. Use counts appropriate for your cluster's capacity and configuration. The account needs creation permissions to create a topic. These settings apply to a new topic, not to changing an existing one.


AMQP 0.9
Use AMQP 0.9 to route measurement messages through RabbitMQ to a receiving application.
| Field | Example | What to enter |
|---|---|---|
| Host | rabbitmq.noorderlicht.example.com:5672 | One hostname with a port. |
| Enable TLS | Off | This example uses a connection without TLS on port 5672. |
| User / Password | User: noorderlicht-watt | Credentials with publishing permissions for the exchange. |
| Exchange | noorderlicht.telemetry | The exchange receiving published messages. |
| Routing key | measurements.zwolle | The key used by the exchange to route messages to the appropriate queue. |
| Max in flight | 64 | The maximum number of concurrent deliveries. |
Ensure the exchange exists and the receiving queue is linked to it with a binding matching the routing key. The exchange receives messages; the application reads them from the queue. Without a matching binding, messages do not reach the intended queue.

AMQP 1.0
Use AMQP 1.0 to send Zwolle's measurement messages to an AMQP 1.0 broker. Ensure the destination exists, your account can send messages to it and your receiving application can read them.
| Field | Example | What to enter |
|---|---|---|
| Host | broker.noorderlicht.example.com | The broker hostname, optionally with :port. Do not include a URL scheme, path or credentials. |
| Enable TLS | On | This example uses TLS. Without a specified port, 5671 is used. |
| User | noorderlicht-watt | The required broker username, with permission to send to the target address. |
| Password | Your broker password | Required when creating the pipeline. Enter it in the separate password field; do not use your Watt password. |
| Target address | queue:/noorderlicht.telemetry | The destination address, in the format required by your broker. |
| Max in flight | 64 | A whole number from 1 through 512. The default is 64. |
Choose the correct target address
The target address is not the host. Host determines which broker the pipeline connects to; Target address determines where that broker receives the measurement messages. Use the address configured for your destination:
- A broker may expect
/noorderlicht.telemetry, for example. - A broker using queue/topic addresses may expect
queue:/noorderlicht.telemetryortopic:/noorderlicht.telemetry. - For an Azure Service Bus destination, use the entity name, such as
noorderlicht-telemetry.
These formats are not interchangeable. Copy the exact address from your receiving system's configuration. Do not enter an HTTP URL, exchange or routing key here. Enter a path only in Target address, not in Host.
Create the AMQP 1.0 pipeline
- Choose Create pipeline, enter
Noorderlicht AMQP 1.0and select Zwolle. - Choose the required delivery mode and select AMQP 1.0.
- Enter Host, User, Password and Target address. Select Enable TLS for this example. Leave Max in flight at
64unless your receiver requires a lower limit. - Choose Save pipeline.
- Check the protocol, connection and target address on the detail page. Then check that the receiving application receives Zwolle's measurement messages.

Change the password or protocol
Stored passwords are not returned. When editing an existing AMQP 1.0 pipeline, leave Password empty to retain the stored password. Enter a new password to replace it. An empty field does not delete the stored password.
When switching from HTTP, Kafka, AMQP 0.9 or MQTT to AMQP 1.0, always enter the new broker details and password. The old connector's password is not reused as an AMQP 1.0 password. Check the target address and receiving permissions before saving. After saving, check the connection and received messages.
MQTT
Use MQTT to publish measurement messages to a topic. The receiving application must subscribe to that topic.
| Field | Example | What to enter |
|---|---|---|
| Host | mqtt.noorderlicht.example.com:1883 | One broker host with a port. |
| Enable TLS | Off | This example uses a connection without TLS on port 1883. |
| Topic | noorderlicht/zwolle/measurements | The topic for measurement messages. |
| User / Password | Empty in this example | Broker credentials, when required. |
| Client ID | noorderlicht-watt-zwolle | A unique client identity for this connection. |
| Max in flight | 64 | The maximum number of concurrent deliveries. |
Do not use the same client ID for different simultaneous connections. Also check that the receiver subscribes to the correct topic and has permission to read it.
Last will
A last will is a status message published by the broker after unexpected connection loss. Enable Last will enabled to configure it.
| Field | Example | Meaning |
|---|---|---|
| Last will topic | noorderlicht/zwolle/status | The topic receiving the status message. |
| Last will payload | offline | The contents of the status message. |
| QoS | 1 | 0: at most one delivery. 1: at least one delivery; duplicates are possible. This applies to the last will. |
| Last will retained | Enabled | The broker stores the last status message for new subscribers. |
Use a separate status topic so the receiver can distinguish status messages from measurements. A retained message may predate a new subscription session. A normal pause is not the same as unexpected connection loss.


Edit a pipeline
- Open the pipeline and choose Edit pipeline.
- Change the name, delivery mode, installation subscriptions or destination. For a broker connector, you can also turn Enable TLS on or off. Check the broker port as well.
- Check all connector fields. Stored passwords are not displayed. For an existing AMQP 1.0 pipeline, leave the password field empty to retain the stored password; switching to AMQP 1.0 requires a password.
- Choose Save pipeline.
- Check the detail page and reception in your own system.
Clear optional fields only when you want to remove those settings, such as headers or username. When switching protocols, check the entire new destination.
For example, change the HTTP path to /measurements/v2. After saving, check that the endpoint contains the new value.


Pause, resume or delete a pipeline
Choose Pause to pause the pipeline and Continue to resume it. Check its status afterwards. Data already at the receiver remains there.
Choose Delete pipeline to remove a pipeline. Check the name in the confirmation dialog before confirming. Deletion does not remove previously received data from your own system.


Check delivery and resolve problems
The detail page distinguishes pipeline status from outgoing connection status. Check both, along with these statistics:
| Statistic | Meaning | What to check |
|---|---|---|
| Messages per minute | Sending rate since you opened the detail page. | The first value may be 0. Wait for subsequent updates before drawing conclusions. |
| Delivery latency (p50) | The median delivery latency: half of measured deliveries take no longer. | A rising value indicates slower delivery; this is not total processing time in your application. |
| Backlog | Messages awaiting delivery. | A persistently growing backlog calls for checking connectivity and receiver capacity. |
Statistics refresh approximately every two seconds. Missing statistics are not zero measurements.
| Problem | Check and next step |
|---|---|
| No messages | Check that the pipeline is active and the intended installation is linked and enabled. Then check the outgoing connection. |
| Connection error | Read the error message and check host/URL, port, TLS setting and destination credentials. Copy the TLS setting and port from the broker configuration. |
| HTTP receives nothing | Check the URL path, method, headers and whether the API accepts the message format. |
| Kafka receives nothing | Check broker addresses, topic and publishing permissions. For a new topic, also check creation permissions. |
| AMQP 0.9 queue stays empty | Check the exchange, routing key and receiving queue's binding. |
| AMQP 1.0 receives nothing | Check that the broker accepts AMQP 1.0, the host/port and TLS setting are correct and the target address exactly matches the broker configuration. Check send permissions and that the application reads the correct destination. |
| AMQP 1.0 cannot be saved | Enter the host, username and target address. Also enter a password when creating the pipeline or switching protocols. Use a whole number from 1 through 512 for Max in flight. |
| MQTT subscriber receives nothing | Check broker, topic, read/write permissions and a unique client ID. |
| Backlog grows | Check errors, latency and receiver load. Change Max in flight only when the receiver can handle the extra concurrency. |
| Installations are missing | Edit the existing pipeline, correct the selection and check the detail table after saving. |
After every correction, check status, subsequent statistic updates and the messages actually received. This checks the entire connection, from installation subscriptions to your receiving application.