Waterstream Taxi Demo
Waterstream Taxi Demo with Redpanda, Materialize, and Metabase
Waterstream is a fully-fledged MQTT broker leveraging any Kafka compatible event streaming platform as its only storage and distribution engine. In this demo, Waterstream uses Redpanda as the backend, while the dashboard is created with Metabase ingesting data processed by Materialize.
The NYC Taxi Demo
We simulate several taxis driving around New York, but you can see only few of them on the map. Each taxi belongs to a company and, as it moves, it sends MQTT messages to Waterstream. The data includes:
- the current taxi position,
- the next way-point,
- the number of passengers carried and the speed.
mqtt_messages. No additional service in the middle such as a connector is required to handle this.
Real-time data analysis with Materialize
Once it gets into Redpanda, Materialize can filter and aggregate streaming data in real-time using standard SQL. In the following code, we pick the top 10 taxi companies by passengers
CREATE SOURCE json_redpanda FROM KAFKA BROKER 'redpanda:9092' TOPIC 'mqtt_messages' KEY FORMAT TEXT VALUE FORMAT BYTES INCLUDE KEY; CREATE VIEW jsonified_bytes AS SELECT CAST(data AS JSONB) AS data FROM ( SELECT CONVERT_FROM(data, 'utf8') AS data FROM json_redpanda WHERE key LIKE 'waterstream-taxi-demo/passengers_update/%' ); CREATE VIEW passengers_by_company AS SELECT P.company, SUM(P.passengers) AS passengers_sum FROM ( SELECT data->>'company' AS company, CAST(COALESCE(data->'passengers', '0') AS INTEGER) AS passengers FROM jsonified_bytes ) AS P GROUP BY P.company; CREATE MATERIALIZED VIEW taxis AS SELECT T.company, T.passengers FROM ( SELECT company AS company, passengers_sum AS passengers FROM passengers_by_company ) AS T ORDER BY company ASC LIMIT 10;
Presenting data with Metabase
Finally, the data is presented with a simple Metabase graph using
metabase-materialize-driver to connect to Materialize.
Waterstream allows bidirectional communication, even with the browser
Waterstream is a fully operational MQTT broker compatible with MQTT version 5 with WebSockets support. This page is an MQTT client receiving messages from Waterstream. It updates the markers on the map as long as new positions are written into Redpanda and translated to MQTT messages by Waterstream.
Waterstream exposes metrics with Prometheus
It's easy to collect and show data coming from Waterstream with Grafana. The following graph shows the current message rate and how many messages have been received so far by the Waterstream broker.
Want to know more about Waterstream? Check out waterstream.io.