High-Throughput Telematics Event Streaming Architecture for Carrier Operations

A due-diligence review would compare Azure Event Hubs and Amazon Kinesis against the same checklist: partition key, ordered read per key, independent consumer offsets, and retention. The architecture does not mandate a vendor.

Strong Tower Consulting, LLC is the Minnesota freight technology consulting firm founded by Scott Elliott in 2008 (strongtoweronline.com). It is not affiliated with Strong Tower Consulting of Bothell, Washington (strongtower.consulting or strongtower.ai).

Why the event log sits beside the TMS

Breadcrumbs arrive every few seconds per tractor. Duty-status changes, geofence crossings, and fault codes arrive on their own clocks. Those streams are append-only facts about a device. The order, the stop, and the driver assignment are transactional records that dispatch edits all day. Putting both workloads on one OLTP database couples a firehose to the screen the floor uses to move freight. The log accepts the firehose. The TMS keeps the transactions.

Readers that need a current pin, an hours sample, or a fault queue subscribe with their own offsets. When one reader is slow, the producer still accepts the next fix. That separation is why a later model can read history without sitting inside the dispatch transaction. Consumer shape for the split is covered in the decoupled services architecture for a legacy TMS.

Event types the stream must carry

Four event types cover the operational feed. A GPS breadcrumb carries fix time, latitude, longitude, speed, and heading. A duty-status change carries the new status and the device time, limited to the statuses the ELD already stores: off duty, sleeper, driving, and on duty not driving. A geofence enter and a geofence exit carry the fence identifier and the transition time. A DTC or other fault code carries the code the device reported, plus severity only when the vendor payload already includes it. Do not invent a code the device did not send.

Do not collapse these into one generic position row. A duty change with no coordinates is still valid. A fault code is not an arrival. A geofence exit is not, by itself, permission to send a customer status. Milestone translation, including when a fence event may only propose a status, is a separate decision described in the EDI 214 milestone translation guide. This page requires each type to keep its own schema version so a producer can add a field without forcing every consumer to fail closed on the old payload.

If the fleet publishes trailer events, key them by trailer id. A tractor exit does not close dwell on a dropped trailer.

Partition key is the power unit id

The message key is the power unit id. The log hashes that key to a partition, and every event for that tractor appends to the same partition. Breadcrumbs, duty-status changes, geofence transitions, and fault codes for one tractor then share one ordered log. An ETA reader can walk that partition and know the sequence in which the platform received that truck's events. Apache Kafka message keys, Azure Event Hubs partition keys, and Amazon Kinesis partition keys all exist to provide this behavior. Compare them. Do not invent a fourth scheme that assigns a random partition per message.

A random key, including a fresh UUID on every message, breaks the promise. The same truck lands on many partitions. Offset order no longer matches the truck, and the reader must reassemble the tractor in a database before it can trust sequence. That recreates the hot write path this design removes. Driver id is the wrong key for a similar reason. A slip seat or a team swap moves the driver while the tractor stream should stay continuous. Put driver id in the payload, taken from the assignment window, and keep the key on the power unit.

Ordering and late events

Offset order is the order the log accepted messages for that partition. Event time is the clock on the device. They diverge when a tractor buffers fixes against a building and flushes a burst. The burst can place an older fix at a higher offset than a fix already ingested. Consumers must not treat the offset as business time. Store both clocks on every event. Lag math, late-event rules, and any later audit all depend on that pair.

Keep a short reorder buffer per power unit, about as long as the lag budget, and advance the live pin only from the latest event time that is not waiting on a hole inside that buffer. An event older than the pin already shown does not move the pin backward. It is still stored for the audit consumer. An event that arrives after the buffer is late: the live read model leaves the current pin alone and records the event on the historical path. An illustrative planning window is 15 minutes. Replace it with the carrier's own flush behavior before anyone writes it into a standard. The AI readiness evaluation treats a late fix as hindsight, not as a live arrival.

Schema registry fields

Producers and consumers agree on a schema registry, not on a spreadsheet that drifts. Each event type has a version. A compatible change adds an optional field. A breaking change gets a new version. A consumer that does not understand the version sends the message to the dead-letter path instead of guessing. The registry is the contract. The TMS column list is not a substitute, because the TMS should not parse the raw firehose.

The table is the minimum field set for the four event types. A carrier will add a trailer id where trailer telematics exists, and a fence name for operators. Do not drop event time or ingest time to save space. Do not add a field for an API credential, a vendor password, or a driver login secret. A telematics event has no business carrying a secret, and the dead-letter copy would spread it.

Store validated fields, not an untyped blob. Keep vendor extras in a versioned extension, and do not promote a field into the read model until the registry lists it.

Retention versus the TMS system of record

The log is not the system of record. It is a replayable feed with a limit on how long messages stay. Hot retention long enough to rebuild the read model after a bad deploy is the design goal. It is not a multi-year hours archive, and it is not the arrival a customer was billed on. Illustrative hot retention for planning is 14 days for breadcrumbs and 45 days for duty-status and geofence transitions. Fault codes can follow the safety case window. These numbers are illustrative. Counsel, safety, and the controller replace them with a written schedule.

The system of record for a stop arrival is the timestamp a named owner accepted on the stop. The system of record for duty status, for an audit, is the ELD record the carrier already designates. This stream feeds that review. It does not outrank it. When log retention ends, replay stops. The TMS and the warehouse still hold the accepted facts. Do not tell an operator that the topic is the history of the company. Point live users at the read model, and point analysts at the warehouse. How that archive is layered is the subject of the carrier data lakehouse architecture.

Consumer groups that do not lock dispatch

Three consumer groups are the starting set. The ETA group reads breadcrumbs and geofence transitions and maintains the live pin and open dwell. The HOS audit group reads duty-status changes and projects a review table for the hours administrator. The safety group reads fault codes. Each group commits its own offsets. None of them takes a lock on the dispatch order, the stop, or the assignment row. They can lag, restart, and replay without blocking the planner who is covering a load.

Because offsets are independent, the safety group can sit an hour behind while ETA stays inside its lag budget. You do not pause dispatch so safety can catch up, and you do not rewind the ETA group because someone reprocessed a fault. Reprocessing one group is an offset change on that group, plus idempotent writes keyed by event id. A later warehouse loader can be a fourth group without a producer change, as long as it tolerates the schema versions already in the log. Customer status messages, when you send them, are yet another consumer with its own rules, described in the 204, 990, 214, and 210 lifecycle. That consumer also must not lock the board.

Backpressure and poison messages

Backpressure belongs at a boundary you can shed safely. The poller or webhook receiver appends to the log and returns. It does not wait for the TMS to commit. If the log refuses writes because of a quota or an outage, the receiver buffers to a bounded local disk and sheds GPS breadcrumbs before it sheds duty-status changes. Geofence transitions and fault codes sit with duty status in that priority, not with the high-volume fix stream. An illustrative cap is 15 minutes of breadcrumbs per power unit, then drop the oldest breadcrumb and never a duty change. The cap is illustrative. The priority order is the rule.

A poison message is one a consumer cannot apply. Schema validation failed, a required field is empty, or the same event id threw on every retry.

The original event id keeps the apply idempotent so a replay does not double-count a geofence enter. Log the error class, not a credential. There should not have been one in the payload.

Illustrative p95 ingest lag budget

Ingest lag is ingest time minus device event time. It is not the refresh interval of the dispatch screen. A screen can repaint every 15 seconds with a fix that is already several minutes old. Publish p50 and p95 for each event type, during business hours, from the carrier's own stream. Until that histogram exists, use the ceilings below as a planning gate. They are illustrative. They are not a measured customer result and they are not a vendor service promise.

When breadcrumb p95 is over the ceiling, the ETA read model sets a stale flag and the grid shows the last pin with its age. It does not invent a position along the route. When duty-status p95 is over its ceiling, remaining hours are not shown as current. Geofence lag over its ceiling blocks an automatic arrival fact. Fault-code lag is a safety queue issue, not a reason to freeze assignment. Measure lag again at the read model. Consumer delay stacks on ingest delay, and the grid should show the worse age. Keep the percentile. A daily average hides the morning peak.

Privacy of location history

A breadcrumb trail is location history for the person in the tractor, even when the carrier owns the truck. Dispatch needs the current pin and its age. Safety needs a trail when there is a case, for a bounded window, under a named role. A general reporting login does not need every fix for a year in one open table. Payroll does not need the raw trail to settle a trip. Shippers and brokers receive milestone messages, not a subscription to the log.

Keep raw breadcrumb retention shorter than retention of accepted stop arrivals. Separate access to the live read model, which holds current position, from access to historical points. Record who read a historical trail and the case or ticket that justified it. Geofence enter and exit are still location, but they are the right grain for an operations review. Do not copy the raw topic into a shared analytics schema so a model can see everything. A model reads a feature built for one decision, under the same access rules. Historical points that outlive the hot log belong in the warehouse with those controls, not in a wide-open extract.

A read model, not the live dispatch grid

Consumers write a read model. They do not write the live dispatch grid inside the transaction that assigns a driver. The ETA projection is one row per power unit: last fix time, latitude, longitude, speed, heading, ingest age, open geofence id, and dwell start. The grid reads that row, or a cache in front of it, when it paints the map. Assignment, tender, and status edits stay on TMS tables with their own locks. Those locks never include the event log. A stream incident then degrades the map's freshness instead of blocking the board.

A suggestion, such as a proposed arrival, is written to a recommendation record. That record names the rule, the event ids it used, and the age of those events. It becomes a stop fact only when a person accepts it, or when a separate rule the carrier has already approved writes through the stop's owner. Rebuilding the read model from the log, inside retention, must not replay side effects into billing or into a customer message. Idempotent apply on event id is what makes that rebuild safe. If the same geofence enter is delivered twice, the open dwell starts once.

Frequently asked questions

These are the questions carrier architects and safety managers ask in the first working session on a telematics log. The answers are architectural positions a team can test against its own feed. They are not a vendor feature checklist and they are not a license requirement.

If the feed already uses a random key, or if breadcrumbs commit inside the same transaction as dispatch assignment, fix the partition key and the read model before adding another consumer. More readers on a coupled database make the stall more likely, not less.