In a factory the network goes down often. Once it stayed down for several days: road works had damaged the fibre in the street. The plant was cut off from the cloud ERP that sends the orders.
Production did not stop. The buffer held without problems. And for long outages a manual mode had already been planned: orders and other information that could not arrive from the ERP were entered by hand, locally. When the fibre came back, the data flowed back to the cloud.
It wasn't luck. It had been decided in advance.
When I design industrial middleware I start from one assumption: the network will go down, and sooner or later an external service will not answer. It is not an exception to handle, it is a normal state of the system. It is designed like the others, by deciding three things: what keeps going, what stops, and how data is put back in order when the network returns.
01What keeps going, what stops
Keeps going
Locally, without a network
- Data acquisition from the PLCs.
- The logic that needs no external information.
- Event recording, in the buffer.
- The manual work planned for long outages.
Stops, and says so
Whatever depends on others
- A new order from the ERP.
- A rescheduling.
- Confirmations only an external system can give.
If I am waiting for a new order or a rescheduling from the central system, I cannot make them up. In that case the system must say the operation is not available, or hand the decision to a person, as in the days without fibre. The worst thing is a system that looks like it works when it can't.
02The buffer doesn't last forever
I size the buffer on three things: how long the system must be able to work without a connection, how much data it generates in that time, and how much space there really is on the edge.
Then I decide what happens when it fills up, because I cannot write until the disk runs out. Depending on the data I can lower the acquisition rate, drop the least important samples, or stop accepting operations that require reliable recording.
A buffer that promises to last forever is a false promise.
03Not all data is worth the same
Telemetry
For a temperature read every second, the latest value may be enough, or a lower rate during a long outage.
Parts, scrap, events
Persistent, complete queue: this is the data that traceability and counting depend on.
Critical events
An explicit retention policy, and a signal when I can no longer guarantee it.
This is not a technical choice. It is a decision about process requirements: first you establish what can be lost, aggregated or recovered and what must be kept. Then you write the code.
04When the network comes back
Resending the accumulated messages is not enough. Every event needs a unique identifier and the moment it was created.
Duplicates. With MQTT at QoS 1, delivery is guaranteed «at least once» (OASIS, MQTT 5.0): the same message can arrive twice. The receiver must recognise events it has already processed, using the identifier and a persistent record of what it has handled. Otherwise one part produced gets counted twice.
Time. I distinguish three moments, and with all three you can reconstruct what happened during the outage.
Three timestamps for every event
- when the machine produced it
- when the middleware put it in the buffer
- when the central system processed it
Order. Arrival order is not always the real order. For a production event I must be able to rebuild the sequence. For an updated measurement, what often matters more is knowing which is the most recent reliable value.
05Who wins, local or cloud?
There is no single rule. It depends on the data and on who owns it. A machine's state is decided locally; a production order may belong to the central system.
That is why I separate data that can be updated locally from data that needs external authorisation or confirmation. When the network returns, the system reconciles according to these rules. A piece of information does not win just because it arrived last.
06What the operator sees
The operator must never face a screen that shows stale data as if it were current. The interface must show the connection status and tell fresh data apart from data that has stopped arriving. If an operation is not available, it must say so.
Whether work can continue by hand depends on the process and the plant's procedures. Sometimes production continues and is recorded locally, to be recovered later. Other times certain operations must be suspended, because information or authorisations are missing.
07The most common mistake
A system that looks offline-first but still depends on external services for its core operations. Acquisition from the PLCs keeps running, but at the first call to an unreachable service the worker blocks. Or one error also stops the messages that never needed that service.
Before go-live
The four tests I run in commissioning
- The backend or the broker not answering.
- The network going down and coming back.
- A long outage, until the buffer fills up.
- An edge restart with data still to send.
The test is not «it reconnects». It is «after recovery, the data and the application state are still consistent, with no duplicates». Planning the reconnection is the easy part; the real work is designing how the whole system behaves while the network is gone.
