Time-Sensitive Networking: Reliable Communication for Industrial IoT

Today’s industrial manufacturing cell can include several robot arms, some cameras, position sensors, and some controllers connected together in a network.  The cameras and sensors send data to the controllers.  The controllers send commands to the robot arms.  What happens when a command from the controller arrives at the robot arm a tenth of a second late?  Or what happens if sensor data never actually reaches the controller?  The robot arm can overshoot its coordinates.  The controller can make the wrong decisions based on omitted data.

The lane keeping / highway driving assistance system in a modern automobile might include GPS sensors, cameras with image recognition systems, radar and lidar sensors, a central controller, which controls the steering, acceleration and braking systems.  The central controller runs sensor fusion computations based upon the various sensor inputs.  It calculates to a high accuracy the current vehicle position and its relationship to other surrounding objects in order to provide the right vehicle commands.  What happens when the various sensor data corresponding to the same data sampling time arrive at different times to the central controller?  This might lead to mistakes in computation, and possibly to errors in steering, braking or acceleration commands which could result in failures and collisions.

In today’s highly competitive environment, business inefficiencies can cause you to go out of business.  Functional safety issues brought on my unreliable communication can even harm human lives, as in the case of a robot working with a human counterpart.  These issues can cause environmental accidents, as in the case of an oil rig that malfunctions.

Industrial and automotive networks like CAN and Profibus have been around for decades to address the need for robust and reliable networked devices.  But these isolated networks require special cabling and wiring that requires maintenance, and can add weight to a vehicle, which are not viable in today’s applications.

A reliable network for industrial and automotive applications requires 100% of commands and data to arrive on time.  Each message containing command and data must arrive at its destination within milliseconds so that the entire chain of events from sensor input, to sensor data processing, to sending of data to controller, to controller decision processing, to sending of command, to receipt of command, to processing of command, to execution of action, all can take place within a small fraction of a second.  Interference from the multitude of sensors and controllers in the system, or from external messages can be deadly.

Time-sensitive networking (TSN) creates a reliable network suitable for critical industrial and automotive systems using existing ethernet hardware infrastructure.  With TSN, the factory floor can use existing IT hardware to connect its devices.  In automotive applications, the vehicle can replace all its heavy wiring with lighter thinner ethernet cables and put all its devices, both critical devices like the emergency braking system, and its non-critical devices, like diagnostic systems, on the same network.

Factory floor system example using TSN IEEE802.1 Qbv for scheduled traffic

TSN includes several specifications.  For the guarantee that 100% of network packets arrive on time, the IEEE 802.1Qbv specification describes a scheduled traffic system.  This article illustrates the use of scheduled traffic using the VxWorks RTOS as an example.

Consider a factory floor system as shown in Figure 1.  This manufacturing cell has 2 industrial controllers, 3 robot arms, 4 conveyor belt controls, 15 sensor devices.    A main control computer for user interaction with the industrial controllers is connected.  There’s a secondary kiosk for operator interaction.  There’s also a gateway connected so that remote users can access the main control computer.  Interspersed around the system are 4 switches to connect all the devices together.

Figure 1.  Factory floor network topology

Let’s look at the critical flow of messages.  The diagram in Figure 2 below shows only the messages that must arrive on time.  Other non-critical flows, like operator configuration of the controller, or factory statistics sent out to the gateway, do not have sub-millisecond deadlines and do not participate in TSN scheduled traffic.

Figure 2.  Factory floor critical flows.

Simplified Topology

The topology and the identification of critical flows are the first steps in TSN implementation.  For demonstration purposes, this article looks at a simplified topology.  The demonstration consists of 2 switches, an industrial controller, and a robot arm.  For testing purposes, the demonstration also includes 2 non-critical devices.

Fig. 3.  Simplified factory floor network topology

The actual hardware used to represent this simplified factory floor are two Cisco IE4000 switches with TSN capabilities, and two Intel Core i7 boards equipped with Intel i210 network controller to represent the industrial controller and the robot arm.  The non-critical devices are 2 Linux laptops.

Figure 4 below shows the simplified topology. The red arrow shows the defined single critical flow, representing critical commands from the industrial controller to the robot arm.

Fig. 4.  Critical flow in simplified factory floor

From the topology, the single line between the two switches creates a bottleneck for traffic.

For this demonstration, the board representing the industrial controller is called the VxWorks talker, and the robot arm is called the VxWorks listener.  The VxWorks talker sends a 1kB packet to the VxWorks listener 100 times a second.  First, the packets are sent as regular ethernet packets without TSN.

Fig. 5.  VxWorks listener console output

interrupt: 500000 bytes in 5 seconds.  Out of sequence:  0

interrupt: 500000 bytes in 5 seconds.  Out of sequence:  0

interrupt: 500000 bytes in 5 seconds.  Out of sequence:  0

interrupt: 500000 bytes in 5 seconds.  Out of sequence:  0

The VxWorks listener console messages in Figure 5 above prints out the number of bytes received at 5 second intervals.   Since the VxWorks talker sends out 500,000 bytes every 5 seconds, we can see that the VxWorks listener receives 100% of its packets.  There is nothing wrong with a regular non-TSN network in this situation.

Now the demonstration uses the Linux laptops to flood network traffic into the system.  The iperf3 application for network bandwidth measurement blasts traffic into the network.

Fig. 6.  VxWorks listener console output with for non-TSN traffic with heavy interfering traffic.

interrupt: 456000 bytes in 5 seconds.  Out of sequence:  40

interrupt: 474000 bytes in 5 seconds.  Out of sequence:  25

interrupt: 466000 bytes in 5 seconds.  Out of sequence:  33

interrupt: 459000 bytes in 5 seconds.  Out of sequence:  36

Looking at the VxWorks listener in Figure 6, some messages out of the 500000 bytes were never received.  When not using TSN flows, heavy traffic interrupts the messages between the VxWorks talker and the VxWorks listener.  About 6-8% of the messages are dropped.  In automated manufacturing applications and vehicle driving assistance applications, the dropped traffic is not acceptable.

Adding Scheduled Traffic

To prevent critical message flow interruption, scheduled traffic is used.  The two Cisco IE4000 switches and the VxWorks devices with Intel I210 network controllers are fully capable of operating on scheduled traffic.

First, all the TSN endpoints (which are the VxWorks talker and the VxWorks listener) and TSN switches (the two Cisco switches) must synchronize their clocks to within 1 microsecond.  Precision Time Protocol, another TSN specification, on the endpoints and the switches effect this synchronization.  A previous blog entry provides details and demonstration of how this is achieved.

Next, the system integrator creates a system-wide schedule that must be shared among all the TSN endpoints and TSN switches.  The schedule depends heavily on the topology and the critical flows.

The simplified demonstration system described above has one defined flow.  The flow originates in VxWorks talker, going to switch 1, to switch 2, to VxWorks listener.  That single flow is the critical flow.  In this demonstration, it is called Critical-Flow-1.

A possible traffic schedule is laid out in Figure 7.    Every millisecond, VxWorks talker can send out Critical-Flow-1 packets between 69 microseconds and 89 microseconds.  Switch 1 will only accept packets from VxWorks talker that belongs to Critical-Flow-1.  Switch 1 will send out Critical-Flow-1 packets to Switch 2 between 146 microseconds and 166 microseconds.  Switch 2 will only accept Critical-Flow-1 packets from Switch 1 between 146 microseconds and 166 microseconds.  Switch 2 will only send out Critical-Flow-1 packets to VxWorks talker between 425 microseconds and 445 microseconds.

Fig. 7  Possible IEEE 802.1Qbv flow schedule

With scheduled traffic for the critical flow, we run the demonstration again using the iperf3 Linux application to flood the network.

Fig. 8.  VxWorks listener console output with for TSN traffic with heavy interfering traffic.

interrupt: 500000 bytes in 5 seconds.  Out of sequence:  0

interrupt: 500000 bytes in 5 seconds.  Out of sequence:  0

interrupt: 500000 bytes in 5 seconds.  Out of sequence:  0

interrupt: 500000 bytes in 5 seconds.  Out of sequence:  0

With TSN, all packets arrive at the VxWorks listener on time, even with interfering traffic flooding.  No packets are dropped.

This simplified demonstration can expand to an actual system, with 10 or 20 or 30 flows defined, with additional switches in complex ring topologies, and with many more talker and listener endpoints hanging off the switches.

TSN Requirements

Scheduled traffic depends on a shared schedule among all critical nodes of the system.  In turn, the ability to follow the schedule depends sub-microsecond level clock synchronization and the ability to send on a hardware timed queue.  Only network controllers with hardware clock synchronization and timed queue capabilities can work with TSN.  The Intel I210 network controller used in this demonstration has hardware timestamping capabilities for clock synchronization.  It also has QAV registers for timed send capabilities.

For endpoint software, TSN talkers must know how to take IEEE 802.1Qbv flow schedules and set up the network controller to send according to schedule.  VxWorks provides such capability.

All switches that direct critical flows must be TSN-enabled, so that the switch can follow the IEEE802.1Qbv flow schedule and queue the TSN traffic at highest priority.  The Cisco IE4000 switch used in this demo provides this capability.

Schedule Creation

The schedule in the demonstration was created using the Centralized Network Configuration (CNC) software tool provided by Cisco for their IE4000 switch.  The tool takes as input the network topology, the flow requirements (talker and listener endpoints, time constraints, size of packets, etc.).  A scheduler computes the schedule, and CNC distributes the schedule to switches.

The specifications under TSN have other mechanisms for automating inputs and outputs to the schedule, including automatic topology discovery, automatic flow requirement input, and automatic distribution of traffic schedule to switches and endpoints.

Read more about IEEE 802.1Qbv.  Also, to view information about VxWorks, visit us here.  Information on Cisco’s TSN switches and other TSN capabilities can be found here.

Wind River and Intel participate in the Industrial Internet Consortium (IIC) TSN Testbed to ensure TSN interoperability with multiple vendors. Learn more here.