Why DevOps is relevant when there is no Ops….Or what is Ops for embedded systems?

“What does DevOps mean for embedded systems and should I care?”  Well, you should – at least if you’re working according to any sort of agile principle.

The merge of Development and Operations, DevOps, is hot in the IT world; still so hot that there is no universally accepted definition. However when you bring up the topic of DevOps for software that is embedded in a system the response is more of, “Really? How could that possibly apply?” After all, for many embedded systems there is no “ops” as it’s known in the IT world. We build a device, or a system that someone else “operates.” Those of us who build the system are rarely involved in the actual operations of the system.

The thing is, DevOps is about organization (amongst other things), and agile is about methodology. And what methodologies you use will eventually affect your organization. Therefore, I see Agile and DevOps as belonging on the same evolutionary path. What we see happening with cross disciplined engineering teams, is proof of this.

Teams that have been working by any sort of agile principle for a while tend to re-organize themselves into cross-discipline, or cross-functional, teams [see this blog post by Nicholas Muldoon. In order to keep a high degree of velocity and keep lead times to a minimum, you want to cut down on all sorts of handovers and synchronization across teams. Therefore, designers, testers, developers and integrators merge into a cross-functional team where they quickly can turn around issues, have fast feedback loops and keep a high degree of autonomy. A cross-functional team has end-to-end responsibility for a certain function or subsystem.  Instead of having to handover a task or a responsibility to another team (maybe even in another organization), you now keep the discussion and work within your own team. You work faster if you’re close to your team mates and can quickly resolve any upcoming issues and discuss solutions.

The picture below illustrates the difference between a functional and cross functional organization. This is very similar to the picture in Nicholas Muldoon’s blog post (referenced above), but modified to be a little more focused on an engineering organization only.

With end-to-end responsibility also comes responsibility for delivery. If your Continuous Integration system has evolved into a Continuous Delivery system [see my earlier post about Continuous Delivery and Deployment], where your code sprints reach production quality when they have passed through the gauntlet of all levels of integration testing, your cross-functional teams could also take responsibility for the delivery of the code sprints. Now, we have a greater end-to-end responsibility, all the way from design through to delivery.

With that, we are moving closer to the “operations” side. What is “operations” for an embedded system? There are several layers of operations here. We can view it as the operation of the actual end-device being used (the router, the gateway, the flight engine, the blood pump, the multifunction printer…), in this case the operations is handled by the user of the system (for example a nurse operates a dialysis machine). We can also view it as operations by the manufacturer of, for example, a dialysis machine. They build the machine and they produce the hardware and the software for it. Operations here could mean to deploy the embedded software into the product. Today this might mean flashing software onto hardware, in the future it might mean updating the software over the Internet. And, one more level of operations could relate to those companies being suppliers of software and hardware to the companies building the machines. For example somebody who builds a solid-state-disk that is included in a high-end server, or a company building networking software being used in routers. With the exception of the first example, the user of the device (the nurse), my point is that   all companies have some sort of operations – we deliver systems, hardware and/or software, to our customers, and as such we have our own operations that we manage. We have a support organization, we have a delivery organization, maybe a service organization, etc. All of this is operations. While a cross-functional team sometimes only refers to a team within the engineering organization, what stops the organizational merge or change from being limited within engineering only? For example, wouldn’t support be more efficient if it was closer to an end-to-end responsibility flow, and maybe even integrated with the development team?

When you include the responsibility for delivery in your cross functional team organization, you will need to take the step towards the organizational changes that DevOps suggests. Somewhere along the way with continuous delivery and deployment, methodology changes will affect how you structure your organization.

Agile practices have managed to increase productivity in the engineering organization to the degree that engineering is no longer the bottleneck for go-to-market and delivery. But bottlenecks still exist, they have just moved elsewhere in the organization. As Gartner’s report “Principles and Practices of DevOps” from March 12, 2015 says

“Agile (and lean) approaches have significantly improved the development process within many enterprises. However, without applying similar concepts to the downstream groups (such as operations), the bottleneck has merely been shifted to the right.”

My point is that, in the IT and web world, “ops” might refer to IT systems only, and as such it makes sense for a company like NetFlix or Facebook to talk about DevOps. But, every company has an operational side, even if you deliver complex electronics systems where software is embedded. We still deliver products to our customers, and that is what DevOps aims at accelerating.

Since I work with Simics, and simulation, I’ve been thinking about how simulation can help in a DevOps journey. We already know well how Simics helps with adopting agile practices [see our agile video]. The DevOps term stresses collaboration, communication and automation. Those are key benefits that you achieve by using a simulation technology like Simics. Also, I think a key observation is what Gartner points out in the earlier mentioned DevOps report:

“the ultimate goal of many DevOps efforts is to enable continuous delivery (into staging), if not continuous deployment (into production), of every source code repository change”.

A staging area is where you test and review your software before you go live to deployment, it resembles the concept of a staging site [see http://en.wikipedia.org/wiki/Staging_site]. For embedded systems, the deployment of software is naturally onto the physical target. But what about the previous steps? In a well-functioning continuous integration (CI) system, you have many levels of CI-loops, and if every one of those loops would only be based on physical target hardware, you would need a lot of hardware to scale out your testing [see Jakob Engblom’s blog post http://blogs.windriver.com/wind_river_blog/2014/09/continuous-integration-with-simics.html]. Basically, each CI loop can be seen as a gradually built-out system, where you test your software’s quality. And here is where simulation is a great fit in order to scale out testing, and automate as much as possible. Before your code is ready for deployment onto physical hardware, you can make it ready for delivery on a staging area which is a simulated system. See the picture below.

So, what is “ops” for embedded systems? And can we claim that DevOps is relevant? Every company has an operational side, and if you’re in the business of building and delivering complex electronics systems with embedded software, you will see market changes that require faster delivery of systems and software. DevOps is about accelerating precisely this. So yes, DevOps is highly relevant for embedded systems. And we have just seen the start of it, by agile practices being adopted. We know there are many challenges ahead, because after all, what we build, is rocket science 🙂  But I think that with simulation, we can get a little closer to the vision of DevOps for software that is embedded in complex electronics systems.