Voices in DevOps – Episode 13: A Conversation with Gareth Rushgrove of Snyk

Guest

Gareth Rushgrove is a Director of Product at Snyk, working remotely from Cambridge, UK, helping to build interesting tools for people to better secure infrastructure and applications. He has previously worked for the UK Government Digital Service focused on infrastructure, operations and information security, as well as at Puppet and Docker. When not working he can be found curating the Devops Weekly newsletter, hiking or reading a good book.

Transcript

Jon Collins: Hello and welcome to this episode of Voices in DevOps, where I’m here to speak to Gareth Rushgrove who’s got a diverse set of experiences in the bag. I’ll leave you to talk about all of that, Gareth. Essentially, as you know, this series is about how we’re looking to help enterprise organizations scale up their DevOps efforts. Why don’t we just start? Just tell us about yourself, Gareth and what you’ve seen in your career to date. How did you end up being on a Voices in DevOps podcast with me? I guess that’s the question of the moment.

Gareth Rushgrove: Yeah. First off, thanks for having us, Jon. Looking forward to chatting. Yeah, I’m Gareth Rushgrove. I’m currently the Director of Product Management at Snyk. That’s a small security software vendor. Prior to that, I was basically working in Product Management at Docker. Before then, mainly on the engineering side in other places. I was also a Technical Architect at the UK Government for a number of years. It was something like on the other side of the fence, if you like.

Can you say which bit?

Yeah. I worked for the Government Digital Service as part of the Cabinet Office for a number of years. I was involved quite heavily early on with gov.uk. That’s the single domain in project in the UK. I then ended up moving into doing more working with other government departments on infrastructure automation and information security, really being part of that help with digital transformation, adopting DevOps practices. Before then, [I had] various software engineering roles. I’ve always been a programmer of some sort, but I generally like the things around the edges. I’ve done a bunch of automations of CI systems.

I did quite a bit of work going back probably ten years ago now with—I was the developer to a couple of other people. They were on the operations side. I didn’t really know much about that. I was a developer looking to get my software deployed. I just built up a really good working relationship with them. Learned a load. That ended up—no telling with when ‘DevOps’ the word was coined in the first DevOps days. I wasn’t at the first one, but one of my colleagues I was working with was. He said, “Oh, you would have loved that.” I was along at the second one in Hamburg, and really just part of that very early conversation that got brought together by the word DevOps.

So you could comfortably say “I was that soldier who sat in that room. I had those conversations, and I watched as they went, ‘ah, we can get them with this guy.’”

To a certain extent I’ve made a lot—at least some of the mistakes as well. The disadvantage I guess of being both a developer at certain times and an operator at certain times is I’ve made a lot of mistakes on both sides as I’ve learned the practices around them. Which I think is then useful when you’re talking to people who were knee-deep in thorny problems that you have some empathy for, like the challenge they’re trying to address.

There are several directions we could take this conversation. You’ve got the background in security which is still in a lot of places the missing trough, if you like. I’d love to call it a link, but putting security into the DevOps process is still something that’s got to be force fitted sometimes. You’ve also got that big organization and big governmental public organization experience. Then you were breaking down little walls of confusion of your own in the very early days.

You’ve seen DevOps go from a thing that people were talking about because it was a good idea to a thing that everyone’s talking about because everyone else is talking about it, and trying to do it. How would you like to—what do you see—out of all of that when you’re working with other organizations today—where things are at, in DevOps terms?

Yeah, good question. Similar to the software evolution of DevOps itself. It wasn’t that there was this word ‘DevOps’ and suddenly people started talking about infrastructure automation or developers and operators working together. Those things were starting to happen anyway. They were starting to happen in individual organizations. They were starting to happen with passionate open source developers. They were starting to happen in large organizations, but it was mainly separate. You’ve got those conversations happening in lots and lots of places, but there’s not one place for it to come together. That’s what the DevOps word and DevOps banner and the DevOps community developed into.

Honestly, I still see that happening today. Terms always cross through that somewhere in the hallways, and successful terms cross over that barrier of, now everyone’s talking about it because everyone needs to be talking about DevOps apparently. Some people get quite cynical about that because they’re like, ‘what does DevOps mean?’ The reality is that, compared to five years ago, six years ago, seven years ago, more people today are talking about the need for development operations to work together fundamentally. The need to automate the patterns that we see that allow us to go faster without sacrificing security, without sacrificing safety. I just think it’s grown—we’re having the same conversations and that’s a good thing because they’re reaching more people.

That’s an interesting philosophical difference compared to some of the people I’ve discussed DevOps with. I’m just going to play this through in my head and make sure I remember it all. You’re saying that this need for—I like what you said: “There are patterns to move faster without sacrificing certain elements to get the two sides to work together.” That’s a very ‘both sides of the divide’ kind of way of presenting it, whereas some of the people I’ve spoken to have come much more from the developer side, and said, “If we’re going to work faster, we need to bring the ops people on board.” I think when we think about what we’re trying to do with enterprises, it is about aligning with the conversations maturing within those enterprises rather than bringing it in as a big new thing.

I’ve seen and talked to people who have said, “oh, DevOps is driven—it’s a plot by developers to get rid of the operations people.” I’ve also seen operations people saying the opposite—where it’s a plot by the ops people to control developers. I’ve seen DevOps events where it’s been mainly about developer conversations. I’ve seen DevOps events where it’s mainly been about operations conversations. Some of this is just about—it’s very local and that might be a local event. It might be local companies. Who is driving this in your organization turns out to be really interesting. Who’s driving the change? In large organizations, sometimes that’s: who’s got the budget? Who’s got the political capital to make this change?

I think where DevOps works it’s fundamentally about acknowledging it a bit more of ‘yes there are specialists and yes, they’re skilled. Yes, there are separate budgets or separate organizations, but we’re better off individually and collectively if we work together.’ Some of it is giving up some of that ‘I’m an operator,’ ‘I’m a developer’ tribalism in pursuit of ‘What does our company do? What does our organization do?’ That’s the primary driver.

It’s funny. It’s a bit like when you say, “Oh, I work with computers,” and they say, “What printer should I buy?” I was speaking to someone who worked for utility at running club last night, and he said, “We’re looking to expand our use of DevOps across the organization.” (I thought, I’m trying to have a run here.) He was asking me things and then there was one moment he said, “Because they...” I said, “There’s your problem. How can you get above that ‘because they?’” Who’s above it that can see it as a ‘we’ thing rather than a ‘they’ thing. It wasn’t just a waffley thing that someone says. He absolutely engaged with that and said, “You know what, that’s absolutely it.”

Often organizations have quite reasonable reasons at the time, structured themselves in such a way that you build those barriers. You build those incentives that literally pit people further down the organizational tree against each other. I think appreciating those incentives are often like, why is someone working like that? Oh, it’s because literally that’s how they grow as a person. They’ve been told that’s how they reach the next level in their career, and not because it affects you, but because that’s the thing you’re optimizing for.

I think if you can go visit people in your organization on ‘the other side’ who you perceive [as ‘other’] organizationally, structurally, socially, and asking them “What are your incentives? What are the main things you’re trying to optimize?” It’s the classic developer thing of... the operations people want to maximize stability and the developers want to maximize velocity.

If you don’t acknowledge that those two things are in opposition, you have incentivized your two bits of your own organization to ultimately never get on. I think incentives are high under the management thing or a big part of embracing DevOps especially in large organizations. Smaller organizations, it’s true, but those people probably sit next to each other. In larger organizations, there’s probably several layers of hierarchy between you and the people setting those incentives.

Yeah, different building, different sites, whatever. I was thinking the security people are there to enforce policy as well which is another dimension to it.

The DevOps banner I think has been just something where people can come and talk about their different incentives and try and build some things together including the security community. The very early DevOps events definitely had no security. They weren’t in the room. Similarly they weren’t in the room as those people went back to their organizations. It was maybe three years in or so, like seven, eight years ago when early inquisitive security people started showing up at some of the DevOps events. That was the topic that was suddenly really interesting in those events. Maybe like chaos engineering today or observability is today. We’ve seen I guess over the last ten years of having that banner, having that lens, security with definitely—it’s still critically important, but it wasn’t until maybe five, six years ago, when people really started showing up to have those joint conversations.

It was almost given where it’s coming from. I was doing this stuff in the ‘90s when dev and ops were certainly not talking to each other. If they were, it was all a bit fraught, and just getting them to talk to each other was—you’re not done, but boy you just need to fall back and regroup and get your breath back before then moving on to other stakeholder groups. We used to, in ISDN days, we use to talk about multifunctional teams, but I’m pretty confident that we didn’t have a security person in that team.

Yeah, and it’s still true in lots of organizations. Again, people need to feel bad about that. I think their acknowledging that ‘wouldn’t it be great if they were there?’ I think that’s the mental model change. Where I’ve seen that work, suddenly the security people are in demand. They’re having things brought to them, and their problems change from playing whack-a-mole after the fact, to going ‘there are too many people asking for my time here,’ and I can really see a sea change there.

Given how you said that different organizations will see it more as a developer thing or an operations thing or a top-down thing, where is the budget, etc, etc. Should all organizations be trying to do DevOps, or should all organizations be just trying to evolve in the right direction?

You could put a digital transformation hat on some of this, but just get to a state where they’re working together better and more agile so on and so forth, and DevOps just naturally falls out of that? Their own blend of DevOps. How does it work?

Yeah. I would say all organizations would surely benefit to a certain extent of having empathy across roles and working together more closely. I think it’s easy to say that; it’s also what was actually needed on the ground. I think it’s easy to be cynical there. To me, to a certain degree, doing that really is DevOps. It’s like that cultural aspect of it because that’s probably the bit that is often more ignored in the pursuit of the other properties.

I quite like the ‘CAMS’ description of DevOps if you like: the ‘culture, automation, measurement, and sharing.’ Ultimately metrics and automation can be tricky, but most organizations will have some pursuit of those. Most people are not today saying “We shouldn’t automate these things.” They might say “It’s expensive to automate these things; we’re automating these things first,” but most of them are not in my experience completely anti-automation.

In a similar way, most are not anti-collection of metrics and actually measuring things. Again, the DevOps community has a lot of material and so do others about improving those things. If you combine them with a culture that really emphasizes sharing, to me that is just DevOps. Whether you’re doing the long list of different practices, the different people like the cloud.

I think DevOps avoids the peril here by being undetailed and unspecified in comparison to something like an Agile process or an SRE more opinionated take on this. I think when it comes to the opinions there is no universal ‘everyone should be doing X.’ I’m a fan of Agile for all sorts of different development activities, but it’s not the panacea. It’s not the only thing you should ever do to your problem. If agile is the solution, it’s great. SRE similarly is a much more opinionated take on the higher-level DevOps practices. Again, that’s very suitable in some organizations. Maybe it’s too much in others.

DevOps I think is such a broad banner. To me, it’s just a word under which we have all these conversations about this space. That has downsides, but I do think it makes it universally that you should be adopting some of those things.

Is it a journey or a state of mind?

To me, it’s a banner. It’s the sign under which we have these conversations at the moment in time. At some point, that will, the banner will move to something else. It’s just a pointer. I wouldn’t say it’s a destination certainly, and I guess ‘state of mind’ isn’t a bad description. I generally just describe it as the banner under which we have the conversation happening at the moment.

Yeah, so it’s more setting the direction than getting to the place.

Yeah, absolutely.

With that in mind, that all just got very, very philosophical. (Laughter) In your experience, is it just the case of literally printing something out in a big piece of cloth and sticking it at the front of the building or given your background, with the Docker side of things, with the automation and CI/CD side of things, etc. What are the things that catalyze the ability to move within that banner? Below that banner that you just do.

I think some of it comes down to you need concrete things at the end of the day, and that’s where the tools come in. Ultimately, finding tools that cross boundaries. I guess the pathological case is when your security team buy separate tools and they’ll do it under a banner of a CIO, and your operations team buys monitoring tools. Again, separate budget, separate organization. The overlap in the tool is actually quite large, but the presentation of the information might be different for a different audience. Your developers now have application level monitoring. Suddenly you’re all using separate tools that overlap quite a bit. The fact you’re using separate tools means you don’t have to collaborate. You don’t have a way of collaborating across data, even though that’s security data about the applications you’re running on your infrastructure.

I think sharing tools brings people together in the same way as sharing physical space brings people together. You suddenly see things that you wouldn’t have seen otherwise. ‘Oh, look. My application—it’s not just that the frontend has been responsive, but actually, look, the backend process is taking up too much disk space’ or ‘oh, look. The cluster it’s running on, why do we need this much capacity versus this other application that another team is running?’ I think shared tooling helps. Continuous integration I think is often at the heart of that because it’s one place where the pipeline metaphor can really be useful for a shared set of tooling and different teams, different types of teams injecting things into a shared pipeline over which we’re deploying software.

The security team introducing security checks is just a step in a pipeline. They’re not trying to take over your application. They’re injecting something that’s useful to everyone, and it’s visible to everyone, but it’s about addressing the security team needs or providing information to the developers.

I think shared tooling is super important. Monitoring, visibility, tooling, logging, tooling similarly. I think a lot of that, when it comes to the software delivery side, continuous integration is key. If you have that and you have that shared across the different disciplines in your organization, and that might be service management people, that might be asset inventory, that might be the security teams, the operations teams, the developers. You can have these conversations often in code in a shared setting versus ‘oh no that’s, only the development team have access to this tool. Only the operations team have access to this tool.’

That’s making it very delivery-centric and as you say, operations is fantastically important, and all that monitoring and so on. First thing you need to do is streamline how you deliver with everyone as part of that streamlined delivery.

I think it depends where—what the pain point is in your organization. There are organizations [where] the priority is delivering new functionality and features or whole new services; where all of those services actually have an active development team and that development team is in-house. There’s also organizations where actually, the majority of the work they’re doing is running software. Actually, the majority of that software isn’t even first party, it’s third party. I think all of these shared tools that benefit which ever end of that spectrum you’re on.

Where you start, I think depends on your starting point. If you are actually starting your applications and you’re undergoing that digital transformation then maybe your insourcing, you’re seeing a lot of that at the moment back in-house.Then step one, you have to get a continuous integration set up in place, share that across your different disciplines and have the conversation in code and configuration about what’s in the pipeline with everyone because then you can then have a really reasoned debate about what’s going on there.

If you’re mainly running things, I would say start with monitoring, start with logging and again, having the visibility into what’s happening, but can be used for debugging applications and capacity planning. It can be used for security analysis. Have one thing like one source code to them. That might be the best place to start. Obviously, there are others. I think it’s contextual.

Excellent. This is not a completely glib question, but it’s going to come across like one. Surely the answer is ‘microservices for everything’ and then the tooling fits with the architecture and the targets and people just need to align around that?

Again, I don’t think there’s generally speaking one size fits all, especially in that case of: ‘Look, we’re not actually building loads of software, we’re just running third party software.’ I guess this is one of those things where surely as a developer earlier in my career, I just knew the software I was building and I thought about the software I was building. I knew other teams adjacent to me were building software. Actually, then you would talk to the operations people and they would be like “We’re running 100 different applications. We have to care about many different things.” I think the trade-offs there were different. If you think about microservices and individual teams going, ‘oh, I’ve gone from managing and writing one application to splitting that into five or six services.’ That is amplified on the operation side.

I think there are operational advantages to that, but there are disadvantages as well. I think in organizations which were embracing a ‘you build it, you run it’ type of approach where you’re pushing the operational responsibilities down to the development teams themselves with often assistance from operators and security teams. In that world, ultimately the trade-offs you’re making from an architecture and development point of view, you’re bearing the cost of them as well. That can work really well because you create the services that work for the domain that make it easy to develop and deploy them independently, but you’re bearing the operational cost, so you understand that it’s not free.

I think if you have a disconnected organization that doesn’t get on, that hasn’t really built up that relationship yet, microservices can just make that worse. Suddenly instead of throwing over the wall one thing every six months, I’m now throwing 100 things every day from the operations side.

It’s gone from granite blocks to tennis balls, isn’t it?

Tennis balls fired via cannon from multiple different angles at a rate that’s going on 24/7. I think…

I just like to say for the record I apologize for putting that in and I in no way think that the whole lot of microservices... but it was a good way to ask the question.

Yeah, I think that’s the thing with all of the, there are patterns and practices around the DevOps community that actually are absolutely the right thing to be knowledgeable about and interested in and consider in your organization. I think asking and answering questions like that is an important way of people realizing: put it in your context. Understand the problems that it’s solving and then say, “Do you have those problems?” or “Is that the most important problem we should solve first?”

Awesome. I think that’s possibly a good point to leave it on, which was absolutely excellent. What have we learned? It’s about understanding the needs of your own organization. It’s about not trying to apply DevOps as a conceptual thing but working with it as a banner below which you should be migrating things that you actually understand and know about. (I hate the word ‘’ by the way.) I mean should in the context of don’t try and do it the wrong way. Do it the easy way. We’ve learned that it’s about getting that continuous integration in place and sharing that across different parts of the organization and building up from there. Have I missed anything?

No. Especially you should start with a focus on individual practices, whether that is getting really good at continuous integration, becoming really good at deployment, and becoming really good at monitoring, becoming really good at securing your applications, becoming really good at asset management. Pick practices that represent problems and focus on those. Be knowledgeable about what’s going on in the DevOps community and the conversation around those things that’s happening. Don’t just do it how you’ve always done it. Actually, yeah, it’s the practices that will move you forward, not ‘doing DevOps.’

Awesome. Gareth, thank you so much for your time. Everyone listening out there, if you’ve got any questions at all for us, do send them over. You know where to find us on Twitter, etc. I really appreciate it. Any last glib single sentence from yourself?

Don’t be too cynical about the benefits of people talking about empathy and talking to each other. There’s just a load of things to learn and generally speaking [it’s] quite interesting at the same time.

Awesome. Thank you so much, Gareth Rushgrove. Thank you all for listening, and speak to you next time.