Voices in DevOps – Episode 1: A Conversation with Thomas Boyles of Sauce Labs

Guest

Thomas Boyles is passionate about building world-class engineering teams, creating innovative products and championing process for facilitating Agile software development. He is the Director of Engineering at Sauce Labs in California.

Transcript

Jon Collins: Hello and welcome to this episode of Voices in DevOps. I'm here to speak to… Thomas Boyles.

Thomas Boyles: I'm the Director of Platform Services at Sauce Labs.

Okay. So we're here at Power Up which is a Source Labs kick off for 2019. So I don't believe it's 2019 already. That's just scary. I can't wait. Yeah suddenly it'll be 2020 and it's life happening far too fast. That's a diversion. So we're here to talk about DevOps. We're here to talk about you. And so maybe just to kick off… tell me a bit about yourself and your role and what you're doing here at Sauce Labs and so on and so forth.

Sure. So as I mentioned, I'm Thomas Boyles. I'm the Director of Platform Services here at Sauce, essentially as an engineering director and responsible for multiple teams. One of the teams is kind of the back end guts of the app and I tend to describe myself as everything that isn't the cloud and isn't the user experience. I do also have a front end facing performance metrics kind of greenfield team that I'm responsible for too. A large part of my energy just goes into making sure that the back end is efficient and that things function and the customers are able to get their tests in good time, and that it's a good user experience for them from the web driver angle.

There's a question I want to ask, but I'm not going to ask it yet, but I'll [say] it now so that you're prepared for it.

Sure, sure.

Are you doing DevOps?

Yes we are. Definitely.

We'll come back to that. So how did you get into this then? What's your background?

Originally I come at this from sort of almost just: ‘I'm good at tech’ and that's kind of what drove me. I don't have a CS degree. I self educated myself. I happened to be friends with a bunch of people who were in tech in the ‘90s in Chicago. And as soon as I finished my film degree, I just didn't want to go into it and they gave me a job and I kind of worked my way up through operations and into developments and into more DevOps-y stuff. When I migrated to California in 2003 a lot of things were starting to kind of come around, including just better CICD processes and that really kind of struck a nerve for me. So I got into release engineering and kind of more DevOps-y practices in advance there. A couple of years ago I had  finished up my commitments at my previous shop and had been sort of stalking Sauce for awhile. They were a very cool brand and I really wanted to see what they were doing. Submitted my resume and it turned out it was a good fit. So lo and behold, two and a half years later, I'm here.

So the job that you're doing now—was that the job you were recruited for?

Yeah, yeah it's more highly scoped. So I started as a manager. I'm now a director.

Okay. And you gave me the title, but what's it mean in terms of [your] role?

For a director it's really you're you're managing managers and you're managing multiple teams. And in this case it's just engineering teams. So a large part of the role is making sure that developers are happy, that they’re productive, that they're really kind of pushing themselves and working well within the constraints of Agile. We’re a big Agile shop. I interface a lot with the product owners. I am also a product owner, which is a bit of an entity problem, but it seems to work for one of the teams. And yet it's really just kind of part coach, part advocate, part ‘Guys like I need to get together and read this; this is important stuff.’

So you're coming it to from the point of view of: you’re a DevOp shop.

Yes, we are.

What does that mean in practice?

For starters, we have a real owner/operator mentality. So when Sauce conceives of a change, an idea, a feature, it's really the job of that team that's working on it to get it from ideation all the way to production. So you know a given team, they're doing their own deploys and they're doing their own config. And one of the things that was kind of a point of maturation for Sauce was getting out of the scope where you just had like 20 engineers that did it all—into some people that were a little more specialized, operations engineers, network people. But we still work incredibly closely with them. And it's a really nice feedback loop.

I've often been at shops that were the antithesis of DevOps where there was a real concrete wall in between dev and operations. And a lot of bad feedback loops result [from] that. This isn't that way. As much as we might have a challenge trying to figure something out, we really are partners in how we move forward and we're always looking for ways to optimize the organizational constructs to improve that and sometimes it's cross-functional. Sometimes you do need specialists like having a cross-functional team with a good network engineer and it might not be the best use of that person's time for example. But in general it seems to work out really well and we can move really fast.

It's always a really nice learning experience too for us because… in the sense that we are at the point where we need specialists: specialized developers, specialized systems engineers. They're coming at things from a different culture. And the fact that there is a lot of transparency and cross-communication [means] they really learn from each other and they expand. It's almost like working with people from different national cultures.

Was it as mature (is that the right word?) as it is now when you when you joined?

No, not at all. Not at all.

That's what I'm interested in: the kind of transition from...

Yeah.

How's the journey been?

The journey has been interesting because when I first started, we were really growing fast and we were really struggling with a lot of structural problems. And they grew the ops team out rather quickly. You know like added a lot of really good top talent. And there was a lot of kind of like ‘storming before norming’ to really kind of figure out what the best balance was. And it's always hard. You know especially if you get people that don't know what DevOps is supposed to be…because everybody's heard the phrase, but they don't really understand necessarily how to implement it.

You can come up with a lot of like signal noise around how you're supposed to do things and what's right and what's wrong. So yeah, I mean I do want to say like one key factor was actually educating people and then coming up with conversations about what the what the definition of it really is and just getting people to read Gene Kim’s books and then you know getting people involved in the community and getting them to let go and actually mix it up with other people and kind of get their experience.

So it's something that's difficult to analyze in survey terms. But I have a feeling that a lot of people sort of ‘talk the talk’ without necessarily walking the walk. They haven't necessarily internalized it, so that they may be doing lots of iterations but yeah who knows how useful they’re being.

Yeah I mean I think you have to have leadership that really understands what that means and that it really ‘bought in’ and then you need to create the right kind of pressure to like make sure people don't fall back on old habits when they're stressed because that's a very human nature kind of thing to do. You have to tell them, “No it's okay—we're going to work through this process, we'll learn. And I understand this isn't like your comfortable spot but it will be and it'll be okay. Trust us.”

I've got a theory on that back in the days. Because we were talking about earlier when I was a DSDM guy. Interestingly thinking about DSDM today, I think it was essentially old school people trying to think how a child thinks. Yeah, it's a bit like you know how elephants might think they should do ballet as opposed to ballet dancers designing it. But it was good as far as it was. So it was a slow and that's unfair because it had nine principles, and as long as you did the nine principles you were okay.

That's a lot of principles.

But, isn't it? But anyway, I had the theory that came up at the time essentially that all the big luminaries… I mean you mentioned Gene Kim. There’s also Kent Beck and Alistair Cockburn and all these people, these names. And so the theory is around that the guru's dilemma. So what happens is any of these people goes into an organization. Have you heard the… This is a bit of a digression: that the queen thinks everywhere smells of fresh paint?

That's very interesting.

Because there was painters just before she goes in. So I think these people go into a place and everyone does the right thing. And then after they leave, it's got to decay curve where they try and keep it going. And then they revert to type.

Yeah.

Over time. And so as far as they're concerned, it always works.

Right. So they never see any bad feedback because why would you? I mean you would be like Kelsey Hightower coming in [and saying] “Show me your one good cluster” and you're like, well great! This is working well, you know?

Yes, exactly.

Now that's interesting.

So, I mean what would you say [are] the biggest lessons that you learned in terms of internalizing and moving away from just trying to ‘talk and talk’ and actually taking it on board.

I don't think you can wing it. This is actually one of the biggest lessons that we learned over the last year: you really actually have to be very structured, and you need real advocates and you need to really train people and you need to find ways to constantly check back in and measure what success is.  I mean it's the same thing with like an Agile transformation. You can't just give people the elements of SCRUM and tell them to read it and expect them to know what they're doing. You actually have to send them to SCRUM training. You really need Agile coaches and then somebody needs to be that that thought leader and that person [who’s] kind of holding it and making sure it's worked into the structural matrix of what that organization is. If you don't have it -- I mean you can get things out of it, but it's not the same as really like owning it.

So that's measuring it? Or?

Measuring it helps because you want to make sure if you're going to give people goals and if you're going to hold them to account, you need to show them how you're holding them to account. And that really gives them a very clear cut sense of like, ‘okay I need to do these things.’ Like how easy was it till I get an iteration between the teams or how cross-functional are we versus how many times do we have to go and reach out to another team in a quarter to get something done? I mean those are those are the kinds of things that you could measure that would show you that. And when you measure, and especially what you evaluate with measurement, -- that can be a really powerful tool because it influences how people behave.

Okay, you don't have to answer this.

Not at all.

But I'm going to ask anyway.

Sure.

Would you put yourself up at the ninja level as a team, as an organization? Or are you still kind of…from zero to 10 in terms of DevOps?

Just out of pure humility: like a six or seven. I mean there's always a ways to go and one thing with Sauce that's always a very interesting conundrum is we are moving very fast. We have real scaling challenges that we have to deal with. And oftentimes we're not able to make the sort of hiring changes that we need to fully go cross-functional in some areas. And in some cases it's just not practical versus the timing. But we’re very serious about moving in that direction. And we also know that the numbers are in on the payoff and we know we're gonna get a huge benefit if we're doing it right. So we're constantly trying to push that envelope and then move it forward.

So it's turning to testing.

Sure.

That's what you guys do. We've been talking about testing in the DevOps process and whether it kind of gets in the way.

Yeah.

What's your view on… I don't quite know how to put this. Does does DevOps actually not exist unless it's a well formed process incorporating testing or do you see organizations go through stages where they kind of reluctantly have to bring in the notion of testing into the DevOp cycle?

I don't know that that software really exists unless you have some semblance of testing. So I think it's even more fundamental than that. I really...

I like that.

Yeah I mean if you think about it, it's like a mathematical proof right? Like yes, two plus two equals four, any child knows that. But when it's at a certain level of complexity, you have to be able to go back and check it because you won't be able to figure it out by yourself. Any  reasonably sized software project is like that. And there's there's just no… Like how could you be successful without even like a certain level of coverage in some sense?

But for us when we think about how we inform DevOps, it's really about helping people shift left and it's really about making that a less friction-full thing. And hopefully it should bring them joy like the ability to do a battery of functional tests before you actually are shipping something into a larger ecosystem—should actually make your lives easier and make you look better. It should make you sleep better at night because you're not getting called by a production operations team saying “Hey you're shit bro. Please come and fix it.” So for us I mean it really is kind of ‘How do you be more self-sufficient? How do you empower yourselves and take pressure off the overall system?’

For someone who didn't do computer science, that’s quite a scientist’s kind of approach of empiricism that if you can't measure it, then it doesn't exist.

It's a big deal.

And also if you can't prove that it's true… then the chances of it being false are quite high.

That's right. Yeah and I've been reading a lot about measurement specifically lately and oftentimes there's this fallacy where people will say like “It's just too expensive to measure” or “I don't know how to measure that.” And really those are exactly fallacies like you should be able to figure out at least something [like] a specific number or a range and that should tell you something about what you're doing. And you can be empowered by that. It expresses your level of uncertainty, and it helps to drive you down to like a lower level so you can be more informed. From an engineering manager’s standpoint that's gold as it is from the product owner’s point of view.

Well, there's two aspects that popped into my head. One is I don't know if you've ever spoken to that kind of formal methods people and so on. They take that stuff way too seriously.

Yes. Well this is the other angle.

That proof's in the…

You have to be comfortable with a lack of precision, because it's not necessarily about being perfectly precise. It's more about reducing your uncertainty. And if you can have that sort of sit easily in your head, it's a completely different game. So but again, you do have to be careful not what you measure but you have to be careful what you evaluate people on, and like if it's just commits, you're going to get people with a ton of commits and their software won't improve. And the same thing with test coverage. Any monkey can write 100% unit test coverage and it's probably of low value.

That aspect of [getting] the measures wrong…  this is a huge question of the wrong measures obviously and therefore skewing behavior and that kind of Heisenbergian aspect to it. Is also a question of just having measures and therefore feeling you're in a better place.

Right. You're not.

And so it's really really important to have measures that are actually of value and therefore you are measuring the measures. Ultimately it gets a bit better, but at least you don't spend too long on it. Just just need to be sure that the measures are actually delivering.

That's right. I mean not all measures are the same. You can say there's a million different dimensions about this area that we're in right now, but I bet only three or four of them are actually of any value to you. Like, how hot is it? How bright is it? You know like that might be it.

So, do you have any concept or visibility on to the customer facing side of things?

More and more recently actually because we've been driving… we've been trying to drive our quality up and that requires a lot of empathy. So I spend a lot of time trying to talk to sales engineers and hopefully customers, and then this really is a year of innovation for Sauce. So we're really trying to find out what would you guys do if we gave you this. What do you want? What's missing when you reach for, when you use our product and just isn't there? And yeah it's getting to be more… It was a little less when I started.

So, where would you then position in your experience your customers in terms of DevOps in general, testing in general, and then dev test ops for lack of a better term?

That's a really good question. It's much more of a wide scale than it was when I started because we weren't going after the same kind of industrial scale customers. And with a smaller shop like you might have a couple of people that are developers and they're very technical and they can write selenium tests in their sleep and you know they were really our target market maybe three or four years ago. Now we're going after enterprises.

So, DevOps were like the first kind of organizations anyway.

A bank, I mean good god like just getting an organizational change for a company that's 100,000 people is like changing what a village thinks or town. And then also they tend to be a little more siloed in terms of their technology, and then also there can be a wide gamut of technical ability within that company. Plus we're not always dealing with independent contributors anymore. Oftentimes it's business people that we're selling to. So it's been interesting. We've had to change our story and really kind of modulate the audience that we were speaking to.

So, what's the story now then? What's the evolution of that in terms of… Let me ask it a different way. When you speak to customers or when Sauce speaks to customers about the role of testing and the role of making DevOps work better.

Yeah.

How do those conversations tend to pan out?

We're getting really good at identifying where people are in this sort of scale of like late adopter to we’ve got it, to we're bleeding edge, we want to do the next thing, and really it's thinking about like what's going to make their lives better and there is a ‘there’ is a part of telling people a story about okay you're here now. Like you could be doing a lot more with a lot less resources if you kind of learn to advance. So let's help you figure out how to do this automated thing you've heard about. And oh by the way, let's find a way to make you—even though you're automated—let's find a way to make it so your developers can run tests. It's not just the final touch. It's very much like partnership to try and figure out how can we help them do what they're doing better. And I like that part of it. You know we have a lot of respect for them.

It’s almost consulting then. Too strong?

Yeah I think we're not quitting the business services yet. We're not going into professional services yet, but we really do spend a lot of time with customers and we really do. We send them technical resources when we go to try and create partnerships with them, and these are people that are actual developers that also happen to be really good with people and have good sales skills. So it's it's an interesting approach, but it's also been really fun to watch Tucker and company develop that org too, because you can just see it growing in this very interesting way. Yeah, a lot of smart, very assertive people.

So looking ahead, where would you say… The reason I'm pausing is trying to avoid asking more business questions. I'm thinking more about maturity questions.

Sure.

Where would you say your customers are going to look to be? Once they start getting their heads around… Every organization I think without exception wants to do DevOps.

Oh, absolutely.

But that's not a question.

Yeah.

Then the question is: How do they do it? and whether or not they're going to be successful. And so when we talk about maturity questions, then it's about well we tried it and it worked to a level. And I'm seeing organizations that kind of falter at that point or then break through into into making it broader.

Yeah.

So, within all that as a context, how do you see your relationships changing with your customers and those conversations evolving?

Oh, I mean it really just depends on the specific customer because every org’s journey is very different and they all have different constraints that they're probably classes of constraints but they're all unique to that org. And for us, we believe what we give people is basically valuable and we believe that we can do it for them better than they can do it for themselves. So in that sense we're really giving them a resource that saves them time and then we're also very much advocating for: ‘Here’s how you should push the envelope on that,’ but you know I mean some some places are these weird conundrums where it's not a linear path, and they're very ahead in some respects and then they're not at all ahead in other respects, and we just really try and find ways to help them evolve as they need.

And there isn't any kind of standard ways that they're likely to be ahead or not? Everyone's different?

I think everybody is different yeah because you have companies that might be practicing CICD but they could have terrible test coverage. They might be like truly cross-functional teams but they might only be deploying once a quarter or something like that. I mean there's a lot of reasons why it just might not look like an end state for them.

And also I mean to be honest, you want to look at what problems the whole DevOps philosophy is trying to solve. If they don't necessarily need to solve all of those problems for themselves, maybe it's okay. We're definitely not there to tell people what they should do. We're there to try and empower them and to kind of show them paradigms that could actually be better for them and by the way, if we can help them get there and if they can buy more Sauce on top of that that's awesome. But for us it's very important that they're there just kind of advancing the canon in a way that's authentic to them.

Same question twice. And it's a kind of wrap up question.

Sure.

That's why I ask it twice: for yourself and for your customers. If you could wave the magic wand. For yourself first in terms of: if you could have one thing right now that would make your life so much easier, and advance things significantly. What would that be?

So the biggest thing that I would say is I would like to find a way to democratize the creation of automated testing in a way that is 10 times easier than how it is now because you can go find a tutorial on writing selenium tests, but the truth is you're going to run into problems and they're related to the construction of selenium as much as infrastructure and network speeds and things like that. It would really be nice to sort of lower that bar to make it some more people are doing it and they're getting more benefit out of it. And it just it doesn't need to be science. It should just be basic engineering and I think it's not quite there yet.

So is that AI or is that overstating there?

Yeah I mean AI is definitely an interesting element. You know this idea that you could throw a robot at a site and it could just spider it and write you a suite. I think that's a noble goal. But I also think languages are advancing and there's always going to be a place for cross-functional browser testing, but there are other paradigms that are coming up like Puppeteer and Espresso and things like that.

We really want to be open to them in terms of what they can bring to the table for the customer is really what we are is this specialized cloud that helps you run any kind of functional testing. I mean we definitely invest in things and we have horses in those races but we want to make sure we're thinking ahead and it really is about how do we make your life easier. It's not about how do we double down on something because we've already invested in it.

So would you answer the question the same way for your customers?

Yeah, definitely. I mean because again for us, the happier they are and the more their experience is useful and frictionless, the better the relationship is and the more we will have enabled that both from a real quality of life and joy perspective, and then obviously it helps us keep the lights on because we're selling more VMs and phones and things.

So it's just ultimately ‘If only we could write tests faster…’

It would really be a game changer.

Is it boils down to…

Yeah, and better.  It's one of those things that people really… they're looking and they're seeking but there's nothing [that] really sits easily in the mind about how we're gonna solve that problem. So it's the task of the year I suppose.

One last question for you. You mentioned the term shift left’ before… and putting testing earlier into the cycle. What about a ‘shift right’?

That depends on what you mean. I think there's always a need for some complete user acceptance test, but really we've definitely seen for a lot of our customers that having a deep end to end testing framework is an antique pattern. It's expensive and it's very difficult to reproduce.

So as you scale, if you're working at Google you're not going to get a whole copy of Google to run on your laptop, you're going to get a bit of a search part piece that you might be working on and you need to stub things out, and if you have to get in an airplane or something and go twelve hours away. Good luck running at all.

So you really want to be there focused about how that works and that's why the shift thing is important. It doesn't change the fact that you still need user acceptance and you still need to find ways to make sure that the whole thing comes together. But if we really want to make that less important…

Okay. Cool. Well that's a twenty five minutes.

Fantastic.

So we're through. Well thank you very much for your time.

Yes.

I'll let you know when it's up.

That's great. Anytime you want to do it.

Cheers. Thank you.

Cheers.