Voices in DevOps – Episode 5: A Conversation with Andi Mann of Splunk

Guest

Andi Mann, Chief Technology Advocate at Splunk, is an accomplished digital executive with global expertise as a strategist, technologist, innovator, marketer, and communicator. With over 30 years' experience, Andi is a sought-after advisor, commentator, and speaker. Andi has co-authored two books, blogs at 'Andi Mann – Übergeek', and tweets as @AndiMann.

Transcript

Jon Collins: Hello and welcome to this episode of Voices in DevOps where I'm here to speak to Andi Mann, who I've known for many years and who is now a Chief Technology Advocate, have I got that right Andi?

Andi Mann: That's right Jon, how're you doing?

Fantastic. It's good to speak to you too, all about DevOps as always but equally all about more ‘OpsDev,’ if I can call it that, the operational side of things, because I always think that that's kind of the poor nephew of the whole DevOps piece. So before we get into that though Andi, why don't you just say a few words about yourself and how on earth you ended up being Chief Technology Advocate at Splunk.

Absolutely... a vast and varied career across multiple continents, you can probably tell from my accent. I'm in the States now but started off in Australia in IT ops, banks, insurance companies, spent some time in the UK, covering like the Pru[dential], NatWest and others, eventually ended up in a in a vendor role in pre-sales. I think I ask too many hard questions of my vendors and in the end, I have come up through automation especially cloud computing in the office of the CTO and some large vendors, and now at Splunk as a Chief Technology Advocate, I advocate for my customers, so [that means] product changes, features, new market opportunities, but I also advocate for Splunk. So just explaining what we do in the world and why it's awesome.

So advocating technology, which is never a bad thing. So specifically at the operational side of things, you've got a huge operational background, and now you're working in an area where Splunk does a lot of things, but it engages a lot with the devops community and devops situations and so on and so forth.

What's your take on the whole thing from an operational perspective? I mean is it just kind of made up stuff? Is it actually all about like say ‘DEVops or is there more to it, and it's not just a kind of made up thing for developers?

You know [referring to] the big Dev, little Ops, I'm totally with you. Devops is not a made up thing. There's real results. I've done personal research on the state of devops report that I've done with Puppet Labs a couple of times, [and it] shows that there's real results. You get better revenues; you get delivery faster of new applications and services. You actually use teams more efficiently.

There's a lot of really significant measurable outcomes from devops, but what I see increasingly in the zeitgeist in the community is exactly what you're saying: ‘big dev, little ops.’ I can't tell you how many times I've been to conferences and people talk about the devops lifecycle and it stops at release. I mean ops is right there in the word devops, and yet increasingly we see so much focus on the development lifecycle. That's not a bad thing, but there's a lot that operations do and there's a lot of modernization that operations needs to go through to work in a devops mode.

And so I think there's a lot of opportunity for us to visit devops from an ops perspective and get a lot of value of innovation, of creativity, of efficiency out of that side of that devops equation.

I'm not going to name and shame here, but there is a school of thought... Let me put it this way: that we don't need ops at all, which is kind of an interesting perspective and I kind of see where it comes from because ultimately automation should be the name of the game. You want to build things that then require massive overhead in terms of their management etc, etc, but are there places that essentially automation can't go currently or going to be forever? Why do we still need operations? I guess is the question.

Yeah I hear this talk about “no ops” and I've actually been quite outspoken against the concept of no ops for a long long time. Even your ‘born in the cloud,’ cloud native companies ultimately still need operations for a lot of things, not least because performance and availability is going to crush your business if you don't get it right. Just because you go to the cloud doesn't mean someone else is doing operations for you. They're doing infrastructure for you maybe.

Even when you go to a SaaS provider, and you think about a Salesforce or NetSuite or something like that, and you've still got operational aspects because your customers will always blame you. They will never blame the providers that you use. If I get a late delivery from someone, I'm not blaming FedEx I'm blaming the company that didn't put it into FedEx in time, and so operational requirements from an on premises perspective in traditional enterprises, are still doing this to a large degree. You've got a lot of the efficiency; you've got the standard operational stuff: performance, availability, response to problems, all this level one type stuff, but then you start to look at the rise of what I'm calling ‘new ops.’

You know, this is the antidote for no ops in the new ops world. It's not that you do the same things but you do apply that automation, right? You're the people that work the automation tools so that developers can get that test server they need in seconds, not hours. You are the people that make sure when you deploy a new application to production, you've done the scalability and load testing in stage or in test, to make sure that application is going to run properly. And you're monitoring it in different ways though in new ops. You're using things like metrics and traces; you're not just using that infrastructure: server response times, utilization. You're actually getting into customer information as well as an IT operations professional in a new ops mode. You're looking at customer experience, customer response time. You're looking at payload information out of your network, things like revenue per minute, sales of particular items. These are your early warnings for whether your system is doomed to fail. if you normally sell 1000 of a particular Tshirt every minute and then this minute you only sell three, that's a problem, but it's not necessarily an infrastructure problem.

So new ops looks at the new infrastructure containers, cloud, serverless, APIs, all this stuff, in a lot of cases also looking at traditional on premises infrastructure. So CPUs and storage is still relevant, but they're also bringing in other aspects. They're looking at data and they're making decisions based on data, and that data includes business goals and customer engagement, amongst other things. This is a very different world from the traditional sys admin.

So I'm just trying to remember back through our conversations before we set this up. I've never heard the term ‘new ops’ before, which probably means I have not read your blogs (and I feel really bad about that ‘cause it's clearly something that you know a lot about). And so the reason I'm saying that is for the audience, that this wasn't set up to be a thing about new ops but I'm absolutely fascinated by it.

And I think it's a really good way of thinking about a more proactive operations, because in my experience a lot of operational stuff is ‘oh my goodness, what the heck have they thrown at us this time?’ And instead it's ‘This is what we bring to the party. We've got all the things that you might need as as a proactive provider, and this is how you can benefit from them.’ So it's a much more empowered operational position.

Yeah absolutely, and don't get me wrong: new ops does not replace devops. It's an extension; it's a more specific and thoughtful approach to IT operations that's part of that larger modern operational context. So it does incorporate aspects of Agile. It does incorporate aspects of devops. But like you say, it is a more thoughtful way of looking at modern IT operations.

And I just want to give a quick shout out to Jayne Groll, who is the CEO of the Devops Institute, who actually helped coin this term; and we're doing a bunch of events at the moment: the New Ops Days events. It's all nascent stuff, it's all just started, and the theory is that we want to help our people in IT ops get up to this new level of skill and feel like they're really contributing in the world. And it's a very powerful concept to be able to train people and help people understand what it means to move from monitoring feeds and speeds: the CPU time, the response time, the CPU utilization storage availability, I-ops, these sorts of things, and moving from there to monitoring and managing customer services, the application that is, how the data getting there? It could be on a split, a multi cloud application: part of it on AWS, part of it on GCP, part of it on premises and this is a whole new world.

So new ops very much keys off the concepts of devops: agility, collaboration, integration, your upskilling, abstracting the mundane and routine and automating that which can be automated so that you free up the humans to make the leaps of faith that they can make, whether it's on an innovation aspect... so ‘I think there's this application could do this’ or whether it's on a troubleshooting problem. ‘I think the problem is probably this... let's go take a look.’ It frees up people to be creative and innovative in all sorts of different ways. I think that's a very powerful thing for people in the ops industry to latch on to.

I used the term empowering once, so I'm not gonna use it again. What I was going to say is, it's quite a refreshing starting point. And when in my experience when I've worked with real companies solving real problems as, believe it or not, I have done in the past, it never worked. That's why I became an analyst. That's another story.

But so when you say “Is it what the customer wants?” and then you find a real customer and work through from there... I'm gonna actually throw in an example. So I did some training for a company called Biorad, that was a microscope company.And I did the whole component based development training Agile list and DSDM list and all that stuff. And I said “What you want to do is you want to get a customer in the room and interview them and find out what they actually want as opposed to what you think they want.” And one of the guys in the training course was the project manager and he said ”That's great, we've got a customer we've got someone in marketing that used to be one of our users. Why don't we just get them in?” (I was thinking, ‘Oh my God. I hope I was right). And so literally that afternoon we interviewed the customer. And with me, it was sweat streaming down my back and so on. And what they said was absolutely profound because this was a electron microscope company. We were trying to build a Windows front end onto their electronics, and what she said was ”You guys have got the interface so completely wrong, because this is how we use your microscopes: the first thing we do is we've got a tiny, tiny window because these things are so expensive and we've got to take the best possible looking picture of whatever it is we're trying to get. And the reason we need that is to put into our funding application. Once we have that, if we get successful funding, then we can use the microscope as long as we like and then we want to open the whole wardrobe of bells and whistles and all that sort of thing. But can you just give us the easiest route to getting a really good photo for our funding?”

And these developers, you had it with 10 of them on the course or so, and it just completely changed their world. It was absolutely fascinating.

Yeah. Look when you can connect closely with your customer, you'll be surprised what you learn. I mean there's the old Henry Ford thing as well. You've got to be careful that it's apocryphal. There's no actual sourcing for this, but Henry Ford supposedly said that if I'd ask my customers what they wanted, they would have asked for a faster horse. So you do have to be careful about preconceived notions and by reinforcing customers’ bias and stuff, you also as a person, as someone who's close to the customer, you get the opportunity to interpret what will make their life easier and take it an extra step. When they say “I want a faster horse,” it's like no no, what outcomes do you want to achieve? And then you can start to think, ‘well that outcome is probably better suited to a car than a faster horse.’ So you can come up with your own stuff. You're absolutely right.

Connecting with customers... and this is something that operations have traditionally not done at all, although they've always been the group at the company who's been closest to customers outside of perhaps your front office staff, but IT operations is the nexus of pretty much everything in a modern organization where every business is a technology business.

So to be able to have that really close engagement with customers, and to see them on an individualized basis. So I can look at your metrics out of all sorts of different systems: front ends, web servers, APM systems and others [to] just see what's happening at even an individualized basis. But now I can start to aggregate and look at metrics, histograms around service quality, I can look at what this release caused or at least correlated closely with a downturn in my business revenue from this website for example. Then you start to get the ability to really make those data driven decisions about what customers actually want. And it's probably going to be a combination of what they say they want and what you interpret that they need.

But that's so much better than sitting in that blank room as for your developers previously did and going, ‘I think my customers might want this. I'm going to presume they're thinking about it you know.’

And that, funnily enough, is the kind of stuff that I did. I've got a theory, which is that a lot of what we're talking [about], as I said before before we started that there's people in the cloud native digital native world that will get this stuff, and then there's a whole bunch of people in enterprises that don't recognize the language. It's not enterprise-y, it's not thought of everything that it needs to think of, all that sort of thing. I've got a theory though in the middle of all of that, which is that there are some really strong enterprise type hooks that we that we can hook into.

So one of my theories is the relationship between micro services and software modularity, which is a very enterprise-y thing. The hook I'm thinking about here is the relationship between what what you're saying and back in the day, when it went from IT service management, which was all about post ITIL kind of stuff. Then people started saying “Well [in] business service management, we need to think about measuring the business value. We need to think about, as you say, the number of shirts that are being sold or the number of bank accounts that are being opened or whatever it is.

So we're kind of going back around that buoy a little bit... I'm not trying to say it's it's all reinventing the wheel. There's a whole bunch of new stuff that we're dealing with now with the cloud. We've got massively scalable systems that we can we can do now, but then it still does come back so that some of those BSM principles.

Yeah. It's interesting isn't it? It's very glib and easy to say the mainframe was cloud earlier on, and SOAs (service oriented architectures) were surplus computing and API driven, you know they weren't. They were, but they weren't. History has moved on a lot. But what we certainly find is that when we look at these new architectures, there are these new ways of operating, right? You've got to understand cloud computing. You've got to do those things differently.

And new ops is about doing new things, but not necessarily throwing away those IT service management, ITIL, BSM kind of disciplines. Yeah I don't think ITIL/BSM, I think the processes we looked at there are fundamentally broken by things like cloud computing serverless architectures, micro services.

But the disciplines are important. Be secure, have governance, own control, don't let things get out of control, have backups. These are principles... understand where things are deployed. You talked about the CMDB: I don't want a static CMDB. As soon as I deploy a container or blow up of microservices it's out of date, but I still need to know what services are running where and why. Maybe CMDB is not the right word for it, but the discipline of asset management, the discipline of service management, of release management. Even things like problem and incident management, certain aspects of the discipline in these ITSM, ITIL categories still matters. But we just can't do it the same way because the infrastructure, the application, the customer engagement models are all just fundamentally different.

Yeah it's interesting, picking up on the CMDB point for the simple reason that I remember reading the original ITIL books the original library of books and that's how old and sad I am. But they didn't mention the CMDB. They were just a bunch of geezers who worked out what worked for them in big enterprise shops and shared that information.

And it was the industry that picked it up and said oh well you need a database because the industry would, and it maybe shouldn't go back to as you say those kind of structures that clearly wouldn't work in this massively orchestratable set up that we have now, that maybe go back to what those guys and girls were thinking about back then in terms of best practices.

So I just am conscious of time. When you speak to your enterprise type customers. To me as an enterprise kind of person,what you're saying makes sense philosophically. What do you think and what kinds of conversations are you having on the ground in terms of making this stuff happen?

Yeah look there is incredible skepticism from buzzwords from the enterprise communities. We've suffered buzzwords for years and years and years. So we take time to wait and see if they're real. So when it comes to devops, there is still skepticism in the enterprise. But I would firstly encourage those skeptics to look at real research. Now there's lots of vendor delivered research, which may or may not have bias and Jon, I'll point to your stuff for example.

Definitely unbiased.

Independent research which validates the concept. I understand what you can get out of it and then start to look at non-threatening ways to implement some of the processes and activities around devops. So what I mean there, think of automation: automate a task, an individual task. Just take a little bit of of what a colleague, a friend of mine Damon Edwards calls toil. You know just that routine mundane work that no one really wants to do but is essential. And it could be things like managing your configurations, managing your golden images. It could be something to do with managing and automating the deployment of software into production.

And I always like to look at the boundary between development and operations. And if you do nothing else, sit down for lunch once a week with teams that are mixed. You know if devops is about anything, it's about communication and collaboration and integration between teams, not software. So get your teams to talk together. If nothing else, bring ops into a planning meeting so that they can understand what's coming down the road for them and they can start to provide feedback and close that feedback loop to developers to say, “Here are some things that you need to consider when you're delivering this new piece of software.”

Having the conversation is the real good start when it comes to technology. Look at automation tooling around that boundary area: provisioning, configuration, release management, especially. You don't have to change the world, but maybe think about setting up a tiger team from a couple of devs, a couple of ops, this isn’t an empty patented devops team, but it does for a large enterprise, if you put it on an individual project maybe a new mobile application, something like that, gives you an opportunity to learn and practice this and see if it's going to work for you.

There are easy ways and this what I see with my large enterprise customers is easy ways to get into this devops mode without destroying everything you've built up before. And I think that's a great way to get learning, get experience, get confidence, prove results and then move forward if it's working for you.

It's said that the flip side of ‘don't throw out the baby with the bathwater’ is ‘don't drown the baby.’

Exactly.

I think enterprises do a lot of things fantastically well because that's what they're designed to do, but because they've been doing it for so long it means that they have the burdens of all that stuff. So it's not about stopping being an enterprise, and I think that's what we're seeing with smaller companies that think ‘why do they do all that boring governance stuff?’ That's because they have to. You will have to soon when you grow up.

Yes once you list at some point non gap?? is not acceptable anymore right. At some point you've got to follow the full accounting principles; you've got to follow audit controls; you've got to follow regulation compliance, all sorts of stuff. At some point you can't move fast and break things anymore.

So to wrap this up very quickly: if I had to ask you, what's the one thing that you would advise any enterprise organization as a starting point for a better way of life in terms of the whole devops, no ops, new ops (forget no ops) world, what would it be?

Honestly if it was one thing it would be a pizza party. Get your teams together in a social or semi social setting just so they realize that the other side are human beings like they are. Break down the wall. That is the most important thing to start within devops.

That's fabulous. Thank you very much, and I think it interesting that I've written about this. The beer and pizza are definitely signs of a well functioning organization, and soft drinks if that's what takes your fancy of course. So on that note, Andi always a pleasure to speak to you. Look forward to meeting you again in person in the not too distant future. Thank you very much for joining me.

Thank you for having me Jon. Really appreciate it. Great to talk to you, mate.