Voices in DevOps – Episode 7: A Conversation with Isaac Sacolick of StarCIO

Guest

Isaac Sacolick is a successful CIO who has led digital transformation, product development, innovation, agile management, and data science programs in multiple organizations. He has transformed underperforming businesses by delivering new digital products, investing in strategic technologies, enabling agile practices, outsourcing commodity services and establishing performance metrics. Isaac has been recognized as a top 100 social CIO, blogger, and industry speaker.

Isaac is currently President and CIO of StarCIO, a consulting services business that helps businesses execute their digital transformation programs. Previously, he held transformational CIO roles at media, financial services, and construction industry businesses and was a founding CTO at media and social networking startups.

Transcript

Jon Collins: Hello and welcome to this episode of Voices in DevOps, where I'm here to speak to Isaac Sacolick, did I get your name right there?

Isaac Sacolick: You've got it pretty close. It's Isaac Sa-CO-lick.

Sacolick, oh I'm terribly sorry. That's my British English accent coming out there and the other thing I'm not going to try and do is introduce your title because I looked at your LinkedIn page, and it's a long list. So why don't you introduce yourself as you would to any person that's wanting to know about you?

Absolutely. Jon, thanks for having me and hello to all the listeners. The best way to understand me is through a few journeys that I've been on. I got my early career, starting in the 90s as a startup CTO, so I have my chops in how to grow organizations and do application development and really run fast. I then pivoted my career and worked for enterprises that wanted to transform their businesses, build customer facing applications, build analytics capabilities, become more efficient with automation.

Along that journey I started I would say my third career as a writer and a speaker. I started a blog about 12 years ago that now has over 400 posts on it. That's a blog called Social, Agile and Transformation. I now write for CIO online, I write for InfoWorld. I also do quite a bit of speaking and nowadays I do a lot of consulting and my consulting is really helping organizations become smarter, faster, innovative with technologies, how to become proficient with everything from application development to automating data pipelines to running things in the cloud. And really my focus is in three different areas: it's how to be agile, how to implement DevOps changes and how to become data driven.

If I'm not wrong, most people (and I'm using that... it's like saying ‘the average company.’ There's no such thing as the average person), my understanding is most people do it completely the other way around from what you've done. They start working for bigger companies and then they go “Actually…” in a midlife crisis moment, “think I'm going to work for a startup.” You've done exactly the opposite which is kind of intriguing.

Yes and I think people somewhat follow the path that's put in front of them and then pivot and get smarter about what they want to do and I like leading organizations. I like seeing them compete and thrive with really good technology, really good data and really good collaborative practices. I've looked at my career as sort of doing it in three different spots. And so it's been fun.

From that point of view, I've got a kind of massively philosophical method question for you first, which is: so essentially DevOps, you say that's one of the areas that you specialize in. When you work in that kind of broader enterprise context, does DevOps actually exist? Given the fact that we're talking about innovation, we're talking about agility, is it a manifestation of that stuff or is it just kind of get on and do this stuff? It's something specific or is it a general thing?

I think it's specific, but I think what it means really changes and is different depending on the type of organization that you're running, the platforms that you're doing, really also the end users that you're targeting. So let's break that down a little bit. So I think what I find most organizations having to do is do a lot more applications, deploy applications faster, some of them are more internally focused, some of them are externally focused, some of them are very data rich, some of them are design focused. And so some of those are the nuances that go into what the business is about.

But I haven't met a single IT leader, CIO digital leader, that comes up and says "oh yeah things are easy. I have plenty of resources and skills and there's no demanding requirements. You know I could just do my work sprint to sprint and everybody's happy." Everybody has too much work to do. There's too much customer demand, there's too much technology. I also don't hear CIOs coming to me and saying “You know what? We've got this legacy stuff and we've got all these new applications to do. Our budgets are significantly increasing so that we can do both.”

Neither of those [scenarios] are happening, and so the cliche is to do more with less. And I think there's a certain reality there. We're doing more app dev; we're doing things that require higher service levels; the security posture is more threatening today than it ever was and we're not getting necessarily any budgets to go and do this at greater scale or breadth upfront. We have to build that up over time. And so I think that's a lot of where DevOps comes to play, right?

It comes into: how does the application team get to deploy things more reliably, more frequently and at scale? How does the operating team collaborate with them and have a stable set of infrastructure to deploy it to? And where the two come together really is around the different types of automation that you can put in place. What level, what expertise, what level of maturity you put that in place really depends on the type of application, the type of risk factors and what the end users need.

And so I've worked with some companies that are doing relatively low risk customer facing type of products and so doing very fast automations that do deployments every few days, if not every day, are very realistic for an organization like that when they're dealing with new technologies. And then I've worked with others that are doing a lot of B2B, a lot of workflow oriented applications, high volumes of data and user experience that requires significant training whenever there's a change to a workflow. And so those organizations still need automation. They need automation at the data level, at the testing level, and maybe they don't need to do deployments as frequently as a B2C startup. And so those are some of the things.

I'm going to say that’s going to cause them literal business change, and you don't want to be doing that every day.

Exactly. That's right.

Really. So essentially if I can summarize what you were just saying, it's not that DevOps is hard in the enterprise necessarily, it's that everything is hard in the enterprise because of the nature of the enterprise [is] doing more with less. There's loads of pressures. It's that there's not enough money to go around, etc. etc. And then equally, DevOps is the solution to some of those pressures. But then it faces exactly the same pressures as anything else you might see.

Yeah I think I would say that. I would also say that the hard thing about enterprises is really picking the types of things to do and what to focus on and what level of focus to put into it. And that's what I think [is] making enterprise workers a little bit more challenging, right? So you know in a startup, I think it's a little bit easier to say, “We're starting with a brand new stack. We're going cloud native; we're going to build up CICD pipelines from the ground up; we're going to put all of our testing into unit testing; these are going to be our development standards and we will evolve them over time as the complexity increases.”

[That’s a] relatively easy way to start with DevOps. When you are working in an enterprise, right? You might have an application that has a mix of new and old technologies. The old technologies may not have built-in testing frameworks. We might have to ‘lift and shift’ those components into the cloud. We might be able to modernize other areas particularly around the user experience and we have to think about the volume of activity in the workflow around those applications from day one.

And so how to orchestrate that is really the decisions that enterprise has to go through and it requires some strategy. It requires some thinking around it, and it requires some coordination you know. So an enterprise that decides to go a full pedal into CICD and doesn't think through the testing ramifications that may not work so well compared to a startup that can fail faster with a testing of a new application and adjust their testing strategy as they learn more.

So as a matter of interest, something that you also cover a lot is digital transformation. Is using DevOp's path of doing do it? I hate to use this term but I can't think of a better one right now: doing this transformation? Is it a symptom? Is it a cause? Is it something that runs in parallel? Or are they just different manifestations of the same progress from the enterprise perspective?

Yeah I would describe it as something that needs to run in parallel. I think the best way to see that is in a digital transformation effort, you're going to pick certain strategic programs that often require doing things in Agile, putting some new platforms out, developing new user experiences. And so you know early on that development team is able to move pretty quickly because everything is ground floor and easy to work with and then as Agile continues on, that team thinks about how to take what it's doing out of dev and move it into testing and moving into production and then later on supporting it in production.

And so what you find if you do not invest in DevOps in parallel to that is that over time that Agile team is spending less and less time on application development and spending more time on the operating side. It's because they haven't automated deployments; they haven't automated testing; they haven't used infrastructure as code to set up their cloud environments. And so instead of you know 80-100% of their time being spent on user experience, you see that percentage dropping off to 60-50-30% as they are having to do what operating teams have to do: manage clouds, manage automation, deployments and things like that, manage response to incidents.

And so I think DevOps needs to be running in parallel to those application development efforts. I think the teams need to think through what level of automation is needed when it needs to be introduced and how to orchestrate those things. And so if those are done correctly, you end up with sort of the best of all three worlds. You have an Agile team working with new platforms. You have it built up in a cloud native environment with the orchestration to set up new environments when you need them, and you have some semblance of deployment integration and testing happening in an automated way. So if you don't do that in digital transformation, essentially what you've done is built another application that becomes another legacy system. It just happens to be on more modernized platforms and maybe running in the cloud.

That's really interesting. So those three things: you've got orchestration, automation and then Agile process. Did I get that right?

Yes.

And what I have seen, I've spoken to companies in certain customer facing sectors let's say (I won't mention any names here), that got part of that right and ended up in exactly the situation that you're talking about. And essentially the DevOps team was less and less able to deliver over time to the extent that it was less and less able to bring value to the organization. And in the end a lot of the projects were wound up because they just weren't delivering as efficiently and therefore, why do it if you're not delivering efficiently? Might as well not bother at all.

Yeah let's dive into that since we're talking to enterprises here, right? What makes an application become a legacy system over time? And it's really over time that application becomes more complex. It has more complexity into its architecture; it has more technical depth. It requires more steps to deploy, more steps to test and then over time, not only do you have that complexity, but you probably have your star people moving on to other applications and you have some attrition in that development group and so you're left with a system that fewer people know how to deploy and fewer people know how to test, and an architecture that has newer capabilities to go out and address technical debt with. That's what makes a legacy system.

Now if you were building up standard automated ways to do your deployments, standard automated ways to do testing and a healthy way of introducing technical debt priorities on top of your business priorities, then you're less likely to see that application become technical debt because as soon as somebody needs to do a deployment there's already a road paved for them to test it, and there's already a road paved for them to deploy it. There's already a standard way to deploy that infrastructure, and to me that's really what separates a legacy application from a modernized one.

I'm trying to get this straight in my head because I think that's the core of this very podcast if I may. This is going to be going in the summary: it’s about not how to stop DevOps things (that’s so hard), but how to stop DevOps becoming just like any other development that delivers what's going to ultimately result in just being yet another legacy system. And it's that wrapping of inefficiency that legacy systems have as you say less people, more effort required just to do any single deployment, and those two factors kind of cross on the diagram. We've got loads of diagrams. There is an imaginary PowerPoint deck in my head with the triangle on it. "Now we've got a picture with crossing lines and there's gonna be a little kind of jaggedy thing in the middle of that, where you know when those lines cross, that's when things get really tough." And so that's what we're trying to avoid here.

So if you're going into an organization, I mean I don't know how this works, but essentially as you look at an organization and look at all the different aspects of that organization, what are you looking for in order to (a) help them deliver the success side of things and then (b) not do the bad, the stupid... not kind of make the obvious, well if you don't have that role or if you don't do ABC, you're already heading in the wrong direction. How do you start to understand that to engage the organization to resolve that?

Yeah it's a great question. Let me talk first in some general practices, some things that I recommend and talk about in my book Driving Digital and then I cover on my blog. I think when you look at enterprises, the reason it's called Dev-Ops actually, I think, is you know this notion that you can develop on the left side and then hand over the application to support to the operating teams on the right side. So the first fundamental thing is that that practice has to break, right? Those operating teams need to be part of the development process as applications are being developed and upgraded. And that doesn't necessarily mean they have to be on the same Agile team or that you have to realign and restructure your organization and change your reporting structure. It means that they need to be part of the process; it means that they need to learn some new skills. Right?

So whatever your CICD tools are going to be, whatever your continuous testing tools, whatever tools you're going to use for infrastructure as code, those are all new skills and technologies that your operating teams need to learn, but they need to be evolving what's being developed in parallel to your application teams building it. And so that's a fundamental difference that I think enterprises have put into their practice and their mindset and their collaboration that's different. Now how they go about doing it is really dependent on circumstance.

And so let me give you a couple of examples: I worked with one university [that was] used to doing things very separately. They asked me to run a workshop for them. They wanted to do some piloting around CICD. I found out very quickly that they weren't inviting their operating teams to the table. The operating teams were already experimenting with Puppet. The development teams wanted to experiment with Jenkins and I said, "Look guys if we're going to do a workshop, we've got to bring both teams together. We have to be operating as one. We have to think through what applications we want to start with. We have to define what success needs to look like in this pilot. And we have to decide which types of stacks to focus on."

So we picked two stacks: one focused on testing, one focused on infrastructure and piloting infrastructure as code. We defined success criteria and we worked through a few month pilot. Why did we go down that path? Because the organization was very distributed. They needed to show that the investment in these technologies had value; they had to show that there was a new form of collaboration their data development and operating teams could have, and so they wanted to start small and show collaboration and success with automation as a starting point.

It's very different from another one of my organizations that is on a rapid timeframe to move to the cloud for various reasons. They want to get that done on a tight timeframe. And I said to [that organization]: “You know, let's talk about what's going to make that migration successful.” What we came away with was “well, we have a huge volume of workflow mix of customer and internal workflow that goes onto these applications. We don't really have a good way to test this thing.” And so even though their initial work was around deployment, we're shifting focus and doing a lot of work around how do we build end-to-end testing so that when an application, a workflow, an API or a service is moved into the cloud, that has a stable way of testing and doing an AB analysis to make sure that it's still performing well in the new cloud environments. So two different ways—both required bringing teams together; both required some prioritization in terms of what to focus on, and both will require road mapping to bring the other disciplines in place over time.

And I'm sorry but I've just invented a new word in the middle of you saying that... One of the things we've been talking about with a lot of people, is DevOps practice seems to be quite heavy on the Dev and not very heavy on the Ops. So I've spoken to Andi Mann at Splunk about how it's like DEVops, and what it needs to be is kind of devOPS, and bringing those things together. The new word for today is “DOPEVS” because it's actually about putting the Ops inside the Devs. I hope you appreciate that.

I do. Andi and I have talked about that a few times, and I referred to him in a blog post I did (this was probably like three years ago) and the image of it is it says “seize the day” and it's really about Ops really understanding and working with more of these technologies than what Dev is doing. It means they have to understand automation, programming, coding, more than they've ever done before. And the reason again is if I have whether it's two developers or 200 developers, I want them focused on moving the business forward. That's you know around data and analytics and user experiences, and that means I really need the Ops teams to pick up the automation, particularly around deployment, around testing, around infrastructure and yes, that's the new skill set for them to do. But that's essentially where I want the alignment in the organizations that I work with.

And that fits with one of the other things you mentioned right up front was around how to make organizations more data driven, so being able to feed data driven-ness into the DevOps cycle as well which fits with everything you're saying around success criteria and then being able to measure that stuff.

Yeah. I mean think about when that application is now in production and you're doing iterative and frequent deployments. One of the disciplines inside DevOps that isn't talked about enough is the monitoring side and the ability to get really good feedback in terms of: maybe a deployment didn't work as well as you thought it was gonna work, and can you find that problem quickly and find root cost to it efficiently? That's part of this, and the question is not only can you do it quickly and efficiently, but who's doing it, right?

If you have to call your developers in every time there's an operating issue that's lost productivity when they're looking into root cause. So monitoring for issues and incidences is a big factor, but also monitoring for capacity and understanding when things need to scale up and scale down. You have all this cloud infrastructure that requires tuning around its parameters and elasticity and giving the tools for the operating groups to understand how to do that well and efficiently and to build the rules in for that, that's the new tool kits that we have to provide our operating world. And that's a whole other level of collaboration that needs to happen as applications are built.

So two questions to round off with, which I'd normally just ask the one. I'm intrigued by what you think about the second one. So the first question: Not everyone's gonna be able to access Isaac Sacolick, so if you were to kind of give people advice on how to take their own organization forward, what would you say on this podcast [about] where they should start looking, where they should start addressing…. how they should take those first steps?

Yeah. So I would say a few things. First is and this is going to sound a little bit cliche, but you need to define your DevOps roadmap based on real true business needs. And so DevOps is a very technical practice. It's very easy for the technologist to read material generically and want again, to do deployments daily if not hourly in some cases that may be well above what the operating needs and the business needs are, and so you have to take a step back and say ”What do we really need from this thing? What's going to move the needle for us and get strategic around that first and use that as a guidepost?”

I said I think that's the first thing, I think the second thing is what we covered around ‘letting ops do ops.’ And so it's very easy for the developers to leave the world of moving the business forward and entered the world of making systems perform better. I think that's a dangerous step to allow that to happen to too quickly and too frequently. And so I think a lot of it is letting operating teams learn the new skill sets, partnering when required and when there's advantages to do that, but let the operating teams do their job. And I think that's a big part of it.

And I think the third thing is: if I were to pick one place to start, that I see people leaving behind. I really like teams starting out with testing. That's counterintuitive from you know focusing on CICD or focusing on infrastructure as code. It's very hard to do things reliably and faster without having some test automation in place, and so you'll see a lot of my writing focus on the test automation and testing practices because I think it sort of is talked about second and third, compared to the automation built into deployment and infrastructure. So those are the three things that I would advise people to look at.

That's awesome. And the other question I was going to ask is: How will those things change over time and how do you think DevOps will change over time? Will it just become... will we still be having these conversations for organizations that are adopting agility best practices in 10 years time, or will it just be intrinsic to the way we're working at that point or will we have just automated the heck out of everything and so we don't talk about it anymore? How do you see things moving?

Well I think the things that we're doing today will become easier. There'll be more tools around it. There'll be better skills around it. And then there'll be a new set of emerging things that you know depending what our strategy and business needs are going to be the emerging things we need to focus on. So I could do automated deployments 10 years ago on my Red Hat environments, using Tomcat, using basic shell scripting and I could do a full deployment. And it was hard, and you didn't find many organizations to do that, that did that.

Today you'll find there's a half a dozen platforms that can automate deployment. The scripting is relatively well understood. There's plug ins for all the major environments whether it's infrastructure or applications. There is a learning curve with it but it's far more accessible to do automation there. If you're telling me we're going to do automated machine learning deployments, models at scale, and push that code to the edge—so a mix of edge computing and cloud computing—and do this on IoT devices as well, we're going to find that’s not as easy, requires a little bit more orchestration and skill set to be able to do those types of things.

And then in you know in 5 years, that will become easier and there'll be a new set of things that, depending on the boundaries your organization is pushing, that you'll have to learn and compete with. That's the nature of technology and I think it's playing out in the cloud space in the development space and in the data DevOps space as well.

Well that's absolutely awesome. And the linking right back to what you were saying at the very beginning, which is: how we can make this just not turn into this next level of legacy if you like. I think that's what I've really taken away from this podcast. Thank you very much for that and I'm not just saying, “hey get with the program, everyone starts doing it,” which is, as I said before, it's easy to say and hard to do.

I'm delighted to speak to you Isaac today and thank you so much for your time. We'll certainly put some links up for these articles and blogs that you've been mentioning, so that people can read those as well. Thank you.

Awesome. Great meeting everyone, and have a great day, and thanks Jon for having me.