Given the buildup regarding edge computing, what should organizations do about their current and future network strategy? How can network teams create or tune their plan to effectively meet these emerging challenges with appropriate efforts in connectivity, network performance, agility, and cost? By adopting a modern cloud NetOps framework, enterprises can eliminate data and networking silos, build for state streaming, incorporate APIs, and leverage advanced data analytics, all while automating network management. Thankfully, technologies and vendors are providing new ways to keep pace with these changes.

In this report, we find:

• Effective network strategy must flow from the overall business goals, and be flexible enough to change as they move.
• Enterprises should understand their national and global footprints from a physical and routing perspective while anticipating ongoing change.
• Networking teams should fully leverage Software-Defined Networking (SDN) as well as implement and plan for Intent-Based Networking (IBN) and Software-Defined Perimeter (SDP).
• Zero Trust network design adds resilience to large and agile networks for less cost. However, legacy networks and data centers still require a strong network perimeter with all the legacy security controls. Consider modernizing these older environments prior to adopting SDN.
• Effective monitoring is essential for situational awareness and decision making. Time-based, operational databases are a powerful solution.
• These key technologies (SDN, IBN, and SDP) are important enablers of mobile and IoT applications.

The post Taking Network Strategy to the Edge appeared first on Gigaom.

Steve speaks with guest Eric Dynowski on helping organizations implement strategy and infrastructure design.

Guest

Eric Dynowski is the chief technology officer at ServerCentral Turing Group (SCTG), which offers cloud-native software development, AWS consulting, cloud infrastructure and global data center services. Eric has a comprehensive background in infrastructure design and integration (including business mergers, spinoffs, acquisitions, and start-ups), and has been credited with reducing IT budgets by millions while adding inherent value to existing departments by focusing on excellence in staff and deployment. At SCTG, he helps Fortune 500s and public enterprises leverage emerging technologies to scale, modernize, optimize, and manage their IT infrastructure.

Transcript

Steve Ginsberg: Hi I‘m Steve Ginsberg, my guest today is Eric Dynowski. Managing partner and chief technology officer at ServerCentral Turing Group. He has a background in tech and helps organizations optimize infrastructure design, strategy and implementation.

Eric, thanks very much for joining me today.

Eric Dynowski: No problem, glad to be on, Steve.

In looking at your background, one of your specialties is infrastructure as a service (IaaS). There’s obviously many ways to approach this. I’m wondering how you’re seeing the solutions play out here.

Yeah, ‘many ways to approach it’ is definitely a good, succinct description. I think that we’re seeing quite a broad range of requests coming from our customers. The one thing I think that’s common across the board in most cases is, our customers are no longer interested in managing their own infrastructure. That’s a pretty broad statement.

Infrastructure could just mean, hey, I don’t really want to be responsible for providing power, cooling, and base networking, but we want to manage our OS. Whereas, other customers say, “Hey, I don’t even want to source my hardware. I want you to provide it and lease it.” Others will say, “I don’t even want to hear the word ‘hardware.’. I just want instances to put my software on and go from there.” Then another group of folks will say, “I don’t even want to hear the word ‘instances.’ I just want to deploy my software somewhere, and containers comes into the discussion.” It’s a pretty broad mix.

I would say that across our entire customer base, our experience has been: there’s no single answer yet that fits any company that’s coming at us. It really depends on their business. It really depends on their workload. It depends on what their business objectives are. Some companies are price sensitive, and other ones are not. They have different drivers. I think that there’s still a role for the major hyper-scaler providers, like Amazon, Azure, and Google. At the same time, there’s still a place for VMware. There’s a place for Hashi[Corp]. There’s a place for all the various Kubernetes solutions. There’s a place still for bare metal.

It’s an interesting time in the environment. Things are evolving quickly, especially in the container space. That’s a rapidly evolving space. I think Cloud Native Foundation’s got a great little map of all the pieces of software that’s out there right now. It’s 150 different components, and they’re all under massive development right now. This space is just exploding. People are doing things in containers that two years ago you would have laughed at. Yeah, it’s a dynamic space right now.

What are some of the more exciting parts on container application deployments that you’re seeing? I’m guessing most folks are going because they want to have microservices that can scale on their own, and less monolithic infrastructures. You said ‘to laugh at’—are there novel uses or particularly excellent uses that you’re seeing?

Yeah, I think customers are definitely looking at containers for those reasons. The microservices architecture is actually—it’s been around for a little while, and people have tried many different strategies, in terms of how they want to host it and run it. Containers were at the forefront of that. I think some of the stuff that’s driving containers today is less that, oh, we want a microservices architecture, so we’re going to go to a container. It’s that we have a microservices architecture, and we’re looking for a cloud-agnostic or cloud-native way of running it, meaning: we want to be able to run it on Amazon. We want to run it on Azure. We want to run it on Google. We want to run it on bare metal. We want to run it on our own KA platform. Originally, the things that you saw in containers were stateless apps that didn’t store data and didn’t need to have access to any local data. They’d connect to a database that was running somewhere in a nice database cluster, a cache layer or an API, or things of that nature.

What we’re seeing now is: customers like what containers are doing for them and how they can move those workloads around, how they can orchestrate them, and are pushing out further beyond just stateless applications. They’re coming back to us and saying, “I want to run my database in a container. I want to run my queuing application in a container. I want to run Elasticsearch in a container, and a Redis cache in a container,” and things like that, which require persistent storage, which was never part of the original container idea. The rule of thumb was: if it used local files or needed access to the file system, you don’t want to do that.

I think the question is: Is the database a good idea in a container for those reasons?

Yeah, and I think a few years ago, I would have said “absolutely not. That’s not a good idea.” Where things are going today, the answer is: Yeah, that’s worth considering. We could think about that and think about what the best way to deploy it is.

Our customers are saying, “We want to run a database in a container, and we want to move it between clusters. We want to have resiliency.” What’s cool is that what’s happening in the container space right now is that there’s a ton of software and different pieces under rapid development that are enabling that to happen. Containers and serverless now are turning into a stack where you can run every aspect of an application. A few years ago, we would have laughed at you and said, “Don’t try that. That’s really risky. You’re asking for trouble.” Today, we’re like, okay let’s think about that. Let’s see if we can come up with a solution.

Are you seeing that play out very predominantly in Kubernetes? Or are your customers also using some of the other container options?

I would say 99 percent is all Kubernetes. There’s some people running Docker Swarm and a handful of other things from other third parties. I would say predominantly today, everything’s coalescing on some flavor of Kubernetes, whether it’s on top of OpenShift, whether it’s in one of the public clouds like GKE, Azure’s Solution, or Amazon’s, or Rancher, or Pivotal, or things of that nature. Everything is coalescing around some form of a Kubernetes core.

Yeah, makes a lot of sense. That’s a lot of what we’re seeing as well. I had noticed that you came up with a bunch of experience in the financial exchange connectivity area with trading systems. I’m wondering what learnings you might think from there are relevant more widely to enterprise architects, especially in light of edge agendas.

I think that space has changed significantly as well, I would say, in the last ten years and what we see the finance industry doing. Ten or 15 years ago, hedge funds were writing their own custom applications. Everything was proprietary. It was latency-sensitive. Applications were being written in C++ dedicated servers, things of that nature. Some might argue in some ways, they were ahead on the technology curve, especially if they were in the high frequency trading space, and they were really pushing the limits of what networking is capable of and what servers were capable of. You were trying to eke every microsecond. I think that a few things have happened since then that have actually put the finance industry in a position where they’re lagging behind other enterprises.

They developed this mentality of running dedicated servers and having ridiculous requirements for performance and availability, and really stuck to their guns with that, whereas the rest of the enterprise moved on and started containerizing applications, going to microservices, and designing software to tolerate downtime. The idea that everything goes down all the time, so instead of trying to build redundant hardware and build redundant solutions that way, let’s build software that can handle failures.

What we’re seeing now is that a lot of these financial firms are realizing that, hey, we’re actually ten years behind where a lot of the upfront internet companies are today. It’s okay for us to start to adopt those technologies. It’s probably akin to how oftentimes, people feel the banking industry is behind in terms of their technology. They’re still using mainframes. There’s this feeling that we need to rely on really resilient hardware.

Even some of the container conversations we just had are—those conversations are coming from financial firms that are approaching us saying, “We’re starting to containerize our applications. We really like what’s happening, and we want these other features and functionality that we’re used to.” I think from that end, the financial services firms are only starting to get caught up. They’re only starting now to say, “You know what? It’s okay to have an infrastructure that’s in the cloud. That extra ten milliseconds of latency doesn’t matter anymore.” I think those are some of the changes we’re seeing there.

Sure. Do you think there are particular points that they should take advantage of regarding security? We’ve been looking at some of the recent developments. We’re seeing reports of implementation problems causing tens of thousands, if not millions, of exploitable points in the cloud infrastructures in general.

Do you think financial companies are going to be up for that challenge? If so, anything in particular you’re noticing to help them leverage, to do a better job in these implementations?

Yeah, personally, I think the cloud solutions, things like AWS, Azure, and Google, AWS specifically actually makes security easier than not. In a traditional environment, especially if you’re managing the data center, there’s so much more ground-level work that you have to handle, take care of, and focus on. Whereas if you’re jumping into one of the public cloud providers or a managed solution where your provider’s handling a lot of the low-level security pieces for you, it offloads a lot of that.

For example, if we spin up an environment in AWS today, we have audit trails built in automatically. Every single change is documented. Every single connection attempt is documented. There’s an auto log that’s immutable and we ship to a different account. A lot of the basic requirements for running a SOC 2 or ISO standard compliant environment are built in by default, and you can’t avoid them, whereas traditional environments, you’ve got to build all the processes, policies, and procedures around that. In many cases, they’re quite manual. Someone’s got to go and update a log somewhere. There’s a manual change control process in place, whereas that stuff was thought about and built into the public cloud platforms. I think in those ways, it makes it easier.

At the same time, because the public Cloud providers make it easy to deploy resources on the internet live in minutes with a credit card, if you’re not thoughtful in what you’re doing, it’s very easy to put things up that are insecure. We’ve seen that time and time again. I can’t tell you how many articles I’ve read about how somebody put a whole bunch of confidential records on an S3 bucket that wasn’t protected. In some way, I feel like the responsibility lies on the consumer of the cloud service that misconfigured it. Also, I feel like there’s a little bit of responsibility on the cloud providers there. Why are you defaulting to a position of ‘open access by default’ rather than: let’s lock everything down and force the customer to open things up as needed?

I think in the last year or two, we’ve seen that shift. If you provision an S3 bucket now in AWS, they really make you jump through a lot of hoops to make it publicly available. That being said, I think really, the message for financial services is that, hey, if you’re going to start using the cloud, make sure you’ve got a good strategy. Make sure you have a good governance model. Make sure you’ve got some experts in place that are going to be there to audit what you’re doing within the cloud. Go through regular security audits. Just because you’re using the cloud doesn’t mean you get to skip all that stuff. The great part is you can automate a lot of it. You can automate security checks. You can automate security scans. The APIs are great. You can automate the analysis and processing of logs, which were things that were much more difficult to do before.

In this hybrid world that we’re talking about – taking advantage of containers for this as well, are your customers architecting well to make partner API connections as they have increasingly business partners as part of their supply chain?

Yes. Here at ServerCentral Turing Group, we’ve got multiple lines of business. We see the conversation around APIs pretty much across every spectrum of our lines of business. For example, on our software development site, our customers are coming to us and saying, “Hey, help us build an API around our product. Help us build an API around our data so that our customers can integrate with us.” That’s a common thread that we see across the board.

Secondly, the customers that are approaching us for our managed services, or infrastructure services, and our cloud services are all saying, “We need all of those services to have some form of API interaction. We don’t want to manually provision anything. We want to use configuration management tools. We want to allow our applications to provision infrastructure and shut it down.” I think that the API thing is a no-brainer. It’s all over the place. Any new product that we build, any new service that we offer, an API-first approach with it is an absolute requirement. Yeah, that’s a no-brainer.

As the world fans out more widely, are you seeing true edge data centers playing out in your customer strategies? Are people looking for thousands of locations? Or are we really just more in the era of seeing people take advantage of some availability zones and a little bit of geographic diversity?

To be completely frank, I think we’re in an era of a lot of edge hype.

Fair enough.

Yeah, I think when we talk about edge, there’s probably several different dimensions or ways to talk about it. I think some are more interesting than others. Maybe I’ll split those up, and then we can let the conversation follow that. I think the first way to talk about it is getting content closer to the consumer. I think that’s, a lot of times, the conclusion that most people jump to when you talk about edge. Okay, I want to make sure my images and my video are as close as possible to the consumer so that my load times are as minimal as possible. I’m not paying for transit across expensive links.. Yes, customers are concerned about that, but I think it’s a problem solved. I don’t think it’s a problem that will be enhanced in any way by having a data center or something that resembles a micro data center every ten miles.

If my data center is in Chicago and my customers are in Milwaukee, that’s probably just fine. That additional latency to get up to Milwaukee, in most cases, is really not going to be a massive challenge. Where we see that content distribution problem is really more of a regional thing. It’s more, are your customers in the Midwest? Your content should be in the Midwest. Are your customers on the West Coast? Your content should be on the West Coast. Are your customers in the UK? Your content should be in the UK. It’s a regional problem. I think it’s a solved problem today.

There are some other underlying issues, especially when it comes to video, that we’ve gotten really good at, that matter, that are less about how fast is the content available and have more to do with latency. Peering and having quality bandwidth and connections between all the major providers, and being able to provide low-latency paths across backhaul networks is important. We do see quite a bit of that.

On the flip side of that as well – the other thing edge does is, it has this property where it aggregates and de-aggregates traffic, and also provides a caching layer. Another reason to have the CDN is not necessarily to bring your content closer to your consumer. It’s actually to serve the content to the consumer and save your back-end from having to do that. We’d limit the number of database queries we have. We maybe limit the number of times a particular file’s being served from a single location. It relieves a load on your back-end infrastructure. That’s a totally legitimate need. As you build highly scalable distributed systems, edge locations play a key role in doing that. Again, I don’t think it’s like, oh, I need a microdata center every ten miles to achieve that. I think you can achieve that through multiple providers and region-based data centers.

We’re doing some writing at GigaOm and coming to these same conclusions, that delivery has been an ongoing process. As you mentioned, for these types of content, anything, especially a lot of this stuff will buffer anyways. You’re talking video – it will buffer from there.

Then some of the applications that you would most want a near response – say, for example, people talk about self-driving cars. You’re not really going to trust much of that beyond… because your car can’t – if it loses connectivity, it can’t be waiting for that. There’s a whole class of things where, it might be useful, but it won’t be trusted. That has to be on the local device.

I think there will be some room in smart building, manufacturing, farming, and things like that where there might be some interesting edge applications, but I agree with your sense that a lot of it is hype currently.

That’s a great lead-in to my last point about edge, Steve. I think the more interesting thing that’s happening with edge, that still has yet to develop, and I spend a lot of time thinking about – hey, what does the internet look like in a post-Amazon, Azure, and Google heyday? We had this massive evolution of moving to public cloud, infrastructure as a service, automation, and consumption of that stuff. What’s the next thing look like?

I think one of the things, in terms of edge, that people oftentimes don’t think about is: right now, all the conversations are about the content. We’re not actually talking about the computing so much. What we’re starting to see, actually, with the deployment of a lot of IoT devices, whether they’re security cameras, locks, things like Alexa, Google Home, and all these other products – there’s a lot of heavy lifting that has to happen locally on that device for those things to be functional.

Also, there’s a lot more heavy lifting that’s just being moved into those devices because we can. For example, take Amazon’s Cloud Cam. You can set it up to get alerts when a person, a human being, enters the frame. The cameras are smart enough to differentiate between the human body and a dog. All that’s happening locally on the camera, which means there’s a lot of compute power in that camera. There’s a ton of compute power. A Tesla’s a super computer on wheels. There’s probably more compute power in a Tesla than there was in an entire data center in 1992. There’s a ton of compute power in your pocket. I think that when we talk about edge computing, I’m fascinated by the idea of: if we can come up with distributed computing models to leverage the spare cycles that are sitting on all these devices, we’ve been able to shrink CPUs and integrated circuits down to such a small level and put so much computing power in these small devices.

There’s so many of them floating around out there. Wireless technologies are evolving quickly. We’ve got 5G on the horizon. We always think about those technologies, about pushing data down to the customer. How fast can they push it down? What if we reverse that paradigm, and leverage that wireless link to push data to a device for processing purposes or serving purposes? I think there’s some really interesting opportunities and evolution of the Internet as a whole if we can take advantage of all of these supercomputers, effectively, that are sitting out there in people’s pockets, parked in parking lots, and hanging from the side of buildings, and distribute compute capability. I think in the sense of edge computing, in the truest sense of compute and processing rather than content serving, I think there’s an interesting thing that could happen there in the next 10, 15 years.

We might actually see, essentially, a P2P computing edge, hopefully versus a bot-net attack, which is what we see –

That’s right. That’s right. Yeah, if you look at things like Ethereum. Ethereum is making – not the cryptocurrency itself, but the platform for distributed computing – it’s in the very, very early stages of development today. It’s effectively a research project at its best. I think some of the potential for what’s happening there is quite interesting, and maybe it’s not for free. Maybe you lease the spare cycles on your phone back to a major provider, in the same way that, hey, when I have solar cells on my house and I have excess power, I sell it back to the provider. I think there’s some really interesting things that can happen there.

Yeah, I think that is really an interesting horizon. I’m wondering with what we’re seeing offerings from public clouds almost arriving daily, even from just the most major cloud providers, and then of course, there’s more specialized clouds beyond that.

How are you advising your enterprise customers to create good strategy, to maybe not get too caught up with the shiny object, but also to make sure that they’re really taking advantage of the good offerings as they’re coming out, whether that be serverless offerings, or as you mentioned, different ways to store data, these type of things?

Yeah, I hate to sound too consultant-y, but usually, most of our engagements start with: what are you trying to do, and why? Then we’re going to tailor our answer based on what some of the goals and objectives are in what they’re trying to do. There’s definitely cases where, yeah, you should build a serverless application on top of Lambda, and other cases where, no, you should not do that, even though it would work. The drivers for that can be both technical; they can be business. They can be financial.

I think that in each of those cases, we really start from that consultative standpoint of understanding what the customer’s trying to accomplish, what their business is, where they’re heading, and really, what the driver is. A great example of that is: we just recently had a conversation with a major dot com software provider that provides productivity software on the Internet. They had a huge push to move to GKE and move everything over. Now they’re pulling it out for various reasons.

I went in and I asked them, “Why did you guys do that? What was the driver behind it?” (thinking that, oh, they wanted performance, or scalability, or they wanted to save costs – that’s a huge piece oftentimes with many companies. They think, oh, if we move to AWS or Google, we can leverage things more efficiently and save money.) No, the answer was: “We needed to enable more rapid iteration within our development group, improve deployment pipelines,” and things of that nature. It wasn’t about scale. It wasn’t about cost. It wasn’t about anything other than flexibility and leveraging an API-first mentality. They were willing to pay two to three times more for their infrastructure because the payback to release new features and keep their customers happy was well worth it. Really, short answer is, why are you trying to do this? What are you trying to do? Then we’ll look at the field of technologies available and come up with something that fits in that very specific use case that that customer’s going to be coming at us with.

One of the ones that you touched on there is serverless. I think for some of our listeners, they might be struggling with even the basics of why they would consider serverless versus not. Obviously, a lot of the considerations you alluded to would be customer-specific. I’m wondering if there’s a bit of an Occam’s razor you might mention of why you would direct customers to start to look at serverless versus not.

Yeah, I think usually the first one is: if you’ve got short-running processes that don’t require a lot of compute, and you don’t want to set up dedicated resources to run them, is definitely a good scenario there because you’re only paying per millisecond of execute time. This is a small process that runs for about a minute every hour. Back in the old days, you set up a server. You set up a chron job or some type of job scheduling system, you ran it, and that server stayed up and ran the entire time. I think in those types of cases, it’s good.

If you’re dealing with any kind of event handling where the event streams are infrequent and you need to do some data massaging or manipulation of the data, they’re really nice. If you’re building a microservices architecture, a lot of the serverless technologies out there available today are pretty amazing. You can build quite a bit and not have to worry about managing the OS, deployment configurations, and things like that.

If you don’t want to manage an internal dev ops and IT team, there’s reasons to consider it. Yeah, I think if there’s particular proprietary services that one of the cloud service providers is offering and you want to leverage them, AWS specifically has awesome integration with Lambda with almost all of their other services. You can roll out solutions to things rather quickly, whereas before, you had to build an entire server and supporting framework to run an application. Today, you can integrate with an event stream and do all these sorts of things. You can do it in a matter of hours or days, depending on your experience level, and not worry about all the other underlying infrastructure – downside is you get locked into a proprietary platform.

This could be the answer to my next question, which was a little bit about looking at the edge offerings from some of the CSPs as they’re starting to push their stacks down. Your comments about whether enterprises need edge are well taken. Say a company wants a – maybe they’re in manufacturing and they need to run local facilities or something similar. How do you feel about the AWS stack, the Azure stack, Google stacks that are moving down to bring that…?

Yeah, I think it’s an interesting space. I think that some of the concerns, when you lock in on the public cloud side of it is, well, my whole solution is based around Amazon’s API, for example. Yeah, I love it, but I need stuff in my data center locally, or I need it in the factory, or I need it in this building for whatever reason. Amazon recognizes that. All the cloud providers are recognizing it. Outpost is a solution to that.

I think the power of the CSPs is really that they have amazing APIs for their platforms, and that you can get services and infrastructure on demand. Really, the idea that I can continue to do that and have some of it locally for those particular use cases is awesome. I think it’s fantastic. I don’t have to then have a different API, or different interface, or different way to consume infrastructure locally.

I can use that same API that I’m used to in the public cloud that I’ve been using all along, but now I can provision stuff that’s sitting, for example, on the factory floor to run machinery or something like that, where you’ve got connectivity constraints and you can’t afford to be down. I think that it’s a good thing. VMware is doing it. You can get an entire stack from Dell that interfaces with the same way that you would on their public cloud. It just gives more options and more choices, so I think it’s good.

You mentioned VMware. Beyond the big three public cloud providers, how do you feel about some of the specialized clouds that are out there? Packet has some interesting capabilities. DigitalOcean seems to be very developer-focused. There are more beyond [that], whether they be Nvidia or VMware. How do you feel about some of the other clouds?

I think VMware’s still interesting, especially in the enterprise space. There’s still a lot of enterprise companies that leverage VMware third-party proprietary products, and really depend on them. There’s no answer for those things on top of any of the other platforms. As a service provider – we still need to have an answer for that, and there’s still a place for that.

I think from the VMware side, there’s still a lot of play. Also, VMware and Dell – they just acquired Hashi. There’s a lot of space happening on the container side. I think interesting things are going to happen there. There’s definitely a place for it. I think the same goes for Packet and DigitalOcean, but I would say that right now, the other three CSPs – their platforms are so much more mature. For things like Amazon, really what adds the value is the level of integration there is across all the different services that they sell.

Having the ability to send a message to a queue, trigger an auto-scaling event, and run a piece of custom software at the exact same time is pretty amazing. I think there’s a lot of catch-up there. Unless you have a specific need that isn’t answered by those other providers, or isn’t answered by the three main CSPs, but is answered by Packet or DigitalOcean, then they’re worth chatting about. Yeah, it just depends on why you’re looking at them.

Great, Eric. I really appreciate your answering my questions today. Thanks very much for joining us.

Anytime. Great questions, Steve, appreciate the time.

Take care. Thanks for listening, everyone.

Bye-bye.

The post CIO Speaks – Episode 11: A Conversation with Eric Dynowski of the ServerCentral Turing Group appeared first on Gigaom.

Steve Ginsberg speaks with David Chou about what enterprises and vendors can learn from his experience in the healthcare and lifestyle realms.

Guest

David Chou is a healthcare industry leader in the digital space. David is the VP, Principal Analyst of Silicon Valley based Constellation Research, Inc. Chou has held executive roles with the Cleveland Clinic, Children’s Mercy Hospital, University Of Mississippi Medical Center, AHMC Healthcare, Prime Healthcare, and is also advising many academic medical centers and healthcare start-ups.

David is a dynamic keynote speaker and industry commentator working with clients to transform their business models using technology. He has spoken around the world at healthcare tech-related conference including keynotes for leading industry events and intimate executive settings. Chou is also one of the most mentioned CIOs in the media and well quoted in outlets such as the Wall Street Journal, Modern Healthcare, HIMSS Media, ZDNet, CIO.com, Huffington Post, and Becker’s Healthcare. David is an active member of both ACHE and HIMSS while serving on the board for CHIME.

Transcript

Steve Ginsberg: Hi, David. Thanks for joining us today.

David Chou: Thanks for having me, excited to be here.

I wanted to ask you, having held both roles, how do you view the positions of CIO and Chief Digital Officer?

I view the CIO–I’ll call it the 2.0 version. They really need to think about digital first, whereas you’re starting to see a lot of organizations that have both a CIO and a Chief Digital Officer. I’m seeing a lot of that use case there, and that’s because the CIO has traditionally been viewed as a back office, keeping the lights on, making sure [that] infrastructure is ready, while the Chief Digital Officer is really trying to build new business policy utilizing the technology. I would say most aspiring CIOs [are] probably thinking they’re trying to fill both roles, more so than ever trying to become that Chief Digital Officer rather than the traditional brand of a Chief Information Officer.

How do you see the two roles working together?

‘Working together’–that’s a really tough one because unless you have a CIO with the personality of… they really want to focus on keeping lights on and are okay with the infrastructure, I don’t see the two aligning well in any organization. There’s usually some sort of hierarchy where either the CIO reports to the Chief Digital Officer or the Chief Digital Officer reports to the CIO.

When you see it in an organization where they’re peers or one above the other in the hierarchy, it doesn’t work out too well because there’s too much overlap. For example, the Chief Digital Officer needs the core foundational infrastructure that the CIO has traditionally been accountable for to be in tip-top shape to really drive these initiatives. Some of those foundational elements, if they’re not there–then the digital officer cannot drive and execute. I would say it really depends on the structure, but if they’re peers, it’s a really hard structure to work out well from an organization perspective.

That makes a lot of sense. Any advice from what you’ve seen on who’s better to have on the top of the food chain, whether it’s better for the CIO to report to the CDO or vice-versa?

I would say forget about the title. Let’s just focus on the personality and let’s focus on that individual who is aspiring to lead digital initiatives and that individual who has the aspiration to realize that their technology investment–they make that a competitive advantage versus other organizations. I would say if you have that talent and you have that aspiration for a single leader, then designate them to lead the digital initiatives regardless of the title.

If you don’t have that, I think that’s what happens when you see an organization go out there trying to recruit a Chief Digital Officer thinking that one individual miraculously is going to transform their organization where in reality, it does not happen that much because one, the individual’s not going to be able to do much and that one individual coming in, they’re probably not going to have a big team either, and they have to build it up. It’s a really hard dynamic for a lot of CEOs out there trying to just pull in this one miraculous individual to solve that digital challenge and drive the transformation.

That sounds like great advice. What learnings from healthcare do you think organizations should consider for organizations that are not in the healthcare field?

I would say the fact that healthcare is a 24×7 business, you really learn and take some of those lessons learned from that vertical. Obviously we’re dealing with people’s lives, so we cannot tolerate any mistakes. A mistake may actually have an impact on life or death. Taking some of those lessons learned, some of those initiatives, some of the approaches, I would say that may be an area to learn from healthcare.

It’s also highly regulated, so a lot of requirements that may or may not be applicable to other industries. On the flipside, I want to say healthcare can learn a lot from other industries as well, specifically in the retail sector, how retail drives the brand stickiness, consumer engagement. That’s really a big thing that’s happening in healthcare. I would say vice-versa; healthcare can also learn a lot from other verticals as well.

Is there something from the data technology specifically in that area that you feel is most promising coming out of healthcare or going the other way?

Data technologies, I would say are pretty consistent[ly] vertical-agnostic. Every organization, every vertical, they’re trying to figure out how to become data-driven. They want to figure out how can they use data to make decisions better versus trusting their gut intuition. I would say regardless of the vertical, everyone’s trying to focus on how to do that.

But I would say the retail side has done a better job in terms of understanding consumer behavior. That’s where healthcare as an industry moves towards the model of keeping patients out and promote wellness and drive a healthier lifestyle. You really have to understand this patient/staff/consumer a lot better, and that’s where the healthcare vertical can learn so much from a retail perspective as far as understanding its whole person approach.

Sounds like you are seeing healthcare companies starting to deploy that type of technology?

Yes, there are a lot of CRM technologies out there tailored toward healthcare. I would say most healthcare organizations are trying to figure out what is the right tool set to help them. There’s the challenge. Most health systems and healthcare providers, hospitals, they have on average between 450 applications to 1,000-plus. One of the things that really needs to be known is really application portfolio rationalization. They need to really look internally first to understand some of the overlaps and then understand which one of their current tools can be that platform or do they need to make another investment.

That’s a struggle that I have seen even personally as a CIO for various healthcare situations as well. We’re managing so many different things. Generally having the large enterprise platform on top, while it makes sense, is just challenging. I’ve seen lots of healthcare platforms out there to drive consumer engagement, to drive what we call population health, which is focusing on this whole person engagement and wellness. That’s where the future of healthcare is moving towards, and there are a lot of solutions for the buyers to really select and choose from.

It’s interesting. I think that organization sprawl in terms of technology is something that a lot of CIOs face across the board these days. Then I think organizational challenges can be for the CIO’s role some of the toughest pieces to actual changes in how the organizations themselves are structured. I saw that at the University of Mississippi Medical Center, you consolidated large organizations into a single one. I’m curious what were the motivations for that and then what were the more interesting parts of the process?

That was a very unique organization where they allowed me to consolidate the three verticals of healthcare: the health system, the hospitals and clinics; research, which is a big component of the academic medical center; and the higher education–to activate the technology over all three areas. I would say that is the right approach because when you look at other academic medical center systems, they usually have a CIO over each vertical.

What happens is at the top level, at the vice-chancellor as they call it or the chancellor layer, they allocate budgets equally or sometimes to all three verticals. Then you’re always competing for dollars to do stuff, rather than trying to create the synergy among all three areas from a technology aspect, or some places that even have it where one area, whether it’s higher education, there’s the technical foundational component. They become a reseller of technology to the other verticals.

You see these different interesting models where the majority of times, the executive leadership team, they’re just fighting for dollars just to do stuff. The fact that they allowed me to consolidate, it really at least took that budget allocation time that I would have to spend away and I could focus more on delivery. That was a great move from that standpoint in terms of organization, governance, and trying to maximize the investments so I could scale out to the other verticals.

Any insight for the audience on how to manage communications during such a big change?

Oh, that is a tough one, because in these organizations, historically, and even now, most of the decisions are done by committee. I would say the I in the Chief Information Officer should really stand for ‘influence’ because the majority of my time out there was trying to influence a decision. Even decisions that you thought would be very simple, I would have to gain executive buy-in and support from all the various verticals. A lot of my time was driving decisions and influencing new behaviors, so I would say the advice is understanding the details of this political landscape and work their way to navigate and drive influence and overly communicate. I think that’s what it comes down to. Even a simple decision that you would think would be a slam dunk requires a lot of effort.

That sounds like the challenge, indeed. Among other things, you focus on enjoyable workplace for employees. In your view, how does that work and can it be consistently done within an organization and between organizations?

Oh, definitely. The key is to make sure you’re driving initiatives or projects that get folks excited. Here’s the reality: people wake up and go to work and if they don’t feel energized, then they’re not going to drive value. No one wakes up to do a bad job, everyone wants to wake up in the morning and get to their profession and do a great job. Number one, you have to entice them with extremely exciting projects that’s in line with what their personal aspirations and their personal interests are. Number two, you need to reward them. I would say thinking about those two factors on the people side is very crucial to employee engagement and making sure you have a workforce that’s jelling.

Most of the work on the executive leadership side is really how do you get people to work together; how do you get them to function as a team; how do you break down silos; and most importantly, how do you make sure they’re spending a lot of fun times at their job? That’s where they are the majority of their day–is at work, so figuring out where you can make it a fun event is crucial.

In your mind, is that in addition to the alignment of the work is also that enjoyable activities at work that are not specifically work-focused, or is it really just about getting the work and doing it in a pleasant way?

Oh, definitely just doing it in a pleasant way. Yes, you can set up a lot of different social activities outside of work just to ensure that the teams get to know each other. In certain large enterprises, you try your best effort to make that happen, but it’s hard to scale. You have a 30,000 employee workforce, how do you get everybody engaged outside of work with busy schedules unless it’s at a departmental function? You can do that from a department aspect outside of work. It’s really important: how do you create those great environments? Internally, you get the entire enterprise on the same page.

What qualities are most important in the people that you’re seeking?

The ability to really learn or the willingness to learn, I would say that’s the most important aspect. We are always reinventing ourselves in this industry; technology’s always changing. Healthcare’s changing. Combining the two [into] healthcare technology, it’s always evolving, and you have to be extremely flexible. You have to be willing to learn, take on new skills. I’ll view the role of the CIO; the role today is going to be different three years from now. It’ll be different five years and ten years from now. As a professional, you need to really seek that self-learning and evolution standpoint. That’s what I look for in employees who have that drive, that willingness, and who can really think outside the box.

Your current work involves advising companies on how to maximize their technology investments while they’re digitally transforming their operations. What are some of the most important pieces of that?

The most important piece of that is the reality is the executives that we work with within our network are so busy with the day-to-day that they don’t have the time to understand what’s reality in the market. They either get upsold the wrong information, or they hear a sales pitch and really need validation. That’s where we come in is: provide that validation for our executive network, and we provide them a guidance as far as where they should be thinking about two, three years ahead. That’s a value from our executive network.

We also work with the vendor side. On the vendor side, they have great solutions but what they’re missing is they don’t understand the operational factor. For example in a hospital system, they don’t have the knowledge of hospital operations. What is the clinicians’ workflow? What happens when a supply chain runs–they’re out of stock or how’s that impact nurses? What is the workflow in day-to-day operations? That’s where we help the vendors’ side from an operational perspective.

I would say I get the luxury right now of working with both sides to create this ecosystem because I really want to see the healthcare vertical make that transition and have some sort of true disruption because the industry has not changed much. I really want to create an ecosystem where patients have access to care conveniently and from a technology side, that’s where it can really enable that. I’m a big believer and am passionate about the work that we’re doing just because I’m able to see both sides of the worldnow, from a vendor’s perspective and also from a buyer’s perspective.

On the companies themselves on the healthcare side, are you seeing movement to the public cloud? How are they viewing cloud across SaaS and also compute?

The cloud has won everywhere. Personally, I’ve always been a believer in ‘cloud first.’ The reality is: a lot of these healthcare enterprise applications are having challenges getting into a public cloud environment and then become a SaaS offering. The reason why they can’t get into a public cloud environment easily is because there are a lot of legacy enterprise systems that were not born cloud-first. That would require these companies to rewrite the entire code and really start from scratch, and that’s a very hard ‘ask’ for businesses that have been profitable for the last 20, 30 years to go back and rewrite their code just to be adaptable to public cloud.

We’re seeing the trend as new systems come out, they’re always going to be a cloud-first model, a SaaS offering. Unfortunately, the majority of the healthcare enterprise applications have been traditionally on-premise but we are starting to see a shift toward a SaaS offering. I’m curious to see what’s going to happen in five, ten years from now whether everything is going to either be SaaS or cloud model or hopefully nothing happens where everything reverses the other way because cloud’s going to get very expensive.

Most folks, they may get to a point where they say “No, this is a lot more expensive than hosting it on premise, even though you get all the agility and the scale things,” and then we see that transition back down [to on-]premise. I hope that never happens, but I would not be surprised. It feels like we’ve seen this wave of application going from mainframes to hosted models and back on premise and now back to SaaS or move to the cloud. It’s an evolution back and forth.

We certainly see some examples of companies whose AWS bills or similar are quite high. There are some surprises for companies that are not planning around managing costs in the cloud. I’m wondering: since you’re describing the healthcare providers moving into essentially a hybrid world, if you think that they’re generally managing that transition well–moving into a hybrid cloud with some on-premise than having some SaaS applications or some public cloud?

Yes, they’re definitely managing. It is really a challenge to even migrate. Just think about an on-premise enterprise system. Now you’re migrating to either a SaaS or a public cloud. That migration’s not easy, and that migration can cause a lot of headaches. Think about when you move toward a cloud multi-tenant environment. The client where the organization, they’re traditionally used to managing their own downtime, something as simple as downtime, now because they’re in multi-tenant environments, the vendor provides the time for them to schedule downtime.

As a hospital, that’s a very hard thing to adapt [to] and requires a lot of communication. Let’s just say our enterprise ERP system has a scheduled downtime twice a week whereas traditionally, the hospital only had a downtime once a month just to do maintenance and so forth. Now you have to tell the nurses: “Twice a week at this time, we’re going to be down so adjust your workflow.” That downhill impact operationally is one that most organizations do not appreciate or understand unless they have gone through it. That’s also from a vendor perspective. That’s something they should also think about when they’re scheduling updates, things of that nature, where the deal is routine, but not really as routine just because of the impact they have. Hospitals are transitioning to the hybrid cloud, but they’re still learning and they’re still maturing their operating model.

That’s a great point to raise, as you say, both for customers and vendors about how they think about maintenance and schedules and how on the customer side, they think about what’s mission-critical in that way. We do try here at GigaOm to advise people to really think, as they move to cloud, what is the SLA they really need and are they getting that proper SLA? I’m wondering as you look at all of this, what do you think is the most valuable thing or things that CIOs and their staff should be working on in 2019?

2019, what we’re seeing is there’s still a lot of catch-up, unfortunately. I think people are focusing a lot on data to try to get their data solutions in place, number one. Number two, most are still trying to get their infrastructure up to date. When I look at almost every healthcare provider organization and health systems, their infrastructure’s not up to date. There’s just never been a big investment for them where they’ve kept it going, but now as security heightens, you have to have the proper hardware and infrastructure in place to have the right security protocols. They have to secure their environments, so because it has been so under-invested, now people are trying to play catch-up. Those are the two main themes from a foundational aspect.

Then of course, people are trying in healthcare to figure out how can they understand this whole-person engagement. ERP’s a hot area that’s coming up as well in healthcare just because every hospital organization has been so focused on the clinical system the last eight-plus years they neglected the back office. Now the back office is due for an upgrade, and they’re looking at ways to cut costs and be more efficient. That’s a major trend as well. Those are some of the major trends that we’re seeing in healthcare for 2019.

That’s really interesting. I guess given your comments earlier, you’re seeing a fair amount of cloud ERP introduction, not just on-premise ERP.

It’s only cloud now. Most of them don’t offer you the option to host it yourself unless you’re still a legacy platform. If you’re not moving towards the latest ERP platform out there, they’re either a SaaS model or in the public cloud. You don’t have an option anymore.

Maybe one final [question] on the security point of view, what are you seeing as the most interesting part of the trend there in terms of what they need to catch up, maybe what the biggest challenge is for healthcare providers or how they’re responding to these things?

One of the biggest challenges is, most of the breaches happen due to human mistakes and human mistakes by internal employees. It’s not because someone externally is hacking into a system. It’s the lack of education, awareness, or let me put it in a different way: the emphasis on education, awareness for security.

Security’s an issue everyone needs to be involved with, and it should be high on everyone’s radar. In the hospital world, there’s a lot of focus on hand-washing from a hygiene perspective. I would say security has to be just as urgent in terms of education as hand-washing. That way, you can keep the internal workforce knowledgeable for making mistakes. I would say that’s a big focus, and that should be the top focus.

Number two is start building a true IT security program. Historically, organizations have done their annual audits and what they have done is they get these audits in; there’s gaps. They’re just checking off the boxes to close out the audit and they think they’re compliant. They’re not building a security program, so I would say the two recommendations are: educate your employees on security; number two, build out this security program versus checking off boxes for audits.

That seems like great advice for healthcare companies and from what I know and beyond, I think a lot of enterprises are in that place, as well. David, I really want to thank you for joining me today and sharing your perspective and your insights with our audience.

Thanks for having me, glad to be here.

Take care. Thanks, everybody.

The post CIO Speaks – Episode 10: A Conversation with David Chou of Constellation Research appeared first on Gigaom.

Steve speaks with Steve Comstock formally CIO of CBS about how companies grow and utilize digital strategy to promote that growth.

Guest

Under Comstock’s leadership, CBS Interactive has advanced productivity and efficiency through the modernization of workforce technology, optimized infrastructure support through strategic vendor product decisions, reduced overall operating costs through improved process/structure, created and implemented incident response/remediation processes, creation of a vendor management organization and implemented a global single sign-on solution helping identify and securing over 130 applications. Prior to joining CBS Interactive in 2010, Steve’s 20-year career in technology developed with Sendmail Inc., Oracle Corporation and the United States Antarctic Program (USAP) in various roles within engineering and IT. Steve earned a Bachelor’s degree in Computer Information Systems from Regis University. Outside of the office, he enjoys spending time with his family, binge-watching television shows/movies and channeling his inner foodie.

Transcript

Steve Ginsberg: My guest today is Steve Comstock, a CIO with deep experience and knowledge in the technologies of modern enterprise. He’s been a leader in real digital transformation and complex high demand operations. In this episode, we’ll get Steve’s take on some new technology as well as some great advice on strategy in these modern times. Well thanks for agreeing to speak with me today. I look forward to our conversation. I thought we might start by my asking you that, in your view, what are some of the best ways to create digital strategy?

Steve Comstock: Thanks, Steve. Yeah, it’s great to be here. Digital strategy, you really have to couple it with the view of the digital strategy of the organization, of the company itself. Digital has a number of meanings that sometimes are in conflict or sometimes aren’t agreed upon. But, you have a digital transformation of an organization. Then you have a digital transformation of the company. And you and I have both been through this.

Where you really – I look at it more from the digital transformation of the company. Where does the company want to go? Where do they want to be? What’s your main product? What are you trying to deliver, and are you trying to make the product from your company be digital, or the other side of digital strategy is are you trying to make your organization and the company more efficient from a digital perspective, better automation, that kind of stuff? Where are you headed as a company? And then, from there, you can really start guiding in what is your digital strategy? Because depending on that answer, it could be you’re looking at operational tools, collaboration tools, etc. which you should always be looking at to optimize your environment, or you could actually be looking at tools that are directly related to what your company produces from a digital strategy perspective. So depending on what the – really, the punchline is on that, it depends on where and how you build that strategy.

Having been through, as you have, the actual making a company digital, do you have any insights you’d want to share with the audience about either kind of major pratfalls or key underpinnings of success?

Yeah, there is a lot of both. We talked a little bit before that some of it’s just organizational. I think that your success or failure will be based on how you structure and look at the organization and have the right people in the right spot. It doesn’t mean you do a wholesale rip and replace. But it does mean that there are people who – in your organization who can make the transformation, the journey with you, and there’s some who just can’t. And you really need to really evaluate what that culture looks like, how people fit into that culture, and really don’t discount people right away. One thing I found which I thought was really interesting was that some of the people who I thought would never make the leap were the first and the best people to make the leap and were some of the best champions of the transformation where others that I thought were going to be highly effective turned out to be not as effective as I wished or expected. So I think that’s really important.

I think the other thing too that’s really important is – depending on the transformation itself, is highlighting and really articulating the level of success that you’re going to have. It’s kind of weird how I said that, but let me see if I can improve on that. If you’re building out a technology that has never existed before or are taking a technology and expanding it in a new way that could have a varied level of success, I think it’s highly important that you highlight and communicate that, and have a plan on how you’re going to measure and improve that success over time. Again, both you and I come from a streaming world where a lot of this stuff didn’t exist. And there were times when things didn’t work as well as you thought they would, and we perhaps are maybe using older techniques, or older philosophies, or older operational playbooks that you have to always be thinking and be agile and pivoting and looking at how you can improve those playbooks as you grow with the technology that you’re maturing. And I think it’s super important that you’re very open about that and have a plan that when things don’t go as expected that you recover quickly. I know that’s overused, but recover quickly and have a plan to move forward again and really build on top of your successes as well as failures to the end goal, but don’t let those small failures keep you from looking at the successes that you’ve had.

Yeah, that sounds like a lot of great things to consider. One I want to touch on that you mentioned was about the – on the staffing piece about some folks surprised you by doing extraordinarily well in the new environment, some going the other way. Would you say that ability to learn new ways of working, or ability to learn tools, or both, are those the two main pieces, or are there other qualities that became important in that way?

I think those two qualities are very good. I think you have to be able to be adaptive in your thinking and in your skillset, and I think that, as a maturity level, you have to always consider that just because it worked in the past doesn’t mean it’s going to work in the future. So, looking at how you operate, how you organize, how you plan, and always try to mature that and improve that. And then also looking at the technology that you have been using and looking at how that’s been effective.And where are there areas that maybe weren’t as effective or that you just never really considered because you always thought as commodity, and it’s just whatever? Really looking and go, oh, but wait. Let’s rethink this. Let’s always keep an eye on the horizon of what is coming and how different technologies are maturing or how different technologies are evolving so that when – and this is important. When you are at the maturity level as an organization and when the technology’s at the maturity level, from a technology perspective, you’re both ready to really execute and really add significant value to what you’re trying to achieve.

I think it’s important to think about it from a maturity curve perspective. That’s one thing that I learned pretty quickly. That even though a tech may be really interesting and shiny object, tinfoil in the corner, it may not have – it may just be tech for tech sake. It might not have a maturity curve that is actually at a point where it’s adding significant value, and on the reverse, your organization – even let’s say that tech is already at that maturity level. Your organization might not be at that maturity level yet. They still might be in a different mindset that is slowing them down or interfering with their ability to kind of either bigger picture or connect the dots and have that aha moment of, hey, wait. This coupled with this coupled with this, that is interesting, and that is not something I expected or something I saw. But now that I see it, we really need to go in this direction.

Again, I keep making it a people thing, but it’s really just more of skillset thing. Do we have the right skill sets in place to see this, execute on it, and are my business partners ready as well for this kind of change? And are they at a maturity level that they’re willing and able to really adopt this change and take it to the next level? I think one of the things that’s super important for all of us to remember is we’re not the technology organization within a business. We are part of the business that’s helping accelerate technology and make technology work for us. I think that’s an important thing.

Yeah, absolutely. I wanted to focus within that. So, you mentioned technical maturity as one of the things that you look at. Among our peers, I know that you are very active at looking at new technology, looking at startups, and those types of things. I know from our conversations. How do you advise that CIOs and IT leaders overall kind of keep up with the breadth? And we know there are – every day, it seems like dozens of new companies coming into existence if not hundreds on some days that want to sell to the CIO suite and want to be part of a company’s digital strategy. What are your recommendations for being effective about what you look at and how you evaluate it?

Obviously, you build a good VC network that you have trusted advisors who can bring you or introduce you to tech or items that you might not have thought of. Maybe areas that are always the bright, shiny object, the blockchain, AI, or whatever but, also, the things that are really taking maybe older technologies or old commodity technology that you wouldn’t of thought of and really taking it to a different level or taking a new twist on it, right? And then looking for, I think it’s very important for us CIOs to have a healthy desire to adopt and look at new technology with a critical eye on that technical maturity. I think there’s a really interesting balance here where, when you look at the technical maturity of something, it may be new, but that doesn’t mean you shouldn’t adopt it. It means that you, perhaps the CIO, one of our jobs is mitigating risks. Also, our other job is to become strategic and help accelerate our companies forward, so maybe you take a more stayed approach and implement these newer technologies in a more defined way where then you become partners with those companies, right? And then you look for the areas. I mean, there’s areas that are very interesting but have really no meat to it, right?

And one that, I’ll probably get some eye rolls or high fives. I don’t know what you get on a podcast. But you talk about AI at this point, right? It’s like AI, to me, is a great concept. It’s a great concept, right? But it’s- from a practical perspective, everybody’s an AI company, and it doesn’t feel like people are dialing into the value of what AI could bring. And so back to your question, you start looking at companies. Everyone talks about AI. We could do this, we could do that.

But then, the companies that are interesting – and I’ll point out one that really has kinda caught my eye, that it made me rethink of how I look at some of this stuff. But we’ll get to that in a minute. But it’s really rethinking, okay, what – AI with what, right? What that adds business value? I ran into this company. I was looking at replacing my phone systems. I mean, probably the most boring commodity, dial tone thing in the world, right? And I was just bored. It’s just boring to me, didn’t want to have anything to do with it. Just hand it off. Delegate it. Go fix it, right?

But then I came across a company that I think the phrase is going to be voice intelligence, right?And I think it’s important as we look at the industry that it’s augmented intelligence. AI is augmented intelligence, not artificial intelligence. How do you make things smarter? So this company, it’s interesting, they started – they bought an AI company and integrated it in with their systems so that when you’re having a conversation in real time it will do natural language processing and then – which makes it really interesting, right? So you do the voice transcription, etc. And then I think, as CIOs, we got to start looking beyond just the sales pitch of that. We got to look beyond – we got to look where it could go, right? So you and I both, again, have had very similar backgrounds of natural language processing. It’s huge. Data’s huge for us.

So this company, Dialpad, I ran into. They started doing this in real time where we could be having a conversation. And think about it, Steve, it’s so cool, right? It’s like, if you have AI with natural language processing and it’s doing real time transcriptions, you can start doing things like key word recognition where I could go, hey, I have an action item, and you could build a summary at the end with action items of that call so you and I – which is the number one thing you and I want to do is have a very authentic conversation and really discuss things, kind of like we are now. Unfortunately, I’m talking more than you are, but it’s have that conversation. But then take it to the next level.

And I know you’ll see this right away. Then you get into sentiment analysis, or you could get into things like sales people having conversations with their customers and understanding objections and having that data be able to be processed and analyzed in real time to determine how many times has my competitor been mentioned? How many times has a price been objected to? You put it to a customer service area where you could go, hey, I’m super angry about this product, and I’m not getting the value out of this, blah, blah, blah. You could save that customer right then.

It’s not unfortunate that you’re speaking more than I. Yeah, the sales part, kind of augmenting the sales process was where my mind went with it pretty quickly when you were saying that having, again, similar. We both supported large sales teams, and in that regard, you could imagine even materials or responses, which would, of course, work for customer service as well, being suggested of where to take the conversation next.

Absolutely, and that’s the thing that’s so cool about it is that it’s all in real time. Again, and it’s taking just that one little piece of artificial intelligence and natural language processing. And I believe, really, it’s going to be voice intelligence. I think that’s going to be the new keyword coming up. But exactly, you have those key things coming up in real time that go, “I’m sorry, Steve, that you had horrible experience. Let me see what I could do. Hey, I have a coupon running right now. Here, let me go ahead and apply that to your account. What else can I do?”

Or, on the sales side, which this is where it gets really exciting, right? On the sales side, they’re having this conversation. They’re running through the marketing message. They’re getting real-time coaching on well what about this, what about this, what about this? But the beauty of it, you have all these data points. It’s not just a sales thing, now. It’s a marketing thing. Is the marketing message right? Are we keying in on the right marketing message? What are we hearing from our consumers and customers in real time about the marketing message? And is it resonating? Are we off target? Are we on target?

And this is the part that gets me real excited. And you can tell I get really animated at this point because guess what? It keys back into my first point. You take a commodity product, a commodity dial tone, dial tone, dial tone, dial tone. It’s like my joke is the old Brady Bunch episode, Marsha, Marsha, Marsha, right? It’s like all we talk about is dial tone. And we look at it so narrowly as a product that when something like this comes along that I think we need, as CIOs, to be open to the idea that maybe some of these products are becoming more intelligent, becoming voice intelligent, or chat intelligent, or whatever. That really can take a commodity product and make it strategic.

Steve, I think you would agree, if we can augment, enhance, improve the customer experience as well as the sales experience and the marketing experience, three areas where CIOs always want to add value, but we have a limited tool set sometimes, that’s pretty fricking incredible. You take that chat, right?. You take that natural language processing. What if it could automatically enter it into SalesForce for you as a sales – that’s perfect. Because your salesperson can now just focus on what they do best, which is build relationships, qualify leads, close deals, that kind of thing. I think with the marketing side –

Even some of the follow-up material could probably be automatically generated based on what it sees. I should mention, also, at Gigaom we use Eva some, which I’m not sure I’ll pronounce it right, hopefully, Voicea, I think, is the company. And it does what you were talking about, which is it can highlight things that sound like action items for it. And I believe you can train it, specifically, to catch key phrases, so that if you wanted to – in a call like this, you can set up a specific, “Hey, Eva, action item,” and then it will record that. And it not only does the transcript, but it prepares a summary.

Yeah, right, and see, this stuff is really emerging. DialTech can do the same thing. And one of our requests at a time was, as a customer service, was look for threats, right? Because you have people calling in. “Hey, I know I have your address. I want to do blah, blah, blah, blah.” It becomes a human thing at this point too. And that’s the cool thing about it. Is all this stuff is maturing. And some people are doing some really interesting things, and some people are – it’s vaporware, right?

But if you could add productivity – if you could improve the productivity of your workforce, if you can improve the consumer experience, if you can help close deals and optimize the sales cycle, it’s amazing, right? This is the joke I’ve been using, is instead of handing out cables, you are now a strategic partner with your Chief Revenue Officer, and your Chief Marketing Officer, to really help push things forward and really open up the art of the possible. I think that as CIOs, this is the stuff that we have to be thinking about looking at. I heard that the other day.

And again, some of my peers might get upset with me, but that’s the risk, is we’re talking about chat They’re like Teams, Teams, Teams. I’m like, no, Slack, Slack, Slack, right? But the reason is, and this is why, it’s like you have – we’ll call it chat intelligence, right? The ability to integrate in chat intelligence, or Slack intelligence, or with like a Salesforce which optimizes your sales team. The ability to do bots and build bots that could automate processes that allow you to do things like deployments, or gives you the ability to do X, Y, Z, right? I think that when we look at the digital transformation, and we look at things moving forward, we have to look outside the silos. And you can’t put voice into the telecom silo, or the old Polycom Avaya-type scenario. We have to look outside of it. You can’t put chat into the same buckets as you would the old instant messenger, or – I’ve moved so far out of it I forget what they used to be, right? But it’s like it’s not that –

Jabber,

Jabber, all of that, yeah, right, Jabber, all of those things that’s not it anymore. And I’m sure there’s hundreds of other examples like that that you start looking outside of that box and was like, what’s interesting? And how mature are these products? Again, voice transcription, even with the product you mentioned, I mean, it’s still a little rusty. But you know what? Everybody’s rusty.

Yeah, it does feel like it’s maturing, not mature.

It’s maturing, not mature, but as the machines learn more and more, it’s going to mature very quickly. I mean, you look at voice transcription, or NLP, I mean, we both have done this before, right? Even three years ago, it was so rudimentary that we wouldn’t really consider it. Fast forward to last year, you could use it for subtitling. And be pretty close to it being accurate, it’s going to be – you’re going to get in the 70% range, 60% range, but then you have people training it to improve it. And as it gets trained, that model just improves better, and better, and better. And with the machines in the back doing all that, the maturity curve shortens very quickly. And it can be very interesting where we end up.

And it goes back to team maturity. I think you can run into a problem where – again, let’s go back to the voice thing for a bit. Where the old Avaya Polycom people on your team are probably going to be pretty resistant to a dial pad, and be like, “Well, why would I want to do that? I mean I’ve got a good job. And I can put all these systems in place, and blah, blah, blah. And I don’t need to worry about it.” But they’re looking at it from a very tactical, day-to-day,\ view instead of looking at some of these technologies of does it add strategic value, and trust the verify. And I think both you and I are cynics. And I know we are because we’ve had lunch many times. But, you trust the verify. Like I’m going to ask you. Like have you done this before? And you’ll tell me it’s like , “Yeah, it’s good, but it’s shaky in this area,” or “It’s almost there. I would put it in certain areas. I wouldn’t put it everywhere.” Or, you really got to look at it and use your network to ask around. Because we have friends who are early adopters on everything. And, we have friends who won’t adopt anything until it’s been in the market for 10 years. I think we’ve got to be somewhere in between.

Yeah, you mentioned managing risk. And I think both controlling scope both initially and depending on what it is, ongoing, and then really looking at the security aspects. That’s where my mind still goes for a lot of the hey, let’s use a cloud service for this or that. Is- are the security underpinnings going to be appropriate for what we’re moving there, for example?

Yeah, right, and even regulatory or compliance, right? Is that obviously, technology, we’ve all seen it, moves much faster than all the compliance regulations. And does it- can you use all this stuff, right? And how do you use it? What’s the proper way? And what is the risk? And what’s the overall risk? And that’s our job. Our job is two-fold. And I think we always have to pay attention to it.

Are you seeing areas the company should keep out of the cloud?

That’s a really good question. And, I have to preface it that I’m a really big cloud advocate. So, my opinion’s going to be – my opinion’s going to be a little skewed. Because I think that our job is to help accelerate our businesses moving forward. And, I think we should always think of view from a compliance security perspective. But, I’ve drank the Kool-Aid on the idea that an Amazon, or a Google, or a –

A Salesforce, for example.

Oh Salesforce has seen more attacks than I will ever see. And pProbably can protect better than – again, I had great security people. I love my security people.And I would put them all against the best. But, go to Salesforce and Google, they’ve probably seen more than I’ve ever seen, right? You look at the spam engine on Google Apps. It’s pretty dang good. That, I could never beat that. We have to manage the risk. And if someone can manage the risk better than we can, then that frees us up to do what we really should be doing, which is accelerate our business, and accelerate our workflow and our processes, and help our business partners move at the speed of technology versus the speed of business, and help them get to their place. So honestly, Steve, to answer your question, I don’t – I’m sure there’s areas that we should never put stuff in the cloud, but I haven’t found it yet.

You’re able to get through it, yeah.

Yeah, right, I mean, it’s like – there’s enough. And it’s matured to the point, now, where people are always thinking about it first and fixing those things. So it’s interesting. I haven’t found it yet, but if I do, I’ll be the first to let you know.

Sure. One of the things that we’ve talked about in the past, that I learned from you to some degree, was that when looking at cloud strategy, one of the advantages is moving from Capex to opex. Part of when looking at the cost of the cloud, that’s one of the things that you can factor. I wonder if you could share a little bit with the audience sort of how you think about that.

Yeah, sure, and since the time we’ve talked, my thoughts are pretty much still the same, but they’ve changed just slightly. Is that when you’re looking at it, the beauty of the space that we were in – and again, for clarification, we were in high volume, unpredictable volume-type scenarios where we would have a lot of traffic, and not a lot of traffic, then a lot of traffic, then not a lot of traffic. So very cyclical based on seasonality, etcetera. But when you have things like that, the opex model is really interesting. Because then with the cloud, you could scale and shrink depending on your workloads.

And so if you’re making more money, you’re spending more money. But you can try to keep your margins as good as possible, where when you’re doing the old Capex model, and you have to buy equipment and all this other stuff, you pretty much have a fixed cost there that you have to build to peak, and that if you aren’t utilizing all those servers for those systems, then you’re really just leaving money on the table. And so I really like the idea of being to be flexible with that model. Now, I’ve changed my opinion a little bit that when you look at some of your workloads, you have to evaluate your workloads. Because if you have really really static workloads that are very consistent, then you really have a model to optimize your cloud build.

And- I don’t even know if it’s true the more I think about it. Because you still have – a system I would think of that would be pretty static would be your financial systems, right? Where it’s going to be every day, it’s pretty much used the same, but that’s not necessarily true. When’s the busiest time for financial systems? Month end, quarter end, year-end, right? Where your CFO probably wants those systems to be at their highest peak cranking this stuff out as quickly as possible, so she or he can get through and look through the numbers to make sure that everything’s right, or figure out what deals are coming in, or what they need to close so they get the books closed quicker, so they can start doing their next financial planning, right?

So again, the Capex versus opex, I think that is the question back to kind of your first digital question, it depends on your business. If your business is a very highly Capex rich company, I’m thinking of real estate, stuff like that, where you have a lot of capital, maybe an operational spend isn’t that good. And the capital is really kind of where you would want it all. But if you’re not a capital business, if you don’t own data centers, if you don’t own real estate, operationally you have a lot more flexibility, in my mind, of how you could move those dollars, etcetera. Now it’s all about the line, and if you’re using an operating income model, or however you’re reporting your numbers is really an important thing. And you’ve got to be really paired with your CFO on that to make sure that she or he understands and can advise you on what’s the best model. And it took me a while to get to that point. But- I still am a huge fan of more of the operational model of the cloud, and then really dialing it in from a financial model of does it fit?

Right, and when you mention operations, one of the things that probably is clear to the listening audience at this point, is that both you and I have backgrounds in operations, which not all CIOs have. And I wonder how you think that informs your path versus some of our peers who might have come over from other disciplines?

Yeah, it’s interesting. I tend to look at everything through that lens. And I look at it through high volume, high traffic, scalability, reliability, in looking for gaps in security. And how can you accelerate engineering processes, and development processes, and really looking at it from a product perspective? Because when you’re in the operations seat, especially if you were coupled very closely with engineering, you are a part of the product process. And, you could affect the roll-out, the speed, the agility of the company as it tries to roll out product.

So, I look at everything from that perspective. And I look at workflows. And I look at ineffective processes, or technologies that could be automated, or what could we do from a team maturity perspective. Instead of building an army of thousands, build an army of tens that are highly highly skilled, probably highly highly expensive too, but that could write and automate all of your processes. And looking at it from that way where maybe not coming from an operational perspective, it’s more of – Well I really can’t speak from it, because I don’t know what that view would look like, right?. Because I look at everything so much through an operational lens.

Sure. One of the pieces of experience that you have, that is somewhat rare air, is that you’ve been involved in premiere streaming events, huge volume, live, streaming events. I know that can be a pretty high-pressure game. How do you prepare for that? Cause I think for some of our enterprise audience will sometimes have to be involved in something like that. They may come to that work in a kind of ever media-rich world. How do you prepare for that kind of pressure both in terms of architecture to make sure things will work, that you’ll pull it off, and then prepare teams to actually perform in those situations?

Oh yeah, that’s a fun one.And that’s one that I matured through as each event came through. I really think the key is to start at the basics, really. And one of the things that we were really maturing into and working into is this – the SRE model that Google uses for their things, Site Reliability Engineering, going from Ops to DevOps to SRE where understand what your service level objective is, especially on these live events and on these big events. What are the number one, two, three, four goals of your company, of your business, on these events? And I think this applies to everything. So for us, it would be the stream has to be available. You cannot lose the stream, right? You have to be able to find the stream. You have to be able to log in if it’s a paywall situation where you have to pay to see it. You have to be able to allow that process to get through the paywall. So if someone wants to register and pay for the event, they go through the process and understand what those objectives are. But once you’ve narrowed down what those objectives are, and it’s a hard one because one of the things that your partners will say is, “Well, everything has to work.” And that’s not necessarily true. Like the thumbs up, thumbs down, for the event, does that have to work? Probably not as important as you’ve got to be able to watch the football game, right?

So defining those, and once you define those then you can define the service level indicators that – and go through your system. And go, okay, what are all the things that support a subscription? What are all the things that support green? What are the things that cannot fail? And then you start going through those. And then what we did is we started – we really refined our monitoring, and alerting, and tracking, and testing on those. And we put in systems that we were able to then measure, to a T, what that was.

And, one of the real big learnings that we had, or I had personally, was instead of trying to hit a number that I would get from marketing or whatever, we flipped the needle, or I ‘m sorry, flipped the playbook and said, you know what? I’m going to tell you what we can do. And we’re going to work with our engineering people. And we’re going to load test the crap out of this stuff. And, I’m going to be able to tell you, I can go all the way to this level of green. Not a problem, got it covered. There should be no failure. Then, I can tell you where we hit yellow. And from yellow to red, what does that look like, and what we will do to put it back into green, which might be turning on or off the thumbs up, thumbs down. But, what do you need to do? And then, what is it at red?

Because now, if you know that, and you walk into that event, you’re not worried about, oh my God, what if we hit four million, five million, six million concurrent streams? We know that if we get four or five million concurrent streams, we’re fine because that’s in the green. And we already have all those tools in place to measure all that, so we already know when we’re getting close to that limit. And, what happens is as you’re putting that all together, you’re maturing your team with that. Because you’re going through all these exercises of load testing, and executions, and failure scenarios where your team is maturing and learning as you’re getting to that event. So when you’re at that event, you’ve already done this 15, 20, 30, 100 times just getting to the point where we know what our load level could be.

That-you’re still going to be nervous as hell, because you, usually on these really big events, have a lot riding on it, especially in my world, and I know in Steve, in your world too. But a confidence level that you can report back to your executive staff, who might be onsite at the event, and go, “We’re doing fine. We’re in the green still. We have plenty of space. And we haven’t had to execute any procedures yet because we’re either good, or hey, we’re not.” And we start to do these things to continue moving forward and get it back to that place. And so it’s a lot of execution – a lot of the execution comes in. A lot of the operational maturity, and planning, and identifying where do you have those gaps. Identifying those SLOs, those SLIs because one thing we ran into –it was a maturity thing-was we were measuring everything.

And what would happen is something would pop up that had nothing to do with anything, and everyone’s chasing that while the ship’s on fire. And it’s like, no no no, wait, just because we can’t raise or lower the flag at the front of the boat, that doesn’t matter. We’re seeing smoke come out of the engine. Let’s go back and fix that, right? So knowing what those things are are critical to the success.

That’s some great advice. You’re advising start-ups and larger organizations today. What trends are you seeing, currently?

I think one of the big trends that I think we hit on earlier is really taking this vague notion of big technologies and really refining them down to really business value add technologies. Again, the AI, the artificial intelligence to voice intelligence, right? The artificial intelligence to chat intelligence I think is really interesting. And I think that you’re going to see more of that where instead of really taking a very broad concept, and really applying it very specifically to a core issue. I think one thing we’ll probably see come up is security intelligence. I think it’s already starting to happen, right? Which is great. And I think that’s a big trend. And I think it’s an area to pay attention to.

I know that we’re all overloaded on the idea of artificial intelligence, and AI, and ML, and block chain, and BPA, blah, blah, blah, right?. I mean it’s – your eyes glass over and you’re like, not this again. But I think pay attention to the things like voice intelligence type stuff, like where you’re going to see people are discovering that these older technologies really need to come out of the commodity stage and go into the strategy stage, and really help optimize environments. I think – I don’t think cloud is dead, or cloud is going to start migrating in the other direction. I don’t believe that.

One of the things I’m seeing is that the use of Kubernetes with like a terraform and a dock are becoming very interesting. And I think you’re going to start seeing multi cloud strategies emerge where as those three technologies mature, the ability to move back and forth between clouds becomes easier. I think the hybrid cloud thing is kind of – I was really poo-pooing it and thinking it was kind of a fad, but again, back with those three technologies, I think that the idea of being able to cross that bridge between a data center and a cloud is going to be interesting. That’s interesting.

And a lot of people are still very focused on kind of the digital strategy of what does that mean for us as companies. Because I just don’t do startups. I do growth companies as well. And some of them are figuring out, am I a digital company from a brand perspective, or am I a digital company from an operational perspective? And really trying to figure out what that means. And I think we have a very big role in helping define that and helping guide companies.

If you’re growing food, your product is food. But the way you process and deliver that food, and deliver it at the most optimal way, could be your digital strategy and your digital brand, improving the food that you’re delivering, right? Which gives you a leg up over your competition. It’s fascinating to me. And I see a lot of that right now, people trying to figure out where they are in that curve. And how can they move quicker? And are they missing out on something?

And I think that’s just, again, to kind of answer another question you asked earlier, I think we’ve got to be very careful of the FOMO problem, the fear of missing out. We’ll always have our friends who will be the first to adopt anything, which is great. And you should always buy those friends a drink because they are taking risks that we probably can’t take. But only take the risks that you think are going to help accelerate your business. Don’t take a risk just to take a risk.

Yeah, that’s great. Thanks very much, Steve. I really appreciate your coming and sharing some deep and wide advice for the listeners, here. So thanks for speaking with me.

Anytime, Steve, it’s always a pleasure.

The post CIO Speaks – Episode 8: A Conversation with Steve Comstock appeared first on Gigaom.

Host Steve Ginsberg speaks with executive digital adviser and consultant Joanna Young about guiding organizations through digital transformation successfully and efficiently.

Guest

Joanna has held CIO and CFO roles in the private and public sector in insurance, healthcare, financial services and higher education. She helps organizations solve big problems such as Agile transformation, Lean management, IT strategy, organizational change management, product selection and large program management. She’s known for her obsession with authentic leadership, digital transformation, customer delight and talent management. She participates often in social media; #CIOchat, #trueleadership, #voiceofthecustomer, #WomenInTech, #PeopleInTech are common hashtags.

In her spare time, she’s with kayaking, running, writing, hiking or sitting on the beach watching the waves roll in.

Transcript

Steve Ginsberg: Hi, I’m Steve Ginsberg. My guest today is Joanna Young, an experienced C-Suite exec with deep experience providing digital leadership and guidance to executives, teams and individuals. In this episode we’ll hear Joanna’s insights, specifically, in organizational efficiency and culture.

Hi, Joanna, thanks for joining me today.

Joanna Young: Great to be here.

You’ve been working with a lot of organizations. You want to tell us a little bit [about] what you’re working on currently?

Yeah, sure, so I’m working with several organizations on large, complex programs. Most of them have some aspect of Agile transformation and also related to moving from legacy architectures and applications to cloud or hybrid or hosted architectures.

That’s great. I wanted to talk a little bit about project management and program management. In your view, what do the project management methodologies bring to the table?

Well when I think about project management today, I think rather broadly all the way from, if you will, more than one traditional PNP-type of methodologies to the more agile, lean types of methodologies that are being deployed. I think what they bring to the table is really these days, the ability to create a minimum viable construct that is going to deliver value as quickly as possible. I think if people are thinking about project management, whether it’s agile or traditional waterfall, that way that it needs to be the most lightweight approach possible to get the team or teams to the point that they’re delivering quickly and accurately, then everybody will be a lot better off. Project management is not for the project managers; it’s not for managers; it’s not for leadership. It’s for the teams. It’s to support the teams.

Yeah, I like your emphasis on ‘lightweight’ having come from waterfall shops and then moved into Agile. It certainly seems like for a lot of projects, it’s a better way to go. How do you feel about communicating to folks–like you said, it’s not for project managers only; that’s not the goal–about bringing in the partners that you’re working with on projects into these methodologies?

Well, though I think the hardest thing for organizations to get their heads around is that these methodologies are, again, not for leaders. They’re not for the managers; they’re not for the project managers. They’re for the teams, and really as I’m bringing different stakeholders together, really the biggest thing is change management, getting people to think differently about how this should be supporting delivery and supporting delivery faster and faster and with higher and higher degrees of quality. A good example is that we’re used to—and when I was a CIO some years back—I was used to getting that weekly or monthly or quarterly red/yellow/green stoplight [about] what’s going on with all of the projects.

It’s things like that–people’s expectations about a lot of the effort going into the plans and the statusing–people need to realize that that is of lower value than the actual delivery, and really change their minds about what they’re looking for, if you will, in terms of metrics and critical success factors in the projects. For example, a team that’s firing on all cylinders and delivering more faster and faster to ever happier customers, whether those customers are internal or external, that’s the sort of thing these people should be looking for and thinking about, not stoplight reports.

Right, it’s the result, not the reporting.

Right, but still there’s so much focus on reporting. Then the other thing I find is that people still get extremely–feel a little wrapped around the axle about resource assignments. One of the things I’m often dealing with in terms of change management is the concept that one of the negative aspects of an organization is the amount of time they spend slicing and dicing people amongst projects. That’s non-value added time and wasted time. Moving people away from that slicing and dicing and focusing on fully resourcing the teams on the projects that are of the highest priority [is the better approach].

I saw you’ve done some nice writing on resourcing and that the planning doesn’t necessarily produce any more resources and that once you commit, organizations should commit to resourcing properly those priority projects.

Exactly. I mean, I’m working on real-time, real-life projects where… I think it’s very obvious but everybody is coming from a different place, so just because I think it’s obvious doesn’t mean other people do. We’re seeing teams that are producing incredible amounts of value in two-week sprints most usually. Then other teams that aren’t and are really struggling to get any sort of velocity or demonstrably higher quality and the largest differentiator between teams in both of those spectrums is that the fully resourced team with the developers, with the product owners, with the BAs and the QA function. Fully resourced teams are just simply so much better positioned than the teams that are not fully resourced and have people sliced and diced between projects, [and] are just simply not going to produce.

I think it’s harder for some organizations to get out of that mindset than others, so just understanding where organizations are standing, going to where they’re standing, trying to help them move forward at a reasonable speed to a better place, if you will.

How do you advise that organizations should balance the ‘always on’ dynamic nature of business? For example, a year or a quarter might be well-planned at first and then depending on how dynamic the business is, there are often new asks coming in.

Yeah, so my thinking these days, what I believe works best for most organizations, is that they should be planning extremely high-level for maybe, let’s call it 12 to 18 months, and then in the 6 to 9 month level or 6 to 12 month level, maybe at a mid-level of planning, and then really only detailed level of planning in what I would call a quarter or two, 3 to 6 months. Then of course when you’re talking about these newer agile or lean constructs, you’re even getting to a more granular level of detail with backlog refinement and assigning things to sprints and what have you. When I see an organization who’s still trying to plan in that beyond 12 to 18 months, like these 2, 3, 5-year horizons and they’re spending a lot of time on that, that usually is what I call a leading indicator of perhaps they’re not focused appropriately on the right things.

That makes a lot of sense. In you view, does the planning tools themselves matter much, or is it as long as an organization is comfortable with the tools, they seem similar or do you feel there’s a vast difference in terms of what can be gained there?

I don’t think the top tools–we all know who they are, Microsoft Teams or a best-in-class suite of products. I don’t think, if you will, top-tiered tools are a ton of difference. It’s like anything else; it’s all about the process, the culture. Are people in the right roles at the right times? Does leadership have the right mindset? Are they supporting teams appropriately? Those things are far, far more important. That being said, a badly implemented tool is going to impede that. I more often look to okay, what are the tools you’re using and are they helping you or are they impeding you? If they’re helping them, let’s put the tool question aside and focus on other things. If I’m seeing some signs that the tools are impediments, then we might put that near the top in terms of things to address.

Sure. What about the human element? For example, you’ve written about hubris and humility. How should IT organizations view those and the other top qualities of the people on the teams?

Well, I think servant leadership is definitely key. Nobody is getting work–very few people, I guess, are getting work done themselves. They’re getting work done with and through other people, and that’s particularly acute for leaders. One of the leading indicators or lead lagging indicators of success that I see are organizations where leaders truly care more about the teams than themselves, and that tends to show up, if you will, in the full life cycle of talent from the attention leaders pay to the quality of the hiring and the onboarding process all the way through, the career building, whether it’s compensations or promotions or giving people different valuable experiences; all the way really to latter stage career and making sure people at all stages of their career are positioned appropriately to contribute.

Leaders who realize the importance of talent and their role in supporting that talent, those people are very valuable and organizations who are doing well–and you can even see this in organizations at the top of the leaderboard. Organizations who pay attention to that, and who value leaders who pay attention to that, tend to do better than others. This is a topic, as you can tell, that I’m pretty passionate about because I think we’ve all seen very sad examples of leaders who have not had this approach and attitude, and what’s happened to some of those organizations.

How does that manifest in your view in terms of communication specifically? What leader style do you prefer in that regard, both what’s effective and what you think is best for all of the constituencies?

I think you want to look for leaders that first of all, have great communication skills and that doesn’t mean that they have to be massive extroverts or anything like that. They are people who are able to very quickly and expertly say, “Okay, this is the meeting. This is the situation. This is the conversation,” and be able to adjust to the audience. I mean, when you’re a leader, if you’re in the C-suite or in a VP or senior director type role, you got to communicate [with] people all the way to boards, potentially external customers depending on your role, all the way down to individual contributors. There’s a lot that goes into that, and it’s critically important. What I think is important about leaders when you’re thinking more about their organizations is that–I’m going to tell a little anecdote.

I was listening to someone speak not that long ago, and it’s someone who manages… the CIO of a mid-sized company, and supply chain is critically important to this enterprise. They were talking about the fact that they had something I believe, that they called the ‘truck driver test.’ If they were able to walk in to, if you will, one of their locations and say to a truck driver, “Do you know what’s going on with this and such?” If that person was able to say, “Oh, yeah. I’ve heard about that,” and that’s important because–or that impacts the company because–and if they were able to articulate that, then that CEO was like, “Okay, I’m doing my job and the people in between me and this truck driver are most likely doing their job as well.” I thought that was just such a super anecdote because it’s so true.

If leaders are able to foster either through their own words and actions, as well as those who work for them and around them, the types of words and actions so that you can get that level of engagement and understanding at the most far-flung edges of the organizations, you’re most likely doing something right. Also, I thought it was cool that here’s someone who’s the CEO, and he’s realizing it’s important to engage and talk to someone who’s at the edge of the organization, if you will.

It’s like QA or monitoring for the quality of the process as well.

Yeah, and there’s a quantitative and a qualitative aspect to this, right? It’s art and it’s science. As leaders we have to know what the right metrics to look at [are]. You can have thousands of metrics, but if most of them aren’t the right ones, you’re wasting a lot of time and energy in the producing of them. Then there’s this qualitative aspect that there’s a lot of art too, and I think CIOs and CEOs and CFOs, they should be looking for the equivalent of the “truck drivers” in their organization including communication with those folks and their quality assessment of how things are going.

I think it’s a great point on the effectiveness of the metrics. In your view, what makes a good KPI?

A good KPI is one that is closest to how the actual customers of your products are consuming, perceiving, net promoting your products. Those are the KPIs we should all be caring most about. Those can range from the financial, revenue numbersthat [are] obviously going to tell you something about the consumption and the perception of your products. But again, there’s metrics about how fast are new desirable features being added and that gets to things like velocity and quality.

People need to figure out what are the KPIs that are really closest to the customer, and you need to make sure that everybody in the organization understands what those KPIs are; how they relate to their particular role and job. Also, I think what’s particularly important, maybe more so for the leaders and certainly the people involved in the metrics and analytics themselves, is the source of those metrics.

If I had a nickel for every meeting or interaction I’ve been in where somebody has said, “Oh thus and so metric is at 75%,” and you end up having a lengthy, sometimes difficult conversation on “Well, I don’t believe that number and what’s the source of that data and what’s the timeframe of the data?” The amount of time that we spiral into conversations about the validity of certain metrics points out the importance of understanding the source of the data and making sure that people understand this is the source we’re using, this is the formula we’re using, and that’s what we’re looking at. We’re not going to debate these things. We may have healthy conversations about altering something about that at periodic intervals, but we’re not going to sit in rooms and argue about these things once the sources and the formulas are set up.

You’ve spent a lot of time talking about delivering value and delivering it quickly. Where are you on the concept of ‘fail fast’?

I’m high on the concept of fail fast as long as leaders understand what that means: the fail fast and the ‘learn fast’ and the role that they play. Nothing will kill the benefits of fail fast sooner than any public or even semi-public wrist slapping for somebody who makes a mistake. We’re not talking about deliberate mistakes or mistakes that are repeated over and over out of carelessness or the inability to collaborate effectively. Talking about things like, ‘hey, we learned to prioritize this feature and this application and what goes into delivering it’ and maybe the feature doesn’t end up being that popular with the end customers. Okay, well, there’s a lot of learning there. Why wasn’t it popular? Why did we think it was going to be popular? What is really the thing that they want and let’s get on that right away. If leaders are having those conversations when something doesn’t go quite right, then that’s good.

Even in dire situations where maybe there’s a bad outage or whatever, if a leader comes in and is like, “Hey, let’s first focus on get[ting] this resolved,” but then after that, having good constructive conversations about what’s happened. What can we learn from it? How can we get better? The value of that again, positive servant leadership is great, but if people are afraid to speak up, if they’re paranoid that they haven’t made the right choice about the way to deliver a feature, then that’s just going to shut things down and slow velocity, impede quality, and you’re probably going to have a problem with your talent too because people these days really don’t want to stay in organizations that aren’t failing fast in a positive way.

For product organizations, understanding who their customer is should be somewhat straightforward. I mean, there’s work that should be done there of course to refine. For CIO organizations, I know internally, it can be a struggle sometimes to even decide who are the customers, and how they should be prioritized.

How do you advise the CIO especially in the idea of: should they also be considering the end customer of the organization as a whole? How do you advise organizations to go about creating a proper priority around all of that?

Right. Obviously, every organization is different so it tends to be, if you will, an ‘of one.’ How I would advise CIOs is [tell them to] look at your portfolio of products and services and make sure you understand who the customers of those things are and make sure you understand the layers of customers, right? To your point, it can be a bit opaque or murky.

Let’s take something fairly common like the applications that support the various human resources processes. Who’s the customer of that? Yeah, it’s the human resources department. It’s the chief human resources officer or whoever it might be in a similar position, but it’s also the employees themselves, right? Is the HR application the one that’s used for employee add, move, change type of thing? Is that creating friction in the organization so that employees feel dragged down by things that they might have to do related to that system, or is it enabling the organization? How does that manifest itself with the individual employees themselves, managers and other leaders in the hierarchy? What does all that mean?

That can be a difficult–even that can be difficult to tease out. And sometimes the HR department can really see we’re the customers, and we’re the conduit, and you need to be working solely with us. Then there are other organizations where HR is what I would say more evolved where they’re [thinking/saying] ‘Okay, we understand that it’s not all about us and it’s really about the employee experience. While we’re the stewards or the product owners, if you will, we understand that it’s important to get actual employee stakeholders involved outside of the HR department on things like onboarding, and how the performance management process works.’

Yeah, certainly a lot to balance there. With that in mind, how do you advise organizations to seek consensus when you have cross-organizational projects in which there are stakeholders throughout or are at least in different parts of the enterprise?

This again is where every organization is very different and ‘of one’ is because consensus does not mean the same thing in all organizations. There are some organizations that they are extremely hierarchical—maybe almost akin maybe to the military in command and in control, and there is a few people at the top who make the decisions. That’s all the consensus that’s going to happen, and then there are other organizations that are very democratic. For example, academic institutions can be a lot like this, and they want to get wide swaths of agreement before proceeding on things, and then there’s everything in between. When I was talking to CIOs, it’s like, “Okay, you need to understand,” particularly new CIOs, “you need to understand what does consensus even mean in your organization, and make sure that you’re talking to existing people in the organization who have been around for a while who can talk to you about that.”

There’s pros and cons at both ends of the spectrum. With something that’s very command and control, maybe decisions and prioritizations [are] made quickly, but maybe you’re only relying on a few people or maybe those few people aren’t always correct. Whereas if you’re doing a very democratic approach, where there’s wide buckets of folks that need to be consulted and have their feedback incorporated and so on and so forth, that can be very, very time consuming and very slow and you can lose time to market opportunities. Again, I think a theme here is, whether it’s consensus or tools or how you’re doing Agile, how you’re managing people, there’s no ‘one size fits all.’

The CIO really needs to do a lot of thinking and exploring and be very curious about how all those things work in an organization, both within the IT organization and outside the IT organization and figure out what of those things are contributing to velocity and equality and delivery, and which of those things are creating barrier conditions that they need to figure out. I’d do a plus one then is that, these days, since most CIOs and their organizations are very dependent on vendors as we move more things to the cloud, that yeah, it’s a good idea to understand some cultural aspect of the vendors you’re engaging with as well because that can have an impact.

Yeah, quite a lot to balance. I did want to ask you that on the human resources front for IT organizations and for companies as a whole. In addition to salary parity, what would you like to see IT organizations doing to promote women in leadership positions, and also support them strongly in direct contributor positions?

I tend to be a pragmatist, and one thing I will say is that until there’s more women in leadership positions, there won’t be more women in leadership positions. Companies need to be turning that wheel much better than they are. One of the things that has been disturbing to me recently and I’ve been talking to colleagues about is that you’ll hear organizations say, “Well, there’s plenty of women in IT.” I’m like, “Okay, let’s talk about the roles that women have in your organization.” There’s still a tendency–you’ll see women in HR roles or project management roles and not so much in the more technical roles or senior roles or leadership roles. I will freely admit that that’s anecdata, but when I look at studies on these things, that does tend to be what you see.

I think there’s a long game and a short game here. The short game is that you need to be looking at what everybody wants their career trajectory to be, and you need to be looking at the organization and actively saying “What women do I have at various levels of the organization who I need to be reskilling,upskilling, focusing on so I can get to a better place because the data clearly states that the more diverse organizations are, the better off they tend to be. That’s the short game. That’s something that organizations should be actively doing now, and a good chief human resource officer and the like should be able to help organizations with that.

Then the long game that companies need to be doing is they need to be working with schools—even down to the primary and the middle school level on the pipeline. There isn’t going to be some silver bullet in the next few years and all of a sudden, we’re going to have turned the corner in the amount of women graduating from universities with STEM or equivalent degrees. There needs to be a lot of change in our educational system to get a pipeline to a far, far better place.

It’s interesting when people talk to me as a woman, they’re like, “Well, are you doing this because you’re a woman? Why is this a thing for you?” I’ll say, “Well, here’s the reason it’s ‘a thing’ for me, is that everybody is screaming for talent. Everybody needs developers and other technologists and IT security folks, and everything else and there’s a dearth of talent. Why would you not be focusing on the entire set of the population, men and women, and making sure that companies are going to be making iterations towards a better place and having enough talent available?” At the end of the day, it comes down to the math and the types of expertise that’s needed in the 21^st century versus the type of expertise that we have available.

Yeah, absolutely. We may have touched on it already but what would you say are the most valuable things that CIOs and their staff should be working on in 2019?

They should be working on figuring out what’s most important and prioritizing. The problem I see is so pervasive every organization that I talk to is that, they don’t have priorities because they’re prioritizing too much. In my mind, they should be working much more specifically and spending more time on what are really the most important things we need to be working on. It’s not a hundred things and it’s not 50 things. It’s less than ten things. In fact, some would argue that it should only be three or five, and so that’s number one. They have got to get that figured out.

Then the other thing is that everybody–I don’t think this is an exaggeration—everybody is somewhere on the “journey to the cloud” which can mean a lot of different things, and there’s a lot of different bits and pieces to it, but I think that organizations need to be doing a better job of not just figuring out technically where they should be with their portfolios. They need to be changing, if you will, working on changing the DNA of the IT organization and the adjacent parts of a given company or organization to be prepared for what that means.

I mean, just take things like IT service management processes. IT service management processes need to be completely changed when you’re talking about moving to software as a service or storage in the cloud or analytics in the cloud or whatever. What I see a lot happening is, everybody gets enamored of the technology and they start down on this technical journey, but the IT organization is not very well prepared in terms of their processes or their talent and so they struggle. The last thing I’d say about this is that a leading indicator that an organization may not be thinking about cloud in the right way is when the leading question or the only question is, “How much money are we going to save? How many fewer FTEs are we going to need?”

Absolutely.

It certainly is a question that needs to be asked and answered but it’s one among many, many questions. I’ve had people ask me many times, “How much do you think we can save if we go to XYZ cloud-based solution?” I’m like, “Well…,” and I will be very honest to people. I’m like, “I can’t answer that question until we have a lot more conversation and a lot more analysis. Let’s talk about a full suite of questions that might be helpful to be asked and answered with that one among them.”

Yeah, glad to hear you’re focused on that. Some of the work that we’re doing at Gigaom is actually looking at what are the real costs of the various cloud technologies, as I think was implied in your statement. A big part of it is, perhaps as you move to the cloud, you’re going to change the nature of your work, and that might be a fantastic business opportunity that saves no money at all. It may be that you’re just going to do much more than you were doing, and in a lot of cases you will then just have to change the nature of the work, but it’s not about less.

Right, exactly. I mean, the question should be around ‘okay, not what’s the amount of money we should be spending on IT, it’s what delivery and quality and value should we be getting out of our IT investments from–particularly again, IT talent?’ I mean, IT talent tends to be, in most organizations between 60% or 80% of the overall budget. Compensation is huge chunks of the budget. People want to change to these new constructs, but what are they doing about reskilling or upskilling their organization? In my opinion, layoffs are a failure of leadership. Not everybody would probably agree with me, but that’s where… I’d feel pretty strongly about that.

Great. Why don’t we leave it there? I want to thank you quite a bit for this discussion. I think you’ve given a lot of great advice for IT leaders, and thank you for speaking with me.

The post CIO Speaks – Episode 7: A Conversation with Joanna Young of JCYCIO.com appeared first on Gigaom.

In this episode of Right Size Security, Simon and Steve discuss bug bounties, penetration testing, some of the tools used to conduct them, and why this important field of information security is so nuanced.

Intro

Welcome to Right Size Security, a podcast where we discuss all manner of infosec: from enterprise security to practical security that any business can use all the way down to end user security. Your hosts for Right Size Security are me, Simon Gibson, longtime CISO—and me, Steve Ginsburg, former head of operations and CIO. For this episode of Right Size Security, we’re discussing bug bounties, penetration testing, some of the tools used to conduct them, and why this important field of information security is so nuanced. We’re going to walk through the ins and outs of this topic and explain some of the pitfalls enterprises should keep in mind before embarking on a bug bounty or penetration test—or pen test, as it’s known—and we’ll get into some of those details. Thanks for listening to Right Size Security.

Transcript

Steve Ginsburg: So those who listen to the podcast know we always start with some short topics, and I wanted to start this week first with just a quick follow-up. Last week we talked a little bit about voting machines; I brought up the interest in a standard for secure voting machines. And I questioned a little bit out loud: “Well, there must be some work done in this field.”

And I just wanted to follow up quickly, without going into all the details, but there are organizations, of course, that are engaged in this, so quick Googling brought up the Organization for Security and Cooperation in Europe, which is actually a very wide mandate organization with 57 countries, including the U.S. and Canada, in addition to European nations, some North American nations as well. And they definitely look at voting security and election security in general. Also, in the U.S.—a reminder, some folks will know—that it’s actually voluntary testing guidelines that are produced, which I thought was interesting.

So the companies do have—and this will lead a little bit into some of our topics today—requirements for disclosure and testing, but still there’s voluntary, and looks like multiple standards, so just thought that was kind of interesting. And there’s also another international nonprofit called the National Democratic Institute, or NDI, which it did not look like the U.S. was a member [of], according to their website. But they had some very clear type of standards, which was really what I was thinking about, like requirements for transparency…and was looking more at the overall technical requirements for a good voting system, guidelines in that regard. So just [an] interesting topic, and one that we’ll hopefully see in the same way we’d like to see corporate security improve overall; in all locations I’d love to see voting machines continue to get more and more rock-solid as much as they can.

Simon Gibson: Yeah. You’d think that as part of our democratic system, it is very intertwined with the government. The postal service is very key to our democracy, because if the mail doesn’t run, people can’t absentee vote. And I think for it being as integral to our country, FIPS or common criteria-like standards or NIST standards around voting machines that are mandatory and transparent is probably a good thing.

Steve Ginsburg: Yes. And this, in fact, does play a role in the U.S. Government standards for voting.

Simon Gibson: Nice.

Steve Ginsburg: And then, on the corporate side, there’s an interesting write-up I saw today on an ongoing issue, which is that a security researcher found what looks like a pretty significant problem with Zoom, the videoconferencing system.

Simon Gibson: Hm. I saw that go by, and I saw that Zoom had a problem with their video and the privacy around the camera, when you’re in the middle of the Zoom meeting, being turned on and off. And I do the thing quite a few researchers do than they care to admit; I thought, ‘Well, I have tape over my cameras all the time, and I’m sure it’s a problem, but my cameras are taped, and I’m generally pretty cautious around any kind of webcam.’ And so I did not dive into it much.

Steve Ginsburg: Yeah, so I’ve similarly opted for the sliding plastic door, and there are times when I thought, ‘Well, perhaps I’m being a little overly paranoid,’ but I also thought, ‘No, it’s probably likely that at some point’—you know, I think a couple of concerning things raised to me there [are] it looks like—and I think Zoom is a great company; their product is excellent, and just to be clear, I’m actually a supporter overall.

Simon Gibson: Same.

Steve Ginsburg: However, it sounds like they’re running a website to get the automatic magic feature of being able to join a meeting.

Simon Gibson: Like a little web server running on the local machine?

Steve Ginsburg: Yeah, it runs on the client, right? And it looks like [that] can be exploited, the researcher was able to show. And then there was a little bit of a concern, I’d say, from the path too, that there’s a note in the timeline that at a certain point, [the researcher] was notified that the security engineer was on vacation.

Simon Gibson: …when he reported the vulnerability.

Steve Ginsburg: Yeah. And I think security engineers should absolutely be able to take vacation, but ideally, there should be enough security researchers that something that looks as serious as this turned out to be, that a company can move quickly towards resolving [it] and really shouldn’t take a delay for staff outage. So I think that just goes under our general theme that we’ve been on about, that companies need to figure out how to provide excellent security. Hopefully, with each one of these events, enterprise listeners and people responsible for these programs will continue to have more fuel to improve them.

Simon Gibson: So interestingly, Steve, with Zoom, there isn’t a way that I was able to find, Googling around [for 5-10 minutes]— to report a vulnerability. There is no responsible disclosure program, they don’t have a portal and any kind of a policy or framework to let you submit a problem, if you happen to find one. Bug bounty program aside, if you are just a user of the service, I expect, as a paid member or even perhaps inside their licensing portal, you can file a support ticket [and] someone will get back to you, but in terms of the engineer, getting a reply [that] their security engineer isn’t in—I honestly am just not in the least bit surprised that that’s their answer. It’s unfortunate; it’s a 25 billion-dollar market cap company, but…

Steve Ginsburg: Yeah. And it really leads into our topic perfectly today. We’re looking at how companies should structure their pen-testing and bug bounty and have a program that’s robust and really improves their overall brand, the brand experience and the product experience, and also really leverages the large security community that’s out there.

Simon Gibson: Yeah. Very topical. So let’s get at it. Let’s get at bug bounties and pen testing and the values and differences between the two, first of all. I think that’s an important one.

Steve Ginsburg: Why don’t I let you do the definition?

Simon Gibson: Sure. I think it’s helpful to understand a little bit—a penetration test, or a bug bounty, they have the same goals. They want to do the same things. They go about them very differently, and I think the nuances in that are things that are the important ones that enterprises understand. My sense is that bug bounties began—my earliest recollection of bug bounties was partially based in the open-source world, with things like Apache and Linux kernels and Free BSD, and then the first commercial version probably was started at Microsoft, and is still arguably the best in the world, where they needed a method for Microsoft Windows end users to report vulnerabilities, and I think that’s how they got going.

Steve Ginsburg: Yeah, there was a period in the not-so-distant past where security was actually—it could have been something that would sink Microsoft operating systems. For awhile, there was just so many Windows security [features] that certainly, those of us in the Unix community felt a vast difference. But sadly, I have to say that over time, there were later some exploits in Unix at the core that were discovered that really took away some of the bragging rights that Unix would have; some very significant problems there, too.

Simon Gibson: I mean, I think security aside—just shelving it for a second—I think it was the ubiquitous Windows desktop and the fact that if a bug happened on one, it happened on all. So that means the entire world in effect, the entire enterprise; every system and every company everywhere, apart from the Unix machines or the SCADA machines or the op mainframes where everybody had that problem.

Steve Ginsburg: That’s right. And it makes it the thing to target, for most of the folks who want to target anything, because it’s also the opportunity—is really there.

Simon Gibson: Yep, good aside. The Chief Security Officer at Microsoft, a guy called Dan Geer, who founded @Stake, I think was at Microsoft a very short amount of time when he made that public statement and was fired promptly for saying that exact thing.

Steve Ginsburg: For representing that they were the big target because…

Simon Gibson: Yeah, absolutely. Went and founded a company called @Stake. Yeah, but that’s another story. For pen testing and bug bounty, really, it’s a way of setting some boundaries and some guidelines about how to report things. With penetration testing, I think that probably sprung up more out of the need for working within a big company and an enterprise. If you hire someone to pen test, you hire a company and you bring them in, or you build a group of employees.

Either way, it’s very much the way companies are used to doing business. There’s a master service agreement, there’s a contract, there’s an NDA, there’s legal teeth around it, there’s a scope, there’s terms of service, there’s not-to-exceeds, there’s timelines, there’s a set of deliverables. And all those things [must have value because] big companies don’t do things unless it’s going to add some sort of value; so somebody somewhere has calculated a value.

Steve Ginsburg: Yeah. In our last episode, we talked about SIEMs and situational awareness that a security team can build themselves. And of course, security teams can do their own pen tests—and we should talk about that—internal pen tests, but when you move to want to leverage other organizations to help you out, this is a great way to do it, and I think both provide pretty powerful models.

Simon Gibson: Yeah. I think it is definitely—it’s a question of the duration, the focus, and the level of comfort. And we can definitely talk about those. So the next big, important thing with this topic, pen test or bug bounties, are ranking vulnerabilities and scoring them.

Steve Ginsburg: Yeah. So one of the things you would ask me when we were looking at this is maybe to share a little bit from the CIO’s perspective of: How do you go into this? Why do you go into this? What are concerns that are going to happen? At least one of them is going to be: ‘Well, we’ve got a lot of different security issues potentially.’ If you have a complex product (and I think the examples we gave up front, both in voting and in the corporate world, say, over time), all digital code is probably exploitable. Maybe some things are so simple, they’re not. But generally speaking, if you have any complex organism in the digital side, there’s going to be some way to pry it open at some point, even if overall you have a good experience.

So I think, looking at that landscape and really being able to cull out priority, that’s one of the things that, as I came to understand it myself and when I’ve talked to peers, as an executive sponsor of that or somebody who’s going to be responsible for that program being financed and being undertaken, knowing that there’s going to be a value return for [it]—I don’t really want to just find a million trivial things that we’re not going to fix.

Simon Gibson: Exactly. And putting a rating system or a ranking system on a vulnerability discovered helps you then measure the risk to your business, the risk [of] it being discovered, the reputational risk, the availability risk, the risk of data being exposed, all those kinds of things. So it’s important to understand the ranking system. A common one is called CVSS, Common Vulnerability Scoring System. It was developed—and it effectively measures the impact of a vulnerability against criteria like availability: Is this vulnerability going to take down the system? It measures the vulnerability’s risk to exposing confidential information, and it measures it against exposing the data to reliability informations; can you trust the data? So those are sort of the three main criteria.

Steve Ginsburg: And you mentioned well before, but maybe worth reminding, is: Those things can be different, depending on the organization and depending on what area of an organization that is. So for example, there’s different types of confidential information, and some might be considered much more of a business risk than others. For example, HR data is an example you’ve given in the past where of course you never want that exposed. But it might not be as business-critical, depending on what it is, as customer data, for example, where…

Simon Gibson: Or exposing C-code, or exposing certain things and inner workings of applications that would then lead to more sophisticated attacks. There are nuances in that as well, but those are the three main things, and there’s a few others. But being able to measure your vulnerability is super important and, again, to your point, if you’re going to do these programs, you’re going to run these programs, you want to understand what the value of actually implementing them is.

And then I think you brought up earlier—what kind of trouble are these going to cause? Are things going to get taken down? Are things going to break?

Steve Ginsburg: Yeah. You know, all the conversations about modern companies involve the rapid rate of change and the increased business responsibility for companies to keep delivering quality product. And so anything that’s going to be [an] interruption of either teams that are developing or teams that are meant to secure or operate the organization, that has to be factored [in].

So on the one hand, there’s very high value in discovering any potential exploit and the trouble it can cause to availability and company reputation; on the other hand, one has to be careful that they’re not just creating busywork or disrupting quality work, even if for a valid reason.

Simon Gibson: Yep. One of the things that I think companies don’t really understand until they start grappling with these is embarrassment. If a vulnerability is found, doesn’t matter if it’s through a pen test or an end user reported it or an engineer found it. If a company realizes they have a critical vulnerability, and they need to patch it and inform customers about this, that—in a world where there’s apps and app updates and just people take rolling security patches all the time, there’s a little bit less of a worry around that, because people are… used to getting security updates, and it just happens and you don’t really need to explain a lot about it.

In a world where there’s routers or there’s data conduits and optics, whatever the thing is, to tell your biggest customers—your 10 million-dollar customers—‘Oh, we have this really risky vulnerability in your core networks, please patch this,’ companies have to be ready to ‘bite down on the rag’ and do that.

Steve Ginsburg: Sure. It’s not a happy discussion at that point. But I think also, folks who are doing vendor management within any organization, they’re going to look to: ‘Are my partners responsible about these things over time? How do they respond to these things ?’ So to your point, I think there is a great understanding that security risks happen, but companies they don’t manage it well do get managed out.

Simon Gibson: Yeah. And I think that we had a section about partner supply chain risk, and I think that that goes absolutely—I think companies really have to sit down and think at an executive level: ‘Do the benefits of me patching this vulnerability; telling my customer it’s critical; is there a risk that they are going to leave—stop buying from us?’ They’re going to not renew their contracts? Or are they going to look at us and think we’re a good partner. And are we really building credibility with them by coming to them ahead of a vulnerability being disclosed?—which is another super nuanced part about this as well.

Steve Ginsburg: Yes. And having their operational story very clear. We’d had engagements in the past where there might be a security issue or a reliability issue and then in calls dialing in to the company, it was clear that their folks actually did not have a clear vision of what was happening. In other words, some of the discussion about remediation, or possible steps to mitigate problems were not accurate.

So really understanding what—and you mentioned cloud security as we were talking about this, too—in a cloud world, that becomes potentially more difficult, which is a lot of companies are leveraging cloud for vast amounts of their infrastructure. Those that are doing it responsibly will understand what are the significant portions of—what are the implications of all that, and the detail, and those that don’t will potentially be in a place where they can’t enforce good security, or good security response.

Simon Gibson: Well, and I think that’s a good place to sort of rip apart responsible disclosure and coordinated disclosure. So in the world of telecom and routing and large interconnected systems, vulnerabilities discovered that could potentially affect the Internet at large, there’s sometimes the notion of a coordinated disclosure, where the people who fix it, who are responsible for maintaining these, get together, release a patch ahead of time of it being public, they go patch all the things, and then the vulnerability gets disclosed, and then everybody runs behind it and patches, but the core stuff is done.

And that’s the real nuance, which is: this vulnerability can be discovered whether or not you have a disclosure program. This will come out if somebody finds it. Or it’s going to be sold and kept quiet, and used on the black market as a zero day, and sold for potentially a lot of money, depending on what kind of vulnerability it is, on what platform and how reliably it triggers.

Steve Ginsburg: Right. And it also brings up for me—the immunity of the herd. And communities can be very helpful in security. And so, just kind of a call there that enterprise teams, people who are running security programs in any way, your security leaders and your IT leaders, they should be talking to other folks at other companies, at other organizations, about what they’re seeing for security, what they’re doing to improve their program.

Simon Gibson: But even if they’re busy guys and they’re not doing as much of that as they should, they should have a really good understanding that if a vulnerability is discovered and it’s brought to their attention, then they now have guilty knowledge that if this is disclosed by someone other than them to their customers, that’s probably worse than it being disclosed by the company that found it, right?

Steve Ginsburg: Absolutely.

Simon Gibson: All things to keep in mind. Again, this is such a nuanced space; I love it just because of specifically that. So we talked about why they’re different, and we talked a bit about cloud, so let’s get into: What are the things you need to do start doing pen tests repeatedly, reliably, or open a bug bounty or, at the very least, a responsible disclosure program, which, in the case of our opening topic, Zoom, they didn’t seem to have one.

Steve Ginsburg: Right. So one of the things that’s at the start is the executive sponsorship. I alluded to it before, but as we talked about earlier, it’s a very important piece, which is: you’re going to create this program, and there are multiple ways to go about it, in terms of what outside parties you use, how you leverage the outside community and your own teams to do these things. But when you raise issues—we talked about resourcing, we talked about priority—how are you going to make your way through all that?

It’s great when direct contributors can just work that out on their own, but they really need a framework, as we talked about, to make that work. And then if they have conflict, or they’re not sure whether work will be prioritized or what approaches should be taken, they need to be able to escalate that through management leadership. And if there’s not a clear path, you can get gridlock right there.

Simon Gibson: Yeah. I mean that’s for sure. Any well-meaning CISO can put a security at their company, and a little bit of indemnification—which we’ll talk about in a second—and start (sort of) a program. But what happens to the product when there’s a real critical vulnerability, and now you have to bring in the general counsel and the CEO, and they have to make a decision about how they talk about it: what would they tell their customers, what they tell their board; it does need executive sponsorship. And also, because if people are going to spend money and hire engineers, or they’re going to take engineers off other projects to work on this, there needs to be some value.

So somebody needs to work out what the value is in having a vulnerability disclosure program. How much does that add to the QA process? Hiring a pen test, they’re not cheap. Pen tests can be many hundreds of thousands of dollars, depending on the scope and the time and who you’re hiring. So what is the actual value proposition? Is it reputational risk? Is it [that] you need to be seen by your shareholders and your board as having done these things? Are you doing M&A, are you buying a company, and do you want to pen test their code and see how they are before you actually sign the deal? Or give them a terms sheet? So there’s a million reasons why these things have value, but a company needs the executive leadership to really work that out. I think the CSO and CISO are the guys that can do a good job explaining it, if they understand this space well.

Steve Ginsburg: Yeah. I think M&A is a perfect example. There’s certainly lots of cases about companies having been acquired, and then greater security risks [have] been discovered after the fact, which is certainly a pretty big business risk, if you’re the one who has done the acquiring, and the asset that you have doesn’t have the value that you thought, because there are security risks present, for example.

Simon Gibson: Yeah, or loses value right after you bought it because something was disclosed; some vulnerability and some piece of medical equipment was…and you were shorted. So the other thing, again, apart from just the reputational aspects and the executive sponsorship for a program, a legal framework is something that you need to understand really clearly before you start wading into pen tests and bug bounties and disclosure programs.

Steve Ginsburg: Yeah, certainly for disclosure, there are national, and certainly there are state, laws which might be different than your overall commitment. I know in California there are strong disclosure laws, for example. So there might be some real important actions that you’re going to need to take. Your legal team, and then your operational team, as a result, need to be clear what those are.

Simon Gibson: Right. And I think it’s important [to unpack this]—we use this word ‘disclosure’ kind of interchangeably in that sense—you know, in the one sense, there’s the company disclosing that they have had a breach and notifying the people…that varies from state to state; there’s a disclosure policy that needs to be around…what you will disclose to the community at large, what you’re willing to expose about how your company works and what happened, and also a disclosure outbound to the researchers who are in the bug bounty, about what you have in scope, and you have disclosed: these are the rules of the road; if we’re going to do a bug bounty or a pen test, this is the scope around it. So there’s a disclosure piece around that as well. So it kind of goes both ways, and the word ‘disclosure’ is an awfully—it’s a very large word. It encompasses a lot of things.

Steve Ginsburg: Right. It’s easy to just say, “the time when you’re going to say something,” but right, it has some very specific context in this realm.

Simon Gibson: Yeah. It’s definitely—it’s very contextual in how you use it. You know, the legal framework around DMCA and the cComputer Fraud and Abuse [Act] is another important thing, and this goes into executive sponsorship, and the executives need to be made aware of this. If I open a bug bounty and take, for example, Simon Widgets does a project, and we’re like, ‘Go ahead and hack it,’ so I’m protected by DMCA and Computer Fraud and Abuse [law].

If somebody hacks into my company, I can prosecute them. And that’s why you don’t just see people attacking websites and, ‘Oh, I’ve attacked a website.’ No, you’re going to jail; you have hacked a website. If you hack a website as part of a bug bounty, somebody has indemnified the work that you’ve done. Otherwise, you’re a hacker, and that’s against the law. It’s a really important thing. So when you do decide to indemnify, are you risking bringing in a hacker? And now you can’t sue them?

Steve Ginsburg: Now you’ve given them a legal cover.

Simon Gibson: Exactly.

Steve Ginsburg: Yeah, and just—this may be obvious, but part of the framework to get involved with having a bug bounty in the first place is—those of us who are involved in security know that you’re basically seeing automated ‘door latch’ attacks…

Simon Gibson: That’s a good analogy, sure, yeah.

Steve Ginsburg: Constantly, right.

Simon Gibson: Yeah. Door-rattling.

Steve Ginsburg: People are charging for an open piece, and at the heart of it, it’s a good example that the bug bounty is really about taking an expert community and saying, ‘Okay, I will provide you a lane in, where we will share that. And when you mention disclosure, one of the problems about not having a bug bounty program in place is what if you do get a responsible request from a security researcher who’s found something?

Security researchers and hackers—there’s a wide range, of course, of personalities out there, and so you’re going to have folks who are really the bad guys, and they’re just going to try to get in and do whatever, and their approach to you is going to be whatever that is, but you really have very concerned, responsible security researchers, and some of those are independent folks who—they do view it as a real, legitimate job in their world. And so they’re going to want to be compensated, but…

Simon Gibson: Or at least recognized.

Steve Ginsburg: Yeah. That’s right. It can be different, depending on what—but you don’t want to do every one as a one-off situation.

Simon Gibson: Yeah. You’re going to want it fixed. If I have a piece of software, especially if I’m paying for it, I want to notify the company: ‘You know, there’s a problem, I can exploit this. You got to get it fixed.’ So there’s definitely a sense of urgency. But you know, at the end of the day, whether or not you have that program, bugs can drop; people will announce those things. Even if there isn’t a Secure [Technology] Act company or a vulnerability disclosure program or a bug bounty, people can just announce it.

Google has a pretty strict policy with their zero day project, which has done a lot to find bugs in software. They actively research and will let the companies know they have 90 days to respond and fix this, and if they don’t, they go public. I think if the company’s working really hard to fix it, they’ll give them some leeway, but Tavis Ormandy could show up with zero day, and everybody better drop everything, ‘cause in 90 days, they’re going to just release vulnerability about your product.

Steve Ginsburg: Yeah. And I think that also, a big part of wanting to fix these things is—we’ve talked about [it] before—a lot of hacks sometimes takes a long time to be discovered, and not only do you want to generally know, but you want to know soon. If there is an exploit on your website and someone get into your internal network, or get into your customer data, if you can find that out—I mean, ideally you might find it out before it’s exploited, right: a responsible security researcher finds it and then tells you, you remediate it, and then your customer data is never threatened, for example. If you don’t have a program like this, sometimes people can be living on your systems for months or years, right?

Simon Gibson: Or just shelving that vulnerability for use when they’re ready to. I mean, that’s the whole market for zero day. Rand just did a big, long—basically a book about it maybe a year ago, maybe a little bit longer—but there’s a whole market for zero days. And it’s an interesting economic incentivization model that some of the more modern pen testing companies have adopted. So in the zero day market model, the better the vulnerability, the more reliably it triggers, and the platform it triggers on—so the scarcity—equals the value of the vulnerability.

So for example: a reasonably good iOS bug that can infect an iPhone that no one knows about is probably on the order of $500,000. It’s got a face value of that, give or take a little, depending on who’s buying it and what the nature of it is. But it’s a lot of money. So the researchers who work on finding these, and if you find two or three of those a year, you’ve got a small company and you work from home, and you’re doing okay. It’s super illegal; you’re probably on a whole lot of strange government lists, but it’s a market. It’s an economy.

What some of the modern pen test companies have done—there’s a couple of them; Synack is one of them—is understood that paying researchers to work on a penetration test by the hour doesn’t necessarily incentivize them; rather, pay the pen tester on the quality of the vulnerability they find during the pen test. And what Synack found was they get many more hours out of their researchers. So imagine, even you’re salaried, and you’re expected to work 8 or 10 hours a day, but you’re incentivized by the vulnerability, you might stay up all night and work on this and come back for… you might spend your weekends and evenings and just be crushing this because you’re finding vulnerabilities. And then what that ends up in is really high-value return for the customer.

Steve Ginsburg: Yeah. A big way that I looked at it was it was essentially taking the diversity of who security hackers—white and black hat—are, the wide diversity. Basically, the way I looked at it was—There will be black hats coming at us; and this is a way to have white hat hackers working for you.

Simon Gibson: Right. And this gets into the product, which is—the ability to vet researchers reliably; the ability to make sure that whatever they’re doing, there’s some controls around it. So you know, some of the companies we looked at in the pen test and bug bounty report have very novel methods for letting researchers get access to the systems that they’re testing, so that they can be monitored. Again, it goes to the point of: if I let somebody hack my system, am I really sure they’ve left it, and they didn’t put anything on the system that I don’t know about now? And can I be sure of that? That’s a difficult question to answer in some environments.

Steve Ginsburg: Right. This doesn’t replace the need for your SIEM and situational awareness from your own direct monitoring at all. But it can certainly enhance my getting a kind of more 360 view, by definition.

Simon Gibson: Yeah. But for sure, opening the door and allowing hackers to come in is not… I think most companies are pretty averse to that. And so understanding the costs/benefits—it’s an important analysis to do. The next thing that’s really, really, really important is an internal process for this kind of stuff. Just the communication between somebody reporting a vulnerability, acknowledging you’ve received it, and then some sort of a guideline as to how long you’re going to take to respond. Just having something that simple—because again, imagine the researcher who finds a vulnerability in a piece of software running on his or her machine that makes their machine vulnerable: well, they’re paying for the software, they want it fixed, or they’re not, it’s—but regardless, they are feeling a little betrayed, and they understand if they have—this piece of software is being used by tens of millions, or hundreds of millions of people, now there starts to be a little bit of pressure on this researcher.

The very least the company can do is say, ‘Thank you for reporting this.’ The company will usually ask you [for] a way to reproduce it, so the company can verify the vulnerability, and then responding back and saying, ‘We’ve taken this in; this is truly a vulnerability. We are going to fix it.’ You need a process to do that. I mean, even finding the right programming group in a big company to address the vulnerability can be challenging. I can submit a bug, but who’s the development team that owns the bug?

Steve Ginsburg: Right. And that communication itself can be an interesting point because of the diversity of the community. That security researcher who’s reporting a bug might have all sorts of different expectations. We already came up with a few different things they might be wanting or needing, and the companies who are going to be responding, they need to be sensitive to that.

One example again—not to dwell on the Zoom situation—I just thought it was very interesting that a security researcher who—overall I would give him high marks from just my personal opinion, from what I saw of how he reported it and the write-up on it—but one thing that just caught my attention was that one of the features that he was talking about that he called to their attention, Zoom gave a very polished answer that they wanted to keep their customers having the flexibility to either use this feature or not, and it was clearly a well thought-out and—he called them on it being a PR answer and essentially said negatively, ‘I don’t really want to hear a PR. answer in the middle of an ongoing security discussion.’ Which is a fair point.

On the other side, that’s one I had to see from both sides, which is: the company is communicating [about] something that, as it turns out, is ultimately going to reach the public. And so perhaps a polished, professional answer is the way to lead in some of these cases. But I think both of those are good points, and striking the right balance is really the way to go. You mentioned that with different PR firms, you might get a different response, too, if you get to the point where you’re going to a fully public discussion on a situation.

Simon Gibson: Yeah. For an incident that—it’s a very different type of public relations to manage a crisis than it is to get your latest feature into the Wall Street Journal. It’s a different company, or it’s a different discipline.

Steve Ginsburg: [Different] team in the company.

Simon Gibson: Yeah, a different team. And not only do you need a process to manage communication; you need to be able to manage internally about—so we’ve got a bug, somebody needs to verify it. Now that it’s verified, there needs to be a ticket. So is there a zero ticket open? Is there a help desk ticket? Where this can fold in nicely is with companies that already have help desk and ticket support systems. This can run right alongside your—when a customer has an outage or a critical severity bug, and you have a way to measure those things and already work on those, a vulnerability program can run alongside of it, but you still need to build it out. You still need to make sure that when the person on the help desk, or the person on the customer service team gets the report, they have a script and they know what to say.

And then they have a tool that they can put the bug into, and you know, there may not be a zero project for security. There just might not be. So maybe, if you don’t create a zero project, maybe you have some special tag that you build in to these kinds of things: you flag them especially so they can be tracked and remediated, and you have a process to report on them and an SLA, and all those kinds of things.

And then I think you brought this up too, in terms of rules of engagement: how much money are you going to spend? And once you rank the severity of the bug, how are you going to—once it’s reported, and based on the severity, what’s the incentivization program?

Steve Ginsburg: Right. And we mentioned, I think, for companies that don’t have active incidents coming through—so some companies will start a program, or they’re already engaged in ad hoc methods of dealing with these security bugs, and then they bring the program… retrofit as things are moving—but for companies that aren’t very active, they definitely need to be practicing these things, too. Because all this communication and incident response team—if those muscles aren’t flexed… Also, you’ll find out people don’t respond the way that you might expect, even if it was written in a plan that everyone agreed to six months ago, that type of thing.

Simon Gibson: Yeah, it’s true, and in some cases a vulnerability—a bug bounty—it may yield a lot of low-hanging fruit that gets repaired quickly, and it didn’t really flex those muscles. A pen test, or an ongoing pen test quarterly, or whatever your release cycle is, that kind of helps keep those muscles going, and that’s something you can hire for and that you keep rolling. And again, it’s a good differentiator; I think it’s important that they have the same goals, but they perform different functions. You know, the pen test and the bug bounty, it’s important to think about them very differently.

Steve Ginsburg: I think one thing too that, again, might be obvious, but I’m not sure that we really made clear here: for smaller companies that are getting involved and are considering pen tests and bug bounties, we mentioned leveraging a community to do that. One of the things that can be very dramatic about this is leveraging a much larger scale than you have. So most companies are struggling to keep an appropriate number of security engineers on staff.

We’ve talked about—[it] depends on the organization, but let’s face it, most organizations would rather pay for whatever is obviously driving revenue than the security aspect, which does enhance revenue, and in some cases can drive revenue by great reputation and great answers for security reviews, and things like that. So security teams can drive revenue, but it’s not as obvious as the core product of many companies, many organizations.

Simon Gibson: For sure.

Steve Ginsburg: So as a result, most security teams are not going to have dozens of extra employees, for example. And so the bug bounty and the pen tests can be used in coordinated methods, either together or alternating, ideally ongoing, to really bring a much larger pool of individuals looking at your website or your public…

Simon Gibson: Yeah, whatever you’re making.

Steve Ginsburg: Yeah. Your public footprint, right?

Simon Gibson: Yeah. Another interesting use case around cloud is…Initially, I looked at different cloud offerings, and I spoke to the CISO—or CSO—and one of my concerns was, ‘Aren’t you a really juicy target? There’s so much stuff up there. Isn’t everyone coming after you?’ ‘Well, yes, but we have a bunch of Fortune 100s, and some Fortune 10s—they’ve all pen-tested us, and they’ve all found different things, and so now every company who uses us benefits from the work they did. And so that’s an interesting way to think about cloud, is that there can be very specific focused pen tests if a large Fortune 10 company wants to go use some SaaS service and it’s an important…public-private cloud…some sort of relationship—you will probably get a pen test on that. And then you, the company that can’t necessarily afford that, will benefit from those things.

Steve Ginsburg: Right. And that’s also a good example of: if you can afford it, you’d like to do it before your next biggest customer or potential customer does it. And finds out that there’s serious problems, that they don’t want to do business [with you].

Simon Gibson: That’s a really interesting one, where I had a team that worked for me. We would routinely test things and find pretty significant problems. I mean, it was a pretty routine thing, and not trivial problems, but real, real serious problems. And it didn’t mean we didn’t want to do business, but what would hurt the vendors we worked with is them taking a long time to fix our problems.

We had a particular vendor where we had millions of dollars embargoed, and all the different leaders around the company agreed not to buy any of their stuff because of these vulnerabilities. And it took them many quarters to fix it. And then they finally did, and we were able to verify that the fix was in, and they ended up becoming a big customer. So the thing that will hurt you isn’t the pen test; it’s either the refusal to fix it or the priorities…those are the things as a big company that will hurt you.

Steve Ginsburg: Yeah. And from the flip side, that’s a great way to show how security can drive revenue. If you do a great job, better than your competitors on that, you’re now in business, and they’re going to come to you and move forward.

Simon Gibson: So let’s get into a couple of the challenges…How you’re messaging things is important. We sort of hit that with the crisis communication plan. I have always really thought it’s important, and implemented these to have a crisis communication plan in the desk. So that once something like this does happen, you have the right lawyers to hire, you have the right firm to hire, you have the right messaging. This terrible thing happened (insert thing here), this is what we’re doing about it; this is how long we expect it to take; here are the people that are engaged on it—have a plan that works with the community, with the rest of the world.

Another thing is that in the executive sponsorship part, having everybody agree to the severity of a vulnerability is very important at the onset, and the priorities that should be given when they’re presented.

Steve Ginsburg: Absolutely. And with both of these, I think, a big part is to consider that when security events happen, one shouldn’t think of them as sort of happening in this vacuum where it’s like, ‘Okay, this security event is going on.’ If there are security events, they’re going to happen at the same time that other things are happening. They’re going to happen when your company has a big trade show or quarterly reporting or the executives are on their ‘off site’ [meeting] or people are traveling overseas, or any…

Simon Gibson: Product launch.

Steve Ginsburg: Any number of things. Companies are very busy. And the individuals who work, especially at the board level and the exec level, they’re incredibly busy. And so you need to know—ideally, you don’t want to pull those people out of meetings, if you don’t need to, and then if you need to, you need to be ready to do that.

Simon Gibson: Well, and they have to have an agreement up front. Because there’s nothing worse than sitting in a room of executives and having three of them agree, ‘This is a vulnerability, we should do something,’ and two of them, ‘Well, I don’t know how important this is, and I think we should just keep on doing the other thing.’ You really need a clear guideline and a clear matrix of, ‘This has now crossed the threshold into sev 1, and we need to do everything or this is sev 3, and we’ll issue a workaround, and we’ll get afix out, but that stuff really needs to be agreed on at once, ‘cause you don’t want to hit that deadlock.

Steve Ginsburg: That’s right. And we talked about in our last episode how having a clear message from your monitoring to have the clear story, and if you’re in an evolving situation, it’s really a combination of having good information coming from the outside, good information coming from the inside, and then having that fit into a clear definition, as you were saying of: what will those things mean if…

Simon Gibson: Right, lacking ambiguity, and it’s not up to one person to say, ‘Well, I think…’ Just: you have crossed the threshold, and it’s empirical. The other thing, you know, we talked about flexing muscles: one of the things that I think is taken for granted but is an important factor is that flexing these muscles is important, and learning from all these things is important. If you do a vulnerability program, a pen test, if you have a bug bounty, every 3-6 months with new releases you’re getting cross-site scripting, or a SQL injection, maybe there’s some training that could happen that would prevent these kinds of things, you know?

Steve Ginsburg: Yeah. There is a lot of security training that can benefit developers and others in organizations. It’s really about how they’re going about their work over time.

Simon Gibson: Yeah, yeah. And flexing these muscles means looking at the trends, and then applying the right security training. We see this one problem recurring; is there a group here that’s doing this, or is this across the company? Do we need some libraries or some way to link things that now sanitize inputs? Do we need a process that needs to be deployed for our programmers?

Steve Ginsburg: Right. And that can sometimes be about seniority of teams, but sometimes it’s not about that at all.

Simon Gibson: The busyness.

Steve Ginsburg: And also, there’s all sorts of specialties. Sometimes coming from webscale companies and dealing with mostly that, I tend to think of software developers of a certain type, but across all enterprises there are software developers who are specialized closer to machine hardware, and closer to any number of things where their sense of the modern web-available security exploits might be very, very different, and yet they might still come across that. A good example would be how web cameras become exploitable in the world. That’s a device that maybe didn’t even consider itself a security device in any way, and yet that can be responsible for some of the biggest storms on the Internet.

Simon Gibson: Yeah. And I think that’s a good way to sort of wrap this up, which is: these are extremely valuable tests, whether it’s physical security—you think you have controls to access to buildings and doors and perimeters—maybe they don’t work the way you think they do. You issue a badge in one place a person has to walk through an area—can they swap out the badge for another color and get access before anybody has a chance to get a look at it?

There’s all sorts of sneaky things that people will do and think outside the box when you believe you have a set of controls that work around controlling access to a database. Is there some hardcoded credential somewhere that somebody is using directory traversal? Is there somebody somewhere looking at something in a way that you’re not? And these tests prove that. These tests show that, and they are super valuable.

Bug bounties can cost very little; you can spend some money on high-quality vulnerabilities that are submitted. You can spend a ton of money on them; you can hire pen tests that do simple things like scan; or pay seriously experienced researchers to work hard on a specific application before you release it and know and find things. There’s a ton of value about understanding the values and the risks is super important. It’s one of those things that companies should not avoid doing, but they need to understand the risks. Fortunately, I think today, this has evolved such that there are a lot of good partners to work with. And that’s what our report’s going to cover.

Steve Ginsburg: Yeah, absolutely, and maybe just a final thought from my side: we talked a bit about targeting it, and I think scope is—for those who are thinking about this and maybe if they’re newer and want to get engaged, is: for each one of these, they can be intelligently scoped to start, and then you can widen them as it makes sense, or launch multiple efforts as it makes sense.

Simon Gibson: And even yes, for very large companies who don’t want to necessarily expose everything wrong with them, I think scoping things is very important, for sure. I think that’s it. This was a good one. Thanks, Steve.

Steve Ginsburg: Thanks, Simon.

Simon Gibson: Thanks for listening to Right Size Security.

The post Right Size Security – Episode 5: Bug Bounties appeared first on Gigaom.

Host Steve Ginsberg speaks with Bill Norton of Equinix about network peering and network strategy from a CIO perspective.

This is the second in a two-part episode. Find the first part here.

Guest

Most know Bill Norton as Co-Founder and former Chief Technical Liaison Equinix, or as DrPeering – author of “The Internet Peering Playbook: Connecting to the Core of the Internet.” Some may remember him as the first chairman of NANOG (1995-1998). He is a thought leader, a passionate Internet engineer with deep interest and expertise in the Internet interconnection and colocation sector.

Today, he helps executive teams strategically leverage SDN, BGP peering and cloud interconnection to their businesses, and assist with market research, sales, lead generation and software / DevOps / network engineering. Recent engagements have him building Internet peering ecosystems in the Middle East and consulting on peering and transit for large-scale global cloud gaming.

Transcript

Steve Ginsberg: I think a lot of enterprises have moved to the cloud as part of a corporate initiative that had a very fast timeline in some cases. When I talk to peers, sometimes it’s driven by the board as much as by the organization itself.

Whoever instantiates it, I think a lot of IT organizations are catching up with the reality, waking up to it, and as we just discussed, with more than one cloud, that kind of traffic that you just mentioned—that’s easy to generate the second you start moving…Part of the reason people move to different clouds is differing capabilities, so you might have developers who want to be in Google Cloud or they might want to be in Azure and their main organization is in Amazon, or it could be the other way exactly flipped. Once that starts happening, it’s easy to start having teams move a lot of resource assets between the different locations.

Bill Norton: Yeah, the other thing I’ve seen is you might find as you look around that there are some people who were just brought up using the AWS portal and they live on that portal. They know how to spin up easy instances like that and move them around and to turn them off. There’s even some logic that people have put into their scripts to do tests of the virtual machine that they just spun up because all virtual machines are not created equal.

And sometimes you’ll find a virtual machine that does not have the IO that you need or does not have the speed for whatever reason that you need and they’ll turn it down and hope to go back to the pool and get a better virtual machine. It’s kind of interesting. Google and Microsoft are pushing heavily to make it easy to do a ‘lift and shift.’ They can be less expensive in some cases; in some cases the network connectivity might be more expensive in another location.

But what you find is interesting Steve, is that you find different personalities leaning towards these different platforms. For example corporate America might say AWS is perfect—it’s what everyone goes for [so] that’s where we’re going to put our information. Other folks would say “You know we’re a Microsoft shop; everything we do day in and day out is Microsoft; we’re dot net and all the tools we use…” I worked with a client in cloud gaming that was entirely focused on Microsoft stuff. Those folks would probably be more comfortable and more familiar with the Azure portal interface. I found the Azure interface to be a bit more complicated and different. See I came up in the AWS side, so Azure looks foreign to me, and that’s why I think maybe people will be comfortable with that which they have started to use because they’re using that particular…

It’s interesting, so I would normally hear that part of the discussion being based on the people throwing up BMs essentially like you know the developers and IT people supporting developers, those teams, but are you saying also that the portals make a difference too for the networking teams? Or are you talking more as a developer working in the cloud in that case?

I’m thinking more about the user and what their preference is. For example I’m spinning up a new version of my website now and I have a choice: Do I want put up on a free AWS account or free Azure account or a free Google Cloud Platform account? I would probably go where I’m most comfortable, where I’m familiar and I can do things fairly quickly.

Sure. So you’re talking really as the corporate end user or developer, folks using cloud service not necessarily setting up the infrastructure underneath?

If you talk to the research folks in an enterprise though, you might find them more comfortable with the Google platform. Yeah because maybe that’s where they did their masters work when they’re off at school. For whatever reason, it does seem to be some kind of categorizations of appeal that I’ve seen in the field today. Maybe it’s going to change, but as it stands now it seems to me that AWS is really strong in the corporate market. Microsoft’s really strong in the dot net space, and if you’re a Microsoft shop that’s probably where you’re gonna be going. And Google is kind of the researcher, ‘the nerd’ for lack of a better word, who loves to play with the toys. Google’s got some great toys.

Yeah I think that’s fair and I think the folks who run each one of those cloud services would want to say “Oh no, we absolutely appeal to these other audiences.” And I think in fairness there probably is some cross-pollination. But I agree that certainly they’re all communities ultimately, and the communities have brought in some types of members first, and then I think as these services are going through their rapid development, some of this changes over time and some may grow stronger.

This is one of the really important points that your audience should know about, and that is that each of these cloud connection techniques Google Cloud interconnect, Express Route, AWS Direct Connect, each of them does essentially the same kind of thing. They allow you to get access to your resources in their particular clouds. The difference though is they use different terms—and that can be really confusing and tends to make people want to stay in their own cloud of comfort. The terminology is one of the forms of lock-in that you have. Whatever terminology, whatever environment she is comfortable with—that’s where you tend to stay.

Yeah I did some research on Kubernetes and containers in the past year and when you start to look at that, it takes a while to kind of wrap your head around ‘At what level is the computing actually happening here?’ And to your point: What services are actually the same as something that’s in a computing cloud and what are actually somewhat distinguished in that way?

And the other decision you have to make is: do you want to download and install open source software yourself or do you want to pay Amazon, who’s already doing that for a lot of other people and pay them a fee? It depends on whether you have the in-house expertise and if it’s strategic for you to control your network and control your software stack like that.

You’ve traveled all over the world to talk to people about peering. Do you have a favorite story from doing that?

Well I’d say my favorite story is probably the 111 8th Street story. There was a power outage in New York. I’m not sure it might have been 2006 or 2007 something like that. The power went out for the entire New York City. Now in downtown New York City there’s a place called 111 8th Street. That’s a major carrier hotel. This is where all the fibers from Europe are coming in and terminating and it’s a major carrier hotel is the point.

Now the power went out for the entire city of New York. And what happens in those cases is, there is a thing called ‘automatic transfer switch’ that’s down in the basement of 111 8th Street that kicks over when the power from the street is turned off; and it kicks over and all of the power for the building is from these UPS uninterruptible power supplies. So for a short period of time the UPS, these batteries are powering the entire building until the generators start up, up on the roof of the building. Those generators—once they kick in, they can power the entire building for days. As long as they have fuel for the generators, they can provide enough power.

Well it seemed like the right things were happening. The power went off, the automatic transfer switch kicked over, UPS took over the building load, generators started up on the roof of the building. The other thing you need to know is up on the roof of the building for those generators were 500 gallon tanks. That was to provide fuel for all of the generators. Now down in the basement after 9/11, they had to have all of the major, the large amounts of fuel underground—not on top of a building for security reasons. So the big 50,000 gallon tank was down underneath. And the idea was that when the power went out, the fuel pump down in the basement would pump the fuel from the 50,000 gallon [tank] in the basement all the way up to the rooftop and continually supply the 500 gallon tanks that supplied the diesel generators. That’s a long setup, are you with me though?

I think so.

OK so what happened was the building owner sent a note out to everyone saying “We’ve lost power, there’s a citywide power outage, [with] no estimated time of repair. But everything is fine, the generators are cranking away. Your equipment is fine at 111 Eighth Street.” Then another note came out. The generator kicked off one of the generators on the roof, (there’s four of them I think, maybe six), one of them kicked off and the building owner said “What’s going on there?” They sent the guy off to the roof to figure out what’s going on and in the time it took him to get to the roof, second generator kicked off and it was off.

Then they sent a note to every tenant saying, “Please turn off unnecessary equipment, we’re having a mechanical problem with our generators” and then the third one kicked off, and then a fourth one kicked off and then they sent another note saying: you have to turn off all nonessential equipment we’re down to two generators for the entire building, that kind of thing. What’s going on here? What is causing these generators to kick out? Well the guy up on the roof is trying to figure out that that very thing. And he goes down to the basement and he sees that the fuel pump is cranking away, so that’s the right sound a fuel pump should be [making] refilling the tanks. He goes up to the roof [and] all the tanks are empty. Last generator kicked off, now they have no power. Why?

Well it turned out that in a previous month or two, they had replaced the fuel pump and they had tested the fuel pump. But the test involved just seeing if the fuel pump made sounds. Turned out they had the polarity wrong on the pump. Instead of pumping fuel up to the rooftop, it was pumping the 500 gallon tank fuel down into the 50,000 gallon it just drained, so it drained the tanks and the guy’s scratching his head—what’s going on here? He goes back down and he hears the fuel pump still cranking away and he tries flipping the polarity and then all of a sudden he hears the fuel going up through the pipes, and they say “Oh my God, they tested it but they didn’t test to make sure that the the polarity was proper and that this was fully working.”

So then he goes up to the roof expecting the generators [are] going to be running, [but] they’re not. They’re actually silent and there’s plenty of fuel, the fuel tanks are all capped off. What happened? Well it turned out during all this going up and down the stairs because remember there’s no power, no elevators, so he had to go up and down the stairs. All this time that was taken going up down the stairs, the generators were constantly trying to turn over and they burned out the starter motors. So they had to go across the partying streets of New York City, everyone was having a big old fun time down there and get to find replacement starter motors for their generators, six of them, and they finally find them, they go back (this is hours and hours and hours go by by the way), and they replace the starter motors and sure enough, the starting motors are cranking over but they can’t seem to get the generators to work. Why not? Well it turns out that when the fuel tanks got drained down to the bottom and all that sludge on the bottom of the diesel tanks clogged the fuel filters on the generators. So now the guy had to go back across the city of partying, celebrating folks to find replacement fuel filters to put back into the generators to get them back and working.

So to me this is an interesting story, not to pooh pooh the systems that they had in place. They had all the same credentials that any CIO would see on a data center: two and plus one redundancy. We have fuel trucks that are contracted take two different routes to deliver fuel. All of this sort of stuff they have, but look what happened here, we have a cascading set of failures one after another, the first one, the polarity caused the fuel filters to get clogged and the starter motors to burn out and these sort of things happen in real life.

Now the Internet service providers and the carrier community’s reaction to this 24 hour+ outage was fascinating to me because they weren’t really annoyed that they had to go without days of power. They were annoyed because they didn’t get scheduled updates from the building owner. And I thought it was interesting—not that there was a problem—I guess everyone understands that there will be things that will break, bad things will happen, but the real question is: ‘How does your vendor treat you during the time when they’re having a really tough time? Are you a partner or are you more like someone that they want to hide stuff from?’ That was really the kind of feeling that I got from that story.

Thanks Bill, yeah that’s a great cautionary tale in testing and in thinking through a failure plan—what goes beyond the first few levels from there. And a good reminder too on peering, the subject of peering that one of the reasons to connect at various peering exchanges is exactly this. So I think what you are saying in part is a lot of the peering community, they weren’t too upset there because their peering traffic would have just failed over to another location and been been kind of picked up somewhere else.

That’s right. That’s another interesting tradeoff that you’ll find in the Internet peering ecosystem. I asked peering coordinators at one point, ”How many exchange points would you like to see in a region for offloading your traffic in either peering or transit relationships?” And I thought the answer would be: “Well, one. We want to build into one location only so we can get our traffic offloaded in that one place, and it turns out that half of the people said “yes, we want exactly one location.” I said, “Well how about redundancy?” They said exactly what you said, they said “Bill for redundancy, we don’t want to have extra cost in our peering infrastructure that we have to pay for. We handle redundancy by routing traffic to another place where we interconnect with the same people, they would say we interconnect with those people in multiple locations. And even if we didn’t, having you know just a short period of time where we don’t have the optimal peering path to that destination, that’s something that we can deal with.”

That was half of the audience saying exactly “one.” The other half the audience said, “We want exactly two exchange points per region,” and I said, “that means double the cost.” They would say “Fine, I don’t care about double the cost, I want the redundancy, I want there to be different exchange points operated by different people, different networks, different companies that operate it, I want them to use different switches, different vendors’ switches and we want them to use different security guards.” They want to have the redundancy be as far into their infrastructure as that peering infrastructure having different everything. So it’s a it’s a way of removing systematic risk.

So it’s great to hear how companies are approaching their peering strategy in that way—that they might have some difference from there. How would you characterize peering exchanges changing over time in the current day? Are things different than they were five years ago? Transit costs have come down, so maybe there’s the idea that peering is a little bit different than it was certainly 10 years ago?

Oh absolutely. You know when I first started working with Equinix, I traveled about 90% of the time and that’s how I developed all these relationships and how I collected all these stories that I put in the book. [I had] a lot of conversations with folks in the field and when I first started I would ask people, “What’s a rough number of the cost to buy internet transit these days?” And back in 1998 when Equinix started, the price was $1200 per megabit per second. Four years later the price was $120 per megabit per second. Four years later the price was $12 a megabit per second. And sure enough we’re now four years after that, and it’s about a $1.20 per meg, and many times, less than that.

So we’re heading towards 12 cents per meg as the next stopping point. But this has been going on since the beginning of the commercial Internet. Every year the price drops, every year the ISPs say “no one’s making any money at these prices.” And every year the prices go down yet again. There are efficiencies in optical equipment that allow you to get things like you know 40 gigs or 100 gigs, 400 gigs is coming pretty soon. So these help make it possible to deliver large amounts of bits for lower and lower prices. So that changes the landscape, but we’ve always had this tension between the ever dropping price of transit and what is the actual cost of peering, because with peering you do need to buy, you need to have a router you need to have transport into a place where you can exchange traffic with somebody else. Maybe you need co-location space for a router. All these different expenses for implementing peering have to go into the mix to figure out: does peering make sense financially? It is a difficult case to make these days that you’re going to save money but it still can be done.

I know some companies that are doing some pretty clever things to minimize their transit costs. I’ll share one of the tactics from my book. What you do is you take a look at the way that you’re doing global traffic distribution, and if you are buying in transit in different regions from different ISPs, you have a bit of a challenge because the 95th percentile—which is how things are are priced in Internet space, the 95th percentile in Europe will be at a different time than the 95th percentile in the United States, at a different time from the 95th percentile in Asia. So you can be paying…

So you’re saying if your peak traffic kind of goes up at the same time it’s because it’s distributed globally and also because audiences will naturally… if a service is commonly used at 9:00 at night or at 1:00pm—that time follows the globe and therefore your peak will move with it?

That’s right. So the clever trick that one enterprise did was they used a single vendor and they told that vendor: “globally I want the same time scale to be used for all of my traffic around the world.” And that way the peaks of the 95th percentile in Europe are offset by the valleys in North America and in Asia and likewise, the peaks in North America are offset by the valleys in the other parts of the world. And by doing so, you end up having a much flatter demand curve that allows you to get better usage at a better price.

So for global enterprises, that’s a pretty powerful piece to get into your contract negotiation early and make sure it’s in the contract?

And you have to ask for it. You have to negotiate for it. There are a whole bunch of really interesting techniques that folks use to try to maximize the price/performance.

Well thanks, Bill. I’m really enjoying our conversation. I wonder if maybe you could kind of bring it all back together. Why should enterprises care about peering? What should they be most hopeful about gaining from having an active peering program, as opposed to say, just letting their transit provider deal with it all?

Yeah it’s an interesting question. As I said before, security is a number one reason that people go down the path of pursuing peering. The second reason is reliability. When people want to have a connection into say Salesforce.com, the direct path is one that has the fewest moving parts in between you and Salesforce.com: fewer routers, fewer links to go down, fewer networks that could be involved in the transaction. So number two is reliability, number one security, the third reason is performance.

I did some consulting work for a cloud gaming company recently and these guys have incredibly tight network requirements, extremely low latency, extremely low jitter, large amount of traffic and almost no tolerance for packet loss. Packets that are dropped will be re-transmitted before TCP could even have a chance to take a look at whether it needs to be retransmitted. So these are the types of enterprises that have specific network requirements that require the kind of reliability that being closer to the eyeballs can give you.

So that’s an interesting point. I’m probably like a lot of folks lulled sometimes into just thinking the Internet is all TCP IP traffic, but increasingly more things are being accomplished over UDP.

Particularly the real time stuff, absolutely. The fourth reason that folks go down the peering path is for a better visibility. When you send your traffic over the wall to your transit provider, you really don’t have any visibility as to how that traffic is being handled by the second ISP or the third ISP that they hand the traffic off to. Contrast that with where you are directly peered with that fourth ISP in the chain, and the traffic goes directly from you to them on to the final destination. So you have the visibility to see how much traffic you’re sending to that fourth ISP in the list and they have visibility into the traffic that’s coming back to you. So when you’re trying to debug a problem, you’re dealing with the principals that are involved in that transaction.

And then finally is cost benefits. This used to be a primary driver for peering, you know everyone wants to be able to offload the traffic for free. But as I said, the price of transit keeps on dropping every single year and the price of peering drops, but maybe not quite as fast. So it becomes a tenuous case in some cases to make the peering a cost justification type of argument. But those are the five reasons that companies generally will go down the peering path.

Well thanks. Appreciate that. And I think I would, having been through it myself in an organization and talked to peers would encourage enterprises to have their network teams look into the detail here and see where the cost benefits really are, because there really are some. One more on a specific point is remote peering. So this is a concept that I know you’ve been involved in, in companies that work with remote peering. This is kind of a special form. Why would enterprises look at remote peering?

This is one of my favorite plays in the Internet Peering Playbook. With remote peering what essentially you can do as an enterprise is contract with a transport provider to get you connected to the most popular peering points that you want to peer your traffic away from. Now you can do this and the cost to you will be the cost of the port and the cost of the transport to get to the exchange point, but you don’t have to pay for a co- location space. You don’t have to pay for a router because the remote peering provider is delivering you directly into that switch.

Now watch what happens here. If you’re smart what you can do is build in and remotely peer at a whole bunch of peering points all over the country or all over the world. And then when you see how much traffic you’re offloading for free at those various different exchanges, you can then make your decision whether you want to build in permanently and establish a co-located presence and buy a router and participate more fully with that exchange point. And those where you’re not delivering a lot of traffic you can disconnect, and it’s just a matter of turning off the transport circuit and the port on the peering fabric that you leased.

One thing that was interesting when we were setting up our peering program related to all that was that it’s sometimes a little difficult to know exactly who is on a net peering exchange. Data centers typically don’t want to give out that information because they want to respect the privacy of their customers. And yet in order to know it’s a good place to peer, you have to have at least some assumption that you have traffic flowing there that you can exchange. Otherwise you’ll show up at a party that there’s no one to talk to essentially.

I know there’s peeringDB as one way to get visibility into that. Are there other ways that peering teams can find out, to develop their peering strategy—where they should locate?

Yeah what I advise my clients to do is to start attending some of these Internet operations conferences in person. And the reason you do that is—a couple reasons actually. The first is you want to be able to have an informal conversation about peering many times with these companies to find out what are their peering requirements? Do they peer openly, would they be receptive to peering with you given the type of traffic that you’re going to be exchanging? You can find out information from other people like maybe you don’t want to talk to a particular cable company because you’re afraid they might say “No” or you want to do that one very carefully, but you can find out from people in the field what it’s like to try and negotiate peering with that particular company.

There’s all kinds of great market intelligence that you mentioned that it might be difficult to find out if people are at a particular data center. It’s a bit of a challenge too, to know how receptive they will be to peering requests. Very often peering requests just don’t go answered. Sometimes you get a no answer, no you don’t meet a peering prerequisites, but face to face you can find out: ‘well you know sometimes they make exceptions and these are the things you can do maybe you’re not in three locations where they want to meet you but you’re in two locations and you plan on going to the third location next quarter.’ Those are the types of things that there’s some wiggle room or not. And that’s the type of ground intelligence you can get at these Internet operations conferences.

Great discussion. Thank you, Bill. That’s it for our episode today. I’m your host Steve Ginsberg and I want to thank our guest Bill Norton. You can find more information from Bill on DrPeering.net, and of course we’re available at GigaOm.com. Thanks for listening.

The post CIO Speaks – Episode 6: A Conversation with Bill Norton of DrPeering Part II appeared first on Gigaom.

In this episode, Simon and Steve discuss what ABA or next-gen SIEMs tell you, how they evolved, and what you need to configure and deploy one.

Intro

Welcome to Right Size Security, a podcast where we discuss all manner of infosec: from enterprise security to practical security every business can use all the way down to end user security. Your hosts for Right Size Security are me, Simon Gibson, longtime CISO — and me, Steve Ginsburg, former head of operations and CIO. For this episode of Right Size Security, we’ll discuss advanced behavioral analytics and threat protection technology — what we’re calling the next generation SIEM (Security Information and Event Management) or user behavioral analytics, UBA. We settled on advanced behavioral analytics because it seemed to more accurately describe what it accomplished and what it did. Today we’re going to be discussing what ABA or next-gen SIEMs tell you, how they evolved, what you need to configure and to do to deploy one, considerations when you’re shopping for one, and how it will fit with the rest of your security program. We’re also going to discuss some of the challenges, and of course, we’ll have our short topics before we get into it. Thanks for tuning in.

Transcript

Simon Gibson: All right, so today on Right Size Security we’re going to discuss SIEMs and advanced behavioral analytics for the next generations, but before we get into it, Steve: voting machines. We had a little conversation about them.

Steve Ginsburg: Sure, yeah. I just thought it would be an interesting topic to maybe address a little bit, both for itself and then for kind of the wider implications for enterprise security and how those things — the approach — might be interlocked. When I think about voting machines, obviously we’re at a point where for years different municipalities have been moving towards digital voting, and there’s always been a lot of concern about, ‘How do you make these things safe?’

It’s something — I’ve only read a little bit about it, but tried to wrap my head around it as well, and what’s interesting… one of the things I saw was that one of the leading voting machine companies, they’ve now flipped and said, ‘Well, you need a digital paper trail.’ So like one of the things in security in that aspect is, ‘Okay, there needs to be some outside-the-system verification,’ and kind of a backup from there. And then when I think about them, I think about, ‘What would make it the most secure?’

So for me, it would seem like open-source software would be great, and something where every push — as we see in the open-source community — every push is done from an open repository. You probably have some thoughts about the security around the tunnels to do it, the cryptology that needs to go to make sure that…sign keys everywhere, obviously, and that type of thing. It’s just that kind of classic thing about: how do you make sure enough that the administrators of the system can’t be corrupted? And then to analyze it, obviously, you have to go up the full stack. So what’s the physical security of each of the facilities? What’s the physical security of the hardware? ‘Cause if there’s wrong chips on the motherboard, then you still might not have security even if the software is secure. Then at the system and OS level, those libraries, then the application level and kind of beyond.

Simon Gibson: Yeah, I have done a little bit of thinking and know some people — again, this goes back to our discussion last episode when we considered privacy — it is a very complicated ecosystem because even when you think about voting machines that produce a digital paper, and you have a human verify that this is indeed what they meant to vote for, and you sort of hope that all works, you still need computers to get people on voter rolls. You still need people to be able to sign up to vote; you need to have a ballot delivered to your house. You still need all that, and that all involves computers.

Now you have that whole part of the universe to consider as well, apart from just — if the voting machines become too hard to attack, somebody will push down somewhere else. And the whole of it is what I think you have to look at holistically, when it comes to protecting this. And that’s where guys like Matt Blaze, for example — he’s a professor who studies nothing but voting machines, and his class is entirely devoted to voting security and the ecosystem around them — and I think if we’re going to have a real serious conversation about this, then this is one of those things where government needs to involve the academics and have a real serious discussion about this. That’s sort of the same way Wassenaar was done. Say what you want about how that’s working right now, but having a real serious conversation about it is what we need to do.

Steve Ginsburg: Yeah, I think one of the things I’m encouraged about is: it is a worldwide problem. So solutions should be coming up all around the globe that could contribute to that and hopefully we’ll see maybe some unified efforts, almost like a U.N. of voting or something like that; an international effort to bring the best practices forward. And perhaps that’s already underway and I’m just not aware of it.

But I think, for our conversation about SIEMs, it’s interesting because… it’s ultimately the same kind of question, which is: ”How do I know everything that’s going on that I should know to make sure that these systems are secure, or at least as secure as they can be given what efforts we are willing to make in this area?”

Simon Gibson: Absolutely. All right, so let’s get into it: advanced behavioral analytics. Let’s get into SIEMs now, or advanced behavioral analytic platforms, user behavioral analytics — call them what you want. My feeling is that these tools are deployed and designed to detect behavior: wanted and unwanted behavior — to be able to aggregate logs, collect telemetry from systems, do analysis of baseline normal or normal-ish, and abnormal or abnormal-ish, and give that information to the security teams, or very often, as is the case, the operations or systems administration team.

When we think about the earliest SIEM, there were the ArcSights and NetWitnesses of the world, but the one that really caught fire and took off was Splunk, and that was built for systems administrators. That was really built to answer the question in a large group of open systems machines: hundreds, thousands, tens of thousands — and an event occurred: what was the first event that caused the cascading failure? Or what was the message that was sent through the system that caused the problem? And Splunk was designed to look at that. And that’s a good way to think about the current generation of advanced behavioral analytics platforms.

Steve Ginsburg: Yeah, I think the correlation was obviously a very central part — and we talked about that before — correlation and search were the key characteristics of Splunk that really moved it forward. So before that, it would seem most administrator’s teams were hopefully moving towards taking their CIS logs into a united place, and that still has its own merit, no matter how you do it, for sure, and then starting to run maybe Regex and custom Perl, that kind of thing.

Simon Gibson: I mean, that was it. It was super awk and grep, and Splunk came along and said, ‘You don’t need to do that. We can make a regular expressions search engine that isn’t based on any data structure. You pick it, and it’s across a time continuum. It’s a regular expression across time. And we will track that for you, whatever you’re looking for.’ And I think that was kind of the heart of the explosion — well, certainly the adoption — of SIEM by security teams.

The question of: ‘What was the first event that caused this problem?’ — and in my case, a lot of the times, it was a directory got removed, or a system got rebooted that shouldn’t have been, or some change occurred on a system that wasn’t meant to [occur] — caused a problem. ‘Whodunnit’? And Splunk was especially good at analyzing those logs across many systems, tracing who logged in to what, what IP they came from, what command was issued, what return code was given, where they went next — Splunk was exceedingly good at being able to quickly take many, many hundreds of gigs of logs and quickly answer those kinds of questions.

Steve Ginsburg: Yeah. I think, in some of my writing, too, we were trying to address that for enterprise teams to think a little, too, about: what is the sophistication of their teams? So for example, some teams, their security operators would have no trouble writing that Regex and that Perl, and could write — I mean, we’ve seen some pretty great examples back in the day, so when you look at something like Splunk, a lot of that is done for you. You’re in the console, and it does two things. So for advanced teams, it frees up their time to do something else. On the one hand, it’s great to hack Regex; on the other hand, that takes time no matter how good you are. So if you’ve got tools that enable you to do that, you can be spending your advanced time on something else.

And then for teams who just don’t have that capability, it’s great to be able to just go into a tool and learn that and make use [of it]. And I think with Splunk, you see a lot of adoption in corporate environments, and also for security teams where maybe the folks aren’t that technical and can make that use as well.

Simon Gibson: Yeah. It is one of those things where — and I think this is especially true in security — it is true to some extent in operations, net ops, CIS ops — very often you don’t know the question you’re going to ask until you need to ask the question; you just have a lot of data. And so, if there’s a particular machine compromised or behaving in a certain way you’d never expected, you may never have a canned query for that question. And the ability to have all the information in aggregate — all the information meaning, if the question you need to ask is about 10,000 systems, can you ask that of 10,000 systems? Splunk was a very good tool for enabling that.

It’s funny, I just read a really good thread on Twitter by Phil Venables, who talked about the need for controls around data inputs. And I can’t tell you how many times I’ve been involved in some sort of incident with a Splunk, and I’ve looked for all the information, and two or three data feeds were down. And we didn’t have any information for a few weeks from a particular machine that was now very critical to whatever it was we were trying to understand. And that the controls you need to put around getting that information are just about as serious — or more serious, even, in some cases — than what you do with the information and how you store it.

Steve Ginsburg: Yeah. You want to practice with all areas of security to make sure when you have an incident — at least as much as you can envision, sometimes there will be an emergent type of incident that you’ve just never seen before, but as much as you can, you want to rehearse and not assume that you’re going to be ready for the forensics that you need.

It also reminds me of what I consider one central issue when I think about Splunk: as a commercial product, it is — or at least was, last time I checked — set up to charge by the amount of data going into the logging system. And it can become inordinately expensive if you have a tremendous amount of logging data.

Simon Gibson: Yeah. It can be real expensive, real quick. But I mean, I think, to be fair, this isn’t just about Splunk. It was a good example. But I do think that does get us into the right part of where we need to begin with this is: what do you need to create a baseline, and what does that really mean? Because again, as successful and as much adoption as Splunk saw and how transformative it was, it didn’t necessarily come with a lot of those analytics; those were community-generated for the most part. Now, Splunk has done a lot of that work; Splunk Cloud is a new offering; a lot of the analytics that were offered as other true SIEMs or behavioral analytics detection platforms, Splunk now has as part of their offering.

But really, talking about a baseline, at the end of the day, the holy grail of information security is observing an event in a system — in an ecosystem of users and computers and data and applications — and understanding, is this normal or not? Is this good or bad? Is this something we should do something about?

My favorite story is: you get a popup to change your password on Monday, you’re busy, you ignore it, you get the pop-up Tuesday, Wednesday, Thursday; finally, Steve, if you don’t change your password on Friday, you’re not going to be able to log-in when… Damn it; you change your password when you leave, and you come back Monday, and you’ve completely forgotten your password; you have no idea what it is. You’ve failed your password five or six times when your phone rings. And it’s your IT Department. And they say, ‘Steve, can I help you?’ And you say, ‘Yeah, I’ve lost my password; I set it on Friday.’ They verify it’s you and promptly reset your password, and you’re off to work. You didn’t put a ticket in, which is always a pain when you don’t have a password, because you can never log in to put a ticket in; that was always sort of one of those catch-22s. But now the help desk has come along, and made your day much easier. But what happened behind the scenes there was there was security [that] just worked perfectly. You failed some control; because that was abnormal, somebody noticed, and they followed up to see if you were you logging in. If they’d called your desk or your cell phone, and you’d said, ‘No, that’s definitely not me,’ they would have alerted the security team and the security team would have gone into action. To me, that is the perfect example of information security working perfectly.

Steve Ginsburg: Yeah. And I see those things becoming even more difficult in the era of ephemeral infrastructure. So — and I’m still wrapping my head around this too, as the year goes on and new products are coming out and just thinking a little bit more about it, even though these kind of concepts are not brand-new by any means.

But when you first start in information security, I think at least I go back far enough that it really was, ‘Okay, I can count how many computers I have; maybe someone’s going to bring something from home later on and it’ll start to be — you have to deal with people BYOD and even BYOD routers and things like that, and hunt those down and kill them.’ And then you might have contractors and visitors; there’s that whole idea of your infrastructure changing. But even still, that’s all almost glacial compared to once the VMs got added in, and then kubernetes clusters and things like that, which is really now the primary mode of everything [that] is meant to be spawned — even entire cloud environments are meant to be spawned spontaneously and dynamically. And then so really, I think, knowing what’s normal, that’s even a greater challenge than it was even a year ago.

Simon Gibson: Yeah. The ability to understand it — fortunately, we’ve seen a lot of companies and infrastructure built around understanding this specifically, and giving the right permissions and credentials sets because of this sort of ephemeral nature, your ability in a microservice to connect to a data source with the right level of permissioning to get the data you’re supposed to — those are all things now that are much more solved than they were in the glacial earlier state of infosec.

Steve Ginsburg: Right. We’re seeing non-perimeter-based security, policy-based AI, and policy intent-based.

Simon Gibson: And definitely identity and role-based [security]. I think a good example of role-based security [is] we talked to some hospitals where there’s teaching going on, and people are students in the daytime, and then at night they’re interns and they’re doctors, and so they need different levels of access depending on what their role is. And that can just be a question of: are they on shift or not? Are they in the class? And that example, you can take to a microservice or a kubernetes or something.

Steve Ginsburg: We had some folks at Pandora who had multiple roles over multiple times, and some of them became literally the poster children in our minds and discussions for: ‘Oh, what about that person who’s been in five different roles in the company?’ Which is great; they had different things to contribute…to your point, their policy roles change.

Simon Gibson: Yeah. And in SIEM, you know, when we think about how to deploy them and what are the most important things…the baselining, and in order to do that, the instrumentation. And I think at the fundamental layer, we can talk about ELF stacks. For example, with a fundamental layer, it’s going to be the storage and the retrieval of the different types of telemetry and information, whether it’s CIS logs, bespoke information from agents, SNMP, net flows, data packets, from the core of your network where you might have the main controllers or server farms, DMZs from corporate to data centers, DMZs from data centers to internets — all of those different places, those ingresses; all those key pivot places logging and sending their data. It has to go somewhere, so that’s a big component of these devices: to collect all that data.

Steve Ginsburg: Yeah. And that’s amazing to be able to see all those streams concurrently and potentially work against… correlating them all in interesting ways and seeing emerging events come from them.

Simon Gibson: Back in the early days of SIEM, the very idea was quite simple. It was: you have a firewall at the perimeter and some sort of a DMZ and a firewall behind that. If an event happens at your perimeter DMZ firewall, maybe it’s okay. It’s when that event happens on the firewall between your interior and your DMZ, now you’ve got an event to correlate. Now you know this thing that was just noise actually got through and triggered an event on another firewall, and you can start to look at it.

But with our early first generation of these, the data was voluminous. It was voluminous amounts of data. Packets were another great way of [organizing] information, but again, just such tremendous amounts of data that storing them became exceedingly difficult.

Steve Ginsburg: Yeah. I mean, there really are so many layers to bring in now. As you said, it’s everything from network flows to threat feeds of external events that you want to compare to what’s happening on your network from there. Maybe you want to talk a little about threat feeds in the ABA world, just ‘cause that continues to evolve in terms of what things are available and how people integrate them into the system.

Simon Gibson: Yeah. So, I think for threat intel, it’s basically data about things that are already ‘known bad.’ Which is a good thing; the problem is, they’re already known bad, and so people who are real[ly] serious about being bad know when they’re burned, generally. That’s why I think you hear about different levels or classifications of threat feeds. Certain threat feeds will identify indicators of compromise, or known bad acts, but they’re not making it very public, ‘cause they don’t want the bad guy to know they’ve been burned. So there’s this sort of different layer, so they’re important to have; they do give you some intelligence about what’s happening on your network…but they’re definitely not the only — they’re not the answer.

I think for understanding those kinds of things, you sort of need a few other things after that telemetry. You sort of need to understand the hierarchy of employees. Because remember, the goal is to create known good from known bad, or good-ish and bad-ish, because you may have an employee who gets a special project, who’s now accessing things that they maybe didn’t before, and it’s not bad because they’re supposed to be doing it. But if you haven’t been told in the security team that somebody from sales is supposed to be looking at HR information, then maybe that would look bad-ish.

But the goal of this is to understand not just the telemetry about the perimeter machine, but also about the hierarchy of who should be doing what, and then honestly, we sort of talk a lot about data loss prevention and understanding data flows. Data classification is a really important factor when we think about baselining as well. What is sensitive data? How can you define it? Certainly, if you’re looking at traffic leaving your perimeter, moving around, and you’re trying to get to that baseline, can you understand the data and who’s sharing what?

One of the things that I think the role of the CIO has changed so drastically too these days is CIOs, up until fairly recently, were very much in charge of data centers and networks and computers; and maybe they were buying machines to run VMs on. But now, I think the CIO maybe still has some of that, but there’s a huge part of the CIO’s responsibility that is understanding the interconnections between cloud services. So we have Salesforce, for example, and what data can be shared with it from our smartsheets deployment, or different employees, or how can we get data from treasury to fulfillment? And payroll to the bank? And it’s really this understanding of data models between these cloud providers.

Steve Ginsburg: Yeah. Absolutely. A lot of the work is spent, as you said, understanding the roles, and roles are something that — for those of my peers who have not already jumped deeply into this — there’s a lot of nuance. You mentioned a salesperson changing categories. When we went down this path, your initial take, of course, is going to be: ‘Engineers should have this class; systems operators, project managers have this class of access to data.’ But then you quickly find: well, project managers are so closely coupled with engineers that if you keep them separated, you very quickly have to start finding the unifying points, or it can’t work. They’re working on the same projects, that kind of thing.

And yet, to your point, you still ultimately need to protect data to a pretty high level, especially depending on what type of organization you are. If you’re a fintech company and you have financial data, you really have to protect it. Every company has HR data; you have to protect that, too.

Simon Gibson: And medical data…

Steve Ginsburg: Right, exactly.

Simon Gibson: Yeah. So I think the point, though, from all of this, is that in order for SIEMs to be effective or ABA threat protection tools to work well, you need to understand what is normal and what your network does: what endpoints are on it, where they communicate, where your data sources are, how human beings interact with the data, how the data is hosted on the applications, whether they’re microservices coming up and down or they’re humans interacting with databases, or license files that you’re hosting in your DMZ — whatever you’re doing. It’s that question of how all this works as an ecosystem, who should access stuff.

And again, like you were just saying, you may have a project manager and an engineer — if you don’t have a data classification program in place, it’s really difficult to explain why I need to give this project manager access to something only engineers have. Then you have to just say, ‘Well, this is what the engineers have, but he’s working with them,’ or ‘she’s working with them.’ If you have a data classification project, you can say, ‘This is definitely not stuff project managers can see, but because we have a data classification program, we can make an exception because now we understand what we’re doing.’

Steve Ginsburg: Yes. And for mid-sized companies and beyond, there’s really a lot of work here because it’s not only understanding what these roles are today, and what these data sources are today, but in most companies, all of this is very dynamic. Every department that you’re working with, they’re changing what software tools they use, they’re changing what outside vendors they use, multiple times a year, often multiple times a month if they’re really, really busy. And even who your experts are in that organization who can tell you what data is being used and how it should be classified, at least relevant to that department, those people are changing as well.

So really catching up with all of that, and making sure, to your point, that you’ve got a good reflection of that in the system, you really need tools that are as dynamic as [they] can be. If it’s not all happening electronically, which it probably isn’t, you need a lot of human effort to really make sure that that stuff is codified.

Simon Gibson: Yeah. And even effort — you need a mandate. You need an edict that says: ‘This shall be important.’ And it’s not. I mean, it is generally not the first thing to come from the board or the CEO or the exec staff.

Steve Ginsburg: Yes. And few folks in a company would choose to have a meeting about data classification if they can be doing whatever their actual job is.

Simon Gibson: Yeah, so it’s a difficult balance to strike. So I think the last part, now that we’ve summed up the need for all the baseline and telemetry collection: the usage of behavioral analytics tools — the two main use cases are forensic and real time.

So real time situational awareness — the ability to monitor end points, user behavior, perimeters, DMZs, proxies, access points — to be able to monitor those in real time and detect something. So a good example is an unexpected large amount of data leaving your network. For some reason, you see many gigs and gigs being transferred and there’s no obvious reason for that. That could be a really good giveaway that your database has been compromised, and somebody’s in the middle of taking a whole bunch of sensitive information. So that’s the sort of real time.

The forensic use case is where somebody comes to you in a month or two, and says, ‘You’ve been breached,’ or ‘We realized this person did something,’ or ‘We have a problem, and we need to go back,’ and now you need to be able to understand exactly what the scope of the problem was — how much damage was done, what was taken.

Steve Ginsburg: Yeah, especially because it’s known that — very common in security circles that you often don’t discover a breach for a long time. It’s often months; weeks if you’re lucky, or hours if you’re really on it. And I think the move towards systems that have automated monitoring around this really is the strongest thing that one can do in this ephemeral world. But that still needs a combination.

I think, in order to know what you want to have automated alerts and even maybe automated remediation on, you probably need visualization. So you need instrumentation first, data collection, visualization, manual interaction, and then move up the food chain towards the best automated action that includes all this awareness of: what are the roles, what is the purpose, what is the criticality?

Simon Gibson: Yeah. So when you’re thinking about buying these things, and we kind of get into what are the mindsets — what do you want to know? There’s sort of the two: cloud and on prem seem to have been much more front and center today. Typically, it was on prem, people were sort of hesitant to use these tools for cloud.

Steve Ginsburg: Yeah, those who have heard me speak at any length know that I certainly started — and I think a good amount of my peers that came from operations, especially — started with a certain amount of, ‘Do I really want to send my key data into a cloud service?’ That becomes another potential concern there. But more and more, when I talk to my peers now, it really is ‘that train’s left the station,’ or whatever analogy you want to use. Increasingly, to your point, as you said, enterprises now are hybrid, multi-cloud, so a good amount of the important data is going to be in the cloud almost regardless of strategy now for enterprises.

There are probably a few holdouts, and certainly, as we talked about, certain types of industry. If I was security with a government arms contractor, nothing would be in the cloud, for example. Some companies still have a reason to have policy that the data’s not up there. But for most companies, they’ve already moved through moving very important enterprise data into the cloud. And so I think the logging goes with that, which is: you’re still going to have to work with that vendor and make sure they can be secure with your log data, and your SIEM, if you’re running it in the cloud. But taking that kind of bet is not considered provocative anymore.

Simon Gibson: I think the concern that always comes up for me is that you get a better set of analytics…you sort of end up using a bit of crowd-sourced intelligence when it’s all shared; you do get this best-of-breed. You get other companies pen-testing cloud infrastructure that you might not — they might be able to afford a pen test that you couldn’t on your budget, and so you benefit from anything those other pen testers found.

The concern I would see with having this kind of data in a cloud is that if you are breached, that means that there’s another company that also knows you were breached. Do you want that CEO calling your CEO to tell them? And what’s the process when they discover you’re breached? And how sure are you that they are going to be confidential and take good duty of care around that information?

Steve Ginsburg: Sure. So it’s a good point. That should be contractually spelled out in a purchase. That’s a good thing to get into the documents there. I think we talked about [it] before: I looked at it as kind of, what is the leverage security model? And you hinted at that in terms of the pen testing, which is: if you have a great operations team, it’s actually very easy to say, ‘Well, a lot of vendors in the world might not be able to do as well on security as my operations team.’ And I think that’s true, and something that I continue to think — my peers I continue to encourage to say, for small companies, you should still really be concerned about that, and even medium-sized companies, because, to your earlier point also, most companies don’t want to prioritize security. That’s not where they see themselves as making money the most, in most cases. There are some places where that wouldn’t be true, but generally they are.

And so when you look at — we’ve given examples of Salesforce, for example — they have enough money leverage to keep secure — that after a certain point, I could no longer feel that it made sense for me to say, ‘Well, I know that my security team can do better than they can.’ They’ve got a multi-billion dollar business around keeping that data secure. They’ve got teams to do it. It doesn’t mean that they’ll never fall down, but it certainly means that they’ve got a good chance at remediating as quickly as anyone would, or that kind of thing. That being said, there are great failures, so one still needs to be careful.

Simon Gibson: Yeah. I mean, I think about the security and the dimensions of confidentiality, integrity, availability, and I really wonder what would happen if Salesforce was down for 72 hours, or there was a breach in confidentiality that was uncovered many months later. I just don’t know what would happen — that would just be a huge thing. To your point, there are a lot of resources there added, but I think one of the things, if we’ve learned anything in infosec, is that if it can happen, it’s probably going to.

Steve Ginsburg: Right. I think it’s safe to manage for everything will fail; it’s just when and what will you do when you do that?. And the question in that case is: Would you feel better if you kept those eggs in your own basket?

Simon Gibson: Absolutely. It’s a question of how you manage the risks you have. I would get asked this question, and it took me a long time to realize — I used to be a little pissed off at it — but I would have boards ask me, ‘Are we secure?’ And I’m the guy in charge of security. So for me to say ‘No’ sort of negates all the work I’m doing. Do you see, that’s this terrible fool’s errand of a question.

And really, what you’re being asked is the wrong question. Because the answer is: ‘No, we’re not secure.’ And if you really expect that we’re going to get you to digital security, you are asking and thinking about this completely differently. What you need to know is: What are our risks? How risky are they? What are we doing about them? And have we looked at all of them? Those are the right questions to ask, not: ‘Are we secure?’

Steve Ginsburg: I think the SIEM provides a valuable role in being able to give continued nuance to your answer there. You can at least say, ‘These are where we have seen events; this is where we have not seen events; this is how we’d summarize them, all that analytics around the data about what kind of events are showing up on our single pane of glass every day.

Simon Gibson: Well, yeah, definitely. And in the deployment of these…part of that is understanding that the SIEMs will not work as well as they’re designed if they have an incomplete picture. So if the SIEM doesn’t understand all the things it needs to get context, you’re going to get not very good efficacy. You’re going to get false positives, false negatives.

The example I gave you about the password, where a human being called…so that’s the problem, right? The human being is the state machine that saw the failed login that an alarm was raised to, and then rather than having an AI or some other sort of smart algorithm decide to make the phone call to your desk, we gave that to a human who then decided that it was really you who was locked out. And that use of humans as state engines is sort of why there’s such a negative unemployment problem in infosec. There isn’t machine learning; there isn’t infrastructure yet for this. Now the good news is, a bunch of the companies that we’re looking at are actually working on these kinds of things.

Steve Ginsburg: Yeah. And even in that example, there’s a parallel for at least part of that, which is: when you log in to Facebook from a new machine, if it has another way to contact you, similar here, an out-of-band way, it’ll say, ‘Hey, was that you?’ and let you either verify it or not. So part of that use case is getting covered these days.

But yeah, I think, to your point here, the central point is the AI and intent-based part of it, and this is true not only for the security component, but even the infrastructure component. I think that’s one of the most exciting areas that’s changing right now as companies are getting more and more machine learning around different use cases, and to your point also, sharing what data they’re seeing across companies and then starting to write some real intelligence to be able to react and be able to remediate if necessary.

Simon Gibson: Yeah. So I think when you’re considering buying, those are the kinds of things to be concerned with, is the amount of data, the level of effort to instrument it, how much data you’re going to send…] Can you get the data if you have remote offices with slow connections and you’ve got a whole bunch of data that you need to ship back from proxies or from different routers or switches or SNM peepholes? Are you going to be able to get it all? Is it going to be confusing if you get an attack and somebody decides to egress out of Jakarta, maybe, and you don’t have telemetry there? How effective is your system after all? So all those are the kinds of things as you’re thinking about these things.

And then I think the next category about this is: Where do these all fit? And you brought up legal earlier, which, this business of]: ‘I’ve been breached; now not only does this other company know about it, but their engineers know about it. Anybody potentially, their sales people know about it. How do we make sure, to your point, that that’s written into the contracts, and is legal okay with it?

Steve Ginsburg: Yeah. And the security teams are very often married in an ongoing effort with the legal department of the company. So there’s the contractual part of it too, and from the legal part, ‘What response does my enterprise want us to have? What kind of things do they want to be notified [of] right away?’

In some cases, of course, there are even state laws about what kind of breaches you need to be notified more publicly about; and then how are you going to go about that, and then the security team, when they’re going to take action, if there’s a concern — and you gave an example about data being exfiltrated; if an employee is involved, then an action is going to have to take part; or even if it’s a third party that’s identified, when do you bring in the FBI, how do you do that? So I think lots of legal touchpoints and interfaces.

Simon Gibson: Yeah, and I have done this on multiple occasions in different companies at different scales, but having a break-glass plan ready to go is nice. I’ve fortunately, knock on wood, not had to really access it or activate it, but having corporate communications involved; having outside counsel involved; having a strike team of people who can come in if you need it involved; understanding who the points of contact are and keeping that list fresh, because people come and go all the time. The minute you put a plan together, someone’s going to change departments and now that point of contact isn’t relevant anymore.

Steve Ginsburg: Yep, and tying it back to the SIEM, and the data visualization, and reporting, is you want those interfaces to be clear. So you want to be able to go to legal with a very clear answer about what you think is happening; not: ‘Hey, we think…’ And there might be times when all you can do is say, ‘Hey, we think this is sort of happening.’ But you want to be able to move quickly to: ‘This is what exactly happened.’

Simon Gibson: And I think that a lot of this — and we are seeing this get more mature — but in the mind of the infosec person, generally speaking, the level of breach and sophistication are things that they get concerned with. I’ve seen a fairly complex attack on a fairly sensitive system from what seems to be a fairly competent adversary; this looks like it’s really bad. And legal might come along and say, ‘Well, is it near our financial systems?’ And you might say, ‘No, it’s actually on a machine that we host documents for stuff on.’ And the legal team might go, ‘Well, tell me when it gets to the financial systems.’ But those are two very different sets of concerns.

Legal has a very different kind of — their job and their incentivization and their resources are totally different from that of infosec, and keeping those aligned is important. And I think we sort of talked about where does the SIEM all fit in with stuff? It can actually be the central routing point — the central place where all information is collected. Data is collected from machines, information about threat intelligence is pushed into it, things are compared, actions are taken, and I think there has been a fair bit of that: where the single pane of glass — which a lot of security operation teams want, a lot of vendors want to sell, for obvious reasons — but just generally to have everybody on the same sheet of music is the goal. Whether or not there’s a mandate or an edict from on high that you shall do this, having that actually does solve a lot of those problems.

Steve Ginsburg: Yeah. And I think it also touches on the idea that a lot of attacks might be multi-headed. And that a SIEM can provide a wider sense of what’s happening across the entire battlefield, not just in one place.

Simon Gibson: Yeah. And not only multi-headed in terms of the technology, but again look at multi-faceted and -dimensional in terms of propaganda and just setting people back from not knowing truth from untruth. And that’s an ‘active measures’ kind of attack, which is maybe very untraditional, at least in this part of the world. But again, not something that somebody was looking for, or even honestly a SIEM might have picked out…yet.

I do think the interesting part about a lot of this — and this is true — I used to always get asked, or get asked a lot, anyway, about how proactive are we being. And the problem is, you can be proactive; you can manage your risk; you can take a risk-based approach and shore things up, but there is a sort of a level of, I want to say, almost like inoculation. When a virus happens and you don’t have any immunities to it, you get sick. And once the thing has happened, you start to build up antibodies.

So like fake news and Twitter bots, and a lot of the stuff that we’ve seen, we have built some immunity to that now. I don’t know that it’s perfect, but it took that to happen. So in the world of infosec, just because it’s dynamic and changing, that there are going to be things where you can be as careful as you want and you’re not going to get everything until you’ve built — until it’s happened to you. And now you’ve been sort of inoculated from it.

Steve Ginsburg: Right. Which is a great reason to get involved, even for, if there are folks who are thinking, ‘I’d like to go down this path, but it seems overwhelming or daunting,’ one approach is to just get going with some scope. Learn how the tools work; get used to practice of looking over some part of your infrastructure; learning how you interact with this, respond to that, instrument it, improve it, evolve it, scale it, grow it. And then as the challenge gets bigger, you’re in a place where you can add from there. The other approach, of course, being: know, go through, understand really what all the inputs are as best you can to start, and pick a solution that’s really scaled for the entire effort.

Simon Gibson: Yeah. I don’t think that’s bad advice either…is to pick a specific area [to] look at. I think for a lot of the work that I’ve done in places I’ve been, it’s the gateways between networks: it’s the proxies, the multi-factor authenticated devices that get you from one-set-of-machines-to-another environment — whether that’s DMZ to the Internet, your corporate environment to your production environment — those are sort of the places that I like to watch because they’re sort of chokeholds; it’s less data.

You can do things like only looking at outbound data from a DMZ, for example. That will tell you lots of stuff. You don’t need to look at inbound and outbound. You’ve just suddenly cut your level of work way, way down. Because if a machine’s infected, it’s going to be freaking out. It’s going to act like an infected machine. I don’t necessarily need to see both directions of communication, because people generally consume more Internet than they produce; Most people surf and look at videos and pictures and read and produce a few emails or a website or some content. Because of that ratio, the amount of traffic you’re looking at is much less if you’re only looking at inbound. So little things like that: picking where your strengths are — it could be a financial system; it could be your treasury system; it could be your fulfillment system, where you build things. Whatever the small universe is, maybe pick that, and start instrumenting that and learning from it.

Steve Ginsburg: Any advice for people on the data sampling? So you said, ‘Hey, go to a place where there’s less data,’ but also in each one of these places, you might be in a situation where you might have to sub-sample; you can’t take all the network traffic, that kind of thing. Any advice on approach there?

Simon Gibson: Only how I’ve been burned by it. Unfortunately, sampling, especially with netflow, it is handy and it will help you, but when there’s an incident, you often want as much as you can get, even if it’s one-directional. So I think it’s helpful; it’s definitely helpful.

I mean, companies like Corelight that has — it’s basically consumer-supported bro. And this is a tool that looks at network packets, summarizes them into texts, and then stores what you want to know about those packets. Again, it’s a tenth of the data, and you get a whole amount of the information. I have with data sampling — again, the few instances I’ve had have been nice, because it was good to have something rather than nothing — but I’d always wished I’d had more.

Steve Ginsburg: Sure.

Simon Gibson: So I think that’s more or less it. I think we’ve kind of covered a fair bit about what you need to do when you’re thinking about a SIEM: why you want one — whether it’s to detect…and I think at the end of the day you want to answer the question, ‘Is this a good event or a bad event? Is this normal, or normal-ish? Is this bad, or bad-ish? But then the challenges around that… they’re not trivial.

Steve Ginsburg: Yeah. I think this kind of situational awareness. Every enterprise should make sure they have a good story here. And I think — we talked about — different enterprises will have different requirements ultimately, and most enterprises will be balancing the, ‘Well, what can I afford to do now versus the other things I need to get done?’ But to your point, when a security event occurs, you’re going to wish you did everything, and so I think moving responsibly towards that is something that pretty much every enterprise should be able to move forward and have a good and evolving story.

Simon Gibson: Yep. Agreed. All right, thanks for tuning in. Thanks, Steve.

Steve Ginsburg: Thanks, Simon.

The post Right Size Security – Episode 4: Advance Behavior Analysis and Threat Detection appeared first on Gigaom.

Host Steve Ginsberg speaks with Bill Norton of Equinix about network peering and network strategy from a CIO perspective.

Guest

Most know Bill Norton as Co-Founder and former Chief Technical Liaison Equinix, or as DrPeering – author of “The Internet Peering Playbook: Connecting to the Core of the Internet.” Some may remember him as the first chairman of NANOG (1995-1998). He is a thought leader, a passionate Internet engineer with deep interest and expertise in the Internet interconnection and colocation sector.

Today, he helps executive teams strategically leverage SDN, BGP peering and cloud interconnection to their businesses, and assist with market research, sales, lead generation and software / DevOps / network engineering. Recent engagements have him building Internet peering ecosystems in the Middle East and consulting on peering and transit for large-scale global cloud gaming.

Transcript

Steve Ginsberg: Welcome to the podcast. I’m your host Steve Ginsberg and today we’re going to talk about network strategy and specifically network peering for the enterprise. My guest today is Bill Norton. I’ve asked Bill to come in and join us as Bill literally wrote the book on internet peering with his book The Internet Peering Playbook: Connecting to the Core of the Internet.” Bill, you’re also the former chairman of the North American Network Operators Group (NANOG), [and] I believe you were the first chairman there. You’ve been an Internet researcher for a long time and you are the co-founder and chief technical liaison in the past for Equinix.

Bill Norton: That’s right.

Is there anything else that you’d like to tell me about your background that I’ve skipped?

Well, I am musician as well. As you know we’ve got a chance to play together on stages in San Francisco, we’ve played on stage at the House of Blues in New Orleans. And I got to tell you there’s an enthusiasm much like we as musicians have towards playing gigs that a lot of the people in the Internet service provider space bring to their art, their skills. Just like we know the best guitars and the newest acoustics, newest equipment that comes out, there are folks that I work with in the ISP industry that know what the latest router is, what the specs are, how you tie these various things together and have that same type of expert knowledge and enthusiasm that we have for music.

How did you get into it in the first place? How did you start connecting with the whole internet peering world?

Yeah. Well I was a programmer actually. I started writing code for network management for the core of the Internet which was called the NSFnet back in the day. That was the core of the North American Internet—was the backbone that we operated connecting all the regional networks together. It was a government sponsored backbone and when the government finally got to the point in 1993/1994 where they decided the government need not run this Internet infrastructure—that it could be commercialized—that’s when I was asked to put together the business plan for what became NANOG, the North American Network Operators Group.

And of course as you know, if you’re going to create a business plan, you’re going to probably appoint yourself to be the chair or the lead of it at least in the beginning to get it started. And that’s what I did. I became the first chair of NANOG and ran that for the first three or four years of its existence. And it was from that that all the contacts in the industry came forward and I was recruited to help Jay Adelson and Al Avery start a company that we came to know, called Equinix.

So for those in the audience who aren’t familiar with NANOG, it’s an organization of network engineers and business people who work in the networking space. It’s an opportunity, a regular conference, and there are educational outreaches from NANOG. There’s a lot of educational efforts and for me as a CIO and as an operations person before that, it was really one of the greatest opportunities to kind of join in to see what was happening on the Internet.

And it continues to be what is happening on the Internet today, the technical issues that are faced really by the largest networks and the largest players on there, but in fact that are also the long tail of anyone who is on the Internet is ultimately affected by those. Were those the original goals when you got involved getting NANOG forward, or are there other things that you would stress that were in the original intentions or that you’ve seen over time?

Yeah, it’s a great question. One of the things I noticed early on in NANOG history was that there were always a bunch of guys in the back of the room that were huddled together working over a laptop computer on something, and I stopped back there and said, “What are you guys working on?” And these are two ISPs that were connecting their networks together. And they did that face to face!

Turns out face to face is one of the fastest ways to get two technical people to work on a single technical issue and get it solved; face to face is just a lot easier. That’s what these guys were doing at NANOG. In fact I would say that probably the greatest value that people get from NANOG are the hallway conversations: the chance to learn from someone else and maybe just deploy a similar thing to what you’re about to deploy and you can learn from those people who are doing the same kind of job. That’s a great place for that.

Absolutely. You know one of the things that the audience will probably have varying familiarity with is just how technically difficult computer networking can be… coming to it from the end user point of view and then running more and more internet dependent organizations. At first I think a lot of people from a distance might think of the Internet more as like a commercial service like their cable service like Comcast, which in fact ultimately is a very complicated network as well, but for an end user it might be more the starting place as I sign up for service, my phone call happens, my internet phone call happens, that’s all I have to worry about.

But of course internet engineering is a very complex and ever changing matter and I think what you’re talking about is at NANOG, there are always a lot of smart people in the room and they’re working out those issues as those complexities arise and continue.

That’s right, and one of the things that’s kind of neat is if you wander around NANOG, I can pretty much guarantee you you’re going to find people that are going to be willing to talk to you about something that they’re working on. And one of the things I did in the early days is I took advantage of that willingness to help other people, and I would document in the form of white papers what people shared with me.

Those white papers would be on operations topics that maybe no one had documented before. For example, how do you connect to a peer if you want to engage in peering not just by Internet transit and get access to the global internet, but actually get direct connects to a particular cloud or a direct connection to a particular destination that you want? That’s the sort of thing that you can pick up going to a NANOG conference, meeting the right people and feeling more comfortable having talked to people who’ve actually gone through the process of building into an exchange point, negotiating peering with another network and bringing that kind of expertise back to the organization is one of the things that NANOG does so well.

Sure. And so peering at its heart is really exchanging traffic. You looked at a bunch of different models for that. Why would any two entities agree to exchange traffic and let traffic flow back and forth, and what are the kind of common concerns that come up in that space?

Sure. Well, you know a lot of times the CIOs are driven to pursue a more controlled way of connecting to the internet besides just buying transit, and some of that is driven by security issues. If you were the CIO at Target, I think you’d be sweating or you’d be fired by now because of the breach that happened. A lot of folks in the CIO office would be doing the sort of things that I’ll share with you later in this podcast. They would do those things mostly to cover their ass (sorry to use that phrase, but it’s the truth).

If you have not done all that is practicable to protect your network infrastructure, you are vulnerable for being the one who gets their head chopped off when that vulnerability is detected. What generally happens is when folks find out there’s been an intrusion—a break in into their network—they disclose internally, they discuss it, they figure out what they’re going to do about it. (Whether they disclose it externally [or not], they’re supposed to, [but] not all companies do that external share of information). But what will very commonly happen, as it did with the Target case, is they’ll bring in a forensics team. And if you’re the CIO, you’d probably be a little bit nervous about what they’re going to uncover as far as what failed under your organization that caused this billion dollar break in to occur.

Sure, and in the case of Target, if I’m remembering correctly, it was internal networks were breached. So a network that was for a lower priority that was basically services network of some type, hackers that were able to come over that network connection and actually establish a path to their commerce network.

That’s right.

How does peering help improve security in general, and then how would peering maybe have helped in that case?

Sure. So the way that peering works is… Well let me start by talking about internet transit first. Most companies, most enterprises buy a service called internet transit, which is really access to the global Internet. You buy a pipe from a provider, maybe AT&T or Sprint or Comcast or someone and they will take care of getting your internet packets out to the rest of the Internet, to the destination and let everyone in the Internet know that you are, your network is on the Internet. And this is the return path back to you, that’s Internet transit.

Now if you are interested in controlling your traffic to a greater degree, you might take a look at Internet peering which is not the same as Internet transit. Internet peering is where you and another network reciprocally agree to exchange traffic with one another typically for free. So if your customers are trying to access something on my network or my customers are trying to access something on your network, at least that traffic can be directly exchanged with one another, typically without any kind of financial exchange.

The way I would look at it is… well just to add to that point is the transit networks will bring your traffic, basically it should bring it everywhere it can go, in terms of of what’s publicly available, and then they themselves are peering. So the top communication networks, your AT&T, Verizon’s of the world and then a number of other ones, they will set up peering arrangements as well so they can more efficiently deliver traffic. And by having your own peering program, you get in on the game.

That’s right. I call it ‘the core of the Internet.’ The core of the Internet is where all of these large backbone networks interconnect the networks directly together. There is a possibility for enterprises that have a particular interest in greater security, better performance, better reliability, for them to say, “You know what? So much of our traffic goes to Salesforce.com, let’s get directly connected to Salesforce.com so our traffic doesn’t traverse the public Internet, which is subject to the security vulnerabilities.”

I can share with you the the top five reasons, if you will. Okay, the top five reasons that companies will connect directly: The first one—and this has been true for the last 10 years—is security. The number one reason people go down the path of connecting directly to their mission critical suppliers is for security. And the argument goes like this: if you buy internet transit today, you’re buying access to the global internet, you’re buying that connectivity. But on average (according to the Europeans) there are four networks on average between you and any destination that you want to reach. That’s four networks, [and] inside of those four networks are dozens or hundreds or thousands of routers each of which could possibly be compromised.

There’s a black market where you can you can basically lease compromised Cisco routers on the internet for denial of service attacks and things like that. Well so a lot of those routers are potentially in the path for your traffic to traverse. Further those routers are interconnected using either cross connects or circuits or potentially long circuits. Those circuits can have traffic mirrored, redirected.

There are cases where traffic was tapped and redirected to China and this was U.S. government traffic. That was why this became such a big ‘to do.’ That’s one of the rules is: U.S. government traffic cannot traverse foreign countries’ infrastructure. So anyway, that type of violation does happen and across the public internet, you have four networks in between you and the destination—each of which has routers and links that could be compromised; and those networks are interconnected themselves to each other using a media that could be tapped as well. So this is what these security guys call a ‘large attack surface’—the public Internet.

And so what you’re talking about is not only how you would deliver traffic in the case of say a content provider like Netflix and Pandora and Spotify’s of the world—how they would deliver traffic, but even for corporate access to systems.

Let’s say I’m a Fortune 500 company and I’m maybe only partly through my digital transformation. So I have a large workforce, but maybe the product is not inherently digital, at least at the scale that Netflix would be for example. Even still, you’re saying that I might want to peer for my Salesforce cloud for example, or to get to my Amazon clouds or to get to my Google clouds?

Yes absolutely. The way that I describe it to my clients is you have to take a look at how important that particular type of traffic is to your company. If you’re salespeople and there’s 100 them in a room and they can’t do anything if Salesforce is not available to them, that might be a good category of traffic where you say “Maybe the cheapest transit provider is not the best path for that particular traffic because we count on it.” And you do the math and you figure out well how much are these salespeople are being paid? What’s the burn on the lights and all this stuff? This is all money that you’re flushing down the toilet while you’re network connectivity is not quite there.

So I’m actually covering some of these topics in a report I’m writing now about connecting clouds, and the premise of the report is that more and more enterprises now are not only connected to a cloud, but starting to be connected to more than one cloud. I think a lot of enterprises are through there or at least they’re well under way, and they’re at first (we’re gonna say) connected to either Amazon or Azure or Google, and then now their businesses are getting more diverse and so now they’re connecting to a second cloud or more. And in fact they’ll either have partnerships or acquisitions that are going to further expand that.

So in that world, one of the things that comes up is cloud exchanges. And then there’s also these connectivity partners that either touch on remote peering or not that can give you… these partners all give you very fast ways to switch. Have you looked at those? I know at Console, you worked for a company that was doing a lot of that work to bring enterprises to corporate applications. How do you look at that market of: ‘should I just let my transit provider take me to all these clouds and all these apps, should I peer directly or should I look at these services that are kind of in the middle of that land?’

Yeah. Oh it’s a really interesting area. As you mentioned, I was a Chief Strategy Officer at Console Connect, which did software defined networking to interconnect the clouds to the enterprise. And I wrote some research papers—I’ll put them up on my website, when I finally get some time to pop it up. But I’ll share with you a couple of interesting data points that CIOs should know about. First off most CIOs that are not in the network space, not in the Internet operation space, a lot of them don’t have the strong or deep network engineering expertise in-house. And as a result, doing things like connecting to the clouds themselves can become a bit of a difficult challenge.

I’ve worked with some clients that only had a single CCIE on staff who is their host master that happened to also know a few Cisco commands—and that would be the level of expertise that you might see inside some of these organizations. And that’s why a company like Console Connect and Megaport and these other companies have spurred up to make the software defined networking piece ease the path for enterprises in to be directly connected to those cloud resources that they care a lot about.

And for those folks who are not familiar with the services that we’re discussing, there’s certainly a wide variety of networking services that are somewhat overlapping, but at the heart of it, they enable you to get a single connection to a service that then can split up through VPNs (virtual networks) or even dedicated circuits at the higher end to other networking points like the big cloud providers, public cloud providers or peering exchanges.

Tell me a little bit about: why did they spring up? So we’ve talked about customers will want to exchange traffic with each other. Why do they want to do it via a peering exchange?

I guess I would say there’s two different things here. We have to be careful about the terminology: the peering exchange I think you’re referring to is a layer 2 type of peering fabric that network operators would connect into so they could peer with those who are on that peering fabric. I think of that as being a peering exchange.

There’s another thing called a cloud exchange which allows you to connect through a platform to potentially many clouds that are also connected on that platform. It’s a little different, peering is a reciprocal exchange of access to each other’s customers—where the cloud exchange really is a way for you as an operator of those cloud services to get access to your resources in the cloud. So it’s you getting access to that which you already have access to over the public Internet.

That’s a good point. So in cloud exchanges, you’re often connecting… you are the network on both sides, and you’re using the cloud exchange to get for example to your own installation in Amazon or your own installation in Google as opposed to on a peering exchange, you’re peering with another customer entity. You are… truly a peer, to reuse the word.

So ISPs use peering exchanges for example and the enterprises would largely only use peering at a peering exchange if they were just delivering a lot of traffic to or from another network organization or another commercial organization who also wanted to reduce their overall transit costs.

There are some folks like that. If you look at the peering ecosystem in my book, I talk a lot about the Internet peering ecosystem as being kind of like an organization—a living breathing organization where if you poke on one side, the reaction happens to the others that are in that Internet ecosystem. For example if you’re in India and the government comes in and mandates anyone within this state has to peer at this location, that effect does in fact change the peering ecosystem within India.

One of the just general things I’ve seen across the world is whenever the government gets involved in any way with Internet peering infrastructure, people back away real fast. And it’s a strange thing, but one of the worst things I think you can do is to have the government get behind and support a new peering exchange in the region, or mandate it—that’s actually the worst is mandate it.

So that’s an interesting difference too. So peering exchanges, to be clear, generally start at a data center and then some of the peering exchanges extend over multiple data centers and even over multiple continents now, right, they’re starting to be connected. But peering is usually… there’s a bit of contract for the circuit.

But the traffic exchanges are very often done with little or no paperwork at all, where a cloud exchange is a commercial service that people buy… in a similar way to how they would buy transit or data center or any other sort of vendor commercial service.

That’s right. If you look at the traditional peering side, you’ll find the ISPs who are seeking to exchange traffic with like minded other ISPs for free, reciprocal, you know: you benefit, I benefit, there’s no paperwork, [and] handshake agreement is typically enough. And then you have the larger ISPs, large guys may not use a public peering fabric. They may just have cross connects between them and their peers in order to have, you know large scalability between themselves. Those very often are signed contract type of agreements. Then you have what are called the hyper giants: the Netflixs, the Amazons, the Apples of the world and these guys are every bit as technically competent as the ISPs that are operating the infrastructure and the core of the Internet. They just have a huge amount of traffic to send out. Almost all of it is outbound, and the core of the Internet is really the only place where a company like Netflix can get direct access to Comcast in the same place. They have direct access to AT&T [subscribers’] eyeballs and the same access to CenturyLink eyeballs and so forth.

So Bill I know you’ve continued to do a lot of consulting in this space. How should enterprises look at connection to clouds?

My view is that the modern enterprise is in a lot of clouds either now or that they will be shortly in the future. We talked a little bit [about how it is] because of partnerships and acquisitions, so even if the main corporation isn’t finding itself split, they may find themselves over time having to deal with an extended network. What do you feel is like the best way to deal with that, and what are the issues that you think are most important?

You know one of the things I find really interesting as I’m consulting with companies that are looking at the cloud is this notion of shadow IT. Turns out that a lot of people within companies are trying to get their job done and using whatever services that are available for free that they can. This includes things like Dropbox and so forth, publicly available services.

So you have a real challenge if you’re the CIO. How do you sign that you are compliant when you don’t really know what all of your people are doing? And it turns out that there’s a lot more access to external network resources than you might think. So very often that’ll be what brings me in is: how can we get that connectivity that our folks depend on even better? We could shut it down, we could say “you can’t use these various different services,” but that is actually probably gonna be worse than just securing the connectivity into the cloud service.

Yeah, I and many of my peers have experienced that. And then it becomes an effort for security teams to have to deal with, and IT teams to deal with in terms of pricing. And it can be a productivity problem also in the enterprise if you have too many overlapping applications where really maybe you should have less.

On the other hand, if you want to enable a workforce, sometimes enabling them as they are is exactly the right thing to do. And I think what you’re saying is then in that case, similar to what you said before, then they become critical services and access to them needs security and availability and resiliency.

Now I’ll tell you one thing I found fascinating when I started working with some of these clients in these different clouds. AWS has a mechanism where you don’t need to use the public internet to get access to what they call your virtual private cloud, your VPC and you can do that with their service—they call it ‘Direct Connect.’ And the idea is that this is a pipe that goes from your datacenter across a private infrastructure—not the public internet—private infrastructure that connects into the AWS side. And they have a mechanism for you to configure this all via soft tools. The software defined networking folks would be able to also use software to configure that path—that private path from your datacenter to the AWS VPC, and do it all with points and clicks.

That’s kind of the way that things are done these days, and it’s really interesting that the different clouds have done different mechanisms for this same type of service. AWS has [called] Direct Connect. Microsoft has what they call ‘Express Route,’ which is a different mechanism for connecting into your resources. Microsoft said that enterprises want to separate out the infrastructure that will be publicly accessible, separate from the infrastructure that is only accessible over a private connection, separate from the Microsoft resources that the company might be using internally; and as a result the Express Route service consists of three peering sessions between endpoints on either side. Not only that—it’s a primary and a backup, you’re required (at least the last time I looked), to have a primary and backup. So that’s six peering sessions set up to reach all your resources on Microsoft as your platform. Really interesting differences between these clouds Google’s different also. They’re just different terminology, different mechanisms and they all claim that these are driven by the enterprise.

And do you think that it would be difficult to achieve that same functionality on Amazon if you needed that?

No, you could actually do that entirely with Direct Connect and…

It’s just not in by the default design, is the difference or the distinction you’re making?

That’s right. The idea is that you have different parts of the organization that may be accessing different things within the cloud, and you need to be able to tease apart these things because, believe it or not, research does not want finance to look inside their their resources and vice versa. These companies are very protective about their information flow.

The other thing that I found really fascinating was having do with price. When you use Direct Connect you’re not paying Amazon for their Internet access for using their Internet access for sending stuff out. And if you do the math, Amazon charges about 8.5 cents per gigabyte for the first large chunk of traffic and they eventually get down to about two cents per gigabyte. But when you do the math and you figure out well what if I use of this full let’s say 50 megabit for a second and how much would that cost? Well 50 megabit per second, if you do the math, it ends up being about $25 per megabit per second for all of your egress traffic out of AWS. That’s a lot compared to public transit today.

The rates are generally below a dollar these days, are they not?

25 cents to a buck and a quarter is kind of the range I’m seeing in the field. So Amazon, Google and Microsoft are all making enormous amounts of money on their Internet service that they charge the cloud operator and by going direct, they don’t make it zero but they drop it down to a point where it’s substantially lower. And I have a business case for direct connect I can share with your audience if you like. Does that crossover point… at what point does it make sense financially to start saving money to go with a direct connect solution? And it’s not all that much traffic—it’s like if you have 20 or 30 megabits per second of egress traffic, I can guarantee and prove to you financially that you’ll be saving money by using a direct connect service. Yeah likewise with Azure and Google cloud, they have different price points, but the same logic exists.

That’s really powerful.

Part two of this interview continues on the next episode of CIO Speaks with Steve Ginsberg.

The post CIO Speaks – Episode 5: A Conversation with Bill Norton of DrPeering appeared first on Gigaom.

In this two part episode, Steve talks to Colin Corbett about his experiences working for companies such as Paypal, Google, Netflix and Youtube.

Guest

Colin is the proprietor of 7 Hills Consulting which helps companies with global network and datacenter architecture and design. With a networking and datacenter background spanning over 25 years, Colin has built and led the entire network and datacenter infrastructure for early stage startups including PayPal, YouTube and Dropbox. Additionally he has had various networking and datacenter roles for growth companies such as eBay, Google, Twitch, and Netflix among others.

Colin has rolled out infrastructure within 5 continents for various sized deployments. He has also designed and built numerous large networks, both in terms of egress traffic, and locations deployed. Additional specialties include: making repeatable cookie-cutter scalable deployments, site selection, lease/contract negotiation, backbone/transit/peering discussions, hardware selection, vendor negotiation etc..

Note

This is the second part of a two part interview, find part one here.

Transcript

So we’ve talked about getting in a great data center contract, making sure the data center is gonna have the spec that you really need for for your location in the best ways. You talked about having integrators kind of roll in a rack, but in terms of what that rack should look like (I know you’re as detail oriented as just about anyone that I’ve talked to about that), what makes for a really great data center rack and rack installation? How do you optimize? What’s your approach there?

So the first thing I try to do is I try to limit it to a certain number of skews. So if you have like a web, a video, a database in a network that’s basically four types of skews so, if your integrator only has to build a certain types of things, that’s great because then they’re not having to stock a bunch of different equipment. You’ve maybe got it down to three or four different types of racks, and then sure, overtime you’re going to end up having to deal with the vision control, but try to do revisions.,

Not every single build should be a unique snowflake. You should try and start to have lots of constant revisions where everything looks pretty much in lockstep for a, three, six or nine month period let’s say. So things like the network rack would have one vendor’s worth of gear and it’s laid out perfectly as well as the webs, the videos and the database and sure as Intel or AMD release new CPUs, you’re going to have to iterate that. As hard drives become obsolete or end of life by the vendor; and you have to swap that. You’re going to have to iterate, but try to do it at a… Certainly never try and say “well this rack’s got half of the old gear and half of the new gear.” Work with your integrator to make sure that you’ve got a very clear delineation of: this is version A and this is version B etc.

Yeah. I’ve seen teams look at that not only to make sure that the rack can be well maintained, but even to get to the fact where even the cage has incredibly similar detail between multiple cages as you start to have lots of data centers and lots of data center cages just to make sure that site operations people can move in and efficiently do the right thing and not spend too much time figuring out how the whole cage installation even works.

Right, so at YouTube for instance, we ended up asking for a very specific type of cage from our vendor—it was basically a nine rack cage. And it started with the first rack as the network rack, the rest of the racks for this [were] all video, full of video POPs, and the rest of the racks were all video. And those looked [good] and that was all very clean. But it got to the point that we even had bins for all of our spare tools—things like fiber, copper, spare hard drives. But across all of our POPs, you knew that the top left most bin was one meter LC-duplex fiber, and the bottom right bin you know, eight rows over five bins down was spare hard drives.

And it became very clear and it was great because you could teach your technicians what it would look like, but also when you would have to go and call for remote hands at night you knew that every data center was exactly the same, and it was really helpful especially when looking for spare parts or optics or things like that.

And then in terms of the rack itself, what’s your approach for optimizing for power —both power that’s available to the rack and providing that power to the rack?

So for that I usually will ask for a circuit that’s going to be bigger than what I need so that I usually either have room to burst, or I’ll ask for something where I’ll never really run in jeopardy of exceeding breakers. You certainly can have cases where, that being said, the nameplate rating on all the hardware is far in excess of what you actually will see in standard use. A lot of that you end up refining over as you actually see the equipment and your workload.

Sometimes though, that does change as you keep having bad pushes, so you need to accommodate headroom for that too because you can’t have bad pushes that increase your workload extensively. But yeah, usually you work with the data center provider and especially in the wholesale world, you can ask to provision whichever circuit you want. And so usually you try to settle on one that has certainly got enough headroom, and it allows me to scale, and as long as there’s sufficient cooling for that, you just try to go as dense as you can within your limitation, because the density—while you end up with [fewer] overall switches, you end up with more rack space free or more rack positions free, etcetera.

You mentioned I believe that you started working with your teams with a kind of standardizing on a DMARC rack? Do you want to tell us about that?

Yes. So that was one of the interesting things we ran into with… as you start doing a lot of wholesale deployments: the idea of eventually we wanted to kind of start being in a wholesale deployment, but also having the flexibility to swap other infrastructure out. So if you had let’s say, network and server racks that you knew were going to keep evolving rapidly, could you go and be in a standard data center and not shut the data center down completely, still keep it operating in some fashion, but still be able to go and upgrade it in bulk?

One of the things we came up with—and this was mainly for retail locations, was land all your cross connects at a unified DMARC, and you and all your fiber from your network racks to the DMARC, and all your cross connects out to network providers, and your peers land there, and then with enough coordination and planning, you can swap out your server racks, you can swap out your network racks and the DMARC stays the same.

Also if you plan really far ahead you have, and in the data center that you’ve never really made your first delivery to, you send the DMARC rack first because it really only has maybe fiber panels and maybe a couple of power strips and not much else in it. If something bad happens to that rack let’s say, it falls off the truck, the overall dollar value of that is actually pretty low. And so you can kind of work out a lot of problems about doing the delivery very far in advance with a rack that is very quickly built and has an overall low dollar value compared to some of the hundred thousand dollar to a million dollar racks you can have out there.

That’s cool. How do you think about the vendor solutions that are out there for customized racks, say the APC racks and Dell racks and rackables, and there’s kind of a lot of consumer options today?

I’ve had some overall good and bad between them. So the main thing is just making sure that they all meet your need. I’ve certainly run into questions where the racks look great and I say “OK can I fully load this and load it into the data center?” They go ‘well this is its static rating,’ which is very different than its rolling rating, so if you actually wanted to put twenty five hundred or three thousand pounds worth of gear, you can’t roll it in with three thousand pounds worth of gear. So that’s one of the interesting things to run into, or you say “hey I have a 60 AMP three phase power plug” and they say “well that’s great but there is no hole in the top of the rack to actually fit that out of all. You need to order a special top or you need to run it with no top to kind of make sure that all fits.”

So those are all interesting issues. For me when I’ve worked with integrators, we’ve also seen things like racks where the customer has come to them saying “I have three different types of airflow either side to side or side to back, or front to back” and all these kind of stranger things. And a good integrator will certainly help you make sure that all of that’s correct. They’ll do the air baffling to make sure that all works. And also things like balancing all the phases, especially when you have things like three phase power.

As for working with some of the larger vendors, I’ve had a lot of strange issues when I want to specify certain types of gear, so things like if I have a favorite RAID or NIC card that I want to use in my infrastructure that I’ve certified from, that we’ve approved from using our white box solutions. We’ve definitely had cases where the bigger vendors have not been able to source it or they can’t get it in a reasonable amount of time.

Also we’ve run into cases where they’ll sell a standard ‘off the shelf’ hard drive but with their custom firmware, and if they no longer support that drive or that drive’s not sold through that vendor, I can’t replace that drive or buy new drives straight from the manufacturer anymore because they don’t have the custom firmware. So I’ve certainly run into those issues and that’s not been fun. Trying to resolve that… there has become a few interesting issues there.

How about as you move to custom storage arrays, blade servers or even hyper converged infrastructure?

So there’s some interesting things there too. A lot of the problems actually end up being physical. Some of the newer ways I’ve seen, they’re basically 36 inches deep, but most data centers especially retail data centers, by standard will only ever give you a 36 inch deep rack—or they have aisles designed around 36 inch deep racks. What that means is that you end up needing like a 42 inch or 48 inch deep rack, which generally does not fit in the data center or doesn’t give you the ability to service them. So if you ever had to go and work on the rack in the back, let’s say to pull out the switch that you’ve rear mounted, you might not be able to either open the door or you might not even be able to pull the switch out to go and service it and swap it.

It’s really interesting that you can run into those issues. Also you’ll see things like, shallower Caldow racks where they’ll try to go and make the Cladow so small that you can’t actually bring out that 36 inch deep drive away because there’s just not enough room in the aisle. So a lot of physical planning goes into supporting some of the blade servers, a lot more than you would think. But as you talk about blade servers you have things like flexibility questions, things like can you get 10/25/40/50 and100GB out of the back of the blade servers? In some cases you do it by just exposing the actual networks onto the chassis or from the back of the chassis, or sometimes if it has dedicated switches in the middle, you end up with limitations in terms of capacity.

For me I’ve usually erred on the side of going with dense servers that don’t have an integrated switch. Something more like the super micro-2 twins or the equivalent Dells or some of the open compute stuff that also looks interesting too.

Right so you bring up that there is a difference between trying to add compute density and then integrating network density into it—as opposed to just using the kind of standard network switching you’ve chosen for your data center?

Yes very much so. Maybe that’s just more of my networking background but I’m very happy with… I would like to dictate the network’s capacity and build it and scale it according to what I want, as opposed to having an integrated switch with limited capacity or limited uplink and you’re not quite sure about what the redundancy looks like. Whereas this way you can build it your own way and dictate it that way.

What’s your take on with storage arrays—particularly there’s alternate networking that becomes available. Is that something that you favor or not? Kind of the iSCSIs of the world and Fibre Channel over Ethernet (FCoE).

I’ve recently looked into a few of things like Fiber Channel, Fiber Channel over Ethernet, Lucky, Lucky-2 things like that. I will say my heart is mainly in the Ethernet world, so I usually predominately stay on Ethernet, and a lot of the technologies that have come out, have been pretty affordable. And now with something like 25GBs really about 20% more expensive than 10GB, and 100GB really seems to be only about a 20% uplift over a 25GB. Eventually you might have some density problems or capacity problems at the switch layer, but those seem to be really good choices, and now you are starting to see things like 400GB out there. And there’s some really interesting developments happening there.

So you’re saying Ethernet is keeping up?

Ethernet seems to be keeping up and there’s things like priority buffering, data center bridging that seem to try to straddle the world of making you not go with dedicated Infinity Band or Fibre Channel switches on the back end.

So when you’re in the data center, connecting it all relies (as you mentioned earlier) on cross connects. I know all of us in the data center world have had a lot of experience with cross connects, portals and making sure we can make those connections. I know you’ve thought a lot about it. What’s your kind of current take on the world there?

One of the hardest things is when you want to order a circuit from a provider, is trying to figure out: is the provider a built in, and more importantly, if they are built and you want multiple circuits from them, are they built in redundantly? To actually see, to actually work with the provider and say, “I have a circuit that’s on path A that lands on equipment A and I want a circuit on path B that’s on equipment B and they both take separate paths out of the building.” That’s something you need to really work with your vendor to find out, and in a lot of cases, your data center vendor, if you ask them for information as to ‘Is that possible?’ they’ll usually just push you back on the provider.

In terms of getting cross connects there’s a lot of hurdles. My strong recommendation is to pre wire panels yourself so that you know where everything is on your side and then you start assigning specific ports on… as you want to cross connects, so that you know exactly what that matches to, exactly what port that goes to, so that everything should theoretically come up sooner. If you can do the pre wiring, the other thing is when you assign a port before you open the cross connect, is to actually go and start sending light down your port, so that then as they’re working on the circuit, you can also put in there standard notes like: ‘I’m sending light. Make sure you see my light all the way down at the far end of this circuit to make sure that it comes up.’

I’ve certainly run into cases though, well I’ll tell you a quick story, I tried working with a large global data center provider to say “How could I do a global deployment with them across multiple continents?” And I said “Is there a globally supported patch panel I can use so that I can do this just standard cookie cutter layout?” And in one case they said, “No, we don’t have a global standard. Everything’s individually reviewed and everything can be individually rejected.” That was kind of a nightmare, so I did eventually get that fixed but it took a good amount of work to get to that point.

Yeah it’s interesting, as their world is evolving to that, maybe at times folks like yourself have had to shape their world—not just the other way around.

Definitely.

Colin, you’ve given us great insight into the data center and the details about working with it and optimizing it. There are some who would say data centers don’t matter to enterprises in the era of the cloud. The data center is someone else’s problem, whether that’s Amazon or Azure or Google or other. Why do you think data centers still matter for corporations and enterprises?

So I think it still has a lot of fits, but depending upon need. So it’s great at the edge as you can work directly at the networks you need to connect to. In the case of content, working with the eyeball networks and interact with them to make sure you have the performance you need to then network. Also with a large amount of egress, as you send your data out it will be cheaper if you send your traffic out of your own infrastructure than if you are paying on a… In the cloud world, you pay on what’s called a gigabyte transfer method, whereas if you are doing it on your own infrastructure, you’re doing it either through a settlement free interconnect through peering or you’re doing paid peering or you’re doing transit, usually on like a 95th percentile basis. That will be cheaper if you have enough traffic to make that account.

This is a key point that enterprises should be aware of if they are not, which is that the public clouds make a lot of their money on this line item that you’re mentioning—on the data coming out of their data center. So if you’re pushing a lot of traffic, it can really become prohibitive.

Very much so. Also some of the other interesting things are if you do something like a lot of expensive cloud processing by running – if you took some of that standardized workload and actually moved it to your infrastructure, you can go and save the money by running it there. That is also kind of a mixed bag. I mean the hardware you’re looking at there is on like a three year depreciation cycle. But some types of the hardware: things like GPUs are changing at a faster cycle. But at the same time you’ve also got the cloud which is charging you a 12 to 18 month depreciation cycle for the hardware anyway.

So depending upon how you do it, it may make sense to go move some of it back to your own infrastructure. But then again, all that being said, you still need to avoid a single point of failure. So, even if you had your own data center and you run your own infrastructure, you still need to be paranoid enough that either the fire marshal comes in, could go and shake your building down and hit the EPO at any point, ora fire could break out that results in the data center having to lose power or things like that. So you should always try and keep everything to a point where you have redundancy and that is the main thing to keep an eye on.

Absolutely. And you mentioned, edge. Mostly we’ve been talking about it in the context I would say is the network edge, as it as you talked about how providers that are moving a lot of traffic and move closer to their customers. Obviously there’s a lot of momentum around just the term “edge” in general and in edge data centers. We’re usually talking about moving sometimes closer to customers but also IoT and industrial IoT. How do you view the concept of edge data centers?

So there’s a lot of really a really cool ideas there. After traditionally most people started in the original six or seven Tier 1 markets and then after they’ve got that solved, they ended up moving to the Tier 2 markets to get better end use performance. The edge data center seems to be solving a lot of the Tier 3 locations as well as things like cell towers and some of the backhaul. You know in the in the quest to get better performance uses, it’s a good idea.

With a lot of things like the offnet caches and all of the updating algorithms, it’ll be very helpful for a lot of the big consumers who can support hundreds of thousands of like edge locations, solve their problems whether it’s to serve more ads, get more use of telemetry, maintain a high quality bitrate for video, that’ll be huge. It’s not clear though, that a lot of the small consumers will be there at the moment.

And you know the corollary to that is—and this is a problem I ran into—is that if this is super successful, how do you guarantee that that space in those small edge data centers doesn’t get gobbled up by the same handful of customers, which then stop the next great startup from getting space there at a reasonable price? And since some of these spaces are as small as a standard semi container, they don’t have a lot of rack space to spare, and they also have cooling power and fiber distribution that… space rapidly becomes a concern.

So Colin, I just kind of met a comment here. I just took a quick look at your answer to ‘composable’ and it’s pretty cool. Do you mind if I just ask you [about] that as a final question?

Sure.

Right. So some would look at the new concept of composable infrastructure as really taking the architecture of the data center apart and reassembling it in a different way. What do you think about composable infrastructure?

So I haven’t really tried this extensively. At the moment Kubernetes seems to be the thing and it looks pretty interesting, but yeah haven’t really had too much time to test it. That being said though, there are many needs where people need both bare metal and NVMs, instead of just one or the other. So being able to utilize the cloud, old wheel in racks, and have them available as additional resources, that’s a great thing. That being said, I do think the ability of capacity plan, which was a core part of doing data center infrastructure originally, has kind of been lost and so, yeah the idea of the bursts is very much needed these days as a result.

So with capacity planning, you’re saying, in the past you’d run an exercise looking at those specific systems and the needs for those systems, and now that’s all kind of too dynamic to plan for, is that what you mean?

So traditionally (getting data center space) if you needed a rack worth of gear from an integrator, usually the lead time there is about 6-8 weeks, with no warning, no planning let’s say. And as a result, usually that means you’re running your infrastructure with either some headroom or with the ability to basically say, “If everything goes south, do we have enough to… if I needed to order a rack today, so I have 8 weeks worth of headroom in my infrastructure?”

Now with things here, I can just go and open up extra, just order additional capacity from the cloud, or just add servers to my cart and all of a sudden I have servers, rather the idea around capacity planning hasn’t really, has kind of been lost. At scale though you really need to know what your hardware needs will be, not just two months away if you order servers today, but also months down the line, and maybe even years down the line, and as you get to a really big scale, you run into cases where the cloud providers may not have capacity of they’ll invite you to their capacity planning meetings as well. They’ll want to know what your forecast is, if you’re big enough for the cloud provider.

Interesting. Well great thanks very much Colin. I appreciate your answers today and ongoing from the past and looking forward to continuing the conversation in the future.

Thank you very much. It was a great time chatting.

The post CIO Speaks – Episode 4: A Conversation with Colin Corbett of 7 Hills Consulting, Part II appeared first on Gigaom.