CIO Speaks – Episode 5: A Conversation with Bill Norton of DrPeering

Guest

Most know Bill Norton as Co-Founder and former Chief Technical Liaison Equinix, or as DrPeering - author of "The Internet Peering Playbook: Connecting to the Core of the Internet." Some may remember him as the first chairman of NANOG (1995-1998). He is a thought leader, a passionate Internet engineer with deep interest and expertise in the Internet interconnection and colocation sector.

Today, he helps executive teams strategically leverage SDN, BGP peering and cloud interconnection to their businesses, and assist with market research, sales, lead generation and software / DevOps / network engineering. Recent engagements have him building Internet peering ecosystems in the Middle East and consulting on peering and transit for large-scale global cloud gaming.

Transcript

Steve Ginsberg: Welcome to the podcast. I'm your host Steve Ginsberg and today we're going to talk about network strategy and specifically network peering for the enterprise. My guest today is Bill Norton. I've asked Bill to come in and join us as Bill literally wrote the book on internet peering with his book The Internet Peering Playbook: Connecting to the Core of the Internet.” Bill, you're also the former chairman of the North American Network Operators Group (NANOG), [and] I believe you were the first chairman there. You've been an Internet researcher for a long time and you are the co-founder and chief technical liaison in the past for Equinix.

Bill Norton: That's right.

Is there anything else that you'd like to tell me about your background that I've skipped?

Well, I am musician as well. As you know we've got a chance to play together on stages in San Francisco, we've played on stage at the House of Blues in New Orleans. And I got to tell you there's an enthusiasm much like we as musicians have towards playing gigs that a lot of the people in the Internet service provider space bring to their art, their skills. Just like we know the best guitars and the newest acoustics, newest equipment that comes out, there are folks that I work with in the ISP industry that know what the latest router is, what the specs are, how you tie these various things together and have that same type of expert knowledge and enthusiasm that we have for music.

How did you get into it in the first place? How did you start connecting with the whole internet peering world?

Yeah. Well I was a programmer actually. I started writing code for network management for the core of the Internet which was called the NSFnet back in the day. That was the core of the North American Internet—was the backbone that we operated connecting all the regional networks together. It was a government sponsored backbone and when the government finally got to the point in 1993/1994 where they decided the government need not run this Internet infrastructure—that it could be commercialized—that's when I was asked to put together the business plan for what became NANOG, the North American Network Operators Group.

And of course as you know, if you're going to create a business plan, you're going to probably appoint yourself to be the chair or the lead of it at least in the beginning to get it started. And that's what I did. I became the first chair of NANOG and ran that for the first three or four years of its existence. And it was from that that all the contacts in the industry came forward and I was recruited to help Jay Adelson and Al Avery start a company that we came to know, called Equinix.

So for those in the audience who aren't familiar with NANOG, it's an organization of network engineers and business people who work in the networking space. It's an opportunity, a regular conference, and there are educational outreaches from NANOG. There's a lot of educational efforts and for me as a CIO and as an operations person before that, it was really one of the greatest opportunities to kind of join in to see what was happening on the Internet.

And it continues to be what is happening on the Internet today, the technical issues that are faced really by the largest networks and the largest players on there, but in fact that are also the long tail of anyone who is on the Internet is ultimately affected by those. Were those the original goals when you got involved getting NANOG forward, or are there other things that you would stress that were in the original intentions or that you've seen over time?

Yeah, it's a great question. One of the things I noticed early on in NANOG history was that there were always a bunch of guys in the back of the room that were huddled together working over a laptop computer on something, and I stopped back there and said, "What are you guys working on?" And these are two ISPs that were connecting their networks together. And they did that face to face!

Turns out face to face is one of the fastest ways to get two technical people to work on a single technical issue and get it solved; face to face is just a lot easier. That's what these guys were doing at NANOG. In fact I would say that probably the greatest value that people get from NANOG are the hallway conversations: the chance to learn from someone else and maybe just deploy a similar thing to what you're about to deploy and you can learn from those people who are doing the same kind of job. That's a great place for that.

Absolutely. You know one of the things that the audience will probably have varying familiarity with is just how technically difficult computer networking can be... coming to it from the end user point of view and then running more and more internet dependent organizations. At first I think a lot of people from a distance might think of the Internet more as like a commercial service like their cable service like Comcast, which in fact ultimately is a very complicated network as well, but for an end user it might be more the starting place as I sign up for service, my phone call happens, my internet phone call happens, that's all I have to worry about.

But of course internet engineering is a very complex and ever changing matter and I think what you're talking about is at NANOG, there are always a lot of smart people in the room and they're working out those issues as those complexities arise and continue.

That's right, and one of the things that's kind of neat is if you wander around NANOG, I can pretty much guarantee you you're going to find people that are going to be willing to talk to you about something that they're working on. And one of the things I did in the early days is I took advantage of that willingness to help other people, and I would document in the form of white papers what people shared with me.

Those white papers would be on operations topics that maybe no one had documented before. For example, how do you connect to a peer if you want to engage in peering not just by Internet transit and get access to the global internet, but actually get direct connects to a particular cloud or a direct connection to a particular destination that you want? That's the sort of thing that you can pick up going to a NANOG conference, meeting the right people and feeling more comfortable having talked to people who've actually gone through the process of building into an exchange point, negotiating peering with another network and bringing that kind of expertise back to the organization is one of the things that NANOG does so well.

Sure. And so peering at its heart is really exchanging traffic. You looked at a bunch of different models for that. Why would any two entities agree to exchange traffic and let traffic flow back and forth, and what are the kind of common concerns that come up in that space?

Sure. Well, you know a lot of times the CIOs are driven to pursue a more controlled way of connecting to the internet besides just buying transit, and some of that is driven by security issues. If you were the CIO at Target, I think you'd be sweating or you'd be fired by now because of the breach that happened. A lot of folks in the CIO office would be doing the sort of things that I'll share with you later in this podcast. They would do those things mostly to cover their ass (sorry to use that phrase, but it's the truth).

If you have not done all that is practicable to protect your network infrastructure, you are vulnerable for being the one who gets their head chopped off when that vulnerability is detected. What generally happens is when folks find out there's been an intrusion—a break in into their network—they disclose internally, they discuss it, they figure out what they're going to do about it. (Whether they disclose it externally [or not], they're supposed to, [but] not all companies do that external share of information). But what will very commonly happen, as it did with the Target case, is they'll bring in a forensics team. And if you're the CIO, you'd probably be a little bit nervous about what they're going to uncover as far as what failed under your organization that caused this billion dollar break in to occur.

Sure, and in the case of Target, if I'm remembering correctly, it was internal networks were breached. So a network that was for a lower priority that was basically services network of some type, hackers that were able to come over that network connection and actually establish a path to their commerce network.

That's right.

How does peering help improve security in general, and then how would peering maybe have helped in that case?

Sure. So the way that peering works is... Well let me start by talking about internet transit first. Most companies, most enterprises buy a service called internet transit, which is really access to the global Internet. You buy a pipe from a provider, maybe AT&T or Sprint or Comcast or someone and they will take care of getting your internet packets out to the rest of the Internet, to the destination and let everyone in the Internet know that you are, your network is on the Internet. And this is the return path back to you, that's Internet transit.

Now if you are interested in controlling your traffic to a greater degree, you might take a look at Internet peering which is not the same as Internet transit. Internet peering is where you and another network reciprocally agree to exchange traffic with one another typically for free. So if your customers are trying to access something on my network or my customers are trying to access something on your network, at least that traffic can be directly exchanged with one another, typically without any kind of financial exchange.

The way I would look at it is... well just to add to that point is the transit networks will bring your traffic, basically it should bring it everywhere it can go, in terms of of what's publicly available, and then they themselves are peering. So the top communication networks, your AT&T, Verizon's of the world and then a number of other ones, they will set up peering arrangements as well so they can more efficiently deliver traffic. And by having your own peering program, you get in on the game.

That's right. I call it ‘the core of the Internet.’ The core of the Internet is where all of these large backbone networks interconnect the networks directly together. There is a possibility for enterprises that have a particular interest in greater security, better performance, better reliability, for them to say, “You know what? So much of our traffic goes to Salesforce.com, let's get directly connected to Salesforce.com so our traffic doesn't traverse the public Internet, which is subject to the security vulnerabilities.”

I can share with you the the top five reasons, if you will. Okay, the top five reasons that companies will connect directly: The first one—and this has been true for the last 10 years—is security. The number one reason people go down the path of connecting directly to their mission critical suppliers is for security. And the argument goes like this: if you buy internet transit today, you're buying access to the global internet, you’re buying that connectivity. But on average (according to the Europeans) there are four networks on average between you and any destination that you want to reach. That's four networks, [and] inside of those four networks are dozens or hundreds or thousands of routers each of which could possibly be compromised.

There's a black market where you can you can basically lease compromised Cisco routers on the internet for denial of service attacks and things like that. Well so a lot of those routers are potentially in the path for your traffic to traverse. Further those routers are interconnected using either cross connects or circuits or potentially long circuits. Those circuits can have traffic mirrored, redirected.

There are cases where traffic was tapped and redirected to China and this was U.S. government traffic. That was why this became such a big ‘to do.’ That's one of the rules is: U.S. government traffic cannot traverse foreign countries’ infrastructure. So anyway, that type of violation does happen and across the public internet, you have four networks in between you and the destination—each of which has routers and links that could be compromised; and those networks are interconnected themselves to each other using a media that could be tapped as well. So this is what these security guys call a ‘large attack surface’—the public Internet.

And so what you're talking about is not only how you would deliver traffic in the case of say a content provider like Netflix and Pandora and Spotify's of the world—how they would deliver traffic, but even for corporate access to systems.

Let's say I'm a Fortune 500 company and I'm maybe only partly through my digital transformation. So I have a large workforce, but maybe the product is not inherently digital, at least at the scale that Netflix would be for example. Even still, you're saying that I might want to peer for my Salesforce cloud for example, or to get to my Amazon clouds or to get to my Google clouds?

Yes absolutely. The way that I describe it to my clients is you have to take a look at how important that particular type of traffic is to your company. If you're salespeople and there's 100 them in a room and they can't do anything if Salesforce is not available to them, that might be a good category of traffic where you say “Maybe the cheapest transit provider is not the best path for that particular traffic because we count on it.” And you do the math and you figure out well how much are these salespeople are being paid? What's the burn on the lights and all this stuff? This is all money that you're flushing down the toilet while you're network connectivity is not quite there.

So I'm actually covering some of these topics in a report I'm writing now about connecting clouds, and the premise of the report is that more and more enterprises now are not only connected to a cloud, but starting to be connected to more than one cloud. I think a lot of enterprises are through there or at least they're well under way, and they're at first (we're gonna say) connected to either Amazon or Azure or Google, and then now their businesses are getting more diverse and so now they're connecting to a second cloud or more. And in fact they'll either have partnerships or acquisitions that are going to further expand that.

So in that world, one of the things that comes up is cloud exchanges. And then there's also these connectivity partners that either touch on remote peering or not that can give you... these partners all give you very fast ways to switch. Have you looked at those? I know at Console, you worked for a company that was doing a lot of that work to bring enterprises to corporate applications. How do you look at that market of: ‘should I just let my transit provider take me to all these clouds and all these apps, should I peer directly or should I look at these services that are kind of in the middle of that land?’

Yeah. Oh it's a really interesting area. As you mentioned, I was a Chief Strategy Officer at Console Connect, which did software defined networking to interconnect the clouds to the enterprise. And I wrote some research papers—I'll put them up on my website, when I finally get some time to pop it up. But I'll share with you a couple of interesting data points that CIOs should know about. First off most CIOs that are not in the network space, not in the Internet operation space, a lot of them don't have the strong or deep network engineering expertise in-house. And as a result, doing things like connecting to the clouds themselves can become a bit of a difficult challenge.

I've worked with some clients that only had a single CCIE on staff who is their host master that happened to also know a few Cisco commands—and that would be the level of expertise that you might see inside some of these organizations. And that's why a company like Console Connect and Megaport and these other companies have spurred up to make the software defined networking piece ease the path for enterprises in to be directly connected to those cloud resources that they care a lot about.

And for those folks who are not familiar with the services that we're discussing, there's certainly a wide variety of networking services that are somewhat overlapping, but at the heart of it, they enable you to get a single connection to a service that then can split up through VPNs (virtual networks) or even dedicated circuits at the higher end to other networking points like the big cloud providers, public cloud providers or peering exchanges.

Tell me a little bit about: why did they spring up? So we've talked about customers will want to exchange traffic with each other. Why do they want to do it via a peering exchange?

I guess I would say there's two different things here. We have to be careful about the terminology: the peering exchange I think you're referring to is a layer 2 type of peering fabric that network operators would connect into so they could peer with those who are on that peering fabric. I think of that as being a peering exchange.

There's another thing called a cloud exchange which allows you to connect through a platform to potentially many clouds that are also connected on that platform. It's a little different, peering is a reciprocal exchange of access to each other's customers—where the cloud exchange really is a way for you as an operator of those cloud services to get access to your resources in the cloud. So it's you getting access to that which you already have access to over the public Internet.

That's a good point. So in cloud exchanges, you're often connecting... you are the network on both sides, and you're using the cloud exchange to get for example to your own installation in Amazon or your own installation in Google as opposed to on a peering exchange, you're peering with another customer entity. You are... truly a peer, to reuse the word.

So ISPs use peering exchanges for example and the enterprises would largely only use peering at a peering exchange if they were just delivering a lot of traffic to or from another network organization or another commercial organization who also wanted to reduce their overall transit costs.

There are some folks like that. If you look at the peering ecosystem in my book, I talk a lot about the Internet peering ecosystem as being kind of like an organization—a living breathing organization where if you poke on one side, the reaction happens to the others that are in that Internet ecosystem. For example if you're in India and the government comes in and mandates anyone within this state has to peer at this location, that effect does in fact change the peering ecosystem within India.

One of the just general things I've seen across the world is whenever the government gets involved in any way with Internet peering infrastructure, people back away real fast. And it's a strange thing, but one of the worst things I think you can do is to have the government get behind and support a new peering exchange in the region, or mandate it—that's actually the worst is mandate it.

So that's an interesting difference too. So peering exchanges, to be clear, generally start at a data center and then some of the peering exchanges extend over multiple data centers and even over multiple continents now, right, they're starting to be connected. But peering is usually... there's a bit of contract for the circuit.

But the traffic exchanges are very often done with little or no paperwork at all, where a cloud exchange is a commercial service that people buy... in a similar way to how they would buy transit or data center or any other sort of vendor commercial service.

That's right. If you look at the traditional peering side, you'll find the ISPs who are seeking to exchange traffic with like minded other ISPs for free, reciprocal, you know: you benefit, I benefit, there's no paperwork, [and] handshake agreement is typically enough. And then you have the larger ISPs, large guys may not use a public peering fabric. They may just have cross connects between them and their peers in order to have, you know large scalability between themselves. Those very often are signed contract type of agreements. Then you have what are called the hyper giants: the Netflixs, the Amazons, the Apples of the world and these guys are every bit as technically competent as the ISPs that are operating the infrastructure and the core of the Internet. They just have a huge amount of traffic to send out. Almost all of it is outbound, and the core of the Internet is really the only place where a company like Netflix can get direct access to Comcast in the same place. They have direct access to AT&T [subscribers’] eyeballs and the same access to CenturyLink eyeballs and so forth.

So Bill I know you've continued to do a lot of consulting in this space. How should enterprises look at connection to clouds?

My view is that the modern enterprise is in a lot of clouds either now or that they will be shortly in the future. We talked a little bit [about how it is] because of partnerships and acquisitions, so even if the main corporation isn't finding itself split, they may find themselves over time having to deal with an extended network. What do you feel is like the best way to deal with that, and what are the issues that you think are most important?

You know one of the things I find really interesting as I'm consulting with companies that are looking at the cloud is this notion of shadow IT. Turns out that a lot of people within companies are trying to get their job done and using whatever services that are available for free that they can. This includes things like Dropbox and so forth, publicly available services.

So you have a real challenge if you're the CIO. How do you sign that you are compliant when you don't really know what all of your people are doing? And it turns out that there's a lot more access to external network resources than you might think. So very often that'll be what brings me in is: how can we get that connectivity that our folks depend on even better? We could shut it down, we could say “you can't use these various different services,” but that is actually probably gonna be worse than just securing the connectivity into the cloud service.

Yeah, I and many of my peers have experienced that. And then it becomes an effort for security teams to have to deal with, and IT teams to deal with in terms of pricing. And it can be a productivity problem also in the enterprise if you have too many overlapping applications where really maybe you should have less.

On the other hand, if you want to enable a workforce, sometimes enabling them as they are is exactly the right thing to do. And I think what you're saying is then in that case, similar to what you said before, then they become critical services and access to them needs security and availability and resiliency.

Now I'll tell you one thing I found fascinating when I started working with some of these clients in these different clouds. AWS has a mechanism where you don't need to use the public internet to get access to what they call your virtual private cloud, your VPC and you can do that with their service—they call it ‘Direct Connect.’ And the idea is that this is a pipe that goes from your datacenter across a private infrastructure—not the public internet—private infrastructure that connects into the AWS side. And they have a mechanism for you to configure this all via soft tools. The software defined networking folks would be able to also use software to configure that path—that private path from your datacenter to the AWS VPC, and do it all with points and clicks.

That's kind of the way that things are done these days, and it's really interesting that the different clouds have done different mechanisms for this same type of service. AWS has [called] Direct Connect. Microsoft has what they call ‘Express Route,’ which is a different mechanism for connecting into your resources. Microsoft said that enterprises want to separate out the infrastructure that will be publicly accessible, separate from the infrastructure that is only accessible over a private connection, separate from the Microsoft resources that the company might be using internally; and as a result the Express Route service consists of three peering sessions between endpoints on either side. Not only that—it's a primary and a backup, you're required (at least the last time I looked), to have a primary and backup. So that's six peering sessions set up to reach all your resources on Microsoft as your platform. Really interesting differences between these clouds Google's different also. They're just different terminology, different mechanisms and they all claim that these are driven by the enterprise.

And do you think that it would be difficult to achieve that same functionality on Amazon if you needed that?

No, you could actually do that entirely with Direct Connect and...

It's just not in by the default design, is the difference or the distinction you're making?

That's right. The idea is that you have different parts of the organization that may be accessing different things within the cloud, and you need to be able to tease apart these things because, believe it or not, research does not want finance to look inside their their resources and vice versa. These companies are very protective about their information flow.

The other thing that I found really fascinating was having do with price. When you use Direct Connect you're not paying Amazon for their Internet access for using their Internet access for sending stuff out. And if you do the math, Amazon charges about 8.5 cents per gigabyte for the first large chunk of traffic and they eventually get down to about two cents per gigabyte. But when you do the math and you figure out well what if I use of this full let's say 50 megabit for a second and how much would that cost? Well 50 megabit per second, if you do the math, it ends up being about $25 per megabit per second for all of your egress traffic out of AWS. That's a lot compared to public transit today.

The rates are generally below a dollar these days, are they not?

25 cents to a buck and a quarter is kind of the range I'm seeing in the field. So Amazon, Google and Microsoft are all making enormous amounts of money on their Internet service that they charge the cloud operator and by going direct, they don't make it zero but they drop it down to a point where it's substantially lower. And I have a business case for direct connect I can share with your audience if you like. Does that crossover point... at what point does it make sense financially to start saving money to go with a direct connect solution? And it's not all that much traffic—it's like if you have 20 or 30 megabits per second of egress traffic, I can guarantee and prove to you financially that you'll be saving money by using a direct connect service. Yeah likewise with Azure and Google cloud, they have different price points, but the same logic exists.

That's really powerful.

Part two of this interview continues on the next episode of CIO Speaks with Steve Ginsberg.