Networks and interconnectivity with Hank Kilmer

Avi: Hi, and welcome to Network AF. Today, we're talking with Hank Kilmer. Hank has run major internet backbones since the early 90s. He got into networking in the late 80s in college. We're going to talk today about both technology and people. The technology, what's been the same over the last decades as the networks and internet has grown, and some of the differences that are good, some of them that are bad. But we're also going to talk about people. What has been helpful to him and his career, what they try to do with Cogent to bring people in, and what we maybe, as a community, could do better to open up networking and help people get in. Thanks and look forward to having you on the episode. Hi everyone, and welcome to Network AF. I'm here with my friend and fellow networker, Hank Kilmer. Hank, can you tell us a little bit about yourself personally and professionally?

Hank Kilmer: Sure. Been doing networking since 1986 when I got into it. So I'm younger than I look, I guess. But needed a way to pay for college, so ended up getting involved back then, running networks and writing device drivers and everything else to make it all work back in the researchy days. Been doing it since. Didn't really view it as a career at the time because didn't know it would be a career, but it's been a fun ride. It's been kind of crazy.

Avi: Awesome. And you're in the greater D. C., Virginia area?

Hank Kilmer: I live in Maryland, so yeah, just outside of D. C. I've been born and raised in Maryland. The school I went to was Rutgers, so that was Jersey. So all East Coast. Lots of travel for the job, but otherwise, all here.

Avi: I've actually done SMDS peering back when Alex Latzko was at Rutgers. We did peering because I was at Philly, and we convinced Bell to moosh the latice together. It was technically tariff that they could. So that was a little funny. So yeah, I was in Ashburn to hug our servers and upgrade from the Sup 720-3BXL in the CEO cab three weeks ago. And I just kept driving around and I expected to see three times the number of data centers as two years ago, but I saw eight times. I just done an IR map of that area, and I remember-

Hank Kilmer: Oh. Yeah. And think of the power draw out there. I mean it's incredible.

Avi: Yeah, gigawatts. And I remember when Bob Gibson from Case took me to the secret mysteries of May East at Gallows, and I saw-

Hank Kilmer: In the parking garage?

Avi: In the parking garage. I think it was maybe when you were still unit, and there was this unit router on a two- prong extension cord plugged in. Meanwhile, I was reading this like," Oh, the NSA has all the secret taps on the internet," and I'm looking at the internet. I'm there, I'm looking at the five routers and it's like the internet and there's no taps, so okay.

Hank Kilmer: There are no taps. Yeah, that's right. That's right. I'll tell you, those kind of rumors all still exist. I mean, it's much more to distributed, it's bigger, but it's still the same thing.

Avi: I mean, we'll get to this. I've gotten to people that are like," Oh, provider X does this, sells their net flows?" Well, it happened that provider X doesn't do anything with their net flow or is just starting to... So unless Cisco is the wawe, which I think we'd see that in networking, it doesn't work. So you mentioned getting into networking. When I showed up at Temple University later than you, so I guess it was actually'87, one year, one year.

Hank Kilmer: One year. Yeah, I was going to say-

Avi: crosstalk. Two years later, I remember I went in and there was a professor who did computer image processing and a professor who did networking. And I was like, " Image processing AI. Oh, that's really interesting. Oh yeah, networking." And then I actually wound up being friendly with both of them, but I found the T1, the prep net access, and I did wind up getting into that. So how did you crosstalk?

Hank Kilmer: And for me, I needed a job. I needed a job. So I was looking for anything, and really just kind of got lucky. I was there for electrical engineering, I eventually switched to computer science, and my degree is in computer science. But I just needed a job. And they were building a new building and they were running some networking in there. This was freshman orientation week, and I saw them trying to get it so that these computers would talk and print, and I could tell they were struggling.

Avi: Was this SICNET and Bridges and-

Hank Kilmer: Oh yeah. Yeah, yeah, yeah. Your taps that you have to dig in, the empire tap type stuff, all that, yeah. And I could tell they were struggling, and so I just went in and offered to help, and they took a chance. They said," All right, if you can get these computers to talk to each other and be able to print by the end of the day, I'll give you a job." So I sat down. I'd hardly been on a computer at that point, and I flipped open the manual, and I found that the way my brain works, it was pretty logical, it was pretty straightforward, and I got it by the end of the day, and they were true to their word and I got a job. And the job was a mixture of running their networking on campuses, some getting more connectivity into the other universities and the ARPANET and that sort of thing, and some repairing of the hardware. I took apart monitors, discharged flyback faster and everything else, right?

Avi: General IT.

Hank Kilmer: Yeah, general IT. But I very much enjoyed that diversity. We got a new four. I think it was a Sun- 2 at the time. We got a new networking card that didn't have drivers, so I spent a couple days writing device driver for that and had to tweak some of the networking stacks so it would deal with the conditions.

Avi: So how many protocols at the time? So it wasn't just TCP, it wasn't just IP.

Hank Kilmer: No, no. It was a-

Avi: Decknet?

Hank Kilmer: ...little bit of everything. It was Decknet, VAX. We had the VAXes, we had Apollos, HP Apollo domain, which that one was crazy, because depending upon how you booted, it was a totally different machine. Sun. We had a good mix of stuff. So it was-

Avi: My college hack was informal. They wanted on the Apple. I think we had Apple laser printers that were hooked up to LPD, but it was a dollar a page. But the mainframe laser printers were free. So that was the grand project, a few of us tried to figure out how to get the CDC Cyber on the net with an LPD- compatible protocol so we could do our laser printouts to it. The dot matrix, that was free. So that was, I guess now we'd say corporate, it was college networking, but IT support, not just crosstalk.

Hank Kilmer: Yeah. We got a connection up, down the East Coast. I remember this was later in my college days, I was doing some research and they were doing some of the same research at University of Delaware. Logged into one of their machines to look at some of their research to work on it, at the time, people are like," What are you doing? Why? What do you do?" Well, you kind of didn't have a way to describe it so," I make it so computers can talk to each other."" Okay. Why?" And you get tired of giving examples like," Okay, well I'm doing this research, so are they, and then we can look and compare data sets." That kind of went over so many people's heads. It just wasn't in their world, because I mean, we're in the 80s here. And so I would just kind of say," Well, it's because I think it's cool," because it was just so much easier, just move on.

Avi: That's fair.

Hank Kilmer: Right? Just move on.

Avi: Even'92, so I was copying where you were at the time, I went to Disclave in'91 or'92 and I saw the DIGEX banners and they were Sun- 3s, and I'm like," Oh, I have Sun- 4s in my basement, I should really..." I'd always wanted to run a multi- line BBS, and I was thinking about how to explain the internet. And that was what I did. I wrote a C program that looked like a BBS, and so you couldn't explain IRC. I couldn't explain IRC to people."It's a 2000 line chat board," and like," Oh okay. I get it."" It's like a 900 line, but with your fingers." And now we have Only Fans, but yeah.

Hank Kilmer: Kind of crazy.

Avi: And then what happened was, people could see there was enough interesting that they wanted a shell account, which was ultimately the goal, was to get them into it. But of course, in'92, we didn't think people would have IP addresses at home.

Hank Kilmer: Well, right. Right. Well, crosstalk.

Avi: All you need is a terminal, right?

Hank Kilmer: Right. Yeah, right. All you need is... And I had a little deck terminal.

Avi: crosstalk.

Hank Kilmer: Yeah, exactly. And I thought it was the best thing. I still love that keyboard. I thought it was the best thing ever. After college, I-

Avi: Actually, where you went to school, I had friends, Dave Chavet, I think his father taught up there too, or no, his father- in- law taught up there too, but he had a Unix PC at home, and I thought that was the coolest thing ever. He had a terminal in every bathroom, and I was telling Gail," We're going to be like that."" Who needs to be connected in the bathroom?" I'm like," Wait, you'll see."

Hank Kilmer: You'll see.

Avi: I do.

Hank Kilmer: Somehow we'll get there, right?

Avi: Yes. So how did that go from... Did you take a special interest in the interconnectivity or was it just general, your college was just general networking across?

Hank Kilmer: So it was very much general, and it was a bit CIS admin, it was a bit of programming, it was a bit of the networking, a bit of the hardware repair and rebuilding things. So it was a bit of a jack- of- all- trades. When I got out of college, I got a job for a little bit as a CIS admin, which was fun, for a software development house. So they also had kind of one of everything to port their software to all the different types of machines, which was a lot of fun. And then UUNET. I got the job at UUNET when they were starting to try and do this alternate thing.

Avi: And so maybe you could describe, the internet at the time was NSFNET, was inaudible, CORE, the commercial. And so UUNET was the little dog at the time, right?

Hank Kilmer: Well yeah, it was, do we make this commercial at first? Right? And you had UUNET and PSI sort of going," Yeah, let's do this," right? You had the government control bits and go," Yes, but we still want to control how you talk to us," right?

Avi: Mm- hmm(affirmative).

Hank Kilmer: UUNET had its UCP side of things, which was a store and forward, not a-

Avi: You can see my Telebit TrailBlazer in the corner there, which was a UCP-optimized modem. Yeah.

Hank Kilmer: Yeah. Which was big and, I mean, it was really cool. It was great at the time, but they're like," No, I think we can make this work." And so they started to hire people to help build out that side of it, and I was one of those early round folks with them. And so you're dealing with companies and it was really more you're dealing with individuals. Yes, they had companies, yes, they had a job, but you're talking to somebody on the phone who's going," Yeah, I just wrote my own PPP implementation or slip implementation to get it so it could work over the Telebit in a dedicated manner." And so when you're trying to get them up and running and working, you're not only debugging the phone lines, you're debugging their implementation.

Avi: The modem serial port, your device driver, your memory end- to- end. Yeah.

Hank Kilmer: Yes. Which is, I think, part of what, at the time, was, having come from a little bit of a jack- of- all- trades, that helped me a lot in those days. But also, UUNET had a lot of really bright people there, which helped me a lot too, right? So it was a great job. It was a lot of fun. It grew so fast.

Avi: Well yeah, I mean, every three months, it was like another era. So at university and UUNET, you mentioned having to figure things out yourself. Really I think a lot of companies still look for people to take the initiative and will beat their head against it until they ger unconfused. But, you mentioned people, did you have people that were real mentors about technology or how to work with people, which can be confusing when you're just entering the workforce or?

Hank Kilmer: Right. So at the university, and I must admit, there were great people there. You mentioned one earlier, Alex and Rick Crispin, there's just a nice list, Mel Pleasant, a nice list of really bright people that I kind of had my own little job to do in my own little world and they would point in a direction, and then I would just run in that direction. So they were the guides. I didn't use them at the time as I kind of learned at UUNET how to use a mentor, because I didn't really know how to use a mentor, because to your point, there's a lot going on when you're a late teen sort of thing, getting into this stuff and figuring it out. But I would not have been successful without them either, but I didn't really know how to tap them the way I should have, or the way I could have, right?

Avi: Yeah.

Hank Kilmer: And the way I learned later, at UUNET, you would have what you would think would be the best idea in the world, and you would just walk down the hall talking about it, and by the time you got to the end, it would look so different. Because everyone was so bright, they could follow exactly what you're doing and give their input and advice and improve, and by the time you're at the end of the hall, you've got this much better idea, right?

Avi: Yeah, it's interesting. And both of us are from the East Coast, it's something I think that we need to be intentional about, which is that there's, especially academia, and people from the East Coast sometimes, not to overgeneralize, there's often the attitude that," Of course you're good. You're here." And then," Of course I like you, you're here, but I hate your idea. Let's talk about that." And that can be jarring for some people. And so I've seen companies try to be more intentional about that, not in the over- validating, but just reminding people that whether it's in that effect," Look at all our happy customers, look at the great product," without doing the everything is awesome, but that can also be intimidating I've seen, and it's something that networking should think about as we welcome people to it.

Hank Kilmer: I think our education system focuses on tasks and on more concrete knowledge, the soft skills of how to talk to somebody, how to know, I can criticize you and I can do it harshly and I can do it gently, and depending upon who you are, you may respond better to one version than the other.

Avi: Privately please, yes. Not in public.

Hank Kilmer: Well, sure. I mean, right, well, you criticize in private, praise in public sort of thing. But the point is, how you talk with people, how you interact with them matters. And it's actually arguably more important than what you know.

Avi: Yeah. My first non- consulting job, the CTO said," Avi, it's not sufficient to be correct. It doesn't matter," right?

Hank Kilmer: Yeah.

Avi: Which is another way of, it's sometimes okay for the customer to be wrong. Which is another way of saying the world is not zero and one, it's not right and wrong, it's not, this is the technically best thing that may be true. That doesn't mean people should change what they're doing, or it may not be true, it may just be a matter of opinion. Your opinions.

Hank Kilmer: Well, right. Well, but also fast forward, in my career, when I moved into management, I had an engineer who really wanted to try his idea out. I knew that it was not going to work the way he wanted it to, but I also knew that wasn't going to harm sort of the customer and product. And I tried to explain for inaudible to think about some of the issues, but in the end, it was," Okay, this is an area he wants to try. It's not going to hurt anything, let him run with it," because that's the way he's going to learn the best. You have to let people make mistakes, and then be there to help them and go," Okay, now, this is what happens, so let's learn from it and move on."

Avi: And we do a follow- up on engineering management, we can talk about what the balance there is, but absolutely. I mean, it's the same thing. If you hear it, well, or if you drive on autopilot, it's very different than making all the connections yourself, right?

Hank Kilmer: Yeah. Yeah.

Avi: I used to drive around without a map and just like," Oh my God, I didn't realize that was connected there." It is very different when you get to actually see it. So at UUNET, did you have people that helped you with the way of communication and people, as well as technology?

Hank Kilmer: Yes, very much so. Had mentors there in terms of both technology and being a team member and communicating and working with others, working with difficult customers, working with crazy employees as things grew, the gamut. UUNET was at a time where not only was the technology exploding, they were very good at selecting people to bring in that not only were really good at what they did, but fit into UUNET's culture, into UUNET's method. And part of that was getting a team that meshed. And as you mesh with people, you instantly build a trust with them, and that gets you to listen better. Because again, I was still young and learning how to work with people. But it was really good. And I think I was there maybe two, three weeks when I made an update to the router that ran in front of our firewall and I locked us out of it. And so I walked into to Rick Adam's office and said," Rick, I'm doing a maintenance this weekend on our main router," and he kind of looked at me and said," Why?"" Because I locked us out. I locked us out and I'm going to fix it and recover this weekend." And he gets this nice little grin on his face, and he's like," Okay, that sounds good. Go do," sort of thing. But he lets you, and he respected I didn't hide it. You learn when you see how he reacted that I happened to approach that one properly, and you go," Okay, that went well." If I tried to shrink and hide what I did, he would've had a very different reaction that wouldn't have gone over well. So I was observant, I learned, but there were also people around that had very good reactions themselves. They clearly had a lot more experience in life than I did, and that really helped me.

Avi: So a long time ago, one of my CIS admin mentors who would do things like send me mail from God @ onhigh to make me realize that SMTP was completely unauthenticated. It was clearly that same academic like," Of course, you're awesome, you're here, but I'm going to torture you anyway." But it used to be the lore that you weren't a real CIS admin until you had destroyed your first machine or real networker until you brought the internet down. But Internet's much more critical now. I mean, was it easier to learn by breaking things, and how do we help people when there's so much money and criticality flowing across networking now?

Hank Kilmer: It is very different in that way. At the time, not only would you break things with trying things out or whatever, but the technology wasn't staple either. So you're fighting both of those battles, and the level-

Avi: How often did you have code with your initials in it running on your routers.

Hank Kilmer: Oh yeah, yeah. All the time. All the time. And you're on the phone with the developers and digging into register entries, going," Okay, that doesn't look right." And you really don't get to do that any more. And one of the differences then versus now, the companies are way better at testing things, so what you get works better. But to your point, there's a lot of money in it, but things still break and things still don't work right all the time. And having a more complete view of how it all is supposed to work can definitely be beneficial. But to learn that is a lot harder. A lot of companies don't have big, massive labs to replicate this stuff, and how do you really replicate the size and scale and instability-

Avi: Of the internet.

Hank Kilmer: ...of the internet, right? And by instability, I mean, just route churn. It's nothing broken or wrong, it's just," I'm here, I'm here, I'm here," right? Okay, good. I see you. Right? But how do you do that in a lab in scale and allow people the opportunity to try different things? It's difficult. And there's an element of fear rolling things out and an element of fearlessness rolling things out. And honestly, different companies have different balances for that. I've been in the telco world where they want you to test something for eight, nine months, take three months to develop the rollout plan, and by the time your idea is actually hitting the network, the technology's moved three or four generations. But I've also been in companies where it's like," That's a good idea. Let's give it a try tonight." And that balance of where do you land is dependent upon your culture, your customer base and your talent pool, right?

Avi: Yeah.

Hank Kilmer: But it's hard. That's a hard balance.

Avi: Yeah. We definitely see that there are... I think there's a lot of evolution, because even people that come from a regulated world where, I mean, they're required to do all this testing and all that, their application developers want to work in this very continuous way. So we sometimes get people saying," Well, no, before kente, before you deploy anything, you need to give us three months notice," and that's not the way SaaS works we're deploying every day, or even the next scale. I remember when we were working with Yahoo, and it was like, well, every time you make a change, you can query the system and see whether anything broke before or after. And Igor is like," Avi, do you know how many changes we make a day?" and I'm like," Uh." He's like," Think about per minute. Think about per minute." I mean, obviously not mostly network, but deployments and roll- outs, the speed of orchestration. So I guess, as you say. Now, there is one thing we didn't have in the 90s and early naughties, which is commit, confirm and roll back, which helps a lot. For those that don't know, the way you used to try to avoid what Hank experienced was, you would tell the router, reboot in 15 minutes, and then you would make the change. And then if you locked yourself out, the router would reboot. Which wasn't good, but it might be better than-

Hank Kilmer: It gave you a chance, right?

Avi: And then you had to remember to cancel that, because the other problem was it took 15 minutes for it to reboot as it propagated everything at serial speed through the device.

Hank Kilmer: But the other thing, we had some failure modes where one router would reboot and it would have enough routing announcements through it that that would make the next router fall over. So the way you would end up solving that is you'd reboot at this time for the whole section of your network. Yeah, those days are behind us, thank goodness.

Avi: Your time at AboveNet and my time at AboveNet, we had the route redistribution incident, which I describe it to mathematicians as a distributed computation that never converges, which is basically crosstalk.

Hank Kilmer: That's good way to describe it.

Avi: That's why you need good inaudible. So you were at UUNET, DIGEX, AboveNet.

Hank Kilmer: crosstalk.

Avi: Did a lot of consulting now at Cogent. I'll take you on a trip down memory lane. If you think about, what were the biggest links that you ran at each company, and then we'll bring you forward to Cogent. Roughly. Roughly.

Hank Kilmer: Well, let's see, UUNET, I left UUNET when it was probably OC-3s at the time. Probably about that. Sprint was kind of a fun tenure. That was OC- 12s probably.

Avi: Okay. So we're still under gigabit. We're still under one gigabit for this. Just following.

Hank Kilmer: Probably, we might have had OC 48s then too. And then you're starting to move up the ladder after that, and it went kind of quickly after that.

Avi: 10 gid and-

Hank Kilmer: Yeah. As you moved away from the point- to- point link into your ethernet based, it went quickly at that point.

Avi: Yeah. Well, that was a great decision that I really... It created some trauma in my time at AboveNet. I mean, but granted, when I came to AboveNet, there was one big broadcast LAN. And bandwidth was expensive enough that people were like," Why do I get 10 megabits of broadcast, and why are you billing me$ 500 a month for that?" So we had to fix that. But the purity of we don't do serial connections, we only do ethernet. Well, ethernet is serial, but we don't do OCAT whatever, the customer terminates in Collo, connects ethernet, the purity of all ethernet to customer was definitely nice. And the-

Hank Kilmer: It's simplified so much. It's simplified. I mean, and kind of how I run Cogent is we keep it simple. It is a very clean design.

Avi: Well, I respect that. I've asked you for interesting ideas a couple times, and well, it's not just you, it comes from Dave, right? Focus.

Hank Kilmer: Oh yeah.

Avi: "Thisis what we do."

Hank Kilmer: Yeah. Our whole business model is focused. We don't have any of the TDM services. We don't have any of these things. He says no to a lot of product ideas, business ideas, because it diverges from what we're good at. What we're good at is taking a packet and delivering it where it's supposed to go. And really, if you ask us to do anything other than that, we will struggle with anything other than that. But that, we will do extremely, extremely well.

Avi: Yeah. No, it's interesting. I just plugged basically my first a hundred gig in, and there was a lot less drama than I thought. And I was just thinking about, this is more bandwidth than... It was like, when I was looking, I don't know, probably eight years at 100 meg card, 48 by 100 in a catalyst, and I'm like," Oh, that's the T1 board." And I was like," Wait, each port is a lot more than a T1," but what is small and what is big changes over time.

Hank Kilmer: That's right. You get your pizza boxes now that the lowest speeds you can get on it is 10 gig. In a multi- tenant office building, still most people want gig E. Okay, so that means I need something else, right? And you think about that, that's a lot of bandwidth.

Avi: It's actually interesting, and maybe something we could have a panel on at some point, how hard it's still is for devices to do a great job of... How much easier it is in networking to limit the physical port than it is to rate limit and traffic shape. It works, but there's just something about the physical limit which works a lot better.

Hank Kilmer: It works a lot better. I think things are more expectant of that in the coding and in the drivers and everything else. And so it's just more baked. It's better.

Avi: Yeah. I mean, even in the Kentik world, the devices do a great job of re- sampling. And so people could like," Oh, can we send you one record per packet?" It's like," Well, you could. I don't think you want to." And generally, things do an okay job. So Cogent, what can you tell us publicly about the scale, size, diversity countries, buildings of Cogent?

Hank Kilmer: Adding new countries as fast as we can, as fast as regulation in the country allows and we can afford to move in.

Avi: You have 10 countries, 100 countries? How distributed is the network?

Hank Kilmer: So Dave will give you the full number rundowns. We're on every continent. We are in hundreds. And I mean, it's huge. We run, it depends how you count. I think Dave tends to use the number 20% of all internet traffic on our network. And it's huge. It's every continent, so.

Avi: And so how many people... You asked the enable question, right?

Hank Kilmer: Yeah.

Avi: But how many people with enough access that they could?

Hank Kilmer: We run an old- school ISP, in that if you're a customer and you pick up the phone and you call in, the person that answers that phone has access to solve your problem, access and training-

Avi: Full stack.

Hank Kilmer: ...and everything.

Avi: Full stack.

Hank Kilmer: So our call center folks, our CST folks, they all have access to make configuration changes.

Avi: To the crosstalk or to the customer attached?

Hank Kilmer: To both. To both, if they need it. Cogent runs... And that's a good question that you ask, because Cogent runs a rather flat network. We're not a telco mindset that has a pure backbone edge division. It doesn't make sense to do that all the time. It doesn't make economic sense to do that all the time. It doesn't even make traffic flow sense to do that all the time. So if you're trying to build... The way it does make sense is potential device management, like what you're talking about. So we have really good internal training, we have developed that over the years. We also do keep track of the mistakes that get made and how to correct them and address it and that sort of thing to continue to refine that. That is a weekly process for us. But no, you call in, the person that answers it can solve that problem. They have access to do it.

Avi: I mean, it is like a SaaS company in some ways. You call it old school, but in a typical SaaS company, any developer can deploy. Now, we trust their wisdom not to do it without testing and not to do it when no one's around and-

Hank Kilmer: Right. And the thing with Cogent is, because we don't offer all these different flavors of whatever the customer wants, it is very standardized. So a customer configuration looks identical across the board. That makes it easier on the call center folks and the knock and my engineers to figure out what's going on, right? Check to make sure it conforms to the standard. Yes, no. Okay. If yes, well then we know how it's going to behave, how it should behave. So now, you're looking at what would cause it to not behave that way. Is there an outage? Are there errors on interfaces? Is there something else odd going on? Is there a bug? And go from there.

Avi: Which we talk about the bet all days, but networking, you still do need to know what it should be doing, because you can find and do find bugs more than in the kernel layer, so.

Hank Kilmer: Yeah. I preach, keep it simple. It is from Dave's point down, but it's also kind of the way I just view engineering in general?

Avi: So how many vendors do you run?

Hank Kilmer: We have a ton of vendors, it's one.

Avi: So you have one IGP?

Hank Kilmer: We have one IGP, we have one vendor, both for the optical layer and for the routed layer. Yes, so we have our one great vendor. And that allows us to have very deep ties within their business units, within their development teams. So when we do find problems, getting the right person on the phone is quick, is painless essentially. Pain other than that you're doing it because there's a problem, right? But when you have multiple vendors, you don't have as much influence with that one vendor, with each vendor than if you had it all aggregated into one, and so there are pros and cons to the different approaches.

Avi: Well, there's some protocols that do better with code diversity, like DNS, there's some protocols that don't always do better with code diversity like BGP.

Hank Kilmer: Like BGP or ISIS or something like that. Right. Right. In the networking world, people will argue that the dual benders, dual routers gives you that redundancy. I would argue that the only way that gives you redundancy really is if you build two-

Avi: Networks?

Hank Kilmer: ...complete networks that might interact and talk to each other back and forth, but you still have to have a path that's all one vendor and a path that's all the other vendor. Because if you have an interaction bug, if you have a bug in one vendor that's tripping up some data flow, whether it's packet- based, whether it's routing- based and you go through that vendor from customer to server or whatever they're talking to, they're going to hit it. So the only way you really mitigate that is to be able to switch up, don't use this vendor anymore.

Avi: Right. I mean, I think that the world has been better. Generally, I see the point. The internet was brought down multiple times in the 90s and naughts by, again, the BGP, where in theory, in networking, we have these things called RFCs, and people look at them to write the code. In practice, that's something that has often, when I talk to academic folks looking at simulation and modeling or network S& M you know, it's like, you know that the actual code has bugs and does not do exactly what the RFCs say.

Hank Kilmer: But even those that do, the RFCs will state shoulds and mays. And if I read a should and a may and I decide to not do, and you decide to do, we can have a conflict that they can cause an outage.

Avi: So if you think about, I know you said you don't have backbone or not, but there must be more than Aaron, who we see on the peering circuit, doing it. From the people that think IGP and picking... I guess we didn't talk about testing, but testing devices and overall architecture, so how big are the various teams, I guess, at cogent, I would say?

Hank Kilmer: So on the routing side of the house, it's in the ballpark of eight people. On the optical side, it's a similar size for the optics. We have very good support groups around us to you other aspects, but it's a small group. It's a smaller group.

Avi: So do you think that focus of design, focus of vendor, focus of product is the key to doing that?

Hank Kilmer: It absolutely is. We get requests," I want to tweak my configuration," and we say no to most of those.

Avi: I want this ACL this way.

Hank Kilmer: Yeah. If I don't have a way to tool it, to write a script that will make it repeatable, and if I don't have a way that even with that, that I can predict how it's going to interact, we say no. Most of the requests that we get, in all honesty, are better done on the customer side anyway. They just find it easier." I don't have to pay my consultant or whatever to do it, it's easier if you do it for me."

Avi: crosstalk firewall management company that just takes the Cogent needs and this customer.

Hank Kilmer: Right. And the other request that we get short of sort of these tweaks like that are things like the ACLs, where you're like," Okay, this will not prevent a DOS attack."

Avi: What you think.

Hank Kilmer: Yeah. Okay. I've got to carry a DOS attack all the way across my network to this port, and I will block it on this port. And you think that's a good thing. Well, I think that's a bad thing, and I would rather come up with ways to block the DOS attack than not carry it across my network.

Avi: Yeah. I know back in 2014, we talked about whether providers could cooperate to say," Hey, I'm going to filter this. Maybe you shouldn't send it to me." But of course, we've had issues operationally with Flowspec, and then, of course, people do want to get billed. People don't really want to generate DDoS, but they do want to get billed for traffic that is sent to them. So there's some complexities there.

Hank Kilmer: And the different hardware and vendors can treat some of this stuff differently. If I do Flowspec and I put out an ACL rule, when does that ACL take effect? Does it take effect before or after I count the packet, if you're a usage- based billing customer? Well, that can vary depending upon your vendor. So," No, I wanted you to block it. You didn't carry that for me, so I don't want to pay for it." But I still acted on it, so should you? It's a good question, right?

Avi: Yeah. No, it's funny how things can often not be as simple as they seem in the-

Hank Kilmer: Right. It seems like an easy request, but you get into some crosstalk.

Avi: Well, and again, there's people that will help customers that either lack the resources or sometimes sophistication to sell on top. And a lot of your competitors would sell a DDoS service and mitigation service, and Chinese vendors would sell a DDoS service. Most of my customers would sell a DDoS mitigation service. But used to be your Eastern European demand gen for the DDoS folks, but now it's global we'll just say, so.

Hank Kilmer: Right. It's spread its wings. Yeah, I would rather sell connectivity to those vendors. I carry bits really well, right? That's what we specialize in. Our design is very thorough in terms of completeness of value sets, how you learn routes, what you do with them, how you tag them, and our design is very predictable, so I know how it's going to behave. There are fiber cuts many times every day globally, so you need to have predictability.

Avi: Is the network down, the answer is yes somewhere.

Hank Kilmer: Always. Yes. But I will still get your packet to where you want it to go.

Avi: Even on the DDoS front, a lot of people, we will work with them and they have normally five gigabits of traffic maybe five years ago, or 20 gigabits or 50 gigabits now, and they get an attack every so often that's five or 10 gigabits. It's like, you know you can get a couple hundred gigabits of capacity at the same Equinix place you're at, not that you don't want scrubbers and cloud solutions, but you don't need to just run so tight. For the cost you're looking at, you can have a many multi- layer, right? You can have more bandwidth, you can do Flowspec yourself, you can engage, you could have on- prem scrubbing, you can do cloud scrubbing. And again, some people not worth it. They're just going to do always on solution or something, which is fine. A lot of diversity

Hank Kilmer: And to your point, part of Cogent's business philosophy is we want to commoditize internet service. So we want to make it. And by that, that means you have to be bringing the price down period, right? And so we focus heavily on that. The flip of that is that means that you should be able to buy whatever bandwidth you need and not worry about these caps, right? Have extra. Buy more. If you use it, great, if you don't, that's okay too. But in the DDoS world, there are different kinds. There's ones I want to take you out and I'm just going to send you as much as I can and they can send it a lot of traffic. But there's also now, and has been for a while, as you know, these little microbus of," I'm going to send you a huge amount of traffic for 15 seconds." And your mitigation systems have to be able to respond really fast for that, right? So the only way really to handle some of that is with extra bandwidth.

Avi: Yeah. Well, and capacity, and a lot of our, again, our more SaaS customers are likely to fire up cloud resource, change DNS, fire up more application layer. It's not just only about network.

Hank Kilmer: That's right. That's all right. Yeah, it's got to work in the end.

Avi: No, that's interesting. So I applaud the focus, even when it's frustrating to me personally, because I'd love you all to do something else. But if you look at Startup Canon, that is the mantra, is focus, focus, focus. If you go too wide, it can be difficult, especially at smaller scale. And I guess-

Hank Kilmer: Well, if you think about your background and mine and where we've worked, they've either, so many of them, UUNET, bankrupt, Sprint has plotted along, not gone bankrupt, DIGEX, bankrupt, AboveNet, bankrupt, right?

Avi: Mm-hmm(affirmative).

Hank Kilmer: And one of the reasons is they spent too much money trying to solve all of these problems or address all of these customer marketplace, right?

Avi: Yeah. And crosstalk.

Hank Kilmer: Cogent's profitable, right.

Avi: The company that bought my ISP, bankrupt. Bought by another company that went bankrupt. Akamai has focus.

Hank Kilmer: Akamai has focus, yes. Kentik has focus. And Cogent has focus. And Cogent is profitable. So it's crosstalk.

Avi: I would get yelled at if we were profitable at this point. So we actually last year had months where we were and I got yelled at. So I'm a bad CEO. Lose more money, grow faster, so.

Hank Kilmer: I'm happy to take more.

Avi: Yeah, no, but you want to know that the business can be, and you want to keep that in control. And that is, it's ultimately up to every company and investors. It's like a lot of people in Silicon valley would say," Oh, that's a lifestyle company." It's like, well, it's their company. Whatever they want it to be is the right thing. It's their company. It's not my company. You ask my opinion, I'll give you my opinion. But I worked at Server Central, which is an awesome lifestyle company and Jordan and Daniel and inaudible, they're happy doing what they're doing. Or Data Foundry just had a great exit, the inaudible prosperity sphere, Texas Net, Giga News, Golden Frog, Data Foundry. They did their thing and have also a lot of great people working for them. So last thoughts on, I guess, the backbone side, what's the same and what's different besides speeds and feeds from DIGEX, UUNET backbone, DIGEX, Sprint forward to now?

Hank Kilmer: So the underlying architectures are really still the same. Your routing architectures, even your trunking, even though it's different interfaces, it bigger, faster and whatnot, that's all still the same. You still have the same issues with the hardware and the vendors and dealing with that and their designs and how much queuing and tool sets you can do on it. So from a purely technical perspective at that level, I mean, there are tweaks, it's growth, but it is still underlying the same. And there's some pros and cons in some of the differences. The difference is what you get is way better tested, way better quality, so that you have less of these catastrophic bugs or interactions or surprises. That said, because it's a very complex setup in terms of your routers and the software and everything, you have fewer of the people, and some of them have retired and whatnot, moved on that, understand-

Avi: See the whole.

Hank Kilmer: ...the whole thing of how it works. So you have a bug in your router and you're dealing with someone who goes," Well, my world's fine," and another person who goes," Well, my world's fine." And in my world, I have one vendor. So I'm the one bridging all of these different people together saying," Okay, now, your world may look fine, your world may look fine, but this is not working, so we need to look deeper." And that is challenging. I also find that true on the networking side. I've got a small team of extremely experienced people, but when you want to bring new people in, you're bringing people in that," Oh, I got my CCIE and I know this bit of networking and, oh, I know this." And one of the things that I always ask... I run the department, so I tend to just look for personality fit and let the technical vetting go elsewhere. But I ask one technical question, you open up your browser and you type dubdubdub. google. com what happens? I want to hear how they think about it. I want hear whether they think about the DNS resolution and how that process works. And I just want to hear how they think about it. There's no right or wrong answer really. I'm trying to see, do they have more than just," I can configure a router"? crosstalk.

Avi: I used to ask people," Draw the internet." And it's a subset of that, but it's like... And again, that's not only a technical question, right?

Hank Kilmer: Yeah.

Avi: If you're selling internet or a CDN, maybe I should know where you're starting from. And I have a friend that worked at, well, my wife's best friend, but a friend of mine too, worked at Deck on some very technical stuff like the thing that did the run time running VAX binaries on Alpha for ULTRAX and BMS. And when I heard what he did, I was like," Oh my God, that's so amazing." And he's like," Yeah, we just ask people to compute polynomial and factorial and stuff. If they can't do the basic stuff fast, then that's the biggest predictor. And master's degree is the biggest anti- predictor." And the other networking question, which is really insightful is just two machines on a network when they find each other, and just understanding ARP and routers, which is passive observation, and cam and switches is subtle and can be subtle, especially if you didn't really get that in your head through debugging. Well, we'll come back to getting people in a couple questions. I've got a question about getting people into it, because, as you said, there's a world of complexity-

Hank Kilmer: There is.

Avi: ...there. So I notice we both took a detour, not a detour, a parallel path into policy. I was asked when the Aaron advisory council was formed, I describe Aaron as the printing press for integers in North America, IP addresses and ASs, and I made it a couple years and I saw you had a detour there. And I marvel at these messages that make it through my white list from John Kern, where I see all these things going on and ran BBN, and now he's helping the community. I mean, what was that like? Why did you decide... How do you think engineering versus policy stuff? Because they both involve people, right?

Hank Kilmer: They both involve people, they both impact what we do. And I think it's important to be involved. I think it's important to do what you can. But I also think priorities can take you in different directions, which is sort of the... I was at Aaron for a little bit, I've been on NANOG program committees and things like that, and so I like helping and I like reviewing, but also there's only so much time I can devote to that that isn't my paying day job. But I found that the policy side is much more people- oriented, less technical knowledge- based. Helps to have a technical background around with it, because it you're trying to make policy that works around or with the technology that you've got.

Avi: With. With.

Hank Kilmer: Sometimes around, right?

Avi: Yeah.

Hank Kilmer: But mostly it's people. It's interacting with people, it's getting consensus, it's figuring out how to listen. I think listening's a skill that I wish they taught in school.

Avi: Yeah. Especially with, again, a lot of smart people that are passionate about their viewpoint, it can be interesting. For me, there was enough talking about how to talk about talking about things that my ADD kicked in and I was twitching to write code. And Alec Peterson was actually," Avi, maybe this is not for you," and I was like," Yeah, you're right." And I'm so happy that people enjoy that and they do the ITF and they do ISOC and all that, but it is a great reason to go travel and see people that we know, but there's other ways to do that.

Hank Kilmer: It is. But there's other ways to do that. I think that the interpersonal relationships are inferred. I mean, I wouldn't be here without them, I don't think you'd be where you are without them. You can't emphasize their importance enough, but if you prioritize that, there are a lot of ways to do it. And like you, I found doing the policy useful to help the community, to help grow this industry. But it's also not... I would not want that to be my career.

Avi: As I said, John, you could probably guilt me or pay me enough to do John's job if someone needed to, but crosstalk do.

Hank Kilmer: Oh yeah. I would do a stint at it. Yeah. But I don't think I could do it as well as he does and for as long as he does, right?

Avi: I wouldn't try to be. I can just by my frustrations with them, I just decide to ignore it.

Hank Kilmer: We run one of the roots, so I'm part of that bit of the community there, but I also have help there with inaudible and crosstalk.

Avi: Well, someone who ran ICAN for a while that I had worked with, we had lunch or something and he's like," They're recruiting me to want to run ICAN." And I was just like,"No, I just can't. I would never do that." People are always going to be unhappy. There's nothing that you could do to make people happy. And I also didn't really agree with the clear direction they were going in terms of domains and stuff. But so bringing all that together, you talked about the complexity of modern networking and stacks. And we talked about primarily the ooze of nerd where you might have to write a device driver, your assistant men. You might help a confused professor do the networking, everything was related. And in networking, I've been, I guess, privileged enough to see iOS back when it was one program. It was calling ARP, and ARP was calling... I mean, in the bad old days. And so how do we bring people into the community or how do you... Because we can't mint people fully formed that have already broken the internet and destroyed the internet, whoever.

Hank Kilmer: No. And it is a good question, and I think that you have companies like mine, Cogent, we love to take people that have maybe a little bit of computer science or computer networking or something at a community college and bring them in and train them. And we've developed some pretty rich training. And then once they get in and they get their feet under them and they figure it out, they can decide what's best for them or what path to go in. Do they want to learn more of the DNS side of the world or the routing side of the world or the optic side of the world and all of that, or is it not right for them? I mean, they get to decide-

Avi: There's people that love supporting customers, and that's their career.

Hank Kilmer: Yes, we have some that you can kind of see it, they're going to do that for the next, however long they want to, and that's what they're really good at. I think that that type of... As a community, we need to recognize and do more of that. The colleges have some good programs, but they tend to want to specialize, and they want to specialize in programming or in networking or in the various different components. And if you specialize too early in a component, you don't understand how it all works together. And you could develop the greatest video processor that doesn't have the buffering to understand that between you and the server and the customer out in the world has latency to it and jitter and everything else. So, I mean, I've seen that repeat itself over and over again, so I'm saying that from history and experience, not just theoretical. And I think that general... It's kind of like, if you want to be a doctor, you become a doctor, then you specialize, right? And you go to more school for your.... We need to recognize that there's an element of this as we evolve that requires some of that. And I feel like at least right now, in our evolution, we're trying to specialize people too early that we're not getting that diverse view. I mean, some of the best engineers that I've worked with and that I have have no computer experience prior to this industry. They're just bright. Maybe they do AV stuff and figure out how all that works. Well, that's a network. It's not a IP packet network, but a signal that goes from there to there is not that different than-

Avi: Well, it's funny because we talk about network observability, but that's where it came from, right?

Hank Kilmer: Yeah.

Avi: Which is understanding an electrical network by observing its outputs and understanding what's going on. Yeah, it's a lot of the same processes. I mean, we have all these labs and testing and virtual things, a lot of it is build this. One of the things that... I don't have a CCIE, and we both come from a world where there was a ton of different protocols, and people would ask me questions like," What's your CCIE number?" It's like,"I don't have one. I have done IPX, but I don't know." I just hack it until it stops barfing on my IP network or whatever, or I did. But one of the things I always liked about CCIE was the lab. I'm going to give you a bunch of broken stuff, I broke this, and you need to figure it out. That's crosstalk.

Hank Kilmer: Go fix it. Yeah.

Avi: And how do we do that up and down and across layers and have that broad interest so it's not just," I want to be a WAN engineer, or I want to do this or that." I hope the maker... I was really encouraged 10, 15 years ago when the maker stuff started, because it takes people down a level. It can be really hard to be fair if you live in this world of all the beautiful UIs and abstraction and everything just works, to start poking at it can be tricky. Although we do have shells on OSX now, so that's at least positive, and Linux running on Windows. Who would've thought-

Hank Kilmer: And Linux running... Yeah, right. Exactly. Exactly. It's kind of converged in a lot of ways. But also, a lot of the younger people I know, I mean, I have a 12, almost 13- year- old daughter and her friend group and that sort of thing, they like that it all works, and they like that it's all easy. And they like that I can go in and fix it when it doesn't. I'm struggling to get them interested in understanding what's going on and why, and how crosstalk.

Avi: I have a lot of friends that are foundational builders, relatives that got me into computers that their kids of all genders are sort of the same thing and their attitude, I mean, just people I tend to hang out with, is whatever they want to do and be, they're not trying to make them be, like they wouldn't make them a doctor or a lawyer or whatever. Yeah, so they'll come from somewhere globally, I mean, but it's different than the hobbyist era. So okay, we'll put a note on that and think about that.

Hank Kilmer: I was going to say, effectively, that's what we've done in our sort of generation, for lack of a better term, is we were hobbyists. And it grew into a career. It wasn't really a plan. And now, if you want a plan, which is where I guess the industry is now, you need to entice people. You need to figure out how to make it interesting.

Avi: Well, it's not just networking. There's a lot of... You talked about medicine, tribal knowledge. There's a lot of professions where we're actually not as far from woodworker and shoemakers and the artisans as we could or should be despite Wikipedia, despite all this information. And networking is one of those areas that we need to think about.

Hank Kilmer: Yeah, networking is still part art, part science. So yeah, absolutely. That's right.

Avi: No, I mean, it's definitely true networking, distributed systems. There are people that lay hands on that couldn't necessarily explain exactly. They could probably reverse engineer it, but there's a lot of pattern match and," Well, how did you know that?"" I looked at the graphs and it just seemed like," but you do anything long enough. But again, medicine, it's a struggle for me to find doctors that will say," I don't know," which is awesome. I love hearing," I don't know." It's like, I love hearing," We don't know. We used to do this, now we do this." Tell me the probability space instead of just the," Go do this and see me in three months." I'm a horrible patient, so.

Hank Kilmer: Well, I get asked networking," How did you know to do that?" And to your point, I don't always have a concrete answer. I mean sometimes, but most of the time, I mean, there's still intuition, and relying on that. Trusting the intuition, but also relying on sort of the history of what I've been through.

Avi: Well, as Kentik grows, we'll definitely think about more the training, especially around internet working, and we see it ourselves in terms of a lot of people want to know about networking who come from software and there definitely could be better resource, so we can't solve it alone, but hopefully as a community, we will.

Hank Kilmer: No. Happy to help. Yeah, absolutely.

Avi: Thank you.

Hank Kilmer: We need it.

Avi: So last question. I think we've both been really fortunate to be able to make a career out of things that are fun, but any advice you'd give younger self on day one," Can you make this thing print?" or a little further on?

Hank Kilmer: I think I would've loved to have told myself early on to pay closer attention to the people around you earlier. I learned it, but earlier to tap into their experience more, to tap into their brains and their personalities more, to see the pros and cons of different approaches more. I was fortunate to get in around some really good people, and I could've learned a lot more than I did earlier. I think I kind of learned that lesson as I went and got better and better and better and better at it. And it's funny, there are times I'll refer to myself at this point in my life as a social engineer, no longer a network engineer, because my job is not necessarily to have my fingers on the routers anymore, it's to get everyone on board, to go in the same direction, to support each other and work as a team and all of that. So the earlier I learned, that the better life would've been, I think, and listening is a big aspect of that. I wish I was a better listener earlier.

Avi: Well, it's interesting because I don't always do this. I could be better. We could all improve. In fact, starting Kentik has made me realize how much I suck at so many things, but it's awesome because I don't like being bored. But I think if you're doing a great job as a leader, people think," Well, all you do is tell people what to do." I think it's actually much more listen, ask and help, which also at the early stage, if you only ask and don't do or help, then that gets crosstalk. If you ask looking for the answer that you want and then you la, la, la, la, that's also crosstalk those things that's generally-

Hank Kilmer: I would add one more word in your circle, trust. You have to trust the people around you. You have to earn their trust that you're going to help. If you just listen and don't do anything, that doesn't get you anywhere. But if you build that trust where they know that if they're going to share something or answer your question or challenge you, I don't know it all. I might be saying," Let's do this," and it's a really horrible idea. And they have to trust that they can say to me," That's a really horrible idea," and that I'm going to accept that and I'm going to listen and I'm going to think about it. I'm not going to just react," Oh, how dare you," and all of that. I mean, I believe strongly in taking the time to reflect, try not to have that,"And he said..." People can tell me that I'm wrong, I'm not.... And they can do that politely or rudely and I accept it. But that's part of who I am. That's what I bring. But it is really important. And I think that helps earn the trust. They know they can say and do anything and I'm going to support them and they're going to support me.

Avi: Or even as you described with Rick Adams a long time ago, you had the trust you could share with him and you established his trust by sharing, being open. And then he's like," Okay," and if he thought," Okay, well, you screwed it up. You should have someone else check it," then he would've told you and you should do it.

Hank Kilmer: Yeah, that's right. That's right. I mean, so I think I earned trust in his eyes, but his reaction taught me a lot about me trusting him as well. So anything moving on, it was... I don't even know if he would remember it. He probably wouldn't in his world, but to me, it was a big deal. Very nervous early in this job, and yeah, it was a big deal. But I learned a lot about that respect and trust and honesty and how to approach things.

Avi: Yeah, interesting.

Hank Kilmer: Mistakes aren't bad. It's how you handle it that can make it bad.

Avi: Or experience is what you get when you didn't get what you wanted. As you said, sometimes you need to get yourself unconfused. Hopefully, with this, is limited a blast radius as possible, which good architecture helps with, which we can have a look at some other time. Well, Hank, thank you so much for sharing.

Hank Kilmer: Thanks, Avi.

Avi: And it's been great to work with you and the people who are at and have come through Cogent. And maybe we'll, as a community, continue to work. We definitely will on trying to help people understand and get in to the community so that, well, we can keep zooming like this and people will crosstalk.

Hank Kilmer: And wee need to keep the crosstalk. That's right. That's right.

Avi: We can't just take people that started as hobbyists in the 80s, that will not work. It needs to be a diverse set of people.

Hank Kilmer: That'll run its course. Yeah, that's right. That's right.

Avi: Yeah. No, there definitely are younger folks, but we need to help. It's possible to get there from the outside, but it needs to be a little bit more ordered and also tell the story better, which hopefully-

Hank Kilmer: I think telling the story, I think, a little more community involvement in the labs and training and teaching. It shouldn't just be the vendors, it shouldn't just be university which has different focuses, right?

Avi: Yeah. Well, plus, it's not actually a technology thing. Peering is not actually a technology thing. No.

Hank Kilmer: No.

Avi: Right. So the business and economics and politics and all that. But again, different topics, different times. Well, thanks again.

Hank Kilmer: And I've been on both sides of that politics.

Avi: Yeah. Well, that's the funny thing is, different companies in the course of their life, they go from the bad gala guy to the inaudible people. So it just shows that people tend to behave in the same patterns when they have the same inputs. Okay. Well, thanks again.

Hank Kilmer: Thank you.

Avi: And I look forward to chatting with you soon.

Hank Kilmer: You bet. Take care.