Building relationships as an internet analyst with Doug Madory

Avi: Hi, everybody. Welcome to Network AF. In this episode, we talk with Doug Madory, a friend of mine, who runs internet analysis at Kentik and did that at Oracle, Dyn, and Renesys. He talks about how he got into technology, his early career in the Air Force, both in technology and leadership, how he got into studying the internet. We talk a lot about building relationships with press, some of the stories that he's covered, what went well, what got more coverage than he thought, what got less, and generally, tips on building relationships, having a great professional career and working with the press overall. Hi, everyone. Welcome to Network AF. Today, we're with my friend, Doug Madory. Doug, would you like to give a brief intro?

Doug Madory: Sure. My name is Doug Madory. I'm the director of internet analysis at Kentik and I've been doing internet analysis for about 12 years now, starting with a company called Renesys. It was a small group that did a lot of BGP analysis, worked in the internet and telecom industry. I started off as the guy that would just write reports that nobody wanted to write and doing data QA that people didn't really want to do, and I was happy to do it, happy to be involved and I got into it. So then, the data QA work was actually a window into analysis because they're related identifying what are the problems in the data? You have to understand the data and how you might make a conclusion of something that's novel. So I worked there, eventually started writing my own blog posts, started to answer media calls on my own, then eventually I took that over from one of the Renesys founders, Jim Cowie, as he made his way out of the company and started his own next company. I've tried to make the most of that. So we were acquired by Dyn, the DNS service provider in 2014 and then at that point, we were rebranded as Dyn Research. Then a couple of years later, Dyn was acquired by Oracle and we were rebranded as Oracle Internet Intelligence. Through all that time I was writing a lot of blogs, going to conferences and just getting everything I could out of our data to tell stories, either with the media or the industry.

Avi: So studying the underbelly of the networks and turning it into interesting and relevant content.

Doug Madory: There's a lot of material there, yeah.

Avi: So how did you get into technology, generally?

Doug Madory: I guess it was in my household growing up. So my father worked at IBM. We grew up in Poughkeepsie, New York where it's the headquarters of the mainframe operations of IBM and my dad worked on mainframes. He wrote a low level code for mainframes for 30 something years, and so we always had computers around the house. I wasn't a big hack- y, coder guy, but it was something that was always there. I remember my older brother and I, we both had PCjrs when those came out in the'80s. They were trying to market computers to kids. We would play games on those. My older brother was a little more into writing code and he would copy it out of the magazine. We get the magazines where the code is in an article and you would try to type it out of the magazine. Okay. You've got one.

Avi: I've got crosstalk compendiums typing basic programs in.

Doug Madory: So I'd helped him. He was seven years older, so I'm just watching him and grabbing things for him and stuff and then we'd typed the whole program in. We would always have all these 10- minute cassette tapes on just how you could store it, which is amazing to think about, but we've got a little tape recorder plugged into the Texas Instruments, computer. He'd type all this in, record it onto a 10- minute cassette tape, then that would be our long- term storage if we wanted to play that back again. Technology's come a long way. Then, let's see. Later on, my father did degree in EE, and so I figured I'd do the same. It seemed I liked the stuff that he was doing and went on to go to do my undergrad at University of Virginia, in the engineering school there.

Avi: Cool, and when did you start using computers and technology professionally?

Doug Madory: I guess it was my first job out of college. So we didn't have a lot of money for school and I guess I never considered taking out a lot of debt, so the other option was I applied for and got a ROTC scholarship from the Air Force and was lucky enough to get a full ride. At that time, you could get a scholarship that whatever the bill came to, they'd pay, if you got the best one. They've capped that, given that university prices have gotten so expensive, but anyways, so it was all- expenses paid and then, I needed to go into the Air Force after that. But my Air Force time was a really good formative experience, so after graduation then I went to San Antonio, Texas. It was my first duty assignment at the Information Warfare Center. I was on the team of guys and gals running the IT infrastructure for the intelligence operation, and I started off as a network engineer. I needed to learn Cisco Gear and they were a well- funded organization, so we go to classes and do start our way through the Cisco certification. I still really liked that whole curriculum, the CCNA, all the certifications. So I went through that for about a year and then we wanted to do a Unix administration in our group. They didn't understand it. They didn't want anything to do with it. crosstalk.

Avi: It was the Windows background people.

Doug Madory: Yeah. I was like,"Really?" Because I prefer to be there, and they're like," Really? Do you want to be in charge of it?" I'm like," Yes." So everybody was happy until I became the head of Solaris Administration for the Information Warfare Center. In undergrad, everything was Unix- based, the coding and all the CS stuff, so that was very familiar territory. Anyway, it was a very technical job, a good one as a first position in the Air Force. Then, I did a good job there. I got rewarded with a command position. It was completely different and was a difficult transition to be in charge of 55 airmen in a tactical unit that has to be able to deploy on a moment's notice. It was just completely different skillset, but I made it work though.

Avi: That's cool. So like me, you like the system and distributed system side of the house and the networking, but what made you choose for the dark side of networking versus staying in or getting into the more CIS admin, now we say SRE and generally distributed system side?

Doug Madory: Oh, I can't say it was a real plan. So after the Air Force, I did a master's degree at Dartmouth. I spent three years in a more leadership management role and I decided I'd like to get back into the technical space, but I had been out of it for a little while and I thought it'd be good to brush up and get another credential in that space. So I came back to Dartmouth to try to get back into computer science. When I finished there, I spent a year at a defense contractor and then became the head of security at Dartmouth- Hitchcock Medical Center, the hospital system up here. That was interesting, and after a couple of years, I was like,"All right, I'm ready to do something else." There was this opportunity came open with, I knew one of the founders of Renesys who was a social friend of my advisor at Dartmouth. I'd seen him at parties at my advisor's house before. He actually was recruiting me in years prior. I never really took him seriously, just too bad. But, and then, I wrote him and I was like," Hey, you still looking for people?" Then, I met with those guys and I was like," This is actually fascinating." I had never knew that such a type of work existed. It definitely wasn't on my roadmap, and then I got in. When I started at Renesys in 2009, there's a good line, and David Carr's the late media critic for the New York Times. He wrote a memoir and he's had a colorful, troubled life, but when he cleaned himself up and got the job at the New York Times he says he had immigrant's love for the place. He loved it more than the people who were there. I feel like that characterized me at Renesys crosstalk I'd done a bunch of different things. I'm like," I've been all over and this is way better. This is way cooler than anything I've done before." The rest is history, I guess. crosstalk I'm still doing similar stuff.

Avi: You didn't think you were going to grow up to be a detective.

Doug Madory: Yeah. That's the challenge today in trying to advise kids, right? If I could talk to myself in 20 years, there's no way I could've predicted that such a job that the internet is such a big industry, that you can have specialties of someone who's actually just analyzing this thing, this technology that we're all dependent on. I guess when I say it that way, it makes sense, but it was hard to crosstalk

Avi: Detective is a much better way of selling it than internet janitor or network proctologist, which some of my friends think of themselves as.

Doug Madory: crosstalk detective. I need a trench coat or something and a fedora crosstalk.

Avi: Yeah, exactly and with the big spy glass. It's interesting because we get so disconnected from all the layers of things. But a couple of weeks ago I was down hugging the servers and putting routers in and forming opinions about which operating systems I liked and hated. It was refreshing because when you live up here and it's nice to remember the joy of... well, you said the immigrants love the actually putting hands on things and getting into it and seeing things. I guess, hopefully things will keep changing so it'll stay fresh and exciting. So you didn't set out to study the internet, but there was already a practice around it crosstalk

Doug Madory: I kind of fell into it, yeah.

Avi: So what does that mean, I guess, from a output side? What are you finding? What are you hoping to find? What kinds of things are you trying to surface and hoping to find for the kinds of analysis? What's your beat?

Doug Madory: Yeah. I guess that's why I've got a public facing thing that maybe people might be more familiar with if there's the internet shuts down or some development on the internet. If we've got data, we tend to send a lot of great data to comment on these things, then that's something I'll keep my ear out for. I try to put us in a good position where we can make an informed comment to inform the discourse, but that's not all my time. That's not a full- time job, so the rest of the time here at Kentik we were in the space of being a vendor to internet monitoring around BGP and Kentik calls it the tentacles of synthetic measurement. We call it performance monitoring, and so we've got a lot of history there. So I'm working with the product managers here at Kentik to try to share some of our experiences and what worked well in developing our next generation of synthetics products. So there's a product management component to it. And then there's also a sales thing of just lead generation, how can you provide material that helps the Salesforce convey the value of the products based on the data that we've got. So there's a lot of ways that you can use an analysis function. Then, I guess, data QA, I haven't done that much in the form of data QA, but that's a potential area as well. The skills of an analysis function can help in all those different ways of either media getting the brand out into the press, fixing data errors, helping sales or helping product management. So I see myself as trying to balance those various needs.

Avi: On the public facing side, are people interested in the technology underneath, the impact about things being down the policy towards the right side as you talk to, well I guess, the reporters that are the proxy for what people should be interested in? Where are you seeing the interest in terms of what's going on with the internet, the digital infrastructure, the applications that connect us all.

Doug Madory: Yeah. For the public side, there's a couple of, I guess, broad buckets. One is news that's accessible to the masses. It's like so country access going offline or something like that. That's understandable by most people without having a CCNA or a background in computer networking, and so on those stories that I'm defining what a CDN is or something to a reporter, that's one bucket of broadly accessible, maybe geopolitical things often fall into that category, things people can relate to. Then there's another one that's like the NANOG talks, I'm not giving NANOG talks on our country going offline. These are gory routing incident autopsies, or some describing what can go wrong and how would we propose to fix things, that's more technical and for those blogs, I have in my head a couple different audiences. Am I writing for a broad audience? Am I writing for an analog audience and where I don't have to define what a Predix is and I don't need to? I'm a point where I know if I have to define what a Predix is, that's not the story for you. I'm not use some of this vernacular in this space. So there's two maybe disjoint areas, and then I guess for the more technical side, what I've spent a lot of time doing is trying to understand how can we get more insight out of all this BGP data that's going on and that's getting passed around and what can we learn when it goes wrong? What caused it? What were the impacts? That's something that because of who we were, Renesys having all the BGP data and analytical tools, that was what we were well- suited to answer those questions, so I got into the habit of a lot of practice.

Avi: For those that don't know, NANOG is the North American Network Operator Group. It's the people that connect things together in the U. S., although it's an international forum and there's similar globally and-

Doug Madory: I guess when I say a NANOG audience, that includes RIPE. I don't want to leave anybody out. crosstalk

Avi: Yeah. AFRINIC yeah. All the crosstalk regional crosstalk

Doug Madory: So I use that in a Royal NANOG crosstalk

Avi: Yeah. It's our background. That's where we fell into it. No, it's interesting because often people come through and they're like," It's time for a new protocol. BGP is..." It's like," Well, do we understand really all the activity in the BGP thing before we go design a new one?" Now, obviously, with cloud and everything going on and infrastructure as code and other people's infrastructure as code, there is innovation and new protocols, but from an internet underlying perspective, occasionally, good at clouds, big companies say," Oh, we don't run BGP. We have our own protocols." It's like," I assure you, you do run BGP. To the internet, you can do run BGP." There's a lot of-

Doug Madory: Those things are never going away. It's a BGP version for one instance for the world forever, forever.

Avi: Well, there was an EGP and there might be something else, but we probably should understand it better before we go replace it. So how do you get from BGP updates to human rights? I've seen you do a bunch of articles that, as you said, shutdowns, or partial shutdowns, how do you go from," I'm watching BGP updates and making a conclusion about how people are behaving and oppressing folks?

Doug Madory: Yeah. So I guess I got started in that space during the Arab Spring, so in January 2011, when Egypt was having their protests and turned off the internet. Let's see, I remember Jim Cowie or resident founders sound a message," Hey, if anybody's around, I need help on a Wednesday evening," or I think it was. So I answered the call and so then we started. Because we had sold these tools to the telecom industry, we had a really good picture of every country. This is our product of understanding from a BGP standpoint, what's the typology of every country? How has that changed second- by- second? What's normal? What's different? So we had all the tools to start digging into this. So as soon as it went offline, then we started," All Right. What went on? What was the timing of everything? What parts are still connected?" So I was racing, doing as much analysis and handing it over to Jim, and who's synthesizing the sense of the blogs and talking to the press. Then, for the Arab Spring, we had Libya and Bahrain, Syria, a bunch of countries had their own events that we were trying to do some coverage on and some adverse event crosstalk sent to the internet of those countries. That was a high water mark, as far as media attention for Renesys. Clearly, we were filling a need that that was not out there of people who could analyze this. People knew the internet was down, this was getting reported, but at some level, there was a desire for a more technical analysis of exactly what was taking place. So we evolved to fill that role of being, we didn't see our role as being a flag carrier for human rights. But what we would do is we had all these technical assets, we would inform the discourse and make sure we've got the right facts. There's an implicit thing that we're showing off that we've got this ability, so that's good for the company, but our contribution will be about just objective technical analysis that makes sure everybody gets the story right. So then I feel like I've been doing that ever since, and then by the end of that, I remember I had spotted Syria. I think in June 2011 was there for a shutdown. We had started setting up more stuff to look for these things back in 2011. It's actually Egypt went offline, we had gotten a tip before it went offline to look for it, but I didn't have anything at the time running that would just alert me when these things would happen. And then crosstalk

Avi: That's interesting you got to tip so that could actually have been from someone that saw what was happening and maybe it was a little bit more libertarian than where their crosstalk

Doug Madory: Yeah, there was an engineer who reached out and who was like,"We've gotten an order to stand by to shut the internet off tomorrow." So we had a day's notice, but I don't know, sometimes you hear those things and you don't know how much faith to put into it, and so like," Okay. We'll see." Then sure enough, it ended up going down and so then, by the end of it, I had automated things that would clue me in when the stuff was happening and also by the end of it, Jim was very gracious as a founder to share some of the limelight with me let me start taking calls, writing my own stuff.

Avi: Mm- hmm(affirmative).

Doug Madory: I'll always be grateful for that. I never would have held it against him if he always kept it for himself. It's his company. It's a-

Avi: He's a great guy. I met him and Andy, actually, doing internet measurement things in 2000, and then when they were starting and tried to buy it when I was at Akamai and didn't, but no, it was fun to see them do that. We actually competed for BGP sessions briefly because Akamai, at the time, we needed BGP for figuring out where we were allowed to send traffic to our ISP partners, but we also did analytics and went and asked people for BGP sessions. So we had about a couple of years of asking IFPs and figuring out who could give better and then Renesys started giving people analytics on it, which was not a business that Akamai was in, although, I wanted it to be. It was not something that we decided to focus on and certainly the network group is not going to go build that product. So, no, it was just fun to see the success there. So did you walk into some pretty mature relationships with journalists that Jim had had, or how did you go about crosstalk

Doug Madory: I did not inherit, really. So once we got going into the 2011 and I started to write my own stuff, because I had been a little bit of an understudy for him, so he would travel around. He liked to go to secure places and give presentations. He could do his own analysis too. I don't want to take anything away from him, but occasionally, I'd have stuff and I could help him where I augment. So I was ghost writing a little bit, but eventually, I had enough, I could start doing this on my own. Then, it was helpful to have two people that would write these things and present at conferences. But the media stuff, I think once I started to get into a rhythm of how to look for a story that's newsworthy, I think I started to just develop it on my own. There may have been a few, they got handed over to me that were previously talking to Jen.

Avi: What was most helpful in building those relationships? Because a lot of companies go down the path of hiring PR agencies and doing pitches and-

Doug Madory: Yeah. Renesys was a 30% company, so probably too small to have a full time PR person, but it needed PR. So this was our solution to it since we had all this data and tools. I guess, definitely the writing, I got a lot of good feedback on how to assemble a story, how to write a blog. I think I'm okay at it. Then, also whether it's the media or a conference presentations, I know, I remember going too early NANOG giving a talk, being very nervous for it, did a crappy job, but you got to start somewhere and crosstalk

Avi: Jim had done a lot of them and Jim gave crosstalk

Doug Madory: Yeah. Right. He was fine. He's a tough act to follow, but he would be like," Listen, we've got the data. You're presenting on our data. The audience doesn't have your data. As long as your caveat is,'Look, this is what we have. This is the conclusion we're making it with our stuff. You're safe there.'" Even when we put out a tweet of regarding something that as per Kentik data, I want to add that caveat, both the point where it's coming from, but also just be like," I can't rule out that we aren't subject to biases in our data sources." That's implicit in that caveat, but that's going to be true for anybody. But anyway, but I guess as far as when you're talking to the media, the lessons are that obviously, you got to have something to say. You got to have something good that if you don't have that, then you don't. But that's not the only thing that need to be very timely. You need to say it quick, be responsive. Time is very important. Usually, if they're worth their salt, then they can size something up pretty quick and work pretty fast. So you need to be able to be responsible and be able to assemble a concise summary of statement about something that took place, and then if you don't know, just say you don't know and maybe make a referral, and it's good for you. It's good for you. If you don't know, tell him that ASAP, don't try to lead them on.

Avi: Or," We're not the best source for this," but as you said-

Doug Madory: Yeah. crosstalk We get queries about DDoS and I'm like," Let me see if I can get you somebody that knows this in that space, because I know enough to know that I don't know what I'm talking about in that space," so but I do know that people, so I'd make referrals. Then that's good to know, keep coming to you knowing that you'd be an honest broker if they had another question, because sometimes they're not in the space. They're not going to know what are the parameters or what are the things that you're an expert at. They're just never going to understand that. So once you've established the trust, then they'll trust that you'll guide them in the right direction and I try to be a good person.

Avi: Yeah. Well, and thank you for your referrals last year, when you were still at Oracle on people looking for traffic, which wasn't a data source that you had. We might've hired you anyway, but we crosstalk it was certainly I appreciated and generally try to do that. If it's an investor, I think that's a good lesson generally for people in building relationships. Certainly, it's the Silicon Valley style; it doesn't mean you have to be there to do it, but-

Doug Madory: What is the Silicon Valley style?

Avi: The Silicon Valley style is pay it forward, help people-

Doug Madory: Okay.

Avi: ...and good things will happen. In the'90s, I was just really frustrated with how crappy the documentation about routing and specifically internet routing was. So I did all those tutorials to help people and still people were like," Oh, I read that tutorial, and that was super helpful." Now, there's a whole industry of content marketing that people do, which is doing good for evil purposes, which I think is probably still okay. It's the same thing if an investor says," We'd like to talk to someone," maybe they're not a good fit for Kentik, but I'll tell them about other startups, try to make a recommendation. Then, you know what? Maybe at some point in the future, one of those other startups will find someone that isn't the right fit for them and refer them to us. Kentik is now seven- years- old, so some of those things that you do early on, eventually, maybe you talk to someone there in their first or second job, and all of a sudden they pop up and they're a leader or they're a good fit. They've moved crosstalk

Doug Madory: I've had that too, especially at Renesys inquiries from somebody who's a fledgling researcher trying to start out and has a question, I'd spent some time with them. Then, years later, this is a prominent person and they've really made it. I was like," Oh good, because I'm glad," especially in the internet space, crosstalk if that's unique to our industry, but if there's a young person who's really into it-

Avi: Yeah. I remember Nick Feemster when he was starting out, super voracious, trying to understand everything and I worked with him. I think I was at Akamai at the time, gave him access to resources. In three years, he knew more than me.

Doug Madory: Okay.

Avi: But it's a side effect if you teach someone something, there tends to be this dynamic where they forever think that you're smarter and more accomplished, even though all you did was know something they didn't yet know at one point. But-

Doug Madory: I think you were an embodiment of this as well, but you can have a lot of success being a decent human being in the space that's not required that you, it's probably better to do that. Then, I've got a bunch of people in the space, actually, there's a few that they have less of a name. I tried to help them out, and then we're still in touch and now they can help me out. But

Avi: Yeah. Yeah, no. Well, I certainly would advise that also there's different people have different styles and some people are less social than others, but certainly works well and I think helps folks in careers if that's something that they're looking for. So be helpful, don't over pitch, guide, even if it's not your own things. So you talked a little bit about talking to journalists. Any other tips for if you get approached, talking to journalists as part of building those relationships?

Doug Madory: One of the things that's usually important to have a sense for is a deadline. So is this something that's going to take a couple of weeks, or is this something that they're to file a story this afternoon, and how are you going to treat that or completely different.

Avi: Different levels of info and caveat.

Doug Madory: Yeah, like everything, everything. Like if it's this afternoon, then this is the conversation and there'll be no more conversation. The next few sentences are what's going in the story or nothing-

Avi: Right.

Doug Madory: ...so it'd be good to know that. Then, if it's in a couple of weeks and you're like," Wow," then it's almost like you have infinite time just to fact check and develop history of something. It's a bigger thing. Over the course of Renesys and Dyn, not so much Oracle, some media relations coaching, and it's been funny because I've done literally hundreds and hundreds of these. So then we've had a couple where I felt like I probably had more experience than the person giving the coaching, but I'll listen to them. Then I'd say," Well, here's all these points you ought to mention. One was the deadline thing. Everything you said was great." One of the things they'll say," There is no off- the- record. Everything you say crosstalk is true, but you do want to..." I think one thing, once you've established some trust with people, they're not going to burn you, because if they burn you, then that's the end of this relationship and it's mutually beneficial, so they're not out to get you. That was definitely the mindset in Oracle was journalists are out to try to stump you and make you say something stupid, or-

Avi: I think that there's investigative journalism, there's people that, again, it helps with context if someone's talking to you because someone told them to talk to you, then that's a middle level of maybe," Good, better watch what you say," but that's why I'm asking about relationships because if you have a relationship, then it's the same with everybody. It doesn't mean you don't have to be aware of where everyone is, but it's another reason why it's important to treat people respectfully and give and you can get. Personally, I am in awe of how it's the narrative journalists, what they do with the fact and the explanation and weave a story. So it's like, how do you give them things and contacts that help with that? I don't understand how what comes out is so great often especially, as you said, for the masses. So that's-

Doug Madory: This is also another thing I learned. If you can put it in terms of," This is the hottest or the wettest in the last decade," or whatever, or the something, some superlative to put it in context, that's something that they can latch on to, then for example, I remember in November 2019, we were covering the internet shut down in Iran. We had been far long scooped by NetBlocks had dedicated to covering this, but one thing I had noticed they missed, this is like the biggest outage in Iran ever. That's what the headline should be. I think I put that on a tweet. I could tweet, but I couldn't blog, or talk to the press about it. Somehow, that was my little loophole. I could tweet, then reporters could see the tweet and they could quote the tweet and there was a couple of headlines like," Oracle says it's the biggest outage ever in Iran crosstalk.

Avi: They decided not to fire you.

Doug Madory: Yeah, not on that day crosstalk

Avi: I did that a fair bit at Akamai, the stuff that it was like," Are they going to fire me? Probably not." So you when you pitch or you work on these stories with journalists, do you have any things you spent a lot of time on and thought were really important that didn't get much reception or really in the reverse, things that you didn't think were a big deal that wound up blowing up? What's been most surprising? crosstalk

Doug Madory: Yeah, definitely on the case of one that I didn't expect to be big, it ended up being the biggest thing I've ever done was the North Korean outage in December of'14. So to set the stage, Sony Pictures was putting together this movie, The Interview, a comedic assassination attempt against the North Korean leader. I still haven't seen the movie. The North Koreans were upset about this. They hacked Sony and all this, secrets came out of Sony. There's all these bombshells that came out and was a big news story. Then, President Obama at the time made some speech saying," At a time of our choosing, we're going to get back," in so many words," We're going to have some sort of response." Around that time, I had been working with a researcher named Martyn Williams, who is the author of North Korea Tech Blog. He was a reporter at IDG for a long time. He is probably one of the foremost experts on source tech information about North Korea. So he and I would often trade notes and collaborate on stuff and so we were already having a conversation of" Is there another way to look at this tiny little internet of North Korea?" I had some theories I wanted to run down with him, and while we were having this conversation then, I'm starting to get little alerts, those alerts that I set up after Egypt and the Arab Spring. So three- and- a- half years, or more, almost four years earlier were pinging me saying," North Korea, their routes are unstable." So BGP is a report- by- exception protocol. Everything's fine. Nothing's changing. There's no messages like this, the way it should work. If the routes could stay up. But if you're receiving a lot of messages, it can, it's not always, but it can be an indication there's some problem, because otherwise if you get a flood of messages on a prefixed, something's going on. It's having some issue. So we were getting alerts that the routes were unstable. They were up, but they were unstable in North Korea. So then, we were talking and I was like," I wonder what's going on there?" They were like,"I don't know, we'll check that out in the morning." Then, in the morning I come in to work on the four routes are down and I sent something over to Martyn again. I was like,"Hey, it looks like North Korea is down." I made some comment like," I wouldn't be surprised if they're under some attack or something." I sent that off and he said,"You mind if I publish that?" I was like," Go For It." So then I went into a meeting for two hours and then I came out and Martyn had published the story and it was huge. It was huge, and my phone was just blowing up and my email inbox was filling up and basically, North Korea had been down. I think I answered Nicole Perlroth from the New York Times. I tried to, in my mind, tried to triage based on my knowledge of the hierarchy of the media landscape. So if it's national TV news, that's probably number one, and NPR is up there pretty high, top- shelf newspapers, New York Times, Wall Street Journal, and-

Avi: Interesting, broadcast over print. crosstalk.

Doug Madory: Well, you think about it, every second is very costly to put something on the air. This is Doug's philosophy. I don't know this is crosstalk Right. So I ended up being on NBC Evening News, and so that's a 30- minute broadcast every second is super valuable; whereas, New York Times can always make another story and there's endless internet space. In the print paper, maybe they've got some bounds for that, but then they're just relying on reputation and stuff. But once you get to analog media, it's hard to break in there. Anyway, so then, it was just getting just inundated that we ended up being essentially, for that day, for that news cycle, the 24- hour New York news cycle, for the world, that was the biggest story that it seemed like Obama's made this implicit threat against North Korea, and then their internet went down. We were the main technical source that everyone was citing and wanted to ask the same questions. That one, from a technical standpoint, we're talking about 4/ 24s were withdrawn, is essentially, this is what took place. There was nine hours of it flapping and then, eventually, it just went offline and it's not super deep from a technical standpoint of what took place, so it wasn't super interesting to me, but the timing was right that it crosstalk

Avi: It wasn't a huge investigatory challenge and-

Doug Madory: Not really crosstalk.

Avi: 4/ 24s is the average in the days when people had offices and leased lines and such which some still do, that's like a small law firms worth of address space. crosstalk

Doug Madory: I bet anybody in this audience, the company you work for definitely has more than this address space, I'm sure and it's like a thousand addresses anyways. It's nutty how tiny their thing is. So then that night, I ended up being NBC News, they had recorded an interview for the CBS Morning the following day. I was on NPR All Things Considered. That was the biggest media thing I ever did and it just goes to show that it's just a little bit timing and you're not in complete control and it's hard to know what gets legs sometimes.

Avi: Mm- hmm( affirmative).

Doug Madory: Then-

Avi: Anything on the other side that you did crosstalk work on and...

Doug Madory: Yeah, I guess let's see. I know that, so one thing that's been a trend in the internet shutdown space is internet shutdowns for reasons to combat student cheating, just a little hard to understand-

Avi: Not of a country. Of a country, or-

Doug Madory: Yeah. Taking a country offline to try to combat cheating on exams and to understand this, you have to understand, and so I've been involved in both Iraq and Syria, I've done this, let's see here, Syria was doing it this year; Iraq, maybe, has stopped. But so what's happening like in the case of Iraq, public education goes up to the sixth grade and then you take an exam. If you don't score high enough on an exam, that's the end of your public education, so the stakes are actually super high. The odds are already stacked against you as a kid in Iraq. You've got a lot of challenges to overcome and if you're without the rest of your high school public education, it's even harder. So what ends up happening is, because the stakes are so high, then parents will resort to anything to try to make sure their kid gets a high mark. There's also just an understanding that everyone else is cheating, and so your kid, even if he's smart, is not going to make it because everyone else is cheating well, then you need to cheat too. So it's devolved into that and so what ends up happening as there's a moment where they publish the exams. The national backbone goes down while they're physically distributing exams. What would happen in the past is as soon as this took place, all of a sudden on social media, there'd be screenshots of every question and people would be solving all the questions and the kids would be trying to memorize all the answers. By the time they stopped for an exam within a couple of hours, maybe they didn't have it written on their arm. I don't know, but the answers would already be out, the exam's compromised. So, Iraq started doing this when ISIS had taken over to cover Mosul, this is back to 2015, and they started shutting the internet down for student exams and they had so many other issues. Anyway, this is a hard story to sell and hard for people to understand. I ended up getting some coverage on that and then when Syria started doing it the following year, I had an insider inside Syria telecom who was very useful in trying to help me explain to people what was going on. I knew when I was writing the one on Syria, I knew that it'd be hard to do a story. I'm just crosstalk doing this because crosstalk I just crosstalk needed to be done.

Avi: Maybe it's your American media relationships. Maybe it's just too foreign, because we don't all take the SATs at the same time. I understand the SATs are now vanishing, but crosstalk the concept of the national tests just being foreign, so interesting.

Doug Madory: I guess another example would be, I guess, we end up getting a little bit of coverage on that, but one that really just dropped like a stone was I thought the opening of Myanmar from being a closed country to rejoining the international community and doing all these political reforms in the telecom sector was a really amazing story of how they liberalized their telecom market and had this open auction of the Republic process. They had a couple of winners that paid top dollar for the ten- year exclusive rights to mobile operators there. They had this internet boom from zero to 60 just overnight, practically. It's an example that I later used to talk about what could happen in Syria. Obviously, Myanmar has gone a little bit sideways this year, so that's too bad. So I had been in touch with a couple of reporters in Myanmar. We were documenting as these things were happening. There would be a couple of outages people would suspect were government- related. I can't remember if there was a one that was a government- directed outage versus just things breaking. But when Telenor came online, we could see that in routing and that was a little milestone in there, but Myanmar was a tough sell as far as getting any coverage. Then, after a little while I was like," All right. Well, I'm just doing this because I'm just writing for Doug at this point. I think it's an important story. I'd like to have this documented," and so it's just crosstalk

Avi: Hopefully, they keep doing that.

Doug Madory: Yeah. crosstalk because then later they had the coup that began of the 1st of February this year, and then like everything, they had total shutdowns, mobile networks, censorship. They even had a BHP leak, a la the Pakistan YouTube thing. They leaked out a Twitter prefix that disrupted Twitter in South Asia for a little bit, just all the above. Everything we'd ever seen all took place within a couple of months in Myanmar. I had a bit of background at that time having knowing the landscape and I still had a few contacts in the country to help tell whatever I could help inform that story.

Avi: So, it's definitely interesting. Things don't always happen exactly as you predict, but there's no way to find out, but to actually do it. I've had pretty cool experiences with different, as I mentioned, narrative journalists, people that are trying to tell a story. George Gilder, as he was doing his Telecosm thing, had a friend and he actually joined us for dinner and then all these observations and stuff, and some quotes as I was trying to crosstalk.

Doug Madory: It's a skill, for sure.

Avi: It was actually BGP. It was at the time that that people were saying," Oh, the internet is going to die." I was trying to explain log- based scaling. We didn't have a route for every IP address. He made fun of me for being a nerd, but in a nice way, it was good. Simpson Garfinkle did this article on HavenCo and SeaLand and said," Welcome to SeaLand, now bugger off," but it was good. Then Charles Fishman for The Atlantic and it's like I talked with him and then at the other end, is this awesomeness that is like," Did I say that?" Have you had any experiences with where you gave an interview and you're happy with the result, but it wasn't exactly [crosstalk 00: 49:47 What was the best or most surprising interview, you've had?

Doug Madory: Well, let's see, a few. I'll mention a couple of little ones and I'll give you a bigger example, but I know that I did have one, I won't mention the publication. I had one, I think it was a Syria or some other country. They modified my quote to make it more crosstalk and I wrote back and I was like, it was basically saying something along," I've never seen anything like this," or something like that, but that's not true. I've seen something like this, but this is just happening today. I wrote back and I was like," That's not what I said." This was an email- based conversation, so-

Avi: Oh, wow.

Doug Madory: ...I was like," It's right here." The guy was like," Oh, the editor punched it up a little." I was like," That makes me sound like I don't know what I'm talking about here." I've had a couple of inaudible and I was leery after that with that particular individual. Then, I have another one. There was a publication during the Syrian Civil War called Syria Deeply that was just dedicated to the Syrian Civil War, and with just any fact, I think there was another journalism concept, or just try to bring every stitch of everything into this one source. I gave an interview there and I had known the journalist, she was at a previous publication. This was just surprising. When it got printed, it was basically just a verbatim of our transcript of our conversation was the interview because usually, they'd clean it up a little bit. So it was like everything, words and all like'ums' and'uhs.' It read like it was literally just the words, like they just put it into a piece of software and just pooped this thing out. I think I would have maybe been a little more careful or something of how I said it. I was surprised, but on the positive side though, in I think 2015, I got asked to join this investigative journalist team from New York Times who was going into trying to study online gambling at the time. That was one where they asked for some assistance. They were basically trying to track down where were these things getting hosted? The law has been changed, but at the time, it was illegal to do sports betting inside of the U. S. unless you're in Vegas or Atlantic City, there's a couple of carve outs, but you couldn't just online place bets. Having said that, everybody's doing it. So then, and basically it would just move off shore, but the understanding was that all of these things had just moved off shore, just being outside the borders of the United States, but in fact, it really wasn't. It was still in the U. S. a variety of things were happening. So I was basically tracing out where are these websites served up from, and trying to explain how the internet works. They actually had me down to Manhattan to the New York Times Building and had a conference room. It was definitely a life highlight crosstalk.

Avi: Were there whiteboards and crosstalk.

Doug Madory: Yeah. So I had a whiteboard and an afternoon in the conference room in the New York Times Building and had the investigative team, a couple of guys from sports journalism, and then a couple of folks from Frontline, which just ended up doing a documentary on it, which I got be interviewed in. Anyway, at the end it was amazing. I'm a huge fan of Frontline, and it usually deals with really, really grave important things like that. I think the episode before me was about infiltrating ISIS, and the next thing it was like something about crosstalk they're all really big, the biggest issues. Then, online gambling was in there too, but that was highlight, so I'm drawing on the inaudible why would I know anything here? How does BGP work and how would I know what's going to happen? How does DNS work and how would I know, and could I be wrong? We went through all that, you want to just suss all that stuff out. After I left, one of the guys on the team was like,"Do you know who the guy in the back was?" I was like," No, I didn't catch his name."" He's the older guy in the back of the..." I was like," No, I don't know who he is." He was like," That's Lowell Bergman." He said,"Do you know who that is?" I was like,"No, I don't," and then he's like,"Well, he's the executive producer of PBS Frontline, but he also, in the movie, The Insider, this is the guy who produced that story and he's a legend in the business." I was like,"Now we got to sit through the explanation of BGP-

Avi: Internet crosstalk.

Doug Madory: ...but that was pretty neat. Anyway, it ended up being a documentary and so it was a bunch of a bunch of work, similar stuff explaining how does the internet work and how does it relate to this story? Of course, a couple of years later, they they modified the law, so the whole issue was crosstalk changed. That was definitely a career highlight.

Avi: It sounds awesome. So what are you looking forward to now, either technology, besides COVID being over and everyone being healthy? Professionally, what are you looking forward to learning or doing or working on over the next year or so?

Doug Madory: So I am fortunate in joining Kentik at the time that it is, where the stage that it's at where it's got lots of data. I've got some pretty good tools I can use to answer questions that are really unique, and so right from day one, I was in a position where I could start, try to pick up where I left off because I had been a little bit limited by Oracle's conservative policies around working with the press, and so I didn't know. I was talking to you about maybe joining Kentik last year. I felt like I was always trying to be careful of what I can promise because. I don't know. Like I said, some of these guys I haven't worked with a long time and people leave the industry and they move on, they find other sources. I think I can get this going again, but I don't know. Sure enough, it looks like it never stopped. So I guess I'm, right now, trying to build out more capability to alert on more types of things that would be interesting both for news media and then also for our clients. A lot of, we would call, the analysis gun that we would use to do stories, all that capability is also useful for customers. I would also be looking out for anybody who was required a white glove treatment, then I was definitely watching anything that could potentially affect them and then, occasionally crosstalk.

Avi: crosstalk while you're collecting all this data and looking at it as right for-

Doug Madory: Yeah, and then we would get questions like," We think you guys would be in a position to answer this. Would somebody answer this and plus we're paying you a lot of money, so you can't say no." Jim would always take those because who else was going to do that? Then when I got proficient, then we could share that task. I guess, I wouldn't say I did it every year until he moved on, then I could help out in those questions where somebody would ask something and it would be like," All right, he's going to pay a lot of money. They're going to pay us a little more for this and we want to make sure they feel like they're taken care of. I'll do it. I'll put together some analysis on it." So probably the majority of my day- to- day work has never seen the light of day because it's more direct for our paying customers, but that's the way we are. So I guess I'm always continuing to try to build out more ability to analyze stuff. I think right now, I'm able to get a lot out of all the net flow that we've got, which is really unique, something I've never had before. We always had BGP and trace route as data sources and those can describe potential paths, but I don't know if I'd do a presentation about a routing leak. In the back of my head I'd be like," I don't even know if a single packet got impacted by this, I just don't know. But here I do know, or at least I've got a pretty good sense because if we missed it, then maybe a single prominent crosstalk.

Avi: Prominent aggregate basis from the people that have crosstalk.

Doug Madory: For sure. Yeah. Yeah. Yeah. Let's be clear on that. But if there's a routing, a leak or something and we can get a sense for where who likely was mostly affected or not. I don't need to rely on just how much did different routes get propagated, then you can just look at aggregate flows to get a sense, so that's cool. I guess the next phase is we have a lot of capability now in the synthetic space of running a lot of performance measurements, and I'm trying to build up more capability there to be able to tell stories in that space as well. That's what I'm spending a bit of my free time doing.

Avi: So looking back, any advice you'd give young Doug at the various points starting out either entering the Air Force or leaving?

Doug Madory: Yeah. A lesson I learned that I guess I could have learned earlier, relationships can be really, really useful. So the story I tell that illustrates this was, so I got to my second assignment in Aviano, Italy in August 2011, so September 11th, which we're coming up on the anniversary tomorrow. That was the following month, and then the rest of my time there being in a mobile unit that's tasked to be able to deploy at a moment's notice. Life was pretty crazy for the next three years and at times, we had stuff we needed to get out the door like send to Pakistan or something like really quick. I was socially friendly with the other lieutenants on the base and they call this in the space the lieutenant mafia. So these are all the young officers, often will go socially, go party together or do something together. But then they also represent the lowest officer, but there's a lot of power even the most junior officer can have and so it's a great network. That layer often knows each other really well. So we had something that needed to go, I had the guys stuck in the logistics line trying to get stuff onto a plane. I knew the Lieutenant who was in charge and so I called him up and I was like," Brandon, I need your help," and he's like," I'm on it." So then he went and personally saw to it to get it on the plane. I was like," Oh, that's helpful. That cost me nothing here." It's just the fact that we were doing the right thing, but those relationships can be super useful. Also, like with the relationships, especially in our space in our industry, there's a lot of people who are really here because they really like it, and if you have a question, they would love to answer your question. So that's maybe a untapped resource because sometimes you can sit and read something and sometimes you have to do that where you have to struggle with it a little bit to understand it, but you can also ask someone. You'd be surprised sometimes how much someone jumps on the chance to answer your question, just because they are really into it. Maybe they want to show off that they know it or prove to themselves that they know it. But if you ask it from a position of humility and be like," Hey, I don't get this. Could you explain this to me?" People will be very helpful. Anyway, I think those are things that I picked up along the way, but I guess I could have learned that a little earlier, but...

Avi: Don't be shy as long as you're genuinely humble about it crosstalk trying to build a relationship.

Doug Madory: Yeah. I guess don't ask someone to explain.

Avi: Right.

Doug Madory: You have to have done a little bit of work on your end.

Avi: Yeah.

Doug Madory: You got to know when crosstalk you're asking too much.

Avi: I found that in college, I went to Temple University, it was a commuter school. So the professors were all outstanding, but there's a lot of the students that just weren't in the position to stay up till 2: 00 AM just playing around with stuff because they had families and other and I was very fortunate. So I found that there was a lot of receptivity to if you are poking at something that's actually the area that they're working in, people get really passionate and excited about talking about it, and especially if you are interested in doing some of that work, professors are always looking for people to help study and do work and crosstalk

Doug Madory: I like to think I've been that for someone else.

Avi: Yeah. Yeah.

Doug Madory: If somebody gets me on the right moment, especially, if I'm in person, I could go for hours. It's how much time do they have, I could explain stuff crosstalk

Avi: Yeah, hopefully they become someone that can help others and maybe you learn, so-

Doug Madory: Definitely.

Avi: Yeah, definitely. Very cool. Well thanks, Doug, for being on and maybe we'll have you on again to talk about evolution over the next few years as things are going and-

Doug Madory: That'd be great.

Avi: Yeah.

Doug Madory: All right. Thanks a lot, Avi.