Understanding data analysis and online activity with David Belson

Avi Freedman: Hi, and welcome to Network AF. On this episode, I'm talking with my friend and once collaborator and hopefully future collaborator again, Dave Belson. We're talking about data analysis, what's going on on the inner internet, and helping people understand all that. David, could you give us a little bit of a background in what you're doing now?

Dave Belson: Sure. So, I recently joined Cloudflare where I'm the Head of Data Insight. Cloudflare, for those who don't know, our mission is to help build a better internet by off offering a variety of content delivery, security acceleration services. My role, right now, is to help do that by using our data to communicate about what's going on in the internet. So, when something happens, an outage or changes in protocol adoption, things like that, we want to be able to communicate about it and inform the industry that this is what's happening out there.

Avi Freedman: Cool. It's been interesting for me to see Cloudflare as they've grown. David Ulevitch, who did OpenDNS, introduced me to Matthew back in, I think, 2009-ish. Actually, when I was at ServerCentral, we actually gave the first six servers, I think, and then later introduced a friend of mine who became a Kentik co-founder who ran ops there doing net-booted, containerized provisioning stuff. I guess that could have been a startup, but just used for technology, and it's been interesting to see the more product- led approach, but then all the growth. It's also...

Dave Belson: Thank you for helping us get started.

Avi Freedman: No worries, but it's fascinating just to see the different approaches that you can take, not only to technology, to business that people have there. So, back in the way back, you've been infrastructure for a while. When I started at Akamai, and I think you predated me a little...

Dave Belson: A little bit, yep.

Avi Freedman: ...Akamai was just becoming the cool place, the brain-sucking cool place in Cambridge, Massachusetts, but you were at BBN, which used to be the place everyone was like, oh, the Mecca in the sky, doing cool stuff, helping build the internet. How did you get into networking and how did you find your way into BBN?

Dave Belson: I guess I'd have to say it started in college. So, I went to Steven's Institute of technology in beautiful Hoboken, New Jersey, which is actually not as bad as people like to joke it is. So, I started there. I was a freshman in 1990, so my first exposure there was probably to BITNET, and I remember we had payphones in the hallway. I had a friend up in Amherst College, so he was on BITNET, also, s we were running back and forth to the payphone.

Avi Freedman: Were you on VAX or Unix?

Dave Belson: We had VAX cluster. Yeah. We had a VAX cluster. All right, try sending it to this address, try sending the data. We finally got it working. So, it was like, okay, this is pretty cool. Then I think somebody else showed me anonymous FTP, so it just kind of grew from there. This was arguably pre-web. I do remember my first experience with the web was with the CERN line mode browser and played with it for probably 10 minutes or so. This is like, this is hard to use. This is not going to go anywhere.

Avi Freedman: When I started, when I was doing the planning for my ISP in'92, I tried using... I started in'92, earlier in'92. I tried using dub- dub- dub, which was not line by line, but I was just like, go for it. There's much more. The Library of Congress was like, go for it. Let's do that.

Dave Belson: Yeah, thankfully, it evolved pretty quickly from there because my bachelor's thesis was on computer media communication, so this book called The Network Nation that had been written about 20 years earlier, so in the mid-70s, and it was this projection of we'll do...

Avi Freedman: The Network Nation or the Network to the Nation?

Dave Belson: The Network Nation by Hiltz and Turoff, but it sort of predated and arguably predicted things like Usenet and discussion boards and all the things we take for granted these days. So, my thesis was looking at all the predictions that had been made and saying... They were basically saying they were predicting them for roughly the mid-90s, so I went and took a look at them and said," Okay, how many of them have come true?" So, in my thesis, there's references to various news groups and to VAX Notes and to VAX gateways. I look back now and I'm like, holy cow, some of it survived, some of it didn't, and some of it, we just don't even think about.

Avi Freedman: My personal passion is to help anybody that wants to make Usenet the basis of Web 3 where we're dealing with these same questions about moderation and that original, oh no, someone's running on the internet. That was actually Usenet it that they were talking about at the time.

Dave Belson: Yes. I mean, not to say these will solve problems, but they're not new problems.

Avi Freedman: Yes.

Dave Belson: Yeah, so I think got into it that way, moved to Boston to go to Northeastern for grad school and was casting about for a job, an internship or something that spring of'95 and found an ad for BBM Planet at the time. That was their internet subsidiary, looking for people to do web stuff. They were spinning up a managed web hosting business. I looked at the ad and I'm like, well, I've done four out of those five things. I always wanted to learn Pearl, so I'm going to apply, and then I realized it was a bike right away. I'm like, holy cow, the people who invented the internet are two miles from where I was living, so this is pretty cool. I joined there, and I think BBN was littered with ex-Deck folks, and then ultimately Akamai was littered with ex-BBN folks.

Avi Freedman: And MIT.

Dave Belson: Yeah, of course. Right.

Avi Freedman: It was the MIA board at MediaLab CS department, which was MIA, missing in Akamai. I think the week I joined Akamai, they hired 200 people.

Dave Belson: Oh, wow.

Avi Freedman: Yeah. InfoSec has gotten better. I think when the network people showed up, I plopped a WiFi gateway just on a random ethernet, just so that we could...

Dave Belson: I'm sure they loved that.

Avi Freedman: ...show them their AirCards.

Dave Belson: I remember looking at... I had pulled out, at some point, my algorithms book and opened it up crosstalk.

Avi Freedman: crosstalk.

Dave Belson: Yep. Yep, and I was looking through the acknowledgements and I'm like, holy cow, three-quarters of these people that are acknowledged here work at Akamai.

Avi Freedman: Well, I remember it was funny. The second week I was at Akamai, Charles came up to me, was like," Avi, did you know that the triangle inequality does not hold on the internet?" I'm like," You mean that going indirectly can be faster than going directly along the hypothesis?" He's like," yes, exactly." It's like," Oh, okay. Yes, I do know that. Frequently, we set static routes to go around peering when we were at NANOG and we needed to get to our own networks, so I am aware of that. It was just interesting, the different cultures. You had to have had a computer science background if you wanted to be a network person working at Akamai so you could communicate with people.

Dave Belson: That realization ultimately turned into a product, or I guess not a product, but a feature.

Avi Freedman: Yeah. It was a feature of the platform and then got pulled out, and Bruce Maggs is the coolest. I had my Blackberry and I had hacked the email client to send me something which Procmail would use to do folders and that without running exchange.

Dave Belson: Okay.

Avi Freedman: Then he got the code. I gave him the code and then he made it like he encrypted it. It was like," Hey, I can still go, too." I was like," Yeah, awesome." Every group actually started inside the engineering for a little bit before growing out of it. So, data analysis, I'm going to say analysis, we can talk about what insight means because there's data and then there's making sense of it. How did you get into that from more tech, hands-on work and even sales engineering effectively and producting it at Akamai?

Dave Belson: Yep. Yeah, so that was just the path I had come through was sales engineering and then into the product team. I was working for Brad Rinklin, CMO at the time. He called me to his office one day and shows me this presentation and says," Hey, we've been thinking about this and we collect a lot of data. We've got a lot of, I shouldn't say collect, but the optimized services throw off a lot of data exhaust.

Avi Freedman: We call it the mapping fumes, the fumes from mapping.

Dave Belson: Yeah, basically, and it's like we should try to do something that we can give back to the community with it. Okay, let me go take a look. So, I gave the presentation, thought about it, and we came up with some ideas about connection speed, connection quality, outages, things like that. I was like, okay, this is interesting. Now I got to figure out, where am I going to find this data? Where does this live in Akamai? Somebody, it might have been you, I don't remember, but said," Hey, there's this team here that does data analysis." I was like,"Hmm, okay." So, they connected me with a couple of folks on the team, went to talk to them and they said," Oh yeah, we analyzed it this way and this way and it can be sort of the milieu. So, we started out with a couple of things. It was ultimately what grew into the state of the internet.

Avi Freedman: Right.

Dave Belson: The first issue that we did back in Q1, 2008, I think didn't even have connection speed data. It was more like... I don't even know why we did it. It was IP... I think it was something to do with the number of IP addresses we saw by geography, but per capita. So, I think the notion was, where were we seeing more internet adoption or proxy for adoption on a geographic level. Then also, I think we started looking at connection speeds, but in aggregate, we defined, I think, what was it called? Low band? I forget what it was called, but looking basically like dial-up percentages and then broadband percentages, and then what we called high broadband, which I think at the time was like 10 megabits. Then as we evolved the report, we also evolved along with the FCC definitions of broadband.

Avi Freedman: Yeah, that makes sense. Back in 2000, we got approached. Actually, they wanted to go meet with Danny, but Richard Clark met with me and I showed him the inaudible and was maybe a little visionary describing what some of our capabilities were to analyze the data. Then six months later, Booz Allen came in.

Dave Belson: Came in for it.

Avi Freedman: There was a presidential task in order to get a synoptic view of the internet from some guy named Avi. So, we figured that out, but now obviously Akamai never looked at customer data, but looking at the internet and name servers. Actually, at the time, everyone was worried about the authoritatives. I was worried about the resolvers. If those are on any casted, don't have enough capacity, then no one can resolve and get to websites. So, we had had some of that going for trying to understand things to protect. So eventually, state of the internet wound up having some security in terms of attack in DDoS and performance. I know, I apologize. When Kentik was starting, we talked about adding a trends in traffic and now we have Doug doing that, but we never quite got that going.

Dave Belson: Yeah, I mean, we incorporated a little bit. I had incorporated, what was it called, internet events and outages, I think, where using the aggregate traffic data, we were able to show when there was an outage, when we were shut down in a given country. I remember Doug and I, when he was at Renesys, we bounced things off of each other. Okay, we just saw this weird traffic shift here or here. Did you see the same thing in BG or vice versa? We also looked at, I got my first taste of how real world events, not outages, but actual events can be seen through traffic. So. I think it was Obama's first inauguration, we were able to look at, and that's probably streaming traffic on the platform. We were able to chart and show that, as the inauguration went on, traffic went up and up and up. I think as soon as he finished speaking and the poet laureate took over, started, the traffic just fell off the cliff, and it was interesting to see that. I think we also had looked at eCommerce traffic, aggregated eCommerce traffic at the time, as well, and you could see eCommerce traffic dropping as the inauguration went on. Then at some point, as the inauguration finished, eCommerce traffic came back up.

Avi Freedman: Twenty years ago, it was Steve Jobs and Victoria's Secret, but you didn't compare those events to... Those were no longer there?

Dave Belson: No, I'm trying to think if I was even there at that point. I must have been, but I think I was busy being an SA and not paying a lot of attention to that side of the house.

Avi Freedman: Yeah. I had started in October. I guess you started in May.

Dave Belson: Yep.

Avi Freedman: It's funny. I'll just share one story. So, in the'90s, I got confused as a sysadmin trying to understand networking. I just started teaching people about BDP, and people are like," Oh, what's this question about SNA and X. 25 and AppleTalk, and how do you do all this stuff?" It's like," I don't have a CCIA." They're like,"But you have to have a CCIA," but I don't. I know stuff and I can talk about it, but at Akamai, I felt like an imposter because I would go out, and I sort of understood how websites are built and I know coding and I know sysadmin, and I knew what Akamai did, but I couldn't tell you how to do it with metadata and all that. I remember, because you were a technical consultant, sales engineering basically, and I remember when Danny took over the product group, he made all the product people take the TC crosstalk test.

Dave Belson: Oh, the test. Yes.

Avi Freedman: Danny was like," Oh, you'd be able to do it fine." I'm like," Danny, no. You have no idea. No, I just bullshit well. I have the connected understanding of what you can do, but I wouldn't know how to...

Dave Belson: You know enough to be dangerous. You don't have to be dangerous. You have the confidence to talk about it. I think that's the...

Avi Freedman: Well, that's the thing. I tried to explain to people," You can do it, too. You don't have to go do all that." So, mad props because I saw that TC Master test, and I was like that's really...

Dave Belson: I remember, I think I was a TC manager by the time we started doing it. The only recollection I have of it was flying to, it was a hotel somewhere. All the TC managers came together at a hotel somewhere that Evan had insisted was halfway between the East Coast and the West Coast. It was like a two-hour flight from San Francisco, like a five-hour flight from inaudible, if that's possible.

Avi Freedman: Right.

Dave Belson: I remember that I was smart enough to make my section multiple choice. So, I was done marking my section in a matter of hours, but there was, I forget who it was, one of the other TC managers had made theirs basically like an essay or multiple essay questions.

Avi Freedman: Right, yeah.

Dave Belson: So, they were stuck all day. I think they literally pulled them all-nighter to get it all marked, but that was a big thing at Akamai.

Avi Freedman: Yes.

Dave Belson: I don't know if they still do it or not.

Avi Freedman: I don't know. I don't know either, but I guess the underlying point is, even if you're not, I wouldn't say master anymore, but even if you don't have complete familiarity with the material, if you put your head into it and start banging your head against it, like what you're doing right now, that was how Doug... Jim did it by learning, and I did it by learning, and then Jim Cowie at Renesys taught Doug and answered his questions. I remember you certainly had the attention of the network group and other people who were supporting you.

Dave Belson: That's been the great part is having those folks around me that can help. I know that I can't do it myself, and I've never claimed to, but building that support team who has access to the data, who has the ability to write those big hairy sequel queries, and they can explain to you, okay, here's why what you think you're looking for is not what you're looking for, and we should approach it this other way, so helping think that way. I think, maybe even more importantly, treating me as an equal. I think that is one of the things I found at both BBN and Akamai was that there were people there who were infinitely smarter than me, but were willing to have that conversation with me as an equal. I think for me, that's one of those really important things.

Avi Freedman: I laugh, not because I think that everyone's infinitely smarter or just more accomplished. You look at people that have done such foundational things, like I could never do that. The truth is, you probably could if that's what you really wanted to do, but it will take time.

Dave Belson: You have the focus and the... Yeah.

Avi Freedman: No, generally I would say, all I know is people that worked at BBN and I was only there a couple times, but Aaron Block who went to Akamai told a story about challenging John Curran, who now runs ARIN, and not letting him in the NOC because he didn't have his badge and was dressed in jeans and whatever. He thought he was going to get fired because," Oh, no! I've challenged the CTO of the company, and John's like," Hey man, that's what the rules say."

Dave Belson: The rules say...

Avi Freedman: I can't do the Boston accent, but you know, that's what you're supposed to do. So, you mentioned collaboration outside the company. I've seen this in networking overall where there's often coopetition. It sounds like you've had an okay time of even people where, in some sense, you could be competing for the scoop or just generally competing, collaborating with, is that because there's something different or because everyone has their different take on it? What would you...

Dave Belson: I'd say yes to both of those where I think different organizations or companies are often collecting different data sets. They may be similar nature, but slightly different. I think maybe we're all competing in some sense, but we also all want to get it right if we're going to put something out there. So, one of the challenges is often when you see something, particularly anomalous in your graphs, knowing is that a thing or is that a database burp?

Avi Freedman: A bug? There's bugs?

Dave Belson: Did something get written into the database twice for some reason or not at all for some reason and figuring out, okay, we saw this in my company and this other contact. This other company saw it another way. So. I think that's... We all want to do the right thing, and we all want to be first to post about it, but I think we also all want to be confident in what we're posting because of the delicate balance of speed versus accuracy.

Avi Freedman: Yeah, that makes that definitely makes sense. So, I guess you could almost have... Maybe that would be a fun combo blog post for you and Doug, like the set of things that could be the noise which might make you think their signal from, as you said, double writing to not writing to the early keynote stuff where agents would inaudible themselves. You'd be trying to debug it, and it turns out it was just an agent that scheduled everything to run at the same time, to bugs in the software that do the telemetry. I don't know. Any other fun categories of missed anomalies?

Dave Belson: I don't know. I think one of the big challenges right now continues to be geolocation, so when you find those anomalies, figuring out where they really occurred. Yes, they occurred on the internet or in a network somewhere, but trying to tie that back to the real world.

Avi Freedman: Any geolocation that updates once a day is always wrong. Well, it's generally fuzzy anyways.

Dave Belson: Right. You're only going to get so good, but the level of granularity that I think we tend to want is really not available in that technology space.

Avi Freedman: Yeah, not unless you're in your Tesla or IOT or other things where, and even then it could be proxied through, but then again, there's latency techniques to help see that.

Dave Belson: Yeah, that's always been a really foundational piece of what so many of us are doing, and I think it's one of those parts of the industry where there hasn't been enough innovation. There hasn't been a lot of light shine on it.

Avi Freedman: We have this as a product to challenge at Kentik, too, where, if you're not careful, you can see something that's data, and it is an answer to the question that you asked, but the question you asked may not be the question you think you were asking because computers, of course, are very literal in what they do. So, you got to really think about that accurately before you draw conclusions. Then you also have to take this, what's going on in the digital world, and try to turn it into what's going on in the physical world where the infrastructure may not be what you think it is, and it sounds like there's also research not into the unique data set that you might have and not just collaborating or corroborating with others things that are seen, but can it be hard to track down what's the digital footprint of something that you think is happening online?

Dave Belson: Yeah, that is a big issue because it's easy to see the change in the graph, but to your point, it's much harder. In many cases, I should say, it's much harder to tie that back to what ultimately caused that change in the graph. In some cases, we know that it's election time in a country that's got a history of messing with the internet or that there was a big storm, a big typhoon in this country or well-publicized power outages somewhere.

Avi Freedman: Right.

Dave Belson: So, you can oftentimes say," Okay, we know there's a typhoon that just made land in this country," Oh look, internet traffic or the routes or whatever..."

Avi Freedman: Or recently Congo, right?

Dave Belson: Exactly, yep, but even that, it was... I'm trying to remember which came first, like the recognition that the internet had fallen off or the fact that people saw the volcano erupt, but yeah. So, in those large scale cases, it is often a fairly short path between changing graph and identification of a real world event. I think it's in the smaller cases, the more localized cases where we're seeing something that occurs on a particular network or we're seeing something in a state or a city, and that's where you really, I think, have to go sleuthing and trying to find news articles or social media or things like that. After Oracle, when I was doing the internet disruption report blog, I would try to track down a lot of those sorts of outages instructions I saw...

Avi Freedman: Localized regional.

Dave Belson: Yeah, and the challenge was that, using various tools, I'd see, oh, the traffic dropped on this particular network. So, I'd go try to find out what happened. I'd say," Oh, good, OviNet has a Facebook page," and then I'd look at the Facebook page and say,"but they haven't posted since 2007." They have the Twitter account, but it's never been used. It's just one of those things where some of the providers are really good about posting status updates or about replying to inquiries. There are a bunch of times I had to explain," No, actually, I don't live in Tonga. I'm living outside the country and trying to understand what caused this problem on your network that I observed," but you can't always find the root cause or some simple different root cause.

Avi Freedman: Yeah. It's interesting, though, that it's a hybrid, so needing to go beyond just the data that we see from internet senses and fumes sometimes to figure out what the root cause is, even just especially something you mentioned and something that comes up a lot in anomaly detection, something isn't there that usually is. What does that mean? That can be, again, even... If the beltway in DC is not busy on a Monday morning and it's not a holiday, then that's probably really bad unless it's COVID, in which case I guess we understand crosstalk.

Dave Belson: Yeah, and especially when we're analyzing traffic shifts for things like outages and disruptions, it's understanding how do you tell when something is not like it's supposed to be and figuring out, do you compare it to just that time the past day or what Mondays are usually like, and there's all the accounting, but it's XYZ Monday. It's a holiday, so that's why it's down. The alerts are still triggering anyway because traffic's not where it's supposed to be. I mean, that's a hard problem.

Avi Freedman: Yeah, a lot of operational products, you have to work in seasonality and things like that and, to some extent, you can use humans a little bit to help with that too. You have data. You can analyze it, making sure that the data actually is complete and you're asking the right questions. Try to correlate it. Maybe you corroborate it with other people doing data analysis, and then you have to explain it.

Dave Belson: Right.

Avi Freedman: One of the issues I always have is, I always need to have this little obvious saying no, no, give the idea of the answer. Don't lecture about every little thing, but no one cares about this BGP thing and, oh, isn't that fascinating about how meds work? You have to tell a story and how do you go about that? You think something has happened. You have these reasons to believe it technically and how it correlates, and you have to up-level that and explain it or, as we say inside Kentik, so what? They need to understand it and then put that into context. How do you go about that?

Dave Belson: Yeah. I mean, the so what's important because throwing up a page with a pile of graphs is interesting to some people, but if you really want to get somebody to understand what's happening or, more importantly, why it's happening, that's what you need to get to. I think you need to be careful around not going too far into the weeds, like you just said. I think it really depends on what you're trying to show. Sometimes, you can set up the thesis and show the chart, the graph, explain what it shows and then talk about the impact of it. So, I think that's one of the things that oftentimes we maybe do a mixed job of doing among the community of folks who follow the internet outages and shutdowns is, yes, this happened here, but what was the impact? What did it mean that all these folks were cut off from the internet for an hour or for a month or for whatever it is. I think that bringing it into the personal, I think, helps provide that context for it.

Avi Freedman: Or early COVID, poor performance of bandwidth means you can't work and have kids get educated, and that's a problem, right?

Dave Belson: That was a big thing. I think many of us are in a position where we don't think about that. We've got our multi hundred megabit connection from our local cable or telecom provider.

Avi Freedman: And phone backup, two phones backups. I've got Sprint and Verizon, and I'll go and...

Dave Belson: You're way more advanced than I am but, again, there are some of the kids who are taking their laptops and sitting outside at McDonald's or a library or something like that. We don't always think about that, but those are the stories. That's the kind of stuff where data can help tell that story and say, Hey, connection speeds in this geography are not to the point where they need to be, both upload and download, to enable Zoom learning or Google Meet learning or Zooms or meets for work or whatever.

Avi Freedman: So, in your career, I also guess recently and over time, have you dealt much with press and analysts or is it more marketing? So, any tips for building relationships and making it more natural and not, I have this thing which I want you to promote my company for me.

Dave Belson: No, I think the important thing there in building those relationships is demonstrating expertise. So, not only personally, because that's what gets them coming back, but also for the company or the organization that you represent. These may be no-brainers, but it's the kind of thing where, if you've shown that, hey, we, as this organization have this kind of insight. In Cloudflare, we have this sort of insight into what's happening on the internet, And I can represent that in a way that you can understand, and you can then use this context or explain to your readers or your audience. I think that's the way you build the relationships and you get them coming back that way. I think part of the challenge there, though, is instantiating those relationships. So, obviously, PR teams are there as support, but I think having an active and intelligent social presence, as well, it really helps there because I think a lot of times the journalist will say," Hey, this person's been out there tweeting or LinkedIning on a particular topic, and I'm going to reach out to them to see if they can provide some data or commentary on this internet event that just happened.

Avi Freedman: So, does it mean, once you choose your profession, that you're in and you can never change your cell phone number?

Dave Belson: Isn't that what Google Voice is for?

Avi Freedman: I guess.

Dave Belson: To keep the cell phone numbers forever?

Avi Freedman: So, I have four Google voice numbers from different numbers that I thought I would never change.

Dave Belson: No, I've had mine since... This particular one I've had since the early Akamai days. That's why I know those spam calls because the call will come in from the...

Avi Freedman: Yeah, 617...

Dave Belson: ...prefix 617, and I'm like, it's unlikely to be any of the three people I know that probably still have an early Akamai cell phone, so I'm just going to let it go to voicemail.

Avi Freedman: Surprising. They don't give you the option to block the NPA/NXX or send it that your phone is in.

Dave Belson: I'd love to do that, especially my home phone, which I look at. 90% of the calls we get...

Avi Freedman: Say that once again. What did you say? What kind of phone do you have? You mean the cell phone on your desk at home? Is that what you're talking about.

Dave Belson: The cordless phone setting in my kitchen.

Avi Freedman: Ah, the 9/ 11 anchor.

Dave Belson: It comes with FIO.

Avi Freedman: I guess I technically have a Comcast phone that crosstalk.

Dave Belson: It's the number I still give out in many cases to people. I don't really want spam calls on my cell phone, so if you don't really need my phone number, I'm giving you my home phone number.

Avi Freedman: Oh, wow. Okay.

Dave Belson: It's just easier to ignore.

Avi Freedman: Okay. I just get a Google Voice and you get the hilarious transcripts. There must be, so there's the people of Walmart, there must be hilarious transcripts from voicemail transcripts website somewhere. I need to take a look for that.

Dave Belson: If not, it's an opportunity.

Avi Freedman: Yeah. So, another thing that I've done that I think has been helpful over time is, if I'm talking to someone, whether it's press or maybe an industry analyst and we're not the source, point them to someone that might be. In fact, that's not how Doug got hired at Kentik, but he was at Oracle and someone wanted data. Well, COVID was going on. People wanted data about the traffic side that they weren't seeing and pointed it over and ultimately it helps build a relationship where you're not always trying to promote. It's like, hey, I can introduce you to someone who actually really knows that. I think I know, but I don't really, whether it's an expert or someone with data could be really helpful, too.

Dave Belson: Absolutely. No, it's actually a great point too, is knowing what you don't know and knowing who probably knows it. I think also, to that point, it's a great way to build the relationships with, not only the press and the analyst community, but those backend folks referring between organizations.

Avi Freedman: Yeah, definitely. It's tough in what we do, and I talk with Doug about this also because sometimes the things that make the best stories, we're not hoping that they will happen. We're not hoping there will be more. We're not hoping for COVID. We're not hoping for internet shutoffs. I don't even know what adjectives to use to describe some of the stuff that's going on today, but with that as backdrop, I will ask, what's the favorite story that you've worked on, helped with?

Dave Belson: It was actually not a...

Avi Freedman: A trauma? Okay.

Dave Belson: No, no. It was not an internet event story. A call had come in to Akamai PR, and they were looking for somebody to talk about submarine cables. So, the PR team turned to me and said, hey, can you talk to them about this? I think once it was for the Atlantic. I don't remember offhand, but I said, yeah I'll talk to them. I should be able to answer their questions. So, we were talking about submarine cables and how they work, and we started talking about how they're protected from damage. I was talking about how sometimes the shocks where they come ashore, sometimes their data centers on guards and so on, and the cables themselves have multiple layers of sheathing and so on. I said, but there are no sharks with lasers down there or anything like that. It was just an offhanded comment. I didn't realize that I was channeling Austin Powers until after the fact, and then the article got published, I think the next day or the next week or whatever it was, and the headline was something to the effect of like, no sharks with lasers. I'm like, oh my God. I can't believe that they did that, and the PR team is never going to let me talk to a reporter again but luckily, my PR team had a good sense of humor and it was like, yep, cool. No problem.

Avi Freedman: Well, at Kentik, it's funny you mentioned the Atlantic because I, again, when COVID was going on, I talked with Charles Fishman, and I really like the narrative, the people that tell stories. The news is important, but it's really fun to read someone's take when they're really weaving it into a story.

Dave Belson: It just makes it more personal.

Avi Freedman: It was entertaining when I worked with some of that in the'90s, but then during COVID, this article almost starts with a quote for me. It's like," Oh, my God," says Avi Freedman, which is not... I don't talk like that, but what was it he said? I said, Oh, my God! It would be all over" says CEO.

Dave Belson: Oh, I remember that article.

Avi Freedman: It's like, what if the internet died? It's like, I don't even remember saying that, but I'm sure it's true, but it's not that we go around alarmists, but you can ask questions and then get responses and it's interesting. So, I guess that's up there with some of the Telecosm stuff that I was in where I think that...

Dave Belson: That was a while ago.

Avi Freedman: I probably said it, but it was taken out of context when I said that Bob Metcalfe was an elder statesman doing more harm than good. That was when he said the internet was going to be destroyed.

Dave Belson: Right, at that point in time, yeah.

Avi Freedman: Then he called me a brash young brat wet behind my, ISP brat wet behind my packets. I was like, okay, whatever, but I wasn't trying to pick a fight. George Gilder just thought that he took this thing, which was about, I just thought it was bullshit. I thought BDP had all these problems and we can skip the crosstalk.

Dave Belson: It was click bait for, or whatever click bait was back in crosstalk.

Avi Freedman: Yeah, so it was click bait as a title. It wasn't like top 10 things that would kill the internet, but it was still... Maybe I'll have to see if I can get sharks with lasers in there somewhere or submarines with wire cutters. I remember Doug Humphreys'...

Dave Belson: More likely these days.

Avi Freedman: ...description of birds as two-year-olds with wire cutters, super smart, going around and cutting things.

Dave Belson: I've heard that one.

Avi Freedman: So, what's the most... Do you have any favorite question you've been asked?

Dave Belson: I think probably when I was doing State of Internet and doing the interviews for that, each quarter we'd come out with, Hey, here are the new rankings. Yours is fastest. A lot of times, reporters would ask," Hey, the US invented the internet. How come we're not the fastest?" It was sort of this interesting, okay, let me explain capitalism to you and why the big telecom providers are not going to invest in bringing fiber everywhere if they don't have a clear ROI for it as opposed to other countries where the government has said...

Avi Freedman: I would say if the government decides it's important, then they supplement.

Dave Belson: Yeah. If there's a mandate or support in other countries, the government has said, thou shalt bring connections with speeds of X to every house or apartment or whatever and, oh, by the way, here's money to help do it. In some of these cases, the telecom there also spun out of the government. They used to be the state-backed TELCO, and then they became independent until the government said we're going to help you do it.

Avi Freedman: Some take it from a view of a human right, and some view it as a strategic. They just make a strategic bet economically. So, Europe a little bit more human right historically, Asia a little bit more economic, but that's not clearcut true either, and often it's for both.

Dave Belson: Right, and especially the US with the geographic challenges also just in running high-speed connectivity.

Avi Freedman: I was in Brazil for 14 hours once when I was at Akamai and requested I come down and talk with Embratel, and we got the result that we wanted bandwidth-wise, but getting into the wilds of Brazil with radio is even more challenging.

Dave Belson: Yeah, you're not going to be able to run fiber everywhere.

Avi Freedman: Yeah. In the US, wherever you're going basically, eventually there would've been fiber. Maybe that wasn't clear in 1996, but would eventually be needed. So, are there any questions that you, I won't say hate, but are not as much a fan of that you get a lot?

Dave Belson: Not really. I mean, I think that they're all asked with good intention.

Avi Freedman: Right. Foundation.

Dave Belson: Maybe I don't always know the answer to them.

Avi Freedman: Yeah, that makes sense. As you do the analysis, I'm curious because Cloudflare has done some really cool stuff with the learning center. We've been doing a lot of work in that regards to it. Kentik, I've been super impressed with that. You could take a look at it. You can let learn about BGP and DDoS and all sorts of things. Then also radar, which I think predated when you joined...

Dave Belson: Yep.

Avi Freedman: ...was interesting. Of course, Sedexis has done some stuff and other people have done things, but Cloudflare's done a pretty good job of making it interactive. Any grand plans or vision for your work will work with that?

Dave Belson: Yeah, so that's one of the parts of my role is to help evolve radar. I was, I think, with the internet society when they launched it and saw it. I was like," Oh my God, this is what I've always wanted to launch," something which just brings together a whole lot of insight about various bits and pieces of the internet. So, I think what we're working on, thinking about the what does the future of Arrow look like? What additional content and data sets can we incorporate into it? Thinking about cycle times, what needs to be real time-ish? What can be updated daily or weekly or whatever the case may be. Of course, there's all the... Then once you think about all that, thinking about the core data analysis piece. Okay, we want to do all this, but you have the data at this sampling level, and then you want to look at it in all these different ways because inaudible cardinality to it. We don't want to build something that's going to make the underlying systems explode. So, thinking about that and then thinking about also, not only the radar tool itself, but I'm also working on plans for regular reports. So, how do we take, for instance, they've got the quarterly DDoS report at Cloudflare. How do we take that and then build that out for other similar areas, and then are there other longer form content pieces that we can put together that are data driven?

Avi Freedman: Interesting.

Dave Belson: Stay tuned, I guess, is the ultimate answer.

Avi Freedman: Okay. We'll see if we can convince the Cloudflare marketing department to link to the resources we're building there, but again, complimentary, but it's very cool what you all have done. Look forward to it. We're thinking about tools and looking glasses and all that, but I think you've probably had to deal with some of these issues, again, maybe before you started. There's also a limit to how much you can show because we all have a duty to protect confidentiality and PII of customer data.

Dave Belson: Yeah, absolutely, and always doing all of this with aggregate data.

Avi Freedman: Yeah. Yes. Please never say anonymize, because you can't really anonymize obfuscated aggregate. Those are all things you can do, and then ultimately when I talk with academics, I'm just like, look, state of the art is, you eyeball it and a human makes an opinion about could someone reasonably... You joked about a Avinet, but yeah. I mean, I have multi-home networks and you could, if someone has one prefix and it's a BGP thing on the internet and you have any source of major CDN or network, you could tell whether they're home or not if you want to rob them or do something else.

Dave Belson: Right.

Avi Freedman: At some level, you can't, but a pretty broad understanding that IP addresses and things are PII and even in many ways aggregates. So, anything you would love to see the internet data and analysis and storytelling community do this year, near future, that we haven't been?

Dave Belson: I mean, I'd like to get back to seeing people again out of my house. I think that there's definitely some good conferences that go on around that, like the Internet Measurement Conference. So, that one is, I think, arguably more academic, but maybe trying to bring together. I'd done this once at Akamai. I tried to bring together an inside baseball for internet measurement with Doug and some of the Renesys folks and some of the MaxMind folks because they were local, and I forget who else was there.

Avi Freedman: I think maybe I heard that once.

Dave Belson: Well, Don used to do the DNS inside baseball.

Avi Freedman: Right. Okay.

Dave Belson: I think Kyle helped support the effort that I did, but I think maybe something where it's folks who are not quite so academic figuring out how we can do a better job across the industry of sharing insights and potentially sharing data, if that's a practical thing. Love to maybe see the industry come together, and this is probably an uphill battle, but maybe some common data formats so that there's some trusted third party, an internal society, like party where everybody could say, here's our data about XYZ so they could aggregate it from Kentik and from Cloudflare, from Akamai, from whoever else has those sorts of measurements and provide a more comprehensive view of what's going on.

Avi Freedman: Well, I have been involved in the recent CADA conferences, and it's tough. It's a tough problem. The thing I've been pushing on that I really would love to see is, subject to being able to do so in privacy-respecting ways, have some understanding of relative importance of IP address ranges, prefixes, because often people look at BGP is the thing which everyone can look at, but they assume that every prefix is created equal or, as you were saying, per capita traffic per IP address or whatever.

Dave Belson: Right.

Avi Freedman: Where we might start is, there are some algorithms that let you take some traffic on a part of a network plus the topology and sort of deduce and, maybe again with customer buy-in, we could validate some of those models, and then that helps people use that. Maybe we could validate that with Kentik customer buy-in and then use the academics...

Dave Belson: crosstalk

Avi Freedman: ...to use that for other purposes, but it's a tricky problem. Then reproducibility or the word longitudinal, which means over time also is something that the academic folks are working on, but keep all the data for paper. Now, if Kentik does something where the result is privacy respecting, but the data underneath might not be, then again, that becomes a challenge. Let's just say trace route data not involving any private agents. So, I get some VPSs and I'm just trace routing from Linode, now Akamai to DigitalOcean and IBM, to name the three that I like. There's no issue there. That data could be that data could be archived and RIPE does that with their data. Those are things being actively looked at, too. Now, the scope of the data is pretty large, but then again, Cloud lets you do, if you have money, do stuff with pretty large data pretty fast.

Dave Belson: Right.

Avi Freedman: Any wishlist for internet infrastructure or industry folks, not the data sharing side over the next year or two?

Dave Belson: I don't know. Let's see. I mean, hopefully, government stops shutting down the internet. I'd love to not have to focus on that.

Avi Freedman: It makes good stories, but that's not how we want to get good stories.

Dave Belson: Yes, yes, absolutely. I mean, as far as the infrastructure side, looking forward to the results of the efforts in the US to help fund improve broadband deployment I'd like to be able to get back to doing another connection speed, connection quality focused report to see, ultimately, the impact of those investments.

Avi Freedman: That'd be great. We're thinking about what we can do at Kentik to help people along the RPKI journey. If you're a customer, we have a whole workflow about it like, see, this is what will drop if you turn on validation by different types, but how could we even do that in an open way? We have some open source stuff that actually does traffic analysis and could plug in and help people with that. Then I'd love to see us get to path validation. I'm glad we started. We have customers that are working. In fact, transit, some of the big wholesale networks that our customers that are focusing on BCP38, for its source address violation, which isn't the primary problem, but it is used in DDoS, is used volumetrically, or just people that have open reflectors. Sometimes, we get put in an awkward position where someone's like," Hey, Kentik, can you tell your other customer they're bad and you need to do this," but there's ways we can do it and say," Look, we have these dashboards, and we have noticed this, but then there's only so pushy we can be," but right at the same time, it's a little embarrassing as an infrastructure person that we've had these issues with the infrastructure for... We've known about them well for 20 years. We've had them forever, but originally, the design was, I could Telnote to a mail server and send mail to you from God. That was like, ha-ha, but crosstalk, but as the internet becomes more production ready, it's time that we dealt with these things.

Dave Belson: I admit to having done something like that a few times, probably.

Avi Freedman: That was how we... I mean, it still is too much of a guild system, which I've talked about with other guests that we won't get into, but the way I was forced to reverse engineer, that was, I got email from God. I was like, how did that happen?

Dave Belson: Wait a minute.

Avi Freedman: I don't think I went to schul last week. Then I was like, wait, let me look at the protocol. I mean, that used to be the way that people did it, and so it's nice that we have much more education. Some of the books behind me that you can actually...

Dave Belson: Yeah. I've been looking at them, trying to figure which ones I have. I think I have a couple of them, but not a lot of them.

Avi Freedman: Do you have a Telebit TrailBlazer modem? I didn't use Bitnet. I was using UECP. So you got to...

Dave Belson: I don't, unfortunately. So, I do have my bin of early internet ephemera, like my Netscape transistor radio crosstalk.

Avi Freedman: Somewhere I have Argon software, TCP/ IP software, and the buttons.

Dave Belson: I have my collection of eight floppy discs to install CompuServe, although I've started making little boxes out of those. Yeah, I have a lot of that stuff that I really probably should throw out. I also have a complete...

Avi Freedman: No, no, no, no. Send it to me. I'll buy it.

Dave Belson: I think you still have my Vec tracks, actually.

Avi Freedman: I do, yes.

Dave Belson: It doesn't work, but you have it.

Avi Freedman: Yes.

Dave Belson: I do. I have a complete set of wired magazines, actually, as well, although they've gotten thinner and thinner and thinner over the years.

Avi Freedman: Yes.

Dave Belson: It makes me sad, but I think next year is 30 years for them, so I have this grand vision of somehow finding enough free time each month to go through...

Avi Freedman: Oh yeah, that'd be cool.

Dave Belson: ...that month's historical issue, and I don't know if it's look at the articles or, for me, it's probably less the articles, more the ads...

Avi Freedman: Yes.

Dave Belson: ...and look at what companies are not around anymore, what's happening to these companies. Even during the conversation today, referencing so many companies that just don't exist anymore.

Avi Freedman: I have news for you. I want to do that with Byte Magazine, which I have multiple complete collections of, and I bet you could do the same thing with Byte where they were talking about networking and risk computing and AI and all this stuff, not the companies, but more the ideas, and you could say we're still...

Dave Belson: You were either in or on the cover of Wired, right?

Avi Freedman: Yeah. So, that was funny, for Sealand. We were doing HavenCo, so I went to the...

Dave Belson: I went to the airport where I saw that, and I was like," Wait a minute. I know him."

Avi Freedman: Well, Gail was, too, to be fair. So, I visited one of my VCs partners and loaned him an inaudible, which I think he still has. I forget. Was it inaudible? No, it was an inaudible, not an inaudible. He saw my storage area, which Gail calls AMOS, Avi's Museum of Shit. Along one wall were unopened boxes. Gail's attitude is, what kind of moron buys a box and buys something and doesn't open it, but this VC, he got me. He was like," That's so awesome. You have an infinite supply of Christmas presents. It's like things you know you will love because you bought them for yourself that you can open!" So, I was like crosstalk.

Dave Belson: When you open them, you lose value.

Avi Freedman: When I went to his office, he had the complete set of Wired, and I was walking over to the 2000. He was like," I get people to do that" because he knows people that have been around for a while. So, we'll get Kentik on there and Wired from Doug's work, but we'll get Kentik, hopefully. It could be Doug. It doesn't need to be me on the cover. So, we'll get that but, before you consider throwing them out, let me know. It's like I have a...

Dave Belson: I will probably keep them.

Avi Freedman: Of every Son workstation up to the 4, 500, every SUN, including a SUN-2, up to the 4, 500, but not the E10k in Virginia that someone donated to me because his wife was like," Out of the house. Now."

Dave Belson: I'll pay the storage, yes.

Avi Freedman: So, last question. Well, second from last question. Any advice you would give your younger self, whether it's entering BBN or in college?

Dave Belson: I think probably two things. I think one would've been useful for college and beyond, which is to get my hands dirtier. I think a lot of the folks that I've interacted with in the industry have had the opportunity to manage a network or, like you've said on past episodes, talking about breaking things, breaking the internet. I've never gone to that point of really configuring a network or doing something with routing. I've got a CS degree, but have not really kept with my ability to code. I definitely wish I'd done a better job there over the years. I think the second piece, probably professionally, is wish I had found a mentor earlier on and really had somebody who could help me figure out what I was doing, help guide me into maybe new areas and help introduce me to the right folks. I think I've done a great job of building a network. I think the places I've been have been fantastic for that, but somebody with a little bit more experience and knowledge helping push me in a given direction, I think would've certainly been useful, as well.

Avi Freedman: It can be difficult because, when you're earlier in career, you think of, you're learning the environment, you need to ask permission.

Dave Belson: Yeah.

Avi Freedman: Then hopefully, you're in a company, which is a separate topic, how to interview to make sure you're in a company where people are approachable, like what you experience at BBN Akamai, and to show interest. You don't have to be, I know Cloudflare used to use the term ninja. I don't really like the 10x Ninja Wizard, but you don't have to be the most super wizard who created the universe. If you're curious and learning, people generally in the field want to help you, but people often come from an environment where it's more hierarchical, whether it's family, cultural, whatever, so except for topic, how do we encourage that, something I talked about on the last podcast with someone.

Dave Belson: Yeah, and I don't have a complex home network. I don't have a rack in the basement where I need to set up separate subnets and whatever, so I just have my, whatever the stock FIOS install is, so I'm not even trying to get my hands dirty with doing something weird with that.

Avi Freedman: Well, but it could be learning about anything, and sometimes the people... Well, I guess the other thing is that you let your nerd passion fly about different topics as a way of connecting. It doesn't have to be the thing that someone is most connected about. I remember, I don't know, 10 or probably 15 years ago at NANOG, I was watching one of the great ones, Steve Bellovin, the original Cheswick and Bellovin firewall book and Bells Labs. I was hovering, as one does. He was talking about something, suitably nerdy, to some other people, and then he turned to me and he's like," So, Avi. My son thinks that I'm not enough of a nerd because of you." I was like," Oh, okay." I didn't know he knew who I was. I didn't know he knew who I was. I guess maybe he saw a inaudible presentation or something. I was like," What are you talking about?" His son saw them playing poker on ESPN for the World Series and was like," Hey, geeks can do that." I guess Steven said," He's a nerd. He does internet stuff." So, there's many things you can connect about and talk about.

Dave Belson: Absolutely. I often describe myself as sort of a mile wide and inch deep, so I think that helps me. I know enough to be dangerous in many cases, but that does help me have the conversations with a variety of different people across the industry.

Avi Freedman: Yeah, absolutely. So, how can people find you if they have something they're curious about, have a tip?

Dave Belson: The best way to find me probably is Dbelson, D- B- E- L- S- O- N on Twitter, or my website, which collecting links to what I've written and stuff. That's Dbe;son. com. So, that's probably the best two ways to find me out there.

Avi Freedman: Okay, cool. I'm Avi Freedman on Twitter and LinkedIn and Avi@Kentik.com. Thanks all for listening to Network AF Podcast. If you like it, please subscribe and tell others and, David, thanks for joining.

Dave Belson: Thank you for having me.