Problem solving and pinball with Jay Adelson
Avi Freedman: Hi, welcome to Network AF today. Today, I'm pleased to have my friend, Jay Adelson, who I've known for many decades and used his services, and I'm proud to have him as an investor in my project with us. And Jay, could you give us a quick intro? Tell us about Jay.
Jay Adelson: Sure. Well, first of all, Avi, I'm honored to be with you today. It is always an honor to be speaking to you and fun. I'm Jay, I've been in the internet business since 1993, give or take, when I was broke, trying to be a filmmaker and audio engineer. And a friend of mine, Jeff Rizzo, got me a job at Netcom installing routers for people. And that was a very long time ago. And then if you just hit me in the back of the head with a bottle and I wake up 30 years later, founded 14 companies, most of them failures, few good ones. I'm known for probably Equinix and Digg and Revision3 and a couple other things in there and ended up leaving that world and becoming a pinball professional. And that is why I'm sitting in my office, filled with pinball machines right now.
Avi Freedman: That's pinball design creation, repair, playing?
Jay Adelson: Well, you know there's an internet twist to this, right? Come on, there has to be an internet twist.
Avi Freedman: Yes, of course.
Jay Adelson: I'm really into retro stuff and so I just thought it would be really fun to be able to connect to old arcade games, mostly pinball but basically anything, and push the scores and achievements like you do on Xbox Live. And didn't quite anticipate just how challenging a technical task that would be, and that was seven years ago, which is crazy to think. But I still operate a venture fund, the one that we invested and Kentik with and that, but I pretty much stepped out of that world in 2017 to do this always, which is, I know-
Avi Freedman: So you're IOT, at the end.
Jay Adelson: I'm IOT, I know we were an internet infrastructure and IOT investor. And I remember learning that IOT is a terrible business and then I went and founded an IOT company.
Avi Freedman: Well, it seems to be doing okay for Samsara and some others that started about the time of Kentik and I don't come from the skater land and all that, but turns out there seems to be a good business and connecting fleets and pumps and stuff like that. I just happen to come from the other side of it. Yeah, Gale, I don't have it all set up. Gale does not prefer, she doesn't love my hobby of collecting vintage 8- bit computers back from when you could understand everything about the hardware and software. If I were actually at home, instead of having a background of at home, I'd show you the Commodore 64 over there, or the Kaypro. Go and get a Kaypro, or an Acorn if I was talking to someone from the UK, but she calls it AMoS, Avi's Museum of Stuff.
Jay Adelson: That's awesome. By the way, if you are interested in taking over my collection of Apple IIs, I have every one, every model, 32 of them.
Avi Freedman: I am very interested.
Jay Adelson: Oh really?
Avi Freedman: I have a warehouse, thanks to a friend of mine, not warehouse, sorry, it's a storage unit. It's effectively on the Ashburn Campus. You know the post office that Equinix kicked out, that's on the original Equinix campus there? Or not the original, but the...
Jay Adelson: I remember it had a catwalk for people with guns.
Avi Freedman: Yeah. Yeah, that all got tore down and that's, I don't know if it was DC 11 or something. It's the Guardian Self Storage there, and so a friend of mine gave me one of every, not a Spark, not a Sun- 1 or, no there is a Sun- 2, and then every Sun- 3 and 4 and Spark up to a 450, I don't have a E10k. Actually, no, no. There is a 4500 which used to be the cheapest way to get a lot of Ram use, but the power bill was not fun.
Jay Adelson: Now are they operational?
Avi Freedman: In theory. I haven't fired them up, but I have a bit image of a [413-U1 00:04:51] and 414 with my custom kernel for SYN flood for SYN cookies that we had to do back when Pennex was a customer who was being attacked. You know, the fun old days, but you worked at Deck.
Jay Adelson: I did. I worked in the network systems laboratory, which sounds like I might know something about networking. Believe me, I don't. The thing about the NSL was it was like, it was like the Mos Eisley-
Avi Freedman: Mos Eisley. We call that Usenet now, a inaudible hive of bugs and flamers. Or Digg or crosstalk.
Jay Adelson: I just couldn't believe I was there. I felt like somebody was going to figure me out and that I was imposter syndrome in the middle of this lab next to Paul Vixie and Steven Stewart and, of course, Al Avery and all of these guys who were really doing cool stuff. And I was so excited and they could tell, they could tell, but I was hopelessly addicted to this idea of internet scale. And I perhaps became a little too obsessed with it, to my wife, Brenda, she still makes a joke that if my phone rings in the middle of the night, she always says the same thing, which is," Tell them to reboot the port master."
Avi Freedman: The port monster. The port monster, yeah.
Jay Adelson: Livingston PortMaster was, most early ISPs use something similar for all-
Avi Freedman: They use that or if I were, again, if I had an updated picture of my actual background at home, I have a Micro Annex Terminal Server. I use the inaudible Wellfleet, I guess, bought them.
Jay Adelson: Yeah, there was a lot of late nights and I was an operations manager, to be clear. Some people have said," Oh, Jay founded PAX." That's not really true. It was operational as an exchange point for easily a year before-
Avi Freedman: Well, I remember, well, we're getting ahead of ourselves because we have to talk about, but yeah, I remember when I first, well, okay, so who did start it? Was it for AltaVista connectivity?
Jay Adelson: It's an interesting, I would love for Paul or Steven to one day get into the really boring details of this because I'll tell you what I know. Was that when I was running network engineering and operations at Netcom, which I again have no idea how that happened but I was operating there, the Deck network systems lab is in Palo Alto and it was in the basement of what is currently now the PAX. And Paul, who was part of the lab working for Brian Reed, who was the director of the lab, had somehow convinced Bill Young and a couple other folks involved with BARNet and AlterNet who had been exchanging traffic in the Stanford computer lab, I believe.
Avi Freedman: Because I remember Paula had showed me the, I think the Ames side. No, no. He showed me the CIX. He showed me the CIX, not the Ames side of MAE- West. He showed me the Commercial Internet eXchange, which was in some office park because it was the secrets of peering and if you go to the CIX, do you get-
Jay Adelson: That's right. No, that's right. And I think that Paul is a visionary in a lot of ways. And I think that one of the things that he recognized around this time when he was helping to, I think, administrate that interconnect, is he figured out that there was a huge issue on where a place was and who was operating it. Because in the case of Stanford, I think they had a squirrel literally take out the data center and take out half of the Western Seaboard of the internet for some period of time. It's a story that everyone tells in some form.
Avi Freedman: I've never heard it, but I'll ask-
Jay Adelson: Everyone's got a squirrel was in the power box story about their data center going down.
Avi Freedman: This is before I hit the EPO button to ex-
Jay Adelson: That would make a lot of sense. You lean against something and it pops off. Stanford set up had never been set up for reliability. Neither was the basement of the network systems lab, but somehow Paul convinced AlterNet and BARNet to move into that basement and connect-
Avi Freedman: For people to keep track. AlterNet was commercial, BARNet was the bay area, regional network. They were all semi- commercial, the regional networks, but it was the government NSF network.
Jay Adelson: This is before the official, or no, there's just after the contract was handed, I think, to crosstalk various carriers, yeah. So he moves it in there and I think he, Deck was also a hardware manufacturer and manufactured network switches and they made a fitty switch or a fiber interconnect. He had them, I think, originally connecting on a Cisco, serial number one of some old Cisco, and then, in parallel, he started the Commercial Internet eXchange with Barb Dooley. And that initially, I think, was a lobbying organization designed to speak on behalf.
Avi Freedman: crosstalk policy side of it and peering.
Jay Adelson: Exactly, because I don't think that at that time, it was anarchy. It was the Wild West. And there wasn't a lot of regulation or oversight or thought given to how this stuff rolled out. And I think that Paul felt very strongly that it should be community- based managed exchange point. And meanwhile though, he was doing this at Digital and so Digital decided to productize it as what business unit inside of the network systems laboratory. And when that happened, that's when Al Avery took over as the general manager and he hired me to basically run operations. By the way, I think that Paul was right on a lot of fronts in that there is challenges when you're commercially driven. Because you're you're on behest of the shareholders and that's an issue. The flip side though, is community- based organizations, just ask all the European exchange point operators, to move anything one port over is a year of debate.
Avi Freedman: I remember, because contemporaneously, Doug Humphrey had showed me a little bit before, a year before, the 1919 Gallows basement MAE- East where unit and PSI routers were on a power strip on a two- prong extension cord plugged into the wall. And I remember visiting PAX and of course Alta Vista was there and Alta Vista was the cool search engine at the time to use the Deck hardware, which I was a fan of. I'm not a big VMS fan, other than architecture, but more of a Unix guy, but I remember thinking," Wow, this is real," because going to CIX, going to MAE- East, going to MAE- West and with no offense to our MFS sisters and brothers, it was a little bit of a hack job. It was like my colo in Philly where I convinced MFS to give me a rack for every T1 that I sold. And suddenly I owned this whole data center that was all DC power and had only AC equipment. And they're like," What are all these computer things that you're putting in our building?" But it was a room in an office building full of lawyers. And I remember going to PAX, I was like," Ooh, they only turn on," because the guard thing was upstairs, you had to go downstairs. They only turn on the lights for the people that are there, so you can't really snoop quite as much, there was a set of rules you had to follow. And I was like," Wow, this is getting real."
Jay Adelson: Yeah, I have to say, I remember that a lot of that feeling was planned and thought through by Brian Reed. Brian, brilliant network engineer who ran the system laboratory there, hired an art designer to come up with colors and patterns and fun lighting to try and make a very small space, evoke some sense of safety and reliability and importance. And the thought there was, you're building an empty exchange point and hoping people will come. And I think the idea was is that if you build it and you enforce these rules and you're think about it in the tradition of maybe the finance and the banking industry.
Avi Freedman: But not as annoying as doing colo in a CO, in a phone company, which was very annoying at the time.
Jay Adelson: There was a certain amount of lipstick on the pig, so to speak. A certain amount of it was dressing in front of what was a very roughly built data center that was incredibly effective given the small, it was 5, 000 square feet, so, people who are building their first data center at 250,000 square feet-
Avi Freedman: And how many gigawatt was it?
Jay Adelson: Oh man, I-
Avi Freedman: 200 crosstalk?
Jay Adelson: Less than that. I think we might have said that and I remember negotiating with crosstalk.
Avi Freedman: crosstalk for software people out there, you can now have the equivalent of 20 houses of power in one cabinet, with or without immersive cooling. But at the time it was not that much more than a few light bulbs per cabinet worth of heat and power that one was dispersing. Yeah, that was the internet, that was the crosstalk.
Jay Adelson: By the way, if you want training on how not to build a data center, one of the things you could do is take someone familiar with building out points of presence, like I was, and put them in this spaghetti setup and tell them that the entire internet depended on its functionality. And we had some great assistance. There was a contracting firm named MCM and this guy named Jim Brush who looked like the prospector from the Toy Story movies, exactly the same, and he would walk in and basically explain how, in the days that they invented electricity, this was how they solved this problem for this or that. And I learned all the lingo and I learned, basically, how to talk to electrical engineers, not necessarily how to be one, but certainly how to understand that. And I think that Al and I began, that was when we started fantasizing over beers and cigars, if money was no object, what would it look like? And how could we grow this thing? Because we were also friends with all of the network operators and they would complain about Tysons Corner parking garage and-
Avi Freedman: 8, 100 Boone, 1919 Gallows...
Jay Adelson: And there was one exchange point, I believe somebody actually perished due to an electrical accident, I don't know if you knew about that. We won't name any names, but some of the folks watching your video will probably remember that. I think it was right around, had to have been'98 or'99, there was an incident where a electrical technician and a carrier exchange point touched the wrong thing. It was not good.
Avi Freedman: I went into the inaudible and I remember, that was a little bit too real. That was closer to the CO level of how things thought, but it was pretty big for what was there.
Jay Adelson: It's arguably true that carriers knew more about compacting a lot of network in a small space better than anybody else. I just think that it was never a question of someone losing a lot of money, well, at least from a commercial internet standpoint, until way later. And once you got to that point where content, eyeballs, and the path between them leveled out in importance in the ecosystem, then you had to change the way you designed all that stuff but that was really fun.
Avi Freedman: I remember being at a NANOG, unlike some people, I can't remember," Yes, NANOG 12 was in this city," and I think it was you who had announced, maybe Bill was up there, had announced Equinix. And I'm like" Equinox? They make serial port cards." But you mentioned rules and so we do have to tell the story about the cutting off of the non Telstra transit of Australia. I definitely try to figure out what are all the ways to interpret the rules to create good for humanity and myself. But we knew one of the rules of PAX was, thou shall not run Zanon cross connects. Now, a lot of people in data centers now will use passive optics to pay for a cross connect and make eight cross connects in it. Or inaudible version one Equinix got upset with me and said we should have shut down. Because it was clear that we were trying to do multi data center, but when people figured out it was cheaper to do exchange in the building over it, this is before OpenIX started to drive the prices down. My friend and a person who's less, he is aware of the rules, but feels less constrained by them, Andrew Ku, I run cross connect between our, we each had a cabinet, NetAccess had a cabinet, not cabinet, sorry, relay rack.
Jay Adelson: That's right.
Avi Freedman: Not even cage or cabinet, I'm thinking modern. And I was giving him transit over, I think actually I was tunneling to make it look like I had at least line to that location. And then he called me and he said," Avi my transit's down," and I think I was running AGS Plus at the time. And yeah, I remember getting a call from you, hugely apologetic. It's like," Avi, I'm really torn here because you guys violated the policy, but you should never interrupt operating networks. We should talk about these things."
Jay Adelson: Oh, no, wait, here's the untold part of this story. All right, to be clear, you end up, by terminating this cross connect, you definitely turn off a good portion of the continent of Australia. Probably not a great decision. We had an operations manager at the time who worked for me, a guy named John Pedro, and almost anybody who was in the internet space back then knew who John was. A very sweet guy who worked at the PAX and was responsible for managing the day- to- day task work of the technicians who were in the cross connects. And he discovered this cross connect which was surreptitiously run between two points and told me about it and asked me what I thought he should do. And I said," Well, I think you should email everyone and let them know." And apparently this went on for several months where John was emailing someone. By the way, I don't know if it was Stewart or Andrew, I don't know who it was he was emailing, if he was doing it right. Probably should have been me. We had no escalation process for this because remember-
Avi Freedman: crosstalk circuit database.
Jay Adelson: But the thing is that I think it, this might be revisionist memory, but the way I remember it was we discovered it and we said," Hey, if you pay us whatever the cross connect price is per month, we're good. Just send us the money and we'll be good." I have no idea how billing worked at Digital, I can't remember. And remember, we had just invented these terms of CNI and GNI and INI and PNI and we were inventing one for, within a cage. And I guess some time passed and, I don't know, it makes me remember when it happened, because it was the middle of the night. I got a call, actually, I got a page.
Avi Freedman: And you said,"Reboot the port master."
Jay Adelson: Basically, this is down and this is a big deal because it affected a lot of other folks.
Avi Freedman: At the time, this was supporting satellite based bandwidth augmentation, so people would buy their E1s in Australia and they'd be full on the inbound side, but they'd have more outbound. We were selling 8 Meg circuits that were basically one- way frame relay for people to get excess capacity for, and they could make SSH and IRC go over terrestrial and use net and web cache and go over satellites. A lot of circuits were, and for latency, it was satellite backed but this was supporting, this was the transit for that satellite thing. So there were a lot of small networks, it wasn't affecting Telstra, but there were a lot of small networks.
Jay Adelson: And the other problem was how we did it, because back then our cross connects were largely, I think, multimode fiber, if I recall, and you cut it with a pair of scissors.
Avi Freedman: Yeah. You didn't just pull it out, yeah.
Jay Adelson: I don't know, he would get angry sometimes, and this must have really made, you have to understand he was a very sweet guy, very peaceful, he would calm you down, very well organized, not the guy that would do something vindictively in any way, shape, or form. And so, for him to cut it with a pair of scissors, which is the part I remember, implies a long history of battle.
Avi Freedman: With prejudice, yes.
Jay Adelson: Yeah. PAX had a cable plant, it had a paper system for knowing what was plugged into what, it had all of these inherited characteristics from the Paul Vixie and BARNet days.
Avi Freedman: And also Deck is definitely real engineering, scientific computing and business computing.
Jay Adelson: The ultimate thing about that whole operation is that it never went away. The PAX would later become acquired by-
Avi Freedman: inaudible data?
Jay Adelson: No, before that it was crosstalk.
Avi Freedman: AboveNet. Yeah. Sorry. I was there. I was running engineering at the time.
Jay Adelson: It was, AboveNet was acquired then though-
Avi Freedman: By MFN.
Jay Adelson: ...by MFN and then MFN acquired, I think, or I don't know which order, but it had the PAX and then later Switch and Data acquired them.
Avi Freedman: Right. Or acquired PAX from MFN, maybe.
Jay Adelson: Yeah. That sounds right.
Avi Freedman: We should look it up.
Jay Adelson: And then ironically, I don't know, a decade later, maybe more-
Avi Freedman: It's like foreign language but with networks.
Jay Adelson: Exactly, everything came round and Equinix ended up owning the PAX and, I hadn't returned to the building the entire time until Equinix acquired PAX, I don't know, 15 years later or something like that. And despite the fact that the original deal, when Al and I left PAX, the deal we struck with Bill Strecker, who was the CTO of Digital at the time, because Digital was getting acquired by Compaq, was that if they were going to sell that we would get a right of first refusal. And let's just say that, and we agreed on a price. I think it was$ 30 million or something like that.
Avi Freedman: That's advantageous, to fix the price.
Jay Adelson: Yeah. And I think it sold to AboveNet for two or three times that, so we were out of the market, but came around.
Avi Freedman: Thank you for Equinix. Equinix is a customer, we have some stuff in DRT, but for the most part I'm an old server hugger. As a SaaS company, we make our own cloud in Equinix. That's where I have my Usenet where I bootstrapped Cloudhelix that Kentik on my Usenet servers. But what was it like during hyper growth?
Jay Adelson: Yeah. I would love to tell you that it was... It's funny because now looking back across my career, I think really only twice or three times I've been operating a business or founding, co- operating a business that had a hyper growth phase at all. And the Equinix one was brutal. It was uncomfortable. We raised just under$ 1 billion.
Avi Freedman: That used to be a lot of money before crypto, before content.
Jay Adelson: Well, remember this is pre NASDAQ crash. We raised$ 1 billion well, not$ 1 billion, but$ 800 million across three years or four years to the credit of Peter Van Camp, who we brought in to run the business and Keith Taylor and Phil Koen, who was the CFO at the time. We did some really crazy financial deals and started an extremely aggressive build out with Bechtel Corporation crosstalk.
Avi Freedman: You paid someone to build the data centers, yeah.
Jay Adelson: Yeah, we came up with a lot of original concepts and we designed them, had our own internal design team and we even hired John Pedro. Yes, he worked for Equinix after that.
Avi Freedman: I remember that when things got tough, they stopped putting game rooms in the inaudible.
Jay Adelson: Yeah, I remember seeing you about the time that I knew that I couldn't stay because I was so stressed out. First of all, you had the NASDAQ crash, and if having your entire net worth and the company's net worth wiped away in the course of six months while your company's locked up from an IPO, that'll do it to anybody. But then throw on top of that the entire community of customers were going out of business.
Avi Freedman: Yep. At Akamai we discovered that, we were watching DSO, day sales outstanding, and it wasn't that our customers had stopped paying, which is what we thought. I remember when Tim Weller who was the CFO came in and he realized that our customers didn't exist anymore. Half our customers just didn't exist.
Jay Adelson: It's crazy the amount of equipment that would go abandoned inside the Equinix data centers. And so we had a bunch of construction projects around the world that were one quarter of the way done. And we realized that the only way to survive as a company was to pull out of those 20- year lease agreements and stop construction and lay off three quarters of our team. And this is all probably two or three weeks after 9/ 11. So, this is hyper growth during that initial phase where I learned what a sales marketing kickoff was and that and watching the contracts come in and...
Avi Freedman: It was like Cisco in the'90s, when the fax machine was out of paper and people would line up in order to shove checks in your pocket because...
Jay Adelson: It was a great spoiling of engineers around the world, because that was when we decided it was okay to spend a quarter million dollars on a box at the core of your network. And I think before that point, ISPs would do that to a certain extent, the BFRs and whatever. But there was this limitless bank account that existed on the part of our customers and, sitting in the middle, you got to play with all this really fun stuff. But then when it ended and I had to basically cut the entire R and D team, and we had some great guys. Ted Hardy and Sean Donnellan and Ian Cooper was there. And we had Dwayne Wessels working on caching back then. And we had this crazy, great, fun team that all had to go at the same time. And I remember seeing you, because I think it was around the time Richard Clark was running the cyber security... What was he running?
Avi Freedman: He was the cyber security advisor for multiple presidents and he got us involved when I was at Akamai with the National Communication System, NCS, before DHS was formed.
Jay Adelson: I think I went to the White House for something, because everything was stressful. We were under attack, we were at war, everything bad was happening. And I was on the road all the time. And I remember I had pulled an all- nighter preparing for something, I don't remember what government thing, and I'm waiting outside some office and I hear your laugh from behind the door. And you came out all chummy with, I think it was Richard Clark. And I'm like," Wow, Akamai got here first. I can't win. There's no way I can win this. He's already here. He already is telling people that peering is overrated."
Avi Freedman: I've always been a big fan of peering.
Jay Adelson: I know, but we were both, you were correctly advising that we needed to revise our infrastructure in the sense-
Avi Freedman: Yes, that was the thing that I did, which is ironic because Job Snijders just asked us at Kentik if we'll support S- BGP, I think it's now called Secure BGP. I'm like," Oh my God, the nineties wants it's protocols back." But we actually have fast computers so we can do path validation. For Richard Clark, I wrote something which is like," We should filter the source addresses of our packets and we should secure the routing and we should use the government purchasing power, which lasted like two minutes until the lobbyists, heard about it." And I was like," Everyone's worried about the authoritative name service. That's not the problem it's on any casted, recursive name servers, and we should support the infrastructure," and he had a good heart for that.
Jay Adelson: Yeah. And I think, thanks to you, called on a bunch of others and Vint Cerf and Kathy Aaronson and I just remember sitting around a table being fully intimidated, and realizing that I used up my resilience, you know what I mean?
Avi Freedman: Yeah.
Jay Adelson: There was a certain sense of responsibility, and I think you probably felt the same thing, where we had had fun building this thing and had been this really fun rise and this was pretty serious stuff. And my I had three little kids that I wasn't seeing ever because I was traveling all the time and I was in DC half the time, because remember I lived in San Francisco back then. And then right around that point, I moved to rural New York to try and escape. That's pretty much when, I think I went to my last RIPE and my last NANOG for a decade was right around that time. And I was planning on getting a teaching certificate and becoming a public school teacher and just disappearing entirely.
Avi Freedman: Interesting.
Jay Adelson: Yeah, that didn't work out as planned. But I remember at that moment, I'll never forget hearing your laugh and thinking," Oh, it's got to be Avi on the other side of that door." And it was.
Avi Freedman: It's funny because my brother, Noam, has a distinctive laugh also, you can tell where wherever he is on a floor. He just moved from running network, I think, at Akamai to Cloud, but they call it Compute because they partner with cloud providers, but after buying inaudible. But it was something I spent a long time at Akamai, not getting them to start up a cloud. I think there's a couple things that I've definitely felt imposter syndrome and with some of the same people. I was not a fan of the MAPS RBL, but my interview for AboveNet was, for various reasons, writing a BGP daemon to do optimize routing and some other stuff that didn't exist. And then Dave Rand gave it to Paul Vixie who, the day after he got it, sent me five, one- line corrections to my 3000 line BGP daemon to make it better, to fix a bug and make it better. And I was like," Wow, I'm not worthy." Then someone was like" Avi, who writes a BGP daemon for an interview?"
Jay Adelson: And by the way, can we just stop right there for a second? You wrote what?
Avi Freedman: Lots of people have, it's a time honored way to take the internet down was to write your own BGP daemon.
Jay Adelson: Well, that's true-
Avi Freedman: crosstalk at that next level and so it's all in context. I was going to say like the Forrest Gump, are you saying you're the Forrest Gump of the internet, you just happen to be there when all these things happened around you?
Jay Adelson: It's a little bit like that, I want to tell my kids-
Avi Freedman: crosstalk Equinix, in a lot of what is still there today, and great culture, is because of you and Bill and other people that decided not, for various reasons, to be there for the whole stay, but made their mark. On the other hand, what I hear you saying, not to put words in your mouth, is you do have to pay attention to what's in your head and not... Sometimes if you need a change, you need a change.
Jay Adelson: I think so. And I think to some extent I recognized how big... Now Equinix is a Fortune 500 company, it's a very different universe. It's, I think, known by its employees to be a wonderful place to work and they treat their people extremely well. They've come back from a position of too much strength to one of a little more humility, despite their size. And I think have done a lot of soul searching as a business over the last 15, 20 years and really, it's a different place and a great place. But my strength was in the people part and the growth, early stage of a business where you have to accept a certain measure, more risk and more product development trials and failures and a certain amount of mentoring. Part of what I loved to do, and I was so grateful that Al Avery gave me the opportunity to do this at PAX and then later as a co- founder, I would find people who were phenomenally good at something, way better than I was, and I would say," Hey, I'll take care of you and make you VP of something, but I have one request. And that is that you allow me to ask you stupid questions in front of your team so that I can learn." And I love that. And then there'd become a point where that was actually inter-
Avi Freedman: I prefer to say foundational questions, not stupid questions.
Jay Adelson: Well, I always presented it on my sleeve. A classic example is I manage coders all the time now. A lot of what I do is small and so I end up contracting coders or hiring them directly. And I learned to code in college, I have no miles whatsoever. I'll be like," Hey, I realize in the public Slack channel, you don't like me asking you stupid questions about syntax, but do you mind if I ask," because I love to do code review and I love to learn about code because I have a nerdy curiosity on how it all works or frameworks. I slowly teach myself to code in the background in private. But if you want to work with me, sorry, you have to be willing to deal with-
Avi Freedman: I found technologists to be better than doctors. I've offered to pay a lot of doctors to answer my questions and they're, but that's to say the doctors are really busy. crosstalk and multitask and whatever.
Jay Adelson: So are developers and engineers.
Avi Freedman: But I've been more frustrated. And my father who is a doctor is like,"I don't know. I just do what they tell me." It's like, well, I'm sorry, that's not me. I need to understand. If I don't understand, I can't. That's always been my problem. I was upset I missed RSA this week because NANOG conflicted. And that's part of what I do at a trade show is go around and try to figure out, I see what value is delivered, I understand the marketing, but if I don't know what it actually does, I can't reason about it and the body is like the internet. It's amazing that it works at all, it's not amazing that it breaks sometimes. It's amazing that it works at all.
Jay Adelson: And all of the things that go wrong with the body also go wrong with the internet.
Avi Freedman: That's also true. And it's a complex distributed system with inaudible dependencies. And the symptom that you're seeing is maybe not the organ or the thing that is the actual problem underneath. But I remember thinking, when you had left and I don't know if it was a year or whatever and then you popped up as CO at Digg, and I remember someone I went to Hebrew school with had started CDnow with his brother and sold it. And I remember sending a note to my ISP saying," I guess the real money's in content, not infrastructure."
Jay Adelson: Is it though, but is it though?
Avi Freedman: I don't know, the jury's out, we'll have to wait and tally up.
Jay Adelson: But that was so much fun. Avi, I cannot tell you how luxurious it is to be a consumer of internet services versus a manager of them. I gave one talk at NANOG as the CEO of Digg where the whole point of it was, wow, it looks different from over here. It looks so different. And I got to play with things that I never thought I would get to play with because I was finally building a presence. And We had our versions of, I don't know what you call it, product innovation or web 2. 0 stuff that we were doing that was interesting. I was really obsessed with egalitarian democratization. After all, neutrality, I was super big on this whole idea that conflicts of interest ruin ecosystems and so-
Avi Freedman: Things are never that simple. Net neutrality, it's like, well, you need to have equal access that maybe the government paid for you to build your network, so it's not really yours. On the other hand, if you can't filter DDoS, then the internet doesn't work. And what is free speech? Was a little different in the nineties before people were doxxing and sending killers to people's houses. As much as you want to be a purist, the world has gotten definitely a lot more gray scale.
Jay Adelson: Well, this is right, you're a hundred percent correct, it's not simple. And the complexity of taking a vision, whether it be an architecture for a network or even a business model, any of that complexity that you introduce to it, your vision has to compromise. It is a nature of how it works. There's a purest view and there's components of it that compromise, and it's that act of introducing complexity to your simple vision that is the challenge, I think, the thing that sucks the resilience out of you. If you can manage to process that, there was this one time at Digg when, I don't know if you would remember this, but remember HD DVD?
Avi Freedman: Yeah.
Jay Adelson: Blue- ray was fighting for dominance over HD, DVD and some users, just for your viewers, Digg was a website where you would submit a story in the form of, you could put a thumbnail on it, a title that you would write, a link, and a description, if you wanted to put a description. It would pull metadata if it existed, remember this is 2008 or 2007, there's only so much metadata. And then you'd submit this to a pooling area called upcoming and people would see that and vote on it by digging and after a certain algorithm filter would pass, it would go to the front page of the website. And then, because of the attention the front page got, it would be like a fire hose of traffic to whatever got there. so there was this huge competition.
Avi Freedman: There was this thing called the sloshed out effect before that.
Jay Adelson: That's right.
Avi Freedman: And then it became Digg.
Jay Adelson: Yeah, that's exactly right. What happened was, Digg attracted a lot of network people, nerds, like me, hackers, people who loved Kevin Rose and his hacker TV show and a whole lot of those kinds of folks. And so it had a very technical bent to it, although it was also, there'd be links that were completely sensational too. But one day somebody posted the encryption key for HD DVD. This is after DMCA take- down notices and chilling effects was out there and the EF-
Avi Freedman: I was running a Usenet provider at the time, so crosstalk.
Jay Adelson: So you know that there was a lot of complexity and debate over who owned the content and whether or not it should be pulled down, who can...
Avi Freedman: Someone was just taken off of Google last week, I forgot who, by DMCA complaint and it was back in a day.
Jay Adelson: We were new to this and we were popular and someone posted it and it got so many votes, it went to the homepage. Then we get the take- down notice and I meet with my buddy, Kevin, about what to do. And we were like,"Oh, well, I guess we'll just take it down." It was already lower and lower and lower, and then was off the homepage anyway, so we took it down. And the users didn't like that, so they reposted it again and this process started to escalate. Finally they were posting it once every two seconds, faster than we could manually take it down. we wrote some code to take it down.
Avi Freedman: The filter for the-
Jay Adelson: To filter it out. And then they started recording themselves on video, playing guitar, singing the code.
Avi Freedman: So you couldn't filter it programmatically.
Jay Adelson: It just started escalating and it got to the point where they took down our website, effectively a DDoS attack. And then we posted on our blog, the title of the blog was the code from HD DVD, and we're like," You're right, it doesn't belong to us. It belongs to you." And that was a big awareness. When I talk about the resilience being, you think that a simple idea like neutrality can be compromised, and to some extent it can be, but when you introduce some of these things, they also become religion.
Avi Freedman: Emergent, yes, that's the word I was looking for. And there's a science fiction series, Enders Game, I don't know if you've read the book, there's a movie, but in the book, these two really bright kids become locked in inaudible. And there's this idea of this worldwide thing but even there, there's a contrast because there's Atheri that was not like Usenet at the time, or Digg or anything or Reddit or anything of rational discourse, like Socrates and arguing with people. We've never seen that. On the other hand, the idea of people smart enough to actually influence the masses and drive emergent behavior, we're seeing that with influencers.
Jay Adelson: That's right.
Avi Freedman: As a company or as not, you have to have the sense of what is, and sometimes yeah, it can get away from you inaudible the users.
Jay Adelson: And it's an evolution of the anarchy or... The internet had the advantage of being an unregulated and anarchistic entity where you would sometimes get defacto standardization way ahead of IETF, way ahead of John and RFC saying there was some official way of doing something because we just didn't have time to wait. It was growing that fast. It's funny because on the sidelines while I'm not operating network companies, I am on the board of Megaport and of course I'm a network geek and I always will be, so I'm watching with amusement, ethernet.
Avi Freedman: These questions we argued about. What happens if there's 200 milliseconds between peers, do bad things happen? It was a question we had in the nineties and now we know.
Jay Adelson: It is a computer science problem, and you introduce real life and randomness and anarchy into these computer science models and things sometimes happen that you don't anticipate.
Avi Freedman: Well, I think that-
Jay Adelson: Cool.
Avi Freedman: It is. And also it's something that, I'm going to say good startups, in my opinion, learn to embrace at a micro level is," Hey, we tried this and it didn't work." Now it can be frustrating when you try it at a macro level and it doesn't work, but some great companies that have created a lot of value, I'm not just saying economically but for users, came from things that didn't work. Slack, I don't really understand, it just looks like chat that doesn't crash, but clearly people like it and that came from a gaming company. Or often it was a whole company that didn't work, and then that taught the lessons. The biggest one that I struggle with is focus because you have customers that want stuff, you want to build it, but then that becomes difficult. Or really what you see is, when you're building a company, you try to engineer what you want that to be. And most companies just become emergently copies of the people, the first employees and how they operate. And that's been a really interesting time for people growing through COVID where stuff that was undocumented and emergent and understood become... And some companies designed for that, very few, before COVID were like," We are remote first, we don't have hybrid meetings. We're very thoughtful about this." And then actually letting people talk over the internet, thank God, works really well but the human dynamics of it obviously is a little tricky.
Jay Adelson: Oh, I agree a hundred percent, I launched my most recent business in September of 2020.
Avi Freedman: What does that do?
Jay Adelson: It's Scorbit, which is the crosstalk. I actually planned on launching it in March of 2020. We have some fun investors, like Matthew Prince and we have a bunch of network guys who are involved in this tiny, by the way, it's a small business, it's not a venture backed business.
Avi Freedman: You say small business, not by the way, I don't view lifestyle business as a negative thing. I know in Silicon Valley it's viewed as a negative word, but I don't-
Jay Adelson: Yeah, I'm definitely building this, let me put it to you this way, in certain markets and certain products, if you don't have a big pool of capital to very quickly land grab and establish a first mover advantage, you can't succeed. And I'm not saying that it's always good to do that, but a lot of the times it makes a lot of sense. I decided to go after something retro and build a network infrastructure for it, largely because I knew that no one else was crazy enough to try it. It would require years of R and D to make work and then once it worked, my audience was this fascinatingly diverse international crowd.
Avi Freedman: Of largely high net worth, but not all.
Jay Adelson: It's true. It's true because pinball tends to be... Of course the inaudible started growing and all these arcades and stuff-
Avi Freedman: Did the new employees have to read Armada and Ready Player One to get the'80s context?
Jay Adelson: Oh, if they haven't read it already, they're not allowed in the door. But I'll tell you, it is interesting if you look at the cast of characters, if we had the Tops playing cards for NANOG or whatever, and you just dealt out a random 10 of from your deck of cards, I guarantee most of those people would probably either own or regularly play a pinball machine.
Avi Freedman: crosstalk. I was more a Dig Dug to relax and inaudible. And I preferred writing stuff than to play. And I think Zaxxon on the, was this cool 3- D game, but I didn't get into pinball for whatever reason.
Jay Adelson: And I didn't get into it as much either. I don't know about you, but the first time a networked game became available, the first one I remember that I played was there was a game called Specter 7 on the Mac, and it was battle zone, but multiplayer battle zone and it was 1994 AppleTalk.
Avi Freedman: You have to remember, I ran Metaverses crosstalk multi- user dungeon games, but that was how I learned IPC and distributed processing and all that. It was an 80 by 24 metaverse.
Jay Adelson: I don't know how many of us fit this model, but from a very young age, the thing that kept me dialing that modem over and over again-
Avi Freedman: It was connection.
Jay Adelson: It was, yeah. The connectivity with other people was just, I was-
Avi Freedman: A lot of ham, a lot of people that were hams that went into started ISCs and stuff like that. Although I do have fondness for," Gauntlet needs more quarters badly." It wasn't network, but you could play with friends so that was-
Jay Adelson: But there is an interesting question which is, if you went back in time to 1984 or'82, and you walked into an arcade and Zaxxon was connected to a Zaxxon somewhere else.
Avi Freedman: Yeah, that would've been the shit. crosstalk someone else's fighter. Robotron was too intense to have any third access of other people participating, but yeah, that's true. Or even Battlezone, you're right.
Jay Adelson: I just feel like there is a nicely contained, limited network problem that I tried to solve. So, on one hand we had to have the latencies of an MMO. We're not doing really head- to- head because we don't really have to, it's a turn- based world, but the visualizations and the response time had to have MMO latencies. And yet we had the volume that was so low that it's, from a financial standpoint, you can't...
Avi Freedman: crosstalk need to buy 800, 000 servers and distribute them around the world.
Jay Adelson: So we're doing it serverless, the whole thing, the whole thing. It is so much fun to do it this way, it's inefficient in some ways, but I got to say it is, thanks to folks like you who did all the hard SaaS work over the last 10 years, we don't have to do anything anymore. I did go to Equinix just a few days ago because our Force10 switch in our one rack that me and a bunch of friends still operate, SV1, decided to kick the bucket.
Avi Freedman: I can hook you up with a used Arista. That'll probably be better than running your Force10.
Jay Adelson: No, we should have shut this off long ago, but we still host net, remember Tasty Lime?
Avi Freedman: Yeah, yeah.
Jay Adelson: It's still operational. And I had to bring Jeff, and it was Jeff Rizzo, and it was so funny because we were walking to the data center and we hadn't been to network campus.
Avi Freedman: crosstalk is basically in LA from, it's so far south in San Jose that I'm-
Jay Adelson: It might as well be.
Avi Freedman: Yes.
Jay Adelson: But we went from a place where it was an apple orchard the last time we were there and now it's three story buildings surrounding us. It was really weird. But yeah, I do occasionally go to a data center, but again, hopefully it's all turnkey now, I could just press a button.
Avi Freedman: That's right, it's all magic. I can't say we only run our own infrastructure because of the economics, because it's a nerding thing that I like. But the reason that it works, because every time half the people that we hire are like," We're doing what we're making a cloud, aren't we a SaaS company, why are we making a cloud?" It's like, look at all these other SaaS companies that make cloud, because they're our customers and they use cloud, like we do, but not for always- on stuff because that's really expensive. Idic, Infinite Diversity in Infinite Combinations. The problem is that is enterprise cloud architecture. But that's a story for another time. If you could go back and tell young Jay, maybe before Netcom or whatever age, a couple things to think about, do, lessons to put into context, do you have any idea what they would be?
Jay Adelson: Well, the first thing I would say to young Jay is, run.
Avi Freedman: Go teach.
Jay Adelson: You do end up doing what you love in the long run in whatever job it is that you end up paying the bills with. If you love to teach, you'll end up being a teacher, if you love to create new things. But if I could go back and talk to Jay, I would probably tell him that you don't have to be so angry. I spent the-
Avi Freedman: DO you mean take it so personally or be so angry?
Jay Adelson: It's really both because the problem is, and I think a lot of investor types try and paint this picture that you need to be somewhat detached in order to be successful. And what really happens is the most successful founders are the ones that literally can't do that. Their people that they work with are their family. They take every customer negative thing totally personally. And I had not developed the ability, and I still have not fully, to really roll with that. As a young entrepreneur, I was not prepared for what that would do to my life and my family and everything else. And I'm proud that I survived it, but if I could give my former self some lessons, it would be around understanding that balance, even if I couldn't control it, just understanding it. Because remember, you've got all these mentors and investors who are telling you you shouldn't feel a certain way.
Avi Freedman: I tried that in my, so Gail and I would be married 30 years, but it's our 31st anniversary of meeting at the Chicago World Science Fiction Convention this year. And I think I tried once or twice saying," Okay, well, we've understood what the problem we were disagreeing about, so don't be angry," and I discovered that doesn't work. It's like, don't be ticklish.
Jay Adelson: Yeah, I think that taking it personal is okay. I think my lesson would be taking it personal is okay, but it's what you do with it and how you channel it and your willingness to express yourself. I think also my early work came out of a lot of insecurity and controlling, trying to control my world and coming to peace with what you can't control is part of that, I'm 51 now.
Avi Freedman: Heavy beat.
Jay Adelson: Yeah. I have three kids. I have a 24, 21 and 19- year- old. My 24- year- old was born two months before we founded Equinix.
Avi Freedman: Oh my God.
Jay Adelson: Right, I think that the subtext of that lesson is stay home. You only get one time and now I'm at 28 years, by the way, so I don't know how Brenda tolerated all those years of it, but that is the very, if there's one shining thing about that, it's the technical details don't really matter so much, you really just need to learn how to deal with that emotion.
Avi Freedman: I think that is a great-
Jay Adelson: You must be feeling it now at Kentik.
Avi Freedman: I think I have a slightly different problem, because I've seen CEOs that are manic, people that are manic that express it. And either everything is always awesome or everything is horrible and that can be really damaging. It could be exhausting for everyone around, now that's not me. Generally, I'm like," Okay, well I fucked up, let me not fuck up the same way." Experience what you get when you didn't get what you want. But what you have to remember when you start a company and especially if you're CEO and there's hundreds of people is, the words that come out of your mouth, people think are more important than you think. If you're just talking and thinking off the cuff, then someone's like," Oh, but the CEO said this." And I even saw this at Akamai where all of a sudden three people will be working on a project because I was like," Wouldn't it be cool to do this?" And the manager would come to me and ask," What did you ask someone to do?" I didn't ask anyone to do it.
Jay Adelson: Oh wow.
Avi Freedman: But that's something that people remind me, it's like, when you talk to people, remember. So it's not as much, and it's focus, again. Jason Lempkin, I call him the great Lempkoni, he's a SaaS VC and he hits on, you could critique his content and say," Well, it's so repetitive," but you look at his content, which is pointing out what they tell CEOs as you grow your company, which is, if you are only just getting sick of hearing yourself say it, then you haven't said it enough yet. And in the early days it's cool to be holding everything in your head and thinking about all the possibilities. But at some point, especially when you have 400 customers and 200 people, not everyone can hold everything in their head and shouldn't. And so you need to carefully limit focus and if you share everything it can be hard for people to see the patterns and get the details.
Jay Adelson: Yeah, your company is not a routing protocol.
Avi Freedman: Exactly. Exactly. Well, except it is, you do have distributed state.
Jay Adelson: That's right. Post pandemic, maybe it is.
Avi Freedman: But I've seen that. I've worked with some companies where I could see that not everyone has the whole picture. I'm like,"Oh, I do that. I should make sure to enable communication." I think that's the word that is, the thing that's most important to engineer around companies, besides some of the core cultural stuff, is communication, which I could be a better communicator. You talked about Matthew Prince, he's a great communicator.
Jay Adelson: Yeah, he's fantastic. The challenge right now is that I grew up and enjoyed building businesses where it didn't, they could be businesses that maybe from the outside in looked a little boring, but because the chemistry and the social part of coming to the office every day was fun and the people I loved to work with when I would come, I would drive to work every day with Jeff Rizzo and Matt Wood and we would sit, in the early Equinix days, and we'd have visits with folks like Martin Levy and we were all seeing each other all the time and riffing off each other and feeling that energy. And in a modern context, and this extends to not just network infrastructure companies but any company, I think that we lose so much of that. Which means that in order to overcome it, we have to over communicate, we have to have a morning stand- up when it's really not, everybody knows what to do already, it's more just a social check- in point.
Avi Freedman: Well, I think we're still figuring that out in conferences now things are getting back. But still, how do you have the whole way track without, how do you do that? And then how do you do that in a company or build the next Digg type thing or with what Elon wants to do, without facing the eternal September of Usenet or the encryption keys of Digg, all these things that happen as emergent behaviors. And any company, if you look at Apple, what happens when people don't like policy, you get emergent behaviors in companies, too. In fact, the more you stand on your principles, the more it's reasonable for employees to say," Hey if you believe this, then we believe this." Topics for future times, but Jay.
Jay Adelson: Yeah.
Avi Freedman: Thank you so much. Thank you for-
Jay Adelson: Hey, it was my pleasure
Avi Freedman: ...for sharing.
Jay Adelson: It's so great to see your face. It really is.
Avi Freedman: Great to see you. We need to get you to a NANOG at some points. You can get Bob crosstalk.
Jay Adelson: I'll do it. I'll do it. And you and I will stand back to back with swords as the-
Avi Freedman: Clue savers, yes.
Jay Adelson: There you go. Perfect.
Avi Freedman: Cool. Well, thanks. And thanks, Jay. And thanks to the audience for listening.
Chairman and Co-Founder of Scorbit, Jay Adelson, sits down with Network AF host Avi Freedman to talk about his history as a serial entrepreneur. Jay founded Equinix, Revision3, Opsmatic, and was the CEO of Digg. Throughout the conversation Jay and Avi touch on problems founders encounter, and discuss their mutual joy for gaming.
Highlights of the conversation include:
- Imposter syndrome and working at DEC
- Server evolution, power efficiency, and presentation
- How not to build a data center
- Terminating a cross connect and turning off service for Australia by accident
- Equinix during its hyper-growth phase
- Richard Clarke and peering
- The Forrest Gump of the Internet
- Amazing that the internet and the human body work at all
- Emergent behavior and the case of HD DVDs
- An evolution of the anarchy started by the internet
- Scorbit, gaming, and problem solving in pinball
- Advice and lessons to a young Jay