Rearchitecting the Cloudflare network for the AI revolution
Tom Strickx
Cloudflare
0:03Hello, welcome. Thank you everybody for being here. There's been a lot of things that we're doing at Cloudflare to manage the way we use IP addresses for the CDN side. So any version of mapping a name to an address. It's been a large evolution over the last five, six years. And I think now is a good marker to start talking about some of these things more publicly so that customers, we can help set their expectations. So first question is just the typical life of a CDN. Raise your hand if you're reasonably familiar with this. Client comes along, makes a request, gets to the authoritative, and a DNS, an IP address is returned in response, right? Is anyone uncomfortable with that idea? Fantastic.
0:48So what we're talking about is this process right here, okay, where the authoritative is going to return an address to the client. the resolve or direct doesn't matter okay and really what this talk should be titled is something like ip addresses at cloudflare are increasingly ephemeral so i know the history is somebody on boards they get a set a pair of ip addresses and the reason we always give two we assign two is for resilience there are two routes on the internet in case one is unavailable or fails for some reason there's always a backup that the client can use um but it is like same address all around the world all the time and increasingly that's going to start changing
1:33now we had an event we had a small incident back in in 2020 and i want to signpost this one so that you have it in the back of your mind as we talk through today and then i'm going to come back to it at the end sort of so you understand why this matters so in 2020 early 2020. This is a screenshot from an internal wiki document. What happened was for a long time, Cloudflare's resolver was number one in terms of performance globally, typically between 1 .1 and 8 .8, top two performers around the planet. A member, one of the engineers on the DNS team just happened to look at one of these public web pages that ranks the performance of DNS.
2:20resolvers around the planet and noticed that for a large region, mostly around Europe, the resolver performance had dropped. So suddenly we went from 1 .2 to 5 .6. And this matters not just to Cloudflare, but matters to customers as well because the DNS is part of the hot path whenever we're retrieving content. As it turned out, there was a route. on the internet no one knew it just by chance somebody happened to look at some rankings after two weeks and noticed something had gone wrong and then we dug in and then we discovered that routes from southern asia were being leaked to europe and so the 1 .1 traffic for most of western europe was flowing all the way down to south asia okay for two weeks is anyone unclear about that idea
3:22all right so this persists it's a page it's we return to it time and time again when we talk about this um now stepping back i'm going to make a claim hopefully uncontroversial it's this notion that ip addresses were only ever designed for routing and reachability and the reason i want to stake that is i think we have Evolved, understandably, in a world where we have one machine, one interface, one IP address. And so there's almost this association between an IP address and some notion of identity or what the machine is supposed to be, right? And the Internet, even back to the original RFCs, they are explicit. And they say very clearly IP addresses have no relationship to the processes that run on top of them, the networks that you might find behind them, and so on and so forth.
4:11Okay? Just happened to be that was... the world that we worked with. And then of course you get things like HTTP host and SNI, and they start to make clear that those boundaries never existed to begin with. In fact, so an old colleague of mine, an old hat, Dave Aran, he once said, exposing IP addresses to applications was a mistake and I couldn't agree more. So we've been spending a lot of effort making sure that IP addresses are decoupled from almost everything we do at Cloudflare. as part of that oh and the reason this matters to us by the way in part is because we're realizing increasingly with the cloudflare network the ip addresses actually don't matter what does matter is the route to cloudflare so the routes are provisioned uh you can imagine enterprise routes are provisioned maybe a little bit differently than the free routes but these are the crucial as long as the routes are safe the ip address within we don't tend to pay too much attention to it.
5:08And the reason is once a packet gets to us, we can send it anywhere within our network whenever we like for whatever reason. Okay. So it is most important for Cloudflare to get packets into the network as quickly as possible. And then we can handle it accordingly. The individual IP address doesn't actually matter for our purposes. We're okay so far.
5:32Fantastic. So starting in 2020 and then into we did this thing we called it the addressing agility experiment there's a blog post about it and there's a peer review paper that's attached to it you're welcome to check it out but fundamentally what we decided to do was to take millions upon millions of your domain names and then randomize the ip address that they got with every query so we stuck them into a slash 20 you could get any one of 4 000 addresses for every query it would change. Okay. And then we said, well, wait, why does it have to be 4 ,000? Why not make it a slash 24 in the V4 case? We do this with the V6, by the way, as well. Slash 24, in which case you get any one of 256 addresses. And then we thought, we've done all this hard work.
6:22Why don't we try random over one address? A full slash 32 or a slash 128. And that got us to a place where it turns out. we were running millions of names on a single address and we could change the address every time a new query came in if we wanted to, so at any rate we chose to, and for any reason. Now, I'm going to pause here and say this has been running since late 2020, early 2021, in a few different parts of the planet. Initially, when we first decided to do this, it was only in and in montreal would anyone like to ask why it was in toronto and montreal and by the way i'm going to point out that a cloudflare when we deploy we have the canaries right like we have these experimental uh pops that we deploy to but we were told explicitly we would not be allowed to do that for this experiment and we narrowed down our choices when we went through the list canada and new zealand and we ruled out new zealand because it just isn't big enough. But why did we narrow it down to those two?
7:29No guesses. No Canadians in the pack, huh? Except for me. So the honest, genuine feeling, anecdotal as it was, is as confident as we were that nothing would break, we genuinely believed that these were the two places in the world where if something broke, if something was catastrophic, we could say we're sorry and and things would ultimately be okay okay yeah there's also by the way next year we might come back there's a lovely incident that was caused by doing this experiment in a new production system rolling out and the two of them together created a a continental scale routing loop which was kind of interesting and it's it's a great teaching talk because everybody did exactly the right thing and still things went wrong.
8:25So now that we've got this, we can ask, what features can we enable? If the address just becomes like a variable effectively in a program that I can write, what can we do with it? Well, the first thing we have to ask is, well, how do we manage it? Now that we know that we can do this, we need to build a system in order to manage it. And that's called Topaz. And we started to talk about it in 24. Again, there's a lovely blog that goes with it and a peer review paper. And effectively, I'll give you the short. version what happens is we write a bunch of programs to do the addressing that effectively get compiled and then shipped off to the name servers at the edge now this is ever so slightly different from most authoritative services where you come up with a mapping so other other large operators will run a program and has some objective functions and they will create a mapping of a name to an address and then that mapping off to the servers themselves that doesn't happen at cloudflare well still does a little bit but increasingly we are we are getting away from that and moving towards this model where we don't ship a mapping we actually ship the program to the edge servers okay and these run on the hot path every time there's a dns query uh and what that means is we've effectively to do this with high performance we've adopted the match action paradigm.
9:49So there's a little program that matches on a set of criteria, for example, service tier or some set of data centers. Okay. And then there's a response function that is completely independent of the match functions. We've decided, designed our responses to be separate from these. Typically they are given address pools to pick from and then a selection mechanism. The selection mechanism can be something like purely random, which we don't do. In practice, because it's hard to debug if you're always giving away new addresses. Typically, we do something much more deterministic, and you're going to see an example of that, one of the things we're thinking of. But then all of these things tend to be parametrized at the same time. Now, the first time we deployed this, en masse, we were changing a whole tier's worth of addresses.
10:42It's a simple merge on a Git repo. text everyone can read it human readable so we start to understand what we're doing why addresses are given to customers under certain circumstances and the first time we did this i think we affected a full address change around the planet in about 10 minutes and then we realized we'd made a slight mistake not anything catastrophic it was just in the in the size of the prefix so things were fine so we corrected it with a three character change we pushed that out and i believe the fix went out in about two and a half minutes globally everywhere. So seconds in some places and then minutes in others. Today, that's much faster. What I want to point out here, though, is by design, going back to those original notions of internet design, where the IP address exists independent of everything else that sits where the IP address is, we've effectively engineered this system so that—oops, can you go back one? Where's our—that's all right.
11:37We've engineered the system— so that names play no role. And maybe I should pause here and say, if anybody in the room uses static, dedicated, or BYOIP, this does not apply. Those are special cases, okay? But for the broad set, we've effectively engineered it so that the policy would have to change if we wanna do something different rather than carve out exceptions for names, right?
12:07So let me get out, remember that route leak example? One of the, it's almost a dirty little secret, but it's not. Anycast is extraordinary. So we've done the analysis and the measurements internally from a performance of, from the perspective of performance, it's really hard to beat BGP routing. So we found maybe single digit percentage places in the world where if we could control routes, we could actually get packets to our network faster. And of course, because you can reach so many different data centers, the reliability and the resilience right but what we lose a little bit of is observability so let me let me give you an example of a system that we're going to be chasing after in order to help us get observability into route leaks so here we have cloudflare on both sides of the atlantic and the way the world exists today that everyone is used to is when a query comes in the same address is returned here as is returned here And the subsequent HTTP request, of course, would use the address.
13:10Presumably it comes here, and the same thing, presumably it comes back here, reasonably close by. What if we change that a little bit? What if instead we say, we only care about the routes, and the name will get some address within the route here, and it will get some other address within the route here? Okay?
13:35Now, what we can do is the following.
13:39If I see roughly the same thing here in the vicinity that I've given away and people are connecting to the address, great. And same thing on this side over here. And by the way, everybody anecdotally knows of these cases where the DNS query might go someplace that's slightly different from a different. So if a 15 is given out in Paris, if we use the Paris pop to give away the 15, see a query in london we probably wouldn't balk at it but what if instead now something different happens and knowing that i give out the dot199 over here and only here and here i'm giving away dot15 but i'm seeing connections on dot199 immediate sorry dns poisoning unlikely to be poisoning let's hope i mean we're on this scale unlikely.
14:31Instead, what has happened is this route from here on this, right, the route advertisement from here has leaked somehow over here, which is exactly what happened in the DNS deluge that I pointed to earlier is one of our peers in South Asia had leaked a route to its peers in Europe. And so a lot of European ISPs were preferring that route instead. Great. So, hopefully this convinced you. It's a good thing that IP addresses are going to be increasingly ephemeral. We just want people to start getting used to the idea. And I want to remind the room, in many parts in the world, we have been doing something like this, changing addresses, moving addresses, and so on for five years, and no one has noticed.
Tom Strickx
Cloudflare
Alex Krivit, Brandon Harris
Cloudflare
Kevin Deems
Cloudflare
Be the first to know when registration opens and the agenda goes live.
--- title: "Cloudflare’s evolution of managing IP addresses for customers at scale" url: "https://www.cloudflare.com/connect/2026/session/cloudflares-evolution-of-managing-ip-addresses-for-customers-at-scale" year: 2026 --- # Cloudflare’s evolution of managing IP addresses for customers at scale [Watch session](https://www.cloudflare.com/connect/2026/session/cloudflares-evolution-of-managing-ip-addresses-for-customers-at-scale) ## Session details - **Track:** Network & Infrastructure - **Topics:** Cloudflare product innovations, Emerging tech & post-quantum, Network modernization & multicloud, Performance & resiliency - **Solution areas:** Core Platform ## About this session Learn about the ways Cloudflare ensures reliability, performance, and global reachability for its customers’ websites, domains, and their users. Contrary to the design of IP, addresses are often tied to domain names and applications in ways that stifle innovation and even cause incidents. However, in 2021, Cloudflare announced trials and technology to decouple names from addresses. In 2024, Cloudflare revealed new technology to manage addresses dynamically and safely and disseminate changes globally, in just seconds. Cloudflare is now using its knowledge and technology to better scale with the Internet and to help advance the evolution of IP addresses. (Note: this does not apply to BYOIP, static, and dedicated addressing.) ## Speakers - [Marwan Fayed](https://www.cloudflare.com/connect/speakers/2421443) — Principal Scientist and Research Lead, Cloudflare ## Transcript [0:03] Hello, welcome. Thank you everybody for being here. There's been a lot of things that we're doing at Cloudflare to manage the way we use IP addresses for the CDN side. So any version of mapping a name to an address. It's been a large evolution over the last five, six years. And I think now is a good marker to start talking about some of these things more publicly so that customers, we can help set their expectations. So first question is just the typical life of a CDN. Raise your hand if you're reasonably familiar with this. Client comes along, makes a request, gets to the authoritative, and a DNS, an IP address is returned in response, right? Is anyone uncomfortable with that idea? Fantastic. [0:48] So what we're talking about is this process right here, okay, where the authoritative is going to return an address to the client. the resolve or direct doesn't matter okay and really what this talk should be titled is something like ip addresses at cloudflare are increasingly ephemeral so i know the history is somebody on boards they get a set a pair of ip addresses and the reason we always give two we assign two is for resilience there are two routes on the internet in case one is unavailable or fails for some reason there's always a backup that the client can use um but it is like same address all around the world all the time and increasingly that's going to start changing [1:33] now we had an event we had a small incident back in in 2020 and i want to signpost this one so that you have it in the back of your mind as we talk through today and then i'm going to come back to it at the end sort of so you understand why this matters so in 2020 early 2020. This is a screenshot from an internal wiki document. What happened was for a long time, Cloudflare's resolver was number one in terms of performance globally, typically between 1 .1 and 8 .8, top two performers around the planet. A member, one of the engineers on the DNS team just happened to look at one of these public web pages that ranks the performance of DNS. [2:20] resolvers around the planet and noticed that for a large region, mostly around Europe, the resolver performance had dropped. So suddenly we went from 1 .2 to 5 .6. And this matters not just to Cloudflare, but matters to customers as well because the DNS is part of the hot path whenever we're retrieving content. As it turned out, there was a route. on the internet no one knew it just by chance somebody happened to look at some rankings after two weeks and noticed something had gone wrong and then we dug in and then we discovered that routes from southern asia were being leaked to europe and so the 1 .1 traffic for most of western europe was flowing all the way down to south asia okay for two weeks is anyone unclear about that idea [3:22] all right so this persists it's a page it's we return to it time and time again when we talk about this um now stepping back i'm going to make a claim hopefully uncontroversial it's this notion that ip addresses were only ever designed for routing and reachability and the reason i want to stake that is i think we have Evolved, understandably, in a world where we have one machine, one interface, one IP address. And so there's almost this association between an IP address and some notion of identity or what the machine is supposed to be, right? And the Internet, even back to the original RFCs, they are explicit. And they say very clearly IP addresses have no relationship to the processes that run on top of them, the networks that you might find behind them, and so on and so forth. [4:11] Okay? Just happened to be that was... the world that we worked with. And then of course you get things like HTTP host and SNI, and they start to make clear that those boundaries never existed to begin with. In fact, so an old colleague of mine, an old hat, Dave Aran, he once said, exposing IP addresses to applications was a mistake and I couldn't agree more. So we've been spending a lot of effort making sure that IP addresses are decoupled from almost everything we do at Cloudflare. as part of that oh and the reason this matters to us by the way in part is because we're realizing increasingly with the cloudflare network the ip addresses actually don't matter what does matter is the route to cloudflare so the routes are provisioned uh you can imagine enterprise routes are provisioned maybe a little bit differently than the free routes but these are the crucial as long as the routes are safe the ip address within we don't tend to pay too much attention to it. [5:08] And the reason is once a packet gets to us, we can send it anywhere within our network whenever we like for whatever reason. Okay. So it is most important for Cloudflare to get packets into the network as quickly as possible. And then we can handle it accordingly. The individual IP address doesn't actually matter for our purposes. We're okay so far. [5:32] Fantastic. So starting in 2020 and then into we did this thing we called it the addressing agility experiment there's a blog post about it and there's a peer review paper that's attached to it you're welcome to check it out but fundamentally what we decided to do was to take millions upon millions of your domain names and then randomize the ip address that they got with every query so we stuck them into a slash 20 you could get any one of 4 000 addresses for every query it would change. Okay. And then we said, well, wait, why does it have to be 4 ,000? Why not make it a slash 24 in the V4 case? We do this with the V6, by the way, as well. Slash 24, in which case you get any one of 256 addresses. And then we thought, we've done all this hard work. [6:22] Why don't we try random over one address? A full slash 32 or a slash 128. And that got us to a place where it turns out. we were running millions of names on a single address and we could change the address every time a new query came in if we wanted to, so at any rate we chose to, and for any reason. Now, I'm going to pause here and say this has been running since late 2020, early 2021, in a few different parts of the planet. Initially, when we first decided to do this, it was only in and in montreal would anyone like to ask why it was in toronto and montreal and by the way i'm going to point out that a cloudflare when we deploy we have the canaries right like we have these experimental uh pops that we deploy to but we were told explicitly we would not be allowed to do that for this experiment and we narrowed down our choices when we went through the list canada and new zealand and we ruled out new zealand because it just isn't big enough. But why did we narrow it down to those two? [7:29] No guesses. No Canadians in the pack, huh? Except for me. So the honest, genuine feeling, anecdotal as it was, is as confident as we were that nothing would break, we genuinely believed that these were the two places in the world where if something broke, if something was catastrophic, we could say we're sorry and and things would ultimately be okay okay yeah there's also by the way next year we might come back there's a lovely incident that was caused by doing this experiment in a new production system rolling out and the two of them together created a a continental scale routing loop which was kind of interesting and it's it's a great teaching talk because everybody did exactly the right thing and still things went wrong. [8:25] So now that we've got this, we can ask, what features can we enable? If the address just becomes like a variable effectively in a program that I can write, what can we do with it? Well, the first thing we have to ask is, well, how do we manage it? Now that we know that we can do this, we need to build a system in order to manage it. And that's called Topaz. And we started to talk about it in 24. Again, there's a lovely blog that goes with it and a peer review paper. And effectively, I'll give you the short. version what happens is we write a bunch of programs to do the addressing that effectively get compiled and then shipped off to the name servers at the edge now this is ever so slightly different from most authoritative services where you come up with a mapping so other other large operators will run a program and has some objective functions and they will create a mapping of a name to an address and then that mapping off to the servers themselves that doesn't happen at cloudflare well still does a little bit but increasingly we are we are getting away from that and moving towards this model where we don't ship a mapping we actually ship the program to the edge servers okay and these run on the hot path every time there's a dns query uh and what that means is we've effectively to do this with high performance we've adopted the match action paradigm. [9:49] So there's a little program that matches on a set of criteria, for example, service tier or some set of data centers. Okay. And then there's a response function that is completely independent of the match functions. We've decided, designed our responses to be separate from these. Typically they are given address pools to pick from and then a selection mechanism. The selection mechanism can be something like purely random, which we don't do. In practice, because it's hard to debug if you're always giving away new addresses. Typically, we do something much more deterministic, and you're going to see an example of that, one of the things we're thinking of. But then all of these things tend to be parametrized at the same time. Now, the first time we deployed this, en masse, we were changing a whole tier's worth of addresses. [10:42] It's a simple merge on a Git repo. text everyone can read it human readable so we start to understand what we're doing why addresses are given to customers under certain circumstances and the first time we did this i think we affected a full address change around the planet in about 10 minutes and then we realized we'd made a slight mistake not anything catastrophic it was just in the in the size of the prefix so things were fine so we corrected it with a three character change we pushed that out and i believe the fix went out in about two and a half minutes globally everywhere. So seconds in some places and then minutes in others. Today, that's much faster. What I want to point out here, though, is by design, going back to those original notions of internet design, where the IP address exists independent of everything else that sits where the IP address is, we've effectively engineered this system so that—oops, can you go back one? Where's our—that's all right. [11:37] We've engineered the system— so that names play no role. And maybe I should pause here and say, if anybody in the room uses static, dedicated, or BYOIP, this does not apply. Those are special cases, okay? But for the broad set, we've effectively engineered it so that the policy would have to change if we wanna do something different rather than carve out exceptions for names, right? [12:07] So let me get out, remember that route leak example? One of the, it's almost a dirty little secret, but it's not. Anycast is extraordinary. So we've done the analysis and the measurements internally from a performance of, from the perspective of performance, it's really hard to beat BGP routing. So we found maybe single digit percentage places in the world where if we could control routes, we could actually get packets to our network faster. And of course, because you can reach so many different data centers, the reliability and the resilience right but what we lose a little bit of is observability so let me let me give you an example of a system that we're going to be chasing after in order to help us get observability into route leaks so here we have cloudflare on both sides of the atlantic and the way the world exists today that everyone is used to is when a query comes in the same address is returned here as is returned here And the subsequent HTTP request, of course, would use the address. [13:10] Presumably it comes here, and the same thing, presumably it comes back here, reasonably close by. What if we change that a little bit? What if instead we say, we only care about the routes, and the name will get some address within the route here, and it will get some other address within the route here? Okay? [13:35] Now, what we can do is the following. [13:39] If I see roughly the same thing here in the vicinity that I've given away and people are connecting to the address, great. And same thing on this side over here. And by the way, everybody anecdotally knows of these cases where the DNS query might go someplace that's slightly different from a different. So if a 15 is given out in Paris, if we use the Paris pop to give away the 15, see a query in london we probably wouldn't balk at it but what if instead now something different happens and knowing that i give out the dot199 over here and only here and here i'm giving away dot15 but i'm seeing connections on dot199 immediate sorry dns poisoning unlikely to be poisoning let's hope i mean we're on this scale unlikely. [14:31] Instead, what has happened is this route from here on this, right, the route advertisement from here has leaked somehow over here, which is exactly what happened in the DNS deluge that I pointed to earlier is one of our peers in South Asia had leaked a route to its peers in Europe. And so a lot of European ISPs were preferring that route instead. Great. So, hopefully this convinced you. It's a good thing that IP addresses are going to be increasingly ephemeral. We just want people to start getting used to the idea. And I want to remind the room, in many parts in the world, we have been doing something like this, changing addresses, moving addresses, and so on for five years, and no one has noticed. ## Related sessions - [Rearchitecting the Cloudflare network for the AI revolution](https://www.cloudflare.com/connect/2026/session/rearchitecting-the-cloudflare-network-for-the-ai-revolution) - [Breaking the Internet (and putting it back together)](https://www.cloudflare.com/connect/2026/session/breaking-the-internet-and-putting-it-back-together) - [Ensuring resiliency during network maintenance](https://www.cloudflare.com/connect/2026/session/ensuring-resiliency-during-network-maintenance) ## More from Connect - [All sessions](https://www.cloudflare.com/connect/) - [Get notified for Connect 2027](https://www.cloudflare.com/connect#2027)