Advertising prefixes for fun and profit: How to anycast

Network & Infrastructure

About the session

Cloudflare’s network uses anycast to advertise the same IP prefixes from locations globally, enabling Internet routing to direct traffic toward an available Cloudflare data center. But operating anycast at global scale takes more than announcing a prefix with BGP. This session traces the evolution of prefix advertisements at Cloudflare, from relatively static routing configurations to increasingly automated, adaptive systems. Explore how routing policy, network topology, capacity, health signals, and traffic engineering influence where prefixes are advertised and how incoming traffic is distributed. Learn about operational challenges that shaped this evolution, including uneven traffic, overloaded locations, routing convergence, and the need to make changes safely across a large network. Leave with an intermediate-level understanding of how anycast works, why prefix-advertisement strategies evolve as networks grow, and what it takes to build reliable, efficient global load balancing.

View transcript

0:03My name is Tom Strix. I'm a Senior Principal Network Engineer here at Cloudflare. I've been with the network team for the last nine years, and I've built some of the systems and tooling that we use now on a day-to-day basis, including some of the systems that we've built around the advertisement of our prefixes. And I'm joined here today by Ash. Hi, I'm Ash Pallarito. I'm a Senior Product Manager at Cloudflare. I cover traffic. management. I've covered routing and IP addressing. So a little bit of everything in the network and infrastructure space. All right, let's get into it. So why are we doing this talk? You're thinking how to Anycast. Okay, we know that. Well, back in 2011, our CEO did a primer on Anycast. You can go into our blog and dig it up if you want to read it. But that was obviously a while ago, and we've grown a lot since then.

0:52And so we wanted to provide that refresher. The core idea hasn't really changed since 2010. You advertise an IP prefix from many different places and BGP picks the closest copy. That core idea hasn't really changed, but what has changed is everything we've built around it and the many different jobs an IP address does for us today. So this session will trace that 16 -year history of all of our routing and traffic management and load balancing changes. So quick level set, just so we're on the same page. We have unicast on the left. One address lives in one space, and the network sends you to that one, so the red dot can only reach the green dot that owns that IP address. On the right, we have anycast. The IP address lives in many places, and which one you get depends on your location and how the routing shakes out.

1:46So we have five parts for you today. First, Tom will take us through our superhero origins. story and a little bit the start of our growing pains. As the product person, I'll obviously double down on the pain part. We love that. And then start getting into how we started growing up in terms of our load balancing solutions. And then Tom will take us home with some of our traffic management changes and where we are now. Great. Awesome. Thank you, Ash. So yeah, talk a tiny bit about the origins, right? 2010, that's when Cloudflare started, 16 years ago. In Belgium, where I'm from, that would mean that we're now legal to drink beer at a bar. It's delightful. Definitely doesn't make me feel older in any way or form. So Cloudflare launched in 2010, and we started using Anycast from day one.

2:35That has always been the guiding principle, at least for the load balancing component. It's always been Anycast has been open standards, and it has been off -the -shelf hardware. That has always been that kind of drive. factor to make sure that cloudflare could scale as easily and as quickly as we've done over those kind of last couple of years so 2011 i think matthew wrote that kind of initial primer on anycast main id as ash outlined it's one ip for many destinations at this point in time cloudflare has more than what's the last number 320 cities i think 320 cities a multitude of that with pops the idea here is that with any cast we can direct all of those end users to the right locations without needing to do anything ourselves we rely entirely on kind of the goodwill of the isps routing routing directly to to our data centers in the optimal way that is not always the most efficient right we've seen that occasionally have, unfortunately, some stories about sending traffic to the wrong locations that occasionally happens.

3:50And that is one of the unfortunate consequences of running Anycast. But we've learned one of the things, though, with Anycast, one of the critical components that people need to really grok sometimes is that it's not just one thing to do. One IP lives in many locations. You also need to make sure that, well, if you have one IP in many locations, and then those many locations need to the exact same services because you don't control where that traffic lands and what traffic for which service so you need to make sure that all of those services run on all of your machines and that's always been that guiding principle again for how we've run things at cloudflare so a key thing here in in the the things that we talk about externally and internally is anycast is how your isp gets you to you know the best location for for cloudflare right like Europe, that can be between Brussels, between London, between Frankfurt, between Amsterdam, between Paris.

4:46That's where Anycast guides that traffic to make sure that we get things there. And then the critical component for Cloudflare for us ourselves is ECMP, that's Equal Cost Multipath. And that is one of the core components or one of the core technologies that we leverage to do load balancing internally to make sure that we can service as many requests as we need at the right time. But, you know, since 2010, always been using BGP, always been trying to make sure that we're always using the open standards. But since 2010 to about 2016, 2017, that network had grown massively. Originally, the network started very small and in four to five locations with a couple of networks to directly interconnect with and a bunch of transit providers to make sure that...

5:36could reach the rest of the internet right and that was workable was scalable but the network kept growing and one of the big problems that um we kept running into was people in tokyo were getting routed to san jose even though that there was a tokyo pop and that was because they were interconnecting better in san jose in in california with the the local japanese networks than they were in in tokyo which meant that a bunch of that traffic started routing to the wrong locations. From a topology perspective, from a BGP or a network topology perspective, it was ideal. It was the shortest path, and it was perfect. But that meant that we needed to start making changes in the way that prefixes were being advertised. The way that's done in BGP, that's done through one of the easiest way of doing this.

6:25Sorry, there's multiple ways, and we're not going to do a deep dive on those. But one of the easiest way of doing this is ASPath prepend. What you can see here is for one of the prefixes that we advertise, 104 .20 .240 .20, we have 1.333.5, that's Cloudflare, occurs three times in there. So we are prepending twice before handing over to our transit providers. At that point in time, that made sense because that way we could manipulate how people were or eyeballs were ingressing into the Cloudflare network just by doing ASPath prepends. And it worked. But over time, those prepens became kind of pointless because the deployments were getting so widely varied and there were so many networks that we could directly interconnect with that those prepens didn't actually make sense anymore.

7:16And we actually needed to get everything equal to make sure that traffic was routing correctly. So about 2016, 2017, every single prefix from Cloudflare was... prepended equally on every single transit provider so at that point your prepends were useless right because everything just had the exact same equal length on every single transit provider there wasn't much point to that anymore so one of the problems with that though is once that started happening whenever we were turning up a new location we were at risk of advertising the anycast prefixes without any pre to one of our transit providers. What then instantly happened is something that became colloquially known as SJCcast when you're sending all of your traffic to San Jose, or TampaCast when all of the traffic arrives in Tampa.

8:12Even at that time, Cloudflare 2017, none of our locations were sized to handle all of Cloudflare's traffic in a single location. And that then turned into delightfully lovely incidents that we'd done. about externally and publicly and this is me in 2018 this was the first talk i ever done for cleftler at ripe talking about exactly this problem um and and this was a growing pain it was a a lesson we had to learn on organically growing a network and that's fine and then we end up in this position and then all of a sudden you're off in a much worse place than you'd to be and the idea was well we need to fix this and we did and we went through a major engineering effort and we removed all of our pre -pens in the span of about 15 minutes and that's what that talk is about um and that it was mostly a ton of fun talking about the automation and the systems that we put in place to allow us to do that but again that was growing and that was evolving and getting right place and one of the other things that needed to get to the right place was the way that we internally looked at the prefixes that we are advertising and how we kind of distribute those internally all right time to double down on the pain prod cfb the whole network was one database

9:44so if you work at cloudflare you might be laughing slash crying at this meme If you don't work at Cloudflare, here's the short of it. ProdCFP was a single shared configuration store for the entire network. It held two kinds of things, keys and values. And those could mean almost anything. And so that means our prefix, advertisement settings, and our load balancing settings lived in the same place. That flexibility helped us early on, but it's also what hurt us later.

10:17Here's how one engineer characterized the process. It's actually got three complaints in one. I aspire to this level of complaining. So like chef's kiss as usual engineering. First, it's painfully slow. And reloading bird, that's the routing daemon, was a manual step you had to do by hand. And then finally nothing double checked it before it went live. So as you can see, mildly terrifying.

10:41Here's a picture of our load balancing at the time. So you can see this tall pillar with a bunch of. sized pieces sacked up and then kind of all propped up here by this little stored procedure. That's redistribute ECMP. And ECMP groups, this is the name of the process that decides which machines in a data center advertise which prefixes. And here's a screenshot of the actual wiki procedure, the SOP, to run this. And I can see why people hated it. And you can see the risk severity if you squint here, says major. So basically what it would do is read a bunch of state, write a small number of changes, and then trigger a whole lot of reloads on the machines. And every reload was a chance for something to go wrong.

11:26So again, terrifying.

11:29This graph is from about five or six years ago, and this kind of visualizes that process. So each row is a machine, and each column is a prefix, and then each color represents a different generation of machine. And so you can see at the time, Gen 9 was our newest, so you see kind of the most activity there. And on paper, it made sense, right? Send the most traffic to the machines with the most capacity. However, in practice, it was scattered. We had prefixes all over the shop. And it was anytime you had to make a change, it was hard to reason about, hard to change and hard to fix when load was uneven.

12:05So we did three things. First, we adopted RPKI to cryptographically. validate which network is allowed to originate a prefix then we standardized and shortened our as paths instead of hand tuning prepens per prefix prepens per prefix if you want a tongue twister and finally we normalized the length so instead of this patchwork of sizes we said for instance let's slash 20 everything so just quick recap where we're at picture it 2018 the before times we had 150 locations cute right 74 countries huh and only 250 IPv4 prefixes.

12:41All right, so we've talked about the pain. Let's get in a little bit of some of the solutions and how we started growing up. Intro Unimog, my fave. This is our layer four load balancer. We started removing those bits and pieces from that tall, wobbly, scary-looking Jenga pillar we showed you of the load balancing earlier. And we removed that active load balancing from prod CFP using Unimog. So each line on this... represents the machine's CPU. And so you can see before the white line where Unimog was introduced right here, it's quite scattered. Each machine generation is doing different amounts of work. After the line, it bunches together, becomes a lot more uniform. That's Unimog doing the load balancing inside of a data center.

13:28But we'd not moved everything over. One year later, we introduced Colo Topologies, a flexible... expression parser that allowed us to express a set of locations from which we'd advertise a prefix from. So before, it was a static list of colo ID arrays in our CIDR distribution list, and it had to be painstakingly messed with by an engineer, a poor engineer, who had to make some edit to it and pray nothing went wrong, right? From there, it just became a simple edit to a rule, an expression, where the locations would kind of figure out the rest based on the rule that you

14:02All right, now I'll hand it over to Tom to talk about the traffic management enhancements.

14:07So like I said earlier, one of the big problems we have at Anycast or one of the big joys we have at Anycast is it's very simple for us to turn up because we literally go to any other place, connect with any other network, and they automatically basically route to the correct location for us. But we don't always want that traffic in that location. Getting capacity in the right locations at the right time is an incredibly complicated. a very complex problem and especially nowadays with lead times going into the the months of eight months nine months 12 months sometimes impossible to get any sort of capacity we started building a system out that allowed us to actually forward some of that traffic that was landing in one location to a different location where we did actually have spare capacity that allowed us to get a much smoother deployment going on the capacity that we want capacity that we need and capacity that we have available But Traffic Manager alone didn't really change everything all that much.

15:03One of the big things that changed massively for Cloudflare in the way that we advertised prefixes was the fact that not all of the prefixes that we were advertising are actually our own prefixes. In 2019, 2020, we started a system or a project or product really named BYUIP. It allowed customers to bring their own IP addresses, their own prefix. do Cloudflare. And since we started this, and you can see this is January 2020, all the way up to October 2022, in just that span of two years, we provisioned over 12 ,000 different prefixes onto the Cloudflare network. Obviously, that is not prefixes that were advertised all at once, all at the same time, but that is just prefixes for customers that we were ready to start provisioning.

15:50And that is massive growth. growth was enabled mostly and uniquely by the Addressing API. The Addressing API was an internal system because one of the engineers on another team, on the DNS team at the time, understood and realized that prefix management is actually going to be a massive concern and a technical problem that we'll need to address both internally for ourselves, for the 250 -ish prefixes that we were advertising and the way that we were doing load balancing. as well as for our customers. We were getting more and more enterprise customers that wanted to be in control of even more of their own stack which included being in control of their prefixes that wanted to onboard onto Cloudflare services but then couldn't because you know they have different they wanted to control over their prefixes.

16:37So we started building BYR IP and it's been a massive success but one of the big things with success is it can have unfortunate consequences and this is what we have to do with the future and this is what we have to do with the is one of the unfortunate consequences we ran into we had one big customer on board more than 10 000 of their own prefixes and our routers were unfortunately not equipped to deal with that massive influx of of prefixes at least not in the the way that we had implemented it at the time so when the customer onboarded you can see this obviously we want to push these things out quite quickly to the edge so there was not a whole lot of space and time between when the first routers started becoming difficult to when most of them started following so we immediately started a task force to make sure that we could understand what was causing this problem how can we onboard this customer safely without bringing up the rest of the network and how can we make sure that we can keep doing this scalably going forward because obviously we want to keep selling this as a product this can be oh yeah one more customer and we'll stop um that's not how we work so instead what we started doing was scale down and rethink the way that we're doing advertisements and the way that we're doing load balancing um one of our key things with ecmp was we wanted to split that into even more prefixes so we disaggregated our prefixes that blows up the the routing table internally for that load balancing so one of the big changes

18:08that we made was what if we just let our compute nodes with bird advertise to our routers and then we just upstream that directly to our peers and our transit providers without any form of aggregation on our edge and that's exactly what we did and just that alone allowed us to completely remove that memory overhead and go back down to kind of the operational overhead that we have from before um these are massive technical changes that involved a lot of different teams and a lot of different um engineers to kind of get us over the line. But this allowed us to scale

18:51without any further issues ever since. So now the thing is, like, where are we now? So let's have a look at the numbers.

19:04All right. And so if you remember back at the 2018 slide, the cute one, we've now grown a ton. years later, we're at 346 cities. They're over almost, let's see, 14 ,000 networks directly connected to Cloudflare with 590 TPS and 31 DDoS attack mitigation in 35 seconds in 2025 with no engineer page. So we're able to handle a lot more load now as it's all distributed. We have more footprint than ever. So in terms of how our IP address strategy involved, you can see here We hand -tued those AS paths, and we changed how we did ingress prefix advertisement to RPK -evalidated, like I said earlier, automated, health and capacity -driven.

19:50We standardized. We went from one building, one colo, to our multi -colo pops, and we introduced traffic manager, egress, soft, any cast addressing. And we scaled our peering relationships, so over 13,000 direct network connections.

20:08so what's next we actually want to hear that from you we have a roadmap but we would love to understand what exactly you want from our network from bring your own ip from kind of if you're a cloud for a customer what are you looking for in this regard all right cool all right thank you

Browse all
Connect 2027

Connect returns in 2027

Be the first to know when registration opens and the agenda goes live.

By submitting this form, you agree to our Privacy Policy.

cloudflare-connect-2026

---
title: "Advertising prefixes for fun and profit: How to anycast"
url: "https://www.cloudflare.com/connect/2026/session/advertising-prefixes-for-fun-and-profit-how-to-anycast"
year: 2026
---

# Advertising prefixes for fun and profit: How to anycast

[Watch session](https://www.cloudflare.com/connect/2026/session/advertising-prefixes-for-fun-and-profit-how-to-anycast)

## Session details

- **Track:** Network & Infrastructure
- **Topics:** Performance & resiliency
- **Solution areas:** Core Platform

## About this session

Cloudflare’s network uses anycast to advertise the same IP prefixes from locations globally, enabling Internet routing to direct traffic toward an available Cloudflare data center. But operating anycast at global scale takes more than announcing a prefix with BGP. This session traces the evolution of prefix advertisements at Cloudflare, from relatively static routing configurations to increasingly automated, adaptive systems. Explore how routing policy, network topology, capacity, health signals, and traffic engineering influence where prefixes are advertised and how incoming traffic is distributed. Learn about operational challenges that shaped this evolution, including uneven traffic, overloaded locations, routing convergence, and the need to make changes safely across a large network. Leave with an intermediate-level understanding of how anycast works, why prefix-advertisement strategies evolve as networks grow, and what it takes to build reliable, efficient global load balancing.

## Speakers

- [Ash Pallarito](https://www.cloudflare.com/connect/speakers/2421440) — Senior Product Manager, Cloudflare
- [Tom Strickx](https://www.cloudflare.com/connect/speakers/2411667) — Principal Network Engineer, Cloudflare

## Transcript

[0:03] My name is Tom Strix. I'm a Senior Principal Network Engineer here at Cloudflare. I've been with the network team for the last nine years, and I've built some of the systems and tooling that we use now on a day-to-day basis, including some of the systems that we've built around the advertisement of our prefixes. And I'm joined here today by Ash. Hi, I'm Ash Pallarito. I'm a Senior Product Manager at Cloudflare. I cover traffic. management. I've covered routing and IP addressing. So a little bit of everything in the network and infrastructure space. All right, let's get into it. So why are we doing this talk? You're thinking how to Anycast. Okay, we know that. Well, back in 2011, our CEO did a primer on Anycast. You can go into our blog and dig it up if you want to read it. But that was obviously a while ago, and we've grown a lot since then.

[0:52] And so we wanted to provide that refresher. The core idea hasn't really changed since 2010. You advertise an IP prefix from many different places and BGP picks the closest copy. That core idea hasn't really changed, but what has changed is everything we've built around it and the many different jobs an IP address does for us today. So this session will trace that 16 -year history of all of our routing and traffic management and load balancing changes. So quick level set, just so we're on the same page. We have unicast on the left. One address lives in one space, and the network sends you to that one, so the red dot can only reach the green dot that owns that IP address. On the right, we have anycast. The IP address lives in many places, and which one you get depends on your location and how the routing shakes out.

[1:46] So we have five parts for you today. First, Tom will take us through our superhero origins. story and a little bit the start of our growing pains. As the product person, I'll obviously double down on the pain part. We love that. And then start getting into how we started growing up in terms of our load balancing solutions. And then Tom will take us home with some of our traffic management changes and where we are now. Great. Awesome. Thank you, Ash. So yeah, talk a tiny bit about the origins, right? 2010, that's when Cloudflare started, 16 years ago. In Belgium, where I'm from, that would mean that we're now legal to drink beer at a bar. It's delightful. Definitely doesn't make me feel older in any way or form. So Cloudflare launched in 2010, and we started using Anycast from day one.

[2:35] That has always been the guiding principle, at least for the load balancing component. It's always been Anycast has been open standards, and it has been off -the -shelf hardware. That has always been that kind of drive. factor to make sure that cloudflare could scale as easily and as quickly as we've done over those kind of last couple of years so 2011 i think matthew wrote that kind of initial primer on anycast main id as ash outlined it's one ip for many destinations at this point in time cloudflare has more than what's the last number 320 cities i think 320 cities a multitude of that with pops the idea here is that with any cast we can direct all of those end users to the right locations without needing to do anything ourselves we rely entirely on kind of the goodwill of the isps routing routing directly to to our data centers in the optimal way that is not always the most efficient right we've seen that occasionally have, unfortunately, some stories about sending traffic to the wrong locations that occasionally happens.

[3:50] And that is one of the unfortunate consequences of running Anycast. But we've learned one of the things, though, with Anycast, one of the critical components that people need to really grok sometimes is that it's not just one thing to do. One IP lives in many locations. You also need to make sure that, well, if you have one IP in many locations, and then those many locations need to the exact same services because you don't control where that traffic lands and what traffic for which service so you need to make sure that all of those services run on all of your machines and that's always been that guiding principle again for how we've run things at cloudflare so a key thing here in in the the things that we talk about externally and internally is anycast is how your isp gets you to you know the best location for for cloudflare right like Europe, that can be between Brussels, between London, between Frankfurt, between Amsterdam, between Paris.

[4:46] That's where Anycast guides that traffic to make sure that we get things there. And then the critical component for Cloudflare for us ourselves is ECMP, that's Equal Cost Multipath. And that is one of the core components or one of the core technologies that we leverage to do load balancing internally to make sure that we can service as many requests as we need at the right time. But, you know, since 2010, always been using BGP, always been trying to make sure that we're always using the open standards. But since 2010 to about 2016, 2017, that network had grown massively. Originally, the network started very small and in four to five locations with a couple of networks to directly interconnect with and a bunch of transit providers to make sure that...

[5:36] could reach the rest of the internet right and that was workable was scalable but the network kept growing and one of the big problems that um we kept running into was people in tokyo were getting routed to san jose even though that there was a tokyo pop and that was because they were interconnecting better in san jose in in california with the the local japanese networks than they were in in tokyo which meant that a bunch of that traffic started routing to the wrong locations. From a topology perspective, from a BGP or a network topology perspective, it was ideal. It was the shortest path, and it was perfect. But that meant that we needed to start making changes in the way that prefixes were being advertised. The way that's done in BGP, that's done through one of the easiest way of doing this.

[6:25] Sorry, there's multiple ways, and we're not going to do a deep dive on those. But one of the easiest way of doing this is ASPath prepend. What you can see here is for one of the prefixes that we advertise, 104 .20 .240 .20, we have 1.333.5, that's Cloudflare, occurs three times in there. So we are prepending twice before handing over to our transit providers. At that point in time, that made sense because that way we could manipulate how people were or eyeballs were ingressing into the Cloudflare network just by doing ASPath prepends. And it worked. But over time, those prepens became kind of pointless because the deployments were getting so widely varied and there were so many networks that we could directly interconnect with that those prepens didn't actually make sense anymore.

[7:16] And we actually needed to get everything equal to make sure that traffic was routing correctly. So about 2016, 2017, every single prefix from Cloudflare was... prepended equally on every single transit provider so at that point your prepends were useless right because everything just had the exact same equal length on every single transit provider there wasn't much point to that anymore so one of the problems with that though is once that started happening whenever we were turning up a new location we were at risk of advertising the anycast prefixes without any pre to one of our transit providers. What then instantly happened is something that became colloquially known as SJCcast when you're sending all of your traffic to San Jose, or TampaCast when all of the traffic arrives in Tampa.

[8:12] Even at that time, Cloudflare 2017, none of our locations were sized to handle all of Cloudflare's traffic in a single location. And that then turned into delightfully lovely incidents that we'd done. about externally and publicly and this is me in 2018 this was the first talk i ever done for cleftler at ripe talking about exactly this problem um and and this was a growing pain it was a a lesson we had to learn on organically growing a network and that's fine and then we end up in this position and then all of a sudden you're off in a much worse place than you'd to be and the idea was well we need to fix this and we did and we went through a major engineering effort and we removed all of our pre -pens in the span of about 15 minutes and that's what that talk is about um and that it was mostly a ton of fun talking about the automation and the systems that we put in place to allow us to do that but again that was growing and that was evolving and getting right place and one of the other things that needed to get to the right place was the way that we internally looked at the prefixes that we are advertising and how we kind of distribute those internally all right time to double down on the pain prod cfb the whole network was one database

[9:44] so if you work at cloudflare you might be laughing slash crying at this meme If you don't work at Cloudflare, here's the short of it. ProdCFP was a single shared configuration store for the entire network. It held two kinds of things, keys and values. And those could mean almost anything. And so that means our prefix, advertisement settings, and our load balancing settings lived in the same place. That flexibility helped us early on, but it's also what hurt us later.

[10:17] Here's how one engineer characterized the process. It's actually got three complaints in one. I aspire to this level of complaining. So like chef's kiss as usual engineering. First, it's painfully slow. And reloading bird, that's the routing daemon, was a manual step you had to do by hand. And then finally nothing double checked it before it went live. So as you can see, mildly terrifying.

[10:41] Here's a picture of our load balancing at the time. So you can see this tall pillar with a bunch of. sized pieces sacked up and then kind of all propped up here by this little stored procedure. That's redistribute ECMP. And ECMP groups, this is the name of the process that decides which machines in a data center advertise which prefixes. And here's a screenshot of the actual wiki procedure, the SOP, to run this. And I can see why people hated it. And you can see the risk severity if you squint here, says major. So basically what it would do is read a bunch of state, write a small number of changes, and then trigger a whole lot of reloads on the machines. And every reload was a chance for something to go wrong.

[11:26] So again, terrifying.

[11:29] This graph is from about five or six years ago, and this kind of visualizes that process. So each row is a machine, and each column is a prefix, and then each color represents a different generation of machine. And so you can see at the time, Gen 9 was our newest, so you see kind of the most activity there. And on paper, it made sense, right? Send the most traffic to the machines with the most capacity. However, in practice, it was scattered. We had prefixes all over the shop. And it was anytime you had to make a change, it was hard to reason about, hard to change and hard to fix when load was uneven.

[12:05] So we did three things. First, we adopted RPKI to cryptographically. validate which network is allowed to originate a prefix then we standardized and shortened our as paths instead of hand tuning prepens per prefix prepens per prefix if you want a tongue twister and finally we normalized the length so instead of this patchwork of sizes we said for instance let's slash 20 everything so just quick recap where we're at picture it 2018 the before times we had 150 locations cute right 74 countries huh and only 250 IPv4 prefixes.

[12:41] All right, so we've talked about the pain. Let's get in a little bit of some of the solutions and how we started growing up. Intro Unimog, my fave. This is our layer four load balancer. We started removing those bits and pieces from that tall, wobbly, scary-looking Jenga pillar we showed you of the load balancing earlier. And we removed that active load balancing from prod CFP using Unimog. So each line on this... represents the machine's CPU. And so you can see before the white line where Unimog was introduced right here, it's quite scattered. Each machine generation is doing different amounts of work. After the line, it bunches together, becomes a lot more uniform. That's Unimog doing the load balancing inside of a data center.

[13:28] But we'd not moved everything over. One year later, we introduced Colo Topologies, a flexible... expression parser that allowed us to express a set of locations from which we'd advertise a prefix from. So before, it was a static list of colo ID arrays in our CIDR distribution list, and it had to be painstakingly messed with by an engineer, a poor engineer, who had to make some edit to it and pray nothing went wrong, right? From there, it just became a simple edit to a rule, an expression, where the locations would kind of figure out the rest based on the rule that you

[14:02] All right, now I'll hand it over to Tom to talk about the traffic management enhancements.

[14:07] So like I said earlier, one of the big problems we have at Anycast or one of the big joys we have at Anycast is it's very simple for us to turn up because we literally go to any other place, connect with any other network, and they automatically basically route to the correct location for us. But we don't always want that traffic in that location. Getting capacity in the right locations at the right time is an incredibly complicated. a very complex problem and especially nowadays with lead times going into the the months of eight months nine months 12 months sometimes impossible to get any sort of capacity we started building a system out that allowed us to actually forward some of that traffic that was landing in one location to a different location where we did actually have spare capacity that allowed us to get a much smoother deployment going on the capacity that we want capacity that we need and capacity that we have available But Traffic Manager alone didn't really change everything all that much.

[15:03] One of the big things that changed massively for Cloudflare in the way that we advertised prefixes was the fact that not all of the prefixes that we were advertising are actually our own prefixes. In 2019, 2020, we started a system or a project or product really named BYUIP. It allowed customers to bring their own IP addresses, their own prefix. do Cloudflare. And since we started this, and you can see this is January 2020, all the way up to October 2022, in just that span of two years, we provisioned over 12 ,000 different prefixes onto the Cloudflare network. Obviously, that is not prefixes that were advertised all at once, all at the same time, but that is just prefixes for customers that we were ready to start provisioning.

[15:50] And that is massive growth. growth was enabled mostly and uniquely by the Addressing API. The Addressing API was an internal system because one of the engineers on another team, on the DNS team at the time, understood and realized that prefix management is actually going to be a massive concern and a technical problem that we'll need to address both internally for ourselves, for the 250 -ish prefixes that we were advertising and the way that we were doing load balancing. as well as for our customers. We were getting more and more enterprise customers that wanted to be in control of even more of their own stack which included being in control of their prefixes that wanted to onboard onto Cloudflare services but then couldn't because you know they have different they wanted to control over their prefixes.

[16:37] So we started building BYR IP and it's been a massive success but one of the big things with success is it can have unfortunate consequences and this is what we have to do with the future and this is what we have to do with the is one of the unfortunate consequences we ran into we had one big customer on board more than 10 000 of their own prefixes and our routers were unfortunately not equipped to deal with that massive influx of of prefixes at least not in the the way that we had implemented it at the time so when the customer onboarded you can see this obviously we want to push these things out quite quickly to the edge so there was not a whole lot of space and time between when the first routers started becoming difficult to when most of them started following so we immediately started a task force to make sure that we could understand what was causing this problem how can we onboard this customer safely without bringing up the rest of the network and how can we make sure that we can keep doing this scalably going forward because obviously we want to keep selling this as a product this can be oh yeah one more customer and we'll stop um that's not how we work so instead what we started doing was scale down and rethink the way that we're doing advertisements and the way that we're doing load balancing um one of our key things with ecmp was we wanted to split that into even more prefixes so we disaggregated our prefixes that blows up the the routing table internally for that load balancing so one of the big changes

[18:08] that we made was what if we just let our compute nodes with bird advertise to our routers and then we just upstream that directly to our peers and our transit providers without any form of aggregation on our edge and that's exactly what we did and just that alone allowed us to completely remove that memory overhead and go back down to kind of the operational overhead that we have from before um these are massive technical changes that involved a lot of different teams and a lot of different um engineers to kind of get us over the line. But this allowed us to scale

[18:51] without any further issues ever since. So now the thing is, like, where are we now? So let's have a look at the numbers.

[19:04] All right. And so if you remember back at the 2018 slide, the cute one, we've now grown a ton. years later, we're at 346 cities. They're over almost, let's see, 14 ,000 networks directly connected to Cloudflare with 590 TPS and 31 DDoS attack mitigation in 35 seconds in 2025 with no engineer page. So we're able to handle a lot more load now as it's all distributed. We have more footprint than ever. So in terms of how our IP address strategy involved, you can see here We hand -tued those AS paths, and we changed how we did ingress prefix advertisement to RPK -evalidated, like I said earlier, automated, health and capacity -driven.

[19:50] We standardized. We went from one building, one colo, to our multi -colo pops, and we introduced traffic manager, egress, soft, any cast addressing. And we scaled our peering relationships, so over 13,000 direct network connections.

[20:08] so what's next we actually want to hear that from you we have a roadmap but we would love to understand what exactly you want from our network from bring your own ip from kind of if you're a cloud for a customer what are you looking for in this regard all right cool all right thank you

## Related sessions

- [Rearchitecting the Cloudflare network for the AI revolution](https://www.cloudflare.com/connect/2026/session/rearchitecting-the-cloudflare-network-for-the-ai-revolution)
- [Breaking the Internet (and putting it back together)](https://www.cloudflare.com/connect/2026/session/breaking-the-internet-and-putting-it-back-together)
- [6 architecture patterns that repeat across enterprise customers](https://www.cloudflare.com/connect/2026/session/6-architecture-patterns-that-repeat-across-enterprise-customers)

## More from Connect

- [All sessions](https://www.cloudflare.com/connect/)
- [Get notified for Connect 2027](https://www.cloudflare.com/connect#2027)