Ensuring resiliency during network maintenance

Network & Infrastructure

About the session

Designed for network and infrastructure leaders, this session explores how Cloudflare maintains resiliency and data integrity while expanding our massive global footprint. Upgrading a rapidly growing network requires rigorous risk analysis and scheduling algorithms to maintain seamless operations. Learn how our engineering teams tackle this issue by replaying historical data via backtesting to prevent recurring issues and by coordinating capacity increases and maintenance schedules. We also share how our stateful edge products, such as Cloudflare R2 Storage, use maintenance-aware placement to automatically rebalance workloads to ensure availability and performance during major hardware migrations. Join us to learn how our infrastructure investments translate into a more resilient platform for your workloads.

View transcript

0:02My name is Kevin Deems and I'm excited to share with you how Cloudflare ensures resiliency during network maintenance. I spent the last four years at Cloudflare specializing on making automated infrastructure that means there's no compromise between resiliency and operational and developer velocity. I'm going to take you. through a little story at Cloudflare here, starting with why we actually do network maintenance. What are some of the risks that might happen when we're doing this on infrastructure that lives at the edge? How do we build our risk analysis engine to make sure that we mitigate any of these possible risks from happening? And then pulling it all together with how we actually build all of this automated resiliency into our actual products to make sure that we improve operational velocity in making this all fully resilient.

1:01What you're looking at here, you may have seen many times at this event already. We were seeing the dots on how much infrastructure lives at the edge. We have over 340 plus cities today with physical data center infrastructure. And you can see these little red X's popping up. Some of these are in two categories. One of them is we can't control, an ISP goes. down, for example, or the internet is just flaky. Public internet is like that. And on the other side, we're actually intentionally taking down small portions of capacity in order to do network maintenance. You may be thinking though, why even have this talk? Why don't we just keep all of our hardware infrastructure running forever until it just inevitably dies, so we don't need to do maintenance? Well, that's pretty risky.

1:48We have to do security tuning, performance tuning. um obviously it's really important to make sure that we're doing sub server reboots as well to make sure that we're actually making the most efficient use of our hardware resources we're squeezing a very large amount of performance out of all of this hardware and we really want to make sure it's up to date with all of the upgrades that you might need so in 2025 we did over 2200 maintenance events individual actions by 2020 last I checked, it was over 2 ,800. Scoping this out to the full 2026 year, we see a 50 % increase year over year. This is in part due to two reasons. We're giving our users and our entire set of customers more capacity at the edge.

2:36Like I said last year, the number of 340 plus cities would be much less. So we're actually giving more capacity in different places around the world. Then the other side was we're just automating a lot more of our operations. So we're making this more efficient. We're scaling our human operations specialists and using automation to make sure that we're delivering more capacity and more maintenance in a single period of time.

3:06Now, I got to take a step back though. Cloudflare operates something called an Anycast network. In other words, what this means is any of our services can run in any of these physical... data centers here. Now the important thing for let's take a product like Cloudflare's cache or CDN. The important thing to know is when your user comes to Cloudflare on your behalf they're actually trying to get a physical asset so we need to have all of that physical cache infrastructure for example a image or an html file sitting in that data center. So it's a combination of two things we have the service running and then the actual data can be into that specific location.

3:48This also means we can route around that failure seamlessly. So any of the internet failures that I talked about, we can easily just move to the next closed data center. There's no impact to performance for your users. And this is fantastic. It's really magical from a customer's perspective.

4:06This changes though. Now what we're looking at is something like an R2 product. R2 is our object storage system. Customers give us a tremendous amount of data. And so all of this data actually lives on the edge. And so in different storage devices. Now, what this also means is that because it's such a tremendous amount of data, we can't feasibly copy it to every single data center. So what we do is we need to still make sure that there's actual data redundancy. You know. Like I said, the internet is a crazy place. We've got to make sure that we're resilient to an entire site going down, for example, like a physical data center site. Now, what we do is we actually wrap your data and your copies of your data into something we call a little home for your data.

4:58So you can see these little homes represent data centers, and we wrap it inside a cluster. So this cluster means that your data would be copied inside that cluster across. many data centers but it wouldn't be like spread across different clusters so now this means that in a situation where we're trying to do network maintenance as well we have to make sure we have enough capacity online inside each individual cluster so we can't do too much simultaneous maintenance inside that cluster so I want you to think back to the number 2 ,800 maintenance can you think of a time where you actually knew what all of your colleagues at your company were doing in any day, I certainly can.

5:43Now, can you imagine all of these operations specialists trying to do maintenance in their head, trying to calculate whether their action is going to be safe or not? Well, they can't take all of this into account. There's just too many operations teams, departments that are spread. And so really, they need a system that allows them to know whether they're going to overlap is what call them, simultaneous maintenance actions inside a cluster. In this scenario, there's no risk. This is totally fine. We can do the little wrenches represent a piece of physical operations maintenance. And we can do this in two different clusters at the same time. No problem whatsoever.

6:26This is what operators want to know. They want to know if their planned maintenance action is actually going to overlap. with someone else at the company, for example. So we need to be able to tell them. So how do we solve this? So the first solution you might be thinking is, well, it's trivial. Why don't we just tell them when it's safe or not safe? And then we just reschedule their event to two days later, for example. So we'd look at these top two maintenance operators and we just tell them one of you needs to move. And then we move them and then suddenly there's no conflict anymore. The one on the bottom doesn't need to move because it's already has no conflict so there is no issue.

7:11Okay, so how do we actually solve this problem? This is great at a high level. Let's break down one of these clusters and simultaneous actions trying to figure out how do we know whether something is safe or unsafe. Okay, so these two operators want to do rack maintenance and this is physical rack in two different data centers. This represents a physical topology graph. What I mean by that is a data center holds racks. Those racks have network devices inside of them. And then those network devices provide network capacity for specific servers. So we can create what's called an impact radius, where if I'm taking down a rack, I know that the servers below it would be impacted. This is not enough to know in both data centers, safe or not? This is just a physical representation.

7:57How do I actually know that those servers are related in any way? Okay, so now the only place that actually knows there's a relationship between those servers is the storage cluster itself. It knows that those two servers have access to the same copy of data. So they are storing your data redundantly on those servers. Well, now we know immediately they cannot do that same meeting. at the same time. And we use this. This is basically what's called the graph. But this is not the only situation we need to take into account. Now, what can also happen is, let's say there's no simultaneous plan maintenance. We've solved that problem, let's say. Five minutes before that operator is going to do their maintenance, let's say a set of network cables dies.

8:52We can't we have to be able to handle that scenario even if we have to plan for things going down. This is the same impact radius, as I said before with the planned operation, but this is just something that we can't control. So what happens here is we're actually making sure that the operator doesn't go through with their maintenance if the real-time health from our edge, we're ingesting all of this data from every single data center to decide whether something is healthy. So this specific product or the network cables in a healthy position. And then we can correlate them using this data. Okay, so let me look at a different view of that timeline that I talked about. So we can replay all of this data because, as I said, it's real time.

9:41So we can put it in a time series database and say at 9 a .m. on August 6th, they tried to do that maintenance. Well, it overlapped. They've rescheduled that operation safely. And then they were told five minutes before that they can't do it again. Rescheduled once again. And then finally, it completes safely. We can replay all of this using a process called backtesting. So we apply algorithms and our risk mitigation engine on top of historical data to replay scenarios. Now, we also do this in continuous integration. So this is... actually much more robust than unit or integration tests because we're actually using raw production grade data inside our continuous integration environment.

10:26We're trying to see if that scenario of the individual cables going down, that's one backtest scenario. It's one of many possible scenarios that could go wrong in a data center. So we replay all of these possible scenarios to make sure that we would say that it's not safe at that specific time.

10:50everything i've talked about already has focused primarily on how do we keep ourselves resilient so let's say now we are fully resilient in this product but everything i've said is now come at the expense of operational velocity how much maintenance can i get done in a specific period of time remember back to the timeline i showed we were rescheduling all of the operations, each one of those is very costly for operational toil. So let's look at a different view of the same cluster. How do we actually move faster with the same resiliency? In this scenario here we have many different data centers which have your data copy and then one in the center that has free capacity. So those storage devices don't have your data on it and we want to see if it's possible to do those two sides.

11:41actions at the same time by copying the data from one of those data centers into the one that has free capacity ahead of the plan maintenance so that we increase the redundancy and eliminate the risk of that simultaneous maintenance having anything take down your customer data. Okay so let's break down what are the inputs required to actually tell us if we can do this operation whether we need to block it or not. Okay, so on the top left here we know that there's 300 terabytes and on the top right data center there's 150 terabytes. Well you can't copy 300 terabytes into a 200 terabyte data center. Just doesn't make any sense. So immediately you know that we have to focus on moving the data from the top right data center into the one in the middle. Great.

12:27So now how much network capacity are we going to allocate to this specific maintenance operation. So let's say we do 10 gigabits per second. We're just gonna allocate this is not the whole capacity of your data center being able to copy to the middle data center, but it's some portion of it. So here, what you might also notice is that we're actually pooling the network resources from many different data centers in order to complete this action. Because as I said, that data copy exists in many data centers, so we can not just have an arrow going from the top right to the middle, but many data centers. Pushing data into that middle data center and what we note here is that it takes about a day and a half Let's say we can compute already that it would take that long. This is at optimal network capacity

13:20So now the final piece is how far in advance is the maintenance operator giving us to do this action? Is it even possible to do it in the time frame that they want if they only give us 24 hours? We can immediately tell them 36 hours of data copying cannot be done in 24 hours. Doesn't make any sense. So we can't rebalance. But what if we told the operators that we wanted to actually change their operational schedule so that now they do things two days in advance? That's a relatively trivial change for their scheduling policies. And suddenly now they know they have an incentive because now they can do two simultaneous maintenances at the same time. This had no impact. to our resiliency but it had a very high impact to our operational velocity remember this is giving more customers more capacity at the edge that's the impact of operational velocity here we want to make sure that we are not like i said compromising the velocity or resiliency for our customers i want to leave you with three takeaways cloudflare takes extreme care and investment to make sure that we have that safe balance of resiliency and efficient operations with all our products we make automated risk mitigation measures to make sure that we are doing all of these decisions and again this is using cloudflare's wealth of network data all over the edge we're computing all of this information in order to tell whether something's safe and the final piece is this as you continue to scale on

14:53cloudflare's global network all of this mission critical infrastructure is yours default you never need to worry about server reboots performance tuning or security patching all of this is immediately done for you and this operational excellence is in our system by default so thank you so much

Browse all
Connect 2027

Connect returns in 2027

Be the first to know when registration opens and the agenda goes live.

By submitting this form, you agree to our Privacy Policy.

cloudflare-connect-2026

---
title: "Ensuring resiliency during network maintenance"
url: "https://www.cloudflare.com/connect/2026/session/ensuring-resiliency-during-network-maintenance"
year: 2026
---

# Ensuring resiliency during network maintenance

[Watch session](https://www.cloudflare.com/connect/2026/session/ensuring-resiliency-during-network-maintenance)

## Session details

- **Track:** Network & Infrastructure
- **Topics:** Performance & resiliency, Network modernization & multicloud
- **Solution areas:** Core Platform

## About this session

Designed for network and infrastructure leaders, this session explores how Cloudflare maintains resiliency and data integrity while expanding our massive global footprint. Upgrading a rapidly growing network requires rigorous risk analysis and scheduling algorithms to maintain seamless operations. Learn how our engineering teams tackle this issue by replaying historical data via backtesting to prevent recurring issues and by coordinating capacity increases and maintenance schedules. We also share how our stateful edge products, such as Cloudflare R2 Storage, use maintenance-aware placement to automatically rebalance workloads to ensure availability and performance during major hardware migrations. Join us to learn how our infrastructure investments translate into a more resilient platform for your workloads.

## Speakers

- [Kevin Deems](https://www.cloudflare.com/connect/speakers/2447009) — Senior Software Engineer, Cloudflare

## Transcript

[0:02] My name is Kevin Deems and I'm excited to share with you how Cloudflare ensures resiliency during network maintenance. I spent the last four years at Cloudflare specializing on making automated infrastructure that means there's no compromise between resiliency and operational and developer velocity. I'm going to take you. through a little story at Cloudflare here, starting with why we actually do network maintenance. What are some of the risks that might happen when we're doing this on infrastructure that lives at the edge? How do we build our risk analysis engine to make sure that we mitigate any of these possible risks from happening? And then pulling it all together with how we actually build all of this automated resiliency into our actual products to make sure that we improve operational velocity in making this all fully resilient.

[1:01] What you're looking at here, you may have seen many times at this event already. We were seeing the dots on how much infrastructure lives at the edge. We have over 340 plus cities today with physical data center infrastructure. And you can see these little red X's popping up. Some of these are in two categories. One of them is we can't control, an ISP goes. down, for example, or the internet is just flaky. Public internet is like that. And on the other side, we're actually intentionally taking down small portions of capacity in order to do network maintenance. You may be thinking though, why even have this talk? Why don't we just keep all of our hardware infrastructure running forever until it just inevitably dies, so we don't need to do maintenance? Well, that's pretty risky.

[1:48] We have to do security tuning, performance tuning. um obviously it's really important to make sure that we're doing sub server reboots as well to make sure that we're actually making the most efficient use of our hardware resources we're squeezing a very large amount of performance out of all of this hardware and we really want to make sure it's up to date with all of the upgrades that you might need so in 2025 we did over 2200 maintenance events individual actions by 2020 last I checked, it was over 2 ,800. Scoping this out to the full 2026 year, we see a 50 % increase year over year. This is in part due to two reasons. We're giving our users and our entire set of customers more capacity at the edge.

[2:36] Like I said last year, the number of 340 plus cities would be much less. So we're actually giving more capacity in different places around the world. Then the other side was we're just automating a lot more of our operations. So we're making this more efficient. We're scaling our human operations specialists and using automation to make sure that we're delivering more capacity and more maintenance in a single period of time.

[3:06] Now, I got to take a step back though. Cloudflare operates something called an Anycast network. In other words, what this means is any of our services can run in any of these physical... data centers here. Now the important thing for let's take a product like Cloudflare's cache or CDN. The important thing to know is when your user comes to Cloudflare on your behalf they're actually trying to get a physical asset so we need to have all of that physical cache infrastructure for example a image or an html file sitting in that data center. So it's a combination of two things we have the service running and then the actual data can be into that specific location.

[3:48] This also means we can route around that failure seamlessly. So any of the internet failures that I talked about, we can easily just move to the next closed data center. There's no impact to performance for your users. And this is fantastic. It's really magical from a customer's perspective.

[4:06] This changes though. Now what we're looking at is something like an R2 product. R2 is our object storage system. Customers give us a tremendous amount of data. And so all of this data actually lives on the edge. And so in different storage devices. Now, what this also means is that because it's such a tremendous amount of data, we can't feasibly copy it to every single data center. So what we do is we need to still make sure that there's actual data redundancy. You know. Like I said, the internet is a crazy place. We've got to make sure that we're resilient to an entire site going down, for example, like a physical data center site. Now, what we do is we actually wrap your data and your copies of your data into something we call a little home for your data.

[4:58] So you can see these little homes represent data centers, and we wrap it inside a cluster. So this cluster means that your data would be copied inside that cluster across. many data centers but it wouldn't be like spread across different clusters so now this means that in a situation where we're trying to do network maintenance as well we have to make sure we have enough capacity online inside each individual cluster so we can't do too much simultaneous maintenance inside that cluster so I want you to think back to the number 2 ,800 maintenance can you think of a time where you actually knew what all of your colleagues at your company were doing in any day, I certainly can.

[5:43] Now, can you imagine all of these operations specialists trying to do maintenance in their head, trying to calculate whether their action is going to be safe or not? Well, they can't take all of this into account. There's just too many operations teams, departments that are spread. And so really, they need a system that allows them to know whether they're going to overlap is what call them, simultaneous maintenance actions inside a cluster. In this scenario, there's no risk. This is totally fine. We can do the little wrenches represent a piece of physical operations maintenance. And we can do this in two different clusters at the same time. No problem whatsoever.

[6:26] This is what operators want to know. They want to know if their planned maintenance action is actually going to overlap. with someone else at the company, for example. So we need to be able to tell them. So how do we solve this? So the first solution you might be thinking is, well, it's trivial. Why don't we just tell them when it's safe or not safe? And then we just reschedule their event to two days later, for example. So we'd look at these top two maintenance operators and we just tell them one of you needs to move. And then we move them and then suddenly there's no conflict anymore. The one on the bottom doesn't need to move because it's already has no conflict so there is no issue.

[7:11] Okay, so how do we actually solve this problem? This is great at a high level. Let's break down one of these clusters and simultaneous actions trying to figure out how do we know whether something is safe or unsafe. Okay, so these two operators want to do rack maintenance and this is physical rack in two different data centers. This represents a physical topology graph. What I mean by that is a data center holds racks. Those racks have network devices inside of them. And then those network devices provide network capacity for specific servers. So we can create what's called an impact radius, where if I'm taking down a rack, I know that the servers below it would be impacted. This is not enough to know in both data centers, safe or not? This is just a physical representation.

[7:57] How do I actually know that those servers are related in any way? Okay, so now the only place that actually knows there's a relationship between those servers is the storage cluster itself. It knows that those two servers have access to the same copy of data. So they are storing your data redundantly on those servers. Well, now we know immediately they cannot do that same meeting. at the same time. And we use this. This is basically what's called the graph. But this is not the only situation we need to take into account. Now, what can also happen is, let's say there's no simultaneous plan maintenance. We've solved that problem, let's say. Five minutes before that operator is going to do their maintenance, let's say a set of network cables dies.

[8:52] We can't we have to be able to handle that scenario even if we have to plan for things going down. This is the same impact radius, as I said before with the planned operation, but this is just something that we can't control. So what happens here is we're actually making sure that the operator doesn't go through with their maintenance if the real-time health from our edge, we're ingesting all of this data from every single data center to decide whether something is healthy. So this specific product or the network cables in a healthy position. And then we can correlate them using this data. Okay, so let me look at a different view of that timeline that I talked about. So we can replay all of this data because, as I said, it's real time.

[9:41] So we can put it in a time series database and say at 9 a .m. on August 6th, they tried to do that maintenance. Well, it overlapped. They've rescheduled that operation safely. And then they were told five minutes before that they can't do it again. Rescheduled once again. And then finally, it completes safely. We can replay all of this using a process called backtesting. So we apply algorithms and our risk mitigation engine on top of historical data to replay scenarios. Now, we also do this in continuous integration. So this is... actually much more robust than unit or integration tests because we're actually using raw production grade data inside our continuous integration environment.

[10:26] We're trying to see if that scenario of the individual cables going down, that's one backtest scenario. It's one of many possible scenarios that could go wrong in a data center. So we replay all of these possible scenarios to make sure that we would say that it's not safe at that specific time.

[10:50] everything i've talked about already has focused primarily on how do we keep ourselves resilient so let's say now we are fully resilient in this product but everything i've said is now come at the expense of operational velocity how much maintenance can i get done in a specific period of time remember back to the timeline i showed we were rescheduling all of the operations, each one of those is very costly for operational toil. So let's look at a different view of the same cluster. How do we actually move faster with the same resiliency? In this scenario here we have many different data centers which have your data copy and then one in the center that has free capacity. So those storage devices don't have your data on it and we want to see if it's possible to do those two sides.

[11:41] actions at the same time by copying the data from one of those data centers into the one that has free capacity ahead of the plan maintenance so that we increase the redundancy and eliminate the risk of that simultaneous maintenance having anything take down your customer data. Okay so let's break down what are the inputs required to actually tell us if we can do this operation whether we need to block it or not. Okay, so on the top left here we know that there's 300 terabytes and on the top right data center there's 150 terabytes. Well you can't copy 300 terabytes into a 200 terabyte data center. Just doesn't make any sense. So immediately you know that we have to focus on moving the data from the top right data center into the one in the middle. Great.

[12:27] So now how much network capacity are we going to allocate to this specific maintenance operation. So let's say we do 10 gigabits per second. We're just gonna allocate this is not the whole capacity of your data center being able to copy to the middle data center, but it's some portion of it. So here, what you might also notice is that we're actually pooling the network resources from many different data centers in order to complete this action. Because as I said, that data copy exists in many data centers, so we can not just have an arrow going from the top right to the middle, but many data centers. Pushing data into that middle data center and what we note here is that it takes about a day and a half Let's say we can compute already that it would take that long. This is at optimal network capacity

[13:20] So now the final piece is how far in advance is the maintenance operator giving us to do this action? Is it even possible to do it in the time frame that they want if they only give us 24 hours? We can immediately tell them 36 hours of data copying cannot be done in 24 hours. Doesn't make any sense. So we can't rebalance. But what if we told the operators that we wanted to actually change their operational schedule so that now they do things two days in advance? That's a relatively trivial change for their scheduling policies. And suddenly now they know they have an incentive because now they can do two simultaneous maintenances at the same time. This had no impact. to our resiliency but it had a very high impact to our operational velocity remember this is giving more customers more capacity at the edge that's the impact of operational velocity here we want to make sure that we are not like i said compromising the velocity or resiliency for our customers i want to leave you with three takeaways cloudflare takes extreme care and investment to make sure that we have that safe balance of resiliency and efficient operations with all our products we make automated risk mitigation measures to make sure that we are doing all of these decisions and again this is using cloudflare's wealth of network data all over the edge we're computing all of this information in order to tell whether something's safe and the final piece is this as you continue to scale on

[14:53] cloudflare's global network all of this mission critical infrastructure is yours default you never need to worry about server reboots performance tuning or security patching all of this is immediately done for you and this operational excellence is in our system by default so thank you so much

## Related sessions

- [Rearchitecting the Cloudflare network for the AI revolution](https://www.cloudflare.com/connect/2026/session/rearchitecting-the-cloudflare-network-for-the-ai-revolution)
- [Breaking the Internet (and putting it back together)](https://www.cloudflare.com/connect/2026/session/breaking-the-internet-and-putting-it-back-together)
- [Cloudflare’s evolution of managing IP addresses for customers at scale](https://www.cloudflare.com/connect/2026/session/cloudflares-evolution-of-managing-ip-addresses-for-customers-at-scale)

## More from Connect

- [All sessions](https://www.cloudflare.com/connect/)
- [Get notified for Connect 2027](https://www.cloudflare.com/connect#2027)