Canary to cloud: Building a flexible traffic strategy
Oscar Moreno Garza, Noah Crouch
Uber · Cloudflare
0:02All right. Hi, everybody. Welcome to breaking the internet and putting it back together again. We're going to play a short video to start.
0:24So while you were sleeping, a global outage brought tech systems to their... An outage at Cloudflare cut off access to several major websites.
0:34Cloudflare runs one of the world's largest networks, handling around one -fifth of web traffic. 5.00 Error and Internet Server Error.
1:04and the very first thing i do is apologize and i take personal responsibility and i really mean it and then we all start full transparency we tell them exactly what happened if you boil it down incident process at cloudflare is really simple this we sure how we we learn as fast as possible that something is wrong we get everyone together we fix it then we report, and then we follow up rigorously. What is the work that we need to do to ensure that this can't happen again? A bug fix is not a prevent sticker. Now, a bug fix with a test is.
1:45We're gonna keep drilling as we keep wanting to get better and better at this.
1:57Hi, my name is Jeremy Hartman, I run infrastructure engineering here at Cloudflare. We're going to start off with our report card. Our goal is five nines, it's a fairly audacious goal, after Code Orange. We really leaned into release managing and how we're doing things safely and fast across our network. There should be another graph that's up into the right. Well, what does it really mean? So we're more change, it's faster, and it's safer all at once.
2:31The frequency of those decreasing. This is over time, over the past six years. We will keep at this until this gets down to zero.
2:46Above all, keep at... high for our customers and high empathy for our teammates and I know that I know that we can do all that because we've done it many times before
3:08oh that was a problem all right all right thank you everybody breaking the internet often makes the news. Putting it back together again doesn't, and that's what we're here to talk about today. Sort of an inside look at how it gets put back together again. My name is Alex Krivet. I work in product here at Cloudflare. Almost a year ago today, on November 18th, a database permissions change doubled the size of a bot management file. The file reached the whole network very quickly. The proxy hit a hard limit, and for a couple of hours. Much of Cloudflare's traffic got an error page. A few days later, a change made to protect folks from a React vulnerability tripped a years -old bug.
3:55And about 28% of HTTP traffic failed for about 25 minutes. Those were two different bugs, but sort of had the same pattern. There were one change everywhere happening in seconds. And so we declared code orange, fail small. And for two quarters... resiliency came sort of before everything else. We changed how configuration ships, how our systems fail, how engineers get in, and when everything is down, and how we talk to our customers during that incident. That program is now done, but the work isn't. And many of you here today sort of lived those incidents with us. And so this isn't necessarily a victory lap. But what we want to cover today is how. How does Cloudflare approach?
4:40which incidents, how they're run internally, how we balance speed and safety when we're fixing things for 20% of the internet, and how we learn from these issues and make sure that they don't happen again. In the video, you heard from our former head of engineering, Usman, describe the incident process, how it works, those four steps. And sort of later in that video, you heard from Jeremy Hartman to sort of show you how that process is. being run, and where we're at today. We want you to leave today's talk with an idea of what's happening internally when Cloudflare declares an incident, the process that we're running, and how things get unbroken. I'm joined today by this panel, this group of people.
5:29Brandon Harris is the VP of Engineering for Infrastructure. He leads SRE in our incident management process. David Tuber, he .. Director of Product for Infrastructure, the physical network, the automation under it, and the systems that ship changes safely. Noah Kennedy, the technical lead for service -to -service load balancing for our edge, and also works on making our foundational systems more reliable. Tom Strix, Principal Network Engineer, spent nine years helping build our global network to be resilient and quick to fix, and he still works in instance every day. All right. Thanks, guys. for being here today to talk through this. As a sort of beginning point, every incident sort of starts the same way, and somebody has to call it.
6:14Brandon, you oversee the incident process. How does Cloudflare declare an incident? I think what makes us super unique is that we allow anybody to declare an incident at any time. And you might be a little scared and wonder, how does that actually work? You know, well, So the implementation is very simple. You know, it's a chat bot, chat operations. I think everybody will have that here. But, you know, it's really the culture around it that makes it important. It's the fact that if you see something, you say something. Something doesn't look right. I'm going to raise an incident. We're going to get somebody to look at it, and we can quickly cancel that incident if, you know, perhaps nothing's wrong. Now, you know, that's the simple way of answering the question. The harder way is, well, there's a lot of data.
7:01behind declaring an incident and knowing when something has been gone off the rails or not working as intended. So we have signals that are, I think the last time we looked, it was over one and a half petabytes a day of information that we're looking at and consuming and determining are we outside of operating ranges. So from those signals, we've got people and staffs and teams that are on call. they receive a page. During that video you might have heard someone up on the stage get paged. Was that? That was me. Okay. So how many people here are on call? Raise your hand. All right. Did anybody hear that page? Did your heart rate go up just a little bit? You know, like, you know, so that's kind of, you know, what on call is about.
7:48But when we go back to, you know, raising an incident, there is a primary incident. or somewhere behind the scenes at Cloudflare. This is a group of volunteers, leaders within our company, who have signed up and said, I want to do more. I want to run the world's largest network and the incidents that occur on the world's largest network. Kind of scary, right? But they signed up for it, and we have 55 of them everywhere. And I think some of them are in this room and have taken part of this, so thank you. Then we go back into the telemetry. So it's about signals. It's about knowing when something has gone off the rails.
8:35We have four major signals that we know there are big problems within the network. If there is a global CPU drop or if there are authentication requests, those are the canaries in the coal mine. Those are automatic signals that get everybody engaged immediately. But yes, that's the long -winded answer going through like how do you raise an incident? It's very simple one chat operation Anybody can do it and we take it from there one one other thing. I want to just add is that like there are Signals that declare incidents but at the end of the day because anybody to anybody can declare an incident even you know All of us can but also our sales teams are going to market teams can declare an incident Our support teams can declare an incident. So it's super helpful because we generally see kind of three different incident types, right?
9:22We see incidents that are declared because a customer saw a problem and we're going to raise an incident, we're going to go figure it out. We see another group of incidents that are caused because an internal employee saw a problem, either they were doing some test or there was something that was broken, they're going to raise an incident, we're going to figure it out. And then the third bucket is what Brandon said, like the monitoring. We have automated systems that detect problems and if a threshold is breached, let's declare an incident and go figure it out. All of those are totally valid. Obviously, customers declaring incidents is something we want less of, and monitoring declaring incidents is something we want more of within reason. But that's kind of how the breakdown happens. So anybody's allowed to actually declare an incident, but when it's kind of a judgment call in sort of that gray area of, do we want this to be escalated or not, which side do you want people to err on?
10:15I mean, there's a chain of command, right? So these things don't work without, you know, some rules and guidelines up front. By default, the incident commander has, you know, the final say. So the person who was paged that says, is this a real problem? They're going to determine after talking with other team members and other people that are brought into the incident, is there actually an issue, right? Is there impact? So they're the ones who can, you know, ultimately say it's canceled. Now, occasionally, these things may run in. to a boundary of leadership, right? So there might be executive decision-making that can override that. And we do that when it makes sense. There could be a single customer who's impacted inappropriately, and we need to dive into that.
11:01But it may just be just a fraction of our requests, so we don't see necessarily the big issue, except we know that one customer might be in pain. And we also have like processes for support right so like not every customer problem turns into an incident but if we're seeing problems with a customer support ticket someone's reporting an issue it's really hard to diagnose it's really really urgent um we'll raise an instant and declare that and so um you know just trust 5 000 people to make the right choice all the time is scary as brandon said but it generally we found that people who are all trying to do the right thing by our customers are doing the right thing for our customers and for our operators to make sure that we're running the best network for our customers at all times.
11:48So a page goes out, an incident is declared, and everybody gets into a room somewhere. Tom, you're a regular sort of on the other end of that, those calls on the other end of those pages. Walk us through sort of what you do first. What is your process? Where do you decide to dig? How do you make sure that this is resolved? quickly. I mean, at the end of the day, it's basically just a decision tree, right? Like we have, and then Ntube's alluded to that earlier, we have like always a specific set of incidents that will always happen at the same kind of issues that we see, right? Like with, if it's a series of 522s that we start seeing a specific region, then it's likely a subseca or a network issue communicating with probably US East 1. It's always US East 1 or it's DNS. But When those issues happen, like, it's very easy and very quick to kind of understand where that problem lies, and we can start debugging, we get the network team involved, and that's kind of where we go.
12:43The more insidious ones are the problems that we're not as used to, right? Like, there used to be a whole set of problems at Cloudflare that we had to deal with on a weekly basis, or almost a daily basis, and that's no longer the case, and now they happen like once every two years, and we forget how to deal with these things. And that's where we have runbooks and that's where we have like kind of making sure that all of that experience that exists within people, that that also resides in documentation that we can kind of fall back on or dashboards. Like I think basically every single engineer at Cloudflare has at least one favorite dashboard. And that dashboard will usually be, this is how broken my service is right now.
13:28And that's, you know, usually what engineers will do is they'll go into an incident room and then they'll look at their dashboard. to make sure it's not them. So everybody has all these tools, all these dashboards, things that they're looking at to make sure it's not them, to help other people that are in that same room with you and knowing where to dig sort of first. As a responder of that page at that time, what's one thing that you would want to change about that process, if you could change one thing? More sleep. I think that's... always the the thing is is we do try with the on-call schedules at least for most of the more operational teams to make sure that it's a follow the sun system but that's not always and it's always possible and then you still have smes and then sometimes you do need to wake up people uh figuring out a way that we not have to do that that'd be great is there something that is that you found overtly useful about that sort of initial process it's it's a really good way if interested in understanding the entire Cloudflare stack to understand the entire Cloudflare stack.
14:34Because it's one of those things where if you don't know what the Cloudflare stack looks like, it's very hard to understand what the problem is or where the problem might lie. But once you start getting involved in these incidents, over time, you start seeing the same questions being asked for the same types of incidents. And from that, it becomes very simple or fairly trivial to kind of understand the stack at least to some level. That's always good fun. I tell all the new PMs who join my team, subscribe to the Incident channel. And for your area, go through the Incidents. You will learn more about your product. You'll learn more about your area running an Incident than you will do shipping anything. And I think that's generally true for also writing code as well. I know when you started, you were in Incidents all the time. Yeah, that is a really common bit of advice that I tell a lot of juniors joining is if you want to get really knowledgeable about how Cloudflare works.
15:26Just join instant rooms for anything vaguely close to your area. You'll learn so much about how things work, how things fit together, and what all the adjacencies are for your systems. This may be a good transition point, but Noah, when your system specifically is on fire, what do you need from people on that call first? Like, what helps you resolve that problem quickly? Yeah, so there's a big difference between an incident commander, as Brandon put it, and what I'm going to call an incident driver. So... If you're like a technical lead or just really anyone on a service team or an SME for something, you don't want to be the one who's doing all the coordination. You don't want to be the one like coordinating and handling customer comms, trying to determine exact impact. You want to be focusing on just debugging issues, digging into what the problems are, and figuring out what you need to do about it.
16:13And so generally, if it's your system that's having issues and you're the SME, you want support from the rest of the room and you want... folks to handle a lot of these sorts of issues that are not directly related to debugging exactly what went wrong and how we can get it back up. How do you go about getting that support from people maybe that you've never worked with before or that you've never even talked to before? Often these calls are virtual and there's 20 people in this call trying to debug, trying to coordinate these things like you're saying. How do you get that support that you need? Yeah, so there's generally there's always going to be an IC in the room so an incident commander we have an on-call rotation for that and so you're always going to have someone who is an incident commander who's been assigned there and so you can always go to them for help and generally they're one of the like main parties who are going to help you with things like customer comms um they'll help like they'll help you like pull on other folks so i don't necessarily need to worry about pulling in uh all sorts of other teams and getting into contact with them because generally an incident commander unless i'm doing that on that particular day is going to be handling that for me.
17:22An incident commander might be a loaded term. Brandon, what's an incident commander? Really, it's kind of a coordinator. It's the administrator. It's the orchestration of, you know, getting to stop the bleeding, right? Primary, you know, resolution is, you know, not necessarily what's the root cause. It's what do we need to do right now to make sure impact goes away? So that's their first job. Really, it could be anybody. And we've opened this up to everyone in our org today can become an incident commander. It's one command. You do it in chat. You say, I see me. And then now you are running the incident. There's a lot of responsibility, as I stated, in doing this. However, it's kind of a rote administrative process.
18:10You're logging status updates. You're communicating information to your customer support. You're driving the problem to solution. You don't stop when you hit a roadblock. You might run into a debugging impasse where, well, we're not sure if this is it. Your AI may not agree with my AI, or we're looking at the things together. The incident commander needs to disambiguate and get to objective reality quickly. So they need to ask the questions, you know. This is why managers are good at this. because they're really great at asking questions. So ask the question, get down to the five whys, get to what do we do right now, and can we do it right now? I like to think of an incident commander as like that app, that hinge ad, like the app that's designed to be deleted.
18:57Like an incident commander, if they're doing their job right, they're finding the person who's actually responsible, right? Like an incident commander is kind of like a triage unit. You're running this thing and getting people stable. and then giving it over to the person who's responsible for the long-term fixed root cause permits. And so a good incident commander can, you know, keep things going until we figure out what's going on. And then somebody says, oh, it's, it's Noah. So we'll give it to Noah and then Noah will drive to completion. I think, I think one thing I'd like to add, it is, it is not necessarily an authoritative position. It is more of a facilitative position, right? If you, the way we haven't modeled. internally uh the commander has the authority to make a decision but they don't really have authority right like they're they're pulling people in they're trying to solve a problem that that indemnifies them from being culpable for responsibility so they're they're delegating responsibility onto multiple streams and ensuring that we converge on the solution yeah like in a lot of these cases the uh ultimate decisions are often made by the technical driver who's sitting there going okay um i need us to do this next, because this is how we're going to stop the bleeding, and this is how this system works, and this is what we're going to do.
20:13And then the rest of the room looks at this, goes, okay, that makes sense, and let's go
20:21Two, you spent seven years, maybe, at Microsoft prior to joining Cloudsler in much of a similar What's universal about incidents and the incident process? They suck.
20:35I think that the thing that's super helpful, and incidents are hard, but one of my old managers used to say that an incident is like a very, very concentrated chip cycle. Something is broken, there's a problem, and you have to solve it very quickly. That's universally true. And I think that regardless of whether or not you have SLOs, SLAs, you have auto -paging, it's just someone yelling at someone else from across the room, there is a customer problem, and it needs to be. be solved very quickly and we are basically saying we are not doing other things and we are focusing on this one thing because that's that's we need to go do and that's universal whether you know you're at microsoft or google or cloudflare or a startup that you know has 100 users or a startup that has a thousand users it's going to be pretty universal we're doing this to solve customer problems fast is there something that unique about cloudflare's process that you've noticed um i really like, it's like Brandon said, like anyone can declare an incident. I really like that.
21:31I think that it means that it's like every, giving people that responsibility means that everybody is invested, right? Like whether you are, you know, the head of SRE or you're a product manager, a tech lead, or, you know, you're someone in graphic design, you are invested in making sure that Cloudflare is running at its, you know, the best it can be. I'm a little surprised you're not mentioning our speed of delivery to our public blog posts and post-mortems. That seems like a... I've blocked that out from my memory a little bit. Well, you know, you've been an author of many of these. I think that that's really helpful as well.
22:17Our transparency is very unparalleled. A lot of people... Actually, a really good example is there was a problem that we had a couple of months ago where we actually know... a problem with between us and a shared provider and we posted on our status page first and said hey there's a problem and we're looking into it and that was good because you want to talk about it but it had this weird side effect of everybody thought that cloudflare was the problem um and so there were a whole bunch of people coming to us and being like why are you broken why are you broken it's it's not us it's a shared dependency um and then the shared dependency posted like an hour later saying oh yeah we're broken up there um we're gonna fix that And that speed is very, very helpful for us because customers really, really want to know everything, all the time, as fast as possible.
23:05And so I think Cloudflare has really leaned into that, and that's very, very helpful. Code Orange gave us sort of a revamp of our communication strategy during incidents, sort of every on a clock, every 30 minutes, every 60 minutes, something like that. Why? So maybe this is a question for Brandon. Why is sometimes we're still looking into it or we're still fixing it or we're still trying to observe this better than no update at all? Yeah, this is, I mean, as a customer, you just want to know very likely, you know, like, what do I need to do and are you working on it? And if there's radio silence, you lose trust, right? And this is something we identified.
23:51We're not doing a great job getting that status out, even if it's the same status. I think internally we look at this as, well, there's nothing to report yet. We're still debugging. But we actually need to respond with, we're still looking into it. Let me take a straw poll here. Who here has read the Cloudflare status page during an incident? You don't count. So of those people who have read the Cloudflare status page during an incident, and please be honest here, because as the person who writes a lot of these... statuses. How many of you feel comfortable with the rate of status or the quality of the status during the instance? Raise your hand if you are comfortable with it. Okay, only a couple. There's not a lot of hands going up. Not a lot of hands going up. And I think that that's because like, and I'm going to pick on you in the front because you've raised your hand, you raised your hand both times. But like, what is for you as a stakeholder, what is the ideal like rate of communication when something is broken?
24:40It depends. It depends on what? shaking my hand a little bit in the second question just like how hard to get hit by it that's fair use your it's like so if you were hard down you probably want to know like as fast as possible like okay what are you doing is it is it fixed yet is it fixed yet right yeah and like i think that that's it's a it's a tough line to walk because for a lot of times especially during incidents it's it's a lot like um schrodinger's cat right like it's a box and there's something in the box and i don't And I know that there's something wrong with the box, but if I open up the box to look at the state of the cat in the box, the state of the cat changes. And so incident communications is a lot like that.
25:27It's like if I'm like there's a lot of people and a lot of folks when incidents are happening, they're always in the incident channel being like, what's the status? It's like, we all know. And we try to get our communications out there very quickly. Whenever I'm running comms on an incident, I'll set a timer for 15 minutes. minutes and every 15 minutes it's like okay cool let's go get a status out there and then let's set the clock again and then repeat because we want to make sure that like we are communicating with you all but we also want to make sure that you know we're not spending all of our time telling people how broken we are because if you're because as Noah said like Noah wants to spend all of his time and Tom wants to spend all of his time fixing the problem and if they have to you know take time every five minutes to be like here's what we're doing here's the status it makes harder it sometimes it doesn't but sometimes it actually does make it harder for us to you know fix the problem faster I think that that's well said and it sort of that balance between speed accuracy is is difficult to to reach sometimes especially when communication and there are a lot of eyes on it but it sort of kind of goes back into the ambiguity of incidents when you don't know what's going on generally maybe Tom When you're paged and you go into these things, you sort of talked about the process before where it's like you're digging, you're looking at your dashboards, you're trying to figure out where the problem lies in the stack.
26:50If there's ever additional ambiguity around that, like how do you resolve that? What's the process of going in to actually like make those things clear? That, I mean, it's a very complex kind of question, right? It depends as with anything and in kind of any SRE or NRE side of things is everything. kind of variable and you never think it's is never sure but one of the things that we tend to be pretty good at cloud for it is if you're not 100 sure you either try to find someone that does so that's either going to be the sme for the thing that you think is impacted and then you page them in and then they'll go oh no actually but we know what that problem is because we've seen that before and it's actually that derivative function that you need to go page and you start chasing down a chain And that's usually how that goes.
27:38And then sometimes, unfortunately, that does mean that you're getting people out of bed that don't need to be out of bed. But people tend to be, as long as you don't do this every day or every week, it's fine. People tend to be pretty good support about it. But then you start kind of basically digging, right? It's like we have this with some of our P0 or P1 incidents, especially the automated ones. We have like, yes, we know we have a metric that's impacted and clearly it's firing. but now the question is why it's like what broke right it's like we have things about our api availability incident gets created it's instantly p1 everyone starts digging it's like well we have the slo we see that the slo is broken no idea what broke and then you just start digging right and that basically means that you have especially with these incidents you have five ten teams all at once digging at the same time And usually, fairly quickly, you can dance into kind of that singular point.
28:38If we don't get there, then you just start paging people in. And that's when people like Brandon or all of the other Incident Commanders come in, and they just kind of go, it's like, look, smells like that. Let's drag them in. And that's how you get there very quickly.
28:54It's being there for a long time. It's pattern matching. You kind of get a sense of where, what it looks like eventually. um yeah that's i mean i find that process kind of interesting because it it just means that you have to sort of continue to follow all of the incidents and continue to build that sense up yeah i mean it's it's with with anything like if it walks like a duck it walks like a duck it's probably the number four but um the the thing here is that cloudflare the product managers love this probably is they ship we keep shipping there's always a new product every other week there's a product every other two weeks there's a birthday week we keep shipping new things and we keep building new things and if you don't keep on top of it it's very easy to start forgetting where the dependencies lie um we have products like hyperdrive where it's like i have no clue what that does and then there's an incident and then all of a sudden it's a network problem it's always a network problem um except when it's tns yes exactly okay Yeah. Okay. So we've talked a lot about the process, how to dig, where to dig.
29:56At some point, a fix goes in and a fix is kind of different than a full remediation. Noah, how do you think about that process? How do you find that balance? So a fix is generally something that you're doing to stop the bleeding. A fix is something that you can do quickly, that you can get out. It may not be necessary. the like right long -term thing maybe it has a couple of trade -offs there are some cases that it maybe doesn't protect against where you can still where people can still get errors or something maybe it's going to cause like a couple of other issues or like maintenance burdens if it were to stay in place and the remediation is okay what are we going to do as our like more long -term thing to fix this up so in the incident you're generally trying to get to find out what the fix is going to be get the fix out and afterwards once you've closed the incident and resolved things then you can spend the rest of the day and maybe the next couple of days doing the proper remediation and getting that out the door days try weeks sometimes oh yeah sometimes it is weeks hopefully it's not but yeah hopefully why why do you think that like that recovery process that you're talking about is sometimes so much slower than actually like pushing the fix out because sometimes like the proper fix for the full root cause is a lot more involved than you can get away with in an incident.
31:22things are time-sensitive, and you're trying to get something up to stop the bleeding and stop customers from having problems as quickly as you can. And the proper remediation might take a few weeks. But in an incident, you don't have a few weeks. You need to find something much faster that you can do, even if there's a couple of trade-offs. Also, the thing that is just as important is that stuff is broken, but if rolling on a fix has a risk around breaking things, you can't just push that out, right? Like the only thing that is worse than breaking something is trying to fix something and then breaking it worse. And so you need to be very deliberate and making sure that like, okay, if we're going to push this out, we need to leverage all of the safe change mechanisms to make sure that like, again, this complex set of interdependencies doesn't degrade further.
32:17than it already has. Yeah, I think a particularly good example that was back in December, where we had a bit of a learning experience around kill switches, where trying to fix one issue cascaded into a far, far larger problem. And that is why fixed, even like an incident fix, you don't just send that out everywhere. You need to take that out slowly, you need to canary that, and you need to make sure that the cure is not going to be worse than the disease. Yeah, I was going to, I was wanting to ask. you know the classic problems that we run into are harder to fix because the things that are quick to roll back well that that's every day so where we have health-mediated deployments right so that means any change that goes out into production goes through a subset of traffic slowly over time and it progresses to you know our different traffic sets our different colors around the world and when we identify a problem or a breach within an SLO, those things are automatically rolled back.
33:15So all the low-hanging fruit of quick recovery, it's already done. So by the time we get paged into a problem that's a P0 or a P1 or something that is negatively impacting our customers, it's usually a pretty hard problem to solve. So it takes a while for us to get in and then release a change safely, right? So that's one of the reasons in rolling back a change that is added from weeks ago can be more harmful than forward patching something on top of it. So as part of that process, sometimes there's a change freeze or a limited deployment window, especially when you're evaluating these things where you need to kind of look across timelines and there are maybe some large-scale incidents.
34:02How do you sort of think about doing those change freezes or limited deployment windows? it's emotion and gut you're just kind of mad and you're like all right no more we're done um not not that often but sometimes um there there are situations where you know things start breaking in precarious ways you'll see like we have five incidents going on right now and they're minor you know there might be an internal um an internal service isn't working like properly you know it's slo breached um and then there's some other thing going on over here in the network some other thing here but then like you squint a little bit and you can see the rate of escalations is increasing so like that requires a little bit of human judgment to say I don't know if we can sustain this like we're actually out of capacity for commandership so we should just call a freeze right now and we do that occasionally like you know this is maybe once or twice a year we'll call an internal ad hoc freeze but For the other ones, they are scheduled, and we set these up around end of quarter, near the holidays, even right now during Connect.
35:15A lot of critical staff are here discussing in panels and meeting with our customers, so we don't want to take any chances. So it's a good idea to call a freeze at that point. Another way to think about this is that this distributed system that we've built and that other people have built is so large. that the knowledge base is distributed. And so that like the things that I know and the things that Noah knows and the things that Brandon and Tom and Alex knows and even you know, are all different to a certain extent. And so like, is it possible for one person to know all of those things like potentially, but then that person would be the one who would be responsible for making the determination that everything is either okay or not. And if that entire corpus of knowledge is either unknowable or very hard to understand then you kind of have to make that evaluation that like if this thing here and this thing here are all broken are they related probably not but if there is you know a critical mass that like maybe something could be at risk if we continue to introduce more change better safe than sorry to stop and roll back as as part of the remediation process and one of the things as that was said by uzman in the video uh is that we try and transparently talk about uh about incidents um oftentimes after big incidents or there is a published blog report that we talk about what we did what we learned and when um the tubes you've written several of those as you were talking about how do you think about what to say and how fast to say it and when that
36:47that blog can and should come out yeah um well it's a team effort so um you know it's not just any one person um but generally they all have a very similar pattern this thing happened we'll talk about how it happened why it happened what we're doing to fix it and also we're sorry right like Usman said that at the top we gotta apologize first we've hurt people we've let them down um was the um I was actually talking to Dane um about the Halo that company that um won the cold start and Dane was like this is really scary there's a whole bunch of people who are wearing health bands that are dependent on do if do were to break that would be really really bad for a lot of people um and so like you've got to keep that in mind and like if you walk into like a blog and be like yeah we broke something deal with it um definitely a way worse opinion so you know lead with empathy and then you know be transparent this is what happens we're doing to fix it i think the the important thing to add to that is that learning in public is really important and that's something that i'd like to pride ourselves on that that we definitely do is we learn in public a lot of the things that we do both from an engineering perspective and an operations perspective is brand new for a lot of the folks here it's it's either brand new for the entire industry or it's brand new for quite the majority of the folks that run it so whenever we make a mistake or we learn something we try to be very transparent and open about that so that other peoples don't run into the exact same mistake right and i think that's that's equally as
38:19important as being transparent to our customers So sort of a dovetail off that, which is, like, we want to be transparent so people don't run into the same problems that we've encountered. How do we make sure that the problems that we've already resolved, the ones that we've fixed and remediated and done all of that, how do we make sure that we don't see those problems again? So generally, a lesson that you've learned and the rules that you've created from this are only as good as, well, how well can you get folks across the company to follow them? So historically, this is actually... been a fair bit of a challenge for not just us but for everyone over time with AI it's actually become slightly easier so nowadays we can have things like reviewers that go in and internal and we can like encode stuff in say code reviewers and go like okay like let's not do this class of thing this has caused issues in the past this is something we did with the notorious unwrap statement that some of you may be familiar with but yeah like it's it's a general, like, it's always a new learning process every time because you have to find ways to say, okay, well, what types of systems do we need to target and like audit and look into just make sure that we don't have any further occurrences of this?
39:30Where might we have similar classes of issues? And you kind of have to take everyone on a case, every new lesson on a case by case basis, because it's always different stuff that's hit in a new way. I think it's really important to add that like AI is. You know, I'm sure that this is a small question in your sheet, but I'm just going to answer and say it right now. I think AI is very helpful as an assistant, right? It can help parse the massive amounts of information. And I emphasize help, right? Like, I think we've all been shown all of those, like, this AI bot can run your incident This AI bot can make it so that you never have to pick up a pager again. It's like, well, here we are picking up the pager. And, like, that's because, I think to a certain extent, problems are subjective, and I don't think that, like, I don't even, and if I don't know what the problem is, and a customer or an end user doesn't know what the problem is, then an agent isn't going to know what the problem is, right?
40:35Like, your AI is only as smart as you. So, I'm sorry I'm going to pick on you again. The thing that you build, do you know all of the... ways in which it can fail? I wish. Yes. Do you know all of the ways in which it can fail to a customer? Yeah, like I wish too. Like I wish, not only do I wish I knew all of the ways that you knew all the ways, but I wish I knew all of the ways your app can fail because then I can figure out that there's a problem and then page myself so that you don't have to get paged if we break you, right? But if I don't know all the ways, you don't. know all the ways then an agent is never going to catch all of those um and so like that's why ai is a very helpful assistant right like there's so much data right like over 1.1 petabytes or some number petabyte petabyte and a half that's a lot of petabytes um any petabyte there's a lot of petabytes i guess but like it helps give me clarity but i still have to make the decisions and we all have to make the decisions ourselves yeah they're useful for discovery and for helping surface things They're very terrible at making decisions on their own.
41:42Agents are very, very good if you want a bit of help trying to find information faster about what's happening, but they are not oracles. They will not understand your systems fully, and so you can't really trust them beyond what you can immediately show and prove based on what they're saying. I mean, that's the IBM slides, computers. computers cannot be held responsible so should not make management decisions computers cannot be held responsible so should not make incident decisions is basically the exact same thing because at the end of the day an AI might make a decision that is blindsided blindsides you in some way or form or reverts a decision that was actively made where it's like hey we know we're gonna impact or we're gonna do this thing because XYZ reasons and then other things happen So it's super helpful, and we have built a ton of, like, systems around kind of AI and making these things easier.
42:43That has definitely, definitely helped. But, yeah, at the end of the day, I think it's going to be humans for a little bit longer, at least. We've heard a lot about how AI is helpful in terms of the investigation of what's happening and as an assistant in trying to find out what's happening. But, Brandon, how are we using AI in terms of the process of incidents? I mean... And we have not adopted AI heavily yet into the process. We are moving forward with this. Like, what we're seeing as a pattern is obvious. Like, every engineer, everybody who's pulled into an incident, jumps in, they throw their agents at it, right? So you've got to wait a little bit. You don't want to create the solution for everybody. So what we're doing is using Cloudflare OS in the future, not yet today, but very, very soon.
43:30will be part of the incident commandership. It will start doing a lot of the administrative tasks that perhaps I don't need to update the status because Noah just mentioned it in chat a minute ago. So that can just be done by an agent. So there's little optimizations we can do there. But at the end of the day, we want to make sure everybody is understanding the information the same. So having one harness. and one system that we share is the correct path forward. Maybe we have, it looks like, one minute left, and maybe we'll stick around at the front for some questions kind of afterwards. But as maybe a closing here, if you were designing an incident process from scratch, if you're sitting in the audience and you're trying to take something home that you can implement at your company, in one sentence, what would you bring?
44:23What would you do?
44:27in one sentence um you you need to bring order to the chaos it's that simple i have a simple set of rules of of who can do what and when and follow that uh and continue to follow it and refine it that would be the one one sentence yeah it's more than one sentence make everyone accountable for instance you
44:57Find and bring in the people who need to be pulled in, and don't create a super, super crowded noisy room when doing so. Democratize where possible. I think that's the key thing for us is the rate at which we detect problems has been really good because we enable folks to do that.
45:19Well, thank you all. We, I think, I don't know if we have- time for questions and we don't see in the back we may not I'm asking everybody to you know feedback is important you know rate us on the session and I think a couple of us will be hanging out to answer some questions up front if you'd like to talk about incident process but thank you all for coming to our session thanks all for coming to connect it's really good to see all the customers and we're excited to meet you afterwards so thank you so much
Oscar Moreno Garza, Noah Crouch
Uber · Cloudflare
Tom Strickx
Cloudflare
David Tuber, Emily Music
Cloudflare
Be the first to know when registration opens and the agenda goes live.
--- title: "Breaking the Internet (and putting it back together)" url: "https://www.cloudflare.com/connect/2026/session/breaking-the-internet-and-putting-it-back-together" year: 2026 --- # Breaking the Internet (and putting it back together) [Watch session](https://www.cloudflare.com/connect/2026/session/breaking-the-internet-and-putting-it-back-together) ## Session details - **Track:** Network & Infrastructure - **Topics:** Network modernization & multicloud, Performance & resiliency, Customer Experience - **Solution areas:** Core Platform, Application Performance ## About this session When you operate a global network, incidents are not just engineering problems, they are coordination, communication, debugging, and trust problems happening under pressure. This session is for infrastructure, SRE, security, and engineering leaders who want a practical look at how Cloudflare manages incidents at scale. Learn how our response process has evolved over the past year, including how AI-assisted tools help teams find signal faster, summarize context, reduce toil, and improve post-incident learning. ## Speakers - [Alex Krivit](https://www.cloudflare.com/connect/speakers/2421418) — PM, Cloudflare - [Brandon Harris](https://www.cloudflare.com/connect/speakers/2421411) — VP, Engineering, Cloudflare - [David Tuber](https://www.cloudflare.com/connect/speakers/2421536) — Director of Product Management, Cloudflare - [Noah Kennedy](https://www.cloudflare.com/connect/speakers/2472425) — Systems Software Engineer, Cloudflare - [Tom Strickx](https://www.cloudflare.com/connect/speakers/2411667) — Principal Network Engineer, Cloudflare ## Transcript [0:02] All right. Hi, everybody. Welcome to breaking the internet and putting it back together again. We're going to play a short video to start. [0:24] So while you were sleeping, a global outage brought tech systems to their... An outage at Cloudflare cut off access to several major websites. [0:34] Cloudflare runs one of the world's largest networks, handling around one -fifth of web traffic. 5.00 Error and Internet Server Error. [1:04] and the very first thing i do is apologize and i take personal responsibility and i really mean it and then we all start full transparency we tell them exactly what happened if you boil it down incident process at cloudflare is really simple this we sure how we we learn as fast as possible that something is wrong we get everyone together we fix it then we report, and then we follow up rigorously. What is the work that we need to do to ensure that this can't happen again? A bug fix is not a prevent sticker. Now, a bug fix with a test is. [1:45] We're gonna keep drilling as we keep wanting to get better and better at this. [1:57] Hi, my name is Jeremy Hartman, I run infrastructure engineering here at Cloudflare. We're going to start off with our report card. Our goal is five nines, it's a fairly audacious goal, after Code Orange. We really leaned into release managing and how we're doing things safely and fast across our network. There should be another graph that's up into the right. Well, what does it really mean? So we're more change, it's faster, and it's safer all at once. [2:31] The frequency of those decreasing. This is over time, over the past six years. We will keep at this until this gets down to zero. [2:46] Above all, keep at... high for our customers and high empathy for our teammates and I know that I know that we can do all that because we've done it many times before [3:08] oh that was a problem all right all right thank you everybody breaking the internet often makes the news. Putting it back together again doesn't, and that's what we're here to talk about today. Sort of an inside look at how it gets put back together again. My name is Alex Krivet. I work in product here at Cloudflare. Almost a year ago today, on November 18th, a database permissions change doubled the size of a bot management file. The file reached the whole network very quickly. The proxy hit a hard limit, and for a couple of hours. Much of Cloudflare's traffic got an error page. A few days later, a change made to protect folks from a React vulnerability tripped a years -old bug. [3:55] And about 28% of HTTP traffic failed for about 25 minutes. Those were two different bugs, but sort of had the same pattern. There were one change everywhere happening in seconds. And so we declared code orange, fail small. And for two quarters... resiliency came sort of before everything else. We changed how configuration ships, how our systems fail, how engineers get in, and when everything is down, and how we talk to our customers during that incident. That program is now done, but the work isn't. And many of you here today sort of lived those incidents with us. And so this isn't necessarily a victory lap. But what we want to cover today is how. How does Cloudflare approach? [4:40] which incidents, how they're run internally, how we balance speed and safety when we're fixing things for 20% of the internet, and how we learn from these issues and make sure that they don't happen again. In the video, you heard from our former head of engineering, Usman, describe the incident process, how it works, those four steps. And sort of later in that video, you heard from Jeremy Hartman to sort of show you how that process is. being run, and where we're at today. We want you to leave today's talk with an idea of what's happening internally when Cloudflare declares an incident, the process that we're running, and how things get unbroken. I'm joined today by this panel, this group of people. [5:29] Brandon Harris is the VP of Engineering for Infrastructure. He leads SRE in our incident management process. David Tuber, he .. Director of Product for Infrastructure, the physical network, the automation under it, and the systems that ship changes safely. Noah Kennedy, the technical lead for service -to -service load balancing for our edge, and also works on making our foundational systems more reliable. Tom Strix, Principal Network Engineer, spent nine years helping build our global network to be resilient and quick to fix, and he still works in instance every day. All right. Thanks, guys. for being here today to talk through this. As a sort of beginning point, every incident sort of starts the same way, and somebody has to call it. [6:14] Brandon, you oversee the incident process. How does Cloudflare declare an incident? I think what makes us super unique is that we allow anybody to declare an incident at any time. And you might be a little scared and wonder, how does that actually work? You know, well, So the implementation is very simple. You know, it's a chat bot, chat operations. I think everybody will have that here. But, you know, it's really the culture around it that makes it important. It's the fact that if you see something, you say something. Something doesn't look right. I'm going to raise an incident. We're going to get somebody to look at it, and we can quickly cancel that incident if, you know, perhaps nothing's wrong. Now, you know, that's the simple way of answering the question. The harder way is, well, there's a lot of data. [7:01] behind declaring an incident and knowing when something has been gone off the rails or not working as intended. So we have signals that are, I think the last time we looked, it was over one and a half petabytes a day of information that we're looking at and consuming and determining are we outside of operating ranges. So from those signals, we've got people and staffs and teams that are on call. they receive a page. During that video you might have heard someone up on the stage get paged. Was that? That was me. Okay. So how many people here are on call? Raise your hand. All right. Did anybody hear that page? Did your heart rate go up just a little bit? You know, like, you know, so that's kind of, you know, what on call is about. [7:48] But when we go back to, you know, raising an incident, there is a primary incident. or somewhere behind the scenes at Cloudflare. This is a group of volunteers, leaders within our company, who have signed up and said, I want to do more. I want to run the world's largest network and the incidents that occur on the world's largest network. Kind of scary, right? But they signed up for it, and we have 55 of them everywhere. And I think some of them are in this room and have taken part of this, so thank you. Then we go back into the telemetry. So it's about signals. It's about knowing when something has gone off the rails. [8:35] We have four major signals that we know there are big problems within the network. If there is a global CPU drop or if there are authentication requests, those are the canaries in the coal mine. Those are automatic signals that get everybody engaged immediately. But yes, that's the long -winded answer going through like how do you raise an incident? It's very simple one chat operation Anybody can do it and we take it from there one one other thing. I want to just add is that like there are Signals that declare incidents but at the end of the day because anybody to anybody can declare an incident even you know All of us can but also our sales teams are going to market teams can declare an incident Our support teams can declare an incident. So it's super helpful because we generally see kind of three different incident types, right? [9:22] We see incidents that are declared because a customer saw a problem and we're going to raise an incident, we're going to go figure it out. We see another group of incidents that are caused because an internal employee saw a problem, either they were doing some test or there was something that was broken, they're going to raise an incident, we're going to figure it out. And then the third bucket is what Brandon said, like the monitoring. We have automated systems that detect problems and if a threshold is breached, let's declare an incident and go figure it out. All of those are totally valid. Obviously, customers declaring incidents is something we want less of, and monitoring declaring incidents is something we want more of within reason. But that's kind of how the breakdown happens. So anybody's allowed to actually declare an incident, but when it's kind of a judgment call in sort of that gray area of, do we want this to be escalated or not, which side do you want people to err on? [10:15] I mean, there's a chain of command, right? So these things don't work without, you know, some rules and guidelines up front. By default, the incident commander has, you know, the final say. So the person who was paged that says, is this a real problem? They're going to determine after talking with other team members and other people that are brought into the incident, is there actually an issue, right? Is there impact? So they're the ones who can, you know, ultimately say it's canceled. Now, occasionally, these things may run in. to a boundary of leadership, right? So there might be executive decision-making that can override that. And we do that when it makes sense. There could be a single customer who's impacted inappropriately, and we need to dive into that. [11:01] But it may just be just a fraction of our requests, so we don't see necessarily the big issue, except we know that one customer might be in pain. And we also have like processes for support right so like not every customer problem turns into an incident but if we're seeing problems with a customer support ticket someone's reporting an issue it's really hard to diagnose it's really really urgent um we'll raise an instant and declare that and so um you know just trust 5 000 people to make the right choice all the time is scary as brandon said but it generally we found that people who are all trying to do the right thing by our customers are doing the right thing for our customers and for our operators to make sure that we're running the best network for our customers at all times. [11:48] So a page goes out, an incident is declared, and everybody gets into a room somewhere. Tom, you're a regular sort of on the other end of that, those calls on the other end of those pages. Walk us through sort of what you do first. What is your process? Where do you decide to dig? How do you make sure that this is resolved? quickly. I mean, at the end of the day, it's basically just a decision tree, right? Like we have, and then Ntube's alluded to that earlier, we have like always a specific set of incidents that will always happen at the same kind of issues that we see, right? Like with, if it's a series of 522s that we start seeing a specific region, then it's likely a subseca or a network issue communicating with probably US East 1. It's always US East 1 or it's DNS. But When those issues happen, like, it's very easy and very quick to kind of understand where that problem lies, and we can start debugging, we get the network team involved, and that's kind of where we go. [12:43] The more insidious ones are the problems that we're not as used to, right? Like, there used to be a whole set of problems at Cloudflare that we had to deal with on a weekly basis, or almost a daily basis, and that's no longer the case, and now they happen like once every two years, and we forget how to deal with these things. And that's where we have runbooks and that's where we have like kind of making sure that all of that experience that exists within people, that that also resides in documentation that we can kind of fall back on or dashboards. Like I think basically every single engineer at Cloudflare has at least one favorite dashboard. And that dashboard will usually be, this is how broken my service is right now. [13:28] And that's, you know, usually what engineers will do is they'll go into an incident room and then they'll look at their dashboard. to make sure it's not them. So everybody has all these tools, all these dashboards, things that they're looking at to make sure it's not them, to help other people that are in that same room with you and knowing where to dig sort of first. As a responder of that page at that time, what's one thing that you would want to change about that process, if you could change one thing? More sleep. I think that's... always the the thing is is we do try with the on-call schedules at least for most of the more operational teams to make sure that it's a follow the sun system but that's not always and it's always possible and then you still have smes and then sometimes you do need to wake up people uh figuring out a way that we not have to do that that'd be great is there something that is that you found overtly useful about that sort of initial process it's it's a really good way if interested in understanding the entire Cloudflare stack to understand the entire Cloudflare stack. [14:34] Because it's one of those things where if you don't know what the Cloudflare stack looks like, it's very hard to understand what the problem is or where the problem might lie. But once you start getting involved in these incidents, over time, you start seeing the same questions being asked for the same types of incidents. And from that, it becomes very simple or fairly trivial to kind of understand the stack at least to some level. That's always good fun. I tell all the new PMs who join my team, subscribe to the Incident channel. And for your area, go through the Incidents. You will learn more about your product. You'll learn more about your area running an Incident than you will do shipping anything. And I think that's generally true for also writing code as well. I know when you started, you were in Incidents all the time. Yeah, that is a really common bit of advice that I tell a lot of juniors joining is if you want to get really knowledgeable about how Cloudflare works. [15:26] Just join instant rooms for anything vaguely close to your area. You'll learn so much about how things work, how things fit together, and what all the adjacencies are for your systems. This may be a good transition point, but Noah, when your system specifically is on fire, what do you need from people on that call first? Like, what helps you resolve that problem quickly? Yeah, so there's a big difference between an incident commander, as Brandon put it, and what I'm going to call an incident driver. So... If you're like a technical lead or just really anyone on a service team or an SME for something, you don't want to be the one who's doing all the coordination. You don't want to be the one like coordinating and handling customer comms, trying to determine exact impact. You want to be focusing on just debugging issues, digging into what the problems are, and figuring out what you need to do about it. [16:13] And so generally, if it's your system that's having issues and you're the SME, you want support from the rest of the room and you want... folks to handle a lot of these sorts of issues that are not directly related to debugging exactly what went wrong and how we can get it back up. How do you go about getting that support from people maybe that you've never worked with before or that you've never even talked to before? Often these calls are virtual and there's 20 people in this call trying to debug, trying to coordinate these things like you're saying. How do you get that support that you need? Yeah, so there's generally there's always going to be an IC in the room so an incident commander we have an on-call rotation for that and so you're always going to have someone who is an incident commander who's been assigned there and so you can always go to them for help and generally they're one of the like main parties who are going to help you with things like customer comms um they'll help like they'll help you like pull on other folks so i don't necessarily need to worry about pulling in uh all sorts of other teams and getting into contact with them because generally an incident commander unless i'm doing that on that particular day is going to be handling that for me. [17:22] An incident commander might be a loaded term. Brandon, what's an incident commander? Really, it's kind of a coordinator. It's the administrator. It's the orchestration of, you know, getting to stop the bleeding, right? Primary, you know, resolution is, you know, not necessarily what's the root cause. It's what do we need to do right now to make sure impact goes away? So that's their first job. Really, it could be anybody. And we've opened this up to everyone in our org today can become an incident commander. It's one command. You do it in chat. You say, I see me. And then now you are running the incident. There's a lot of responsibility, as I stated, in doing this. However, it's kind of a rote administrative process. [18:10] You're logging status updates. You're communicating information to your customer support. You're driving the problem to solution. You don't stop when you hit a roadblock. You might run into a debugging impasse where, well, we're not sure if this is it. Your AI may not agree with my AI, or we're looking at the things together. The incident commander needs to disambiguate and get to objective reality quickly. So they need to ask the questions, you know. This is why managers are good at this. because they're really great at asking questions. So ask the question, get down to the five whys, get to what do we do right now, and can we do it right now? I like to think of an incident commander as like that app, that hinge ad, like the app that's designed to be deleted. [18:57] Like an incident commander, if they're doing their job right, they're finding the person who's actually responsible, right? Like an incident commander is kind of like a triage unit. You're running this thing and getting people stable. and then giving it over to the person who's responsible for the long-term fixed root cause permits. And so a good incident commander can, you know, keep things going until we figure out what's going on. And then somebody says, oh, it's, it's Noah. So we'll give it to Noah and then Noah will drive to completion. I think, I think one thing I'd like to add, it is, it is not necessarily an authoritative position. It is more of a facilitative position, right? If you, the way we haven't modeled. internally uh the commander has the authority to make a decision but they don't really have authority right like they're they're pulling people in they're trying to solve a problem that that indemnifies them from being culpable for responsibility so they're they're delegating responsibility onto multiple streams and ensuring that we converge on the solution yeah like in a lot of these cases the uh ultimate decisions are often made by the technical driver who's sitting there going okay um i need us to do this next, because this is how we're going to stop the bleeding, and this is how this system works, and this is what we're going to do. [20:13] And then the rest of the room looks at this, goes, okay, that makes sense, and let's go [20:21] Two, you spent seven years, maybe, at Microsoft prior to joining Cloudsler in much of a similar What's universal about incidents and the incident process? They suck. [20:35] I think that the thing that's super helpful, and incidents are hard, but one of my old managers used to say that an incident is like a very, very concentrated chip cycle. Something is broken, there's a problem, and you have to solve it very quickly. That's universally true. And I think that regardless of whether or not you have SLOs, SLAs, you have auto -paging, it's just someone yelling at someone else from across the room, there is a customer problem, and it needs to be. be solved very quickly and we are basically saying we are not doing other things and we are focusing on this one thing because that's that's we need to go do and that's universal whether you know you're at microsoft or google or cloudflare or a startup that you know has 100 users or a startup that has a thousand users it's going to be pretty universal we're doing this to solve customer problems fast is there something that unique about cloudflare's process that you've noticed um i really like, it's like Brandon said, like anyone can declare an incident. I really like that. [21:31] I think that it means that it's like every, giving people that responsibility means that everybody is invested, right? Like whether you are, you know, the head of SRE or you're a product manager, a tech lead, or, you know, you're someone in graphic design, you are invested in making sure that Cloudflare is running at its, you know, the best it can be. I'm a little surprised you're not mentioning our speed of delivery to our public blog posts and post-mortems. That seems like a... I've blocked that out from my memory a little bit. Well, you know, you've been an author of many of these. I think that that's really helpful as well. [22:17] Our transparency is very unparalleled. A lot of people... Actually, a really good example is there was a problem that we had a couple of months ago where we actually know... a problem with between us and a shared provider and we posted on our status page first and said hey there's a problem and we're looking into it and that was good because you want to talk about it but it had this weird side effect of everybody thought that cloudflare was the problem um and so there were a whole bunch of people coming to us and being like why are you broken why are you broken it's it's not us it's a shared dependency um and then the shared dependency posted like an hour later saying oh yeah we're broken up there um we're gonna fix that And that speed is very, very helpful for us because customers really, really want to know everything, all the time, as fast as possible. [23:05] And so I think Cloudflare has really leaned into that, and that's very, very helpful. Code Orange gave us sort of a revamp of our communication strategy during incidents, sort of every on a clock, every 30 minutes, every 60 minutes, something like that. Why? So maybe this is a question for Brandon. Why is sometimes we're still looking into it or we're still fixing it or we're still trying to observe this better than no update at all? Yeah, this is, I mean, as a customer, you just want to know very likely, you know, like, what do I need to do and are you working on it? And if there's radio silence, you lose trust, right? And this is something we identified. [23:51] We're not doing a great job getting that status out, even if it's the same status. I think internally we look at this as, well, there's nothing to report yet. We're still debugging. But we actually need to respond with, we're still looking into it. Let me take a straw poll here. Who here has read the Cloudflare status page during an incident? You don't count. So of those people who have read the Cloudflare status page during an incident, and please be honest here, because as the person who writes a lot of these... statuses. How many of you feel comfortable with the rate of status or the quality of the status during the instance? Raise your hand if you are comfortable with it. Okay, only a couple. There's not a lot of hands going up. Not a lot of hands going up. And I think that that's because like, and I'm going to pick on you in the front because you've raised your hand, you raised your hand both times. But like, what is for you as a stakeholder, what is the ideal like rate of communication when something is broken? [24:40] It depends. It depends on what? shaking my hand a little bit in the second question just like how hard to get hit by it that's fair use your it's like so if you were hard down you probably want to know like as fast as possible like okay what are you doing is it is it fixed yet is it fixed yet right yeah and like i think that that's it's a it's a tough line to walk because for a lot of times especially during incidents it's it's a lot like um schrodinger's cat right like it's a box and there's something in the box and i don't And I know that there's something wrong with the box, but if I open up the box to look at the state of the cat in the box, the state of the cat changes. And so incident communications is a lot like that. [25:27] It's like if I'm like there's a lot of people and a lot of folks when incidents are happening, they're always in the incident channel being like, what's the status? It's like, we all know. And we try to get our communications out there very quickly. Whenever I'm running comms on an incident, I'll set a timer for 15 minutes. minutes and every 15 minutes it's like okay cool let's go get a status out there and then let's set the clock again and then repeat because we want to make sure that like we are communicating with you all but we also want to make sure that you know we're not spending all of our time telling people how broken we are because if you're because as Noah said like Noah wants to spend all of his time and Tom wants to spend all of his time fixing the problem and if they have to you know take time every five minutes to be like here's what we're doing here's the status it makes harder it sometimes it doesn't but sometimes it actually does make it harder for us to you know fix the problem faster I think that that's well said and it sort of that balance between speed accuracy is is difficult to to reach sometimes especially when communication and there are a lot of eyes on it but it sort of kind of goes back into the ambiguity of incidents when you don't know what's going on generally maybe Tom When you're paged and you go into these things, you sort of talked about the process before where it's like you're digging, you're looking at your dashboards, you're trying to figure out where the problem lies in the stack. [26:50] If there's ever additional ambiguity around that, like how do you resolve that? What's the process of going in to actually like make those things clear? That, I mean, it's a very complex kind of question, right? It depends as with anything and in kind of any SRE or NRE side of things is everything. kind of variable and you never think it's is never sure but one of the things that we tend to be pretty good at cloud for it is if you're not 100 sure you either try to find someone that does so that's either going to be the sme for the thing that you think is impacted and then you page them in and then they'll go oh no actually but we know what that problem is because we've seen that before and it's actually that derivative function that you need to go page and you start chasing down a chain And that's usually how that goes. [27:38] And then sometimes, unfortunately, that does mean that you're getting people out of bed that don't need to be out of bed. But people tend to be, as long as you don't do this every day or every week, it's fine. People tend to be pretty good support about it. But then you start kind of basically digging, right? It's like we have this with some of our P0 or P1 incidents, especially the automated ones. We have like, yes, we know we have a metric that's impacted and clearly it's firing. but now the question is why it's like what broke right it's like we have things about our api availability incident gets created it's instantly p1 everyone starts digging it's like well we have the slo we see that the slo is broken no idea what broke and then you just start digging right and that basically means that you have especially with these incidents you have five ten teams all at once digging at the same time And usually, fairly quickly, you can dance into kind of that singular point. [28:38] If we don't get there, then you just start paging people in. And that's when people like Brandon or all of the other Incident Commanders come in, and they just kind of go, it's like, look, smells like that. Let's drag them in. And that's how you get there very quickly. [28:54] It's being there for a long time. It's pattern matching. You kind of get a sense of where, what it looks like eventually. um yeah that's i mean i find that process kind of interesting because it it just means that you have to sort of continue to follow all of the incidents and continue to build that sense up yeah i mean it's it's with with anything like if it walks like a duck it walks like a duck it's probably the number four but um the the thing here is that cloudflare the product managers love this probably is they ship we keep shipping there's always a new product every other week there's a product every other two weeks there's a birthday week we keep shipping new things and we keep building new things and if you don't keep on top of it it's very easy to start forgetting where the dependencies lie um we have products like hyperdrive where it's like i have no clue what that does and then there's an incident and then all of a sudden it's a network problem it's always a network problem um except when it's tns yes exactly okay Yeah. Okay. So we've talked a lot about the process, how to dig, where to dig. [29:56] At some point, a fix goes in and a fix is kind of different than a full remediation. Noah, how do you think about that process? How do you find that balance? So a fix is generally something that you're doing to stop the bleeding. A fix is something that you can do quickly, that you can get out. It may not be necessary. the like right long -term thing maybe it has a couple of trade -offs there are some cases that it maybe doesn't protect against where you can still where people can still get errors or something maybe it's going to cause like a couple of other issues or like maintenance burdens if it were to stay in place and the remediation is okay what are we going to do as our like more long -term thing to fix this up so in the incident you're generally trying to get to find out what the fix is going to be get the fix out and afterwards once you've closed the incident and resolved things then you can spend the rest of the day and maybe the next couple of days doing the proper remediation and getting that out the door days try weeks sometimes oh yeah sometimes it is weeks hopefully it's not but yeah hopefully why why do you think that like that recovery process that you're talking about is sometimes so much slower than actually like pushing the fix out because sometimes like the proper fix for the full root cause is a lot more involved than you can get away with in an incident. [31:22] things are time-sensitive, and you're trying to get something up to stop the bleeding and stop customers from having problems as quickly as you can. And the proper remediation might take a few weeks. But in an incident, you don't have a few weeks. You need to find something much faster that you can do, even if there's a couple of trade-offs. Also, the thing that is just as important is that stuff is broken, but if rolling on a fix has a risk around breaking things, you can't just push that out, right? Like the only thing that is worse than breaking something is trying to fix something and then breaking it worse. And so you need to be very deliberate and making sure that like, okay, if we're going to push this out, we need to leverage all of the safe change mechanisms to make sure that like, again, this complex set of interdependencies doesn't degrade further. [32:17] than it already has. Yeah, I think a particularly good example that was back in December, where we had a bit of a learning experience around kill switches, where trying to fix one issue cascaded into a far, far larger problem. And that is why fixed, even like an incident fix, you don't just send that out everywhere. You need to take that out slowly, you need to canary that, and you need to make sure that the cure is not going to be worse than the disease. Yeah, I was going to, I was wanting to ask. you know the classic problems that we run into are harder to fix because the things that are quick to roll back well that that's every day so where we have health-mediated deployments right so that means any change that goes out into production goes through a subset of traffic slowly over time and it progresses to you know our different traffic sets our different colors around the world and when we identify a problem or a breach within an SLO, those things are automatically rolled back. [33:15] So all the low-hanging fruit of quick recovery, it's already done. So by the time we get paged into a problem that's a P0 or a P1 or something that is negatively impacting our customers, it's usually a pretty hard problem to solve. So it takes a while for us to get in and then release a change safely, right? So that's one of the reasons in rolling back a change that is added from weeks ago can be more harmful than forward patching something on top of it. So as part of that process, sometimes there's a change freeze or a limited deployment window, especially when you're evaluating these things where you need to kind of look across timelines and there are maybe some large-scale incidents. [34:02] How do you sort of think about doing those change freezes or limited deployment windows? it's emotion and gut you're just kind of mad and you're like all right no more we're done um not not that often but sometimes um there there are situations where you know things start breaking in precarious ways you'll see like we have five incidents going on right now and they're minor you know there might be an internal um an internal service isn't working like properly you know it's slo breached um and then there's some other thing going on over here in the network some other thing here but then like you squint a little bit and you can see the rate of escalations is increasing so like that requires a little bit of human judgment to say I don't know if we can sustain this like we're actually out of capacity for commandership so we should just call a freeze right now and we do that occasionally like you know this is maybe once or twice a year we'll call an internal ad hoc freeze but For the other ones, they are scheduled, and we set these up around end of quarter, near the holidays, even right now during Connect. [35:15] A lot of critical staff are here discussing in panels and meeting with our customers, so we don't want to take any chances. So it's a good idea to call a freeze at that point. Another way to think about this is that this distributed system that we've built and that other people have built is so large. that the knowledge base is distributed. And so that like the things that I know and the things that Noah knows and the things that Brandon and Tom and Alex knows and even you know, are all different to a certain extent. And so like, is it possible for one person to know all of those things like potentially, but then that person would be the one who would be responsible for making the determination that everything is either okay or not. And if that entire corpus of knowledge is either unknowable or very hard to understand then you kind of have to make that evaluation that like if this thing here and this thing here are all broken are they related probably not but if there is you know a critical mass that like maybe something could be at risk if we continue to introduce more change better safe than sorry to stop and roll back as as part of the remediation process and one of the things as that was said by uzman in the video uh is that we try and transparently talk about uh about incidents um oftentimes after big incidents or there is a published blog report that we talk about what we did what we learned and when um the tubes you've written several of those as you were talking about how do you think about what to say and how fast to say it and when that [36:47] that blog can and should come out yeah um well it's a team effort so um you know it's not just any one person um but generally they all have a very similar pattern this thing happened we'll talk about how it happened why it happened what we're doing to fix it and also we're sorry right like Usman said that at the top we gotta apologize first we've hurt people we've let them down um was the um I was actually talking to Dane um about the Halo that company that um won the cold start and Dane was like this is really scary there's a whole bunch of people who are wearing health bands that are dependent on do if do were to break that would be really really bad for a lot of people um and so like you've got to keep that in mind and like if you walk into like a blog and be like yeah we broke something deal with it um definitely a way worse opinion so you know lead with empathy and then you know be transparent this is what happens we're doing to fix it i think the the important thing to add to that is that learning in public is really important and that's something that i'd like to pride ourselves on that that we definitely do is we learn in public a lot of the things that we do both from an engineering perspective and an operations perspective is brand new for a lot of the folks here it's it's either brand new for the entire industry or it's brand new for quite the majority of the folks that run it so whenever we make a mistake or we learn something we try to be very transparent and open about that so that other peoples don't run into the exact same mistake right and i think that's that's equally as [38:19] important as being transparent to our customers So sort of a dovetail off that, which is, like, we want to be transparent so people don't run into the same problems that we've encountered. How do we make sure that the problems that we've already resolved, the ones that we've fixed and remediated and done all of that, how do we make sure that we don't see those problems again? So generally, a lesson that you've learned and the rules that you've created from this are only as good as, well, how well can you get folks across the company to follow them? So historically, this is actually... been a fair bit of a challenge for not just us but for everyone over time with AI it's actually become slightly easier so nowadays we can have things like reviewers that go in and internal and we can like encode stuff in say code reviewers and go like okay like let's not do this class of thing this has caused issues in the past this is something we did with the notorious unwrap statement that some of you may be familiar with but yeah like it's it's a general, like, it's always a new learning process every time because you have to find ways to say, okay, well, what types of systems do we need to target and like audit and look into just make sure that we don't have any further occurrences of this? [39:30] Where might we have similar classes of issues? And you kind of have to take everyone on a case, every new lesson on a case by case basis, because it's always different stuff that's hit in a new way. I think it's really important to add that like AI is. You know, I'm sure that this is a small question in your sheet, but I'm just going to answer and say it right now. I think AI is very helpful as an assistant, right? It can help parse the massive amounts of information. And I emphasize help, right? Like, I think we've all been shown all of those, like, this AI bot can run your incident This AI bot can make it so that you never have to pick up a pager again. It's like, well, here we are picking up the pager. And, like, that's because, I think to a certain extent, problems are subjective, and I don't think that, like, I don't even, and if I don't know what the problem is, and a customer or an end user doesn't know what the problem is, then an agent isn't going to know what the problem is, right? [40:35] Like, your AI is only as smart as you. So, I'm sorry I'm going to pick on you again. The thing that you build, do you know all of the... ways in which it can fail? I wish. Yes. Do you know all of the ways in which it can fail to a customer? Yeah, like I wish too. Like I wish, not only do I wish I knew all of the ways that you knew all the ways, but I wish I knew all of the ways your app can fail because then I can figure out that there's a problem and then page myself so that you don't have to get paged if we break you, right? But if I don't know all the ways, you don't. know all the ways then an agent is never going to catch all of those um and so like that's why ai is a very helpful assistant right like there's so much data right like over 1.1 petabytes or some number petabyte petabyte and a half that's a lot of petabytes um any petabyte there's a lot of petabytes i guess but like it helps give me clarity but i still have to make the decisions and we all have to make the decisions ourselves yeah they're useful for discovery and for helping surface things They're very terrible at making decisions on their own. [41:42] Agents are very, very good if you want a bit of help trying to find information faster about what's happening, but they are not oracles. They will not understand your systems fully, and so you can't really trust them beyond what you can immediately show and prove based on what they're saying. I mean, that's the IBM slides, computers. computers cannot be held responsible so should not make management decisions computers cannot be held responsible so should not make incident decisions is basically the exact same thing because at the end of the day an AI might make a decision that is blindsided blindsides you in some way or form or reverts a decision that was actively made where it's like hey we know we're gonna impact or we're gonna do this thing because XYZ reasons and then other things happen So it's super helpful, and we have built a ton of, like, systems around kind of AI and making these things easier. [42:43] That has definitely, definitely helped. But, yeah, at the end of the day, I think it's going to be humans for a little bit longer, at least. We've heard a lot about how AI is helpful in terms of the investigation of what's happening and as an assistant in trying to find out what's happening. But, Brandon, how are we using AI in terms of the process of incidents? I mean... And we have not adopted AI heavily yet into the process. We are moving forward with this. Like, what we're seeing as a pattern is obvious. Like, every engineer, everybody who's pulled into an incident, jumps in, they throw their agents at it, right? So you've got to wait a little bit. You don't want to create the solution for everybody. So what we're doing is using Cloudflare OS in the future, not yet today, but very, very soon. [43:30] will be part of the incident commandership. It will start doing a lot of the administrative tasks that perhaps I don't need to update the status because Noah just mentioned it in chat a minute ago. So that can just be done by an agent. So there's little optimizations we can do there. But at the end of the day, we want to make sure everybody is understanding the information the same. So having one harness. and one system that we share is the correct path forward. Maybe we have, it looks like, one minute left, and maybe we'll stick around at the front for some questions kind of afterwards. But as maybe a closing here, if you were designing an incident process from scratch, if you're sitting in the audience and you're trying to take something home that you can implement at your company, in one sentence, what would you bring? [44:23] What would you do? [44:27] in one sentence um you you need to bring order to the chaos it's that simple i have a simple set of rules of of who can do what and when and follow that uh and continue to follow it and refine it that would be the one one sentence yeah it's more than one sentence make everyone accountable for instance you [44:57] Find and bring in the people who need to be pulled in, and don't create a super, super crowded noisy room when doing so. Democratize where possible. I think that's the key thing for us is the rate at which we detect problems has been really good because we enable folks to do that. [45:19] Well, thank you all. We, I think, I don't know if we have- time for questions and we don't see in the back we may not I'm asking everybody to you know feedback is important you know rate us on the session and I think a couple of us will be hanging out to answer some questions up front if you'd like to talk about incident process but thank you all for coming to our session thanks all for coming to connect it's really good to see all the customers and we're excited to meet you afterwards so thank you so much ## Related sessions - [Canary to cloud: Building a flexible traffic strategy](https://www.cloudflare.com/connect/2026/session/canary-to-cloud-building-a-flexible-traffic-strategy) - [Rearchitecting the Cloudflare network for the AI revolution](https://www.cloudflare.com/connect/2026/session/rearchitecting-the-cloudflare-network-for-the-ai-revolution) - [How we built Cloudflare’s global network](https://www.cloudflare.com/connect/2026/session/how-we-built-cloudflares-global-network) ## More from Connect - [All sessions](https://www.cloudflare.com/connect/) - [Get notified for Connect 2027](https://www.cloudflare.com/connect#2027)