As I write this, it’s been barely 4 days since an RV exploded in downtown Nashville. I’m 15-ish miles away and was never in any danger. Interestingly, I was walking the dog when it happened and I should have heard it, but didn’t.
There’s still a ton of speculation–some in bad-faith, some in ignorance–about why he did what he did, but I’m not going to talk about that.
My day job revolves around being prepared for disasters, and this can be treated exactly like a disaster due to the nature of the target. In fact, some of my colleagues have spent much of the last 4 days implementing disaster mitigation plans due to the failure of the AT&T network. I am incredibly proud of the work that they have done both in the last four days and the last four years in preparation for an event like this.
So, looking at it from a pure communications disaster/widespread failure standpoint, I wanted to note a few things for you to think about.
Know your local infrastructure
When I first saw reports of an explosion downtown, I thought it was a transformer explosion. That has happened a couple of time in the last 30 years so it seemed logical, especially considering the area. Like every other IT nerd in Nashville, I knew exactly what that building was and what it’s purpose is. It needs a lot of power, and it’s been there for a gazillion years.
Once the police said it was “an intentional act” I knew that the AT&T switching station was the target. There was nothing else important in the area (really, who bombs a Hooters?), and a much more politically appealing target–the Federal Courthouse–is only about 10 blocks away. Whoever it was, their purpose was to disrupt communications.
The AT&T network is not as resilient as you’d think, but it is pretty resilient. A lot of people locally are angry with AT&T because of the loss of cell service, 911 service, internet, etc…but AT&T can only do so much to prepare for something like this. It took almost 5 hours from the time of the explosion until the first widespread outage was noticed. My theory–and I cannot stress enough that this is a theory and not based on actual facts–is that the automated systems started re-routing traffic exactly as intended, but that the load was too much for the secondary systems. Essentially, they moved calls from going through Nashville to going through Louisville, but the Louisville system did not have enough capacity to handle the load of the entire Nashville infrastructure going down. That’s why you had outages up that way, too: Network overload and nowhere to go.
Sure, they could have planned for this, but then your $80 cell phone bill would be $200. It’s a calculation that every company has to go through and I’m not going to fault them with less than perfect resilience.
Anyway, the point is that you should know your local failure points. I expected a comms breakdown because I knew the importance of that building. This is important for natural disasters, too. When we had a flood in 2010 and a news helicopter showed footage of the local gasoline depot under water, I knew we were about to have a gas shortage.
At a minimum, you should know the locations of the following facilities:
- The power substation that feeds your home and/or office.
- The power plant(s) that feed that substation.
- The water treatment facility that feeds your home and/or office.
- The distribution point that your fuel comes from. Gasoline, heating oil, etc.
- The distribution point that your food comes from.
- The switching station or local exchange office that handles your phones.
If there was some sort of widespread natural disaster in your area, knowing these things may help you make a decision to leave much easier and much faster. In that 2010 flood, for example, two of Nashville’s three water treatment facilities were taken out. The third came very, VERY close to also being rendered inoperable. If that one had been breached, we would have evacuated within an hour or so.
Communications Redundancy
This one can be confusing but I’ll try my best. I’m not talking about the need for everyone in your family to have a HAM license and a pre-programmed handheld and mobile radio (although that is useful as we’ll see later). I’m talking about the poor folks here who had AT&T for Voice, Data, TV, Cell Service…the works.
Those people were completely cut off from the world for 24 hours+.
The day after my wife and I had our very first date, T-Mobile had an outage that affected something like 8 states. I really needed to call this super-cool girl for a follow-up date because she was leaving town the day after, so I ran down to Target and bought a pre-paid cell phone on AT&T’s network. The day was saved, I got the second date, and now we have two kids, a dog, and a house in the suburbs.
When we got married, she kept her Sprint phone for a while and I had switched to AT&T after the outage. This was important, because Sprint and AT&T used completely different network technologies at the time. There were no scenarios outside of a widespread long-term blackout where both of us would be out of cell service. That’s a communications redundancy.
Today, we’re both on AT&T for cell service and we have Comcast for internet. I have Google Voice, so that gives me VoIP for my “cell phone” number and I could still send and receive text messages during the outage. We had some communication ability, but it was limited.
Then, the day after the explosion, Tactical Dog had a medical issue and had to go to the vet, like, now. She’s fine, but the experience exposed a problem.
My phone worked on wifi, but only on wifi. When I left the house I was completely isolated. If we’d still had separate cell phone carriers, I could have just taken the other phone, but we’d consolidated to one carrier to save money.
Everything worked out, I just had to live in 1994 for a few hours. I remember 1994 so I was fine. But it brings me to my final point.
Have printouts for emergencies
Right now, can you put your hands on a piece of paper with your local EMS/Fire dispatch’s phone number? Or your doctor’s? Or your veterinarian’s?
One of the effects of the bombing was a widespread, multistate 911 outage. Everyone had to call either the non-emergency dispatch number or the local precinct/fire hall/whatever. There were lists floating around twitter and the Emergency Management websites but you had to know where to look. And you had to have internet, which you may not have had.
I was fortunate enough to have been to the vet’s so many times that I knew how to get there without the aid of GPS (which would have required downloading map data since our vet is outside the area we normally roam around in–which is a story for another day). If the way I knew how to get to the vet’s office had been blocked, as might be likely during a more widespread disaster, I’m not familiar enough with the area to have routed around it quickly without the aid of a map.
A pre-printed map, or a paper map with important locations already marked, could have been a literal lifesaver for about a 36-hour period. I bought a good old-fashioned paper map last January just to keep in the car. Something to think about.
The last thing I wish I’d had beforehand was a list of local repeater frequencies, either pre-programmed or printed out. This is amateur radio specific, but as I sat in the vet’s parking lot waiting on them to treat the dog, I realized that my little handheld was way, way out of range of the repeaters I normally use. Again, the vet is outside our normal operating area, but it’s close enough that I really should have put those frequencies somewhere handy.
So that’s all I have for now. Just a few lessons-learned from someone on the ground with a somewhat unique professional perspective.
Great points