Originally published in 2600: The Hacker Quarterly, Winter 2023-2024
Hello, and greetings from the Central Office! It’s winter, which means power interruptions here in the Great Northwest. Most of our power (and telephone) lines are above ground, and the whole region is covered with trees that average over 100 feet tall. Trees and branches are falling constantly causing power interruptions, especially during the fall and winter. It has been getting worse in recent years, though, given that summer weather keeps getting hotter and the rainy season is ever shorter. Making matters worse, we have been getting more “atmospheric rivers” as of late, which deliver several inches of soaking rain at a time, saturating the ground. When a windstorm happens after this, large trees like cedars with shallow root systems simply blow over. They fall onto cars, houses and (of course) power lines. I’ll give you one guess whose power line was taken out by a tree today. If you guessed a Central Office with “Forest” in the name, you’d be right. I’m acting in an incident response capacity today, dealing with possibly the strangest incident that I have ever experienced in my career. But we’ll get deeper into that later. For now, let’s talk about power engineering.
Obviously, downed trees and power interruptions are nothing new in the Pacific Northwest, and we have been prepared for them for a long time–including in the Central Office where I am working today. Now, you probably don’t think of the USDA as a telecommunications regulator (the FCC writes most of the rules), but they have made a significant mark on the telecommunications landscape. I doubt that the company would have been so well prepared for emergencies if it wasn’t contractually required!
If you’re scratching your head, I’ll explain. The USDA, through the Rural Electrification Administration provides subsidized financing to telephone companies. These subsidies were intended to serve rural areas, but as the population of the Pacific Northwest grew, the Company was very effective in its lobbying to secure financing for suburban and exurban locations in its service territory. This saved hundreds of millions of dollars in interest, which could instead be used to buy back shares of company stock and help executives meet their bonus targets.
In a rare case of CEO incentives aligning with public good, the USDA loan program has underwriting requirements which enforce minimum Central Office construction standards. In fact, they publish an entire reference engineering guide, and this includes power supply and backup power requirements. With peak loads assumed, the USDA requires either 8 hours of backup battery power, or 3 hours of battery power plus a diesel generator. In 2007, the FCC eventually weighed in after Hurricane Katrina with Order 07-177, releasing a loophole-ridden, watered down rule applying to all telecommunications facilities (not just ones funded by the USDA). It was then immediately challenged in court.
This particular Central Office is equipped with a 3 hour backup battery system and a diesel backup generator. One room on the ground floor of the Central Office is dedicated to our backup battery system, and the generator is located outdoors. Both systems are required to provide enough power to run the Central Office during peak load, and do so effectively. There are varying requirements on how much of a fuel supply we need, but this Central Office was constructed in 1982, not long after the Mount St. Helens eruption (which happened in 1980). This was obviously fresh in the minds of the engineers who designed subsequent Central Offices. Without knowing what regulations might be forthcoming, they provisioned 72 hours of peak load fuel storage on-site. In practice, we have about 5 days of fuel, because the Central Office doesn’t run at peak load at all anymore, and especially doesn’t do so for 24 hours per day. As with our backup battery contract, we have a fuel services and maintenance contract with an outside vendor.
This all sounds great, right? It’s all good in theory. It’s also good in practice: this stuff is regularly used! We have weather events all the time during Pacific Northwest winters. Most of the problems I’d normally encounter would result from deferred maintenance or a component failing. This time, though, a tree is down, we don’t have utility power, and the clock is ticking. You see, our compliance department abruptly ended our contracts with our fuel service vendor who also happens to be the only fuel depot in the region. The fuel depot doesn’t know why, Compliance wouldn’t tell me why, and it finally took a Washington Utilities and Transportation Commission rule to prompt a solution.
I showed up 3 days in, while we were running on backup power. A landslide had taken out our utility feed to the Central Office, road access to the utility lines, and several utility poles along with it. Although some of our outside plant was impacted, we were lucky: we only had a few subscribers in that direction. Our power was out, though, and it was going to be out for awhile—at least two more days. This wouldn’t normally be a problem; after all, we are well prepared with backup power, and the Central Office itself was accessible via other routes. When I arrived, we had about 1/3 of our fuel remaining for the backup generator, and a battery backup operating with a failed (but redundant) PDU. Normally, none of this would be a big deal. PDU failures happen, especially when switching over to generator power, which is why we have redundant ones. We’d want to fix it before we switched back to utility power (in case the same thing happened again with the switchover), but that was manageable. Fuel is normally no problem, since roads were open and critical utilities like telephone service have priority on constrained supplies (which, in this case, weren’t even constrained). We just needed to call the fuel depot for delivery, and call our electrical vendor to fix the PDU.
This is all easy stuff—well, not easy, but manageable. We have a standard operating procedure for it. All of the contracts are in place. Should be a cake walk, right? That’s what I thought, until I called the fuel depot. “No truck. You guys terminated our contract!” said my contact at the fuel depot. “What? That doesn’t make any sense. I’ll get with the contract guys, but in the meantime, can I get the truck out? We’re running pretty low,” I said, my jaw almost hitting the floor. “Not a chance,” said my contact at the fuel depot. “Cash up front is the only way we can do business without a contract on file, and you guys terminated the contract. Send us a wire, and we’ll deliver fuel. Here are the wire instructions” said my contact at the fuel depot.
Obviously, I didn’t have the ability to wire thousands of dollars of company money to buy fuel, so I called Accounts Payable. They could pay an invoice, but it needed a purchase order. That was the responsibility of Procurement. I contacted Procurement, and they couldn’t issue a purchase order without a contract. They referred me to Contracting. Contracting notified me that they couldn’t issue a contract because they had terminated the contract at the instruction of Compliance. They suggested I talk to Compliance. I called Compliance and left a message, marking it as urgent. In our voicemail back-end system, I could see that Compliance had 143 other urgent voice messages, so I was guessing this might take awhile. OK, fine. Time to notify Compliance Legal. In the state of Washington, we’re required to notify the Utilities and Transportation Commission of critical utility outages, with an explanation as to what caused them. I left a voicemail for Compliance Legal letting them know that I would be filing a major outage report with the UTC if I didn’t hear back from them.
Well, that set off a firestorm. Compliance Legal called me back, and fast! They absolutely did not want me to file an outage report. Could I do anything to prevent this? “Yes. Help me get some fuel,” I said. “Our fuel service contract was cancelled by your department. I have no idea why.” Now, as it turns out, people in Compliance pick up the phone when it’s their own Legal team calling. The next morning, with only 15% fuel remaining, I was finally able to arrange fuel delivery and find out what happened (we paid up front just like the fuel depot asked).
Our fuel services vendor has the same name as a different fuel services company in Malta. It’s not the same company, isn’t run by the same people, and has nothing to do with them. However, there is also an obscure division of the Department of the Treasury called the Office of Foreign Assets Control, aka OFAC. They enforce financial sanctions using a watchlist—you know, sort of like the TSA no fly list and various terrorist watchlists. The OFAC list used to have a few obvious terrorists and terrorist organizations on it, along with countries sanctioned by the US (think Iran, North Korea, Cuba, etc.). These days, it’s a 13.4 megabyte file with thousands of names (including Mike. Just “Mike.” Sorry if your name is Mike). And as you have probably guessed, a soundalike Maltese fuel services company is on the OFAC block list. Now, if anyone involved in this had any common sense at all, they would have investigated a bit more before cutting off my fuel supply. But this is the corporate office we’re talking about, and they bought an AI tool to ensure compliance. So, compliance with an outage report was nearly ensured.
And with that, enjoy your winter. For my part, I still haven’t figured out why our PDU tech hasn’t arrived.
References
USDA Rural Electrification Administration Power Requirements For Digital Central Office Equipment: https://www.rd.usda.gov/files/UTP_Bulletins_1751E-302.pdf
FCC Order 07-177 (Hurricane Katrina Commission): https://docs.fcc.gov/public/attachments/FCC-07-177A1.pdf
Washington Utilities and Transportation Commission Outage Reporting: https://www.utc.wa.gov/regulated-industries/utilities/telecommunications/telecom-reporting-requirements/telephone-outage-reports
OFAC Specially Designated Entity list: https://www.treasury.gov/ofac/downloads/sdnlist.txt
https://ofac.treasury.gov/faqs/topic/1516 – What to do if your credit report is OFAC flagged
Disclaimer
Telecom Informer columns are fictional accounts based on true stories from the world of telecom. Any resemblance to actual people or events is entirely coincidental.