How a Foreign Sanctions Block List Nearly Shut Down An Entire US Community’s Phone Service

Originally published in 2600: The Hacker Quarterly, Winter 2023-2024

Hello, and greetings from the Central Office! It’s winter, which means power interruptions here in the Great Northwest. Most of our power (and telephone) lines are above ground, and the whole region is covered with trees that average over 100 feet tall. Trees and branches are falling constantly causing power interruptions, especially during the fall and winter. It has been getting worse in recent years, though, given that summer weather keeps getting hotter and the rainy season is ever shorter. Making matters worse, we have been getting more “atmospheric rivers” as of late, which deliver several inches of soaking rain at a time, saturating the ground. When a windstorm happens after this, large trees like cedars with shallow root systems simply blow over. They fall onto cars, houses and (of course) power lines. I’ll give you one guess whose power line was taken out by a tree today. If you guessed a Central Office with “Forest” in the name, you’d be right. I’m acting in an incident response capacity today, dealing with possibly the strangest incident that I have ever experienced in my career. But we’ll get deeper into that later. For now, let’s talk about power engineering.

Obviously, downed trees and power interruptions are nothing new in the Pacific Northwest, and we have been prepared for them for a long time–including in the Central Office where I am working today. Now, you probably don’t think of the USDA as a telecommunications regulator (the FCC writes most of the rules), but they have made a significant mark on the telecommunications landscape. I doubt that the company would have been so well prepared for emergencies if it wasn’t contractually required!

If you’re scratching your head, I’ll explain. The USDA, through the Rural Electrification Administration provides subsidized financing to telephone companies. These subsidies were intended to serve rural areas, but as the population of the Pacific Northwest grew, the Company was very effective in its lobbying to secure financing for suburban and exurban locations in its service territory. This saved hundreds of millions of dollars in interest, which could instead be used to buy back shares of company stock and help executives meet their bonus targets.

In a rare case of CEO incentives aligning with public good, the USDA loan program has underwriting requirements which enforce minimum Central Office construction standards. In fact, they publish an entire reference engineering guide, and this includes power supply and backup power requirements. With peak loads assumed, the USDA requires either 8 hours of backup battery power, or 3 hours of battery power plus a diesel generator. In 2007, the FCC eventually weighed in after Hurricane Katrina with Order 07-177, releasing a loophole-ridden, watered down rule applying to all telecommunications facilities (not just ones funded by the USDA). It was then immediately challenged in court.

This particular Central Office is equipped with a 3 hour backup battery system and a diesel backup generator. One room on the ground floor of the Central Office is dedicated to our backup battery system, and the generator is located outdoors. Both systems are required to provide enough power to run the Central Office during peak load, and do so effectively. There are varying requirements on how much of a fuel supply we need, but this Central Office was constructed in 1982, not long after the Mount St. Helens eruption (which happened in 1980). This was obviously fresh in the minds of the engineers who designed subsequent Central Offices. Without knowing what regulations might be forthcoming, they provisioned 72 hours of peak load fuel storage on-site. In practice, we have about 5 days of fuel, because the Central Office doesn’t run at peak load at all anymore, and especially doesn’t do so for 24 hours per day. As with our backup battery contract, we have a fuel services and maintenance contract with an outside vendor.

This all sounds great, right? It’s all good in theory. It’s also good in practice: this stuff is regularly used! We have weather events all the time during Pacific Northwest winters. Most of the problems I’d normally encounter would result from deferred maintenance or a component failing. This time, though, a tree is down, we don’t have utility power, and the clock is ticking. You see, our compliance department abruptly ended our contracts with our fuel service vendor who also happens to be the only fuel depot in the region. The fuel depot doesn’t know why, Compliance wouldn’t tell me why, and it finally took a Washington Utilities and Transportation Commission rule to prompt a solution.

I showed up 3 days in, while we were running on backup power. A landslide had taken out our utility feed to the Central Office, road access to the utility lines, and several utility poles along with it. Although some of our outside plant was impacted, we were lucky: we only had a few subscribers in that direction. Our power was out, though, and it was going to be out for awhile—at least two more days. This wouldn’t normally be a problem; after all, we are well prepared with backup power, and the Central Office itself was accessible via other routes. When I arrived, we had about 1/3 of our fuel remaining for the backup generator, and a battery backup operating with a failed (but redundant) PDU. Normally, none of this would be a big deal. PDU failures happen, especially when switching over to generator power, which is why we have redundant ones. We’d want to fix it before we switched back to utility power (in case the same thing happened again with the switchover), but that was manageable. Fuel is normally no problem, since roads were open and critical utilities like telephone service have priority on constrained supplies (which, in this case, weren’t even constrained). We just needed to call the fuel depot for delivery, and call our electrical vendor to fix the PDU.

This is all easy stuff—well, not easy, but manageable. We have a standard operating procedure for it. All of the contracts are in place. Should be a cake walk, right? That’s what I thought, until I called the fuel depot. “No truck. You guys terminated our contract!” said my contact at the fuel depot. “What? That doesn’t make any sense. I’ll get with the contract guys, but in the meantime, can I get the truck out? We’re running pretty low,” I said, my jaw almost hitting the floor. “Not a chance,” said my contact at the fuel depot. “Cash up front is the only way we can do business without a contract on file, and you guys terminated the contract. Send us a wire, and we’ll deliver fuel. Here are the wire instructions” said my contact at the fuel depot.

Obviously, I didn’t have the ability to wire thousands of dollars of company money to buy fuel, so I called Accounts Payable. They could pay an invoice, but it needed a purchase order. That was the responsibility of Procurement. I contacted Procurement, and they couldn’t issue a purchase order without a contract. They referred me to Contracting. Contracting notified me that they couldn’t issue a contract because they had terminated the contract at the instruction of Compliance. They suggested I talk to Compliance. I called Compliance and left a message, marking it as urgent. In our voicemail back-end system, I could see that Compliance had 143 other urgent voice messages, so I was guessing this might take awhile. OK, fine. Time to notify Compliance Legal. In the state of Washington, we’re required to notify the Utilities and Transportation Commission of critical utility outages, with an explanation as to what caused them. I left a voicemail for Compliance Legal letting them know that I would be filing a major outage report with the UTC if I didn’t hear back from them.

Well, that set off a firestorm. Compliance Legal called me back, and fast! They absolutely did not want me to file an outage report. Could I do anything to prevent this? “Yes. Help me get some fuel,” I said. “Our fuel service contract was cancelled by your department. I have no idea why.” Now, as it turns out, people in Compliance pick up the phone when it’s their own Legal team calling. The next morning, with only 15% fuel remaining, I was finally able to arrange fuel delivery and find out what happened (we paid up front just like the fuel depot asked).

Our fuel services vendor has the same name as a different fuel services company in Malta. It’s not the same company, isn’t run by the same people, and has nothing to do with them. However, there is also an obscure division of the Department of the Treasury called the Office of Foreign Assets Control, aka OFAC. They enforce financial sanctions using a watchlist—you know, sort of like the TSA no fly list and various terrorist watchlists. The OFAC list used to have a few obvious terrorists and terrorist organizations on it, along with countries sanctioned by the US (think Iran, North Korea, Cuba, etc.). These days, it’s a 13.4 megabyte file with thousands of names (including Mike. Just “Mike.” Sorry if your name is Mike). And as you have probably guessed, a soundalike Maltese fuel services company is on the OFAC block list. Now, if anyone involved in this had any common sense at all, they would have investigated a bit more before cutting off my fuel supply. But this is the corporate office we’re talking about, and they bought an AI tool to ensure compliance. So, compliance with an outage report was nearly ensured.

And with that, enjoy your winter. For my part, I still haven’t figured out why our PDU tech hasn’t arrived.

References

USDA Rural Electrification Administration Power Requirements For Digital Central Office Equipment: https://www.rd.usda.gov/files/UTP_Bulletins_1751E-302.pdf

FCC Order 07-177 (Hurricane Katrina Commission): https://docs.fcc.gov/public/attachments/FCC-07-177A1.pdf

Washington Utilities and Transportation Commission Outage Reporting: https://www.utc.wa.gov/regulated-industries/utilities/telecommunications/telecom-reporting-requirements/telephone-outage-reports

OFAC Specially Designated Entity list: https://www.treasury.gov/ofac/downloads/sdnlist.txt

https://ofac.treasury.gov/faqs/topic/1516 – What to do if your credit report is OFAC flagged

Disclaimer

Telecom Informer columns are fictional accounts based on true stories from the world of telecom. Any resemblance to actual people or events is entirely coincidental.

 

A New Connected World Of Mobile-Enabled Sensors (Winter 2014-2015)

Hello, and greetings from the Central Office! Since I wrote last, I have been around the world clockwise once again. It was good to catch up with friends and fellow hackers in Europe and China, and to visit the amazing technology markets in Beijing. Technology changes very rapidly in China and despite being only 6 months from my previous visit, I was really surprised to see how much has changed.

basic GSM handset image

This low cost Arcci GSM phone sells for under $8

One of the most exciting recent developments in telecommunications is the astonishing price drop in mobile phone chipsets, particularly for basic GSM technology. This is combined with massive improvements in both battery technology (which has gotten much greater), charging technology (which can reliably operate off of inexpensive solar cells), and power consumption (which has dropped). In Beijing, you can now buy a brand new, quad band, unlocked GSM world phone for less than $8. These phones can remain powered on, able to make and receive calls, with a standby time of up to two weeks in between charges. Talk time is also truly astonishing. I remember when I barely got an hour of talk time on my enormous Motorola brick analog cellular phone, but basic GSM phones now boast talk time of up to 8 hours of continuous usage—if your voice can hold up for that long!

Just stop for a minute and think about that. For under $15, you can buy a phone that works anywhere in the world for voice, text, and data, and a solar charger to go with it, and even if you don’t charge the phone for 2 weeks, it’ll still be able to make and receive text messages and can even log onto the Internet. It’s completely mind-blowing when you think about it. I think the only reason that most people in Western countries haven’t noticed is that because handsets like these aren’t widely available in wealthier places. When your mobile phone carrier’s lineup is populated with the latest smartphones, it’s hard to notice the availability of no-name Chinese brands at astonishingly low prices.

Now, let me be clear: these inexpensive phones aren’t smart phones, and they don’t support even 3G, let alone 4G technologies. However, they do work just fine for voice, low-speed GPRS data and SMS messaging. And this is the retail price, and even includes value added tax! The wholesale price is about half of this, and it’s for a fully assembled phone. So, you can infer that the component parts are even less expensive than this. Want to support the latest networks and fastest data speeds? The price is about 5 times as much, but we’re still talking about $30 for the components. Making things even more interesting, you don’t necessarily need all of the component parts involved in building a phone when you consider GSM scenarios that aren’t phone calls.

“Wait, a minute,” you may ask. “GSM scenarios that use mobile phone components but don’t involve making phone calls, you say? What might those be?” Well, actually, that’s where things have gotten really interesting. Given the confluence of low cost, low power requirements, and creative charging solutions, some new and really exciting scenarios have been unlocked. Sensors are quietly but steadily being deployed to help automate everything from water and electric meter reading to weather monitoring.

Sure, sensors have existed in various forms and in various places for many years, and there have even been previous efforts at “smart meters.” However, there have been a number of key issues. First of all, most sensors had very limited computing power, because the availability of low-cost microcontrollers with low power consumption was limited. So, the technology was there to gather data, but interpreting it had to be done in a centralized location somewhere; you couldn’t fit enough computing power on a sensor to do much meaningful interpretation. Today, with the availability of Arduino and similar microcontrollers, it’s possible to build sensors with substantial onboard computing resources, without needing a whole lot of energy to do it. This means that sensors don’t necessarily have to upload as much data to centralized locations for real-time processing anymore, because software can be more capable at making real-time decisions. Even if you didn’t need to continuously gather data, or needed to centralize processing, the capability didn’t exist to process data over a wireless WAN at high speed. Nowadays, GSM coverage is available almost everywhere, and 4G allows data transfer at speeds similar to WiFi. This combined with the plummeting cost of sensor technology has unlocked some really incredible new scenarios. Some of the most interesting innovations are in utilities and—oddly enough—agriculture.

Arduino picture

Arduino is an ideal platform for many embedded systems.

Many utilities around the country are starting to deploy smart meters, to which the tinfoil hat crowd has responded with predictable fury (they’re mainly concerned about RF emissions). The Salt River Project in Phoenix has already deployed them in most areas, and the Los Angeles Department of Water and Power is beginning to deploy these as well. While the key reason to (and most important application for) implementing the technology is eliminating the need for meter readers, smart meter technology also allows more data to be collected about energy usage, and more creative billing to take place. You might recall that long distance charges used to vary by time of day and day of week. Calls were billed based on a day rate (the highest price), evening rate (around 20% less), and nights or weekends (around 50% less). This was done to provide an incentive to shift usage to off-peak times, so the phone company didn’t have to build a lot of peak capacity that was otherwise underutilized. Your electric utility could offer similar incentives to use power during off-peak times. For example, Sunday evening is the period of lowest power usage in most cities. So, you might choose to do your laundry on Sunday evening if the rate were half as much as doing it on Monday morning.

crop moisture sensor

This sensor monitors crops for moisture

Agriculture is also seeing a lot of really interesting new scenarios in wireless sensors, which are helping to reduce waste and improve efficiency. For example, farmers waste hundreds of millions of dollars a year replacing spoiled livestock feed. Farmers buy feed and put it in storage. The feed gets wet for one reason or another, and then it spoils. Typically, farmers will find out that this happened when they go to use the feed and find that it has spoiled. So, a company called Kongskilde has developed several types of moisture, temperature and humidity sensors that can be stored with the feed. So, if a leak in the roof develops, the sensors will detect this and notify the farmer before his feed becomes spoiled.

Both of the above smart devices typically rely on a local mesh network, typically WiFi, which then uplinks data to a centralized location via mobile Internet. However, there has been a lot of recent research (with some development) on sensors that communicate directly via mobile Internet. Given the water crisis in California, one of the most interesting pieces of research I have seen involves irrigation systems that are sensor-controlled. Most irrigation systems today operate on timers, and the amount of water used isn’t an exact match for what is actually needed. So, most farmers over-water or under-water their crops (typically the former), which isn’t good for either the crops or the water supply. However, given the vast distances, mesh networks don’t make a lot of sense. These devices, along with other smart devices such as pH monitoring, can literally be “planted” along with crops. The power source? Often solar. In the case of irrigation, the amount of water sprayed can be precisely correct for the exact soil moisture level, leading to both higher crop yields and lower water usage. How can we continue to feed a rapidly expanding human population? Technologies like these will go a long way toward doing so, and they’re all enabled by telecommunications.

And with that, it’s time for me to finish eating this turkey sandwich. Hope you had a happy Thanksgiving, and best wishes for the new year! The world only gets more exciting every day.

References

https://www.youtube.com/watch?feature=player_detailpage&v=qTqi9xnMf3U – A Smart Meter video from BC Hydro, which provides a good overview of the features and services brought by smart meters.

http://www.bchydro.com/energy-in-bc/projects/smart_metering_infrastructure_program.html?WT.mc_id=rd_smartmeters – Excellent FAQ and information from BC Hydro which in particular describes the science of smart meters. Designed for the tinfoil hat crowd.

http://www.cityofgreensburg.com/MiNet.pdf – Excellent technical whitepaper on Mueller Systems smart meters.

http://www.kongskilde.com/Agriculture/Grain/SensSeed/SensSeed/Wireless%20sensor%20system# -Many technical whitepapers, along with sales brochures, for the Kongskilde agricultural sensor system.

http://ijarcsms.com/docs/paper/volume2/issue1/V2I1-0007.pdf – Detailed academic paper describing a prototype GPRS-based sensor network for irrigation.