Infrastructure Lessons from Texas

TL:DR Design for graceful failure. Avoid global dependencies.

This past week, millions of Texans were left without power or water during the worst cold snap in a decade. There are so many lessons here.

As I understand it, the bulk of Texas power comes from natural gas. As many people know by now, electric grids cannot store electricity. Supply must equal demand at all times. During the storms, unexpected demand from the cold temperatures met an unexpected lack of generation due to multiple causes, and the only way to keep the grid up at all was to drastically shed load. This is supposed to be an organized process, but things you don’t test test not to work when you need them. Instead of everyone suffering short outages, some people lost power for days.

Texas “conservatives” were quick to blame, for example, AOC (huh?) for this, and then to blame wind power for dropping offline. In fact, wind and renewables account for less than 10% of Texas generation and the availability of renewables was better than fossil and nuclear. The actual cause of a lot of fossil generation losses was a lack of winterized equipment, so instrumentation froze, and a lack of natural gas to power generators, due to … high demand because it was cold and a lack of winterized equipment in the gas industry, so wells froze. It turns out that Texas natural gas generation plants do not store much gas locally, so they had to shut down.

The next problem is water. The low temperatures and lack of winterization caused unexpected pipe and water main breaks, and treatment plants shut down due to lack of power. Shut down plants and low pressure have caused uncertainty about the safety of what water there is, so people are supposed to boil water. Of course they have no electricity or gas, so…?

I think the first lesson for public infrastructure is that you have to spend money to make your system reliable even when conditions are unusual. Cold areas have utilities, water, and electricity that work in subzero temperatures. That part is well understood.

Second, your part of the system must not depend on remote services that you do not control. A water treatment plan must have local emergency power with fuel enough to ride through an extended outage. Gas power plants should either have several days local storage of gas or reliable access to stored gas. Probably you don’t want giant gas storage tanks in the middle of cities.

Third, hospitals, police and so forth must have emergency power and communications that does not depend on outside services.

There has been a lot of humorous writing about Texas’ decision to not join the national grids. I don’t think the interconnections would have been enough to overcome a lack of local generation, but what would have helped is adherence to national standard for reliability. El Paso did winterize their services, according to standards, and they did just fine.

It’s fine with me if Texas wants its own grid, don’t mess with Texas! But it is kind of sensible to track the national standards anyway. Texas had a similar cold event in 2011 or something, and many reports were written and ignored. That part is on Texas politicians.

It is tempting to point out that Republicans claim that government is useless, and then once they get into power proceed to prove it, but I’ll resist.

Here in Massachusetts, the power and gas has been reasonably reliable since the utilities started actually trimming the trees that tend to fall on the wires. Nevertheless, we have a wood stove and a pile of split wood and a 6KW generator that gets used once or twice a year, and a second small generator in case the first one is busted. As soon as we can afford it, we plan to get batteries to be charged by the solar panels on the roof as well. Central utilities are great, but you shouldn’t depend on them 100%.

In computing, there’s a saying that a distributed system is one in which the failure of a machine you didn’t even know existed can keep you from getting your work done.

I think it is quite hard to predict the results of unusual combinations of events. It is much easier to provide backup systems to prevent failures from cascading and becoming disasters. It takes money and periodic testing.


TLDR – when someone calls you and then asks you to authenticate yourself, they are doing it wrong. DO NOT ANSWER.

A while ago, I got a call from a brokerage house I use (Hello Vanguard!). The caller asked me for the answer for one of my challenge questions, to make sure I was actually me.

I burst out laughing.

This a surprisingly subtle issue, and to have a major brokerage get it wrong is both sad and scary.

The caller is the unknown party. The called person is not, at least with the current way the phone system works. Caller ID is easily spoofable. You cannot trust that a caller is who they claim to be.

As long as the phone system is ringing the correct phone, the recipient should be, if not the exact person you want, then someone nearby. There are certainly exceptions to this, such as SIM card hijacking, which is sadly easy as well, but for the most part, if you call 1-800-BIG-CORP from a phone, you are going to get the right people.

The reverse is not true. If 1-800-BIG-CORP calls you, you have no reason to believe it is really them. You must not give away ANY secret information. You must call back, using a number you find out by yourself, NOT one given to you over the phone.

Why is this important? If the caller is actually a scammer trying to break into your account, when they come to the “secret question”, they just call you pretending to be the bank and ask you for the answer! Don’t give it out.

The caller can be quite inventive about trying to convince you they are legitimate. Krebs on Security reports on recent cases in which callers knew details of recent transactions, for example, (see Krebs gives the right advice, look up the number from public sources and call back, but he doesn’t explain the general principle.

The caller must authenticate themselves, not the callee.

This is also the reason why you should never click a link in an email message. There is no reason to trust such a thing. You must look up the link yourself, from public sources.