Caution is dangerous, build guardrails

Too many times I’ve heard the phrase, “Just be careful.” Whether running queries on production, tweaking live code, or searching through files, all too often we are asked to take needless risks. If something goes awry, it’s our fault. We weren’t “cautious enough.” That’s just wrong.

Years ago, I was working on a VB.NET application and discovered a major problem. The application was leaking PII into the Application Logs. This meant that unencrypted sensitive information was being stored in the least secured environment. It was a hacker’s dream.

The offending code was simple:

public void Routine(Customer customer)
{
    _logger.Info($"Executing routine for {customer.first} {customer.last} - {customer.ssn}");

    // Application Logic
}

There are numerous fixes for this, the simplest being to remove the logging statement. That’s not, however, the right fix. That fix is predicated on bad assumptions:

The issue exists because the previous developer was incompetent.

We have no way of knowing if the previous developer was incompetent. For all we know, they may have spent two days without sleep in a caffeine fueled rush to solve a critical production issue. That logging may have been integral to fixing the problem and they may have intended but forgot to remove it. That’s not incompetence, it’s exhaustion.

The current developers won’t make the same mistake.

The current developers may find themselves in the exact same circumstances and repeat the mistake. Assuming that you won’t screw up is more than hubris, it’s courting disaster. You ignore, rather than account for, your fallibility.

Fixing this issue is a complete solution.

There may be other leaks in the code. We can purge the logs, grep the code for every variant of “ssn”, “social”, “socialSecurityNumber”, etc. and still miss a leak. It’s entirely possible that the value is passed into some variable called “subjectString” or a similarly obfuscated name, and then logged. Even if we grep the logs for leaked SSNs and use it to explore the code, there may be some condition which has not yet occurred but will cause a leak.

This fix and these assumptions are an example of the “be cautious”, “don’t do bad things”, and “don’t make mistakes” approach. They eliminate a single problem, ignore the possibility of unknown or potential problems, and blame a person instead of a process. This is especially wrong when the solution was so simple!

public class SingleUseString
{
    private string _value;
    public SingleUseString(string value)
        => _value = value;

    public override string ToString() => "IMPROPER USE OF SingleUserString!";

    public string Consume()
    {
        if (_value is null)
            throw new ApplicationException("An attempt has been made to use a SingleUseString more than once!");
        var value = _value;
        _value = null;
        return value;
    }
} 

In dotnet, every object implements ToString(). It’s an essential feature and fundamental to any logging framework out there. No matter what object you have, a logging framework can get some text representation of it. By overriding the ToString method for this type and passing a constant message, we ensure that we can pass objects of this type to any logging framework without worry. The internals won’t leak.¹

Even better, we now have a non-standard method for retrieving the value. This means we have a single (and traceable) method across the entire codebase. The compiler can instantly tell us every place where the underlying value is exposed.

Could we still have a leak? Yes! It’s possible that some developer could call Consume(), grab the SSN, and log it. It’s dramatically less likely. It would need to be a deliberate choice instead of an easy mistake. Beyond that, we’d know it had happened. Rather than (hopefully) stumbling across PII in the logs, we’d get a process crash when the appropriate method tried to consume the (now purged) inner value.

That’s the power of guardrails. They’re not perfect but they’re dramatically more reliable than tiptoeing. Transporting PII as SingleUseStrings gave us protection from common logging leaks, reduced the chance of any accidental leak, and a created leak detection system. Would you rather have all that, or yell at someone for being human?

This is a simple example of defensive coding but the mindset applies everywhere. When joining a team (especially as a lead), I’m often handed a set of credentials for the production database. My first question, “Why?” Why give me something that allows me to accidentally cause a major production issue? If I must have access, why give me anything but read access?

On that, never issue a single set of production DB credentials. Issue read-only credentials and separate (preferably temporary and task-scoped) write credentials. It dramatically reduces risk.

If someone tries to give you unnecessary write access to production, refuse. Would you rather your customer be upset that you’re not cooperating, or that you accidentally took prod offline?

Caution alone is a losing strategy. I’d say it’s a gamble but mistakes are inevitable. At best, you’re gambling on what mistake will be made and how much damage will be done.

Perfect protection doesn’t exist but guardrails are the next best thing. Accept that mistakes will happen and build systems which are resilient in the face of human error.

1. Also, if we see "IMPROPER USE OF SingleUserString!" in the logs, we know someone’s at risk of making a mistake. Time for training.