Tag Archives: security

Equifax Data Breach: Lessons We Must Learn

Learn more about transforming people, process and culture with the Real Agility Program

I woke up this morning to the news that my personal data has been leaked by Equifax to…who knows!?  I was reminded of the following tweet:

So, what “awful circumstances” led to Equifax’s recent breach?

Let’s reflect:  News has surfaced (TechCrunch, Reuters) that hackers have scraped untold amounts of sensitive data from Equifax systems.  143+ million people are affected as hackers have amassed a huge database of names, addresses, credit records, license numbers, banking histories.  (That probably includes you too!)

I want to be clear though, the news broke yesterday but the problem started long ago.  The security vulnerability has existed for (probably) years and I have no doubt some Equifax staff have known about it.

Equifax!  We’re not talking about some high-school project with junior coders and tech newbs.  We’re talking about one of the world’s most trusted organizations.  We’re talking about a company whose very existence (their whole business!) is to protect our collateral.  This is supposed to be one of the best-funded, most secure, most technologically-advanced companies on the planet.

But I’m not surprised.   Here’s why…

I teach Scrum and my classrooms are filled regularly with people who work in companies exactly like Equifax.  I hear their stories every day:

“Our managers don’t provide the tools we need to do the job.”

“Our managers don’t understand the time required to deliver high-quality software.  We’re always pressured to cut corners to meet arbitrary and impossible deadlines.”

“Our systems are broken, everyone knows it, but managers continue to outsource and off-shore our QA.”

“We don’t have authority to decide the implementation, we’re often told what to implement by architects and supervisors, even if we know it to be rotten.”

“Our managers never ask us about quality…they ask us only ‘when will this be done’?”

And that’s the crux of the problem: people are hired by companies like Equifax to provide technical expertise, then their advice is ignored and their implementation decisions aren’t trusted.

Some important questions to consider…

1. Does Equifax lack the money to hire excellent technical staff?

No, their offices are filled with some of the best programmers in the world.  I meet them (or people like them) regularly in my classes and I have no doubt that the technical staff at Equifax have warned the managers for years of security holes and technical defects.  I have no doubt those managers have ignored the alarms and have pushed the staff to deliver deficient code.

2. Does Equifax lack the time to build high-quality systems?

No, last I checked they’ve been at it a long time and their existing contracts will carry their operation years into the future.  And as mentioned earlier, securing our data is the reason the company exists.  It’s  the one thing they’re supposed to get right – I’d think their time should be entirely devoted to building high-quality systems.

3. Does Equifax lack the financial resources to invest in proper tools and training for security/quality testing?

No, such techniques and tools are widely available and inexpensive (even hackers can afford them!)  Managers at Equifax have denied budgets for training, tools, and upgrades because “it’s too costly” – hmm…I wonder the cost of this data breach?

4. And my favourite question of the day: Are the hackers “smarter”?

Absolutely not.  But they’re more dedicated and have equipped themselves with good techniques and tools for penetration testing.  In my personal experience as a hacker (er, I use that term loosely) security holes are all around us if we look for them.  Equifax simply wasn’t looking!

What to do about it…

First, it’s clear to me the problem isn’t technical or financial.  It’s cultural.  I see it all the time.  Teams of good product developers are surrounded by bureaucracy instead of support.   Teams of good coders aren’t trusted to see the source code of the systems used by the company – yes, that’s as crazy as it sounds!  Teams of good coders are kept silent about the security vulnerabilities they see.  Solutions are ignored by management and the arguments are: “improving the security isn’t a priority right  now” or “we know there are some possible security concerns and we are in discussion with vendors or outside agencies to address it” or “we have a budget for security improvements scheduled for next quarter; let’s focus for now on new features instead”.  Managers are more concerned with deadlines than with quality.  Managers scrutinize the number of hours a developer works on a task, and outsource or off-shore the quality assurance and testing!  Managers conduct endless planning activities then compress the implementation into tight budgets and timelines – evidently, a lot of energy is spent getting the plan “right” but getting the software right is not a priority.  I could go on.

If you’re interested to know how things work at Equifax, just think of the Dilbert cartoons.  I mean it.  I am very serious.  Dilbert isn’t funny because it’s fiction; it’s funny because it’s NON-fiction.  Sadly. Typically, for enterprises like Equifax, their technical staff and customers take a back seat to management “theatre”.  This needs to be fixed and it starts by asking the technical staff a single, simple question:  “Who among you have raised concerns about technical debt with your managers/supervisors and were ignored?”  That question will unearth bugs which have been deprioritized by managers, budgets that have been denied for technical training and automated testing, projects which have been reported as “done” before they were actually ready for deployment – in other words, that question will reveal the many (fixable) ways managers get in the way of quality.

Second, executive staff at Equifax need a crash course in automated testing.  Yes, THE EXECUTIVE STAFF!  It’s is essential they understand and see with their own eyes that:

  1. Automated testing is cheaper and exponentially more effective than manual testing;
  2. ALL defects are discoverable and fixable before hitting production environments;
  3. Quality is not something one outsources
  4. and the techniques to achieve ZERO DEFECTS are well-known, teachable, repeatable, and proven.   I’m of course referring to techniques like Test-Driven Development, Continuous Integration, Refactoring, and Swarming.  For example, these technical topics form the bulk of our Certified Scrum Developer classes. (Shameless plug.)

And third, technical staff need to stop behaving like sheep.  So far in this article, I’ve been very critical of managers, sure, and anyone who knows me personally knows I have no time for inept management.  But too often I meet smart, well-meaning developers who deliver shoddy code – perhaps at under pressure and against their better judgement, but in the end whose code is it?  Developers! I understand you might feel trapped in a pattern of quantity-over-quality and you are frequently coerced by your management to cut corners.  I get it… I understand it… it’s easy to feel that deadlines are some sort of immutable truth and that managers wield all the power.  But the fact is, developers, YOU hold all the responsibility and therefore you need to be the professional.  You need to say “no” and demand the latitude you require to deliver high quality.  You’re the one closest to the code and therefore directly responsible for the safety and well-being of your users.

So, Equifax and enterprises everywhere, I’m speaking now as your user or stakeholder or customer…

Equifax has failed. Miserably. The company deserves all the class-action suits coming there way. From leaders to developers. Everyone.

Most members of society are unwilling participants in all this.  Most of us aren’t your direct customers.  Example: I’m not a direct customer of Equifax – nobody has chosen Equifax as their personal agent.  Instead, our banks and our governments have selected Equifax on our behalf.  This presents a problem: if I were a direct customer of Equifax I’d call them today and close my accounts; but I can’t do that.  Instead, the best I can do as an individual is contact my banks, lenders, and insurance agents to demand change.  (Yes, I likely will do that.  I’m that sort.)

However, the larger issue is that we are at the mercy of YOU who produce software.  I’m talking about the software in our vehicles, in our heart-monitors, in our subway systems, in our air-traffic-control centres, in our banks – this is serious stuff!  We must be able to trust those systems…with our lives, with our security.  We must be able to trust you even though we don’t and won’t ever know you.

A hacker friend of mine once said, “if self-driving cars are being produced without complete automated test coverage, then that’s a future I don’t want.”

In this day and age, low quality is intolerable.

Please share!

Quality is not an attribute, it’s a mindset

Learn more about transforming people, process and culture with the Real Agility Program

This was actually cribbed from a Bruce Schneier blog post about security…

Security engineers see the world differently than other engineers. Instead of focusing on how systems work, they focus on how systems fail, how they can be made to fail, and how to prevent–or protect against–those failures. Most software vulnerabilities don’t ever appear in normal operations, only when an attacker deliberately exploits them. So security engineers need to think like attackers.People without the mindset sometimes think they can design security products, but they can’t. And you see the results all over society–in snake-oil cryptography, software, Internet protocols, voting machines, and fare card and other payment systems. Many of these systems had someone in charge of “security” on their teams, but it wasn’t someone who thought like an attacker.  

There’s an interesting parallel between this statement and how most software quality is handled. Quality and Security are similar. In fact, I see security as a very specific subset of quality-mindedness. Certainly both require the same mindset to ensure – rather than thinking merely “how will this work”, a quality-focused person will also, or perhaps alternately think: “how might this be breakable”. From this simple change in thinking flows several important approaches

  • Constraint-based thinking (as opposed to solution based thinking): allows an architect/developer to conceive of the set of possible solutions, rather than an enumeration of solutions. By looking at constraints, a developer implements the lean principle of deciding as late as possible, with as full information as possible.
  • Test-First: As one thinks of how it might break, scenarios emerge that can form the basis of test cases. These cases form a sort of executable acceptance criteria
  • Lateral Thinking: The constraint+test approach starts to get people into a very different mode, where vastly different kinds of solutions show up. The creative exercise of trying to break something provides insights that can change the whole approach of the system.

 Schneier goes on to ponder 

This mindset is difficult to teach, and may be something you’re born with or not. But in order to train people possessing the mindset, they need to search for and find security vulnerabilities–again and again and again. And this is true regardless of the domain. Good cryptographers discover vulnerabilities in others’ algorithms and protocols. Good software security experts find vulnerabilities in others’ code. Good airport security designers figure out new ways to subvert airport security. And so on.  

 Here again – I think it’s possible to help people get a mind-set about quality, but some do seem to have a knack. It’s important to have some of these people on your teams, as they’ll disturb the waters and identify potential failure modes. These are going to be the ones who want to “mistake proof” (to borrow Toyota’s phrase) the system by writing more unit tests and other executable proofs of the system. But most importantly (and I can personally testify to this) it is critical that people just write more tests. It is a learned skill to start to think of “how might this fail” until it becomes a background mental thread, always popping up risk models.A related concept is Demmings’ “systems-thinking”, which, applied to software quality, causes one to start looking at whole ecosystems of error states. This is when fearless re-factoring starts to pay off, because the elimination of duplication allows one to catch classes of error in fewer and fewer locations, where they’re easier to fix. There are many and multifarious spin-off effects of this inverted questioning and the mindset it generates. Try it yourself. When you’re writing code, ask yourself how you might break it? What inputs, external state, etc. might cause it to fail, crash, or behave in odd ways. This starts to show you where you might have state leaking into the wild, or side-effects from excessively complex interactions in your code. So quality focus can start to improve not only the external perception of your product, but also its fitness to new requirements by making it more resilient and less brittle. Cleaner interactions and less duplication allow for much faster implementation of new features.I could go on, but I just wanted to convey this sense of “attitude” or “mindset,” over mere technique. Technique can help you get to a certain level, but you have to let it “click”, and the powerful questions can sometimes help.

Please share!