Tag Archives: risk

Equifax Data Breach: Lessons We Must Learn

I woke up this morning to the news that my personal data has been leaked by Equifax to…who knows!?  I was reminded of the following tweet:

So, what “awful circumstances” led to Equifax’s recent breach?

Let’s reflect:  News has surfaced (TechCrunch, Reuters) that hackers have scraped untold amounts of sensitive data from Equifax systems.  143+ million people are affected as hackers have amassed a huge database of names, addresses, credit records, license numbers, banking histories.  (That probably includes you too!)

I want to be clear though, the news broke yesterday but the problem started long ago.  The security vulnerability has existed for (probably) years and I have no doubt some Equifax staff have known about it.

Equifax!  We’re not talking about some high-school project with junior coders and tech newbs.  We’re talking about one of the world’s most trusted organizations.  We’re talking about a company whose very existence (their whole business!) is to protect our collateral.  This is supposed to be one of the best-funded, most secure, most technologically-advanced companies on the planet.

But I’m not surprised.   Here’s why…

I teach Scrum and my classrooms are filled regularly with people who work in companies exactly like Equifax.  I hear their stories every day:

“Our managers don’t provide the tools we need to do the job.”

“Our managers don’t understand the time required to deliver high-quality software.  We’re always pressured to cut corners to meet arbitrary and impossible deadlines.”

“Our systems are broken, everyone knows it, but managers continue to outsource and off-shore our QA.”

“We don’t have authority to decide the implementation, we’re often told what to implement by architects and supervisors, even if we know it to be rotten.”

“Our managers never ask us about quality…they ask us only ‘when will this be done’?”

And that’s the crux of the problem: people are hired by companies like Equifax to provide technical expertise, then their advice is ignored and their implementation decisions aren’t trusted.

Some important questions to consider…

1. Does Equifax lack the money to hire excellent technical staff?

No, their offices are filled with some of the best programmers in the world.  I meet them (or people like them) regularly in my classes and I have no doubt that the technical staff at Equifax have warned the managers for years of security holes and technical defects.  I have no doubt those managers have ignored the alarms and have pushed the staff to deliver deficient code.

2. Does Equifax lack the time to build high-quality systems?

No, last I checked they’ve been at it a long time and their existing contracts will carry their operation years into the future.  And as mentioned earlier, securing our data is the reason the company exists.  It’s  the one thing they’re supposed to get right – I’d think their time should be entirely devoted to building high-quality systems.

3. Does Equifax lack the financial resources to invest in proper tools and training for security/quality testing?

No, such techniques and tools are widely available and inexpensive (even hackers can afford them!)  Managers at Equifax have denied budgets for training, tools, and upgrades because “it’s too costly” – hmm…I wonder the cost of this data breach?

4. And my favourite question of the day: Are the hackers “smarter”?

Absolutely not.  But they’re more dedicated and have equipped themselves with good techniques and tools for penetration testing.  In my personal experience as a hacker (er, I use that term loosely) security holes are all around us if we look for them.  Equifax simply wasn’t looking!

What to do about it…

First, it’s clear to me the problem isn’t technical or financial.  It’s cultural.  I see it all the time.  Teams of good product developers are surrounded by bureaucracy instead of support.   Teams of good coders aren’t trusted to see the source code of the systems used by the company – yes, that’s as crazy as it sounds!  Teams of good coders are kept silent about the security vulnerabilities they see.  Solutions are ignored by management and the arguments are: “improving the security isn’t a priority right  now” or “we know there are some possible security concerns and we are in discussion with vendors or outside agencies to address it” or “we have a budget for security improvements scheduled for next quarter; let’s focus for now on new features instead”.  Managers are more concerned with deadlines than with quality.  Managers scrutinize the number of hours a developer works on a task, and outsource or off-shore the quality assurance and testing!  Managers conduct endless planning activities then compress the implementation into tight budgets and timelines – evidently, a lot of energy is spent getting the plan “right” but getting the software right is not a priority.  I could go on.

If you’re interested to know how things work at Equifax, just think of the Dilbert cartoons.  I mean it.  I am very serious.  Dilbert isn’t funny because it’s fiction; it’s funny because it’s NON-fiction.  Sadly. Typically, for enterprises like Equifax, their technical staff and customers take a back seat to management “theatre”.  This needs to be fixed and it starts by asking the technical staff a single, simple question:  “Who among you have raised concerns about technical debt with your managers/supervisors and were ignored?”  That question will unearth bugs which have been deprioritized by managers, budgets that have been denied for technical training and automated testing, projects which have been reported as “done” before they were actually ready for deployment – in other words, that question will reveal the many (fixable) ways managers get in the way of quality.

Second, executive staff at Equifax need a crash course in automated testing.  Yes, THE EXECUTIVE STAFF!  It’s is essential they understand and see with their own eyes that:

  1. Automated testing is cheaper and exponentially more effective than manual testing;
  2. ALL defects are discoverable and fixable before hitting production environments;
  3. Quality is not something one outsources
  4. and the techniques to achieve ZERO DEFECTS are well-known, teachable, repeatable, and proven.   I’m of course referring to techniques like Test-Driven Development, Continuous Integration, Refactoring, and Swarming.  For example, these technical topics form the bulk of our Certified Scrum Developer classes. (Shameless plug.)

And third, technical staff need to stop behaving like sheep.  So far in this article, I’ve been very critical of managers, sure, and anyone who knows me personally knows I have no time for inept management.  But too often I meet smart, well-meaning developers who deliver shoddy code – perhaps at under pressure and against their better judgement, but in the end whose code is it?  Developers! I understand you might feel trapped in a pattern of quantity-over-quality and you are frequently coerced by your management to cut corners.  I get it… I understand it… it’s easy to feel that deadlines are some sort of immutable truth and that managers wield all the power.  But the fact is, developers, YOU hold all the responsibility and therefore you need to be the professional.  You need to say “no” and demand the latitude you require to deliver high quality.  You’re the one closest to the code and therefore directly responsible for the safety and well-being of your users.

So, Equifax and enterprises everywhere, I’m speaking now as your user or stakeholder or customer…

Equifax has failed. Miserably. The company deserves all the class-action suits coming there way. From leaders to developers. Everyone.

Most members of society are unwilling participants in all this.  Most of us aren’t your direct customers.  Example: I’m not a direct customer of Equifax – nobody has chosen Equifax as their personal agent.  Instead, our banks and our governments have selected Equifax on our behalf.  This presents a problem: if I were a direct customer of Equifax I’d call them today and close my accounts; but I can’t do that.  Instead, the best I can do as an individual is contact my banks, lenders, and insurance agents to demand change.  (Yes, I likely will do that.  I’m that sort.)

However, the larger issue is that we are at the mercy of YOU who produce software.  I’m talking about the software in our vehicles, in our heart-monitors, in our subway systems, in our air-traffic-control centres, in our banks – this is serious stuff!  We must be able to trust those systems…with our lives, with our security.  We must be able to trust you even though we don’t and won’t ever know you.

A hacker friend of mine once said, “if self-driving cars are being produced without complete automated test coverage, then that’s a future I don’t want.”

In this day and age, low quality is intolerable.


Affiliated Promotions:

Try our automated online Scrum coach: Scrum Insight - free scores and basic advice, upgrade to get in-depth insight for your team. It takes between 8 and 11 minutes for each team member to fill in the survey, and your results are available immediately. Try it in your next retrospective.

Please share!
Facebooktwittergoogle_plusredditpinterestlinkedinmailFacebooktwittergoogle_plusredditpinterestlinkedinmail

Link: The Cost of Turnover on an Agile Team

Great article by Mike Griffiths: http://leadinganswers.typepad.com/leading_answers/2015/10/agile-talent-management.html


Affiliated Promotions:

Try our automated online Scrum coach: Scrum Insight - free scores and basic advice, upgrade to get in-depth insight for your team. It takes between 8 and 11 minutes for each team member to fill in the survey, and your results are available immediately. Try it in your next retrospective.

Please share!
Facebooktwittergoogle_plusredditpinterestlinkedinmailFacebooktwittergoogle_plusredditpinterestlinkedinmail

Report average velocity and fail 50% of the time

The question of “expected velocity” and long-term planning has come up at more than one client. A recent client conversation got me thinking, however, questioning how to interpret velocity when estimating and plotting a roadmap based on a current backlog of features. Assume, for a moment, a backlog of story-pointed features, and 10 good iterations (consistent team, no odd occurrences that would affect velocity). Mathematically average velocity (well, a mean really) is a 50/50 proposition for any subsequent iteration. Some organizations don’t find this level of confidence acceptable. What velocity should be reported as expected for iteration/sprint planning and roadmap forecasting, and how should it be used?

Context

Interpreting velocity, before anything else, requires some context. An agile organization that sees estimates as hypothetical might find this article is of less use. In fact, a good question is whether estimation is even a value-added activity. For this post assume an organization that sees strong value in estimation and planning.

Culture

The biggest piece of context is to know the organizational culture. This is important in two respects, and both of these cultural factors are important because they impact how Velocity is understood within the organization.

What is Failure?

First is the meaning of failure in the organization. Is failure to deliver what was committed to by the planned date considered a failure of the team, or is it simply a fact to be understood and accounted for in future planning? Even in Agile organizations, the former is often true and a hard habit to break. If not delivering to expectations is considered failure and has negative consequences, then that means that estimation is being treated not as estimation, but as prediction and contract. Velocity is therefore a commitment, and should therefore be used conservatively.

Consistency or Speed?

The second item to know is whether consistency and predictability of delivery is of a higher strategic value than the actual rate of delivery. This is often un-stated. Usually people want fast and consistent delivery. The truth is that you can get consistent, or fast software development, or a balance between the two. Lack of trust is usually a strong motivation to encourage consistency over speed, or a history of quality problems, etc. In this case, as well, Velocity is more of a boundary than an indicator.

Emotional Loading in Estimation (or why not Low-ball?)

If estimation is seen as binding, contractual, or limiting, then additional emotions get overloaded. Trust, promise, and betrayal are words used in such organizational cultures. Distrust is usually a strong factor, especially between silos (business vs. technology, company vs. project management vs. customer, etc.). So when people are asked to give estimates, even using agile-friendly mechanisms such as story points, there is usually a process of cementing that estimate into a part of an accountability model, so estimates start to get conservative. People are then accused of low-balling, others are accused of irrational expectations… we’ve all seen this. The language clearly becomes one of contention and blame. Even the term low-balling is often an outright pejorative term for estimating too conservatively.

This doesn’t happen only in agile environments, and project managers in traditional PMBOK frameworks have long factored risk into “contingency budgets”. Interestingly, however, if a Project Manager were to factor risk into the task estimates, they’d be “low-balling capacity,” yet if they were to factor it out and layer it on top of the project work, it’s “contingency budgeting” (At least in a few experiences I’ve had). Either way, someone’s adding a factor for uncertainty, based on the need to predict conservatively or liberally or somewhere in between.

That’s the point of the article: how can Agile projects use velocity to estimate as conservatively (or liberally) as is appropriate?

An average is a 50% chance to succeed (or fail)

Velocity is not a constant. It’s a set of instantaneous values on a curve, with instances being iterations. That means that it varies, and is therefore only meaningful statistically. So how do you reasonably use velocity statistically, and improve confidence? One way is to stop delivering against “average” velocity.

A lot of coaches use average velocity over the previous N iterations. This is not helpful for all sorts of reasons, if estimation is a commitment. By definition, average (well, actually a mean, but they’re close) is a 50/50 proposition. If you report the average team velocity (assuming it’s accurate), then about half the time the team will be under and about half the time the team will be over, statistically. So basically an average is a crap shoot, when taken in any given instance. It’s can only be good in the long run. For this to work, the long-haul has to include permission to fail and a lot of trust. Teams need to be able to go miss dates but will sometimes exceed dates and it should all wash out in the end. In organizations such as I’m describing, that trust isn’t there, so. Additionally, if the language of commitment is around meeting instantaneous iteration commitments (as opposed to delivering high-quality customer value as quickly as is sustain-ably possible) then you aren’t playing the long-game, you’re playing a very short-game.

Simulate Velocity, not work

In a PMI training course I took when I was at Sun Microsystems, we were nicely informed that two point estimates of tasks are a perfect way to fail half the time, per the above logic. One point estimates are just idiotic. Three point estimates were better. We simulated with a monte-carlo algorithm and found a curve and a distribution, and then determined a confidence level yadda yadda. Well, we’re trying to avoid wasting a lot of time estimating up-front, but one way to start representing velocity properly is to do the same kind of statistical modelling done in traditional product management, only simulate velocity, not work items.

In this approach, you take the last N iterations (say 10). Determine the maximum velocity (optimistic) and the minimum velocity (pessimistic), and then the mode (the velocity value that seems to occur most frequently). Then you do monte-carlo simulation so you get a statistical pattern. Now, you actually can determine an answer based on confidence. If you want to be right with an 80% confidence, you pick a velocity where 80% of the simulated runs were successful. (Note – there are a paucity of excel templates to do this math automatically, and often they are for sale. It would be nice to have a few functions with arbitrary distributions based on min-max-mode to help this along.)

It’s not perfect, and it’s a potentially huge amount of administrative overhead. Elsewhere I’ve referenced blogs that entirely oppose any estimation at all, but if you are gong to, then working statistically with simulation is the only way to take small sample numbers meaningful.

Commitment Velocity: Low-Ball as a policy.

Another approach, one perhaps controversial, but taught by some Scrum trainers is to pick the lowest historical delivered velocity. This is a commitment-based approach, on the assumption that building trust around consistent delivery is critical to building sound relationships where product owners and teams can safely state their needs and get things done with a minimum of contractual behaviour. By taking the minimum, you force a low-ball capacity, which means you can have high-confidence of success after a few iterations. You have, likely, after a while, some spare time on your hands. Teams can then choose to pull more work in (without adjusting their commitment velocity), work on “technical debt”, improve their skills, etc. A team could raise their commitment velocity in certain inflection points in the project. A new team member is added that provides a necessary skill not previously available, and after a few iterations the team is consistently hitting a higher number, but this is a careful process to ensure that they are committing, and if they don’t make their new number, it goes down to what they got accomplished.

Indemnify teams’ learning

An arguably healthier option, if you have built enough trust, is to simply indemnify a team from failing to meet the estimate. Since you’re doing mathematics on actuals to generate an expected future number, everyone can acknowledge that past behaviour is no guarantee of future behaviour, and simply use it for capacity planning. In this case, estimation is actually estimation, not commitment or contract. The team is expected to be ahead sometimes, and behind sometimes. The upside of this is that a lot of extra time isn’t spent playing with fictional numbers. Teams are spending their efforts on delivery as quickly-yet-sustain-ably as they can, and the organization treats them as trusted professionals in this. The temptation to assume you can predict the future is seen as folly, and the estimates are used to guide overall direction, not to make outward customer commitments.

Don’t be mindless

There may be other approaches, I’m sure. The agile community is certainly not short of people who love this topic and can talk for hours on “proper” estimation. The point of this post is merely to point out some options, and ask you to look at your organizational culture, team culture, customer culture, the meaning of terms like commitment, failure, success, consistency, speed, etc. As you understand the culture, balance consistency vs. speed, trust, and other factors to choose a method of estimation that meets your goals. Don’t do estimation based on your own, internal cultural assumptions, as you may have developed or been taught techniques that are useful when and where they were taught, but may no longer be so. Or maybe they weren’t so useful then either. Regardless, this because estimation cuts at the heart of the dialogue between producer and consumer, and establishes parameters for that discussion, it’s critical that you think your choice through.

[Christian also blogs at http://www.geekinasuit.com/]


Affiliated Promotions:

Try our automated online Scrum coach: Scrum Insight - free scores and basic advice, upgrade to get in-depth insight for your team. It takes between 8 and 11 minutes for each team member to fill in the survey, and your results are available immediately. Try it in your next retrospective.

Please share!
Facebooktwittergoogle_plusredditpinterestlinkedinmailFacebooktwittergoogle_plusredditpinterestlinkedinmail

Article on Risk and the Brain

This article called “To Bet Or Not To Bet: How The Brain Learns To Estimate Risk” is interesting, particularly because agile methods treat risk completely differently than traditional methods.


Affiliated Promotions:

Try our automated online Scrum coach: Scrum Insight - free scores and basic advice, upgrade to get in-depth insight for your team. It takes between 8 and 11 minutes for each team member to fill in the survey, and your results are available immediately. Try it in your next retrospective.

Please share!
Facebooktwittergoogle_plusredditpinterestlinkedinmailFacebooktwittergoogle_plusredditpinterestlinkedinmail