Tag Archives: ETL

Scrum Data Warehouse Project

May people have concerns about the possibility of using Scrum or other Agile methods on large projects that don’t directly involve software development.  Data warehousing projects are commonly brought up as examples where, just maybe, Scrum wouldn’t work.
I have worked as a coach on a couple of such projects.  Here is a brief description of how it worked (both the good and the bad) on one such project:
The project was a data warehouse migration from Oracle to Teradata.  The organization had about 30 people allocated to the project.  Before adopting Scrum, they had done a bunch of up-front analysis work.  This analysis work resulted in a dependency map among approximately 25,000 tables, views and ETL scripts.  The dependency map was stored in an MS Access DB (!).  When I arrived as the coach, there was an expectation that the work would be done according to dependencies and that the “team” would just follow that sequence.
I learned about this all in the first week as I was doing boot-camp style training on Scrum and Agile with the team and helping them to prepare for their first Sprint.
I decided to challenge the assumption about working based on dependencies.  I spoke with the Product Owner about the possible ways to order the work based on value.  We spoke about a few factors including:
  • retiring Oracle data warehouse licenses / servers,
  • retiring disk space / hardware,
  • and saving CPU time with new hardware
The Product Owner started to work on getting metrics for these three factors.  He was able to find that the data was available through some instrumentation that could be implemented quickly so we did this.  It took about a week to get initial data from the instrumentation.
In the meantime, the Scrum teams (4 of them) started their Sprints working on the basis of the dependency analysis.  I “fought” with them to address the technical challenges of allowing the Product Owner to work on the migration in order based more on value – to break the dependencies with a technical solution.  We discussed the underlying technologies for the ETL which included bash scripts, AbInitio and a few other technologies.  We also worked on problems related to deploying every Sprint including getting approval from the organization’s architectural review board on a Sprint-by-Sprint basis.  We also had the teams moved a few times until an ideal team workspace was found.
After the Product Owner found the data, we sorted (ordered) the MS Access DB by business value.  This involved a fairly simple calculation based primarily on disk space and CPU time associated with each item in the DB.  This database of 25000 items became the Product Backlog.  I started to insist to the teams that they work based on this order, but there was extreme resistance from the technical leads.  This led to a few weeks of arguing around whiteboards about the underlying data warehouse ETL technology.  Fundamentally, I wanted to the teams to treat the data warehouse tables as the PBIs and have both Oracle and Teradata running simultaneously (in production) with updates every Sprint for migrating data between the two platforms.  The Technical team kept insisting this was impossible.  I didn’t believe them.  Frankly, I rarely believe a technical team when they claim “technical dependencies” as a reason for doing things in a particular order.
Finally, after a total of 4 Sprints of 3 weeks each, we finally had a breakthrough.  In a one-on-one meeting, the most senior tech lead admitted to me that what I was arguing was actually possible, but that the technical people didn’t want to do it that way because it would require them to touch many of the ETL scripts multiple times – they wanted to avoid re-work.  I was (internally) furious due to the wasted time, but I controlled my feelings and asked if it would be okay if I brought the Product Owner into the discussion.  The tech lead allowed it and we had the conversation again with the PO present.  The tech lead admitted that breaking the dependencies was possible and explained how it could lead to the teams touching ETL scripts more than once.  The PO basically said: “awesome!  Next Sprint we’re doing tables ordered by business value.”
A couple Sprints later, the first of 5 Oracle licenses was retired, and the 2-year $20M project was a success, with nearly every Sprint going into production and with Oracle and Teradata running simultaneously until the last Oracle license was retired.  Although I don’t remember the financial details anymore, the savings were huge due to the early delivery of value.  The apprentice coach there went on to become a well-known coach at this organization and still is a huge Agile advocate 10 years later!

Affiliated Promotions:

Try our automated online Scrum coach: Scrum Insight - free scores and basic advice, upgrade to get in-depth insight for your team. It takes between 8 and 11 minutes for each team member to fill in the survey, and your results are available immediately. Try it in your next retrospective.

Please share!
Facebooktwittergoogle_plusredditpinterestlinkedinmailFacebooktwittergoogle_plusredditpinterestlinkedinmail
Berteig
Upcoming Courses
View Full Course Schedule
Introduction to REALagility™ with ICP® (ICAgile Certified Professional®) [Virtual Learning]
Online
C$1185.75
Dec 8
2021
Details
Real Agility for Managers
Online
C$149.00
Dec 10
2021
Details
ICAgile Certified Professional in Agile Coaching® (ICP-ACC)
Online
C$1995.00
Dec 13
2021
Details
Scrum Master Bootcamp with CSM® (Certified Scrum Master®) [Virtual Learning]
Online
C$1795.00
Dec 14
2021
Details
Certified Scrum Master® (CSM) [Virtual Learning]
Online
C$595.00
Dec 14
2021
Details
Kanban Systems Improvement® (KSI) [Virtual Learning]
Online
C$1795.00
Dec 14
2021
Details
BERTEIG Professional No Agenda Networking [Exclusive Event]
Online
C$0.00
Dec 16
2021
Details
Certified Scrum Product Owner® (CSPO) [Virtual Learning]
Online
C$1795.00
Dec 16
2021
Details
Kanban for Product Owners - Improve Your Project Ownership Skills
Online
C$149.00
Dec 16
2021
Details
Product Owner Bootcamp with CSPO® (Certified Scrum Product Owner®) [Virtual Learning]
Online
C$1795.00
Dec 16
2021
Details
Advanced Certified Scrum Product Owner® (A-CSPO) [Virtual Learning]
Online
C$1695.00
Dec 21
2021
Details
Real Agility for Managers
Online
C$149.00
Dec 22
2021
Details
Practicing Agile Estimation Techniques
Online
C$149.00
Jan 6
2022
Details
Real Agility for Managers
Online
C$149.00
Jan 11
2022
Details
Certified Scrum Master® (CSM) [Virtual Learning]
Online
C$595.00
Jan 12
2022
Details
Scrum Master Bootcamp with CSM® (Certified Scrum Master®) [Virtual Learning]
Online
C$1525.75
Jan 12
2022
Details
Certified Scrum Product Owner® (CSPO) [Virtual Learning]
Online
C$1525.75
Jan 18
2022
Details
Product Owner Bootcamp with CSPO® (Certified Scrum Product Owner®) [Virtual Learning]
Online
C$1525.75
Jan 18
2022
Details
BERTEIG Professional No Agenda Networking [Exclusive Event]
Online
C$0.00
Jan 20
2022
Details
Certified Scrum Master® (CSM) [Virtual Learning]
Online
C$595.00
Jan 20
2022
Details
Scrum Master Bootcamp with CSM® (Certified Scrum Master®) [Virtual Learning]
Online
C$1525.75
Jan 20
2022
Details
Real Agility™ Team Performance Coaching with BERTEIG
Online
C$750.00
Jan 24
2022
Details
Real Agility for Managers
Online
C$149.00
Jan 25
2022
Details
Certified Scrum Product Owner® (CSPO) [Virtual Learning]
Online
C$1525.75
Jan 25
2022
Details
Product Owner Bootcamp with CSPO® (Certified Scrum Product Owner®) [Virtual Learning]
Online
C$1525.75
Jan 25
2022
Details
Leadership for Managers - How to be an Effective Leader
Online
C$149.00
Jan 25
2022
Details
Kanban for Scrum Masters
Online
C$149.00
Jan 27
2022
Details
Advanced Certified Scrum Product Owner® (A-CSPO) [Virtual Learning]
Online
C$1440.75
Jan 28
2022
Details
ICAgile Certified Professional in Agile Coaching® (ICP-ACC)
Online
C$1995.00
Jan 31
2022
Details
ICAgile Certified Professional in Leading With Agility® (ICP-LEA) [Virtual]
Online
C$1865.75
Feb 1
2022
Details
Certified Scrum Master® (CSM) [Virtual Learning]
Online
C$595.00
Feb 2
2022
Details
Scrum Master Bootcamp with CSM® (Certified Scrum Master®) [Virtual Learning]
Online
C$1525.75
Feb 2
2022
Details
Real Agility for Managers
Online
C$149.00
Feb 7
2022
Details
Advanced Certified ScrumMaster® (A-CSM) [Virtual Learning]
Online
C$1440.75
Feb 8
2022
Details
Kanban for Product Owners - Improve Your Project Ownership Skills
Online
C$149.00
Feb 9
2022
Details
Certified Scrum Product Owner® (CSPO) [Virtual Learning]
Online
C$1525.75
Feb 9
2022
Details
Product Owner Bootcamp with CSPO® (Certified Scrum Product Owner®) [Virtual Learning]
Online
C$1525.75
Feb 9
2022
Details
Certified Scrum Master® (CSM) [Virtual Learning]
Online
C$595.00
Feb 14
2022
Details
Scrum Master Bootcamp with CSM® (Certified Scrum Master®) [Virtual Learning]
Online
C$1525.75
Feb 14
2022
Details
Real Agility™ Team Performance Coaching with BERTEIG
Online
C$750.00
Feb 15
2022
Details
Certified Scrum Product Owner® (CSPO) [Virtual Learning]
Online
C$1525.75
Feb 16
2022
Details
Product Owner Bootcamp with CSPO® (Certified Scrum Product Owner®) [Virtual Learning]
Online
C$1525.75
Feb 16
2022
Details
ICAgile Certified Professional in Adaptive Org Design® (ICP-ORG)
Online
C$977.55
Feb 21
2022
Details
Certified Scrum Master® (CSM) [Virtual Learning]
Online
C$595.00
Feb 23
2022
Details
Scrum Master Bootcamp with CSM® (Certified Scrum Master®) [Virtual Learning]
Online
C$1525.75
Feb 23
2022
Details
Real Agility for Managers
Online
C$149.00
Feb 25
2022
Details
Certified Scrum Product Owner® (CSPO) [Virtual Learning]
Online
C$1795.00
Mar 9
2022
Details
Product Owner Bootcamp with CSPO® (Certified Scrum Product Owner®) [Virtual Learning]
Online
C$1525.75
Mar 9
2022
Details
Kanban for Scrum Masters
Online
C$149.00
Mar 10
2022
Details
Certified Scrum Master® (CSM) [Virtual Learning]
Online
C$595.00
Mar 16
2022
Details
Scrum Master Bootcamp with CSM® (Certified Scrum Master®) [Virtual Learning]
Online
C$1525.75
Mar 16
2022
Details
Real Agility™ Team Performance Coaching with BERTEIG
Online
C$750.00
Mar 16
2022
Details
Certified Scrum Product Owner® (CSPO) [Virtual Learning]
Online
C$1795.00
Mar 22
2022
Details
Product Owner Bootcamp with CSPO® (Certified Scrum Product Owner®) [Virtual Learning]
Online
C$1525.75
Mar 22
2022
Details
*NEW* Certified Scrum Professional - ScrumMaster® (CSP-SM)
Online
C$1695.75
Mar 24
2022
Details
Scrum Master Bootcamp with CSM® (Certified Scrum Master®) [Virtual Learning]
Online
C$1525.75
Mar 24
2022
Details
Certified Scrum Master® (CSM) [Virtual Learning]
Online
C$595.00
Mar 24
2022
Details
Advanced Certified Scrum Product Owner® (A-CSPO) [Virtual Learning]
Online
C$1440.75
Mar 29
2022
Details
Certified Scrum Master® (CSM) [Virtual Learning]
Online
C$595.00
Apr 6
2022
Details
Scrum Master Bootcamp with CSM® (Certified Scrum Master®) [Virtual Learning]
Online
C$1525.75
Apr 6
2022
Details
Certified Scrum Product Owner® (CSPO) [Virtual Learning]
Online
C$1795.00
Apr 12
2022
Details
Product Owner Bootcamp with CSPO® (Certified Scrum Product Owner®) [Virtual Learning]
Online
C$1525.75
Apr 12
2022
Details
Certified Scrum Master® (CSM) [Virtual Learning]
Online
C$595.00
Apr 19
2022
Details
Scrum Master Bootcamp with CSM® (Certified Scrum Master®) [Virtual Learning]
Online
C$1525.75
Apr 19
2022
Details
Real Agility™ Team Performance Coaching with BERTEIG
Online
C$750.00
Apr 25
2022
Details
Advanced Certified Scrum Product Owner® (A-CSPO) [Virtual Learning]
Online
C$1440.75
Apr 26
2022
Details
Certified Scrum Product Owner® (CSPO) [Virtual Learning]
Online
C$1795.00
Apr 27
2022
Details
Product Owner Bootcamp with CSPO® (Certified Scrum Product Owner®) [Virtual Learning]
Online
C$1525.75
Apr 27
2022
Details
Kanban for Scrum Masters
Online
C$149.00
Apr 28
2022
Details
Advanced Certified ScrumMaster® (A-CSM) [Virtual Learning]
Online
C$1440.75
May 5
2022
Details
ICAgile Certified Professional in Agile Coaching® (ICP-ACC)
Online
C$1995.00
May 16
2022
Details
Real Agility™ Team Performance Coaching with BERTEIG
Online
C$750.00
May 30
2022
Details
Advanced Certified Scrum Product Owner® (A-CSPO) [Virtual Learning]
Online
C$1440.75
Jun 21
2022
Details