This afternoon I took the Beta version of the knowledge exam for the Certified ScrumMaster credential. I’m not allowed to provide any details on the questions, but I will provide my impressions.

I’ll start with a story.

Microsoft Certified Application Developer

About six years ago, near the end of my career as a technical contributor, the company I was working for, Solution Architects (who still have my profile on their “people” page), decided that I should become a Microsoft Certified Application Developer. At the time, I was doing .NET development and I had a long background in Java and Objective-C development. The approach we decided on was for me to go to a “Boot Camp” where I would be immersed in all things .NET and after nine days of solid training, write the Microsoft exam.

I arrived at the Boot Camp (which was very much a outdoorsy camp environment) quite excited. I got a room for myself, and it looked like I would be treated very well. One the first day of classes, the instructor gave us some strong advice: come to class, and then in all your spare time, do the practice exams and study them hard. I was a bit baffled by this. We were also given the huge Microsoft Press books to study for the exam (I kept them for a few years, but recently got rid of them). My first night I studied the books and my notes from class. To be frank, the instructor spent most of the time going over exactly what was in the books and giving us all a little time on computers to do the “exercises” in the books. Instruction was really limited to rote recital of the book content. Any time someone would ask a question that was in any way deep, the instructor would simply redirect with another reminder to study the practice exams.

The second night I decided I would try the practice exam since the first of three real exams was in the afternoon of the third day. It was fairly simple multiple choice test questions. I went through all the questions, made sure I found the answers for ones I didn’t know in the books or in my notes, and then after I had done a once-through, I did a quick second pass.

And then, the next day, I took the real exam. I was utterly, completely shocked. The real exam was exactly like the sample exam. The only difference was that the word problems change the names of the fictional people and companies used in the problems. The structure of the questions was identical. The answers – including the ordering of the multiple choice answers – were exactly the same. And of course, it was a breeze. Anyone could have passed. In fact, it was completely unnecessary to attend the classroom training part. I was extremely dis-illusioned.

Why do I mention this experience with a certification exam? Simple: it has made me extremely sceptical of exams. They simply cannot measure any level of competency. They simple measure people’s ability to pass exams. And since there are many fair and unfair ways to do that, exams are not relevent.

Now I will say that I have changed my mind just a wee bit about this… but that’s a topic for a completely different blog post.

So, when I heard that the Scrum Alliance was going to add an exam to the CSM certification, I felt that it was a waste of time, and probably would encourage all sorts of bad behaviors. I still think that.

The Beta CSM Exam

Okay. A few facts about the exam. It was administered in a room in the convention center here in Orlando. There was a registration desk and when you sign in you are given a password. You then go to a workstation which has Internet Explorer running pointed at the exam site. The exam has bookends: at the start an experience self-assessment that is used to help interpret the exam results, and at the end a satisfaction survey. Throughout the course of the exam, you are able to comment on the questions. These bookends and the feedback along the way are a great way to help improve the exam and I really like that.

As I mentioned, I am not allowed to discuss the details of the questions. I will make some general comments about the questions. Some questions are about Agile, some are about Scrum principles and some are about Scrum practices. Some are fairly standard fact-based kinds of question like: what are the roles in Scrum, while others are more scenario-based question like: you are the ScrumMaster and X-bad-behavior is happening… what do you do?

There were 99 questions in total and I was told that it would take approximately one hour to go through the questions. Now, just so you know, I normally do _really_ _really_ well on multiple choice exams, and I normally complete them extremely quickly. I read fast, and my mind seems to be able to eliminate incorrect options almost subconsciously. So, for this exam, I completed it in 35 minutes including the time it took me to comment on about a third of the questions. If I hadn’t been commenting as I went, I estimate it would have taken me about 20 minutes.

How Did I Score?

Well, I got 84%. Not bad. The summary page of the exam said this was a “mastery” level. I should explain why I didn’t score higher (after all, Certified Scrum Trainer (TM) Mishkin Berteig should be able to do 100%!!!).

I decided before I even started, that I would answer the questions as if I was a “perfect” student of my own training. In other words, I would deliberately get things wrong if I taught them differently than the “right way” that the question implied. As well, if I didn’t cover a topic in my training, I would do a best guess putting myself in the shoes of someone who had attended my class. There were two broad topic areas that I don’t teach about that showed up: Product Vision and Release Planning. As well, there were a few topics that I teach slightly differently: Scrum Team membership, burndown charts, and Sprint Planning/Sprint Backlog Tasks.

Apparently, despite these differences, a student of my class would do pretty well on the exam.

The Problem

When I first became a Certified Scrum Trainer (no TM, this was before the existance of the Scrum Alliance), Ken Schwaber had a clear policy that as a trainer I was encouraged to integrate into my training materials and approach things that I had discovered through actual practice about Scrum. I loved this. It meant that Scrum was not a Canonized Body of Knowledge, but rather a living framework for doing excellent work. When we put in place an exam like this, it changes the nature of Scrum. Is this good or bad? I think it has aspects of both. The clear down side is that it will have the tendency of freezing Scrum which might make it less relevent.

Another problem is more personal: as a trainer, there will be clear pressure for me to teach to the exam. If a student of mine goes and does the exam, and fails because (in part) I have taught things differently than what is on the exam, then does that mean this person can blame me? Sure! Why not?! So then I am faced with a problem: do I teach what I know works or do I teach what I know will be tested?

Solution

There is a simple way to avoid this second problem and in fact to mitigate the first problem at the same time: the exam should be taken before taking the CSM course. The exam is clearly based on the reading materials: Agile Software Development with Scrum and Agile Project Management with Scrum. Then, if people don’t pass the exam, they can blame only themselves for not studying these excellent books deeply enough. And, it will simplify training since as trainers we will know that people coming into the class are already _knowledgeable_ about Scrum. We can then teach our variations, see the dynamic of people in the class, and offer Certification based on that.

This solves the trainer’s dilema easily and obviously. What is not so obvious is that it also helps prevent Scrum from ossifying. The Certification becomes based on living interaction with an experienced Scrum trainer rather than an exam. The long term effect of this is that people will place less importance on the exam (rightly) and more importance on making a good showing in the course (rightly) and then we have a relationship-based Certification. Since it is based on a relationship, it can live more easily as an organically changing framework rather than a defined (simple) methodology.

After all: Individuals and Interations are valued over Processes and Tools (Agile Manifesto).