Running big retrospectives at Spotify

Running big projects often hurts. Big projects mean more dependencies, more coordination, more risk, more uncertainty, and so on and so forth. It’s safe to say that you should avoid big projects if you can. But sometimes you can’t. You could say that Spotify is organized to avoid big projects, but in order to build great stuff fast sometimes a lot of people need to be involved and then we suddenly have a big project on our hands. As with everything else we constantly ask ourselves how we can become better at running big projects. One part of that is to run a project retrospective at the end of the project. In this blog post I am going to describe my experience from participating in the latest of these retrospectives. Hopefully it will help someone with inspiration for running a similar event.

[box] How big is “big”? Big projects at Spotify are still pretty small, maybe 25-50 engineers for 3-6 months depending on how you count. More on that in another blog post. You can also read more about our product development process here.[/box]

Background

December 2012 we launched a new web site and introduced a new way of discovering music.

Both these things, as well as a couple of others, were part of the project we now met to retrospect about. All in all it involved about six squads in three different locations (Stockholm, New York, Gothenburg) directly and a number of other squads and departments, such as Operations and PR, indirectly. My colleagues, agile coaches Henrik Kniberg, Martin Wasielewski & Daisy Pilbrow, and Petter Weiderholm from People Operations, all did an ambitious job preparing the retrospective; interviewing key people, collecting data, handling invitations, planning and preparations, organizing a dinner, etc. I have never before been at a big retrospective that felt this well prepared.

A total of about 65 people turned up at the retrospective; several people flew in from New York and engineers, ops people, agile coaches, management (e.g., CTO and VP Product) and product owners all had good representation. The retrospective was held at an off site, lasted a whole day and ended with a really nice dinner.

Opening the retrospective

When we arrived we were met by a greeter who handed out printed agendas and told us to get some breakfast and write our own name tags.

After a while we were ushered in to a big room with two rows of chairs in circles. One of the facilitators placed himself in the centre and opened the retrospective. He started with a short welcome and then he had us empty our minds by writing all the things preoccupying us on an index card and tucking it in a pocket for tomorrow: “today we need you to be 100% present”.

To raise the energy level and to celebrate that we made it through the project, Henrik the facilitator asked us to join together in a big, loud, uninhibited cheer, like if our favorite band was on stage. We did and we probably scared the crap out of the venue staff. Henrik even had to scrap his original plan on requesting an even louder cheer

After a quick round of brief presentations – name, role in project, two word summarizing how we feel being here today (the number one word turned out to be curious, followed by tired) – the agenda was presented:

The Agenda

Part I – The past
Visualize the project
Tell the story
How did the project feel
How successful was the project

Part II – The present
Generate discussion topics
Breakout discussions
Report back

Part III – The future
Output cleanup
Prioritize
Present the message

The last part of the opening was to read the suggested working agreement, item by item, and get agreement through a simple thumb vote.

Part I – The Past

Our Chief Product Officer started by giving his view on the expected project outcomes, everything from increased collaboration between different teams to actual deliverables, and project costs like frustrations, tech debt and alternative costs. He posted them on flip chart sheets and anyone could add their own ones during the day.

You could say that the project actually was four different smaller projects and during the next part of the retrospective we divided into four breakout groups based on these.

Visualize the project

We had all been given a home work assignment before the retrospective to gather artifacts related to the project. Things like important e-mails, early design print-outs and screen dumps of prototypes.We used these and our insights, memories of events, feelings, challenges, and so on, written on stickies to build a timeline on sheets prepared on the wall. After about half an hour we reconvened in the big room and put all the timelines on one wall, running parallel to each other.

Story telling

During the breakout sessions we had appointed storytellers for different periods and they now took turns summarizing and explaining the timelines in front of the whole group.

How did the project feel?

“How did you feel about this project?” These were the questions we were asked to answer through a drawing on a sticky, for example a smiley or a frowney of some kind. We put the stickies on our chests and mingled with the others, discussing our drawings, preferably with someone with a picture that was different from our own. I noticed a couple of drawings of stick figure heads with drops around them, especially from the New York colleagues – maybe they all got wet from Sandy? After the mingle we put the stickies on a matrix indicating our role (dev/qa/manager/product/other) and our project within the project.

How successful was the project?

Next we were asked how successful we felt the project was on a scale from 1 to 10. We put the number on a sticky and next to it wrote one positive thing that pushed the grade up and one negative thing that pushed the grade down. To get a feeling for how the grades were distributed we all lined up in the room with one side of it representing low and the other high. Our facilitators interviewed one of the low, one of the average and one of the high ones, to learn why they voted like they did. We wrapped up by putting the stickies oin order on a wall where we could later browse through them and see what others had written.

Before going to lunch we were encouraged to think about topics we wanted to discuss later and write them down on index cards. We were also handed a lot of “kudo cards” with graphics and titles such as “Awesome”, “Thank you” and “Congratulations” that we could fill out during lunch and hand out to people we wanted to show our appreciation to.

Part II – The Present

Generate discussion topics

The first part of the afternoon was dedicated to the present. We gathered in front of a schedule matrix with time slots and rooms to briefly present the topics prepared during lunch, merge similar ones and find a good slot on the schedule.

Breakout discussions

The goal for each session was to generate insights and recommendations on what to keep doing and what to do differently in the future. The topic originators were encouraged to make sure these things were captured during the sessions. If we bumped into something important that couldn’t be solved here and now, we noted that as a mystery. In an Open Space inspired style we chose what we wanted to attend and were reminded of the law of the two feet: if you feel you’re not learning or contributing, move on. Among the topics were classic difficulties such as how to better overcome the challenges of coordination multiple squads in different time zones and locations, how to involve [insert role/department here] earlier/more frequently and how to improve branching/release management.

Report back

Two 45 minutes sessions later, the insights, recommendations and mysteries that had been captured were presented briefly, 2-3 minutes per session, in front of the whole group.

Part III – The Future

Output cleanup

During the break that followed, the topic presenters gathered and cleaned the output: removed duplicates, made sure notes had clear handwriting and full understandable sentences, and put each note on its own bigger paper to allow for space for dot voting and signing up. All insights were posted in one room, all recommendations in another, and the mysteries in a third room.

Prioritize

After the break we paired up with someone we didn’t know that well, grabbed two sheets of green and black dot stickers and walked through the rooms discussing the notes. If we believed a note to be very important and strongly agreed with the statement, we attached a green dot sticker to it, if we disagreed strongly we attached a black sticker. For the mysteries we could also sign our names for the ones we wanted to solve. Using the dot votes we chose the top five or so for each room and posted them to a flip chart paper.

Present the key messages

Everyone gathered in the big room again where a human billboard presented each flip chart while someone read the notes out loud from him or her. We also had a short discussion about the few notes that had several black dots and several green dots to better understand why people had such different views on these.

Closing

To finally close the retrospective we spent a couple of minutes writing down two things on a card, and then shared it with our neighbor: What is your main take-away from today? What will you do to help these messages stick? One of my main takeaways was the deep consensus on key issues like using real data as early as possible to drive decisions, perform user testing as early as possible, and involving as many different people/roles as possible in coming up with a solution (“how do we cross the water” vs “build a bridge”). A few people shared what they wrote with the entire group, we declared the retrospective done and applauded ourselves for the effort. On our way out we were told to add a mark on a feedback chart: from 1 to 5, how valuable was this day to you in return of time invested? Most of us stayed for a really nice dinner at restaurant Jonas where the discussions and the getting to know each other better continued into the night.

Future improvements

As I’ve already mentioned I was impressed with how well prepared and well executed this retrospective was. There is of course always room for improvement though. These are some of the things I would consider doing differently next time:

• We spent less than 30 minutes on creating the timelines and none, or only a very small amount, of that time was used to prepare the story telling in front of the whole group. The result was a somewhat disorganized and lengthy story telling at times. Next time we should set aside time for preparing, possibly also coaching the story tellers on what is expected and how much time is allotted for this.
• Allow for more space on the timeline to put up the artifacts people had brought.
• During the mingling with post-its during “how did the project feel” I felt that people weren’t really into it. Maybe a bit more structure is needed, such as prepared questions to ask the others or small groups that are changed after half the time.
• When people lined up from 1 to 10 in the “how successful was this project” exercise it was difficult to see the scale and, because of the big group around the average, unclear if people actually were in the right place or just standing where they could find a space. We should clear out the space in front of the wall, have numbers on the wall and have people stand in a long line from that wall. Kind of like a human histogram.
• I know time is a scarce commodity at these kind of events, but I still would have wanted more time for generating topics for the breakout sessions. Everyone should be participating rather than having people come up with topics during lunch and then only having the ones that did convene to build the schedule.
• The output cleanup and choosing the key messages could also benefit from some hands-on coaching. Now at least one of them was at the same time as a coffee break and the people who were “voluntold” to do it of course also wanted to take a break and get some coffee. Have someone bring the coffee and sweets to the cleanup rooms and have a facilitator in each room leading the work maybe?
• While I think really encouraging the law of the two feet to the point that you suggest people do whatever they want during the sessions is a really great idea in a real Open Space, I found that in this context it gave even more permission to “busy people” to choose “real work” like making phone calls, answering e-mails or talking to others about the business as usual. Given the difficulty to get all of these people together for a focused event like this I think it is better to demand focus on the task at hand.
• I was personally involved in the planning before the retrospective, gave feedback on the agenda, ran a pre-retrospective with one of my squads, and so on. I have also run retrospectives like this several times. I knew what to expect. Others did not and they were not sure how to prepare. I think we should share more about the agenda and the expectations before the next big retrospective.

All the good photographs in this blog post were taken by colleague Fredric Sannebro, the others by me.

If there is something you want to know more about, wether it’s about this retrospective or anything else, please add a comment to this post. The same goes if you have feedback on how to improve this post.