I wanted to understand employee’s mental model of a certain process (guided by an application) at a blue-chip retail company, and besides a qualitative interview, I ran a sequential card sorting activity. I have been searching high and low for guidance on how to analyze data. Surprisingly I did not find anything relevant to my situation, so here is what I ended up doing.
Preparing the Cards
After breaking down the task flow into small steps, verified by former retail employees at the company, I created a deck of cards with the individual tasks typed on each card. As the actual work is proprietary information, I am using a made-up scenario for a public library just because most people have an understanding of library processes.
I agonized a lot on the specifics of the cards. I ended up just printing on regular paper and cut out the cards. I considered other options, which then I discarded:
often used in card sorting but (1) they cannot be printed on without a lot of hassle; (2) they need to be moved around very easily. Sticky notes are sticky by definition, which does not work well when creating a workflow. Test subjects discover a new task they want to add, say, early in the task flow – they would have to peel off and re-position lots of notes. This is cumbersome and may discourage people to change their task flow, leading to invalid data.
cheap and use to use, but (1) they cannot be printed on; (2) their size makes them impractical in settings where there is no big table. In the original study, I used 24 cards, and the offices behind the retail spaces were very small, with narrow desks.
Collecting the Data
Here is the file for the library_cards for each task one can do when using the digital library system. Users are invited to sort the cards in the order that makes the most sense to them. They can add additional cards to the sequence if they think some steps are missing, and/or toss the ones that they don’t consider important. You can gain more insight into your users’ thinking by encouraging them to think aloud when they are arranging the cards. I asked the participants’ permission to record their card sorting and their voices. I made sure I was not recording their faces, just their hands on the desk as they moved the cards around.
Once participants are done with their card sorting, take pictures of the final arrangements. This will make it easy to transfer the sequence into a spreadsheet.
Analyzing the Data
The first step is to enter all the subtasks in order into a spreadsheet, something like this, in the order respondent put them. To develop insights into the mental models of your user, you can color code different aspects of the tasks (say, some focused on finding the item, some on checking the patron’s records, etc.) and look for patterns. The color coding helps when you have a lot of subtasks – you can see the patterns without reading the individual tasks. In my experience, the more steps a task consists of, the larger your sample needs to be so that you can detect tendencies.
By using the spreadsheet, each task has a specific value assigned to them, for each row. The earlier the subtask was placed, the higher the value. In the example below (made-up, and probably not realistic), “identifying the title” would have the following scores: 24, 24, 21, 21, 24, 21.
Then create a card for each subtask, with the following information:
Once you have a card for each task, you can arrange them starting with the largest sum of scores all the way to the lowest. If the spread of the individual scores is relatively large, then you can consider that task to be placed either early on, or late in the process.