“8 Tips for Measuring the Impact of Serious Games”

Serious games are great! When you play them, the good ones (!!), you get the feeling that they are a breakthrough in learning. They “feel” like they are doing much more than traditional teaching and training approaches have done in the past. You think, “Everyone should be learning through games and they will replace textbooks in the classroom and brochures in the doctor’s office!”

But that is not enough.

Games are expensive to make, they come in all different levels of quality, and sometimes when you hear that a martian on an imaginary planet is teaching kids how to become successful entrepreneurs, you have to wonder if they really “work.”  So we are still at a point in time where we have to demonstrate what we do and show investors that they are getting a return on their investment.

But because serious games seem to have an impact beyond the outcomes we have traditionally measured in training and teaching, this often means that we have to be smart, thoughtful and innovative to do a good job of measuring what they can do.

Based on my experience having successfully used a range of traditional and non-traditional measures in assessing behavior change, I’d like to share some of my thoughts and advice on finding and developing measures to assess what your serious game does.

Get your brain power-up because here are some tips about how to go about measuring what serious games can do. They go from the mundane to the interesting during different stages of game development.

Before game design…..

1. BEFORE DESIGNING YOUR GAME, COME UP WITH A SET OF EXISTING MEASURES THAT COULD TAP INTO THE INSTRUCTIONAL GOALS OF YOUR GAME.  This is mundane but essential. Look at your game design, instructional design documents and even your grant proposals that you are working on  before you actually get into developing your game. These documents should contain practical and theoretical goals and objectives for the impact you wanted your serious game to have on the outcomes of the players. Go to the literature and find studies that have also tried to measure these outcomes and make a list of what they have used successfully. “Successfully” means that the measure has been able to tap into a change in behavior or cognitions. If you find a measure that comes up in every study but doesn’t move across time or with an intervention, then it might not be sensitive to changes. Don’t take the risk that this happens in your outcomes evaluation as well. Ideally, you should find measures that have tapped into changes that have come about with interventions or games that are similar to yours.

This step seems obvious because it is just about finding available measures to assess your outcomes of interest. I list this  step explicitly because most serious game projects don’t even take this step before they make their game. They think of it AFTER they make the game. This is not wise because serious games should be cleverly and skillfully designed from the beginning to impact outcomes of interest.

2. FIND STANDARD MEASURES.  This is basically an extension of the first tip. Look for any existing reliable and standard measures that can further tap into these potentially interesting “things” you found in the process of making the game that might change through playing the game yourself or watching members of the target audience. Also keep up with the literature during the process of developing your game to see if any new measures come out or any new studies have used some measures you haven’t heard of before. Stay focused on those “things” that are important to demonstrate an impact. Do you see an increase in self-esteem? There are standard measures out there to assess that.  Just make sure that you find the self-esteem measure that maps most closely to the type of self-esteem you are trying to improve with your game.

3. DEVELOP YOUR OWN MEASURES. But maybe you are making a game to empower  patients who are older to be proactive in avoiding medical errors in their interactions with the health system. One thing you might want to see is if their knowledge of what actually constitutes a medical error (e.g., misdiagnosis, botched blood draw, inaccurate dosing of a drug with no resulting harm, etc.) changes during the course of the intervention. You will likely find that a standard measure of patient knowledge of medical errors does not exist. You may need to create your own measure.

People look at me like I’m crazy when I say this because they know that standard, published measures are highly credible. Their psychometric properties are usually well researched (e.g., test-retest reliability, correlations with other measures, internal validity, etc.). Also, because they are standard, scores on them from one study can be compared to scores on them from another study with a different population to get a general idea of how they compare (example, the SAT in the US). They also know that you can’t just flat-out come up with your own measure and expect people to believe or journals to accept your research with your made up measures because you haven’t evaluated the reliability or validity of the measures. However, in some circumstances, this is EXACTLY what you need to do. This is really at the heart of demonstrating the powerful and important things that serious games can do. In sum, it’s ok to develop your own measure. You can be the first.

4. SEE A PSYCHOLOGIST. If you ever hear someone say, “That can’t be measured,” it is a sign that they need to go see a psychologist. Not because they are nuts (although they might be that as well). But because psychologists go ballistic geek over measuring intangible things like stress, self-esteem, motivation, happiness, etc. I was once in a meeting with people doing an intervention to increase “citizenship.” When asked  if they were going to evaluate their intervention, they said that they weren’t because the things they were impacting “couldn’t be measured.” I breathed an inaudible secret sigh to myself. They needed to come see me, a psychologist, because right off the bat I would have suggested that they assess whether or not people in their intervention register to vote in higher numbers, actually vote, have greater knowledge of the political candidates, volunteer for more activities in the community and donate more often to political causes just as start! If they hired me to help them measure citizenship, I would also comb the literature for existing measures and find one with the best likelihood of tapping into the type of citizenship they were trying to improve in their intervention in addition to having them asses objective behaviors. So please, give a psychologist a call when you want to measure something that “can’t be measured.”

You will want to find psychologists that have experience doing this. I would recommend finding someone with some kind of background in social psychology rather than a clinical psychologist (including neuropsychologists) who tend to rely heavily on standard measures in their work.  Social psychologists do freaky things like measuring “altruism” by looking at whether or not students at a theological seminary will stop to help a victim slumped in a doorway if they are in a hurry or had to give a talk to an audience on the “Good Samaritan.” Pretty clever, eh? (turns out it didn’t matter how religious they were or whether or not they were giving a talk on the “Good Samaritan.” They only stopped to help if they weren’t in a hurry. See Darley, J. M., and Batson, C.D., “From Jerusalem to Jericho”: A study of Situational and Dispositional Variables in Helping Behavior”. JPSP.)

5. FOCUS ON OBJECTIVE MEASURES!!! This is the most important tip I can give you. First of all, if you can show that your serious game had an impact on a behavior rather than just something the player SAID it did, you have some pretty damn convincing evidence that your intervention worked. For example, in the Re-Mission study, all of the young patients with cancer in our study reported high levels of adherence to their medication at every assessment time point regardless of whether or not they played the game designed to improve their adherence. However, computerized chips on their bottles of antibiotics and regular blood draws sent to labs to measure the amount of chemotherapy in their blood told a different story. These objective measures showed that patients who played Re-Mission were more adherent than patients in the control group. The Re-Mission players took more of their antibiotics as prescribed as indicated by computer chip records of when and how often they opened their pill bottles. The lab reports of blood levels of oral chemotherapy  showed that Re-Mission players maintained higher levels of chemotherapy metabolites in their blood than patients who played a commercial game. Nobody ever argued that we should have relied on patient self-report of adherence over these measures. In fact, if we didn’t have these objective measures in our study, we would have not only concluded that Re-Mission didn’t work, but that it was an unnecessary intervention because the patients already had high levels of adherence. Let this be a lesson as you plan your outcome evaluations!

During game development in a rapid prototyping process…..

6.  PLAY THE SERIOUS GAME! This sounds obvious to many but in reality, I have worked on many projects where content experts and even people working on the game aren’t even sitting down to play the game. So do it. When you play the game, notice any ways you might think about things differently or new insights you might be having into problems or situation. You might even be feeling a bit different in terms of your emotional state. Then after you play the game, notice anything you might be doing, thinking or feeling differently that you think is related to playing the game.

This is a start for finding or even creating measures to tap into the powerful things your serious game might be doing. This is also a start at going beyond traditional ways of measuring outcomes.

And by the way, if you are a researcher in charge of evaluating the game, you really shouldn’t skip this part. Because no matter what you planned to do with your game, there will be things your game will do better than others and not everything you planned to have in the game will make it in there. Accept it, adjust your expectations and revise your measures accordingly.

Also during game development….

7. CLOSELY OBSERVE AND INVESTIGATE WHAT MEMBERS OF THE TARGET AUDIENCE ARE DOING, FEELING AND GETTING FROM THE GAME PROTOTYPES.

If you read my top 10 tips for making a successful serious game, you saw this one coming from a mile away! A rapid prototyping process is helpful to use in making serious games in part so that key members of your target audience can give feedback. I also like to include members of the target audience on the QA team so I can also get their reactions to the game beyond clearing up any technical problems. They will give you clues into any potentially powerful things your game is doing to and for them if you watch and listen carefully. This might also give you some ideas for how you could do an objective assessment of behaviors that change from playing the game, rather than just assessing what the participants are telling you has changed after playing the game.

This being said, be careful not to be fooled into thinking that what people do when they are playing a game is an “outcome.” For example, playing a game about brushing your teeth regularly does not necessarily mean that the person is more motivated to brush their teeth. It could be a clue that they are. But the ultimate outcome of your game you will want to measure is a change in their tooth brushing behaviors. On the other hand, if people are getting more exercise as a result of playing your game, this actually could be an outcome you are interested in assessing. Just be careful when you are observing people playing your game to be clear about the outcome you are interested in.

After the game is finished and before you submit your outcomes study protocol for ethics approval….

8. DO A TRIAL RUN AS A PARTICIPANT IN YOUR OWN STUDY. I think this is really important because as researchers, we get insecure and want to measure everything hoping that we find something that is able to tap into what our serious game does. But you will get crap data if your participants find your measures confusing, frustrating and tiring. If you can’t participate in the study you designed, then make changes. Take out non-sense measures and reduce the requirements load on your participants. Pilot testing of your study will also help you do this but I think it’s really important to participate yourself (and of course you wouldn’t use your own data). I also get the people carrying out the study to also  go through the process of being a participant. They will also provide important feedback and they will also gain more empathy of what it is like to be a participant in the study. But I think it is really important for the principal investigators of trials do be in the studies themselves. It is leading by doing and makes for a better research project all around.

By now you should have a good idea of how to come up with good measures for assessing the impact your serious game may have on outcomes. You know it’s a good idea to come up with measures at the very beginning of the project, find standard measures, develop your own measures when needed, see a psychologist for help (with developing and identifying measures), focus on objective measures, play the serious game yourself and watch members of the target group playing it for further insight into game effects that could be measured, and be a participant in the study you designed.

I look forward to seeing your publications in the literature. Good luck!