Serious games are great! When you play them, the good ones (!!), you get the feeling that they are a breakthrough in learning. They “feel” like they are doing much more than traditional teaching and training approaches have done in the past. You think, “Everyone should be learning through games and they will replace textbooks in the classroom and brochures in the doctor’s office!”
But that is not enough.
Games are expensive to make, they come in all different levels of quality, and sometimes when you hear that a martian on an imaginary planet is teaching kids how to become successful entrepreneurs, you have to wonder if they really “work.” So we are still at a point in time where we have to demonstrate what we do and show investors that they are getting a return on their investment.
But because serious games seem to have an impact beyond the outcomes we have traditionally measured in training and teaching, this often means that we have to be smart, thoughtful and innovative to do a good job of measuring what they can do.
Based on my experience having successfully used a range of traditional and non-traditional measures in assessing behavior change, I’d like to share some of my thoughts and advice on finding and developing measures to assess what your serious game does.
Get your brain power-up because here are some tips about how to go about measuring what serious games can do. They go from the mundane to the interesting during different stages of game development.
Before game design…..
1. BEFORE DESIGNING YOUR GAME, COME UP WITH A SET OF EXISTING MEASURES THAT COULD TAP INTO THE INSTRUCTIONAL GOALS OF YOUR GAME. This is mundane but essential. Look at your game design, instructional design documents and even your grant proposals that you are working on before you actually get into developing your game. These documents should contain practical and theoretical goals and objectives for the impact you wanted your serious game to have on the outcomes of the
This step seems obvious because it is just about finding available measures to assess your outcomes of interest. I list this step explicitly because most serious game projects don’t even take this step before they make their game. They think of it AFTER they make the game. This is not wise because serious games should be cleverly and skillfully designed from the beginning to impact outcomes of interest.
2. FIND STANDARD MEASURES. This is basically an extension of the first tip. Look for any existing reliable and standard measures that can further tap into these potentially interesting “things” you found in the process of making the game that might change through playing the game yourself or watching members of the target audience. Also keep
3. DEVELOP YOUR OWN MEASURES. But maybe you are making a game to empower patients who are older to be proactive in avoiding medical errors in their interactions with the health system. One thing you might want to see is if their knowledge of what actually constitutes a medical error (e.g., misdiagnosis, botched blood draw, inaccurate dosing of a drug with no resulting harm, etc.) changes during the course of the intervention. You will likely find that a standard measure of patient knowledge of medical errors does not exist. You may need to create your own measure.
People look at me like I’m crazy when I say this because they know that standard, published measures are highly credible. Their psychometric properties are usually well researched (e.g., test-retest reliability, correlations with other measures, internal validity,
4. SEE A PSYCHOLOGIST. If you ever hear someone say, “That can’t be measured,” it is a sign that they need to go see a psychologist. Not because they are nuts (although they might be that as well). But because psychologists go ballistic geek over measuring
You will want to find psychologists that have experience doing this. I would recommend finding someone with some kind of background in social psychology rather than a clinical psychologist (including neuropsychologists) who tend to rely heavily on standard measures in their work. Social psychologists do freaky things like measuring “altruism” by looking at whether or not students at a theological seminary will stop to help a victim slumped in a doorway if they are in a hurry or had to give a talk to an audience on the “Good Samaritan.” Pretty clever, eh? (turns out it didn’t matter how religious they were or whether or not they were giving a talk on the “Good Samaritan.” They only stopped to help if they weren’t in a hurry. See Darley, J. M., and Batson, C.D., “From Jerusalem to Jericho”: A study of Situational and Dispositional Variables in Helping Behavior”. JPSP.)
5. FOCUS ON OBJECTIVE MEASURES!!! This is the most important tip I can give you. First of all, if you can show that your serious game had an impact on a behavior rather than just something the player SAID it did, you have some pretty damn convincing evidence that your intervention worked. For example, in the Re-Mission study, all of the young patients with cancer in our study reported high levels of adherence to their medication at every assessment time point regardless of whether or not they played the game designed to improve their adherence. However, computerized chips on their bottles of antibiotics and regular blood
During game development in a rapid prototyping process…..
This is a start for finding or even creating measures to tap into the powerful things your serious game might be doing. This is also a start at going beyond traditional ways of measuring outcomes.
And by the way, if you are a researcher in charge of evaluating the game, you really shouldn’t skip this part. Because no matter what you planned to do with your game, there will be things your game will do better than others and not everything you planned to have in the game will make it in there. Accept it, adjust your expectations and revise your measures accordingly.
Also during game development….
7. CLOSELY OBSERVE AND INVESTIGATE WHAT MEMBERS OF THE TARGET AUDIENCE ARE DOING, FEELING AND GETTING FROM THE GAME PROTOTYPES.
If you
This being said, be careful not to be fooled into thinking that what people do when they are playing a game is an “outcome.” For example, playing a game about brushing your teeth regularly does not necessarily mean that the person is more motivated to brush their teeth. It could be a clue that they are. But the ultimate outcome of your game you will want to measure is a change in their tooth brushing behaviors. On the other hand, if people are getting more exercise as a result of playing your game, this actually could be an outcome you are interested in assessing. Just be careful when you are observing people playing your game to be clear about the outcome you are interested in.
After the game is finished and before you submit your outcomes study protocol for ethics approval….
8. DO A TRIAL RUN AS A PARTICIPANT IN YOUR OWN STUDY. I think this is really important because as researchers, we get insecure and want to measure everything hoping that we find something that is able to tap into what our serious game does. But you will get crap data if your participants find your measures confusing, frustrating and tiring. If you can’t participate in the study you designed, then make changes. Take out non-sense measures and reduce the requirements load on your participants. Pilot testing of your study will also help you do this but I think it’s really important to participate yourself (and of course you wouldn’t use your own data). I also get the people carrying out the study to also go through the process of being a participant. They will also provide important feedback and they will also gain more empathy of what it is like to be a participant in the study. But I think it is really important for the principal investigators of trials do be in the studies themselves. It is leading by doing and makes for a better research project all around.
By now you should have a good idea of how to come up with good measures for assessing the impact your serious game may have on outcomes. You know it’s a good idea to come up with measures at the very beginning of the project, find standard measures, develop your own measures when needed, see a psychologist for help (with developing and identifying measures), focus on objective measures, play the serious game yourself and watch members of the target group playing it for further insight into game effects that could be measured, and be a participant in the study you designed.
I look forward to seeing your publications in the literature. Good luck!
3 comments