Background

I asked readers to share their scores with me. I would like to thank all of you who took advantage of social media and let me know how you did.

It is now time to announce the winner of the Hardcore Serious Games Research Award.

THE RUNNER-UP AWARD….

First, I would like to recognize Priscilla Haring as the Runner-Up in this competition. Priscilla endorsed eight out of ten items with a heavy emphasis on her heart soaring upon hearing of randomized trials of serious games and her genuine interest in figuring out what serious games can do.

Priscilla Haring is a Media Psychologist and Consultant and the owner of Ph(?) in Amsterdam, the Netherlands. In addition to conducting research on serious games, she also writes a blog called ‘Ph(?) Research and Advice’ and posts frequently on Twitter (@ThePrisca). Congratulations, Priscilla!

AND THE WINNER IS….

And now the moment you have all waited for, the announcement of the winner of the Hardcore Serious Games Research Award. Without further ado (and it is “ado” and not “adieu”), I am pleased to present the award to Jim Hensman of the Serious Games Institute!

Jim gets the award for two reasons…

First, because he said he really like the approach I took when I wrote “You might be a serious games researcher if….” proving once again that flattery will get you everywhere with the award judge (me).

But Jim blew the competition out of the water when he said, “You had a trick question in there. Your last item said that a serious games researcher would find several typos in the post. I looked through it several times and there were no typos.”

Whoa, Jim! You not only qualify as a serious games researcher but you win the Hardcore Serious Games Researcher Award!! You were the only person that I know of to actually go through the document to check for typos and let me know. This clearly indicates your embodiment of the core skills of a Hardcore Serious Games Researcher.  *wink*  But seriously, that is HARDCORE! This definitely got you bonus points in the competition.

In addition to his attention to detail, it is not surprising that Jim Hensman is winning this award. Jim is a very cool guy. I think he has a very unique insight into the complexities of making serious games.

According to his profile, Jim works as an Innovation Specialist at Coventry University and is an Associate of the Serious Games Institute. He has a background in the electronics, computer and games industries, and was responsible for the design and development of a stand-alone commercial computer game, and the research and development for what was believed to be the world’s first “blitter” video graphics microchip system. He has also worked as a designer and consultant for a number of major companies in the wider games and toys sector, such as Hasbro, Waddingtons and Matchbox.

His research interests in the area of games include serious games design and pedagogic principles, and how practice in this area can be elicited and shared using a pattern language approach. He also has a particular interest in how physical and virtual worlds can be merged, including in the area of games and performance, where he has run improvisational theatre and performance groups and workshops, worked on alternate reality games, and created machinima performances in virtual worlds.

Once again, congratulations, Jim Hensman!

To all of you who may have checked every box on the questionnaire AND checked the document for typos, please let me know how you do the next time I post test (use social media!) so you can compete for the prestigious awards (like the above certificates) offered on the pamkato.com blog.

This concludes the awards ceremonies for the Hardcore Serious Games Research Award. Please tune into this blog on a regular basis for information on all upcoming award competitions.

Thank you very much.

Here goes…

You might be a serious games researcher if…

1. You want to crawl under a rock when people hold up poorly designed studies of serious games and proclaim that ALL serious games work.

2. You feel duped when you find out that when someone said they did “research,” it meant that they did extensive reading on the topic on Wikipedia and did a Google search for websites dealing with the topic. You lose it completely when you see the written summary on the topic they “researched” and the references are all hyperlinked to second-hand sources on the internet.

Lee, C. (2010, November 18). How to Cite Something You Found on a Website in APA Style [blog post]. Retrieved from http://blog.apastyle.org/apastyle/2010/11/how-to-cite-something-you-found-on-a-website-in-apa-style.html.

3. You know what a File Drawer Problem is.

4. You secretly wish you had the power to prescribe elementary statistics classes (or an effective serious game on statistics) to people. You painfully feel this urge when someone looks at a graph of two averages with error bars that overlap and they tell you that there actually IS a difference between the averages. But you don’t make the prescription. Instead you spend hours thinking about how you can explain summary statistics, distributions, hypothesis testing, Type I and II Errors and probabilities in less than 4000 characters (because that is the limit in LinkedIn discussion groups).

from Open Educational Resources (2013). Comparing average and variability for several datasets [online guide]. Retrieved from http://www.numeracy-bank.net/?q=t/std/cgc/7.

5. You are a bit gullible. Your heart drops when you find out that someone who told you that research on the efficacy of serious games is unnecessary, also has a company making and selling serious games (that of course have never been evaluated for efficacy). You actually read the non-scientific articles they sent you supporting their point. You are so gullible!

6. You wonder how far the field will advance when money is being put into investigating mechanisms of how serious games “work” and not into research to evaluate which serious games “work” in the first place.

7. You secretly wonder if people who show a lack of critical thinking skills and a disregard for empirical research on serious games got their information from Fox News.

8. Your heart fills with joy when you hear that researchers conducted a randomized trial of a serious game that was powered to detect differences between experimental groups.

9. You are genuinely interested in finding out what serious games can and cannot do. You love challenges and the process of discovery. This is precisely why you continue to pursue research on serious games.

10. Finally, due to your extensive experience reading, writing and editing; you can find several errors in this post (including typos and formatting errors).

Scoring:

Count the number of statements with which you agree.

• If you agreed with 0 to 3 items on this list, you are definitely not a serious games researcher.  You have a disdain for anyone with letters after their name. You are probably puzzled by how hard serious games researchers work for so little money, how much attention they pay to little details, and why they seem so frustrated when they should be forever grateful that, unlike other researchers, they get to work on fun games for a living!
• If you agreed with 4 to 7 items, you might have an interest in research on serious games but you haven’t been immersed in the area long enough to have your buttons pushed yet.  You may be currently taking a course on statistics and research design. If so, it will only be a few years for you to develop a better appreciation of the probabilistic nature of science and the extent to which researchers can get caught up in seemingly petty details.
• If you agreed with 8 to 10 items on this list, you are probably a fully fledged serious games researcher! Congratulations!

Please feel free to share your scores in the comments section below.

Note:  Ironically, the criteria above are not based on empirical studies but from subjective impressions and anecdotal reports. The criteria have not been empirically evaluated for experimental, test, internal, external, concurrent, predictive, criterion, face, construct, content or any other type of validity there is. The lack of scientific rigor in this list really irritates you if you are a hard-core serious games researcher.

Finally, if you realize the above criteria were made all in good fun, then you are a serious games researcher with a sense of humor.

The end.

p.s. Just in case you scrolled down this low, click here to find out who won the Hardcore Serious Games Researcher Award based on their responses to this questionnaire.

David Crookall, Ph.D. read my post and pointed out two things. 1) That there is another meaning of  serious game “validation” that is contributing to confusion around the term and 2) that a distinction should be made between “validation” as a process and “valid” as a label. Prof. Crookall knows what he is talking about. He is the Editor-in-Chief of  Simulation & Gaming: An Interdisciplinary Journal and is on the faculty of Université de Nice Sophia Antipolis, France.

Can I say that Prof. Crookall’s points are valid?

Yet another meaning of validating a serious game

Prof. Crookall pointed out that there have been several papers published on validating serious games that focus exclusively on the process of software validation. In these cases, the validation of the serious games  involved a process of quality testing the software.

I confess that I knew about the use of the term validation to address quality testing of software programs in the industry. But I thought I didn’t have to discuss this in my last post because surely people at conferences weren’t referring to this process when they said their game was “validated.” I read some articles that Prof. Crookall sent me and realized I was wrong! There have been several excellent papers written on validating serious games that focus exclusively on quality assurance and beta testing processes. It is therefore highly likely that people had this meaning in mind when they said their serious game had been “validated.”

This revelation left me feeling even more frustrated. This means that when people go around saying they “validated” their game they may also mean that their game underwent quality assurance processes. So now it doesn’t just mean that their game is “valid” as an assessment tool or is “valid” as an effective intervention, it can also be “valid” as a software program that will technically work and be accepted by the target users. I have a sneaking suspicion that it won’t stop here. *Sigh*

The importance of validation through quality assurance processes

Please let me clarify. My frustration in no way means that I think that quality assurance processes are not important when it comes to serious games delivered as digital content. I recall that at a Games for Health Europe Conference presentation a few years ago, Tim Laning, of Grendel Games, pointed out that most serious games do not undergo a thorough process of quality assurance testing that is standard in the software industry. He called for more serious games to undergo this industry standard process (among other things) in order to advance the overall success of serious games. I heartily agree! The quality of most serious games has massive room for improvement. Also, serious games ultimately won’t “work” (experimental validity) if they crash, contain errors, and are frustrating to play.

Quality assurance IS a validation process

It does make sense to call the quality assurance process a validation process. The quality assurance process has many things in common with the goals of Experimental and Test Validity. For example, among other things, the process helps developers and stakeholders in the serious game industry determine whether or not the software does what it says it does (see internal validity), will be accepted by the target audience (see external validity), and contains the contents required in the program (see content validity).

Suggested reading on quality assurance processes for serious games

If you would like to learn more about how the software validation process can be applied to serious games, I would recommend reading “The Validity and Effectiveness of a Business Game Beta Test” by Gold and Wolfe (2011) published in Simulation and Gaming. It provides a really nice detailed description of the process along with a presentation of the results of the quality assurance process they undertook with their business game. I was quite impressed with the article and I think it would be very helpful for people who plan to pursue this process (that should be standard in making serious games!).

Another important distinction: “Validation” as a process versus “valid” as a label

There is something else that David Crookall brought up. We need to separate the “validation process” in serious games from labeling something as being in a state of being “valid.” Confusion arises when we talk about something being “validated.” When someone says that their serious game has been validated, I bet that most of us assume that the game underwent a process of validation for X, Y, or Z AND (there is a big emphasis on this logical operator “AND”) that the serious game was determined to be valid.

No matter what kind of validation we are taking about, any of us can take a serious game through a validation process for a particular purpose and end up finding that our game is not valid for that purpose.

Example 1:  I can take a game, put it through a randomized trial to determine its experimental validity as an intervention to impact outcomes, and find that the serious game had no effect on outcomes.

Example 2: I can take another serious game and do some validation studies consisting of focus groups and correlational studies and find that performance on the game is not related to performance in the real world, it is not correlated with any similar game measures, and experts don’t even think that the game contains the necessary content to train and educate players.

Example 3: I can make another serious game that I put through quality assurance testing but I have failed to be able to resolve major bugs with the program, users find it completely frustrating to use, and I ran out of money to fulfill all of the content requirements for the game.

In all of the above three examples, the serious games were “validated,” in the sense that they went through a validation process. But none of these games would likely be considered to be “valid” for the purposes they were being tested.

It would therefore not only be helpful for people to be very clear about what they validated their games for, but people should try to be clear about what the results of the validation process were.

Communication

The business of making serious games is inherently interdisciplinary. People from different cultures (software, research, education, arts), backgrounds, and languages come together to make games that educate and train in addition to being entertaining. Mutual understanding and clear communication are inherent challenges. Luckily, there are ways to address them.

Suggested behaviors:

When we talk about our serious games, we can begin by completing these phrases appropriately,

1) “My serious game underwent a validation process for _______.”

2) “We found that the serious game was valid for______.”

And if we hear someone say that their game has been validated, we can ask,

1) “What did you validate your game for?”and

2)  ”What did you find that your serious game was valid for?”

Seems easy enough. Let’s see if these simple behaviors can clear up some of the confusion.

Conclusion

To sum things up, my frustration is that people talk about validating games and valid games without clarifying what they are validated for. I am further frustrated that “validation” and “validity” can mean such different things and processes. And to be quite honest, mixed in with the frustration is some embarrassment that I had assumed it meant one thing when I have very slowly over time learned it meant so many other things (with a power-up from David Crookall).

So readers, I would like to publicly apologize for my previous talks and papers in which I assumed that a validation process for a game meant an outcomes trial and that a game was valid if the trial showed was effective in addressing the outcomes intended. I was guilty of not being specific and of assuming you knew what I was talking about. I was contributing to the problem. My efforts now focus on trying to be a part of the solution.

Time to practice good communication skills.

But when I listen more closely, sometimes I  hear people say that they have “validated” their serious game at various steps of the development process. Humph. How do you validate an incomplete game for effectiveness? Then it turns out they never conducted a trial to evaluate the efficacy of their game to impact outcomes.  But they still say they “validated” their game. How can that be?

I was confused by this for a long time until I dug a bit deeper into the use of the term “validation.” I think I found the root of my confusion.

The confusion surrounding the use of the term “validity” with serious games often arises because serious games can be used as interventions to impact outcomes or as measures to assess performance. Someone could say, “My Kinect exercise program has been validated.”  This could mean that the Kinect exercise program is a valid intervention shown to increase physical fitness outcomes. It could also mean that their Kinect exercise program has been determined to be a good way to measure how much a person exercises. The game can be a good measure of physical exercise regardless of whether or not  people who play it are exercising more as a result of interacting with the game.

It’s basically the difference between a pedometer that has been shown to lead to increases in physical activity versus a pedometer shown to give  accurate readings of physical activity in the people who use it, whether or not these people get off the couch and start exercising more.

Both types of validity are important but obviously indicate very different things. Experimental validity refers to treatments or interventions that have gone through the process of an experimental study or studies that provided adequate evidence that they “work” as intended.  Test validity refers to assessment tools or measures that are evaluated and developed with a combination of subjective assessments and correlational studies that demonstrated that they do a good job of “measuring” what they are intended to measure.

I am going to explain the details of experimental validity and test validity below. I realize that for many of you this discussion will be very academic and boring. I will try to make it as interesting as possible. I am doing this because I hope you can slog through it so you might gain some insights that in the least change the way you think about things. Ultimately, I hope they might change your  behavior. Specifically, I hope that it will inspire you to be more specific about what you mean by validation when you use the word.  I also hope that if you hear someone say that their serious game has been validated, you will be confident enough to ask if that means it has been validated as intervention or as an assessment tool.  So if you can bear with me through the following academic discussion, I will follow-it up with my thoughts on why I think this distinction between experimental and test validity is important to clarify in the area of serious games.

EXPERIMENTAL Validity

Experimental validity refers to whether or not experimental condition or treatment has an effect for its intended purpose. We can talk about experimental validity in terms of having differing levels of internal and external validity.

Internal Validity

Internal validity refers to whether or not an experimental treatment (in our case, a serious game) makes a difference on outcomes AND whether or not there is sufficient evidence to support the claim that it makes a difference. People often say that Re-Mission, a game for young people with cancer, has been validated as an adherence intervention because we did a very large randomized trial with objective measures. The results showed that patients who played the game increased their adherence to oral chemotherapy and antibiotics. The randomized trial design nailed the evidence for drawing causal conclusions about the game with a slam dunk of scientific evidence (yes, criticize randomized trials all you want but they do a great job of testing causal claims). There was therefore good scientific evidence in this one trial to support claims that Re-Mission actually “worked” as intended. We can therefore say that Re-Mission is a serious game that has internal validity as an intervention for treatment adherence among young people with cancer.

External Validity

External validity refers to whether or not the claimed effects of an experimental treatment or serious game will transfer to people, environments and situations outside the original scientific study setting. In the case of Re-Mission, the game’s effects probably have high external validity for young patients with cancer in the US, Australia and Canada where the study was conducted. It has questionable external validity for older patients with cancer or other diseases in other countries and cultures where the effects were not evaluated.

When I give talks at conferences where I try to argue that we need more quality research on serious games, I ask audience members to raise their hands  if they cited the Re-Mission study in their proposals to get funding to develop their own serious game. I then ask these people what type of serious game they were proposing to develop. They have told me that they were proposals to develop a classroom game to teach math, a game for cystic fibrosis, and even a game for driver education (among many topics). I then ask them if they really think that the findings from a treatment adherence game for young cancer patients will generalize to their groups. There is nervous laughter and some heads look at the floor. The findings may or may not generalize to their game. This is an issue of external validity. By the way, this doesn’t mean these people shouldn’t cite the Re-Mission study. It does mean that any new serious game should plan to scientifically evaluate whether or not their new serious game does actually have an effect on intended outcomes. It certainly is not a given that one study proved that all serious games should work.

Taken together, internal and external validity as components of experimental validity allow one to make claims that their serious game works in light of the evidence they have to support that claim while acknowledging the limits of how much that claim might generalize beyond the existing evidence.

Where the Confusion Sets In

Serious Games as Assessment Tools

There is confusion about what a “validated” serious game means because serious games are not only being used as interventions to impact outcomes but they are also used as measures to assess performance. It is confusing for me personally because my initial assumption is that serious games are made to impact outcomes. BUT, people do make and use serious games as assessments to measure something about the player.

If we look at the field of medical education, we find many examples of serious games used  to train aspiring doctors. Most of these serious games are actually simulation training tools, such as a surgical simulator (note: we can argue about whether or not these are really “serious games” elsewhere). Surgical simulators are digital interactive tools that can be used to help doctors in training learn surgical skills. They are educational tools designed to train and shape target behaviors.  Confusion arises around issues of validity because these same training simulators can also be used to assess student performance. The validity of a surgical simulator as an assessment tool involves efforts to determine what is known as “test validity.”

Test Validity

If we use a surgical simulation as an example, the simulator would in general be a valid measure of surgical skill if performance on the simulation was a good indicator of surgical performance on real patients in a real operating room (or “theater” as the Brits like to say). There are several components of test validity that one would evaluate to determine the validity of a surgical simulation as a test of surgical skill.

Construct validity in testing

A measure has construct validity if is actually measuring what it theoretically set out to do. For example, a surgical simulator could set out to measure the technical aspects of surgical skill. It could provide situations that allow for the measurement of  hand-eye coordination, fine and gross motor skills, mechanical knowledge (how to tie a knot or make a cut), and efficient performance with economy of movement and minimal errors. Surgical skill as a construct might also set out to assess some non-technical skills shown to be related to surgical success in the real world such as communication skills, stress management, or teamwork.

Content Validity

Content validity indicates the extent to which an assessment tool contains content that relates to the knowledge or skills that are required in the area of assessment focus. A surgical simulator would have content validity if group of experts agreed that the content it content allowed for the evaluation of critical technical and perhaps even non-technical skills that would be evident in someone skilled in surgery. If the simulator left out opportunities to evaluate a critical skill such as fine motor skills, then content validity would be lower.

Criterion Validity (Predictive and Concurrent)

Criterion validity refers to the extent that performance on a test relates to criteria in the real world. If a surgical simulator has high criterion-related validity, then performance scores on the surgical simulator would be highly correlated with surgical performance in the real world on real patients in a real operating room.

Criterion validity can also be determined by its concurrent and predictive validity. The simulator would have concurrent validity if it was highly correlated with other assessments given at the same time as the surgical simulator that  have been shown to be highly correlated with surgical performance in the real world. Furthermore, the surgical simulator would have high predictive ability if performance on the simulator was determined to be highly correlated with surgical performance months or even years later.

Face Validity

Face Validity (consensus) is basically whether or not you can subjectively take a look at the assessment tool and it seems to be “getting at” what you want it to get at. Many times, face validity involves experts agreeing on whether or not the assessment contains critical items that will get at the core construct in a way that makes sense. Face validity can be important for people taking the test. It influences whether or not they think the test is fair and therefore plays a part in their motivation to take the test.

Overall, different aspects of test validity “get at” whether or not an assessment is measuring what it is intended to measure. It consists of qualitative and empirical approaches. These approaches share some similarities but many differences from processes used to evaluate experimental validity.

The importance of distinguishing experimental validity from test validity

OK, here is why I think it is REALLY important to be really clear whether or not the a serious game is a valid treatment or a valid measure.

Important point #1

When we say we need validation studies of serious games, I think most people mean that we need good evidence that our serious games work. Thus, most people are talking about experimental validity. While it is important to validate serious games used as assessments for the test validity, at this point in time, the future of the field depends more critically on validating their use as tools to train. There are overwhelmingly more serious games that have been developed as intervention tools than as performance measures. The fact that most of these have not been validated for the impact they have on intended outcomes weakens the basic premise of serious games that they provide engaging and entertaining ways to train and educate. While it is important to the field that  serious games used as performance measures are validated, I think that efforts to validate serious games as interventions provide more powerful support to the strength of our endeavor as whole.

When I hear someone say that they validated their game as an assessment tool, I can almost audibly hear people breathe a sigh of relief that someone else has done the hard work of validating a serious game. But watch out, they validated it as an assessment tool and not as an intervention. So you can breathe but not a sigh of relief. There is still work to be done in evaluating whether or not most of our serious games can be considered valid intervention tool that impact outcomes. As stated above, I believe this is a critical issue for the future of serious games.

Important point #2

We should be clear about what we are validating because valid measures can creep into becoming interventions through the magic of Testing Effects. I explained the magic of Testing Effects in more detail in last week’s post on the efficacy of Brain Games. Brain Games basically take assessment tools that psychologists have used to measure cognitive functioning, put them in a game format,  get players to practice them and get better on that test. They then claim these tests-turned-interventions made players better in the area of cognitive functioning that the test is supposed to reflect. That ain’t necessarily so, my friends. The DMV (Department of Motor Vehicles) may fall for that type of reasoning when people take their driving test multiple times and improve on it, but hopefully we’re smarter than the DMV. We know a bit more about the difference between a test effect and an intervention effect. A validated measure is NOT the same thing as a validated treatment. In fact, a well-validated measure can be used to show that a treatment is NOT valid. These beasts are related but they are not the same beast.

Conclusion

Thank you for sticking with me through this very academic discussion of validity. Moving forward, the field is wide open to have serious games that have been validated as interventions AND/OR as assessment tools. We just need to use the word “validity” more carefully when we talk about our serious games. And let’s not be afraid to get other people to clarify what they mean when they say their serious game has been validated. We can simply ask, “Are you referring to experimental or test validity when you say your serious game has been validated?” or “Are you saying that your serious game was validated as an intervention or a measure?” Or as Dr. Lyle A. Brenner, Jack Faricy Professor of Marketing at University of Florida is likes to say, “‘Validated’ always needs a modifier.  For example ‘validated as a measure / predictor of X’ or ‘validated as affecting Y.’”  

By asking these questions and providing these reminders, you’ll probably clear up a lot of confusion not only for yourself but for other people in the room as well. And finally, let’s try to do more validation studies on serious games as effective tools to make people smarter, healthier, stronger, and maybe even kinder. The future of serious games depends on it.

(Note: Please read my next post for Part 2 on validating serious games http://wp.me/p299Wi-dW)

References

Campbell, D. T., Stanley, J. C., & Gage, N. L. (1963). Experimental and quasi-experimental designs for research (pp. 171-246). Boston: Houghton Mifflin.

Carter, F. J., Schijven, M. P., Aggarwal, R., Grantcharov, T., Francis, N. K., Hanna, G. B., & Jakimowicz, J. J. (2005). Consensus guidelines for validation of virtual reality surgical simulators. Surgical Endoscopy and Other Interventional Techniques, 19(12), 1523-1532.

Kato, P. M. (2012). Evaluating efficacy and validating games for health. GAMES FOR HEALTH: Research, Development, and Clinical Applications,1(1), 74-76.

Kato, P. M. (2010). Video games in health care: Closing the gap. Review of General Psychology, 14(2), 113-121.

Kato, P. M., Cole, S. W., Bradlyn, A. S., & Pollock, B. H. (2008). A video game improves behavioral outcomes in adolescents and young adults with cancer: A randomized trial. Pediatrics, 122(2), e305-e317.

When is the answer “YES”?

Arthur Rubenstein, often considered the greatest pianist of his generation, was apparently approached on the streets of New York City by someone who asked him how they could get to Carnegie Hall. The story goes that Arthur Rubenstein replied “Practice, practice, practice!”

And so it, goes. If you want to get better at something, you have to practice.

Video games are great tools that offer endless opportunities to practice and learn. In contrast, in traditional schooling approaches, you take a test, you don’t do so well, you get a low score, you feel defeated and diminished, and then you move on to the next assignment. You’re not allowed to take the test again and master the material. No wonder kids love video games. They love the FUN of mastering challenges and the feeling of accomplishment! As far as I’m concerned, video games prepare people much better for the real world in which success is based on learning from failures, mastering challenges, and reaching goals despite the number of times they failed.

The ability to practice and get better is at the core of what is fun about games and why serious games can be used to teach and train.

Now let’s turn to brain games. They do the same thing. They present players with cognitive tasks in a game format that players can take multiple times to improve their score on these tasks. So it seems at face value to make sense that you can improve your cognitive functioning by playing these games.

The cognitive tasks that are in brain games have been around for quite a while. Psychologists have used them for years to assess things like attention and memory. For example the Stroop Test has been around since John Ridley Stroop first published his seminal paper on the Stroop Effect in 1935. Brain games often include Stroop Tests among the many cognitive tasks in their lineup.

Quite unfortunately for the players, the Stroop Test is not about how many addictive caramel waffle cookies you can eat in Holland (stroopwafels…YUM!). It is about how quickly you can correctly name the colors of words. It is not as easy as it sounds. You can give a try yourself.

Take a look at the graphic below and as quickly as possible, say out loud the COLORS on left hand side and then the COLORS (not the words) on the right hand side.

It was probably really easy for you to name the colors on the left hand side where the colors corresponded to the words. But you probably struggled to name the colors on the right hand side when the colors differed from the meaning of the words. Pretty much everyone struggles with naming the colors on the right hand side because reading and gathering meaning from words is a much more ingrained and practiced habit than naming colors. The brain, through years of practice and learning, is automatically ready for you to process the meaning of word first. This ingrained process interferes with your intention to disregard the meaning of the word and focus on only the color that you see.

Psychologists count how many errors people make in naming the colors on the right hand side. They also measure how much longer it takes people to name the colors on the right compared to the left hand side. This information gives them an idea of how well someone can focus their attention, processes information and inhibit unwanted responses. People who don’t do so well on this task compared to other people might also have difficulties in the real world staying focused on boring tasks or in keeping their mouth shut when a compelling but socially unacceptable thought comes to mind.

With brain games you basically do tasks like the Stroop Test and get a score on how well you did based on errors and how long it takes you to respond. You can then try to improve your score which is often indicated as your game IQ (BPI with Lumosity) or  age (with Brain Age). When you start a game like Brain Age, your mentor, may tell you that your brain age is something like 80 years old. Your fear of being an old person with an old brain probably motivates you (even if you are very old!) to practice many times to improve your performance.

This game mechanic of feedback loops and motivation incentives engages players to practice cognitive tests like the Stroop Test over and over.  With practice, players improve and their game IQ or brain age improves.

In sum, brain training games are really good at getting you to practice a cognitive test like the StroopTest and improve your performance on THAT cognitive test. So YES! Brain games  can improve your performance on THAT cognitive task. They are “good” in this way without a doubt.

However, the assumption is that as your score improves, you are improving your ability to focus your attention, process information, and inhibit unwanted responses not just on this task but in other areas as well.

The question is whether or not taking a test again and again and improving one’s score in brain games indicates a REAL improvement.

Are performance improvements on a cognitive task “REAL” improvements?

Testing Effects

Claims that brain games actually improve cognitive abilities that translate to other domains of functioning can be viewed as a “Testing Effect.” The phenomenon of practicing a test and getting a higher score on the test as a result of that practice is  known as a “Testing Effect” in research design.

For example, I can take an IQ test and get a score of 100. Then I can figure out what I did wrong and take the IQ test again and again until I get a score of 115. The question is then, do I REALLY have a higher IQ? Or does my new IQ of 115 simply reflect the fact that I took the IQ test multiple times. Most people would say that my IQ improved because I practiced the test but I’m still as smart (or dumb) as I was before I practiced and got better on the IQ test.

Research Note

We can do things in research to help us rule out testing effects. I know research is boring for most people but please indulge me to talk a bit about research design.

If I am evaluating the impact of an intervention on outcomes, I want to make sure that the improvements I see in test performance are a result of the intervention and not the result of having previously taken a test. This is one of many reasons why it is VERY important to have a control comparison group in evaluations of interventions. If there is an improvement in performance simply due to having taken a test before and not the intervention, you will see a similar improvement in performance in the control group that is no different from the intervention group. If the intervention “works” you still might see a testing effect in the control group with improved scores, but the improvement in the intervention group should be much bigger.  Get the picture?

The NO

A good study of brain games that used a control group

A very large scientific study conducted through the BBC that was published in Nature had very convincing findings. It showed that if one trained and improved their performance cognitive tasks in brain games, one did not show improvements in similar tasks of those cognitive abilities when presented with tasked they were not trained on.  D’oh!

This video does a really nice job of summarizing the results. Please also pay attention to what the participants in the research thought the game was helping them do and how surprised they were by the results (see 0:15 for people’s self-report of how they thought they did when they played the games and their surprise when they see the scientific evidence later).

The results  showed that the apparent cognitive gains on tasks that people were trained on did not transfer to cognitive tasks they were not trained on. Humph, it looks like a testing effect. This looks like a big NO for brain games being “good.”

Brain games and the research on them point out some problematic thinking when people design games and when they set out to do research on them.

Big Problem #1: Using in-game metrics to show efficacy

They use in-game metrics to show that their game “works.” They say that improvements on cognitive tasks in their game means that they improve in the real world without measuring what happens in the real world. Or they say that increases in a physical skill like balance on the Wii balance board means that these people don’t trip and fall in the real world. The problem is that they didn’t go out into the real world and  see if these people had better balance there as well.

Listen. I love games and I believe in the power of serious games. But I could give a @!#’s !@# about what people do in a game. I care about what the game actually leads them to do in the real world. And yes it is difficult to measure things in the real world, but can we at least give it a try?

Big Problem #2: Using self-report qualitative assessments  to show efficacy

Big Problem #2 happens when researchers attempt to get at what is going on in the real world by asking people, “Did you think the game made you change the way you do things in the real world?” When a significant number of people say yes and believe it very strongly, the researchers then conclude that their game “works.” I don’t believe these statements if they are not supported by similar changes in observable objective behaviors or physiological changes compared to placebo control group. There is just too much pressure on people to say positive things about an intervention in an experimental situation and to be very bad at objectively evaluating their own performance (have you watched the auditions for “Idols” on TV recently?). It is not that people lie, and they do that sometimes to make themselves feel good, but they also have cognitive biases to focus on evidence that supports their wishes and views of themselves. And like I said before, if people self-report something but the objective evidence says something else, people will always go with the objective evidence, not the other way around. (Note: You can read my 2008 study of Re-Mission to see how the objective measure of adherence told a different story than the self-report measures of adherence and decide for yourself which measure you believe.)

Also, think of the BBC study discussed above. If you saw the video, there probably would have been an intervention effect shown in the study if the researchers only asked for the self-assessment of the impact of the games on the people playing the games. Great. But objective measures did not bear it out much to their surprise! And by the way, the study would not have been published in Nature nor would it have been considered as scientifically rigorous by most scientists who read it if they only evaluated the games based on subjective self-reports.

The MAYBE

Caveat. Now, the BBC was just one study. There are other studies that look at brain training games. The findings are mixed and the scientific quality of the studies varies. There are also many, many anecdotal reports that brain games work and transfer to other areas of real life. These are usually put out by the makers of the game.

BUT, as much as I  bring up criticism of brain games and the tactics they use to get people to believe they work, I am actually not fully convinced that they don’t work. Science is probabilistic and depends on converging evidence to help us understand how things work. There simply have not been enough good studies of brain games to convince me they don’t work (or do work for that matter). The time is now to do a few really good studies to gather more good evidence to get at whether or not brain games are any good.

In the meantime, I will probably still enjoy playing a brain game now and then with the hope it is helping to keep my brain young. But I would probably enjoy them even more if saw some more really good research coming out examining its claims in the “real world.”

Research, research, research!!!

One of the first things to do as part of your serious game homework is to figure out what problem or problems you want it to address. Don’t be the person who responds with a blank stare to the question, “Is this actually a problem with the target group for your serious game?” Choose a problem that is big enough for you to measure a reliable change in it when people play it.

How to find big problems to address

Let’s say that I want to change the world by making a serious game to help people live longer. Ultimately, I want to reduce premature deaths thereby improving people’s average life expectancy through my intervention. I will walk you through this example to show you how to think about big problems.

Homework assignment 1: Figure out where the problem is

The problem may be found in a certain location.  Because I’m interested in prolonging life, I read up on lots of things. I find out what causes short or long life expectancy because that gives me clues about what I need to change to increase life expectancy. I also need to find out where life expectancy is a big problem.

I read  articles that compare average life expectancy across countries. It turns out that there is a relationship between income and life expectancy.  Luckily, I find a cool site that shows these relationships nicely. You can see the relationship in the graph below. You can also go to the site for the interactive graph to get more information and details. 

The dots that represent countries are clustering around a line that positively slopes upwards. As I look more closely at the horizontal and vertical axes of the graph, it appears that countries in which people make more money on average (have a higher per capita income) live longer on average (have a higher average life expectancy).

This graph also demonstrates that average life expectancy pretty much is at its best around 85 year of age. I can go to those countries where average life expectancy is around 85 years of age to improve life expectancy because, of course, not everyone in those countries lives such a long life. But let’s face it, to have a big impact on life expectancy, I can have a much bigger impact if I go to the countries with a low per capita income where life expectancy is in the 40′s and 50′s. I have lots of room to make huge improvements there. Big problems are characterized by lots of room for improvement.

What is “room for improvement”?

Let’s take a look at the numbers of how big of an impact I can have by prolonging life in places where people have a relatively shorter life expectancy.

Think of it this way, if I have a barrel of people with high incomes whose average life expectancy is 80 years and I improve their life expectancy by 20 years, I have increased their life expectancy by 25% (26 years is 25% of 80 years). A 25% increase is not bad! But do I really have room to get people who live on average to 80 years of age to live to 100? Is that even biologically possible??? Seems like a difficult task if you ask me.

However, if I take a barrel of people with low incomes whose average life expectancy is 50 years and I similarly prolong their lives for 20 years, THEIR life expectancy has increased by 40% (20 years is 40% of 50 years). For the same increase in the average number of years lived, I have almost doubled my impact with this group.

And practically speaking, doesn’t it seem more “biologically” possible to increase the average life expectancy of a group of people from 50 to 70 years of age?. Lots of people live until they are 70 so it is a more reasonable goal than the effort it must take to get people to live until 100. More on this later.

To summarize, if I can prolong life for 20 years with my serious game intervention, I can go to a country with high per capita income and prolong their lives by 25% or I can go to a low per capita income country and prolong their lives by 40%. Obviously, I can have a bigger impact by picking the place where the problem is bigger. And I have more room to make a big improvement by going where the problem is bigger.

Bonus benefits of focusing on big problems

You will often find that there are more things you can do to make changes where the problems are bigger.

People with who live in countries with high per capita incomes already have lots of things working for them that support a long life expectancy. They have almost 100% access to excellent health care that is affordable (with one notable exception but hopefully that will change in the next few years), sanitary living conditions, and environments with low rates of violent crimes. Because conditions are so good for them, it is difficult for me to improve on an already good situation with my serious game.  I would have to search around a lot to find VERY effective ways of making their small problem even smaller.

CAVEATS

I used countries as the unit of analysis to in this example. I could have looked at life expectancy in urban versus suburban environments, hospices versus elementary schools, or countries with strict gun control laws versus the United States. The point is that you have to do some kind of homework on where the problem is and go there. And if you only have access to people and places where the problem is small, consider focusing on a different problem if you want to have an impact that is measurable.

By focusing on income, I was able to identify a problem that was big not only in severity, but it also more widespread. In the world and even within countries, there are relatively more people with lower incomes than higher incomes. Thus, my target audience and my eventual research sample is big. Unfortunately, this isn’t always the case. Lots of problems are big with lots of room for improvement because they only affect a small number of people. The argument still holds that you should go for these big problems if they affect a small number if you want to have a big impact. You will need to do your homework in other ways (e.g., partnering with associations that have access to the target group, coming up with innovative and tailored solution for a problem that virtually nobody has tried to address before). I am a big fan of this approach because I am quite fond of having a big impact by boldly going where no serious game has gone before.

Conclusions

If you want to change the world, you have to go where change is needed. I hope I have inspired you to find big problems to address in your serious game that will change the world….one big problem at a time.

This health care model is rooted in history. It worked well over 100 years ago when people lived to the age of 49 on average and they died either of an acute illness or of a chronic disease that had no cure at the time (e.g., cancer or heart disease).

This acute care model has less value in advanced societies of today in which the average life expectancy has almost doubled and most people are living a great portion of their lives with chronic illnesses. These people have to interact with the health care system on an ongoing basis. Then, when they are sent home from the hospital or doctor’s office, they still have a disease that they need to manage on their own outside of the watchful eyes of a health professional.

In sum, health care has shifted from the care of a passive patient to one that requires patients to be empowered to manage their health by engaging in behaviors that prevent disease and manage chronic diseases and disability over time.

What does this new model look like and where can serious games and gamification fit in?

There are efforts to move health care to a new model of care called the Chronic Care Model (CCM). The Chronic Care Model was conceptualized to address the growing burden of chronic diseases in a health system that was structured to treat acute illnesses and injury as described above. It focuses on restructuring health care systems to provide higher quality care and increase patient engagement in self-management of chronic illnesses in a way that addresses their physical, psychological and social needs.

The CCM model is based on a review of diabetes interventions that showed that multifaceted interventions that included one or more of four categories led to the greatest improvements in outcomes for organizations and patients. These four categories are

• Education programs, audits and feedback, peer reviews, and reminders focused on the physician,
• Patient-focused interventions that provided information or support,
• Interventions that promote more team-based approaches to care delivery, especially ones that enhance the role of the nurse, and
• The use of registry-based computer information systems to track patients.

One thing that the Chronic Care Model does not specifically acknowledge but is clearly needed, are strategies that are engaging, challenging and even enjoyable for patients and members of the medical team. These strategies should move patients from being passive to being more active. This is precisely where gamified approaches can step in to play a role. They may be able to support efforts within the Chronic Care Model to be more effective and appealing. There is already good scientific evidence that serious games for health can improve adherence to cancer treatment and help improve depressive symptoms as well as traditional face-to-face psychotherapy. These games put the power of health actively in the hands of patients. Reviews of games for health also show that games to improve physical activity and promoting physiotherapy are showing a pattern of positive results.

Health care is overdue for a change. By implementing serious games and gamification in the movement to improve healthcare, we may not only make the change more enjoyable, we might even make healthcare fun!

Let’s say you made a serious game to increase the engagement of seniors in regular physical activity at a gym. One of the “research goals” of your game was to increase player’s self-efficacy to engage in regular physical activity. You included role models in the game, you separated tasks into manageable chunks, you presented players with mastery experiences, you provided encouraging feedback, and you helped players manage their stress and anxiety related to engaging in the target behaviors. The game is engaging and fun. Done!

You also designed a study in which you measure self-efficacy before and after seniors play your serious game (compared to a control group of seniors that doesn’t play your game). You hypothesize that your game will increase player self-efficacy to engage in regular physical activity at the gym (note: you may further hypothesize that increases in self-efficacy will mediate or “explain” increases in actual behavioral engagement in physical activity).

You want to make sure you have a really good measure of self-efficacy. What do you do next?

Please pick the BEST answer:

1. Use a published measure of general Self-Efficacy that has acceptable levels of reliability and validity.
2. Use a published measure of Self-Efficacy to exercise regularly that has acceptable levels of reliability and validity.
3. Generate your own measure of Self-Efficacy.
4. Forget about measuring Self-Efficacy. Just get an objective assessment of actual  engagement in regular exercise (e.g., pedometer, frequency of attending a gym). Objective measures are always better than self-report!

Believe it or not, the correct answer is…. (Please scroll down)

#3. GENERATE YOUR OWN MEASURE OF SELF-EFFICACY!

Shocking, isn’t it? Developing your own measure should be the last resort shouldn’t it? Take a deep breath and let me help you reduce your galvanic skin response by walking you through why the other responses are not ideal. I will also show you how you can develop a really good self-efficacy measure on your own.

Why 1. ”Use a published measure of General Self-Efficacy that has acceptable levels of reliability and validity” is INCORRECT.

You designed your game based on self-efficacy theory to help players address specific challenges in specific situations regarding engaging in regular exercise. Although many general measures of self-efficacy exist, they will not help you fully capture increases in self-efficacy in this domain. You were not trying to help your players increase their self-efficacy across varied domains that can include public speaking, paying taxes on time, and remembering to take out the trash regularly so why measure that? You can if you want to, but I suspect you won’t fully capture the power your game has to increase self-efficacy with a general measure.

Why 2. “Use a published measure of Self-Efficacy to Exercise that has acceptable levels of reliability and validity” is INCORRECT.

This was the tricky option because it is almost correct. It could be that someone developed a measure of self-efficacy to engage in regular exercise that you could use in your study. This would happen if the measure were developed with a similar target population (e.g., older people and not school-age children), target outcome (e.g., regular physical exercise at a gym and not hiking in the woods), facing similar challenges (e.g., finding a gym that is open early enough to suit their schedule and not finding alternative means to work out if it rains).

It is usually not the case that a published measure is perfectly suited to assess the domain and specific situation that you targeted in your intervention. Therefore, #3 is the MOST correct answer.

Why 3. “Generate your own measure of Self-Efficacy” is CORRECT.

In the words of Al Bandura, “Scales of perceived self efficacy must be tailored to the particular domain of functioning that is the object of interest” (Bandura, 2006). That is his published statement and this is why it is best to generate your own measure of self-efficacy.

In addition, I also have a personal anecdote to support this claim. I was fortunate enough to be able to take a class from Al Bandura on self efficacy in the 1990′s. He was putting the finishing touches on his book, Self-Efficacy: The Exercise of Control while teaching this class. I vividly remember how he laughed when he described how he frequently received phone calls and requests for a global measure of self-efficacy.  He chuckled as he talked about how these people had a hard time understanding that he could not point them to a published scale of self-efficacy that he would recommend them to use.  They seemed incredulous that he could not offer them any help with finding a measure. They had to create one themselves.

So if you thought that you had to use a published “validated” measure of self-efficacy, you are in good company. Hopefully this information has saved you hours of combing through the literature and attempts to contact Al Bandura personally.

The good news is that there are good published guidelines for creating self-efficacy scales that you can use and cite when you write-up your research for publication. Prof. Bandura published a helpful guide for constructing self-efficacy scales that gives very specific guidelines for creating a good self-efficacy scale. Chapter 2 in his book, Self-Efficacy: The Exercise of Self-Control contains a discussion of conceptual and methodological issues involved in developing self-efficacy scales.

A final note on why 4.  ”Forget about measuring Self-Efficacy. Just get an objective assessment of actual  engagement in regular exercise (e.g., pedometer, frequency of attending a gym). Objective measures are always better than self-report!” is INCORRECT.

This is another tricky one because really, who really cares if the game increased self-efficacy as long as people who play your game are engaging in more regular physical activity? Ultimately, the behavior and the health outcome is what we should be concerned about it if we want to change the world. I agree there.

However, we researchers are interested in mechanisms of action that explain what we see.  We want to go beyond the magic of waving a wand with a game and getting an improvement in health outcomes. We want to know why it happened (or not) and either replicate the findings or tweak the approach to get even better outcomes the next time. We want to learn and we want to be able to explain things.

Also, people who develop serious games, the game developers, SHOULD be interested in maximizing the power of their serious games to make a difference by building in these active ingredients (and leaving out the inert fillers) that are found in research studies.  It always surprises me to hear that there are game developers out there who aren’t interested in seeing what their games are actually doing. Luckily, that seems to be becoming more rare as the field is growing and becoming more “legitimate.”

So off you go to fearlessly develop your own self-efficacy measure for your study! People will say you are “mad” but you can defend yourself with your knowledge of self-efficacy and documented approaches to developing these scales yourself.  You are ready to level up!

But that is not enough.

Games are expensive to make, they come in all different levels of quality, and sometimes when you hear that a martian on an imaginary planet is teaching kids how to become successful entrepreneurs, you have to wonder if they really “work.”  So we are still at a point in time where we have to demonstrate what we do and show investors that they are getting a return on their investment.

But because serious games seem to have an impact beyond the outcomes we have traditionally measured in training and teaching, this often means that we have to be smart, thoughtful and innovative to do a good job of measuring what they can do.

Based on my experience having successfully used a range of traditional and non-traditional measures in assessing behavior change, I’d like to share some of my thoughts and advice on finding and developing measures to assess what your serious game does.

Get your brain power-up because here are some tips about how to go about measuring what serious games can do. They go from the mundane to the interesting during different stages of game development.

Before game design…..

1. BEFORE DESIGNING YOUR GAME, COME UP WITH A SET OF EXISTING MEASURES THAT COULD TAP INTO THE INSTRUCTIONAL GOALS OF YOUR GAME.  This is mundane but essential. Look at your game design, instructional design documents and even your grant proposals that you are working on  before you actually get into developing your game. These documents should contain practical and theoretical goals and objectives for the impact you wanted your serious game to have on the outcomes of the players. Go to the literature and find studies that have also tried to measure these outcomes and make a list of what they have used successfully. “Successfully” means that the measure has been able to tap into a change in behavior or cognitions. If you find a measure that comes up in every study but doesn’t move across time or with an intervention, then it might not be sensitive to changes. Don’t take the risk that this happens in your outcomes evaluation as well. Ideally, you should find measures that have tapped into changes that have come about with interventions or games that are similar to yours.

This step seems obvious because it is just about finding available measures to assess your outcomes of interest. I list this  step explicitly because most serious game projects don’t even take this step before they make their game. They think of it AFTER they make the game. This is not wise because serious games should be cleverly and skillfully designed from the beginning to impact outcomes of interest.

2. FIND STANDARD MEASURES.  This is basically an extension of the first tip. Look for any existing reliable and standard measures that can further tap into these potentially interesting “things” you found in the process of making the game that might change through playing the game yourself or watching members of the target audience. Also keep up with the literature during the process of developing your game to see if any new measures come out or any new studies have used some measures you haven’t heard of before. Stay focused on those “things” that are important to demonstrate an impact. Do you see an increase in self-esteem? There are standard measures out there to assess that.  Just make sure that you find the self-esteem measure that maps most closely to the type of self-esteem you are trying to improve with your game.

3. DEVELOP YOUR OWN MEASURES. But maybe you are making a game to empower  patients who are older to be proactive in avoiding medical errors in their interactions with the health system. One thing you might want to see is if their knowledge of what actually constitutes a medical error (e.g., misdiagnosis, botched blood draw, inaccurate dosing of a drug with no resulting harm, etc.) changes during the course of the intervention. You will likely find that a standard measure of patient knowledge of medical errors does not exist. You may need to create your own measure.

People look at me like I’m crazy when I say this because they know that standard, published measures are highly credible. Their psychometric properties are usually well researched (e.g., test-retest reliability, correlations with other measures, internal validity, etc.). Also, because they are standard, scores on them from one study can be compared to scores on them from another study with a different population to get a general idea of how they compare (example, the SAT in the US). They also know that you can’t just flat-out come up with your own measure and expect people to believe or journals to accept your research with your made up measures because you haven’t evaluated the reliability or validity of the measures. However, in some circumstances, this is EXACTLY what you need to do. This is really at the heart of demonstrating the powerful and important things that serious games can do. In sum, it’s ok to develop your own measure. You can be the first.

4. SEE A PSYCHOLOGIST. If you ever hear someone say, “That can’t be measured,” it is a sign that they need to go see a psychologist. Not because they are nuts (although they might be that as well). But because psychologists go ballistic geek over measuring intangible things like stress, self-esteem, motivation, happiness, etc. I was once in a meeting with people doing an intervention to increase “citizenship.” When asked  if they were going to evaluate their intervention, they said that they weren’t because the things they were impacting “couldn’t be measured.” I breathed an inaudible secret sigh to myself. They needed to come see me, a psychologist, because right off the bat I would have suggested that they assess whether or not people in their intervention register to vote in higher numbers, actually vote, have greater knowledge of the political candidates, volunteer for more activities in the community and donate more often to political causes just as start! If they hired me to help them measure citizenship, I would also comb the literature for existing measures and find one with the best likelihood of tapping into the type of citizenship they were trying to improve in their intervention in addition to having them asses objective behaviors. So please, give a psychologist a call when you want to measure something that “can’t be measured.”

You will want to find psychologists that have experience doing this. I would recommend finding someone with some kind of background in social psychology rather than a clinical psychologist (including neuropsychologists) who tend to rely heavily on standard measures in their work.  Social psychologists do freaky things like measuring “altruism” by looking at whether or not students at a theological seminary will stop to help a victim slumped in a doorway if they are in a hurry or had to give a talk to an audience on the “Good Samaritan.” Pretty clever, eh? (turns out it didn’t matter how religious they were or whether or not they were giving a talk on the “Good Samaritan.” They only stopped to help if they weren’t in a hurry. See Darley, J. M., and Batson, C.D., “From Jerusalem to Jericho”: A study of Situational and Dispositional Variables in Helping Behavior”. JPSP.)

5. FOCUS ON OBJECTIVE MEASURES!!! This is the most important tip I can give you. First of all, if you can show that your serious game had an impact on a behavior rather than just something the player SAID it did, you have some pretty damn convincing evidence that your intervention worked. For example, in the Re-Mission study, all of the young patients with cancer in our study reported high levels of adherence to their medication at every assessment time point regardless of whether or not they played the game designed to improve their adherence. However, computerized chips on their bottles of antibiotics and regular blood draws sent to labs to measure the amount of chemotherapy in their blood told a different story. These objective measures showed that patients who played Re-Mission were more adherent than patients in the control group. The Re-Mission players took more of their antibiotics as prescribed as indicated by computer chip records of when and how often they opened their pill bottles. The lab reports of blood levels of oral chemotherapy  showed that Re-Mission players maintained higher levels of chemotherapy metabolites in their blood than patients who played a commercial game. Nobody ever argued that we should have relied on patient self-report of adherence over these measures. In fact, if we didn’t have these objective measures in our study, we would have not only concluded that Re-Mission didn’t work, but that it was an unnecessary intervention because the patients already had high levels of adherence. Let this be a lesson as you plan your outcome evaluations!

During game development in a rapid prototyping process…..

6.  PLAY THE SERIOUS GAME! This sounds obvious to many but in reality, I have worked on many projects where content experts and even people working on the game aren’t even sitting down to play the game. So do it. When you play the game, notice any ways you might think about things differently or new insights you might be having into problems or situation. You might even be feeling a bit different in terms of your emotional state. Then after you play the game, notice anything you might be doing, thinking or feeling differently that you think is related to playing the game.

This is a start for finding or even creating measures to tap into the powerful things your serious game might be doing. This is also a start at going beyond traditional ways of measuring outcomes.

And by the way, if you are a researcher in charge of evaluating the game, you really shouldn’t skip this part. Because no matter what you planned to do with your game, there will be things your game will do better than others and not everything you planned to have in the game will make it in there. Accept it, adjust your expectations and revise your measures accordingly.

Also during game development….

7. CLOSELY OBSERVE AND INVESTIGATE WHAT MEMBERS OF THE TARGET AUDIENCE ARE DOING, FEELING AND GETTING FROM THE GAME PROTOTYPES.

If you read my top 10 tips for making a successful serious game, you saw this one coming from a mile away! A rapid prototyping process is helpful to use in making serious games in part so that key members of your target audience can give feedback. I also like to include members of the target audience on the QA team so I can also get their reactions to the game beyond clearing up any technical problems. They will give you clues into any potentially powerful things your game is doing to and for them if you watch and listen carefully. This might also give you some ideas for how you could do an objective assessment of behaviors that change from playing the game, rather than just assessing what the participants are telling you has changed after playing the game.

This being said, be careful not to be fooled into thinking that what people do when they are playing a game is an “outcome.” For example, playing a game about brushing your teeth regularly does not necessarily mean that the person is more motivated to brush their teeth. It could be a clue that they are. But the ultimate outcome of your game you will want to measure is a change in their tooth brushing behaviors. On the other hand, if people are getting more exercise as a result of playing your game, this actually could be an outcome you are interested in assessing. Just be careful when you are observing people playing your game to be clear about the outcome you are interested in.

After the game is finished and before you submit your outcomes study protocol for ethics approval….

8. DO A TRIAL RUN AS A PARTICIPANT IN YOUR OWN STUDY. I think this is really important because as researchers, we get insecure and want to measure everything hoping that we find something that is able to tap into what our serious game does. But you will get crap data if your participants find your measures confusing, frustrating and tiring. If you can’t participate in the study you designed, then make changes. Take out non-sense measures and reduce the requirements load on your participants. Pilot testing of your study will also help you do this but I think it’s really important to participate yourself (and of course you wouldn’t use your own data). I also get the people carrying out the study to also  go through the process of being a participant. They will also provide important feedback and they will also gain more empathy of what it is like to be a participant in the study. But I think it is really important for the principal investigators of trials do be in the studies themselves. It is leading by doing and makes for a better research project all around.

By now you should have a good idea of how to come up with good measures for assessing the impact your serious game may have on outcomes. You know it’s a good idea to come up with measures at the very beginning of the project, find standard measures, develop your own measures when needed, see a psychologist for help (with developing and identifying measures), focus on objective measures, play the serious game yourself and watch members of the target group playing it for further insight into game effects that could be measured, and be a participant in the study you designed.

I look forward to seeing your publications in the literature. Good luck!

I keep in touch with Richard Tate who is the Director of Communications at HopeLab, the non-profit company I co-founded with Pam Omidyar back in 2001. When I expressed my amazement not just at Re-Mission’s continued popularity as a serious game but the media’s continued fascination with it, he said, “Yes. That story has legs!”  In journalism when a story has legs, it lasts for a long time. The story about Re-Mission sure does have legs!

I recently did a video interview with Ben Rooney of the Wall Street Journal. In this you can hear why I think Re-Mission has been so successful for so long. (Click here to link to the story and video.)

In summary, I think it is a combination of three factors.

#1 The high production factors of the game which means not just the artistic value but the design approach that combined serious learning and behavior goals with fun gameplay. It is by now an old PC game and the controls aren’t perfect but you do feel immersed inside the human body and engaged in the story of Roxxi the nanobot fighting cancer in young patients, which brings me to the second point…

#2 The concept is brilliant. Too many people mistakenly think I came up with the concept but I didn’t. It was Pam Omidyar’s dream to make a game for kids with cancer where you go inside the body and engage in the fight against cancer by shooting cancer cells. Pam O.’s goal was to engage kids in an epic battle against cancer in the game. Epic games are what Jane McGonigal says engage people on a large scale and help them feel part of something bigger than themselves. It’s almost a spiritual experience. I think Re-Mission does that. 

#3 As I say in the video, Re-Mission is still popular and people are talking about it because we did research on it. This made the game much more than an amusing and entertaining distraction for kids with cancer. The research findings confirmed that this game was actually a powerful tool in the fight against cancer. The game “worked.” Kids who played the game took more of their antibiotics and oral chemotherapy as prescribed by their doctors than kids who just played a control game. I don’t think anyone would be talking to us today if we didn’t have these research findings. It wasn’t just a fun game where you shoot cancer cells. It was something that led you to do things in the real world like take your medication more regularly so that you could actually beat YOUR cancer, not just the cancer of a character in a game. 

Overall, I think we need to make really great serious games that combine our serious goals with fun and we need to make sure we do good research on them.

It’s not easy but it can be done. And we can change the world with serious games. That is OUR epic battle!