So you mean something else when you say your serious game has been “validated”? Confusion Part 2

In my previous post on validating serious games (“What do you mean when you say your serious game has been validated? Experimental vs. Test Validity“) I tried to clear up some confusion around what it means when someone says their game has been “validated.” I called for people to specify whether or not their game had undergone a validation process as an intervention or as an assessment measure. Seems that there is even more confusion out there to clear up.

David Crookall, Ph.D. read my post and pointed out two things. 1) That there is another meaning of  serious game “validation” that is contributing to confusion around the term and 2) that a distinction should be made between “validation” as a process and “valid” as a label. Prof. Crookall knows what he is talking about. He is the Editor-in-Chief of  Simulation & Gaming: An Interdisciplinary Journal and is on the faculty of Université de Nice Sophia Antipolis, France.

Can I say that Prof. Crookall’s points are valid?

Yet another meaning of validating a serious game

Prof. Crookall pointed out that there have been several papers published on validating serious games that focus exclusively on the process of software validation. In these cases, the validation of the serious games  involved a process of quality testing the software.

I confess that I knew about the use of the term validation to address quality testing of software programs in the industry. But I thought I didn’t have to discuss this in my last post because surely people at conferences weren’t referring to this process when they said their game was “validated.” I read some articles that Prof. Crookall sent me and realized I was wrong! There have been several excellent papers written on validating serious games that focus exclusively on quality assurance and beta testing processes. It is therefore highly likely that people had this meaning in mind when they said their serious game had been “validated.”

This revelation left me feeling even more frustrated. This means that when people go around saying they “validated” their game they may also mean that their game underwent quality assurance processes. So now it doesn’t just mean that their game is “valid” as an assessment tool or is “valid” as an effective intervention, it can also be “valid” as a software program that will technically work and be accepted by the target users. I have a sneaking suspicion that it won’t stop here. *Sigh*

The importance of validation through quality assurance processes

Please let me clarify. My frustration in no way means that I think that quality assurance processes are not important when it comes to serious games delivered as digital content. I recall that at a Games for Health Europe Conference presentation a few years ago, Tim Laning, of Grendel Games, pointed out that most serious games do not undergo a thorough process of quality assurance testing that is standard in the software industry. He called for more serious games to undergo this industry standard process (among other things) in order to advance the overall success of serious games. I heartily agree! The quality of most serious games has massive room for improvement. Also, serious games ultimately won’t “work” (experimental validity) if they crash, contain errors, and are frustrating to play.

Quality assurance IS a validation process

It does make sense to call the quality assurance process a validation process. The quality assurance process has many things in common with the goals of Experimental and Test Validity. For example, among other things, the process helps developers and stakeholders in the serious game industry determine whether or not the software does what it says it does (see internal validity), will be accepted by the target audience (see external validity), and contains the contents required in the program (see content validity).

Suggested reading on quality assurance processes for serious games

If you would like to learn more about how the software validation process can be applied to serious games, I would recommend reading “The Validity and Effectiveness of a Business Game Beta Test” by Gold and Wolfe (2011) published in Simulation and Gaming. It provides a really nice detailed description of the process along with a presentation of the results of the quality assurance process they undertook with their business game. I was quite impressed with the article and I think it would be very helpful for people who plan to pursue this process (that should be standard in making serious games!).

Another important distinction: “Validation” as a process versus “valid” as a label

There is something else that David Crookall brought up. We need to separate the “validation process” in serious games from labeling something as being in a state of being “valid.” Confusion arises when we talk about something being “validated.” When someone says that their serious game has been validated, I bet that most of us assume that the game underwent a process of validation for X, Y, or Z AND (there is a big emphasis on this logical operator “AND”) that the serious game was determined to be valid.

No matter what kind of validation we are taking about, any of us can take a serious game through a validation process for a particular purpose and end up finding that our game is not valid for that purpose.

Example 1:  I can take a game, put it through a randomized trial to determine its experimental validity as an intervention to impact outcomes, and find that the serious game had no effect on outcomes.

Example 2: I can take another serious game and do some validation studies consisting of focus groups and correlational studies and find that performance on the game is not related to performance in the real world, it is not correlated with any similar game measures, and experts don’t even think that the game contains the necessary content to train and educate players.

Example 3: I can make another serious game that I put through quality assurance testing but I have failed to be able to resolve major bugs with the program, users find it completely frustrating to use, and I ran out of money to fulfill all of the content requirements for the game.

In all of the above three examples, the serious games were “validated,” in the sense that they went through a validation process. But none of these games would likely be considered to be “valid” for the purposes they were being tested.

It would therefore not only be helpful for people to be very clear about what they validated their games for, but people should try to be clear about what the results of the validation process were.

Communication

The business of making serious games is inherently interdisciplinary. People from different cultures (software, research, education, arts), backgrounds, and languages come together to make games that educate and train in addition to being entertaining. Mutual understanding and clear communication are inherent challenges. Luckily, there are ways to address them.

Suggested behaviors:

When we talk about our serious games, we can begin by completing these phrases appropriately,

1) “My serious game underwent a validation process for _______.”

2) “We found that the serious game was valid for______.”

And if we hear someone say that their game has been validated, we can ask,

1) “What did you validate your game for?”and

2)  “What did you find that your serious game was valid for?”

Seems easy enough. Let’s see if these simple behaviors can clear up some of the confusion.

Conclusion

To sum things up, my frustration is that people talk about validating games and valid games without clarifying what they are validated for. I am further frustrated that “validation” and “validity” can mean such different things and processes. And to be quite honest, mixed in with the frustration is some embarrassment that I had assumed it meant one thing when I have very slowly over time learned it meant so many other things (with a power-up from David Crookall).

So readers, I would like to publicly apologize for my previous talks and papers in which I assumed that a validation process for a game meant an outcomes trial and that a game was valid if the trial showed was effective in addressing the outcomes intended. I was guilty of not being specific and of assuming you knew what I was talking about. I was contributing to the problem. My efforts now focus on trying to be a part of the solution.

Time to practice good communication skills.