Gamification: Evidence on its Effectiveness After the Hype

 

 

Author: Jordi Moretón Galí >> @jordimgali

Spanish Version / Versión en Castellano: Descarga

The hype that surrounded Gamification visually materialized in 2011 when it appeared for the first time in Gartner’s Hype Cycle for Emerging Technologies. In 2013 was already in the Peak of Inflated Expectations zone and this year 2014 it’s in the Trough of Disillusionment, a zone that doesn’t really mean anything specially pejorative but rather means for the first time the community acknowledges that Gamification can’t fix all the problems in the world. This “landing” in 2014 implies a period of success only for those solutions that could set in the market (and frustration for the rest) but the relevant fact is that we are entering a period of consideration and deep analysis about the application of Gamification, beyond marketing hype. The following Hype Cycle stages will be –potentially– the Slope of Enlightenment and the Plateau of Productivity, that fundamentally will mean that Gamification as a whole will have found his niche in the market.

One of the hypotheses we at Gecon.es have identified to explain why Gamification is in this stage is the lack of solid experimental evidences about its effectiveness.

Beyond reports and articles that have been popping in –and continue to do so- that advocate the success of Gamification in the following years or that enhance game techniques as powerful motivators in plenty of fields of application, there is a real lack of experimental evidence to support all these claims.

One of the most visited posts in Yu-Kai Chow blog presents us with an exhaustive recollection of Gamification cases with its ROI. It’s an excellent article, no doubt about that, but it’s also a clear signal about a stage that is starting to lag behind, where Gamification “fed the multitude”. This suggests us that there is a problem:

1. the majority of published Gamification cases are successful cases,
2. many of these are published by corporations like consulting firms that, perhaps, don’t have much of a motivation to show bad results and,
3. these don’t explain much about methodologies, encountered problems or limitations.

The kind of documents we need is found near the academia rather than businesses but, be aware!, there is bias also in academic papers. Anyway, academia uses more rigorous methodologies but, because of this, needs more time to publish experimental results that could validate the effectiveness of a theory or technique.

Now is the time these publications are appearing:

The Gamification business grows and there is a consensus that the technique works, but the reality is that it’s difficult to find solid justifications on why Gamification is effective and, moreover, in what circumstances. It’s necessary to connect theories with experiments and publish openly, both successful and unsuccessful results.

On the issue that brings us here today, one of the academic authors we follow closely and read with more attention is Juho Hamari, Jonna Koivisto and their collaborators, from the Game research lab de la Universidad de Tampere. Among others of their relevant papers about experiments and Gamification, we selected two that we believe are key to understand the issue:

Hamari, J., Koivisto, J. & Sarsa, H., 2014. Does Gamification Work? — A Literature Review of Empirical Studies on Gamification. In 2014 47th Hawaii International Conference on System Sciences. Waikoloa, HI: IEEE, pp. 3025–3034.

Hamari, J., Koivisto, J. & Pakkanen, T., 2014. Do Persuasive Technologies Persuade? – A Review of Empirical Studies. In A. Spagnolli, L. Chittaro, & L. Gamberini, eds. Persuasive Technology. Lecture Notes in Computer Science. Cham: Springer International Publishing, pp. 118–136.

In open access here and here (the second article goes beyond Gamification and includes other persuasive techniques; these differ from Gamification by emphasizing the social and communicative persuasion rather than motivation through play experiences, even though both concepts originate from the same place).

Both articles are reviews of other scientific publications about cases of Gamification, filtered by those that include experimental studies and methodology, among other selection criteria. The authors conclude, in both cases, that the analysis of these publications reveals that Gamification generally works and its effects are positive but also warn that this is not entirely conclusive as negative details appear where in general the results are positive, and therefore there are still underlying factors to research, like 1) the gamified context and 2) the qualities of users. Furthermore, the authors identify several methodological flaws in the analyzed experiments, such as too small sample sizes, lack of control groups, short experimental time, Gamification novelty bias or lack of clarity in the results.

As we see, Gamification is a technique that is in the process of maturing and has many challenges ahead. In late 2014 we published a report on the Gamification of 3DWire event and, in the line that we describe in this article, we found several of the challenges Hamari & Co. highlight, many of them could be overcome while others not yet.

In any case, it is clear that now is the time when Gamification starts to be openly accounted. And preliminary results look good!