Spanish Version: Download here (bajar aquí)
This article speaks about the extended development of our methodology to identify and to train Soft Skills through commercial video games. This detailed methodology comes from bibliography analysis about the difficulties not only to measure but also to definy and improve Complex Problem Solving (CPS) and how is possible to identify this important Soft Skills using Botanicula Game.
We also like to talk about why we decided to use this particular video game instead others and our effort to identify actions and events within the game that allows us to provide a minimum guarantee of this soft skill acquisition playing this game.
About xBadges Project
For whom still don’t know what xBadges is, just saying is our research project (AEESD TSI-100600-2015-30) for identification and training of Soft Skills through commercial video games playing. This is what we like to call “Reversed Gamification”. We started from the (already tested) premise of any game is useful to train a huge number of soft skills in some way, we think in fact that is not needed (but too expensive) to produce specific (serious) games for specific purposes when video games market is already overflowing of them.
The main gap we had was to figure out how to track what players are doing within the game while playing in order to know both what skills they are training and which they already have.
We used embedded scripts code in HTML5 classic games (PACMAN, Tetris and Flappy Bird) in our first experiment, this approach allows us to track the interactions and timing of the players, however to our disappointment, we realized that modern commercial video games are not able to be embedded with our code, at that very moment a bulb lit up…
In certain video games online stores there is a way to know what players are doing within their favorite games, this piece of information is called “badge” or “achievement”. We realized that such information is verifiable in real time through API (Application Programming Interface) request, allowing us to track the interaction in that way.
Despite of embedding code to know what skills is being developed we were forced to do the opposite, this is, to know which interactions are needed to win one particular achievement and what that means in terms of Skills acquisition. That was how we started tracking the achievements of certain games from Steam and feeding a Soft Skills-Achievements database.
“Complex Problem-Solving-a field in search of a definition?” a paper by some authors from the Institute of Cognitive Science of the University of Colorado and the Department of Experimental Psychology of Universidad de Granada is the most relevant paper we have used to link CPS to a particular video game (Botanicula).
The cited paper mentions that CPS can be considered when 3 specific features are present:
1- Dynamic, because early actions determine the environment in which subsequent decisions must be made and features of the task environment may change independently of the solver’s actions;
2- Time-dependent, because decisions must be made at the correct moment in relation to environmental demands; and
3- Complex, in the sense that most variables are not related to each other in a one-to-one manner.
The paper insists “in current society, humans spend most of their time interacting with complex systems and more knowledge about this interaction is demanded, makes their arguments inherently interesting and relevant”. In this regard we can say ICT organizations must dealing constantly with complex problems, so we deduce that both ICT Society and CPS Society are barely the same.
A report from the World Economic Forum called “The Future of Jobs”, supports this idea about the importance of CPS Soft Skills. In this report executives from more than 350 employers across nine industries in 15 of the world’s largest economies have been asked to uncover the top 10 skill, respondents said Soft Skills will be most in demand by 2020. Whilst the list includes skills like cognitive flexibility and negotiation skills, the one in poll position was the ability to problem solve [CPS], with 36% of all jobs across all industries expected to require complex problem-solving abilities as a core skill by 2020 (McCormack, 2016).
How is CPS skill measured?
After the explanation of what is CPS and the importance of training this Skills we would like to explain what elements allow us to measure CPS Soft Skill. In Quesada et al. paper some blocks that help us to know how complex is a problem are identified as following:
- Time Related.
- Variable Related.
- System Behavior Related.
- Psychological Description.
Trying not to overextend it too much, we would like to explain each one a bit:
- Time Variant.
- Static Systems.
- Continuous Time.
- Discrete Time.
- Number and Type of Variables.
- Linear Relationships.
- Non-Linear Relationships.
SYSTEM BEHAVIOR RELATED
- Opaque Feedback.
- Transparent Feedback.
- Stochastic System.
- Deterministic System.
- Delayed Feedback.
- Immediate Feedback.
- Expert Model.
- Generalist Model.
- Ill-defined Problem.
- Well-defined Problem.
We think is better just to enunciate each attribute instead of explaining one by one in detail, avoiding, in this way, falling into a tedious article. On the other hand it is interesting for us to keep clear the basic structure we are based on to know which actions or achievements within a game are adding weight to our argument of CPS acquisition.
Botanicula is a very beautiful and cute independent point’n’click exploration video game created by Amanita Design Studio, they are also authors of other very successfully games as Samorost and Machinarium. Besides the very peculiar graphics, it has got an exceptional and ‘organic’ soundtrack. We play this game by controlling 5 ‘botanic’ creatures, exploring the world and solving countless puzzles through them, also gathering some objects to be combined with the environment and going forward in that way.
There are a lot of treasures hidden in this game, also there are a lot of interaction components which do not affect the development of gameplay but they introduce a lot of diversity and abundant diversity. The time needed to complete every level and beat the game is about 3 hours for expert players.
Botanícula has 43 Achievements and 6 different Levels approximately. Level 4 have achievements the most, with up to 14. The names of the achievements correspond to fantastics beings that inhabit this world as “Zaba” and “Mura” and are represented by virtual cards with the image of them, you obtain a new card (achievement) every time you solve a puzzle.
This video game has not been chosen by chance, the decision is based on player profiling. In comparison with our previous options (CS:GO and TF2) we have chosen a very easy-to-play game for casual gamers. This allows us to include in the experience almost any gamer with very basic knowledge about the use of physical interfaces, indeed it is enough to know how to use the mouse.
Botanicula versus CPS
And now… the most complicated part, this is, to analyze achievement by achievement related with a number of values that we have included in each element (that we internally call vectors) which tell us the CPS level in each achievement. In this way we add or subtract a specific number of points in order to evaluate the CPS level of training by player in Botanicula.
Both playing meticulously and writing a qualitative definition of every puzzle were needed to get into our own vector matrix of CPS elements afterward. ie: “Hopik Achievement: You get this achievement after moving mouse cursor and pushing over some red objects, the achievement is activated after putting one of these red elements into a funnel”.
Maybe is easier to understand the process explaining it in a narrative way: We know that there is transparent feedback in a puzzle/problem (ie. Jepice Achievement), this reduces complexity in the problem, hence that is a -1 in our scheme of complexity values. By other hand there is an opaque feedback in Mesto Achievement, a huge quantity and type of variables, hence our internal vectors indicate there is +4 balance in this case.
Thanks to this numeric evaluation we are able to achieve two different goals:
- To evaluate qualitative and quantitative each achievement.
- To translate each value to the algorithm which allow us to know the level of Soft Skill acquisition.
Initially we check every game using psychological standard tests availables, this is, taking into account an external evaluation as reference that is useful to establish controls. And here is where a surprise comes…
In general, the standard tests to measure CPS levels are used to be something similar to a kind of mini-games (sometimes called micro-world), which are easier to play than most of the commercial video games we usually like to play in our PCs or gaming consoles. We don’t know at a high level of detail how are all of these mini-games used to test one subject’s CPS skill but, reading on the descriptions of Moro, Lohhausen or FireChief (just giving some examples) we have realized that games as Civilization would be able to be catalogued as both very complex and complete “standar test” not just for CPS evaluation but a kind of SCPS one (Super Computer Problem Solving) according with the super-complexity of its gameplay.
Under our humble opinion this difference on complexity between available standard tests and actual commercial video games we use drives us to the conclusion that there is a new generation of gamers who have a very high level of CPS skill. According to the available information of the number of hours an average teenager is spending in resolving very complex puzzles and problems playing commercial video games (up to 25h approx), we are barely sure there is a new profile with a lot of CPS skill arising.
Our methodology is trying to be as quantitative as possible for sure, even then the research topic is not able to be qualified with a 100% of accuracy. Moreover there is no any kind of soft skills leveling standar and even more each country has its own definition of them. According to all of that, on one hand we are making our own Soft Skills levelling, and on the other we are mining stats data in order to establish more accurate soft skills-achievements relationships.
As a summary to conclude:
- We are increasingly convinced of commercial video games capability to develop soft skills in gamers.
- We are still involved in tuning up our methodology, the same that allow us to track achievements in order to measure a specific Soft Skill level.
- We have noticed that standar tests used by professionals and researchers on psychology are lagging behind if compared to their cousins from digital leisure platforms, they are: video games.
- As much information we are able to collect, more we will be able to improve our methodology accuracy, hence, increasing our confidence of playing particular video games help us to be better professionals.
- J. Quesada , W. Kintsch & E. Gomez (2005) Complex problem-solving: a field in search of a definition?, Theoretical Issues in Ergonomics Science, 6:1, 5-33, DOI: 10.1080/14639220512331311553
To link to this article: http://dx.doi.org/10.1080/14639220512331311553
- McCormack (2016). Why Complex Problem Solving Will Be The Skill Most Employers Want By 2020. https://www.hrzone.com/community/blogs/mark-mccormack/why-complex-problem-solving-will-be-the-skill-most-employers-want-by