Softskills.games in the Master in Financial and Management at UPF

Authors: Sergio Alloza
Reviewed by: Dr. Flavio Escribano

Spanish Version:Download here (bajar aquí)

Introduction

After watching the project presentation video, you can imagine what is this article about. Here  we are to share with you some details of the project we have carried out a few months ago in collaboration with meHRs at the UPF.

The collaboration between both organizations consisted in a program of both evaluation and training of certain soft skills identified as key by the UPF máster staff. In this experience, the main goal was students to play the video games that were recommended by us to demonstrate that they can effectively improve not only the target demanded skills but others, thus establishing one more experience to refine and strengthen the methodological validity of the Soft Skills Games project.

This project was framed within the Master of Financial and Finance Management of the Pompeu Fabra University (UPF) of the 2018/2019 academic year for the identification of certain soft skills in students through the use of coaching techniques (by Sergio Carmona, Director of the Training and Development Area of meHRs) and the soft skills evaluation platform with commercial video games (softskills.games).

Methodology

Next we will show you the methodology that was carried out by us during the project, this includes each one of the relevant aspects, such as the key skills identified, the usual procedure for its evaluation and training and the tools that were used for it.

Soft Skills

In relation to which skills were trained, requested by UPF, the project was focused on evaluating and training the following competency, that is, a package of specific skills:

Plasticity

Within the Soft Skills Games platform and after analyzing it with our methodology, the Plasticity competition was broken down into the following soft skills:

• Cognitive Flexibility. Ability to generate or use a different set of rules to combine or to group elements or actions in a different way than usual.
• Judgement and Decision Making. To consider the relative costs and benefits of the potentially most optimal action within a broad spectrum of choices

Within each skill trained we can observe some other skills that also came into play, such as Goal Setting, Creativity as a pillar of Cognitive Flexibility and Critical Thinking within Decision Making.

As you can see, defining a skill is not a simple action but something that is not standardized or normalized in the scientific community (in addition to the added problem of taxonomy variability according to different countries and even to local clients). For this particular pilot, we wanted to “keep it simple” to avoid possible complications due to the nature of the skills, so we were focused on measuring only a small set of them.

It should be noticed that this is still ongoing research so the description of the skills and the breakdown is in the process of continuous adjustment for greater accuracy in the model thanks to the acquisition of new data and the benchmarking processes that we performed.

Sample

The experience was initially carried out with a group of 16 students of the UPF Master in Financial and Management.

Of these 16 students, the majority of users were casual gamers (players of games as Candy Crush, Sims, etc.) and there were two high gaming profiles precisely on Steam (which was reflected in their graphics). Due to technical issues related to Steam, there was an user whose profile was not set as public and their data could not be accessed, so there was a final sample of 15 subjects.

Procedure

The structure of the experience was designed as follows:

• Pre-activity measurement (Pre test). Where several standardized psychological tests have been used to check the previous level (baseline) of the students in the two skills to train (test showed below).
• Interaction with Soft Skills Games platform. With a weekly minimum of 3.3 hours of play, the total count of hours in the experience amounted to 20 being 6 weeks of play available to users. Specifically, they were able to start playing from the beginning of November 2018 and the project was completed on December 14, 2018 with a closing session. To decide what amount of hours should the users play we based on a review of some of the studies that use video games to enhance some aspects of the human mind hence we established a play-time according to what we found in those studies considered as normal and sufficient amount of time to obtain significant data (Quiroga et al., 2016 & Oei & Patterson, 2013).
• Data presentation and closing dynamic. With the individual reports generated and all the technical or methodological problems solved, the students were presented with their own progress in a face-to-face session along with a dynamic in the form of a game so that the students can see the relevance of the skills trained and how video game training helps to improve these skills.
  • Closing Dynamic: with 15 users, the group was divided into 3 teams. A team that was isolated in another room with their eyes covered, another team that was left thinking about how to make the “blind” team solve a task and another team of observers. This division of teams was not random, because the students were distributed according to their results in softskills.games to then be able to link their behavior in the dynamics to the games they have played. Specifically, users with high scores in Goal Setting and bad or low People/Team Management were assigned to the team of leaders, in order to see their shortcomings when managing people while focusing on the task. On the other hand, users with less data (because they started playing late, among other reasons) were assigned to the group of observers. After 70 minutes of dynamics, the individual reports were distributed and individual action plans were drawn up.

• Data Benchmark (Post test). After the Play-time and the closing dynamic, the students had to fill the questionnaires once again, the same that they had filled at the beginning of the experience in order to compare the data and let us check if there have been any changes according to these indicators as well. Additionally, a satisfaction survey was sent to them to provide feedback and their opinion about the project.

Measuring tools

Standardized tests (pre and post tests)

• Cognitive Flexibility. For the evaluation of this soft skill we have used the I-ADAPT-M instrument (Reduced version of 47 items, translated and without the “physical” subscale). Based on the study conducted by Ployhart & Bliese (2006) where they use this test to measure “Adaptability”. This test tries to measure the individual adaptability with its questions distributed in subscales (cultural, interpersonal, etc.). Since we wanted to focus on measuring the adaptability or flexibility of each individual only in psychic aspects, we excluded the physical subscale.
• Judgement and Decision Making. For the evaluation of this soft skill we have used the Life Skills Development Scale for Adolescents instrument (Darden et al., 1996) (modified version for adults and translated from English). We emphasize that this specific soft skill can be much more complex than others, since decision-making can be influenced by other soft skills at the same time. However, with the aforementioned desire to “keep it simple” we wanted to focus on using a specific measurement instrument with a Decision Making subscale.

For the completion of the questionnaires, the users were sent a digital version of them in Google Form format, which allowed us to collect the answers in an agile and comfortable way.

We will not discuss the feasibility of measuring soft skills through questionnaires from the 90s, where many factors intervene, among them the Hawthorne effect (among others, McCarney et al., 2007), which implies that the knowledge on the part of the respondent that is being evaluated will modify the result, generating a bias. This topic could be discussed on an entire article but for now we will plant the seed of doubt here, since it is much more effective for us to measure a behavior simulating and observing than not asking about it, which leads us to the next point.

The selected video games

A series of video games were initially proposed to train the commented skills (some of them oriented to hardcore gamers profiles). However, due to factors outside the experiment (mainly budget) and to improve the users’ motivation to play, finally it was decided to use only 2 simple video games oriented to casual gamer profiles (among other things because the majority profile of the sample was casual):

Fallout Shelter (Bethesda Softworks, 2015).
https://softskills.games/en/game/fallout-shelter-588430/

2D simulator of a refuge where the player has to manage both resources and workers to ensure the proper functioning and progress of the base as well as avoid or mitigate possible accidents or external attacks. The skills that stand out, among others, in this video game are the following:

• Creativity. Since the game allows us to create the refuge at our whim.
• Self Discipline/Perseverance. Shown above all the more you play the game.
• Management of Material Resources. Main mechanics where you have to optimize the position of each worker according to their statistics and take into account the global resources of the refuge.
• Cognitive Flexibility. For the rapidity of reaction when responding to unexpected events or readjusting the system with the integration of a new worker.

Gems of War (Infinite Interactive, 2014).
https://softskills.games/en/game/gems-of-war-329110/

Match3 game with turn-based mechanic, levels and collection of cards as characters. Players have to match pieces of the same color taking into account the color they need to load attacks. The skills that stand out, among others, in this video game are the following:

• Self Discipline/Perseverance. Again taking as indicator the playing time, the number of cards collected, games played, etc.
• Spatial Scanning. Main mechanics of match3 genre where the user has to scan the screen in search of pieces to put them together.
• Goal Setting. Because of the complexity of the game, putting pieces together has to have a meaning, in this case, a goal (to charge attacks or prevent them from attacking you).
• Judgment and Decision Making. Considering each action as a decision, being a turn-based combat where each action means defeat or victory.

Results

Play time and distribution

The group of students played an average of 17.46 hours with a standard deviation of 16.64. A standard deviation almost equal to the average indicates that the group is very distributed, with 5 users above the minimum recommendation (20 hours) and 5 users below the half (10 hours).

Not all users have played both games. In fact, the distribution is as follows:

• Users who played Fallout Shelter: 6
• Users who played Gems of War: 9

The following table shows the dedication of each user in hours for each game:

SSG Group data

For the elaboration of the group graph, a sum of the most boosted skills for each user has been made, independently of the game that they choose to play and the hours dedicated:

Thus, in the following graph, the following skills are shown by the number of subjects who enhanced them:

• People/Team Management = 8 users
• Management of Material Resources = 5 users
• Self Discipline/Perseverance = 12 users
• Creativity = 5 users
• Spatial Scanning = 10 users
• Problem Sensitivity = 4 users
• Judgement and Decision Making = 9 users
• Goal Setting = 9 users

As mentioned, the main objective of the project was to enhance the “Plasticity” competence. The skills highlighted by the group are related to the processes of Decision Making, Management of Resources and People and the effective execution of previously organized strategies, as well as sufficient Cognitive Flexibility to be adaptable to new situations using some others skills such as Creativity, Sensitivity to the Problems and Goal Setting.

SSG individual data

Here the individual data of each user is exposed, with the amount of hours that they played (and its relationship to the obtained experience). Note that the trained skills are not the same in one group of users as in the other. While the Fallout Shelter group has trained both more management and analysis skills, the Gems of War group has improved skills more related to tracking, logic and Spatial Scanning. These differences are normal since the games are different and therefore have different mechanics and rules that also influence certain soft skills.

It also highlights the absence of data from certain users due to technical problems when synchronizing the mobile game with Steam, resulting in hours played but not achievements (or partially) obtained. As it is the case of the last two users (14 and 15, exposed in the 2nd table) that despite having more hours of playing than some users, they have less experience acquired).

Data (XP * per skill) of Fallout Shelter players:
*experience points

Data (XP * per skill) of Gems of War players:
*experience points

Standardized tests (pre and post tests)

• Cognitive Flexibility — I- ADAPT-M.
  • Pre test: 11 Users
  • Post test: 1 User

Only one user answered the test a second time, giving a positive difference (+14) in the questionnaire scores, notice that this user improved Cognitive Flexibility skill. In the absence of more data to evaluate this difference at a statistical level, it is inferred that a change of 14 points on a scale of more than 200 is relatively relevant.

If we compare this data with the data provided by the platform, we observe that this user has played 11.6 hours to Fallout Shelter and has improved 200 xp points in Cognitive Flexibility (compared to the total of 1000 xp), which correspond to +14 test points standard. According to the standard test we could establish the following relationship: this player has played 11.6 hours and has improved his Cognitive Flexibility ability by 7% (according to the test) and by 20% (according to softskills.games).

If we had had more data from other users it would be interesting to be able to benchmark some of them to clarify better some other hypotheses, such as where is this user located at the distribution level (if he is a good player and with 11.6 hours it is impossible to improve more the skill or if there are better users who have improved more skill percentage with the same time).

• Judgement and Decision Making — Life Skills Development Scale
  • Pre test: 8 Users
  • Post test: 1 User

Only one user has answered the test a second time, giving us a positive difference (+18) in the score of the questionnaire, notice that this user has improved in Judgment and Decision Making. In the absence of more data to evaluate this difference at the statistical level, it is inferred that a change of 18 on a scale of 45 is quite enough relevant.

Again, if we compare this data with the data provided by the platform, we observe that this user has played 10.8 hours to Gems of War and has improved 501 points in Judgment and Decision Making (with respect to the total of 1500), which correspond at +18 points of the standard test. According to the standard test we could then establish the following relationship: this player has played 10.8 hours and has improved his Judgment and Decision Making ability by 40% (according to the test) and 33.4% (according to softskills.games).

As with the previous case, if we had had more data from other users here, it would also be interesting to be able to benchmark this data to clarify the aforementioned hypotheses.

In any case both results match a lot, since thanks to both indicators (test and game experience) the skills improve in a limited way in the first case (7% – 20%) and regular in the second (40% – 33.4%) we could say that there is some accuracy in the measurement of skills by the platform.

At this point we are struck by the discrepancy between percentage improvements according to the different indicators. We did not expect a 100% synchronization and we accept the differences exposed as normal, since we are facing the measurement of the same skill with different instruments, some more reliable than others (and yet the percentages do not diverge so much), and the lack of information since only 2 people generated data with the questionnaires.

Satisfaction survey

Of the total number of students who participated in the experience, only two people answered the satisfaction questionnaire. The questions were assessed on a Likert scale from 1 to 5, with 1 being the lowest or negative score and 5 the highest or positive. Since the data received are so few, they lack statistical significance, but nevertheless they are useful to be able to infer certain aspects of the project that we will see in the next section.

Discussion

Overall, the experience has been well perceived and valued by the students. The following points stand out:

Standardized tests

Very few people have answered the questionnaires that were presented at the beginning and end of the experience, but there were people who completed the questionnaires the first time (11 and 8 people) but nevertheless the second time, when completing the same questionnaires, only 2 people did it all. For what it is inferred that they are heavy questionnaires to be completed, they are not given importance or simply that the users were not motivated to complete them (which is also understandable due to the boring nature of the tests in front of video games as a training tool). Alternatives for the future are proposed such as presenting the questionnaires in the face-to-face session of the project, to be sure of the data collection, and in the same way at the end of the experience in the closing session. So at least we will collect enough data to carry out a statistical analysis and better contrast our hypotheses despite the fact questionnaires are BORING!.

SSG data

In short, the group played less than we would have liked except for some users who played much more, this entails a very limited data generation and data collection that was reflected in simple graphics like the following:

However, the 5 students who played more than 20 hours had more data and richer graphics, such as the following:

This difference between graphics was highlighted by the students in the face-to-face session, so they realized that if they played more hours they would have more complete and meaningful graphics. For future applications, one solution is to make a control session in the middle of the experience to save potential problems in person and avoid the late incorporation of users who had technical problems.

Another aspect to comment on the data of softskills.games is the diversity of profiles. Although the students have been able to play the same game, there has been variety in the profiles, highlighting some students in skills different to others. This can be seen clearly in the profiles with few hours of play, which got different achievements to the rest because they performed different actions. You can also see in the two profiles that already had Steam and, despite having so many hours of play, they improve skills that had nothing to do with the game they have played (such as they have great levels of Time Management and Coordination with Others for having played shooters, but had barely developed skills of Self Discipline or Monitoring Self and Others).

Also comment on the issue of hours of play and achievements. As a general rule, the hypothesis is fulfilled that the more a user plays, the more achievements he gets and therefore the more skills he trains. However you can see interesting exceptions that show the different levels among the students, having students with a similar amount of hours, have more achievements and more experience in the same skills, standing out from their teammates.

Regarding the data, finally notice that the students in the face-to-face session presented some doubts about some soft skills definition. Specifically, four students asked what the following soft skills meant: Spatial Scanning, Cognitive Flexibility and Goal Setting. From what is extracted it is important to improve the explanation and definition of each skill with examples so that users understand at all times what they are training and what it is for. In fact we also anticipate that in the coming months we want to add an explanatory section on the web related to the skills of our catalog.

Satisfaction survey

The few data received indicate that the sample is satisfied with the project, valuing the experience in a 4 out of 5. With the precarious coordination between the actors involved it is understood as the perception about the organization of the project has been average. However, we found a difference of opinions regarding the duration of the project, with students who preferred to continue playing and others who preferred not to have had to play in the exam period, confirming the initial suspicion that we had relating the period in which the project was applied and its influence on the behavior of users.

Similarly, the evaluation of the manuals of the platform (games and platform tutorials that we made available to users), were both well valued and poorly valued by the sample. Here we have a difficulty in interpreting this data, since the negative evaluation of the usefulness of a manual may be because the user has actually consulted and it has not been useful, or simply the user has not had the need to look at it.

We find ourselves with a dichotomy related to the schedule and the number of hours, while a user prefers to play when she wants and believes that the number of minimum hours is few, another would have preferred to have a schedule and have fewer hours to dedicate to the project.

In general, students have not noticed many changes in their behavior or abilities in a normal context, but they do agree that they would notice changes if they had continued training, emphasizing the need to establish a program with specific milestones.

The attention received from the softskills.games team as well as the web, the SSG graphics and the ease of connectivity between SSG and Steam accounts have been valued in a medium-high manner, while the face-to-face closing session was perceived as moderately-unhelpful.

The students do not manifest any desire to change any aspect of the softskills.games website, although one does highlight that they would like to work better on an individual plan, that is to say, once they have evaluated their skills with the project, continue training with the platform and other complementary tools.

*Being an anonymous survey we do not have the data to contrast but the internal hypothesis is that those users who have valued the project best and wanted to continue playing were the gamers users who obtained the best results, while the users with the most technical problems and methodology are the ones who have rated the experience the worst.

Technical problems

Within this category, the following problems that emerged during the experience are highlighted:

• Gameplay on Mac. Many of the users have Mac computers and did not have access to a Windows PC with Steam installed, so at first they could not participate in the project (since unfortunately the selected games were not available for Mac). However, an alternative was proposed: play on mobile to either of the two games. This options has also technical problems which are discussed below.

• Mobile data connection with Steam. After playing on mobile and getting the achievements in the PlayStore or AppStore, when synchronizing with Steam (through the manual and instructions provided by the softskills.games team) it should be shared both the game hours and mobile achievements to Steam. The problem was that this communication is not 100% effective, being valid in certain cases and not in others, so there were certain users who could not obtain their real data due to this problem that we can not take charge since it depends solely and exclusively of technical issues between two companies outside the participants in the project.
• Delay privacy setting. This problem appeared when users created Steam accounts at the beginning of the experience. By default, when a Steam account is created, the profile is set as private, so we can not access the data it generates while playing. The solution is to configure the profile in public mode manually, which all students did with the help of the manual and the instructions. However, the problem appears when manually putting a profile in public, it seems that this update is not in real time. This time is variable, since there were cases in which within a few hours the data could be accessed while in others days passed and data access still appeared as private.

In conclusion, in spite of the technical problems and the fact that we have to keep a close watch on some students due to their limited commitment, the experience is valued as very positive, because it has served us as learning once again in the “real world”.

From the students, although in the satisfaction survey there are not many answers, in the closing session it was possible to collect qualitative information where the users stated that they liked the project a lot, participating in something different and with video games (such as as they comment in the video), besides seeing the relevance of the skills and their transfer to their daily life through the dynamics of the proposed games.

We do not want to end without first mentioning a factor that seems relevant to us. We found a substantial difference between the hardcore gamers and the casual gamer profiles. Being the most achievers gamers, that is, they were more committed to every action demanded within the project, dedicating more hours of playing games and also generating more data, showing more interest and completing all the tasks adjacent to the game playing, such as the completion of the questionnaires. Which leads us to think of two new hypotheses:

1. Do gamers really adhere more to the project because it is more attractive to them and in fact they already have some trained soft skills that are related to persistence and commitment?
2. Could it be that the projects where the user average is a gamer will generate different results to those projects where the average of users is not?

References

• Darden, C. A., Ginter, E. J., and Gazda, G. M. (1996). Life-skills development scale – adolescent form: the theoretical and therapeutic relevance of life-skills. Journal of Mental Health Counseling, 18, pp. 142-163.
• McCarney, R., Warner, J., Iliffe, S., van Haselen, R., Griffin, M. & Fisher, P. (2007). The Hawthorne Effect: a randomised, controlled trial. BMC Medical Research Methodology. 7 (30).
• Oei, A. C. & Patterson, M. D. (2013). Enhancing Cognition with Video Games: A Multiple Game Training Study. PlosOne, 8 (3).
• Ployhart, R, & Bliese, P. D. (2006). Theory: Conceptualizing the Antecedents, Consequences, and Measurement of Individual Differences in Adaptability. Emerald Group Publishing Limited, pp.3 – 39.
• Quiroga, M. A., Román, F. J., De la Fuente, J., Privado, J. & Colom, R. (2016) The Measurement of Intelligence in the XXI Century using Video Games. The Spanish Journal of Psychology, 19 (89).