The "messy" world of video game research: Video Games Industry Memo, 14/03/2024
Tidying up a tricky topic with Professor Pete Etchells
Prof Pete Etchells unlocks the “messiness” of screen and games research
Former Activision head honcho eyes up buying TikTok in the US
Jeff Minter playable documentary leads a quiet week of pre-GDC launches
Good morning all,
Given that the big read is a whopper, I’ll forego my usual jaunty intro to quickly plough through two points of order.
The schedule for the London Developer Conference is popping online shortly and I can say here that it’ll feature speakers from Roblox, Sharkmob, ustwo Games, nDreams, Newzoo, Politico, The BFI, Ukie and more. Grab a ticket to next month’s event here.
I also popped up on the Making Games is Fun Podcast, where I discussed the inspiration for VGIM, why I think video games are more like mini nation states than platforms and how Athletico Madrid coach Diego Simeone looks like a Red Dead Redemption 2 character. Link here.
Anyway, enough of all that - let’s get on with the newsletter.
The big read - The "messy" world of video game research
As many of you know, I’m a mild-mannered diplomatic person who is willing to engage in good faith on just about anything video games related.
But there is one thing that I really struggle to deal with constructively: crap, or misleading, research about the impact of video games on peoples’ lives.
During my time on the industry beat, I’ve watched a report linking video games to gambling get picked up by the national press despite it containing the deeply factually incorrect line of argument that “Fortnite is a loot box.”
I once had to explain to John Pienaar on his Times Radio show why the NHS Gaming Disorder Clinic’s decision to count parents of young people being treated in the clinic as patients in its statistics is probably a little bit unfair if you want to actually understand the prevalence of gaming disorder.
And I can remember fielding some, ahem, interesting questions about whether playing video games really can cause deep vein thrombosis (spoiler alert: they can’t) from one broadsheet journalist who was inspired to cover the story by one particularly shonky and scaremongering piece of research.
Yet even though all of these stories sound ridiculous, I also know they all ended up shaping public perception of games to some extent - despite the clear problems with the research and data underneath them.
But why is research into the impact of games on our lives, and screens more generally, tricky to get right? How do these problems feed into the ways we perceive, interact with and regulate games? And can we do anything to stop knee-jerk reactions towards the medium feeding a cycle of shaky research and bad outcomes?
Pete Etchells is a Professor of Psychology and Science Communication at Bath Spa University. His new book Unlocked: The Real Science Behind Screen Time explores the “messy” science that shapes our perception of screens, the consequences of that messiness and what we need to do to change the way we both research and interact with digital technology.
So I caught up with him to try to get to the bottom of these challenging questions.
Etch-Communication
Pete was inspired to write Unlocked for a similar reason that he wrote his last book Lost in a Good Game: a desire to provide rigour and balance to the wider digital debate.
“The thing that I really care about is science communication,” Pete explained. “I'm interested in trying to reframe some of the conversations around digital technology based on an objective and critical reading of the research literature.”
“Unlocked was born out of the frustrations of seeing a very one sided conversation in “the mainstream media” about screens, social media, smartphones and video games being inherently bad or unwholesome for us. It didn't really feel like there was a counterpoint to that.”
And sitting at the heart of Pete’s argument is what he described as a “messiness” within screen research that means that some of the truisms of our digital lives - such as screens being “addictive” - struggle to stand up to reasonable critical and methodological scrutiny.
I asked him what was the cause of this messiness and he broke it down for me into three interrelated points.
First, the debate around screens and games is shaped by an oxymoronic public perception of them whereby they’re simultaneously both an existential threat and banal - preventing a reasonable assessment of their harm and increasing the perception of risk around them.
“We worry that games are addictive or that they're melting kids’ brains, so we have these really big societal conversations about video games or screen time being bad,” Pete said. “But at the same time we don't treat them with any sort of seriousness.”
The best way to overcome this perception problem is through gathering robust empirical research to test or challenge these ideas. The problem, and the second issue making screen research messy, is that it’s methodologically “tricky” to effectively gather that information because screen use is so embedded into our everyday lives.
“What you're trying to do is capture quite a complex aspect of human behaviour,” says Pete. “And it's really hard to do that in a solid and robust way.”
This can be seen in some of the methodology challenges facing people who work within screen research fields.
Laboratory based experiments, where a hypothesis is tested in a carefully controlled setting, are often the preferred way to conduct scientific tests across research areas. But when it comes to assessing the impact of screens, Pete says that lab tests are poor indicators of actual behaviour because experiments in these conditions are “so detached from the reality” of day-to-day device usage that the data that comes from them is of limited use.
So researchers try to gather data more organically through routes such as self-reported surveys, where respondents monitor and answer questions about their screen usage. This sounds great on paper, but according to Pete’s research across Unlocked - and perhaps unsurprisingly when you think about it - people are much worse at reporting what they’re actually doing compared to a device or a piece of software.
“If you take anything that you're worried about like social media and mental health and do some sort of study where you get them to self-report measures, you get stronger correlations than when you use more objective measures like reporting screen time on their iPhone generally,” he says.
So if you can’t test the impact of screen time on our lives in a lab effectively and self-reported studies are to be trusted as much as your mate saying that their Strava was definitely not recording properly on their most recent run, what do you turn to? The answer for many researchers is to run longitudinal studies, which survey a large group of people over years or even decades, to create a robust and reliable data set.
This, generally, is the best route forward for screen researchers. However, Pete explains that longitudinal studies are not the “be all and end all”. This is because they tend to result in the production of a “glorified correlation” - which tells an interesting story about a screen issue but is necessarily open to argument or requires further research to fully drill into - or they move so slowly that they report long after the policy and media debate has concluded.
And this leads into the third and final problem behind the messiness of screen research: the nuanced complexities of gathering research butts up against the ‘act now’ impulse created by the cultural conversation around games and screens.
This creates an environment in which individuals, campaign groups, the media and politicians either overstate what screen research says to allow them to take action or they move on a topical issue without any evidence at all on a ‘precautionary’ basis - resulting in problems down the line.
Unintended consequences
Pete provided a few examples of how messiness has led to unfortunate issues in video games research and policy around the world.
His first example was South Korea’s Shutdown Law. The law, which came into effect in 2011, introduced a curfew on playing online games late at night over concerns that addictive titles were preventing young people from sleeping.
The law provoked outcry from civil liberties groups, who felt it was an unfair restriction on young peoples’ freedoms, leading to the Government amending it in 2014 to give parents the choice to exempt their children from the ban.
But it wasn’t until 2018 that researchers were able to fully measure the impact of the law, eventually discovering that the “shutdown policy had practically insignificant effects in reducing Internet use for target adolescents”. It was eventually abolished in 2021.
The next example is the World Health Organization’s (WHO) decision to include gaming disorder as a mental health condition within the International Classification of Diseases (ICD) in May 2019.
It was undoubtedly a serious moment for the industry and it has been presented as such by people within policy and media circles. But the science remains remarkably shaky for something endorsed by the WHO.
“Gaming disorder is the only formal clinical digital addiction that exists,” Pete explained. “We essentially have no idea what its unique characteristics are or what the prevalence rate is.”
“On the WHO's own website, there's a link to a meta analysis from 2020 that puts the prevalence rate of gaming disorder anywhere between 0.7% and somewhere around 60% of the gaming population. That's wild, right? You're saying that this thing could affect either nobody or pretty much everybody or somewhere in between.”
Pete says that this shows that the WHO doesn’t know “what it’s looking at” now, let alone in 2017 when it first proposed its inclusion in the ICD. Yet instead of pausing for thought, the WHO instead conducted a flimsy literature review to allow it to eventually go through in 2019. Gaming disorder has since formed part of the Chinese government’s rationale for limiting young people’s access to online games - showing again how messy research can be quickly misdirected for political gain.
We then come to the final and most controversial topic Pete addressed: the relationship, or otherwise, between loot boxes and gambling.
The issue here is a little different to the others. Pete says that a range of researchers, including himself, have consistently found a correlation between the self-reported (uh-oh) spend of players on loot boxes and problem gambling scores. This should, however, be treated cautiously.
“That doesn't tell us anything useful about loot boxes and what their impact is. It just tells us that there's something going on somewhere,” Pete explains.
However, a lack of access to industry data, which is caused by both commercial caution and what Pete calls a “totally understandable” concern from industry that they’ll be kicked by academics in the backside if they hand it over, has mostly frozen efforts to advance the agenda.
Yet with the atmosphere around loot boxes remaining utterly putrid, researchers still feel compelled to contribute to the debate.
This leads in Pete’s words to “everybody doing the same study” over and over again: swelling the research base with new papers and literature reviews that don’t move the debate forward but does feed the media outrage cycle - causing policy makers around the world to react to the problem in a scattergun fashion that’s leading to regulatory fragmentation.
Belgium and Netherlands famously nested loot boxes (or certain formulations of them) under gambling law, which goes against the grain within the EU thus far towards treating concerns as a consumer protection matter.
Video games in Australia that feature loot boxes must now carry an M rating (essentially a 15+), whereas other video game age rating authorities have resisted that to prevent their wider content rating processes - which assess games based on, say, violence or sexual content - from being diluted.
And the UK’s efforts to manage the issue through self-regulatory industry principles has descended into an “ absolute shit show” according to one VGIM source, with Government and industry mired in circular arguments over the practicalities of administering them nearly five years after the loot box controversy first truly broke into the national debate.
Play nicely
These examples each show how the messiness of screen research and the debate around it can lead to negative outcomes.
So how do we move forwards? Pete doesn’t claim to have an easy answer (namely, because there isn’t one) but he does have a few suggestions.
The first is to build trust between industry and academia to enable sharing of actual in-game data to overcome a large chunk of the methodological issues affecting research into the medium.
The good news is that this is already underway in the UK. The creation of the world’s first Video Games Research Framework last year provided sensible rules of engagement to both parties, with research from The Oxford Internet Institute into a number of in-game topics already showing the benefits of such an approach.
However, Pete believes we could build this further by introducing independent brokers to deepen the conversation between academia and industry. This could help stop academics from “bulldozing” games companies with unreasonable or unachievable data requests and convince the sector that opening up its data is much better than leaving itself to the mercy of, say, self-reported surveys.
Second, Pete believes that research must go deeper to unlock the nuance at the heart of key digital issues.
This applies particularly to the loot box argument, where Pete believes that the possible risks to industry of deeper investigation into the mechanic will likely be outweighed by more measured outcomes.
“There is the ‘disaster for industry’ scenario, which is that there's nothing about loot boxes which is good” Pete said. “I suspect the actual answer is, it's much more complicated in that there are certain elements of loot boxes which are really not good for people's wellbeing…and there are other aspects which are actually quite good for their wellbeing. So can we weed out the stuff that's hurting people? Can we keep the stuff that's good? And how does that happen?”
And finally, Pete suggests that the games industry should play a more active role in fostering digital literacy to empower players and take the sting out of the public debate.
Pete argues in Unlocked that we need to think less about screen time as a blunt metric and more about a habit. There’s a big difference between screen time that adds to your life - such as video calling a friend to stave off loneliness - and times when it doesn’t - such as scrolling videos aimlessly late at night when you know you should be in bed.
Games companies already know the importance of talking about play as a balanced part of life but there may be value in pushing that argument even more strongly in the public debate to counter the knee-jerk urge.
I will admit that I left my conversation with Pete uncertain that we can ever truly break out of the cycle that we discussed.
But if it is to happen, it does feel like it will only occur if industry and wider society come together to tidy up the messiness that still currently sits behind screen research in the years to come.
Unlocked: The Real Science Behind Screen Time comes out on Thursday 21st March and can be ordered here.
News in brief
Bobby Kotick-tok?: Is the Artist Formerly Known as The Angriest Executive in Video Games about to buy TikTok? The Wall Street Journal reports that Bobby Kotick floated the idea of buying the US part of the business from ByteDance to a group of interested parties, including Sam Altman of OpenAI, if legislation to ban the app passes in the country.
DMA FTW: Apple has restored Epic Games’s developer account on Friday after dramatically removing it last Wednesday, paving the way for the Fortnite maker to launch its alternative app store in the coming months. The retreat, which came in large part due to pressure from the EU to comply with the Digital Markets Act, has been quickly followed by another concession to allow app developers to directly distribute apps via web pages (provided they agree to Apple’s new business terms).
Gaming Safety Dance: Keywords Studios, Moderate, ActiveFence and not-for-profit Take This have come together to form a new organisation called The Gaming Safety Coalition dedicated to “creating more robust, safer and more resilient game environments.” An important development, given the next story…
Gamergate - round 2?: Keza Macdonald has written a piece about the disturbing parallels between conspiracy theories currently being peddled about Sweet Baby Inc, a narrative design consultancy that has worked on a number of leading games, and the original Gamergate movement. It’s a worrying story and something the industry must pay attention to.
Super Mario Bros: Part 2: Nintendo has confirmed that a sequel to the Super Mario Bros film is on the way and will arrive in cinemas in the US in April 2026. This technically makes the film Super Mario Bros 2, which means I am expecting the plucking of a lot of onions and plenty of outstanding Birdo content to delight/terrify the kids.
On the move
Stig Asmussen, who was the Series Director for the Star Wars Jedi series, has founded Giant Skull…Tobias Sjögren has stepped down as CEO at Starbreeze Entertainment. Juergen Goeldner is in the interim hotseat…Greg Essig has returned to Apple in a games role…Rebecca Stow has joined the team at 71 Consulting after stints at Bastion and Honest PR…
Jobs, jobs, jobs
Epic Games is hiring for a Director, Public Policy Engagement…Scopely is seeking a Director of Content Design to support Monopoly Go…Moonbug’s move into games continues apace with it recruiting a Director, Product Strategy, Gaming & Interactive…Futurlab is looking for an Experienced Games Producer who is in the UK (but can work remotely)...YRS TRULY is searching for a Marketing Assistant who is up for occasionally viewing interesting material in the workplace…And it’s a G’day to apply for the Policy Officer role at Aussie and NZ games trade body IGEA…
Events and conferences
Pocket Gamer Connects, San Francisco - 18th-19th March
Game Developers Conference, San Francisco - 18th - 22nd March
Mobile Games Intelligence Conference, London - 27th March
London Games Festival, London - 9th-25th April
London Developer Conference, London - 11th April
BAFTA Games Awards, London - 11th April
Games of the week
Llamasoft: The Jeff Minter Story - Interactive documentary that allows you to play legendary designer Jeff Minter’s most famous games arrives this Friday.
Star Wars Battlefront: Classic Collection - Re-release of the classic 64 player multiplayer shooter arrives across all platforms.
Outcast: A New Beginning - Open world third person adventure game that is disappointingly not based upon Andre 3000 and Big Boi’s double act releases on Steam.
Before you go…
The comedian Phil Wang has been confirmed as the host of this year’s BAFTA Games Awards on Thursday 11th April.
This provides me with the perfect excuse to resurface Phil’s effort to recreate a classic video game as part of the TV show Taskmaster.
It is James Acaster’s GTA recreation that really stands out for me though…