research-article

A metric for automatically flagging problem levels in games from prototype walkthrough data

Authors:

Christoph Lürig,

Regan L. MandrykAuthors Info & Claims

AcademicMindTrek '15: Proceedings of the 19th International Academic Mindtrek Conference

Pages 33 - 40

https://doi.org/10.1145/2818187.2818281

Published: 22 September 2015 Publication History

Abstract

Playtesting early and often is important for all game developers, but especially for the growing number of indie teams producing commercial games; however, playtesting game prototypes remains an expensive and time-consuming process. In this paper, we present a new game metric, automatically generated from prototype walkthrough data, which flags problematic levels so that developers know where to invest their effort in fixing the game. Created during the development of the commercial game Angus Hates Aliens, in collaboration with indie developer Team Stendec, our death-related problem level likelihood indicator (DPLI) is interpretable and actionable, i.e., it easily allowed the developer to know where to fix the game levels. Finally, DPLI correlated to enjoyment ratings for the game levels, suggesting that it was a good indicator of problems in the context of our prototype evaluation.

References

[1]

Ajzen, I., & Fishbein, M. (1977). Attitude-behavior relations: A theoretical analysis and review of empirical research. Psychological bulletin, 84(5), 888.

[2]

Andersen, E., Liu, Y. E., Apter, E., Boucher-Genesse, F., & Popović, Z. (2010). Gameplay analysis through state projection. In Proc. of FDG, pp. 1--8.

Digital Library

[3]

Ambinder, M. (2014). Making the Best of Imperfect Data: Reflections on an Ideal World. Keynote at CHI Play'14. http://goo.gl/OFWGz5, Accessed April 2015

[4]

Brandt, E. (2007). How tangible mock-ups support design collaboration. Knowledge, Technology & Policy, 20(3), 179--192.

[5]

Brockmyer, J. H., Fox, C. M., Curtiss, K. A., McBroom, E., Burkhart, K. M., & Pidruzny, J. N. (2009). The development of the Game Engagement Questionnaire: A measure of engagement in video game-playing. Journal of Experimental Social Psychology, 45(4), 624--634.

[6]

Canossa, A., Drachen, A., & Sørensen, J. R. M. (2011). Arrrgghh!!!: blending quantitative and qualitative methods to detect player frustration. In Proc. of FuGa'11, 61--68.

Digital Library

[7]

Davison, G. C., Vogel, R. S., & Coffman, S. G. (1997). Think-aloud approaches to cognitive assessment and the articulated thoughts in simulated situations paradigm. Journal of Consulting and Clinical Psychology, 65(6), 950.

[8]

De Grove, F., Cauberghe, V., & Van Looy, J. (2014). Development and validation of an instrument for measuring individual motives for playing digital games. Media Psychology, 1--25.

[9]

Desurvire, H., Caplan, M., & Toth, J. A. (2004). Using heuristics to evaluate the playability of games. In CHI'04 EA, 1509--1512.

Digital Library

[10]

Drachen, A., & Canossa, A. (2009). Analyzing spatial user behavior in computer games using geographic information systems. In Proc. MindTrek'13, 182--189.

Digital Library

[11]

Drachen, A., & Canossa, A. (2011). Evaluating motion: Spatial user behaviour in virtual environments. International Journal of Arts and Technology, 4(3), 294--314.

[12]

El-Nasr, M. S., Drachen, A., & Canossa, A. (2013).Game analytics: Maximizing the value of player data. Springer Science & Business Media.

Digital Library

[13]

Fullerton, T. (2014). Game design workshop: a playcentric approach to creating innovative games. CRC Press.

[14]

Fulton, B., & Medlock, M. (2003). Beyond focus groups: Getting more useful feedback from consumers. In Proc. GDC'03.

[15]

Games User Research Summit Schedule (2015). http://goo.gl/36M1t6, Accessed March, 2015.

[16]

Gerling, K. M., Birk, M., Mandryk, R. L., & Doucette, A. (2013). The effects of graphical fidelity on player experience. In Proc. MindTrek'13, 229--236.

Digital Library

[17]

Hopf, M., & Ertl, T. (2003). Hierarchical splatting of scattered data. In Proc. IEEE VIS'03, 57

Digital Library

[18]

Kaplan, R., & Saccuzzo, D. (2012). Psychological testing: Principles, applications, and issues. Cengage Learning.

[19]

Kim, J. H., Gunn, D. V., Schuh, E., Phillips, B., Pagulayan, R. J., & Wixon, D. (2008). Tracking real-time user experience (TRUE): a comprehensive instrumentation solution for complex systems. In Proc. CHI'08, 443--452.

Digital Library

[20]

Lazzaro, N. (2004). Why we play games: 4 keys to more emotion. In Proc. GDC'04.

[21]

McAuley, E., Duncan, T., & Tammen, V. V. (1989). Psychometric properties of the Intrinsic Motivation Inventory in a competitive sport setting: A confirmatory factor analysis. Research Quarterly for Exercise and Sport, 60, 48--58.

[22]

Mandryk, R. L., Atkins, M. S., & Inkpen, K. M. (2006). A continuous and objective evaluation of emotional experience with interactive play environments. In Proc. of CHI'06, 1027--1036.

Digital Library

[23]

Myers, G. J., Sandler, C., & Badgett, T. (2011). The art of software testing. John Wiley & Sons.

Digital Library

[24]

Ryan, R. M., Rigby, C. S., & Przybylski, A. (2006). The motivational pull of video games: A self-determination theory approach. Motivation and Emotion, 30(4), 344--360.

[25]

Sefelin, R., Tscheligi, M., & Giller, V. (2003, April). Paper prototyping-what is it good for?: a comparison of paper-and computer-based low-fidelity prototyping. In Proc. of CHI'03, pp. 778--779.

Digital Library

[26]

The ESA (2014). Essential Facts about the Canadian Video Game Industry. http://theesa.ca/wp-content/uploads/2014/11/ESAC-Essential-Facts-2014.pdf, Accessed Mar 30, 2015.

[27]

Wallner, G., & Kriglstein, S. (2013). Visualization-based analysis of gameplay data--a review of literature. Entertainment Computing, 4(3), 143--155.

[28]

Valve Corporation, Statistics Half-Life, Website http://goo.gl/g1Jf6c, Accessed Mar 30, 2015.

[29]

Vermeulen,. L. & McGibney, J. (2014). Finding the Fun -- Usability Testing as an Indie Studio. Presentation at CHI Play'14.

Cited By

Dechant MWelsch RFrommel JMandryk R(2022)(Don’t) stand by me: How trait psychopathy and NPC emotion influence player perceptions, verbal responses, and movement behaviours in a gaming taskProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502014(1-17)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3502014
Miller MMandryk RHancock MMarquardt NSchöning JTory M(2016)Differentiating in-Game Frustration from at-Game Frustration using Touch PressureProceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces10.1145/2992154.2992185(225-234)Online publication date: 6-Nov-2016
https://dl.acm.org/doi/10.1145/2992154.2992185

Index Terms

A metric for automatically flagging problem levels in games from prototype walkthrough data

Recommendations

A Postmortem on Playtesting: Exploring the Impact of Playtesting on the Critical Reception of Video Games
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

Game studios aim to develop titles that deliver a fun and engaging experience for players. Playtesting promises to help identify opportunities to improve player experience and assist developers in achieving their design intent. However, a lack of ...
Playtesting for indie studios
AcademicMindtrek '16: Proceedings of the 20th International Academic Mindtrek Conference

Creating video games is a lengthy and demanding process. Financial success for games studios often depends on making games that deliver a fun and engaging experience for a diverse audience of players. Therefore, understanding how players interact and ...
Towards Democratisation of Games User Research: Exploring Playtesting Challenges of Indie Video Game Developers
CHI PLAY

Playtesting is a games user research (GUR) method used to evaluate design decisions based on feedback gathered from players with the goal to improve player experience. HCI games research has been actively working on and promoting best practices in GUR. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

AcademicMindTrek '15: Proceedings of the 19th International Academic Mindtrek Conference

September 2015

230 pages

ISBN:9781450339483

DOI:10.1145/2818187

Conference Chair:
Markku Turunen
University of Tampere

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 September 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

AcademicMindTrek'15

AcademicMindTrek'15: Academic Mindtrek Conference 2015

September 22 - 24, 2015

Tampere, Finland

Acceptance Rates

AcademicMindTrek '15 Paper Acceptance Rate 33 of 51 submissions, 65%;

Overall Acceptance Rate 110 of 207 submissions, 53%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
97
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)1

Reflects downloads up to 23 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Dechant MWelsch RFrommel JMandryk R(2022)(Don’t) stand by me: How trait psychopathy and NPC emotion influence player perceptions, verbal responses, and movement behaviours in a gaming taskProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502014(1-17)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3502014
Miller MMandryk RHancock MMarquardt NSchöning JTory M(2016)Differentiating in-Game Frustration from at-Game Frustration using Touch PressureProceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces10.1145/2992154.2992185(225-234)Online publication date: 6-Nov-2016
https://dl.acm.org/doi/10.1145/2992154.2992185

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents