Your Clones Want to Tear You to Pieces, and Analytics is Cool

While I was studying at Cornell for my undergraduate degree, I was fortunate to get to design and build games with other students in semester-long projects. One of these games was Clone Combat, a Flash puzzle platformer game in which the player is tasked with killing all of their clones using objects around them. The clones mirror the player’s actions exactly, except the actions are all flipped along the vertical axis.

Oh, and if you stumble across one of your clones on the way, fun things happen:

You can play Clone Combat here.

The Unintended Solution Debate

The puzzle component of the game mostly stems from the fact that most levels are impossible to beat without causing moments of asymmetry between character actions on the map. Examples include the main character hitting a wall while a clone being able to walk freely, a clone throwing a knife while another clone not having a knife to throw, and a clone falling down from the edge of a platform while the main character has a solid platform under it. This meant that as we built levels with increasingly complex platform layouts, the puzzles’ decision trees got exponentially larger. Needless to say, level design was a big challenge. The two level designers on the team – one of them being me – would come up with a good number of puzzles with intricate solutions, only to find out during playtests that players who have never played the game before would go ways we never thought of and still be able to solve the level from there.

This fired up a debate among the team: Should we make unintended solutions impossible? We could have tracked down each unintended solution we observed during playtests and reworked levels to kill these solutions off, but the sheer amount of time needed for this would have made finishing the game in a semester impossible. Another way of tackling this problem could be to prevent solutions, whether intended or unintended, from diverging too much from each other simply by restricting space. What that meant, however, was that designing hard puzzles would be much more challenging since the space itself would give away the solutions.

We ultimately figured that what we were really targeting were the solutions that disrupted the feeling of being challenged. If a player found an unintended solution but spent as much time trying to devise that solution as the time we would expect them to come up with the intended solution, it would indicate that we still achieved our goal in terms of how the game makes the player feel. Since the emotional disruption we were looking for is very subjective, we had to test the game with a large number of playtesters and find systematic patterns to identify flaws that we needed to fix.

Soon after the start of the debate, we released our earliest online version on a playtest website provided by Cornell. While this allowed us to reach the greater number of playtesters we needed, this was an unsupervised playtest, meaning testers would play the game on their own devices without any direct observation or intervention on our behalf and would then leave ratings and comments under our page. We also, however, had a very powerful tool that allowed us to remotely observe what players were doing in the game. The project course management team had given us an API to log in-game data onto Cornell’s server. That’s right – this is where the second part of the title steps in.

Analytics!

Game analytics is just a fancy name for interpreting playtest findings, whether it be before public release or post-release. Dictionary.com gives three definitions for analytics:

Logic. the science of logical analysis.
The analysis of data, typically large sets of business data, by the use of mathematics, statistics, and computer software
The patterns and other meaningful information gathered from the analysis of data

The term, in line with the second definition, is mostly used to refer to interpreting mass amounts of data from what I have observed. While I will be sticking to this particular usage of the term throughout the post, I have no problem using the term in the case of individually going over a small pool of data as well. Reading over the notes you took as you watched your little cousins screaming when neither of them could beat the boss in your game involves analytics just as doing a regression analysis of a graph of play times across a thousand players does.

Our way of tackling unintended solutions using analytics was to look at the frequency at which players found unintended solutions for each level and decide whether the solutions with higher frequencies took short enough to figure to warrant a major retune. We recorded player locations across timestamps and compared them to the amount of time it took the player to beat the level. We also checked whether players were pausing long enough, indicating that they were spending time deliberating the solution. We could not search these trends on raw data, however, since the increasing number of players made it too impractical.

In data analytics, representing big chunks of data in a way that is easily analyzable is often achieved through data visualization since humans have a high visual processing capability. That is what we exactly ended up doing – we constructed temporal heat maps out of random samples we gathered from the data, overlaid the heat maps onto the level backgrounds and then looked for patterns in the resulting series of images. From there on, we were easily able to identify which levels had problems.

It looks like we did a good job with design, since a lot of player comments after our final release were directed at how well crafted the puzzles were. Clone Combat went on to receive 20,000 hits in its first week of release on various Flash game websites, and was featured on the homepage of Newgrounds.