A brief look into the world of quality assurance testing.

What is Quality Assurance?

Hey all, Kai here! A stint in quality assurance is how many of us got started in the games industry, and today I wanted to shed some light on the profession.

Broadly speaking, quality assurance testers ensure the final quality of a game. Does this feature work? Is this mechanic fun? Did we fix that bug? Answering these questions is part of the QA tester’s job.

While we currently don’t employ a QA, we can’t simply forego testing and we all chip in to test each other’s contributions to the project. Since we do have prior QA experience, we put in place some methods to make this process as efficient as possible.

Assuring Quality

As we discussed in our Game Production 101 post, the development of Legion Hearts follows an agile development methodology. So instead of waiting until the very end to test the game, we test every single feature and iteration of the game independently instead.

This process is streamlined through the use of version control – Git in our case. Think of it like a shared save file for the Legion Hearts project that everyone can upload their progress to.

In practice, this translates to all new additions to the game being developed in a separate branches. And when we finish an addition, we create what’s called a pull request to hopefully merge that branch into the main game.

Feature Testing

Testing a PR often starts with checking the actual requirements or design to see if all the requested elements have been implemented. A feature just working isn’t enough – it has to work as intended!

If that’s the case, we can begin the actual testing process and try to identify issues. In practice, that means going over all the possible interactions of a given feature.

  • What happens if I place a unit on an occupied hex?
  • What happens if I enter full screen while dragging a unit on the grid?
  • What happens if I rapidly pause and unpause the battle?
  • What happens if press the menu hotkey while I have the skill popup open?
  • What happens if I do all of this at the same time?

In my experience, it helps to avoid thinking like a developer. Developers tend to be logical and predictable, spending their days building logical and predictable systems for logical and predictable humans.

That means the feature almost always works as intended, and most of the issues instead arise when doing things that are possible but not intuitive at all – sometimes called edge cases.

Once we’ve found an issue, it’s time to describe it a clearly as possible. This should enable the developer to reproduce the issue on their end and make debugging easier. Screenshots or videos are always welcome!

Bug: After repositioning an already placed unit on an empty hex, dragging it onto an occupied hex will cause the units to stack instead of swapping positions.

The developers will likely know which parts of the of code caused this interaction to happen, and clearly explaining how the issue came about is paramount to make the debugging process easier.

Code Review

Part of the testing process is the code review, where we go over the code that makes up to feature to point out errors, avenues for optimization and compatibility issues. At larger studios, this part is often handled by other programmers.

A big help when reviewing code is the fabled unit test. This refers to a set of small automated tests that we create to see if a given part of the system works. An example would checking that a character is indeed removed from battle when it has 0 or less HP.

While these unit tests alone do not replace manual testing, they can be helpful especially when working with code that touches a lot of other systems and could potentially break something seemingly unrelated.

While Legion Hearts isn’t all that CPU intensive, the client (the game installed on your device) will often check in with our servers, meaning small networking optimizations add up if we want to deliver a smooth experience.

Lastly, consistency and readability of the code become important factors when many different developers work with the same code at some point, and adhering to set practices makes everyone’s life easier.

Smoke Testing

Every now and then, we release a new version of our game in the form of an update. Unfortunately, it simply isn’t possible to test the entire game again with every build.

So instead we follow a checklist that goes over most of the critical systems to make sure the game is stable. A typo on the sound settings is annoying, but not critical. The game hard crashing when entering full-screen on the other hand would have to be addressed.

The goal of this process isn’t to find and fix all bugs. But by quickly checking all the core systems, we know whether or not we should release a given release candidate.

Compliance Testing

This type of testing is less about the game itself and more more about its compatibility with other platforms. Most games nowadays use an entire library of third-party services and integrations called SDKs, and we have to make sure they all play nice with each other.

  • Does the Steam overlay appear?
  • Do all PlayStation Trophies trigger correctly?
  • Did any profanity slip through that could make Nintendo mad?
  • Do we comply with Apple’s new privacy update?
  • Does this SDK conflict with that SDK?

The good part about third-party issues is that said third-party usually fixes issues by themselves, while the bad part is that you have to rely on a third-party to fix your immediate issue. Digging through documentation to figure out why something that always worked out of the box suddenly doesn’t is also a challenge on its own!

User Testing

Last on our list in user testing, which revolves around the behavior of players and gathering outside feedback. At its most basic form, user testing is what we’re doing right now by having you play our test version.

Inherent marketing benefits aside, this helps us sanity check our concepts and see if the game has potential, as well as revealing general sentiments about the game or frequent issues with it.

This type of testing can be much more involved however, and usually involves having a select number of people fitting a certain criteria play the game in a controlled environment.

By recording the play session of a tester and meticulously reviewing every moment of it, we can see their experience first hand. Why did they hover on this? What did they think this button would do? How did they react to it?

As this is a very time-consuming and expensive process, organizing such tests might be part of a more experienced QA’s responsibilities.

Breaking Stuff

All in all, QA testers tend to be those who are best informed about the state of the project at any given time. That’s because they’re intimately familiar with every single system in the game.

Over time, great QAs will develop a specific mindset towards breaking things. For example, whenever I play through a tutorial in a game, I instinctively think about all the things I’m not supposed to do.

This thought process extends to sanity checking designs, and being able to recognize potential issues already during the conceptual phase can save a lot of time during implementation.

Experienced QA testers might also spend a lot of time supervising the act of quality assurance itself to spot bottlenecks and develop a more efficient testing process with the other departments.

It’s an unfortunate reality therefore that QA testers are often underappreciated. As it’s considered an easily replaceable entry level position in an already competitive industry, the working conditions are often poor.

That said, QA testers are an indispensable part of any development team. Fixing bugs and polishing small annoyances may sound a lot less exciting than making new features, but that’s what separates good games from great games.

Game Dev 101

And this concludes our third post in our ongoing game dev 101 series. I hope you found it interesting! As always, feel free to reach out if you’re curious about anything game dev – we’re happy to chat.

Previous posts: