Review of our code review process

August 19th 2016, twelve days before the deadline, the release of another expansion of the game world of Kweetet. I read this article online that absolutely convinced me that we had to start with code reviews, and we had to start now. The stats spoke for themselves. 80% of the bugs are found during code review? Sign me up!

Did you read the above line? “80% of the bugs are found during code review”. I’m still baffled by that.

When you’re rushing for a deadline that’s when the most mistakes are introduced, so I wanted to start with code review immediately, to capture as much of them as we can. It turned out this was very successful, we did find bugs that would have been very embarrassing in a release.

There are a lot of options in setting up a code review process, for example as described in this article, which is a pre-merge review process with pull requests and is by far the most common model, certainly when using git.

At die Keure we use perforce for the development of Kweetet, which does not know the concept of pull requests. This forced us to use a post-check-in process. More specifically, we used a MOB code review process (how cool a name is that?).

Now, a good year later, we’re still doing code reviews and didn’t change much to our initial process.

Context

Maybe I should start with a little context, because although our process works well in our team, there’s no guarantee it would work in another. Actually, it turned out it didn’t work for another team - more on that later.

Kweetet is played at school and at home during the six years that a child attends primary school. It is a story driven adventure game, where children complete quests, interact with numerous NPC’s, collect stuff, play puzzles and minigames and a lot more. A child starts playing the game in the first grade and only finishes it in the sixth. Every school year they start a new chapter in the story. And every chapter is played in another world, which means a lot of content. And a lot of code.

We have apps for Windows, Mac, iOS and Android plus a “lite” version for Android and iOS. Kweetet can't be called a small project; I didn’t count the number of lines, but we have a good 7000+ files with code, not counting the server code. This code has been written by a team of 4-5 people (seniors, juniors and interns) over the course of four years. In total, 10 programmers worked on the code.

We could have done this faster, but we faced a major technical hiccup in the middle of our development: the game used to be played directly in the browser via the Unity webplayer plugin, but alas, browsers decided they didn’t want any plugins anymore forcing the webplayer to retire. We tried to convert the game to WebGL, but we learned the hard way that this technology, promising as it is, has no way near the performance (yet) as the webplayer had. This forced us to convert the game to installed versions on several platforms. We lost a lot of time on that.

So in short the context is: lots of code, a small team with interns coming and going, over four years. Only the last 1.5 years we had code reviews. By now it’s safe to say: we should’ve done this by the start, we would’ve been ready perhaps a year earlier.

As said before we don’t use git, which is a common tool in code review processes, but Perforce. Completely unlike git-flow, the whole team works on the trunk (and for good reasons, but not in the scope of this post to elaborate on). So we’re actually reviewing commits rather than pull requests, if put in a git context. There is collective code ownership.

Our process for MOB code reviews

• Right after lunch, we sit down with some team members for maximum 45 minutes (it’s often shorter) and review all commits since the last review session. Right after lunch, because then we don’t interrupt anyone’s “zone”.
• Anyone who wants and/or has time can join, but you’re not obliged. In general we’re doing these reviews with 3-4 people; with more, the discussions can become lengthy.
• We do this reviews on a projector, with someone behind the keyboard going through the files. There’s no designated driver, anyone who wants can take the wheel.
• We don’t exceed the 45 minutes limit, anything that isn’t reviewed we review later. Programmers note the bugs that are found and fix them afterwards before continuing on what they were doing.

These are not hard rules, if no one is free, or there haven’t been many commits, the review is postponed. Sometimes we skip to important parts first and then review the other things later.

We try to keep to a daily review, but at least once a week it happens that we skip one.

The good

Less bugs

Obviously, we have less bugs. We don’t measure anything, but we do find mistakes regularly, so these are all bugs that are squashed. This is actually the least important advantage (but an advantage nonetheless).

Better code

The more important advantage is not so much that there are less bugs, but that the code is just better. It becomes more readable, it is more performant, it is better structured.

The above situation is one that could occur in the past, but doesn’t anymore.

Because we talk with each other on a daily basis about our code, we discuss the chosen solutions on problems. We share a lot of views and opinions with each other, we discuss why we prefer certain approaches, we share ideas, we tell each other what we perceive as readable code, we have many good laughs.

Thus the code became very consistent, all members of the team write code in pretty much the same way now, and we all changed our ways to get to that point. Voluntarily.

Better developers

This made us all better developers, we learned solutions to problems we would’ve never thought of, we write better and more performant code, we all know and understand almost the entire code base.

Better team

It made us a better team. We have our share of heated discussions, certainly at the start, but it all happens in mutual respect and it made a stronger team. There was also a lot of humor. A lot of it :)

Respect and team spirit among team members can really improve the quality of the code, since everyone is open for improvement and feedback. These code reviews introduce that.

More resilient

We are more resilient to changes, by which I mean members of the team entering or leaving. When a team member joined us (often interns) they quickly learned the codebase and also learned our approaches. On the other hand we were challenged by these new team members; we had to defend to them why we did the things the way we did them.This brought new insights to the team and to the new member.

When we weren’t doing these code reviews, we often ended up with very alien pieces of code in our codebase written by interns who didn’t know any better. We still suffer from that.

And when a team member left, everyone knew the code he/she worked on so there is little to no need of knowledge transfer sessions.

Collective code ownership

Because everyone has seen the code grow, has discussed the changes, had the chance to contribute (even without writing any code), there is a stronger sense of code ownership among the team over the complete codebase.

We talk to each other

Instead of writing our remarks behind a computer, we sit together and talk to each other (Of course, this is only possible when your team is not spread over different locations in the world.). Differences in opinion are voiced immediately and can be discussed in an open manner. Misunderstandings are detected early. It creates a stronger team spirit.

It’s fast

Typing a sentence is not as fast as saying it. And when you’re typing, you really need to weigh your words, lest they are not misinterpreted. Discussions in text can generate a lot of back and forth messages, costing more time.

Less delays

The review of pull request can take time before someone had time to do them, and by the time you get the feedback you’re already on working on something else, as happened with the author in this article. By reviewing commits rather than pull requests, we review code very early, even while it’s still in development. This can seem obsolete, since the programmer might detect his own errors before he’s done, but more often the programmer is just helped by this. Even better: if we notice that a programmer is implementing a system fundamentally wrong, we detect it before much work has been wasted. This is especially the case when interns join the fray.

The not so good

It didn’t work for the webdev team

We also have a team of four web developers who tried to implement the same process, but they stopped after a few months. They considered it a waste of time and instead started to review pull requests in git from time to time via the computer. Discussions happen via slack.

Perhaps important to note is that they used to work via a weak code ownership model (each programmer is responsible for a separate part of the code or a separate project), this makes the open code review process not such a good fit.

We mostly review code

Since we review the commits one by one, file by file, we’re mostly considering small local changes to files and new code files. We’re not looking at the big picture in a structural manner. However this is not such a bad downside; although we don’t review systems in a consequent process, we often discuss the architecture of the game when a related issue pops up

We can get behind

Sometimes we get behind with our reviews, when most of us can’t attend the review, we skip it. We try to keep this to a minimum, but sometimes we skip a few days, and then it’s possible we have a lot on our plate. Nothing to do about that, we just do the work and sometimes extend the duration of the review session a bit. The problem is that this can become a drag and that causes us to be less attentive. We try to avoid this problem as much as possible, by having review sessions nearly every day.

No history

An advantage of written code reviews is that you have a history of them. You can refer to previous reviews for motivation of arguments. You can cross check how you dealt with similar issues in the past. This is something we don’t have in the process. Can’t say I miss it though.

On the spot

In the proposed process, we need to review and spot defects on the spot. This can be hard sometimes. It’s a big advantage of reviewing pull requests on the computer: you can take all the time you need to review a file/codebase and you can even take a complete checklist and go over the code at length.

Conclusion

The team and myself all feel that this MOB code review process improved our code, our product and ourselves. For a small co-located team, working on the same project, I think this is a good approach.

Feel free to share your own code review experiences below! I'm starting as a lector in game programming shortly and I'm playing with the idea to incorporate code reviews in the lessons, so any feedback is more than welcome!

Nordic Game 2017 takeaways - part 2

(Read part 1 here)

The second day was even hotter, the talks as well!

Analytics

The first talk of the day was about an algorithm for fun and profit, turned out it wasn't really an algorithm but rather a (very) quick overview of analytics concepts. All in all it was just promoting their analytics platform.

The slide here illustrates the metrics that drive revenue and this was the algorithm. I missed the fun though.

Design patterns for distributed game servers

Say "design patterns" and you have my attention :). This was also a very quick and short talk about three aspects of game servers. Again it was a bit promotion for their own server software Nakama, which is freely available on Github.

It was about session management and JWT tokens, which is a en encoder format for session data that can detect tampering. If you put non-identifiable user data into tokens and after authentication, you cache the token on the device. That way you have some info about the user without having to authenticate again, saving on database queries.
Have a field to track when the token is expired.

About databases the tip was to understand the database engine you use and how it behaves for each kind of query you need. Postgres has the most options when it comes to indexing. In general, use key-value tables as much as possible.

The talk made the software look good, if I ever have the time, I would love to port Kweetet over to this software.

Mundaun

I had some time left so I started to check out some of the games on display. It was my intent to check as much of these games as possible, but I didn't get very far. Anyway, I started talking with Michel Ziegler, the developer of one of the Swiss games, "Mundaun".

The game was still a bit rough and needs some more visual love, but it already had a very beautiful aesthetic that made it look promising. What was really impressive is that he made this game all by himself and was now promoting it at this event! Good luck to him.

Soft launch

Next up was "How to soft launch successfully" by Paul Conway.

There are three phases to a soft launch:

1. Planning (smart!)
2. Retention soft launch, where you try to optimize your retention and daily active users (DAU)
3. Monetization soft launch

Two to three months is enough for the retention soft launch, as long as you can update easily so you can perform a/b testing on all the important features. Good retention rates are 40% for day 1, 20% for day 7 and 10% for day 30 retention.

The slide on the right says it all to get these retention: "Cater for different skill sets", make sure your game doesn't get boring for skilled players and too hard for less skilled players, in other words: adapt.

Denmark, for some reason, is a land with high retention rates, so you can start there for the soft launch.

Consider returning players, when a player returns to your game, make sure he has a pleasant experience: make the game a bit easier for players who have been away for quite some time, make it challenging for players who return regularly.

And then follows the monetization soft launch, you can start experimenting with in app purchases. The conversion rate of your game will (or should) be around 2% of the total number of players, in other words: 2% of the players make an in-app purchase in your game. The best purchase that you can offer your players is the starter bundle. Make sure that the starter bundle has clear value.

In the end you must ensure that the LTV is greater than the CPI. (Life Time Value and Cost Per Install)

Quite some stats came from this blog and the gamesparks blog itself has also some interesting topics.

Hitman

Senses of the Hitman AI was a cool talk! They showed how the line of sight system works for the AI, how their current state influences the view, how they respond on what they see, and how they hear. I didn't write as much down as I should have, I think I was too busy looking at the visual candy :)

Some of the content was similar to this GDC '13 talk, but they've improved a lot since then.

Maybe trivial, but a very important idea is how they clearly differentiate between what the state of an AI object is and what other AI's know about that object. By seeing or hearing things, what they know about an object changes.

PBR on mobile

We're not there yet when it comes to high end graphics on mobile devices, but we're getting there. It's only a matter of time. In the meantime I had to put on my thinking cap because the session "Bringing physically based shading to mobile" was a very technical and deep session. It's a repeat from his session at GDC '16 and this one I think, but the topics differed somewhat.

Full PBR on mobile devices is not possible yet, but this talk explained in detail how they achieved triple-A visuals (not graphics) on a mobile in CSR2. They started with a full PBR rendered concept and given the game's characteristics skipped or optimized where possible.

For example they don't do dynamic shadows, since the game goes to fast to notice them. Also they made sure that light always remained static, from one direction, enabling to use pre baked image based lighting in cubemaps, or rather prefiltered mipmaped radiance environment maps. To generate these cubemaps they used this adapted version of AMD's cubemapgen tool. A detailed explanation can be found in that article.

These cubemaps can best be sampled with the texCUBElod shader instruction, however not many mobile devices support it, they had to use texCUBEbias instead. It's supported, but heavier on the shader.

For AAA visuals you need HDR, but mobile devices don't support these kind of textures. Therefor they changed the cubemaps to contain the low range values in the RGB channels and the high range values in the alpha channel and change the appropriate shader code accordingly.

For the cars they could get the albedo values from the manufacturers (obviously), but for all the other scenery they needed to get the values elsewhere. You can try and capture values yourselves, but it turns out that photographers often provide these values with their work and they were able to use these. I haven't been able to find a site like this, so if anyone knows such a link please do share in the comments!

More mobile graphics

Right after this heavy session, I went to "Optimizing our renderer for Metal on iOS" by Timo Heinäpurola. He shared his slides for this talk here.

The concept of triple buffering here is interesting and there are lots of good optimization tips in the slides. I for one will try what the effect is of using smaller texture types in Kweetet.

Unboxing Unreal 4

Last talk of the day, by a bearded lady. The team started a new project with Unreal and the talk was a collection of lessons learned.

• Unreal is C+++blueprints, you can never have one without the other.
• Always overload CharacterActor and MovementComponent
• Use the store! The game they showed was almost completely made with store assets
• But put them in a separate project and import only the content you need in your own game. This is actually a tip that is true for Unity too.
• They gave the grass more depth with contact shadows.
• They had a cool post process effect for their fog of war
• The camera transition system was custom made and very cool

Awards

Then came the Nordic Game Awards, which was a very weird show. The host Niels Forsberg was great! Apparently he's a Danish geek/comedian bit like our own Lieven Scheire.

The award winners were just weird, it seemed as if they were not so happy about it, saying just "thank you" when they received a reward or nothing at all. One said: "Backup your files, eat your vegetables, visit your parents". Another one was very emotional and asked all the other nominees also on stage. Very weird, but good fun :). One winner didn't show up, so some random guy came up the stage and took the bottle of champagne :)

Booze

I couldn't afford to be tired in the weekend so I skipped the Nordic Party, nevertheless there was good food and good beer and good company!

Nordic Game 2017 takeaways - part 1

It was hot in Malmö, which is weird for a country in the North. Nevertheless, these are the other interesting tidbits I saw at Nordic Game 2017.

The Huddle

The show began with a detailed overview of how "the Huddle" was made in Inside. This was a really inspiring talk, it was totally why I wanted to come to this event: to learn about the inner workings of cool things in games. The best insight about this blob is how the animation was done; the legs and arms don't drive the blob, although it looks that way. Instead there is an ingenious IK system that drive the animations.

Six

Another inspiring talk was about the implementation of Six, the character in Little Nightmares by Tarsier studios. An interesting idea was their "Locomotion Animation Overrides", they change specific animations in character animation setup in function of the state of Six.

Also worth noting is that the character interacts with objects in the world, but the objects also interact with the character. This two-way animation communication makes a very rich world.

The IK of Six was done with IKinema, which is a full body IK solution. This makes Six standing on stairs and other objects in a natural way, and makes her grabbing or climbing naturally. Because this can look sometimes weird they added something they called "Positional fixup": the player character will walk closer to the object that it wants to pickup or interact with, making the animation look smooth and natural.

"Things look better with audio" was a very inspiring quote from the talk. They discussed how sometimes when objects interacted with the player character you could notice intersecting meshes or other visual artifacts, but by adding sound somehow you don't notice these misses in the visuals, making it look good.

Mobile games

I attended a talk by Jonathan Jordan about the past, present and future of mobile games. Nothing much I didn't know there, but it was good to hear and see. Maybe the best bit was this:

• 70% of the successful mobile games use portrait mode
• They feature short play sessions

The full talk is on youtube, so check it out, definitely check out the part about "Where are we going"

Experiments and mobile games

The next talk was about how a/b testing was applied at King for their games. Not a really good talk for me, it was very general without much (if any) concrete examples.

Twitch

Then there was the session from Jamie Jackson of the slingshot cartel.

He said that there are three types of gamers nowadays:

• Players, people who play your game
• Viewers, people who watch games and want to learn - they're also players.
• Broadcasters (on twitch and youtube), who also play

He then described how their game provides for these types of players in DRG Initiative. The idea resembled a bit what happens in the hunger games: viewers can vote and steer what happens to the players in the game, while the broadcaster is the game master who can set up the votes. Cool idea!

Social Media

The next talk by Natalie Griffith was about how to handle social networks to promote your game. The slides were the same as at Develop Conference 2016 and can be found here. I learned a lot from this talk; how important a plan is, what AIDA is and what social networks are best applied at each stage.

Then there were a few very useful Facebook tips:

1. Post regularly
2. Don't post too much (no more than 1 post per day average)
3. Make sure you create enough reach by posting smaller news before posting something important.
4. Choose good moments to post (not at night, for example, when everyone is asleep)
5. Mix up media types
6. Use native FB video, no links to youtube
7. Use subtitles in your movies (58% of the audience whatches the movie without sound)

She provided this link to some resources at the end.

Booze

And then happy hour began!

And then a German Games Party!

Hopper - a "finished" project

Many times I've been tempted to take part in a game jam, but up until now I always had something on my agenda that was in the way.

A friend of mine, set to defeat adulthood (and perhaps even puberty), pointed me in the direction of the "Finally Finish Something Jam". A jam that lasted a month where the goal was to finish a personal project that you had lying around. That's the ideal type of jam for me :)

I happened to be busy with a small test project, I was creating a small test bed where I could play with AI code in a puzzle game. So with a good time frame for a Jam (31 days) and a good project, I joined the jam.

In the upper right corner of this blog you have a link to the result. It's a first version. I'd like to add more and smarter AI models, the option to add you own levels and the option to add your own AI. I also would like to look into writing an API on Google cloud to register high scores, levels and AI's. But all these are quite out of scope for this jam.

How did I go along

For source control I used my own perforce server on Droplet and the game was developed with Unity Personal.

For the UI I used the free Jolly UI kit. I had enough with the free version, but in future versions I'll need the full kit.

It's hard to find nowadays, but I used the free Photoshop CS2 for all the textures. It does the job, but the current Photoshop version is quite a lot easier (and more expensive of course).

I tried adding sound effects, but let's face it, I'm not good at that, so the sound effects are out (they're in, but disabled).

I also need to work on the levels I think, I ordered them a bit by difficulty, but that can be tweaked. And I can definitely add more.

Anyway, give it a go, tell me what you liked, what you didn't like, was it too hard or way too easy? Do you understand how it works without much trouble? What do you think I should add?

And foremost: have fun!