Why I can't recommend Git for game projects.

As mentioned before on this blog, I have worked with many source control systems in the past 15 years; SVN, CSV (boy that's old now I that look up that link, I was a student while I last had to use this), Plastic SCM, Git and Perforce.

At work, the idea has started to rise to use Git for our projects. Most of these projects are websites, written in php, javascript, html and css. I am very fond of the idea to start using Git for those, it opens many many doors to easier ways of life.

We can use Travis CI, we deploy everything automagically on AWS, we can work offline and most importantly: we can branch all we want.

Definitely read this article, "A successfull Git branching model". This branching model known as "Git-Flow" does indeed ensure that you have very few merge conflicts. It is the basic model that is also recommended by Sourcetree.

Basically, it boils down to this:

• There is a master branch
• There is a dev branch
• Create branches per feature, only merge them with dev
• Create branches per release, merge them with dev and master
• Create branches per hotfix, merge them with dev or the active release branch

And that looks like this:

I believe this is a decent model for using git. In web development.

In game development however, I don't think its the right tool for the job. Since the discussion to start using git began to rise at work, I found myself defending this point of view more and more often. I believe git isn't fit for game development, but it is hard to come up with a compelling case why this would be, when challenged at the spot.

Notice that I write "I don't think it's the right tool". I am not sure. And by writing this article and doing research for it, I hope to be more sure and build a good argument for this thesis.

The situation

First, try and follow me on this. Say you have a photoshop file, a psd. This psd file consists of one hand-painted layer. This psd is in the product that we want to ship, build features for, fix mistakes in, etc.

(I agree this is a contrived example, since you would never create a psd like that, but I'll make an analogy later that will justify this.)

I want this psd in source control, so I don't lose any progress. Thus, first commit on the master branch, we push to dev and start working on the psd. We draw a landscape, we add a tree, we add the sky. Cool, first version is finished, we merge with master.

We start working on the next feature: a tree house in the tree, we do this on a feature branch.

Feature

But! We also need to hotfix the release while we're working on the treehouse! It was required that the tree had leafs and seven apples but we forgot. So we create a hotfix branch to add the apples, and merge that back to master and to dev.

Hotfix

Up until now, no problemo. But when we want to merge the feature branch with the tree house into dev we have an issue: binary files don't merge, so we need to choose one of both, the apples or the treehouse. And even if binary files could be merged, how would you merge hand painted strokes in an image? Probably best if we just redo the apples on the feature branch and merge that into dev.

Do we just multiply?

A possible "solution"

One psd file containing the whole project clearly won't work, so let's split the features in the file into layers, and let each layer reference another psd in which we draw each separate feature. To follow the same example, we start with a psd in the master branch, push into dev and add a layer with the landscape, a layer with the sky and a layer with the tree. This goes back to master, first version released.

Now the same thing happens: we start a feature branch where we add a tree house. We also do the hotfix and create a layer with apples. Both get merged into dev but we have a merge conflict. We need to add both the apples layer and the treehouse layer into the same file. But which one goes on top of the other? It may very well be that there is a possible order, but it could also be that this doesn't work either way:

No 7 apples

The house is behind the leaves

It's the same for games

Now the idea is as follows: the psd is a lot like a game. In fact, the situation for a game is a lot worse than with the psd.

• A game consists of many textures, who all have the problem as the layer-less psd. They can't be merged.
• Animations idem dito
• Models idem dito
• Audio files idem dito
• Prefabs idem dito
• Scenes idem dito

Prefab and scenes are a lot like the psd with the layers, they reference other assets. Even if you would be able to merge scenes and prefabs, changing parts of a scene or a prefab in different branches can easily break the scene/prefab.

Point is, code files and text files are easily merged, but colors, images, shapes and sound are impossible to merge in a reliable way.

Are there any solutions?

(I focus on Unity here, but most of the items here are applicable to all other game engines too.)

• A typical workaround is to submit these files only when they're completely done, so you will never need to merge them. But this is a workaround, when you do need to have an unforeseen change then you're back in the same situation. And if the files never change, why are they in version control in the first place?
• Another workaround is to try and have as many files as possible in a mergeable format. Scenes and prefabs for example can be stored as text instead of binary data. With Unity's SmartMerge you can even easily merge the complex YAML files in the editor! You would have to, because a merge conflict in GIT makes the YAML file unparseable and you can't review the changes anymore in the editor.
  This tool is a god-send, but still won't fix all issues. It can make a file parseable again, but it won't stop treehouses being placed in the wrong tree. The artist will have to redo a part of his work.
• Another and often used workaround is to put as little as possible in a scene and split everything into small prefabs. This will indeed avoid many problems, but it can be very cumbersome to manage. And it's just another case of the psd with layers, eventually the parts won't mix at some point and you'll have to redo work.
• The same article presents another workaround: ensure that only one person can work at the same time on the same file. Git has no support for this. And even if it would (like Perforce does), it still wouldn't make it impossible to change the same file in different branches. Even Perforce can't prevent that.

Other issues

• Games can grow into large asset- and codebases. Kweetet for example has +300K files, 25GB in size on the client, +250GB on the server. And that's only counting the game, without all related websites, assets, scripts etc. Divinity II also has +300K files, 70GB in size on the client and +500GB on the server. As this article points out, git needs to examine all files to check whether they changed, so this is not ideal for large codebases. (Actually, reading that article, I might want to try Mercurial!)
• Another often heard issue is that git can't handle big files very well. Since every client stores the entire history of a file this can grow out of control with many large files. This has recently been solved with git-lfs. This will only store the latest revision of a file and keep the rest on a centralized location. (Thus breaking the whole decentralized idea of git...)

Conclusion

In conclusion, I cannot recommend Git to be used for large game development projects with medium to large teams. Large projects with many large binary files cannot be easily merged, so you need to keep branching to a minimum. On the other hand, if you have a small and short-lived game project and a small team (fourish people) Git would be a valid choice.

So then what?

So what do we do for Kweetet? We use Perforce for version control and have two branches; master and dev. We only develop on dev, and work in cycles from stable dev state to stable dev state. Each time when the dev branch is stable (usually when a new feature is ready), we integrate to the master branch. We clean up any issues we find on the master branch and integrate those back into dev. These are mostly small so it does not happen often that we have conflicts. In practice, we only work on one branch.

One of the advantages of using Perforce is that the developers can open a scene exclusively, do what they need to do and check it back in. That way, there are never conflicts in the scenes.

Another big advantage is that Perforce does not need to check which files changed, you need to tell Perforce that. Of course you don't do that by hand, all our tools do this automatically when we change a file. Unity, Visual Studio, Notedpad++, Sublime and others all have perforce plugins.

If you know of a better way to deal with the issues I mentioned in this post, I'm more than happy to hear it, because I would love to give Git a second chance, or improve our Perforce workflow.

Other interesting articles/forums I read for this post

• Branching For The Win: How Black Rock Uses Code Branches
• GitFlow considered harmful
• A Method for Working with Shared Code with Unity and Git
• Working on workflow for Unity and git
• Unity and GIT confusion
• Why isn't Git used by more video game developers?
• 22 Cans – Old Dogs Reject New Tricks

[EDIT]

Other articles I found since posting the above:

• FeatureBranch
• Why Google Stores Billions of Lines of Code in a Single Repository
• What to think about when choosing a VCS for game development

Setting up a perforce server on a droplet instance.

A colleague of mine was looking for a solution to host a perforce server somewhere reliably, that could be used for small personal projects. There are not many hosted perforce solutions, Assembla is the only one I know and is expensive if you only want to use it for personal small projects, it is a complete project management tool so it's more than I need.
My colleague found this project by DigitalOcean called 'droplets' where you can have a small vps for a reasonable low price. 20GB is not big, but is enough to start personal projects on. Wanting a personal perforce server myself, I immediately started with a new droplet!

So first I created a so called 'droplet' with 20GB disk space and CentOS7. It is amazing how fast this was set up, within 5 minutes I was up and running, with putty connected to my new server, cool!

To access the server I've also created a free .tk domain and pointed it to the IP address of the server.

So far the easy part. And as it turned out the rest was easy too. A perforce installation used to be a pain in the ass because the perforce documentation is hard to find for the correct version, there is no clear path on their website to the correct documentation on any server version. It used to be copying binaries to your server, chmod +x them and start configuring. But then I happened to find this page and I literally did what it said:

rpm --import https://package.perforce.com/perforce.pubkey

vi /etc/yum.repos.d/perforce.repo
and copy pasted the repo config from the website:
[perforce]
name=Perforce
baseurl=http://package.perforce.com/yum/rhel/7/x86_64
enabled=1
gpgcheck=1

yum install helix-p4d

Done! So we list all services to see if there was a p4d instance running:

chkconfig --list

No p4d to be seen, but a "perforce-p4dctl", what is that? The best info I could find is this page. Basically it says: "documentation is in the man page" and that's it. That's so weird, why wouldn't they put the info also in their knowledge base? It would be so much easier! Ah well, back to the command line:

man p4dctl

Basically it's a service to manage multiple perforce instances with easy config files. Cool, way better than it used to be! So I created a .conf file according to the template in the /etc/perforce/p4dctl.conf.d folder. I created a /home/perforce folder for the P4ROOT and switched the ownership to the perforce user

mkdir /home/perforce

chown perforce:perforce /home/perforce

And then running a list gave me this output:

[root@alex-droplet-1 home]# p4dctl list
Type Owner Name Config
p4d perforce p4-ava port=1666 root=/home/perforce


Allright! Only one thing left to do:

p4dctl start p4-ava

Aaand done! Wow, this was easy!

Drawing game for mobile

In Kweetet there is a donkey in Initia (the main scene) called "schildersezel", which is Dutch for a painter's easel. "Ezel" is in dutch actually a donkey and it's a painting donkey, so there's the pun.

When you click on the sign above the donkey you start the drawing game of Kweetet, which you can use to create drawings that are displayed throughout the world, or that you even need to complete certain missions.

In this game you can either draw lines or use a flood fill. The code for drawing the lines is based on Aron Granberg's UnityPaint. For the floodfill I used an algorithm found on the internet, but I lost the original link.

I also used the floodfill algorithm for the lines; that way, when drawing lines, the color does not cross a black line, making it easier to stay within the lines for small children (and basically anyone who wants to draw between the lines with thick fingers on a tablet).

There is only one downside to Granberg's implementation: the algorithm has the texture in memory as an array of pixels which is manipulated by the algorithm and when done painting there is a call to SetPixels and Apply on the target texture, which copies the pixels from RAM to VRAM. Aside from being a memory hog, it worked in the webplayer and standalone, even on webgl. Some older devices were not capable to run it though. Let alone mobile devices. On mobile devices, when drawing lines the fps drops to +/- five. Uploading a texture from CPU to GPU memory is not very performant but you can get away with it on computers. On mobile devices on the other hand, it appears not.

We had a couple of students at the office who's task was to find a better solution and they suggested to use GL calls directly on the texture, so we wouldn't need to upload anything to VRAM.

The flood fill algorithm scans the texture in horizontal lines finding the left and right edges and coloring the pixels in between. So I changed this to drawing lines between the found left and right edges and using GL calls to draw the pixels in a render texture. The coloring code (with a Gaussian falloff) needed to be applied on the GPU, so we used a custom shader that executed the exact same rendering code, but now directly on the GPU.

Result: 60 fps all the time! Checkout this example.

Ps: I know the drawing in above example is probably copyrighted by Studio 100, but I love the childishly innocence of it so much that I use it anyway, it's too beautiful. They'll contact me if they want to have it removed.

The unity project of the example can be downloaded here.

Edit: I uploaded new versions of the webgl demo and the project, because I noticed that on some pc's and mobile devices, there was a problem with the fill algorithm. It appeared there was a one pixel offset on the top of the fill surface. After quite some searching I discovered that this is caused by a bug that has been reported and reproduced, but not yet fixed in Unity.

Edit: This project has now been integrated in my general UnityToolset project on bitbucket.

Forwarding trigger events

Sometimes when developing any kind of game you want to have a function called on an object A (not necessarily a GameObject) when object B hits a trigger defined by object C. For example, A can be your game controller, B your player's avatar and C a trigger next to a door. When the avatar walks into the trigger you want to call something like

GameController.Instance.LoadLevel("nameofthelevel");

So you write a "DoorTrigger" monobehaviour with a public string member for the name of the level that should be loaded when the avatar walks into the trigger. Perfect.

The things is that this is a very common behaviour; often you just want to call one simple function on another object when the player (or something else) runs into a trigger and often the trigger object and the callee for the function is not the same object. Writing a new class for every occasion is cumbersome.

In the UnityEngine.UI namespace we have the EventTrigger class that solves exactly this issue, but then for input events. When a ui button is clicked the code for doing something with that event is often not in a monobehaviour on that button, but likely in some PanelController class that controls the panel. So you use the button's functionality to forward the event to that controller. The EventTrigger class does the same thing for other ui objects than buttons.

So we thought to take a peek at the open source implementation of EventTrigger and see if we could mimic the same easy editor interaction for physics trigger events. We finally came up with a class called ForwardTrigger that can be found here. It is very similar to the code of EventTrigger but a lot less complicated, since we don't have to deal with InputModules but simple trigger events.

We introduced a PhysicsTriggerEventData class that tells you what object the trigger was and what object the collider was. We also copied with very little changes the code of the editor for that class so that we can now forward physics trigger events as easily as we could forward click events:

A working example is hosted on this bitbucket repository called "UnityToolset". This repository will become the place where I'll add all code that I frequently use in multiple Unity projects (as long as it is not under NDA) and also all code that I write about on my blog. I welcome all feedback on this project!

Unity crash dumps to the rescue!

My last post ended with me being disappointed about the state of our webgl build of Kweetet. I'm happy to say that I solved a lot of the remaining issues in the past holidays, against my expectations! The WebGL build might be a viable production target after all.

Nevertheless we still need a standalone build, because it's clear that WebGL is not fit for older systems. The memory usage has become less (around 1GB) but it's still a lot and we have a target audience that sometimes has only that.

Creating a standalone version went fast, i was done in a day, but that doesn't come as a surprise since standalone windows and webplayer builds are very alike.

A colleague of mine always had the weirdest crashes in the webplayer. It was always only on her pc and we could never reliably reproduce it. The log of the webplayer pointed in various directions, but we could never determine the cause of the crashes. Now with the standalone build she had the same crash and behold: there was a crash.dmp file! I've always made builds for mobile and web platforms so I had no idea that standalone build generated dump files. I opened it in Visual Studio immediately. I couldn't do much with it at first, but googling around brought me to this thread on the Unity forum where I learned that Unity has a symbol server! There you can download the pdb files for a standalone build, or even better, you can add the server to your debugger settings in Visual Studio:

Once you've done that you have an accurate callstack of the crash. The callstack was completely different than anything reported in the webplayer log files, but it explained completely every crash my colleague has had. It was an easy fix after that.

Trying to reproduce hard to find crashes in the webplayer or even the webgl builds with a standalone build can really pay off.