Host Unreal Engine 4 projects on Microsoft Azure DevOPS with unlimited cost-free Git LFS quota

Host Unreal Engine 4 projects on Microsoft Azure DevOPS with unlimited cost free Git LFS quota

Host Unreal Engine 4 projects on Microsoft Azure DevOPS with unlimited cost free Git LFS quota

UPDATE 1 [2021/07/25]: It seems that Git LFS is able to resume your pushes after a network failure. At least it’s like that on Microsoft Azure DevOPS. So, it should be totally redundant to divide huge commits into smaller ones. How have I noticed this? Today, I pushed a huge single commit (around 53GBs) and it failed at 39GB due to a connection error without me noticing it for some time. A few hours later, when I made another attempt by issuing the push command again, it picked up and resumed the push at 39GB, which was really exciting.

UPDATE 2 [2021/07/25]: After pushing the repository to Azure DevOPS, if you find your self stuck in git pull without doing anything, the following command will fix the consecutive pulls:

$ git pull origin master

Or, alternatively:

$ git fetch origin master
$ git reset --hard FETCH_HEAD

UPDATE 3 [2021/07/28]: I’ve noticed due to the fact that the files modification times affect how Rsync and Git work by default, my approach in writing the original script was totally wrong, which in turn caused a bug where on each update it committed all tracked files over again causing huge bloat in the repository, despite the fact that the content of the files was unchanged. Thus, it led me to completely rewrite the script. Hopefully, the new script has been extensively tested with two repositories/projects and works as expected. In addition to that, the script now shows progress for every step, which is a nice addition in order to keep you informed and give an estimation of the time it is going to take to get the job done. And, last but not least, I have edited and improved the blog post a bit.

UPDATE 4 [2021/08/04]: Due to nested .gitignore files inside the Unreal Engine dependencies, I noticed tiny bits of dependencies for building UE4/UE5 on Microsoft Windows are not getting copied over to the repository. As a result, I fixed the script in order to also take care of that.


Among the gamedev industry, it’s a well-known fact that Unreal Engine projects sizes have always been huge and a pain to manage properly. And it becomes more painful by the day as your project moves forward and grows in size. Some even keep the Engine source and its monstrous binary dependencies inside their source control management software. In case you are a AAA game development company or you are working for one, there’s probably some system in place with an unlimited quota to take care of that. But, for most of us indie devs, or individual hobbyists, it seems there are not lots of affordable options, especially that your team is scattered across the globe.

There are plenty of costly solutions to keep UE4 projects under source control; ranging from maintaining a local physical server or renting a VPS with plenty of space on the cloud, equipped with a self-hosted Git, SVN, or Perforce, to use cloud SCM providers such as GitHub, GitLab, BitBucket, or Perforce. Since I prefer cloud SCM providers and Git + Git LFS (which also supports file locking), let’s take a look at some popular ones such as GitHub and GitLab.

GitHub for one, provides data packs, but the free offering is far from enough for collaborative UE4 projects:

Every account using Git Large File Storage receives 1 GB of free storage and 1 GB a month of free bandwidth. If the bandwidth and storage quotas are not enough, you can choose to purchase an additional quota for Git LFS. Unused bandwidth doesn’t roll over month-to-month.

Additional storage and bandwidth is offered in a single data pack. One data pack costs $5 per month, and provides a monthly quota of 50 GB for bandwidth and 50 GB for storage. You can purchase as many data packs as you need. For example, if you need 150 GB of storage, you’d buy three data packs.

For GitLab, although the initial generous 10GB repository size is way beyond the 1GB repository size offer by GitHub, the LFS pricing is insanely high:

  1. Additional repository storage for a namespace (group or personal) is sold in annual subscriptions of $60 USD/year in increments of 10GB. This storage accounts for the size calculated from Repositories, which includes the git repository itself and any LFS objects.

  2. When adding storage to an existing subscription, you will be charged the prorated amount for the remaining term of your subscription. (ex. If your subscription ends in 6 months and you buy storage, you will be charge for 6 months of the storage subscription, i.e. $30 USD)

Well, before this all get you disappointed, let’s hear the good news from the Microsoft Azure DevOPS team:

In uncommon circumstances, repositories may be larger than 10GB. For instance, the Windows repository is at least 300GB. For that reason, we do not have a hard block in place. If your repository grows beyond 10GB, consider using Git-LFS, Scalar, or Azure Artifacts to refactor your development artifacts.

Before we proceed any further, there are some catches to consider about Microsoft Azure DevOPS:

1. The maximum Git repository size is 10GB, which considering that we keep binary assets and huge files in LFS, is way beyond any project’s actual needs. For Git LFS it seems that Microsoft since at least 2015 has been providing unlimited free storage. For comparison, the engine source code for 4.27 is 1.4GB, which in turn when it’s getting committed to the git repo becomes less than 230MB:

$ cd /path/to/ue4.27/source

$ du -h
1.4G	.

$ git init
$ git add .
$ git commit -m "add unreal engine 4.27 source code"

$ git count-objects -vH

count: 97545
size: 900.25 MiB
in-pack: 110815
packs: 1
size-pack: 227.80 MiB
prune-packable: 97545
garbage: 0
size-garbage: 0 bytes

2. The maximum push size is limited to 5GB at a time. The 5GB limit is only for files in the actual repository and it won’t affect LFS objects. Thus, there are no limits for LFS objects’ pushes. Despite that, if your internet connection is not stable, you could divide your files into multiple commits and push them separately. For example, the initial git dependencies for UE 4.27 is around 40GB spanned across ~70,000 files. Instead of committing and pushing a 40GB chunk all at once, one could divide that into multiple commits and push those commits one by one using the following command:

$ git rev-list --reverse master \
    | ruby -ne 'i ||= 0; i += 1; puts $_ if i % 1 == 0' \
    | xargs -I{} git push origin +{}:refs/heads/master

3. Sadly, at the moment Azure DevOPS does not support LFS over SSH. So, you are bound to git push/pull over https, which for some might be annoying. Especially, that it keeps asking for the https token 3 consecutive times on any push or pull!

Q: I’m using Git LFS with Azure DevOps Services and I get errors when pulling files tracked by Git LFS.

A: Azure DevOps Services currently doesn’t support LFS over SSH. Use HTTPS to connect to repos with Git LFS tracked files.

4. Last but not least, there is an issue with the Microsoft implementation of LFS, which rejects large LFS objects and spits out a bunch of HTTP 413 and 503 errors at the end of your git push. It happened to me when I was pushing 40GB of UE4 binary dependencies. The weird thing was I tried twice and both times it took a few good hours till the end of the push operation and based on measuring the bandwidth usage, the LFS upload size appeared to be more than the actual upload size. According to some answers on this GitHub issue and this Microsoft developer community question, it seems the solution is running the following command inside the root of your local repository, before any git pull/push operations:

$ git config http.version HTTP/1.1

Well, not only it did the trick and worked like a charm, but also the push time on the following git push dropped dramatically to 30 minutes for that hefty 40GB UE4 binary dependencies.

OK, after getting ourselves familiarized with all the limits, if you deem this solution a worthy one for managing UE4 projects along with the engine source in the same repository, in the rest of this blog post I’m going to share my experiences and a script to keep the engine updated with ease using a Git + LFS setup.

[Read More...]

HackYourFuture First Game Development Workshop Report

HackYourFuture First Game Development Workshop Report

HackYourFuture First Game Development Workshop Report

OK! Today, HackYourFuture held their first-ever gamedev workshop titled “Introduction to Unreal Engine 4 Game Development Using Blueprint” at their Amsterdam office, which I’ve been given the opportunity to present to their alumni on HackYourFuture’s behalf.

Since I’m still very excited and fascinated by the culture of sharing and the friendly environment governed by HackYourFuture, I decided to share the presentation file which I did prepare for this four hour workshop under the terms of (CC BY-SA 3.0) Creative Commons Attribution-ShareAlike 3.0 Unported License, to be used by anyone who is either new to game development or is intersted in the topic. It’s jam packed with so much information for absolute beginners. Feel free to share and adapt according to your own needs under the terms of the license.

The presentation can be found at the end of this post in PDF, OpenDocument Presentation, or Microsoft PowerPoint formats.

[Read More...]

Post-Processing Effects in Cocos2d-x

Picture A: Gray (black and white) post-processing effect in Cocos2d-x

Picture A: Gray (black and white) post-processing effect in Cocos2d-x

Update [2020/10/23]: Since I wrote this tutorial many things has changed in Cocos2d-x and I haven’t been using it in years as I switched to Unreal Engine 4. According to this GitHub issue opened by my blog readers, in order for this code to work there has to be some modifications to the code which is provided on that issue by @lazerfalcon and @madrazo. I was also having trouble with render to texture on Sprite3D which seems to have been solved in newer versions.

Despite the fact that Cocos2d-x uses the OpenGL ES Shading Language v1.0 for the shaders (in order to learn more about the language, please refer to: OpenGL ES Shading Language v1.0 Spec) the engine does not provide an out-of-the-box API to apply post-processing effects to the whole game scene output at once.

In this tutorial we will learn an easy way to apply multiple post-processing effects to the game scene output all at once.

[Read More...]

Get nFringe to Work with Visual Studio 2012

As you’ve noticed there’s still no official support for Microsoft Visual Studio 2012 in recent Pixel Mine nFringe releases. While ago I came across an awesome forum post at Epic Games Forums which describs a simple process of getting nFringe to work with VS2012. Since then I’ve used it in my day to day development tasks and I had no difficulties at all using it. And damn, it’s pretty stable despite the fact that not officially supported by Pixel Mine. Even nFringe version 1.1 which I’ve tested is playing nice with VS2012.

The only prerequisite that you need is your previous VS2010 + nFringe installation to obtain some files from it. Once you’ve acquired these files you don’t need VS2010 or nFringe installer for further installations.

1. Open Windows Command Prompt (cmd.exe) and run the following commands:

> xcopy /E "C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Pixel Mine" "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\Extensions\"
> xcopy /E "C:\Program Files (x86)\Microsoft Visual Studio 10.0\UnrealScript" "C:\Program Files (x86)\Microsoft Visual Studio 11.0\"

2. Open up extension.vsixmanifest in Notepad or your favorite editor and change VisualStudio Version to 11 (Note: In the following path change 1.1 with your nfringe version, e.g. 1.2).

> notepad "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\Extensions\Pixel Mine nFringe (UnrealScript)\1.1\extension.vsixmanifest"
extension.vsixmanifest
    <InstalledByMsi>true</InstalledByMsi>
    <SupportedProducts>
      <VisualStudio Version="11.0">

3. Run the following command to register nFringe extension in Visual Studio 2012:

> "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\devenv.exe" /setup

4. Finally, you need to re-validate your nFringe license in VS2012.

Happy coding ;)