Host Unreal Engine 4 projects on Microsoft Azure DevOPS with unlimited cost-free Git LFS quota

UPDATE 1 [2021/07/25]: It seems that Git LFS is able to resume your pushes after a network failure. At least it’s like that on Microsoft Azure DevOPS. So, it should be totally redundant to divide huge commits into smaller ones. How have I noticed this? Today, I pushed a huge single commit (around 53GBs) and it failed at 39GB due to a connection error without me noticing it for some time. A few hours later, when I made another attempt by issuing the push command again, it picked up and resumed the push at 39GB, which was really exciting.

UPDATE 2 [2021/07/25]: After pushing the repository to Azure DevOPS, if you find your self stuck in git pull without doing anything, the following command will fix the consecutive pulls:

$ git pull origin master

Or, alternatively:

$ git fetch origin master
$ git reset --hard FETCH_HEAD

UPDATE 3 [2021/07/28]: I’ve noticed due to the fact that the files modification times affect how Rsync and Git work by default, my approach in writing the original script was totally wrong, which in turn caused a bug where on each update it committed all tracked files over again causing huge bloat in the repository, despite the fact that the content of the files was unchanged. Thus, it led me to completely rewrite the script. Hopefully, the new script has been extensively tested with two repositories/projects and works as expected. In addition to that, the script now shows progress for every step, which is a nice addition in order to keep you informed and give an estimation of the time it is going to take to get the job done. And, last but not least, I have edited and improved the blog post a bit.

UPDATE 4 [2021/08/04]: Due to nested .gitignore files inside the Unreal Engine dependencies, I noticed tiny bits of dependencies for building UE4/UE5 on Microsoft Windows are not getting copied over to the repository. As a result, I fixed the script in order to also take care of that.

UPDATE 5 [2021/11/30]: Sometimes it’s possible that the amount of renamed Unreal Engine files surpass the Git’s optimal rename limit inside the Sync repository (the intermediary local git repository that we are going to use for syncing the engine source code with upstream):

warning: exhaustive rename detection was skipped due to too many files.
warning: you may want to set your diff.renameLimit variable to at least 13453 and retry the command.

So, you could set that to a really large number in order to keep track of file renames:

$ cd ~/dev/MamadouArchives-Sync
$ git config diff.renameLimit 999999
$ git config merge.renameLimit 999999

Note: You will get this warning only when the Git option diff.renames is set to true (default behavior). Likewise, the above settings does not have any effects when the copy/rename detection is turned off. Nonetheless, you can always check your settings with:

$ git config -l

UPDATE 6 [2021/12/18]: I’ve added a step regarding EngineAssociation in the project’s .uproject file, which I forgot to mention in the original post.

UPDATE 7 [2023/03/03]: In UE5 UE4Games.uprojectdirs file shas been renamed to Default.uprojectdirs. Though the syntax and the contents of the file has remained the same.

UPDATE 8 [2023/03/04]: After upgrading my project to Unreal Engin 5.1 despite the fact that I’ve already set the git configuration http.version to HTTP/1.1 as instructed in this article, despite the commit size of no bigger than 166.30 MB and the acceptable upload bandwidth I’ve got, I was getting HTTP 413 Request Entity Too Large error:

Enumerating objects: 190058, done.
Counting objects: 100% (164439/164439), done.
Delta compression using up to 16 threads
Compressing objects: 100% (113439/113439), done.
Writing objects: 100% (138834/138834), 166.30 MiB | 47.32 MiB/s, done.
Total 138834 (delta 35613), reused 121343 (delta 22206), pack-reused 0
error: RPC failed; HTTP 413 curl 22 The requested URL returned error: 413
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly
Everything up-to-date

I tried every suggestion that I came across in order to debug and resolve the issue to no avail. Including enabling git verbose logging:

$ export GIT_TRACE_PACKET=1                      
$ export GIT_TRACE=1
$ export GIT_CURL_VERBOSE=1

And then maxing out on all the size limits, buffer, packet sizes, and other hints:

[core]
	compression = 0
    packedGitLimit = 512m
    packedGitWindowSize = 512m
[http]
    postBuffer = 2147483648
[https]
    postBuffer = 2147483648
[init]
    defaultBranch = master
[pack] 
    deltaCacheSize = 2047m
    packSizeLimit = 2047m
	window = 1
    windowMemory = 2047m

Then I tried to change the origin URL to SSH:

[remote "origin"]
	url = https://SOME-ORGANIZATION@dev.azure.com/SOME-ORGANIZATION/MamadouArchives/_git/MamadouArchives
	#url = git@ssh.dev.azure.com:v3/SOME-ORGANIZATION/MamadouArchives/MamadouArchives

And then pushing without LFS:

$ git push --set-upstream origin 5.1 --no-verify

This attempt was futile as well, that made me revert back to https. Then I tried to push commit by commit since I had made a few commits using (5.1 which has been repeated twice in the following command, is the name of the new local branch intended to be pushed):

$ git rev-list --reverse 5.1 | ruby -ne 'i ||= 0; i += 1; puts $_ if i % 1 == 0' | xargs -I{} git push origin +{}:refs/heads/5.1 --no-verify

And sadly, the approach of pushing one commit at a time was frutiless as well :/

Thus, for the time being I’m stuck pushing the updated project from my Linux machine and pulling it from my Windows machine. I’ll do another updated once I’ve figured what’s going wrong.

UPDATE 9 [2023/03/04]: As an experiment, I did create a new organiation and a new repository inside it. Then prior to changing the origin URL, I fetched all LFS objects issuing:

$ git lfs fetch --all
fetch: 78618 objects found, done.                                                                      
fetch: Fetching all references...

Then, I decided to first push the Git comnmits without the LFS objects, so after updating the origin URL inside the .git/config:

$ git push --set-upstream origin 5.1 --no-verify
Enumerating objects: 296578, done.
Counting objects: 100% (296578/296578), done.
Delta compression using up to 16 threads
Compressing objects: 100% (210384/210384), done.
Writing objects: 100% (296578/296578), 360.11 MiB | 5.78 MiB/s, done.
Total 296578 (delta 78143), reused 296578 (delta 78143), pack-reused 0
remote: Analyzing objects... (296578/296578) (77738 ms)
remote: Storing packfile... done (10303 ms)
remote: Storing index... done (3740 ms)
To https://dev.azure.com/SOME-ORGANIZATION/MamadouArchives/_git/MamadouArchives
 * [new branch]            5.1 -> 5.1
branch '5.1' set up to track 'origin/5.1'

¯\_(ツ)/¯ as unexpected as it seems, it worked! As you can see my actual Git objects without the LFS objects on this repository are in no way near the 10 GB size limit:_

$ git count-objects -vH                         
count: 0               
size: 0 bytes
in-pack: 296583
packs: 1
size-pack: 366.76 MiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

The issue might be that I’ve reached some kind of limit on the main organization that I’m not aware of.

Anyways, then I pushed the master and checked out back the new brnach for continuation on the upgrade:

$ git checkout master
$ git push origin master --no-verify
$ git checkout 5.1

And, then proceeded to pushing all LFS objects:

$ git lfs push origin --all

UPDATE 10 [2023/03/05]: Yesterday, I removed a large redundant repository from the previous organization, in order to see if I could still push my updates and the error I am getting was not due to hitting some kind of ceiling limit. It didn’t work. I did also cleanup the limit hacks I’ve added to my ~/.gitconfig in UPDATE 8. Then, after successfully pushing to the new organization/repository, I’ve decided to revert back the URL section for the orgin inside my .git/config inside the local repository and try to push once more to the old repository and guess what? It worked! Weird Microsoft/Azure! Not sure what fixed the issue. It could be even I had to wait for Microsoft to clean up the repository’s space I’ve deleted if the organization size limit was the issue. Don’t really know.

Among the gamedev industry, it’s a well-known fact that Unreal Engine projects sizes have always been huge and a pain to manage properly. And it becomes more painful by the day as your project moves forward and grows in size. Some even keep the Engine source and its monstrous binary dependencies inside their source control management software. In case you are a AAA game development company or you are working for one, there’s probably some system in place with an unlimited quota to take care of that. But, for most of us indie devs, or individual hobbyists, it seems there are not lots of affordable options, especially that your team is scattered across the globe.

There are plenty of costly solutions to keep UE4 projects under source control; ranging from maintaining a local physical server or renting a VPS with plenty of space on the cloud, equipped with a self-hosted Git, SVN, or Perforce, to use cloud SCM providers such as GitHub, GitLab, BitBucket, or Perforce. Since I prefer cloud SCM providers and Git + Git LFS (which also supports file locking), let’s take a look at some popular ones such as GitHub and GitLab.

GitHub for one, provides data packs, but the free offering is far from enough for collaborative UE4 projects:

Every account using Git Large File Storage receives 1 GB of free storage and 1 GB a month of free bandwidth. If the bandwidth and storage quotas are not enough, you can choose to purchase an additional quota for Git LFS. Unused bandwidth doesn’t roll over month-to-month.

…

Additional storage and bandwidth is offered in a single data pack. One data pack costs $5 per month, and provides a monthly quota of 50 GB for bandwidth and 50 GB for storage. You can purchase as many data packs as you need. For example, if you need 150 GB of storage, you’d buy three data packs.

For GitLab, although the initial generous 10GB repository size is way beyond the 1GB repository size offer by GitHub, the LFS pricing is insanely high:

1. Additional repository storage for a namespace (group or personal) is sold in annual subscriptions of $60 USD/year in increments of 10GB. This storage accounts for the size calculated from Repositories, which includes the git repository itself and any LFS objects.

2. When adding storage to an existing subscription, you will be charged the prorated amount for the remaining term of your subscription. (ex. If your subscription ends in 6 months and you buy storage, you will be charge for 6 months of the storage subscription, i.e. $30 USD)

Well, before this all get you disappointed, let’s hear the good news from the Microsoft Azure DevOPS team:

In uncommon circumstances, repositories may be larger than 10GB. For instance, the Windows repository is at least 300GB. For that reason, we do not have a hard block in place. If your repository grows beyond 10GB, consider using Git-LFS, Scalar, or Azure Artifacts to refactor your development artifacts.

Before we proceed any further, there are some catches to consider about Microsoft Azure DevOPS:

1. The maximum Git repository size is 10GB, which considering that we keep binary assets and huge files in LFS, is way beyond any project’s actual needs. For Git LFS it seems that Microsoft since at least 2015 has been providing unlimited free storage. For comparison, the engine source code for 4.27 is 1.4GB, which in turn when it’s getting committed to the git repo becomes less than 230MB:

$ cd /path/to/ue4.27/source

$ du -h
1.4G	.

$ git init
$ git add .
$ git commit -m "add unreal engine 4.27 source code"

$ git count-objects -vH

count: 97545
size: 900.25 MiB
in-pack: 110815
packs: 1
size-pack: 227.80 MiB
prune-packable: 97545
garbage: 0
size-garbage: 0 bytes

2. The maximum push size is limited to 5GB at a time. The 5GB limit is only for files in the actual repository and it won’t affect LFS objects. Thus, there are no limits for LFS objects’ pushes. Despite that, if your internet connection is not stable, you could divide your files into multiple commits and push them separately. For example, the initial git dependencies for UE 4.27 is around 40GB spanned across ~70,000 files. Instead of committing and pushing a 40GB chunk all at once, one could divide that into multiple commits and push those commits one by one using the following command:

$ git rev-list --reverse master \
    | ruby -ne 'i ||= 0; i += 1; puts $_ if i % 1 == 0' \
    | xargs -I{} git push origin +{}:refs/heads/master

3. Sadly, at the moment Azure DevOPS does not support LFS over SSH. So, you are bound to git push/pull over https, which for some might be annoying. Especially, that it keeps asking for the https token 3 consecutive times on any push or pull!

Q: I’m using Git LFS with Azure DevOps Services and I get errors when pulling files tracked by Git LFS.

A: Azure DevOps Services currently doesn’t support LFS over SSH. Use HTTPS to connect to repos with Git LFS tracked files.

4. Last but not least, there is an issue with the Microsoft implementation of LFS, which rejects large LFS objects and spits out a bunch of HTTP 413 and 503 errors at the end of your git push. It happened to me when I was pushing 40GB of UE4 binary dependencies. The weird thing was I tried twice and both times it took a few good hours till the end of the push operation and based on measuring the bandwidth usage, the LFS upload size appeared to be more than the actual upload size. According to some answers on this GitHub issue and this Microsoft developer community question, it seems the solution is running the following command inside the root of your local repository, before any git pull/push operations:

$ git config http.version HTTP/1.1

Well, not only it did the trick and worked like a charm, but also the push time on the following git push dropped dramatically to 30 minutes for that hefty 40GB UE4 binary dependencies.

OK, after getting ourselves familiarized with all the limits, if you deem this solution a worthy one for managing UE4 projects along with the engine source in the same repository, in the rest of this blog post I’m going to share my experiences and a script to keep the engine updated with ease using a Git + LFS setup.

OK! Let’s get started

OK, in this tutorial I assume we are going to create a new project from scratch. Migration is a similar story if one learns how to set up a Git LFS repository from scratch. We briefly go over that at the end.

In the rest of this tutorial, I will name the new project MamadouArchives.

Requirements

The script that I am going to represent in this tutorial is a Bash shell script and runs on Linux. I haven’t tested it on Git Bash for Windows. It may work on macOS or Microsoft Windows or not. In fact, you don’t even need this script for following this tutorial. It could all be done manually, or it could be automated using your favorite automation scripting language once you know what are you doing.

In order to effectively follow this tutorial and use the script, you may need the following:

• Bash (Tested on 5.1, but it probably works on 4.x, too)
• Git + LFS
• rsync
• A github account correctly setup with access to the Unreal Engine repository on Epic’s official GitHub account
• A Microsoft Azure DevOPS account

The directory structure

MamadouArchives (the project name containing the engine source + binary dependencies + the actual UE4 project)
  │
  ├── .git/ (Git metadata and objects database directory)
  │     │
  │     └── *
  │
  ├── Engine/
  │     │
  │     └── *
  │
  ├── FeaturePacks/
  │     │
  │     └── *
  │
  ├── MamadouArchives/ (your actual UE4 project sits here)
  │     │
  │     ├── Config/
  │     │     │
  │     │     └── *
  │     │
  │     ├── Content/
  │     │     │
  │     │     └── *
  │     │
  │     ├── Plugins/
  │     │     │
  │     │     └── *
  │     │
  │     ├── Source/
  │     │     │
  │     │     └── *
  │     │
  │     ├── .gitignore (your project's .gitignore, which overrides the the engine one)
  │     │
  │     ├── MamadouArchives.uproject
  │     │
  │     └── README.md
  │
  ├── Samples/
  │     │
  │     └── *
  │
  ├── Templates/
  │     │
  │     └── *
  │
  ├── .editorconfig
  │
  ├── .gitattributes (used by Git LFS to keep track of the LFS objects)
  │
  ├── .gitignore
  │
  ├── .tgitconfig
  │
  ├── .ue4dependencies
  │
  ├── cpp.hint
  │
  ├── GenerateProjectFiles.bat
  │
  ├── GenerateProjectFiles.command
  │
  ├── GenerateProjectFiles.sh
  │
  ├── LICENSE.md
  │
  ├── README.md
  │
  ├── Setup.bat
  │
  ├── Setup.command
  │
  ├── Setup.sh
  │
  ├── UE4Games.uprojectdirs (scanned by UnrealBuildTool in order to find projects)
  │
  └── UpdateEngine.sh (the engine update script from this tutorial)

Three copies of the Engine

OK, in order to keep things less complex and as easy as we could, we need three copies of the engine (I use the actual paths on my computer):

• /opt/UnrealEngine: the untouched engine source from Epic Games GitHub account which tracks the Unreal Engine development.
• ~/dev/MamadouArchives: the actual engine and project directory I build and work with daily. One should never ever run the engine update script from this directory, or there may be hazardous effects ¯\_(ツ)_/¯. See the next one for more details.
• ~/dev/MamadouArchives-Sync: due to using rsync since the script has destructive effects on any uncommitted changes, or is cleaning up the engine builds, we use this intermediary copy of our project which tracks our project’s development. We run the engine update script only from this directory and git pull/push this repository in order to keep it in sync with the actual project /dev/MamadouArchives.

Note: Running the engine update script gives you the following warning before actually doing anything. It’s safe until you press y. And, by default the answer is no:

WARNING: RUNNING THIS SCRIPT MAY HAVE DESTRUCTIVE EFFECTS
ON THIS REPOSITORY!

MAKE A BACKUP FIRST, BEFORE YOU PROCEED ANY FURTHER!

Are you sure? [y/N] 

The UE4 update script

Here is the aforementioned engine update script we are going to use in a moment. Just remember we have to modify a few variables later on.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166

#!/usr/bin/env bash

#  Author: Mamadou Babaei <info@babaei.net>
#
#  (The MIT License)
#
#  Copyright (c) 2019 - 2021 Mamadou Babaei
#  Copyright (c) 2021 @mamadou_archive
#  Copyright (c) 2019 - 2021 @Barandazstorm
#
#  Permission is hereby granted, free of charge, to any person obtaining a copy
#  of this software and associated documentation files (the "Software"), to deal
#  in the Software without restriction, including without limitation the rights
#  to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
#  copies of the Software, and to permit persons to whom the Software is
#  furnished to do so, subject to the following conditions:
#
#  The above copyright notice and this permission notice shall be included in
#  all copies or substantial portions of the Software.
#
#  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
#  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
#  FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
#  AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
#  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
#  OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
#  THE SOFTWARE.

set -e

readonly UPSTREAM_ENGINE_DIR="/opt/UnrealEngine"
readonly PROJECT_DIR_NAME="MamadouArchives"
readonly UPDATE_SCRIPT_NAME="$(basename "$0")"
readonly PROJECT_ENGINE_DIR="$(dirname $(realpath "$0"))"
readonly GIT_ATTRIBUTES_FILE_NAME=".gitattributes"
readonly GIT_IGNORE_FILE_NAME=".gitignore"
readonly GIT_DIR_NAME=".git"
readonly UE4_GAMES_UPROJECTDIRS_FILE_NAME="UE4Games.uprojectdirs"
readonly PROJECT_ENGINE_GIT_ATTRIBUTES_FILE="${PROJECT_ENGINE_DIR}/${GIT_ATTRIBUTES_FILE_NAME}"

function displayWarning() {
    echo ""
    echo "WARNING: RUNNING THIS SCRIPT MAY HAVE DESTRUCTIVE EFFECTS"
    echo "ON THIS REPOSITORY!"
    echo ""
    echo "MAKE A BACKUP FIRST, BEFORE YOU PROCEED ANY FURTHER!"
    echo ""

    read -p "Are you sure? [y/N] " -n 1 -r

    echo ""
    echo ""

    if [[ ! $REPLY =~ ^[Yy]$ ]] ;
    then
        echo "Aborted!"

        [[ "$0" = "$BASH_SOURCE" ]] && exit 1 || return 1
    fi

    update
}

function update() {
    echo "Cleaning up the upstream engine '${UPSTREAM_ENGINE_DIR}'..." \
        && echo "" \
        && cd "${UPSTREAM_ENGINE_DIR}" \
        && git clean -dfx \
        && git reset --hard \
        && echo "" \
        && echo "Updating the upstream engine '${UPSTREAM_ENGINE_DIR}'..." \
        && echo "" \
        && cd "${UPSTREAM_ENGINE_DIR}" \
        && git pull \
        && bash Setup.sh \
        && echo "" \
        && echo "Syncing the ${PROJECT_DIR_NAME} engine '${PROJECT_ENGINE_DIR}' with upstream '${UPSTREAM_ENGINE_DIR}'..." \
        && echo "" \
        && cd "${UPSTREAM_ENGINE_DIR}" \
        && rsync --checksum --delete-after --ignore-errors \
            --ignore-times --links --perms --recursive --verbose \
            --exclude="/${GIT_DIR_NAME}" \
            --exclude="/${UE4_GAMES_UPROJECTDIRS_FILE_NAME}" \
            --exclude="/${UPDATE_SCRIPT_NAME}" \
            --exclude="/${PROJECT_DIR_NAME}" \
            "${UPSTREAM_ENGINE_DIR}/" "${PROJECT_ENGINE_DIR}/" \
        && echo "" \
        && echo "Extracting engine's git dependencies list from '${UPSTREAM_ENGINE_DIR}'..." \
        && echo "" \
        && cd "${UPSTREAM_ENGINE_DIR}" \
        && for GIT_IGNORE_FILE in $(find . -type f -name "${GIT_IGNORE_FILE_NAME}"); do \
            rm -f "${GIT_IGNORE_FILE}"; \
            done \
        && ENGINE_DEPS_STR=$(git ls-files -z --others --exclude-standard | xargs -0 -I %s -r printf '[[ "%s" != "" ]] && printf "%s\n";\0' | xargs -0 -n1 -r bash -c 2> /dev/null) \
        && IFS=$'\n' && ENGINE_DEPS=(${ENGINE_DEPS_STR}) && unset IFS \
        && unset ENGINE_DEPS_STR \
        && ENGINE_DEPS_COUNT="${#ENGINE_DEPS[@]}" \
        && echo "" \
        && echo "Tracking engine's git dependencies as LFS objects..." \
        && echo "" \
        && echo "# UE4 Git Dependencies" >> "${PROJECT_ENGINE_GIT_ATTRIBUTES_FILE}" \
        && for i in "${!ENGINE_DEPS[@]}"; do \
            [[ -z "${ENGINE_DEPS[${i}]}" ]] && continue; \
            echo "Tracking $((${i}+1))/${ENGINE_DEPS_COUNT} '${ENGINE_DEPS[${i}]}'..."; \
            echo "\"${ENGINE_DEPS[${i}]}\" filter=lfs diff=lfs merge=lfs -text" \
                >> "${PROJECT_ENGINE_GIT_ATTRIBUTES_FILE}"; \
            done \
        && echo "" \
        && echo "Extracting engine's changelist from '${PROJECT_ENGINE_DIR}'..." \
        && echo "" \
        && cd "${PROJECT_ENGINE_DIR}" \
        && ENGINE_CHANGELIST_STR=$(git diff -z --name-only | xargs -0 -I %s -r printf '[[ "%s" != "" ]] && printf "%s\n";\0' | xargs -0 -n1 -r bash -c 2> /dev/null) \
        && IFS=$'\n' && ENGINE_CHANGELIST=(${ENGINE_CHANGELIST_STR}) && unset IFS \
        && unset ENGINE_CHANGELIST_STR \
        && ENGINE_CHANGELIST_UNTRACKED_STR=$(git ls-files -z --others --exclude-standard | xargs -0 -I %s -r printf '[[ "%s" != "" ]] && printf "%s\n";\0' | xargs -0 -n1 -r bash -c 2> /dev/null) \
        && IFS=$'\n' && ENGINE_CHANGELIST_UNTRACKED=(${ENGINE_CHANGELIST_UNTRACKED_STR}) && unset IFS \
        && unset ENGINE_CHANGELIST_UNTRACKED_STR \
        && ENGINE_CHANGELIST=("${ENGINE_CHANGELIST[@]}" "${ENGINE_CHANGELIST_UNTRACKED[@]}") \
        && unset ENGINE_CHANGELIST_UNTRACKED \
        && ENGINE_CHANGELIST_COUNT="${#ENGINE_CHANGELIST[@]}" \
        && echo "" \
        && echo "Staging engine's changes inside '${PROJECT_ENGINE_DIR}'..." \
        && echo "" \
        && for i in "${!ENGINE_CHANGELIST[@]}"; do \
            [[ -z "${ENGINE_CHANGELIST[${i}]}" ]] && continue; \
            echo "Staging $((${i}+1))/${ENGINE_CHANGELIST_COUNT} '${ENGINE_CHANGELIST[${i}]}'..."; \
            git add -f "${ENGINE_CHANGELIST[${i}]}" &> /dev/null; \
            done \
        && unset ENGINE_CHANGELIST \
        && unset ENGINE_CHANGELIST_COUNT \
        && echo "" \
        && echo "Staging engine's git dependencies inside '${PROJECT_ENGINE_DIR}'..." \
        && echo "" \
        && cd "${PROJECT_ENGINE_DIR}" \
        && for i in "${!ENGINE_DEPS[@]}"; do \
            [[ -z "${ENGINE_DEPS[${i}]}" ]] && continue; \
            echo "Staging LFS object $((${i}+1))/${ENGINE_DEPS_COUNT} '${ENGINE_DEPS[${i}]}'..."; \
            git add -f "${ENGINE_DEPS[${i}]}" &> /dev/null; \
            done \
        && unset ENGINE_DEPS \
        && unset ENGINE_DEPS_COUNT \
        && echo "" \
        && echo "Cleaning up the upstream engine '${UPSTREAM_ENGINE_DIR}'..." \
        && echo "" \
        && cd "${UPSTREAM_ENGINE_DIR}" \
        && git clean -dfx \
        && git reset --hard \
        && echo "" \
        && echo "Counting staged files inside '${PROJECT_ENGINE_DIR}'..." \
        && echo "" \
        && cd "${PROJECT_ENGINE_DIR}" \
        && STAGED_FILES_COUNT=$(git diff --name-only --cached | wc -l 2> /dev/null) \
        && echo "${STAGED_FILES_COUNT} file(s) have been staged and ready to be committed!" \
        && unset STAGED_FILES_COUNT \
        && echo "" \
        && echo "Done!" \
        && echo "" \
        && echo "In order to delete old LFS files from local storage, thus" \
        && echo "freeing up space, after commiting and pushing your changes," \
        && echo "do not forget to run:" \
        && echo "" \
        && echo "   $ git lfs prune --verbose --verify-remote" \
        && echo ""
}

displayWarning

The steps

1. Clone the Unreal Engine source code from the official GitHub repository. Here we check out the 4.27 branch:

$ git clone git@github.com:EpicGames/UnrealEngine.git -b 4.27 /opt/UnrealEngine

2. Create your project’s git repository on Microsoft Azure DevOPS and generate your https access token.

3. Prepare the initial intermediary local git repository we are going to use for syncing the engine source code with upstream:

$ rsync -arv --delete-before \
    --exclude='/.git' \
    /opt/UnrealEngine/ /home/mamadou/dev/MamadouArchives-Sync/

WARNING: Never run this command ever again after this step!

4. Initialize your project’s git repository, commit the engine source, and push it to Microsoft Azure DevOPS. As we mentioned earlier we have to use https instead of ssh and you might be asked for password 3 times:

$ cd ~/dev/MamadouArchives-Sync/
$ git init
$ git remote add origin https://SOME-ORGANIZATION@dev.azure.com/SOME-ORGANIZATION/MamadouArchives/_git/MamadouArchives
$ git add -f .
$ git commit -m "add unreal engine 4.27 source code"
$ git push -u origin --all

Note: In case one needs to store their git credentials on their computer they could simply do:

$ git config credential.helper store

This will add the following to .git/config:

[credential]
	helper = store

Upon your next pull/push, after typing your password once, it will be saved in plain text inside ~/.git-credentials in the following format:

https://SOME-ORGANIZATION:PASSWORD@dev.azure.com

WARNING: Although the permission for this file is set to 0600 by Git, which means only the owner can read/write it. This is not a recommended approach for saving Git credentials, at all! Any malicious program running by the user owning ~/.git-credentials could steal your Git credentials.

$ ls -lah ~/.git-credentials
-rw------- 1 mamadou mamadou 88 Jul 17 09:18 /home/mamadou/.git-credentials

There are better ways to store Git credentials, but, it is out of the scope of this article.

5. Add the script from the previous section to the root of the repository, and modify the following variables with your project directory name inside the root directory and the Epic Game’s untouched engine source path:

1
2

readonly UPSTREAM_ENGINE_DIR="/opt/UnrealEngine"
readonly PROJECT_DIR_NAME="MamadouArchives"

For example in the above, Epic Game’s untouched engine source code is inside /opt/UnrealEngine on my computer. For the project’s directory name I am going to create my project inside ~/dev/MamadouArchives-Sync/ with the name MamadouArchives. So, it will be located here on my computer: ~/dev/MamadouArchives-Sync/MamadouArchives.

6. Make the script executable, then stage, commit, and push the script to the remote repository on Microsoft Azure DevOPS:

$ chmod a+x UpdateEngine.sh
$ git add UpdateEngine.sh
$ git commit -m "add engine update script"
$ git push

7. Prepare and initialize Git LFS for our repository. And do not forget to run the HTTP/1.1 fix for large LFS objects on the repository:

$ git lfs install
$ git config http.version HTTP/1.1

This will add the following to .git/config:

[http]
	version = HTTP/1.1

8. Now, in order to fetch engine’s binary dependencies, run the script by issuing either:

$ cd ~/dev/MamadouArchives-Sync/
$ ./UpdateEngine.sh

Or (because the script is aware of how to find its repository path, one code safely run it like this):

$ ~/dev/MamadouArchives-Sync/UpdateEngine.sh

9. It may take a long time for the script to finish. If everything goes fine, we can commit and push engine’s binary dependencies. Note that the script does not so automatically because I prefer to review it before committing or pushing it. But, the script stage the retrieved new or updated files for you automatically. So:

$ git status           # in order to see the changes
$ git lfs ls-files -s  # in order to list LFS tracked files with size
$ git lfs track        # another way to list LFS tracked files
$ git commit -m "add unreal engine 4.27 binary dependencies"
$ git push

10. Form now on, you can always follow steps #8 #9 in order to keep your engine updated.

11. Create a UE4 project with the desired name, in my case MamadouArchives, or copy over an already existing project (without the .git directory if it’s already hosted on git) into your newly created repository. e.g. ~/dev/MamadouArchives-Sync/.

12. Make sure to set EngineAssociation to "". This is of chief importance!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

{
	"FileVersion": 3,
	"EngineAssociation": "",
	"Category": "",
	"Description": "",
	"Modules": [
		{
			"Name": "MamadouArchives",
			"Type": "Runtime",
			"LoadingPhase": "Default"
		}
	]
}

13. Now, it’s time to adjust Git LFS for our project. In order to touch less engine files, we are not going to touch engine’s .gitattributes located at ~/dev/MamadouArchives-Sync/.gitattributes. Instead, we could override this file by creating the exact same file in our project’s directory, e.g. ~/dev/MamadouArchives-Sync/MamadouArchives/.gitattributes. If you are going to do so, keep in mind that from now on you must avoid tracking files by running git lfs track “pattern or file path” in the engine directory ~/dev/MamadouArchives-Sync, because it will probably get truncated by the engine update script in consecutive runs. Thus, etither run this command in the project directory ~/dev/MamadouArchives-Sync/MamadouArchives or it’s subdirectories, and after that by issuing cat ~/dev/MamadouArchives-Sync/MamadouArchives/.gitattributes always make sure your pattern is being tracked. Or, you code modify ~/dev/MamadouArchives-Sync/MamadouArchives/.gitattributes directly. Here is a sample .gitattributes, which overrides or adds to engine’s .gitattributes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

# Keep CRLF out of the repository
* text=auto eol=lf

# In case the git binary/text auto-detection algorithm fails, in order to avoid
# line-ending normalization, treat these files as binary no matter what
*.{uasset,umap,3ds,fbx,flac,gif,jif,jiff,jpe,jpeg,jpg,m4a,mp3,mp4,ogg,png,psd,ttf,wav,webp,xcf} binary

# LFS: UE4 file types
*.uasset filter=lfs diff=lfs merge=lfs -text
*.umap filter=lfs diff=lfs merge=lfs -text

# LFS: raw content types
*.3ds filter=lfs diff=lfs merge=lfs -text
*.fbx filter=lfs diff=lfs merge=lfs -text
*.flac filter=lfs diff=lfs merge=lfs -text
*.gif filter=lfs diff=lfs merge=lfs -text
*.jif filter=lfs diff=lfs merge=lfs -text
*.jiff filter=lfs diff=lfs merge=lfs -text
*.jpe filter=lfs diff=lfs merge=lfs -text
*.jpeg filter=lfs diff=lfs merge=lfs -text
*.jpg filter=lfs diff=lfs merge=lfs -text
*.m4a filter=lfs diff=lfs merge=lfs -text
*.mp3 filter=lfs diff=lfs merge=lfs -text
*.mp4 filter=lfs diff=lfs merge=lfs -text
*.ogg filter=lfs diff=lfs merge=lfs -text
*.png filter=lfs diff=lfs merge=lfs -text
*.psd filter=lfs diff=lfs merge=lfs -text
*.ttf filter=lfs diff=lfs merge=lfs -text
*.wav filter=lfs diff=lfs merge=lfs -text
*.webp filter=lfs diff=lfs merge=lfs -text
*.xcf filter=lfs diff=lfs merge=lfs -text

# LFS: everything under Content/
Content/**/* filter=lfs diff=lfs merge=lfs -text

# LFS: every asset source under RawContent/
RawContent/**/* filter=lfs diff=lfs merge=lfs -text

# LFS: all third-party libraries
ThirdParty/**/* filter=lfs diff=lfs merge=lfs -text

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124

# Explcitly ignore annoying Windows files, regardless of where they are
[Dd]esktop.ini
Thumbs.db

# Explcitly ignore annoying macOS files, regardless of where they are
.DS_Store

# Makefile; only in the project root
/Makefile

# CMake; only in the project root
/CMakeLists.txt

# QMake / Qt Creator
*.pro
*Config.pri
*Defines.pri
*Header.pri
*Includes.pri
*Source.pri
.qmake.stash
*.autosave
*.pro.user
*.pro.user.*
CMakeLists.txt.user
CMakeLists.txt.user.*

# Visual Studio
.vs/
*.sln
*.suo
*.opensdf
*.sdf
*.VC.db
*.VC.opendb

# Visual Studio Code
.vscode/
*.code-workspace

# Code::Blocks
*.workspace

# CodeLite
*CodeLitePreProcessor.txt

# KDevelop
.kdev4/
*.kdev4

# CLion
*CodeCompletionFolders.txt

# IntelliJ IDE
.idea/

# Xcode
*.xcodeproj
*.xcworkspace

# Compiled Object files
*.slo
*.lo
*.o
*.obj

# Precompiled Headers
*.gch
*.pch

# Compiled Dynamic libraries
*.so
*.dylib
*.dll

# Fortran module files
*.mod

# Compiled Static libraries
*.lai
*.la
*.a
*.lib

# Executables
*.exe
*.out
*.app
*.ipa

# Precompiled Assets
SourceArt/**/*.png
SourceArt/**/*.tga

# Binary Files
Binaries/*
Plugins/*/Binaries/*

# Builds
Build/*

# Whitelist PakBlacklist-<BuildConfiguration>.txt files
!Build/*/
Build/*/**
!Build/*/PakBlacklist*.txt

# Don't ignore icon files in Build
!Build/**/*.ico

# Built data for maps
*_BuiltData.uasset

# Configuration files generated by the Editor
Saved/*

# Compiled source files for the engine to use
Intermediate/*
Plugins/*/Intermediate/*

# Cache files for the editor to use
DerivedDataCache/*

# Misc
.ignore

$ git add .gitattributes
$ git commit -m "track even more lfs objects"
$ git push

15. Well done! Now, we need to take one more step in order to enable the engine to pick up our project automatically at the time of generating the project files and building the engine, so that our project is going to be built along with the engine. Open the UE4Games.uprojectdirs file in the root of our repository:

1
2
3
4
5
6
7
8
9

; These folders will be searched 1 level deep in order to find projects
; UnrealBuildTool will store the following information:
; - Project name
; - Location of project
; - Whether it has code or not
; - TargetNames contains at the project location
;
./
Engine/Source/

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

; These folders will be searched 1 level deep in order to find projects
; UnrealBuildTool will store the following information:
; - Project name
; - Location of project
; - Whether it has code or not
; - TargetNames contains at the project location
;
./
Engine/Source/
MamadouArchives/

$ git add -f UE4Games.uprojectdirs
$ git commit -m "add MamadouArchives to UE4Games.uprojectdirs"
$ git push

Note: Adding the -f or –force parameter to git add is helpful when a file such as UE4Games.uprojectdirs is being ignored by one of the engine .gitignore files, but you want to track it nonetheless.

17. Well, congratulations! Now we have everything set up. You can make a copy of the intermediary ~/dev/MamadouArchives-Sync/ in order to start your development:

$ cp -vr ~/dev/MamadouArchives-Sync ~/dev/MamadouArchives

Or, you could clone a completely new copy of it from Microsoft Azure DevOPS:

$ git clone https://SOME-ORGANIZATION@dev.azure.com/SOME-ORGANIZATION/MamadouArchives/_git/MamadouArchives ~/dev/MamadouArchives

18. Now you can head to ~/dev/MamadouArchives and run GenerateProjectFiles.sh for GNU/Linux, GenerateProjectFiles.bat for Windows, GenerateProjectFiles.command for macOS, in order to generate the project files for your platform or toolchain of choice and build the engine and your project with it.

19. On both GNU/Linux and Microsoft Windows, sometimes the ShaderCompileWorker fails to build. You may notice this at build time or when running the engine with an error similar to the following:

Couldn't launch /home/mamadou/dev/MamadouArchives/Engine/Binaries/Linux/ShaderCompileWorker! Make sure you build ShaderCompileWorker.
make ShaderCompileWorker

The solution is to find the ShaderCompileWorker project inside Microsoft Visual Studio’s solution explorer and build it. Or, on GNU/Linux open up a terminal, navigate to the engine directory and issue the make command:

$ cd ~/dev/MamadouArchives
$ make ShaderCompileWorker

20. If everything went fine until this step, you can directly open your project inside the engine by issuing the following command or making it a script file in your project’s engine root directory and then run that:

$ cd ~/dev/MamadouArchives
$ ./Engine/Binaries/Linux/UE4Editor "./MamadouArchives/MamadouArchives.uproject"

Note: In some cases depending on your build type, the name of the UE4 editor or its parameter may require adjustments:

$ cd ~/dev/MamadouArchives
$ /home/mamadou/dev/Animus/Engine/Binaries/Linux/UE4Editor-Linux-DebugGame "./MamadouArchives/MamadouArchives.uproject" -vulkan -debug

21. Lastly, in case you need the useful file locking feature for LFS objects, it could be enabled by running the following command:

$ git config lfs.https://SOME-ORGANIZATION@dev.azure.com/SOME-ORGANIZATION/MamadouArchives/_git/MamadouArchives.git/info/lfs.locksverify true

This will add the following to .git/config

[lfs "lfs.https://SOME-ORGANIZATION@dev.azure.com/SOME-ORGANIZATION/MamadouArchives/_git/MamadouArchives.git/info/lfs"]
	locksverify = true

Migrating an existing repository

After setting up your current repository for LFS as we have explained in this tutorial, in case you need your git history, it could be as easy as running a command like this:

$ git lfs migrate import --everything --include="*.uasset,*.umap"

Adjust the –include according to your project’s specific needs.

Since migration does overwrite your reoisutiry’s history, you’ll need to force push your repository:

$ git push --force

Note: In case you don’t desire a history rewrite, –no-rewrite comes to the rescue in order to create new commits that move your files to Git LFS instead:

$ git lfs migrate import --everything --include="*.uasset,*.umap" --no-rewrite

Now, it’s time for cleaning up your local .git directory after the migration to Git LFS:

$ git reflog expire --expire-unreachable=now --all
$ git gc --prune=now

Final thoughts

Despite the shortcomings of Git in handling binary files, it’s my SCM of choice for game development over currently available solutions such as SVN or Perforce. In the past, I used to heavily relying on nested Git Submodules in order to divide UE4 git projects’ repositories into smaller repositories and keep the history clean. Since I’ve found Microsoft Azure DevOPS, life has never been easier with UE4 project maintenance under git.