• C++ / Unreal Engine game developer
  • VR and Oculus developer
  • FreeBSD ports contributor
  • *nix enthusiast

Host Unreal Engine 4 projects on Microsoft Azure DevOPS with unlimited cost-free Git LFS quota

Host Unreal Engine 4 projects on Microsoft Azure DevOPS with unlimited cost free Git LFS quota

Host Unreal Engine 4 projects on Microsoft Azure DevOPS with unlimited cost free Git LFS quota

Among the gamedev industry, it’s a well-known fact that Unreal Engine projects sizes have always been huge and a pain to manage properly. And it becomes more painful by the day as your project moves forward and grows in size. Some even keep the Engine source and its monstrous binary dependencies inside their source control management software. In case you are a AAA game development company or you are working for one, there’s probably some system in place with an unlimited quota to take care of that. But, for most of us indie devs, or individual hobbyists, it seems there are not lots of affordable options, especially that your team is scattered across the globe.

There are plenty of costly solutions to keep UE4 projects under source control; ranging from maintaining a local physical server or renting a VPS with plenty of space on the cloud, equipped with a self-hosted Git, SVN, or Perforce, to use cloud SCM providers such as GitHub, GitLab, BitBucket, or Perforce. Since I prefer cloud SCM providers and Git + Git LFS (which also supports file locking), let’s take a look at some popular ones such as GitHub and GitLab.

GitHub for one, provides data packs, but the free offering is far from enough for collaborative UE4 projects:

Every account using Git Large File Storage receives 1 GB of free storage and 1 GB a month of free bandwidth. If the bandwidth and storage quotas are not enough, you can choose to purchase an additional quota for Git LFS. Unused bandwidth doesn’t roll over month-to-month.

Additional storage and bandwidth is offered in a single data pack. One data pack costs $5 per month, and provides a monthly quota of 50 GB for bandwidth and 50 GB for storage. You can purchase as many data packs as you need. For example, if you need 150 GB of storage, you’d buy three data packs.

For GitLab, although the initial generous 10GB repository size is way beyond the 1GB repository size offer by GitHub, the LFS pricing is insanely high:

  1. Additional repository storage for a namespace (group or personal) is sold in annual subscriptions of $60 USD/year in increments of 10GB. This storage accounts for the size calculated from Repositories, which includes the git repository itself and any LFS objects.

  2. When adding storage to an existing subscription, you will be charged the prorated amount for the remaining term of your subscription. (ex. If your subscription ends in 6 months and you buy storage, you will be charge for 6 months of the storage subscription, i.e. $30 USD)

Well, before this all get you disappointed, let’s hear the good news from the Microsoft Azure DevOPS team:

In uncommon circumstances, repositories may be larger than 10GB. For instance, the Windows repository is at least 300GB. For that reason, we do not have a hard block in place. If your repository grows beyond 10GB, consider using Git-LFS, Scalar, or Azure Artifacts to refactor your development artifacts.

Before we proceed any further, there are some catches to consider about Microsoft Azure DevOPS:

1. The maximum Git repository size is 10GB, which considering that we keep binary assets and huge files in LFS, is way beyond any project’s actual needs. For Git LFS it seems that Microsoft since at least 2015 has been providing unlimited free storage. For comparison, the engine source code for 4.27 is 1.4GB, which in turn when it’s getting committed to the git repo becomes less than 230MB:

$ cd /path/to/ue4.27/source

$ du -h
1.4G	.

$ git init
$ git add .
$ git commit -m "add unreal engine 4.27 source code"

$ git count-objects -vH

count: 97545
size: 900.25 MiB
in-pack: 110815
packs: 1
size-pack: 227.80 MiB
prune-packable: 97545
garbage: 0
size-garbage: 0 bytes

2. The maximum push size is limited to 5GB at a time. The 5GB limit is only for files in the actual repository and it won’t affect LFS objects. Thus, there are no limits for LFS objects’ pushes. Despite that, if your internet connection is not stable, you could divide your files into multiple commits and push them separately. For example, the initial git dependencies for UE 4.27 is around 40GB spanned across ~70,000 files. Instead of committing and pushing a 40GB chunk all at once, one could divide that into multiple commits and push those commits one by one using the following command:

$ git rev-list --reverse master \
    | ruby -ne 'i ||= 0; i += 1; puts $_ if i % 1 == 0' \
    | xargs -I{} git push origin +{}:refs/heads/master

3. Sadly, at the moment Azure DevOPS does not support LFS over SSH. So, you are bound to git push/pull over https, which for some might be annoying. Especially, that it keeps asking for the https token 3 consecutive times on any push or pull!

Q: I’m using Git LFS with Azure DevOps Services and I get errors when pulling files tracked by Git LFS.

A: Azure DevOps Services currently doesn’t support LFS over SSH. Use HTTPS to connect to repos with Git LFS tracked files.

4. Last but not least, there is an issue with the Microsoft implementation of LFS, which rejects large LFS objects and spits out a bunch of HTTP 413 and 503 errors at the end of your git push. It happened to me when I was pushing 40GB of UE4 binary dependencies. The weird thing was I tried twice and both times it took a few good hours till the end of the push operation and based on measuring the bandwidth usage, the LFS upload size appeared to be more than the actual upload size. According to some answers on this GitHub issue and this Microsoft developer community question, it seems the solution is running the following command inside the root of your local repository, before any git pull/push operations:

$ git config http.version HTTP/1.1

Well, not only it did the trick and worked like a charm, but also the push time on the following git push dropped dramatically to 30 minutes for that hefty 40GB UE4 binary dependencies.

OK, after getting ourselves familiarized with all the limits, if you deem this solution a worthy one for managing UE4 projects along with the engine source in the same repository, in the rest of this blog post I’m going to share my experiences and a script to keep the engine updated with ease using a Git + LFS setup.

[Read More...]

Gregorian / Jalali (a.k.a. Persian Calendar) Date Conversion in C++ using boost::locale

Well, anyone who has ever developed a C++ game or application with Gregorian to Jalali date conversion (or, vice versa) requirement is well aware of the hurdles of doing such a task. I, for one, have been maintaining my own cross-platform C++ library for almost two decades now, with occasional bugs coming up from time to time.

Recently, I have been playing with boost::locale (developed by Artyom Beilis and contributed to Boost project) a bit more in order to utilize it in a personal project called Barandazstorm, a home-grown social media analysis tool. Browsing the docs, I noticed the following example which does not even compile on my compiler:

boost::locale Gregorian to Hebrew date conversion example
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
using namespace boost::locale;
using namespace boost::locale::period;
generator gen;
// Create locales with Hebrew and Gregorian (default) calendars.
std::locale l_hebrew=gen("en_US.UTF-8@calendar=hebrew");
std::locale l_gregorian=gen("en_US.UTF-8");
// Create a Gregorian date from fields
date_time greg(year(2010) + february() + day(5),l_gregorian);
// Assign a time point taken from the Gregorian date to date_time with
// the Hebrew calendar
date_time heb(greg.time(),l_hebrew);
// Now we can query the year.
std::cout << "Hebrew year is " << heb / year << std::endl;


So, I tried to make a guess and replaced the en_US.UTF-8@calendar=hebrew part with en_US.UTF-8@calendar=jalali which didn’t work. But, on the second try replacing that with en_US.UTF-8@calendar=persian actually worked! Which is sheer delight; due to the fact that now I found a solution to convert dates between both calendars as efortless as techonologies such as .NET in C++:

Two-way Gregorian / Jalali date conversion using boost::locale
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
/**
* @file
* @author  Mamadou Babaei <info@babaei.net>
* @version 0.1.0
*
* @section LICENSE
*
* (The MIT License)
*
* Copyright (c) 2020 Mamadou Babaei
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* @section DESCRIPTION
*
* Two-way Gregorian / Jalali date conversion using boost::locale example
*/

#include <iomanip>
#include <iostream>
#include <cstdlib>
#include <boost/locale.hpp>

void from_gregorian_to_jalali()
{
    boost::locale::generator generator;

    std::locale locale_gregorian = generator("en_US.UTF-8");
    std::locale locale_jalali = generator("en_US.UTF-8@calendar=persian");

    boost::locale::date_time gregorian(
            boost::locale::period::year(2020)
            + boost::locale::period::february()
            + boost::locale::period::day(25),
            locale_gregorian);

    boost::locale::date_time jalali(gregorian.time(), locale_jalali);

    std::cout << "Persian date is " << jalali / boost::locale::period::year()
            << "/" << std::setfill('0') << std::setw(2)
            << (jalali / boost::locale::period::month()) + 1
            << "/" << std::setfill('0') << std::setw(2)
            << jalali / boost::locale::period::day()
            << "." << std::endl;
}

void from_jalali_to_gregorian()
{
    boost::locale::generator generator;

    std::locale locale_gregorian = generator("en_US.UTF-8");
    std::locale locale_jalali = generator("en_US.UTF-8@calendar=persian");

    boost::locale::date_time jalali(
            boost::locale::period::year(1398)
            + boost::locale::period::month(11)
            + boost::locale::period::day(06),
            locale_jalali);

    boost::locale::date_time gregorian(jalali.time(), locale_gregorian);

    std::cout << "Gregorian date is " << gregorian / boost::locale::period::year()
            << "/" << std::setfill('0') << std::setw(2)
            << (gregorian / boost::locale::period::month()) + 1
            << "/" << std::setfill('0') << std::setw(2)
            << gregorian / boost::locale::period::day()
            << "." << std::endl;
}

int main()
{
    from_gregorian_to_jalali();
    from_jalali_to_gregorian();

    return EXIT_SUCCESS;
}


On a side note, for the above code to work your boost::locale libraries has to be built with ICU support; otherwise boost::locale throws an exception. On most Linux or BSD distros this is the default when you install Boost libraries from your distro’s package manager. On Windows, it requires a bit of effort if you are trying to build Boost binaries yourself, which is another story for another time.

I did test the above code on FreeBSD, Linux, and Microsoft Windows, using MSVC, GCC, LLVM/Clang, and MinGW, and it’s working as expected on all of these platforms.

A workaround for udevd 100% CPU usage and blank screen freeze on Gentoo GNU/Linux with recent NVIDIA drivers

A few months back due to various changes in how Funtoo is being managed, I migrated back from Funtoo to Gentoo after almost a decade. After some time I realized my laptop randomly gets stuck on a blank screen and freezes just right before my login manager (SDDM) starts. I noticed the hard-disk LED is blinking and the system is actually not freezed and probably is working and stuck on something. Checking the system or Xorg logs did not reveal anything unusual.

I even posted my issue on the Gentoo Forums and when I thought the issue is gone I marked it as SOLVED (well, I don’t turn off this laptop or reboot too much). But, the problem came back and hunted me over again.

Finally, I decided to install JuiceSSH on my phone since I do not have access to another PC for the time being. When it did freeze, I did ssh into my Gentoo installation and noticed udevd’s CPU usage is at 100%. I looked up the forums to see if someone else having this issue or not. I cannot recall where on the forums I saw it, but it seems this was a known issue to some users with recent NVIDIA drivers and someone suggested blacklisting the NVIDIA drivers, so the kernel won’t load them at boot time as it is going to be loaded by X later on.

Well, I did the following changes in order to blacklist the NVIDIA modules, so the kernel won’t load them at boot itme:

/etc/modprobe.d/blacklist.conf
blacklist nvidia
blacklist nvidia_drm
blacklist nvidia_modeset
blacklist nvidia_uvm

And, viola! It has been a month without any issues so far. It did solve the issue for me, once and for all. Hope it helps someone with a similar issue until this bug is officially fixed.

FOSS  FLOSS  Funtoo  Gentoo  GNU  Linux  Unix 

The long awaited FreeBSD www/wt and www/wt3 ports updates

For those who don’t know, I maintain various official FreeBSD ports and for almost 18 months my submitted updates to www/wt has been stuck due to lack of a review. Hopefully, tonight it has been committed to the official FreeBSD Ports tree and has been divided into two ports:

  1. www/wt for following the 4.x releases
  2. www/wt3 for following the 3.x releases

Although Wt Webtoolkit 4.x is a significant update which brings in more modern C++ and performance improvements (read more about the changes between releases on their archive), I haven’t migrated any project to the 4.x release yet. I guess the 3.x releases will live on for the time being and any project using them should be fine for some time.

Keep Crashing Daemons Running on FreeBSD


UPDATE 1 [2019/05/11]: Thanks to @mirrorbox’s suggestion, I refactored the script to use service status instead of ps aux | grep which makes the script even more simple. As a result, the syntax has changed. Since I keep the article untouched, for the updated code visit either the GitHub or GitLab repositories. The new syntax is as follows:

# Syntax
$ /path/to/daemon-keeper.sh

Correct usage:

    daemon-keeper.sh -d {daemon} -e {extra daemon to (re)start} [-e {another extra daemon to (re)start}] [... even more -e and extra daemons to (re)start]

# Example
$ /path/to/daemon-keeper.sh -d "clamav-clamd" -e "dovecot"

# Crontab
$ sudo -u root -g wheel crontab -l

# At every minute
*   *   *   *   *   /usr/local/cron-scripts/daemon-keeper.sh -d "clamav-clamd" -e "dovecot"

UPDATE 2 [2019/05/11]: Another thanks to @mirrorbox for mentioning sysutils/daemontools which seems a proven solution for restarting a crashing daemon. It makes this hack redundant.

Daemontools is a small set of /very/ useful utilities, from Dan
Bernstein.  They are mainly used for controlling processes, and
maintaining logfiles.

WWW: http://cr.yp.to/daemontools.html

UPDATE 3 [2019/05/11]: Thanks to @dlangille for mentioning sysutils/py-supervisor, which seems to be a viable alternative to sysutils/daemontools.

Supervisor is a client/server system that allows its users
to monitor and control a number of processes on UNIX-like
operating systems.

It shares some of the same goals of programs like launchd,
daemontools, and runit. Unlike some of these programs, it is
not meant to be run as a substitute for init as "process id 1".
Instead it is meant to be used to control processes related to
a project or a customer, and is meant to start like any
other program at boot time.

WWW: http://supervisord.org/

UPDATE 4 [2019/05/13]: Thanks to @olevole for mentioning sysutils/fsc. It is minimalistic, dependency free and designed for FreeBSD:

The FreeBSD Services Control software provides service
monitoring, restarting, and event logging for FreeBSD
servers.  The core functionality is a daemon (fscd)
which is interfaced with using fscadm.  See manual pages
for more information.

UPDATE 5 [2019/05/13]: Thanks to @jcigar for bringing daemon(8) to my attention, which is available in the base system and it seems perfectly capable of doing what I was going to achieve in my script and more.


Amidst all the chaos in the current stage of my life, I don’t know exactly what got into me that I thought it was a good idea to perform a major upgrade on a production FreeBSD server from 11.2-RELENG to 12.0-RELENG, when I even did not have enough time to go through /usr/src/UPDATING thoroughly or consult the Release Notes or the Errata properly; let alone hitting some esoteric changes which technically crippled my mail server, when I realized it has been over a week that I haven’t been receiving any new emails.

At first, I did not take it seriously. Just rebooted the server and prayed to the gods that it won’t happen again. It was a quick fix and it seemed to work. Until after a few days, I noticed that it happened again. This time I prayed to the gods even harder - both the old ones and the new ones ¯\_(ツ)_/¯ - and rebuilt every installed ports all over again in order to make sure I did not miss anything. I went for another reboot and, ops! There it was again laughing at me. Thus, losing all faith in the gods, which led me to take up responsibility and investigate more on this issue or ask the experts on the FreeBSD forums.

After messing around with it, it turned out that the culprit is clamav-clamd service crashing without any apparent reason at first. I fired up htop after restarting clamav-clamd and figured even at idle times it devours around ~ 30% of the available memory. According to this Stack Exchange answer:

ClamAV holds the search strings using the classic string (Boyer Moore) and regular expression (Aho Corasick) algorithms. Being algorithms from the 1970s they are extemely memory efficient.

The problem is the huge number of virus signatures. This leads to the algorithms’ datastructures growing quite large.

You can’t send those datastructures to swap, as there are no parts of the algorithms’ datastructures accessed less often than other parts. If you do force pages of them to swap disk, then they’ll be referenced moments later and just swap straight back in. (Technically we say “the random access of the datastructure forces the entire datastructure to be in the process’s working set of memory”.)

The datastructures are needed if you are scanning from the command line or scanning from a daemon.

You can’t use just a portion of the virus signatures, as you don’t get to choose which viruses you will be sent, and thus can’t tell which signatures you will need.

I guess due to some arcane changes in 12.0-RELEASE, FreeBSD kills memory hogs such as clamav-clamd daemon (don’t take my word for it; it is just a poor man’s guess). I even tried to lower the memory usage without much of a success. At the end, there were not too many choices or workarounds around the corner:

A. Pray to the gods that it go away by itself, which I deemed impractical

B. Put aside laziness, and replace security/clamsmtp with security/amavisd-new in order to be able to run ClamAV on-demand which has its own pros and cons

C. Write a quick POSIX-shell script to scan for a running clamav-clamd process using ps aux | grep clamd, set it up as a cron job with X-minute(s) interval, and then start the server if it cannot be found running, and be done with it for the time being.

For the sake of slothfulness, I opted to go with option C. As a consequence, I came up with a generic simple script that is able to not only monitor and restart the clamav-clamd service but also is able to keep any other crashing services running on FreeBSD.

[Read More...]

My Reddit Wallpaper Downloader Script

My i3wm setup with amazing gruvbox color scheme and a wallpaper from Reddit

i3wm setup with amazing gruvbox color scheme and a wallpaper from Reddit

Update [2019/05/08]: Many people have been asking for the wallpaper in the above screenshot. It is from System Failure II, oil on canvas, 31x43” on r/Art.

Well, I am really fascinated by Reddit art and genuine creative ideas such as Scrolller which was made possible thanks to gazillions of art pieces scattered throughout various art subreddits. I am also fascinated by Unix philosophy and have been a *nix enthusiast for as long as I can remember. In addition to all this, the discovery of r/unixporn - realizing I am not the only one who cares about aesthetics of their Unix box - was a huge blow for me; to the point that studying the GitHub dotfiles posted along the screenshots on r/unixporn by fellow nix-enthusiast redditors felt like a day to day hubby for me.

All the while, I had a successful experiment with writing a complex piece of real-world software in pure Bash with an amazingly wide range of features for around 3.5K lines of code. The real excitement came when it made to the official FreeBSD Ports Tree. In spite of the fact that many people find Bash syntax annoyingly ugly and unmaintainable and often wonder why do people still write shell scripts by asking it on Quora, since MS-DOS 6.22 era, I did develop a certain love–hate relationship with shell scripting languages such as Batch Files, Bash, etc. Thus, still I do automate almost everything with these ancient technologies.

So, here is my fully-configurable wallpaper changer software written in bash which automagically fetches and display wallpapers from your favorite subs. It has been powering and brightening up my i3wm setup for the past eight months which led me to the conclusion that it deserves a proper introduction.

[Read More...]

HackYourFuture First Game Development Workshop Report

HackYourFuture First Game Development Workshop Report

HackYourFuture First Game Development Workshop Report

OK! Today, HackYourFuture held their first-ever gamedev workshop titled “Introduction to Unreal Engine 4 Game Development Using Blueprint” at their Amsterdam office, which I’ve been given the opportunity to present to their alumni on HackYourFuture’s behalf.

Since I’m still very excited and fascinated by the culture of sharing and the friendly environment governed by HackYourFuture, I decided to share the presentation file which I did prepare for this four hour workshop under the terms of (CC BY-SA 3.0) Creative Commons Attribution-ShareAlike 3.0 Unported License, to be used by anyone who is either new to game development or is intersted in the topic. It’s jam packed with so much information for absolute beginners. Feel free to share and adapt according to your own needs under the terms of the license.

The presentation can be found at the end of this post in PDF, OpenDocument Presentation, or Microsoft PowerPoint formats.

[Read More...]

How to Run Multiplayer Call of Duty 4: Modern Warfare (Promod LIVE) using Wine on GNU/Linux

Call of Duty 4: Modern Warfare

Call of Duty 4: Modern Warfare

Well, I haven’t played a multiplayer game in ages until recently, when my cool boss announced regular playtimes for all the employees in our company as a group activity in order to put the fun back into work. Since I’m a die-hard COD4 fan and I used to play Promod LIVE heavily with colleagues and friends, I proposed Call of Duty 4: Promod LIVE 220 which happened to be favored by everybody; except there was one issue: everyone uses Microsoft Windows while I’m developing UE4 games on a single-boot Funtoo Linux system.

Naturally, my first attempt was running it inside a Windows 7 virtual machine under VMWare Workstation for Linux which supports up to Direct3D 10 (the exact API used by COD4). Sadly, the experience was very poor and painful with lots of unbearable stuttering on my decent hardware. Thus, the last resort was running it under Wine, which I used to happily run many Windows applications and games under it for many years. Though, throughout those years I replaced almost every Windows application with an equivalent or an alternative Linux application until I gradually stopped using it. In the meanwhile, I also distanced myself from traditional desktop environments such as GNOME, KDE, Xfce, and LXDE, while experimenting with various window managers specially i3wm, which caught my attention for many good reasons. So, in the end I made up my mind and alienated myself from desktop environments once and for all.

Running a fully-fledged game engine such as Unreal Engine 4, I expected COD4, Wine, and i3 combination to work fine out of the box as it would under any other DE. Well, it turned out that I was too simple-minded about running a fullscreen game such as COD4 under Wine/i3wm. Hopefully, as the Wine FAQ states the workaround is super easy.

Here is the full guide on running COD4 v1.7 with Promod LIVE 2.20 on GNU/Linux.

[Read More...]