# Kongregate Developers

## Unity WebGL Memory and Performance Optimization

Many Unity developers have taken the plunge and started exporting WebGL versions of their games. Unfortunately, many of them also run into issues that can lead to performance degredation and instability. This is a particularly tricky problem on older machines that have less resources available.

This article will outline several techniques for resolving some common WebGL performance issues we’ve seen with games published on Kongregate, and hopefully get you off to a good start when it comes to delivering a great plugin-free experience with your WebGL game.

## Measure Twice, Optimize Once

One of the most common mistakes developers make when trying to optimize content is making assumptions about what is taking up lots of time or space. With Unity and WebGL this happens often when deciding how much heap space to allocate for the WebGL application in the player settings. Without doing research on the subject, it can be easy to assume that increasing the memory available to the application will lead to better performance. In reality you generally want to allocate the smallest amount of memory possible, which can be counter-intuitive.

Reducing the size of the heap can greatly reduce or eliminate the number of “The browser could not allocate enough memory for the WebGL content” errors your players see, but how do you know what to set it to? Luckily, you can get a pretty good feel for how much memory your application is using with a simple JavaScript snippet which you can either paste into the browser console (for ad-hoc debugging), or as a permanent addition to your index.html template:

setInterval(function() {
if (typeof TOTAL_MEMORY !== 'undefined') {
try {
var totalMem = TOTAL_MEMORY/1024.0/1024.0;
var usedMem = (TOTAL_STACK + (STATICTOP - STATIC_BASE) +
(DYNAMICTOP - DYNAMIC_BASE))/1024.0/1024.0;
console.log('Memory stats - used: ' + Math.ceil(usedMem) + 'M' +
' free: ' + Math.floor(totalMem - usedMem) + 'M');
} catch(e) {}
}
}, 5000);

This will output the amount of used and free memory into the console every 5 seconds, like so: Memory stats - used: 155M free: 37M. With this very simple addition, you now have some insight into how to tune your heap size, which is fantastic. You can watch these numbers as you play your game and get a feeling for what you should cap your memory usage at. Be careful not to lower it too much, though, or you’ll start seeing memory allocation errors that tell you to increase your heap size.

#### A Note on Dynamic Heap Sizing

When searching around for solutions to out-of-memory issues, you may stumble upon an Emscripten flag called ALLOW_MEMORY_GROWTH, which sounds like the perfect solution to your woes. These days enabling this flag can have a negative impact on performance, and it can also cause out-of-memory issues, as the memory manager has to allocate even more space when trying to grow the heap. The Unity team has never recommended enabling this flag, and we do not recommend it either.

When using Unity 5.3 and above, your game assets will be compressed with gzip by default, and will have a gz prefix on the filenames. As described in a previous WebGL blog post, if you upload these to Kongregate, we will automatically set the Content-Encoding: gzip header so that the assets can be decompressed by the browser instead of JavaScript, leading to a much better loading experience.

However, if you are hosting your game externally, you will need to manually ensure that the assets are being served with the correct encoding. You will know if this is an issue because Unity will log messages like Decompressed Release/MyGame-WebGL.jsgz in 500ms in the console, the browser may appear to lock up during loading, and your progress bar will not work properly.

If you are using apache or a compatible server, you can use the .htaccess file that Unity generates for you to resolve this issue. However, if you are hosting on S3 or something similar, you can force the encoding by changing the filenames in your index.html to add the gz suffix, like so:

var Module = {
TOTAL_MEMORY: 201326592,
errorhandler: null,
compatibilitycheck: null,
dataUrl: "Release/MyGame-WebGL.datagz",
codeUrl: "Release/MyGame-WebGL.jsgz",
memUrl: "Release/MyGame-WebGL.memgz",
};

You will also need to ensure that S3 is set up to send the Content-Encoding: gzip header for each of the files that end in gz, or you will get decompression errors upon loading the game. You can use the browser network inspector to ensure the header is being sent properly by the server.

Another unexpected source of performance problems is caused by loading asset bundles. For Unity versions lower than 5.5, asset bundles are compressed using LZMA by default. Decompressing the LZMA content for WebGL builds causes a very noticeable pause. These pauses not only make your game run very poorly, but can bring up the dreaded “this content is not responding” error dialog. Luckily, Unity 5.3 and above support the faster LZ4 format for asset bundles (and it is the default for WebGL in 5.5+), so you can resolve this issue relatively easily.

It should be noted that LZ4 assets can be a bit larger than LZMA, but the tradeoff is generally acceptable. If it is not, you can use gzip compression on top of the asset bundles, and serve them with the Content-Encoding: gzip header (like mentioned above) to have the browser decompress them automatically.

One of our developers was kind enough to share the C# code they’re using in their build script to enforce LZ4 compression for WebGL:

BuildAssetBundleOptions options = (buildTarget == BuildTarget.WebGL) ?
BuildAssetBundleOptions.ChunkBasedCompression :
BuildAssetBundleOptions.None;
BuildPipeline.BuildAssetBundles(bundlePath, buildMap,  options, buildTarget);

## Keep Up on Unity Patch Releases

The Unity team releases patches for supported versions of Unity frequently. Keep an eye on them and see if any of the WebGL-related changes might resolve issues you are experiencing.

For example, Unity 5.3.6 patch 6 resolved an issue where WebGL builds were allocating much more memory than needed, which is definitely a bug you’ll want to have fixed when you build your game. If upgrading is not a viable option, they also generally have you covered with workarounds. In this case, you can reduce your footprint on lower versions with the following build setting:

PlayerSettings.SetPropertyString("emscriptenArgs",
" -s MEMFS_APPEND_TO_TYPED_ARRAYS=1",
BuildTargetGroup.WebGL);

## Other Resources

• Read the Unity Blog and documentation religiously -- such a vast amount of knowledge is available there. Many of the concepts in this article are discussed in much more technical detail here, for example.
• Check out this Kongregate Developer Blog entry focused on managing in-game memory for Unity WebGL projects.

### Author

#### Ben Vinson

While studying for his B.S. in Computer Science from the University of Colorado, Ben was a major contributor to the book Software Engineering for Game Developers.

After graduating, Ben started work as a game developer at Electronic Arts. He was also the creator of the Leeroy Jenkins video, which is probably his greatest achievement to date.

## The Math of Idle Games, Part III

This three-part series is intended to break apart my MIGS 2016 slides into what are hopefully detailed but accessible angles on the math and mechanics behind idle and incremental games. The sections draw heavily from a collection of spreadsheet models I built and have opened up for you to play around with.

In Part I we looked at some of the standard math behind rapid-growth idle games, primarily at the relationships between exponential and polynomial growth and some methods of checking and adjusting the balance of the various generators over time.

In Part II we explored different options for growth outside of the Cookie Clicker exponential model, as well as other options for interactions of generators.

Now, in the final part, we will look at prestige loops, including analyzing the design behind real-world examples and how to vary them to keep things interesting for the player.

The ability to reset your game with a multiplier to progress, often called a “prestige,” is one of the crucial mechanics behind most modern idle games, originally popularized (and conceived?) by Orteil’s Cookie Clicker. At a high level, these systems serve two primary purposes:

1) They create the “ladder climbing” effect that contributes to why idle games are so compelling. It gives players the ability to reset with a huge boost that gives a sense of power and progres. This is similar to how “launch games” (like Burrito Bison: Launcha Libre or Curl Up and Fly) have the player go as far as possible, then upgrade, then launch again.

2) They rein growth back into a more manageable number. This can give you as a developer a more stable number to key progress and upgrades off of, since growth here will be slower.

Note: I have incorrectly called this “taking a log” in the past. Most prestiges are a fractional exponent (e.g. a square or cube root) rather than a mathematical log function, so growth is still very large in some cases. Clicker Heroes is an exception that really does have a log-type effect. That said, some of these games have since added systems that do substantially rein in numbers, such as Realm Grinder’s Reincarnations and AdVenture Capitalist’s Mega Bucks.

The amount of prestige currency generated can be calculated off a variety of factors, including (but not limited to):

• Max earnings (Realm Grinder)
• Earnings since last reset (Egg, Inc.)
• Number of upgrades purchased since last reset (Clicker Heroes)

These fall into two primary categories: lifetime stats and since-reset. This fundamental difference guides general reset behavior in a significant way. Let’s dig into each of these examples a bit more to see how. For these formulas, the variables will be:

$Δp$ = prestige currency to gain
$p$ = total prestige currency
$c_L$ = lifetime currency
$c_M$ = max currency
$c_R$ = currency this run

First up, Realm Grinder:

$p = \frac{\sqrt{1 + 8 \cdot\frac{c_M}{10^{12}}} - 1}{2}$

That seems like a pretty bizarre formula, but it's actually very simple. Why? The equation for the amount $c_M$ required for $p$ is:

$c_M=10^{12}\cdot\frac{p(p+1)}{2}$

You may recognize that as the summation formula. So now we need to solve for $p$. Hmm, let's see...

$p^2+p-\frac{2c_M}{10^{12}}=0$

Yup, that's right. You're about to use the quadratic formula. For your job/hobby. In the real world. I bet you never thought you would! Plug it in and you'll get that first equation up there. (This legitimately got me so excited that I posted about it on Facebook.)

© Saturday Morning Breakfast Cereal (SMBC), used with permission

Since this formula is based on the square root of the max currency earned, it means that if you reset at the same point you will earn literally no more prestige currency. To double prestige currency, a player would need to earn 4 times as much as the previous run, so that gives you a rough target to keep in mind when modeling and balancing.

$p = 150\cdot\sqrt{\frac{c_L}{10^{15}}}$

Here we similarly have a square root, but this time based on lifetime earnings. Again assuming we want to double prestige currency each run, how much more than the previous run do we need to earn? It depends a bit on your assumptions about the previous run, but it’s likely somewhere in the similar 3x - 4x range. However, a fundamental difference is that you can keep resetting at the same point and gain currency. That said, the amount of currency diminishes each time, which means that players do need to be able to advance to make appreciable gains.

$p = \sqrt[3]{\frac{c_L}{10^{12}}}$

It is also based on lifetime earnings, but using a cube root, meaning to double prestige currency, a player needs to earn roughly 8 times the previous run.

On the other side we have the systems that are agnostic to previous performance -- each run is an independent event as far as the prestige currency calculation is concerned. This means that if you reset at the same point a few times in a row you will get the same amount of currency each time. This is a very different dynamic that can create ideal strategies that don’t involve much progress in the game beyond a certain point.

These types of systems must be balanced quite differently. They have very flat curves for calculating prestige currency (that is, getting farther in a run has rapidly diminished returns).

Let's look at Egg, Inc.:

$\Delta p = (\frac{c_R}{10^6})^{0.14}$

That’s roughly an exponent of $\frac{1}{7}$. So where doubling in Realm Grinder would require 4 times as much as the previous run, Egg, Inc. would require 128 ($2^7$) times as much! This could be done for a variety of design reasons, but I think one of the clearest is to nudge players into more active play and reduce the influence of time on progression. Or conversely, to accelerate progress in an offline-limited system.

In the case of Egg, Inc. I believe it to be the latter. Players are limited to 2 hours of offline play by the game. By allowing players to earn equivalent currency on equivalent runs, the game minimizes the dependency of idle gains, which is important since it has created hard limits on those gains. Progress then happens in staircase tiers through the sub-prestige loop (which roughly multiplies earnings by 5x each time) and general upgrades.

Clicker Heroes, on the other hand, has no such limits to offline gains. Instead it has a very steep difficulty curve that makes progression through the main series of “hero zones” fairly slow. As a result, the ability to keep earning prestige currency at about the same point gives players the ability to progress despite a pretty substantial progress wall. The currency itself is based on the number of upgrades purchased for the generators (heroes). Because the cost of upgrades grows at an exponential rate, this does effectively “take a log” of the overall growth curve.

Pulling these all together, in the chart below we look at the relative prestige currency gains based on earnings relative to the previous run. So the $x$-axis is a multiple of the previous run, meaning $x$=1 is earning the same amount as the previous run, $x$=2 is double, etc. The $y$-axis is prestige currency, normalized to the amount currently owned. So $y$=1 would mean that you would double your prestige currency (i.e. you earn +100%). For the sake of sanity, we’re making an assumption that the previous earnings for the lifetime calculations were ¾ of the total earnings (i.e. the previous run quadrupled the lifetime earnings to that point), so lifetime earning is $\frac{4}{3}$ of the previous run.

Looking at $y$=1 you can see rough relative earnings required to double your current prestige currency. As expected, this occurs at $x$=1 for both Egg, Inc. and Clicker Heroes, since they are independent from previous runs. Both of those curves flatten out quickly, though, and are overtaken by the other three before long.

So, that was a quick look at a variety of ways to calculate prestige currency and the design implications of them. But once you settle on a method for your game (or even use a few methods if you want to get crazy with multiple prestige currencies), how do you balance it?

To try to get a bit of a handle on this I put together “yet another spreadsheet”™ to let you play around with implications of varying the parameters. It can be found in this collection of idle game models, specifically sheet 3a. This is a very simple prestige system with only a single generator, but it can help understand the dynamics at play.

One thing to consider is that in the same way that you want to make progress through a run somewhat “bumpy” with slow parts and fast parts, you likely want to have similar variation in your prestige resets too. Much of this variation comes from the interplay of multipliers/upgrades and the rate at which prestige currency is gained (as well as the player’s targets for when to reset). Predicting player behavior in this case is difficult, but you can likely capture most players by varying within reasonable ranges.

In this chart, generated by that spreadsheet, the $y$-axis is the number of generators owned. Each time that hits 0 it’s a prestige reset. You can see bumps of rapid purchases at 300, 400, 500, etc. generators owned -- those are due to multipliers kicking in at those totals.

There’s also a fairly long prestige starting at 120 minutes. It finally breaks at 500 generators, which is when a large 16x multiplier triggers. This is a case in which the model likely would be off -- a player who just hit a big milestone is likely to keep going for a while since that’s a fast section of the game. The point being that you should not use this model as an accurate simulation but instead as a way to try to get a rough feel for how your system would behave.

So, that’s three blog posts full of equations, graphs, and spreadsheets. While I hope that those will be useful tools for anyone working on an idle game, I think the meta takeaways are:

• There’s a lot more variety possible than the two most-common models (Cookie Clicker and Clicker Heroes). In particular, think about ways that generators can interact with one another. Draw a flowchart, then mess around with the arrows.
• Balancing progression is hard. Spreadsheet models can help, but there’s a lot of iteration required and you have many tools at your disposal.
• When designing your game, determine where the “fun” is and focus on that. Is it unfolding new features? Collecting tons of achievements? Optimizing the prestige loop? The novelty of big numbers on their own is largely gone, but there are still many opportunities for surprising and delighting players.

And with that we’ll conclude this series of idle math blog posts. I hope they have been helpful, and as always please reach out if you have any questions or cool games you want to share. :)

If you would like to contact me, shoot me an email at anthony@kongregate.com. I also want to give credit to the tools used in this post: MathJax did the awesome live-rendering of LaTeX and Desmos is a slick in-browser graphing calculator with some great educational functionality.

### Author

#### Anthony Pecorella

Anthony has been at Kongregate for 7 years and is the director of virtual goods for browser games. He works to bring great games to the platform and help developers make a living with their games. He was the lead producer for the mobile version of AdVenture Capitalist.

Anthony also designs and develops games on the side with his indie game studio Level Up Labs, including the acclaimed tower defense RPG Defender's Quest: Valley of the Forgotten, and a MacArthur-grant-funded educational biology game, CellCraft.

## BattleHand Q&A with Another Place Productions

To celebrate the 1-year anniversary of the charming fantasy RPG BattleHand, we sat down with the extremely talented development team behind the game, Another Place Productions, to discuss inspiration, process, and, of course, BattleHand!

The London-based indie development studio started surprisingly enough at Lionhead Studios when the 3 founders, who were working on the Fable franchise, decided to create their own games. After forming in 2012, they quickly realized that developing mobile games would give them more freedom to take risks and innovate.

Their latest mobile game, BattleHand, is a return to their fantasy roots where players enter the world of FellCrest to rescue the kingdom from darkness. And after 1 year, the game is still going strong!

Q. BattleHand is celebrating its 1-year anniversary today! It’s amazing how quickly the year flew by, and we’re super excited to say that not only was BattleHand chosen as an Apple Editor’s Choice title at launch, but Apple also just included it in its Best of App Store 2016! How did the team react when you found out?

A. It’s a huge achievement and we are SUPER excited! We’re extremely proud for BattleHand to be acknowledged like that, and it’s a privilege to be listed with some amazing titles that we love playing too!

Q. It was the first and still only 1 of 2 games we’ve published to get the Editor's Choice nod (The Trail being the second)! Did being an Apple Editor’s Choice game affect BattleHand?

A. We always intend to create the best games we can, but being an Apple Editor’s Choice certainly sets a standard. It is a great honour for any developer and only spurs you on to get better and better.

Q. The game reached 2 million downloads back in April. How did the team celebrate the milestone?

A. We were working on releasing Red, our first new Hero since launch, at the time! Passing the milestone then only made us more eager to deliver more awesome content and features to the 2M BattleHand fans out there!

Q. And were there any other big milestones that compared or exceeded the 2 million download mark?

A. Releasing Guild Wars was a massive milestone for us. It took a huge effort for the team and we were delighted with the result for players. Bringing some social competitiveness to the game whilst keeping the fun aspect of BattleHand was what the game needed. It felt like launching the game again to a degree!

Q. How did you approach creating a fantasy universe for BattleHand? Did you try and diversify your world from other popular fantasy franchises?

A. The world of FellCrest is essentially a jocular love letter to Dungeons & Dragons and the early fantasy worlds that many of the team grew up on. We wanted to embrace a lot of the familiar tropes and characteristics of board games and fantasy books and add our own unique twist of humour and charm to it, as well as a unique visual style. Like any parody or homage, pulling away from the source material too far can burst the bubble, so we made sure that we were innovating in the right places.

Q. Who is the team's favorite character in BattleHand and why?

A. We each have our own favourites, but we do all have a soft spot for Peg! A new Hero is always exciting for us and we’d been saving the idea for the Kraken card for some time, plus her Ghost Ship trait is pretty awesome! Or maybe we just all want to be pirates!

Q. Several members of your team worked on the Fable franchise. What made you switch from that core/console experience to developing mobile games? And did you have trouble adjusting to the more casual audience/shorter game sessions?

A. We were very much driven to build game experiences that are relevant and can evolve by engaging with our players. Console is not the space for that, and those games also take far too long to make. Fable games, although for core players in principle, have always attracted a much larger, less "core" audience thanks to its charm, humour and infinite possibilities. It is very much with that in mind that we tackled working on mobile, and we think it worked!

Q. What game or entertainment property is your team most inspired by?

A. Visually, we looked at Don Bluth’s work on the classic '80s video game "Dragon’s Lair". This seemed like a suitable reference for our particularly "cartoonish" take on the genre and something that still looks fantastic today. Gameplay-wise, there is a heap of influence from Dungeons & Dragons and the classic JPRG’s like Final Fantasy. As a team, we’ve played everything in the RPG genre to be honest, but it was important that the mechanic and game design was built specifically for mobile, so we made sure we researched just as thoroughly in that area. And that list is endless!

Galvanic Games is a small company with few resources. As our primary programmer on the team, it’s my job to figure out how we can deliver content as quickly as possible. We had a goal to release our game, Questmore, after just a few months of development. This makes picking our tools one of the most valuable steps in the process, and Unity is a phenomenal tool for rapid development. However, it isn’t without problems.

One of the biggest issues right now with using WebGL and Unity is memory management. Unity needs to allocate a certain amount of memory up front, and as developers we hope the browser grants it. Sometimes this is a little unpredictable, as it really depends on what’s happening with the browser at the time (how much memory has it already given to its tabs?). On top of that, the architecture of the browser may give different results.

Early on we saw a much higher failure rate on 32-bit browsers than we did on 64-bit. Which makes sense! But still frustrating. There is really only one solution to this problem. Ask for as little memory as possible. Easy!

Of course after having the browser accept the desired amount of memory you then have to make sure you never go up over this amount or you’ll nab yourself an out of memory exception. We ended up settling on an allocation block of 256 MB for Questmore and we pretty easily sit under that amount when the game is playing. So problem solved? Not quite.

A quirk of Unity’s memory management in WebGL is that the garbage collector can only run in between frames instead of during a frame. If you allocate too much memory (even if it becomes garbage) in the middle of a frame, you’ll hit that game-ending out of memory exception.

“Be pro! Don’t generate garbage! The collector causes hitches anyways!” And that’s good! We keep our frame-by-frame garbage generation as low as we can to avoid those nasty hitches, but that’s not where the issue is. When setting up the scene, populating our object pool, or otherwise just doing the expensive initialization needed to run the game, it turns out a ton of garbage is created. This isn’t really surprising and usually isn’t much of an issue since we’re pretty okay with doing a lot of expensive stuff up front if that means the actual play experience for the player is super smooth. But we’re never getting that smooth experience if we run out of memory initializing the game. We could play the game at 256 MB, but we couldn’t launch the game at 256 MB.

The memory when our game idles is manageable but requires far too much when initializing. Since we don’t have threading in WebGL (Unity doesn’t play nicely with multi-threading anyways), the solution we came up with was to initialize as much as we can in a frame before yielding to the next frame. This allowed us to avoid spiking up over the memory limit as well as spin with enough frames that we could show a loading animation or progress bar so it doesn’t look like the game has locked up while loading.

The solution isn’t too complicated, but we enjoy giving back to the game development community, so we open-sourced the loader we created for Unity. While it was created to solve our WebGL memory issue, it’s built to be functional for any platform when it’s desired to initialize a scene while showing progress.

To set up the loader, add the Load Manager component to any GameObject that exists in the scene. The Load Manager has a couple of properties that will need to be set before loading can occur.

Seconds Allowed Per Frame - The amount of time, in seconds, that may pass before the loader will yield to the next frame. This can be thought of as a target framerate so animations can play while loading. So for 30 fps then a value 0.0333 would be used. However, this framerate is in no way guaranteed. If a registered enumerator takes more time than the allowed seconds, then that enumerator will finish before the yield will occur.

Memory Threshold For Yield - This is specific to WebGL and is ignored for other platforms. This specifies memory threshold before the loader will yield to the next frame (to allow the garbage collector to kick in). The value queried to check against this is returned by System.GetTotalMemory() which, according to Unity, is “the total managed memory currently used.” For Questmore a value of 128MB worked out pretty well, but experimentation for each game will be needed to find a value that allows as much to be loaded in a frame as possible (important!) without going over the limit and crashing the program.

Now that the loader is set up, there are two primary ways of loading content. Directly registering enumerators and registering GameObjects/classes that implement the IAssetLoader interface. When told to load, the loader will step through the registered enumerators and IAssetLoaders to load them in order.

Registering Enumerators

To register an enumerator, call the RegisterEnumerator method on LoadManager with the enumerator that will need to load.

LoadManager.instance.RegisterEnumerator(EnumeratorToLoad());

Registering Game Objects

Components that implement the IAssetLoader interface may also be registered to load. Invoke the method RegisterObject, and all the components that implement the interface will be registered.

LoadManager.instance.RegisterObject(gameObject);

Once registration is complete, then all that’s left is to inform the Load Manager to begin loading.

LoadManager.instance.LoadRegistered(OnLoadComplete);

And we’re done! The registered enumerators and GameObjects have been loaded and the frames will yield when appropriate. Below is a full example of how this could look.

public class Example : MonoBehaviour, IAssetLoader
{
public GameObject prefabToCreate;
public int numberToCreate;

public GameObject secondPrefabToCreate;
public int secondNumberToCreate;

// Use this for initialization
void Start()
{
}

{
}

{
for (int i = 0; i < numberToCreate; i++)
{
Instantiate(prefabToCreate);
yield return null;
}

// Can call into other enumerators

// Can continue to load here, will be invoked when LoadOtherEnumerator finishes
}

{
yield return null;
}

//
//
{
// Can also have registered the object and had this
// enumerator called
for (int i = 0; i < secondNumberToCreate; i++)
{
Instantiate(secondPrefabToCreate);
yield return null;
}

// Can also yield to other enumerators
}
}

That’s the system we came up with for Questmore to handle loading resources to avoid the WebGL memory limitations. To learn about more advanced topics (such as loading control, forcing yields, step progressions, and multiple phases) check out the readme on the github page.

### Author

#### C.J. Kimberlin

C.J. Kimberlin is a game designer, lead programmer, and rock star at Galvanic Games. Manage a bunch of heroes and become rich in their newly released game Questmore.

## Building Better AI with Kongregate Games

Web games like Kongregate's may be one of the key ingredients for developing powerful artificial intelligence. Read on to find out why this is, where they fit in, and how you can help!

Hi! I'm Jack Clark, strategy and communications director at OpenAI -- a non-profit artificial intelligence research lab. I want to tell you about a new AI project from us called "Universe" that relies in part on games from Kongregate.

At OpenAI, we've spent the past few months building software called Universe to let us introduce our nascent artificial intelligence systems to the digital world. We're making the main components of Universe available as open source code, so anyone can use it to develop their own AI systems. Our initial release includes a thousand different environments to train your AI on, including many Kongregate games.

Universe makes it simple to hook up an AI agent to a program running on a computer, and lets the AI system manipulate the program in the same way a human does -- by observing the pixels on the screen and manipulating the controls.

This kind of infrastructure is pretty crucial to the development of AI systems, because it lets us expose our AI agents to a variety of different stimuli and lets them interact with a range of novel environments.

If you can't easily introduce your AI to new scenarios then you have a fairly expensive brain in a jar (well, code on a server, but you get the point). With Universe, you can link this AI to a variety of different programs, giving it the same interface each time, letting it learn to perceive and act in a variety of different environments.

### A Brief Aside About How Our AI Works

We built Universe so we could teach our increasingly capable AI systems about the world. The technique we're using is called deep reinforcement learning. Deep RL is a popular technology in AI -- it's already been used to let software master a range of Atari videogames, beat a world champion at the ancient boardgame of Go, and make data centers more efficient.

The approach has two main ingredients: neural networks, which is the machinery that lets our AI perceive the world and develop an internal representation of it; and reinforcement learning, which lets us teach our AI to associate the actions it takes with changes in how it perceives the world. RL works by rewarding correct courses of action, like gaining a high score, letting its neural network components learn which actions it can take to generate rewards.

We've also used another trick, called behavior cloning. This lets us use data we've gathered from human players mastering various games to train our AI to imitate the actions they took. This is helpful for games where a complex sequence of actions stands between you and your first reward. Without this, RL approaches tend to get stuck, as they need to perform a long series of actions in sequence to attain their first reward, and the chance of doing this is fairly slim. Therefore, we seed our Universe agents with some examples of good play, so they can learn by supervision in combination with RL.

To get an idea of how unsuccessful a random RL agent is, you can view this animation of an algo getting stuck on the Kongregate game Neon Racer.

Now, after we've trained the agent using our Universe system, we're able to create an agent that attains the same scores a talented human player might. It develops rapid reactions, is able to figure out it can gain points by hitting the red cars and avoiding the purple ones, and even learns to use the "boost" special ability.

But it doesn’t stop there. We can then train the AI further until it achieves mastery of the game, attaining top scores and displaying a decent aptitude for some of the more strategic elements, like hitting certain types of cars, or using boost more strategically.

### Where Kongregate Fits In

So, where does Kongregate come in? With Universe we want to not only train our AI system to master individual games, but we want to be able to develop a curriculum for them to let them learn about more and more complex aspects of reality. That requires a lot of diverse environments, many of them games. Our initial release has over a thousand different environments to give our machines a diverse world to learn and experiment in.

Web games like those showcased on Kongregate are an excellent resource for this because they consist of genres -- say, racing games -- with lots of variation, difficulty differences, and differing levels of fidelity. Therefore, we can not only train our brain-in-a-jar to take the correct course of actions to win at a single Kongregate game, but can also teach it to master the basic principles of racing by playing a variety of games. We'll have more to share about our transfer learning experiments in the coming months.

Obviously, it's not feasible to train every agent to master every game if it takes several hours to do so for each game. That's where transfer learning comes in. The more games we have in Universe, the easier it is for us to train an agent on one, then transfer some of that knowledge over to another one, then another one, and so on. Each time it will take less time for the agent to master the game and, hopefully, it will develop some general "common sense" knowledge about games over time.

So, if you're interested in helping us build the next generation of AI in an open and transparent way, then there's an easy way to help: make your games available to Universe by hopping over to our website and filling out this form. We look forward to building artificial intelligence with you and your peers. Thanks!

### Author

#### Jack Clark

Jack Clark is the Strategy and Communications Director of OpenAI. Previously, he worked as a technical journalist for publications including BusinessWeek and The Register. He runs an AI newsletter in his (limited) spare time called Import AI.

December 5th, 2016 12:23 4 Comments