Summary

The Outer Worlds is a first person action RPG released in 2019 initially on PC, PS4, and Xbox One. Being an RPG the game has a large focus on creating/building your own character and making your own decisions as you see the story through. Most of the game supports approaching situations using either combat, stealth, or dialogue (with conversations usually having various branches you can take depending on your character/decisions). Most of the game is played in ‘open zones’, with the game containing multiple larger open areas for the player to explore. Combat includes both ranged and melee weapons, with pretty typical FPS style gameplay.

Combat

For a large part of The Outer World’s development, I was on the combat team (more specifically focused on the player side of things)- I implemented various features related to weapons and combat, this included: Weapon appearance support (asset management/loading, VFX/anim support, mods, etc.), projectile weapons, beam weapons, charge weapons, improvements to our shot triangulation, and more.

Appearance

Along with work on combat I also did work for our character appearances- this involved all the technical work to build our characters (both the player and NPCs, which used mostly the same system) and support any requests from art for character customization support. This involved implementing a component that manages assets and controls the construction of our characters- including their head and body mesh (non-humans just use a single body mesh), any material swaps, customizations like age/scars/make up/etc, appearance changes from armor, helmets, armor mods, etc. We also put a lot of effort into making sure pre-placed characters had their appearance fully constructed (so nothing needed to be done at runtime for them) and that blueprints had a cache of the appearance assets they would need for construction (this allowed us to be able to immediately spawn a character as long as their blueprints are loaded- there was no need for an additional load prior to their appearance being able to construct when spawning).

Theft

As an RPG player myself I was a bit upset that The Outer Worlds was originally going to ship without item ownership/theft (time for it could not be found in the schedule and it was cut). I ended up using a bit of my own time to implement a theft system for The Outer Worlds- with the implementation focused on making it as easy as possible to add into the game (as getting this implemented would require time from other teams to mark ownership for items) while also being simple but good enough to ship. After implementing a system in about a day’s work I was able to get buy in from the other teams on the feature and it ended up shipping.

Incremental Garbage Collection

While I did not originally implement it, I (along with another engine team member) ended up inheriting an incremental garbage collection implementation after the original author left (and it ended up needing a lot of work). This engine change took the reachability phase of garbage collection and split it up over multiple frames. Now-a-days this is something Epic supports out of the box (thankfully, it’s still new and I even reported some issues with it to Epic when it first came out), but this was something we ended up building ourselves and getting it stable enough to ship for The Outer Worlds. I am not more familiar with how garbage collection in Unreal works than I would like to be.

‘AutoPlayer’

The ‘autoplayer’ was a system that was thrown together in a couple days to start- but ended up getting more attention as it was instrumental to the project both for stability and information gathering. The autoplayer effectively controls the player (like a bot but it doesn’t use anything I could consider actual AI) to move them around the world (mostly via navigation), engage with enemies, press random buttons, load between maps, collect stats, etc. This was useful for the following reasons:

  1. By just ‘playing’ the game the autoplayer was able to weed out stability issues unattended. When we first put the autoplayer in it would crash in less than an hour- after a week or two of reporting crashes and getting fixes it was up for multiple hours. It’s now a cornerstone of our testing for ensuring the game is stable.
  2. We had telemetry for our game for tracking performance- we had some automated tests for this to collect performance into telemetry (move around the map and spin)- but found it tended to not accurately represent performance as it was not actually engaging with gameplay in the area. The autoplayer generated telemetry as well just by running (like any of our players did during development) and we ended up leaning on this data more than the more automated test data as we found it more useful/accurate.
  3. Offline data collection- the autoplayer tracked _a lot_ of data that would be too much to shove into telemetry. For example, it tracked (every 15 seconds) general memory usage, memory stats (from UE’s stat system), render target pool usage, audio memory usage, texture streaming pool info, and more. We could then easily look through this data to find maps that had outlying information, identify locations where memory peaked (and investigate them further), ensure that our texture pool was well tuned for the whole game, identify trends in memory over time, etc.

Incremental Garbage Collection

While I did not originally implement it, I (along with another engine team member) ended up inheriting an incremental garbage collection implementation after the original author left (and it ended up needing a lot of work). This engine change took the reachability phase of garbage collection and split it up over multiple frames. Now-a-days this is something Epic supports out of the box (thankfully, it’s still new and I even reported some issues with it to Epic when it first came out), but this was something we ended up building ourselves and getting it stable enough to ship for The Outer Worlds. I am not more familiar with how garbage collection in Unreal works than I would like to be.

Optimizations

While I originally started working on the project as a gameplay programmer (/not focused on performance aspects of the game) later during the project I shifted heavily to optimization/memory work- commonly working as part of the engine team to help optimize the game across all platforms (PC/Xbox One/PS4). One of the major areas that I focused on for performance was weapon performance (which isn’t surprising given how much I worked on them)- specifically weapon firing and impacts could be quite expensive due to the amount of VFX they would spawn, the actual traces/spawns for firing the weapon, damage calculations, UI damage number spawning, ‘death’s being triggered, and more.
While a lot of minor optimizations were made throughout the code the main focuses were on.

  1. We created a particle pooling system- this was used for most weapon and death VFX (so they pulled from a pool instead of spawning from scratch).
  2. We used actors pools for projectiles as well as UI damage numbers (these used actors with widget components so they could ‘move’ in 3D space)- instead of spawning a unique actor each time they were needed we instead pulled actors from a pre-spawned pool and reused them.
  3. Shots from weapons that spawned multiple shots was ran over multiple frames- our shotguns for example used to make 6+ shots in a single frame (and it was almost as expensive as just shooting 6+ separate weapons- with only the firing VFX being shared). By swapping us to run a single shot per frame for these weapons we were able to spread the cost out over multiple frames with no noticeable impact for the player.

Memory

Memory ended up being a major focus of mine as well later on the project, with me pretty much being the main expert on the team for dealing with memory. There are two main things I did worth discussing here.

Custom Memory Report

I made our own memory report command that was like Epic’s but extended in many ways (basically any time we found something new causing a memory problem we added a report to the command to dump info about it). This command gave general memory information (overall usage, free, etc), memory info about every object in memory, a summary for memory usage per class, audio memory usage, texture memory usage (including mip based streaming info such as desired size, loaded size, etc), information about loaded sublevels, a render target dump, and more. When investigating memory in an area the first thing we do is load it up and dump one of these reports to pick through it and look for issues.

‘LLMTrack’

I extended the LLM from Epic to help us track down memory leaks. The LLM was somewhat new when we started using it on The Outer Worlds (it’s intended as a way to tag memory allocations into certain categories and then it will track memory usage across these categories. The ‘low level’ comes from the fact that it hooks into memory management/allocation at the lowest level and tags memory as it’s allowed/freed so nothing is missed- it avoids a lot of the problems that manual tracking has and it ensures everything in the engine is tracked (even if some of that memory is just categorized as ‘untagged’). Because the autoplayer was tracking memory stats over time we could see in builds running the LLM how tags were behaving over time. We found that certain tags had continuous memory growth- indicating a leak. I extended the LLM system via modifications to allow marking a specific tag as ‘tracked’- when tracked all allocations made under that tag will record their callstack (and then these will be forgotten when the allocation is freed). These callstacks could then be dumped to the log- letting you analyze outstanding memory allocations under a certain tag. (This was done as an extension of the LLM as we already knew we needed to focus our attention on specific tags from the stats we had, and by limiting what allocations we are tracking the game can stay generally performant while doing this tracking). This system led to us fixing the vast majority of memory leaks on The Outer Worlds prior to shipping (many of which were leaks in the engine itself that we reported back to Epic). This work was instrumental in getting us to hit our stability goal for The Outer Worlds- we could be up 2+ days without crashing (either due to bugs or from running out of memory).