Create
Abort Abort If Abort If Condition Is False Abort If Condition Is True Add Health Pool To Player Allow Button Apply Impulse Attach Players Big Message Break Call Subroutine Cancel Primary Action Chase Global Variable At Rate Chase Global Variable Over Time Chase Player Variable At Rate Chase Player Variable Over Time Clear Status Communicate Continue Create Beam Effect Create Dummy Bot Create Effect Create HUD Text Create Homing Projectile Create Icon Create In-World Text Create Progress Bar HUD Text Create Progress Bar In-World Text Create Projectile Create Projectile Effect Damage Declare Match Draw Declare Player Victory Declare Round Draw Declare Round Victory Declare Team Victory Destroy All Dummy Bots Destroy All Effects Destroy All HUD Text Destroy All Icons Destroy All In-World Text Destroy All Progress Bar HUD Text Destroy All Progress Bar In-World Text Destroy Dummy Bot Destroy Effect Destroy HUD Text Destroy Icon Destroy In-World Text Destroy Progress Bar HUD Text Destroy Progress Bar In-World Text Detach Players Disable Built-In Game Mode Announcer Disable Built-In Game Mode Completion Disable Built-In Game Mode Music Disable Built-In Game Mode Respawning Disable Built-In Game Mode Scoring Disable Death Spectate All Players Disable Death Spectate Target HUD Disable Game Mode HUD Disable Game Mode In-World UI Disable Hero HUD Disable Inspector Recording Disable Kill Feed Disable Messages Disable Movement Collision With Environment Disable Movement Collision With Players Disable Nameplates Disable Scoreboard Disable Text Chat Disable Voice Chat Disallow Button Else Else If Enable Built-In Game Mode Announcer Enable Built-In Game Mode Completion Enable Built-In Game Mode Music Enable Built-In Game Mode Respawning Enable Built-In Game Mode Scoring Enable Death Spectate All Players Enable Death Spectate Target HUD Enable Game Mode HUD Enable Game Mode In-World UI Enable Hero HUD Enable Inspector Recording Enable Kill Feed Enable Messages Enable Movement Collision With Environment Enable Movement Collision With Players Enable Nameplates Enable Scoreboard Enable Text Chat Enable Voice Chat End For Global Variable For Player Variable Go To Assemble Heroes Heal If Kill Log To Inspector Loop Loop If Loop If Condition Is False Loop If Condition Is True Modify Global Variable Modify Global Variable At Index Modify Player Score Modify Player Variable Modify Player Variable At Index Modify Team Score Move Player To Team Pause Match Time Play Effect Preload Hero Press Button Remove All Health Pools From Player Remove Health Pool From Player Remove Player Reset Player Hero Availability Respawn Restart Match Resurrect Return To Lobby Set Ability 1 Enabled Set Ability 2 Enabled Set Ability Charge Set Ability Cooldown Set Ability Resource Set Aim Speed Set Ammo Set Crouch Enabled Set Damage Dealt Set Damage Received Set Environment Credit Player Set Facing Set Global Variable Set Global Variable At Index Set Gravity Set Healing Dealt Set Healing Received Set Invisible Set Jump Enabled Set Jump Vertical Speed Set Knockback Dealt Set Knockback Received Set Match Time Set Max Ammo Set Max Health Set Melee Enabled Set Move Speed Set Objective Description Set Player Allowed Heroes Set Player Health Set Player Score Set Player Variable Set Player Variable At Index Set Primary Fire Enabled Set Projectile Gravity Set Projectile Speed Set Reload Enabled Set Respawn Max Time Set Secondary Fire Enabled Set Slow Motion Set Status Set Team Score Set Ultimate Ability Enabled Set Ultimate Charge Set Weapon Skip Skip If Small Message Start Accelerating Start Assist Start Camera Start Damage Modification Start Damage Over Time Start Facing Start Forcing Dummy Bot Name Start Forcing Player Outlines Start Forcing Player Position Start Forcing Player To Be Hero Start Forcing Spawn Room Start Forcing Throttle Start Game Mode Start Heal Over Time Start Healing Modification Start Holding Button Start Modifying Hero Voice Lines Start Rule Start Scaling Barriers Start Scaling Player Start Throttle In Direction Start Transforming Throttle Stop Accelerating Stop All Assists Stop All Damage Modifications Stop All Damage Over Time Stop All Heal Over Time Stop All Healing Modifications Stop Assist Stop Camera Stop Chasing Global Variable Stop Chasing Player Variable Stop Damage Modification Stop Damage Over Time Stop Facing Stop Forcing Dummy Bot Name Stop Forcing Player Outlines Stop Forcing Player Position Stop Forcing Player To Be Hero Stop Forcing Spawn Room Stop Forcing Throttle Stop Heal Over Time Stop Healing Modification Stop Holding Button Stop Modifying Hero Voice Lines Stop Scaling Barriers Stop Scaling Player Stop Throttle In Direction Stop Transforming Throttle Teleport Unpause Match Time Wait Wait Until While
Ability Charge Ability Cooldown Ability Icon String Ability Resource Absolute Value Add All Damage Heroes All Dead Players All Heroes All Living Players All Players All Players Not On Objective All Players On Objective All Support Heroes All Tank Heroes Allowed Heroes Altitude Of Ammo And Angle Between Vectors Angle Difference Append To Array Arccosine In Degrees Arccosine In Radians Arcsine In Degrees Arcsine In Radians Arctangent In Degrees Arctangent In Radians Array Array Contains Array Slice Assist Count Attacker Backward Button Char In String Closest Player To Color Compare Control Mode Scoring Percentage Control Mode Scoring Team Cosine From Degrees Cosine From Radians Count Of Cross Product Current Array Element Current Array Index Current Game Mode Current Map Custom Color Custom String Damage Modification Count Damage Over Time Count Direction From Angles Direction Towards Distance Between Divide Dot Product Down Empty Array Entity Count Entity Exists Evaluate Once Event Ability Event Damage Event Direction Event Healing Event Player Event Was Critical Hit Event Was Environment Event Was Health Pack Eye Position Facing Direction Of False Farthest Player From Filtered Array First Of Flag Position Forward Game Mode Global Global Variable Has Spawned Has Status Heal Over Time Count Healee Healer Healing Modification Count Health Health of Type Hero Hero Being Duplicated Hero Icon String Hero Of Horizontal Angle From Direction Horizontal Angle Towards Horizontal Facing Angle Of Horizontal Speed Of Host Player Icon String If-Then-Else Index Of Array Value Index Of String Char Input Binding String Is Alive Is Assembling Heroes Is Between Rounds Is Button Held Is CTF Mode In Sudden Death Is Communicating Is Communicating Any Is Communicating Any Emote Is Communicating Any Spray Is Communicating Any Voice line Is Control Mode Point Locked Is Crouching Is Dead Is Dummy Bot Is Duplicating Is Firing Primary Is Firing Secondary Is Flag At Base Is Flag Being Carried Is Game In Progress Is Hero Being Played Is In Air Is In Alternate Form Is In Line of Sight Is In Setup Is In Spawn Room Is In View Angle Is Jumping Is Match Complete Is Meleeing Is Moving Is Objective Complete Is On Ground Is On Objective Is On Wall Is Portrait On Fire Is Reloading Is Standing Is Team On Defense Is Team On Offense Is True For All Is True For Any Is Using Ability 1 Is Using Ability 2 Is Using Ultimate Is Waiting For Players Last Assist ID Last Created Entity Last Created Health Pool Last Damage Modification ID Last Damage Over Time ID Last Heal Over Time ID Last Healing Modification ID Last Of Last Text ID Left Local Player Local Vector Of Magnitude Of Map Mapped Array Match Round Match Time Max Max Ammo Max Health Max Health of Type Min Modulo Multiply Nearest Walkable Position Normalize Normalized Health Not Null Number of Dead Players Number of Deaths Number of Eliminations Number of Final Blows Number of Heroes Number of Living Players Number of Players Number of Players On Objective Number of Slots Objective Index Objective Position Opposite Team Of Or Payload Position Payload Progress Percentage Player Carrying Flag Player Closest To Reticle Player Hero Stat Player Stat Player Variable Players In Slot Players On Hero Players Within Radius Players in View Angle Point Capture Percentage Position Of Raise To Power Random Integer Random Real Random Value In Array Randomized Array Ray Cast Hit Normal Ray Cast Hit Player Ray Cast Hit Position Remove From Array Right Round To Integer Score Of Server Load Server Load Average Server Load Peak Sine From Degrees Sine From Radians Slot Of Sorted Array Spawn Points Speed Of Speed Of In Direction Square Root String String Contains String Length String Replace String Slice String Split Subtract Tangent From Degrees Tangent From Radians Team Of Team Score Text Count Throttle Of Total Time Elapsed True Ultimate Charge Percent Up Update Every Frame Value In Array Vector Vector Towards Velocity Of Vertical Angle From Direction Vertical Angle Towards Vertical Facing Angle Of Vertical Speed Of Victim Weapon Workshop Setting Combo Workshop Setting Hero Workshop Setting Integer Workshop Setting Real Workshop Setting Toggle World Vector Of X Component Of Y Component Of Z Component Of

Help, my server is crashing! A guide on how to improve server stability for your codes Last updated March 16, 2024

The workshop provides a lot of super useful tools, but start to use too many of them at once and suddenly the server will start crashing. This tutorial will serve as a guide to improve the stability of your codes and to prevent the server from crashing.

Content

Disabling the inspector

The inspector is used for debugging, and can contribute to server load, particularly when modifying arrays.

If you have finished debugging and are releasing the gamemode, it is recommended to disable the inspector using the Disable Inspector Recording action.

Waitless loops

A waitless loop is any kind of loop without a Wait action. Depending on the duration and number of actions being executed in the loop, the server could crash.

For Global Variable(A, 0, 100, 1);
  Small Message(All Players(All Teams), Custom String("I'm looping!"));
End;

A waitless for loop. This will loop 100 times in 1 server tick, likely crashing your server.


rule("Looping") {
  conditions {
    Is Button Held(Event Player, Button(Interact)) == True;
  }

  actions {
    Small Message(All Players(All Teams), Custom String("I'm looping!"));
    Loop If Condition Is True;
  }
}

A rule with a waitless loop. This will show a Small Message while the player is holding down the Interact button, and repeat it continuously via the Loop If action until the button is no longer held. The action will execute hundreds of times, potentially resulting in a server crash. A server crash is more likely if ran for multiple players at once.

Solution

The easiest way to solve this is to add a Wait action to your loop. Adding a Wait will improve the overall server load of your mode by delaying the actions running in the loop. It does not matter where you add this wait, but it is typically added just before the end of the loop.

If it is not necessary for things to happen at the same time, batching actions can be considered:

< Your actions >

Global.batch++
if Global.batch == 20
  Wait(0.016)
  Global.batch == 0
end

In this example, we loop over something 20 times, then wait for least amount of time possible (0.016 seconds), and then continuing the loop.


Condition ordering

Conditions in a rule are not checked all at once but sequentially one by one in order. While a condition remains false, the conditions below it will not be checked.

rule("My Rule")
{

    event
    {
        Ongoing - Each Player;
        All;
        All;
    }

    conditions
    {
        Is Button Held(Event Player, Button(Interact)) == True;
        Is Alive(Event Player) == True;
    }
}

A rule that runs when the Interact button is held by a player and the player is alive. The first condition is checked before the second condition. The player will never be checked if they're alive until after they hold the Interact button.

Why Condition order matters

Since condition order determines how conditions are checked, we need to be careful about what conditions we use and how we use them.

The examples provided below are super simple and the condition order will not make a noticeable difference, but with more rules improper ordering can increase server load.

rule("I'm alive") {
  conditions {
    Is Button Held(Event Player, Button(Interact)) == True;
    Is Alive(Event Player) == True;
  }
}

rule("I'm dead") {
  conditions {
    Is Button Held(Event Player, Button(Interact)) == True;
    Is Dead(Event Player) == True;
  }
}

In this example, we have two rules where one fires if the Interact button is held and the player is alive, and one that fires if the Interact button is held and the player is dead. The condition for checking if the button is held is first in both rules, therefore both rules will be checked, even though only one of them can run, since the player cannot be both dead and alive.


Is True For Any(Global.PositionsArray, Distance Between(Current Array Element, Event Player) <= 2) == True;
Is Button Held(Event Player, Button(Interact)) == True;

In this example, if the player is within range of any position within the array and is holding Interact, something happens. The condition for checking if the player is within range of a position is a more complex calculation. It will be constantly checking, regardless if the player is holding Interact. Only when the player is within range of a position will it check if the player is also holding down Interact.

Solution

  • Avoid multiple rules sharing the same conditions as the first condition.
  • Avoid having conditions that involve complex calculations as the first condition in a rule.

With this in mind we would want to swap conditions around, with the least frequently changing condition first.

Optimal ordering of conditions can ensure that conditions are only checked when they need to be, reducing the server load and potential for the server to crash.

Condition order can have a huge impact in larger modes, particularly those that utilize many rules.

Is Button Held(Event Player, Button(Interact)) == True;
Is True For Any(Global.PositionsArray, Distance Between(Current Array Element, Event Player) <= 2) == True;

In this example, if the player is holding Interact and is within range of any position within the array, something happens. The condition for checking if the player is holding Interact is first, therefore checking the player's position is done after the player is holding down the button.


Player filters

All player rules include a Player filter, which determines what hero or slot the rule will run for.

A player rule

A player rule in the Workshop with the Player filter set to All. This rule will run for all players, on all heroes, and all slots.

Using the Player filter of a player rule can be an alternative to checking for the hero via conditions. The examples below provide two similar rules: one that uses conditions, and one that uses the Player filter.

rule("Is Mercy") {
  event {
    Ongoing - Each Player;
    All;
    All;
  }
  conditions {
    Hero Of(Event Player) == Hero(Mercy);
  }
}

A player rule that checks if a player is playing as Mercy.
This rule's condition will be checked for every player, even if they are not Mercy.


rule("Is Mercy")
{
    event
    {
        Ongoing - Each Player;
        All;
        Mercy;
    }
}

A player rule with the Player filter set to Mercy. This rule will run for any players on the hero Mercy, and only for those players.

By limiting the number of rules and conditions that are being reevaluated for each player, it will be easier for the server to handle large number of rules that are used for different heroes/slots.

Notes

  • When a hero or slot is selected in the Player filter, the rule will abort the moment the player swaps to a different hero or different slot.

Too many actions

The server can only handle so many actions at once (the exact number will completely depend on what you're doing). Let's say we have a gamemode, and when you press Interact a whole bunch of stuff starts happening. You're initializing all sorts of variables, you're teleporting all players, etc. You end up with over 20 actions in 1 rule. All these actions will try to execute in the same frame. Meaning they will essentially all try to execute before moving on to whatever is next.

The solution here is simple; add a Wait action! After a certain number of actions, simply add a short wait. Anything goes, no matter how short.


Too many conditions at start up

Is your server crashing right from the start? You may have too many conditions that need to be checked at the start. When the server first starts it will need to go by every single rule and check all of their conditions. If you have 100+ rules this may start to be a problem.

In some cases this can be difficult to fix, but there's a few things you can try.

  • Defer expensive actions to a later point. If you have rules that for example create HUDs, In-World text, or simply set a variable, consider delaying them a little bit. For example only initiate them when the player has actually spawned. Or simply add a start Wait.
  • Make sure expensive conditions that don't need to be checked yet don't get checked yet. Chances are you have some expensive conditions (with complex calculations or checking large arrays) that don't need to be checked the very second the server starts. Hide them behind other conditions that are definitely not true at start up so these more expensive condition don't need to be checked. Refer back to Condition Order for more info. You could use conditions such as Total Time Elapsed or Number Of Living Players to defer these rules to when they are actually needed.
  • Merge multiple rules in to 1. You might be able to get away with merging rules with similar conditions in to 1, and using If Else statements in the actions instead. These If Else actions are more expensive, but if all you need is more headroom at start up, this could be an option.
  • Avoid putting all conditions in the same event type (Ongoing - Each Player, Ongoing - Global ...). A healthy mix of events is generally easier for the server to handle.

De-syncing expensive actions

Say you have a Ongoing - Each Player rule that sets up a large number of variables, creates new effects and several hud texts. A rule like this might be perfectly fine for the server to handle one its own, but not when attempting to run it several times on the same server tick (which is often the case at startup).
A quick peak from 100 to 255 server load can be worse than a consistent 200 server load, and by de-syncing expensive actions we can reduce such peaks.
One way to de-sync actions at startup is to use a wait that is inconsistent between players.

    actions
    {
        Wait(Slot Of(Event Player) * 0.016, Ignore Condition);
    }

There are a number of ways to make sure expensive actions dont all line up on the same server tick, such as using variables to check if certain actions are currently running, or by combining several rules and adding waits inbetween actions. But in the end it all comes down to making good use of the Wait action.

Anti-crash rules

A way to reduce the chance of crash is to use anti-crash rules. It works by setting slow motion if the server load goes past a certain threshold ; slow motion lowers the tick rate, letting the server with more time to do the calculations.

variables
{
    global:
        127: antiCrashActivated
}

rule("anti crash") {
    event  {
        Ongoing - Global;
    }
    conditions {
        Global Variable(antiCrashActivated) == False;
        Server Load > 230;
    }
    actions {
        Wait(1, Abort When False);
        Small Message(All Players(All Teams), Custom String("Anti crash system activated", Null, Null, Null));
        Set Slow Motion(10);
        Set Global Variable(antiCrashActivated, True);
    }
}
rule("disable anti crash") {
    event {
        Ongoing - Global;
    }
    conditions {
        Global Variable(antiCrashActivated) == True;
        Server Load < 200;
    }
    actions {
        Set Slow Motion(100);
        Set Global Variable(antiCrashActivated, False);
    }
}

Although this is not a silver bullet (as your players might not like playing at 10% speed), it is very useful to prevent crashes due to spikes in load (eg if too many players use your custom ability at the same time).


Disabled rules or actions

Disabled rules, actions, or conditions do not affect performance. Nor do completely empty rules.

For these code examples we will use the @for format of the Workshop.codes editor to quickly create many rules.

Having too many conditions will instantly crash the server on start up, no matter how simple the condition is.

@for (1 through 500) {
    rule("Rule Name") { 
        conditions {
            Global.a == 1;
        }
    }
}

How many conditions the server can handle on startup before crashing is going to depend on how advanced they are, as well as what type of event triggers the condition. Combining Global and Each Player events allow for more conditions to be used before the server crashes. This is likely caused by players loading in later than the Global Events, causing a de-sync between condition checks.

@for (1 through 400) {
    rule("Rule Name") { 
        event {
            Ongoing - Each Player;
            All;
            All;
        }
        conditions {
            Global.a == 1;
        }
    }
}

@for (1 through 400) {
    rule("Rule Name") { 
        event {
            Ongoing - Global;
        }
        conditions {
            Global.a == 1;
        }
    }
}

Disabling the rule or the condition means they are not processed at all and they will no longer affect performance as a result.

@for (1 through 5000) {
  disabled rule("Rule Name") {  
        conditions {
            Global.a == 1;
        }
    }
}

@for (1 through 5000) {
  rule("Rule Name") {   
        conditions {
            disabled Global.a == 1;
        }
    }
}

Similarly, empty rules have no impact on performance and act the same as if they were disabled. They do still show up in the inspector unlike the disabled rules.

@for (1 through 5000) {
    rule("Rule Name") { 
        conditions {
        }
    }
}

Workshop quirks and pitfalls

Avoid using large arrays in conditions.
when any value within an array changes, every condition in any rule containing that variable is checked.
For example, if you set Event Player.upgrades[73] to = 1, the conditions will also need to be checked in rules with Event Player.upgrades[0], Event Player.upgrades[1] and so on. This can potentially trigger a lot of unnessecary condition checks.

Avoid using too many/expensive Player Dealt/Took Damage events
Certain abilities and weapons trigger significantly more damage events than others (attacks with a lot of damage instances such as SMGs, dots and beams). If possible, add a short wait at the end of the rule to prevent them from triggering too often. Just keep in mind that each rule can only run one at the time, so if you add a wait to a Player Dealt Damage rule, and that player damages several players on the same server tick, the rule will only apply to one target (there is only one attacker who can trigger the rule). If you swap the event to Player Took Damage, the rule can now trigger once per target (several victims, each one triggering their own respective rule). Because of this behavior you will need to be careful with how you use your waits, or you might prevent your rules from running when you want them to.

Using the built in array-actions
To a programmer coming to the workshop it might not make a lot of sense, but iterating over an array with For Global Variable can be a lot more expensive than doing the exact same thing with actions like Mapped Array / Filtered Array / Sorted Array. Working with arrays in general is pretty expensive in the workshop, especially multidimensional arrays. Using the built in array actions can be a good way to reduce server load when performing a lot of calculations on arrays.

"My mode is crashing today but worked fine yesterday??"
Server stability changes from day to day, and server load is generally higher in the evening than it is in the morning/night. It is also not uncommon to see more crashes on patch days, not because the patch is unstable, but because of the increase in player numbers. The best thing you can do is to improve the general performance of your mode so that it remains stable even when the servers are struggling.

Server load peaks every few seconds
This is a problem that has long been discussed within the workshop community. The cause is most likely the snapshots that are sent to the server containing data for the replay system, kill cam, play of the game. After the game has been running for 1h 30min the replay system and kill cams will turn off, and the periodic server load peaks will disappear along with it. Unfortunately there is no other known way to turn off or avoid this system. The best you can do is to reduce its impact by not having too many bots/players, texts and effects.

"Heroes are randomly disappearing from the hero select screen"
This is something that has been happening since the launch of OW2 to modes that are dangerously close to breaking the limits of what the workshop can handle. Working to improve the performance of your mode and reducing server load should fix the issue.

Testing your changes in the Practice range
Unknown to many, the practice range runs on a different tick rate than the remaining maps. Instead of the usual 62.5 ticks/s the practice range runs at a third of that, ~20.8333 ticks/s.
Unless you plan to build your mode around the practice range, it is generally better to test your changes on any other map.

Join the Workshop.codes Discord