Outriders - Post Launch Dev News Update

Hello everyone,

We would like to thank everyone in the Outriders community for your patience, support and assistance. Everyone on the Outriders team is continuing to work hard on improving the game and we'd like to share news about the things we are focusing on.

Please use CTRL + F to jump to the information that interests you the most.

• Patch News

  • tl;dr*: Patches are inbound for crashes, matchmaking issues and more

• Community Appreciation Package

  • tl;dr: Launch window players are getting a “thank you” package

• Inventory Wipes & Restoration

  • tl;dr: We’re working on restorations

• Re-balancing

  • tl;dr: We’re rebalancing a handful of certain skills

• Multiplayer Stability and Connection

  • tl;dr: We’re continuing to work hard on improving this

• Connectivity Post-Mortem

We have been continuing to track and post known workarounds for encountered issues in our Launch Tracking Mega-Thread as well as our Steam forums, but our intention is of course that workarounds shouldn't be necessary. So here is our latest patch news:

Patch News:

First Patch:

• This is our first big patch for the main Outriders game.
• As of today, the deployment of this patch is scheduled for next week, as we are using this week to identify as many severe issues as possible, fix them and then thoroughly test the changes across all platforms. Submission through platform holders adds on a little bit of time as well, which is why we are not able to deploy this patch any earlier.
• We are doing our utmost to deliver you a stable patch as soon as possible and we are continually looking at ways to bring the patch release day earlier. This is why we are not yet committing to a patch release date, as we may still be able to release the patch before such a date.
• We will let you know as soon as we have a confirmed release schedule, so please do keep an eye on our social channels and in particular our Twitter.

First Patch - Patch Notes:

• Will Fix a performance issue where GPU is not being fully utilized. This should help with stuttering and DX11/12 issues
• Once platforms have been updated to the same patch version, cross-play across platforms will become viable again
• Overall stability improvements for the matchmaking service
• Crash Fixes
  • Will Fix a multiplayer crash that could result in client players having their inventory wiped
  • Will fix the crash when completing the "A Bad Day" side quest.
  • Will fix the crash that occurs in No Man's Land when your language is set to Spanish (Yes, we know. Video game code is a magical thing).
  • Will fix crash on launch issues
  • Will include many more "random" crash fixes
  • We are confident that these fixes will address the majority of crashes reported, as there are only a handful of root causes but the crashes they generate appear in a number of places.

• Will fix the HUD disappearing in certain cases
• Will fix bugs that interfere with players re-spawning in multiplayer Expeditions
• Will fix bugs with players getting stuck on geometry (including when using Gravity Leap) or falling out of the world.
• Will change the default matchmaking setting from "Open" to "Closed".
  • You will still be able to manually change this setting to "Open" through your game settings
  • This change will prevent players from joining games where the host didn't intend to play in multiplayer. It will also cut down on AFK lobbies
  • This will also help improve matchmaking times, as the queues will be less likely to be overwhelmed by the sheer volume of constant matchmaking requests generated by "open" games.

• Many other minor fixes and improvements

Additional fixes and changes are being worked upon to be delivered via our later follow up patch

Outriders Community Appreciation Package:

After working on Outriders for more than five years, we were as disappointed as you that the launch weekend did not go as planned. We ran into connectivity issues, which we have explained in full further below.

Throughout all of this we really appreciated all your messages of support and encouragement and they helped us power through and improve the situation.

We would like to thank each and every one of you. Beyond just saying it, however, we would like to confirm that we are working on a small "appreciation package" for our launch window players. We are still working through the exact details of it, but our intentions are as follows:

• All players who played between March 31st and April 11th (UTC timezones) are eligible
• Any players outside the above window, but for whom we will have performed an automatic inventory wipe restoration (LINK) are also eligible
• Your highest levelled character will receive:
• A level appropriate Legendary Weapon
• A level appropriate amount of Titanium
• The emote "Frustration", which is otherwise unobtainable at this moment in time. The irony here was not intentional but is fitting.

These are our intentions, but these details are subject to change based on what is technically feasible for us to do.

We are still determining when the appreciation package will be delivered.

Inventory Wipes & Restoration:

We believe that the rate of this bug appearing should be greatly reduced once the upcoming patch has been published. With crash fixes in place, we intend to run a one-off mass restoration. Here are our intentions for this restoration:

• Restoration should take place on a specific day and should be a one-time event
• The exact date is TBC but we’re hoping for this to occur in the coming week(s)
• We will update you when we have confirmation of date and timings
• We will only be able to restore the most valuable tiers of items lost, e.g. Epics and Legendaries.
• We will be unlikely to restore an inventory full of blue gear.
• The restoration should be additive to your inventory. It should not replace your current inventory.
• Restored items may not be have the exact same stats as the lost items

Note that restoration will not be possible for items that were acquired but where the game connection was terminated before an autosave could complete as those items would never have made it into your server inventory/our logs. Such suspended autosaving is noticeable when the yellow autosave icon stays active for a longer period of time.

For any individual restorations that were missed, or that occur in future, we plan to equip our teams to help on an ad hoc basis.

Please note that these are our intentions that we are providing here for transparency. While we will try our hardest to make things right, we unfortunately cannot make guarantees for the specifics of the above or the timeline of this, as we may run into technical difficulties that delay our work.

The Accolades Wipe issue is something we're again looking into whether we can run as part of such a one-off restoration or whether we will need to tackle these cases on a one-to-one basis. Accolades are a more complex piece of data per profile than a piece of equipment, so working through a solution here is taking a little more time.

Re-balancing:

This re-balancing work can be implemented through backend updates, does not require a patch, and will be run during today’s maintenance window.

IMPORTANT: Balancing is done by a separate team to those who are working hard on preparing the patches. Re-balancing does not impact our progress towards patches or reduce our ability to resolve bugs.

General Balance Notes:

• Historian, bounty hunt, and monster hunt quests no longer grant Legendary items for each subsequent completion
  • While rewarding players for completing all the Historian, bounty hunt, and monster hunt quests is still our intention, we did observe unintended results when players repeated this activity in combination with a specific multiplayer bug.
  • With this rewards adjustment we want to eliminate this exploit behaviour while still making that quest line worth the effort.
  • Players who took advantage of this exploit will not be punished.

• Adjusted loot on specific enemies: Tooka Beast, Splittooth, Sandshifter and Coldclaw
  • These particular beasts are being too generous considering the required effort to take them down.

• Chem Plant - Expedition time for Gold/Silver/Bronze adjusted to 650 / 1150 / 1500 seconds (Previously: 957 / 1303 / 1650)
  
• Boom Town - Expedition time for Gold/Silver/Bronze adjusted to 390 / 589 / 776 seconds (Previously: 422 / 607 / 792)
  • Some Expeditions seem to be too generous in their required completion times when compared to others.
  • We want to avoid such disproportionate discrepancies, as we believe making all expeditions viable leads to a more diverse and thus more interesting endgame.

• Crit Rating Damage Multiplier on Tactical Assault Rifle reduced to 165% (Previously: 175%)
• Critical Rating Damage Multiplier on Marksman Rifle reduced to 250% (Previously: 300%)
  • Marksman Rifles with a 300% crit rate showed up to be dominating multiple crit builds. By making the modifier the same as for a Bolt Action Rifle, we want to offer players more options when choosing what gun to use.

• Duration of the Massacre AP buff has been reduced to 3 seconds
  • This is a temporary measure until we have fixed an issue with the stacking cap.

• Bullet abilities

  (Twisted Rounds, Blighted Rounds and Volcanic Rounds):

  • While we fully intend to provide the overpowered feel for the best builds out there, we are also committed to making sure that there is build diversity in the top performing builds.
  • Late in the demo lifecycle, we discovered a bug with regards to the bullet augmentation abilities damage calculation formula. While we fixed this bug, some unforeseen imbalances were unfortunately introduced, which has led to bullet based abilities vastly outperforming other skills. The main problem with this is that such bullet based builds require much less investment to make them "top tier", compared to other builds.
  • To achieve this, we have made a couple of adjustments for the Trickster and Technomancer classes, who benefit the most from their bullet abilities power. It's uncertain as to whether these changes will prove too strong or too weak, so please do consider this to be only the first step, as we will continue monitoring and adjusting them in the future. Just like you, we hope to see more inspiring and crazy combinations of Abilities and Mods!

• The base cooldown of the Twisted Rounds skill has increased to 25 seconds (Previously: 16 seconds)
  • While infinite uptime on bullets abilities was always our intention, it is intended to be high risk, high reward playstyle. However, given the current power of this skill, there is very little risk involved in this ability given its short cooldown, even when players fail to trigger ammo replenishment. Increasing its cooldown in combination with adjusting its damage should increase that risk and make this playstyle a more thrilling experience.

• The Trickster's Hero Tree nodes: - Disruptive Firepower, Scion of Power, Outrider Executioner - have had their power reduced to 35% (Previously: 50%)
  • The combination of Twisted Rounds' increase of firepower on top of the multipliers provided by the class tree lead to huge increases of damage dealt by this ability. While we do understand the satisfaction this kind of power can bring, it is currently vastly outperforming our wildest expectations and we therefore need to bring it down a notch.

• "Vulnerable" status - Power reduced to 15% (Previously: 25%)
• The Technomancer Hero Tree node -"Damage Against Poison" - has had its power reduced to 15% (Previously: 30%)
  • While the Technomancer's Blighted Rounds do not provide any damage increase on their own other than ignoring armor as Anomaly Damage, they do provide an excellent tool for spreading the Vulnerable and Toxic statuses. which in turn can scale out of proportion.

• The Pyromancer Hero Tree Node - "Trial of Ashes" - has been reduced 15% extra damage (Previously: 30%)

Multiplayer Stability and Connection:

Co-op and multiplayer sessions are a separate matter from solo play connectivity.
Launching a multiplayer session, requires multiple different partners and services. For example, matchmaking and queuing are hosted by one service partner, while the peer-to-peer relay is handled by another. Other components are hosted by other partners. This is a normal setup for multiplayer games but is important to mention because when we attempt to track multiplayer or co-op issues, knowing exactly what issue appears at what point is vital.

The upcoming patches should help alleviate crashes in multiplayer as well as improve stability and speed. Because of the complexity of the multiple systems involved, we will continue to improve and iterate upon these systems in order to deliver you a better multiplayer experience. In most cases, this can and has already been done via the backend without patching the game, but it is something we are keeping a close eye on.

Connectivity Post-Mortem:

tl;dr: Our team worked throughout the Easter weekend and around the clock to resolve the server issues players were experiencing. We completely understand how frustrating this experience will have been especially given the huge amount of players eagerly anticipating the launch. We had enough server scaling capacity but our externally hosted database was seeing issues that only appeared at extreme loads.

We’re committed to full transparency with you. Today, just as we have been over the past year.
So we won’t give you the expected “server demand was too much for us”.

We were in fact debugging a complex issue with why some metric calls were bringing down our externally hosted database. We did not face this issue during the demo launch earlier this year.
Our database is used to hold onto everyone’s gear, legendaries, profile and progression.

Tech-heavy insight:

We managed to understand that many server calls were not being managed by RAM but were using an alternative data management method ("swap disk"), which is too slow for the flow of this amount of data. Once this data queued back too far, the service failed. Understanding why it was not using RAM was our key challenge and we worked with staff across multiple partners to troubleshoot this.
We spent over two days and nights applying numerous changes and improvement attempts: we both doubled the database servers and vertically scaled them by approximately 50% (“scale-up and scale out”). We re-balanced user profiles and inventories to new servers. Subsequent to the scale-up and scale-out, we also increased disk IOPS on all servers by approximately 40%. We also increased the headroom on the database, multiplied the number of shards (not the Anomalous kind) and continued to do all we were able to in order to force data into RAM.

Each of these steps helped us improve the resilience of the database when under extreme loads, but none of them were the "fix" we were looking for.

At this moment in time we are still waiting for a final Root Cause Analysis (RCA) from our partners, but ultimately what really helped resolve the overloading issue was configuring our database cache cleaning, which was being run every 60 seconds. At this frequency the database cache cleaning operation demanded too many resources which in turn led to the above mentioned RAM issues and a snowball effect that resulted in the connectivity issues seen.

We reconfigured the database cache cleanup operations to run more often with fewer resources, which in turn had the desired result of everything generally running at a very comfortable capacity.

All of this has enabled the servers to recover and sustain significantly more concurrent user loads.