Random Thoughts.....What Are You Thinking?

Hello, this is Naoki Yoshida, Producer & Director of FINAL FANTASY XIV.

Thank you all for participating in the Early Access and official launch of Endwalker.

Since the start of Early Access, we have been seeing an unprecedented number of active players, the highest ever seen since the launch of FFXIV, as well as record numbers of simultaneous logins, but along with this, the number of connections has far surpassed the scale of our services which has resulted in long wait times when attempting to log in.
　
Even during this congestion, we’ve received words of support and encouragement, for which we’re truly grateful. Thank you very much. On the other hand, we sincerely apologize for the long waiting times and the resulting frustration experienced by players attempting to log in.

I’d like to share some information regarding the ongoing congestion, the reasons behind errors that have been appearing while waiting in queues, and the current status of our fixes. For information on error codes not listed below, please see my previous announcement on December 5 (https://sqex.to/QDFMi).

■ Lobby server maintenance to address errors

Allow me to share some information regarding specific errors. We are planning for a full maintenance on the lobby servers at the earliest timing possible, in order to resolve these errors. Although the maintenance time isn’t fully solidified, it is estimated to take about 30 minutes. This will not affect any players who are already logged into the game, but during these 30 minutes, players will not be able to make new login attempts, and the login queue will be unavailable.

We deeply apologize for this but ask for your cooperation as we address these server errors. The exact maintenance times will be shared in an upcoming announcement.

■ All Worlds Maintenance

In the early morning of Wednesday, December 8, we will be conducting maintenance across all Worlds to address server errors aside from those mentioned above, as well as a few issues that can be resolved on the server side. This requires all servers to go offline and is estimated to take about four hours to complete. Details on maintenance timing will be announced in the future. I’d like to ask for your understanding and cooperation regarding this matter as well.

Below are some details regarding errors that have been occurring as a result of congestion.

■ Disconnects by Error 4004 right before logging in

Currently, I believe the most frustrating cases are when you’ve been waiting in the queue and get disconnected by an error when your turn comes around, forcing you to line up in the queue again. These disconnects are often happening as a result of Error 4004 and 3001, and here I’d like to provide the details and statuses of countermeasures against Error 4004.

・ How the Process Works

To prevent servers from going down as a result of mass logins or overloading, the process of logging into a World that has reached the login cap involves waiting until the World can admit 100 players, then admitting the first 100 players in the login queue, a process that is constantly being repeated.

・ The Issue

We have confirmed an issue where during the process of admitting these 100 players, if certain conditions coincide in combination while the servers are experiencing extremely high loads, up to 25 of these players will receive an error. This issue is not guaranteed to occur even when the servers are experiencing extremely high loads, and the number of players who experience the issue varies. However, considering login queues can take up to several hours and how an error can cause players to have to return to the back of the queue, we currently recognize this as the most frustrating issue.

・ Resolving the Issue

We have been able to locate the cause of this error that occurs due to extremely high server loads and are currently fixing the source code for the lobby servers. Although the fix itself is finished, the process of checking whether the fix has actually resolved this issue requires time as it is an extremely complex process. In addition, due to the complexity of the conditions of occurrence itself, it is possible that we have not yet addressed more detailed conditions, but we will reflect the fixes and continue to monitor the situation. As soon as our preparations are ready, we will carry out the work through the aforementioned lobby server maintenance. We ask that you wait a little longer until the fix is made.

■ Disconnects by Error 3001 right before logging in

I’d also like to provide the details and countermeasures regarding Error 3001.

・ Error 3001 Cause #1

A login cap is in place on every World in order to prevent them from going down due to high loads. Despite this, a number of extremely overloaded servers have allowed players to log in beyond this cap, resulting in the allowed number of lobby server connections to fail and cause Error 3001 and disconnect players in the queue.

・ Resolution for Cause #1

We currently have a general idea of what is causing this issue with extremely overloaded Worlds, and a fix is underway. However, these are areas where complex processing is carried out and careful fixing is required for this process. As such, fixing/confirming/implementation into the lobby server will require time. This will be implemented in the maintenance scheduled for December 8, and we are putting our full effort into it, but haven’t been able to determine the full details. Further details regarding this will be announced at a later timing as well.

・ Error 3001 Cause #2

This is an issue that is occurring only on the Tonberry World on the Elemental data center. The Tonberry World has the largest number of login requests of all the game Worlds, and players are helping to mitigate this issue by using the World Visit System and logging into other Worlds. However, due to the way chat connections work when using the World Visit System, we cannot completely cut off a player’s connection with their character’s Home World, causing the total number of active connections to become enormous, and we have confirmed that as a result, a situation similar to Cause #1 above can occur.

・ Resolution for Cause #2

Although we are aware of the relative risk of the Tonberry World going down, we’ve increased the number of network connections to its absolute maximum. Your patience would be greatly appreciated as we monitor the situation based on this tentative fix for the time being.

■ Error 2002

Now I would like to explain about the details related to Error 2002 and our countermeasures.

・ Details of the Process

In order to prevent a major login server outage, FFXIV has a limit of "no more than 17,000 total players waiting to log in per logical data center". This cap is set uniformly regardless of how many Worlds exist in a logical data center.

・ The Issue

If the total number of players waiting in the logical data center exceeds 17,000, Error 2002 will be displayed and you will not be able to queue up for login.

・ The Resolution for this Issue

We are now ready to deploy the backup development servers to the public lobby servers, which will improve the management capacity of the lobby server and raise the maximum number of players in the queue to around 21,000. Since this will not require any maintenance, we will err on the side of caution and proceed with the deployment outside of peak hours. It is scheduled to be completed late in the night of December 7, Japan time.


■ Regarding Measures Towards a Fundamental Resolution for Congestion

When it comes to fundamental measures against this type of large-scale congestion—and this is something that was brought up in the Lodestone announcement (https://sqex.to/4IkYs) we made ahead of early access—the most effective method would be a “large-scale addition of new Worlds in all regions.”

On the one hand, following the explosive rise in players coming to FFXIV beginning in spring 2021, we spent over a half a year scrambling to address the need to add new Worlds. However, due to policies surrounding COVID-19, which has had an effect on the entire world, even now we have been unable to procure the required hardware for adding new Worlds.

■ Regarding the Global Semiconductor Shortage (Supply Chain Issues)

When it comes to adding new Worlds, we need tens of “server machines” for every World that we add. Server machines are high-performance computers, which utilize numerous semiconductors. However, due to COVID-19 countermeasures currently put in pace, many factories across the globe which produce semiconductors have halted production, or have faced labor shortages—this has ultimately led to a decrease in the number of semiconductors being produced, and has resulted in a worldwide semiconductor shortage.

Furthermore, this does not only pertain to production of semiconductors, but from a logistics standpoint as well, COVID-19 countermeasures continue to explosively push up demand and create this turbulent situation. We have put in considerable investment—even more so than usual—to secure the needed hardware, but even so, the current fact of the matter is that a long lead time is needed to prepare the server hardware.

If we try to look to when this semiconductor shortage may be solved, unfortunately given the state of things, we aren’t able to estimate when we might reach a resolution. The FFXIV Server and Infrastructure teams accelerated work to implement the additional high-performance machines we had secured, as well as optimized the server source code and increased the number of maximum logins per World. I promise that we will continue to put in this type of effort.

Given the current situation, we anticipate that it may take several months or more to add new Worlds. I want to express my sincerest apologies to all of our players for the inconvenience this is causing. I appreciate your continued understanding and support regarding this matter.