client side mods not working when enabled on dedicated server

[brunobf]

Hello all,

 

I'm trying to run a dedicated server which is enabling both server and client side mods. For reference, here's a copy of my modconfig.lua:

modLocation = ""
forceEnabling = false

-- Docs: https://avorion.gamepedia.com/Using_Mods_on_Dedicated_Servers

mods =
{
    {workshopid = "1731575231"},  -- Auto Research By: Rinart73
    {workshopid = "1722652757"},  -- AzimuthLib - Library for modders By: Rinart73
    {workshopid = "1722261398"},  -- Compass-like Gate Pixel Icons By: Rinart73
    {workshopid = "1756645965"},  -- Crewboard Tweaks By: Rinart73
    {workshopid = "1765619430"},  -- Galaxy Map QoL By: Rinart73
    {workshopid = "1734791653"},  -- Gate Founder By: Rinart73
    {workshopid = "1723188393"},  -- Improved Energy Suppressor By: Rinart73
    {workshopid = "1728695059"},  -- NPC Respawn By: Rinart73
    {workshopid = "1769379152"},  -- Resource Display By: Rinart73
    {workshopid = "1722256729"},  -- Sector Overview By: Rinart73
    {workshopid = "1776283892"},  -- Trading Overview Fixes By: Rinart73
    {workshopid = "1754114031"},  -- Trading Tweaks By: Rinart73
    {workshopid = "1722263986"},  -- Transfer Cargo Tweaks By: Rinart73
    {workshopid = "1691591293"},  -- claim.lua claim() hook By: Laserzwei
    {workshopid = "1695671502"},  -- Laserzwei's Advanced Shipyard By: Laserzwei
    {workshopid = "1691539727"},  -- Laserzwei's Move Asteroids By: Laserzwei
    {workshopid = "1694550170"},  -- Laserzwei's Sector Manager By: Laserzwei
    {workshopid = "1692998037"},  -- Laserzwei's simple asteroid respawn mod By: Laserzwei
    {workshopid = "1745027189"},  -- Sell Asteroid - Extended By: Finite
    {workshopid = "1746985221"},  -- IFTL Library [Current v1.3] By: IFoundTheLight
    {workshopid = "1747690472"},  -- IFTL Loot Display [Current v1.3] By: IFoundTheLight
    {workshopid = "1746985158"},  -- IFTL Player Gate [Current v1.3] By: IFoundTheLight
    {workshopid = "1751636748"},  -- Detailed Turret Tooltips By: MassCraxx
    {workshopid = "1751540023"},  -- Fleet Jump through Gate Command By: MassCraxx
    {workshopid = "1751798934"},  -- System Scanner Upgrade By: MassCraxx
}

-- these are allowed mods, that can run on the client only (ie. UI mods)
allowed =
{
}

 

On the server, I don't see any issues, and the console doesn't produce any errors or warnings.

 

When I try to connect a client to the server though, I see the following issues:

Warning: Couldn't add script from stream: 12ClientPlayer: Script 'H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua' not found. Tried the following paths:
data/scripts/player/H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua
data/scripts/player/H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua.lua
data/scripts/playerH:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua
data/scripts/playerH:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua.lua
data/scripts/quest/H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua
data/scripts/quest/H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua.lua
data/scripts/questH:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua
data/scripts/questH:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua.lua
H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua
H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua.lua
data/scripts/H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua
data/scripts/H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua.lua
data/scriptsH:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua
data/scriptsH:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua.lua

Modders: Make sure to enable Dev-Mode while writing scripts, or files you created while the game was already running might get ignored.
Warning: Couldn't add script from stream: 12ClientPlayer: Script 'H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1694550170\data/scripts/player/sectorOpener.lua' not found. Tried the following paths:
data/scripts/player/H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1694550170\data/scripts/player/sectorOpener.lua
<...snip....>
data/scriptsH:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1694550170\data/scripts/player/sectorOpener.lua.lua

Modders: Make sure to enable Dev-Mode while writing scripts, or files you created while the game was already running might get ignored.
Warning: Couldn't add script from stream: 12ClientPlayer: Script 'H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1734791653\data/scripts/player/gatefounder.lua' not found. Tried the following paths:
<...snip....>

Modders: Make sure to enable Dev-Mode while writing scripts, or files you created while the game was already running might get ignored.
Warning: Couldn't add script from stream: 12ClientPlayer: Script 'H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1765619430\data/scripts/player/map/galaxymapqol.lua' not found. Tried the following paths:
<...snip....>

Modders: Make sure to enable Dev-Mode while writing scripts, or files you created while the game was already running might get ignored.
<Server> Player BBFz0r joined the galaxy
Player BBFz0r joined the galaxy
Warning: Couldn't add script from stream: 12ClientPlayer: Script 'H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua' not found. Tried the following paths:
data/scripts/player/H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua
<...snip....>

 

The server is being spawned with the following command:

bin\AvorionServer.exe --galaxy-name GalaxyModSeason --seed ModSeason234 --datapath H:/Games/Avorion --admin <...snip...> --collision-damage 0.25 --use-steam-networking 1 -t all

 

So the save for the galaxy is located at:

H:\Games\Avorion\GalaxyModSeason

 

And  the location that is being used for the workshop mods is:

H:\Games\Avorion\GalaxyModSeason\workshop\content\445220

 

Somehow, that absolute server path is being sent to the client side, and being expanded by package.path.

Am I'm missing something in my configuration, or is this a bug?

 

 

 

 

[brunobf]

By the way, I noticed that there is some log corruption happening on the server. It seems like calls to the logger are not thread safe.

 

Example:

Call to 'sectorOpener.initialize' in '"H:\GameFilesecurer destination content size: s\A649274vor
ion\GalaxyModSeason\workshop\content\445220\1694550170\data/scripts/player/sectorOpener.lua"' successful.

I think this is supposed to be:

Filesecurer destination content size: 649274
Call to 'sectorOpener.initialize' in '"H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1694550170\data/scripts/player/sectorOpener.lua"' successful.

 

[koonschi]

For some reason the server is sending script information to the connecting clients with full paths, instead of relative paths. Did you add scripts manually at some point and use the full path (H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua) instead of a relative path (data/scripts/player/azimuthlib-clientdata.lua)?

[koonschi]

Second question: Is it possible that you've disabled a mod? Does this issue happen only with specific mods or with all mods?

[koonschi]

The issue has been resolved, thanks for the report!

[brunobf]

Hi koonschi,

 

Sorry for not noticing your previous requests for more info, I never received any email from the forum that I had replies on this thread.

 

Although the bug has been fixed, to address your previous questions, I had created a brand new galaxy when providing the reproduction steps.

On the client side I had also unsubscribed from all mods after I noticed the issue, and the issue did not manifest on all mods, only 4 of them.

 

Anyways, thank you for fixing the bug. Is the fix coming out on the 0.23.x stable branch, or only eventually on 0.24.x?

[koonschi]

No worries! Thanks for the infos. The fix will be coming to all live branches, including 0.23.*

[brunobf]

So I noticed the new manifests in the Steam depots, and decided to try out the new build in hopes the fixes were included. And apparently they were :)

 

Unfortunately though, once I did  that the server begun crashing. Initially I thought I would get away by just disabling the offending mods, which did not work. I noticed the logs had references even to the disabled mods. So I renamed the workshop download folders, and that made some difference, but not enough to fix it.

 

Since lua scripts are attached to game objects/entities, and they contain absolute paths, the server is very unhappy with it's current state on this new build. Are there any asserts() for invalid state? Player profiles are not being loaded correctly, and blank ones are been attached to existing players. It's not a happy sight.

 

I had to rollback the galaxy save to before updating, and then manually download the previous depots to keep players on the server playing while something can be done.

 

At this point I'm considering if I should write a mod that removes scripts from the entities when it detects absolute paths on them, and even if that will be enough to prevent the latest crashes. I'm attaching the logs from the crashes in case anyone has a better idea on how to fix the state in the saves.

 

Thanks

Server_Logs.zip

[brunobf]

It doesn't seem like I will be able to fix this using a mod. I think the engine just throws an exception when it detects an absolute path attached  to an entity.

 

It would be great if instead it either:

A) Removed the offending script from the entity automatically (Alliance, Entity, EntityDescriptor, Player, Sector)

B) Ignored the script entry from the entity

 

Option A would be best since it would fix the corrupted saves.

Option B would at least allow me to write a script to clean up the save :) (although while I can check for colon on Windows saves and fix that, I would not be able to test a Linux/Mac path using lua file system restrictions, or assuming that we can never have /data/scripts twice)

 

if onServer() then
local p = Player()
for sid, script in pairs(p:getScripts()) do
	printlog("Player() sid %d script %s", sid, script)
	if string.find(script, ':') then -- <driver>:<path> <-- Absolute on Windows
		printlog("Found invalid char in script, deleting due to absolute path: %s", script)
		p:removeScript(script)
	end
end
end

 

Update: Apparently running on the previous version of the executable, when I list all the scripts in the player, they are all relative paths. So I don't think Option B is viable at all. The only way I can expect to recover the save if I want to upgrade is if some fix is implemented in the engine. :(

 

 

Thanks

[koonschi]

I noticed in your logs that you removed a mod with ID 1694550170 from the mod config. That's probably why it's crashing. I've now fixed the crash, but disabling mods can still lead to errors and error messages.

 

Patch will go live soon!

[brunobf]

Hi koonschi,

 

In some of those logs I had temporarily removed some of the mods from modconfig.lua because they were the one that were causing the warnings in the logs. After I discovered that the references (absolute) to the file paths were stored directly on the entities, I left all them back on my modconfig.lua.

 

I've updated to your latest build, 0.23.4.16335, and the server still crashes.

 

On the previous build, I had noticed on the logs that following pattern:

 

.... (snip) ....
Avorion server Beta 0.23.4 r16262 79f7b3aecb15 running on Windows 10 starting up in "H:/Games/Avorion\GalaxyModSeason_alt"
.... (snip) ....
Trying to load player 76561197993124810 [bBFz0r]
Warning: Found an absolute path while loading script information: "H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua"

=== STACKTRACE =====================================
#1: ?? [0xafb41c]
#2: ?? [0xb732a9]
.... (snip) ....
Trying to load player 76561197993124810 [bBFz0r]
.... (snip) ....
Trying to load player 76561197993124810 [bBFz0r]
.... (snip) ....
FactionDataBase failed to load player 76561197993124810 [bBFz0r], error: 12ServerPlayer: Script 'H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua' not found. Tried the following paths:
.... (snip) ....
New Player registered: 76561197993124810 [bBFz0r] index: 81

 

Player's were getting a new profile when logging in because of their corrupted profile, which after a few retries (3?) the engine would just give up.

That was happening for Players and Sectors , and they are being wiped whenever they could not  be loaded.

 

Please note that the galaxy for the server is actually under H:\Games\Avorion\GalaxyModSeason_alt, and the reference above is to the original location when the server was first started (which I had to rename to make sure it wasn't actually loading content from that path).

 

I noticed on the new build / log a few entries of the following line:

Script will be removed from this scriptable object to avoid further errors.

 

I still could not find an entry that logged the successful load of the player profile, or a new one being created, before the crash though.

 

I've attached an updated server log below. Let me know if there's anything else that could also help.

 

Would a sample player.dat help ? :)

 

 

serverlog_2019-07-22_20-14-32.zip

[koonschi]

I'm afraid there is not much I can do by now. I can see in the logs that you've moved the server around quite a bit and it seems that absolute paths were saved a lot (a bug that is now fixed). Those absolute paths are now incorrect because the server was moved and thus cause the errors.

 

If you manage to reproduce this error with the current version you're having, then please post the steps to reproduce.

 

The errors with the broken scripts should all be corrected after some shutting down and restarting the server and having players log in. Does this happen to players who register as completely new players?

[brunobf]

I can't say what happens with new players on the current version since I'm stuck at running the previous version.

 

Unfortunately the errors with the broken scripts aren't being fixed by the server because the server still crashes before it can save its state.

It also doesn't seem that a minidump is being generated and uploaded with the crash.