chore: Add linux support and containerize the project

[clouths]

Context

Setting up development environment for OpenPerpetuum requires Windows, clone of 3 repositories (OPResource, OPDB, PerpetuumServer2), Steam with the official PerpetuumServer installed to get access to database seed + some data files (customDictionnary, map layers, plantrules). Developing on linux is not possible without modifications. We also need to setup the database with migration script, setup the OPResource as a file server, then you can start the PerpetuumServer2 to connect a game client.

The goal of this PR is to reduce that friction to get started more quickly and support multiple development platform by using containers to be compatible on windows, mac, linux. Note: for windows, using linux containers would be a requirement to keep the initial implementation simple.

When porting the application to linux, one issue came up with the database DistributedTransactions. It turns out the .NET on linux does not support it. For now, it has been disabled but I'll try to find a solution or an alternative.

Description

• Add git submodules for OPResource (as asset) and OPDB (as db)
• Add .env.local where you can configure db password, container ports, asset url, various paths, connection string, type of .net build
• Add template for perpetuum.ini to get dynamic content based on the config
• Add dockerfile for the asset using nodejs to host the file server
• Add dockerfile for the migration that act as the OPDB/apply_all.bat but for linux + also handle updating the PerpetuumData content
• Add multi-stage dockerfile for the server
• Add Makefile to help run commands using docker-compose or equivalent
• Refactor System.Drawing into SkiaSharp
• Added 77 unit tests
• Added 5 integration tests

How to test

• Run unit tests

make test-unit

• Start the local stack (docker compose)

make up

• Run integration tests

make test-integration

Some technical information

• The database uses a local directory as the persistent storage in mssql-data
• The perpetuum-data folder is used during migration but the one used by the server is in a docker volume openperpetuum-data
• DistributedTransactions is not compatible with linux but is enabled by default in the server if you don't pass it to the server environment so the windows build is unchanged.

Improvements

• Ability to detect database migration and run if some change is detected by adding a database schema version to compare with
• Move the template/restore_DB_to_original_state.sql somewhere else
• Able to run DistributedTransactions enabled for linux or find an alternative
• readme to explain how to set up and document how it works
• move database files into a docker volume instead of mounted one

[aqpanaciy]

Regarding using DistributedTransactions, we use a single database and the connection string doesn't change. Therefore, using DistributedTransactions is not necessary. I'm not very knowledgeable about the intricacies of database operation, but after reading the documentation, I found out that:

1. After closing the connection to the database, it is not closed immediately, but is returned to the pool, and when reopened, it is taken from there.
2. For each request inside a TransactionScope, you can open and close your own connection; this is a well-known technique.

Therefore, you can remove: TransactionManager.ImplicitDistributedTransactions = true; in PerpetuumBootstrapper.cs
AND
Rremove: dbConnection.EnlistTransaction(Transaction.Current); in DbQuery.cs (This line of code is responsible for forcing the use of distributed transactions for the current transaction)

With these changes, I was able to launch the server on Linux. And there are no errors when working with the database. I hope this information will at least be useful.

[clouths]

Regarding using DistributedTransactions, we use a single database and the connection string doesn't change. Therefore, using DistributedTransactions is not necessary. I'm not very knowledgeable about the intricacies of database operation, but after reading the documentation, I found out that:

1. After closing the connection to the database, it is not closed immediately, but is returned to the pool, and when reopened, it is taken from there.

2. For each request inside a TransactionScope, you can open and close your own connection; this is a well-known technique.

Therefore, you can remove: TransactionManager.ImplicitDistributedTransactions = true; in PerpetuumBootstrapper.cs AND Rremove: dbConnection.EnlistTransaction(Transaction.Current); in DbQuery.cs (This line of code is responsible for forcing the use of distributed transactions for the current transaction)

With these changes, I was able to launch the server on Linux. And there are no errors when working with the database. I hope this information will at least be useful.

This PR disables it but I have left a default value (true) to keep the same behavior to keep compatibility for the Windows build. If removing the EnlistTransaction do the trick, we might be able to disable it if the live server using the Windows build don't have any issue.

[clouths]

At this point, the PR is almost ready for review.

The client is able to connect and perform all actions tested (chat, enter/leave chat, buy from market, unpack item, activate bot, undock/dock, move, assignments, looting, combat, tutorial, rift)

What is left todo?

• Add readme how how to setup development environment (server, assets, gamma layers) (can be done in a follow-up PR)

[clouths]

At this point, the documentation has been added and the DistributedTransactions config is now correctly applied in the codebase.

[aqpanaciy]

I successfully tested this branch in Docker.Desktop on Windows. I did have to manually update the submodules to the latest versions, though.

[clouths]

All unit and integration tests pass now.

Added Live_36 to the migration with the Raw_SQL for now until the patch is complete.

[aqpanaciy]

I successfully completed the "make test integration". However, I had to convert two files (17_consolidate_statistics.sql and 37_Market_decoupling_4.sql in Live_36) from UTF-16 LE to UTF-8.

Also, live_patch_35.sql doesn't contain data from the Raw_SQL files (15_... through 19_...).

I plan to make corrections to the OPDB in the near future.

[clouths]

The UTF-16 LE fix has been done in this PR: OpenPerpetuum/OPDB#389.

I'll also check for the SQL file.

[clouths]

I've reworked the migration to load files automatically without having to add new patch in the migration.sh file with a cache mechanism to accelerate migration (development QoL).

I also had some instances of Distributed transaction happening due to Enlist some times which was causing errors, to fix this issue, I refactored the DbConnectionManager to avoid multiple connections in the same scope to reuse them. When distributed transactions is enabled, the behavior is unchanged. This refactor will allow in theory better scaling with many players/npcs.

I also added SIMD acceleration, did some optimization on the loaded assets by caching the essential content. I also added idle optimization to slow down tick when no player is on an island from from 20hz to 1hz while keeping precision on events on the island. Overall saving (40-60% reduction) 1.5-2.5GB on RAM and (60-90% reduction) 0.6-0.9 CPU core. One notable change is that a zone metadata can now fit in L2 cache (from 36MB to 512KB per zone).