[TOOL] AVCML - Automated Mod Installation*

[bit]

AVCML: Avorion Community Mod Linker

AVCML is a set of scripts I've been using to quickly install mods in a nice, clean way.  Before you continue on, this is not a full-fledged installer with all the bells and whistles of a full mod manager.  If you're looking for that, I believe Steam workshop support should be arriving which will officially support a much more robust, featureful system.  I wrote the scripts originally for myself, but I figured since it's working pretty well, I'd release it for those who might be interested.

 

What is does:

AVCML helps with the installation of mods, but more importantly, with the rapid removal or swapping of mod content.  Rather than editting files inside the Steam directory, AVCML will automatically snapshot the scripts directory and install mods from an isolated directory.  This prevents unnecessary data loss (forgetting to back up files) while also providing the ability to selectively install sets of mods by reverting to older scripts directories.  While useful to any user, it was designed with developers in mind, keeping the environment clutter free, organized, and well-archived.

 

How would it be used:

Instead of installing a mod to the Steam directory, the user is able to install it to a local (more easily accessible) directory (I periodically refer to this directory as the local scripts directory).  Then, by running AVCML, the locally installed mods are mapped onto the Steam installation's scripts folder.  The process is conducted in such a way that backups are maintained so that if the user wants to revert back to an earlier installation, either an unaltered one or a previous setup of mods, the user maintains the ability to quickly do so.  Because it takes a full snapshot, even if you made changes in the local scripts folder, prior versions are maintained.

 

About the program:

Things AVCML (the program) handles:

- Installing sets of mods to the relevent directories

- Keeping meticulous backups of editted directories

 

Things AVCML does not handle:

- Installing a particular mod

- Uninstalling a particular mod

- Merging mods

- Handling conflicting mods

- Handling dependencies

 

Requirements:

- Python 3.6 (support for 3.0-3.5 available upon request)

- Windows 7, 8, 10; or Linux with Bash

 

Additional Requirements:

Windows: Admin access or ability to change user permissions*

 

Installation:

Download the files (avcml.py for everyone, avcml.bat and avcml.sh for Windows and *nix respectively)

 

* For Windows, in its current form, the program will require the ability to create symbolic links.  The Python script is written to be as language-agnostic as possible to make it easy to write for everybody, but unfortunately at least on Win7, symbollic links are not provided to all users.  One option is to run avcml.bat as an administrator.  Another option is to give yourself the correct privileges.  Refer to here for more info on that process.

 

---

 

Setup:

There will be commented text at the top of avcml.bat and avcml.sh. Read it some time during installation, preferably before you start.

 

Windows

 

1. Make a few folder somewhere on your harddrive.  This will contain the mods (local mods folder) you install.  Inside, make another folder and name it scripts (local scripts folder).  My setup looks like:

C:\Users\bit\Documents\AvorionMods\scripts

2. Compress a copy of your Steam installation data/scripts folder (installation folder).  This is different from your APPDATA Avorion folder.  The default installation directory should be C:\Program Files (x86)\Steam\steamapps\common\Avorion\data\scripts, though those with custom setups may have it placed elsewhere.  Any mods installed in this compressed copy will be persist into every installation, possibly causing problems down the line.  You can use Steam's "Verify integrity..." to retrieve an unaltered copy.

3.  Store this compressed copy (snapshot) preferably somewhere inside your local mod folder, but NOT in the local scripts folder you just made.

3. Open avcml.bat in your favourite editor.

4. Uncomment lines 10, 20, 24, 38 and fill in the relevant details.  They correspond to SET STEAM_AVORION_..., etc.  An example will apear below.

5. Set STEAM_AVORION_SCRIPTS_PATH to point to wherever your Avorion installation folder is.  Remember, this is the one under the Steam directory.

6. Set AVORION_MOD_PATH to the folders you made earlier.

7. Optional: Set ARCHIVE_MODE to "zip".  The program will default on "zip", but for the adventurous seeking those sweet extra bytes of compression can also set it to "gz", "gztar", "xztar", "xztar", or "bztar".    The available compression methods depend on what your system and python installation offers.

8. Set ORIG_SNAPSHOT to the path of the compressed copy you made in step 2.

* It is imperative that you do not add any spacing to the fields in steps 4-8 except for the spaces in your paths.  You may need to embed your paths in quotation marks.

 

By now, it should look something like

...
:: Usually installed by default under
:: C:\Program Files (x86)\Steam\steamapps\common\Avorion\data\scripts
::
SET STEAM_AVORION_SCRIPTS_PATH=C:\Program Files (x86)\Steam\steamapps\common\Avorion\data\scripts

:: Here is where your local mod directory should be.  This can be whatever folder you want, but the program will 
:: mirror ALL files inside.
:: It will also backup your current vanilla scripts to the parent directory.
:: Make sure that the only files contained are mod files.
:: It'll back everything up just to be safe, but you don't want to risk anything.
::
::
::
SET AVORION_MOD_PATH=C:\Users\bit\Documents\AvorionMods\scripts

::  Backup mode (type of archive used for backups, unrelated to ORIG_SNAPSHOT
::
:: ARCHIVE_MODE=
::
::
::
::  This is where your base snapshot of the scripts folder should be stored.
::  If you have not already done so, archive your Steam installation's scripts folder to the directory
::  provided in ORIG_SNAPSHOT.
::  Make sure your Steam installation's scripts folder is cleared of all mods
::  as this is the base-state the script will revert to.
::  It should include a filename and type. E.g. ~/.avorion/mods/base_snapshot.tar.gz
::  It must be conformant with the types the script can read, namely tar, zip, gz, tar.gz, tar.bz, tar.lzma.  Further,
::  your computer must have the appropriate Python modules to process these.  zip, gz, and tar and well supported and so
::  I would recommend sticking to those.
::
:: ORIG_SNAPSHOT=
::
...

 

Try running it.  If you get an error about Python not being found or pythonexec not being a command, or error messages and you are sure you have Python 3.6, edit line 39 from

:: set pythonexec=

to

set pythonexec=C:\Python36\python.exe

Or wherever your Python 3.6 installation is.

 

Linux

1. Make nested folders.  The parent can be named anything (usually mods, and i'll refer to it as the local mods folder), but the child should be named scripts (which I refer to as the local scripts folder).

1. Make a snapshot of your data/scripts folder in your Steam installation and place it somewhere, preferably inside the local mods folder from Step 1.  Do NOT place it inside the local scripts folder.  Any mods installed in this compressed copy will be persist into every installation, possibly causing problems down the line.  You can download a copy using Steam's "Verify integrity..." to retrieve an unaltered copy.  The Steam installation directory is trickier on Linux, but it is definitely not ~/.avorion.  Usually it is under ~/.local/share or ~/.steam or ~/.steamapps but this is highly distro dependent.

2. Just like with Windows, you'll edit your avcml.sh.  Uncomment lines 12, 20, 24, 36.

3. Fill in the uncommented lines of with your Steam Avorion folder, a local nested folder, compression type, and base snapshot.  On Linux, ARCHIVE_MODE defaults to "gz" and so if this is satisfactory, there is no need to uncomment that line unless you wish to try a different compression method.  The available compression methods depend on what your system and python installation offers.

* It is imperative that you do not add any spacing to the fields in steps 4-8 except for the spaces in your paths.  You may need to embed your paths in quotation marks.

 

...
#
STEAM_AVORION_SCRIPTS_PATH=~/.local/share/Steam/steamapps/common/Avorion/data/scripts

# Here is where your local mod directory should be.  This can be whatever folder you want, but the program will mirror ALL files inside.
# It will also backup your current vanilla scripts to the parent directory.
# Make sure that the only files contained are mod files.  It'll back everything up just to be safe, but you don't want to risk anything.
#
# I keep mine in ~/.avorion/mods/scripts
#
AVORION_MOD_PATH=~/.avorion/mods/scripts

# Backup mode (type of archive)
ARCHIVE_MODE="gztar"
#
# This is where your base snapshot of the scripts folder should be stored.
# If you have not already done so, archive your Steam installation's scripts folder to the directory
# provided in ORIG_SNAPSHOT.
# Make sure your Steam installation's scripts folder is cleared of all mods
# as this is the base-state the script will revert to.
# It should include a filename and type. E.g. ~/.avorion/mods/base_snapshot.tar.gz
# It must be conformant with the types the script can read, namely tar, zip, gz, tar.gz, tar.bz, tar.lzma.  Further,
# your computer must have the appropriate Python modules to process these.  zip, gz, and tar and well supported and so
# I would recommend sticking to those.
#
ORIG_SNAPSHOT=~/.avorion/mods/snapshot.tar.gz
...

 

Execution

Terminal

The .bat and .sh files should now contain all configuration required to run without problem.

Instead of installing scripts to your /data/scripts folder, install them into the relevant places in your mod folder.

For example, if I am supposed to install a script to scripts/entity/ai/flee.lua, I would install it to ~/.avorion/mods/scripts/entity/ai/flee.lua instead of the longer Steam path.  I would confirm everything inside of my local scripts folder is what I want to be in there; everything not there shouldn't be as well.  Then, I would call avcml.sh (or avcml.bat).  It can be done from anywhere as long as you used absolute paths in your configuration, although I like to keep mine in my local mods folder.

 

For those interested in access to the Python script, the -h/--help were quickly written but should provide enough information to allow for use.

 

What It's Doing

1. Backs up the target directory provided by STEAM_AVORION_SCRIPT_PATH.

2. Removes all symbolic links from the target directory.  As of the most recent release, the scripts folder is normally free of symbolic links and so all symbolic links were previously installed mods.

3. Unarchives a pristine scripts folder (the snapshot) to fill in any files that are now missing.

4. Symbollically maps all files in the local mods folder (AVORION_MODS_PATH) to the target directory.

 

Advanced usage

With access to the raw Python, the user could do any number of useful tasks.  But, there are some interesting use-cases for those who want to stick strictly to the .sh/.bat scripts.  By maintaining multiple copies of local scripts (the one where mods are installed) (using various filenames), whole sets of mods can be swapped out extremely quickly.  On Linux, this might look like:

 

./mods

|-avcml.py

|-avcml.sh

|-scripts

|-scripts_loadout_1

|-scripts_loadout_2

|-scripts_loadout_3

|-...

 

To install scripts_loadout_2, the user might use rm -r scripts && cp -r scripts_loadout_2 scripts && source avcml.sh.

Similar operations exist for Windows.

 

---

 

Future updates and closing remarks

If there is great demand for Python 3.0-3.5 support, I can certainly try porting it over, although the process will take time.  There was quite a bit of hoop-jumping to make this as obscure-dependency-free and so the permission settings on Windows is quite infuriating.  I don't have a finished solution at the moment, but if people are running into this problem often, let me know and I'll push a fix for that.

 

There may be those who look at this and wonder if the effort of installing all of this is worth it for the enduser (usually themselves).  And they make a valid point.  This tool is primarily for developers.  For the enduser who only installs mods once in a while, it may not be worth it.

 

In the works is a much more extensive mod manager which will handle the functions of AVCML along with more advanced diffpatch-based mod-merging, repository searching, etc (AVCML was actually originally written as a library for the larger project).  While the layout, list of features, vague architecture are developed by now, there is still quite a bit more work to be done.  Of course, that takes a tremendous amount of time to write, time I personally do not have to complete and so I am opening up the project to development within the community.  If there are those interested in helping develop this mod manager, please contact me either through the forum's messaging support or through Discord.

 

Earlier today I came across an interesting tool already posted on the Mods page.  I'd like to give a shoutout to NMaster's Mod Patch Generate & Apply tool.  I havn't used it personally, but for those looking for an already developed tool which handles the patchdiff system, I would encourage you to look there.

 

---

 

Notice

The provided code was developed and primarily tested on Linux using Python 3.6 and only received partial testing on Windows 7.  While all IO operations are performed in Python with an excellent track record for cross-platform behavior, I cannot promise that AVCML will work under all conditions, especially in custom or complex architectures.  I do believe, though, that with proper configuration, AVCML regardless of operational behavior, should not cause any data loss other than what is expected (refer to Step 2 of What It's Doing)

 

The usual lack of warranty, responsibility, and promise of data integrity apply.  I am releasing this as a resource for others to use under Creative Commons Attribution-NonCommerical-ShareAlike 4.0 International (CC BY-NC-SA 4.0).  A link to the license is provided below.

https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode

 

TLDRLegal is an excellent resource for understanding what the various licenses mean.  While not a replacement for true legal advice, generally speaking I find them to be decently accurate.

TLDRLegal (CC BY-NC-SA 4.0)

 

Update (6 Feb. 2017)

Removed AVCML.zip attachment to prevent use of outdated version.

[bit]

Bugfix:

 

Users may have been unable to start the game after running the script.  This was due to the unpacking of the snapshot one directory level too far.  The attached AVCML.ZIP should provide a fix for this.

AVCML.zip

[Strongground]

Hi!

 

First: Thank you for your tool and the dedication put into it!

Also the documentation is superb, whereas often you get useful tools with zero documentation and must resort to digging into code to see what it does, you perfectly describe what happens, what to do etc...

 

Now in my ignorance and enthusiasm when finding Avorion some weeks ago and playing it for a while, I started this reddit:

 

Only afterwards I stumbled upon your thread/tool and also MMaster's "Mods Patch Generate & Apply"-Tool. Now I planned to do largely the same - using diffs and merging (no conflict-resolving though) as well as backup and quarantine for mod-scripts. Only combined into one tool with user-friendly UI, to spread the safe usage of mods for this awesome game and making it more popular in the process.

I wanted to enable easy "one click" mod de-/activation and also thought about some way to show versions, author, compatibility etc. based on a scrape of this forum (specifically the mod-section) as each thread already has nice prefixes ("[REQ], [MOD]", etc.) and some contain the necessary information in a way, that a clever regex should be able to catch it.

 

Now I'd love to not have to re-implement all of your work (effectively only duplicating it) and rather would like to implement a GUI for your and MMaster's tool, with some enhancements like interactive folder choosing, dialogs walking the user trough the available choices and steps etc.

 

I'd be happy to offer support if you don't want to be bothered by all the inevitable "Tis duz not worrk!" spam too. ;)

 

Would you be fine with me doing this, based on your work? Of course due credit will be given,  any form you and I deem appropriate. It is your work, after all.

 

You are welcome to contact me via mail, PM or Telegram :)

 

Best regards

 

[bit]

I'll move this over to a PM.