New Development: ModHelper and Utilities

Kick back and discuss whatever takes your fancy.
Post Reply
Mystical Panda
Posts: 750
Joined: Wed Mar 30, 2016 2:02 pm

[CreateModArchive] It was an

Post by Mystical Panda » Mon Feb 12, 2018 9:41 pm

[CreateModArchive] It was an interesting evening and a half; the base utility is coded. Here's some output from the test. It basically looks at the specified plugins (either within a specified folder, or individually- more than one plugin is supported, though they should be for the same mod for now) for defined assets, then, if the assets are present as a loose file, creates a mod archive that includes them plus:


o - The plugins specified.


o - Any BSA archives associated with the plugins.


This should produce a 'ready to use' standalone mod archive file; by default it's setup to use 7z, but can be changed if needed with the included config file. 


I'll need to think about the best way to use the MAST records for a bit, plus I'll also need to include in the archive file any textures associated with nifs. These are going to be a lot more tricky.


You do not have the required permissions to view the files attached to this post.

Mystical Panda
Posts: 750
Joined: Wed Mar 30, 2016 2:02 pm

Here's the problem with using

Post by Mystical Panda » Tue Feb 13, 2018 2:01 am

There a few inherent problems with this type of system, so it wouldn't work for all circumstances.


Here's the problem with using the MAST records to cross-reference assets... If there's a loose file that is referenced by both a vanilla game plugin asset (easily filtered out), and the modder's plugin, and the asset doesn't belong to another modder (a custom override), we'd want to include it, otherwise no. Therein lay the problem; how to tell to whom the loose file belongs. And vice versa.


If a modder's just testing his mod only, the system would, at least to the point of including custom assets that aren't found in any of the plugins. Then there's no way it'd know to include those. Unless, we auto-add all custom assets- but the modder would have to be careful (like adding from the games "data" folder instead of a separate mod folder). A way around this is to keep the modder's custom assets in a separate folder and sym or hard link the game to the files. This would allow auto-adding the entire folder, and checking for missing assets, and bypass the need to sync to dependent or master plugins; what's in it's separate folder is what it uses, otherwise report it as missing. 


Plugins can be specified in one of two ways; specifically using the "-p" parameter (no limit), or within a folder using the "-f" parameter. If a folder was specified, the utility loads the plugins from the 'root' of the folder specified (presumed to be "data"). Either can be used simultaneously or interchangeably.


Without doing much modding, I'm guessing the normal procedure would be to place the modder's custom assets. plugins and BSA (if present) in the games installation "data" folder (so to speak) for testing. From there, just select the plugins, and the utility will do the rest, but, if there are assets whose names aren't found in the specified plugins, there's no way the utility can pick those out- thus back to a sym or hard link solution.


What's the normal procedure modders use when creating and packaging mods? This might help me to find tune the process better.



User avatar
jlf65
Posts: 1324
Joined: Wed Aug 10, 2016 9:10 pm

When I work on a mod, I make

Post by jlf65 » Tue Feb 13, 2018 4:22 am

When I work on a mod, I make a folder for it with everything that goes with it. If I change a file, it gets copied back to the folder so that the folder is always up to date. Once I'm done, I just zip the folder. I keep track of which files are "mine" and which are being used but a part of another mod - if that mod has a plugin, I make it a master of the mod. If it's just files, I make a note of the dependence somewhere - usually a notebook.



Mystical Panda
Posts: 750
Joined: Wed Mar 30, 2016 2:02 pm

jlf65 wrote:

Post by Mystical Panda » Tue Feb 13, 2018 12:20 pm

[quote=jlf65]


 


When I work on a mod, I make a folder for it with everything that goes with it. If I change a file, it gets copied back to the folder so that the folder is always up to date. Once I'm done, I just zip the folder. I keep track of which files are "mine" and which are being used but a part of another mod - if that mod has a plugin, I make it a master of the mod. If it's just files, I make a note of the dependence somewhere - usually a notebook.


[/quote]


Then to work both ways, we'd need a switch to include all the files found in the specified folder. This way an archive could be built from a custom folder, or by the plugins found in the game's installation folder (or custom folder if there aren't any loose files not used by a plugin- it would work there also).


Either FindMissingPluginAssets or CreateModArchive will report if assets were used by a plugin and wasn't found as a loose file, or in the vanilla assets 'catalog', with the latter doing an extra step of building the archive.


An extra thought... to 'track' down potentially missing assets, we'd need a larger system to collect asset names from the various mod archives then store and access them from a central database. Maybe if there was a system to submit a 'catalog' from an archive in text format (just asset file names), it could be auto added, indexed and searched.



User avatar
jlf65
Posts: 1324
Joined: Wed Aug 10, 2016 9:10 pm

Yeah, if you tracked where

Post by jlf65 » Tue Feb 13, 2018 1:36 pm

Yeah, if you tracked where every file installed in the game came from, then you know who it "belongs to" when you later make mod archives. Of course, that would require people install everything through the manager; no manual installs.



Mystical Panda
Posts: 750
Joined: Wed Mar 30, 2016 2:02 pm

jlf65 wrote:

Post by Mystical Panda » Tue Feb 13, 2018 7:07 pm

[quote=jlf65]


 


Yeah, if you tracked where every file installed in the game came from, then you know who it "belongs to" when you later make mod archives. Of course, that would require people install everything through the manager; no manual installs.


[/quote]


I'm thinking along the lines of... a program that the user could run on their end, and would 'get' the asset names from the mod archive specified and submit the archive name, game it's used for, and asset list to a central online 'hub' (server), which would add that information to an online 'on-demand' database of sorts. This could then be queried remotely as needed.


[CreateModArchive] Here's a new test with the -includefiles switch. It also grabs all the texture names from each loose file nif, then verifies that the texture is present as either a loose file, or within the specified catalog. If present as a loose file, it's included in final mod archive that's automatically built. If it's not found to be an asset to any plugin, it'll still be included in the final mod archive. Without that switch, it'll just include those assets found in the plugins and it's respective nifs.


With the new test, you can see some additional texture assets show up as missing, since nifs are now also being checked.


You do not have the required permissions to view the files attached to this post.

Mystical Panda
Posts: 750
Joined: Wed Mar 30, 2016 2:02 pm

[CreateModArchive] Here's a

Post by Mystical Panda » Wed Feb 14, 2018 9:23 pm

[CreateModArchive] Here's a quick test against URWLNV. A good use for reporting the MAST records is to show which masters are associated with each plugin; an asset is missing from the second plugin because it should be installed by "Project Brazil" (shown by it's MAST record). Also, the CK can leave MAST records in a plugin un-necessarily (from what I've gathered) where there are no associated assets (thus would need to be 'master' cleaned).


Keeping mods in a separate folder, and using sym/ hard links or a regular copy to install (manage the wip mod), so the game engine can find them, seems, at least to me, to be the best development practice. This should isolate all the changes from the actual game folder, and allow the mod to be developed from start to finish apart from the game. In this scenario, this program along with the 'include files' switch would work just fine.


The following test was ran against an extracted mod archive (it's folder), with and includes loose files and using the full FNV asset 'catalog'. 


You do not have the required permissions to view the files attached to this post.

User avatar
jlf65
Posts: 1324
Joined: Wed Aug 10, 2016 9:10 pm

Yeah, that's a great idea,

Post by jlf65 » Wed Feb 14, 2018 10:30 pm

Yeah, that's a great idea, but I didn't know about links working in the game when I first started. That was a good find.



Mystical Panda
Posts: 750
Joined: Wed Mar 30, 2016 2:02 pm

I'll need to start laying out

Post by Mystical Panda » Wed Feb 14, 2018 11:13 pm

I'll need to start laying out the 'server' side before I hop back on ModHelper; it'll be useful for a variety of things, and I'll need to build that functionality in across the board application wise. It shouldn't take too awful long to get at least an initial working construct up and running, then I can shift back around, then come back when needed to upgrade any online functions.


Hopefully as time permits, I'll still be able to pick apart the remaining utilities that need updated code- circling the wagons on some of the most important ones (BSA, ESX, etc.,.). In fact, I was tossing around another idea concerning scripts which would become another utility all together, and take us one step closer.



Mystical Panda
Posts: 750
Joined: Wed Mar 30, 2016 2:02 pm

jlf65 wrote:

Post by Mystical Panda » Wed Feb 14, 2018 11:16 pm

[quote=jlf65]


 


Yeah, that's a great idea, but I didn't know about links working in the game when I first started. That was a good find.


[/quote]


Hard-links work for all files, but sym-links (at least for me) only worked with loose (I still had to hard copy the plugins and BSAs to the game's installation "data" folder). The limitation with hard-links is, both the actual file and the 'link' have to exist on the save device, which is unlike sym-links (which can be anywhere, across multiple devices is necessary). 


A hard-link is really another VTOC (if you will, going old school here) entry where the first sector points to a single source file. So, multiple file names which are all different (but on the same devices) will all have the same first sector (on the HDD) address. The caveat is, you can't delete the source file until all hard-links have first been deleted. Because hard-links aren't an actual file, only a directory (VTOC type) entry, the actual source file only exists once and doesn't take up space for additional copies. When you create hard-links, then see it with a "dir" command, it looks like the file is actually there, but the OS is just referencing the source file and it's attributes; the same with reading and writing to it. 


A sym-link is similar but the 'link' isn't in the directory (VTOC), it's on the disk in the form of a file (usually listed as 0 bytes), and that's what's used to point to the source file. 


 



Post Reply