Build System Refactor: Master Discussion

[stellarpower]

So, I've been making a number of small edits to the Engine/Studio, and in general I feel I am constantly fighting the buildsystem. I haven't committed anything in or attempted to contribute it back yet, cause I seem to spend hours building and rebuilding the same components that haven't changed, whilst the code I have changed seems not to be included into the build, and/or an error flew past my screen and I don't see it. I've ended up writing some wrapper scripts, but things like this screenshot I saved are pretty common:

Obviously it's not the case, but, let's say Neuromore given to me as a project to work on, and I were asked what changes I'd make. The first thing I'd have down as needing work is modernising the buildsystem, I don't feel it's an efficient use of time if every dependency has to be taken, split out, committed in as files, and every single file that we want to build needs to be added into the make recipe by hand. I feel it's only gonna bog down adding new features or fixing bugs over time. Obviously, I know it needs to build for diverse platforms, including MacOS and mobile, so I understand if that limits some options. But I feel this is an amount of manual labour that we don't need at this point in time, as higher-level tools like CMake will do all this for us (and I'd argue even CMake is getting a bit too old to be usable now). Or if we do have to fall back to Unix Makefiles, many higher tools have some compatibility options to output these.

I think it massively helps, and what I would normally expect form other projects, if to have one command that I type, and anything that hasn't changed doesn't get built, and anything that has does. This makes it much faster to develop than things are now, as, I'm not sure what is going on, but make seems to keep scanning and rebuilding objects whose sources haven't changed. I also know plain Makefiles can't be run in parallel and stop on the first error, not without a good amount of supporting code that makefile generators like the one in CMake produce automatically.

I'm making this issue as a more open master discussion point. Then individual issues can be sprung off it for specific components. See [TBC] for one on CI and Earthly; this is the first thing I'd like to try.

Here are come comments from #280, which I'm pasting verbatim for reference, as they don't belong there:

---

Am wondering, is it easier if I alter the MR to be pulled into a feature branch, and that way you can use what's there as a springboard, and if you know what you wanna do or how you want things set out, you can work on it directly? And then I can also pull back from there to my fork to make any edits there and then submit those back. And then there's potentially only one set of conflicts to fix when you're happy with how things are and ready to merge it all back into the master?

(1) was just as I was trying to get it to build on Windows for someone who uses it there, so I've pushed those off to a branch on the side now. On (3) also, am not a Windows user, so not sure if there's much I can usefully do.

Longer-term, I think I'd quite like to take a look at the build and dependency setup, as I think the way things are now probably all creates quite a bit more work than is needed. Ideally I think bumping a dependency should be a new version in one or two lines of configs and then just any breaking changes this new versions introduces that need fixed in our code. So I don't know if you have any opinions on this, or requirements for how things need to work at neuromore, but I was going to suggest using something similar to how I have set some of my own projects out.

Earthly + VCPkg is a combination I've used before with pretty good success. VCPkg is also designed around building self-standing static binaries, which seems to suit how Studio is set up presently. Earthly is a tool that tries to bridge the world between CI and local development, so that you get reproducible builds both when working on code on your local machine as well as then building in the cloud. This also extends to cross-compiling, something I've not tried yet but this was the reason for the junk commits under (1), as I couldn't get the CI going for various reasons. I have also hit problems that were hard to reproduce when running parallel builds using make, and I think this is a known issue, with how make by default splits up jobs and carries on even if one has failed - so I think it would probably make sense to migrate to a higher-level build system. But I understand this may complicate things for integration with Windows/MSVC.

So at a high-level the rough process I am thinking of would involve:

Replacing the hardcoded makefiles with a more modern buildsystem
  * CMake would be an obvious choice
  * However there are more modern alternatives to this that make life easier available. I am looking to give [xmake](https://xmake.io/#/) a run for its money in the near future myself.
Using a package manager for dependencies and removing third-party files that are checked in at specific versions and then altered:
  * Bumping any package version should be as simple as e.g. changing a version string in some JSON
  * I'd need to analyse how many of Neuromore's dependencies are available in repositories, but I think if any are not available, provisioning these as packages should be no more work than it is now manually integrating these into studio, and then the wider community can benefit aswell from the recipe.
  * This is also where xmake may help, as it can pull dependencies from all the major package managers (vcpkg, conan, build2) as well as its own repository
Using earthly as an option for performing local builds, and using this as the main mechanism for the cloud build
  * Earthfiles are similar to Dockerfiles, with added features, and Earthly has been designed for very simple integration with CI pipelines, so most of the work on this is basically already done within the GitHub actions you guys have.
  * And ideally this would make it relatively trivial to perform a reproducible buuld on a development machine, possibly for a differing OS, to check that nothing in the local setup is impacting the build process and thus make it more likely CI checks will pass upon pushing up.

Do you have any thoughts on all this? Appreciate it's quite a major refactor I'm suggesting, but at the same time, I think as you mentioned, if upgrading a version is a task that ends up on the backlog list, IMO it hopefully should be worth it to make those sorts of tasks much quicker to tick off.

---

Since writing that, I have got my hands dirty with xmake a bit more. I think I would recommend it as the best option; without having tried all the various other buildsystems out there, it has reduced the amount of pain in getting things set up for the builds I've tried it on. Lua is nice and expressive, and the recipes even for complex packages are relatively terse. Settings management is handled really well, sensible defaults are provided for most things. Overall, a number of configurable options Just Worked :TM: when in CMake, they don't (because there are so many ways to skin a cat and anyhting you google is not the same as your exact setup). The main problem I hit was including dependent projects that had non-standard CMake recipes, and there's no real way around this that doesn't cause some pain. I think if all the dependencies can be committed into some repository, then all that becomes contained, and it's possible to keep a relatively clean build in maybe a few hundred lines for something as large and targetting as many platforms as Neuromore.

Anyway, I leave the floor open for comments.

[stellarpower]

Is there any consensus on this? As if not, I will probably just start working on my own fork, and if you want anything, you can cherry-pick it back from there. I have multiple fixes committed locally but the way a lot of this is architected it's prohibitive to trying to get those pulled out neatly, committed, and then opened as an MR. Ideally I'd like everyone to benefit, but I can't trust a build unless I scroll all the way back in the terminal because make just won't stop when it hits an error, and will link against something stale if it has to.