Dev Diary #3: Writing Complicated Code to Avoid Writing Simple Code

Welcome to the third dev diary for Substructure.

Last time, we looked at how we build up the layers of the planet visually: the terrain tiles, decals, decoratives, and generation logic that slowly turn a flat surface into something that looks like a strange, usable planet.

This time, we are going in a somewhat different direction.

Substructure is a game about automation, but, unfortunately, making an automation game also involves quite a lot of automation that players will never see. Some of that work is glamorous. Some of it is writing code so that we do not have to keep writing other code by hand.

This diary is about the latter.

More specifically, we are going to talk about how Substructure handles data, how Lua definitions become C++ structures, why that process can turn into a small data-shoving factory if we are not careful, and how reflection and annotations help us keep the whole thing a little more sane.

Writing Complicated Code to Avoid Writing Simple Code

There is an old meme at Google:

For the non-programmers or otherwise uninitiated among you, Protobuf is a ubiquitous data serialization format.

The butt of the joke is that Google's engineers spend a large portion of their time writing code that translates data from one shape to another, slightly different one.

Fortunately, we are not a data company. Unfortunately, we still have a few different ways to talk about the same or related data, and ideally we like to spend our time doing the fun bits of game development rather than painstakingly translate from one data format to another.

As you might have already guessed, this post is fairly technical in nature. Substructure is written in C++, and this post is very much going to go into the weeds of the language, though we'll leave the most gritty bits for the end. That said, hopefully this'll prove an interesting insight into the problems we deal with here at Dubious even for those of you who are not C++ developers by daytime.

Entities and Prototypes

In Substructure, most things you can interact with in the world are entities.

In the picture above, there are four entities: from left to right a Fabricator, a Fluid processing plant, and two Chests.

The two chests are different entities, but have the same prototype (Chest). The Fabricator and the Fluid processing plant each have their own prototype, but they are more similar to each other than to a chest. They are both structures that, among other things, consume power and use recipes which turn some items and/or fluids into other items and/or fluids.

The base game does not in fact know anything about Fabricators and Fluid processing plants. It implements behaviour for various entity types. Each of these has a datastructure associated with it that defines what kind of properties and parameters that type of entity can have. That is, what the prototype for a particular entity type looks like.

Both the Fabricator and the Fluid processing plant are Machines, which is one of the entity types. The definition of Machine's prototype data looks like this (simplified, subset of):

struct machine_data_t {
  sprite_4way image;
  sprite_4way working_animation;
  small_vector<collision_data> collision;
  watts energy_usage;
  small_vector<ref<recipe_category>> recipe_categories;
};

A Machine has an image for 4 world orientations, an animation that plays when the machine is working, a number of collisions, energy_usage, and a number of recipe categories compatible with it. This defines the shape of what makes a Machine.

To actually tell the game about Fabricators and Fluid processing plants, we have to write some Lua which the game runs during the startup. This is how the base game defines all its content, but also how mods can modify or add content:

data.machine.fabricator = {
  image = {
    ...
  },
  working_animation = {
    ...
  },
  collision = {
    mask = defines.collision.player | defines.collision.structure_placement
           | defines.collision.structure_collision,
    box = {{-1.625, -1.625}, {1.625, 1.625}},
  },
  energy_usage = "125kW",
  recipe_categories = {"fabricator"},
}data.machine.fluid_processing_plant = {
  image = {
    ...
  },
  working_animation = {
    ...
  },
  collision = {
    mask = defines.collision.player | defines.collision.structure_placement
           | defines.collision.structure_collision,
    box = {{-2.499, -2.499}, {2.499, 2.499}},
  },
  energy_usage = "200kW",
  recipe_categories = {"fluid_processing_plant"},
}

So our very first task as developers is to take some Lua data, and translate it to C++. In practice, this actually happens in two phases as we load the data first, then resolve any cross-references like recipe_categories in the example above. We might later need to expose this data to Lua at runtime (although as of writing this, that doesn't exist yet), so we better translate back from C++ to Lua. And since Substructure is a multiplayer game, we should calculate a checksum (basically, a unique-ish number) from our mod data to make sure that if two players play together, the game agrees on how it should behave.

So we might end up with something that looks a bit like this (simplified, illustrative)

[]
result<void> read(const tracked_lua_value& v, machine_data_t& data) {
  if (!v.is_table()) {
    return unexpected_format("'{}' must be a table (was {})", v.name(), v.type_string());
  }
  if (auto r = read(v.field("image"), data.image); !r) {
    return r;
  }
  if (auto r = read_if(v.field("working_animation"), data.working_animation); !r) {
    return r;
  }
  if (auto r = read_single_or_list(v.field("collision"), data.collision); !r) {
    return r;
  }
  if (auto r = read_if(v.field("energy_usage"), data.energy_usage); !r) {
    return r;
  }
  if (auto r = read(v.field("recipe_categories"), data.recipe_categories); !r) {
    return r;
  }
  return {};
}[]
lua_value write(LuaContext& context, const machine_data_t& data) {
  auto table = context.make_table();
  table.set_field("image", write(context, data.image));
  table.set_field("working_animation", write(context, data.working_animation));
  table.set_field("collision", write(context, data.collision));
  table.set_field("energy_usage", write(context, data.energy_usage));
  table.set_field("recipe_categories", write(context, data.recipe_categories));
  return table;
}void write(checksum_t& checksum, const machine_data_t& data) {
  write(checksum, data.image);
  write(checksum, data.collision);
  write(checksum, data.energy_usage);
  write(checksum, data.recipe_categories);
}

Let's say we wanted to add a power source to machine_data_t so we can specify whether the machine needs fuel or electricity. We have to remember to modify, or at least consider, three or four different functions, probably spread across a few different files before we can even do anything with it.

So how do we not end up the Stuart & Michael data shoving company?

Reflection: Making C++ Look at Itself

Most of the content of these functions is pretty repetitive and straightforward, i.e. what we in the programming world would call boilerplate. It's neither particularly hard to write, nor is it particularly fun. The human brain is very good at learning patterns, so once it's seen enough of these things and realises they are usually fairly trivial, it'll stop paying attention. This makes errors or inconsistencies more likely to slip through review.

However, the exact same properties also make it a fine candidate for writing some generic code. All we need is the ability to iterate over the fields or members of some datatype (i.e. image, working_animation, collision, energy_usage, recipe_categories of machine_data_t). This feature exists in lots of programming languages as reflection. However, it's been painfully missing from C++, and only got standardised (added) with the finalisation of C++26 in March 2026. As of writing this, it's only been implemented in one major C++ compiler, GCC 16.1 released 30th April 2026. This basically means we can't use it, yet.

If you really wanted reflection in C++ in the past, the main choice has been macros. But these have a slew of problems and can result in very ugly code that's borderline unreadable.

However, C++ is very powerful. With some major hacks and tricks, it's possible to introspect on the fields of datatypes in at least C++20, without having to resort to any macros polluting the data definition. We use a library called reflect-cpp that implements this functionality.

With that, we can write code that is much more complicated, but we only have to write it once (simplified, illustrative):

template <typename T>
[]
result<void> read(const tracked_lua_value& t, T& data) {
  auto process_member =
      [&t]<string_literal Name, typename U>(anno::member_ptr<Name, U> member) -> result<void> {
    return read(t.field(Name.c_str()), *member);
  };  result<void> r;
  std::apply([&]<typename... Ts>(Ts&&... members) { ((r ? r = process_member(members) : r), ...); },
             anno::to_view(data));
  return r;
}template <typename T>
[]
lua_value write(LuaContext& context, const T& data) {
  auto table = context.make_table();  auto process_member = [&]<string_literal Name, typename U>(anno::member_ptr<Name, U> member) {
    table.set_field(Name.c_str(), write(context, *member));
  };  std::apply([&]<typename... Ts>(Ts&&... members) { (process_member(members), ...); },
             anno::to_view(data));
  return table;
}template <typename T>
void write(checksum_t& checksum, const T& data) {
  auto process_member = [&]<string_literal Name, typename U>(anno::member_ptr<Name, U> member) {
    write(checksum, *member);
  };  std::apply([&]<typename... Ts>(Ts&&... members) { (process_member(members), ...); },
             anno::to_view(data));
}

Annotations: Telling the Code What Matters

This gets us a long way, but we are not quite there yet. Our generic functions are great at ensuring consistency; everything behaves exactly the same way. But what if we don't want it to? Notice the generic functions above do some things differently, most notably in the from-Lua (read) conversions.

Going back to our example, of the 5 fields of our simplified Machine, we want to make sure a modder doesn't forget to add the main sprite image or the recipe_categories. If they forget these, it's almost certainly an error. However, we don't want to error if they don't specify any of the other fields. What's worse, we also don't necessarily want to error whenever we see a field of type sprite_4way - in particular, we want our working_animation field to be entirely optional. (In this instance we could make the type std::optional<sprite_4way> instead, but similar solutions are not as practical in all cases).

What we want is the ability to attach some (ideally) structured data to each field of a type. We want to do this in a way that's unobtrusive, ideally local (as close to the field definition as possible), and not error-prone.

Basically, what we want is a language feature called annotations.

This already exists in languages like Go, where this feature is called struct tags:

struct {
	microsec  uint64 `protobuf:"1"`
	serverIP6 uint64 `protobuf:"2"`
}

It's also coming to C++ (P3394), but again this isn't quite available yet so we are once again out of luck with core language or standard library support.

Unlike with reflection, we didn't find any solution we were quite happy with. We didn't want macros, and we didn't want to put any annotations into the type of the field itself, which seem like the two most common solutions.

So we wrote our own small library. An annotation might look something like this:

struct machine_data_t {
  sprite_4way image;
  auto annotate(anno::member<"image">) -> anno::annotations<anno::deserialization ::required>;  ...
}

This annotates the member image with a value anno:: deserialization::required where anno::deserialization is an enum. If we add another value anno:: deserialization::single_or_list, we can make the read function respect the annotation values and choose the appropriate conversion function:

template <typename T>
[]
result<void> read(const tracked_lua_value& t, T& data) {
  auto process_member =
      [&t]<string_literal Name, typename U>(anno::member_ptr<Name, U> member) -> result<void> {
    constexpr auto mode = anno::member_annotation<T, Name, anno:: deserialization>.value_or(
        anno:: deserialization::none);
    if constexpr (mode == anno:: deserialization::required) {
      return read(t.field(Name.c_str()), *member);
    } else if constexpr (mode == anno:: deserialization::single_or_list) {
      return read_single_or_list(t.field(Name.c_str()), *member);
    } else {
      return read_if(t.field(Name.c_str()), *member);
    }
  };  result<void> r;
  std::apply([&]<typename... Ts>(Ts&&... members) { ((r ? r = process_member(members) : r), ...); },
             anno::to_view(data));
  return r;
}

The Lua to C++ conversion code now does a hard check for the image and recipe_categories fields in Lua (or any other field that has the anno:: deserialization::required annotation), erroring if it's not present, without affecting other fields.

The fully annotated version of the (subset of) machine_data_t as it appears in our source code today is:

struct machine_data_t {
  sprite_4way image;
  auto annotate(anno::member<"image">) -> anno::annotations<anno:: deserialization::required>;  sprite_4way working_animation;
  auto annotate(anno::member<"working_animation">) -> anno::annotations<anno:: checksum::omit>;  small_vector<collision_data> collision;
  auto annotate(anno::member<"collision">) -> anno::annotations<anno:: deserialization::single_or_list>;  watts energy_usage;  small_vector<ref<recipe_category>> recipe_categories;
  auto annotate(anno::member<"recipe_categories">) -> anno::annotations<anno:: deserialization::required>;
};

That is, when converting machine_data_t from Lua to C++, the image and recipe_categories fields are required; collision accepts a single collision directly, or a list of collisions - both collision = some_collision and collision = {some_collision} are acceptable for a single collision some_collision. Separately, working_animation is omitted from the checksum.

Going deeper on annotations (for C++ nerds)

If you don't know C++, you might not be surprised that the code above looks like total gibberish. However, if you do know C++, you might be surprised that the code above looks like total gibberish. What is going on?

To annotate a member m with some values a, b, c, you declare a member function called annotate that has a single argument of type anno::member<"m"> and returns a value of type anno::annotations<a, b, c>. A keen eye might notice that auto annotate(anno::member<"m">) -> anno::annotations<a, b, c>; is equivalent to anno::annotations<a, b, c> annotate(anno::member<"m">);, and the latter is actually shorter. While we don't use trailing return type too often, we think the former stylistically reads much better in this context, so we use that syntax as a convention.

You might also notice that we only declared the function; so where does the definition go? Perhaps surprisingly, nowhere. We only ever use the type signature of the annotate function (though importantly still pass values around). The function is never called, so we don't have to bother defining it. There is also no real reason for the function not to be static. It would in fact likely make a little more sense if it was, but again we never actually call these anyway and this saves us some extra syntax.

To us, this solution is local enough. We would welcome something like what's in C++26, but if we can't have that, writing the annotation directly below the field is as good as it gets in our books. Note you could write the annotation wherever you want in the class/struct definition, but we like to put it straight under the member as a convention.

It's also pretty unobtrusive. The annotation does not touch the member definition itself, and poses no requirements on the type of the member. We have to have the ability to add the annotation to the parent type (machine_data_t in our examples) itself, but that's not a problem for us.

The remaining question now is: is it hard to mess this up? This is quite important - once something is generic or automated, no one is really going to expect it to fail. Any potential errors could be hard to detect, so we want to have compiler errors for the most obvious ones.

Bar misspelling the function name (e.g. anotate) which we can't really do much about, we think this mechanism is pretty robust. The annotations themselves are typed, which eliminates one class of problems. Most of the errors we see in practice happen because we copy-paste an annotation and forget to change the name of the annotated member.

For example, one might end up with something like this:

struct foo {
    int bar;
    auto annotate(anno::member<"bar">) -> anno::annotations<>;    int baz;
    auto annotate(anno::member<"bar">) -> anno::annotations<>;
};

Luckily, this is just plain invalid C++, so the compiler helps us out here:

post/post.cc: 198:8 : error: class member cannot be redeclared
  198 |   auto annotate(anno::member<"bar">) -> anno::annotations<>;
      |        ^
post/post.cc: 195:8 : note: previous declaration is here
  195 |   auto annotate(anno::member<"bar">) -> anno::annotations<>;
      |        

But what about something like this:

struct foo {
    int bar;
    auto annotate(anno::member<"baz">) -> anno::annotations<>;
};

That is, trying to annotate a member that does not exist? This is somewhat trickier. We could instead write auto annotate(anno::member<&foo::baz>) -> anno::annotations<>;, i.e. use pointers to members in the values, which would turn this problem into something the compiler can help us with directly; you can't create a pointer to a member which does not exist. But typing out the &class_name:: is just a little awkward, and we can do without it.

But how? Without proper reflection, we can't iterate over all the annotated function declarations (and their overload sets) to validate them. As far as we know, no tricks to iterate over all member functions existed before C++26.

What we need is a way to check if any annotated function that accepts anything but anno::member<"name_of_member"> exists.

A relatively well-known trick is a special type that converts to anything:

struct any_device {
  template <typename T>
  constexpr operator T();
};

We can't hope the conversion would do anything useful (or indeed exist for all types), but that's not important. This type is really only used at compile time, and we don't really need the definition for the conversion operator, or indeed care whether it is implementable in practice or not.

For example, we could check if the free function foo can be called with a single argument of some type:

constexpr bool foo_takes_one_arg = requires(any_device d) { foo(d); };

Taking inspiration from this, we can write a special type that converts to any type, except for anno::member<Name> for a list of names given:

template <string_literal... MemberNames>
struct any_member_but {
  template <typename T>
  requires((!std::is_same_v<std::remove_cvref_t<T>, anno::member<MemberNames>>) && ...)
  constexpr operator T();
};

If we combine this with the reflection library that lets us extract (non-function) member names of a type, we can use this to query whether or not a particular type has an incorrect annotation. A naive implementation can look something like this:

template <typename T>
constexpr bool is_annotated_correctly = for_member_names<T>([]<string_literal... MemberNames>() {
  return !requires(any_member_but_device<MemberNames...> d, T t) { t.annotate(d); };
});

This can now correctly detect that:

struct foo {
  int bar;
  auto annotate(anno::member<"bar">) -> anno::annotations<>;
};static_assert(is_annotated_correctly<foo>);struct bar {
  int bar;
  auto annotate(anno::member<"baz">) -> anno::annotations<>;
};static_assert(!is_annotated_correctly<bar>);

The real version in our codebase is a bit more complicated, but it's conceptually the same.

Was This Worth It?

You might ask, was this really worth it? We think it is. All of the translations mentioned used to be manual, and when transitioning to the use of reflection, we found several inconsistencies and some outright errors.

Not having to write all the boilerplate makes making changes faster, and we are less at risk of introducing new errors. When related functionality is spread across several places in the codebase, it's easy to make a subtle mistake like accidentally omitting a field from the checksum. And one of the most powerful things about all this is the ability to add a new conversion function like converting the same data to JSON - having to handle all the existing types is a tall order, and much harder than adding a bunch of generic code, especially if we want everything to be consistent.

At the same time, we really wish we didn't have to do this ourselves and could use the standard facilities provided by C++. At the moment, we compile with Clang, which still seems a way off from providing support for reflection. One thing in particular we hope language support for reflection will help with is faster compile times.

As it often is in engineering, though, there is no right or wrong answer or approach. We could have just as easily done nothing and lived with manual conversions, opted for code generation, or jumped on the bandwagon of asking your favourite LLM to generate all the boilerplate. Call us old school, but we prefer code we know and control and would rather write a few lines of complicated-looking code once (it's also fun!) than wait on billions of multiplications to complete every time we want to modify a piece of code.

That’s it for this diary. If you missed the previous one, you can check it out as well, where we looked at how we build the planetary layers of Substructure, from concept art and terrain tiles through to decals, decoratives, and the generation logic that pulls it all together in-game.

https://store.steampowered.com/news/app/3012600/view/702145613257509672

Join the Substructure Community

If you’d like to follow along with development, ask questions, or just chat with us about the game, you can find us on our socials and Discord.
Discord | Reddit | Twitter | BlueSky | Website

https://store.steampowered.com/app/3012600/Substructure/