Hot reloading: polymorphism

This is part of an ongoing series on hot reloading, a core feature of Sanable Engine. It is closely related to Unreal’s live coding and Visual Studio’s edit-and-continue: those systems only rewrite changed bytecode, whereas Sanable’s hot reload feature unloads and reloads an entire DLL while also fixing object layouts.

The most fundamental problem unloading then reloading code is that it can move. If new code is added or removed, the relative positions of functions or variables in the image may move, or the entire image may not load in the same location. This means that if any polymorphic objects from that DLL are still alive, they are now ticking time bombs. The next time our program uses those objects’ virtual functions, dynamic_cast, or typeid, we’d crash¹–the built-in mechanisms for facilitating those now point to bad memory.

Some programs like Visual Studio/MSVC pad their functions ahead of time so they can keep bytecode loaded–if a function shrinks they can just rewrite in place, and if it grows they can have it redirect to a new block of memory. Others like Unity avoid this entirely by serializing before a reload, then deserializing to restore state–but anything that can’t be serialized or doesn’t define a custom serialization method is lost.

Almost all modern compilers use a vtable for all type info, which is a mechanism for resolving override polymorphism. Short for “virtual function table”, it contains pointers to the correct version of each virtual function for the corresponding type. By convention, compilers quietly insert a pointer to that type’s vtable (a vptr) at the start of the object layout, but in our case this pointer is the cause of the access violation exception.

Although it gets unwieldy, we can implement vtables by hand, which has the bonus of working in C:

//End-user sees:

class Animal
{
public:
    virtual void speak() = 0;
};

class Dog
{
public:
    Color coatColor;
    virtual void speak() { printf("Woof\n"); }
};

class Cat
{
public:
    bool hairless;
    virtual void speak(); { printf("Meow\n"); }
};

void do_speak(Animal* animal)
{
    animal->speak();
}

int main()
{
    Dog dog;
    Cat cat;
    do_speak(&dog);
    do_speak(&cat);
}

//Compiler level:

struct Animal_vtbl;

struct Animal
{
    Animal_vtbl* vptr;
};
//No function is generated for Animal::speak

struct Dog : Animal
{
    Color coatColor;
};
void _Dog_speak_impl(Animal* _this) { printf("Woof\n"); }

struct Cat : Animal
{
    bool hairless;
};
void _Cat_speak_impl(Animal* _this) { printf("Meow\n"); }


// BEGIN COMPILER GENERATED

struct Animal_vtbl
{
    using speak_t = void(*)(Animal*);
    speak_t speak = nullptr; //This is what = 0 means
};

Animal_vtbl vtbl_dog_as_animal { &_Dog_speak_impl };
Animal_vtbl vtbl_cat_as_animal { &_Cat_speak_impl };

//Polymorphic types are never trivially constructible:
//vptr will be implicitly set as part of all ctors
void _Dog_ctor(Dog* _this) { _this->vptr = &vtbl_dog_as_animal; }
void _Cat_ctor(Cat* _this) { _this->vptr = &vtbl_cat_as_animal; }

// END COMPILER GENERATED


void do_speak(Animal* animal)
{
    auto fp_speak = animal->vptr->speak;
    fp_speak(animal);
}

int main()
{
    Dog dog; _Dog_ctor(&dog);
    Cat cat; _Cat_ctor(&cat);
    do_speak(&dog);
    do_speak(&cat);
}

So how can we fix this?

Sanable addresses the reload problem using a technique called vptr jamming or pointer hydration, which copies vptrs from one object to all living objects. The naive implementation used for Sanable’s first iteration rewrote just the first 8 bytes:

MyObject dummy; //Never used directly, just stealing a fresh vptr
memcpy(badObj, &dummy, sizeof(void*)); //All nonmember pointers have the same size, regardless of the pointed-to type
badObj->update(); //No longer crashes!

While this is easy to implement and test, it isn’t consistent. With multiple inheritance or virtual inheritance, we’d have multiple vptrs to overwrite, locking us out of using interface classes (IUpdatable). Additionally, compilers are allowed to store additional metadata² however they see fit: MSVC adds another pointer to type info right after the vptr, whereas GCC and Clang store that information in the vtable. and a compiler could theoretically skip the vtable entirely and put function pointers in the object itself.

We can refine our approach if we know the exact layout of an object in memory, which normally would mean writing a compiler plugin. However, that would only yield information for that specific compiler. Thanks to the offsetof macro, we can identify which bytes correspond to explicitly defined fields, and by process of elimination which bytes were generated by the compiler. These gaps must be either compiler-generated constants (metadata we should capture), or padding generated to respect alignment requirements (safe to write/ignore).

Writing out the members of every class is fragile and hard to reliably check for errors, so I wrote a dedicated tool for Sanable v2. While this uses Clang’s Abstract Syntax Tree, it runs as a pre-build step and generates source code that any compiler can use.

We can take this approach one step even further, differentiating between what is padding and what is an implicit field. We can create multiple dummy objects, first filling the memory they will occupy to some known value unique to that dummy. Constructors shouldn’t³ touch padding bytes (see this post), so the bytes that always match the known value must be padding. Bytes that change but don’t match the known value are instead marked as unknown/anomalous, and ignored.

Further steps

A significant drawback with this approach is that constructors must be called. Default constructors are the easy option, followed by constructors with trivial arguments that can be cast from 0, and default-constructible arguments. Nested nontrivial constructors are almost impossible to programmatically resolve. This also assumes that the constructors have no side effects.

One potential workaround would be to analyze bytecode at runtime. Although it would require per-platform definitions for instructions, it could be performed with any constructor without any side effects. The main instructions to look for would be mov/lea for setting vptrs, and call/ret to follow the full constructor hierarchy.

1. An access violation (Windows) or segmentation fault (*nix). C and C++ let you register a function for when a crash like this has happened, but there’s no well-defined way to recover from something like this. atexit hooks will still be called. ↩︎
2. The C++ specification does not specify any particular implementation for dynamic dispatch. Vtables are simply the de-facto standard. Some compilers store type metadata entirely separately from the object. Bjarne Stroustrup himself wrote a paper describing single-TU dynamic_cast using primes instead of vtables and extending it to dynamic dispatch. For the sake of simplicity, this article treats non-vtable-based compilers that still work as a nice bonus. Since Stroustrup’s scheme stores type information (the prime) in each object, it would conveniently work. ↩︎
3. But some do even though it’s breaking from the spec. GCC is known to pre-zero objects’ entire memory, padding included, before any constructors run. ↩︎