Carbonite Data Serialization

Carbonite Data Serialization

Carbonite is a library for serializing assets for use in native code.

The idea for Carbonite came to me while I was working on yet another interactive graphics project. I had once again found myself importing/generating complex data in C# and then writing code to load it into the actual C++ application, and I thought to myself, “I really ought to make a standalone library for this kind of thing.”

So I set out to make one definitive, reuseable C# to C++ serialization library, inspired by my experience working on the content tools at Bungie and my work reversing Frostbite, Havok, FaceFX, and Unreal 3 & 4 files. A new library starting from scratch, exactly how I would want the library to work if I were consuming it.

So, I wrote Carbonite, which is focused around an effective blend of these guiding principles:

• Be intuitive.
• Be opinionated.
• Be simple.

In other words, it should only take a few lines of code to effectively use Carbonite the way it’s supposed to be used, and it should rather clear what those lines of code ought to be.

Taking the guiding principles together (and maintaining balance where they are in tension with one another) led me to some interesting design decisions:

• Type information is not serialized whatsoever. It is assumed that the native code knows the types of the objects in the Carbonite Image it is reading. This is because the unfreezing process does not require type information, and C++ does not provide a robust reflection system anyways. Adding type information would massively bloat the library for very little practical gain. If a developer needs to store their own type information within the image to correctly determine how to read the objects, they can handle that themselves trivially.
• Serialization only works one way. Carbonite Images are only written using C#, and are only read using C++. I am considering adding support for reading images using C#, for use in developing diagnostic tools, however that would still require the code to know the data types ahead of time (see: above).

The implementation of Carbonite is very straightforward: the C# library includes a source generator to generate the serialization code for a developer’s data types, and then the C++ library consists of a single function that simply adjusts the relative offsets in the serialized image to be valid pointers.

Carbonite is available as NuGet packages and from its GitHub repository.