Serializing C++ with X-Macros

Stable Diffusion prompt: A twisting maze of passages, dark and scary.

Once upon a time I was yanked out of my cozy safe space working on a video transport SDK and was told, “Guess what? You’re going to be working on FPGA firmware now.” Being an agreeable fellow I said, “Sure, why not?” and rolled up my sleeves. It soon became apparent, however, that there were several problems standing in the way of my can-do attitude, not the least of which being that we didn’t actually have the hardware yet. The other problem was that I barely even knew how to spell FPGA and had no idea how one developed firmware for one of them.

Here’s a quick primer for those of you who haven’t already tabbed away to watch cat videos. An FPGA (field programmable gate array) is like programmable hardware. Clever designers can make them do specialized extremely high performance tasks like processing and transmitting high resolution uncompressed video with almost no latency.

Because FPGAs look a lot like specialized pieces of hardware, you need ‘regular’ code to control them. This could be a full operating system with drivers or just a simple runtime that pokes away at the hardware registers. This is what we’d call ‘the firmware’. FPGA designers are greedy fellows who want to use all the circuitry for themselves but they roll their eyes, sigh and say, “Fine…you can have a little processor to run some code on.” They set aside a bit of space to run a virtual (soft) CPU for the firmware. In the case of Altera FPGAs it’s a thing called a NIOS processor and it can run various systems like Linux or FreeRTOS.

Great, in theory, but I still don’t have a device to work with so how do I even get started? Plus, those FPGA guys are so stingy they hardly gave us any memory or processing power. I’ve got this huge SNMP specification with hundreds of settings which needs a back end in C++ that can store settings and control the hardware. Plus it needs human readable config files that can be installed onto the devices after they’re built. There’s not a lot of space and everything gets baked into a single image so we can’t just add libraries willy-nilly either. What to do?

In talking to some of the other team members it seems like there is a library on the device that can read and write YAML, a fairly simple config file format. This library also can be used on Windows so now we have an entry point. We can start writing generic C++ on Windows driven by unit tests and just make a nice hardware abstraction layer for reading/writing to the hardware once it’s running on the FPGA.

Finally we arrive at the point of the article. How do I serialize these settings to and from these huge C++ classes? The answer my twisted brain came up with is x-macros. This is a clever way to use the preprocessor to write a whole bunch of repetitive code for you. For example:

// Define a single macro with some colors.
#define COLORS \
    X(RED)     \
    X(BLACK)   \
    X(WHITE)   \
    X(BLUE)
  
// Expand the colors as an enum
enum colors {
    #define X(value) value,
        COLORS
    #undef X
};
  
// Expand the colors into a switch statement
char* toString(enum colors value)
{
    switch (value) {
        #define X(color) \
            case color:  \
                return #color;
                COLORS
        #undef X
    }
}

You first create a global list, a single point of truth for the set of things you want to work with, in this case some colors. Then you define the X macro as some arbitrary bit of code, include the global list and the preprocessor expands it into code. For the enum this expansion is just the value with a comma after it. For the toString function it’s a switch statement returning the string value (#color). The preprocessor has a few tricks like ‘#’ to make something a string or ## to concatenate things together. Once this preprocessing happens the above code is expanded like so:

enum colors {
    RED,
    BLACK,
    WHITE,
    BLUE,
};
  
char* toString(enum colors value)
{
    switch (value) {
    case RED:
        return "RED";
    case BLACK:
        return "BLACK";
    case WHITE:
        return "WHITE";
    case BLUE:
        return "BLUE";
    }
}

Not too complicated and a great way to avoid repeating yourself. If you add another color it only happens in the one place and all the utility functions and enums are automatically updated. There is also a variation where instead of defining and undefining an expansion macro you use it as an argument. So the above code could be rewritten as this:

#define COLORS(X) \
    X(RED)     \
    X(BLACK)   \
    X(WHITE)   \
    X(BLUE)

#define EXPAND_ENUM(value) value,
#define EXPAND_STRING(color) \
  case color:  \
    return #color;

enum colors {
  COLORS(EXPAND_ENUM)
};
  
// Expand the colors into a switch statement
char* toString(enum colors value)
{
    switch (value) {
      COLORS(EXPAND_STRING)
    }
}

You’re probably asking yourself how one might use this to serialize/deserialize a C++ class into YAML. The answer to that, my friend, is a twisted descent into some of the most egregious abuse of the preprocessor that I would ever perform. I cannot in any way recommend this approach as anything other than an exercise in masturbatory mental gymnastics. Either way, let’s dive right in shall we?

As I struggle to imagine how one might explain this it occurs to me to start at the finish and work back towards the beginning. This way you can see the result and then judge for yourself if it was worth it in the end. So let’s start out with our C++ class that describes a device. It’s got a bunch of accessors that maybe do things to the underlying device or maybe just store information. So you’ve got a bunch of GetThis(), GetThat(), SetThis(this), SetThat(that) methods. You want these things to be stored in a YAML file and loaded/saved on demand. So at the very bottom of your DeviceInfo class you add this code:

#define PRESET_MEMBERS(_) \
 _(Desc, "Description", string) \
 _(Manufacturer, "Manufacturer", string) \
 _(PartNumber, "Part Number", string) \
 _(SerialNumber, "Serial Number", string) \
 ...
#define PERSIST_MEMBERS(_) \
 _(UserDesc, "User Description", string) \
 _(SAPAddr, "SAP Broadcast IP", uint32) \
 _(SAPPort, "SAP Broadcast Port", uint) \
 _(SAPIPv6Addr, "SAP Broadcast IPv6", string) \
 ...

#define ISETTINGS_CLASS_OVERRIDE DeviceInfo
#define ISETTINGS_CHILD_TABLE mEthernetEntryVector
#define ISETTINGS_ADD_ENTRY AddEthernetIfEntry
#define ISETTINGS_INDEX_OFFSET 1 // Ethernet entries are indexed from 1
#define ISETTINGS_NESTED

#include "ISettingsPersist.def"

Because I’m cantankerous I use ‘_’ instead of ‘X’ for my macros but essentially these are two lists of all the members you want to write to YAML. The preset vs persist versions are for read-only presets that are hard coded and user modifiable settings respectively. Any member that has a Get and Set method can use be saved and restored simply by adding a line to these definitions. So we have a GetDesc/SetDesc which get and set a string and a GetSAPAddr/SetSAPAddr which get and set a uint32.

That’s it. If you want a new member in the class, create get/set methods and add a line to one of these lists. In terms of ease for future modification, it doesn’t get much better than that. The defines underneath those lists are where the rabbit hole begins.

The first entry ISETTINGS_CLASS_OVERRIDE is just the name of this class, DeviceInfo. The next few defines I’m going to gloss over but essentially you can have nested tables in the SNMP specification (the ‘MIB’) which means that the DeviceInfo can have one or more ethernet entries. Also, the MIB has cute little ‘quirks’ such as how some tables are zero indexed and others are one indexed. Never mind any of that because this is complicated enough already. We’re going to dive into ISettingsPersist.def in a minute but first let’s consider the macro expansions we’re going to use for each member of our lists above.

If you recall, when we wanted to expand something we need an expansion macro. In our case we need one for loading from YAML and one for saving to YAML. These macros look like this:

// ISettingsPersist.h (persistence interface class)
// These macros get expanded for each of the member variables
// we want to save or restore.
#define ISETTINGS_EMIT_MEMBERS(member, name, type) \
  yaml_scalar_event_initialize(&lEvent, NULL, NULL, (yaml_char_t *)(name), strlen((name)), 1, 1, YAML_ANY_SCALAR_STYLE); \
  if (!yaml_emitter_emit(aEmitterPtr, &lEvent)) goto error; \
  ISettingsPersist::init_##type(&lEvent, Get##member()); \
  if (!yaml_emitter_emit(aEmitterPtr, &lEvent)) goto error;

#define ISETTINGS_STORE_MEMBERS(member, name, type) \
  if (lKey == (name)) { \
    Set##member(ISettingsPersist::parse_##type(lValue)); \
  } else

So now consider these lines being expanded with one of the member lines from above, for example _(Desc, “Description”, string).

// Emit (writing to file)
yaml_scalar_event_initialize(&lEvent, NULL, NULL,
  (yaml_char_t *)"Description", strlen("Description"), 1, 1, YAML_ANY_SCALAR_STYLE); \
if (!yaml_emitter_emit(aEmitterPtr, &lEvent)) goto error; \
ISettingsPersist::init_string(&lEvent, GetDesc()); \
if (!yaml_emitter_emit(aEmitterPtr, &lEvent)) goto error;

// Store (loading from file)
if (lKey == "Description") { \
  SetDesc(ISettingsPersist::parse_string(lValue)); \
} else

You can see the text name of the property from the YAML file as well as the Get/SetDesc calls and the init_string/parse_string being created by the macro. It’s instructive to look at some of these helper functions, some of which do almost nothing but are there for the sake of consistency.

static bool parse_bool(std::string & aValue) {
  static const char *lTruthValues[] = {
    "true",
    "y",
    "yes",
    "on",
    NULL
  };
  int i = 0;
  const char *lValue = aValue.c_str();

  while (NULL != lTruthValues[i]) {
    if (strcasecmp(lValue, lTruthValues[i++]) == 0) {
      return true;
    }
  }

  return false;
}
static std::string parse_string(std::string &aValue) { return aValue; }
static int parse_int(std::string &aValue) { return strtol(aValue.c_str(), NULL, 0); }

static void init_bool(yaml_event_t *aEvent, bool aBool) {
  char const *lBuf = aBool ? "true" : "false";
  yaml_scalar_event_initialize(aEvent, NULL, NULL, (yaml_char_t *)lBuf, strlen(lBuf), 1, 1, YAML_ANY_SCALAR_STYLE);
}

static void init_int(yaml_event_t *aEvent, int aInt) {
  char lBuf[BUF_SIZE];
  sprintf_s(lBuf, "%d", aInt);
  yaml_scalar_event_initialize(aEvent, NULL, NULL, (yaml_char_t *)lBuf, strlen(lBuf), 1, 1, YAML_ANY_SCALAR_STYLE);
}

Before getting to the ISettingsPersist.def which we include after defining all our members we should look at the ISettingsPersist interface. It implements a bunch of common features like an isDirty flag for knowing when to save but it’s main features are the interfaces it needs child classes to override. Looking at the DeviceInfo class definition we see this:

// Persistence
bool SaveTo(yaml_emitter_t *aEmitterPtr, SettingsType_t aType);

bool LoadFrom(yaml_parser_t *aParserPtr, SettingsType_t aType);
 
char const *GetStartTag() { return "Device Information"; }

char const *GetPresetCfgFileName() { return "DeviceInfo.preset"; }
char const *GetPersistCfgFileName() { return "DeviceInfo.persist"; }

void SetBulkMode(bool aMode);
bool IsDirty(void) const;

So we see that DeviceInfo can set the start tag for its YAML file sections as well as their file names. ‘Bulk mode’ is used to temporarily turn off all the hardware side effects while loading a file. The SaveTo and LoadFrom are where we finally come back to ISettingsPersist.def. It gets included after we set up the member list and various ISETTINGS_ macros above. So for each C++ class we’re planning to serialize this code gets included (only showing the simpler non-nested version). Keep your eyes open for the PRESET_MEMBERS() and PERSIST_MEMBERS() where the member lists get expanded. For the LoadFrom() it is a series of if/else statements so you can see that if it falls through then the very last case is an error handler. Non-fatal so that extra settings will be ignored for forward compatibility.

// Definition file for generic SaveTo() and LoadFrom() methods that are
// suitable for classes needing no fancy handling. ie. Nothing but member
// variables

#define _xstr_(x) _str_(x)
#define _str_(x) #x

#ifdef ISETTINGS_SIMPLE

// A 'simple' settings file that only has member variable to persist.
// Just define PRESET_MEMBERS, PERSIST_MEMBERS and ISETTINGS_CLASS_OVERRIDE

bool ISETTINGS_CLASS_OVERRIDE::SaveTo(yaml_emitter_t * aEmitterPtr, SettingsType_t aType)
{
  yaml_event_t lEvent;

  ISettingsPersist::init_int(&lEvent, (int)mMyIx);
  if (!yaml_emitter_emit(aEmitterPtr, &lEvent)) goto error;

  yaml_mapping_start_event_initialize(&lEvent, NULL, NULL, 1, YAML_ANY_MAPPING_STYLE);
  if (!yaml_emitter_emit(aEmitterPtr, &lEvent)) goto error;

  if (aType == PRESET) {
    PRESET_MEMBERS(ISETTINGS_EMIT_MEMBERS)
  }
  else {
    PERSIST_MEMBERS(ISETTINGS_EMIT_MEMBERS)
  }

  yaml_mapping_end_event_initialize(&lEvent);
  if (!yaml_emitter_emit(aEmitterPtr, &lEvent)) goto error;
  return true;
error:
  ISettingsPersist::ProcessSyntaxError("Unable to emit YAML event\n", NULL);
  return false;
}

bool ISETTINGS_CLASS_OVERRIDE::LoadFrom(yaml_parser_t * aParserPtr, SettingsType_t aType)
{
  yaml_event_t lEvent;
  bool lDone = false;
  std::string lKey("");
  std::stringstream lError;

  SetBulkMode(true);
  while (!lDone) {
    if (!yaml_parser_parse(aParserPtr, &lEvent)) {
      lError << "Parse error";
      goto error;
    }
    switch (lEvent.type) {
    case YAML_NO_EVENT:
    case YAML_ALIAS_EVENT:
      break;
    case YAML_MAPPING_START_EVENT:
      break;
    case YAML_MAPPING_END_EVENT:
      lDone = true;
      break;
    case YAML_SCALAR_EVENT:
      if (lKey.length() == 0) {
        lKey = (char *)lEvent.data.scalar.value;
      }
      else {
        std::string lValue((char *)lEvent.data.scalar.value);
        if (aType == PERSIST) {
          PERSIST_MEMBERS(ISETTINGS_STORE_MEMBERS) {
            lError << "Unknown setting \"" << lKey << " = " << lValue << "\"";
            ISettingsPersist::ProcessSyntaxError(lError.str().c_str(), aParserPtr);
            lError.str("");
          }
        }
        else {
          PRESET_MEMBERS(ISETTINGS_STORE_MEMBERS) {
            lError << "Unknown setting \"" << lKey << " = " << lValue << "\"";
            ISettingsPersist::ProcessSyntaxError(lError.str().c_str(), aParserPtr);
            lError.str("");
          }
        }
        lKey.clear();
      }
      break;
    default:
      lError << "Unexpected YAML " << ISettingsPersist::DecodeYamlEvent(lEvent.type) << "event";
      goto error;
      break;
    }
  }

  SetBulkMode(false);
  return true;
error:
  SetBulkMode(false);
  // decode yaml error with lError
  ISettingsPersist::ProcessSyntaxError(lError.str().c_str(), aParserPtr);
  return false;
}

The final piece of the interface lies in the ISettingsPersist class:

static bool RunEmitter(ISettingsPersist *aPersistTbl[], SettingsType_t aType, char *aBufPtr, int aLen, size_t &aBytesWrittenRef);
static bool RunEmitter(ISettingsPersist *aPersistTbl[], SettingsType_t aType, char const *aPath);

static bool RunParser(ISettingsPersist *aPersistTbl[], SettingsType_t aType, char const *aBufPtr, size_t aLen);
static bool RunParser(ISettingsPersist *aPersistTbl[], SettingsType_t aType, char const *aPath);

These are the main ‘driver’ functions that run the parser or emitter for an array of classes which implement the ISettingsPersist interface. They run on both files and strings which means that it’s very easy for unit tests or the firmware to simply create any one of these objects and load/save their configs. For example a unit test might look like this:

TEST_F(DeviceInfoTest, SavePresetConfig)
{
  char lBuf[0x10000];
  size_t lBytesWritten;
  // Write the config to a string
  bool lRet = ISettingsPersist::RunEmitter(mSettings, ISettingsPersist::PRESET, lBuf, 0x10000, lBytesWritten);
  ASSERT_TRUE(lRet);

  std::cout << "Preset Config:\n";
  std::cout << lBuf << std::endl;

  // Change the device info (mDeviceInfo is in mSettings above)
  mDeviceInfo->SetDesc("Some random description");

  // Run the parser on the string and observe that the description
  // has been reverted to the original DEVICE_DESC default
  lRet = ISettingsPersist::RunParser(mSettings, ISettingsPersist::PRESET, lBuf, lBytesWritten);
  ASSERT_TRUE(lRet);
  EXPECT_EQ(mDeviceInfo->GetDesc(), DEVICE_DESC);
}

There are a few glossed over details like nested tables and the details of some of the yaml helper functions but this is pretty much the whole thing. It’s complicated but from the future user’s perspective it’s very simple. To add a new member, implement Get/Set and add one line. To read/write configs simply call RunParser/RunEmitter.

I hope you enjoyed reading about this as much as I did writing it.