Decode gob output without knowing concrete types

Question

I'm using gob to serialize structs to disk. The struct in question contains an interface field, so the concrete type needs to be registered using gob.Register(...).

The wrinkle here is that the library doing the gob-ing should be ignorant of the concrete type in use. I wanted the serialization to be possible even when callers have defined their own implementations of the interface.

I can successfully encode the data by registering the type on the fly (see trivial example below), but upon trying to re-read that data, gob refuses to accept the un-registered type. Its frustrating, because it feels like all the data is there - why isn't gob just unpacking that as a main.UpperCaseTransformation struct if it's labelled as such?

package main

import (
    "encoding/gob"
    "fmt"
    "os"
    "strings"
)

type Transformation interface {
    Transform(s string) string
}

type TextTransformation struct {
    BaseString     string
    Transformation Transformation
}

type UpperCaseTransformation struct{}

func (UpperCaseTransformation) Transform(s string) string {
    return strings.ToUpper(s)
}

func panicOnError(err error) {
    if err != nil {
        panic(err)
    }
}

// Execute this twice to see the problem (it will tidy up files)
func main() {
    file := os.TempDir() + "/so-example"

    if _, err := os.Stat(file); os.IsNotExist(err) {
        tt := TextTransformation{"Hello, World!", UpperCaseTransformation{}}

        // Note: didn't need to refer to concrete type explicitly
        gob.Register(tt.Transformation)

        f, err := os.Create(file)
        panicOnError(err)

        defer f.Close()

        enc := gob.NewEncoder(f)
        err = enc.Encode(tt)
        panicOnError(err)
        fmt.Println("Run complete, run again for error.")
    } else {

        f, err := os.Open(file)
        panicOnError(err)

        defer os.Remove(f.Name())
        defer f.Close()

        var newTT TextTransformation
        dec := gob.NewDecoder(f)

        // Errors with: `gob: name not registered for interface: "main.UpperCaseTransformation"'
        err = dec.Decode(&newTT)
        panicOnError(err)
    }
}

My work-around would be to require implementers of the interface to register their type with gob. But I don't like how that reveals my serialization choices to the callers.

Is there any route forward that avoids this?

Answer 1

Philosophical argumentation

The encoding/gob package cannot (or rather should not) make that decision on its own. Since the gob package creates a serialized form independent of / detached from the app, there is no guarantee that values of interface types will exist in the decoder; and even if they do (matched by the concrete type name), there is no guarantee that they represent the same type (or the same implementation of a given type).

By calling gob.Register() (or gob.RegisterName()) you make that intent clear, you give green light to the gob package to use that type. This also ensures that the type does exist, else you would not be able to pass a value of it when registering.

Technical requirement

There's also a technical point of view that dictates this requirement (that you must register prior): you cannot obtain the reflect.Type type descriptor of a type given by its string name. Not just you, the encoding/gob package can't do it either.

So by requiring you to call gob.Register() prior, the gob package will receive a value of the type in question, and therefore it can (and it will) access and store its reflect.Type descriptor internally, and so when a value of this type is detected, it is capable of creating a new value of this type (e.g. using reflect.New()) in order to store the value being decoded into it.

The reason why you can't "lookup" types by name is that they may not end up in your binary (they may get "optimized out") unless you explicitly refer to them. For details see Call all functions with special prefix or suffix in Golang; and Splitting client/server code. When registering your custom types (by passing values of them), you are making an explicit reference to them and thus ensuring that they won't get excluded from the binaries.