Protobuf.net Exception - Timeout while inspecting

问题:

I am sometimes receiving the following exception when attempting to deserialise an object using protobuf.net. I'm surprised as I never have more than a single thread deserialising the same object at the same time and the protobuf.net source does not seem to use any static objects for deserialising. The exception does suggest a solution but I am unsure as to how to implement so would welcome an example.

Base Exception Type:
System.TimeoutException: Timeout while inspecting metadata; this may indicate a deadlock. This can often be avoided by preparing necessary serializers during application initialization, rather than allowing multiple threads to perform the initial metadata inspection
at ProtoBuf.Meta.RuntimeTypeModel.TakeLock(Boolean& lockTaken)
at ProtoBuf.Meta.RuntimeTypeModel.FindOrAddAuto(Type type, Boolean demand, Boolean addWithContractOnly, Boolean addEvenIfAutoDisabled)
at ProtoBuf.Meta.RuntimeTypeModel.GetKey(Type type, Boolean demand, Boolean getBaseKey)

Inner Exception Type:
System.TimeoutException: Timeout while inspecting metadata; this may indicate a deadlock. This can often be avoided by preparing necessary serializers during application initialization, rather than allowing multiple threads to perform the initial metadata inspection
at ProtoBuf.Meta.RuntimeTypeModel.TakeLock(Boolean& lockTaken)
at ProtoBuf.Meta.RuntimeTypeModel.FindOrAddAuto(Type type, Boolean demand, Boolean addWithContractOnly, Boolean addEvenIfAutoDisabled)
at ProtoBuf.Meta.RuntimeTypeModel.GetKey(Type type, Boolean demand, Boolean getBaseKey)

Stack Trace: 
at ProtoBuf.Meta.RuntimeTypeModel.GetKey(Type type, Boolean demand, Boolean getBaseKey)
at ProtoBuf.Meta.TypeModel.GetKey(Type& type)
at ProtoBuf.Meta.TypeModel.Deserialize(Stream source, Object value, Type type)

Regards, Marc

edit to add: I define my serialisable objects thus:

[ProtoContract]
public class Job
{
    [ProtoMember(1)]
    public long JobId { get; private set; } 
}

It would be difficult for me to easily call PrepareSerialiser on each one of my serialisable objects as I have many in different namespaces. However thinking about it what happens if protobuf is asked to deserialise two objects of the same type, a type it has not seen before, at exactly the same time?

回答1:

RuntimeTypeModel.Default (the default model) is static, and underpins the static Serializer class (re the comment that there wasn't any static state). Despite adding this check due to paranoia, I have never seen this error raised. I would absolutely love to see an example that will repro this. Are you sure you aren't threading? If not for threading, I can only wonder: is the type-model really really big?

Indeed, even with many threads hitting it aggressively at startup (i.e. here on stackoverflow) it is well behaved. As the error message hints, you might try calling Serializer.PrepareSerializer during app-startup, which will pre-initialize everything, avoiding any threading concerns.

But hey! At least it didn't deadlock!

The odd thing here, though, is that it still shouldn't be possible to deadlock - it deliberately uses a coarse lock to avoid issues from the order it takes lock. Again - I'd really love to see a sample.

回答2:

Old question, but if somebody happens to get this error, please check the DLL version you are using. Chances for this exception to show up in portable version are very high.

There are a couple of PRs related to this issue with the Portable version that are https://github.com/mgravell/protobuf-net/pull/98 and https://github.com/mgravell/protobuf-net/pull/114.

回答3:

For me preparing Serializers using

Serializer.PrepareSerializer<Type>();

didn't really helped me.

For me, solution (aka workaround) was to serialize type at startup which was causing trouble at app startup :

MessageSerialization.Serialize(new Type());

Where MessageSerialization.Serialize is serialization method using protobuf Serializer.Serialize(stream, o)

回答4:

I'm getting this same exact error on my servers. Though I'm not sure what is causing the error. A few days ago it happened twice a few hours apart while our servers were under the highest load we've experienced. Running 8 servers the CPU across all of them went from 70% to 100% in seconds but a slightly different times. For example each of the servers might have started this spike 1-5 minutes after the first.

Never seen this happen before and i've had this code in production for a few months. Still haven't been able to reproduce it and I can't tell if the error is thrown because the server was at 100% CPU or if this is what caused the server to spike. Stopping all the connections to the servers and letting the cpu drop back to 0 fixed the issue. No iis restart was required.

When IIS starts i run the following code once for each type.

var type = this.GetType();
RuntimeTypeModel.Default.Add(type, true);
Int32 i = 1;
foreach(PropertyInfo info in type.GetProperties())
{
    if(info.CanWrite)
    {
        RuntimeTypeModel.Default[type].AddField(i++, info.Name);
    }
}