我可以使用 protobuf-net 将 ConcurrentDictionaries 序列化和反序列化为同一个文件,然后读取它吗?

Can I use protobuf-net to serialize and deserialize ConcurrentDictionaries to the same file and then read it?

提问人:MrBott_a 提问时间:1/13/2023 最后编辑:MrBott_a 更新时间:1/13/2023 访问量:120

问:

因此,我正在开发一个海军模拟器的通信管理器,该模拟器管理通信并存储所有数据,我需要序列化以快速且每秒最多 10 次提交大型并发字典,我需要这样做的次数是可变的,因为这是我们实现重放功能的尝试,序列化的数量取决于模拟花费的时间。经过一番研究,我登陆了 protobuf-net,但我无法弄清楚。 我需要处理的对象是具有许多属性的非常复杂的类,其中一些是自定义类型,并且继承自从其他类继承的类。并发词典看起来像这样 我认为我唯一做对的是用属性装饰类。其余的我都想不通。ConcurrentDictionary<string, MyClass>

我需要多次序列化到同一个文件,然后读取该文件以重新创建 ConcurrentDictionaries。我尝试了不同的方法,但没有取得任何成功......我需要一个方向。

我认为问题在于我多次序列化到同一个文件,当我反序列化时,我只是将文件作为流提供给方法。我认为它试图读取整个文件并分解......Serializer.Deserialize

编辑:按照建议,我添加了更多细节来保护程序。

C# 性能 IO protobuf-net concurrentdictionary

评论

0赞 JonasH 1/13/2023
是否打算覆盖或追加数据?在这种情况下,什么是“大”?为什么要序列化并立即反序列化?这似乎是一堆不必要的工作。您可能还希望将并发词典转换为常规词典,并将对象转换为数据传输对象,以便更轻松地进行序列化。
0赞 MrBott_a 1/13/2023
@JonasH 我想附加 ConcurrentDictionary 的每个序列化,然后以“节奏”阅读它。对我来说,Big 就像一个包含 ~100 个对象的并发字典,每个对象有 ~100 个属性。我不需要立即反序列化它,重点是实现类似重放功能的东西,我在其中获取数据并在以后重新加载它。我正在考虑为 ConcurrentDictionary 制作一个类似代理类的东西,在那里我使用字典并自己管理线程安全,但作为一个拥有 ~10000 行项目的初级开发人员,这似乎有点令人生畏和恐惧。
0赞 Theodor Zoulias 1/13/2023
有多少?您是否将它们存储在字典列表中?ConcurrentDictionary<string, MyClass>
0赞 MrBott_a 1/13/2023
@TheodorZoulias Atm 我有 11 个,每个都包含不同类型的对象,但将来可能会增长。不,我为每种类型都对它们进行了硬编码,因为对公司来说,性能比代码复杂性更重要。ConcurrentDictionary<string, CustomClass>
0赞 Theodor Zoulias 1/13/2023
假设您要添加第 12 个字典,并且该文件当前包含 11 个字典。您的程序的理想行为是什么?崩溃并显示错误消息,建议删除现有文件?加载 11 个字典,将第 12 个字典留空?别的?

答:

1赞 Theodor Zoulias 1/13/2023 #1

您可以声明一个容器(结构或类),其中包含您的字典作为成员,然后序列化/反序列化此容器。下面是一个示例:

[ProtoContract]
struct MyDictionaries
{
    [ProtoMember(1)]
    public ConcurrentDictionary<string, int> A;
    [ProtoMember(2)]
    public ConcurrentDictionary<string, char> B;
}

public static void Main()
{
    MyDictionaries dictionaries = new();
    dictionaries.A = new();
    dictionaries.A.TryAdd("A", 1);
    dictionaries.A.TryAdd("B", 2);
    dictionaries.B = new();
    dictionaries.B.TryAdd("C", 'c');
    dictionaries.B.TryAdd("D", 'd');
    Console.WriteLine($"dictionaries.A: {String.Join(", ", dictionaries.A)}");
    Console.WriteLine($"dictionaries.B: {String.Join(", ", dictionaries.B)}");
    MemoryStream stream = new();
    Serializer.Serialize(stream, dictionaries);
    Console.WriteLine();

    stream.Position = 0;
    var deserialized = Serializer.Deserialize<MyDictionaries>(stream);
    Console.WriteLine($"deserialized.A: {String.Join(", ", deserialized.A)}");
    Console.WriteLine($"deserialized.B: {String.Join(", ", deserialized.B)}");
}

输出:

dictionaries.A: [A, 1], [B, 2]
dictionaries.B: [D, d], [C, c]

deserialized.A: [A, 1], [B, 2]
deserialized.B: [D, d], [C, c]

在线演示

文档明确指出支持类型,源代码中也有类型。IDictionary<TKey,TValue>ConcurrentDictionarySerializer

评论

0赞 MrBott_a 1/13/2023
是的,但这只在流上序列化一次,然后反序列化它,我需要沿着这条线做更多的事情。但是我得到的输出如下所示:dict1: [A, 1], [B, 2] dictX: [C, 3], [D, 4] dict1PD: [C, 3], [D, 4], [A, 1], [B, 2] dictXPD:
0赞 Theodor Zoulias 1/13/2023
@MrBott_a啊,您想在安全文件中存储多个词典。您是否确切知道文件中将存储多少个字典,或者它们的数量是可变的?
0赞 Theodor Zoulias 1/13/2023
使用 ValueTuple<T1,T2>链接的@MrBott_a示例
0赞 MrBott_a 1/13/2023
It is variable, I'm working on the communication manager in a naval simulation system and my program manages both communication and keeps all the data about the entire simulation. So the number of times an object is serialized depends on how long the simulation is running...it could be even hours
0赞 Theodor Zoulias 1/13/2023
MrBott_a I think that this is important information, that should be included in the question.
1赞 JonasH 1/13/2023 #2

There is a bit to unpack here.

To serialize multiple independent objects to the same stream you can use / . This should allow you to serialize objects one after each other. See ProtoInclude for how to handle inheritance.Serializer.SerializeWithLengthPrefixSerializer.DeserializeWithLengthPrefix

To serialize each object I would tend to prefer to convert the objects into a separate type that is only used for serialization. Sometimes called a Data Transfer Object or DTO. This lets you separate the concerns of serialization from all kinds of domain logic, at the cost of some duplication of code.

There are a few ways to manage size. One approach is to only change changes to state, not the entire state. Something similar is sometimes used for games, where you only need to record the user input to allow you to replay the entire game. Another approach is compression, since states probably do not change that much. LZ4 claims to be one of the faster algorithms around. You might also want to keep things in memory if possible, since even the fastest SSD is much slower than memory.

I would highly recommend setting up a simple test environment. I.e. start by serializing a simple object, continue with a complex object, a dictionary of complex objects, and so on. This is also a good opportunity to measure performance.

评论

0赞 MrBott_a 1/13/2023
I did as you suggested and built a small test environment emulating this part of my program and succeded in decorating the classes with ProtoInclude and (i think) serializing with , the problem lies in the deserialization part. When I call the deserialization method I expect to have the first instance of the dictionary deserialized. Somehow I get the last one.Serializer.SerializeWithLengthPrefix
0赞 MrBott_a 1/13/2023
Here I pasted both the serialization and deserialization methods I use
0赞 JonasH 1/13/2023
@MrBott_a If you intend to append multiple objects you need to use , If you use you will just overwrite the file. But if you are going to write many objects you might want to keep the stream open.File.Open(path, FileMode.Append)File.Create