In-depth Analysis of Serialization of Millions of C# Objects: How to Achieve the Perfect Balance Between Speed and Size in Network Transmission

Table of Contents

1. Background
2. Building Test Data
3. Comparison of Approaches
   • JSON Serialization
   • Custom Binary Serialization
   • BinaryWriter / BinaryReader
   • ProtoBuf
   • MessagePack
   • Approach Analysis
   • Benchmark Tests
4. Summary

1. Background

Hello everyone, I'm Dotnet9.

In network communication, data serialization is the process of converting an object's state into a form that can be stored or transmitted. This is especially critical for TCP network transmission. In projects where tens of thousands of data items need to be transmitted, the traditional JSON serialization approach, due to its redundant field names and string format, results in large binary packet sizes and inefficient serialization/deserialization. To solve these problems, I considered more efficient serialization methods to reduce packet size and improve processing speed. This article will explore four serialization methods: custom binary serialization, BinaryWriter/BinaryReader, MessagePack, and ProtoBuf. By comparing their performance, I aim to provide what I currently consider best practices.

2. Building Test Data

Create a C# console application. Add two classes, Organization and Member, containing basic data types and List<T>. Other arrays and dictionaries can be extended as needed:

public class Organization
{
    public int Id { get; set; }

    public string[]? Tags { get; set; }

    public List<Member>? Members { get; set; }
}

public class Member
{
    public int Id { get; set; }

    public string? Name { get; set; }

    public string? Description { get; set; }

    public string? Address { get; set; }

    public double Value { get; set; }

    public long UpdateTime { get; set; }
}

Create 1 million test data items to align with the title:

public class BenchmarkTest
{
    /// <summary>
    /// Number of test data items
    /// </summary>
    private const int DataCount = 1000000;

    private static readonly Random RandomShared = new(DateTime.Now.Millisecond);

    /// <summary>
    /// Test data
    /// </summary>
    private static readonly Organization TestData = new()
    {
        Id = 1,
        Tags = Enumerable.Range(0, 5).Select(index => $"TestTag{index}").ToArray(),
        Members = Enumerable.Range(0, DataCount).Select(index => new Member()
        {
            Id = index,
            Name = $"TestName{index}",
            Description = $"TestDescription{RandomShared.Next(1, int.MaxValue)}",
            Address = $"TestAddress{RandomShared.Next(1, int.MaxValue)}",
            Value = RandomShared.Next(1, int.MaxValue),
            UpdateTime = ((DateTimeOffset)DateTime.Now).ToUnixTimeMilliseconds()
        }).ToList()
    };
}

3. Comparison of Approaches

First, create a serialization interface ISerializeHelper, which each serialization provider must implement:

public interface ISerializeHelper
{
    byte[] Serialize(Organization data);

    Organization? Deserialize(byte[] buffer);
}

Then create the BenchmarkTest class and add the RunSerialize method to execute the serialization provider. This method calls the serializer and deserializer of the provider in sequence, and prints statistics on time consumption and packet size:

public class BenchmarkTest
{
    // ... omitted previous code
    private static void RunSerialize(ISerializeHelper helper)
    {
        Stopwatch sw = Stopwatch.StartNew();

        var buffer = helper.Serialize(TestData);

        sw.Stop();
        Log($"{helper.GetType().Name} Serialize {sw.ElapsedMilliseconds}ms {buffer.Length}byte");

        sw.Restart();

        var data = helper.Deserialize(buffer);

        sw.Stop();

        Log($"{helper.GetType().Name} Deserialize {sw.ElapsedMilliseconds}ms {data?.Members?.Count} items");
    }

    private static void Log(string log)
    {
        Console.WriteLine($"{DateTime.Now:yyyy-MM-dd HH:mm:ss fff}: {log}");
    }
}

3.1. JSON Serialization

First, test JSON serialization. Install the System.Text.Json package:

<PackageReference Include="System.Text.Json" Version="8.0.0" />

Create the JsonSerializeHelper provider implementing the ISerializeHelper interface:

using ByteTest.Core.Models;
using System.Text.Json;

namespace ByteTest.Core.Helpers;

public class JsonSerializeHelper : ISerializeHelper
{
    public byte[] Serialize(Organization data)
    {
        return JsonSerializer.SerializeToUtf8Bytes(data);
    }

    public Organization? Deserialize(byte[] buffer)
    {
        var data = JsonSerializer.Deserialize<Organization>(buffer);
        return data;
    }
}

The JsonSerializer.SerializeToUtf8Bytes method directly generates byte data instead of first generating a string and then converting to bytes. This is useful for scenarios where JSON data needs to be written to a file or network stream, as these scenarios usually require byte data rather than a string. Additionally, it can improve performance and reduce memory pressure by avoiding unnecessary string allocations.

Then add the test method Test to the BenchmarkTest class:

public static void Test()
{
    RunSerialize(new JsonByteSerializeHelper());
}

Call Test() in Program:

BenchmarkTest.Test();

Program output:

2023-12-10 22:28:24 880: JsonByteSerializeHelper Serialize 2813ms 196227181byte
2023-12-10 22:28:26 858: JsonByteSerializeHelper Deserialize 1964ms 1000000 items

JSON serialization of 1 million data items takes 2.8s, packet size 187.14 MB – truly large and slow.

3.2. Custom Binary Serialization

Next, test custom binary serialization, which I commonly used before and now see as verbose. First, define the data packet field specification:

Add CustomSerializeHelper implementing the ISerializeHelper interface:

using ByteTest.Core.Models;

namespace ByteTest.Core.Helpers;

public class CustomSerializeHelper : ISerializeHelper
{
    public byte[] Serialize(Organization data)
    {
        // 1. Calculate Id
        var idBuffer = BitConverter.GetBytes(data.Id);

        // 2. Calculate Tags array
        var tagBuffer = GetBytes(data.Tags);

        // 3. Calculate Members
        var membersBuffer = GetBytes(data.Members);

        return GetBytes(new[] { idBuffer, tagBuffer, membersBuffer });
    }

    public Organization? Deserialize(byte[] buffer)
    {
        var data = new Organization();
        var index = 0;

        data.Id = BitConverter.ToInt32(buffer, index);
        index += sizeof(int);

        data.Tags = GetTags(buffer, ref index);
        data.Members = GetMembers(buffer, ref index);
        return data;
    }

    /// <summary>
    /// Get byte[] from string array
    /// </summary>
    private byte[] GetBytes(string[]? data)
    {
        var dataCount = data?.Length ?? 0;
        var dataCountBuffer = BitConverter.GetBytes(dataCount);

        if (dataCount <= 0)
        {
            return dataCountBuffer;
        }

        var dataValueBuffers = data!.Select(item => ByteHelper.GetBytes(item)!).ToArray();
        var dataValueBuffer = GetBytes(dataValueBuffers);
        return GetBytes(new[] { dataCountBuffer, dataValueBuffer });
    }

    /// <summary>
    /// Get byte[] from list of Members
    /// </summary>
    private byte[] GetBytes(List<Member>? data)
    {
        var dataCount = data?.Count ?? 0;
        var dataCountBuffer = BitConverter.GetBytes(dataCount);

        if (dataCount <= 0)
        {
            return dataCountBuffer;
        }

        var dataValueBuffers = data!.Select(item =>
        {
            var idBuffer = BitConverter.GetBytes(item.Id);
            var nameBuffer = ByteHelper.GetBytes(item.Name);
            var descriptionBuffer = ByteHelper.GetBytes(item.Description);
            var addressBuffer = ByteHelper.GetBytes(item.Address);
            var valueBuffer = BitConverter.GetBytes(item.Value);
            var updateTimeBuffer = BitConverter.GetBytes(item.UpdateTime);

            var buffer = GetBytes(new byte[][]
                { idBuffer, nameBuffer, descriptionBuffer, addressBuffer, valueBuffer, updateTimeBuffer });
            return buffer;
        }).ToArray();
        var dataValueBuffer = GetBytes(dataValueBuffers);
        return GetBytes(new[] { dataCountBuffer, dataValueBuffer });
    }

    private byte[] GetBytes(byte[][] data)
    {
        var dataBufferLen = data.Sum(itemBuffer => itemBuffer.Length);
        var dataBuffer = new byte[dataBufferLen];
        var dataIndex = 0;
        for (var i = 0; i < data.Length; i++)
        {
            var itemBuffer = data[i];
            Array.Copy(itemBuffer, 0, dataBuffer, dataIndex, itemBuffer.Length);

            dataIndex += itemBuffer.Length;
        }

        return dataBuffer;
    }

    private string[]? GetTags(byte[] buffer, ref int index)
    {
        var count = BitConverter.ToInt32(buffer, index);
        index += sizeof(int);

        if (count <= 0)
        {
            return default;
        }

        var data = new string[count];
        for (var i = 0; i < count; i++)
        {
            data[i] = GetString(buffer, ref index);
        }

        return data;
    }

    private List<Member>? GetMembers(byte[] buffer, ref int index)
    {
        var count = BitConverter.ToInt32(buffer, index);
        index += sizeof(int);

        if (count <= 0)
        {
            return default;
        }

        var data = new List<Member>();
        for (var i = 0; i < count; i++)
        {
            var people = new Member();

            people.Id = BitConverter.ToInt32(buffer, index);
            index += sizeof(int);

            people.Name = GetString(buffer, ref index);

            people.Description = GetString(buffer, ref index);

            people.Address = GetString(buffer, ref index);

            people.Value = BitConverter.ToDouble(buffer, index);
            index += sizeof(double);

            people.UpdateTime = BitConverter.ToInt64(buffer, index);
            index += sizeof(long);

            data.Add(people);
        }

        return data;
    }

    private string GetString(byte[] buffer, ref int index)
    {
        var count = BitConverter.ToInt32(buffer, index);
        index += sizeof(int);

        if (count <= 0)
        {
            return string.Empty;
        }

        var data = ByteHelper.DefaultEncoding.GetString(buffer, index, count);
        index += count;
        return data;
    }
}

There’s quite a bit of code, but the idea is to use BitConverter.GetBytes and BitConverter.ToXXX to get or set byte[] for basic data types. For copying multiple byte[] arrays, if the data is small, use Array.Copy; if larger, Buffer.BlockCopy is recommended.

The code uses a helper class ByteHelper to compute string byte[]:

public static class ByteHelper
{
    public static Encoding DefaultEncoding = Encoding.UTF8;

    /// <summary>
    /// Get binary data for a string: string binary = 4 bytes (int) representing length + n bytes (actual string value)
    /// </summary>
    public static byte[] GetBytes(string? str)
    {
        if (string.IsNullOrEmpty(str))
        {
            return BitConverter.GetBytes(0);
        }

        var strValueBuffer = DefaultEncoding.GetBytes(str);
        var strValueLen = strValueBuffer.Length;
        var strValueLenBuffer = BitConverter.GetBytes(strValueLen);

        var strBufferLen = sizeof(int) + strValueLen;
        var strBuffer = new byte[strBufferLen];

        var index = 0;
        Array.Copy(strValueLenBuffer, 0, strBuffer, index, sizeof(int));
        index += sizeof(int);

        Array.Copy(strValueBuffer, 0, strBuffer, index, strValueLen);

        return strBuffer;
    }
}

We modify the Test method to include CustomSerializeHelper:

public static void Test()
{
    var serializeHelpers = new List<ISerializeHelper>
    {
        new JsonSerializeHelper(),
        new CustomSerializeHelper()
    };

    serializeHelpers.ForEach(RunSerialize);
}

Program output:

2023-12-10 22:45:14 701: JsonSerializeHelper Serialize 2774ms 196225588byte
2023-12-10 22:45:16 613: JsonSerializeHelper Deserialize 1898ms 1000000 items
2023-12-10 22:45:17 414: CustomSerializeHelper Serialize 801ms 92854209byte
2023-12-10 22:45:18 072: CustomSerializeHelper Deserialize 657ms 1000000 items

We can see significant optimization: custom binary serialization reduces packet size by about 100 MB, serialization is ~2s faster, deserialization ~1s faster.

3.3. BinaryWriter / BinaryReader

The BinaryWriter and BinaryReader classes are used to write and read data in binary format. They provide methods to write and read various primitive data types (int, float, double, string, etc.) in binary representation. These classes are typically used with FileStream but can also be used with other streams like MemoryStream.

The custom approach is fully manual, requiring byte array copying and conversions—somewhat primitive. Using BinaryWriter/BinaryReader for serialization is the standard approach for binary serialization.

Create BinarySerializeHelper implementing ISerializeHelper:

public class BinarySerializeHelper : ISerializeHelper
{
    public byte[] Serialize(Organization data)
    {
        using var stream = new MemoryStream();
        using var writer = new BinaryWriter(stream, ByteHelper.DefaultEncoding);
        writer.Write(data.Id);
        Write(writer, data.Tags);
        Write(writer, data.Members);

        return stream.ToArray();
    }

    public Organization Deserialize(byte[] buffer)
    {
        var data = new Organization();
        using var stream = new MemoryStream(buffer);
        using var reader = new BinaryReader(stream, ByteHelper.DefaultEncoding);
        data.Id = reader.ReadInt32();
        data.Tags = ReadStringList(reader);
        data.Members = ReadPeopleList(reader);

        return data;
    }

    private static void Write(BinaryWriter writer, string[]? data)
    {
        var count = data?.Length ?? 0;
        writer.Write(count);
        if (count <= 0)
        {
            return;
        }

        foreach (var item in data!)
        {
            writer.Write(item);
        }
    }

    private static void Write(BinaryWriter writer, List<Member>? data)
    {
        var count = data?.Count ?? 0;
        writer.Write(count);
        if (count > 0)
        {
            foreach (var item in data)
            {
                writer.Write(item.Id);
                writer.Write(item.Name ?? string.Empty);
                writer.Write(item.Description ?? string.Empty);
                writer.Write(item.Address ?? string.Empty);
                writer.Write(item.Value);
                writer.Write(item.UpdateTime);
            }
        }
    }

    private static string[]? ReadStringList(BinaryReader reader)
    {
        var count = reader.ReadInt32();
        if (count <= 0)
        {
            return default;
        }

        var values = new string[count];
        for (int i = 0; i < count; i++)
        {
            values[i] = reader.ReadString();
        }

        return values;
    }

    private static List<Member>? ReadPeopleList(BinaryReader reader)
    {
        var count = reader.ReadInt32();
        if (count <= 0)
        {
            return default;
        }

        var values = new List<Member>();
        for (int i = 0; i < count; i++)
        {
            var item = new Member();
            item.Id = reader.ReadInt32();
            item.Name = reader.ReadString();
            item.Description = reader.ReadString();
            item.Address = reader.ReadString();
            item.Value = reader.ReadDouble();
            item.UpdateTime = reader.ReadInt64();

            values.Add(item);
        }

        return values;
    }
}

There is still quite a bit of code. When dealing with custom binary serialization earlier, we could have encapsulated list serialization/deserialization into methods using reflection for generality. Same for this section, but we won't revisit that now. Let's add BinarySerializeHelper to the Test method and run the program:

2023-12-10 22:52:56 986: JsonSerializeHelper Serialize 2715ms 196225584byte
2023-12-10 22:52:58 910: JsonSerializeHelper Deserialize 1910ms 1000000 items
2023-12-10 22:52:59 730: CustomSerializeHelper Serialize 819ms 92853722byte
2023-12-10 22:53:00 389: CustomSerializeHelper Deserialize 659ms 1000000 items
2023-12-10 22:53:00 660: BinarySerializeHelper Serialize 269ms 83853707byte
2023-12-10 22:53:01 466: BinarySerializeHelper Deserialize 806ms 1000000 items

BinaryWriter/BinaryReader reduces packet size by about 8.5 MB compared to custom binary serialization, serialization is ~0.5s faster, and deserialization is slightly slower. Not bad.

3.4. ProtoBuf

Have you heard of or used Google's Protocol Buffers?

This section introduces the protobuf-net library, which provides support for Google's Protocol Buffers data serialization format in .NET. Protocol Buffers is a lightweight, efficient structured data serialization mechanism, commonly used for cross-service or application communication and data storage.

Install the protobuf-net package:

<PackageReference Include="protobuf-net" Version="3.2.30" />

Add class serialization attribute [ProtoContract] and property serialization attributes [ProtoMember(serialization order)] to the test classes:

[ProtoContract]
public class Organization
{
    [ProtoMember(1)] public int Id { get; set; }

    [ProtoMember(2)] public string[]? Tags { get; set; }

    [ProtoMember(3)] public List<Member>? Members { get; set; }
}

[ProtoContract]
public class Member
{
    [ProtoMember(1)] public int Id { get; set; }

    [ProtoMember(2)] public string? Name { get; set; }

    [ProtoMember(3)] public string? Description { get; set; }

    [ProtoMember(4)] public string? Address { get; set; }

    [ProtoMember(5)] public double Value { get; set; }

    [ProtoMember(6)] public long UpdateTime { get; set; }
}

Add the ProtoBufSerializeHelper class implementing ISerializeHelper:

using ByteTest.Core.Models;
using ProtoBuf;

namespace ByteTest.Core.Helpers;

public class ProtoBufSerializeHelper : ISerializeHelper
{
    public byte[] Serialize(Organization data)
    {
        using var stream = new MemoryStream();
        Serializer.Serialize(stream, data);
        return stream.ToArray();
    }

    public Organization? Deserialize(byte[] buffer)
    {
        using var stream = new MemoryStream(buffer);
        return Serializer.Deserialize<Organization>(stream);
    }
}

One more step: add ProtoBufSerializeHelper to the Test method. Program output:

2023-12-10 23:01:17 478: JsonSerializeHelper Serialize 2767ms 196225803byte
2023-12-10 23:01:19 556: JsonSerializeHelper Deserialize 2064ms 1000000 items
2023-12-10 23:01:20 350: CustomSerializeHelper Serialize 793ms 92853782byte
2023-12-10 23:01:21 012: CustomSerializeHelper Deserialize 662ms 1000000 items
2023-12-10 23:01:21 271: BinarySerializeHelper Serialize 258ms 83853767byte
2023-12-10 23:01:22 086: BinarySerializeHelper Deserialize 815ms 1000000 items
2023-12-10 23:01:22 629: ProtoBufSerializeHelper Serialize 542ms 88837248byte
2023-12-10 23:01:23 688: ProtoBufSerializeHelper Deserialize 1058ms 1000000 items

Hmm, ProtoBuf produces a larger packet than BinaryWriter and is slower. Perhaps I'm using it incorrectly? Maybe a compression algorithm is needed. We'll leave it at that for now, and move to the last one, MessagePack. If you have any usage tips, please point them out.

3.5. MessagePack

Finally, introduce MessagePack, an efficient binary serialization format that allows fast and compact data transfer between different systems. It is similar to JSON but smaller, faster, and more space-efficient.

Install the MessagePack package:

<PackageReference Include="MessagePack" Version="2.6.100-alpha" />

Add the MessagePackSerializeHelper class implementing ISerializeHelper:

using ByteTest.Core.Models;
using MessagePack;

namespace ByteTest.Core.Helpers;

public class MessagePackSerializeHelper : ISerializeHelper
{
    // This approach requires attributes on classes and fields, slightly cumbersome, but with compression options gives smaller packet size and faster serialization/deserialization.
    //readonly MessagePackSerializerOptions _options = MessagePackSerializerOptions.Standard.WithCompression(MessagePackCompression.Lz4BlockArray);

    // This approach does not require attributes on the transfer objects but can still add compression options.
    readonly MessagePackSerializerOptions _options =
        MessagePack.Resolvers.ContractlessStandardResolver.Options.WithCompression(MessagePackCompression
            .Lz4BlockArray);

    public byte[] Serialize(Organization data)
    {
        return MessagePackSerializer.Serialize(data, _options);
    }

    public Organization? Deserialize(byte[] buffer)
    {
        return MessagePackSerializer.Deserialize<Organization>(buffer, _options);
    }
}

As seen in the commented code, the provided options may be more optimal, yielding smaller compressed size. We'll add tests later.

Finally, modify the Test method:

   public static void Test(List<ISerializeHelper>? moreHelpers = null)
    {
        var serializeHelpers = new List<ISerializeHelper>
        {
            new JsonSerializeHelper(),
            new CustomSerializeHelper(),
            new BinarySerializeHelper(),
            new ProtoBufSerializeHelper(),
            new MessagePackSerializeHelper(),
        };
        if (moreHelpers?.Count() > 0)
        {
            serializeHelpers.AddRange(moreHelpers);
        }

        serializeHelpers.ForEach(RunSerialize);
    }

Run the program, output:

2023-12-11 21:34:47 782: JsonSerializeHelper Serialize 2456ms 196225500byte
2023-12-11 21:34:51 215: JsonSerializeHelper Deserialize 3430ms 1000000 items
2023-12-11 21:34:52 186: CustomSerializeHelper Serialize 970ms 92853911byte
2023-12-11 21:34:52 711: CustomSerializeHelper Deserialize 526ms 1000000 items
2023-12-11 21:34:53 734: BinarySerializeHelper Serialize 1022ms 83853896byte
2023-12-11 21:34:54 354: BinarySerializeHelper Deserialize 620ms 1000000 items
2023-12-11 21:34:55 170: ProtoBufSerializeHelper Serialize 815ms 88837377byte
2023-12-11 21:34:56 205: ProtoBufSerializeHelper Deserialize 1035ms 1000000 items
2023-12-11 21:34:57 123: MessagePackSerializeHelper Serialize 917ms 43583878byte
2023-12-11 21:34:58 527: MessagePackSerializeHelper Deserialize 1403ms 1000000 items

Now switch to the commented option. Modify the code as follows:

using ByteTest.Core.Models;
using MessagePack;

namespace ByteTest.Core.Helpers;

public class MessagePackSerializeHelper : ISerializeHelper
{
    // This approach requires attributes on classes and fields, slightly cumbersome, but with compression gives smaller packet size and faster serialization/deserialization.
    readonly MessagePackSerializerOptions _options = MessagePackSerializerOptions.Standard.WithCompression(MessagePackCompression.Lz4BlockArray);

    // This approach does not require attributes on transfer objects but can also add compression options.
    //readonly MessagePackSerializerOptions _options =
    //    MessagePack.Resolvers.ContractlessStandardResolver.Options.WithCompression(MessagePackCompression
    //        .Lz4BlockArray);

    public byte[] Serialize(Organization data)
    {
        return MessagePackSerializer.Serialize(data, _options);
    }

    public Organization? Deserialize(byte[] buffer)
    {
        return MessagePackSerializer.Deserialize<Organization>(buffer, _options);
    }
}

And add attributes [MessagePackObject] to the classes and [Key(serialization index)] to the properties that need serialization:

using MessagePack;
using ProtoBuf;

namespace ByteTest.Core.Models;

[ProtoContract]
[MessagePackObject]
public class Organization
{
    [ProtoMember(1)] [Key(0)] public int Id { get; set; }

    [ProtoMember(2)] [Key(1)] public string[]? Tags { get; set; }

    [ProtoMember(3)] [Key(2)] public List<Member>? Members { get; set; }
}

[ProtoContract]
[MessagePackObject]
public class Member
{
    [ProtoMember(1)] [Key(0)] public int Id { get; set; }

    [ProtoMember(2)] [Key(1)] public string? Name { get; set; }

    [ProtoMember(3)] [Key(2)] public string? Description { get; set; }

    [ProtoMember(4)] [Key(3)] public string? Address { get; set; }

    [ProtoMember(5)] [Key(4)] public double Value { get; set; }

    [ProtoMember(6)] [Key(5)] public long UpdateTime { get; set; }
}

Program output:

2023-12-11 21:49:34 153: JsonSerializeHelper Serialize 2383ms 196226140byte
2023-12-11 21:49:37 736: JsonSerializeHelper Deserialize 3581ms 1000000 items
2023-12-11 21:49:38 720: CustomSerializeHelper Serialize 983ms 92854251byte
2023-12-11 21:49:39 250: CustomSerializeHelper Deserialize 530ms 1000000 items
2023-12-11 21:49:40 273: BinarySerializeHelper Serialize 1023ms 83854236byte
2023-12-11 21:49:40 907: BinarySerializeHelper Deserialize 632ms 1000000 items
2023-12-11 21:49:41 660: ProtoBufSerializeHelper Serialize 754ms 88837717byte
2023-12-11 21:49:42 676: ProtoBufSerializeHelper Deserialize 1014ms 1000000 items
2023-12-11 21:49:43 357: MessagePackSerializeHelper Serialize 681ms 38706475byte
2023-12-11 21:49:44 344: MessagePackSerializeHelper Deserialize 986ms 1000000 items

For comparison, here's the output from the previous MessagePack option:

2023-12-11 21:34:57 123: MessagePackSerializeHelper Serialize 917ms 43583878byte
2023-12-11 21:34:58 527: MessagePackSerializeHelper Deserialize 1403ms 1000000 items

Clearly, the first option (with attributes) yields better packet size and speed.

Approach Analysis

With 1 million test data items, the statistics for the 5 serialization methods are shown in the table:

From the table, after serialization, MessagePack has the smallest packet at 36.91 MB, while JSON is the largest at 187.13 MB, and the other three are around 80 MB. In terms of serialization efficiency, MessagePack is best, but interestingly, my custom binary approach is fastest for deserialization. Let's remove JSON and run again:

2023-12-11 21:55:47 813: CustomSerializeHelper Serialize 1263ms 92854890byte
2023-12-11 21:55:48 804: CustomSerializeHelper Deserialize 989ms 1000000 items
2023-12-11 21:55:49 215: BinarySerializeHelper Serialize 410ms 83854875byte
2023-12-11 21:55:50 081: BinarySerializeHelper Deserialize 866ms 1000000 items
2023-12-11 21:55:50 726: ProtoBufSerializeHelper Serialize 644ms 88838356byte
2023-12-11 21:55:51 725: ProtoBufSerializeHelper Deserialize 999ms 1000000 items
2023-12-11 21:55:52 426: MessagePackSerializeHelper Serialize 701ms 38701799byte
2023-12-11 21:55:53 427: MessagePackSerializeHelper Deserialize 999ms 1000000 items

Packet size unchanged. BinaryWriter is fastest for serialization, and BinaryReader for deserialization. However, test results can be unreliable; we should use benchmarking.

Benchmark Tests

Install BenchmarkDotNet for benchmarking:

<PackageReference Include="BenchmarkDotNet" Version="0.13.11" />

Modify the BenchmarkTest class:

using BenchmarkDotNet.Attributes;
using ByteTest.Core.Helpers;
using ByteTest.Core.Models;
using MessagePack;
using System.Diagnostics;

namespace ByteTest.Core.Test;

[MemoryDiagnoser, RankColumn]
public class BenchmarkTest
{
    // test data code omitted

    //[Benchmark]
    //public void JsonByteSerialize()
    //{
    //    RunSerialize(new JsonSerializeHelper());
    //}

    [Benchmark]
    public void CustomSerialize()
    {
        RunSerialize(new CustomSerializeHelper());
    }

    [Benchmark]
    public void BinarySerialize()
    {
        RunSerialize(new BinarySerializeHelper());
    }

    [Benchmark]
    public void ProtoBufPackSerialize()
    {
        RunSerialize(new ProtoBufSerializeHelper());
    }

    [Benchmark]
    public void MessagePackSerialize()
    {
        RunSerialize(new MessagePackSerializeHelper());
    }
	
    // statistics related code omitted
}

Modify Program.cs:

using BenchmarkDotNet.Running;
using ByteTest.Core.Test;

// Run benchmark
BenchmarkRunner.Run<BenchmarkTest>();

// Normal test
//BenchmarkTest.Test();

Benchmark results:

Roughly speaking, MessagePack wins on size, meaning fewer network fragments and less round-trip time. For packet building (serialization) and unpacking (deserialization), native BinaryWriter and BinaryReader are superior.

4. Summary

In summary, the data packet size should be set appropriately based on the network environment and device capabilities to ensure efficient data transmission. At the same time, efficient packet building and unpacking processing capabilities are crucial for maintaining network transfer performance. For the former, MessagePack with compression can be considered; for the latter, native BinaryWriter and BinaryReader are recommended.

Do you have any better suggestions? Feel free to leave a comment. You can also submit a PR to the test code at ByteTest.

References

• msgpack/msgpack: MessagePack is an extremely efficient object serialization library. It's like JSON, but very fast and small. (github.com)
• Introduction and usage of MessagePack - Jianshu (jianshu.com)