NativeBuferring，一種零分配的數(shù)據(jù)類型[下篇]

上文說(shuō)到Unmanaged、BufferedBinary和BufferedString是NativeBuffering支持的三個(gè)基本數(shù)據(jù)類型，其實(shí)我們也可以說(shuō)NativeBuffering只支持Unmanaged和IReadOnlyBufferedObject<T>兩種類型，BufferedString、NativeBuffering和通過(guò)Source Generator生成的BufferedMessage類型，以及下面介紹的幾種集合和字典類型，都實(shí)現(xiàn)了IReadOnlyBufferedObject<T>接口。

一、IReadOnlyBufferedObject<T>
二、集合
三、字典
四、為什么不直接返回接口?

一、IReadOnlyBufferedObject<T>

顧名思義，IReadOnlyBufferedObject<T>表示一個(gè)針對(duì)緩沖字節(jié)序列創(chuàng)建的只讀數(shù)據(jù)類型。如下面的代碼片段所示，該接口只定義了一個(gè)名為Parse的靜態(tài)方法，意味著對(duì)于任何一個(gè)實(shí)現(xiàn)了該接口的類型，對(duì)應(yīng)的實(shí)例都可以利用一個(gè)代表緩沖字節(jié)序列的NativeBuffer的對(duì)象進(jìn)行創(chuàng)建。

public interface IReadOnlyBufferedObject<T> where T: IReadOnlyBufferedObject<T>
{
    static abstract T Parse(NativeBuffer buffer);
}

public unsafe readonly struct NativeBuffer
{
    public byte[] Bytes { get; }
    public void* Start { get; }

    public NativeBuffer(byte[] bytes, void* start)
    {
        Bytes = bytes ?? throw new ArgumentNullException(nameof(bytes));
        Start = start;
    }

    public NativeBuffer(byte[] bytes, int index = 0)
    {
        Bytes = bytes ?? throw new ArgumentNullException(nameof(bytes));
        Start = Unsafe.AsPointer(ref bytes[index]);
    }
}

由于IReadOnlyBufferedObject<T>是NativeBuffering支持的基礎(chǔ)類型，而生成的BufferedMessage類型也實(shí)現(xiàn)了這個(gè)接口。通過(guò)這種“無(wú)限嵌套”的形式，我們可以定義一個(gè)具有任意結(jié)構(gòu)的數(shù)據(jù)類型。比如我們具有如下這個(gè)表示聯(lián)系人的Contact類型，我們需要利用它作為“源類型”生成對(duì)應(yīng)BufferedMessage類型。

[BufferedMessageSource]
public partial class Contact
{
    public Contact(string id, string name, Address address)
    {
        Id = id;
        Name = name;
        ShipAddress = address;
    }

    public string Id { get; }
    public string Name { get; }
    public Address ShipAddress { get; }
}

[BufferedMessageSource]
public partial class Address
{
    public string Province { get; }
    public string City { get; }
    public string District { get; }
    public string Street { get; }
    public Address(string province, string city, string district, string street)
    {
        Province = province ?? throw new ArgumentNullException(nameof(province));
        City = city ?? throw new ArgumentNullException(nameof(city));
        District = district ?? throw new ArgumentNullException(nameof(district));
        Street = street ?? throw new ArgumentNullException(nameof(street));
    }
}

Contact具有Id、Name和ShipAddress 三個(gè)數(shù)據(jù)成員，ShipAddress 對(duì)應(yīng)的Address又是一個(gè)復(fù)合類型，具有四個(gè)表示省、市、區(qū)和介紹的字符串類型成員。現(xiàn)在我們?yōu)镃ontact和Address這兩個(gè)類型生成對(duì)應(yīng)的ContactBufferedMessage和AddressBufferedMessage。

public unsafe readonly struct ContactBufferedMessage : IReadOnlyBufferedObject<ContactBufferedMessage>
{
    public NativeBuffer Buffer { get; }
    public ContactBufferedMessage(NativeBuffer buffer) => Buffer = buffer;
    public static ContactBufferedMessage Parse(NativeBuffer buffer) => new ContactBufferedMessage(buffer);
    public BufferedString Id => Buffer.ReadBufferedObjectField<BufferedString>(0);
    public BufferedString Name => Buffer.ReadBufferedObjectField<BufferedString>(1);
    public AddressBufferedMessage ShipAddress => Buffer.ReadBufferedObjectField<AddressBufferedMessage>(2);
}

public unsafe readonly struct AddressBufferedMessage : IReadOnlyBufferedObject<AddressBufferedMessage>
{
    public NativeBuffer Buffer { get; }
    public AddressBufferedMessage(NativeBuffer buffer) => Buffer = buffer;
    public static AddressBufferedMessage Parse(NativeBuffer buffer) => new AddressBufferedMessage(buffer);
    public BufferedString Province => Buffer.ReadBufferedObjectField<BufferedString>(0);
    public BufferedString City => Buffer.ReadBufferedObjectField<BufferedString>(1);
    public BufferedString District => Buffer.ReadBufferedObjectField<BufferedString>(2);
    public BufferedString Street => Buffer.ReadBufferedObjectField<BufferedString>(3);
}

如下的程序演示了如何將一個(gè)Contact對(duì)象轉(zhuǎn)換成字節(jié)數(shù)組，然后利用這這段字節(jié)序列生成一個(gè)ContactBufferedMessage對(duì)象。給出的調(diào)試斷言驗(yàn)證了Contact和ContactBufferedMessage對(duì)象承載了一樣的數(shù)據(jù)，fixed關(guān)鍵字是為了將字節(jié)數(shù)組“固定住”。（源代碼從這里下載）

using NativeBuffering;
using System.Diagnostics;

var address = new Address("Jiangsu", "Suzhou", "Industory Park", "#328, Xinghu St");
var contact = new Contact("123456789", "John Doe", address);
var size = contact.CalculateSize();
var bytes = new byte[size];
var context = new BufferedObjectWriteContext(bytes);
contact.Write(context);

unsafe
{
    fixed (byte* _ = bytes)
    {
        var contactMessage = ContactBufferedMessage.Parse(new NativeBuffer(bytes));
        Debug.Assert(contactMessage.Id == "123456789");
        Debug.Assert(contactMessage.Name == "John Doe");
        Debug.Assert(contactMessage.ShipAddress.Province == "Jiangsu");
        Debug.Assert(contactMessage.ShipAddress.City == "Suzhou");
        Debug.Assert(contactMessage.ShipAddress.District == "Industory Park");
        Debug.Assert(contactMessage.ShipAddress.Street == "#328, Xinghu St");
    }
}

二、集合

NativeBuffering同樣支持集合。由于Unmanaged和IReadOnlyBufferedObject<T>是兩種基本的數(shù)據(jù)類型，它們的根據(jù)區(qū)別在于：前者的長(zhǎng)度有類型本身決定，是固定長(zhǎng)度類型，后者則是可變長(zhǎng)度類型。元素類型為Unmanaged和IReadOnlyBufferedObject<T>的集合分別通過(guò)ReadOnlyFixedLengthTypedList<T>和ReadOnlyVaraibleLengthTypedList<T>類型（結(jié)構(gòu)體）表示，它們同樣實(shí)現(xiàn)了IReadOnlyBufferedObject<T>接口。ReadOnlyFixedLengthTypedList<T>采用如下的字節(jié)布局：集合元素?cái)?shù)量（4字節(jié)整數(shù)）+所有元素的字節(jié)內(nèi)容（下圖-上）。對(duì)于ReadOnlyVaraibleLengthTypedList<T>類型，我們會(huì)在前面為每個(gè)元素添加一個(gè)索引（4字節(jié)的整數(shù)），該索引指向目標(biāo)元素在整個(gè)緩沖區(qū)的偏移量（下圖-下）。

以如下所示的Entity為例，它具有兩個(gè)數(shù)組類型的屬性成員Collection1和Collection2，數(shù)組元素類型分別為Foobar和double，它們分別代表了上述的兩種集合類型。

[BufferedMessageSource]
public partial class Entity
{
    public Foobar[] Collection1 { get; }
    public double[] Collection2 { get; }
    public Entity(Foobar[] collection1, double[] collection2)
    {
        Collection1 = collection1;
        Collection2 = collection2;
    }
}

[BufferedMessageSource]
public partial class Foobar
{
    public int Foo { get; }
    public string Bar { get; }
    public Foobar(int foo, string bar)
    {
        Foo = foo;
        Bar = bar;
    }
}

NativeBuffering.Generator會(huì)將作為“源類型”的Entity和Foobar類型的生成對(duì)應(yīng)的BufferedMessage類型（EntityBufferredMessage和FoobarBufferedMessage）。從EntityBufferredMessage類型的定義可以看出，兩個(gè)集合屬性的分別是ReadOnlyVariableLengthTypeList<FoobarBufferedMessage>和ReadOnlyFixedLengthTypedList<double>。

public unsafe readonly struct EntityBufferedMessage : IReadOnlyBufferedObject<EntityBufferedMessage>
{
    public NativeBuffer Buffer { get; }
    public EntityBufferedMessage(NativeBuffer buffer) => Buffer = buffer;
    public static EntityBufferedMessage Parse(NativeBuffer buffer) => new EntityBufferedMessage(buffer);
    public ReadOnlyVariableLengthTypeList<FoobarBufferedMessage> Collection1 => Buffer.ReadBufferedObjectCollectionField<FoobarBufferedMessage>(0);
    public ReadOnlyFixedLengthTypedList<System.Double> Collection2 => Buffer.ReadUnmanagedCollectionField<System.Double>(1);
}

public unsafe readonly struct FoobarBufferedMessage : IReadOnlyBufferedObject<FoobarBufferedMessage>
{
    public NativeBuffer Buffer { get; }
    public FoobarBufferedMessage(NativeBuffer buffer) => Buffer = buffer;
    public static FoobarBufferedMessage Parse(NativeBuffer buffer) => new FoobarBufferedMessage(buffer);
    public System.Int32 Foo => Buffer.ReadUnmanagedField<System.Int32>(0);
    public BufferedString Bar => Buffer.ReadBufferedObjectField<BufferedString>(1);
}

兩個(gè)集合類型都實(shí)現(xiàn)了IEnumerable<T>接口，還提供了索引。下面的代碼演示了以索引的形式提取集合元素（源代碼從這里下載）。

using NativeBuffering;
using System.Diagnostics;

var entity = new Entity(
    collection1: new Foobar[] { new Foobar(1, "foo"), new Foobar(2, "bar") },
    collection2: new double[] { 1.1, 2.2 });
var bytes = new byte[entity.CalculateSize()];
var context = new BufferedObjectWriteContext(bytes);
entity.Write(context);

unsafe
{
    fixed (byte* p = bytes)
    {
        var entityMessage = EntityBufferedMessage.Parse(new NativeBuffer(bytes));
        var foobar = entityMessage.Collection1[0];
        Debug.Assert(foobar.Foo == 1);
        Debug.Assert(foobar.Bar == "foo");

        foobar = entityMessage.Collection1[1];
        Debug.Assert(foobar.Foo == 2);
        Debug.Assert(foobar.Bar == "bar");

        Debug.Assert(entityMessage.Collection2[0] == 1.1);
        Debug.Assert(entityMessage.Collection2[1] == 2.2);
    }
}

三、字典

從數(shù)據(jù)的存儲(chǔ)來(lái)看，字典就是鍵值對(duì)的集合，所以我們采用與集合一致的存儲(chǔ)形式。NativeBuffering對(duì)集合的Key作了限制，要求其類型只能是Unmanaged和字符串（String/BufferredString）。按照Key和Value的類型組合，我們一共定義了四種類型的字典類型，它們分別是：

ReadOnlyUnmanagedUnmanagedDictionary<TKey, TValue>：Key=Unmanaged; Value = Unmanaged
ReadOnlyUnmanagedBufferedObjectDictionary<TKey, TValue>：Key=Unmanaged; Value = IReadOnlyBufferedObject<TValue>
ReadOnlyStringUnmanagedDictionary<TValue>：Key=String/BufferredString; Value = Unmanaged
ReadOnlyStringBufferedObjectDictionary<TValue>：Key=String/BufferredString; Value = IReadOnlyBufferedObject<TValue>

如果Key和Value的類型都是Unmanaged，鍵值對(duì)就是定長(zhǎng)類型，所以我們會(huì)采用類似于ReadOnlyFixedLengthTypedList<T>的字節(jié)布局方式（下圖-上），至于其他三種字典類型，則采用類似于ReadOnlyVaraibleLengthTypedList<T>的字節(jié)布局形式（下圖-下）。

但是這僅僅解決了字段數(shù)據(jù)存儲(chǔ)的問(wèn)題，字典基于哈希檢索定位的功能是沒(méi)有辦法實(shí)現(xiàn)的。這里我們不得不作出妥協(xié)，四種字典的索引均不能提供時(shí)間復(fù)雜度O(1)的哈希檢索方式。為了在現(xiàn)有的數(shù)據(jù)結(jié)構(gòu)上使針對(duì)Key的查找盡可能高效，在生成字節(jié)內(nèi)容之前，我們會(huì)按照Key對(duì)鍵值對(duì)進(jìn)行排序，這樣我們至少可以采用二分法的形式進(jìn)行檢索，所以四種類型的字典的索引在根據(jù)指定的Key查找對(duì)應(yīng)Value，對(duì)應(yīng)的時(shí)間復(fù)雜度為Log(N)。如果字典包含的元素比較多，這樣的查找方式不能滿足我們的需求，我們可以I將它們轉(zhuǎn)換成普通的Dictionary<TKey, TValue>類型，但是這就沒(méi)法避免內(nèi)存分配了。

我們照例編寫一個(gè)簡(jiǎn)答的程序來(lái)演示針對(duì)字典的使用。我們定義了如下這個(gè)Entity作為“源類型”，它的四個(gè)屬性對(duì)應(yīng)的字典類型剛好對(duì)應(yīng)上述四種鍵值對(duì)的組合。從生成的EntityBufferedMessage類型可以看出，四個(gè)成員的類型正好對(duì)應(yīng)上述的四種字典類型。

[BufferedMessageSource]
public partial class Entity
{
    public Dictionary<int, long> Dictionary1 { get; set; }
    public Dictionary<int, string> Dictionary2 { get; set; }
    public Dictionary<string, long> Dictionary3 { get; set; }
    public Dictionary<string, string> Dictionary4 { get; set; }
}

public unsafe readonly struct EntityBufferedMessage : IReadOnlyBufferedObject<EntityBufferedMessage>
{
    public NativeBuffer Buffer { get; }
    public EntityBufferedMessage(NativeBuffer buffer) => Buffer = buffer;
    public static EntityBufferedMessage Parse(NativeBuffer buffer) => new EntityBufferedMessage(buffer);
    public ReadOnlyUnmanagedUnmanagedDictionary<System.Int32, System.Int64> Dictionary1 => Buffer.ReadUnmanagedUnmanagedDictionaryField<System.Int32, System.Int64>(0);
    public ReadOnlyUnmanagedBufferedObjectDictionary<System.Int32, BufferedString> Dictionary2 => Buffer.ReadUnmanagedBufferedObjectDictionaryField<System.Int32, BufferedString>(1);
    public ReadOnlyStringUnmanagedDictionary<System.Int64> Dictionary3 => Buffer.ReadStringUnmanagedDictionaryField<System.Int64>(2);
    public ReadOnlyStringBufferedObjectDictionary<BufferedString> Dictionary4 => Buffer.ReadStringBufferedObjectDictionaryField<BufferedString>(3);
}

如下的代碼演示了基于四種字典類型基于“索引”的檢索方式（源代碼從這里下載）。

using NativeBuffering;
using System.Diagnostics;

var entity = new Entity
{
    Dictionary1 = new Dictionary<int, long> { { 1, 1 }, { 2, 2 }, { 3, 3 } },
    Dictionary2 = new Dictionary<int, string> { { 1, "foo" }, { 2, "bar" }, { 3, "baz" } },
    Dictionary3 = new Dictionary<string, long> { { "foo", 1 }, { "bar", 2 }, { "baz", 3 } },
    Dictionary4 = new Dictionary<string, string> { { "a", "foo" }, { "b", "bar" }, { "c", "baz" } }
};

var bytes = new byte[entity.CalculateSize()];
var context = new BufferedObjectWriteContext(bytes);
entity.Write(context);
unsafe
{
    fixed (void* _ = bytes)
    {
        var bufferedMessage = EntityBufferedMessage.Parse(new NativeBuffer(bytes));

        ref var value1 = ref bufferedMessage.Dictionary1.AsRef(1);
        Debug.Assert(value1 == 1);
        ref var value2 = ref bufferedMessage.Dictionary3.AsRef("baz");
        Debug.Assert(value2 == 3);

        var dictionary1 = bufferedMessage.Dictionary1;
        Debug.Assert(dictionary1[1] == 1);
        Debug.Assert(dictionary1[2] == 2);
        Debug.Assert(dictionary1[3] == 3);

        var dictionary2 = bufferedMessage.Dictionary2;
        Debug.Assert(dictionary2[1] == "foo");
        Debug.Assert(dictionary2[2] == "bar");
        Debug.Assert(dictionary2[3] == "baz");

        var dictionary3 = bufferedMessage.Dictionary3;
        Debug.Assert(dictionary3["foo"] == 1);
        Debug.Assert(dictionary3["bar"] == 2);
        Debug.Assert(dictionary3["baz"] == 3);

        var dictionary4 = bufferedMessage.Dictionary4;
        Debug.Assert(dictionary4["a"] == "foo");
        Debug.Assert(dictionary4["b"] == "bar");
        Debug.Assert(dictionary4["c"] == "baz");
    }
}

四、為什么不直接返回接口

針對(duì)集合，NativeBuffering提供了兩種類型；針對(duì)字典，更是定義了四種類型，為什么不直接返回IList<T>/IDictionary<TKey,TValue>（或者IReadOnlyList<T>/IReadOnlyDictionary<TKey,TValue>）接口呢？這主要有兩個(gè)原因，第一：為了盡可能地減少內(nèi)存占用，我們將四種字典類型都定義成了結(jié)構(gòu)體，如果使用接口的話會(huì)導(dǎo)致裝箱；第二，四種字典類型的提供的API是有差異的，比如ReadOnlyFixedLengthTypedList<T> 和ReadOnlyUnmanagedUnmanagedDictionary<TKey, TValue>都提供了一個(gè)額外的AsRef方法，它直接返回值的引用（只讀）。如果這個(gè)值被定義成一個(gè)成員較多的結(jié)構(gòu)體，傳引用的方式可以避免較多的拷貝。

public readonly unsafe struct ReadOnlyFixedLengthTypedList<T> : IReadOnlyList<T>, IReadOnlyBufferedObject<ReadOnlyFixedLengthTypedList<T>>
    where T: unmanaged
{
    public readonly ref T AsRef(int index);
    ...
}

public unsafe readonly struct ReadOnlyUnmanagedUnmanagedDictionary<TKey, TValue> : IReadOnlyDictionary<TKey, TValue>, IReadOnlyBufferedObject<ReadOnlyUnmanagedUnmanagedDictionary<TKey, TValue>>
    where TKey : unmanaged, IComparable<TKey>
    where TValue : unmanaged
{
    public readonly ref TValue AsRef(TKey index) ;
    ...
}

posted @ 2023-08-01 08:30 Artech 閱讀(1275) 評(píng)論(4) 收藏舉報(bào)

刷新頁(yè)面返回頂部

NativeBuferring，一種零分配的數(shù)據(jù)類型[下篇]

一、IReadOnlyBufferedObject<T>

二、集合

三、字典

四、為什么不直接返回接口

公告