.Net: Switch MEVD MongoDB Driver to v3.51

[damieng]

Motivation and Context

Switches the MongoDB Driver to the latest 3.5 release which had a number of breaking changes.

This affects both the MongoDB vector data project and the CosmosMongoDB project.

This fixes #11652 and likely addresses #12707 and partly #10291.

Description

Switches the MongoDB driver to 3.5. Part of the breaking changes is that GUIDs in BSON no longer have a default storage format specified due to the need to switch from the C#-only format to the cross-MongoDB-driver standard format. Setting this is achieved in this PR by way of both a convention for the registry based mode and an alternative to BsonValue.Create in the scenarios where we don't have access to conventions/serialization such as key creation and using the dynamic mapper.

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
• All unit tests pass, and I have added new tests where possible
• I didn't break anyone 😄

cc @roji

[roji]

Great to see this @damieng, thanks. All looks good, see mainly the question about netstandard2.1 and some nits.

[roji]

LGTM, pinged others for a 2nd review necessary for merging.

[westey-m]

A number of the conformance tests are failing for me, all with a similar error, e.g.
MongoRecordConformanceTests_Guid.GetAsync_WithVectors

Note that these tests have be enabled manually to be run by commenting out the following line in assembly info.
[assembly: DisableTests(Skip = "The MongoDB container is intermittently timing out at startup time blocking prs, so these test should be run manually.")]

Error:

Message: 
MongoDB.Bson.BsonSerializationException : An error occurred while serializing the Id property of class VectorData.ConformanceTests.Models.SimpleRecord`1[[System.Guid, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]: GuidSerializer cannot serialize a Guid when GuidRepresentation is Unspecified.
---- MongoDB.Bson.BsonSerializationException : GuidSerializer cannot serialize a Guid when GuidRepresentation is Unspecified.

Stack Trace: 
BsonClassMapSerializer1.SerializeMember(BsonSerializationContext context, Object obj, BsonMemberMap memberMap) BsonClassMapSerializer1.SerializeClass(BsonSerializationContext context, BsonSerializationArgs args, TClass document)
BsonClassMapSerializer1.Serialize(BsonSerializationContext context, BsonSerializationArgs args, TClass value) BsonExtensionMethods.ToBsonDocument(Object obj, Type nominalType, IBsonSerializer serializer, Action1 configurator, BsonSerializationArgs args)
BsonExtensionMethods.ToBsonDocument[TNominalType](TNominalType obj, IBsonSerializer1 serializer, Action1 configurator, BsonSerializationArgs args)
MongoMapper1.MapFromDataToStorageModel(TRecord dataModel, Int32 recordIndex, IReadOnlyList1[] generatedEmbeddings) line 53
MongoCollection2.UpsertCoreAsync(TRecord record, Int32 recordIndex, IReadOnlyList1[] generatedEmbeddings, CancellationToken cancellationToken) line 270
MongoCollection2.UpsertAsync(IEnumerable1 records, CancellationToken cancellationToken) line 261
VectorStoreCollectionFixture2.SeedAsync() line 52 VectorStoreCollectionFixture2.InitializeAsync() line 41
----- Inner Stack Trace -----
GuidSerializer.Serialize(BsonSerializationContext context, BsonSerializationArgs args, Guid value)
IBsonSerializer.Serialize(BsonSerializationContext context, BsonSerializationArgs args, Object value)
IBsonSerializerExtensions.Serialize(IBsonSerializer serializer, BsonSerializationContext context, Object value)
BsonClassMapSerializer1.SerializeNormalMember(BsonSerializationContext context, Object obj, BsonMemberMap memberMap) BsonClassMapSerializer1.SerializeMember(BsonSerializationContext context, Object obj, BsonMemberMap memberMap)

[damieng]

The changes made to MongoCollection would need to be applied to CosmosCollection. I don't have authorisation to work on that at my end nor access to a Cosmos instance/emulator to test it with. Sorry.

[westey-m]

@damieng, these failures are also when testing MongoCollection using e.g. MongoRecordConformanceTests.

[damieng]

I just ran the tests again and saw four types of failure. The first one was caused by the .NET 10/C# 14 work that got merged after I created my PR. The others are not related to this PR but I have included a fix for the second.

1. A single test with that Guid failure - An array of guids is now coming through on as readonlyspan. I've included a fix for this that now supports all enumerable types not just array.
2. The index service not available - I've includes a new waiter that will ensure that testcontainers waits for the index service.
3. Some tests failing as data isn't ready - this has been a problem for a while and I don't have a solution but it's not caused by this PR.
4. Some Contains calls failing - again C# 14 changes that I think need to be fixed independent of the mongodb provider.

[roji]

For the Contains failures (point 4), something odd happened, as these tests were passing until very recently (likely related to #13395. In any case, submitted #13407 to fix those - that's indeed unrelated to this PR.

[roji]

Pushed a commit to this PR which takes care of remaining Cosmos Mongo issues; I confirmed that after this commit, the only conformance tests still failing are the Contains tests fixed by #13407.

@damieng with this PR, I'm unable to run tests for the (regular, non-Cosmos) MongoDB provider, whereas I'm able to on main (see two run logs below) - the container setup times out after waiting for a minute. It looks like this is because of these changes in this PR.

Failed test run (after this PR)

➜  MongoDB.ConformanceTests git:(switch-mongodb-to-3-5) ✗ dotnet test -f net10.0 --filter MongoDB.ConformanceTests.MongoIndexKindTests
  VectorData.Abstractions net10.0 succeeded (0.1s) → /Users/roji/projects/semantic-kernel/dotnet/src/VectorData/VectorData.Abstractions/bin/Debug/net10.0/Microsoft.Extensions.VectorData.Abstractions.dll
  VectorData.ConformanceTests net10.0 succeeded (0.1s) → /Users/roji/projects/semantic-kernel/dotnet/test/VectorData/VectorData.ConformanceTests/bin/Debug/net10.0/VectorData.ConformanceTests.dll
  MongoDB net10.0 succeeded (0.1s) → /Users/roji/projects/semantic-kernel/dotnet/src/VectorData/MongoDB/bin/Debug/net10.0/Microsoft.SemanticKernel.Connectors.MongoDB.dll
  MongoDB.ConformanceTests net10.0 succeeded with 2 warning(s) (0.6s) → bin/Debug/net10.0/MongoDB.ConformanceTests.dll
    /Users/roji/projects/semantic-kernel/dotnet/test/VectorData/MongoDB.ConformanceTests/Support/MongoTestStore.cs(122,13): warning CA1031: Modify 'UntilAsync' to catch a more specific allowed exception type, or rethrow the exception (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca1031)
    /Users/roji/projects/semantic-kernel/dotnet/test/VectorData/MongoDB.ConformanceTests/Support/MongoTestStore.cs(131,13): warning CA1031: Modify 'UntilAsync' to catch a more specific allowed exception type, or rethrow the exception (https://learn.microsoft.com/dotnet/fundamentals/code-analysis/quality-rules/ca1031)
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v3.1.5+1b188a7b0a (64-bit .NET 10.0.0)
[xUnit.net 00:00:00.05]   Discovering: MongoDB.ConformanceTests
[xUnit.net 00:00:00.11]   Discovered:  MongoDB.ConformanceTests
[xUnit.net 00:00:00.12]   Starting:    MongoDB.ConformanceTests
[testcontainers.org 00:00:00.06] Connected to Docker:
  Host: unix:///var/run/docker.sock
  Server Version: 28.5.2
  Kernel Version: 6.12.54-linuxkit
  API Version: 1.51
  Operating System: Docker Desktop
  Total Memory: 7.65 GB
  Labels: 
    com.docker.desktop.address=unix:///Users/roji/Library/Containers/com.docker.docker/Data/docker-cli.sock
[testcontainers.org 00:00:00.12] Docker container f296ead3f331 created
[testcontainers.org 00:00:00.14] Start Docker container f296ead3f331
[testcontainers.org 00:00:00.29] Wait for Docker container f296ead3f331 to complete readiness checks
[testcontainers.org 00:00:00.30] Docker container f296ead3f331 ready
[testcontainers.org 00:00:00.32] Searching Docker registry credential in CredsStore
[testcontainers.org 00:00:00.32] Searching Docker registry credential in CredHelpers
[testcontainers.org 00:00:00.32] Searching Docker registry credential in Auths
[testcontainers.org 00:00:00.32] Searching Docker registry credential in Auths
[testcontainers.org 00:00:00.41] Docker registry credential https://index.docker.io/v1/ not found
[xUnit.net 00:01:00.28]     MongoDB.ConformanceTests.MongoIndexKindTests.Flat [FAIL]
[xUnit.net 00:01:00.28]       System.OperationCanceledException : The operation was canceled.
[xUnit.net 00:01:00.29]       ---- System.OperationCanceledException : The operation was canceled.
[xUnit.net 00:01:00.29]       Stack Trace:
[xUnit.net 00:01:00.29]            at System.IO.Pipelines.StreamPipeReader.<ReadInternalAsync>g__Core|40_0(StreamPipeReader reader, Nullable`1 minimumSize, CancellationTokenSource tokenSource, CancellationToken cancellationToken)
[xUnit.net 00:01:00.29]            at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
[xUnit.net 00:01:00.29]         /_/src/Docker.DotNet/JsonSerializer.cs(50,0): at Docker.DotNet.JsonSerializer.DeserializeAsync[T](Stream stream, CancellationToken cancellationToken)+MoveNext()
[xUnit.net 00:01:00.29]            at Docker.DotNet.JsonSerializer.DeserializeAsync[T](Stream stream, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
[xUnit.net 00:01:00.29]         /_/src/Docker.DotNet/Endpoints/StreamUtil.cs(71,0): at Docker.DotNet.Models.StreamUtil.MonitorStreamForMessagesAsync[T](Task`1 streamTask, DockerClient client, CancellationToken cancellationToken, IProgress`1 progress)
[xUnit.net 00:01:00.29]         /_/src/Docker.DotNet/Endpoints/StreamUtil.cs(71,0): at Docker.DotNet.Models.StreamUtil.MonitorStreamForMessagesAsync[T](Task`1 streamTask, DockerClient client, CancellationToken cancellationToken, IProgress`1 progress)
[xUnit.net 00:01:00.29]         /_/src/Docker.DotNet/Endpoints/StreamUtil.cs(82,0): at Docker.DotNet.Models.StreamUtil.MonitorResponseForMessagesAsync[T](Task`1 responseTask, DockerClient client, CancellationToken cancel, IProgress`1 progress)
[xUnit.net 00:01:00.29]         /_/src/Testcontainers/Clients/DockerImageOperations.cs(73,0): at DotNet.Testcontainers.Clients.DockerImageOperations.CreateAsync(IImage image, IDockerRegistryAuthenticationConfiguration dockerRegistryAuthConfig, CancellationToken ct)
[xUnit.net 00:01:00.29]         /_/src/Testcontainers/Clients/TestcontainersClient.cs(422,0): at DotNet.Testcontainers.Clients.TestcontainersClient.PullImageAsync(IImage image, CancellationToken ct)
[xUnit.net 00:01:00.29]         /_/src/Testcontainers/Clients/TestcontainersClient.cs(335,0): at DotNet.Testcontainers.Clients.TestcontainersClient.RunAsync(IContainerConfiguration configuration, CancellationToken ct)
[xUnit.net 00:01:00.29]         /_/src/Testcontainers/Containers/DockerContainer.cs(462,0): at DotNet.Testcontainers.Containers.DockerContainer.UnsafeCreateAsync(CancellationToken ct)
[xUnit.net 00:01:00.29]         /_/src/Testcontainers/Containers/DockerContainer.cs(306,0): at DotNet.Testcontainers.Containers.DockerContainer.StartAsync(CancellationToken ct)
[xUnit.net 00:01:00.29]         /Users/roji/projects/semantic-kernel/dotnet/test/VectorData/MongoDB.ConformanceTests/Support/MongoTestStore.cs(55,0): at MongoDB.ConformanceTests.Support.MongoTestStore.StartMongoDbContainerAsync()
[xUnit.net 00:01:00.29]         /Users/roji/projects/semantic-kernel/dotnet/test/VectorData/MongoDB.ConformanceTests/Support/MongoTestStore.cs(37,0): at MongoDB.ConformanceTests.Support.MongoTestStore.StartAsync()
[xUnit.net 00:01:00.29]         /Users/roji/projects/semantic-kernel/dotnet/test/VectorData/VectorData.ConformanceTests/Support/TestStore.cs(43,0): at VectorData.ConformanceTests.Support.TestStore.ReferenceCountingStartAsync()
[xUnit.net 00:01:00.29]         ----- Inner Stack Trace -----
[xUnit.net 00:01:00.29]            at System.Threading.CancellationToken.ThrowOperationCanceledException()
[xUnit.net 00:01:00.29]            at System.Threading.CancellationToken.ThrowIfCancellationRequested()
[xUnit.net 00:01:00.29]            at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
[xUnit.net 00:01:00.29]            at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
[xUnit.net 00:01:00.29]         --- End of stack trace from previous location ---
[xUnit.net 00:01:00.29]         /_/src/Docker.DotNet/Microsoft.Net.Http.Client/BufferedReadStream.cs(176,0): at Microsoft.Net.Http.Client.BufferedReadStream.ReadLineAsync(CancellationToken cancellationToken)
[xUnit.net 00:01:00.29]         /_/src/Docker.DotNet/Microsoft.Net.Http.Client/ChunkedReadStream.cs(83,0): at Microsoft.Net.Http.Client.ChunkedReadStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
[xUnit.net 00:01:00.29]            at System.IO.Pipelines.StreamPipeReader.<ReadInternalAsync>g__Core|40_0(StreamPipeReader reader, Nullable`1 minimumSize, CancellationTokenSource tokenSource, CancellationToken cancellationToken)
[xUnit.net 00:01:00.29]   Finished:    MongoDB.ConformanceTests
  MongoDB.ConformanceTests test net10.0 failed with 1 error(s) (60.8s)
    /_/src/Docker.DotNet/JsonSerializer.cs(50): error TESTERROR: 
      MongoDB.ConformanceTests.MongoIndexKindTests.Flat (1ms): Error Message: System.OperationCanceledException : The operation was canceled.
      ---- System.OperationCanceledException : The operation was canceled.
      Stack Trace:
         at System.IO.Pipelines.StreamPipeReader.<ReadInternalAsync>g__Core|40_0(StreamPipeReader reader, Nullable`1 minimumSize, CancellationTokenSource tok
      enSource, CancellationToken cancellationToken)
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.G
      etResult(Int16 token)
         at Docker.DotNet.JsonSerializer.DeserializeAsync[T](Stream stream, CancellationToken cancellationToken)+MoveNext() in /_/src/Docker.DotNet/JsonSeria
      lizer.cs:line 50
         at Docker.DotNet.JsonSerializer.DeserializeAsync[T](Stream stream, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSou
      rce<System.Boolean>.GetResult()
         at Docker.DotNet.Models.StreamUtil.MonitorStreamForMessagesAsync[T](Task`1 streamTask, DockerClient client, CancellationToken cancellationToken, IPr
      ogress`1 progress) in /_/src/Docker.DotNet/Endpoints/StreamUtil.cs:line 71
         at Docker.DotNet.Models.StreamUtil.MonitorStreamForMessagesAsync[T](Task`1 streamTask, DockerClient client, CancellationToken cancellationToken, IPr
      ogress`1 progress) in /_/src/Docker.DotNet/Endpoints/StreamUtil.cs:line 71
         at Docker.DotNet.Models.StreamUtil.MonitorResponseForMessagesAsync[T](Task`1 responseTask, DockerClient client, CancellationToken cancel, IProgress`
      1 progress) in /_/src/Docker.DotNet/Endpoints/StreamUtil.cs:line 82
         at DotNet.Testcontainers.Clients.DockerImageOperations.CreateAsync(IImage image, IDockerRegistryAuthenticationConfiguration dockerRegistryAuthConfig
      , CancellationToken ct) in /_/src/Testcontainers/Clients/DockerImageOperations.cs:line 73
         at DotNet.Testcontainers.Clients.TestcontainersClient.PullImageAsync(IImage image, CancellationToken ct) in /_/src/Testcontainers/Clients/Testcontai
      nersClient.cs:line 422
         at DotNet.Testcontainers.Clients.TestcontainersClient.RunAsync(IContainerConfiguration configuration, CancellationToken ct) in /_/src/Testcontainers
      /Clients/TestcontainersClient.cs:line 335
         at DotNet.Testcontainers.Containers.DockerContainer.UnsafeCreateAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:l
      ine 462
         at DotNet.Testcontainers.Containers.DockerContainer.StartAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 306
         at MongoDB.ConformanceTests.Support.MongoTestStore.StartMongoDbContainerAsync() in /Users/roji/projects/semantic-kernel/dotnet/test/VectorData/Mongo
      DB.ConformanceTests/Support/MongoTestStore.cs:line 55
         at MongoDB.ConformanceTests.Support.MongoTestStore.StartAsync() in /Users/roji/projects/semantic-kernel/dotnet/test/VectorData/MongoDB.ConformanceTe
      sts/Support/MongoTestStore.cs:line 37
         at VectorData.ConformanceTests.Support.TestStore.ReferenceCountingStartAsync() in /Users/roji/projects/semantic-kernel/dotnet/test/VectorData/Vector
      Data.ConformanceTests/Support/TestStore.cs:line 43
      ----- Inner Stack Trace -----
         at System.Threading.CancellationToken.ThrowOperationCanceledException()
         at System.Threading.CancellationToken.ThrowIfCancellationRequested()
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Microsoft.Net.Http.Client.BufferedReadStream.ReadLineAsync(CancellationToken cancellationToken) in /_/src/Docker.DotNet/Microsoft.Net.Http.Client
      /BufferedReadStream.cs:line 176
         at Microsoft.Net.Http.Client.ChunkedReadStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken) in /_/src/Do
      cker.DotNet/Microsoft.Net.Http.Client/ChunkedReadStream.cs:line 83
         at System.IO.Pipelines.StreamPipeReader.<ReadInternalAsync>g__Core|40_0(StreamPipeReader reader, Nullable`1 minimumSize, CancellationTokenSource tok
      enSource, CancellationToken cancellationToken)

Test summary: total: 1, failed: 1, succeeded: 0, skipped: 0, duration: 60.8s
Build failed with 1 error(s) and 2 warning(s) in 62.3s

Successful test run (before this PR, on main)

➜  MongoDB.ConformanceTests git:(main) ✗ dotnet test -f net10.0 --filter MongoDB.ConformanceTests.MongoIndexKindTests
  VectorData.Abstractions net10.0 succeeded (2.0s) → /Users/roji/projects/semantic-kernel/dotnet/src/VectorData/VectorData.Abstractions/bin/Debug/net10.0/Microsoft.Extensions.VectorData.Abstractions.dll
  VectorData.ConformanceTests net10.0 succeeded (1.5s) → /Users/roji/projects/semantic-kernel/dotnet/test/VectorData/VectorData.ConformanceTests/bin/Debug/net10.0/VectorData.ConformanceTests.dll
  MongoDB net10.0 succeeded (1.7s) → /Users/roji/projects/semantic-kernel/dotnet/src/VectorData/MongoDB/bin/Debug/net10.0/Microsoft.SemanticKernel.Connectors.MongoDB.dll
  MongoDB.ConformanceTests net10.0 succeeded (1.0s) → bin/Debug/net10.0/MongoDB.ConformanceTests.dll
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v3.1.5+1b188a7b0a (64-bit .NET 10.0.0)
[xUnit.net 00:00:00.09]   Discovering: MongoDB.ConformanceTests
[xUnit.net 00:00:00.19]   Discovered:  MongoDB.ConformanceTests
[xUnit.net 00:00:00.20]   Starting:    MongoDB.ConformanceTests
[testcontainers.org 00:00:00.13] Connected to Docker:
  Host: unix:///var/run/docker.sock
  Server Version: 28.5.2
  Kernel Version: 6.12.54-linuxkit
  API Version: 1.51
  Operating System: Docker Desktop
  Total Memory: 7.65 GB
  Labels: 
    com.docker.desktop.address=unix:///Users/roji/Library/Containers/com.docker.docker/Data/docker-cli.sock
[testcontainers.org 00:00:00.31] Docker container 4ed4739756bb created
[testcontainers.org 00:00:00.37] Start Docker container 4ed4739756bb
[testcontainers.org 00:00:00.56] Wait for Docker container 4ed4739756bb to complete readiness checks
[testcontainers.org 00:00:00.62] Docker container 4ed4739756bb ready
[testcontainers.org 00:00:00.69] Docker container 80674f5fabe4 created
[testcontainers.org 00:00:01.17] Start Docker container 80674f5fabe4
[testcontainers.org 00:00:01.37] Wait for Docker container 80674f5fabe4 to complete readiness checks
[testcontainers.org 00:00:03.54] Docker container 80674f5fabe4 ready
[testcontainers.org 00:00:10.73] Stop Docker container 80674f5fabe4
[xUnit.net 00:00:11.36]   Finished:    MongoDB.ConformanceTests
  MongoDB.ConformanceTests test net10.0 succeeded (12.8s)

Test summary: total: 1, failed: 0, succeeded: 1, skipped: 0, duration: 12.8s
Build succeeded in 18.8s

[zarusz]

Great that this is being ported to the latest MDB driver 3.x.

When could we expect a new build to be available?
Currently trying to use the VectorStore abstractions with the SemanticKernel MDB implementation.

Thanks!

[markwallace-microsoft]

@damieng Can you take a look at the merge conflicts