From c751d70dadba96c52ba4a9c3333b12fa1853e2da Mon Sep 17 00:00:00 2001 From: Quezlatch Date: Mon, 6 Jul 2026 08:56:20 +0100 Subject: [PATCH 1/3] add azure blob storage section --- .../dotnet-next/infra/azure-blob-storage.md | 114 ++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100644 src/content/docs/dotnet-next/infra/azure-blob-storage.md diff --git a/src/content/docs/dotnet-next/infra/azure-blob-storage.md b/src/content/docs/dotnet-next/infra/azure-blob-storage.md new file mode 100644 index 0000000..1fc0266 --- /dev/null +++ b/src/content/docs/dotnet-next/infra/azure-blob-storage.md @@ -0,0 +1,114 @@ +--- +title: "Azure Blob Storage" +description: "Projections for Azure Blob Storage" +sidebar: + order: 8 +--- + +[Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs/) is a fully managed object storage in the cloud. Eventuous supports Azure Service Bus for projections using the `Eventuous.Azure.Storage.Blobs` package. +It allows you to project event store events to Azure Blob Storage as state objects, maintaining a separate state document for each event stream. + +## Using projections + +Create your own projection class that inherits from `BlobStorageProjector` where `T` is your state type. The state type must be a class with a parameterless constructor. + +Register event handlers using the `On` methods. When an event is received, the projector retrieves the current state blob (or creates a new state instance if the blob doesn't exist), applies the event to the state using the registered event handler, and uploads the updated state back to Blob Storage. + +```csharp +public class BookingProjection : BlobStorageProjector { + public BookingProjection(BlobServiceClient client, IOptions serializerOptions) + : base(client, "bookings-container", serializerOptions.Value) { + + // Uses default blob ID from stream + On((state, evt) => { + state.RoomId = evt.RoomId; + state.CheckInDate = evt.CheckIn; + return state; + }); + + // Custom blob ID using event data + On( + (state, evt) => { + state.PaidAmount += evt.AmountPaid; + return state; + }, + context => new ValueTask($"custom-{context.Message.BookingId}") + ); + } +} +``` + +By using `IOptions` we can also use the Json serialization options as set in ASP DI. + +The blob name itself is constructed using the projection type name and stream id. This can be overriden. + + +## Projector options + +The `BlobStorageProjectorOptions` class provides several configuration options for fine-tuning the projector behavior. + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `JsonOptions` | `JsonSerializerOptions?` | `null` (uses `JsonSerializerOptions.Web`) | JSON serializer options for state serialization/deserialization. Controls formatting, naming policies, etc. | +| `RaceRetries` | `int` | `0` | Number of retry attempts for optimistic concurrency conflicts. Increase when concurrent updates are likely. | +| `IdempotencyMode` | `IdempotencyMode` | `IdempotencyMode.None` | Controls duplicate message detection behavior. | + +### Idempotency modes + +The `IdempotencyMode` enum controls how the projector handles duplicate messages: + +- **`None`** - No idempotency checks. Always processes messages and updates blobs. +- **`ByGlobalPosition`** - Skips processing if existing blob has matching global position metadata. +- **`ByMessageId`** - Skips processing if existing blob has matching message ID metadata. + +### Custom blob naming + +By default, blob names are generated using `GetBlobName(string id)` which creates names in the format `{id}/{T}.json`, where `id` defaults to the stream ID from `context.Stream.GetId()`. + +You can customize blob naming in two ways: + +**1. Override the virtual methods globally for all events:** + +```csharp +protected override string GetBlobName(string id, IMessageConsumeContext context) { + // Use stream name and type in the path + var streamName = context.Stream.ToString(); + return $"projections/{streamName}/{id}.json"; +} + +protected override string GetBlobName(string id) { + return $"{id}/{typeof(T).Name}.json"; +} +``` + +**2. Override blob ID per event handler using `getBlobId`:** + +```csharp +On( + (state, evt) => { + state.PaidAmount += evt.AmountPaid; + return state; + }, + // Custom blob ID for this specific event only + context => new ValueTask($"payments/{context.Message.BookingId}.json") +); +``` + +Use per-event blob ID overrides when you need different events to target different blob paths or naming conventions within the same projector, such as when the business identifier differs from the stream identifier. +## Features + +- **Automatic state management** - Creates new state instances when blobs don't exist +- **Optimistic concurrency control** - Uses ETags for safe concurrent updates +- **Idempotency** - Prevents duplicate processing with configurable modes +- **Retry handling** - Automatic retries for race conditions +- **Flexible blob naming** - Customizable blob ID and naming conventions +- **Metadata storage** - Automatically stores stream info, positions, and message IDs + +## Background + +The projector stores each state as a separate blob in Azure Blob Storage. Each blob contains: +- The serialized state object (JSON by default) +- Metadata including stream name, message ID, stream position, and global position +- Content type set to `application/json` + +This approach provides natural partitioning by stream and enables efficient state retrieval for individual streams. \ No newline at end of file From ef2e11fe4a157c9b2989315a73290b4533b8c5f4 Mon Sep 17 00:00:00 2001 From: Quezlatch Date: Sun, 2 Aug 2026 18:23:48 +0100 Subject: [PATCH 2/3] update idempotency mode info --- .../docs/dotnet-next/infra/azure-blob-storage.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/src/content/docs/dotnet-next/infra/azure-blob-storage.md b/src/content/docs/dotnet-next/infra/azure-blob-storage.md index 1fc0266..36f30c5 100644 --- a/src/content/docs/dotnet-next/infra/azure-blob-storage.md +++ b/src/content/docs/dotnet-next/infra/azure-blob-storage.md @@ -45,7 +45,7 @@ The blob name itself is constructed using the projection type name and stream id ## Projector options -The `BlobStorageProjectorOptions` class provides several configuration options for fine-tuning the projector behavior. +The `BlobStorageProjectorOptions` class provides several configuration options for fine-tuning the projector behavior. | Option | Type | Default | Description | |--------|------|---------|-------------| @@ -57,9 +57,13 @@ The `BlobStorageProjectorOptions` class provides several configuration option The `IdempotencyMode` enum controls how the projector handles duplicate messages: -- **`None`** - No idempotency checks. Always processes messages and updates blobs. -- **`ByGlobalPosition`** - Skips processing if existing blob has matching global position metadata. -- **`ByMessageId`** - Skips processing if existing blob has matching message ID metadata. +- **`None`** - No idempotency checks. Will process messages and updates blob, without +checking for duplicates. +- **`ByGlobalPosition`** - Skips processing if the existing blob has a global position set in +its metadata that indicates it has already been processed. The event global position must be greater than that stored in the blob. +- **`ByMessageId`** - Use this when building projections directly from integration events. Skips +processing if the message ID in the blob metadata matches that in the event. +Note, this means the idempotency is weaker as only the last message ID is checked. Older messages that are replayed will be processed as normal. ### Custom blob naming From f9a0eb9588c483b217d2861a03d9bcfdfc6398d6 Mon Sep 17 00:00:00 2001 From: Alexey Zimarev Date: Fri, 21 Aug 2026 13:14:50 +0200 Subject: [PATCH 3/3] Correct the Azure Blob Storage projector page against the shipped API The page was written against the pre-review API of Eventuous/eventuous#550, so it carried mistakes that the package README had already been corrected for. Fixes that make the samples compile: - the third constructor argument is BlobStorageProjectorOptions, not JsonSerializerOptions, so the IOptions example was invalid; show BlobStorageProjectorOptions.JsonOptions instead - typeof(T) is not in scope in a concrete projector deriving from BlobStorageProjector; drop it from the GetBlobName example Corrections: - the intro said Eventuous supports "Azure Service Bus" for projections - getBlobId returns a blob ID, not a full name, so the example producing "payments/{id}.json" actually wrote payments/{id}.json/BookingState.json - overriding both GetBlobName overloads is misleading, because the default two-argument implementation delegates to the one-argument one Content missing from the page but present in the README: - the container must exist; the projector doesn't create it - ByGlobalPosition silently ignores everything after the first event on subscriptions whose global position is always zero - both constructor overloads - stream name and message ID are stored percent-encoded, as Azure requires ASCII metadata values Also add a registration section matching the other infra pages, list the projector on the supported projectors page, and fix a typo, a missing blank line before a heading, and the missing trailing newline. Co-Authored-By: Claude Opus 5 --- .../dotnet-next/infra/azure-blob-storage.md | 98 ++++++++++++++----- .../read-models/supported-projectors.md | 1 + 2 files changed, 75 insertions(+), 24 deletions(-) diff --git a/src/content/docs/dotnet-next/infra/azure-blob-storage.md b/src/content/docs/dotnet-next/infra/azure-blob-storage.md index 36f30c5..5d57397 100644 --- a/src/content/docs/dotnet-next/infra/azure-blob-storage.md +++ b/src/content/docs/dotnet-next/infra/azure-blob-storage.md @@ -5,7 +5,7 @@ sidebar: order: 8 --- -[Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs/) is a fully managed object storage in the cloud. Eventuous supports Azure Service Bus for projections using the `Eventuous.Azure.Storage.Blobs` package. +[Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs/) is a fully managed object storage service in the cloud. Eventuous supports Blob Storage as a [projection](../../read-models/rm-concept) target using the `Eventuous.Azure.Storage.Blobs` package. It allows you to project event store events to Azure Blob Storage as state objects, maintaining a separate state document for each event stream. ## Using projections @@ -14,11 +14,16 @@ Create your own projection class that inherits from `BlobStorageProjector` wh Register event handlers using the `On` methods. When an event is received, the projector retrieves the current state blob (or creates a new state instance if the blob doesn't exist), applies the event to the state using the registered event handler, and uploads the updated state back to Blob Storage. +The class provides two constructors: + +* `BlobStorageProjector(BlobContainerClient container, ...)` where the container client is passed directly +* `BlobStorageProjector(BlobServiceClient serviceClient, string containerName, ...)` where the service client comes from DI and the container name is set by the projection + ```csharp public class BookingProjection : BlobStorageProjector { - public BookingProjection(BlobServiceClient client, IOptions serializerOptions) - : base(client, "bookings-container", serializerOptions.Value) { - + public BookingProjection(BlobServiceClient client) + : base(client, "bookings-container") { + // Uses default blob ID from stream On((state, evt) => { state.RoomId = evt.RoomId; @@ -38,10 +43,41 @@ public class BookingProjection : BlobStorageProjector { } ``` -By using `IOptions` we can also use the Json serialization options as set in ASP DI. +:::caution +The blob container must exist before the projector handles events. The projector doesn't create it. +::: + +### Registration -The blob name itself is constructed using the projection type name and stream id. This can be overriden. +The projector needs an Azure `BlobServiceClient` registered in the DI container: +```csharp +builder.Services.AddSingleton(new BlobServiceClient(connectionString)); +``` + +Then add the projection to a [subscription](../../subscriptions/subs-concept) as an event handler: + +```csharp +builder.Services.AddSubscription( + "BookingsBlobProjection", + b => b.AddEventHandler() +); +``` + +Give the projection its own subscription and [checkpoint](../../subscriptions/checkpoint) when you add it to a system that already has data, so it replays from the beginning of the log and backfills the blobs instead of resuming from another projection's position. + +### JSON serialization + +JSON serialization is configured via `BlobStorageProjectorOptions.JsonOptions`. When it isn't set, the projector uses `JsonSerializerOptions.Web`. If you keep serializer options in DI, pass them on. The example below uses `Microsoft.AspNetCore.Http.Json.JsonOptions`, which is what minimal APIs configure: + +```csharp +public BookingProjection(BlobServiceClient client, IOptions options) + : base( + client, + "bookings-container", + new BlobStorageProjectorOptions { JsonOptions = options.Value.SerializerOptions } + ) { } +``` ## Projector options @@ -53,38 +89,47 @@ The `BlobStorageProjectorOptions` class provides several configuration options f | `RaceRetries` | `int` | `0` | Number of retry attempts for optimistic concurrency conflicts. Increase when concurrent updates are likely. | | `IdempotencyMode` | `IdempotencyMode` | `IdempotencyMode.None` | Controls duplicate message detection behavior. | +When the projector loses the optimistic concurrency race and `RaceRetries` is exhausted, it returns `EventHandlingStatus.Failure` for that event. + ### Idempotency modes The `IdempotencyMode` enum controls how the projector handles duplicate messages: -- **`None`** - No idempotency checks. Will process messages and updates blob, without -checking for duplicates. -- **`ByGlobalPosition`** - Skips processing if the existing blob has a global position set in -its metadata that indicates it has already been processed. The event global position must be greater than that stored in the blob. -- **`ByMessageId`** - Use this when building projections directly from integration events. Skips -processing if the message ID in the blob metadata matches that in the event. +- **`None`** - No idempotency checks. Will process messages and update the blob, without checking for duplicates. +- **`ByGlobalPosition`** - Skips processing if the existing blob has a global position set in its metadata that indicates it has already been processed. The event global position must be greater than that stored in the blob. +- **`ByMessageId`** - Use this when building projections directly from integration events. Skips processing if the message ID in the blob metadata matches that in the event. Note, this means the idempotency is weaker as only the last message ID is checked. Older messages that are replayed will be processed as normal. +:::caution +`ByGlobalPosition` requires a subscription that provides real global positions, such as an all-stream subscription. +Don't use it with message broker subscriptions where the global position is always zero — the first event would store position `0` and every subsequent event would be treated as a duplicate and silently ignored. Use `ByMessageId` instead. +::: + ### Custom blob naming -By default, blob names are generated using `GetBlobName(string id)` which creates names in the format `{id}/{T}.json`, where `id` defaults to the stream ID from `context.Stream.GetId()`. +By default, blob names are generated using `GetBlobName(string id)` which creates names in the format `{id}/{StateType}.json`, where `id` defaults to the stream ID from `context.Stream.GetId()`. You can customize blob naming in two ways: **1. Override the virtual methods globally for all events:** ```csharp -protected override string GetBlobName(string id, IMessageConsumeContext context) { - // Use stream name and type in the path - var streamName = context.Stream.ToString(); - return $"projections/{streamName}/{id}.json"; -} +public class BookingProjection : BlobStorageProjector { + // ... -protected override string GetBlobName(string id) { - return $"{id}/{typeof(T).Name}.json"; + protected override string GetBlobName(string id) => $"bookings/{id}.json"; } ``` +When the blob name depends on the event, override the overload that takes the consume context instead: + +```csharp +protected override string GetBlobName(string id, IMessageConsumeContext context) + => $"projections/{context.Stream}/{id}.json"; +``` + +The default implementation of the two-argument overload calls the one-argument overload, so overriding the two-argument version replaces the naming completely — a one-argument override is then never called. + **2. Override blob ID per event handler using `getBlobId`:** ```csharp @@ -94,11 +139,14 @@ On( return state; }, // Custom blob ID for this specific event only - context => new ValueTask($"payments/{context.Message.BookingId}.json") + context => new ValueTask($"payments-{context.Message.BookingId}") ); ``` -Use per-event blob ID overrides when you need different events to target different blob paths or naming conventions within the same projector, such as when the business identifier differs from the stream identifier. +Note that `getBlobId` returns a blob _ID_, not a full blob name: the result is still passed to `GetBlobName`, so with the default naming the example above produces `payments-{id}/BookingState.json`. To change the full blob path, override `GetBlobName` as well. + +Use per-event blob ID overrides when you need different events to target different blobs within the same projector, such as when the business identifier differs from the stream identifier. + ## Features - **Automatic state management** - Creates new state instances when blobs don't exist @@ -111,8 +159,10 @@ Use per-event blob ID overrides when you need different events to target differe ## Background The projector stores each state as a separate blob in Azure Blob Storage. Each blob contains: + - The serialized state object (JSON by default) -- Metadata including stream name, message ID, stream position, and global position +- Metadata including stream name, message ID, stream position, and global position; + because Azure requires metadata values to be ASCII, the stream name and message ID are stored percent-encoded - Content type set to `application/json` -This approach provides natural partitioning by stream and enables efficient state retrieval for individual streams. \ No newline at end of file +This approach provides natural partitioning by stream and enables efficient state retrieval for individual streams. diff --git a/src/content/docs/dotnet-next/read-models/supported-projectors.md b/src/content/docs/dotnet-next/read-models/supported-projectors.md index f4be324..f8f2bc6 100644 --- a/src/content/docs/dotnet-next/read-models/supported-projectors.md +++ b/src/content/docs/dotnet-next/read-models/supported-projectors.md @@ -9,5 +9,6 @@ Eventuous supports the following projection targets: - [PostgreSQL projections](../../infra/postgres#projections) - [Microsoft SQL Server projections](../../infra/mssql#projections) - [SQLite projections](../../infra/sqlite#projections) +- [Azure Blob Storage projections](../../infra/azure-blob-storage) You can project to any other database using a custom projector, which can be built as a [custom event handler](../../subscriptions/eventhandler#custom-handlers). \ No newline at end of file