diff --git a/src/content/docs/dotnet-next/infra/azure-blob-storage.md b/src/content/docs/dotnet-next/infra/azure-blob-storage.md new file mode 100644 index 0000000..5d57397 --- /dev/null +++ b/src/content/docs/dotnet-next/infra/azure-blob-storage.md @@ -0,0 +1,168 @@ +--- +title: "Azure Blob Storage" +description: "Projections for Azure Blob Storage" +sidebar: + order: 8 +--- + +[Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs/) is a fully managed object storage service in the cloud. Eventuous supports Blob Storage as a [projection](../../read-models/rm-concept) target using the `Eventuous.Azure.Storage.Blobs` package. +It allows you to project event store events to Azure Blob Storage as state objects, maintaining a separate state document for each event stream. + +## Using projections + +Create your own projection class that inherits from `BlobStorageProjector` where `T` is your state type. The state type must be a class with a parameterless constructor. + +Register event handlers using the `On` methods. When an event is received, the projector retrieves the current state blob (or creates a new state instance if the blob doesn't exist), applies the event to the state using the registered event handler, and uploads the updated state back to Blob Storage. + +The class provides two constructors: + +* `BlobStorageProjector(BlobContainerClient container, ...)` where the container client is passed directly +* `BlobStorageProjector(BlobServiceClient serviceClient, string containerName, ...)` where the service client comes from DI and the container name is set by the projection + +```csharp +public class BookingProjection : BlobStorageProjector { + public BookingProjection(BlobServiceClient client) + : base(client, "bookings-container") { + + // Uses default blob ID from stream + On((state, evt) => { + state.RoomId = evt.RoomId; + state.CheckInDate = evt.CheckIn; + return state; + }); + + // Custom blob ID using event data + On( + (state, evt) => { + state.PaidAmount += evt.AmountPaid; + return state; + }, + context => new ValueTask($"custom-{context.Message.BookingId}") + ); + } +} +``` + +:::caution +The blob container must exist before the projector handles events. The projector doesn't create it. +::: + +### Registration + +The projector needs an Azure `BlobServiceClient` registered in the DI container: + +```csharp +builder.Services.AddSingleton(new BlobServiceClient(connectionString)); +``` + +Then add the projection to a [subscription](../../subscriptions/subs-concept) as an event handler: + +```csharp +builder.Services.AddSubscription( + "BookingsBlobProjection", + b => b.AddEventHandler() +); +``` + +Give the projection its own subscription and [checkpoint](../../subscriptions/checkpoint) when you add it to a system that already has data, so it replays from the beginning of the log and backfills the blobs instead of resuming from another projection's position. + +### JSON serialization + +JSON serialization is configured via `BlobStorageProjectorOptions.JsonOptions`. When it isn't set, the projector uses `JsonSerializerOptions.Web`. If you keep serializer options in DI, pass them on. The example below uses `Microsoft.AspNetCore.Http.Json.JsonOptions`, which is what minimal APIs configure: + +```csharp +public BookingProjection(BlobServiceClient client, IOptions options) + : base( + client, + "bookings-container", + new BlobStorageProjectorOptions { JsonOptions = options.Value.SerializerOptions } + ) { } +``` + +## Projector options + +The `BlobStorageProjectorOptions` class provides several configuration options for fine-tuning the projector behavior. + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `JsonOptions` | `JsonSerializerOptions?` | `null` (uses `JsonSerializerOptions.Web`) | JSON serializer options for state serialization/deserialization. Controls formatting, naming policies, etc. | +| `RaceRetries` | `int` | `0` | Number of retry attempts for optimistic concurrency conflicts. Increase when concurrent updates are likely. | +| `IdempotencyMode` | `IdempotencyMode` | `IdempotencyMode.None` | Controls duplicate message detection behavior. | + +When the projector loses the optimistic concurrency race and `RaceRetries` is exhausted, it returns `EventHandlingStatus.Failure` for that event. + +### Idempotency modes + +The `IdempotencyMode` enum controls how the projector handles duplicate messages: + +- **`None`** - No idempotency checks. Will process messages and update the blob, without checking for duplicates. +- **`ByGlobalPosition`** - Skips processing if the existing blob has a global position set in its metadata that indicates it has already been processed. The event global position must be greater than that stored in the blob. +- **`ByMessageId`** - Use this when building projections directly from integration events. Skips processing if the message ID in the blob metadata matches that in the event. +Note, this means the idempotency is weaker as only the last message ID is checked. Older messages that are replayed will be processed as normal. + +:::caution +`ByGlobalPosition` requires a subscription that provides real global positions, such as an all-stream subscription. +Don't use it with message broker subscriptions where the global position is always zero — the first event would store position `0` and every subsequent event would be treated as a duplicate and silently ignored. Use `ByMessageId` instead. +::: + +### Custom blob naming + +By default, blob names are generated using `GetBlobName(string id)` which creates names in the format `{id}/{StateType}.json`, where `id` defaults to the stream ID from `context.Stream.GetId()`. + +You can customize blob naming in two ways: + +**1. Override the virtual methods globally for all events:** + +```csharp +public class BookingProjection : BlobStorageProjector { + // ... + + protected override string GetBlobName(string id) => $"bookings/{id}.json"; +} +``` + +When the blob name depends on the event, override the overload that takes the consume context instead: + +```csharp +protected override string GetBlobName(string id, IMessageConsumeContext context) + => $"projections/{context.Stream}/{id}.json"; +``` + +The default implementation of the two-argument overload calls the one-argument overload, so overriding the two-argument version replaces the naming completely — a one-argument override is then never called. + +**2. Override blob ID per event handler using `getBlobId`:** + +```csharp +On( + (state, evt) => { + state.PaidAmount += evt.AmountPaid; + return state; + }, + // Custom blob ID for this specific event only + context => new ValueTask($"payments-{context.Message.BookingId}") +); +``` + +Note that `getBlobId` returns a blob _ID_, not a full blob name: the result is still passed to `GetBlobName`, so with the default naming the example above produces `payments-{id}/BookingState.json`. To change the full blob path, override `GetBlobName` as well. + +Use per-event blob ID overrides when you need different events to target different blobs within the same projector, such as when the business identifier differs from the stream identifier. + +## Features + +- **Automatic state management** - Creates new state instances when blobs don't exist +- **Optimistic concurrency control** - Uses ETags for safe concurrent updates +- **Idempotency** - Prevents duplicate processing with configurable modes +- **Retry handling** - Automatic retries for race conditions +- **Flexible blob naming** - Customizable blob ID and naming conventions +- **Metadata storage** - Automatically stores stream info, positions, and message IDs + +## Background + +The projector stores each state as a separate blob in Azure Blob Storage. Each blob contains: + +- The serialized state object (JSON by default) +- Metadata including stream name, message ID, stream position, and global position; + because Azure requires metadata values to be ASCII, the stream name and message ID are stored percent-encoded +- Content type set to `application/json` + +This approach provides natural partitioning by stream and enables efficient state retrieval for individual streams. diff --git a/src/content/docs/dotnet-next/read-models/supported-projectors.md b/src/content/docs/dotnet-next/read-models/supported-projectors.md index f4be324..f8f2bc6 100644 --- a/src/content/docs/dotnet-next/read-models/supported-projectors.md +++ b/src/content/docs/dotnet-next/read-models/supported-projectors.md @@ -9,5 +9,6 @@ Eventuous supports the following projection targets: - [PostgreSQL projections](../../infra/postgres#projections) - [Microsoft SQL Server projections](../../infra/mssql#projections) - [SQLite projections](../../infra/sqlite#projections) +- [Azure Blob Storage projections](../../infra/azure-blob-storage) You can project to any other database using a custom projector, which can be built as a [custom event handler](../../subscriptions/eventhandler#custom-handlers). \ No newline at end of file