Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .changeset/free-rabbits-clap.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
---
---
94 changes: 60 additions & 34 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,42 @@
# @effectmq/core
# effectmq

It's a task queue built on [Effect](https://effect.website): typed payloads, typed results, typed errors, all the way down. You describe a unit of work as a schema, hand it to a queue, and process it with a handler that is just an `Effect`. Retries, delays, idempotency, cron schedules: handled. The available engine is backed by Redis, but, like many things in Effect, it can be swapped for a different implementation.
**A typed, Redis-backed task queue for Effect 4.** Define work with schemas, run
handlers as Effects, and keep payloads, results, and failures typed from producer
to worker.

[Website](https://docs-one-eta-87.vercel.app/) ·
[Documentation](https://docs-one-eta-87.vercel.app/docs) ·
[Getting started](https://docs-one-eta-87.vercel.app/docs/tutorials/getting-started) ·
[API reference](https://docs-one-eta-87.vercel.app/docs/reference/task-queue) ·
[npm](https://www.npmjs.com/package/@effectmq/core)

effectmq is for background jobs that need durable Redis state without becoming a
workflow engine: send an email, resize an image, refresh a cache, or materialize
a scheduled report. It provides:

- schema-checked payloads, successes, and failures;
- at-least-once delivery with fenced attempts and stalled-worker recovery;
- Effect `Schedule` retries, delayed offers, deduplication, and durable cron;
- bounded local worker concurrency, graceful draining, and maintenance;
- typed lifecycle streams plus `wait` and `execute` for durable results.

## Install

```bash
pnpm add @effectmq/core effect@4.0.0-beta.107 @effect/platform-node@4.0.0-beta.107
pnpm add @effectmq/core@0.3.0-rc.0 effect@4.0.0-beta.107 @effect/platform-node@4.0.0-beta.107
```

This library is built on the Effect 4 beta and doesn't work with the current stable Effect release. The examples below use the bundled `NodeRedisPool` layer, a connection-pooled Redis client that ships with the package (`@effect/platform-node` is only needed for `NodeRuntime`). This is beta-era software riding beta-era Effect; pin accordingly.
> [!IMPORTANT]
> effectmq currently targets the Effect 4 beta and is not compatible with the
> stable Effect 3 release. Pin the versions shown above. Node.js 22.19 or newer
> is required; CI verifies Node.js 22 and 24.

Node.js 22.19 or newer is required; the release matrix verifies Node.js 22 and 24.
The package includes its pooled `NodeRedisPool` implementation.
`@effect/platform-node` is only needed by these examples for `NodeRuntime`.

---

## In thirty seconds
## Quick start

Define a task, enqueue work, process it. The whole loop:

Expand Down Expand Up @@ -46,11 +70,13 @@ const AppLayer = TaskEngine.layer({
program.pipe(Effect.provide(AppLayer), NodeRuntime.runMain);
```

That's the shape of it. The rest of this README explains the pieces (typed errors, retries, worker pools, schedules) and the one thing the library deliberately *doesn't* do.
That is the complete producer-to-worker loop. For a clean-room walkthrough with
Redis startup and expected output, follow the
[getting-started tutorial](https://docs-one-eta-87.vercel.app/docs/tutorials/getting-started).

---

## The setup, once
## Runtime setup

`TaskEngine.layer()` is the complete Node live graph: it provides the engine,
cryptographic identity generation, and the retained Redis pool, role, and
Expand All @@ -74,7 +100,9 @@ ACL, bounded-pool, persistence, failover, health, and shutdown guidance.

The tested platform matrix is in the [support policy](./docs/support-policy.md), and reproducible throughput/tail-latency results are published as [performance evidence](./docs/performance.md).

`TaskEngine` is the machinery underneath: atomic Lua scripts, locks, the lists tasks move between. Provide its layer and forget it; the API you live in is `TaskQueue` and `Scheduler`.
`TaskEngine` is the machinery underneath: atomic Lua scripts, leases, and the
lists tasks move between. Provide its layer once; application code normally
lives in `TaskQueue`, `Worker`, and `Scheduler`.

---

Expand Down Expand Up @@ -135,7 +163,10 @@ One `complete` processes one task. To process *many*, set your workers up accord

### Predefine the handler

Handlers are just functions and workers are just Effects, so both are values you can name once and reuse. Type a handler with `TaskHandler` to declare it next to the task definition, before any queue exists; bind it to a queue with `complete` and you have a worker effect you can run, repeat, or fork like any other:
Handlers are functions and workers are Effects, so both are values you can name
once and reuse. Type a handler with `TaskHandler` to declare it next to the task
definition before any queue exists. Bind it with `complete` for one task, or use
the managed `Worker` shown below for a long-running process:

```ts docs-check=email
// Declared against the task definition — no queue in sight yet.
Expand Down Expand Up @@ -189,7 +220,7 @@ const offerAndWait = Effect.gen(function* () {

---

## On concurrency
## Run a worker

`complete` processes exactly one task. For a long-running process, `Worker`
provides bounded local concurrency, lease supervision, maintenance, and graceful
Expand All @@ -200,11 +231,11 @@ const worker = Worker.make(emails, handleSendEmail, { concurrency: 5 });
const program = Worker.run(worker);
```

The built-in worker does not impose distributed/global concurrency or rate
limits. Compose those policies from Effect primitives or external coordination,
and run more worker processes to fan out. The queue preserves eligible work and
fences the current attempt; handlers remain at-least-once and must make external
side effects idempotent.
`concurrency` is local to one worker process (valid values: 1–1000). Run more
processes to fan out. Distributed/global concurrency and rate limits require
external coordination; effectmq does not pretend a process-local semaphore can
enforce them. The queue fences each attempt, but handlers remain at-least-once,
so make external side effects idempotent.

---

Expand Down Expand Up @@ -260,28 +291,23 @@ const worker = Worker.make(reportQueue, ({ payload }) =>
## Notes

- **Completion policies.** `offer` accepts `onSuccessPolicy` and `onFailurePolicy`, each one of `delete` | `keep` | `mark-as-success` | `mark-as-failure`. They decide where a finished task lands: gone, quietly retained, or parked on the success/failed list for inspection. Defaults are `delete`.
- **Retries.** Declare `retry` on the task definition (`Task.make`) as an Effect `Schedule` — or a `{ while, until, times, schedule }` options object. On failure the next run time is computed from the schedule and the task lands on the scheduled list until then; when the schedule is exhausted, the failure policy applies. `maxRetries` caps the attempts so an unbounded schedule (e.g. `Schedule.forever`) can't loop forever: it defaults to `5`, is overridable per-`offer` (the per-offer value wins), and set it to `null` for truly unbounded retries. A `Canceled` error skips remaining retries.
- **Retries.** Declare `retry` on the task definition (`Task.make`) as an Effect `Schedule` — or a `{ while, until, times, schedule }` options object. On failure the next run time is computed from the schedule and the task lands on the scheduled list until then; when the schedule is exhausted, the failure policy applies. The task-level `maxRetries` cap defaults to `5`; pass `null` there for an intentionally unbounded cap. An `offer` may override the cap with a finite non-negative number. Built-in canceled or stalled failures are not retried by the handler schedule.
- **Idempotency.** The `idempotencyKey` is the task id. By default, offering the same key returns the existing generation unchanged; replacement requires explicit new-generation mode.
- **Delays.** `offer(..., { delay })` schedules the task for the future; it sits on the scheduled list until its time comes.
- **The engine.** `TaskEngine` is the low-level, Lua-backed layer all of this sits on. You provide its layer; you rarely call it directly.

## Production guides

- [Architecture](./docs/architecture.md)
- [Runtime boundaries](./docs/runtime-boundaries.md)
- [API reference](./docs/api-reference.md)
- [Delivery guarantees](./docs/delivery-guarantees.md)
- [Idempotent offers](./docs/idempotent-offers.md)
- [Task relationships](./docs/task-relationships.md)
- [Durable scheduling](./docs/scheduler.md)
- [Storage protocol v1](./docs/storage-protocol-v1.md)
- [Operations](./docs/operations.md)
- [Support policy](./docs/support-policy.md)
- [Upgrade and rollback](./docs/upgrade-and-rollback.md)
- [Performance evidence](./docs/performance.md)
- [Soak evidence](./docs/soak.md)
- [Release process](./docs/releasing.md)
- [Current release record](./docs/release-readiness.md)
## Go deeper

| If you need to… | Read… |
| --- | --- |
| learn the library from a running example | [Getting started](https://docs-one-eta-87.vercel.app/docs/tutorials/getting-started) |
| process, schedule, retry, or await tasks | [How-to guides](https://docs-one-eta-87.vercel.app/docs/how-to/process-tasks) |
| look up exact API behavior | [API reference](./docs/api-reference.md) |
| understand delivery and task identity | [Delivery guarantees](./docs/delivery-guarantees.md) · [Idempotent offers](./docs/idempotent-offers.md) |
| operate Redis and plan upgrades | [Operations](./docs/operations.md) · [Upgrade and rollback](./docs/upgrade-and-rollback.md) |
| inspect architecture and storage contracts | [Architecture](./docs/architecture.md) · [Runtime boundaries](./docs/runtime-boundaries.md) · [Storage protocol v1](./docs/storage-protocol-v1.md) |
| evaluate support and performance | [Support policy](./docs/support-policy.md) · [Performance evidence](./docs/performance.md) · [Soak evidence](./docs/soak.md) |
| release the package | [Release process](./docs/releasing.md) · [Current release record](./docs/release-readiness.md) |

---

Expand Down
18 changes: 16 additions & 2 deletions apps/docs/app/landing.css
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,7 @@
.lp a.lp-ghost-btn {
display: flex;
align-items: center;
white-space: nowrap;
color: #ededf2;
border: 1px solid #2a2a33;
border-radius: 10px;
Expand All @@ -236,6 +237,15 @@
text-decoration: none;
}

.lp a.lp-ghost-btn:focus-visible {
outline: 2px solid var(--accent);
outline-offset: 2px;
}

.lp a.lp-ghost-btn:active {
border-color: var(--accent);
}

/* ---- sections ---- */

.lp-eyebrow {
Expand Down Expand Up @@ -399,11 +409,15 @@
padding-top: 96px;
padding-bottom: 96px;
display: grid;
grid-template-columns: minmax(320px, 5fr) 6fr;
grid-template-columns: minmax(320px, 5fr) minmax(0, 6fr);
gap: 56px;
align-items: center;
}

.lp-conc-grid > * {
min-width: 0;
}

.lp-conc-body {
margin: 0 0 22px;
font:
Expand Down Expand Up @@ -708,7 +722,7 @@

@media (max-width: 960px) {
.lp-conc-grid {
grid-template-columns: 1fr;
grid-template-columns: minmax(0, 1fr);
gap: 36px;
}
.lp-stats,
Expand Down
57 changes: 27 additions & 30 deletions apps/docs/app/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,11 @@ import "./landing.css";
export const metadata: Metadata = {
title: "effectmq — typed task queue built on Effect",
description:
"A task queue built on Effect. Describe work as a schema, process it with a handler that is just an Effect. Retries, delays, idempotency, cron schedules: handled. Redis keeps unfinished work recoverable.",
"A typed, Redis-backed task queue for Effect 4 with schema-checked payloads, results and failures, fenced attempts, retries, delays and durable cron.",
};

const GITHUB_URL = "https://github.com/julia-script/effectmq";
const NPM_URL = "https://www.npmjs.com/package/@effectmq/core";

const FEATURES = [
{
Expand Down Expand Up @@ -43,8 +44,8 @@ const FEATURES = [
code: "lease: unique token per attempt",
},
{
title: "Swappable engine",
body: "The Redis engine is a Layer built on atomic Lua scripts. Provide it and forget it — or swap it for a different implementation, like anything in Effect.",
title: "One runtime layer",
body: "TaskEngine.layer wires the Redis pools, health services, cryptographic identity and Lua-backed engine. Provide it once; work through TaskQueue, Worker and Scheduler.",
code: "TaskEngine.layer({ redis })",
},
];
Expand Down Expand Up @@ -73,7 +74,7 @@ export default function HomePage() {
<div className="lp-hero-glow" />
<div className="lp-container lp-hero-inner">
<div className="lp-chips">
<span className="lp-chip lp-chip-accent">BUILT ON EFFECT 4</span>
<span className="lp-chip lp-chip-accent">EFFECT 4 BETA</span>
<span className="lp-chip">REDIS-BACKED</span>
<span className="lp-chip">MIT</span>
</div>
Expand All @@ -85,16 +86,18 @@ export default function HomePage() {
Typed errors<span className="lp-accent">.</span> All the way down.
</h1>
<p className="lp-hero-sub">
A task queue built on <a href="https://effect.website">Effect</a>.
Describe work as a schema, process it with a handler that is just an
Effect. Retries, delays, idempotency, cron schedules: handled. Redis
keeps unfinished work recoverable.
Define background work with schemas and process it with handlers
that are ordinary Effects. Redis keeps unfinished attempts
recoverable; payloads, results and failures stay typed end to end.
</p>
<div className="lp-cta-row">
<InstallCommand />
<Link href="/docs" className="lp-ghost-btn">
Read the docs →
</Link>
<a href={NPM_URL} className="lp-ghost-btn">
View on npm ↗
</a>
</div>
</div>
</section>
Expand All @@ -103,13 +106,12 @@ export default function HomePage() {
className="lp-container"
style={{ paddingTop: 96, paddingBottom: 84 }}
>
<p className="lp-eyebrow">01 — Lifecycle</p>
<h2 className="lp-h2">How a task lives.</h2>
<h2 className="lp-h2">Durable work, explicit states.</h2>
<p className="lp-lede">
The state machine every task moves through: offered, queued, leased
under a unique fence token, and acknowledged — or routed back through
its retry Schedule. Unfinished work survives worker loss. (An
illustration of the engine's internals, not a product dashboard.)
A task is offered, queued, leased under a unique fence token, then
acknowledged or rescheduled by its retry policy. If a worker
disappears, maintenance recovers the unfinished attempt. This diagram
shows the engine's state transitions.
</p>
<div className="lp-panel lp-canvas-panel">
<LifecycleCanvas />
Expand Down Expand Up @@ -138,12 +140,11 @@ export default function HomePage() {
className="lp-container"
style={{ paddingTop: 12, paddingBottom: 96 }}
>
<p className="lp-eyebrow">02 — The API</p>
<h2 className="lp-h2">The whole loop, in thirty seconds.</h2>
<p className="lp-lede" style={{ marginBottom: 30 }}>
A task is a schema, not a function. Your handler receives a fully
decoded payload — the real object, not a JSON stringand its
failures are pattern-matchable typed errors.
decoded payloadnot a JSON stringand typed failures remain
pattern-matchable downstream.
</p>
<ApiTabs />
</section>
Expand All @@ -158,12 +159,11 @@ export default function HomePage() {
className="lp-container"
style={{ paddingTop: 96, paddingBottom: 96 }}
>
<p className="lp-eyebrow">04 — Receipts</p>
<h2 className="lp-h2">Numbers, not vibes.</h2>
<h2 className="lp-h2">A baseline you can reproduce.</h2>
<p className="lp-lede" style={{ marginBottom: 40, maxWidth: 640 }}>
Every release ships reproducible performance evidence: full end-to-end
task lifecycles — atomic create, fenced acquire, acknowledge —
measured per payload size and concurrency.
The release baseline measures full task lifecycles—atomic create,
fenced acquire and acknowledge—by payload size and concurrency. Use it
to detect regressions, not to size production infrastructure.
</p>
<div className="lp-stats">
<div className="lp-stat-card">
Expand Down Expand Up @@ -207,9 +207,8 @@ export default function HomePage() {
className="lp-container"
style={{ paddingTop: 96, paddingBottom: 96 }}
>
<p className="lp-eyebrow">05 — Batteries</p>
<h2 className="lp-h2" style={{ marginBottom: 44 }}>
Handled, so you don't.
What the queue handles.
</h2>
<div className="lp-cards">
{FEATURES.map((f) => (
Expand All @@ -227,9 +226,10 @@ export default function HomePage() {
<div className="lp-cta-glow" />
<div className="lp-container lp-cta-inner">
<h2>
Put work in. Take attempts.
Define the work. Run the worker.
<br />
Redis keeps the rest recoverable<span className="lp-accent">.</span>
Redis keeps unfinished attempts recoverable
<span className="lp-accent">.</span>
</h2>
<InstallCommand />
<a href={GITHUB_URL} className="lp-star-link">
Expand All @@ -250,10 +250,7 @@ export default function HomePage() {
<a href={GITHUB_URL} className="lp-footer-link">
GitHub
</a>
<a
href="https://www.npmjs.com/package/@effectmq/core"
className="lp-footer-link"
>
<a href={NPM_URL} className="lp-footer-link">
npm
</a>
</div>
Expand Down
Loading
Loading