Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/inline-source-parsers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@tanstack/markdown": minor
---

Add opt-in source-level inline parsers to Markdown extensions. Declare starting characters and return a standard inline node with an explicit consumed length, before built-in formatting changes the source. Preserve escape and code precedence, expose link-label context, and share parser budgets with nested parsing.
5 changes: 5 additions & 0 deletions .changeset/optional-http-autolinks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@tanstack/markdown": minor
---

Add a separately imported HTTP(S) autolink extension using the inline source parser API. Recognize pasted URLs and angle-bracket URLs before Markdown punctuation changes their contents, preserve code and explicit links, and apply the existing application URL policy. Core and docs-preset defaults are unchanged.
8 changes: 4 additions & 4 deletions docs/comparison.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,11 @@ These repository benchmarks bundle representative browser entry points from pinn

| Entry | Gzip | Brotli |
| --- | ---: | ---: |
| `@tanstack/markdown/parser` | 5.0 KB | 4.6 KB |
| `@tanstack/markdown/html` | 6.8 KB | 6.2 KB |
| `@tanstack/markdown/react` | 6.7 KB | 6.2 KB |
| `@tanstack/markdown/parser` | 5.3 KB | 4.9 KB |
| `@tanstack/markdown/html` | 7.1 KB | 6.5 KB |
| `@tanstack/markdown/react` | 7.0 KB | 6.5 KB |
| React with streaming extension | 6.9 KB | 6.4 KB |
| `@tanstack/markdown/octane` | 6.7 KB | 6.2 KB |
| `@tanstack/markdown/octane` | 7.0 KB | 6.5 KB |
| Marked | 12.5 KB | 11.5 KB |
| micromark | 15.4 KB | 13.7 KB |
| markdown-wasm JS + WASM | 31.3 KB | 26.4 KB |
Expand Down
4 changes: 2 additions & 2 deletions docs/core-concepts/syntax-profile.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ TanStack Markdown implements a documented subset aimed at repository-authored bl
| Reference links and images | Yes | Full, collapsed, and shortcut forms; normalized reference labels |
| Hard breaks | Yes | Backslash before a newline |
| Raw inline HTML | Opt-in | Requires `allowHtml: true` |
| Autolink literals | No | Write an explicit link |
| Autolink literals | Opt-in | HTTP(S) only via the [autolinks extension](../guides/extensions#optional-url-linking) |
| Entity decoding | Partial | HTML is escaped; full CommonMark entity behavior is not a goal |

## Docs metadata
Expand Down Expand Up @@ -69,7 +69,7 @@ The following are not project goals:

- complete CommonMark or GFM conformance
- MDX, JSX parsing, or arbitrary code evaluation
- automatic URL linking
- automatic URL linking by default or full GFM autolink conformance
- a complete HTML parser or sanitizer
- every delimiter, indentation, entity, or reference-label interaction
- syntax highlighting, themes, or language grammars
Expand Down
63 changes: 61 additions & 2 deletions docs/guides/extensions.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,68 @@ The context includes:

Nested parsing shares the parser depth budget and heading slugger.

## Inline source parsing

Use `inlineParser` when syntax must see source characters before they become emphasis, links, or decoded escapes. `transformInline` cannot recover an escaped opener or the original spelling of an already-parsed node.

```ts
import type { MarkdownExtension } from '@tanstack/markdown'

const issueReferences: MarkdownExtension = {
name: 'issue-references',
inlineParser: {
markers: '#',
parse({ source, index, inLink }) {
if (inLink) return undefined
const match = /^#([0-9]+)\b/.exec(source.slice(index))
if (!match) return undefined
return {
length: match[0].length,
node: {
type: 'link',
href: `/issues/${match[1]}`,
children: [{ type: 'text', value: match[0] }],
},
}
},
},
}
```

`markers` lists literal possible first characters, not a regular expression. The parser skips ordinary text to the next built-in or extension marker. At a matching position, extensions run in array order after built-in escapes and code spans, before the other built-in inline rules. The first returned result owns that range. Return `undefined` to let the next extension or built-in rule handle it.

The context provides `source`, `index`, `options`, `inLink`, and a nested `parseInline(value)` helper. Source and UTF-16 indices refer to the current inline container, not offsets in the original document. Hooks also run within emphasis and explicit link labels; `inLink` remains true through their nested content. They do not run inside code, image alt text, or link destinations. A hook cannot consume across an enclosing inline or block boundary.

Return one standard `InlineNode` and a positive integer `length` within the remaining source. Invalid lengths throw `RangeError`. The child parser shares the existing depth and scan limits; use that helper instead of calling the top-level parser recursively. Hook dispatch counts against the scan budget. Extension code remains trusted: these limits do not bound work done inside a callback. Keep recognition deterministic and avoid repeatedly scanning a suffix after unsuccessful matches.

Returned nodes follow the same trust contract as supplied ASTs: URL destinations and component names/properties must be validated by the extension. The fixed, numeric issue path above needs no user-supplied URL. Extensions accepting arbitrary URLs should use the application URL policy. Return a portable `InlineComponentNode` for custom presentation; no HTML renderer changes are required.

## Optional URL linking

```ts
import { renderHtml } from '@tanstack/markdown/html'
import { autolinksExtension } from '@tanstack/markdown/extensions/autolinks'

const html = renderHtml('See https://example.com/~alice~/notes.', {
extensions: [autolinksExtension()],
})
```

This extension recognizes bare HTTP(S) URLs and `<http://…>` / `<https://…>` notation, without changing the core or docs-preset defaults. URLs retain their original source spelling, including Markdown punctuation. Host/port validity follows the platform `URL` implementation. Link nodes pass through the existing `urlTransform(url, 'link', defaultUrl)` policy and render consistently in HTML, React, and Octane; `null` keeps only the URL label. Application replacements remain trusted, as with explicit links.

The bounded profile is intentionally smaller than GFM autolink literals:

- Bare links must start at the container boundary or after punctuation/whitespace, excluding letters, numbers, `_`, `/`, `@`, `<`, and backslash. They stop at whitespace, controls, quotes, backticks, backslashes, angle brackets, or an unmatched closing parenthesis/bracket/brace.
- Balanced parentheses, brackets, and braces stay in bare URLs. Trailing `. , ! ? ; :` characters stay outside the link. Use angle notation or an explicit Markdown link when those trailing characters belong to the URL.
- Angle notation preserves trailing punctuation and requires a closing `>` before whitespace, controls, quotes, backticks, backslashes, or another `<`. Escaping the opening `<` keeps it literal.
- Explicit links (including their formatted labels), images, code, destinations, and enabled raw HTML are handled by the existing parser. Autolinks also work in ordinary emphasis, headings, lists, quotes, and table cells, within the enclosing inline boundaries.
- `www.` addresses, email detection, other schemes, entity decoding, and full CommonMark/GFM autolink conformance are not included. Malformed HTTP(S) candidates are consumed as literal text to avoid rescanning their suffixes.

This uses the same public `inlineParser` contract as third-party extensions and adds no runtime dependency. Import it only where URL linking is wanted.

## Inline transformation

`transformInline` receives built-in inline nodes after parsing. Return the replacement array. Keep transforms deterministic and avoid repeated full-array scans for every node.
`transformInline` receives built-in and extension inline nodes after parsing. Return the replacement array. Keep transforms deterministic and avoid repeated full-array scans for every node.

The hook runs once per inline container. Recurse through inline `children` when your transform also needs to handle content inside emphasis or links. Code spans and image alt text are not separate inline containers.

Expand Down Expand Up @@ -93,4 +152,4 @@ Document transforms are already represented in a pre-parsed AST. HTML render hoo

## Admission rule

An extension is appropriate when syntax is broadly useful to docs, has a deterministic block boundary, and does not justify cost in the core entry. Use a larger processing ecosystem when the job requires async plugins, arbitrary tree pipelines, compiler integration, or MDX evaluation.
An extension is appropriate when syntax is broadly useful to docs, has a deterministic source boundary, and does not justify cost in the core entry. Use a larger processing ecosystem when the job requires async plugins, arbitrary tree pipelines, compiler integration, or MDX evaluation.
8 changes: 4 additions & 4 deletions docs/guides/performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ The generated browser bundle report records:

| Entry | Gzip | Brotli |
| --- | ---: | ---: |
| parser | 5.0 KB | 4.6 KB |
| HTML renderer | 6.8 KB | 6.2 KB |
| React adapter | 6.7 KB | 6.2 KB |
| Octane adapter | 6.7 KB | 6.2 KB |
| parser | 5.3 KB | 4.9 KB |
| HTML renderer | 7.1 KB | 6.5 KB |
| React adapter | 7.0 KB | 6.5 KB |
| Octane adapter | 7.0 KB | 6.5 KB |
| React adapter with streaming extension | 6.9 KB | 6.4 KB |
| Streaming extension | 0.3 KB | 0.3 KB |
| docs preset | 2.3 KB | 2.1 KB |
Expand Down
4 changes: 2 additions & 2 deletions docs/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,13 @@ General Markdown processors optimize for broad conformance, plugin ecosystems, o
- code metadata for documentation UI
- a small browser bundle

TanStack Markdown spends its complexity budget on that path. It deliberately does not implement every CommonMark edge case, MDX evaluation, automatic linkification, or a general asynchronous processing ecosystem.
TanStack Markdown spends its complexity budget on that path. It deliberately does not implement every CommonMark edge case, MDX evaluation, automatic linkification by default, or a general asynchronous processing ecosystem.

## Core properties

### Small entry points

Current minified browser bundles are 5.0 KB gzip for the parser, 6.8 KB for HTML rendering, and 6.7 KB for either UI adapter with its framework runtime externalized. The generated [bundle report](https://github.com/TanStack/markdown/blob/main/reports/sizes.md) is the source of truth.
Current minified browser bundles are 5.3 KB gzip for the parser, 7.1 KB for HTML rendering, and 7.0 KB for either UI adapter with its framework runtime externalized. The generated [bundle report](https://github.com/TanStack/markdown/blob/main/reports/sizes.md) is the source of truth.

### Parse once, render many

Expand Down
12 changes: 12 additions & 0 deletions docs/reference/extensions.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,18 @@ title: Extensions

Every built-in extension is available through a separate package entry.

## Autolinks

Import from `@tanstack/markdown/extensions/autolinks`.

### `autolinksExtension`

```ts
function autolinksExtension(): MarkdownExtension
```

Creates an inline source parser for bare `http://` / `https://` URLs and explicit `<https://…>` links. Returns ordinary link nodes and applies the active `urlTransform` policy. It is not included in the core entry or docs preset. See the [autolink profile](../guides/extensions#optional-url-linking) for boundaries and deliberate limits.

## Callouts

Import from `@tanstack/markdown/extensions/callouts`.
Expand Down
1 change: 1 addition & 0 deletions docs/reference/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ TanStack Markdown uses explicit subpath exports so applications can import only
| `@tanstack/markdown/html` | HTML rendering functions |
| `@tanstack/markdown/react` | React component and React rendering functions |
| `@tanstack/markdown/octane` | Octane component and descriptor rendering functions |
| `@tanstack/markdown/extensions/autolinks` | optional HTTP(S) URL linking |
| `@tanstack/markdown/extensions/callouts` | callout block parser |
| `@tanstack/markdown/extensions/comment-components` | comment-delimited component parser |
| `@tanstack/markdown/extensions/docs` | composed docs extension preset |
Expand Down
14 changes: 13 additions & 1 deletion docs/reference/types.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,12 +158,24 @@ Configures anchor `content`, `className`, `ariaHidden`, and `tabIndex`.

### `MarkdownExtension`

Named hook object with optional `parseBlock`, `transformDocument`, `transformInline`, and `renderHtml` functions.
Named hook object with optional `parseBlock`, `transformDocument`, `transformInline`, and `renderHtml` functions, plus an optional `inlineParser`.

### `BlockParseContext`

Provides source `lines`, current `index`, active `options`, nested `parseInline` and `parseBlocks` helpers, and `consume`.

### `InlineParser`

Contains literal first-character `markers` and a synchronous `parse(context)` callback returning `InlineParseResult | undefined`. Runs at matching source positions after escapes and code spans, before other built-in inline rules.

### `InlineParseContext`

Provides current-container `source`, UTF-16 `index`, active `options`, `inLink`, and a nested `parseInline(value)` helper that shares the recursion and scan budget.

### `InlineParseResult`

Contains one standard `InlineNode` as `node` and a positive integer `length` in UTF-16 code units. The range must fit in the current source. Invalid lengths throw `RangeError`.

### `InlineTransformContext`

Provides active parse `options` to inline transforms.
Expand Down
4 changes: 4 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,10 @@
"./extensions/tabs": {
"types": "./dist/extensions/tabs.d.ts",
"import": "./dist/extensions/tabs.js"
},
"./extensions/autolinks": {
"types": "./dist/extensions/autolinks.d.ts",
"import": "./dist/extensions/autolinks.js"
}
},
"scripts": {
Expand Down
28 changes: 28 additions & 0 deletions reports/autolinks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Optional HTTP(S) autolinks

This proposal depends on the inline source parser API. It keeps automatic URL linking opt-in, addressing the core profile’s existing non-goal without changing default rendering. It intentionally does not promise full GFM autolink behavior.

The separate public extension bundle is 994 bytes minified, 632 bytes gzip, and 568 bytes Brotli. All 22 existing measured import shapes are byte-for-byte unchanged in size relative to the inline-parser commit. No runtime dependency or renderer-specific implementation is added.

The revision comparison retains all 403 established CommonMark matches with the extension disabled. The 45 extension tests cover URL punctuation, delimiters, source spelling, code/link exclusions, URL policy, malformed input, serialization, and HTML/React/Octane rendering.

These synthetic measurements use `v26.7.0`, seven alternating warmed rounds, and median milliseconds per render. Enabled and disabled profiles perform different work and can produce different HTML; this is an overhead measurement, not an equivalent-feature renderer comparison. Browser performance is unmeasured.

| Fixture | Core (ms) | Autolinks enabled (ms) |
| --- | ---: | ---: |
| plain comment | 0.00094 | 0.00305 |
| URL comment | 0.00211 | 0.00394 |
| punctuation | 0.00124 | 0.00319 |
| malformed 9000 chars | 0.00336 | 0.13189 |
| malformed 18000 chars | 0.00585 | 0.27066 |
| malformed 36000 chars | 0.01047 | 0.49776 |

The repeated malformed-prefix cases exercise increasing input sizes; the implementation consumes a failed HTTP(S) candidate as one text range instead of retrying every prefix in its suffix. These observations cover this fixture, not arbitrary extension code.

Reproduce:

```sh
pnpm exec tsx scripts/compare-revision.mjs b70affe --no-bench
pnpm exec tsx scripts/bench-autolinks.ts
pnpm run verify
```
Loading