Skip to content

feat(irr): add IRR data support with RPSL parsing and AS info enrichment - #39

Open
digizeph wants to merge 3 commits into
mainfrom
feat/irr-data-support
Open

feat(irr): add IRR data support with RPSL parsing and AS info enrichment#39
digizeph wants to merge 3 commits into
mainfrom
feat/irr-data-support

Conversation

@digizeph

Copy link
Copy Markdown
Member

Summary

Add IRR (Internet Routing Registry) data support to bgpkit-commons, including a new irr module for parsing RPSL data and extending asinfo with structured IRR and delegated-stats enrichment.

New irr module

Library-level RPSL parsing toolkit using rpsl-rs v3.0:

  • 14 known IRR registries (RIPE, APNIC, ARIN, LACNIC, AFRINIC, NTTCOM, RADB, + 7 more) with per-registry dump URLs, transport (HTTPS/FTP), and format (split files vs whole-DB)
  • Typed structs for 7 RPSL object types: AutNum, Route, Route6, AsSet, RouteSet, Mntner, Organisation
  • Streaming parser via oneio with lossy UTF-8 handling for legacy IRR data
  • Single-pass download for whole-DB sources (RADB, ARIN, etc. downloaded once, all object types extracted)

Extended asinfo module

  • DelegatedInfo struct: structured RIR delegated-stats data (registry, country, date, status) for every allocated ASN
  • IrrAsnInfo struct: per-source IRR data (as-name, descr, mnt-by, route prefixes, route6 prefixes, as-set memberships) stored as a Vec so callers pick which source(s) to trust
  • AsInfoProfile enum with three presets:
    • Minimum: asn.txt only (~1s load)
    • Default: + as2org, population, hegemony, peeringdb (~30s) — matches current production
    • Full: + delegated stats + IRR data with route prefixes (~75s)
  • AsInfoBuilder for fine-grained control: .with_irr_sources(), .with_irr_route_prefixes(), etc.
  • Route prefix collection is opt-in — off by default, saving ~90MB in serialized output
  • Unknown ASN names filled from IRR aut-num as-name (363 remaining UNKNOWN out of ~999 gap ASNs)

Breaking changes

  • load_asinfo(bool, bool, bool, bool)load_asinfo_with_profile(AsInfoProfile)
  • AsInfo gains delegated: Option<DelegatedInfo> and irr: Vec<IrrAsnInfo> fields
  • asinfo feature now depends on irr feature

Performance (live data, release build)

Source Time
asn.txt only ~1s
Delegated stats (5 RIRs) ~12s
IRR RIPE (4 split files) ~7s
IRR AFRINIC (whole-DB) ~18s (slowest)
IRR RADB (FTP, 25MB) ~10s
All IRR combined ~43s
Full profile ~73s

Output sizes

Profile Raw JSONL Gzipped
Production (current) 37 MB 6.3 MB
Full, no routes 160 MB 24 MB
Full, with routes 203 MB 31 MB

Test plan

  • 48 lib unit tests pass (including 6 delegated-stats parser tests, 4 IRR streaming tests)
  • 4 non-network integration tests pass (source registry coverage, URL correctness, transport verification)
  • Live network tests verified: RIPE aut-num (AS13335=Cloudflare, AS15169=Google, AS1299=Arelion), RIPE routes, RADB FTP routes (1.1.1.0/24 via AS13335), ARIN whole-DB, APNIC
  • cargo fmt --check clean
  • cargo clippy --all-features -- -D warnings clean
  • cargo build --no-default-features and --features irr both compile

Closes #38

digizeph and others added 2 commits July 30, 2026 10:23
Add a new irr module and extend asinfo with IRR and delegated-stats
data. Uses rpsl-rs (v3.0) for RPSL parsing and oneio for remote/FTP
downloads.

New irr module (library-level toolkit):
- sources.rs: 14 known IRR registries (RIPE, APNIC, ARIN, LACNIC,
  AFRINIC, NTTCOM, RADB, ALTDB, BELL, BBOI, JPIRR, TC, CANARIE, REACH)
  with per-registry dump URLs, transport (HTTPS/FTP), and format
  (split files vs whole-DB)
- types.rs: Typed structs for 7 RPSL object types (AutNum, Route,
  AsSet, RouteSet, Mntner, Organisation) with serde serialization
- extract.rs: Extracts typed objects from rpsl::Object<Raw>, handles
  comma-separated members, case-insensitive ASN parsing, inline comments
- stream.rs: Streaming parser via oneio with read_until for UTF-8
  robustness, blank-line object splitting, lossy conversion for legacy
  IRR data with non-UTF-8 bytes

Extended asinfo module:
- DelegatedInfo: structured RIR delegated-stats data (registry,
  country, date, status) attached to every allocated ASN
- IrrAsnInfo: per-source IRR data (as-name, descr, mnt-by, route
  prefixes, route6 prefixes, as-set memberships) as a Vec so callers
  pick which source(s) to trust
- AsInfoProfile enum (Minimum, Default, Full) for curated presets
- AsInfoBuilder retained for fine-grained control: with_delegated(),
  with_irr(), with_irr_sources(IrrSourceConfig), with_irr_route_prefixes()
- IRR route prefix collection is opt-in (off by default) to control
  output size (~90MB savings when disabled)
- Unknown ASN names filled from IRR aut-num as-name when available
  (363 remaining UNKNOWN out of ~999 delegated-stats gap ASNs)

Performance (release build, live data):
- asn.txt only: ~1s
- Default profile: ~30s
- Full profile: ~75s (AFRINIC 17.5s, RADB 10.3s are slowest sources)
- Whole-DB sources download once (single-pass extraction)
- Split-file sources skip route/route6 files when prefixes not requested

Output sizes:
- Minimum: ~37 MB JSONL (6.3 MB gzipped)
- Default: ~50 MB JSONL
- Full: ~210 MB JSONL (31 MB gzipped)

Integration tests verified with live IRR data: RIPE aut-num (AS13335,
AS15169, AS1299), RIPE routes, RADB FTP routes, ARIN whole-DB, APNIC.

Closes #38
@digizeph
digizeph marked this pull request as ready for review July 30, 2026 23:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add IRR (RPSL) data support: aut-num names, route objects, as-set resolution

1 participant