Skip to content

Add credential redaction and markdown export#31

Open
jrzmurray wants to merge 2 commits into
DissectMalware:mainfrom
jrzmurray:markdown-export
Open

Add credential redaction and markdown export#31
jrzmurray wants to merge 2 commits into
DissectMalware:mainfrom
jrzmurray:markdown-export

Conversation

@jrzmurray

Copy link
Copy Markdown

Builds on #29. That PR's page-text-extraction fixes are included in this branch and are a hard dependency: without the TextExtendedAscii latin-1 decode, Mac-authored sections are garbled by the existing UTF-16 decode before the exporter ever sees them (~16% of output comes back as CJK noise and hex dumps). Merge #29 first and this diff reduces to just the feature.

What this adds

An opt-in credential redaction pass that applies to every output mode, and a markdown text export for .one sections.

pyonenote -r -f section.one            # default dump, redacted
pyonenote -r -j -f section.one         # JSON, redacted
pyonenote -m -f notebook/ -o out/      # export section text to markdown
pyonenote -r -m -f notebook/ -o out/   # markdown, redacted

Redaction (-r, off by default)

Replaces credentials with visible [redacted:*] markers. Deliberately precise rather than aggressive:

  • Matched: prefix/structure-keyed vendor sentinels (AWS, GitHub, GitLab, OpenAI/Anthropic, Slack, Stripe, Google, npm, PyPI, SendGrid, Twilio, JWT, Bearer) and PEM PRIVATE KEY blocks.
  • Deliberately not matched: public certificates, public keys, and CSRs — they are meant to be shared and carry no secret. And there is no generic entropy, keyword-proximity, or base64/blob detection, so ordinary note text, identifiers, and embedded data (images, encoded content) are untouched. The pass is effectively free of false positives.

Coverage is complete, not partial. -r applies to the default dump, JSON (-j), markdown (-m), and text attachments written to --output-dir. Binary attachments (images, Office docs) have no text to scrub and are written byte-identical. The redaction summary goes to stderr so it cannot corrupt JSON on stdout.

Markdown export (-m)

There was no direct path to the plain readable text of a notebook, and revision noise made naive extraction unusable: OneNote keeps every historical revision of a page inline, so a section's property stream repeats each page 2-20x. get_pages() segments on jcidPageNode, titles each page via its CachedTitleString, and collapses revisions to one entry each (de-dup is scoped per title, so genuinely distinct pages that share text are preserved). Text runs are read per platform — RichEditTextUnicode (Windows) and TextExtendedAscii (Mac) — and joined into one .md per section.

Tests

tests/test_redact.py — stdlib unittest, no new dependencies:

python3 -m unittest discover -s tests

Covers full-span key redaction (a partial match that leaves half a live key behind is worse than none), public-certificate preservation, nested property-tree scrubbing, and binary attachment passthrough.

Scope

Both flags are additive and opt-in; no existing output or behavior changes.

jrzmurray and others added 2 commits July 10, 2026 16:27
- Implement PropertyID.type 0x10 (ArrayOfPropertyValues), which was a
  bare `raise NotImplementedError`. Any section containing a page with
  an array-typed property failed to parse at all, silently dropping
  all text in that section.

- Handle ObjectDeclaration2LargeRefCountFND file nodes, which had no
  matching branch in FileNode.__init__ at all. self.data was never
  set for such nodes, which (a) made their content invisible to
  get_properties() and (b) crashed with
  "'FileNode' object has no attribute 'data'" if the node had
  baseType == 2 (i.e. had children). Also load propertySet for the
  two ReadOnlyObjectDeclaration2*RefCountFND variants, which set
  self.data but never parsed out the property set, and broaden
  OneDocment.get_properties()'s hardcoded single-type filter to
  include all four declaration node types.

- Decode the TextExtendedAscii property as latin-1 instead of
  utf-16. OneNote for Mac stores plain-text runs under this
  single-byte-per-character property instead of Windows OneNote's
  RichEditTextUnicode (UTF-16). Decoding it as utf-16 either garbles
  it into CJK-looking noise (even byte count) or throws and falls
  back to a raw hex dump (odd byte count) -- in both cases the
  original text is unrecoverable from the output.

Tested against a real, multi-year OneNote notebook with pages
authored on both Windows and Mac OneNote clients. Sections that
previously extracted 0 characters (crashing on the ArrayOfPropertyValues
error) now extract hundreds of KB to over 1MB of real text each;
Mac-authored sections that previously returned only bare page
timestamps (no body text at all) now extract their full text content.
…odes

Builds on the page-text-extraction fixes in the preceding commit (PR DissectMalware#29),
which this feature depends on: without the TextExtendedAscii latin-1 decode,
Mac-authored sections are garbled by the existing UTF-16 decode before the
exporter ever sees them.

Markdown export (-m/--markdown):
- OneDocument.get_pages()/get_markdown() reconstruct de-duplicated page text.
  OneNote keeps every historical revision of a page inline, so the raw property
  stream repeats each page 2-20x; pages are segmented on jcidPageNode, titled
  via their CachedTitleString, and collapsed to one entry each.
- Text runs are read per platform -- RichEditTextUnicode (Windows) and
  TextExtendedAscii (OneNote for Mac) -- and joined into clean markdown, one
  .md per section.

Redaction (-r/--redact-secrets), off by default:
- Replaces high-confidence credential shapes with visible [redacted:*] markers.
- Applies to ALL output modes -- the default dump, JSON (-j), markdown (-m), and
  text attachments written to --output-dir. Binary attachments have no text to
  scrub and are written unchanged.
- Matching is limited to prefix/structure-keyed vendor sentinels (AWS, GitHub,
  GitLab, OpenAI/Anthropic, Slack, Stripe, Google, npm, PyPI, SendGrid, Twilio,
  JWT, Bearer) and PEM PRIVATE KEY blocks. Public certificates, public keys, and
  CSRs are shareable and deliberately left intact, and there is no generic
  entropy/keyword/base64 detection, so ordinary note text and embedded data are
  untouched.

tests/test_redact.py: stdlib-unittest coverage for full-span redaction, public
material preservation, nested-tree scrubbing, and binary passthrough.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jrzmurray jrzmurray changed the title Add credential redaction across all output modes, plus markdown export Add credential redaction and markdown export Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant