Add credential redaction and markdown export#31
Open
jrzmurray wants to merge 2 commits into
Open
Conversation
- Implement PropertyID.type 0x10 (ArrayOfPropertyValues), which was a bare `raise NotImplementedError`. Any section containing a page with an array-typed property failed to parse at all, silently dropping all text in that section. - Handle ObjectDeclaration2LargeRefCountFND file nodes, which had no matching branch in FileNode.__init__ at all. self.data was never set for such nodes, which (a) made their content invisible to get_properties() and (b) crashed with "'FileNode' object has no attribute 'data'" if the node had baseType == 2 (i.e. had children). Also load propertySet for the two ReadOnlyObjectDeclaration2*RefCountFND variants, which set self.data but never parsed out the property set, and broaden OneDocment.get_properties()'s hardcoded single-type filter to include all four declaration node types. - Decode the TextExtendedAscii property as latin-1 instead of utf-16. OneNote for Mac stores plain-text runs under this single-byte-per-character property instead of Windows OneNote's RichEditTextUnicode (UTF-16). Decoding it as utf-16 either garbles it into CJK-looking noise (even byte count) or throws and falls back to a raw hex dump (odd byte count) -- in both cases the original text is unrecoverable from the output. Tested against a real, multi-year OneNote notebook with pages authored on both Windows and Mac OneNote clients. Sections that previously extracted 0 characters (crashing on the ArrayOfPropertyValues error) now extract hundreds of KB to over 1MB of real text each; Mac-authored sections that previously returned only bare page timestamps (no body text at all) now extract their full text content.
…odes Builds on the page-text-extraction fixes in the preceding commit (PR DissectMalware#29), which this feature depends on: without the TextExtendedAscii latin-1 decode, Mac-authored sections are garbled by the existing UTF-16 decode before the exporter ever sees them. Markdown export (-m/--markdown): - OneDocument.get_pages()/get_markdown() reconstruct de-duplicated page text. OneNote keeps every historical revision of a page inline, so the raw property stream repeats each page 2-20x; pages are segmented on jcidPageNode, titled via their CachedTitleString, and collapsed to one entry each. - Text runs are read per platform -- RichEditTextUnicode (Windows) and TextExtendedAscii (OneNote for Mac) -- and joined into clean markdown, one .md per section. Redaction (-r/--redact-secrets), off by default: - Replaces high-confidence credential shapes with visible [redacted:*] markers. - Applies to ALL output modes -- the default dump, JSON (-j), markdown (-m), and text attachments written to --output-dir. Binary attachments have no text to scrub and are written unchanged. - Matching is limited to prefix/structure-keyed vendor sentinels (AWS, GitHub, GitLab, OpenAI/Anthropic, Slack, Stripe, Google, npm, PyPI, SendGrid, Twilio, JWT, Bearer) and PEM PRIVATE KEY blocks. Public certificates, public keys, and CSRs are shareable and deliberately left intact, and there is no generic entropy/keyword/base64 detection, so ordinary note text and embedded data are untouched. tests/test_redact.py: stdlib-unittest coverage for full-span redaction, public material preservation, nested-tree scrubbing, and binary passthrough. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
An opt-in credential redaction pass that applies to every output mode, and a markdown text export for
.onesections.Redaction (
-r, off by default)Replaces credentials with visible
[redacted:*]markers. Deliberately precise rather than aggressive:Bearer) and PEMPRIVATE KEYblocks.Coverage is complete, not partial.
-rapplies to the default dump, JSON (-j), markdown (-m), and text attachments written to--output-dir. Binary attachments (images, Office docs) have no text to scrub and are written byte-identical. The redaction summary goes to stderr so it cannot corrupt JSON on stdout.Markdown export (
-m)There was no direct path to the plain readable text of a notebook, and revision noise made naive extraction unusable: OneNote keeps every historical revision of a page inline, so a section's property stream repeats each page 2-20x.
get_pages()segments onjcidPageNode, titles each page via itsCachedTitleString, and collapses revisions to one entry each (de-dup is scoped per title, so genuinely distinct pages that share text are preserved). Text runs are read per platform —RichEditTextUnicode(Windows) andTextExtendedAscii(Mac) — and joined into one.mdper section.Tests
tests/test_redact.py— stdlibunittest, no new dependencies:Covers full-span key redaction (a partial match that leaves half a live key behind is worse than none), public-certificate preservation, nested property-tree scrubbing, and binary attachment passthrough.
Scope
Both flags are additive and opt-in; no existing output or behavior changes.