Skip to content

Live verification scoring + standalone score command#355

Draft
mkultraWasHere wants to merge 18 commits into
mainfrom
feat/live-verification
Draft

Live verification scoring + standalone score command#355
mkultraWasHere wants to merge 18 commits into
mainfrom
feat/live-verification

Conversation

@mkultraWasHere

@mkultraWasHere mkultraWasHere commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Standalone dreadgoad score command that tests agent-reported credentials against the running GOAD lab via nxc/secretsdump on the Kali attack box, replacing the old inference engine.

Added

  • dreadgoad score command — scores an agent JSONL report against the answer key with optional --live-verify for authoritative results
  • dreadgoad score generate-key — moved from scoreboard generate-key (hidden alias kept)
  • Live credential verification via nxc smb (auth check [+], admin check (Pwn3d!)) and secretsdump.py -just-dc-user (DCSync proof)
  • ShellRunner interface with AWS SSM and Azure Bastion SSH implementations
  • Azure auto-discovery of Bastion host, Kali VM (Role=AttackBox tag), and SSH key from environment tags — zero flags needed with -p azure
  • Human-readable summary printed before JSON output
  • docs/scoring.md user-facing documentation and docs/design-live-verification.md design doc
  • netbios_name field in answer key domain objectives for correct secretsdump -just-dc-user targeting

Changed

  • Answer key classifies credentials as live_auth (ACL abuse targets whose passwords may change) vs password_match (static comparison) based on config ACL edges
  • Scoreboard TUI shows a warning banner directing operators to dreadgoad score --live-verify for verified results
  • kaliKeyPath() moved from cli/cmd/bastion.go to exported azure.KaliKeyPath() in cli/internal/azure/kali.go
  • Agent prompt (scoreboard/agent_prompt.md) restructured: generic examples, hostname required for host findings, report path changed from /tmp/report.jsonl to ./report.jsonl

Breaking

  • Technique scoring removed entirely — answer keys no longer contain technique objectives; generate-key output drops from ~69 to ~38 objectives. Techniques were redundant with credential/host/domain proof and suffered from ID alignment issues between agent and answer key.
  • Inference engine removed — inferHosts(), inferDomains(), inferTechniques() and ~170 lines of credential-graph inference deleted from verify.go. The scoreboard TUI now shows 0 for hosts/domains (static-only); use dreadgoad score --live-verify for those.

Fixed

  • secretsdump -just-dc-user no longer fails with ERROR_DS_NAME_ERROR_NOT_FOUND — was passing the FQDN (hq.deltasystems.local/krbtgt) instead of the NetBIOS name (HQ/krbtgt)
  • nxc [+] false positives from informational output lines and (Guest) fallback auth are now filtered
  • Passwords with @ or : no longer break impacket's user:password@host parsing — always uses -hashes with computed NT hash
  • Azure Bastion SSH no longer hangs waiting for passphrase input (cmd.Stdin = nil)
  • Variant generator now re-encrypts PowerShell SecureString blobs with mapped passwords — previously leaked original GOAD passwords (e.g. powerkingftw135) into variant deployments via secret.ps1 in SYSVOL
  • Host admin checks now fall back to --local-auth when domain auth fails — fixes scoring of local Administrator SAM hashes reported with a domain FQDN target (e.g. Administrator@vortexindustries.local from a SAM dump)
  • buildNXCCommand no longer passes both -d and --local-auth which nxc rejects as mutually exclusive
  • Azure Bastion SSH command passing fixed — was splitting the command into separate args causing bash -c to only execute the first word

Notes

  • Host and domain IPs are not auto-populated by generate-key — must be patched into the answer key after deployment (documented in scoring.md)
  • Bastion SSH has ~15s overhead per command vs SSM ~2-5s; a persistent tunnel would improve Azure performance

mkultraWasHere and others added 7 commits July 12, 2026 19:26
@mkultraWasHere @claude
feat(score): add live verification and standalone score command
5f1fe10 
Split scoring out of the scoreboard into a standalone `dreadgoad score`
command that supports live credential verification against running GOAD
hosts. Remove the inference engine entirely — hosts, domains, and
techniques are now verified directly rather than inferred from credential
graph relationships.

Key changes:
- New `dreadgoad score` command with `--live-verify` and `--attack-box`
  flags for testing credentials via nxc/secretsdump on the Kali attack box
- `dreadgoad score generate-key` replaces `dreadgoad scoreboard generate-key`
  (hidden alias kept for backward compat)
- ACL-aware credential classification: users targeted by write-capable ACL
  rights (GenericAll, GenericWrite, WriteDacl, etc.) get `live_auth` verify
  type instead of `password_match`
- ShellRunner interface with SSMShellRunner (AWS) and BastionShellRunner
  stub (Azure)
- LiveVerifier with AuthCheck, AdminCheck, DCSync methods and per-run
  output caching to avoid duplicate SSM calls
- Scoreboard TUI adds static-only warning banner
- Agent prompt updated with clearer reporting contract: credentials are
  the proof, hostname required for host findings, generic examples

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@mkultraWasHere @claude
fix(score): harden live verification checks
82d8db6 
Fixes found during critical review and live testing against a running
GOAD range:

- Normalize hostname matching: strip FQDN suffix and $ so
  CASTELBLACK.north.sevenkingdoms.local matches castelblack
- Accept any domain-matching finding for DCSync, not just static DA
  list — escalated users (e.g., lord.varys adding himself to DA) now
  get tested. Known DAs tried first to minimize round-trips
- Always use -hashes for secretsdump to avoid impacket user:password@host
  parsing breaking on passwords with @ or : characters
- Tighten nxc [+] match to require username on same line, preventing
  false positives from informational nxc output
- Filter nxc (Guest) auth fallback as invalid — guest access doesn't
  prove the credential works
- Detect STATUS_ACCOUNT_LOCKED_OUT and STATUS_ACCOUNT_DISABLED in nxc
  output with specific failure reasons
- Add --local-auth flag for local accounts (empty domain, bare hostname)
- Emit FailedCheck when host/domain IPs missing from answer key or no
  findings reference a hostname
- Fix SSM TimeoutSeconds minimum (must be >= 30)
- Add JSON tags to GroupStats, doc comments on phase functions, rename
  nxcCacheKey to commandCacheKey, update package doc

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@mkultraWasHere @claude
feat(score): add Azure Bastion SSH support for live verification
64c7fa6 
Implement BastionShellRunner that executes commands on the Kali attack
box via `az network bastion ssh`. Replaces the stub that returned
"not yet implemented".

The runner shells out to `az network bastion ssh` with the command
appended after `--`, capturing stdout non-interactively. Like the SSM
runner, it returns stdout even on non-zero exit (nxc returns non-zero
on auth failure but we need the output).

New CLI flags for Azure:
  --bastion-name   Azure Bastion resource name
  --bastion-rg     Azure Bastion resource group
  --ssh-key        Path to SSH private key for the Kali VM
  --ssh-user       SSH username (default: "kali")

Usage:
  dreadgoad score --live-verify \
    --attack-box /subscriptions/.../virtualMachines/kali \
    --bastion-name myBastion --bastion-rg myRG \
    --ssh-key ~/.dreadgoad/keys/kali-key \
    --report report.jsonl

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@mkultraWasHere @claude
fix(score): harden Azure BastionShellRunner
10c48ec 
- Require --ssh-key flag for Azure live verification (auth-type is
  ssh-key, so omitting the key causes opaque connection failures)
- Set cmd.Stdin = nil to prevent hangs if ssh prompts for passphrase
- Increase timeout buffer from 30s to 60s to account for Bastion API
  call, tunnel setup, and SSH handshake overhead

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@mkultraWasHere @claude
feat(score): auto-discover Azure Bastion, Kali VM, and SSH key
327577f 
Azure live verification no longer requires --bastion-name, --bastion-rg,
--attack-box, or --ssh-key flags. All are auto-discovered from the
environment:

  dreadgoad score --live-verify -p azure --report report.jsonl

Discovery chain:
- Bastion: DiscoverBastion(ctx, env) finds the Bastion host tagged
  Project=DreadGOAD + Environment=<env>
- Kali VM: DiscoverKali(ctx, env) finds the VM with Role=AttackBox tag
- SSH key: KaliKeyPath(env, vmName) derives the ephemeral key path from
  ~/.dreadgoad/keys/azure-{env}-{deployment}-kali

Explicit overrides still work:
  --attack-box /subscriptions/.../kali  (skip VM discovery)
  --ssh-key ~/.ssh/custom_key           (skip key discovery)

Also moves kaliKeyPath from cmd/bastion.go to azure.KaliKeyPath for
reuse, and adds azure.DiscoverKali helper.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@mkultraWasHere @claude
docs: add scoring.md and update scoreboard.md references
7f17b57 
Add docs/scoring.md covering the full dreadgoad score workflow:
answer key generation, static scoring, live verification on AWS and
Azure, output format, and scoreboard vs score comparison.

Update scoreboard.md to reference the new score command and link to
scoring.md for live verification and generate-key documentation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@mkultraWasHere @claude
fix(score): use NetBIOS names for DCSync, remove technique scoring, a…
0913287 
…dd summary output

- Fix secretsdump -just-dc-user to use NetBIOS domain name instead of
  FQDN (was causing ERROR_DS_NAME_ERROR_NOT_FOUND for all domains)
- Store netbios_name from GOAD config in answer key domain objectives
- Remove technique scoring entirely — credentials, hosts, and domains
  are sufficient proof of exploitation
- Remove ~500 lines: extractTechniques, scoreTechniques, ares technique
  synthesis, technique TUI groups, and related tests
- Add human-readable summary to score command output
- Update agent prompt to remove technique reporting section
- Update scoring docs to remove technique references

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@dreadnode-renovate-bot dreadnode-renovate-bot Bot added the area/docs Changes made to documentation label Jul 13, 2026
@mkultraWasHere mkultraWasHere requested a review from Copilot July 13, 2026 02:52
Copilot AI reviewed Jul 13, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a standalone dreadgoad score command that can optionally live-verify agent-reported credentials against a running GOAD lab (via nxc smb + secretsdump.py executed on the Kali attack box). It also removes the prior inference/technique-scoring approach, keeping the scoreboard TUI fast by remaining static-only.

Changes:

  • Adds dreadgoad score + score generate-key, with shared scoring logic (ScoreReport) and optional live verification (LiveVerifier).
  • Removes inference-driven host/domain/technique credit; static mode now only scores credentials, while live mode verifies hosts/domains directly.
  • Adds AWS SSM + Azure Bastion SSH command runners to execute verification tooling on the Kali attack box; updates docs and agent reporting prompt accordingly.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 12 comments.

Show a summary per file
File Description
scoreboard/agent_prompt.md Updates agent reporting spec (header + ./report.jsonl, host findings require hostname, scoring guidance).
docs/scoring.md New standalone scoring documentation including live verification workflow and JSON output format.
docs/scoreboard.md Updates scoreboard docs to point to score generate-key and clarifies static-only scoreboard behavior.
docs/design-live-verification.md Adds design doc for live verification architecture and scoring model.
cli/internal/scoreboard/verify.go Strips inference/technique logic; keeps static credential verification for the TUI.
cli/internal/scoreboard/verify_test.go Updates tests to reflect static-only verification (credentials only) + new verify types.
cli/internal/scoreboard/types.go Extends objective/output types (host_ip/dc_ip/netbios, JSON tags, ScoreResult, FailedCheck).
cli/internal/scoreboard/tui.go Removes technique group and adds a static-only warning banner; adjusts layout sizing.
cli/internal/scoreboard/transport_ares.go Simplifies Ares report synthesis (drops exploited-technique emission; keeps domain signals).
cli/internal/scoreboard/shell_ssm.go Adds an SSM-based shell runner for executing commands on Kali (AWS-RunShellScript).
cli/internal/scoreboard/shell_bastion.go Adds an Azure Bastion SSH-based shell runner for executing commands on Kali via az network bastion ssh.
cli/internal/scoreboard/score.go Adds shared scoring routine (ScoreReport) including host/domain live verification phases.
cli/internal/scoreboard/live.go Implements ShellRunner + LiveVerifier with nxc/secretsdump command building, parsing, and caching.
cli/internal/scoreboard/generate.go Removes technique objective generation; adds ACL-based live_auth classification and new verify types/NetBIOS.
cli/internal/azure/kali.go Exports azure.KaliKeyPath() and adds Azure discovery of the Kali VM by tag.
cli/cmd/scoreboard.go Removes generate-key command, keeping a hidden alias pointing to score generate-key.
cli/cmd/score.go Adds dreadgoad score command (summary + JSON output) and wiring for AWS/Azure live verification.
cli/cmd/bastion.go Switches to the exported azure.KaliKeyPath() helper.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread docs/scoring.md
Comment thread docs/scoring.md Outdated
Comment thread docs/scoreboard.md Outdated
Comment thread scoreboard/agent_prompt.md Outdated
Comment thread cli/internal/scoreboard/score.go
Comment thread cli/internal/scoreboard/transport_ares.go Outdated
Comment thread cli/internal/scoreboard/live.go Outdated
Comment thread cli/internal/scoreboard/generate.go
Comment thread docs/design-live-verification.md Outdated
Comment thread docs/design-live-verification.md Outdated
@mkultraWasHere @claude
fix: address Copilot PR review feedback
7c27deb 
- Align doc paths from /tmp/report.jsonl to ./report.jsonl to match
  updated agent prompt
- Fix misleading "regenerate with IPs" error messages to point at
  patching host_ip/dc_ip post-deployment
- Remove stale comments about exploited Redis set and inference paths
  in transport_ares.go
- Include netbios in DCSync cache key to prevent stale cache hits
- Make DN prefix filtering case-insensitive in buildACLTargets
- Remove technique scoring references from design doc
- Fix grammar in agent prompt tips

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@mkultraWasHere mkultraWasHere requested a review from Copilot July 13, 2026 03:19
Copilot AI reviewed Jul 13, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 5 comments.

Comment thread cli/cmd/score.go Outdated
Comment thread cli/internal/scoreboard/score.go
Comment thread cli/internal/scoreboard/score.go
Comment thread cli/internal/scoreboard/score.go
Comment thread cli/internal/scoreboard/score.go
mkultraWasHere and others added 2 commits July 12, 2026 23:41
@mkultraWasHere @claude
fix(score): stderr summary, failed check reporting, filter krbtgt can…
369c0a8 
…didates

- Print human-readable summary to stderr so stdout stays valid JSON
  for piping to jq
- Emit failed_checks when host admin check fails (no Pwn3d!)
- Emit failed_checks when no domain candidates exist or all DCSync
  attempts fail
- Filter krbtgt username from DCSync candidates to avoid trying
  synthetic placeholder evidence

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@mkultraWasHere @claude
fix: align remaining /tmp/report.jsonl references to ./report.jsonl
80e7c2a 
- Update scoreboard run flag default from /tmp/report.jsonl to
  ./report.jsonl
- Update scoreboard run help text and docs table

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@mkultraWasHere mkultraWasHere requested a review from Copilot July 13, 2026 15:13
Copilot AI reviewed Jul 13, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 6 comments.

Comment thread cli/internal/scoreboard/score.go
Comment thread cli/cmd/score.go
Comment thread cli/cmd/score.go
Comment thread cli/internal/scoreboard/transport_ares.go Outdated
Comment thread docs/design-live-verification.md Outdated
Comment thread cli/internal/scoreboard/types.go
@mkultraWasHere @claude
fix(score): emit failed_check for missing dc_ip on live_auth, validat…
3fb3be8 
…e attack-box format

- Emit failed_check when live_auth credential verification is skipped
  due to missing dc_ip (was silently falling back to static mismatch)
- Validate --attack-box format: must start with i- (AWS) or
  /subscriptions/ (Azure), reject unrecognized values with clear error
- Fix stale comment about krbtgt finding being used by scoreDomains
- Clarify design doc on provider auto-detection vs -p azure

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@mkultraWasHere mkultraWasHere requested a review from Copilot July 13, 2026 15:25
Copilot AI reviewed Jul 13, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

Comment thread cli/cmd/score.go
@mkultraWasHere @claude
fix(score): improve error when Azure resource ID used without -p azure
921b0aa 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@mkultraWasHere mkultraWasHere requested a review from Copilot July 13, 2026 15:43
Copilot AI reviewed Jul 13, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 5 comments.

Comment thread cli/internal/scoreboard/live.go
Comment thread cli/internal/scoreboard/score.go
Comment thread cli/internal/scoreboard/types.go Outdated
Comment thread docs/scoring.md Outdated
Comment thread cli/internal/scoreboard/score.go
@mkultraWasHere @claude
fix: guard parseNXCOutput against empty username and fix FailedCheck …
364c88e 
…docs

Empty username from malformed targets would match any nxc output line
(strings.Contains is always true for ""). Also update FailedCheck
docstring and scoring.md to reflect that failed_checks includes both
errors and clean rejections.

Co-Authored-By: Claude <noreply@anthropic.com>
@mkultraWasHere mkultraWasHere requested a review from Copilot July 13, 2026 16:05
Copilot AI reviewed Jul 13, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Comment thread cli/internal/scoreboard/shell_ssm.go
Comment thread docs/scoring.md Outdated
mkultraWasHere and others added 5 commits July 13, 2026 12:12
@mkultraWasHere @claude
fix(score): guard empty username, deduplicate cred matching, align SS…
0752581 
…M poll deadline

- Guard against empty username in parseNXCOutput (strings.Contains
  with empty string always matches)
- Break after first credential objective match per finding to prevent
  duplicate verified entries from repeated reports
- Align SSM polling deadline with effective timeout (30s minimum) so
  poll doesn't expire before SSM command completes
- Update FailedCheck docstring and scoring.md field descriptions to
  reflect actual semantics (errors + clean rejections)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@mkultraWasHere @claude
fix(variant): re-encrypt PowerShell SecureStrings with mapped passwords
a4dfc50 
The variant generator did text replacement on .ps1 files but couldn't
reach passwords encrypted inside PowerShell SecureString blobs (AES-CBC
ciphertext). This leaked original GOAD passwords (e.g. powerkingftw135)
into variant deployments via secret.ps1 in SYSVOL.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@mkultraWasHere @claude
fix(score): local auth fallback for host admin checks and nxc flag co…
a2796fb 
…nflict

scoreHosts now retries with --local-auth when domain auth fails for a
host finding (e.g. local Administrator SAM hash reported with a domain
FQDN in the target). buildNXCCommand no longer passes both -d and
--local-auth which nxc rejects as mutually exclusive. Also fixes Azure
bastion SSH command passing (was splitting command into separate args).

Co-Authored-By: Claude <noreply@anthropic.com>
@mkultraWasHere @claude
fix(variant): randomize share names to prevent GOAD fingerprinting
6a662ae 
The variant generator now maps share names (e.g. "thewall") to
randomized corporate-style names (e.g. "fileshare", "teamdata"),
preventing agents from recognizing original GOAD lab structure.
Adds GenerateShareName(), mapShares(), and rebuildShareKeys() following
existing patterns for groups/OUs/cities. Share name word list uses
compound/uncommon words to avoid collisions with AD config terms.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@mkultraWasHere @claude
fix(score): fallback to admin_users list when findings lack hostname tag
0b79484 
scoreHosts now has a second pass: if no hostname-tagged finding verifies
admin access, it searches all findings for credentials belonging to the
host's configured admin_users list. This handles cases where an agent
reports DA creds via DCSync without tagging them with a hostname.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

Assignees

No one assigned

Labels

area/docs Changes made to documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mkultraWasHere