Hashes for artifacts

[falcon217836]

Referencing issue #457, this submission will generate hash values for all artifacts processed.

[stark4n6]

@falcon217836 this appears to ingest all files in a path/container and hashes, is that correct? If so how are the speeds? I would imagine that would take a bit of time

[falcon217836]

@stark4n6 That is correct. And processing was much quicker than I expected. 30K files processed in:

Fs: 5-15 sec
Zip: 10-20 sec
Tar: 10-20 sec
gz: 10-20 sec

There was an additional 15-20 sec required when loading the html report for the artifacts hashed. All in all not too bad for hashing 30K files.

[stark4n6]

@falcon217836 I ran it on a test CTF image from MVS2024 and it took 22 minutes just with this parser turned on. I got 69k entries for results with over thousands erroring out

I don't know if this is optimal on a fully utilized FFS extraction. Also the temp folder is almost 22gb

[falcon217836]

@stark4n6 It wasn't nearly that exhaustive when I ran it in Ubuntu2204, but this brings up something I hadn't considered. Many use ALEAPP on Windows systems so let me make some adjustments, run some regression testing in a Winenv, and I'll update you when I've got a more efficient commit.

[JamesHabben]

moving to draft for now. where are we at with this? can the speed be improved to have minimal impact? if not but we still want to merge this in, we probably need to gate it behind a cmd line parameter or GUI check to enable - off by default

[abrignoni]

Thanks for the contribution!

This PR changes artifact modules without test data for them. A small fixture with each artifact change lets reviewers run the module against real data, and the committed case keeps guarding the module after merge.

• hashvals.py: please include a fixture with this PR.

Adding a fixture

Generate it from your extraction with the helper (details in create_module_test_cases.md):

python admin/test/scripts/make_test_data.py <module> --case <case_number> --input <extraction.zip>

It writes admin/test/cases/testdata.<module>.json and one zip per artifact under admin/test/cases/data/<module>/.

Size rules:

• Under 10 MB per zip: commit the files in this PR.
• 10 to 25 MB: commit the case JSON in the PR and attach the zip to a comment here.
• Over 25 MB: say so here and a maintainer will arrange a handoff.

If your extraction cannot be shared:

• If the app appears on a public research image, generate the fixture from that instead. public_corpus_images.md lists the images and where to download them.
• Or sanitize the real file in place: keep the file the app wrote and overwrite only the personal values, which keeps the format honest.
• Or script a known session: install the app on a test device with a throwaway account, perform documented actions, and extract that.

If none of those fit, say so here and we will work it out. The PR can still be reviewed and merged with the gap recorded in the artifact's notes.

This is a request, not a gate. Nothing here blocks review.