H5json by jreadey · Pull Request #420 · HDFGroup/hsds

jreadey · 2025-04-23T17:56:04Z

Use h5json package for typing and objids

Important

Migrated HSDS to use the h5json library for core utilities, restructured utility modules, added support for client-provided object IDs and timestamps, and updated dependencies to require Python 3.10+ with h5json 1.0.0+.

Library Migration and Utility Restructuring

h5json Library Integration: Migrated from local utility modules to h5json library for data type, array, object ID, shape, dataset, filter, link, and time utilities across 30+ files.
Deleted Utility Modules: Removed hsds/util/idUtil.py, hsds/util/timeUtil.py, hsds/util/hdf5dtype.py, and hsds/util/arrayUtil.py as their functionality is now provided by h5json.
New Utility Module: Created hsds/util/nodeUtil.py with node ID generation, partitioning, and datanode URL resolution functions.
Updated Imports: Changed all references from local util modules to h5json equivalents (e.g., util.idUtil â�� h5json.objid, util.timeUtil â�� h5json.time_util).

Object ID and Timestamp Handling

Client-Provided Object IDs: Added support for creating objects with client-specified IDs in POST_Dataset, POST_Group, POST_Datatype, and related functions in dset_dn.py, group_dn.py, ctype_dn.py, and dset_sn.py.
Timestamp Validation: Added max_timestamp_drift configuration parameter to validate client-provided timestamps in attr_dn.py, link_dn.py, and related modules, with fallback to server-generated timestamps when skew exceeds threshold.
Deleted Object Tracking: Added logic to check and remove previously deleted object IDs from deleted_ids set when creating new objects with the same ID.

Configuration and Dependencies

New Configuration Parameters: Added default_vlen_type_size, predate_maxtime, posix_delay, max_compact_dset_size, and max_timestamp_drift to admin/config/config.yml.
Updated Dependencies: Modified pyproject.toml to require Python 3.10+, add h5json 1.0.0+, update numpy to 2.0.0+, and constrain numcodecs to â�¤0.15.1.
Removed Python 3.9: Removed Python 3.9 from CI/CD test matrix in .github/workflows/python-package.yml.

API and Function Refactoring

Object Creation Functions: Refactored POST_Dataset, POST_Group, and POST_Datatype handlers to support batch creation of multiple objects using new helper functions (createDatasets, createGroups, createDatatypeObjs) and DomainCrawler for writing initial data.
Layout Handling: Changed getChunkLayout calls to getChunkDims throughout codebase; moved layout from top-level response to nested under creationProperties.
Link Handling: Changed external link field from h5domain to file in link_dn.py, link_sn.py, and servicenode_lib.py; added per-link timestamp validation in PUT_Links.
Attribute Initialization: Added support for initializing attributes from request body in POST_Dataset, POST_Group, and POST_Datatype instead of always creating empty objects.

New Functionality

PostCrawler Class: Added hsds/post_crawl.py with PostCrawler class for asynchronously creating multiple HDF5 objects with configurable worker count and error handling.
Domain Metadata Consolidation: Added getConsolidatedMetaData function in async_lib.py to create consolidated metadata summaries for all objects in a domain.
Data Writing: Added put_data method to DomainCrawler for writing one-chunk dataset values; added doPointWrite and doHyperslabWrite functions in dset_lib.py for writing point and hyperslab selections.
Domain Objects Retrieval: Added getobjs parameter to getDomainResponse function to optionally return domain objects from S3 summary file.

Bug Fixes and Improvements

Typo Fixes: Fixed multiple typos including "coniguous" â�� "contiguous", "seperated" â�� "separated", "heirarchy" â�� "hierarchy", "inital" â�� "initial", and various attribute/link-related typos.
Error Handling: Changed error responses from HTTPInternalServerError to HTTPBadRequest for duplicate object IDs and invalid configurations in ctype_dn.py, dset_dn.py, and group_dn.py.
Logging Improvements: Added debug logging for request bodies, object creation, and metadata processing; updated log message prefixes for consistency.
POSIX Delay Support: Added posix_delay configuration support to fileClient.py for simulating cloud storage latencies in get_object, put_object, and list_keys methods.
Version Update: Updated HSDS_VERSION from 0.9.2 to 1.0.0 in basenode.py.

Test Updates

New Test Methods: Added tests for client-provided object IDs (testPostDatasetWithId, testPostTypeWithId, testPostWithId), attribute initialization (testPostDatasetWithAttributes, testPostWithAttributes), timestamp handling (testUseTimestamp), and batch creation (testPostMulti, testDatasetPostMulti).
Test Refactoring: Updated tests to access layout from creationProperties instead of top-level; removed CHUNK_MIN/CHUNK_MAX constants and moved them to local scope; updated external link tests to use file field instead of h5domain.
Removed Tests: Deleted array_util_test.py, hdf5_dtype_test.py, and id_util_test.py as their functionality is now tested through h5json library.
Import Updates: Updated test imports to use h5json functions (e.g., createObjId, getFilterItem) instead of local utilities.

^{This description was created by}^{for 2bafb51. You can customize this summary. It will automatically update as commits are pushed.}

+def _getIdHash(id):
+    """Return md5 prefix based on id value"""
+    m = hashlib.new("md5")
+    m.update(id.encode("utf8"))


To fix the issue, replace the use of the MD5 hashing algorithm in _getIdHash with a stronger algorithm, such as SHA-256. This ensures that the hash function is resistant to pre-image and collision attacks. The change involves:

Updating the _getIdHash function to use hashlib.sha256 instead of hashlib.new("md5").

Ensuring that the rest of the code remains functional by keeping the truncation to the first 5 characters of the hash.

No additional imports are required since hashlib already supports SHA-256.

mattjala · 2025-05-07T14:56:23Z

Since we now depend on hdf5-json to do this testing, it might be a good idea to include hdf5-json's tests as a step in the CI

mattjala

Besides a few minor comments and questions, this is good to go in. I'll try to get the outstanding PRs on hdf5-json reviewed this week so that we can avoid having HSDS depend on a specific branch.

mattjala · 2025-05-07T21:34:07Z

+                            created = link_item["created"]
+                            # allow "pre-dated" attributes if recent enough
+                            predate_max_time = config.get("predate_max_time", default=10.0)
+                            if now - created > predate_max_time:


This comparison seems backwards. If I understand correctly, the difference between current time and creation time should need to be under the max time, not above it

mattjala · 2026-08-14T20:45:35Z

 azure-storage-blob
 cryptography
 h5py>=3.6.0
+git+https://github.com/HDFGroup/hdf5-json.git@abstract#egg=h5json


It's fine to leave this as a git ref during development, but it needs to be changed to a real version after merge/release

mattjala · 2026-08-14T20:45:50Z

    "bitshuffle >=0.5.2",
    "cryptography",
    "h5py >= 3.6.0",
+    "h5json@git+https://github.com/HDFGroup/hdf5-json@abstract",


It's fine to leave this as a git ref during development, but it needs to be changed to a real version after merge/release

mattjala · 2026-08-14T20:56:25Z

+from h5json.array_util import bytesToArray, squeezeArray, getBroadcastShape
+from h5json.objid import isValidUuid
+from h5json.shape_util import isNullSpace, isScalar, getShapeDims, getMaxDims, getRank
+from h5json.dset_util import getChunkDims, isExtensible


isExtensible is defined in h5json's shape_util, and then imported into h5json's dset_util. HSDS should import it directly from shape_util.

mattjala · 2026-08-14T20:57:36Z

+from h5json.hdf5dtype import getBaseTypeJson, validateTypeItem, createDataType, getItemSize
+from h5json.shape_util import getShapeDims, getShapeClass, getShapeJson
+from h5json.dset_util import getChunkSize, generateLayout
+from h5json.dset_util import getDataSize, validateDatasetCreationProps


getDataSize is defined in h5json's shape_util, and then h5json imports it into dset_util. HSDS should import getDataSize directly from h5json.shape_util.

mattjala · 2026-08-14T21:43:06Z

        uses: actions/checkout@v4
        with:
          repository: HDFGroup/h5pyd
          path: ${{github.workspace}}/h5pyd


Between the merging of hdf5-json's new branch into master and the associated/follow-up HSDS and h5pyd release, this should be changed to point at h5pyd's h5json branch.

mattjala · 2026-08-21T15:54:22Z

 ]
-requires-python = ">=3.8"
+requires-python = ">=3.10"
 version = "0.9.2"


Should be bumped to 1.0.0 for consistency

mattjala · 2026-08-21T16:17:23Z

@@ -1,2 +1,2 @@
 aiohttp==3.9.4
 aiobotocore==2.13.0


The requirements.txt in this branch should be updated to reflect all the dependency updates that have gone into master

mattjala · 2026-08-21T16:20:00Z

@@ -0,0 +1,2973 @@
+openapi: 3.1.0


The README points at HDFGroup/hdf5-rest-api as an authoritative description of the API, but it's now out of date. We should remove any references to it and flag it as out of date.

mattjala · 2026-08-21T16:25:40Z

 # link related functions
 #
+from h5json.time_util import getNow
+from h5json.link_util import validateLinkName, getLinkClass, getLinkPath, getLinkFilePath


validateLinkName and isEqualLink moved to h5json.link_util, but their analogues for attributes (validateAttributeName and isEqualAttr) stayed behind in hsds/util/attrUtil.py. Is there a reason the two are split differently?

mattjala · 2026-08-21T16:26:41Z

 max_rangeget_gap: 1024 # max gap in byte for intelligent range get requests
+predate_maxtime: 10.0 # max delta between object created timestamp in request and actual time
+posix_delay: 0.0  # delay for POSIX IO operations for simulating cloud storage latencies
+max_compact_dset_size: 65536  # size in bytes for maximum compact storage size


It seems that nothing ever uses this field.

mattjala · 2026-08-21T16:28:19Z

 allow_any_bucket_write: true # enable writes to buckets other than default bucket
 bit_shuffle_default_blocksize: 2048 # default blocksize for bitshuffle filter
 max_rangeget_gap: 1024 # max gap in byte for intelligent range get requests
+predate_maxtime: 10.0 # max delta between object created timestamp in request and actual time


All the places that try to use this field (servicenode_lib.py:1059, servicenode_lib.py:1347 and link_sn.py:301 spell it as predate_max_time instead of the correct predate_maxtime, find a missing key, and always get the default value.

mattjala · 2026-08-21T16:29:26Z

+    """ create dataset objects based on parameters in items list """
+
+    if not root_id:
+        msg = "no root_id given for createDatatypeObjs"


Incorrect function name in error msg - should be createDatasets

mattjala · 2026-08-21T16:30:03Z

+    post_crawler = PostCrawler(app, root_id=root_id, bucket=bucket, items=items)
+    await post_crawler.crawl()
+    if post_crawler.get_status() > 201:
+        msg = f"createGroups returning status from crawler: {post_crawler.get_status()}"


Incorrect function name in error msg - should say _createObjects returning status...

mattjala · 2026-08-21T16:30:30Z

+
+    obj_list = post_crawler.get_rsp_objs()
+    if not isinstance(obj_list, list):
+        msg = f"createGroups expected list but got: {type(obj_list)}"


Incorrect function name in error msg - should be _createObjects expected...

mattjala · 2026-08-21T16:32:57Z

+requires-python = ">=3.10"
 version = "0.9.2"

 dependencies = [


pyproject/requirements.txt is missing openapi_spec_validator, so CI skips the openapi spec validation step

mattjala · 2026-08-21T16:34:05Z

+        multiple groups in one request. `type` is not permitted in the body
+        (groups have no datatype).
+
+        **Bug:** `implicit` is only forwarded to argument construction for


Should be moved to a GH issue instead of a note in openapi

mattjala · 2026-08-21T16:34:23Z

+        (multi-item list) create; it is not restricted to non-batch
+        creates.
+
+        **Bug:** for a batch create (list with more than one item),


Should be moved to a GH issue instead of being in the openapi spec

mattjala · 2026-08-21T16:34:41Z

+          schema: { type: boolean, default: false }
+          description: |
+            Include an `alias` list of h5paths that resolve to this
+            dataset. Bug: on this specific route, the flag that gates


Bug desc should be moved to a GH issue

mattjala · 2026-08-21T16:35:02Z

+                    properties:
+                      bytes_sent:
+                        type: integer
+                        description: "Note: a source-level bug (hsds/basenode.py) assigns bytes_recv over this key immediately after setting it, so this actually reports received bytes, and sent-byte count is not exposed."


Bug desc should be moved to a GH issue

mattjala · 2026-08-21T16:35:41Z

+          description: |
+            Include an `alias` list of h5paths that resolve to this group.
+            Note: parsed with a raw truthiness check rather than real
+            boolean parsing, so `?getalias=0` is truthy and turns this on.


Truthiness issue seems like a bug and should be moved from here to a GH issue

mattjala · 2026-08-21T16:36:00Z

+          description: |
+            Include an `alias` list of h5paths that resolve to this
+            datatype. Note: parsed with a raw truthiness check, so
+            `?getalias=0` is truthy and turns this on.


Truthiness issue seems like a bug and should be moved from here to a GH issue

mattjala · 2026-08-21T16:37:12Z

-unit_tests = ('array_util_test', 'chunk_util_test', 'compression_test', 'domain_util_test',
-              'dset_util_test', 'hdf5_dtype_test', 'id_util_test', 'lru_cache_test',
+unit_tests = ('chunk_util_test', 'compression_test', 'domain_util_test',
+              'dset_util_test', 'lru_cache_test', 'openapi_test',


glob_parser_test and _stor_util_test` are omitted from this list.

mattjala · 2026-08-21T16:37:59Z

+    # in use and block its removal) actually gets torn down.
+    DOWN_COMPOSE_FILES="${COMPOSE_FILES}"
+    if [[ -z ${SWAGGER} ]]; then
+      DOWN_COMPOSE_FILES="${DOWN_COMPOSE_FILES} -f admin/docker/docker-compose.swagger.yml"


The referenced swagger file doesn't seem to exist, which will break teardown

jreadey added 10 commits April 14, 2025 16:58

add h5json package

978e672

temp use of github branch for h5json ref

abb5d0c

remove array_util test

ed44afa

use h5json for ndarray_compare function

bdff6e4

use h5json objid funcs

3904cf9

add nodeUtil.py

e1926c0

fix parameter for createObjId call

ae4579f

fix collection name for use with h5json

d6cad74

use connsistent collection name for isValidUuid

6add48a

fix flake8 format errors

b13321c

github-advanced-security AI found potential problems Apr 23, 2025

View reviewed changes

jreadey added 6 commits April 23, 2025 19:59

fix flake8 error in testall

fee9390

use h5json for unit test id

f1b1cab

restrict version on numcodecs

5dc3f76

allow client to generate obj ids

fb17e10

enable attributes to be included with POST req

3be18a0

add create timestamps for attributes in obj create

00d7c96

mattjala reviewed May 7, 2025

View reviewed changes

Comment thread tests/integ/attr_test.py Outdated

mattjala reviewed May 7, 2025

View reviewed changes

Comment thread hsds/ctype_sn.py Outdated

mattjala reviewed May 7, 2025

View reviewed changes

mattjala previously approved these changes May 7, 2025

View reviewed changes

jreadey dismissed mattjala’s stale review via 47b9a6e May 7, 2025 16:45

enable links to be initialized in post groups

47b9a6e

mattjala reviewed May 7, 2025

View reviewed changes

jreadey added 6 commits May 8, 2025 20:46

support dataset value init in post request

d9c3e87

add compound init value test

4ab24fc

added post data with compound data initializer

fc3ad68

add post_crawler class

8a18945

avoid exception for mkdir race condition

a8ec66d

use domain crawler to create links for post group multi

41e23e9

mattjala reviewed Aug 14, 2026

View reviewed changes

jreadey added 6 commits August 16, 2026 15:01

test case cleanup - dead code/misspells

85bd7fb

added regionref support, updated GET/PUT Value api

b3d52ce

added tests for opaque types

f7720a7

fixed log messages missing f format

2303f22

added openapi unit test

884ba94

removed non-relevant env from ci yml

3a73530

mattjala reviewed Aug 21, 2026

View reviewed changes

brtnfld added this to the HSDS 1.0.0 milestone Aug 21, 2026

mattjala reviewed Aug 21, 2026

View reviewed changes

Uh oh!

Uh oh!

Conversation

jreadey commented Apr 23, 2025 • edited by ellipsis-dev Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Use h5json package for typing and objids

Library Migration and Utility Restructuring

Object ID and Timestamp Handling

Configuration and Dependencies

API and Function Refactoring

New Functionality

Bug Fixes and Improvements

Test Updates

Uh oh!

Check failure

Copilot Autofix

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattjala left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jreadey commented Apr 23, 2025 •

edited by ellipsis-dev Bot

Loading