diff --git a/CHANGELOG.md b/CHANGELOG.md index b4632bb..f86bd37 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- feat: support async table loads and append mode - feat(uploads): support streaming uploads with on-demand part URLs +- **Breaking:** results and query runs are now scoped to a database via the + required `X-Database-Id` header. The generated `ResultsApi` / `QueryRunsApi` + methods (`get_result`, `list_results`, `get_query_run`, `list_query_runs`) + gain a required `x_database_id` argument, and the hand-written ergonomic + helpers match: `hotdata.arrow.ResultsApi.get_result_arrow` / + `stream_result_arrow` and `hotdata.query.QueryApi.wait_for_result` take the + database id, and `QueryApi.query`'s truncation auto-follow forwards the + query's database scope (the `X-Database-Id` header, or the request-body + `database_id` when no header is set) to its result and query-run fetches. ## [0.5.0] - 2026-06-28 diff --git a/README.md b/README.md index 91abc4d..bb93c56 100644 --- a/README.md +++ b/README.md @@ -77,17 +77,21 @@ from hotdata.arrow import ResultsApi with ApiClient(Configuration(api_key="...", workspace_id="...")) as client: results = ResultsApi(client) + # Results are scoped to a database via the required `X-Database-Id` header, + # so pass the id of the database the query ran in. + database_id = "your_database_id" + # Buffered: returns a pyarrow.Table. - table = results.get_result_arrow(result_id) + table = results.get_result_arrow(result_id, database_id) # Streaming: yields a pyarrow.RecordBatchStreamReader without # materializing the full table in memory. - with results.stream_result_arrow(result_id) as reader: + with results.stream_result_arrow(result_id, database_id) as reader: for batch in reader: ... ``` -Both methods accept `offset` and `limit` for pagination. They raise `hotdata.arrow.ResultNotReadyError` if the result is still pending or processing — poll `results.get_result(result_id)` until `status == "ready"` first. +Both methods accept `offset` and `limit` for pagination. They raise `hotdata.arrow.ResultNotReadyError` if the result is still pending or processing — poll `results.get_result(result_id, database_id)` until `status == "ready"` first. ## File uploads diff --git a/docs/AsyncQueryResponse.md b/docs/AsyncQueryResponse.md index 446f964..4d4c5d9 100644 --- a/docs/AsyncQueryResponse.md +++ b/docs/AsyncQueryResponse.md @@ -1,6 +1,6 @@ # AsyncQueryResponse -Response returned when a query is submitted asynchronously (202 Accepted). Poll GET /query-runs/{id} to track progress. Once status is \"succeeded\", retrieve results via GET /results/{result_id}. +Response returned when a query is submitted asynchronously (202 Accepted). Poll GET /query-runs/{id} to track progress, sending the same `X-Database-Id` header used to submit the query — the endpoint is scoped to that database and returns 400 without it. Once status is \"succeeded\", retrieve results via GET /results/{result_id}. ## Properties @@ -9,7 +9,7 @@ Name | Type | Description | Notes **query_run_id** | **str** | Unique identifier for the query run. | **reason** | **str** | Human-readable reason why the query went async (e.g., caching tables for the first time). | [optional] **status** | **str** | Current status of the query run. | -**status_url** | **str** | URL to poll for query run status. | +**status_url** | **str** | URL to poll for query run status. Requires the same `X-Database-Id` header used to submit the query. | ## Example diff --git a/docs/ConnectionsApi.md b/docs/ConnectionsApi.md index 8341092..43b18db 100644 --- a/docs/ConnectionsApi.md +++ b/docs/ConnectionsApi.md @@ -810,7 +810,7 @@ This endpoint does not need any parameter. Load managed table from upload -Publish a previously-uploaded file as the new contents of a managed table. CSV, JSON, and Parquet uploads are supported; the format is auto-detected from the upload's `Content-Type` and file contents, or set explicitly via the `format` field. Only `mode = "replace"` is supported. Concurrent loads against the same upload return 409. +Publish a previously-uploaded file as the new contents of a managed table. CSV, JSON, and Parquet uploads are supported; the format is auto-detected from the upload's `Content-Type` and file contents, or set explicitly via the `format` field. If the target table (or its schema) has not been declared yet, it is created automatically as part of the load — declaring tables up front is optional. `mode` selects how the upload is applied: `replace` overwrites the table's contents, `append` inserts the uploaded rows on top of the existing data. Concurrent loads against the same upload return 409. Set `async` to run the load in the background and get back a job ID to poll; add `async_after_ms` to wait briefly for it to finish before falling back to a job ID. ### Example @@ -894,9 +894,10 @@ Name | Type | Description | Notes | Status code | Description | Response headers | |-------------|-------------|------------------| **200** | Managed table loaded | - | -**400** | Invalid request (bad mode, non-managed connection, bad parquet) | - | -**404** | Connection or upload not found | - | -**409** | Upload already consumed or in flight | - | +**202** | Load accepted and running in the background; poll the returned job for status and result | - | +**400** | Invalid request (bad mode, non-managed connection, invalid identifier, bad parquet) | - | +**404** | Connection or upload not found, or the table was deleted | - | +**409** | Upload already consumed or in flight, or the uploaded data changes a column's type incompatibly (only widening to a larger compatible type can be applied automatically); the existing data is unchanged and remains queryable | - | [[Back to top]](#) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to Model list]](../README.md#documentation-for-models) [[Back to README]](../README.md) diff --git a/docs/CreateIndexRequest.md b/docs/CreateIndexRequest.md index baa0276..553808a 100644 --- a/docs/CreateIndexRequest.md +++ b/docs/CreateIndexRequest.md @@ -7,6 +7,7 @@ Request body for POST .../indexes Name | Type | Description | Notes ------------ | ------------- | ------------- | ------------- **var_async** | **bool** | When true, create the index as a background job and return a job ID for polling. | [optional] +**async_after_ms** | **int** | If set (requires `async` = true), wait up to this many milliseconds for the index build to finish: if it completes in time the index is returned (201), otherwise a 202 with a job ID to poll. Must be between 1000 and the server maximum; a value out of that range, or set without `async` = true, is rejected with 400. | [optional] **columns** | **List[str]** | Columns to index. Required for all index types. | **description** | **str** | User-facing description of the embedding (e.g., \"product descriptions\"). | [optional] **dimensions** | **int** | Output vector dimensions. Some models support multiple dimension sizes (e.g., OpenAI text-embedding-3-small supports 512 or 1536). If omitted, the model's default dimensions are used | [optional] diff --git a/docs/DatabasesApi.md b/docs/DatabasesApi.md index 40151d3..20c45c0 100644 --- a/docs/DatabasesApi.md +++ b/docs/DatabasesApi.md @@ -712,7 +712,7 @@ This endpoint does not need any parameter. Load database table from upload -Publish a previously-uploaded file as the new contents of a table on the database's default catalog. The database-scoped equivalent of the connection-scoped managed-table load — addressed by `database_id`, so no `default_connection_id` is needed. CSV, JSON, and Parquet uploads are supported; the format is auto-detected or set via `format`. Only `mode = "replace"` is supported. Concurrent loads against the same upload return 409. +Publish a previously-uploaded file as the new contents of a table on the database's default catalog. The database-scoped equivalent of the connection-scoped managed-table load — addressed by `database_id`, so no `default_connection_id` is needed. CSV, JSON, and Parquet uploads are supported; the format is auto-detected or set via `format`. If the target table (or its schema) has not been declared yet, it is created automatically as part of the load — declaring tables up front is optional. `mode` selects how the upload is applied: `replace` overwrites the table's contents, `append` inserts the uploaded rows on top of the existing data. Concurrent loads against the same upload return 409. Set `async` to run the load in the background and get back a job ID to poll; add `async_after_ms` to wait briefly for it to finish before falling back to a job ID. ### Example @@ -796,9 +796,10 @@ Name | Type | Description | Notes | Status code | Description | Response headers | |-------------|-------------|------------------| **200** | Table loaded | - | -**400** | Invalid request (bad mode, bad parquet) | - | -**404** | Database, table, or upload not found | - | -**409** | Upload already consumed or in flight | - | +**202** | Load accepted and running in the background; poll the returned job for status and result | - | +**400** | Invalid request (bad mode, invalid identifier, bad parquet) | - | +**404** | Database or upload not found, or the table was deleted | - | +**409** | Upload already consumed or in flight, or the uploaded data changes a column's type incompatibly (only widening to a larger compatible type can be applied automatically); the existing data is unchanged and remains queryable | - | [[Back to top]](#) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to Model list]](../README.md#documentation-for-models) [[Back to README]](../README.md) diff --git a/docs/IndexesApi.md b/docs/IndexesApi.md index ed516fe..e205dec 100644 --- a/docs/IndexesApi.md +++ b/docs/IndexesApi.md @@ -99,6 +99,7 @@ Name | Type | Description | Notes | Status code | Description | Response headers | |-------------|-------------|------------------| **201** | Index created | - | +**202** | Index build accepted and running in the background; poll the returned job for status | - | **400** | Invalid request | - | **404** | Table not found | - | **500** | Internal server error | - | diff --git a/docs/JobResult.md b/docs/JobResult.md index f3c1cfb..0e74c55 100644 --- a/docs/JobResult.md +++ b/docs/JobResult.md @@ -24,6 +24,8 @@ Name | Type | Description | Notes **source_column** | **str** | Source text column for an embedding-backed vector index. A query searches it via `vector_distance(<source_column>, …)`; the indexed `columns` hold the generated embedding column instead. Absent for BM25, sorted, and direct (existing-column) vector indexes. | [optional] **status** | [**IndexStatus**](IndexStatus.md) | | **updated_at** | **datetime** | | +**arrow_schema_json** | **str** | Schema of the loaded table, as JSON. | +**row_count** | **int** | Total number of rows in the table after the load. | ## Example diff --git a/docs/LoadManagedTableRequest.md b/docs/LoadManagedTableRequest.md index 24184cb..0d36ef6 100644 --- a/docs/LoadManagedTableRequest.md +++ b/docs/LoadManagedTableRequest.md @@ -1,13 +1,15 @@ # LoadManagedTableRequest -Request body for `POST /v1/connections/{connection_id}/schemas/{schema}/tables/{table}/loads`. Publishes a previously-uploaded file as the new contents of the named managed table. CSV and JSON uploads are converted to columnar storage on load; Parquet uploads are published directly. `mode` is fixed to `\"replace\"` today; the field is kept in the request body so future modes (e.g. append) are an additive change. +Request body for the managed-table load endpoints — the connection-scoped `POST /v1/connections/{connection_id}/schemas/{schema}/tables/{table}/loads` and the database-scoped equivalent. Publishes a previously-uploaded file to the named table. CSV and JSON uploads are converted to columnar storage on load; Parquet uploads are published directly. `mode` selects whether the upload replaces the table's contents or is appended on top of them. ## Properties Name | Type | Description | Notes ------------ | ------------- | ------------- | ------------- +**var_async** | **bool** | When true, run the load as a background job and return a job ID to poll instead of blocking until it finishes. Recommended for large uploads, which can take longer than an HTTP request should stay open. | [optional] +**async_after_ms** | **int** | If set (requires `async` = true), wait up to this many milliseconds for the load to finish: if it completes in time the full result is returned (200), otherwise a 202 with a job ID to poll. Must be between 1000 and the server maximum; a value out of that range, or set without `async` = true, is rejected with 400. | [optional] **format** | **str** | File format of the upload: `\"csv\"`, `\"json\"`, or `\"parquet\"`. Optional — when omitted, the format is auto-detected from the upload's `Content-Type` and, failing that, from the file contents. Provide it explicitly to override detection or when the contents are ambiguous. `\"json\"` expects newline-delimited JSON (one object per line), not a JSON array. | [optional] -**mode** | **str** | Load mode. Only `\"replace\"` is supported in this release. | +**mode** | **str** | How the upload is applied: `\"replace\"` overwrites the table's contents, `\"append\"` inserts the uploaded rows on top of the existing data. | **upload_id** | **str** | ID of a previously-staged upload (see `POST /v1/files`). The upload is claimed atomically; concurrent loads against the same `upload_id` return 409. | ## Example diff --git a/docs/LoadManagedTableResponse.md b/docs/LoadManagedTableResponse.md index 3d29b6c..9a6f68d 100644 --- a/docs/LoadManagedTableResponse.md +++ b/docs/LoadManagedTableResponse.md @@ -1,6 +1,6 @@ # LoadManagedTableResponse -Response body for `POST /v1/connections/{connection_id}/schemas/{schema}/tables/{table}/loads`. +Result of a managed-table load: row count after the load plus the published table schema. Returned inline (`200`) for a synchronous load, and as the `GET /v1/jobs/{id}` result payload for a completed background load. ## Properties @@ -8,7 +8,7 @@ Name | Type | Description | Notes ------------ | ------------- | ------------- | ------------- **arrow_schema_json** | **str** | Schema of the loaded table, as JSON. | **connection_id** | **str** | | -**row_count** | **int** | Total rows in the published parquet file. | +**row_count** | **int** | Total number of rows in the table after the load. | **schema_name** | **str** | | **table_name** | **str** | | diff --git a/docs/QueryRequest.md b/docs/QueryRequest.md index 6853b86..b11fb25 100644 --- a/docs/QueryRequest.md +++ b/docs/QueryRequest.md @@ -7,7 +7,7 @@ Request body for POST /query Name | Type | Description | Notes ------------ | ------------- | ------------- | ------------- **var_async** | **bool** | When true, execute the query asynchronously and return a query run ID for polling via GET /query-runs/{id}. The query results can be retrieved via GET /results/{id} once the query run status is \"succeeded\". | [optional] -**async_after_ms** | **int** | If set with async=true, wait up to this many milliseconds for the query to complete synchronously before returning an async response. Minimum 1000ms. Ignored if async is false. | [optional] +**async_after_ms** | **int** | If set (requires `async` = true), first attempt the query synchronously and wait up to this many milliseconds: if it finishes in time the full result is returned, otherwise an async response (a run id to poll) is returned. Must be at least 1000 and at most the server's configured maximum; a value out of that range, or set without `async` = true, is rejected with 400. | [optional] **database_id** | **str** | Database to scope the query to (its id). Alternative to the `X-Database-Id` header — exactly one source must be provided. If both this field and the header are set and they disagree, the request is rejected with a 400. | [optional] **default_catalog** | **str** | Catalog that unqualified table references resolve against within the query's database scope. Must name a catalog visible in the database (`default`, an attached catalog alias, or a system catalog). Defaults to `default` when omitted. | [optional] **default_schema** | **str** | Schema that unqualified table references resolve against within the query's database scope. Defaults to `main` when omitted. Existence is not validated up front — an unknown schema surfaces as a \"table not found\" error at planning time. | [optional] diff --git a/docs/QueryRunsApi.md b/docs/QueryRunsApi.md index c29c400..3e3fc34 100644 --- a/docs/QueryRunsApi.md +++ b/docs/QueryRunsApi.md @@ -9,11 +9,11 @@ Method | HTTP request | Description # **get_query_run** -> QueryRunInfo get_query_run(id) +> QueryRunInfo get_query_run(id, x_database_id) Get query run -Get the status and details of a specific query run by ID. +Get the status and details of a specific query run by ID, scoped to the database named by the required X-Database-Id header. ### Example @@ -53,10 +53,11 @@ with hotdata.ApiClient(configuration) as api_client: # Create an instance of the API class api_instance = hotdata.QueryRunsApi(api_client) id = 'id_example' # str | Query run ID + x_database_id = 'x_database_id_example' # str | Database the query run belongs to (required) try: # Get query run - api_response = api_instance.get_query_run(id) + api_response = api_instance.get_query_run(id, x_database_id) print("The response of QueryRunsApi->get_query_run:\n") pprint(api_response) except Exception as e: @@ -71,6 +72,7 @@ with hotdata.ApiClient(configuration) as api_client: Name | Type | Description | Notes ------------- | ------------- | ------------- | ------------- **id** | **str**| Query run ID | + **x_database_id** | **str**| Database the query run belongs to (required) | ### Return type @@ -90,15 +92,18 @@ Name | Type | Description | Notes | Status code | Description | Response headers | |-------------|-------------|------------------| **200** | Query run details | - | -**404** | Query run not found | - | +**400** | Missing or malformed X-Database-Id header | - | +**404** | Query run or database not found | - | [[Back to top]](#) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to Model list]](../README.md#documentation-for-models) [[Back to README]](../README.md) # **list_query_runs** -> ListQueryRunsResponse list_query_runs(limit=limit, cursor=cursor, status=status, saved_query_id=saved_query_id) +> ListQueryRunsResponse list_query_runs(x_database_id, limit=limit, cursor=cursor, status=status, saved_query_id=saved_query_id) List query runs +List query runs for the database named by the required X-Database-Id header. + ### Example * Api Key Authentication (WorkspaceId): @@ -136,6 +141,7 @@ configuration = hotdata.Configuration( with hotdata.ApiClient(configuration) as api_client: # Create an instance of the API class api_instance = hotdata.QueryRunsApi(api_client) + x_database_id = 'x_database_id_example' # str | Database to scope the query runs to (required) limit = 56 # int | Maximum number of results (optional) cursor = 'cursor_example' # str | Pagination cursor (optional) status = 'status_example' # str | Filter by status (comma-separated, e.g. status=running,failed) (optional) @@ -143,7 +149,7 @@ with hotdata.ApiClient(configuration) as api_client: try: # List query runs - api_response = api_instance.list_query_runs(limit=limit, cursor=cursor, status=status, saved_query_id=saved_query_id) + api_response = api_instance.list_query_runs(x_database_id, limit=limit, cursor=cursor, status=status, saved_query_id=saved_query_id) print("The response of QueryRunsApi->list_query_runs:\n") pprint(api_response) except Exception as e: @@ -157,6 +163,7 @@ with hotdata.ApiClient(configuration) as api_client: Name | Type | Description | Notes ------------- | ------------- | ------------- | ------------- + **x_database_id** | **str**| Database to scope the query runs to (required) | **limit** | **int**| Maximum number of results | [optional] **cursor** | **str**| Pagination cursor | [optional] **status** | **str**| Filter by status (comma-separated, e.g. status=running,failed) | [optional] @@ -180,6 +187,8 @@ Name | Type | Description | Notes | Status code | Description | Response headers | |-------------|-------------|------------------| **200** | List of query runs | - | +**400** | Missing or malformed X-Database-Id header | - | +**404** | Database not found | - | [[Back to top]](#) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to Model list]](../README.md#documentation-for-models) [[Back to README]](../README.md) diff --git a/docs/RefreshApi.md b/docs/RefreshApi.md index e0d2c4f..f6f09e8 100644 --- a/docs/RefreshApi.md +++ b/docs/RefreshApi.md @@ -97,8 +97,10 @@ Name | Type | Description | Notes | Status code | Description | Response headers | |-------------|-------------|------------------| **200** | Refresh completed | - | +**202** | Refresh accepted and running in the background; poll the returned job for status | - | **400** | Invalid request | - | **404** | Connection not found | - | +**409** | A column's type changed incompatibly and can't be applied automatically (only widening to a larger compatible type is). The existing data is unchanged and remains queryable. | - | [[Back to top]](#) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to Model list]](../README.md#documentation-for-models) [[Back to README]](../README.md) diff --git a/docs/RefreshRequest.md b/docs/RefreshRequest.md index 5182a48..a7d4e85 100644 --- a/docs/RefreshRequest.md +++ b/docs/RefreshRequest.md @@ -7,6 +7,7 @@ Request body for POST /refresh Name | Type | Description | Notes ------------ | ------------- | ------------- | ------------- **var_async** | **bool** | When true, submit the refresh as a background job and return immediately with a job ID for status polling. Only supported for data refresh operations. | [optional] +**async_after_ms** | **int** | If set (requires `async` = true), wait up to this many milliseconds for the refresh to finish: if it completes in time the full result is returned, otherwise a `202` with a job ID to poll. Must be between 1000 and the server maximum; a value out of that range, or set without `async` = true, is rejected with 400. Only applies to data refresh. | [optional] **connection_id** | **str** | | [optional] **data** | **bool** | | [optional] **include_uncached** | **bool** | Controls whether uncached tables are included in connection-wide data refresh. - `false` (default): Only refresh tables that already have cached data. This is the common case for keeping existing data up-to-date. - `true`: Also sync tables that haven't been cached yet, essentially performing an initial sync for any new tables discovered since the connection was created. This field only applies to connection-wide data refresh (when `data=true` and `table_name` is not specified). It has no effect on single-table refresh or schema refresh operations. | [optional] diff --git a/docs/RefreshResponse.md b/docs/RefreshResponse.md index 75de0b5..4dd1b73 100644 --- a/docs/RefreshResponse.md +++ b/docs/RefreshResponse.md @@ -21,9 +21,6 @@ Name | Type | Description | Notes **tables_failed** | **int** | | **tables_refreshed** | **int** | | **total_rows** | **int** | | -**id** | **str** | Job ID for status polling. | -**status** | [**JobStatus**](JobStatus.md) | Current status of the submitted job. | -**status_url** | **str** | URL to poll for job status. | ## Example diff --git a/docs/ResultsApi.md b/docs/ResultsApi.md index e1ade8d..b4c933d 100644 --- a/docs/ResultsApi.md +++ b/docs/ResultsApi.md @@ -9,7 +9,7 @@ Method | HTTP request | Description # **get_result** -> GetResultResponse get_result(id, offset=offset, limit=limit, format=format) +> GetResultResponse get_result(id, x_database_id, offset=offset, limit=limit, format=format) Get result @@ -78,13 +78,14 @@ with hotdata.ApiClient(configuration) as api_client: # Create an instance of the API class api_instance = hotdata.ResultsApi(api_client) id = 'id_example' # str | Result ID + x_database_id = 'x_database_id_example' # str | Database the result belongs to (required) offset = 56 # int | Rows to skip (default: 0) (optional) limit = 56 # int | Maximum rows to return (default: unbounded) (optional) format = hotdata.ResultsFormatQuery() # ResultsFormatQuery | `arrow`, `json`, `csv`, `md`, or `parquet` — overrides the `Accept` header. `markdown` is also accepted at runtime as an alias for `md`. (optional) try: # Get result - api_response = api_instance.get_result(id, offset=offset, limit=limit, format=format) + api_response = api_instance.get_result(id, x_database_id, offset=offset, limit=limit, format=format) print("The response of ResultsApi->get_result:\n") pprint(api_response) except Exception as e: @@ -99,6 +100,7 @@ with hotdata.ApiClient(configuration) as api_client: Name | Type | Description | Notes ------------- | ------------- | ------------- | ------------- **id** | **str**| Result ID | + **x_database_id** | **str**| Database the result belongs to (required) | **offset** | **int**| Rows to skip (default: 0) | [optional] **limit** | **int**| Maximum rows to return (default: unbounded) | [optional] **format** | [**ResultsFormatQuery**](.md)| `arrow`, `json`, `csv`, `md`, or `parquet` — overrides the `Accept` header. `markdown` is also accepted at runtime as an alias for `md`. | [optional] @@ -129,10 +131,12 @@ Name | Type | Description | Notes [[Back to top]](#) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to Model list]](../README.md#documentation-for-models) [[Back to README]](../README.md) # **list_results** -> ListResultsResponse list_results(limit=limit, offset=offset) +> ListResultsResponse list_results(x_database_id, limit=limit, offset=offset) List results +List stored results for the database named by the required X-Database-Id header. + ### Example * Api Key Authentication (WorkspaceId): @@ -177,12 +181,13 @@ configuration = hotdata.Configuration( with hotdata.ApiClient(configuration) as api_client: # Create an instance of the API class api_instance = hotdata.ResultsApi(api_client) + x_database_id = 'x_database_id_example' # str | Database to scope the results to (required) limit = 56 # int | Maximum number of results (default: 100, max: 1000) (optional) offset = 56 # int | Pagination offset (default: 0) (optional) try: # List results - api_response = api_instance.list_results(limit=limit, offset=offset) + api_response = api_instance.list_results(x_database_id, limit=limit, offset=offset) print("The response of ResultsApi->list_results:\n") pprint(api_response) except Exception as e: @@ -196,6 +201,7 @@ with hotdata.ApiClient(configuration) as api_client: Name | Type | Description | Notes ------------- | ------------- | ------------- | ------------- + **x_database_id** | **str**| Database to scope the results to (required) | **limit** | **int**| Maximum number of results (default: 100, max: 1000) | [optional] **offset** | **int**| Pagination offset (default: 0) | [optional] @@ -217,6 +223,8 @@ Name | Type | Description | Notes | Status code | Description | Response headers | |-------------|-------------|------------------| **200** | List of results | - | +**400** | Missing or malformed X-Database-Id header | - | +**404** | Database not found | - | [[Back to top]](#) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to Model list]](../README.md#documentation-for-models) [[Back to README]](../README.md) diff --git a/hotdata/api/connections_api.py b/hotdata/api/connections_api.py index 74e2f3a..3102a41 100644 --- a/hotdata/api/connections_api.py +++ b/hotdata/api/connections_api.py @@ -2603,7 +2603,7 @@ def load_managed_table( ) -> LoadManagedTableResponse: """Load managed table from upload - Publish a previously-uploaded file as the new contents of a managed table. CSV, JSON, and Parquet uploads are supported; the format is auto-detected from the upload's `Content-Type` and file contents, or set explicitly via the `format` field. Only `mode = \"replace\"` is supported. Concurrent loads against the same upload return 409. + Publish a previously-uploaded file as the new contents of a managed table. CSV, JSON, and Parquet uploads are supported; the format is auto-detected from the upload's `Content-Type` and file contents, or set explicitly via the `format` field. If the target table (or its schema) has not been declared yet, it is created automatically as part of the load — declaring tables up front is optional. `mode` selects how the upload is applied: `replace` overwrites the table's contents, `append` inserts the uploaded rows on top of the existing data. Concurrent loads against the same upload return 409. Set `async` to run the load in the background and get back a job ID to poll; add `async_after_ms` to wait briefly for it to finish before falling back to a job ID. :param connection_id: Connection ID (required) :type connection_id: str @@ -2648,6 +2648,7 @@ def load_managed_table( _response_types_map: Dict[str, Optional[str]] = { '200': "LoadManagedTableResponse", + '202': "SubmitJobResponse", '400': "ApiErrorResponse", '404': "ApiErrorResponse", '409': "ApiErrorResponse", @@ -2685,7 +2686,7 @@ def load_managed_table_with_http_info( ) -> ApiResponse[LoadManagedTableResponse]: """Load managed table from upload - Publish a previously-uploaded file as the new contents of a managed table. CSV, JSON, and Parquet uploads are supported; the format is auto-detected from the upload's `Content-Type` and file contents, or set explicitly via the `format` field. Only `mode = \"replace\"` is supported. Concurrent loads against the same upload return 409. + Publish a previously-uploaded file as the new contents of a managed table. CSV, JSON, and Parquet uploads are supported; the format is auto-detected from the upload's `Content-Type` and file contents, or set explicitly via the `format` field. If the target table (or its schema) has not been declared yet, it is created automatically as part of the load — declaring tables up front is optional. `mode` selects how the upload is applied: `replace` overwrites the table's contents, `append` inserts the uploaded rows on top of the existing data. Concurrent loads against the same upload return 409. Set `async` to run the load in the background and get back a job ID to poll; add `async_after_ms` to wait briefly for it to finish before falling back to a job ID. :param connection_id: Connection ID (required) :type connection_id: str @@ -2730,6 +2731,7 @@ def load_managed_table_with_http_info( _response_types_map: Dict[str, Optional[str]] = { '200': "LoadManagedTableResponse", + '202': "SubmitJobResponse", '400': "ApiErrorResponse", '404': "ApiErrorResponse", '409': "ApiErrorResponse", @@ -2767,7 +2769,7 @@ def load_managed_table_without_preload_content( ) -> RESTResponseType: """Load managed table from upload - Publish a previously-uploaded file as the new contents of a managed table. CSV, JSON, and Parquet uploads are supported; the format is auto-detected from the upload's `Content-Type` and file contents, or set explicitly via the `format` field. Only `mode = \"replace\"` is supported. Concurrent loads against the same upload return 409. + Publish a previously-uploaded file as the new contents of a managed table. CSV, JSON, and Parquet uploads are supported; the format is auto-detected from the upload's `Content-Type` and file contents, or set explicitly via the `format` field. If the target table (or its schema) has not been declared yet, it is created automatically as part of the load — declaring tables up front is optional. `mode` selects how the upload is applied: `replace` overwrites the table's contents, `append` inserts the uploaded rows on top of the existing data. Concurrent loads against the same upload return 409. Set `async` to run the load in the background and get back a job ID to poll; add `async_after_ms` to wait briefly for it to finish before falling back to a job ID. :param connection_id: Connection ID (required) :type connection_id: str @@ -2812,6 +2814,7 @@ def load_managed_table_without_preload_content( _response_types_map: Dict[str, Optional[str]] = { '200': "LoadManagedTableResponse", + '202': "SubmitJobResponse", '400': "ApiErrorResponse", '404': "ApiErrorResponse", '409': "ApiErrorResponse", diff --git a/hotdata/api/databases_api.py b/hotdata/api/databases_api.py index 1117008..8fe63f6 100644 --- a/hotdata/api/databases_api.py +++ b/hotdata/api/databases_api.py @@ -2317,7 +2317,7 @@ def load_database_table( ) -> LoadManagedTableResponse: """Load database table from upload - Publish a previously-uploaded file as the new contents of a table on the database's default catalog. The database-scoped equivalent of the connection-scoped managed-table load — addressed by `database_id`, so no `default_connection_id` is needed. CSV, JSON, and Parquet uploads are supported; the format is auto-detected or set via `format`. Only `mode = \"replace\"` is supported. Concurrent loads against the same upload return 409. + Publish a previously-uploaded file as the new contents of a table on the database's default catalog. The database-scoped equivalent of the connection-scoped managed-table load — addressed by `database_id`, so no `default_connection_id` is needed. CSV, JSON, and Parquet uploads are supported; the format is auto-detected or set via `format`. If the target table (or its schema) has not been declared yet, it is created automatically as part of the load — declaring tables up front is optional. `mode` selects how the upload is applied: `replace` overwrites the table's contents, `append` inserts the uploaded rows on top of the existing data. Concurrent loads against the same upload return 409. Set `async` to run the load in the background and get back a job ID to poll; add `async_after_ms` to wait briefly for it to finish before falling back to a job ID. :param database_id: Database ID (required) :type database_id: str @@ -2362,6 +2362,7 @@ def load_database_table( _response_types_map: Dict[str, Optional[str]] = { '200': "LoadManagedTableResponse", + '202': "SubmitJobResponse", '400': "ApiErrorResponse", '404': "ApiErrorResponse", '409': "ApiErrorResponse", @@ -2399,7 +2400,7 @@ def load_database_table_with_http_info( ) -> ApiResponse[LoadManagedTableResponse]: """Load database table from upload - Publish a previously-uploaded file as the new contents of a table on the database's default catalog. The database-scoped equivalent of the connection-scoped managed-table load — addressed by `database_id`, so no `default_connection_id` is needed. CSV, JSON, and Parquet uploads are supported; the format is auto-detected or set via `format`. Only `mode = \"replace\"` is supported. Concurrent loads against the same upload return 409. + Publish a previously-uploaded file as the new contents of a table on the database's default catalog. The database-scoped equivalent of the connection-scoped managed-table load — addressed by `database_id`, so no `default_connection_id` is needed. CSV, JSON, and Parquet uploads are supported; the format is auto-detected or set via `format`. If the target table (or its schema) has not been declared yet, it is created automatically as part of the load — declaring tables up front is optional. `mode` selects how the upload is applied: `replace` overwrites the table's contents, `append` inserts the uploaded rows on top of the existing data. Concurrent loads against the same upload return 409. Set `async` to run the load in the background and get back a job ID to poll; add `async_after_ms` to wait briefly for it to finish before falling back to a job ID. :param database_id: Database ID (required) :type database_id: str @@ -2444,6 +2445,7 @@ def load_database_table_with_http_info( _response_types_map: Dict[str, Optional[str]] = { '200': "LoadManagedTableResponse", + '202': "SubmitJobResponse", '400': "ApiErrorResponse", '404': "ApiErrorResponse", '409': "ApiErrorResponse", @@ -2481,7 +2483,7 @@ def load_database_table_without_preload_content( ) -> RESTResponseType: """Load database table from upload - Publish a previously-uploaded file as the new contents of a table on the database's default catalog. The database-scoped equivalent of the connection-scoped managed-table load — addressed by `database_id`, so no `default_connection_id` is needed. CSV, JSON, and Parquet uploads are supported; the format is auto-detected or set via `format`. Only `mode = \"replace\"` is supported. Concurrent loads against the same upload return 409. + Publish a previously-uploaded file as the new contents of a table on the database's default catalog. The database-scoped equivalent of the connection-scoped managed-table load — addressed by `database_id`, so no `default_connection_id` is needed. CSV, JSON, and Parquet uploads are supported; the format is auto-detected or set via `format`. If the target table (or its schema) has not been declared yet, it is created automatically as part of the load — declaring tables up front is optional. `mode` selects how the upload is applied: `replace` overwrites the table's contents, `append` inserts the uploaded rows on top of the existing data. Concurrent loads against the same upload return 409. Set `async` to run the load in the background and get back a job ID to poll; add `async_after_ms` to wait briefly for it to finish before falling back to a job ID. :param database_id: Database ID (required) :type database_id: str @@ -2526,6 +2528,7 @@ def load_database_table_without_preload_content( _response_types_map: Dict[str, Optional[str]] = { '200': "LoadManagedTableResponse", + '202': "SubmitJobResponse", '400': "ApiErrorResponse", '404': "ApiErrorResponse", '409': "ApiErrorResponse", diff --git a/hotdata/api/indexes_api.py b/hotdata/api/indexes_api.py index b99b463..11bf376 100644 --- a/hotdata/api/indexes_api.py +++ b/hotdata/api/indexes_api.py @@ -109,6 +109,7 @@ def create_index( _response_types_map: Dict[str, Optional[str]] = { '201': "IndexInfoResponse", + '202': "SubmitJobResponse", '400': "ApiErrorResponse", '404': "ApiErrorResponse", '500': "ApiErrorResponse", @@ -191,6 +192,7 @@ def create_index_with_http_info( _response_types_map: Dict[str, Optional[str]] = { '201': "IndexInfoResponse", + '202': "SubmitJobResponse", '400': "ApiErrorResponse", '404': "ApiErrorResponse", '500': "ApiErrorResponse", @@ -273,6 +275,7 @@ def create_index_without_preload_content( _response_types_map: Dict[str, Optional[str]] = { '201': "IndexInfoResponse", + '202': "SubmitJobResponse", '400': "ApiErrorResponse", '404': "ApiErrorResponse", '500': "ApiErrorResponse", diff --git a/hotdata/api/query_runs_api.py b/hotdata/api/query_runs_api.py index 10c69d7..72b73ac 100644 --- a/hotdata/api/query_runs_api.py +++ b/hotdata/api/query_runs_api.py @@ -44,6 +44,7 @@ def __init__(self, api_client=None) -> None: def get_query_run( self, id: Annotated[StrictStr, Field(description="Query run ID")], + x_database_id: Annotated[StrictStr, Field(description="Database the query run belongs to (required)")], _request_timeout: Union[ None, Annotated[StrictFloat, Field(gt=0)], @@ -59,10 +60,12 @@ def get_query_run( ) -> QueryRunInfo: """Get query run - Get the status and details of a specific query run by ID. + Get the status and details of a specific query run by ID, scoped to the database named by the required X-Database-Id header. :param id: Query run ID (required) :type id: str + :param x_database_id: Database the query run belongs to (required) (required) + :type x_database_id: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of @@ -87,6 +90,7 @@ def get_query_run( _param = self._get_query_run_serialize( id=id, + x_database_id=x_database_id, _request_auth=_request_auth, _content_type=_content_type, _headers=_headers, @@ -95,6 +99,7 @@ def get_query_run( _response_types_map: Dict[str, Optional[str]] = { '200': "QueryRunInfo", + '400': "ApiErrorResponse", '404': "ApiErrorResponse", } response_data = self.api_client.call_api( @@ -112,6 +117,7 @@ def get_query_run( def get_query_run_with_http_info( self, id: Annotated[StrictStr, Field(description="Query run ID")], + x_database_id: Annotated[StrictStr, Field(description="Database the query run belongs to (required)")], _request_timeout: Union[ None, Annotated[StrictFloat, Field(gt=0)], @@ -127,10 +133,12 @@ def get_query_run_with_http_info( ) -> ApiResponse[QueryRunInfo]: """Get query run - Get the status and details of a specific query run by ID. + Get the status and details of a specific query run by ID, scoped to the database named by the required X-Database-Id header. :param id: Query run ID (required) :type id: str + :param x_database_id: Database the query run belongs to (required) (required) + :type x_database_id: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of @@ -155,6 +163,7 @@ def get_query_run_with_http_info( _param = self._get_query_run_serialize( id=id, + x_database_id=x_database_id, _request_auth=_request_auth, _content_type=_content_type, _headers=_headers, @@ -163,6 +172,7 @@ def get_query_run_with_http_info( _response_types_map: Dict[str, Optional[str]] = { '200': "QueryRunInfo", + '400': "ApiErrorResponse", '404': "ApiErrorResponse", } response_data = self.api_client.call_api( @@ -180,6 +190,7 @@ def get_query_run_with_http_info( def get_query_run_without_preload_content( self, id: Annotated[StrictStr, Field(description="Query run ID")], + x_database_id: Annotated[StrictStr, Field(description="Database the query run belongs to (required)")], _request_timeout: Union[ None, Annotated[StrictFloat, Field(gt=0)], @@ -195,10 +206,12 @@ def get_query_run_without_preload_content( ) -> RESTResponseType: """Get query run - Get the status and details of a specific query run by ID. + Get the status and details of a specific query run by ID, scoped to the database named by the required X-Database-Id header. :param id: Query run ID (required) :type id: str + :param x_database_id: Database the query run belongs to (required) (required) + :type x_database_id: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of @@ -223,6 +236,7 @@ def get_query_run_without_preload_content( _param = self._get_query_run_serialize( id=id, + x_database_id=x_database_id, _request_auth=_request_auth, _content_type=_content_type, _headers=_headers, @@ -231,6 +245,7 @@ def get_query_run_without_preload_content( _response_types_map: Dict[str, Optional[str]] = { '200': "QueryRunInfo", + '400': "ApiErrorResponse", '404': "ApiErrorResponse", } response_data = self.api_client.call_api( @@ -243,6 +258,7 @@ def get_query_run_without_preload_content( def _get_query_run_serialize( self, id, + x_database_id, _request_auth, _content_type, _headers, @@ -268,6 +284,8 @@ def _get_query_run_serialize( _path_params['id'] = id # process the query parameters # process the header parameters + if x_database_id is not None: + _header_params['X-Database-Id'] = x_database_id # process the form parameters # process the body parameter @@ -308,6 +326,7 @@ def _get_query_run_serialize( @validate_call def list_query_runs( self, + x_database_id: Annotated[StrictStr, Field(description="Database to scope the query runs to (required)")], limit: Annotated[Optional[Annotated[int, Field(strict=True, ge=0)]], Field(description="Maximum number of results")] = None, cursor: Annotated[Optional[StrictStr], Field(description="Pagination cursor")] = None, status: Annotated[Optional[StrictStr], Field(description="Filter by status (comma-separated, e.g. status=running,failed)")] = None, @@ -327,7 +346,10 @@ def list_query_runs( ) -> ListQueryRunsResponse: """List query runs + List query runs for the database named by the required X-Database-Id header. + :param x_database_id: Database to scope the query runs to (required) (required) + :type x_database_id: str :param limit: Maximum number of results :type limit: int :param cursor: Pagination cursor @@ -359,6 +381,7 @@ def list_query_runs( """ # noqa: E501 _param = self._list_query_runs_serialize( + x_database_id=x_database_id, limit=limit, cursor=cursor, status=status, @@ -371,6 +394,8 @@ def list_query_runs( _response_types_map: Dict[str, Optional[str]] = { '200': "ListQueryRunsResponse", + '400': "ApiErrorResponse", + '404': "ApiErrorResponse", } response_data = self.api_client.call_api( *_param, @@ -386,6 +411,7 @@ def list_query_runs( @validate_call def list_query_runs_with_http_info( self, + x_database_id: Annotated[StrictStr, Field(description="Database to scope the query runs to (required)")], limit: Annotated[Optional[Annotated[int, Field(strict=True, ge=0)]], Field(description="Maximum number of results")] = None, cursor: Annotated[Optional[StrictStr], Field(description="Pagination cursor")] = None, status: Annotated[Optional[StrictStr], Field(description="Filter by status (comma-separated, e.g. status=running,failed)")] = None, @@ -405,7 +431,10 @@ def list_query_runs_with_http_info( ) -> ApiResponse[ListQueryRunsResponse]: """List query runs + List query runs for the database named by the required X-Database-Id header. + :param x_database_id: Database to scope the query runs to (required) (required) + :type x_database_id: str :param limit: Maximum number of results :type limit: int :param cursor: Pagination cursor @@ -437,6 +466,7 @@ def list_query_runs_with_http_info( """ # noqa: E501 _param = self._list_query_runs_serialize( + x_database_id=x_database_id, limit=limit, cursor=cursor, status=status, @@ -449,6 +479,8 @@ def list_query_runs_with_http_info( _response_types_map: Dict[str, Optional[str]] = { '200': "ListQueryRunsResponse", + '400': "ApiErrorResponse", + '404': "ApiErrorResponse", } response_data = self.api_client.call_api( *_param, @@ -464,6 +496,7 @@ def list_query_runs_with_http_info( @validate_call def list_query_runs_without_preload_content( self, + x_database_id: Annotated[StrictStr, Field(description="Database to scope the query runs to (required)")], limit: Annotated[Optional[Annotated[int, Field(strict=True, ge=0)]], Field(description="Maximum number of results")] = None, cursor: Annotated[Optional[StrictStr], Field(description="Pagination cursor")] = None, status: Annotated[Optional[StrictStr], Field(description="Filter by status (comma-separated, e.g. status=running,failed)")] = None, @@ -483,7 +516,10 @@ def list_query_runs_without_preload_content( ) -> RESTResponseType: """List query runs + List query runs for the database named by the required X-Database-Id header. + :param x_database_id: Database to scope the query runs to (required) (required) + :type x_database_id: str :param limit: Maximum number of results :type limit: int :param cursor: Pagination cursor @@ -515,6 +551,7 @@ def list_query_runs_without_preload_content( """ # noqa: E501 _param = self._list_query_runs_serialize( + x_database_id=x_database_id, limit=limit, cursor=cursor, status=status, @@ -527,6 +564,8 @@ def list_query_runs_without_preload_content( _response_types_map: Dict[str, Optional[str]] = { '200': "ListQueryRunsResponse", + '400': "ApiErrorResponse", + '404': "ApiErrorResponse", } response_data = self.api_client.call_api( *_param, @@ -537,6 +576,7 @@ def list_query_runs_without_preload_content( def _list_query_runs_serialize( self, + x_database_id, limit, cursor, status, @@ -580,6 +620,8 @@ def _list_query_runs_serialize( _query_params.append(('saved_query_id', saved_query_id)) # process the header parameters + if x_database_id is not None: + _header_params['X-Database-Id'] = x_database_id # process the form parameters # process the body parameter diff --git a/hotdata/api/refresh_api.py b/hotdata/api/refresh_api.py index 19d46de..0e2b587 100644 --- a/hotdata/api/refresh_api.py +++ b/hotdata/api/refresh_api.py @@ -92,8 +92,10 @@ def refresh( _response_types_map: Dict[str, Optional[str]] = { '200': "RefreshResponse", + '202': "SubmitJobResponse", '400': "ApiErrorResponse", '404': "ApiErrorResponse", + '409': "ApiErrorResponse", } response_data = self.api_client.call_api( *_param, @@ -161,8 +163,10 @@ def refresh_with_http_info( _response_types_map: Dict[str, Optional[str]] = { '200': "RefreshResponse", + '202': "SubmitJobResponse", '400': "ApiErrorResponse", '404': "ApiErrorResponse", + '409': "ApiErrorResponse", } response_data = self.api_client.call_api( *_param, @@ -230,8 +234,10 @@ def refresh_without_preload_content( _response_types_map: Dict[str, Optional[str]] = { '200': "RefreshResponse", + '202': "SubmitJobResponse", '400': "ApiErrorResponse", '404': "ApiErrorResponse", + '409': "ApiErrorResponse", } response_data = self.api_client.call_api( *_param, diff --git a/hotdata/api/results_api.py b/hotdata/api/results_api.py index ab006b0..da1a78e 100644 --- a/hotdata/api/results_api.py +++ b/hotdata/api/results_api.py @@ -45,6 +45,7 @@ def __init__(self, api_client=None) -> None: def get_result( self, id: Annotated[StrictStr, Field(description="Result ID")], + x_database_id: Annotated[StrictStr, Field(description="Database the result belongs to (required)")], offset: Annotated[Optional[Annotated[int, Field(strict=True, ge=0)]], Field(description="Rows to skip (default: 0)")] = None, limit: Annotated[Optional[Annotated[int, Field(strict=True, ge=0)]], Field(description="Maximum rows to return (default: unbounded)")] = None, format: Annotated[Optional[ResultsFormatQuery], Field(description="`arrow`, `json`, `csv`, `md`, or `parquet` — overrides the `Accept` header. `markdown` is also accepted at runtime as an alias for `md`.")] = None, @@ -67,6 +68,8 @@ def get_result( :param id: Result ID (required) :type id: str + :param x_database_id: Database the result belongs to (required) (required) + :type x_database_id: str :param offset: Rows to skip (default: 0) :type offset: int :param limit: Maximum rows to return (default: unbounded) @@ -97,6 +100,7 @@ def get_result( _param = self._get_result_serialize( id=id, + x_database_id=x_database_id, offset=offset, limit=limit, format=format, @@ -128,6 +132,7 @@ def get_result( def get_result_with_http_info( self, id: Annotated[StrictStr, Field(description="Result ID")], + x_database_id: Annotated[StrictStr, Field(description="Database the result belongs to (required)")], offset: Annotated[Optional[Annotated[int, Field(strict=True, ge=0)]], Field(description="Rows to skip (default: 0)")] = None, limit: Annotated[Optional[Annotated[int, Field(strict=True, ge=0)]], Field(description="Maximum rows to return (default: unbounded)")] = None, format: Annotated[Optional[ResultsFormatQuery], Field(description="`arrow`, `json`, `csv`, `md`, or `parquet` — overrides the `Accept` header. `markdown` is also accepted at runtime as an alias for `md`.")] = None, @@ -150,6 +155,8 @@ def get_result_with_http_info( :param id: Result ID (required) :type id: str + :param x_database_id: Database the result belongs to (required) (required) + :type x_database_id: str :param offset: Rows to skip (default: 0) :type offset: int :param limit: Maximum rows to return (default: unbounded) @@ -180,6 +187,7 @@ def get_result_with_http_info( _param = self._get_result_serialize( id=id, + x_database_id=x_database_id, offset=offset, limit=limit, format=format, @@ -211,6 +219,7 @@ def get_result_with_http_info( def get_result_without_preload_content( self, id: Annotated[StrictStr, Field(description="Result ID")], + x_database_id: Annotated[StrictStr, Field(description="Database the result belongs to (required)")], offset: Annotated[Optional[Annotated[int, Field(strict=True, ge=0)]], Field(description="Rows to skip (default: 0)")] = None, limit: Annotated[Optional[Annotated[int, Field(strict=True, ge=0)]], Field(description="Maximum rows to return (default: unbounded)")] = None, format: Annotated[Optional[ResultsFormatQuery], Field(description="`arrow`, `json`, `csv`, `md`, or `parquet` — overrides the `Accept` header. `markdown` is also accepted at runtime as an alias for `md`.")] = None, @@ -233,6 +242,8 @@ def get_result_without_preload_content( :param id: Result ID (required) :type id: str + :param x_database_id: Database the result belongs to (required) (required) + :type x_database_id: str :param offset: Rows to skip (default: 0) :type offset: int :param limit: Maximum rows to return (default: unbounded) @@ -263,6 +274,7 @@ def get_result_without_preload_content( _param = self._get_result_serialize( id=id, + x_database_id=x_database_id, offset=offset, limit=limit, format=format, @@ -289,6 +301,7 @@ def get_result_without_preload_content( def _get_result_serialize( self, id, + x_database_id, offset, limit, format, @@ -329,6 +342,8 @@ def _get_result_serialize( _query_params.append(('format', format.value)) # process the header parameters + if x_database_id is not None: + _header_params['X-Database-Id'] = x_database_id # process the form parameters # process the body parameter @@ -374,6 +389,7 @@ def _get_result_serialize( @validate_call def list_results( self, + x_database_id: Annotated[StrictStr, Field(description="Database to scope the results to (required)")], limit: Annotated[Optional[Annotated[int, Field(strict=True, ge=0)]], Field(description="Maximum number of results (default: 100, max: 1000)")] = None, offset: Annotated[Optional[Annotated[int, Field(strict=True, ge=0)]], Field(description="Pagination offset (default: 0)")] = None, _request_timeout: Union[ @@ -391,7 +407,10 @@ def list_results( ) -> ListResultsResponse: """List results + List stored results for the database named by the required X-Database-Id header. + :param x_database_id: Database to scope the results to (required) (required) + :type x_database_id: str :param limit: Maximum number of results (default: 100, max: 1000) :type limit: int :param offset: Pagination offset (default: 0) @@ -419,6 +438,7 @@ def list_results( """ # noqa: E501 _param = self._list_results_serialize( + x_database_id=x_database_id, limit=limit, offset=offset, _request_auth=_request_auth, @@ -429,6 +449,8 @@ def list_results( _response_types_map: Dict[str, Optional[str]] = { '200': "ListResultsResponse", + '400': "ApiErrorResponse", + '404': "ApiErrorResponse", } response_data = self.api_client.call_api( *_param, @@ -444,6 +466,7 @@ def list_results( @validate_call def list_results_with_http_info( self, + x_database_id: Annotated[StrictStr, Field(description="Database to scope the results to (required)")], limit: Annotated[Optional[Annotated[int, Field(strict=True, ge=0)]], Field(description="Maximum number of results (default: 100, max: 1000)")] = None, offset: Annotated[Optional[Annotated[int, Field(strict=True, ge=0)]], Field(description="Pagination offset (default: 0)")] = None, _request_timeout: Union[ @@ -461,7 +484,10 @@ def list_results_with_http_info( ) -> ApiResponse[ListResultsResponse]: """List results + List stored results for the database named by the required X-Database-Id header. + :param x_database_id: Database to scope the results to (required) (required) + :type x_database_id: str :param limit: Maximum number of results (default: 100, max: 1000) :type limit: int :param offset: Pagination offset (default: 0) @@ -489,6 +515,7 @@ def list_results_with_http_info( """ # noqa: E501 _param = self._list_results_serialize( + x_database_id=x_database_id, limit=limit, offset=offset, _request_auth=_request_auth, @@ -499,6 +526,8 @@ def list_results_with_http_info( _response_types_map: Dict[str, Optional[str]] = { '200': "ListResultsResponse", + '400': "ApiErrorResponse", + '404': "ApiErrorResponse", } response_data = self.api_client.call_api( *_param, @@ -514,6 +543,7 @@ def list_results_with_http_info( @validate_call def list_results_without_preload_content( self, + x_database_id: Annotated[StrictStr, Field(description="Database to scope the results to (required)")], limit: Annotated[Optional[Annotated[int, Field(strict=True, ge=0)]], Field(description="Maximum number of results (default: 100, max: 1000)")] = None, offset: Annotated[Optional[Annotated[int, Field(strict=True, ge=0)]], Field(description="Pagination offset (default: 0)")] = None, _request_timeout: Union[ @@ -531,7 +561,10 @@ def list_results_without_preload_content( ) -> RESTResponseType: """List results + List stored results for the database named by the required X-Database-Id header. + :param x_database_id: Database to scope the results to (required) (required) + :type x_database_id: str :param limit: Maximum number of results (default: 100, max: 1000) :type limit: int :param offset: Pagination offset (default: 0) @@ -559,6 +592,7 @@ def list_results_without_preload_content( """ # noqa: E501 _param = self._list_results_serialize( + x_database_id=x_database_id, limit=limit, offset=offset, _request_auth=_request_auth, @@ -569,6 +603,8 @@ def list_results_without_preload_content( _response_types_map: Dict[str, Optional[str]] = { '200': "ListResultsResponse", + '400': "ApiErrorResponse", + '404': "ApiErrorResponse", } response_data = self.api_client.call_api( *_param, @@ -579,6 +615,7 @@ def list_results_without_preload_content( def _list_results_serialize( self, + x_database_id, limit, offset, _request_auth, @@ -612,6 +649,8 @@ def _list_results_serialize( _query_params.append(('offset', offset)) # process the header parameters + if x_database_id is not None: + _header_params['X-Database-Id'] = x_database_id # process the form parameters # process the body parameter diff --git a/hotdata/arrow.py b/hotdata/arrow.py index 5d3c5f8..6e1d1c5 100644 --- a/hotdata/arrow.py +++ b/hotdata/arrow.py @@ -87,6 +87,7 @@ class ResultsApi(_GeneratedResultsApi): def get_result_arrow( self, id: str, + x_database_id: str, *, offset: Optional[int] = None, limit: Optional[int] = None, @@ -99,6 +100,9 @@ def get_result_arrow( iterate batches without materializing the whole table. :param id: Result ID. + :param x_database_id: Database the result belongs to. Results are scoped + to a database via the required ``X-Database-Id`` header — pass the + same database the query ran in. :param offset: Rows to skip (default: 0). :param limit: Maximum rows to return (default: unbounded). :raises ResultNotReadyError: result is still pending or processing. @@ -106,7 +110,8 @@ def get_result_arrow( (400 invalid params, 404 not found, 409 failed result). """ ipc = _import_pyarrow() - response = self._call_arrow(id=id, offset=offset, limit=limit, + response = self._call_arrow(id=id, x_database_id=x_database_id, + offset=offset, limit=limit, _request_timeout=_request_timeout) try: return ipc.open_stream(response).read_all() @@ -117,6 +122,7 @@ def get_result_arrow( def stream_result_arrow( self, id: str, + x_database_id: str, *, offset: Optional[int] = None, limit: Optional[int] = None, @@ -132,15 +138,22 @@ def stream_result_arrow( Example:: - with results.stream_result_arrow(result_id) as reader: + with results.stream_result_arrow(result_id, database_id) as reader: for batch in reader: process(batch) + :param id: Result ID. + :param x_database_id: Database the result belongs to. Results are scoped + to a database via the required ``X-Database-Id`` header — pass the + same database the query ran in. + :param offset: Rows to skip (default: 0). + :param limit: Maximum rows to return (default: unbounded). :raises ResultNotReadyError: result is still pending or processing. :raises hotdata.exceptions.ApiException: for other HTTP errors. """ ipc = _import_pyarrow() - response = self._call_arrow(id=id, offset=offset, limit=limit, + response = self._call_arrow(id=id, x_database_id=x_database_id, + offset=offset, limit=limit, _request_timeout=_request_timeout) try: yield ipc.open_stream(response) @@ -151,6 +164,7 @@ def _call_arrow( self, *, id: str, + x_database_id: str, offset: Optional[int], limit: Optional[int], _request_timeout: Any, @@ -158,9 +172,12 @@ def _call_arrow( # Build the request via the generator's private serialize helper so # path/query/auth handling stays in lockstep with the generated client. # Override only what we need: the Accept header and the format query. + # `GET /v1/results/{id}` is database-scoped, so the required + # X-Database-Id header flows through the generated serializer too. headers: Dict[str, Any] = {"Accept": ARROW_STREAM_MEDIA_TYPE} params = self._get_result_serialize( id=id, + x_database_id=x_database_id, offset=offset, limit=limit, format=ResultsFormatQuery.ARROW, diff --git a/hotdata/models/async_query_response.py b/hotdata/models/async_query_response.py index 67c6590..d42927b 100644 --- a/hotdata/models/async_query_response.py +++ b/hotdata/models/async_query_response.py @@ -25,12 +25,12 @@ class AsyncQueryResponse(BaseModel): """ - Response returned when a query is submitted asynchronously (202 Accepted). Poll GET /query-runs/{id} to track progress. Once status is \"succeeded\", retrieve results via GET /results/{result_id}. + Response returned when a query is submitted asynchronously (202 Accepted). Poll GET /query-runs/{id} to track progress, sending the same `X-Database-Id` header used to submit the query — the endpoint is scoped to that database and returns 400 without it. Once status is \"succeeded\", retrieve results via GET /results/{result_id}. """ # noqa: E501 query_run_id: StrictStr = Field(description="Unique identifier for the query run.") reason: Optional[StrictStr] = Field(default=None, description="Human-readable reason why the query went async (e.g., caching tables for the first time).") status: StrictStr = Field(description="Current status of the query run.") - status_url: StrictStr = Field(description="URL to poll for query run status.") + status_url: StrictStr = Field(description="URL to poll for query run status. Requires the same `X-Database-Id` header used to submit the query.") __properties: ClassVar[List[str]] = ["query_run_id", "reason", "status", "status_url"] model_config = ConfigDict( diff --git a/hotdata/models/create_index_request.py b/hotdata/models/create_index_request.py index 95010d5..95bf39b 100644 --- a/hotdata/models/create_index_request.py +++ b/hotdata/models/create_index_request.py @@ -29,6 +29,7 @@ class CreateIndexRequest(BaseModel): Request body for POST .../indexes """ # noqa: E501 var_async: Optional[StrictBool] = Field(default=None, description="When true, create the index as a background job and return a job ID for polling.", alias="async") + async_after_ms: Optional[Annotated[int, Field(strict=True, ge=1000)]] = Field(default=None, description="If set (requires `async` = true), wait up to this many milliseconds for the index build to finish: if it completes in time the index is returned (201), otherwise a 202 with a job ID to poll. Must be between 1000 and the server maximum; a value out of that range, or set without `async` = true, is rejected with 400.") columns: List[StrictStr] = Field(description="Columns to index. Required for all index types.") description: Optional[StrictStr] = Field(default=None, description="User-facing description of the embedding (e.g., \"product descriptions\").") dimensions: Optional[Annotated[int, Field(strict=True, ge=0)]] = Field(default=None, description="Output vector dimensions. Some models support multiple dimension sizes (e.g., OpenAI text-embedding-3-small supports 512 or 1536). If omitted, the model's default dimensions are used") @@ -37,7 +38,7 @@ class CreateIndexRequest(BaseModel): index_type: Optional[StrictStr] = Field(default=None, description="Index type: \"sorted\" (default), \"bm25\", or \"vector\"") metric: Optional[StrictStr] = Field(default=None, description="Distance metric for vector indexes: \"l2\", \"cosine\", or \"dot\". When omitted, defaults to \"l2\" for float array columns or the provider's preferred metric for text columns with auto-embedding.") output_column: Optional[StrictStr] = Field(default=None, description="Custom name for the generated embedding column. Defaults to `{column}_embedding`.") - __properties: ClassVar[List[str]] = ["async", "columns", "description", "dimensions", "embedding_provider_id", "index_name", "index_type", "metric", "output_column"] + __properties: ClassVar[List[str]] = ["async", "async_after_ms", "columns", "description", "dimensions", "embedding_provider_id", "index_name", "index_type", "metric", "output_column"] model_config = ConfigDict( populate_by_name=True, @@ -78,6 +79,11 @@ def to_dict(self) -> Dict[str, Any]: exclude=excluded_fields, exclude_none=True, ) + # set to None if async_after_ms (nullable) is None + # and model_fields_set contains the field + if self.async_after_ms is None and "async_after_ms" in self.model_fields_set: + _dict['async_after_ms'] = None + # set to None if description (nullable) is None # and model_fields_set contains the field if self.description is None and "description" in self.model_fields_set: @@ -116,6 +122,7 @@ def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]: _obj = cls.model_validate({ "async": obj.get("async"), + "async_after_ms": obj.get("async_after_ms"), "columns": obj.get("columns"), "description": obj.get("description"), "dimensions": obj.get("dimensions"), diff --git a/hotdata/models/job_result.py b/hotdata/models/job_result.py index 796c742..7945a5d 100644 --- a/hotdata/models/job_result.py +++ b/hotdata/models/job_result.py @@ -20,12 +20,13 @@ from typing import Any, List, Optional from hotdata.models.connection_refresh_result import ConnectionRefreshResult from hotdata.models.index_info_response import IndexInfoResponse +from hotdata.models.load_managed_table_response import LoadManagedTableResponse from hotdata.models.table_refresh_result import TableRefreshResult from pydantic import StrictStr, Field from typing import Union, List, Set, Optional, Dict from typing_extensions import Literal, Self -JOBRESULT_ONE_OF_SCHEMAS = ["ConnectionRefreshResult", "IndexInfoResponse", "TableRefreshResult"] +JOBRESULT_ONE_OF_SCHEMAS = ["ConnectionRefreshResult", "IndexInfoResponse", "LoadManagedTableResponse", "TableRefreshResult"] class JobResult(BaseModel): """ @@ -37,8 +38,10 @@ class JobResult(BaseModel): oneof_schema_2_validator: Optional[ConnectionRefreshResult] = Field(default=None, description="Result of a connection-wide data refresh.") # data type: IndexInfoResponse oneof_schema_3_validator: Optional[IndexInfoResponse] = Field(default=None, description="Result of an index creation.") - actual_instance: Optional[Union[ConnectionRefreshResult, IndexInfoResponse, TableRefreshResult]] = None - one_of_schemas: Set[str] = { "ConnectionRefreshResult", "IndexInfoResponse", "TableRefreshResult" } + # data type: LoadManagedTableResponse + oneof_schema_4_validator: Optional[LoadManagedTableResponse] = Field(default=None, description="Result of a managed-table load (row count + published schema).") + actual_instance: Optional[Union[ConnectionRefreshResult, IndexInfoResponse, LoadManagedTableResponse, TableRefreshResult]] = None + one_of_schemas: Set[str] = { "ConnectionRefreshResult", "IndexInfoResponse", "LoadManagedTableResponse", "TableRefreshResult" } model_config = ConfigDict( validate_assignment=True, @@ -76,12 +79,17 @@ def actual_instance_must_validate_oneof(cls, v): error_messages.append(f"Error! Input type `{type(v)}` is not `IndexInfoResponse`") else: match += 1 + # validate data type: LoadManagedTableResponse + if not isinstance(v, LoadManagedTableResponse): + error_messages.append(f"Error! Input type `{type(v)}` is not `LoadManagedTableResponse`") + else: + match += 1 if match > 1: # more than 1 match - raise ValueError("Multiple matches found when setting `actual_instance` in JobResult with oneOf schemas: ConnectionRefreshResult, IndexInfoResponse, TableRefreshResult. Details: " + ", ".join(error_messages)) + raise ValueError("Multiple matches found when setting `actual_instance` in JobResult with oneOf schemas: ConnectionRefreshResult, IndexInfoResponse, LoadManagedTableResponse, TableRefreshResult. Details: " + ", ".join(error_messages)) elif match == 0: # no match - raise ValueError("No match found when setting `actual_instance` in JobResult with oneOf schemas: ConnectionRefreshResult, IndexInfoResponse, TableRefreshResult. Details: " + ", ".join(error_messages)) + raise ValueError("No match found when setting `actual_instance` in JobResult with oneOf schemas: ConnectionRefreshResult, IndexInfoResponse, LoadManagedTableResponse, TableRefreshResult. Details: " + ", ".join(error_messages)) else: return v @@ -114,13 +122,19 @@ def from_json(cls, json_str: str) -> Self: match += 1 except (ValidationError, ValueError) as e: error_messages.append(str(e)) + # deserialize data into LoadManagedTableResponse + try: + instance.actual_instance = LoadManagedTableResponse.from_json(json_str) + match += 1 + except (ValidationError, ValueError) as e: + error_messages.append(str(e)) if match > 1: # more than 1 match - raise ValueError("Multiple matches found when deserializing the JSON string into JobResult with oneOf schemas: ConnectionRefreshResult, IndexInfoResponse, TableRefreshResult. Details: " + ", ".join(error_messages)) + raise ValueError("Multiple matches found when deserializing the JSON string into JobResult with oneOf schemas: ConnectionRefreshResult, IndexInfoResponse, LoadManagedTableResponse, TableRefreshResult. Details: " + ", ".join(error_messages)) elif match == 0: # no match - raise ValueError("No match found when deserializing the JSON string into JobResult with oneOf schemas: ConnectionRefreshResult, IndexInfoResponse, TableRefreshResult. Details: " + ", ".join(error_messages)) + raise ValueError("No match found when deserializing the JSON string into JobResult with oneOf schemas: ConnectionRefreshResult, IndexInfoResponse, LoadManagedTableResponse, TableRefreshResult. Details: " + ", ".join(error_messages)) else: return instance @@ -134,7 +148,7 @@ def to_json(self) -> str: else: return json.dumps(self.actual_instance) - def to_dict(self) -> Optional[Union[Dict[str, Any], ConnectionRefreshResult, IndexInfoResponse, TableRefreshResult]]: + def to_dict(self) -> Optional[Union[Dict[str, Any], ConnectionRefreshResult, IndexInfoResponse, LoadManagedTableResponse, TableRefreshResult]]: """Returns the dict representation of the actual instance""" if self.actual_instance is None: return None diff --git a/hotdata/models/load_managed_table_request.py b/hotdata/models/load_managed_table_request.py index a93b9df..9ef868c 100644 --- a/hotdata/models/load_managed_table_request.py +++ b/hotdata/models/load_managed_table_request.py @@ -18,19 +18,22 @@ import re # noqa: F401 import json -from pydantic import BaseModel, ConfigDict, Field, StrictStr +from pydantic import BaseModel, ConfigDict, Field, StrictBool, StrictStr from typing import Any, ClassVar, Dict, List, Optional +from typing_extensions import Annotated from typing import Optional, Set from typing_extensions import Self class LoadManagedTableRequest(BaseModel): """ - Request body for `POST /v1/connections/{connection_id}/schemas/{schema}/tables/{table}/loads`. Publishes a previously-uploaded file as the new contents of the named managed table. CSV and JSON uploads are converted to columnar storage on load; Parquet uploads are published directly. `mode` is fixed to `\"replace\"` today; the field is kept in the request body so future modes (e.g. append) are an additive change. + Request body for the managed-table load endpoints — the connection-scoped `POST /v1/connections/{connection_id}/schemas/{schema}/tables/{table}/loads` and the database-scoped equivalent. Publishes a previously-uploaded file to the named table. CSV and JSON uploads are converted to columnar storage on load; Parquet uploads are published directly. `mode` selects whether the upload replaces the table's contents or is appended on top of them. """ # noqa: E501 + var_async: Optional[StrictBool] = Field(default=None, description="When true, run the load as a background job and return a job ID to poll instead of blocking until it finishes. Recommended for large uploads, which can take longer than an HTTP request should stay open.", alias="async") + async_after_ms: Optional[Annotated[int, Field(strict=True, ge=1000)]] = Field(default=None, description="If set (requires `async` = true), wait up to this many milliseconds for the load to finish: if it completes in time the full result is returned (200), otherwise a 202 with a job ID to poll. Must be between 1000 and the server maximum; a value out of that range, or set without `async` = true, is rejected with 400.") format: Optional[StrictStr] = Field(default=None, description="File format of the upload: `\"csv\"`, `\"json\"`, or `\"parquet\"`. Optional — when omitted, the format is auto-detected from the upload's `Content-Type` and, failing that, from the file contents. Provide it explicitly to override detection or when the contents are ambiguous. `\"json\"` expects newline-delimited JSON (one object per line), not a JSON array.") - mode: StrictStr = Field(description="Load mode. Only `\"replace\"` is supported in this release.") + mode: StrictStr = Field(description="How the upload is applied: `\"replace\"` overwrites the table's contents, `\"append\"` inserts the uploaded rows on top of the existing data.") upload_id: StrictStr = Field(description="ID of a previously-staged upload (see `POST /v1/files`). The upload is claimed atomically; concurrent loads against the same `upload_id` return 409.") - __properties: ClassVar[List[str]] = ["format", "mode", "upload_id"] + __properties: ClassVar[List[str]] = ["async", "async_after_ms", "format", "mode", "upload_id"] model_config = ConfigDict( populate_by_name=True, @@ -71,6 +74,11 @@ def to_dict(self) -> Dict[str, Any]: exclude=excluded_fields, exclude_none=True, ) + # set to None if async_after_ms (nullable) is None + # and model_fields_set contains the field + if self.async_after_ms is None and "async_after_ms" in self.model_fields_set: + _dict['async_after_ms'] = None + # set to None if format (nullable) is None # and model_fields_set contains the field if self.format is None and "format" in self.model_fields_set: @@ -88,6 +96,8 @@ def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]: return cls.model_validate(obj) _obj = cls.model_validate({ + "async": obj.get("async"), + "async_after_ms": obj.get("async_after_ms"), "format": obj.get("format"), "mode": obj.get("mode"), "upload_id": obj.get("upload_id") diff --git a/hotdata/models/load_managed_table_response.py b/hotdata/models/load_managed_table_response.py index 10eafca..dd19105 100644 --- a/hotdata/models/load_managed_table_response.py +++ b/hotdata/models/load_managed_table_response.py @@ -26,11 +26,11 @@ class LoadManagedTableResponse(BaseModel): """ - Response body for `POST /v1/connections/{connection_id}/schemas/{schema}/tables/{table}/loads`. + Result of a managed-table load: row count after the load plus the published table schema. Returned inline (`200`) for a synchronous load, and as the `GET /v1/jobs/{id}` result payload for a completed background load. """ # noqa: E501 arrow_schema_json: StrictStr = Field(description="Schema of the loaded table, as JSON.") connection_id: StrictStr - row_count: Annotated[int, Field(strict=True, ge=0)] = Field(description="Total rows in the published parquet file.") + row_count: Annotated[int, Field(strict=True, ge=0)] = Field(description="Total number of rows in the table after the load.") schema_name: StrictStr table_name: StrictStr __properties: ClassVar[List[str]] = ["arrow_schema_json", "connection_id", "row_count", "schema_name", "table_name"] diff --git a/hotdata/models/query_request.py b/hotdata/models/query_request.py index 9e525dd..7553db4 100644 --- a/hotdata/models/query_request.py +++ b/hotdata/models/query_request.py @@ -29,7 +29,7 @@ class QueryRequest(BaseModel): Request body for POST /query """ # noqa: E501 var_async: Optional[StrictBool] = Field(default=None, description="When true, execute the query asynchronously and return a query run ID for polling via GET /query-runs/{id}. The query results can be retrieved via GET /results/{id} once the query run status is \"succeeded\".", alias="async") - async_after_ms: Optional[Annotated[int, Field(strict=True, ge=1000)]] = Field(default=None, description="If set with async=true, wait up to this many milliseconds for the query to complete synchronously before returning an async response. Minimum 1000ms. Ignored if async is false.") + async_after_ms: Optional[Annotated[int, Field(strict=True, ge=1000)]] = Field(default=None, description="If set (requires `async` = true), first attempt the query synchronously and wait up to this many milliseconds: if it finishes in time the full result is returned, otherwise an async response (a run id to poll) is returned. Must be at least 1000 and at most the server's configured maximum; a value out of that range, or set without `async` = true, is rejected with 400.") database_id: Optional[StrictStr] = Field(default=None, description="Database to scope the query to (its id). Alternative to the `X-Database-Id` header — exactly one source must be provided. If both this field and the header are set and they disagree, the request is rejected with a 400.") default_catalog: Optional[StrictStr] = Field(default=None, description="Catalog that unqualified table references resolve against within the query's database scope. Must name a catalog visible in the database (`default`, an attached catalog alias, or a system catalog). Defaults to `default` when omitted.") default_schema: Optional[StrictStr] = Field(default=None, description="Schema that unqualified table references resolve against within the query's database scope. Defaults to `main` when omitted. Existence is not validated up front — an unknown schema surfaces as a \"table not found\" error at planning time.") diff --git a/hotdata/models/refresh_request.py b/hotdata/models/refresh_request.py index 6961732..d0ea5f1 100644 --- a/hotdata/models/refresh_request.py +++ b/hotdata/models/refresh_request.py @@ -20,6 +20,7 @@ from pydantic import BaseModel, ConfigDict, Field, StrictBool, StrictStr from typing import Any, ClassVar, Dict, List, Optional +from typing_extensions import Annotated from typing import Optional, Set from typing_extensions import Self @@ -28,12 +29,13 @@ class RefreshRequest(BaseModel): Request body for POST /refresh """ # noqa: E501 var_async: Optional[StrictBool] = Field(default=None, description="When true, submit the refresh as a background job and return immediately with a job ID for status polling. Only supported for data refresh operations.", alias="async") + async_after_ms: Optional[Annotated[int, Field(strict=True, ge=1000)]] = Field(default=None, description="If set (requires `async` = true), wait up to this many milliseconds for the refresh to finish: if it completes in time the full result is returned, otherwise a `202` with a job ID to poll. Must be between 1000 and the server maximum; a value out of that range, or set without `async` = true, is rejected with 400. Only applies to data refresh.") connection_id: Optional[StrictStr] = None data: Optional[StrictBool] = None include_uncached: Optional[StrictBool] = Field(default=None, description="Controls whether uncached tables are included in connection-wide data refresh. - `false` (default): Only refresh tables that already have cached data. This is the common case for keeping existing data up-to-date. - `true`: Also sync tables that haven't been cached yet, essentially performing an initial sync for any new tables discovered since the connection was created. This field only applies to connection-wide data refresh (when `data=true` and `table_name` is not specified). It has no effect on single-table refresh or schema refresh operations.") schema_name: Optional[StrictStr] = None table_name: Optional[StrictStr] = None - __properties: ClassVar[List[str]] = ["async", "connection_id", "data", "include_uncached", "schema_name", "table_name"] + __properties: ClassVar[List[str]] = ["async", "async_after_ms", "connection_id", "data", "include_uncached", "schema_name", "table_name"] model_config = ConfigDict( populate_by_name=True, @@ -74,6 +76,11 @@ def to_dict(self) -> Dict[str, Any]: exclude=excluded_fields, exclude_none=True, ) + # set to None if async_after_ms (nullable) is None + # and model_fields_set contains the field + if self.async_after_ms is None and "async_after_ms" in self.model_fields_set: + _dict['async_after_ms'] = None + # set to None if connection_id (nullable) is None # and model_fields_set contains the field if self.connection_id is None and "connection_id" in self.model_fields_set: @@ -102,6 +109,7 @@ def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]: _obj = cls.model_validate({ "async": obj.get("async"), + "async_after_ms": obj.get("async_after_ms"), "connection_id": obj.get("connection_id"), "data": obj.get("data"), "include_uncached": obj.get("include_uncached"), diff --git a/hotdata/models/refresh_response.py b/hotdata/models/refresh_response.py index c9132b9..3ca404c 100644 --- a/hotdata/models/refresh_response.py +++ b/hotdata/models/refresh_response.py @@ -20,13 +20,12 @@ from typing import Any, List, Optional from hotdata.models.connection_refresh_result import ConnectionRefreshResult from hotdata.models.schema_refresh_result import SchemaRefreshResult -from hotdata.models.submit_job_response import SubmitJobResponse from hotdata.models.table_refresh_result import TableRefreshResult from pydantic import StrictStr, Field from typing import Union, List, Set, Optional, Dict from typing_extensions import Literal, Self -REFRESHRESPONSE_ONE_OF_SCHEMAS = ["ConnectionRefreshResult", "SchemaRefreshResult", "SubmitJobResponse", "TableRefreshResult"] +REFRESHRESPONSE_ONE_OF_SCHEMAS = ["ConnectionRefreshResult", "SchemaRefreshResult", "TableRefreshResult"] class RefreshResponse(BaseModel): """ @@ -38,10 +37,8 @@ class RefreshResponse(BaseModel): oneof_schema_2_validator: Optional[TableRefreshResult] = None # data type: ConnectionRefreshResult oneof_schema_3_validator: Optional[ConnectionRefreshResult] = None - # data type: SubmitJobResponse - oneof_schema_4_validator: Optional[SubmitJobResponse] = None - actual_instance: Optional[Union[ConnectionRefreshResult, SchemaRefreshResult, SubmitJobResponse, TableRefreshResult]] = None - one_of_schemas: Set[str] = { "ConnectionRefreshResult", "SchemaRefreshResult", "SubmitJobResponse", "TableRefreshResult" } + actual_instance: Optional[Union[ConnectionRefreshResult, SchemaRefreshResult, TableRefreshResult]] = None + one_of_schemas: Set[str] = { "ConnectionRefreshResult", "SchemaRefreshResult", "TableRefreshResult" } model_config = ConfigDict( validate_assignment=True, @@ -79,17 +76,12 @@ def actual_instance_must_validate_oneof(cls, v): error_messages.append(f"Error! Input type `{type(v)}` is not `ConnectionRefreshResult`") else: match += 1 - # validate data type: SubmitJobResponse - if not isinstance(v, SubmitJobResponse): - error_messages.append(f"Error! Input type `{type(v)}` is not `SubmitJobResponse`") - else: - match += 1 if match > 1: # more than 1 match - raise ValueError("Multiple matches found when setting `actual_instance` in RefreshResponse with oneOf schemas: ConnectionRefreshResult, SchemaRefreshResult, SubmitJobResponse, TableRefreshResult. Details: " + ", ".join(error_messages)) + raise ValueError("Multiple matches found when setting `actual_instance` in RefreshResponse with oneOf schemas: ConnectionRefreshResult, SchemaRefreshResult, TableRefreshResult. Details: " + ", ".join(error_messages)) elif match == 0: # no match - raise ValueError("No match found when setting `actual_instance` in RefreshResponse with oneOf schemas: ConnectionRefreshResult, SchemaRefreshResult, SubmitJobResponse, TableRefreshResult. Details: " + ", ".join(error_messages)) + raise ValueError("No match found when setting `actual_instance` in RefreshResponse with oneOf schemas: ConnectionRefreshResult, SchemaRefreshResult, TableRefreshResult. Details: " + ", ".join(error_messages)) else: return v @@ -122,19 +114,13 @@ def from_json(cls, json_str: str) -> Self: match += 1 except (ValidationError, ValueError) as e: error_messages.append(str(e)) - # deserialize data into SubmitJobResponse - try: - instance.actual_instance = SubmitJobResponse.from_json(json_str) - match += 1 - except (ValidationError, ValueError) as e: - error_messages.append(str(e)) if match > 1: # more than 1 match - raise ValueError("Multiple matches found when deserializing the JSON string into RefreshResponse with oneOf schemas: ConnectionRefreshResult, SchemaRefreshResult, SubmitJobResponse, TableRefreshResult. Details: " + ", ".join(error_messages)) + raise ValueError("Multiple matches found when deserializing the JSON string into RefreshResponse with oneOf schemas: ConnectionRefreshResult, SchemaRefreshResult, TableRefreshResult. Details: " + ", ".join(error_messages)) elif match == 0: # no match - raise ValueError("No match found when deserializing the JSON string into RefreshResponse with oneOf schemas: ConnectionRefreshResult, SchemaRefreshResult, SubmitJobResponse, TableRefreshResult. Details: " + ", ".join(error_messages)) + raise ValueError("No match found when deserializing the JSON string into RefreshResponse with oneOf schemas: ConnectionRefreshResult, SchemaRefreshResult, TableRefreshResult. Details: " + ", ".join(error_messages)) else: return instance @@ -148,7 +134,7 @@ def to_json(self) -> str: else: return json.dumps(self.actual_instance) - def to_dict(self) -> Optional[Union[Dict[str, Any], ConnectionRefreshResult, SchemaRefreshResult, SubmitJobResponse, TableRefreshResult]]: + def to_dict(self) -> Optional[Union[Dict[str, Any], ConnectionRefreshResult, SchemaRefreshResult, TableRefreshResult]]: """Returns the dict representation of the actual instance""" if self.actual_instance is None: return None diff --git a/hotdata/query.py b/hotdata/query.py index 53555a1..91a17b4 100644 --- a/hotdata/query.py +++ b/hotdata/query.py @@ -337,11 +337,20 @@ def query( return response if not follow or not response.truncated: return response - return self._materialize_full(response, row_guard, byte_guard) + # Auto-follow re-fetches the persisted result and the query run, both of + # which are scoped to a database via the required X-Database-Id header. + # /v1/query takes exactly one database source, so the effective database + # is the header value when present, otherwise the request-body + # database_id. + effective_db = x_database_id or query_request.database_id or "" + return self._materialize_full( + response, effective_db, row_guard, byte_guard + ) def wait_for_result( self, result_id: str, + x_database_id: str, *, results_api: Optional[_ResultsApi] = None, ) -> Any: @@ -355,6 +364,10 @@ def wait_for_result( every poll, materializing the whole result into memory before the size guards can act. + :param result_id: Result ID to poll. + :param x_database_id: Database the result belongs to. Results are scoped + to a database via the required ``X-Database-Id`` header — pass the + same database the query ran in. :raises ResultFailedError: the result failed (delivered as HTTP 409). :raises ResultTimeoutError: ``ready`` not reached within the poll deadline. """ @@ -365,7 +378,7 @@ def wait_for_result( last_status = "pending" while True: try: - result = api.get_result(result_id, limit=0) + result = api.get_result(result_id, x_database_id=x_database_id, limit=0) except ApiException as exc: # A failed result is delivered as HTTP 409: the generated client # raises (response_deserialize raises on any non-2xx) rather than @@ -449,6 +462,7 @@ def _backoff_delay(self, exc: ApiException, attempt: int) -> float: def _materialize_full( self, preview: QueryResponse, + x_database_id: str, max_auto_rows: Optional[int], max_auto_bytes: Optional[int], ) -> QueryResponse: @@ -456,9 +470,11 @@ def _materialize_full( raise ResultUnavailableError(warning=preview.warning) results_api = _ResultsApi(self.api_client) - self.wait_for_result(preview.result_id, results_api=results_api) + self.wait_for_result( + preview.result_id, x_database_id, results_api=results_api + ) - total = self._authoritative_total(preview) + total = self._authoritative_total(preview, x_database_id) # Auto-follow does extra round-trips (poll + paginate) and materializes # the full result; log it so the hidden work behind one query() call is # observable without being noisy (info, not a warning). @@ -477,7 +493,12 @@ def _materialize_full( ) rows = self._fetch_all_rows( - preview.result_id, total, max_auto_rows, max_auto_bytes, results_api + preview.result_id, + x_database_id, + total, + max_auto_rows, + max_auto_bytes, + results_api, ) # Replace the bounded preview with the full row set. truncated / # total_row_count stay as the server reported them so the caller can @@ -487,15 +508,22 @@ def _materialize_full( preview.total_row_count = total return preview - def _authoritative_total(self, preview: QueryResponse) -> Optional[int]: + def _authoritative_total( + self, preview: QueryResponse, x_database_id: str + ) -> Optional[int]: """The grand total row count. ``total_row_count`` is null while a truncated result is still persisting, so fall back to the query-run record, which carries the authoritative count once the run succeeds. + + The query run is scoped to a database via the required ``X-Database-Id`` + header, so ``x_database_id`` is forwarded to the lookup. """ if preview.total_row_count is not None: return preview.total_row_count try: - run = _QueryRunsApi(self.api_client).get_query_run(preview.query_run_id) + run = _QueryRunsApi(self.api_client).get_query_run( + preview.query_run_id, x_database_id=x_database_id + ) except ApiException: return None return run.row_count @@ -503,6 +531,7 @@ def _authoritative_total(self, preview: QueryResponse) -> Optional[int]: def _fetch_all_rows( self, result_id: str, + x_database_id: str, total: Optional[int], max_auto_rows: Optional[int], max_auto_bytes: Optional[int], @@ -515,6 +544,7 @@ def _fetch_all_rows( while True: page = results_api.get_result( result_id, + x_database_id=x_database_id, offset=offset, limit=page_size, format=ResultsFormatQuery.JSON, diff --git a/test/test_create_index_request.py b/test/test_create_index_request.py index dbf1f69..e41dc19 100644 --- a/test/test_create_index_request.py +++ b/test/test_create_index_request.py @@ -37,6 +37,7 @@ def make_instance(self, include_optional) -> CreateIndexRequest: if include_optional: return CreateIndexRequest( var_async = True, + async_after_ms = 1000, columns = [ '' ], diff --git a/test/test_job_result.py b/test/test_job_result.py index f8fc9b5..e4d5489 100644 --- a/test/test_job_result.py +++ b/test/test_job_result.py @@ -65,7 +65,9 @@ def make_instance(self, include_optional) -> JobResult: metric = '', source_column = '', status = 'ready', - updated_at = datetime.datetime.strptime('2013-10-20 19:20:30.00', '%Y-%m-%d %H:%M:%S.%f') + updated_at = datetime.datetime.strptime('2013-10-20 19:20:30.00', '%Y-%m-%d %H:%M:%S.%f'), + arrow_schema_json = '', + row_count = 0 ) else: return JobResult( @@ -85,6 +87,8 @@ def make_instance(self, include_optional) -> JobResult: index_type = '', status = 'ready', updated_at = datetime.datetime.strptime('2013-10-20 19:20:30.00', '%Y-%m-%d %H:%M:%S.%f'), + arrow_schema_json = '', + row_count = 0, ) """ diff --git a/test/test_load_managed_table_request.py b/test/test_load_managed_table_request.py index b72c030..08f1d8a 100644 --- a/test/test_load_managed_table_request.py +++ b/test/test_load_managed_table_request.py @@ -36,6 +36,8 @@ def make_instance(self, include_optional) -> LoadManagedTableRequest: model = LoadManagedTableRequest() if include_optional: return LoadManagedTableRequest( + var_async = True, + async_after_ms = 1000, format = '', mode = '', upload_id = '' diff --git a/test/test_refresh_request.py b/test/test_refresh_request.py index 1af46c2..2038a55 100644 --- a/test/test_refresh_request.py +++ b/test/test_refresh_request.py @@ -37,6 +37,7 @@ def make_instance(self, include_optional) -> RefreshRequest: if include_optional: return RefreshRequest( var_async = True, + async_after_ms = 1000, connection_id = '', data = True, include_uncached = True, diff --git a/test/test_refresh_response.py b/test/test_refresh_response.py index 6066422..9db2b9b 100644 --- a/test/test_refresh_response.py +++ b/test/test_refresh_response.py @@ -60,10 +60,7 @@ def make_instance(self, include_optional) -> RefreshResponse: ], tables_failed = 0, tables_refreshed = 0, - total_rows = 0, - id = '', - status = 'pending', - status_url = '' + total_rows = 0 ) else: return RefreshResponse( @@ -80,9 +77,6 @@ def make_instance(self, include_optional) -> RefreshResponse: tables_failed = 0, tables_refreshed = 0, total_rows = 0, - id = '', - status = 'pending', - status_url = '', ) """ diff --git a/tests/integration/test_query_async_polling.py b/tests/integration/test_query_async_polling.py index a4d797b..791fe32 100644 --- a/tests/integration/test_query_async_polling.py +++ b/tests/integration/test_query_async_polling.py @@ -56,7 +56,7 @@ def test_query_async_polling( deadline = time.monotonic() + POLL_TIMEOUT_S run = None while time.monotonic() < deadline: - run = query_runs_api.get_query_run(query_run_id) + run = query_runs_api.get_query_run(query_run_id, x_database_id=database_id) if run.status in TERMINAL_STATUSES: break time.sleep(POLL_INTERVAL_S) @@ -70,13 +70,13 @@ def test_query_async_polling( ) assert run.row_count == 1 - runs_listing = query_runs_api.list_query_runs(limit=50) + runs_listing = query_runs_api.list_query_runs(x_database_id=database_id, limit=50) assert any(r.id == query_run_id for r in runs_listing.query_runs), ( f"query run {query_run_id} not surfaced by list_query_runs" ) if run.result_id: - result = results_api.get_result(run.result_id) + result = results_api.get_result(run.result_id, x_database_id=database_id) assert result.result_id == run.result_id assert result.status in {"ready", "processing"} if result.status == "ready": @@ -85,7 +85,7 @@ def test_query_async_polling( # ResultInfo (list_results) exposes the id as `id`, not `result_id` — # only GetResultResponse uses `result_id`. - results_listing = results_api.list_results(limit=50) + results_listing = results_api.list_results(x_database_id=database_id, limit=50) assert any(r.id == run.result_id for r in results_listing.results), ( f"result {run.result_id} not surfaced by list_results" ) diff --git a/tests/integration/test_results_arrow.py b/tests/integration/test_results_arrow.py index c2d561b..9a99ac1 100644 --- a/tests/integration/test_results_arrow.py +++ b/tests/integration/test_results_arrow.py @@ -70,7 +70,7 @@ def test_results_arrow( deadline = time.monotonic() + POLL_TIMEOUT_S run = None while time.monotonic() < deadline: - run = query_runs_api.get_query_run(query_run_id) + run = query_runs_api.get_query_run(query_run_id, x_database_id=database_id) if run.status in TERMINAL_STATUSES: break time.sleep(POLL_INTERVAL_S) @@ -85,7 +85,7 @@ def test_results_arrow( # ResultNotReadyError on 202. deadline = time.monotonic() + POLL_TIMEOUT_S while time.monotonic() < deadline: - result = results_api.get_result(result_id) + result = results_api.get_result(result_id, x_database_id=database_id) if result.status == "ready": break time.sleep(POLL_INTERVAL_S) @@ -93,7 +93,7 @@ def test_results_arrow( pytest.fail(f"result {result_id} never became ready") # Buffered: returns a full pyarrow.Table. - table = results_api.get_result_arrow(result_id) + table = results_api.get_result_arrow(result_id, database_id) assert isinstance(table, pa.Table) assert table.num_rows == 2 assert set(table.column_names) == {"x", "msg"} @@ -101,11 +101,11 @@ def test_results_arrow( assert table.column("msg").to_pylist() == ["hello", "world"] # Streaming: same data via RecordBatchStreamReader. - with results_api.stream_result_arrow(result_id) as reader: + with results_api.stream_result_arrow(result_id, database_id) as reader: streamed = pa.Table.from_batches(list(reader), schema=reader.schema) assert streamed.equals(table) # Pagination forwards correctly. - sliced = results_api.get_result_arrow(result_id, offset=1, limit=1) + sliced = results_api.get_result_arrow(result_id, database_id, offset=1, limit=1) assert sliced.num_rows == 1 assert sliced.column("x").to_pylist() == [2] diff --git a/tests/test_arrow.py b/tests/test_arrow.py index 3accdbc..8e1f238 100644 --- a/tests/test_arrow.py +++ b/tests/test_arrow.py @@ -148,7 +148,7 @@ def test_get_result_arrow_returns_table(monkeypatch: pytest.MonkeyPatch) -> None _install_fake_response(monkeypatch, fake, captured) results, _ = _make_results_api() - got = results.get_result_arrow("res_123") + got = results.get_result_arrow("res_123", "db_x") assert got.equals(table) # Connection is drained (not just released) so a partially-read body can't @@ -166,6 +166,8 @@ def test_get_result_arrow_returns_table(monkeypatch: pytest.MonkeyPatch) -> None assert "format=arrow" in call["url"] # Accept header overrides the JSON default. assert call["headers"]["Accept"] == ARROW_STREAM_MEDIA_TYPE + # Results are database-scoped: the required X-Database-Id header is sent. + assert call["headers"]["X-Database-Id"] == "db_x" def test_get_result_arrow_forwards_offset_and_limit( @@ -180,7 +182,7 @@ def test_get_result_arrow_forwards_offset_and_limit( _install_fake_response(monkeypatch, fake, captured) results, _ = _make_results_api() - results.get_result_arrow("res_123", offset=10, limit=100) + results.get_result_arrow("res_123", "db_x", offset=10, limit=100) url = captured[0]["url"] assert "offset=10" in url @@ -199,7 +201,7 @@ def test_stream_result_arrow_yields_reader( _install_fake_response(monkeypatch, fake, []) results, _ = _make_results_api() - with results.stream_result_arrow("res_123") as reader: + with results.stream_result_arrow("res_123", "db_x") as reader: batches = list(reader) assert batches, "expected at least one RecordBatch" roundtrip = pa.Table.from_batches(batches, schema=reader.schema) @@ -232,7 +234,7 @@ def test_get_result_arrow_raises_when_not_ready( results, _ = _make_results_api() with pytest.raises(ResultNotReadyError) as ei: - results.get_result_arrow("res_pending") + results.get_result_arrow("res_pending", "db_x") assert ei.value.status == "processing" assert ei.value.result_id == "res_pending" @@ -252,7 +254,7 @@ def test_get_result_arrow_raises_api_exception_on_404( results, _ = _make_results_api() with pytest.raises(ApiException) as ei: - results.get_result_arrow("res_missing") + results.get_result_arrow("res_missing", "db_x") assert ei.value.status == 404 assert fake.release_conn_called @@ -278,7 +280,7 @@ def test_get_result_arrow_raises_api_exception_on_409_failed( results, _ = _make_results_api() with pytest.raises(ApiException) as ei: - results.get_result_arrow("res_failed") + results.get_result_arrow("res_failed", "db_x") assert ei.value.status == 409 diff --git a/tests/test_query.py b/tests/test_query.py index 1b6b60d..a890cc5 100644 --- a/tests/test_query.py +++ b/tests/test_query.py @@ -378,6 +378,77 @@ def get_result(result_id, offset=None, limit=None, format=None, **kw): assert ei.value.status == "processing" +# --- database-scoped auto-follow ---------------------------------------------- + + +def test_autofollow_forwards_x_database_id_header() -> None: + # Results and query runs are database-scoped (the required X-Database-Id + # header). A truncated result followed via query(..., x_database_id=...) must + # carry that database id on every follow-up request: the readiness poll, the + # query-run total lookup, and each page fetch. + preview = _preview(truncated=True, total=None, rows=[[1]]) + full = [[1]] + seen_result_dbs: List[Any] = [] + seen_run_dbs: List[Any] = [] + + def get_result(result_id, offset=None, limit=None, format=None, **kw): + seen_result_dbs.append(kw.get("x_database_id")) + if format is None: # readiness poll + return _result(status="ready") + return _result(status="ready", rows=full[offset:offset + limit]) + + def get_query_run(query_run_id, **kw): + seen_run_dbs.append(kw.get("x_database_id")) + return SimpleNamespace(row_count=1) + + with ExitStack() as stack: + stack.enter_context( + patch.object(_GeneratedQueryApi, "query", return_value=preview) + ) + stack.enter_context( + patch.object(_ResultsApi, "get_result", side_effect=get_result) + ) + stack.enter_context( + patch.object(_QueryRunsApi, "get_query_run", side_effect=get_query_run) + ) + stack.enter_context(patch("hotdata.query.time.sleep")) + out = _api(poll=PollPolicy(page_size=1000)).query(REQ, x_database_id="db_x") + + assert out.rows == full + # Every follow-up request carried the header database id. + assert seen_result_dbs and all(db == "db_x" for db in seen_result_dbs) + assert seen_run_dbs == ["db_x"] + + +def test_autofollow_uses_body_database_id_when_no_header() -> None: + # When no header is passed, the effective database for auto-follow falls back + # to the request-body database_id (the spec requires exactly one database + # source on /v1/query): query() sends that on the follow-up fetches. + preview = _preview(truncated=True, total=1, rows=[[1]]) + full = [[1]] + seen_result_dbs: List[Any] = [] + + def get_result(result_id, offset=None, limit=None, format=None, **kw): + seen_result_dbs.append(kw.get("x_database_id")) + if format is None: + return _result(status="ready") + return _result(status="ready", rows=full[offset:offset + limit]) + + req_with_db = QueryRequest(sql="SELECT 1 AS x", database_id="db_body") + with ExitStack() as stack: + stack.enter_context( + patch.object(_GeneratedQueryApi, "query", return_value=preview) + ) + stack.enter_context( + patch.object(_ResultsApi, "get_result", side_effect=get_result) + ) + stack.enter_context(patch("hotdata.query.time.sleep")) + out = _api(poll=PollPolicy(page_size=1000)).query(req_with_db) + + assert out.rows == full + assert seen_result_dbs and all(db == "db_body" for db in seen_result_dbs) + + # --- forward-compatible deserialization --------------------------------------