Feat: Enhance GPU Tests, Docs & Private Cluster Setup

gcloud/.gitignore

@@ -1,5 +1,17 @@
-init/*/
-tls/
-#tmp/
+# Env files
+env.json
+*.env
+
+# Logs
+logs/
+*.log
+
+# Temp files
+tmp/
+.DS_Store
 *~
-env.json
+
+# State
+state.db
+state.json
+*.db-journal

gcloud/README.md

@@ -1,6 +1,6 @@
 <!--
 
-Copyright 2021 Google LLC and contributors
+Copyright 2021-2026 Google LLC and contributors
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -16,73 +16,102 @@ limitations under the License.
 
 -->
 
-## Dataproc Environment Reproduction Scripts
+# Dataproc Environment Reproduction Scripts
 
-This collection of bash scripts helps create and manage Google Cloud environments to reproduce and test Google Dataproc cluster setups, particularly useful for troubleshooting issues related to startup scripts, initialization actions, and network configurations.
+This collection of bash scripts facilitates the rapid provisioning, management, and teardown of Google Cloud environments to reproduce and test Google Dataproc cluster setups. It is particularly useful for troubleshooting initialization actions, custom images, complex network topologies (like SWP and NAT), and GPU driver integrations.
 
-## Setup
+**Core Principles:**
 
-1.  **Clone the repository:**
-    ```bash
-    git clone https://github.com/GoogleCloudDataproc/cloud-dataproc
-    cd cloud-dataproc/gcloud
-    ```
+*   **Declarative & Idempotent:** The `create-dpgce` script operates declaratively. It audits the current state of the cloud environment and only creates missing resources required to reach the desired configuration state (defined by CLI flags and `env.json`).
+*   **Audit-Driven:** The `audit-dpgce` (and `audit-dpgke`) script forms the foundation, performing a comprehensive, concurrent scan of your GCP project to populate a local SQLite database (`.state/state.db`).
+*   **Stateful Cache:** A persistent local SQLite database (`.state/state.db`) caches audit results and persists your operational configuration flags between runs (e.g., remembering if you deployed with `--gpu` and `--custom`).
+*   **Modular:** Core infrastructure logic is organized cleanly into functions within the `lib/` directory.
 
-2.  **Configure Environment:**
-    *   Copy the sample configuration: `cp env.json.sample env.json`
-    *   Edit `env.json` with your specific Google Cloud project details, region, network ranges, etc. Key fields include:
-        *   `PROJECT_ID`
-        *   `REGION`
-        *   `ZONE` (often derived from REGION, e.g., `us-west4-b`)
-        *   `BUCKET` (for staging)
-        *   `TEMP_BUCKET`
-        *   Other fields as needed for your test case.
+## Supported Scenarios
 
-3.  **Review Script Libraries:** The core logic is now modularized into files within the `lib/` directory, categorized by function (e.g., `lib/gcp`, `lib/network`, `lib/dataproc`).
+These scripts deploy Dataproc clusters in various configurations, supporting both Compute Engine (DPGCE) and Kubernetes Engine (DPGKE) architectures:
 
-## Scripts
+*   **Standard Dataproc on GCE (DPGCE):** Clusters with default or advanced network settings.
+*   **Dataproc on GKE (DPGKE):** Provisioning of GKE clusters and registration of Dataproc virtual clusters.
+*   **Egress Control:** Options for `--nat-egress` (Cloud NAT) or `--swp-egress` (Secure Web Proxy) to test isolated network environments.
+*   **Custom Images & Secure Boot:** Support for deploying clusters using pre-built custom images (`--custom`) and validating Shielded VM Secure Boot constraints.
+*   **GPU-Enabled Clusters:** Facilitates testing hardware accelerators (`--gpu`), integrating seamlessly with local or remote GPU initialization scripts.
 
-The main scripts are located in the `bin/` directory:
+## Setup & Configuration
 
-*   **`bin/create-dpgce`**: Creates a standard Dataproc on GCE cluster environment, including VPC, subnets, NAT, router, and firewall rules.
-*   **`bin/create-dpgce-private`**: Creates a private Dataproc on GCE cluster environment. This setup uses a Secure Web Proxy (SWP) for controlled egress and does *not* include a Cloud NAT or default internet route.
-*   **`bin/create-dpgke`**: Sets up a Dataproc on GKE environment.
-*   **`bin/destroy-dpgce`**: Tears down the environment created by `bin/create-dpgce`.
-*   **`bin/destroy-dpgce-private`**: Tears down the environment created by `bin/create-dpgce-private`.
-*   **`bin/destroy-dpgke`**: Tears down the DPGKE environment.
-*   **`bin/recreate-dpgce`**: Quickly deletes and recreates the Dataproc cluster within the existing `dpgce` environment.
-*   **`bin/recreate-dpgke`**: Quickly deletes and recreates the DPGKE cluster.
+1.  **Prerequisites:**
+    *   `gcloud` CLI and `gsutil`
+    *   `jq` (for JSON parsing)
+    *   `sqlite3` (for state cache queries)
+    *   `perl` (used in some robust text-manipulation utilities)
 
-### Common Flags
+2.  **Configure Environment (`env.json`):**
+    Copy the sample configuration file to begin:
+    ```bash
+    cp env.json.sample env.json
+    ```
+    Edit `env.json` with your specific details. Critical fields include:
+    *   `PROJECT_ID`, `REGION`, `ZONE`: Your target GCP deployment topology.
+    *   `IMAGE_VERSION`: The Dataproc OS/Version to test (e.g., `2.2-ubuntu22`, `2.0-rocky8`).
+    *   `CUSTOM_IMAGE_URI`: The specific GCP image URI to use when deploying with `--custom`.
+    *   `ACCELERATOR_TYPE`: The GPU hardware type to attach when deploying with `--gpu` (e.g., `nvidia-tesla-t4`).
+    *   `BUCKET` / `TEMP_BUCKET`: Target GCS buckets for staging initialization scripts and staging large files.
+    *   `RANGE`, `PRIVATE_RANGE`, `SWP_RANGE`: Subnet CIDR blocks.
 
-*   `--no-create-cluster`: Used with `create-*` scripts. Sets up all networking and dependencies but skips the final `gcloud dataproc clusters create` command. Useful for preparing an environment.
-*   `--force`: Used with `destroy-*` scripts. By default, GCS buckets and versioned SWP Certificate Authority components are not deleted. Use `--force` to remove these as well.
-*   `--quiet-gcloud`: Used with `create-*` scripts. Suppresses the pretty-printing of the `gcloud dataproc clusters create` command.
-*   `DEBUG=1`: Set this environment variable before running any script to enable verbose debug output (e.g., `DEBUG=1 bash bin/create-dpgce`).
-*   `TIMESTAMP=<number>`: Set this to a specific Unix timestamp to attempt to resume a previous `create` operation or to target specific versioned resources for deletion. If not set, a new timestamp is generated for each run.
+## Main Lifecycle Scripts (`bin/`)
 
-## Customizing Cluster Creation
+The core workflow centers around the following lifecycle management scripts. Note that scripts ending in `-dpgce` target Dataproc on Compute Engine, while `-dpgke` target Dataproc on GKE.
 
-The parameters for the `gcloud dataproc clusters create` command are primarily defined within `lib/dataproc/cluster.sh` in the `create_dpgce_cluster` function. You can adjust machine types, accelerators, metadata, properties, and initialization actions in this function.
+*   **`bin/audit-dpgce` / `bin/audit-dpgke`**: Queries the live cloud environment to discover deployed resources and updates the local SQLite state cache. Typically called automatically, but useful for manual state inspection.
+*   **`bin/create-dpgce` / `bin/create-dpgke`**: The idempotent creation script. Generates a deployment plan based on missing infrastructure and creates the necessary resources (networks, routers, proxies, node pools, clusters). 
+*   **`bin/destroy-dpgce` / `bin/destroy-dpgke`**: The teardown script. Audits the environment and de-provisions all discovered resources in a safe dependency order. Add `--force` to forcefully delete persistent storage (GCS buckets) and SWP policies.
+*   **`bin/recreate-dpgce` / `bin/recreate-dpgke`**: Utility script to rapidly delete and recreate *only* the Dataproc cluster (or GKE node pools) while leaving the underlying network infrastructure intact. It intelligently loads the *last used flags* from `.state/state.db`.
 
-Numerous examples of alternative configurations and common options can be found in `docs/dataproc_cluster_examples.md`.
+## Utilities (`bin/`)
 
-## Idempotency and Sentinels
+*   **`bin/ssh-m [node-index] [command...]`**: SSH into the -m node. Target HA -m nodes using numeric indexes (e.g., `bin/ssh-m 1` for `-m-1`).
+*   **`bin/ssh-w [node-index] [command...]`**: SSH into a worker node (e.g., `bin/ssh-w 0` for `-w-0`).
+*   **`bin/scp-m` / `bin/scp-w`**: Optimized file transfer to cluster nodes. These scripts bypass slow IAP TCP windowing by staging files to a GCS `TEMP_BUCKET` and invoking a remote `gcloud storage cp` pull on the node, dramatically reducing transfer times.
+*   **`bin/setup-cicd.sh`**: Automates the provisioning of a Cloud Build CI/CD pipeline, connecting Cloud Source Repositories, and configuring necessary IAM service accounts for remote integration testing.
 
-The `create-*` scripts use sentinel files to track the completion of major steps. These sentinels are stored in `/tmp/dataproc-repro/${RESOURCE_SUFFIX}/sentinels/`. This allows you to re-run a `create-*` script, and it will skip steps that were already completed successfully in a previous run with the same `TIMESTAMP`.
+## Fast Iterative Development (Initialization Actions)
 
-The `destroy-*` scripts remove the corresponding sentinel files.
+When developing or debugging complex initialization actions (like GPU drivers), destroying and recreating the entire Dataproc cluster takes too much time. Use this optimized workflow for rapid manual testing:
 
-## Logging
+1.  **Provision a Bare Cluster:** Deploy the cluster with hardware attached but bypass the initialization action execution during boot.
+    ```bash
+    ./bin/recreate-dpgce --gpu --no-init-action
+    ```
+2.  **Stage Your Script:** Use the optimized `scp-m` command to transfer your local development script to the node quickly.
+    ```bash
+    ./bin/scp-m /path/to/your/install_gpu_driver.sh
+    ```
+4.  **Execute and Monitor (Robust Execution):** Instead of standard SSH, use the `install-in-screen.sh` wrapper to execute the script. This safely encapsulates the execution in a detached `screen` session. If your SSH connection drops, running the command again will instantly re-attach you without interrupting the build.
+    ```bash
+    cd ../initialization-actions
+    ./gpu/install-in-screen.sh
+    ```
+5.  *(Idempotent Retries)*: If your script uses completion sentinels, purge them before testing your fix to ensure the specific phase executes again.
+    ```bash
+    cd ../cloud-dataproc/gcloud
+    ./bin/ssh-m 'sudo rm -rf /opt/install-dpgce/complete'
+    ```
 
-All `gcloud` commands executed via the `run_gcloud` helper function have their stdout and stderr redirected to log files within the `/tmp/dataproc-repro/${RESOURCE_SUFFIX}/` directory. Check these logs for details on any failures.
+## Common CLI Flags
 
-## Troubleshooting
+Applicable primarily to `create-dpgce` and `recreate-dpgce`:
 
-*   **"command not found"**: Ensure the `bin/` script you are running sources the necessary files from the `lib/` subdirectories.
-*   **Resource Deletion Failures:** Check the logs in `/tmp/dataproc-repro/${RESOURCE_SUFFIX}/` for the specific `gcloud` error. Often, dependencies prevent deletion. Use `--force` with destroy scripts to be more aggressive.
-*   **Service Account Permissions:** Cluster creation can fail if the service account doesn't have the required roles. The `create_service_account` function attempts to bind these, but errors can occur. Check the `bind_*.log` files.
+*   `--custom` / `--no-custom`: Toggle between the `CUSTOM_IMAGE_URI` and standard `IMAGE_VERSION` defined in `env.json`.
+*   `--nat-egress` / `--no-nat-egress`: Enable/disable Cloud NAT for outbound internet access on the cluster subnet.
+*   `--swp-egress` / `--no-swp-egress`: Enable/disable Secure Web Proxy (SWP) for restricted, proxied internet egress.
+*   `--gpu` / `--no-gpu`: Enable/disable attachment of the `ACCELERATOR_TYPE` to the cluster nodes.
+*   `--no-init-action`: Provisions the cluster but skips appending initialization action URIs to the cluster creation command. Useful for development scenarios where init actions are applied manually after boot.
+*   `--no-create-cluster`: Sets up all underlying networking, proxies, and dependencies but skips the final `gcloud dataproc clusters create` command.
 
-## Private Cluster Networking
+## Debugging
 
-The `create-dpgce-private` script sets up a VPC with no default internet route. Egress is intended to be handled by the Secure Web Proxy. Nodes in this cluster should not have direct internet access.
+*   **Trace Execution:** Set `DEBUG=1` before running any script to enable verbose bash execution tracing (`set -x`).
+    ```bash
+    DEBUG=1 bash bin/create-dpgce --nat-egress
+    ```
+*   **Audit Logs:** Detailed execution logs for the scripts are stored in timestamped directories under `/tmp/dataproc-repro/`.