diff --git a/gcloud/.gitignore b/gcloud/.gitignore
index e6a83821..43a7e150 100644
--- a/gcloud/.gitignore
+++ b/gcloud/.gitignore
@@ -1,5 +1,17 @@
-init/*/
-tls/
-#tmp/
+# Env files
+env.json
+*.env
+
+# Logs
+logs/
+*.log
+
+# Temp files
+tmp/
+.DS_Store
 *~
-env.json
\ No newline at end of file
+
+# State
+state.db
+state.json
+*.db-journal
diff --git a/gcloud/README.md b/gcloud/README.md
index 59ccbbaa..e77aec50 100644
--- a/gcloud/README.md
+++ b/gcloud/README.md
@@ -1,6 +1,6 @@
 <!--
 
-Copyright 2021 Google LLC and contributors
+Copyright 2021-2026 Google LLC and contributors
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -16,73 +16,102 @@ limitations under the License.
 
 -->
 
-## Dataproc Environment Reproduction Scripts
+# Dataproc Environment Reproduction Scripts
 
-This collection of bash scripts helps create and manage Google Cloud environments to reproduce and test Google Dataproc cluster setups, particularly useful for troubleshooting issues related to startup scripts, initialization actions, and network configurations.
+This collection of bash scripts facilitates the rapid provisioning, management, and teardown of Google Cloud environments to reproduce and test Google Dataproc cluster setups. It is particularly useful for troubleshooting initialization actions, custom images, complex network topologies (like SWP and NAT), and GPU driver integrations.
 
-## Setup
+**Core Principles:**
 
-1.  **Clone the repository:**
-    ```bash
-    git clone https://github.com/GoogleCloudDataproc/cloud-dataproc
-    cd cloud-dataproc/gcloud
-    ```
+*   **Declarative & Idempotent:** The `create-dpgce` script operates declaratively. It audits the current state of the cloud environment and only creates missing resources required to reach the desired configuration state (defined by CLI flags and `env.json`).
+*   **Audit-Driven:** The `audit-dpgce` (and `audit-dpgke`) script forms the foundation, performing a comprehensive, concurrent scan of your GCP project to populate a local SQLite database (`.state/state.db`).
+*   **Stateful Cache:** A persistent local SQLite database (`.state/state.db`) caches audit results and persists your operational configuration flags between runs (e.g., remembering if you deployed with `--gpu` and `--custom`).
+*   **Modular:** Core infrastructure logic is organized cleanly into functions within the `lib/` directory.
 
-2.  **Configure Environment:**
-    *   Copy the sample configuration: `cp env.json.sample env.json`
-    *   Edit `env.json` with your specific Google Cloud project details, region, network ranges, etc. Key fields include:
-        *   `PROJECT_ID`
-        *   `REGION`
-        *   `ZONE` (often derived from REGION, e.g., `us-west4-b`)
-        *   `BUCKET` (for staging)
-        *   `TEMP_BUCKET`
-        *   Other fields as needed for your test case.
+## Supported Scenarios
 
-3.  **Review Script Libraries:** The core logic is now modularized into files within the `lib/` directory, categorized by function (e.g., `lib/gcp`, `lib/network`, `lib/dataproc`).
+These scripts deploy Dataproc clusters in various configurations, supporting both Compute Engine (DPGCE) and Kubernetes Engine (DPGKE) architectures:
 
-## Scripts
+*   **Standard Dataproc on GCE (DPGCE):** Clusters with default or advanced network settings.
+*   **Dataproc on GKE (DPGKE):** Provisioning of GKE clusters and registration of Dataproc virtual clusters.
+*   **Egress Control:** Options for `--nat-egress` (Cloud NAT) or `--swp-egress` (Secure Web Proxy) to test isolated network environments.
+*   **Custom Images & Secure Boot:** Support for deploying clusters using pre-built custom images (`--custom`) and validating Shielded VM Secure Boot constraints.
+*   **GPU-Enabled Clusters:** Facilitates testing hardware accelerators (`--gpu`), integrating seamlessly with local or remote GPU initialization scripts.
 
-The main scripts are located in the `bin/` directory:
+## Setup & Configuration
 
-*   **`bin/create-dpgce`**: Creates a standard Dataproc on GCE cluster environment, including VPC, subnets, NAT, router, and firewall rules.
-*   **`bin/create-dpgce-private`**: Creates a private Dataproc on GCE cluster environment. This setup uses a Secure Web Proxy (SWP) for controlled egress and does *not* include a Cloud NAT or default internet route.
-*   **`bin/create-dpgke`**: Sets up a Dataproc on GKE environment.
-*   **`bin/destroy-dpgce`**: Tears down the environment created by `bin/create-dpgce`.
-*   **`bin/destroy-dpgce-private`**: Tears down the environment created by `bin/create-dpgce-private`.
-*   **`bin/destroy-dpgke`**: Tears down the DPGKE environment.
-*   **`bin/recreate-dpgce`**: Quickly deletes and recreates the Dataproc cluster within the existing `dpgce` environment.
-*   **`bin/recreate-dpgke`**: Quickly deletes and recreates the DPGKE cluster.
+1.  **Prerequisites:**
+    *   `gcloud` CLI and `gsutil`
+    *   `jq` (for JSON parsing)
+    *   `sqlite3` (for state cache queries)
+    *   `perl` (used in some robust text-manipulation utilities)
 
-### Common Flags
+2.  **Configure Environment (`env.json`):**
+    Copy the sample configuration file to begin:
+    ```bash
+    cp env.json.sample env.json
+    ```
+    Edit `env.json` with your specific details. Critical fields include:
+    *   `PROJECT_ID`, `REGION`, `ZONE`: Your target GCP deployment topology.
+    *   `IMAGE_VERSION`: The Dataproc OS/Version to test (e.g., `2.2-ubuntu22`, `2.0-rocky8`).
+    *   `CUSTOM_IMAGE_URI`: The specific GCP image URI to use when deploying with `--custom`.
+    *   `ACCELERATOR_TYPE`: The GPU hardware type to attach when deploying with `--gpu` (e.g., `nvidia-tesla-t4`).
+    *   `BUCKET` / `TEMP_BUCKET`: Target GCS buckets for staging initialization scripts and staging large files.
+    *   `RANGE`, `PRIVATE_RANGE`, `SWP_RANGE`: Subnet CIDR blocks.
 
-*   `--no-create-cluster`: Used with `create-*` scripts. Sets up all networking and dependencies but skips the final `gcloud dataproc clusters create` command. Useful for preparing an environment.
-*   `--force`: Used with `destroy-*` scripts. By default, GCS buckets and versioned SWP Certificate Authority components are not deleted. Use `--force` to remove these as well.
-*   `--quiet-gcloud`: Used with `create-*` scripts. Suppresses the pretty-printing of the `gcloud dataproc clusters create` command.
-*   `DEBUG=1`: Set this environment variable before running any script to enable verbose debug output (e.g., `DEBUG=1 bash bin/create-dpgce`).
-*   `TIMESTAMP=<number>`: Set this to a specific Unix timestamp to attempt to resume a previous `create` operation or to target specific versioned resources for deletion. If not set, a new timestamp is generated for each run.
+## Main Lifecycle Scripts (`bin/`)
 
-## Customizing Cluster Creation
+The core workflow centers around the following lifecycle management scripts. Note that scripts ending in `-dpgce` target Dataproc on Compute Engine, while `-dpgke` target Dataproc on GKE.
 
-The parameters for the `gcloud dataproc clusters create` command are primarily defined within `lib/dataproc/cluster.sh` in the `create_dpgce_cluster` function. You can adjust machine types, accelerators, metadata, properties, and initialization actions in this function.
+*   **`bin/audit-dpgce` / `bin/audit-dpgke`**: Queries the live cloud environment to discover deployed resources and updates the local SQLite state cache. Typically called automatically, but useful for manual state inspection.
+*   **`bin/create-dpgce` / `bin/create-dpgke`**: The idempotent creation script. Generates a deployment plan based on missing infrastructure and creates the necessary resources (networks, routers, proxies, node pools, clusters). 
+*   **`bin/destroy-dpgce` / `bin/destroy-dpgke`**: The teardown script. Audits the environment and de-provisions all discovered resources in a safe dependency order. Add `--force` to forcefully delete persistent storage (GCS buckets) and SWP policies.
+*   **`bin/recreate-dpgce` / `bin/recreate-dpgke`**: Utility script to rapidly delete and recreate *only* the Dataproc cluster (or GKE node pools) while leaving the underlying network infrastructure intact. It intelligently loads the *last used flags* from `.state/state.db`.
 
-Numerous examples of alternative configurations and common options can be found in `docs/dataproc_cluster_examples.md`.
+## Utilities (`bin/`)
 
-## Idempotency and Sentinels
+*   **`bin/ssh-m [node-index] [command...]`**: SSH into the -m node. Target HA -m nodes using numeric indexes (e.g., `bin/ssh-m 1` for `-m-1`).
+*   **`bin/ssh-w [node-index] [command...]`**: SSH into a worker node (e.g., `bin/ssh-w 0` for `-w-0`).
+*   **`bin/scp-m` / `bin/scp-w`**: Optimized file transfer to cluster nodes. These scripts bypass slow IAP TCP windowing by staging files to a GCS `TEMP_BUCKET` and invoking a remote `gcloud storage cp` pull on the node, dramatically reducing transfer times.
+*   **`bin/setup-cicd.sh`**: Automates the provisioning of a Cloud Build CI/CD pipeline, connecting Cloud Source Repositories, and configuring necessary IAM service accounts for remote integration testing.
 
-The `create-*` scripts use sentinel files to track the completion of major steps. These sentinels are stored in `/tmp/dataproc-repro/${RESOURCE_SUFFIX}/sentinels/`. This allows you to re-run a `create-*` script, and it will skip steps that were already completed successfully in a previous run with the same `TIMESTAMP`.
+## Fast Iterative Development (Initialization Actions)
 
-The `destroy-*` scripts remove the corresponding sentinel files.
+When developing or debugging complex initialization actions (like GPU drivers), destroying and recreating the entire Dataproc cluster takes too much time. Use this optimized workflow for rapid manual testing:
 
-## Logging
+1.  **Provision a Bare Cluster:** Deploy the cluster with hardware attached but bypass the initialization action execution during boot.
+    ```bash
+    ./bin/recreate-dpgce --gpu --no-init-action
+    ```
+2.  **Stage Your Script:** Use the optimized `scp-m` command to transfer your local development script to the node quickly.
+    ```bash
+    ./bin/scp-m /path/to/your/install_gpu_driver.sh
+    ```
+4.  **Execute and Monitor (Robust Execution):** Instead of standard SSH, use the `install-in-screen.sh` wrapper to execute the script. This safely encapsulates the execution in a detached `screen` session. If your SSH connection drops, running the command again will instantly re-attach you without interrupting the build.
+    ```bash
+    cd ../initialization-actions
+    ./gpu/install-in-screen.sh
+    ```
+5.  *(Idempotent Retries)*: If your script uses completion sentinels, purge them before testing your fix to ensure the specific phase executes again.
+    ```bash
+    cd ../cloud-dataproc/gcloud
+    ./bin/ssh-m 'sudo rm -rf /opt/install-dpgce/complete'
+    ```
 
-All `gcloud` commands executed via the `run_gcloud` helper function have their stdout and stderr redirected to log files within the `/tmp/dataproc-repro/${RESOURCE_SUFFIX}/` directory. Check these logs for details on any failures.
+## Common CLI Flags
 
-## Troubleshooting
+Applicable primarily to `create-dpgce` and `recreate-dpgce`:
 
-*   **"command not found"**: Ensure the `bin/` script you are running sources the necessary files from the `lib/` subdirectories.
-*   **Resource Deletion Failures:** Check the logs in `/tmp/dataproc-repro/${RESOURCE_SUFFIX}/` for the specific `gcloud` error. Often, dependencies prevent deletion. Use `--force` with destroy scripts to be more aggressive.
-*   **Service Account Permissions:** Cluster creation can fail if the service account doesn't have the required roles. The `create_service_account` function attempts to bind these, but errors can occur. Check the `bind_*.log` files.
+*   `--custom` / `--no-custom`: Toggle between the `CUSTOM_IMAGE_URI` and standard `IMAGE_VERSION` defined in `env.json`.
+*   `--nat-egress` / `--no-nat-egress`: Enable/disable Cloud NAT for outbound internet access on the cluster subnet.
+*   `--swp-egress` / `--no-swp-egress`: Enable/disable Secure Web Proxy (SWP) for restricted, proxied internet egress.
+*   `--gpu` / `--no-gpu`: Enable/disable attachment of the `ACCELERATOR_TYPE` to the cluster nodes.
+*   `--no-init-action`: Provisions the cluster but skips appending initialization action URIs to the cluster creation command. Useful for development scenarios where init actions are applied manually after boot.
+*   `--no-create-cluster`: Sets up all underlying networking, proxies, and dependencies but skips the final `gcloud dataproc clusters create` command.
 
-## Private Cluster Networking
+## Debugging
 
-The `create-dpgce-private` script sets up a VPC with no default internet route. Egress is intended to be handled by the Secure Web Proxy. Nodes in this cluster should not have direct internet access.
\ No newline at end of file
+*   **Trace Execution:** Set `DEBUG=1` before running any script to enable verbose bash execution tracing (`set -x`).
+    ```bash
+    DEBUG=1 bash bin/create-dpgce --nat-egress
+    ```
+*   **Audit Logs:** Detailed execution logs for the scripts are stored in timestamped directories under `/tmp/dataproc-repro/`.
\ No newline at end of file
diff --git a/gcloud/bin/audit-dpgce b/gcloud/bin/audit-dpgce
new file mode 100755
index 00000000..468916bc
--- /dev/null
+++ b/gcloud/bin/audit-dpgce
@@ -0,0 +1,268 @@
+#!/bin/bash
+#
+# Universal audit script for all Dataproc on GCE environment variations.
+#
+# This script populates the state.db SQLite database with the current state of GCP resources.
+
+# Exit on failure
+set -e
+
+# --- Argument Parsing ---
+ARG_TIMESTAMP=""
+while (( "$#" )); do
+  case "$1" in
+    --timestamp)
+      if [[ -n "$2" && "$2" != --* ]]; then
+        ARG_TIMESTAMP="$2"
+        shift 2
+      else
+        echo "Error: --timestamp requires an argument" >&2
+        exit 1
+      fi
+      ;;
+    -*)
+      echo "Error: Unsupported flag $1" >&2
+      exit 1
+      ;;
+    *)
+      # Ignore other params
+      shift
+      ;;
+  esac
+done
+
+export ARG_TIMESTAMP
+
+# --- Get script's real directory ---
+SCRIPT_DIR=$(dirname "${BASH_SOURCE[0]}")
+GCLOUD_DIR="$(realpath "${SCRIPT_DIR}/..")"
+export GCLOUD_DIR
+
+# --- Source environment variables and utility functions ---
+source "${GCLOUD_DIR}/lib/env.sh"
+
+# Override TIMESTAMP if provided as an argument
+if [[ -n "${ARG_TIMESTAMP}" ]]; then
+  export TIMESTAMP="${ARG_TIMESTAMP}"
+  # Re-set variables derived from TIMESTAMP
+  export REPRO_TMPDIR="/tmp/dataproc-repro/${TIMESTAMP}"
+  export LOG_DIR="${REPRO_TMPDIR}/logs"
+  mkdir -p "${REPRO_TMPDIR}"
+  mkdir -p "${LOG_DIR}"
+  echo "Using provided TIMESTAMP for resources: ${TIMESTAMP}" >&2
+fi
+
+source "${GCLOUD_DIR}/lib/script-utils.sh"
+
+# Initialize the state database and table
+init_state_db
+
+AUDIT_TEMP_DIR="${REPRO_TMPDIR}/audit_results"
+rm -rf "${AUDIT_TEMP_DIR}"
+mkdir -p "${AUDIT_TEMP_DIR}"
+PIDS=()
+
+# Function to run a check in the background, writing output to a temp file
+run_check() {
+  local key="$1"
+  local source_file="$2"
+  local func_name="$3"
+  shift 3
+  local cmd_out="${AUDIT_TEMP_DIR}/${key}.json"
+
+  (
+    cd "${GCLOUD_DIR}"
+    source "./lib/env.sh" # Ensure env vars are set in subshell
+    if [[ -n "${ARG_TIMESTAMP}" ]]; then # Re-apply override if needed
+      export TIMESTAMP="${ARG_TIMESTAMP}"
+      export REPRO_TMPDIR="/tmp/dataproc-repro/${TIMESTAMP}"
+      export LOG_DIR="${REPRO_TMPDIR}/logs"
+    fi
+    source "./lib/script-utils.sh"
+    source "./${source_file}" # Relative to GCLOUD_DIR
+    "${func_name}" "$@" > "${cmd_out}" 2> "${cmd_out%.json}.err"
+  ) &
+  PIDS+=($!)
+}
+
+# Function to run a check for commands in script-utils.sh or env.sh
+run_shared_check() {
+  local key="$1"
+  local func_name="$2"
+  shift 2
+  local cmd_out="${AUDIT_TEMP_DIR}/${key}.json"
+  (
+    cd "${GCLOUD_DIR}"
+    source "./lib/env.sh" # Ensure env vars are set in subshell
+    if [[ -n "${ARG_TIMESTAMP}" ]]; then # Re-apply override if needed
+      export TIMESTAMP="${ARG_TIMESTAMP}"
+      export REPRO_TMPDIR="/tmp/dataproc-repro/${TIMESTAMP}"
+      export LOG_DIR="${REPRO_TMPDIR}/logs"
+    fi
+    source "./lib/script-utils.sh"
+    "${func_name}" "$@" > "${cmd_out}" 2> "${cmd_out%.json}.err"
+  ) &
+  PIDS+=($!)
+}
+
+# --- Infrastructure State Checks ---
+run_shared_check "project" _check_exists gcloud projects describe "${PROJECT_ID}" --format="json(lifecycleState,projectId)"
+run_shared_check "billing" _check_exists gcloud beta billing projects describe "${PROJECT_ID}" --format="json(billingEnabled)"
+
+# --- Resource Existence Checks ---
+run_check "vpcNetwork" "lib/network/network.sh" exists_network
+run_check "standardSubnet" "lib/network/subnet.sh" exists_subnet "${SUBNET}"
+run_check "privateSubnet" "lib/network/subnet.sh" exists_subnet "${PRIVATE_SUBNET}"
+run_check "swpSubnet" "lib/network/subnet.sh" exists_subnet "${SWP_SUBNET}"
+run_check "cloudRouter" "lib/network/router.sh" exists_router
+run_check "cloudRouterNAT" "lib/network/router.sh" exists_router_nat "nat-config"
+run_check "firewallRule-ssh" "lib/swp/firewall.sh" exists_firewall_rule "fw-${CLUSTER_NAME}-in-ssh"
+run_check "firewallRule-internal" "lib/swp/firewall.sh" exists_firewall_rule "fw-${CLUSTER_NAME}-in-internal"
+run_shared_check "routes" _check_exists gcloud compute routes list --project="${PROJECT_ID}" --filter="network~/${NETWORK}$" --format="json(name,selfLink)"
+run_check "serviceAccount" "lib/gcp/iam.sh" exists_service_account
+run_check "serviceAccountBindings" "lib/gcp/iam.sh" get_service_account_bindings
+run_check "serviceAccountRolesReady" "lib/gcp/iam.sh" check_service_account_bindings
+run_check "autoscalingPolicy" "lib/dataproc/autoscaling.sh" exists_autoscaling_policy
+run_check "dataprocCluster" "lib/dataproc/cluster.sh" exists_dpgce_cluster
+run_check "dataprocClusterVMs" "lib/dataproc/cluster.sh" exists_dataproc_cluster_vms
+run_check "anyNetworkVms" "lib/gcp/misc.sh" get_any_network_vms_state
+run_check "debugVms" "lib/gcp/misc.sh" exists_debug_vms
+run_check "bigtableInstance" "lib/bigtable.sh" exists_bigtable_instance
+run_check "legacyMssqlInstance" "lib/database/mssql.sh" exists_legacy_mssql_instance
+run_check "mssqlInstance" "lib/database/mssql.sh" exists_mssql_instance
+run_check "mysqlInstance" "lib/database/mysql.sh" exists_mysql_instance
+run_check "oracleVm" "lib/database/oracle.sh" exists_oracle_vm
+run_check "pgsqlInstance" "lib/database/pgsql.sh" exists_pgsql_instance
+run_check "artifactsRepository" "lib/gcp/gcr.sh" exists_artifacts_repository
+run_check "gcsBucket" "lib/gcp/gcs.sh" exists_gcs_bucket "${BUCKET}"
+run_check "gcsTempBucket" "lib/gcp/gcs.sh" exists_gcs_bucket "${TEMP_BUCKET}"
+run_check "mysqlSecret" "lib/gcp/kms.sh" exists_secret "${MYSQL_SECRET_NAME}"
+run_check "kmsKeyring" "lib/gcp/kms.sh" exists_kms_keyring
+run_check "kdcKmsKey" "lib/gcp/kms.sh" exists_kms_key "${KDC_ROOT_PASSWD_KEY}"
+run_check "gkeCluster" "lib/gke.sh" exists_gke_cluster
+run_check "dpgkeCluster" "lib/gke.sh" exists_dpgke_cluster
+run_check "kdcServer" "lib/kerberos.sh" exists_kdc_server
+run_check "ipAllocation" "lib/network/peering.sh" exists_ip_allocation
+run_check "vpcPeering" "lib/network/peering.sh" exists_vpc_peering
+run_check "phsCluster" "lib/phs.sh" exists_phs_cluster
+run_check "swpCaPool" "lib/swp/certs.sh" exists_swp_ca_pool
+run_check "swpRootCa" "lib/swp/certs.sh" exists_swp_root_ca
+run_check "swpCic" "lib/swp/certs.sh" exists_swp_cic
+run_check "swpManagedCertificate" "lib/swp/certs.sh" exists_swp_managed_certificate
+run_check "swpFirewallIngress" "lib/swp/firewall.sh" exists_firewall_rule "allow-swp-ingress-${CLUSTER_NAME}"
+run_check "swpFirewallInternal" "lib/swp/firewall.sh" exists_firewall_rule "allow-internal-${CLUSTER_NAME}"
+run_check "swpGateway" "lib/swp/gateway.sh" exists_swp_gateway
+run_check "swpPolicy" "lib/swp/policy.sh" exists_gateway_security_policy
+
+# Wait for all background jobs to finish
+for pid in "${PIDS[@]}"; do
+  wait "${pid}" || true # Ignore errors from wait
+done
+
+echo "Populating SQLite Database..."
+# Populate STATE_DB from temp files
+for key_file in $(find "${AUDIT_TEMP_DIR}" -type f -name "*.json"); do
+  key=$(basename "${key_file}" .json)
+  value=$(cat "${key_file}")
+
+  if [[ -z "${value}" ]]; then
+    value="null"
+  fi
+  # Handle empty list from gcloud for routes
+  if [[ "${key}" == "routes" && "${value}" == "[]" ]]; then
+    value="null"
+  fi
+  update_state "${key}" "${value}"
+done
+
+# --- Human Readable Report ---
+echo "--------------------------------------"
+echo "  DPGCE Environment Audit Report      "
+echo "--------------------------------------"
+PROJECT_ID_RAW=$(get_state "project")
+if [[ "${PROJECT_ID_RAW}" == "null" || -z "${PROJECT_ID_RAW}" ]]; then
+  PROJECT_ID_VAL="N/A"
+else
+  # Check if it's valid JSON before jq
+  if echo "${PROJECT_ID_RAW}" | /usr/bin/jq empty > /dev/null 2>&1; then
+    PROJECT_ID_VAL=$(echo "${PROJECT_ID_RAW}" | /usr/bin/jq -r '.projectId // "N/A"')
+  else
+    PROJECT_ID_VAL="N/A" # Not valid JSON
+  fi
+fi
+echo "Project: ${PROJECT_ID_VAL}"
+echo
+
+print_resource_status() {
+  local display_name="$1"
+  local key="$2"
+  local value
+  value=$(get_state "${key}")
+
+  echo -n "- ${display_name}: "
+  if [[ "${value}" == "null" || "${value}" == "" ]]; then
+    report_audit_status "Not Found"
+  else
+    report_audit_status "Exists"
+  fi
+}
+print_resource_status "VPC Network (${NETWORK})" "vpcNetwork"
+print_resource_status "Standard Subnet (${SUBNET})" "standardSubnet"
+print_resource_status "Private Subnet (${PRIVATE_SUBNET})" "privateSubnet"
+print_resource_status "SWP Subnet (${SWP_SUBNET})" "swpSubnet"
+print_resource_status "Cloud Router (${ROUTER_NAME})" "cloudRouter"
+print_resource_status "Cloud NAT (nat-config)" "cloudRouterNAT"
+print_resource_status "Firewall Rule (SSH)" "firewallRule-ssh"
+print_resource_status "Firewall Rule (Internal)" "firewallRule-internal"
+print_resource_status "Service Account (${GSA})" "serviceAccount"
+print_resource_status "Dataproc Autoscaling Policy (${AUTOSCALING_POLICY_NAME})" "autoscalingPolicy"
+print_resource_status "Dataproc Cluster (${CLUSTER_NAME})" "dataprocCluster"
+print_resource_status "Dataproc Cluster ${CLUSTER_NAME} VMs" "dataprocClusterVMs"
+print_resource_status "Bigtable Instance (${BIGTABLE_INSTANCE})" "bigtableInstance"
+print_resource_status "Cloud SQL MSSQL (Legacy VM)" "legacyMssqlInstance"
+print_resource_status "Cloud SQL MSSQL (${MSSQL_INSTANCE})" "mssqlInstance"
+print_resource_status "Cloud SQL MySQL (${MYSQL_INSTANCE})" "mysqlInstance"
+print_resource_status "Oracle VM (${ORACLE_VM_NAME})" "oracleVm"
+print_resource_status "Cloud SQL PostgreSQL (${PGSQL_INSTANCE})" "pgsqlInstance"
+print_resource_status "Artifact Repository (${ARTIFACT_REPOSITORY})" "artifactsRepository"
+print_resource_status "GCS Bucket (${BUCKET})" "gcsBucket"
+print_resource_status "GCS Temp Bucket (${TEMP_BUCKET})" "gcsTempBucket"
+print_resource_status "Secret (MySQL)" "mysqlSecret"
+print_resource_status "KMS Keyring (${KMS_KEYRING})" "kmsKeyring"
+print_resource_status "KMS Key (KDC)" "kdcKmsKey"
+print_resource_status "GKE Cluster (${GKE_CLUSTER_NAME})" "gkeCluster"
+print_resource_status "DPGKE Cluster (${DPGKE_CLUSTER_NAME})" "dpgkeCluster"
+print_resource_status "KDC Server (${KDC_NAME})" "kdcServer"
+print_resource_status "IP Allocation (${ALLOCATION_NAME})" "ipAllocation"
+print_resource_status "VPC Peering" "vpcPeering"
+print_resource_status "PHS Cluster (${CLUSTER_NAME}-phs)" "phsCluster"
+print_resource_status "SWP CA Pool" "swpCaPool"
+print_resource_status "SWP Root CA" "swpRootCa"
+print_resource_status "SWP CIC" "swpCic"
+print_resource_status "SWP Managed Certificate" "swpManagedCertificate"
+print_resource_status "SWP Firewall (Ingress)" "swpFirewallIngress"
+print_resource_status "SWP Firewall (Internal)" "swpFirewallInternal"
+print_resource_status "SWP Gateway (${SWP_INSTANCE_NAME})" "swpGateway"
+print_resource_status "SWP Policy (${SWP_POLICY_NAME})" "swpPolicy"
+print_resource_status "Debug VMs" "debugVms"
+
+# Routes are a list, so handle differently
+routes=$(get_state "routes")
+echo -n "- Routes for ${NETWORK}: "
+if [[ "${routes}" == "null" || "${routes}" == "[]" || -z "${routes}" ]]; then
+  report_audit_status "Not Found"
+else
+  # Check if the JSON array is not empty
+  tmp_routes_file="${AUDIT_TEMP_DIR}/routes.json.tmp"
+  echo "${routes}" > "${tmp_routes_file}"
+  routes_len=$(/usr/bin/jq '. | length' "${tmp_routes_file}")
+  rm "${tmp_routes_file}"
+  if [[ "${routes_len}" -gt 0 ]]; then
+    report_audit_status "Exists"
+  else
+    report_audit_status "Not Found"
+  fi
+fi
+
+echo "--------------------------------------"
+echo "State DB at: ${STATE_DB}"
diff --git a/gcloud/bin/audit-dpgce-create b/gcloud/bin/audit-dpgce-create
deleted file mode 100644
index ba049144..00000000
--- a/gcloud/bin/audit-dpgce-create
+++ /dev/null
@@ -1,135 +0,0 @@
-#!/bin/bash
-
-# Script to audit resource creation after create-dpgce.sh
-
-source lib/env.sh
-source lib/gcp/misc.sh # Source the file containing configure_gcloud
-
-if (( DEBUG != 0 )); then
-  set -x
-fi
-
-configure_gcloud # Set gcloud context
-
-if [[ -z "${CLUSTER_NAME}" || "${CLUSTER_NAME}" == "null" ]]; then
-  echo "ERROR: CLUSTER_NAME is not set. Please source lib/env.sh after env.json is configured."
-  exit 1
-fi
-
-LOG_DIR="tmp/create_audit_${CLUSTER_NAME}_$(date +%Y%m%d-%H%M%S)"
-mkdir -p "${LOG_DIR}"
-echo "Detailed logs will be saved in ${LOG_DIR}"
-
-RED='\033[0;31m'
-GREEN='\033[0;32m'
-YELLOW='\033[1;33m'
-NC='\033[0m' # No Color
-
-# Function to print status updates
-print_status() {
-  local message="$1"
-  local first_word=$(echo "${message}" | awk '{print $1}')
-  local rest_of_message=$(echo "${message}" | cut -d' ' -f2-)
-  echo -en "${YELLOW}${first_word}${NC} ${rest_of_message}"
-}
-
-# Function to print result
-print_result() {
-  local result="$1"
-  if [[ "${result}" == "Pass" ]]; then
-    echo -e " [${GREEN}Pass${NC}]"
-  elif [[ "${result}" == "Exists" ]]; then
-    echo -e " [${GREEN}Exists${NC}]"
-  elif [[ "${result}" == "Fail" ]]; then
-    echo -e " [${RED}Fail${NC}]"
-  else
-    echo -e " [${YELLOW}${result}${NC}]"
-  fi
-}
-
-# Function to check if a resource exists and log details
-function check_exists() {
-  local test_name="$1"
-  local command_to_run="$2"
-  local safe_test_name=$(echo "$test_name" | tr ' /:' '___')
-  local log_file="${LOG_DIR}/${safe_test_name}.log"
-
-  print_status "Checking: ${test_name}... "
-
-  # Run the command, redirect output to log file
-  if eval "${command_to_run}" > "${log_file}" 2>&1; then
-    if [[ $(wc -l < "${log_file}") -eq 0 ]]; then
-      print_result "Fail"
-      echo "  -> ${test_name} NOT FOUND. Check ${log_file}"
-      return 1
-    else
-      print_result "Exists"
-      return 0
-    fi
-  else
-    print_result "Fail"
-    echo "  -> Command failed for ${test_name}. Check ${log_file}"
-    return 1
-  fi
-}
-
-function check_exists_grep() {
-  local test_name="$1"
-  local command_to_run="$2"
-  local grep_pattern="$3"
-  local log_file="${LOG_DIR}/$(echo "$test_name" | tr ' /:' '___').log"
-
-  print_status "Checking: ${test_name}... "
-
-  # Run the command, redirect output to log file
-  eval "${command_to_run}" > "${log_file}" 2>&1
-
-  # Grep the log file quietly
-  if grep -q "${grep_pattern}" "${log_file}"; then
-    print_result "Exists"
-    return 0
-  else
-    print_result "Fail"
-    echo "  -> ${test_name} NOT FOUND or pattern mismatch. Check ${log_file}"
-    return 1
-  fi
-}
-
-# --- Start Audit ---
-echo "Starting resource creation audit for cluster: ${CLUSTER_NAME}"
-
-# 1. VPC Network
-check_exists "VPC Network ${NETWORK}" "gcloud compute networks describe '${NETWORK}' --project='${PROJECT_ID}'"
-
-# 2. Subnet
-check_exists "Main Subnet ${SUBNET}" "gcloud compute networks subnets describe '${SUBNET}' --region='${REGION}' --project='${PROJECT_ID}'"
-
-# 3. Service Account
-check_exists "Service Account ${GSA}" "gcloud iam service-accounts describe '${GSA}' --project='${PROJECT_ID}'"
-
-# 4. GCS Buckets
-check_exists "GCS Staging Bucket gs://${BUCKET}" "gcloud storage ls --buckets 'gs://${BUCKET}'"
-check_exists "GCS Temp Bucket gs://${TEMP_BUCKET}" "gcloud storage ls --buckets 'gs://${TEMP_BUCKET}'"
-
-# 5. Cloud Router
-check_exists "Cloud Router ${ROUTER_NAME}" "gcloud compute routers describe '${ROUTER_NAME}' --region='${REGION}' --project='${PROJECT_ID}'"
-
-# 6. NAT Policy
-check_exists "NAT Policy nat-config" "gcloud compute routers nats describe nat-config --router='${ROUTER_NAME}' --region='${REGION}' --project='${PROJECT_ID}'"
-
-# 7. Firewall Rules
-check_exists_grep "Firewall Rule ${FIREWALL}-out" "gcloud compute firewall-rules list --project='${PROJECT_ID}' --filter=\"network ~ ${NETWORK}$ AND name='${FIREWALL}-out'\" --format='value(name)'" "${FIREWALL}-out"
-
-# 8. Autoscaling Policy
-check_exists "Autoscaling Policy ${AUTOSCALING_POLICY_NAME}" "gcloud dataproc autoscaling-policies describe '${AUTOSCALING_POLICY_NAME}' --region='${REGION}' --project='${PROJECT_ID}'"
-
-# 9. Dataproc Cluster (Optional)
-print_status "Checking: Dataproc Cluster ${CLUSTER_NAME}... "
-cluster_log_file="${LOG_DIR}/Dataproc_Cluster_${CLUSTER_NAME}.log"
-if gcloud dataproc clusters describe "${CLUSTER_NAME}" --region="${REGION}" --project="${PROJECT_ID}" > "${cluster_log_file}" 2>&1; then
-  print_result "Exists"
-else
-  print_result "Not Found"
-fi
-
-echo -e "\nAudit complete."
diff --git a/gcloud/bin/audit-dpgce-destroy b/gcloud/bin/audit-dpgce-destroy
deleted file mode 100644
index 5943bc78..00000000
--- a/gcloud/bin/audit-dpgce-destroy
+++ /dev/null
@@ -1,71 +0,0 @@
-#!/bin/bash
-
-# Script to audit resource cleanup after destroy-dpgce.sh
-
-source lib/env.sh
-source lib/gcp/misc.sh # Source the file containing configure_gcloud
-
-if (( DEBUG != 0 )); then
-  set -x
-fi
-
-configure_gcloud # Set gcloud context
-
-parse_args "$@" # Sets FORCE_AUDIT based on --force
-
-if [[ -z "${CLUSTER_NAME}" || "${CLUSTER_NAME}" == "null" ]]; then
-  echo "ERROR: CLUSTER_NAME is not set. Please source lib/env.sh after env.json is configured."
-  exit 1
-fi
-
-LOG_DIR="tmp/destroy_audit_${CLUSTER_NAME}_$(date +%Y%m%d-%H%M%S)"
-mkdir -p "${LOG_DIR}"
-echo "Detailed logs will be saved in ${LOG_DIR}"
-
-# --- Start Audit ---
-echo "Starting resource cleanup audit for cluster: ${CLUSTER_NAME}"
-if [[ "${FORCE_AUDIT}" == "true" ]]; then
-  echo "--force flag detected, expecting ALL resources to be deleted."
-fi
-
-# 1. Dataproc Clusters
-check_resource_exact "Dataproc Clusters" \
-  "gcloud dataproc clusters list --region=\"${REGION}\" --project=\"${PROJECT_ID}\" --filter=\"clusterName = ${CLUSTER_NAME}\" --format=\"value(clusterName)\""
-
-# 2. Service Accounts
-check_resource_exact "Service Account" \
-  "gcloud iam service-accounts list --project=\"${PROJECT_ID}\" --filter=\"email = ${GSA}\" --format=\"value(email)\""
-
-# 3. Autoscaling Policies
-check_resource "Autoscaling Policies" \
-  "gcloud dataproc autoscaling-policies list --region=\"${REGION}\" --project=\"${PROJECT_ID}\" --format=\"value(id)\"" \
-  "${AUTOSCALING_POLICY_NAME}"
-
-# 4. Cloud Routers & NAT
-check_resource "Cloud Router ${ROUTER_NAME}" \
-  "gcloud compute routers list --regions=\"${REGION}\" --project=\"${PROJECT_ID}\" --filter=\"network ~ ${NETWORK}$\" --format=\"value(name)\"" \
-  "${ROUTER_NAME}"
-check_resource "NAT on ${ROUTER_NAME}" \
-  "gcloud compute routers nats list --router='${ROUTER_NAME}' --region='${REGION}' --project='${PROJECT_ID}' --format='value(name)'" \
-  "nat-config"
-
-# 5. Firewall Rules
-check_resource "Cluster Firewall Rules" \
-  "gcloud compute firewall-rules list --project=\"${PROJECT_ID}\" --filter='network ~ \"${NETWORK}\" AND name ~ \"${CLUSTER_NAME}\"' --format=\"value(name)\"" \
-  "${CLUSTER_NAME}"
-
-# 6. Subnets
-check_resource "Main Subnet" \
-  "gcloud compute networks subnets list --network=\"${NETWORK}\" --project=\"${PROJECT_ID}\" --format=\"value(name)\"" \
-  "${SUBNET}"
-
-# 7. VPC Network
-check_resource_exact "VPC Network ${NETWORK}" "gcloud compute networks describe \"${NETWORK}\" --project=\"${PROJECT_ID}\""
-
-# 8. GCS Buckets (Optional without --force)
-check_resource_exact "GCS Staging Bucket gs://${BUCKET}" "gcloud storage ls --buckets 'gs://${BUCKET}'" true
-check_resource_exact "GCS Temp Bucket gs://${TEMP_BUCKET}" "gcloud storage ls --buckets 'gs://${TEMP_BUCKET}'" true
-
-echo -e "\nAudit complete."
-echo -e "[${YELLOW}Pass*${NC}] indicates the resource was not found (which is expected after destroy)."
-echo -e "[${BLUE}Kept${NC}] indicates the resource was found, which is expected as --force was not used."
\ No newline at end of file
diff --git a/gcloud/bin/audit-dpgke b/gcloud/bin/audit-dpgke
new file mode 100755
index 00000000..11bd8c7b
--- /dev/null
+++ b/gcloud/bin/audit-dpgke
@@ -0,0 +1,123 @@
+#!/bin/bash
+#
+# Universal audit script for Dataproc on GKE (DPGKE) environments.
+#
+# This script inspects the live state of GCP resources to generate a report,
+# ignoring potentially stale local sentinel files.
+
+# Exit on failure
+set -e
+
+# --- Get script's real directory ---
+SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+GCLOUD_DIR="$(realpath "${SCRIPT_DIR}/..")"
+source "${GCLOUD_DIR}/lib/env.sh"
+source "${GCLOUD_DIR}/lib/script-utils.sh"
+source "${GCLOUD_DIR}/lib/gke.sh"
+source "${GCLOUD_DIR}/lib/dpgke.sh"
+source "${GCLOUD_DIR}/lib/gcp/misc.sh"
+
+# --- Argument Parsing ---
+AUDIT_DESTROY=false
+OUTPUT_FORMAT="text"
+
+while [[ "$#" -gt 0 ]]; do
+    case $1 in
+        --destroy) AUDIT_DESTROY=true ;;
+        --format=json) OUTPUT_FORMAT="json" ;;
+        *) echo "Unknown parameter passed: $1"; exit 1 ;;
+    esac
+    shift
+done
+
+# --- Main Audit Logic ---
+if [[ "${OUTPUT_FORMAT}" != "json" ]]; then
+  configure_gcloud
+fi
+
+declare -A AUDIT_RESULTS
+
+if [[ "${OUTPUT_FORMAT}" != "json" ]]; then
+  echo "================================================="
+  echo "Starting Comprehensive DPGKE Environment Audit"
+  echo "================================================="
+fi
+
+if [[ "$AUDIT_DESTROY" == "true" ]]; then
+    # --- Verify Destruction ---
+    if [[ "${OUTPUT_FORMAT}" != "json" ]]; then
+      echo
+      echo "Verifying Complete Resource Destruction..."
+    fi
+    check_not_exists "Dataproc on GKE Cluster '${DPGKE_CLUSTER_NAME}'" "exists_dpgke_cluster"
+    check_not_exists "GKE Cluster '${GKE_CLUSTER_NAME}'" "exists_gke_cluster"
+    check_not_exists "Default NodePool" "exists_gke_nodepool '${DP_POOLNAME_DEFAULT}'"
+    check_not_exists "Spark Driver NodePool" "exists_gke_nodepool '${DP_DRIVER_POOLNAME}'"
+    check_not_exists "Spark Executor NodePool" "exists_gke_nodepool '${DP_EXEC_POOLNAME}'"
+
+    # Correct sentinels
+    if [[ "${OUTPUT_FORMAT}" != "json" ]]; then
+      print_status "Updating sentinel files based on audit findings..."
+      report_result "Done"
+    fi
+else
+    # --- Verify Creation ---
+    if [[ "${OUTPUT_FORMAT}" != "json" ]]; then
+      print_status "Auditing environment state..."
+    fi
+    AUDIT_RESULTS["GKE Cluster"]=$(_check_exists "GKE Cluster '${GKE_CLUSTER_NAME}'" "exists_gke_cluster")
+    AUDIT_RESULTS["Dataproc on GKE Cluster"]=$(_check_exists "Dataproc on GKE Cluster '${DPGKE_CLUSTER_NAME}'" "exists_dpgke_cluster")
+    if [[ ${AUDIT_RESULTS["GKE Cluster"]} == "Exists" ]]; then
+        AUDIT_RESULTS["Default NodePool"]=$(_check_exists "Default NodePool" "exists_gke_nodepool '${DP_POOLNAME_DEFAULT}'")
+        AUDIT_RESULTS["Spark Driver NodePool"]=$(_check_exists "Spark Driver NodePool" "exists_gke_nodepool '${DP_DRIVER_POOLNAME}'")
+        AUDIT_RESULTS["Spark Executor NodePool"]=$(_check_exists "Spark Executor NodePool" "exists_gke_nodepool '${DP_EXEC_POOLNAME}'")
+    fi
+    if [[ "${OUTPUT_FORMAT}" != "json" ]]; then
+      report_result "Done"
+    fi
+
+    # Correct sentinels
+    if [[ "${OUTPUT_FORMAT}" != "json" ]]; then
+      print_status "Updating sentinel files based on audit findings..."
+    fi
+    if [[ ${AUDIT_RESULTS["Dataproc on GKE Cluster"]} == "Exists" ]]; then
+    else
+    fi
+    if [[ "${OUTPUT_FORMAT}" != "json" ]]; then
+      report_result "Done"
+    fi
+fi
+
+# --- Output Results ---
+if [[ "${OUTPUT_FORMAT}" == "json" ]]; then
+  # Build a JSON object with jq
+  json_output=$(jq -n \
+    --arg gke_cluster "${AUDIT_RESULTS["GKE Cluster"]:-"Not Checked"}" \
+    --arg dpgke_cluster "${AUDIT_RESULTS["Dataproc on GKE Cluster"]:-"Not Checked"}" \
+    --arg default_nodepool "${AUDIT_RESULTS["Default NodePool"]:-"Not Checked"}" \
+    --arg driver_nodepool "${AUDIT_RESULTS["Spark Driver NodePool"]:-"Not Checked"}" \
+    --arg exec_nodepool "${AUDIT_RESULTS["Spark Executor NodePool"]:-"Not Checked"}" \
+    '{
+      "resources": {
+        "gkeCluster": $gke_cluster,
+        "dataprocOnGkeCluster": $dpgke_cluster,
+        "defaultNodePool": $default_nodepool,
+        "sparkDriverNodePool": $driver_nodepool,
+        "sparkExecutorNodePool": $exec_nodepool
+      }
+    }')
+  echo "${json_output}"
+else
+  # Print the human-readable summary
+  echo
+  echo "================================================="
+  echo "               Audit Summary"
+  echo "================================================="
+  for resource in "GKE Cluster" "Dataproc on GKE Cluster" "Default NodePool" "Spark Driver NodePool" "Spark Executor NodePool"; do
+      if [[ -v AUDIT_RESULTS[${resource}] ]]; then
+          printf "%-25s: %s
+" "${resource}" "${AUDIT_RESULTS[${resource}]}"
+      fi
+  done
+  echo "================================================="
+fi
diff --git a/gcloud/bin/audit-private-create b/gcloud/bin/audit-private-create
deleted file mode 100644
index 7a47de07..00000000
--- a/gcloud/bin/audit-private-create
+++ /dev/null
@@ -1,78 +0,0 @@
-#!/bin/bash
-
-# Script to audit resource creation after create-dpgce-private.sh
-# Expected: All resources should be found.
-
-source lib/env.sh
-source lib/gcp/misc.sh
-
-if (( DEBUG != 0 )); then
-  set -x
-fi
-
-configure_gcloud # Set gcloud context
-
-parse_args "$@" # Not really used here, but good practice
-
-if [[ -z "${CLUSTER_NAME}" || "${CLUSTER_NAME}" == "null" ]]; then
-  echo "ERROR: CLUSTER_NAME is not set. Please source lib/env.sh after env.json is configured."
-  exit 1
-fi
-
-LOG_DIR="tmp/create_audit_${CLUSTER_NAME}_$(date +%Y%m%d-%H%M%S)"
-mkdir -p "${LOG_DIR}"
-echo "Detailed logs will be saved in ${LOG_DIR}"
-
-# --- Start Audit ---
-echo "Starting resource creation audit for cluster: ${CLUSTER_NAME}"
-
-# 1. VPC Network
-check_exists "VPC Network ${NETWORK}" "gcloud compute networks describe '${NETWORK}' --project='${PROJECT_ID}'"
-
-# 2. Subnets
-check_exists "Main Subnet ${SUBNET}" "gcloud compute networks subnets describe '${SUBNET}' --region='${REGION}' --project='${PROJECT_ID}'"
-check_exists "Private Subnet ${PRIVATE_SUBNET}" "gcloud compute networks subnets describe '${PRIVATE_SUBNET}' --region='${REGION}' --project='${PROJECT_ID}'"
-check_exists "SWP Subnet ${SWP_SUBNET}" "gcloud compute networks subnets describe '${SWP_SUBNET}' --region='${REGION}' --project='${PROJECT_ID}'"
-
-# 3. Service Account
-check_exists "Service Account ${GSA}" "gcloud iam service-accounts describe '${GSA}' --project='${PROJECT_ID}'"
-
-# 4. GCS Buckets
-check_exists "GCS Staging Bucket gs://${BUCKET}" "gcloud storage ls --buckets 'gs://${BUCKET}'"
-check_exists "GCS Temp Bucket gs://${TEMP_BUCKET}" "gcloud storage ls --buckets 'gs://${TEMP_BUCKET}'"
-
-# 5. SWP Certificate Components
-SUFFIX=${RESOURCE_SUFFIX}
-CA_POOL_NAME="swp-ca-pool-${CLUSTER_NAME}-${SUFFIX}"
-CIC_NAME="swp-cic-${CLUSTER_NAME}-${SUFFIX}"
-CA_NAME="swp-root-ca-${CLUSTER_NAME}-${SUFFIX}"
-CERT_NAME="swp-cert"
-
-check_exists "CA Pool ${CA_POOL_NAME}" "gcloud privateca pools describe '${CA_POOL_NAME}' --location='${REGION}' --project='${PROJECT_ID}'"
-check_exists "Root CA ${CA_NAME}" "gcloud privateca roots describe '${CA_NAME}' --pool='${CA_POOL_NAME}' --location='${REGION}' --project='${PROJECT_ID}'"
-check_exists "CIC ${CIC_NAME}" "gcloud certificate-manager issuance-configs describe '${CIC_NAME}' --location='${REGION}' --project='${PROJECT_ID}'"
-check_exists "Static Certificate ${CERT_NAME}" "gcloud certificate-manager certificates describe '${CERT_NAME}' --location='${REGION}' --project='${PROJECT_ID}'"
-
-# 6. Gateway Security Policy
-check_exists "Gateway Security Policy ${SWP_POLICY_NAME}" "gcloud network-security gateway-security-policies list --location='${REGION}' --project='${PROJECT_ID}' --filter='name ~ /${SWP_POLICY_NAME}$' --format='value(name)'"
-check_exists "GSP Rule allow-all-rule" "gcloud network-security gateway-security-policies rules list --gateway-security-policy='${SWP_POLICY_NAME}' --location='${REGION}' --project='${PROJECT_ID}' --filter='name ~ /allow-all-rule$' --format='value(name)'"
-
-# 7. SWP Gateway
-check_exists "SWP Gateway ${SWP_INSTANCE_NAME}" "gcloud network-services gateways describe '${SWP_INSTANCE_NAME}' --location='${REGION}' --project='${PROJECT_ID}'"
-
-# 8. Firewall Rules
-check_exists "Firewall Rule allow-swp-ingress-${CLUSTER_NAME}" "gcloud compute firewall-rules describe allow-swp-ingress-${CLUSTER_NAME} --project='${PROJECT_ID}'"
-
-# 9. Autoscaling Policy
-check_exists "Autoscaling Policy ${AUTOSCALING_POLICY_NAME}" "gcloud dataproc autoscaling-policies describe '${AUTOSCALING_POLICY_NAME}' --region='${REGION}' --project='${PROJECT_ID}'"
-
-# 10. Dataproc Cluster (Optional)
-print_status "Checking: Dataproc Cluster ${CLUSTER_NAME}... "
-cluster_log_file="${LOG_DIR}/Dataproc_Cluster_${CLUSTER_NAME}.log"
-if gcloud dataproc clusters describe "${CLUSTER_NAME}" --region="${REGION}" --project="${PROJECT_ID}" > "${cluster_log_file}" 2>&1; then
-  print_result "Exists"
-else
-  print_result "Not Found"
-fi
-
-echo -e "\nAudit complete."
\ No newline at end of file
diff --git a/gcloud/bin/audit-private-destroy b/gcloud/bin/audit-private-destroy
deleted file mode 100644
index 1bd974e9..00000000
--- a/gcloud/bin/audit-private-destroy
+++ /dev/null
@@ -1,139 +0,0 @@
-#!/bin/bash
-
-# Script to audit resource cleanup after destroy-dpgce-private.sh
-# Expected: No resources matching the patterns should be found.
-
-source lib/env.sh
-source lib/gcp/misc.sh # Source the file containing configure_gcloud
-
-if (( DEBUG != 0 )); then
-  set -x
-fi
-
-configure_gcloud # Set gcloud context
-
-parse_args "$@" # Sets FORCE_AUDIT based on --force
-
-if [[ -z "${CLUSTER_NAME}" || "${CLUSTER_NAME}" == "null" ]]; then
-  echo "ERROR: CLUSTER_NAME is not set. Please source lib/env.sh after env.json is configured."
-  exit 1
-fi
-
-LOG_DIR="tmp/destroy_audit_${CLUSTER_NAME}_$(date +%Y%m%d-%H%M%S)"
-mkdir -p "${LOG_DIR}"
-echo "Detailed logs will be saved in ${LOG_DIR}"
-
-NOT_FOUND_LOGS=()
-
-# --- Start Audit ---
-echo "Starting resource cleanup audit for cluster: ${CLUSTER_NAME}"
-if [[ "${FORCE_AUDIT}" == "true" ]]; then
-  echo "--force flag detected, expecting ALL resources to be deleted."
-fi
-
-# 1. Dataproc Clusters
-check_resource_exact "Dataproc Clusters" \
-  "gcloud dataproc clusters list --region=\"${REGION}\" --project=\"${PROJECT_ID}\" --filter=\"clusterName = ${CLUSTER_NAME}\" --format=\"value(clusterName)\""
-
-# 2. Service Accounts
-check_resource_exact "Service Account" \
-  "gcloud iam service-accounts list --project=\"${PROJECT_ID}\" --filter=\"email = ${GSA}\" --format=\"value(email)\""
-
-# 3. Autoscaling Policies
-check_resource "Autoscaling Policies" \
-  "gcloud dataproc autoscaling-policies list --region=\"${REGION}\" --project=\"${PROJECT_ID}\" --format=\"value(id)\"" \
-  "${AUTOSCALING_POLICY_NAME}"
-
-# 4. Cloud Routers
-check_resource "Cloud Routers" \
-  "gcloud compute routers list --regions=\"${REGION}\" --project=\"${PROJECT_ID}\" --filter=\"network ~ ${NETWORK}$\" --format=\"value(name)\"" \
-  "${ROUTER_NAME}"
-check_resource "SWG Autogen Routers" \
-  "gcloud compute routers list --regions=\"${REGION}\" --project=\"${PROJECT_ID}\" --filter=\"network ~ ${NETWORK}$\" --format=\"value(name)\"" \
-  "swg-autogen-router-"
-
-# 5. Firewall Rules
-check_resource "Cluster Firewall Rules" \
-  "gcloud compute firewall-rules list --project=\"${PROJECT_ID}\" --filter=\"network ~ ${NETWORK}$\" --format=\"value(name)\"" \
-  "${CLUSTER_NAME}"
-check_resource "SWP Ingress Firewall Rule" \
-  "gcloud compute firewall-rules list --project=\"${PROJECT_ID}\" --filter=\"network ~ ${NETWORK}$\" --format=\"value(name)\"" \
-  "allow-swp-ingress-${CLUSTER_NAME}"
-check_resource "S8S Internal Firewall Rule" \
-  "gcloud compute firewall-rules list --project=\"${PROJECT_ID}\" --filter=\"network ~ ${NETWORK}$\" --format=\"value(name)\"" \
-  "allow-internal-s8s"
-
-# 6. SWP Gateway
-check_resource "SWP Gateway" \
-  "gcloud network-services gateways list --location=\"${REGION}\" --project=\"${PROJECT_ID}\" --format=\"value(name)\"" \
-  "${SWP_INSTANCE_NAME}"
-
-# 7. Gateway Security Policies
-check_resource "Gateway Security Policies" \
-  "gcloud network-security gateway-security-policies list --location=\"${REGION}\" --project=\"${PROJECT_ID}\" --filter=\"name ~ /${SWP_POLICY_NAME}$ \" --format=\"value(name)\"" \
-  "${SWP_POLICY_NAME}"
-
-# 8. Certificate Manager Certificate (Static)
-check_resource "Static Cert Manager Certificate" \
-  "gcloud certificate-manager certificates list --location=\"${REGION}\" --project=\"${PROJECT_ID}\" --format=\"value(name)\"" \
-  "${SWP_CERT_NAME}" \
-  true # Optional
-
-# 9. Certificate Issuance Configs (Optional without --force)
-check_resource "Cert Issuance Configs" \
-  "gcloud certificate-manager issuance-configs list --location=\"${REGION}\" --project=\"${PROJECT_ID}\" --format=\"value(name)\"" \
-  "swp-cic-${CLUSTER_NAME}-" \
-  true # Optional
-
-# 10. CA Pools (Optional without --force)
-check_resource "CA Pools" \
-  "gcloud privateca pools list --location=\"${REGION}\" --project=\"${PROJECT_ID}\" --format=\"value(name)\"" \
-  "swp-ca-pool-${CLUSTER_NAME}-" \
-  true # Optional
-
-# 11. Root CAs (Optional without --force)
-ca_pool_prefix="swp-ca-pool-${CLUSTER_NAME}-"
-pool_log="${LOG_DIR}/CA_Pools_for_Root_CA_check.log"
-print_status "Checking: Root CAs in any lingering Pools... "
-gcloud privateca pools list --location="${REGION}" --project="${PROJECT_ID}" --format="value(name)" > "${pool_log}" 2>&1
-pool_names=$(grep "${ca_pool_prefix}" "${pool_log}" || true)
-if [[ -n "${pool_names}" ]]; then
-  if [[ "${FORCE_AUDIT}" == "false" ]]; then
-    report_result "Kept"
-    echo "  -> Found lingering CA Pools (expected without --force)."
-  else
-    report_result "Fail"
-    echo "  -> Found lingering CA Pools, CAs might exist. Check logs in ${LOG_DIR}"
-    while read -r pool_full_name; do
-      short_pool_name=$(basename "${pool_full_name}")
-      check_resource "Root CAs in ${short_pool_name}" \
-        "gcloud privateca roots list --pool=\"${short_pool_name}\" --location=\"${REGION}\" --project=\"${PROJECT_ID}\" --format=\"value(name)\"" \
-        "swp-root-ca-${CLUSTER_NAME}-" \
-        true # Optional
-    done <<< "${pool_names}"
-  fi
-else
-  report_result "Not Found"
-fi
-
-# 12. Subnets
-check_resource "SWP Subnet" \
-  "gcloud compute networks subnets list --network=\"${NETWORK}\" --project=\"${PROJECT_ID}\" --format=\"value(name)\"" \
-  "${SWP_SUBNET}"
-check_resource "Private Subnet" \
-  "gcloud compute networks subnets list --network=\"${NETWORK}\" --project=\"${PROJECT_ID}\" --format=\"value(name)\"" \
-  "${PRIVATE_SUBNET}"
-check_resource "Main Subnet" \
-  "gcloud compute networks subnets list --network=\"${NETWORK}\" --project=\"${PROJECT_ID}\" --format=\"value(name)\"" \
-  "${SUBNET}"
-
-# 13. VPC Network
-check_resource_exact "VPC Network ${NETWORK}" "gcloud compute networks describe \"${NETWORK}\" --project=\"${PROJECT_ID}\""
-
-# 14. GCS Buckets (Optional without --force)
-check_resource_exact "GCS Staging Bucket gs://${BUCKET}" "gcloud storage ls --buckets 'gs://${BUCKET}'" true
-check_resource_exact "GCS Temp Bucket gs://${TEMP_BUCKET}" "gcloud storage ls --buckets 'gs://${TEMP_BUCKET}'" true
-
-echo -e "\nAudit complete."
-echo -e "[${YELLOW}Pass*${NC}] indicates the resource was not found (which is expected after destroy)."
-echo -e "[${BLUE}Kept${NC}] indicates the resource was found, which is expected as --force was not used."
\ No newline at end of file
diff --git a/gcloud/bin/connectivity-test b/gcloud/bin/connectivity-test
new file mode 100755
index 00000000..bd623430
--- /dev/null
+++ b/gcloud/bin/connectivity-test
@@ -0,0 +1,35 @@
+#!/bin/bash
+
+source lib/env.sh
+
+gcloud network-services gateways describe ${SWP_INSTANCE_NAME} --location ${REGION}
+
+gcloud network-security gateway-security-policies rules export allow-all-rule \
+  --gateway-security-policy=${SWP_POLICY_NAME} \
+  --location ${REGION}
+
+# Look back over the last day for Gateway logs
+gcloud logging read \
+  "resource.type=\"networkservices.googleapis.com/Gateway\" \
+  resource.labels.location=\"${REGION}\" \
+  resource.labels.gateway_id=\"${SWP_INSTANCE_NAME}\"" \
+  --project="${PROJECT_ID}" \
+  --limit=500 \
+  --freshness=1d
+
+# Delete the old test if it exists
+gcloud network-management connectivity-tests delete swp-test --quiet || true
+
+# Create a new connectivity test from the master node to the SWP gateway
+gcloud network-management connectivity-tests create swp-test \
+  --source-instance=projects/${PROJECT_ID}/zones/${ZONE}/instances/${CLUSTER_NAME}-m \
+  --destination-ip-address=${SWP_IP} \
+  --destination-port=${SWP_PORT} \
+  --protocol=TCP \
+  --source-network=projects/${PROJECT_ID}/global/networks/${NETWORK}
+
+echo "Waiting for connectivity test to complete..."
+sleep 10
+
+# Get the results
+gcloud network-management connectivity-tests describe swp-test
diff --git a/gcloud/bin/create-dpgce b/gcloud/bin/create-dpgce
index e91ad095..7a3ec570 100755
--- a/gcloud/bin/create-dpgce
+++ b/gcloud/bin/create-dpgce
@@ -1,112 +1,366 @@
 #!/bin/bash
-#
-# Copyright 2021 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS-IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
 # Exit on failure
+unset REGION ZONE
+set -e
 set -e
 
-source lib/env.sh
+# --- Get script's real directory ---
+SCRIPT_DIR=$(dirname "${BASH_SOURCE[0]}")
+GCLOUD_DIR="$(realpath "${SCRIPT_DIR}/..")"
+export GCLOUD_DIR
+DP_EVOLUTION_DIR=$(realpath "${GCLOUD_DIR}/../..")
+export DP_EVOLUTION_DIR
+
+# --- Source utility functions ---
+source "${GCLOUD_DIR}/lib/script-utils.sh"
+
+echo "INFO: Using TIMESTAMP ${TIMESTAMP} for this run's logs." >&2
+
+# --- Argument Parsing ---
+IS_CUSTOM=false
+CREATE_CLUSTER=true
+SWP_EGRESS=false
+NAT_EGRESS=true
+ENABLE_GPU=false
+
+while [[ "$#" -gt 0 ]]; do
+    case $1 in
+        --custom) IS_CUSTOM=true; shift ;;
+        --no-custom) IS_CUSTOM=false; shift ;;
+        --swp-egress) SWP_EGRESS=true; shift ;;
+        --no-swp-egress) SWP_EGRESS=false; shift ;;
+        --nat-egress) NAT_EGRESS=true; shift ;;
+        --no-nat-egress) NAT_EGRESS=false; shift ;;
+        --gpu) ENABLE_GPU=true; shift ;;
+        --no-gpu) ENABLE_GPU=false; shift ;;
+        --no-create-cluster) CREATE_CLUSTER=false; shift ;;
+        *) echo "Unknown parameter passed: $1"; exit 1 ;;
+    esac
+done
+
+# Export boolean flags for use in functions
+export IS_CUSTOM
+export SWP_EGRESS
+export NAT_EGRESS
+export ENABLE_GPU
+
+# --- Source environment variables AFTER arguments are parsed ---
+source "${GCLOUD_DIR}/lib/env.sh"
+
+# GPU Configuration - only applied if ENABLE_GPU is true
+export ACCELERATOR_TYPE="$(jq -r .ACCELERATOR_TYPE "${GCLOUD_DIR}/env.json")"
+if [[ "${ENABLE_GPU}" == "true" ]]; then
+  if [[ -z "${ACCELERATOR_TYPE}" || "${ACCELERATOR_TYPE}" == "null" ]]; then
+    export ACCELERATOR_TYPE="nvidia-tesla-t4"
+  fi
+
+  # Read from env.json, but do not export if not set, to allow init action defaults
+  CUDA_VERSION_FROM_ENV=$(jq -r .CUDA_VERSION "${GCLOUD_DIR}/env.json")
+  DRIVER_VERSION_FROM_ENV=$(jq -r .DRIVER_VERSION "${GCLOUD_DIR}/env.json")
+  CUDA_URL_FROM_ENV=$(jq -r .CUDA_URL "${GCLOUD_DIR}/env.json")
+  GPU_DRIVER_URL_FROM_ENV=$(jq -r .GPU_DRIVER_URL "${GCLOUD_DIR}/env.json")
+
+  if [[ -n "${CUDA_VERSION_FROM_ENV}" && "${CUDA_VERSION_FROM_ENV}" != "null" ]]; then
+    export CUDA_VERSION="${CUDA_VERSION_FROM_ENV}"
+  else
+    unset CUDA_VERSION
+  fi
+  if [[ -n "${DRIVER_VERSION_FROM_ENV}" && "${DRIVER_VERSION_FROM_ENV}" != "null" ]]; then
+    export DRIVER_VERSION="${DRIVER_VERSION_FROM_ENV}"
+  else
+    unset DRIVER_VERSION
+  fi
+  if [[ -n "${CUDA_URL_FROM_ENV}" && "${CUDA_URL_FROM_ENV}" != "null" ]]; then
+    export CUDA_URL="${CUDA_URL_FROM_ENV}"
+  else
+    unset CUDA_URL
+  fi
+  if [[ -n "${GPU_DRIVER_URL_FROM_ENV}" && "${GPU_DRIVER_URL_FROM_ENV}" != "null" ]]; then
+    export GPU_DRIVER_URL="${GPU_DRIVER_URL_FROM_ENV}"
+  else
+    unset GPU_DRIVER_URL
+  fi
+  echo "INFO: GPU enabled: Type=${ACCELERATOR_TYPE}, CUDA_VER=${CUDA_VERSION:-Default}, DRIVER_VER=${DRIVER_VERSION:-Default}, CUDA_URL=${CUDA_URL:-None}, DRIVER_URL=${GPU_DRIVER_URL:-None}" >&2
+else
+  export ACCELERATOR_TYPE=""
+  unset CUDA_VERSION
+  unset DRIVER_VERSION
+  unset CUDA_URL
+  unset GPU_DRIVER_URL
+fi
+export M_ACCELERATOR_TYPE="${ACCELERATOR_TYPE}"
+export PRIMARY_ACCELERATOR_TYPE="${ACCELERATOR_TYPE}"
+export SECONDARY_ACCELERATOR_TYPE="${ACCELERATOR_TYPE}"
+
+# Upload local GPU init action if --gpu is enabled
+unset GPU_INIT_ACTION_URI
+if [[ "${ENABLE_GPU}" == "true" ]]; then
+  TIMESTAMP=$(date +%s)
+  export GPU_INIT_ACTION_URI="gs://${BUCKET}/tmp_init_actions/install_gpu_driver_${TIMESTAMP}.sh"
+  echo "INFO: Uploading local GPU init action to ${GPU_INIT_ACTION_URI}" >&2
+  gsutil cp "${DP_EVOLUTION_DIR}/initialization-actions/gpu/install_gpu_driver.sh" "${GPU_INIT_ACTION_URI}"
+fi
 
-parse_args "$@"
+if [[ "${IS_CUSTOM}" == "true" ]]; then
+  if [[ -z "${CUSTOM_IMAGE_URI}" || "${CUSTOM_IMAGE_URI}" == "null" ]]; then
+    echo "ERROR: --custom flag is set but CUSTOM_IMAGE_URI is not defined in env.json" >&2
+    exit 1
+  fi
+fi
 
 if (( DEBUG != 0 )); then
   set -x
 fi
 
-# Source function files
-source lib/gcp/project.sh
-source lib/gcp/misc.sh
-source lib/misc.sh
-source lib/gcp/iam.sh
-source lib/gcp/gcs.sh
-source lib/network/network.sh
-source lib/network/subnet.sh
-source lib/network/router.sh
-source lib/network/routes.sh
-source lib/network/firewall.sh
-source lib/dataproc/cluster.sh
-source lib/dataproc/autoscaling.sh
-# Add others as needed, e.g.:
-# source lib/network/peering.sh
-# source lib/bigtable.sh
-# source lib/database/mysql.sh
-# source lib/database/mssql.sh
-
-create_project
-
+# Source all function files needed for checks and creation
+source "${GCLOUD_DIR}/lib/script-utils.sh"
+source "${GCLOUD_DIR}/lib/gcp/project.sh"
+source "${GCLOUD_DIR}/lib/gcp/gcs.sh"
+source "${GCLOUD_DIR}/lib/network/network.sh"
+source "${GCLOUD_DIR}/lib/network/subnet.sh"
+source "${GCLOUD_DIR}/lib/network/router.sh"
+source "${GCLOUD_DIR}/lib/network/firewall.sh"
+source "${GCLOUD_DIR}/lib/network/routes.sh"
+source "${GCLOUD_DIR}/lib/gcp/iam.sh"
+source "${GCLOUD_DIR}/lib/gcp/misc.sh"
+source "${GCLOUD_DIR}/lib/misc.sh"
+source "${GCLOUD_DIR}/lib/dataproc/autoscaling.sh"
+source "${GCLOUD_DIR}/lib/dataproc/cluster.sh"
+
+# Source SWP functions
+for swp_script in $(ls "${GCLOUD_DIR}/lib/swp/"*.sh); do
+  source "${swp_script}"
+done
+
+# --- Main Logic ---
 configure_gcloud
+init_state_db # Ensures the fixed state.db and table exist
+
+# Store config flags in the database
+update_state "config.isCustom" "${IS_CUSTOM}"
+update_state "config.swpEgress" "${SWP_EGRESS}"
+update_state "config.natEgress" "${NAT_EGRESS}"
+update_state "config.enableGpu" "${ENABLE_GPU}"
+
+print_status "Auditing environment to ensure cache is fresh..."
+"${GCLOUD_DIR}/bin/audit-dpgce" &> /dev/null
+report_result "Done"
+
+# --- Set derived resource names for SWP ---
+# This must happen after audit to reuse existing versioned resources (certs/policies)
+if [[ "${SWP_EGRESS}" == "true" ]]; then
+  get_or_construct_swp_cert_uri
+  get_or_construct_swp_policy_uri
+fi
 
-set_cluster_name
-
-enable_services
-
-create_service_account
-
-create_bucket
-
-create_vpc_network
-
-#create_ip_allocation
-
-#create_vpc_peering
-
-# Create a cloud router
+echo "--- Current Environment State (from Audit) --- "
+# Function to display resource status from DB
+display_status() {
+  local display_name="$1"
+  local key="$2"
+  local value=$(get_state "${key}")
+  echo -n "- ${display_name}: "
+  if [[ "${key}" == *RolesReady ]]; then
+    if [[ "${value}" == "true" ]]; then
+      report_audit_status "True"
+    elif [[ "${value}" == "false" ]]; then
+      report_audit_status "False"
+    else
+      report_audit_status "Unknown"
+    fi
+  elif [[ "${value}" == "" ]]; then
+    report_audit_status "Not Found"
+  else
+    report_audit_status "Exists"
+  fi
+}
 
-create_vpc_network
+# Display status for all potentially relevant resources
+display_status "Service Account" "serviceAccount"
+if [[ $(get_state "serviceAccount") != "" ]]; then
+  display_status "  Roles Ready" "serviceAccountRolesReady"
+fi
+display_status "GCS Bucket (${BUCKET})" "gcsBucket"
+display_status "GCS Temp Bucket (${TEMP_BUCKET})" "gcsTempBucket"
+display_status "VPC Network (${NETWORK})" "vpcNetwork"
+display_status "Standard Subnet (${SUBNET})" "standardSubnet"
+display_status "Cloud Router" "cloudRouter"
+if [[ "${NAT_EGRESS}" == "true" || $(get_state "cloudRouterNAT") != "" ]]; then
+  display_status "Cloud NAT (nat-config)" "cloudRouterNAT"
+fi
+display_status "Firewall Rules" "firewallRule-ssh" # Assuming this key represents all base rules
+display_status "Autoscaling Policy" "autoscalingPolicy"
+if [[ "${CREATE_CLUSTER}" == "true" ]]; then
+  display_status "Dataproc Cluster (${CLUSTER_NAME})" "dataprocCluster"
+fi
 
-create_subnet
+# SWP Resources
+if [[ "${SWP_EGRESS}" == "true" || $(get_state "swpSubnet") != "" ]]; then
+  display_status "SWP Subnet" "swpSubnet"
+fi
+if [[ "${SWP_EGRESS}" == "true" || $(get_state "swpCaPool") != "" ]]; then
+  display_status "SWP CA Pool" "swpCaPool"
+fi
+if [[ "${SWP_EGRESS}" == "true" || $(get_state "swpRootCa") != "" ]]; then
+  display_status "SWP Root CA" "swpRootCa"
+fi
+if [[ "${SWP_EGRESS}" == "true" || $(get_state "swpCic") != "" ]]; then
+  display_status "SWP CIC" "swpCic"
+fi
+if [[ "${SWP_EGRESS}" == "true" || $(get_state "swpManagedCertificate") != "" ]]; then
+  display_status "SWP Managed Certificate" "swpManagedCertificate"
+fi
+if [[ "${SWP_EGRESS}" == "true" || $(get_state "swpPolicy") != "" ]]; then
+  display_status "SWP Policy" "swpPolicy"
+fi
+if [[ "${SWP_EGRESS}" == "true" || $(get_state "swpGateway") != "" ]]; then
+  display_status "SWP Gateway" "swpGateway"
+fi
+if [[ "${SWP_EGRESS}" == "true" || $(get_state "swpFirewallIngress") != "" ]]; then
+  display_status "SWP Firewall Ingress" "swpFirewallIngress"
+fi
+if [[ "${SWP_EGRESS}" == "true" || $(get_state "swpFirewallInternal") != "" ]]; then
+  display_status "SWP Firewall Internal" "swpFirewallInternal"
+fi
+echo "----------------------------------------------"
+# --- Determine Intended State & Generate Plan ---
+PLAN=()
+INTENDED=()
 
-create_router
+# Basic resources for all types
+INTENDED+=("gcsBucket" "gcsTempBucket" "vpcNetwork" "standardSubnet" "firewallRule-ssh" "autoscalingPolicy" "defaultRoute")
 
-create_default_route
+if [[ "${NAT_EGRESS}" == "true" ]]; then
+  INTENDED+=("cloudRouter")
+fi
 
-add_nat_policy
+# Service Account and Roles are handled separately
+if [[ $(get_state "serviceAccount") == "" ]]; then
+  PLAN+=("CREATE_serviceAccount")
+  PLAN+=("ENSURE_serviceAccountRoles") # If we create it, we need to ensure roles
+elif [[ $(get_state "serviceAccountRolesReady") != "true" ]]; then
+  PLAN+=("ENSURE_serviceAccountRoles")
+fi
 
-create_firewall_rules
+if [[ "${SWP_EGRESS}" == "true" ]]; then
+  INTENDED+=("swpSubnet" "swpPolicy" "swpCaPool" "swpManagedCertificate" "swpGateway" "swpFirewallIngress" "swpFirewallInternal")
+fi
+if [[ "${CREATE_CLUSTER}" == "true" ]]; then
+  INTENDED+=("dataprocCluster")
+fi
 
-# Create logging firewall rules
+# Helper to check if a resource needs creation
+needs_create() {
+  local key="$1"
+  [[ $(get_state "${key}") == "" ]]
+}
+
+# Add to plan if resource is missing
+for resource in "${INTENDED[@]}"; do
+  if [[ "${resource}" == "dataprocCluster" ]]; then continue; fi
+  if [[ "${resource}" == "defaultRoute" ]]; then
+    if [[ "${NAT_EGRESS}" == "true" ]] && ! check_default_route; then
+      PLAN+=("ENSURE_DEFAULT_ROUTE")
+    fi
+    continue
+  fi
 
-#create_logging_firewall_rules
+  # For other resources, check if they need creation based on audit
+  if needs_create "${resource}"; then
+    PLAN+=("CREATE_${resource}")
+  fi
+done
 
-#create_bigtable_instance
 
-#create_mysql_instance
-#create_legacy_mssql_instance
+# NAT Egress logic
+if [[ "${NAT_EGRESS}" == "true" ]]; then
+  if [[ $(get_state "cloudRouter") != "" ]]; then
+    NAT_STATE=$(get_state "cloudRouterNAT")
+    if [[ "${NAT_STATE}" == "" ]]; then
+      PLAN+=("ADD_NAT_TO_ROUTER")
+    fi
+  fi
+else
+  # --no-nat-egress passed, remove NAT and Router if they exist
+  if [[ $(get_state "cloudRouterNAT") != "" ]]; then
+    PLAN+=("REMOVE_NAT_FROM_ROUTER")
+  fi
+  if [[ $(get_state "cloudRouter") != "" ]]; then
+    PLAN+=("DELETE_ROUTER")
+  fi
+fi
 
-# Create PHS dataproc cluster
+# Special handling for dataprocCluster
+if [[ " ${INTENDED[*]} " =~ " dataprocCluster " ]]; then
+  if needs_create "dataprocCluster"; then
+    PLAN+=("CREATE_dataprocCluster")
+  else
+    echo "INFO: Dataproc Cluster '${CLUSTER_NAME}' already exists. Skipping creation." >&2
+  fi
+fi
 
-#create_phs_cluster
+if [[ ${#PLAN[@]} -eq 0 ]]; then
+  echo "No actions needed. Environment matches target state."
+  exit 0
+fi
 
-# Create normal dataproc cluster
+echo "--- Execution Plan ---"
+for action in "${PLAN[@]}"; do
+  echo "- ${action}"
+done
+echo "----------------------"
+
+# --- Plan Execution ---
+echo "Executing plan..."
+
+# Order matters here for dependencies
+if [[ " ${PLAN[*]} " =~ " CREATE_serviceAccount " ]]; then create_service_account; fi
+if [[ " ${PLAN[*]} " =~ " ENSURE_serviceAccountRoles " ]]; then ensure_service_account_roles; fi
+
+# GCS Buckets - No strong dependencies between them
+if [[ " ${PLAN[*]} " =~ " CREATE_gcsBucket " ]]; then create_gcs_bucket "${BUCKET}" "Standard"; grant_gcs_bucket_perms "${BUCKET}"; fi
+if [[ " ${PLAN[*]} " =~ " CREATE_gcsTempBucket " ]]; then create_gcs_bucket "${TEMP_BUCKET}" "Standard"; grant_gcs_bucket_perms "${TEMP_BUCKET}"; fi
+upload_init_actions
+
+if [[ " ${PLAN[*]} " =~ " CREATE_vpcNetwork " ]]; then create_vpc_network; fi
+if [[ " ${PLAN[*]} " =~ " ENSURE_DEFAULT_ROUTE " ]]; then ensure_default_internet_route; fi
+if [[ " ${PLAN[*]} " =~ " CREATE_standardSubnet " ]]; then create_subnet "${SUBNET}" "standardSubnet" "${RANGE}"; fi
+
+# SWP Resources
+if [[ " ${PLAN[*]} " =~ " CREATE_swpSubnet " ]]; then create_swp_subnet; fi
+if [[ " ${PLAN[*]} " =~ " CREATE_swpPolicy " ]]; then create_gateway_security_policy; fi
+if [[ " ${PLAN[*]} " =~ " CREATE_swpCaPool " ]]; then create_managed_certificate; fi # This creates CaPool, RootCa, and CIC
+if [[ " ${PLAN[*]} " =~ " CREATE_swpGateway " ]]; then create_swp_gateway; fi
+if [[ " ${PLAN[*]} " =~ " CREATE_swpFirewallIngress " ]]; then create_allow_swp_ingress_rule; fi
+if [[ " ${PLAN[*]} " =~ " CREATE_swpFirewallInternal " ]]; then create_allow_internal_subnets_rule; fi
+
+if [[ " ${PLAN[*]} " =~ " CREATE_cloudRouter " ]]; then
+  create_router
+  if [[ "${NAT_EGRESS}" == "true" ]]; then add_nat_to_router; fi
+elif [[ " ${PLAN[*]} " =~ " ADD_NAT_TO_ROUTER " ]]; then
+  echo "INFO: NAT Egress enabled, ensuring NAT on existing router..."
+  add_nat_to_router
+fi
+if [[ " ${PLAN[*]} " =~ " REMOVE_NAT_FROM_ROUTER " ]]; then
+  remove_nat_from_router
+fi
+if [[ " ${PLAN[*]} " =~ " DELETE_ROUTER " ]]; then
+  delete_router
+fi
 
-create_autoscaling_policy
+if [[ " ${PLAN[*]} " =~ " CREATE_firewallRule-ssh " ]]; then create_firewall_rules; fi
+if [[ " ${PLAN[*]} " =~ " CREATE_autoscalingPolicy " ]]; then create_autoscaling_policy; fi
 
-if [[ "${CREATE_CLUSTER}" = true ]]; then
-  print_status "Creating Dataproc Cluster ${CLUSTER_NAME}..."
-  if create_dpgce_cluster; then
-    print_result "Pass"
-  else
-    print_result "Fail"
-    exit 1
-  fi
+# Cluster Creation if CREATE is in plan
+if [[ " ${PLAN[*]} " =~ " CREATE_dataprocCluster " ]]; then
+    create_dpgce_cluster
 else
-  echo -e "${YELLOW}Skipping Cluster Creation due to --no-create-cluster flag.${NC}"
+  echo "INFO: Skipping cluster creation, already exists or not requested."
 fi
 
-# Perform some connectivity tests
-
-#perform_connectivity_tests
+echo "Plan execution finished."
 
+# --- Final Audit ---
+print_status "Running final audit to update cache..."
+"${GCLOUD_DIR}/bin/audit-dpgce" &> /dev/null
+report_result "Done"
diff --git a/gcloud/bin/create-dpgce-private b/gcloud/bin/create-dpgce-private
deleted file mode 100755
index 373fdf7d..00000000
--- a/gcloud/bin/create-dpgce-private
+++ /dev/null
@@ -1,120 +0,0 @@
-#!/bin/bash
-#
-# Copyright 2021 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS-IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# Exit on failure
-set -e
-
-source lib/env.sh
-
-source lib/gcp/project.sh
-source lib/gcp/iam.sh
-source lib/gcp/gcs.sh
-source lib/gcp/misc.sh
-source lib/misc.sh # Added this line
-source lib/network/network.sh
-source lib/network/subnet.sh
-source lib/network/router.sh
-source lib/network/firewall.sh
-source lib/swp/subnet.sh
-source lib/swp/certs.sh
-source lib/swp/policy.sh
-source lib/swp/gateway.sh
-source lib/swp/firewall.sh # Added this line
-source lib/dataproc/cluster.sh
-source lib/dataproc/autoscaling.sh
-# source lib/database/mysql.sh
-# source lib/database/mssql.sh
-
-parse_args "$@"
-
-if (( DEBUG != 0 )); then
-  set -x
-fi
-
-create_project
-
-configure_gcloud
-
-set_cluster_name
-
-enable_services
-
-create_service_account
-
-create_bucket
-
-create_vpc_network
-
-#create_ip_allocation
-
-#create_vpc_peering
-
-# Create a cloud router
-
-# create subnet
-
-create_subnet
-
-create_private_subnet
-
-create_swp_subnet
-
-#create_certificate
-
-create_managed_certificate
-
-create_gateway_security_policy
-
-create_swp_gateway
-
-create_firewall_rules
-
-create_allow_swp_ingress_rule
-
-# Create logging firewall rules
-
-#create_logging_firewall_rules
-
-#create_bigtable_instance
-
-#create_mysql_instance
-#create_legacy_mssql_instance
-
-# Create PHS dataproc cluster
-
-#create_phs_cluster
-
-# Create normal dataproc cluster
-
-create_autoscaling_policy
-
-if [[ "${CREATE_CLUSTER}" = true ]]; then
-  print_status "Creating Dataproc Cluster ${CLUSTER_NAME}..."
-  if create_dpgce_cluster; then
-    print_result "Pass"
-  else
-    print_result "Fail"
-    exit 1
-  fi
-else
-  echo -e "${YELLOW}Skipping Cluster Creation due to --no-create-cluster flag.${NC}"
-fi
-
-# Perform some connectivity tests
-
-#perform_connectivity_tests
-
diff --git a/gcloud/bin/create-dpgke b/gcloud/bin/create-dpgke
index 9ffe8fa5..01123673 100755
--- a/gcloud/bin/create-dpgke
+++ b/gcloud/bin/create-dpgke
@@ -67,17 +67,17 @@ grant_gke_roles
 if [[ "${CREATE_CLUSTER}" = true ]]; then
   print_status "Creating GKE Cluster ${GKE_CLUSTER_NAME}..."
   if create_gke_cluster; then
-    print_result "Pass"
+    report_result "Pass"
   else
-    print_result "Fail"
+    report_result "Fail"
     exit 1
   fi
 
   print_status "Creating Dataproc on GKE Cluster ${DPGKE_CLUSTER_NAME}..."
   if create_dpgke_cluster; then
-    print_result "Pass"
+    report_result "Pass"
   else
-    print_result "Fail"
+    report_result "Fail"
     exit 1
   fi
 else
diff --git a/gcloud/bin/create-github-trigger.sh b/gcloud/bin/create-github-trigger.sh
new file mode 100755
index 00000000..8252dcf8
--- /dev/null
+++ b/gcloud/bin/create-github-trigger.sh
@@ -0,0 +1,41 @@
+#!/bin/bash
+
+source lib/env.sh
+
+# --- Configuration ---
+PROJECT_ID="${CI_PROJECT_ID}"
+REPO_NAME="${CI_CSR_REPO_NAME}"
+REPO_OWNER="${CI_REPO_OWNER}"
+# BRANCH_NAME is not used for PR triggers with comment control
+REGION="${CI_CSR_REGION}"
+BYOSA_EMAIL="${CI_BYOSA_EMAIL}"
+
+if [[ -z "$PROJECT_ID" || "$PROJECT_ID" == "your-ci-test-project-id" ]]; then
+  echo "ERROR: CI_PROJECT_ID not set or is placeholder in env.json" >&2
+  exit 1
+fi
+if [[ -z "$REPO_NAME" ]]; then
+  echo "ERROR: CI_CSR_REPO_NAME not set in env.json" >&2
+  exit 1
+fi
+if [[ -z "$REPO_OWNER" ]]; then
+  echo "ERROR: CI_REPO_OWNER not set in env.json" >&2
+  exit 1
+fi
+if [[ -z "$BYOSA_EMAIL" ]]; then
+  echo "ERROR: CI_BYOSA_EMAIL not set in env.json" >&2
+  exit 1
+fi
+
+gcloud beta builds triggers create github --project="$PROJECT_ID" \
+  --name="${REPO_NAME}-pr-trigger" \
+  --repo-name="$REPO_NAME" \
+  --repo-owner="$REPO_OWNER" \
+  --pull-request-pattern=".*" \
+  --comment-control=COMMENTS_ENABLED \
+  --build-config="gcloud/cloudbuild.yaml" \
+  --region="$REGION" \
+  --service-account="projects/${PROJECT_ID}/serviceAccounts/${BYOSA_EMAIL}"
+
+echo "Trigger creation command for '/gcbrun' comments on PRs executed."
+echo "Ensure the Google Cloud Build GitHub App has permissions to read PRs and comments."
diff --git a/gcloud/bin/debug-init-action b/gcloud/bin/debug-init-action
new file mode 100755
index 00000000..8442e678
--- /dev/null
+++ b/gcloud/bin/debug-init-action
@@ -0,0 +1,33 @@
+#!/bin/bash
+source lib/env.sh
+
+echo "--- Debug Init Action Workflow ---"
+echo "Step 1: Ensure your local 'init/' directory is populated with the scripts you want to test."
+echo "If you need to sync from the public repository, run something like:"
+echo "  mkdir -p init"
+echo "  cp -r ../../initialization-actions/gpu init/"
+echo "  cp -r ../../initialization-actions/spark-rapids init/"
+echo " "
+
+if [[ ! -d init ]]; then
+  echo "ERROR: 'init/' directory not found. Please create and populate it before running this script."
+  exit 1
+fi
+
+echo "Step 2: Copying local 'init/' directory to the master node's /tmp via IAP..."
+bash bin/scp-m init
+
+echo " "
+echo "Step 3: SSH to the master node using the helper script:"
+echo "  bash bin/ssh-m"
+echo " "
+echo "Step 4: Once SSHed, run the script directly from /tmp/init:"
+echo "----"
+echo "sudo -i"
+echo "# The 'init' directory is available at /tmp/init"
+echo "chmod +x /tmp/init/gpu/install_gpu_driver.sh"
+echo "time bash -x /tmp/init/gpu/install_gpu_driver.sh 2>&1 | tee /tmp/install.log"
+echo "----"
+echo " "
+echo "Step 5: After the script finishes, you can download the log with this command:"
+echo "  gcloud compute scp --zone ${ZONE} ${CLUSTER_NAME}-m:/tmp/install.log ./tmp --tunnel-through-iap --project ${PROJECT_ID}"
diff --git a/gcloud/bin/destroy-dpgce b/gcloud/bin/destroy-dpgce
index 858248c7..936998bc 100755
--- a/gcloud/bin/destroy-dpgce
+++ b/gcloud/bin/destroy-dpgce
@@ -1,77 +1,275 @@
 #!/bin/bash
-#
-# Copyright 2021 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS-IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
 
+# Exit on failure
 set -e
 
-source lib/env.sh
-
-# Source function files
-source lib/dataproc/cluster.sh
-source lib/gcp/iam.sh
-source lib/dataproc/autoscaling.sh
-source lib/network/routes.sh
-source lib/network/router.sh
-source lib/network/firewall.sh
-source lib/network/subnet.sh
-source lib/network/network.sh
-source lib/gcp/gcs.sh
-# source lib/database/mysql.sh
-# source lib/database/mssql.sh
-
-parse_args "$@"
-
-if (( DEBUG != 0 )); then
-  set -x
+# --- Get script's real directory ---
+SCRIPT_DIR=$(dirname "${BASH_SOURCE[0]}")
+GCLOUD_DIR="$(realpath "${SCRIPT_DIR}/..")"
+export GCLOUD_DIR
+
+# --- Source environment variables and utility functions ---
+source "${GCLOUD_DIR}/lib/env.sh"
+source "${GCLOUD_DIR}/lib/script-utils.sh"
+
+echo "INFO: Using TIMESTAMP ${TIMESTAMP} for this run." >&2
+source "${GCLOUD_DIR}/lib/gcp/misc.sh"
+source "${GCLOUD_DIR}/lib/network/network.sh"
+source "${GCLOUD_DIR}/lib/network/subnet.sh"
+source "${GCLOUD_DIR}/lib/network/router.sh"
+source "${GCLOUD_DIR}/lib/network/routes.sh"
+source "${GCLOUD_DIR}/lib/network/firewall.sh"
+source "${GCLOUD_DIR}/lib/gcp/iam.sh"
+source "${GCLOUD_DIR}/lib/gcp/gcs.sh"
+source "${GCLOUD_DIR}/lib/dataproc/cluster.sh"
+source "${GCLOUD_DIR}/lib/dataproc/autoscaling.sh"
+# Add other libs as needed for full cleanup
+source "${GCLOUD_DIR}/lib/phs.sh"
+source "${GCLOUD_DIR}/lib/gke.sh"
+source "${GCLOUD_DIR}/lib/database/mssql.sh"
+source "${GCLOUD_DIR}/lib/database/mysql.sh"
+source "${GCLOUD_DIR}/lib/database/oracle.sh"
+source "${GCLOUD_DIR}/lib/database/pgsql.sh"
+source "${GCLOUD_DIR}/lib/kerberos.sh"
+source "${GCLOUD_DIR}/lib/swp/certs.sh"
+source "${GCLOUD_DIR}/lib/swp/firewall.sh"
+source "${GCLOUD_DIR}/lib/swp/gateway.sh"
+source "${GCLOUD_DIR}/lib/swp/policy.sh"
+source "${GCLOUD_DIR}/lib/swp/subnet.sh"
+source "${GCLOUD_DIR}/lib/network/peering.sh"
+
+# --- Argument Parsing ---
+FORCE_DELETE=false
+IS_CUSTOM=false
+IS_PRIVATE=false
+SWP_EGRESS=false
+NAT_EGRESS=false
+
+while [[ "$#" -gt 0 ]]; do
+    case $1 in
+        --force) FORCE_DELETE=true; shift ;;
+        --custom) IS_CUSTOM=true; shift ;; # Infer intent
+        --nat-egress) NAT_EGRESS=true; shift ;; # Infer intent
+        --swp-egress) SWP_EGRESS=true; IS_PRIVATE=true; shift ;; # Infer intent
+        --private) IS_PRIVATE=true; shift ;; # Infer intent
+        *) echo "Unknown parameter passed: $1"; exit 1 ;;
+    esac
+done
+
+# --- Main Logic ---
+configure_gcloud
+
+if [[ "${FORCE_DELETE}" == "true" ]]; then
+  echo "INFO: --force set. Clearing state database and generating fresh TIMESTAMP." >&2
+  rm -f "${STATE_DB}"
+  unset TIMESTAMP
+  # Re-source env.sh to get a fresh TIMESTAMP and recreate state.db
+  source "${GCLOUD_DIR}/lib/env.sh"
+  init_state_db
+  # Update the persistent timestamp so future runs use this new one
+  update_state "config.timestamp" "${TIMESTAMP}"
+else
+  init_state_db
 fi
 
-delete_dpgce_cluster
+echo "========================================"
+echo "Starting DPGCE Environment Teardown"
+echo "========================================"
 
-delete_service_account
+# Run audit to get the current state
+print_status "Auditing environment..."
+"${GCLOUD_DIR}/bin/audit-dpgce" --timestamp "${TIMESTAMP}" &> /dev/null
+report_result "Done"
 
-delete_autoscaling_policy
+echo "--- Current Environment State (from Audit) --- "
+# Function to display resource status from DB
+display_status() {
+  local display_name="$1"
+  local key="$2"
+  local value=$(get_state "${key}")
+  echo -n "- ${display_name}: "
+  if [[ -z "${value}" || "${value}" == "null" ]]; then
+    report_audit_status "Not Found"
+  else
+    report_audit_status "Exists"
+  fi
+}
 
-#delete_phs_cluster()
+# Display status for contextually relevant resources
+display_status "Service Account" "serviceAccount"
+display_status "GCS Bucket (${BUCKET})" "gcsBucket"
+display_status "GCS Temp Bucket (${TEMP_BUCKET})" "gcsTempBucket"
+display_status "VPC Network (${NETWORK})" "vpcNetwork"
+display_status "Standard Subnet (${SUBNET})" "standardSubnet"
+if [[ "${SWP_EGRESS}" == "true" || $(get_state "swpSubnet") != "null" ]]; then
+  display_status "SWP Subnet (${SWP_SUBNET})" "swpSubnet"
+fi
+display_status "Cloud Router" "cloudRouter"
+if [[ "${NAT_EGRESS}" == "true" || $(get_state "cloudRouterNAT") != "null" ]]; then
+  display_status "Cloud NAT (nat-config)" "cloudRouterNAT"
+fi
+if [[ "${SWP_EGRESS}" == "true" || $(get_state "swpGateway") != "null" || $(get_state "swpCaPool") != "null" ]]; then
+    display_status "SWP CA Pool" "swpCaPool"
+    display_status "SWP Root CA" "swpRootCa"
+    display_status "SWP CIC" "swpCic"
+    display_status "SWP Managed Certificate" "swpManagedCertificate"
+    display_status "SWP Gateway (${SWP_INSTANCE_NAME})" "swpGateway"
+    display_status "SWP Policy (${SWP_POLICY_NAME})" "swpPolicy"
+    display_status "SWP Firewall (Ingress)" "swpFirewallIngress"
+    display_status "SWP Firewall (Internal)" "swpFirewallInternal"
+fi
+display_status "Firewall Rules (SSH)" "firewallRule-ssh"
+display_status "Firewall Rules (Internal)" "firewallRule-internal"
+display_status "Autoscaling Policy" "autoscalingPolicy"
+display_status "Dataproc Cluster (${CLUSTER_NAME})" "dataprocCluster"
+echo "----------------------------------------------"
 
-#delete_mysql_instance
-#delete_legacy_mssql_instance
+# --- Teardown Plan Generation ---
+PLAN=()
 
-delete_default_route
+# Helper to check if a resource exists
+exists() {
+  local key="$1"
+  local val=$(get_state "${key}")
+  [[ -n "${val}" && "${val}" != "null" ]]
+}
 
-delete_nat_configs # Should be called before delete_router
+# Order matters for dependencies
+if exists "dataprocCluster"; then PLAN+=("DELETE_dataprocCluster"); fi
+if exists "phsCluster"; then PLAN+=("DELETE_phsCluster"); fi
+if exists "autoscalingPolicy"; then PLAN+=("DELETE_autoscalingPolicy"); fi
 
-delete_router
+# Warning for GKE resources - DO NOT DELETE IN THIS SCRIPT
+GKE_RESOURCES_FOUND=false
+if exists "dpgkeCluster" || exists "gkeCluster"; then
+  GKE_RESOURCES_FOUND=true
+fi
 
-delete_firewall_rules
 
-#delete_logging_firewall_rules
+if exists "anyNetworkVms"; then
+  PLAN+=("DELETE_all_network_vms")
+fi
 
-#delete_ip_allocation
+ROUTES_RAW=$(get_state "routes")
+if [[ "${ROUTES_RAW}" != "null" && "${ROUTES_RAW}" != "[]" ]]; then
+  if [[ $(echo "${ROUTES_RAW}" | jq 'map(select(.name | test("^default-route-") | not)) | length') -gt 0 ]]; then
+    PLAN+=("DELETE_custom_routes")
+  fi
+fi
+
+if exists "kdcServer"; then PLAN+=("DELETE_kdcServer"); fi
+if exists "oracleVm"; then PLAN+=("DELETE_oracleVm"); fi
+if exists "legacyMssqlInstance"; then PLAN+=("DELETE_legacyMssqlInstance"); fi
+if exists "mssqlInstance"; then PLAN+=("DELETE_mssqlInstance"); fi
+if exists "mysqlInstance"; then PLAN+=("DELETE_mysqlInstance"); fi
+if exists "pgsqlInstance"; then PLAN+=("DELETE_pgsqlInstance"); fi
+if exists "debugVms"; then PLAN+=("DELETE_debugVms"); fi
+
+# SWP Components
+if exists "swpGateway"; then PLAN+=("DELETE_swpGateway"); fi
+if exists "swpFirewallIngress"; then PLAN+=("DELETE_swpFirewallIngress"); fi
+if exists "swpFirewallInternal"; then PLAN+=("DELETE_swpFirewallInternal"); fi
 
-delete_subnet
+# Networking
+if exists "firewallRule-ssh" || exists "firewallRule-internal"; then PLAN+=("DELETE_firewallRules"); fi
+if exists "cloudRouter"; then PLAN+=("DELETE_cloudRouter"); fi
+if exists "vpcPeering"; then PLAN+=("DELETE_vpcPeering"); fi
+if exists "ipAllocation"; then PLAN+=("DELETE_ipAllocation"); fi
+if exists "swpSubnet"; then PLAN+=("DELETE_swpSubnet"); fi
+if exists "privateSubnet"; then PLAN+=("DELETE_privateSubnet"); fi
+if exists "standardSubnet"; then PLAN+=("DELETE_standardSubnet"); fi
+if exists "vpcNetwork"; then PLAN+=("DELETE_vpcNetwork"); fi
 
-delete_vpc_network
+# Other Resources
+if exists "serviceAccount"; then PLAN+=("DELETE_serviceAccount"); fi
 
-#delete_vpc_peering
+# Persistent resources requiring --force to delete
+PROTECTED_RESOURCES_EXIST=false
+if exists "gcsBucket" || exists "gcsTempBucket" || exists "swpCaPool" || exists "swpRootCa" || exists "swpCic" || exists "swpManagedCertificate" || exists "swpPolicy"; then
+  PROTECTED_RESOURCES_EXIST=true
+fi
 
 if [[ "${FORCE_DELETE}" = true ]]; then
-  delete_bucket
+  if exists "gcsBucket"; then PLAN+=("DELETE_gcsBucket"); fi
+  if exists "gcsTempBucket"; then PLAN+=("DELETE_gcsTempBucket"); fi
+  if exists "swpManagedCertificate"; then PLAN+=("DELETE_swpManagedCertificate"); fi
+  if exists "swpPolicy"; then PLAN+=("DELETE_swpPolicy"); fi
 else
-  print_status "Skipping Bucket Deletion. Use --force to delete buckets."
-  report_result "Skipped"
+  if [[ "${PROTECTED_RESOURCES_EXIST}" = true ]]; then
+    echo "INFO: Persistent resources (Buckets, SWP Certs/Policy) exist but --force not specified, skipping their deletion."
+  fi
+fi
+
+if [[ ${#PLAN[@]} -eq 0 ]]; then
+  echo "No resources found to delete."
+  exit 0
 fi
 
-set +x
\ No newline at end of file
+echo "--- Teardown Plan ---"
+for action in "${PLAN[@]}"; do
+  echo "- ${action}"
+done
+echo "----------------------"
+
+# --- Plan Execution ---
+echo "Executing teardown plan..."
+
+for action in "${PLAN[@]}"; do
+  case ${action} in
+    DELETE_dataprocCluster) delete_dpgce_cluster ;;
+    DELETE_phsCluster) delete_phs_cluster ;;
+    DELETE_autoscalingPolicy) delete_autoscaling_policy ;;
+    DELETE_all_network_vms) delete_all_network_vms ;;
+    DELETE_kdcServer) delete_kdc_server ;;
+    DELETE_oracleVm) delete_oracle_vm ;;
+    DELETE_legacyMssqlInstance) delete_legacy_mssql_instance ;;
+    DELETE_mssqlInstance) delete_mssql_instance ;;
+    DELETE_mysqlInstance) delete_mysql_instance ;;
+    DELETE_pgsqlInstance) delete_pgsql_instance ;;
+    DELETE_debugVms) 
+      print_status "Deleting Debug VMs..."
+      if exists "debugVms"; then
+        gcloud compute instances delete $(get_state "debugVms" | jq -r '.[].name') --zone "${ZONE}" --quiet || true
+        update_state "debugVms" "null"
+      fi
+      report_result "Done"
+      ;;
+    DELETE_swpGateway) delete_swp_gateway ;;
+    DELETE_swpPolicy) delete_gateway_security_policy ;;
+    DELETE_swpManagedCertificate) delete_managed_certificate ;;
+    DELETE_swpFirewallIngress) delete_allow_swp_ingress_rule ;;
+    DELETE_swpFirewallInternal) delete_allow_internal_subnets_rule ;;
+    DELETE_firewallRules) delete_firewall_rules ;;
+    DELETE_custom_routes)
+      ROUTES_RAW=$(get_state "routes")
+      if [[ "${ROUTES_RAW}" != "null" && "${ROUTES_RAW}" != "[]" ]]; then
+          mapfile -t route_names < <(echo "${ROUTES_RAW}" | jq -r '.[].name')
+          for route_name in "${route_names[@]}"; do
+              # Ignore all auto-generated default peering/network routes
+              if [[ ! "${route_name}" =~ ^default-route- ]]; then
+                  delete_route "${route_name}"
+              fi
+          done
+      fi
+      ;;
+    DELETE_cloudRouter) delete_router ;;
+    DELETE_vpcPeering) delete_vpc_peering ;;
+    DELETE_ipAllocation) delete_ip_allocation ;;
+    DELETE_swpSubnet) delete_subnet "${SWP_SUBNET}" "swpSubnet" ;;
+    DELETE_privateSubnet) delete_subnet "${PRIVATE_SUBNET}" "privateSubnet" ;;
+    DELETE_standardSubnet) delete_subnet "${SUBNET}" "standardSubnet" ;;
+    DELETE_vpcNetwork) delete_vpc_network ;;
+    DELETE_serviceAccount) delete_service_account ;;
+    DELETE_gcsBucket) delete_gcs_bucket "${BUCKET}" ;;
+    DELETE_gcsTempBucket) delete_gcs_bucket "${TEMP_BUCKET}" ;;
+  esac
+done
+
+echo "========================================"
+echo "DPGCE Environment teardown complete"
+echo "========================================"
+
+if [[ "${GKE_RESOURCES_FOUND}" == "true" ]]; then
+  echo -e "\n${YELLOW}WARNING:${NC} GKE-specific resources were detected but NOT deleted."
+  echo -e "Please run ${BLUE}bash bin/destroy-dpgke${NC} to clean up GKE components."
+fi
diff --git a/gcloud/bin/destroy-dpgce-private b/gcloud/bin/destroy-dpgce-private
deleted file mode 100755
index 00073e57..00000000
--- a/gcloud/bin/destroy-dpgce-private
+++ /dev/null
@@ -1,90 +0,0 @@
-#!/bin/bash
-#
-# Copyright 2021 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS-IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-set -e
-
-source lib/env.sh
-
-source lib/dataproc/cluster.sh
-source lib/dataproc/autoscaling.sh
-source lib/gcp/iam.sh
-source lib/gcp/gcs.sh
-source lib/network/network.sh
-source lib/network/subnet.sh
-source lib/network/router.sh
-source lib/network/firewall.sh
-source lib/swp/subnet.sh
-source lib/swp/certs.sh
-source lib/swp/policy.sh
-source lib/swp/gateway.sh
-source lib/swp/firewall.sh
-
-parse_args "$@"
-
-if (( DEBUG != 0 )); then
-  set -x
-fi
-
-delete_dpgce_cluster
-
-delete_service_account
-
-delete_autoscaling_policy
-
-#delete_phs_cluster()
-
-#delete_mysql_instance
-#delete_legacy_mssql_instance
-
-delete_nat_configs
-
-delete_router
-
-delete_firewall_rules
-
-delete_allow_swp_ingress_rule
-
-#delete_logging_firewall_rules
-
-#delete_ip_allocation
-
-delete_swp_gateway
-
-delete_gateway_security_policy
-
-#delete_certificate
-
-delete_managed_certificate
-
-delete_swp_subnet
-
-delete_private_subnet
-
-delete_subnet
-
-delete_vpc_network
-
-#delete_vpc_peering
-
-if [[ "${FORCE_DELETE}" = true ]]; then
-  delete_bucket
-else
-  print_status "Skipping Bucket Deletion. Use --force to delete buckets."
-  report_result "Skipped"
-fi
-
-set +x
diff --git a/gcloud/bin/destroy-dpgke b/gcloud/bin/destroy-dpgke
index 98758760..b4309554 100755
--- a/gcloud/bin/destroy-dpgke
+++ b/gcloud/bin/destroy-dpgke
@@ -15,29 +15,23 @@
 # limitations under the License.
 #
 
-source lib/env.sh
-
-# perform_connectivity_tests
-
-# Perform some connectivity tests
-
+# --- Get script's real directory ---
+SCRIPT_DIR=$(dirname "${BASH_SOURCE[0]}")
+GCLOUD_DIR="$(realpath "${SCRIPT_DIR}/..")"
+export GCLOUD_DIR
+
+source "${GCLOUD_DIR}/lib/env.sh"
+source "${GCLOUD_DIR}/lib/script-utils.sh"
+source "${GCLOUD_DIR}/lib/gke.sh"
+source "${GCLOUD_DIR}/lib/network/network.sh"
+source "${GCLOUD_DIR}/lib/network/subnet.sh"
+source "${GCLOUD_DIR}/lib/network/router.sh"
+source "${GCLOUD_DIR}/lib/network/firewall.sh"
+source "${GCLOUD_DIR}/lib/gcp/iam.sh"
+source "${GCLOUD_DIR}/lib/gcp/gcs.sh"
+
+init_state_db
+
+# Only destroy GKE-specific components
+delete_dpgke_cluster
 delete_gke_cluster
-
-# Delete gke dataproc cluster
-
-delete_service_account
-
-delete_firewall_rules
-
-delete_subnet
-
-# Delete a cloud router
-delete_router
-
-# create_vpc_peering
-# create_ip_allocation
-delete_vpc_network
-
-#enable_services
-delete_bucket
-
diff --git a/gcloud/bin/recreate-dpgce b/gcloud/bin/recreate-dpgce
index d32818cf..e3472c2d 100755
--- a/gcloud/bin/recreate-dpgce
+++ b/gcloud/bin/recreate-dpgce
@@ -1,41 +1,209 @@
 #!/bin/bash
-#
-# Copyright 2021 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS-IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-source lib/env.sh
-source lib/dataproc/cluster.sh
-source lib/gcp/iam.sh
-
-# Copy latest initialization action scripts
-#echo -n "copying actions to gcs bucket..."
-#gcloud storage cp \
-#       --manifest-path action-update.log \
-#       --recursive init/* gs://${BUCKET}/dataproc-initialization-actions
-#if [[ $? == 0 ]]; then
-#  echo "done"
-#else
-#  echo "fail"
-#  exit 1
-#fi
-
-# re-create dpgce dataproc cluster
+
+# Exit on failure
+unset REGION ZONE
+set -e
+set -e
+
+# --- Get script's real directory ---
+SCRIPT_DIR=$(dirname "${BASH_SOURCE[0]}")
+GCLOUD_DIR="$(realpath "${SCRIPT_DIR}/..")"
+DP_EVOLUTION_DIR=$(realpath "${GCLOUD_DIR}/../..")
+export DP_EVOLUTION_DIR
+
+# --- Source utility functions first ---
+source "${GCLOUD_DIR}/lib/script-utils.sh"
+
+# --- Argument Parsing ---
+# Load current state to use as defaults
+# Note: get_state is from script-utils.sh, so must be sourced first.
+IS_CUSTOM=$(get_state "config.isCustom")
+NAT_EGRESS=$(get_state "config.natEgress")
+SWP_EGRESS=$(get_state "config.swpEgress")
+ENABLE_GPU=$(get_state "config.enableGpu")
+
+# Apply defaults if not set in state.db
+IS_CUSTOM=${IS_CUSTOM:-false}
+NAT_EGRESS=${NAT_EGRESS:-true}
+SWP_EGRESS=${SWP_EGRESS:-false}
+ENABLE_GPU=${ENABLE_GPU:-false}
+
+while [[ "$#" -gt 0 ]]; do
+    case $1 in
+        --custom) IS_CUSTOM=true; shift ;;
+        --no-custom) IS_CUSTOM=false; shift ;;
+        --swp-egress) SWP_EGRESS=true; shift ;;
+        --no-swp-egress) SWP_EGRESS=false; shift ;;
+        --nat-egress) NAT_EGRESS=true; shift ;;
+        --no-nat-egress) NAT_EGRESS=false; shift ;;
+        --gpu) ENABLE_GPU=true; shift ;;
+        --no-gpu) ENABLE_GPU=false; shift ;;
+        --no-init-action) export SKIP_INIT_ACTIONS=true; shift ;;
+        *) echo "Unknown parameter passed: $1"; exit 1 ;;
+    esac
+done
+
+# --- Source environment variables AFTER arguments are parsed ---
+source "${GCLOUD_DIR}/lib/env.sh"
+
+# GPU Configuration - only applied if ENABLE_GPU is true
+export ACCELERATOR_TYPE="$(jq -r .ACCELERATOR_TYPE "${GCLOUD_DIR}/env.json")"
+if [[ "${ENABLE_GPU}" == "true" ]]; then
+    if [[ -z "${ACCELERATOR_TYPE}" || "${ACCELERATOR_TYPE}" == "null" ]]; then
+      export ACCELERATOR_TYPE="nvidia-tesla-t4"
+    fi
+
+    # --- Determine CUDA and Driver Versions ---
+    CUDA_VERSION=""
+    DRIVER_VERSION=""
+
+    # 1. Try to parse from env.json URLs
+    CUDA_URL=$(jq -r .CUDA_URL "${GCLOUD_DIR}/env.json" 2>/dev/null)
+    GPU_DRIVER_URL=$(jq -r .GPU_DRIVER_URL "${GCLOUD_DIR}/env.json" 2>/dev/null)
+    [[ "${GPU_DRIVER_URL}" == "null" ]] && GPU_DRIVER_URL=""
+
+    if [[ -n "${CUDA_URL}" && "${CUDA_URL}" != "null" ]]; then
+      CUDA_VERSION=$(echo "${CUDA_URL}" | sed -n 's/.*cuda_\([0-9.]*\)_.*_linux\.run/\1/p')
+    fi
+    if [[ -n "${GPU_DRIVER_URL}" && "${GPU_DRIVER_URL}" != "null" ]]; then
+      DRIVER_VERSION=$(echo "${GPU_DRIVER_URL}" | sed -n 's/.*NVIDIA-Linux-x86_64-\([0-9.]*\)\.run/\1/p')
+    fi
+
+    # 2. Fallback to version fields in env.json
+    if [[ -z "${CUDA_VERSION}" ]]; then
+      CUDA_VERSION=$(jq -r .CUDA_VERSION "${GCLOUD_DIR}/env.json" 2>/dev/null)
+      [[ "${CUDA_VERSION}" == "null" ]] && CUDA_VERSION=""
+    fi
+    if [[ -z "${DRIVER_VERSION}" ]]; then
+      DRIVER_VERSION=$(jq -r .DRIVER_VERSION "${GCLOUD_DIR}/env.json" 2>/dev/null)
+      [[ "${DRIVER_VERSION}" == "null" ]] && DRIVER_VERSION=""
+    fi
+
+    # 3. Fallback to defaults based on DATAPROC_IMAGE_VERSION
+    DATAPROC_IMAGE_VERSION=$(jq -r .IMAGE_VERSION "${GCLOUD_DIR}/env.json" | cut -d- -f1)
+    if [[ -z "${CUDA_VERSION}" || -z "${DRIVER_VERSION}" ]]; then
+      DEFAULT_CUDA_VERSION=""
+      DEFAULT_DRIVER_VERSION=""
+
+      case "${DATAPROC_IMAGE_VERSION}" in
+        "1.5" ) DEFAULT_CUDA_VERSION="11.6.2" ; DEFAULT_DRIVER_VERSION="510.47.03" ;;
+        "2.0" ) DEFAULT_CUDA_VERSION="12.1.1" ; DEFAULT_DRIVER_VERSION="530.30.02" ;; # Cuda 12.1.1 - Driver v530.30.02 is the latest version supported by Ubuntu 18)
+        "2.1" ) DEFAULT_CUDA_VERSION="12.4.1" ; DEFAULT_DRIVER_VERSION="550.135" ;;
+        "2.2" ) DEFAULT_CUDA_VERSION="13.1.1" ; DEFAULT_DRIVER_VERSION="550.142" ;;
+        "2.3" ) DEFAULT_CUDA_VERSION="13.1.1" ; DEFAULT_DRIVER_VERSION="550.142" ;;
+        *   )
+          echo "ERROR: Unrecognized Dataproc image version: ${DATAPROC_IMAGE_VERSION}" >&2
+          exit 1
+          ;;
+      esac
+
+      if [[ -z "${CUDA_VERSION}" ]]; then
+        CUDA_VERSION="${DEFAULT_CUDA_VERSION}"
+        echo "INFO: Using default CUDA_VERSION=${CUDA_VERSION} for image ${DATAPROC_IMAGE_VERSION}" >&2
+      fi
+      if [[ -z "${DRIVER_VERSION}" ]]; then
+        DRIVER_VERSION="${DEFAULT_DRIVER_VERSION}"
+        echo "INFO: Using default DRIVER_VERSION=${DRIVER_VERSION} for image ${DATAPROC_IMAGE_VERSION}" >&2
+      fi
+    fi
+
+    # 4. Error if versions are still not set
+    if [[ -z "${CUDA_VERSION}" || -z "${DRIVER_VERSION}" || "${DRIVER_VERSION}" == "<"* ]]; then
+      echo "ERROR: Could not determine CUDA or DRIVER version." >&2
+      echo "  Checked env.json: CUDA_URL, GPU_DRIVER_URL, CUDA_VERSION, DRIVER_VERSION" >&2
+      echo "  Checked defaults for DATAPROC_IMAGE_VERSION: ${DATAPROC_IMAGE_VERSION}" >&2
+      exit 1
+    fi
+
+    export CUDA_VERSION
+    export DRIVER_VERSION
+    echo "INFO: GPU enabled: Type=${ACCELERATOR_TYPE}, CUDA=${CUDA_VERSION}, Driver=${DRIVER_VERSION}" >&2
+else
+  export CUDA_VERSION=""
+  export DRIVER_VERSION=""
+fi
+export M_ACCELERATOR_TYPE="${ACCELERATOR_TYPE}"
+export PRIMARY_ACCELERATOR_TYPE="${ACCELERATOR_TYPE}"
+export SECONDARY_ACCELERATOR_TYPE="${ACCELERATOR_TYPE}"
+
+# Upload local GPU init action if --gpu is enabled
+unset GPU_INIT_ACTION_URI
+if [[ "${ENABLE_GPU}" == "true" ]]; then
+  TIMESTAMP=$(date +%s)
+  export GPU_INIT_ACTION_URI="gs://${BUCKET}/tmp_init_actions/install_gpu_driver_${TIMESTAMP}.sh"
+  echo "INFO: Uploading local GPU init action to ${GPU_INIT_ACTION_URI}" >&2
+  gsutil cp "${DP_EVOLUTION_DIR}/initialization-actions/gpu/install_gpu_driver.sh" "${GPU_INIT_ACTION_URI}"
+fi
+
+# --- Source other necessary function files ---
+source "${GCLOUD_DIR}/lib/gcp/gcs.sh"
+source "${GCLOUD_DIR}/lib/dataproc/cluster.sh"
+source "${GCLOUD_DIR}/lib/gcp/misc.sh"
+
+# Export boolean flags for use in functions
+export IS_CUSTOM
+export SWP_EGRESS
+export NAT_EGRESS
+export ENABLE_GPU
+
+# --- Configuration Validation ---
+if [[ "${IS_CUSTOM}" == "true" ]]; then
+  if [[ -z "${CUSTOM_IMAGE_URI}" || "${CUSTOM_IMAGE_URI}" == "null" ]]; then
+    echo "ERROR: --custom flag is set but CUSTOM_IMAGE_URI is not defined in env.json" >&2
+    exit 1
+  fi
+fi
+
+if [[ "${ENABLE_GPU}" == "true" ]]; then
+  if [[ -z "${ACCELERATOR_TYPE}" || "${ACCELERATOR_TYPE}" == "null" ]]; then
+    echo "ERROR: --gpu flag is set but ACCELERATOR_TYPE is not defined in env.json" >&2
+    exit 1
+  fi
+  echo "INFO: GPU enabled with ACCELERATOR_TYPE=${ACCELERATOR_TYPE}" >&2
+fi
+
+# --- Main Logic ---
+configure_gcloud
+init_state_db # Initialize the database if it doesn't exist
+
+# Store NEW config flags in the database
+update_state "config.isCustom" "${IS_CUSTOM}"
+update_state "config.swpEgress" "${SWP_EGRESS}"
+update_state "config.natEgress" "${NAT_EGRESS}"
+update_state "config.enableGpu" "${ENABLE_GPU}"
+
+echo "INFO: Configuration to be applied - IS_CUSTOM=${IS_CUSTOM}, NAT_EGRESS=${NAT_EGRESS}, SWP_EGRESS=${SWP_EGRESS}, ENABLE_GPU=${ENABLE_GPU}" >&2
+
+# --- Run an audit to refresh the current state of resources ---
+# The audit will use the same TIMESTAMP as defined in env.sh for its own logs
+print_status "Auditing environment to refresh state..."
+"${GCLOUD_DIR}/bin/audit-dpgce" > /dev/null
+report_result "Done"
+
+if (( DEBUG != 0 )); then
+  set -x
+fi
+
+echo "========================================"
+echo "Starting DPGCE Cluster Recreation"
+echo "========================================"
+
+print_status "Attempting to ensure any pre-existing cluster named '${CLUSTER_NAME}' is deleted..."
 delete_dpgce_cluster
-create_service_account # Ensure SA and roles exist
+report_result "Done"
+
+# Upload init actions before recreation to ensure latest scripts are used
+upload_init_actions
+
+# Re-create the cluster based on the loaded config
+print_status "Creating cluster ${CLUSTER_NAME}..."
 create_dpgce_cluster
 
+# After creation, run audit again to update state file with new resource details
+print_status "Running final audit to update cache..."
+"${GCLOUD_DIR}/bin/audit-dpgce" > /dev/null
+report_result "Done"
+
 echo "========================================"
-echo "General Purpose DPGCE Cluster re-created"
+echo "DPGCE Cluster re-created"
 echo "========================================"
diff --git a/gcloud/bin/scp-m b/gcloud/bin/scp-m
index 09c78937..930cb351 100755
--- a/gcloud/bin/scp-m
+++ b/gcloud/bin/scp-m
@@ -17,25 +17,26 @@
 source lib/env.sh
 
 if [[ -z "$1" ]]; then
-  echo "$0 [master number] <files to copy to worker:/tmp>"
+  echo "$0 [-m node index] <local_path>"
   exit 1
 fi
 
-# If the first argument is a number, assume it indicates which master to send file to
-MASTER_HOSTNAME="${CLUSTER_NAME}-m"
+# If the first argument is a number, assume it indicates which m node to send file to
+M_HOSTNAME="${CLUSTER_NAME}-m"
 if [[ $(perl -e "print 1 if q{$1} =~ /^\d+$/") == "1" ]]; then
-  master_num="$1"
-  echo "master_num: $master_num"
-  MASTER_HOSTNAME="${MASTER_HOSTNAME}-${master_num}"
+  m_num="$1"
+  echo "m_num: $m_num"
+  M_HOSTNAME="${M_HOSTNAME}-${m_num}"
   shift 1
 else
-  MASTER_HOSTNAME="${MASTER_HOSTNAME}"
+  M_HOSTNAME="${M_HOSTNAME}"
 fi
 
 date
 
-gcloud compute scp --recurse "$*" \
-  --zone ${ZONE} \
-  ${MASTER_HOSTNAME}:/tmp  \
-  --tunnel-through-iap \
-  --project ${PROJECT_ID}
+STAGING_URI="gs://${TEMP_BUCKET}/tmp_scp_m/"
+echo "Staging files to ${STAGING_URI}..."
+gcloud storage cp "$@" "${STAGING_URI}"
+
+echo "Fetching files to ${M_HOSTNAME}:/tmp..."
+gcloud compute ssh --zone "${ZONE}" "${M_HOSTNAME}" --tunnel-through-iap --project "${PROJECT_ID}" --command "gcloud storage cp ${STAGING_URI}* /tmp/"
diff --git a/gcloud/bin/setup-cicd.sh b/gcloud/bin/setup-cicd.sh
new file mode 100755
index 00000000..b1f5b290
--- /dev/null
+++ b/gcloud/bin/setup-cicd.sh
@@ -0,0 +1,222 @@
+#!/bin/bash
+
+# One-time setup script for CI/CD environment using Cloud Build
+
+set -e
+set -u
+
+export TIMESTAMP="${TIMESTAMP:-$(date +%s)}"
+source lib/env.sh # Sources variables from env.json
+
+# --- Configuration ---
+CI_PROJECT_ID="${CI_PROJECT_ID}"
+CI_GCP_CREDENTIALS_PATH="${CI_GCP_CREDENTIALS_PATH}"
+CI_CSR_REPO_NAME="${CI_CSR_REPO_NAME}"
+CI_CSR_REGION="${CI_CSR_REGION}"
+CI_GITHUB_CONNECTION_NAME="${CI_GITHUB_CONNECTION_NAME}"
+CI_TRIGGER_BRANCH="${CI_TRIGGER_BRANCH}"
+
+if [[ "$CI_PROJECT_ID" == "your-ci-test-project-id" || -z "$CI_PROJECT_ID" ]]; then
+  echo "ERROR: Please update CI_PROJECT_ID in env.json"
+  exit 1
+fi
+if [[ "$CI_GCP_CREDENTIALS_PATH" == "/path/to/your/ci-service-account.json" || ! -f "$CI_GCP_CREDENTIALS_PATH" ]]; then
+  echo "ERROR: Please update CI_GCP_CREDENTIALS_PATH in env.json and ensure the file exists."
+  exit 1
+fi
+if [[ -z "$CI_CSR_REPO_NAME" ]]; then
+  echo "ERROR: Please update CI_CSR_REPO_NAME in env.json."
+  exit 1
+fi
+
+echo "Setting up CI/CD for project: $CI_PROJECT_ID"
+gcloud config set project "$CI_PROJECT_ID"
+
+# --- Enable Services ---
+echo "Enabling necessary services in $CI_PROJECT_ID..."
+gcloud services enable \
+  secretmanager.googleapis.com \
+  cloudbuild.googleapis.com \
+  containerregistry.googleapis.com \
+  compute.googleapis.com \
+  dataproc.googleapis.com \
+  container.googleapis.com \
+  networksecurity.googleapis.com \
+  networkservices.googleapis.com \
+  privateca.googleapis.com \
+  certificatemanager.googleapis.com \
+  sourcerepo.googleapis.com \
+  --project="$CI_PROJECT_ID"
+
+# --- Create Secrets ---
+echo "Creating secrets in $CI_PROJECT_ID..."
+
+# Use the main env.json for the test environment secret
+gcloud secrets create test-env-json --replication-policy=automatic --project="$CI_PROJECT_ID" --quiet || echo "Secret test-env-json already exists."
+gcloud secrets versions add test-env-json --data-file="env.json" --project="$CI_PROJECT_ID"
+
+gcloud secrets create gcp-credentials --replication-policy=automatic --project="$CI_PROJECT_ID" --quiet || echo "Secret gcp-credentials already exists."
+gcloud secrets versions add gcp-credentials --data-file="$CI_GCP_CREDENTIALS_PATH" --project="$CI_PROJECT_ID"
+
+echo "Secrets created."
+
+# --- Create BYOSA Service Account ---
+BYOSA_EMAIL="${CI_BYOSA_EMAIL}"
+BYOSA_NAME=$(echo "$BYOSA_EMAIL" | cut -d @ -f 1)
+echo "Checking for BYOSA: $BYOSA_EMAIL"
+if ! gcloud iam service-accounts describe "$BYOSA_EMAIL" --project="$CI_PROJECT_ID" > /dev/null 2>&1; then
+  echo "Creating BYOSA: $BYOSA_EMAIL..."
+  gcloud iam service-accounts create "$BYOSA_NAME" \
+    --description="Service Account for Cloud Build CI/CD" \
+    --display-name="Dataproc Repro CICD" \
+    --project="$CI_PROJECT_ID"
+else
+  echo "BYOSA $BYOSA_EMAIL already exists."
+fi
+
+# Grant Owner role to BYOSA (SCOPE DOWN FOR PRODUCTION)
+echo "WARNING: Granting roles/owner to the BYOSA $BYOSA_EMAIL in $CI_PROJECT_ID."
+gcloud projects add-iam-policy-binding "$CI_PROJECT_ID" \
+  --member="serviceAccount:$BYOSA_EMAIL" \
+  --role='roles/owner'
+
+# --- Grant Permissions to Cloud Build SA ---
+PROJECT_NUMBER=$(gcloud projects describe "$CI_PROJECT_ID" --format="value(projectNumber)")
+CLOUD_BUILD_SA="${PROJECT_NUMBER}@cloudbuild.gserviceaccount.com"
+
+echo "Granting permissions to Cloud Build service account: $CLOUD_BUILD_SA"
+
+# Grant access to secrets
+gcloud secrets add-iam-policy-binding test-env-json \
+  --member="serviceAccount:${CLOUD_BUILD_SA}" \
+  --role='roles/secretmanager.secretAccessor' \
+  --project="$CI_PROJECT_ID"
+gcloud secrets add-iam-policy-binding gcp-credentials \
+  --member="serviceAccount:${CLOUD_BUILD_SA}" \
+  --role='roles/secretmanager.secretAccessor' \
+  --project="$CI_PROJECT_ID"
+
+echo "WARNING: Granting roles/owner to the Cloud Build SA in $CI_PROJECT_ID. Scope down for production."
+gcloud projects add-iam-policy-binding "$CI_PROJECT_ID" \
+  --member="serviceAccount:${CLOUD_BUILD_SA}" \
+  --role='roles/owner'
+
+# Grant Cloud Build SA permission to ACT AS the BYOSA
+echo "Granting Service Account User role to $CLOUD_BUILD_SA on $BYOSA_EMAIL..."
+gcloud iam service-accounts add-iam-policy-binding "$BYOSA_EMAIL" \
+  --member="serviceAccount:${CLOUD_BUILD_SA}" \
+  --role='roles/iam.serviceAccountUser' \
+  --project="$CI_PROJECT_ID"
+
+echo "Permissions granted."
+
+# --- Write cloudbuild.yaml ---
+CLOUDBUILD_FILE="cloudbuild.yaml"
+echo "Writing $CLOUDBUILD_FILE..."
+cat > "$CLOUDBUILD_FILE" << EOF
+steps:
+  - name: 'gcr.io/cloud-builders/gcloud'
+    entrypoint: 'bash'
+    args:
+      - '-c'
+      - |
+        echo "\$TEST_ENV_JSON" > env.json
+        echo "CI/CD env.json content:"
+        cat env.json
+  # --- Test Standard DPGCE ---
+  - name: 'gcr.io/cloud-builders/gcloud'
+    id: 'Test Standard Create'
+    entrypoint: 'bash'
+    args: ['-c', 'source lib/env.sh && ./bin/create-dpgce && ./bin/audit-dpgce-create']
+    env:
+      - 'PROJECT_ID=\$PROJECT_ID'
+  - name: 'gcr.io/cloud-builders/gcloud'
+    id: 'Test Standard Destroy'
+    entrypoint: 'bash'
+    args: ['-c', 'source lib/env.sh && ./bin/destroy-dpgce --force && ./bin/audit-dpgce-destroy --force']
+    env:
+      - 'PROJECT_ID=\$PROJECT_ID'
+    waitFor: ['Test Standard Create']
+
+  # --- Test Private DPGCE ---
+  - name: 'gcr.io/cloud-builders/gcloud'
+    id: 'Test Private Create'
+    entrypoint: 'bash'
+    args: ['-c', 'source lib/env.sh && ./bin/create-dpgce-private && ./bin/audit-private-create']
+    env:
+      - 'PROJECT_ID=\$PROJECT_ID'
+  - name: 'gcr.io/cloud-builders/gcloud'
+    id: 'Test Private Destroy'
+    entrypoint: 'bash'
+    args: ['-c', 'source lib/env.sh && ./bin/destroy-dpgce-private --force && ./bin/audit-private-destroy --force']
+    env:
+      - 'PROJECT_ID=\$PROJECT_ID'
+    waitFor: ['Test Private Create']
+
+  # --- Test Custom Standard DPGCE ---
+  - name: 'gcr.io/cloud-builders/gcloud'
+    id: 'Test Custom Std Create'
+    entrypoint: 'bash'
+    args: ['-c', 'source lib/env.sh && ./bin/create-dpgce-custom && ./bin/audit-dpgce-create-custom']
+    env:
+      - 'PROJECT_ID=\$PROJECT_ID'
+  - name: 'gcr.io/cloud-builders/gcloud'
+    id: 'Test Custom Std Destroy'
+    entrypoint: 'bash'
+    args: ['-c', 'source lib/env.sh && ./bin/destroy-dpgce --force && ./bin/audit-dpgce-destroy --force']
+    env:
+      - 'PROJECT_ID=\$PROJECT_ID'
+    waitFor: ['Test Custom Std Create']
+
+  # --- Test Custom Private DPGCE ---
+  - name: 'gcr.io/cloud-builders/gcloud'
+    id: 'Test Custom Pvt Create'
+    entrypoint: 'bash'
+    args: ['-c', 'source lib/env.sh && ./bin/create-dpgce-custom-private && ./bin/audit-dpgce-create-custom-private']
+    env:
+      - 'PROJECT_ID=\$PROJECT_ID'
+  - name: 'gcr.io/cloud-builders/gcloud'
+    id: 'Test Custom Pvt Destroy'
+    entrypoint: 'bash'
+    args: ['-c', 'source lib/env.sh && ./bin/destroy-dpgce-private --force && ./bin/audit-private-destroy --force']
+    env:
+      - 'PROJECT_ID=\$PROJECT_ID'
+    waitFor: ['Test Custom Pvt Create']
+
+availableSecrets:
+  secretManager:
+  - versionName: projects/\$PROJECT_ID/secrets/test-env-json/versions/latest
+    env: 'TEST_ENV_JSON'
+
+options:
+  env:
+  - 'PROJECT_ID=$CI_PROJECT_ID' # Use CI_PROJECT_ID here
+  - 'CLOUDSDK_CORE_DISABLE_PROMPTS=1'
+substitutions:
+  _CI_PROJECT_ID: "$CI_PROJECT_ID"
+timeout: 3600s # 60 minutes
+EOF
+echo "$CLOUDBUILD_FILE written."
+
+# --- Configure Triggers ---
+echo "Instructions to create Cloud Build Trigger:"
+echo "1. Go to Cloud Build Triggers in project $CI_PROJECT_ID."
+echo "2. Connect your repository if not already done."
+echo "3. Create a trigger:"
+echo "   - Name: ${CI_CSR_REPO_NAME}-trigger"
+echo "   - Event: Push to a branch"
+echo "   - Source: Your repository, Branch: ^${CI_TRIGGER_BRANCH}$"
+echo "   - Configuration: Cloud Build configuration file (yaml)"
+echo "   - Location: /cloudbuild.yaml"
+echo "   - Service Account: Default Cloud Build SA should be fine with above permissions."
+
+echo "Alternative gcloud command to create trigger (adjust connection details if not using CSR):"
+echo "gcloud beta builds triggers create github --project="$CI_PROJECT_ID" "
+echo "  --name="${CI_CSR_REPO_NAME}-trigger" "
+echo "  --repo-name="${CI_CSR_REPO_NAME}" "
+echo "  --repo-owner="YOUR_GITHUB_USERNAME" "
+echo "  --branch-pattern="^${CI_TRIGGER_BRANCH}$" "
+echo "  --build-config="cloudbuild.yaml" "
+echo "  --region="$CI_CSR_REGION""
+
+echo "Setup script finished. PLEASE REVIEW AND ADJUST env.json and the trigger commands."
diff --git a/gcloud/bin/ssh-m b/gcloud/bin/ssh-m
index 4bfbfbb9..53bf4979 100755
--- a/gcloud/bin/ssh-m
+++ b/gcloud/bin/ssh-m
@@ -14,14 +14,21 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
-source lib/env.sh
+# --- Get script's real directory ---
+SCRIPT_DIR=$(dirname "${BASH_SOURCE[0]}")
+GCLOUD_DIR="$(realpath "${SCRIPT_DIR}/..")"
+export GCLOUD_DIR
 
-master_num="$1"
+source "${GCLOUD_DIR}/lib/env.sh"
 
-MASTER_HOSTNAME="${CLUSTER_NAME}-m"
-if [[ -n "$master_num" ]]; then
-  echo "master_num: $master_num"
-  MASTER_HOSTNAME="${MASTER_HOSTNAME}-${master_num}"
+M_HOSTNAME="${CLUSTER_NAME}-m"
+# If the first argument is a number, treat it as the master index for HA
+if [[ "$1" =~ ^[0-9]+$ ]]; then
+  M_HOSTNAME="${M_HOSTNAME}-${1}"
+  shift # Remove the index from the arguments
 fi
 
-gcloud compute ssh --zone ${ZONE} ${MASTER_HOSTNAME}  --tunnel-through-iap --project ${PROJECT_ID} -- -o ConnectTimeout=360
+# The rest of the arguments are the command to be executed
+COMMAND_TO_RUN="$@"
+
+gcloud compute ssh --zone "${ZONE}" "${M_HOSTNAME}" --tunnel-through-iap --project "${PROJECT_ID}" -- -t -o ConnectTimeout=360 -AY ${COMMAND_TO_RUN}
diff --git a/gcloud/cloudbuild.yaml b/gcloud/cloudbuild.yaml
new file mode 100644
index 00000000..c09a9a4b
--- /dev/null
+++ b/gcloud/cloudbuild.yaml
@@ -0,0 +1,81 @@
+steps:
+  - name: 'gcr.io/cloud-builders/gcloud'
+    entrypoint: 'bash'
+    args:
+      - '-c'
+      - |
+        echo "$TEST_ENV_JSON" > env.json
+        echo "CI/CD env.json content:"
+        cat env.json
+  # --- Test Standard DPGCE ---
+  - name: 'gcr.io/cloud-builders/gcloud'
+    id: 'Test Standard Create'
+    entrypoint: 'bash'
+    args: ['-c', 'source lib/env.sh && ./bin/create-dpgce && ./bin/audit-dpgce-create']
+    env:
+      - 'PROJECT_ID=$PROJECT_ID'
+  - name: 'gcr.io/cloud-builders/gcloud'
+    id: 'Test Standard Destroy'
+    entrypoint: 'bash'
+    args: ['-c', 'source lib/env.sh && ./bin/destroy-dpgce --force && ./bin/audit-dpgce-destroy --force']
+    env:
+      - 'PROJECT_ID=$PROJECT_ID'
+    waitFor: ['Test Standard Create']
+
+  # --- Test Private DPGCE ---
+  - name: 'gcr.io/cloud-builders/gcloud'
+    id: 'Test Private Create'
+    entrypoint: 'bash'
+    args: ['-c', 'source lib/env.sh && ./bin/create-dpgce-private && ./bin/audit-private-create']
+    env:
+      - 'PROJECT_ID=$PROJECT_ID'
+  - name: 'gcr.io/cloud-builders/gcloud'
+    id: 'Test Private Destroy'
+    entrypoint: 'bash'
+    args: ['-c', 'source lib/env.sh && ./bin/destroy-dpgce-private --force && ./bin/audit-private-destroy --force']
+    env:
+      - 'PROJECT_ID=$PROJECT_ID'
+    waitFor: ['Test Private Create']
+
+  # --- Test Custom Standard DPGCE ---
+  - name: 'gcr.io/cloud-builders/gcloud'
+    id: 'Test Custom Std Create'
+    entrypoint: 'bash'
+    args: ['-c', 'source lib/env.sh && ./bin/create-dpgce-custom && ./bin/audit-dpgce-create-custom']
+    env:
+      - 'PROJECT_ID=$PROJECT_ID'
+  - name: 'gcr.io/cloud-builders/gcloud'
+    id: 'Test Custom Std Destroy'
+    entrypoint: 'bash'
+    args: ['-c', 'source lib/env.sh && ./bin/destroy-dpgce --force && ./bin/audit-dpgce-destroy --force']
+    env:
+      - 'PROJECT_ID=$PROJECT_ID'
+    waitFor: ['Test Custom Std Create']
+
+  # --- Test Custom Private DPGCE ---
+  - name: 'gcr.io/cloud-builders/gcloud'
+    id: 'Test Custom Pvt Create'
+    entrypoint: 'bash'
+    args: ['-c', 'source lib/env.sh && ./bin/create-dpgce-custom-private && ./bin/audit-dpgce-create-custom-private']
+    env:
+      - 'PROJECT_ID=$PROJECT_ID'
+  - name: 'gcr.io/cloud-builders/gcloud'
+    id: 'Test Custom Pvt Destroy'
+    entrypoint: 'bash'
+    args: ['-c', 'source lib/env.sh && ./bin/destroy-dpgce-private --force && ./bin/audit-private-destroy --force']
+    env:
+      - 'PROJECT_ID=$PROJECT_ID'
+    waitFor: ['Test Custom Pvt Create']
+
+availableSecrets:
+  secretManager:
+  - versionName: projects/$PROJECT_ID/secrets/test-env-json/versions/latest
+    env: 'TEST_ENV_JSON'
+
+options:
+  env:
+  - 'PROJECT_ID=cjac-2025-01' # Use CI_PROJECT_ID here
+  - 'CLOUDSDK_CORE_DISABLE_PROMPTS=1'
+substitutions:
+  _CI_PROJECT_ID: "cjac-2025-01"
+timeout: 3600s # 60 minutes
diff --git a/gcloud/env.json.sample b/gcloud/env.json.sample
index 49d1fff9..7062d4da 100644
--- a/gcloud/env.json.sample
+++ b/gcloud/env.json.sample
@@ -7,12 +7,15 @@
   "PRIV_DOMAIN": "privileged-domain-here.com",
   "BILLING_ACCOUNT": "100000-000000-000001",
   "FOLDER_NUMBER": "100000000001",
-  "REGION": "us-west4",
-  "ZONE": "us-west4-a",
+  "REGION": "us-east4",
+  "ZONE": "us-east4-a",
+  "UNIVERSE_DOMAIN": "googleapis.com",
   "RANGE": "10.0.1.0/24",
   "IDLE_TIMEOUT": "30m",
   "ASN_NUMBER": "65531",
   "IMAGE_VERSION": "2.2",
+  "CUSTOM_IMAGE_URI": "projects/your-project/global/images/your-custom-image",
+  "ACCELERATOR_TYPE": "nvidia-tesla-t4", // Only used if --gpu is passed to create-dpgce. Check Cloud Docs for compatible machine types.
   "BUCKET": "myproject-dataproc-repro-bucket",
   "TEMP_BUCKET": "myproject-dataproc-repro-temp-bucket",
   "CLUSTER_NAME": "cluster-name-here",
@@ -23,5 +26,13 @@
   "SWP_RANGE": "10.0.3.0/24",
   "SWP_HOSTNAME": "swp.internal.local",
   "SWP_IP": "10.0.2.245",
-  "SWP_PORT": 3128
+  "SWP_PORT": 3128,
+  "CI_PROJECT_ID": "your-ci-test-project-id",
+  "CI_GCP_CREDENTIALS_PATH": "/path/to/your/ci-service-account.json",
+  "CI_CSR_REPO_NAME": "your-cloud-source-repo",
+  "CI_CSR_REGION": "us-central1",
+  "CI_GITHUB_CONNECTION_NAME": "github-connection",
+  "CI_TRIGGER_BRANCH": "main",
+  "CI_REPO_OWNER": "your-github-username",
+  "CI_BYOSA_EMAIL": "your-byosa-email@your-ci-test-project-id.iam.gserviceaccount.com"
 }
diff --git a/gcloud/init/gce-proxy-setup.sh b/gcloud/init/gce-proxy-setup.sh
index 0a49202d..dd31c15c 100644
--- a/gcloud/init/gce-proxy-setup.sh
+++ b/gcloud/init/gce-proxy-setup.sh
@@ -55,167 +55,310 @@ function is_debuntu()  {  [[ "$(os_id)" == "debian" || "$(os_id)" == "ubuntu" ]]
 function is_rocky()    {  [[ "$(os_id)" == "rocky" ]] ; }
 # --- End OS Detection Helpers ---
 
+# --- Version Comparison Helpers ---
+function version_le(){ [[ "$1" = "$(echo -e "$1\n$2"|sort -V|head -n1)" ]]; }
+function version_lt(){ [[ "$1" = "$2" ]] && return 1 || version_le "$1" "$2"; }
+# --- End Version Comparison Helpers ---
+
+function execute_with_retries() {
+  local -r cmd="$*"
+  local retries=3
+  local delay=5
+  for ((i = 0; i < retries; i++)); do
+    eval "${cmd}" && return 0
+    echo "Command failed. Retrying in ${delay} seconds..." >&2
+    sleep "${delay}"
+  done
+  echo "Command failed after ${retries} retries: ${cmd}" >&2
+  return 1
+}
+
 function set_proxy(){
-  METADATA_HTTP_PROXY="$(get_metadata_attribute http-proxy '')"
+  # Idempotency Check for Proxy
+  if grep -q "http_proxy=" /etc/environment && [[ -n "${http_proxy:-}" ]]; then
+    echo "INFO: Proxy already configured in /etc/environment. Skipping proxy setup portion."
+    return 0
+  fi
+
+  local meta_http_proxy meta_https_proxy meta_proxy_uri
+  meta_http_proxy=$(get_metadata_attribute 'http-proxy' '')
+  meta_https_proxy=$(get_metadata_attribute 'https-proxy' '')
+  meta_proxy_uri=$(get_metadata_attribute 'proxy-uri' '')
+
+  echo "DEBUG: set_proxy: meta_http_proxy='${meta_http_proxy}'"
+  echo "DEBUG: set_proxy: meta_https_proxy='${meta_https_proxy}'"
+  echo "DEBUG: set_proxy: meta_proxy_uri='${meta_proxy_uri}'"
+
+  local http_proxy_val=""
+  local https_proxy_val=""
+
+  # Determine HTTP_PROXY value
+  if [[ -n "${meta_http_proxy}" ]] && [[ "${meta_http_proxy}" != ":" ]]; then
+    http_proxy_val="${meta_http_proxy}"
+  elif [[ -n "${meta_proxy_uri}" ]] && [[ "${meta_proxy_uri}" != ":" ]]; then
+    http_proxy_val="${meta_proxy_uri}"
+  fi
+
+  # Determine HTTPS_PROXY value
+  if [[ -n "${meta_https_proxy}" ]] && [[ "${meta_https_proxy}" != ":" ]]; then
+    https_proxy_val="${meta_https_proxy}"
+  elif [[ -n "${meta_proxy_uri}" ]] && [[ "${meta_proxy_uri}" != ":" ]]; then
+    https_proxy_val="${meta_proxy_uri}"
+  fi
+
+  if [[ -z "${http_proxy_val}" && -z "${https_proxy_val}" ]]; then
+    echo "DEBUG: set_proxy: No valid proxy metadata found. Skipping proxy config."
+    return 0
+  fi
+
+  local default_no_proxy_list=(
+    "localhost"
+    "127.0.0.1"
+    "::1"
+    "metadata.google.internal"
+    "169.254.169.254"
+    ".google.com"
+    ".googleapis.com"
+    ".internal"
+  )
 
-  if [[ -z "${METADATA_HTTP_PROXY}" ]] ; then return ; fi
+  # Add project-specific internal domain
+  local project_id
+  project_id=$(get_metadata_attribute 'project-id' "${PROJECT_ID:-}")
+  if [[ -n "${project_id}" ]]; then
+    default_no_proxy_list+=( ".c.${project_id}.internal" )
+  fi
 
-default_no_proxy_list=("localhost" "127.0.0.0/8" "::1" "*.googleapis.com"
-			"metadata.google.internal" "169.254.169.254")
+  # Add cluster-specific hostnames
+  local cluster_name
+  cluster_name=$(get_metadata_attribute 'dataproc-cluster-name' '')
+  if [[ -n "${cluster_name}" ]]; then
+    # Add wildcard patterns (supported by some tools like Go/Java)
+    default_no_proxy_list+=( "${cluster_name}-m" "${cluster_name}-m-*" "${cluster_name}-w-*" "${cluster_name}-sw-*" )
+    # Add FQDN suffixes to ensure bypass for tools like curl/wget
+    default_no_proxy_list+=( "${cluster_name}-m.c.${project_id}.internal" )
+    default_no_proxy_list+=( ".c.${project_id}.internal" )
+  fi
 
+  local user_no_proxy
   user_no_proxy=$(get_metadata_attribute 'no-proxy' '')
-  user_no_proxy_list=()
+  local user_no_proxy_list=()
   if [[ -n "${user_no_proxy}" ]]; then
-    # Replace spaces with commas, then split by comma
     IFS=',' read -r -a user_no_proxy_list <<< "${user_no_proxy// /,}"
   fi
 
-  combined_no_proxy_list=("${default_no_proxy_list[@]}" "${user_no_proxy_list[@]}")
+  local combined_no_proxy_list=( "${default_no_proxy_list[@]}" "${user_no_proxy_list[@]}" )
+  local no_proxy
   no_proxy=$( IFS=',' ; echo "${combined_no_proxy_list[*]}" )
-
-  export http_proxy="http://${METADATA_HTTP_PROXY}"
-  export https_proxy="http://${METADATA_HTTP_PROXY}"
-  export no_proxy
-  export HTTP_PROXY="http://${METADATA_HTTP_PROXY}"
-  export HTTPS_PROXY="http://${METADATA_HTTP_PROXY}"
   export NO_PROXY="${no_proxy}"
+  export no_proxy="${no_proxy}"
 
-  # configure gcloud
-  # There is no no_proxy config for gcloud so we cannot use these settings until https://github.com/psf/requests/pull/7068 is merged
-#  gcloud config set proxy/type http
-#  gcloud config set proxy/address "${METADATA_HTTP_PROXY%:*} "
-#  gcloud config set proxy/port "${METADATA_HTTP_PROXY#*:}"
+  # Export environment variables
+  if [[ -n "${http_proxy_val}" ]]; then
+    export HTTP_PROXY="http://${http_proxy_val}"
+    export http_proxy="http://${http_proxy_val}"
+  fi
+  if [[ -n "${https_proxy_val}" ]]; then
+    export HTTPS_PROXY="http://${https_proxy_val}"
+    export https_proxy="http://${https_proxy_val}"
+  fi
 
-  # add proxy environment variables to /etc/environment
-  grep http_proxy /etc/environment || echo "http_proxy=${http_proxy}" >> /etc/environment
-  grep https_proxy /etc/environment || echo "https_proxy=${https_proxy}" >> /etc/environment
-  grep no_proxy /etc/environment || echo "no_proxy=${no_proxy}" >> /etc/environment
-  grep HTTP_PROXY /etc/environment || echo "HTTP_PROXY=${HTTP_PROXY}" >> /etc/environment
-  grep HTTPS_PROXY /etc/environment || echo "HTTPS_PROXY=${HTTPS_PROXY}" >> /etc/environment
-  grep NO_PROXY /etc/environment || echo "NO_PROXY=${NO_PROXY}" >> /etc/environment
+  # Clear existing proxy settings in /etc/environment
+  sed -i -e '/^http_proxy=/d' -e '/^https_proxy=/d' -e '/^no_proxy=/d' \
+    -e '/^HTTP_PROXY=/d' -e '/^HTTPS_PROXY=/d' -e '/^NO_PROXY=/d' /etc/environment
 
-  local pkg_proxy_conf_file
-  if is_debuntu ; then
-    # configure Apt to use the proxy:
-    pkg_proxy_conf_file="/etc/apt/apt.conf.d/99proxy"
-    cat > "${pkg_proxy_conf_file}" <<EOF
-Acquire::http::Proxy "http://${METADATA_HTTP_PROXY}";
-Acquire::https::Proxy "http://${METADATA_HTTP_PROXY}";
-EOF
-elif is_rocky ; then
-    pkg_proxy_conf_file="/etc/dnf/dnf.conf"
+  # Add current proxy environment variables to /etc/environment
+  if [[ -n "${HTTP_PROXY:-}" ]]; then echo "HTTP_PROXY=${HTTP_PROXY}" >> /etc/environment; fi
+  if [[ -n "${http_proxy:-}" ]]; then echo "http_proxy=${http_proxy}" >> /etc/environment; fi
+  if [[ -n "${HTTPS_PROXY:-}" ]]; then echo "HTTPS_PROXY=${HTTPS_PROXY}" >> /etc/environment; fi
+  if [[ -n "${https_proxy:-}" ]]; then echo "https_proxy=${https_proxy}" >> /etc/environment; fi
+  if [[ -n "${NO_PROXY:-}" ]]; then echo "NO_PROXY=${NO_PROXY}" >> /etc/environment; fi
+  if [[ -n "${NO_PROXY:-}" ]]; then echo "no_proxy=${no_proxy}" >> /etc/environment; fi
 
-    touch "${pkg_proxy_conf_file}"
+  # Persist for all shell sessions
+  local profile_script="/etc/profile.d/proxy.sh"
+  echo "# Proxy settings from Dataproc init action" > "${profile_script}"
+  if [[ -n "${HTTP_PROXY:-}" ]]; then echo "export HTTP_PROXY='${HTTP_PROXY}'" >> "${profile_script}"; fi
+  if [[ -n "${http_proxy:-}" ]]; then echo "export http_proxy='${http_proxy}'" >> "${profile_script}"; fi
+  if [[ -n "${HTTPS_PROXY:-}" ]]; then echo "export HTTPS_PROXY='${HTTPS_PROXY}'" >> "${profile_script}"; fi
+  if [[ -n "${https_proxy:-}" ]]; then echo "export https_proxy='${https_proxy}'" >> "${profile_script}"; fi
+  if [[ -n "${NO_PROXY:-}" ]]; then echo "export NO_PROXY='${NO_PROXY}'" >> "${profile_script}"; fi
+  if [[ -n "${no_proxy:-}" ]]; then echo "export no_proxy='${no_proxy}'" >> "${profile_script}"; fi
+
+  # Source the script to apply settings to the current shell
+  source "${profile_script}"
 
-    if grep -q "^proxy=" "${pkg_proxy_conf_file}"; then
-      sed -i.bak "s@^proxy=.*@proxy=${HTTP_PROXY}@" "${pkg_proxy_conf_file}"
-    elif grep -q "^\\\[main\\\\]" "${pkg_proxy_conf_file}"; then
-      sed -i.bak "/^\\\[main\\\\]/a proxy=${HTTP_PROXY}" "${pkg_proxy_conf_file}"
+  # Configure gcloud proxy
+  local gcloud_version
+  local -r min_gcloud_proxy_ver="547.0.0"
+  gcloud_version=$(gcloud version --format="value(google_cloud_sdk)" 2>/dev/null || echo "0.0.0")
+  if version_ge "${gcloud_version}" "${min_gcloud_proxy_ver}"; then
+    if [[ -n "${http_proxy_val}" ]]; then
+      local proxy_host=$(echo "${http_proxy_val}" | cut -d: -f1)
+      local proxy_port=$(echo "${http_proxy_val}" | cut -d: -f2)
+      gcloud config set proxy/type http
+      gcloud config set proxy/address "${proxy_host}"
+      gcloud config set proxy/port "${proxy_port}"
     else
-      local TMP_FILE=$(mktemp)
-      printf "[main]\nproxy=%s\n" "${HTTP_PROXY}" > "${TMP_FILE}"
+      gcloud config unset proxy/type
+      gcloud config unset proxy/address
+      gcloud config unset proxy/port
+    fi
+  fi
+
+  # Install the HTTPS proxy's certificate
+  local proxy_ca_pem=""
+  local trusted_pem_path=""
+  METADATA_HTTP_PROXY_PEM_URI="$(get_metadata_attribute http-proxy-pem-uri '')"
+  if [[ -n "${METADATA_HTTP_PROXY_PEM_URI}" ]] ; then
+    if [[ ! "${METADATA_HTTP_PROXY_PEM_URI}" =~ ^gs:// ]] ; then echo "ERROR: http-proxy-pem-uri value must start with gs://" ; exit 1 ; fi
+    echo "DEBUG: set_proxy: Processing http-proxy-pem-uri='${METADATA_HTTP_PROXY_PEM_URI}'"
+    local trusted_pem_dir
+    if is_debuntu ; then
+      trusted_pem_dir="/usr/local/share/ca-certificates"
+      proxy_ca_pem="${trusted_pem_dir}/proxy_ca.crt"
+      mkdir -p "${trusted_pem_dir}"
+      gsutil cp "${METADATA_HTTP_PROXY_PEM_URI}" "${proxy_ca_pem}" || { echo "ERROR: Failed to download proxy CA cert from GCS." ; exit 1 ; }
+      update-ca-certificates
+      trusted_pem_path="/etc/ssl/certs/ca-certificates.crt"
+    elif is_rocky ; then
+      trusted_pem_dir="/etc/pki/ca-trust/source/anchors"
+      proxy_ca_pem="${trusted_pem_dir}/proxy_ca.crt"
+      mkdir -p "${trusted_pem_dir}"
+      gsutil cp "${METADATA_HTTP_PROXY_PEM_URI}" "${proxy_ca_pem}" || { echo "ERROR: Failed to download proxy CA cert from GCS." ; exit 1 ; }
+      update-ca-trust
+      trusted_pem_path="/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem"
+    fi
+    export REQUESTS_CA_BUNDLE="${trusted_pem_path}"
+    echo "DEBUG: set_proxy: trusted_pem_path set to '${trusted_pem_path}'"
+
+    # Add to Java/Conda trust stores
+    if [[ -f "/etc/environment" ]]; then
+        JAVA_HOME="$(awk -F= '/^JAVA_HOME=/ {print $2}' /etc/environment)"
+        if [[ -n "${JAVA_HOME:-}" && -f "${JAVA_HOME}/bin/keytool" ]]; then
+            "${JAVA_HOME}/bin/keytool" -import -cacerts -storepass changeit -noprompt -alias swp_ca -file "${proxy_ca_pem}"
+        fi
+    fi
+    if command -v conda &> /dev/null ; then
+      local conda_cert_file="/opt/conda/default/ssl/cacert.pem"
+      if [[ -f "${conda_cert_file}" ]]; then
+        local ca_subject=$(openssl crl2pkcs7 -nocrl -certfile "${proxy_ca_pem}" | openssl pkcs7 -print_certs -noout | grep ^subject)
+        openssl crl2pkcs7 -nocrl -certfile "${conda_cert_file}" | openssl pkcs7 -print_certs -noout | grep -Fxq "${ca_subject}" || {
+          cat "${proxy_ca_pem}" >> "${conda_cert_file}"
+        }
+      fi
+    fi
+  fi
+
+  if [[ -n "${http_proxy_val}" ]]; then
 
-      cat "${TMP_FILE}" "${pkg_proxy_conf_file}" > "${pkg_proxy_conf_file}".new
-      mv "${pkg_proxy_conf_file}".new "${pkg_proxy_conf_file}"
+    local proxy_host=$(echo "${http_proxy_val}" | cut -d: -f1)
+    local proxy_port=$(echo "${http_proxy_val}" | cut -d: -f2)
 
-      rm "${TMP_FILE}"
+    echo "DEBUG: set_proxy: Testing TCP connection to proxy ${proxy_host}:${proxy_port}..."
+    if ! nc -zv -w 5 "${proxy_host}" "${proxy_port}"; then
+      echo "ERROR: Failed to establish TCP connection to proxy ${proxy_host}:${proxy_port}."
+      exit 1
+    fi
+
+    echo "DEBUG: set_proxy: Testing external site access via proxy..."
+    local test_url="https://www.google.com"
+    local curl_test_args=()
+    if [[ -n "${trusted_pem_path}" ]]; then
+      curl_test_args+=(--cacert "${trusted_pem_path}")
+    fi
+    if curl "${curl_test_args[@]}" -vL --retry 3 --retry-delay 5 -o /dev/null "${test_url}"; then
+      echo "DEBUG: set_proxy: Successfully fetched ${test_url} via proxy."
+    else
+      echo "ERROR: Failed to fetch ${test_url} via proxy ${HTTP_PROXY}."
+      exit 1
     fi
-  else
-    echo "unknown OS"
-    exit 1
   fi
-  # configure gpg to use the proxy:
-  if ! grep 'keyserver-options http-proxy' /etc/gnupg/dirmngr.conf ; then
-    mkdir -p /etc/gnupg
-    cat >> /etc/gnupg/dirmngr.conf <<EOF
-http-proxy http://${METADATA_HTTP_PROXY}
-EOF
+
+  # Configure package managers
+  local pkg_proxy_conf_file
+  local effective_proxy="${http_proxy_val:-${https_proxy_val}}"
+  if [[ -z "${effective_proxy}" ]]; then
+      echo "DEBUG: set_proxy: No HTTP or HTTPS proxy set for package managers."
+  elif is_debuntu ; then
+    pkg_proxy_conf_file="/etc/apt/apt.conf.d/99proxy"
+    echo "Acquire::http::Proxy \"http://${effective_proxy}\";" > "${pkg_proxy_conf_file}"
+    echo "Acquire::https::Proxy \"http://${effective_proxy}\";" >> "${pkg_proxy_conf_file}"
+  elif is_rocky ; then
+    pkg_proxy_conf_file="/etc/dnf/dnf.conf"
+    touch "${pkg_proxy_conf_file}"
+    sed -i.bak '/^proxy=/d' "${pkg_proxy_conf_file}"
+    if grep -q "^\[main\]" "${pkg_proxy_conf_file}"; then
+      sed -i.bak "/^\\\[main\\\\]/a proxy=http://${effective_proxy}" "${pkg_proxy_conf_file}"
+    else
+      echo -e "[main]\nproxy=http://${effective_proxy}" >> "${pkg_proxy_conf_file}"
+    fi
   fi
 
-  # Install the HTTPS proxy's certificate in the system and Java trust databases
-  METADATA_HTTP_PROXY_PEM_URI="$(get_metadata_attribute http-proxy-pem-uri '')"
+  # Configure dirmngr
+  if is_debuntu ; then
+    if ! dpkg -l | grep -q dirmngr; then
+      execute_with_retries apt-get install -y -qq dirmngr
+    fi
+  elif is_rocky ; then
+    if ! rpm -q gnupg2-smime; then
+      execute_with_retries dnf install -y -q gnupg2-smime
+    fi
+  fi
+  mkdir -p /etc/gnupg
+  local dirmngr_conf="/etc/gnupg/dirmngr.conf"
+  touch "${dirmngr_conf}"
+  sed -i.bak '/^http-proxy/d' "${dirmngr_conf}"
+  if [[ -n "${HTTP_PROXY:-}" ]]; then
+    echo "http-proxy ${HTTP_PROXY}" >> "${dirmngr_conf}"
+  fi
+}
 
-  if [[ -z "${METADATA_HTTP_PROXY_PEM_URI}" ]] ; then return ; fi
-  if [[ ! "${METADATA_HTTP_PROXY_PEM_URI}" =~ ^gs ]] ; then echo "http-proxy-pem-uri value should start with gs://" ; exit 1 ; fi
+function repair_boto() {
+  local boto_file="/etc/boto.cfg"
+  if [[ -f "${boto_file}" ]]; then
+    echo "DEBUG: repair_boto: Repairing and deduplicating ${boto_file}" >&2
+    
+    # 1. Deduplicate sections (fix for DuplicateSectionError)
+    # Use a more robust perl one-liner that also handles the content within duplicate sections
+    # by only keeping the first occurrence of each section and its variables.
+    perl -i -ne '
+      if (/^\[(.*)\]/) {
+        $section = $1;
+        $skip = $seen{$section}++;
+      }
+      print unless $skip;
+    ' "${boto_file}"
+    
+    # 2. Fix universe_domain if it is still a variable
+    local universe_domain
+    universe_domain=$(get_metadata_attribute 'universe-domain' 'googleapis.com')
+    # Use a more robust replacement that handles potential escaping issues
+    UNIVERSE_DOMAIN="${universe_domain}" perl -i -pe 's/\$\{universe_domain\}/$ENV{UNIVERSE_DOMAIN}/g' "${boto_file}"
+    # Also fix cases where it might have been partially expanded to storage.$
+    UNIVERSE_DOMAIN="${universe_domain}" perl -i -pe 's/storage\.\$/storage.$ENV{UNIVERSE_DOMAIN}/g' "${boto_file}"
 
-  local trusted_pem_dir
-  # Add this certificate to the OS trust database
-  # When proxy cert is provided, speak to the proxy over https
-  if is_debuntu ; then
-    trusted_pem_dir="/usr/local/share/ca-certificates"
-    mkdir -p "${trusted_pem_dir}"
-    proxy_ca_pem="${trusted_pem_dir}/proxy_ca.crt"
-    gsutil cp "${METADATA_HTTP_PROXY_PEM_URI}" "${proxy_ca_pem}"
-    update-ca-certificates
-    trusted_pem_path="/etc/ssl/certs/ca-certificates.crt"
-    sed -i -e 's|http://|https://|' "${pkg_proxy_conf_file}"
-elif is_rocky ; then
-    trusted_pem_dir="/etc/pki/ca-trust/source/anchors"
-    mkdir -p "${trusted_pem_dir}"
-    proxy_ca_pem="${trusted_pem_dir}/proxy_ca.crt"
-    gsutil cp "${METADATA_HTTP_PROXY_PEM_URI}" "${proxy_ca_pem}"
-    update-ca-trust
-    trusted_pem_path="/etc/ssl/certs/ca-bundle.crt"
-    sed -i -e 's|^proxy=http://|proxy=https://|' "${pkg_proxy_conf_file}"
-  else
-    echo "unknown OS"
-    exit 1
-  fi
-
-  # configure gcloud to respect proxy ca cert
-  #gcloud config set core/custom_ca_certs_file "${proxy_ca_pem}"
-
-  ca_subject="$(openssl crl2pkcs7 -nocrl -certfile "${proxy_ca_pem}" | openssl pkcs7 -print_certs -noout | grep ^subject)"
-  # Verify that the proxy certificate is trusted
-  local output
-  output=$(echo | openssl s_client \
-           -connect "${METADATA_HTTP_PROXY}" \
-           -proxy "${METADATA_HTTP_PROXY}" \
-           -CAfile "${proxy_ca_pem}") || {
-    echo "proxy certificate verification failed"
-    echo "${output}"
-    exit 1
-  }
-  output=$(echo | openssl s_client \
-           -connect "${METADATA_HTTP_PROXY}" \
-           -proxy "${METADATA_HTTP_PROXY}" \
-           -CAfile "${trusted_pem_path}") || {
-    echo "proxy ca certificate not included in system bundle"
-    echo "${output}"
-    exit 1
-  }
-  output=$(curl --verbose -fsSL --retry-connrefused --retry 10 --retry-max-time 30 --head "https://google.com" 2>&1)|| {
-    echo "curl rejects proxy configuration"
-    echo "${output}"
-    exit 1
-  }
-  output=$(curl --verbose -fsSL --retry-connrefused --retry 10 --retry-max-time 30 --head "https://developer.download.nvidia.com/compute/cuda/12.6.3/local_installers/cuda_12.6.3_560.35.05_linux.run" 2>&1)|| {
-    echo "curl rejects proxy configuration"
-    echo "${output}"
-    exit 1
-  }
-
-  # Instruct conda to use the system certificate
-  echo "Attempting to install pip-system-certs using the proxy certificate..."
-  export REQUESTS_CA_BUNDLE="${trusted_pem_path}"
-  pip install pip-system-certs
-  unset REQUESTS_CA_BUNDLE
-
-  # For the binaries bundled with conda, append our certificate to the bundle
-  openssl crl2pkcs7 -nocrl -certfile /opt/conda/default/ssl/cacert.pem | openssl pkcs7 -print_certs -noout | grep -Fx "${ca_subject}" || {
-    cat "${proxy_ca_pem}" >> /opt/conda/default/ssl/cacert.pem
-  }
-
-  sed -i -e 's|http://|https://|' /etc/gnupg/dirmngr.conf
-  export http_proxy="https://${METADATA_HTTP_PROXY}"
-  export https_proxy="https://${METADATA_HTTP_PROXY}"
-  export HTTP_PROXY="https://${METADATA_HTTP_PROXY}"
-  export HTTPS_PROXY="https://${METADATA_HTTP_PROXY}"
-  sed -i -e 's|proxy=http://|proxy=https://|'  -e 's|PROXY=http://|PROXY=https://|' /etc/environment
-
-  # Instruct the JRE to trust the certificate
-  JAVA_HOME="$(awk -F= '/^JAVA_HOME=/ {print $2}' /etc/environment)"
-  "${JAVA_HOME}/bin/keytool" -import -cacerts -storepass changeit -noprompt -alias swp_ca -file "${proxy_ca_pem}"
+    # 3. Apply proxy if set
+    local meta_http_proxy=$(get_metadata_attribute 'http-proxy' '')
+    local meta_proxy_uri=$(get_metadata_attribute 'proxy-uri' '')
+    local effective_proxy="${meta_http_proxy:-${meta_proxy_uri}}"
+    
+    if [[ -n "${effective_proxy}" ]]; then
+      local proxy_host="${effective_proxy%:*}"
+      local proxy_port="${effective_proxy##*:}"
+      
+      sed -i -e '/^proxy =/d' -e '/^proxy_port =/d' "${boto_file}"
+      if grep -q "^\[Boto\]" "${boto_file}"; then
+        sed -i "/^\[Boto\]/a proxy = ${proxy_host}\nproxy_port = ${proxy_port}" "${boto_file}"
+      else
+        echo -e "\n[Boto]\nproxy = ${proxy_host}\nproxy_port = ${proxy_port}" >> "${boto_file}"
+      fi
+    fi
+    echo "DEBUG: repair_boto: Updated ${boto_file}" >&2
+  fi
 }
 
-set_proxy
\ No newline at end of file
+# --- Execution ---
+set_proxy
+repair_boto
+echo "DEBUG: gce-proxy-setup.sh complete." >&2
diff --git a/gcloud/lib/bigtable.sh b/gcloud/lib/bigtable.sh
index 1744d914..4f6015ef 100644
--- a/gcloud/lib/bigtable.sh
+++ b/gcloud/lib/bigtable.sh
@@ -1,60 +1,33 @@
 #!/bin/bash
+#
 # Bigtable functions
 
 function exists_bigtable_instance() {
-  BIGTABLE_INSTANCES="$(gcloud bigtable instances list --format=json)"
-  JQ_CMD=".[] | select(.name | test(\"${BIGTABLE_INSTANCE}$\"))"
-  OUR_INSTANCE=$(echo ${BIGTABLE_INSTANCES} | jq -c "${JQ_CMD}")
-
-  if [[ -z "${OUR_INSTANCE}" ]]; then
-    return 1
-  else
-    return 0
-  fi
+    _check_exists gcloud bigtable instances describe "${BIGTABLE_INSTANCE}" --format="json(name,displayName)"
 }
 export -f exists_bigtable_instance
 
 function create_bigtable_instance() {
-  local phase_name="create_bigtable_instance"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Checking Bigtable Instance ${BIGTABLE_INSTANCE}..."
-    report_result "Exists"
-    return 0
-  fi
-
   print_status "Creating Bigtable Instance ${BIGTABLE_INSTANCE}..."
-  if exists_bigtable_instance; then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
+  local log_file="create_bigtable_${BIGTABLE_INSTANCE}.log"
+  if run_gcloud "${log_file}" gcloud bigtable instances create "${BIGTABLE_INSTANCE}" \
+    --display-name="${BIGTABLE_DISPLAY_NAME}" \
+    --cluster-config="${BIGTABLE_CLUSTER_CONFIG}"; then
+    report_result "Created"
   else
-    local log_file="create_bigtable_${BIGTABLE_INSTANCE}.log"
-    if run_gcloud "${log_file}" gcloud bigtable instances create ${BIGTABLE_INSTANCE} \
-      --display-name "${BIGTABLE_DISPLAY_NAME}" \
-      --cluster-config="${BIGTABLE_CLUSTER_CONFIG}"; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
+    report_result "Fail"
+    return 1
   fi
 }
 export -f create_bigtable_instance
 
 function delete_bigtable_instance() {
-  local phase_name="create_bigtable_instance"
-  remove_sentinel "${phase_name}" "done"
-
   print_status "Deleting Bigtable Instance ${BIGTABLE_INSTANCE}..."
   local log_file="delete_bigtable_${BIGTABLE_INSTANCE}.log"
-  if exists_bigtable_instance; then
-    if run_gcloud "${log_file}" gcloud bigtable instances delete --quiet ${BIGTABLE_INSTANCE}; then
-      report_result "Deleted"
-    else
-      report_result "Fail"
-    fi
+  if run_gcloud "${log_file}" gcloud bigtable instances delete --quiet "${BIGTABLE_INSTANCE}"; then
+    report_result "Deleted"
   else
-    report_result "Not Found"
+    report_result "Fail"
   fi
 }
-export -f delete_bigtable_instance
\ No newline at end of file
+export -f delete_bigtable_instance
diff --git a/gcloud/lib/database/mssql.sh b/gcloud/lib/database/mssql.sh
index 10d1f88d..a8637107 100644
--- a/gcloud/lib/database/mssql.sh
+++ b/gcloud/lib/database/mssql.sh
@@ -3,107 +3,72 @@
 # MS SQL Cloud SQL functions
 
 function create_legacy_mssql_instance() {
-  local phase_name="create_legacy_mssql_instance"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating Legacy MSSQL Instance ${MSSQL_INSTANCE}..."
-    report_result "Exists"
-    return 0
-  fi
-
   print_status "Creating Legacy MSSQL Instance ${MSSQL_INSTANCE}..."
-  if gcloud compute instances describe "${MSSQL_INSTANCE}" --zone "${ZONE}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
+  local log_file="create_legacy_mssql_${MSSQL_INSTANCE}.log"
+  local METADATA="kdc-root-passwd=${INIT_ACTIONS_ROOT}/${KDC_ROOT_PASSWD_KEY}.encrypted"
+  METADATA="${METADATA},kms-keyring=${KMS_KEYRING}"
+  METADATA="${METADATA},kdc-root-passwd-key=${KDC_ROOT_PASSWD_KEY}"
+  METADATA="${METADATA},startup-script-url=${INIT_ACTIONS_ROOT}/kdc-server.sh"
+  METADATA="${METADATA},service-account-user=${GSA}"
+  if run_gcloud "${log_file}" gcloud compute instances create "${MSSQL_INSTANCE}" \
+    --zone "${ZONE}" \
+    --subnet "${SUBNET}" \
+    --service-account="${GSA}" \
+    --boot-disk-type pd-ssd \
+    --image-family="${MSSQL_IMAGE_FAMILY}" \
+    --image-project="${MSSQL_IMAGE_PROJECT}" \
+    --machine-type="${MSSQL_MACHINE_TYPE}" \
+    --scopes='cloud-platform' \
+    --metadata "${METADATA}"; then
+    report_result "Created"
   else
-    local log_file="create_legacy_mssql_${MSSQL_INSTANCE}.log"
-    local METADATA="kdc-root-passwd=${INIT_ACTIONS_ROOT}/${KDC_ROOT_PASSWD_KEY}.encrypted"
-    METADATA="${METADATA},kms-keyring=${KMS_KEYRING}"
-    METADATA="${METADATA},kdc-root-passwd-key=${KDC_ROOT_PASSWD_KEY}"
-    METADATA="${METADATA},startup-script-url=${INIT_ACTIONS_ROOT}/kdc-server.sh"
-    METADATA="${METADATA},service-account-user=${GSA}"
-    if run_gcloud "${log_file}" gcloud compute instances create "${MSSQL_INSTANCE}" \
-      --zone "${ZONE}" \
-      --subnet "${SUBNET}" \
-      --service-account="${GSA}" \
-      --boot-disk-type pd-ssd \
-      --image-family="${MSSQL_IMAGE_FAMILY}" \
-      --image-project="${MSSQL_IMAGE_PROJECT}" \
-      --machine-type="${MSSQL_MACHINE_TYPE}" \
-      --scopes='cloud-platform' \
-      --metadata "${METADATA}"; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
+    report_result "Fail"
+    return 1
   fi
 }
 export -f create_legacy_mssql_instance
 
-function delete_legacy_mssql_instance() {
-  local phase_name="create_legacy_mssql_instance"
-  remove_sentinel "${phase_name}" "done"
+function exists_legacy_mssql_instance() {
+  _check_exists gcloud compute instances describe "${MSSQL_INSTANCE}" --zone "${ZONE}" --project="${PROJECT_ID}" --format="json(name,status)"
+}
+export -f exists_legacy_mssql_instance
 
+function delete_legacy_mssql_instance() {
   print_status "Deleting Legacy MSSQL Instance ${MSSQL_INSTANCE}..."
-  if gcloud compute instances describe "${MSSQL_INSTANCE}" --zone "${ZONE}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    local log_file="delete_legacy_mssql_${MSSQL_INSTANCE}.log"
-    if run_gcloud "${log_file}" gcloud compute instances delete "${MSSQL_INSTANCE}" --zone "${ZONE}" --project="${PROJECT_ID}" --quiet; then
-      report_result "Deleted"
-    else
-      report_result "Fail"
-    fi
+  local log_file="delete_legacy_mssql_${MSSQL_INSTANCE}.log"
+  if run_gcloud "${log_file}" gcloud compute instances delete "${MSSQL_INSTANCE}" --zone "${ZONE}" --project="${PROJECT_ID}" --quiet; then
+    report_result "Deleted"
   else
-    report_result "Not Found"
+    report_result "Fail"
   fi
 }
 export -f delete_legacy_mssql_instance
 
 function create_mssql_instance() {
-  local phase_name="create_mssql_instance"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating MSSQL Instance ${MSSQL_INSTANCE}..."
-    report_result "Exists"
-    return 0
-  fi
-
   print_status "Creating MSSQL Instance ${MSSQL_INSTANCE}..."
-  if gcloud sql instances describe "${MSSQL_INSTANCE}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
+  local log_file="create_mssql_${MSSQL_INSTANCE}.log"
+  if run_gcloud "${log_file}" gcloud sql instances create "${MSSQL_INSTANCE}" \
+    --no-assign-ip \
+    --project="${PROJECT_ID}" \
+    --network="${NETWORK_URI_PARTIAL}" \
+    --database-version="${MSSQL_DATABASE_VERSION}" \
+    --activation-policy=ALWAYS \
+    --zone "${ZONE}"; then
+    report_result "Created"
   else
-    local log_file="create_mssql_${MSSQL_INSTANCE}.log"
-    if run_gcloud "${log_file}" gcloud sql instances create "${MSSQL_INSTANCE}" \
-      --no-assign-ip \
-      --project="${PROJECT_ID}" \
-      --network="${NETWORK_URI_PARTIAL}" \
-      --database-version="${MSSQL_DATABASE_VERSION}" \
-      --activation-policy=ALWAYS \
-      --zone "${ZONE}"; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
+    report_result "Fail"
+    return 1
   fi
 }
 export -f create_mssql_instance
 
 function delete_mssql_instance() {
-  local phase_name="create_mssql_instance"
-  remove_sentinel "${phase_name}" "done"
-
   print_status "Deleting MSSQL Instance ${MSSQL_INSTANCE}..."
-  if gcloud sql instances describe "${MSSQL_INSTANCE}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    local log_file="delete_mssql_${MSSQL_INSTANCE}.log"
-    if run_gcloud "${log_file}" gcloud sql instances delete --quiet "${MSSQL_INSTANCE}" --project="${PROJECT_ID}"; then
-      report_result "Deleted"
-    else
-      report_result "Fail"
-    fi
+  local log_file="delete_mssql_${MSSQL_INSTANCE}.log"
+  if run_gcloud "${log_file}" gcloud sql instances delete --quiet "${MSSQL_INSTANCE}" --project="${PROJECT_ID}"; then
+    report_result "Deleted"
   else
-    report_result "Not Found"
+    report_result "Fail"
   fi
 }
 export -f delete_mssql_instance
diff --git a/gcloud/lib/database/mysql.sh b/gcloud/lib/database/mysql.sh
index 76202d6e..01ea79bd 100644
--- a/gcloud/lib/database/mysql.sh
+++ b/gcloud/lib/database/mysql.sh
@@ -3,50 +3,35 @@
 # MySQL Cloud SQL functions
 
 function create_mysql_instance() {
-  local phase_name="create_mysql_instance"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating MySQL Instance ${MYSQL_INSTANCE}..."
-    report_result "Exists"
-    return 0
-  fi
-
   print_status "Creating MySQL Instance ${MYSQL_INSTANCE}..."
-  if gcloud sql instances describe "${MYSQL_INSTANCE}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
+  local log_file="create_mysql_${MYSQL_INSTANCE}.log"
+  if run_gcloud "${log_file}" gcloud sql instances create "${MYSQL_INSTANCE}" \
+    --no-assign-ip \
+    --project="${PROJECT_ID}" \
+    --network="${NETWORK_URI_PARTIAL}" \
+    --database-version="${MYSQL_DATABASE_VERSION}" \
+    --activation-policy=ALWAYS \
+    --zone "${ZONE}"; then
+    report_result "Created"
   else
-    local log_file="create_mysql_${MYSQL_INSTANCE}.log"
-    if run_gcloud "${log_file}" gcloud sql instances create "${MYSQL_INSTANCE}" \
-      --no-assign-ip \
-      --project="${PROJECT_ID}" \
-      --network="${NETWORK_URI_PARTIAL}" \
-      --database-version="${MYSQL_DATABASE_VERSION}" \
-      --activation-policy=ALWAYS \
-      --zone "${ZONE}"; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
+    report_result "Fail"
+    return 1
   fi
 }
 export -f create_mysql_instance
 
 function delete_mysql_instance() {
-  local phase_name="create_mysql_instance"
-  remove_sentinel "${phase_name}" "done"
-
   print_status "Deleting MySQL Instance ${MYSQL_INSTANCE}..."
-  if gcloud sql instances describe "${MYSQL_INSTANCE}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    local log_file="delete_mysql_${MYSQL_INSTANCE}.log"
-    if run_gcloud "${log_file}" gcloud sql instances delete --quiet "${MYSQL_INSTANCE}" --project="${PROJECT_ID}"; then
-      report_result "Deleted"
-    else
-      report_result "Fail"
-    fi
+  local log_file="delete_mysql_${MYSQL_INSTANCE}.log"
+  if run_gcloud "${log_file}" gcloud sql instances delete --quiet "${MYSQL_INSTANCE}" --project="${PROJECT_ID}"; then
+    report_result "Deleted"
   else
-    report_result "Not Found"
+    report_result "Fail"
   fi
 }
 export -f delete_mysql_instance
+
+function exists_mysql_instance() {
+  _check_exists gcloud sql instances describe "${MYSQL_INSTANCE}" --project="${PROJECT_ID}" --format="json(name,state)"
+}
+export -f exists_mysql_instance
diff --git a/gcloud/lib/database/oracle.sh b/gcloud/lib/database/oracle.sh
new file mode 100644
index 00000000..0065033e
--- /dev/null
+++ b/gcloud/lib/database/oracle.sh
@@ -0,0 +1,130 @@
+#!/bin/bash
+#
+# Oracle DB on GCE VM functions
+
+function create_oracle_vm() {
+  local phase_name="create_oracle_vm"
+    print_status "Checking Oracle VM ${ORACLE_VM_NAME}..."
+    report_result "Exists"
+    return 0
+
+  print_status "Creating Oracle VM ${ORACLE_VM_NAME}..."
+  local log_file="create_oracle_vm_${ORACLE_VM_NAME}.log"
+
+  local ova_url="https://yum.oracle.com/templates/OracleLinux/OL10/u0/x86_64/OL10U0_x86_64-olvm-b266.ova"
+  local ova_file="OL10U0_x86_64-olvm-b266.ova"
+  local machine_image_name="oracle-linux-10u0"
+  local os_type="oraclelinux-10"
+  local gcs_uri="gs://${BUCKET}/${ova_file}"
+  local startup_script="${REPRO_TMPDIR}/oracle_startup.sh"
+
+  # Create startup script content
+  cat << 'EOF' > "${startup_script}"
+#!/bin/bash
+# oracle_startup.sh
+echo "Starting Oracle DB setup..."
+
+# !!! IMPORTANT: These commands are placeholders. You need to add the actual
+# Oracle Database installation steps here for your specific version. !!!
+
+# 1. Add Oracle Yum Repos if needed
+# Example for 19c:
+# sudo yum install -y oracle-database-preinstall-19c
+
+# 2. Download Oracle Database Software (e.g., from Oracle website or GCS)
+# Example: wget ... or gsutil cp ...
+
+# 3. Unzip and Run Oracle Installer
+# Example: unzip ...
+# Example: ./runInstaller -silent -responseFile ...
+
+# 4. Configure Listener
+# Example: Using netca
+
+# 5. Create Database
+# Example: Using dbca
+
+# 6. Open Firewall Port
+sudo firewall-cmd --zone=public --add-port=1521/tcp --permanent
+sudo firewall-cmd --reload
+
+echo "Oracle DB setup script finished."
+EOF
+
+  print_status "  Downloading OVA..."
+  if wget -O "${REPRO_TMPDIR}/${ova_file}" "${ova_url}" > "${REPRO_TMPDIR}/wget_${ova_file}.log" 2>&1; then
+    report_result "Pass"
+  else
+    report_result "Fail"
+    return 1
+  fi
+
+  print_status "  Uploading OVA to GCS..."
+  if run_gcloud "${log_file}" gsutil cp "${REPRO_TMPDIR}/${ova_file}" "${gcs_uri}"; then
+    report_result "Pass"
+  else
+    report_result "Fail"
+    return 1
+  fi
+
+  print_status "  Importing Machine Image ${machine_image_name}..."
+  if run_gcloud "${log_file}" gcloud compute machine-images import "${machine_image_name}" \
+    --project="${PROJECT_ID}" \
+    --source-uri="${gcs_uri}" \
+    --os="${os_type}" \
+    --zone="${ZONE}"; then
+    report_result "Imported"
+  else
+    # Continue if image already exists
+    if gcloud compute machine-images describe "${machine_image_name}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
+      report_result "Exists"
+    else
+      report_result "Fail"
+      return 1
+    fi
+  fi
+
+  print_status "  Creating VM instance ${ORACLE_VM_NAME}..."
+  if run_gcloud "${log_file}" gcloud compute instances create "${ORACLE_VM_NAME}" \
+    --project="${PROJECT_ID}" \
+    --zone="${ZONE}" \
+    --source-machine-image="${machine_image_name}" \
+    --metadata-from-file=startup-script="${startup_script}" \
+    --scopes=cloud-platform; then
+    report_result "Created"
+  else
+    report_result "Fail"
+    return 1
+  fi
+
+  print_status "  Cleaning up local OVA file..."
+  rm "${REPRO_TMPDIR}/${ova_file}"
+  report_result "Done"
+}
+export -f create_oracle_vm
+
+function delete_oracle_vm() {
+  local phase_name="create_oracle_vm"
+  print_status "Deleting Oracle VM ${ORACLE_VM_NAME}..."
+  local log_file="delete_oracle_vm_${ORACLE_VM_NAME}.log"
+
+  if gcloud compute instances describe "${ORACLE_VM_NAME}" --zone "${ZONE}" --project "${PROJECT_ID}" > /dev/null 2>&1; then
+    if run_gcloud "${log_file}" gcloud compute instances delete "${ORACLE_VM_NAME}" \
+      --project="${PROJECT_ID}" \
+      --zone="${ZONE}" \
+      --quiet; then
+      report_result "Deleted"
+    else
+      report_result "Fail"
+      return 1
+    fi
+  else
+    report_result "Not Found"
+  fi
+}
+export -f delete_oracle_vm
+
+function exists_oracle_vm() {
+  _check_exists gcloud compute instances describe "${ORACLE_VM_NAME}" --zone "${ZONE}" --project="${PROJECT_ID}" --format="json(name,status)"
+}
+export -f exists_oracle_vm
diff --git a/gcloud/lib/database/pgsql.sh b/gcloud/lib/database/pgsql.sh
index f1cdcb93..c0adfd53 100644
--- a/gcloud/lib/database/pgsql.sh
+++ b/gcloud/lib/database/pgsql.sh
@@ -3,51 +3,36 @@
 # PostgreSQL Cloud SQL functions
 
 function create_pgsql_instance() {
-  local phase_name="create_pgsql_instance"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating PostgreSQL Instance ${PGSQL_INSTANCE}..."
-    report_result "Exists"
-    return 0
-  fi
-
   print_status "Creating PostgreSQL Instance ${PGSQL_INSTANCE}..."
-  if gcloud sql instances describe "${PGSQL_INSTANCE}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
+  local log_file="create_pgsql_${PGSQL_INSTANCE}.log"
+  if run_gcloud "${log_file}" gcloud sql instances create "${PGSQL_INSTANCE}" \
+    --no-assign-ip \
+    --project="${PROJECT_ID}" \
+    --network="${NETWORK_URI_PARTIAL}" \
+    --database-version="${PGSQL_DATABASE_VERSION}" \
+    --activation-policy=ALWAYS \
+    --root-password="${PGSQL_ROOT_PASSWORD}" \
+    --zone "${ZONE}"; then
+    report_result "Created"
   else
-    local log_file="create_pgsql_${PGSQL_INSTANCE}.log"
-    if run_gcloud "${log_file}" gcloud sql instances create "${PGSQL_INSTANCE}" \
-      --no-assign-ip \
-      --project="${PROJECT_ID}" \
-      --network="${NETWORK_URI_PARTIAL}" \
-      --database-version="${PGSQL_DATABASE_VERSION}" \
-      --activation-policy=ALWAYS \
-      --root-password="${PGSQL_ROOT_PASSWORD}" \
-      --zone "${ZONE}"; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
+    report_result "Fail"
+    return 1
   fi
 }
 export -f create_pgsql_instance
 
 function delete_pgsql_instance() {
-  local phase_name="create_pgsql_instance"
-  remove_sentinel "${phase_name}" "done"
-
   print_status "Deleting PostgreSQL Instance ${PGSQL_INSTANCE}..."
-  if gcloud sql instances describe "${PGSQL_INSTANCE}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    local log_file="delete_pgsql_${PGSQL_INSTANCE}.log"
-    if run_gcloud "${log_file}" gcloud sql instances delete --quiet "${PGSQL_INSTANCE}" --project="${PROJECT_ID}"; then
-      report_result "Deleted"
-    else
-      report_result "Fail"
-    fi
+  local log_file="delete_pgsql_${PGSQL_INSTANCE}.log"
+  if run_gcloud "${log_file}" gcloud sql instances delete --quiet "${PGSQL_INSTANCE}" --project="${PROJECT_ID}"; then
+    report_result "Deleted"
   else
-    report_result "Not Found"
+    report_result "Fail"
   fi
 }
 export -f delete_pgsql_instance
+
+function exists_pgsql_instance() {
+  _check_exists gcloud sql instances describe "${PGSQL_INSTANCE}" --project="${PROJECT_ID}" --format="json(name,state)"
+}
+export -f exists_pgsql_instance
diff --git a/gcloud/lib/dataproc/autoscaling.sh b/gcloud/lib/dataproc/autoscaling.sh
index fee77024..a84cd666 100644
--- a/gcloud/lib/dataproc/autoscaling.sh
+++ b/gcloud/lib/dataproc/autoscaling.sh
@@ -2,45 +2,32 @@
 #
 # Dataproc Autoscaling Policy functions
 
-function create_autoscaling_policy() {
-  local phase_name="create_autoscaling_policy"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating Autoscaling Policy ${AUTOSCALING_POLICY_NAME}..."
-    report_result "Exists"
-    return 0
-  fi
+function exists_autoscaling_policy() {
+  _check_exists gcloud dataproc autoscaling-policies describe "${AUTOSCALING_POLICY_NAME}" --region="${REGION}" --format="json(id,name)"
+}
+export -f exists_autoscaling_policy
 
+function create_autoscaling_policy() {
   print_status "Creating Autoscaling Policy ${AUTOSCALING_POLICY_NAME}..."
   local log_file="create_autoscaling_${AUTOSCALING_POLICY_NAME}.log"
-  if gcloud dataproc autoscaling-policies describe ${AUTOSCALING_POLICY_NAME} --region ${REGION} > /dev/null 2>&1; then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
+  if run_gcloud "${log_file}" gcloud dataproc autoscaling-policies import "${AUTOSCALING_POLICY_NAME}" --region="${REGION}" --source="${GCLOUD_DIR}/autoscaling-policy.yaml" --quiet; then
+    report_result "Created"
+    refresh_resource_state "autoscalingPolicy" "lib/dataproc/autoscaling.sh" exists_autoscaling_policy
   else
-    if run_gcloud "${log_file}" gcloud dataproc autoscaling-policies import ${AUTOSCALING_POLICY_NAME} --region ${REGION} --source autoscaling-policy.yaml; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
+    report_result "Fail"
+    return 1
   fi
 }
 export -f create_autoscaling_policy
 
 function delete_autoscaling_policy() {
-  local phase_name="create_autoscaling_policy"
-  remove_sentinel "${phase_name}" "done"
-
   print_status "Deleting Autoscaling Policy ${AUTOSCALING_POLICY_NAME}..."
-  if gcloud dataproc autoscaling-policies describe ${AUTOSCALING_POLICY_NAME} --region ${REGION} > /dev/null 2>&1; then
-    local log_file="delete_autoscaling_${AUTOSCALING_POLICY_NAME}.log"
-    if run_gcloud "${log_file}" gcloud dataproc autoscaling-policies delete --quiet ${AUTOSCALING_POLICY_NAME} --region ${REGION}; then
-      report_result "Deleted"
-    else
-      report_result "Fail"
-    fi
+  local log_file="delete_autoscaling_${AUTOSCALING_POLICY_NAME}.log"
+  if run_gcloud "${log_file}" gcloud dataproc autoscaling-policies delete --quiet "${AUTOSCALING_POLICY_NAME}" --region="${REGION}"; then
+    report_result "Deleted"
+    update_state "autoscalingPolicy" "null"
   else
-    report_result "Not Found"
+    report_result "Fail"
   fi
 }
-export -f delete_autoscaling_policy
\ No newline at end of file
+export -f delete_autoscaling_policy
diff --git a/gcloud/lib/dataproc/cluster.sh b/gcloud/lib/dataproc/cluster.sh
index a0953dd2..6a351377 100644
--- a/gcloud/lib/dataproc/cluster.sh
+++ b/gcloud/lib/dataproc/cluster.sh
@@ -3,90 +3,143 @@
 # Dataproc Cluster Management Functions
 
 function exists_dpgce_cluster() {
-  # print_status "  Checking if cluster ${CLUSTER_NAME} exists..."
-  if gcloud dataproc clusters describe "${CLUSTER_NAME}" --region="${REGION}" --project="${PROJECT_ID}" > /dev/null 2>&1;
- then
-    # report_result "Exists"
-    return 0 # Found
-  else
-    # report_result "Not Found"
-    return 1 # Not found
-  fi
+  _check_exists gcloud dataproc clusters describe "${CLUSTER_NAME}" --region="${REGION}" --project="${PROJECT_ID}" --format="json(clusterName,clusterUuid,status.selfLink,config.softwareConfig.imageVersion,config.masterConfig.imageUri)"
 }
 export -f exists_dpgce_cluster
 
 function create_dpgce_cluster() {
-  local phase_name="create_dpgce_cluster"
-  # Note: No sentinel check here because this is the main resource being created/recreated.
-  # We rely on the --no-create-cluster flag to skip this if needed.
-
   print_status "Creating Dataproc Cluster ${CLUSTER_NAME}..."
 
+  local metadata_array=(
+    "public_secret_name=${public_secret_name}"
+    "private_secret_name=${private_secret_name}"
+    "secret_project=${secret_project}"
+    "secret_version=${secret_version}"
+    "modulus_md5sum=${modulus_md5sum}"
+    "init-actions-repo=${INIT_ACTIONS_ROOT}"
+    "debug=true"
+    "enable-oslogin=TRUE"
+    "universe-domain=${UNIVERSE_DOMAIN}"
+    "startup-script=gcloud config set core/universe_domain '${UNIVERSE_DOMAIN}'"
+    "startup-script-url=${GCE_PROXY_SETUP_URI}"
+  )
+
+  if [[ "${ENABLE_GPU}" == "true" ]]; then
+    metadata_array+=(
+      "install-gpu-agent=true"
+      "gpu-driver-provider=NVIDIA"
+      "gpu-conda-env=dpgce"
+      "include-pytorch=yes"
+      "include-tensorflow=yes"
+      "include-rapids=yes"
+      "dask-runtime=standalone"
+      "rapids-runtime=SPARK"
+      "bigtable-instance=${BIGTABLE_INSTANCE}"
+      "include-gpus=1"
+    )
+    if [[ -n "${CUDA_URL}" && "${CUDA_URL}" != "null" ]] ; then
+      metadata_array+=( "cuda-url=${CUDA_URL}" )
+    elif [[ -n "${CUDA_VERSION}" && "${CUDA_VERSION}" != "null" ]] ; then
+      metadata_array+=( "cuda-version=${CUDA_VERSION}" )
+    fi
+    if [[ -n "${GPU_DRIVER_URL}" && "${GPU_DRIVER_URL}" != "null" ]] ; then
+      metadata_array+=( "gpu-driver-url=${GPU_DRIVER_URL}" )
+    elif [[ -n "${DRIVER_VERSION}" && "${DRIVER_VERSION}" != "null" ]] ; then
+      metadata_array+=( "gpu-driver-version=${DRIVER_VERSION}" )
+    fi
+  fi
+
+  if [[ "${SWP_EGRESS}" == "true" ]]; then
+    metadata_array+=(
+      "http-proxy=${SWP_IP}:${SWP_PORT}"
+      "https-proxy=${SWP_IP}:${SWP_PORT}"
+      "proxy-uri=${SWP_IP}:${SWP_PORT}"
+      "no-proxy=metadata.google.internal,${PROJECT_ID}.svc.id.goog"
+    )
+  fi
+
+  local all_metadata
+  all_metadata="$(IFS='|'; echo "${metadata_array[*]}")"
+  all_metadata="^|^${all_metadata}"
+
+  echo "DEBUG: all_metadata string: ${all_metadata}" >&2
+
   local gcloud_cmd=(
     gcloud dataproc clusters create "${CLUSTER_NAME}"
     --single-node
-    --master-accelerator "type=${MASTER_ACCELERATOR_TYPE}"
-    --worker-accelerator "type=${PRIMARY_ACCELERATOR_TYPE}"
-    --secondary-worker-accelerator "type=${SECONDARY_ACCELERATOR_TYPE}"
-    --master-machine-type "${MASTER_MACHINE_TYPE}"
-    --worker-machine-type "${PRIMARY_MACHINE_TYPE}"
-    --master-boot-disk-size 60
-    --worker-boot-disk-size 60
-    --secondary-worker-boot-disk-size 60
+    --master-machine-type "${M_MACHINE_TYPE}"
+    --master-boot-disk-size 600
+    --master-local-ssd-interface=NVME
+    --num-master-local-ssds=1
     --master-boot-disk-type pd-ssd
-    --worker-boot-disk-type pd-ssd
-    --secondary-worker-boot-disk-type pd-ssd
     --region "${REGION}"
     --zone "${ZONE}"
     --subnet "${SUBNET}"
     --no-address
-    --service-account="${GSA}"
-    --tags="${TAGS}"
+    --service-account "${GSA}"
+    --tags "${TAGS}"
     --bucket "${BUCKET}"
     --temp-bucket "${TEMP_BUCKET}"
     --enable-component-gateway
-    --metadata "public_secret_name=${public_secret_name}"
-    --metadata "private_secret_name=${private_secret_name}"
-    --metadata "secret_project=${secret_project}"
-    --metadata "secret_version=${secret_version}"
-    --metadata "modulus_md5sum=${modulus_md5sum}"
-    --metadata "install-gpu-agent=true"
-    --metadata "gpu-driver-provider=NVIDIA"
-    --metadata "gpu-conda-env=dpgce"
-    --metadata "rapids-mirror-disk=${RAPIDS_MIRROR_DISK_NAME}"
-    --metadata "rapids-mirror-host=${RAPIDS_REGIONAL_MIRROR_ADDR[${REGION}]}"
-    --metadata "init-actions-repo=${INIT_ACTIONS_ROOT}"
-    --metadata "dask-cloud-logging=true"
-    --metadata "debug=true"
-    --metadata "http-proxy=${SWP_IP}:${SWP_PORT}"
-    --metadata dask-runtime="standalone"
-    --metadata rapids-runtime="SPARK"
-    --metadata bigtable-instance=${BIGTABLE_INSTANCE}
-    --metadata include-gpus=1
-    --image "projects/${PROJECT_ID}/global/images/dataproc-2-2-deb12-20251108-180659-tf" \
-    --initialization-action-timeout=90m
-    --optional-components DOCKER,JUPYTER
-    --max-idle="${IDLE_TIMEOUT}"
+    --metadata "${all_metadata}"
+    # NO --image or --image-version here
+    --initialization-action-timeout 90m
+#    --optional-components "DOCKER,JUPYTER"
     --properties "spark:spark.history.fs.logDirectory=gs://${BUCKET}/phs/eventLog"
     --scopes 'https://www.googleapis.com/auth/cloud-platform,sql-admin'
   )
 
-#    --no-shielded-secure-boot
-#    --image-version "${IMAGE_VERSION}"
-#    --initialization-actions ${INIT_ACTIONS_ROOT}/gpu/install_gpu_driver.sh
-#    --metadata=startup-script-url="gs://dataproc-staging-us-west4-kf7bmp/dataproc-initialization-actions/gce-proxy-setup.sh"
+  if [[ "${ENABLE_GPU}" == "true" ]]; then
+    gcloud_cmd+=(--master-accelerator "type=${M_ACCELERATOR_TYPE}")
+  fi
+
+  if [[ "${IS_CUSTOM}" == "true" ]]; then
+    if [[ -z "${CUSTOM_IMAGE_URI}" || "${CUSTOM_IMAGE_URI}" == "null" ]]; then
+      echo "ERROR: --custom flag is set but CUSTOM_IMAGE_URI is not defined in env.json" >&2
+      exit 1
+    fi
+    gcloud_cmd+=(--image "${CUSTOM_IMAGE_URI}")
+    gcloud_cmd+=(--shielded-secure-boot)
+    echo "INFO: Using Custom Image URI: ${CUSTOM_IMAGE_URI} with Secure Boot enabled."
+  else
+    gcloud_cmd+=(--no-shielded-secure-boot)
+    gcloud_cmd+=(--image-version "${IMAGE_VERSION}")
+    if [[ "${ENABLE_GPU}" == "true" ]]; then
+      echo "INFO: Using Image Version: ${IMAGE_VERSION}. GPU Init actions enabled from default repo."
+    else
+      echo "INFO: Using Image Version: ${IMAGE_VERSION}. No additional init actions."
+    fi
+  fi
 
+  # Add init actions if any are specified
+  INIT_ACTIONS_LIST=()
+  if [[ "${SKIP_INIT_ACTIONS:-}" != "true" ]]; then
+    if [[ "${ENABLE_GPU}" == "true" ]] && [[ -n "${GPU_INIT_ACTION_URI:-}" ]]; then
+      INIT_ACTIONS_LIST+=("${GPU_INIT_ACTION_URI}")
+      echo "INFO: Using custom GPU init action: ${GPU_INIT_ACTION_URI}" >&2
+    elif [[ "${ENABLE_GPU}" == "true" ]]; then
+      INIT_ACTIONS_LIST+=("${INIT_ACTIONS_ROOT}/gpu/install_gpu_driver.sh")
+      echo "INFO: Using default GPU init action: ${INIT_ACTIONS_ROOT}/gpu/install_gpu_driver.sh" >&2
+    fi
+  else
+    echo "INFO: Skipping initialization actions because SKIP_INIT_ACTIONS is set to true." >&2
+  fi
+
+  if [[ ${#INIT_ACTIONS_LIST[@]} -gt 0 ]]; then
+    gcloud_cmd+=(--initialization-actions $(IFS=,; echo "${INIT_ACTIONS_LIST[*]}"))
+  fi
 
   if [[ "${GCLOUD_QUIET}" != "true" ]]; then
     echo
     echo "Command to be executed:"
     cmd_str=$(printf "%s " "${gcloud_cmd[@]}")
-    # Replace " --" with " \\\n  --" for pretty printing
-    echo "${cmd_str}" | perl -pe 's/ --/ \\\n  --/g'
+    echo "${cmd_str}" | perl -pe 's/ --/ 
+  --/g'
   fi
 
-  if "${gcloud_cmd[@]}"; then
+  if time "${gcloud_cmd[@]}"; then
     report_result "Created"
+    refresh_resource_state "dataprocCluster" "lib/dataproc/cluster.sh" exists_dpgce_cluster
   else
     report_result "Fail"
     return 1
@@ -96,16 +149,16 @@ export -f create_dpgce_cluster
 
 function delete_dpgce_cluster() {
   print_status "Deleting Dataproc Cluster ${CLUSTER_NAME}..."
-  if exists_dpgce_cluster;
- then
-    local log_file="delete_dpgce_cluster_${CLUSTER_NAME}.log"
-    if run_gcloud "${log_file}" gcloud dataproc clusters delete --quiet --region ${REGION} ${CLUSTER_NAME}; then
-      report_result "Deleted"
-    else
-      report_result "Fail"
-    fi
+  local log_file="delete_dpgce_cluster_${CLUSTER_NAME}.log"
+  if run_gcloud "${log_file}" gcloud dataproc clusters delete --quiet "${CLUSTER_NAME}" --region "${REGION}"; then
+    report_result "Deleted"
   else
-    report_result "Not Found"
+    report_result "Fail"
   fi
 }
 export -f delete_dpgce_cluster
+
+function exists_dataproc_cluster_vms() {
+  _check_exists gcloud compute instances list --project="${PROJECT_ID}" --filter="labels.goog-dataproc-cluster-name=${CLUSTER_NAME}" --format="json(name,zone,status)"
+}
+export -f exists_dataproc_cluster_vms
diff --git a/gcloud/lib/env.sh b/gcloud/lib/env.sh
index a4fe4d5d..b8bc7303 100644
--- a/gcloud/lib/env.sh
+++ b/gcloud/lib/env.sh
@@ -14,43 +14,80 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
-# Set RESOURCE_SUFFIX based on TIMESTAMP env var or generate new
-if [[ -n "${TIMESTAMP}" ]]; then
-  export RESOURCE_SUFFIX="${TIMESTAMP}"
-  echo "Using provided TIMESTAMP for resources: ${RESOURCE_SUFFIX}"
-else
-  export RESOURCE_SUFFIX="$(date +%s)"
-  echo "Generated new TIMESTAMP for resources: ${RESOURCE_SUFFIX}"
+# --- Get lib's real directory ---
+LIB_DIR=$(realpath "$(dirname "${BASH_SOURCE[0]}")")
+GCLOUD_DIR="$(realpath "${LIB_DIR}/..")"
+export GCLOUD_DIR
+
+export STATE_DIR="${GCLOUD_DIR}/.state"
+mkdir -p "${STATE_DIR}"
+export STATE_DB="${STATE_DIR}/state.db"
+
+source "${GCLOUD_DIR}/lib/script-utils.sh"
+init_state_db
+
+# Load or generate persistent TIMESTAMP
+if [[ -z "${TIMESTAMP:-}" ]]; then
+  # Try to load from state database
+  PERSISTENT_TIMESTAMP=$(get_state "config.timestamp")
+  if [[ "${PERSISTENT_TIMESTAMP}" != "null" && -n "${PERSISTENT_TIMESTAMP}" ]]; then
+    export TIMESTAMP="${PERSISTENT_TIMESTAMP}"
+    # echo "INFO: Loaded persistent TIMESTAMP: ${TIMESTAMP}" >&2
+  else
+    # Fallback to current time and save it
+    export TIMESTAMP=$(date +%s)
+    update_state "config.timestamp" "${TIMESTAMP}"
+    # echo "INFO: Generated new persistent TIMESTAMP: ${TIMESTAMP}" >&2
+  fi
 fi
-export REPRO_TMPDIR="${REPRO_TMPDIR:-/tmp/dataproc-repro/${RESOURCE_SUFFIX}}"
-mkdir -p "${REPRO_TMPDIR}"
-export SENTINEL_DIR="${SENTINEL_DIR:-${REPRO_TMPDIR}/sentinels}"
-mkdir -p "${SENTINEL_DIR}"
 
-source lib/script-utils.sh
+# Ensure paths are based on the correct persistent TIMESTAMP
+export REPRO_TMPDIR="/tmp/dataproc-repro/${TIMESTAMP}"
+mkdir -p "${REPRO_TMPDIR}"
+export LOG_DIR="${REPRO_TMPDIR}/logs"
+mkdir -p "${LOG_DIR}"
 
 export PATH_SEPARATOR=";"
-export FOLDER_NUMBER="$(jq -r .FOLDER_NUMBER env.json)"
-export DOMAIN="$(jq -r .DOMAIN env.json)"
-export USER="$(jq -r .USER env.json)"
-export PRIV_DOMAIN="$(jq -r .PRIV_DOMAIN env.json)"
-export PRIV_USER="$(jq -r .PRIV_USER env.json)"
-export PROJECT_ID="$(jq -r .PROJECT_ID env.json)"
+export FOLDER_NUMBER="$(jq -r .FOLDER_NUMBER "${GCLOUD_DIR}/env.json")"
+export DOMAIN="$(jq -r .DOMAIN "${GCLOUD_DIR}/env.json")"
+export USER="$(jq -r .USER "${GCLOUD_DIR}/env.json")"
+export PRIV_DOMAIN="$(jq -r .PRIV_DOMAIN "${GCLOUD_DIR}/env.json")"
+export PRIV_USER="$(jq -r .PRIV_USER "${GCLOUD_DIR}/env.json")"
+export PROJECT_ID="$(jq -r .PROJECT_ID "${GCLOUD_DIR}/env.json")"
 if [[ "${PROJECT_ID}" == "ldap-example-yyyy-nn" ]]; then
   export PROJECT_ID="${USER}-example-$(date +%Y-%U)"
 fi
-export BILLING_ACCOUNT="$(jq -r .BILLING_ACCOUNT env.json)"
-export CLUSTER_NAME="$(jq -r .CLUSTER_NAME env.json)"
-export BUCKET="$(jq -r .BUCKET env.json)"
-export TEMP_BUCKET="$(jq -r .TEMP_BUCKET env.json)"
-export RANGE="$(jq -r .RANGE env.json)"
-export IDLE_TIMEOUT="$(jq -r .IDLE_TIMEOUT env.json)"
-export ASN_NUMBER="$(jq -r .ASN_NUMBER env.json)"
-export IMAGE_VERSION="$(jq -r .IMAGE_VERSION env.json)"
-export REGION="$(jq -r .REGION env.json)"
-export DEBUG="${DEBUG:-0}"
+export BILLING_ACCOUNT="$(jq -r .BILLING_ACCOUNT "${GCLOUD_DIR}/env.json")"
+export CLUSTER_NAME="$(jq -r .CLUSTER_NAME "${GCLOUD_DIR}/env.json")"
+export BUCKET="$(jq -r .BUCKET "${GCLOUD_DIR}/env.json")"
+export TEMP_BUCKET="$(jq -r .TEMP_BUCKET "${GCLOUD_DIR}/env.json")"
+export RANGE="$(jq -r .RANGE "${GCLOUD_DIR}/env.json")"
+export SWP_RANGE="$(jq -r .SWP_RANGE "${GCLOUD_DIR}/env.json")"
+export SWP_SUBNET="swp-subnet-${CLUSTER_NAME}"
+export IDLE_TIMEOUT="$(jq -r .IDLE_TIMEOUT "${GCLOUD_DIR}/env.json")"
+export ASN_NUMBER="$(jq -r .ASN_NUMBER "${GCLOUD_DIR}/env.json")"
+if [[ "${ASN_NUMBER}" == "null" ]]; then
+  export ASN_NUMBER="65531"
+fi
+export IMAGE_VERSION="$(jq -r .IMAGE_VERSION "${GCLOUD_DIR}/env.json")"
+export REGION="$(jq -r .REGION "${GCLOUD_DIR}/env.json")"
+export ZONE="$(jq -r .ZONE "${GCLOUD_DIR}/env.json")"
+export REGION ZONE
+export SWP_IP="$(jq -r .SWP_IP "${GCLOUD_DIR}/env.json")"
+export SWP_PORT="$(jq -r .SWP_PORT "${GCLOUD_DIR}/env.json")"
+export SWP_HOSTNAME="$(jq -r .SWP_HOSTNAME "${GCLOUD_DIR}/env.json")"
+export UNIVERSE_DOMAIN="$(jq -r .UNIVERSE_DOMAIN "${GCLOUD_DIR}/env.json")"
+if [[ "${UNIVERSE_DOMAIN}" == "null" ]]; then
+  export UNIVERSE_DOMAIN="googleapis.com"
+fi
 
-export ZONE="${REGION}-b"
+# Manual overrides for testing specific driver/CUDA packages during cluster creation
+# export CUDA_URL="$(jq -r .CUDA_URL "${GCLOUD_DIR}/env.json")"
+# export GPU_DRIVER_URL="$(jq -r .GPU_DRIVER_URL "${GCLOUD_DIR}/env.json")"
+export SWP_POLICY_NAME="swp-policy-${CLUSTER_NAME}"
+export SWP_INSTANCE_NAME="swp-gateway-${CLUSTER_NAME}"
+export SWP_CERT_NAME="swp-cert-${CLUSTER_NAME}-${RESOURCE_SUFFIX}"
+export DEBUG="${DEBUG:-0}"
 #export ZONE="${REGION}-b"
 #export IMAGE_VERSION="2.0"
 #export IMAGE_VERSION="2.0.67-debian10" # final proprietary gpu support - April 26, 2024 - 5.10.0-0.deb10.16-amd64
@@ -86,6 +123,7 @@ else
 fi
 
 export INIT_ACTIONS_ROOT="gs://${BUCKET}/dataproc-initialization-actions"
+export GCE_PROXY_SETUP_URI="${INIT_ACTIONS_ROOT}/gce-proxy-setup.sh"
 export YARN_DOCKER_IMAGE="gcr.io/${PROJECT_ID}/${USER}/cudatest-ubuntu18:latest"
 export SPARK_PROPERTIES="spark:spark.yarn.unmanagedAM.enabled=false,spark:spark.task.resource.gpu.amount=1,spark:spark.executor.cores=1,spark:spark.task.cpus=1,spark:spark.executor.memory=4G"
 export DOCKER_PROPERTIES="dataproc:docker.yarn.enable=true"
@@ -109,15 +147,15 @@ export PRINCIPAL="${USER}@${DOMAIN}"
 export ARTIFACT_REPOSITORY="${PROJECT_ID}-dataproc-repro"
 
 # BigTable
-export BIGTABLE_INSTANCE="$(jq -r .BIGTABLE_INSTANCE env.json)"
+export BIGTABLE_INSTANCE="$(jq -r .BIGTABLE_INSTANCE "${GCLOUD_DIR}/env.json")"
 if [[ "${BIGTABLE_INSTANCE}" == "null" ]]; then
   BIGTABLE_INSTANCE="${USER}-bigtable0"
 fi
-export BIGTABLE_DISPLAY_NAME="$(jq -r .BIGTABLE_DISPLAY_NAME env.json)"
+export BIGTABLE_DISPLAY_NAME="$(jq -r .BIGTABLE_DISPLAY_NAME "${GCLOUD_DIR}/env.json")"
 if [[ "${BIGTABLE_DISPLAY_NAME}" == "null" ]]; then
   BIGTABLE_DISPLAY_NAME="bigtable-${CLUSTER_NAME}"
 fi
-export BIGTABLE_CLUSTER_CONFIG="$(jq -r .BIGTABLE_CLUSTER_CONFIG env.json)"
+export BIGTABLE_CLUSTER_CONFIG="$(jq -r .BIGTABLE_CLUSTER_CONFIG "${GCLOUD_DIR}/env.json")"
 if [[ "${BIGTABLE_CLUSTER_CONFIG}" == "null" ]]; then
   BIGTABLE_CLUSTER_CONFIG="id=${BIGTABLE_DISPLAY_NAME},zone=${ZONE},nodes=3"
 fi
@@ -151,7 +189,12 @@ export MR_HISTORY_BUCKET="${BUCKET}/*/mapreduce-job-history/done"
 #export MACHINE_TYPE="n1-highmem-8"
 #export MACHINE_TYPE="n1-standard-16"
 export MACHINE_TYPE="n1-standard-32"
+#export MACHINE_TYPE="n1-standard-64"
 #export MACHINE_TYPE="n1-standard-96"
+#export MACHINE_TYPE="n1-highmem-32"
+#export MACHINE_TYPE="a4x-highgpu-4g"
+#export MACHINE_TYPE="a4x-maxgpu-4g-metal"
+#export MACHINE_TYPE="a4-highgpu-8g"
 #export MACHINE_TYPE="e2-standard-2"
 # g2- are for l4 GPUs
 #export MACHINE_TYPE="g2-standard-4"
@@ -161,15 +204,20 @@ export MACHINE_TYPE="n1-standard-32"
 #export MACHINE_TYPE="a2-highgpu-1g"
 #export MACHINE_TYPE="a2-highgpu-2g"
 # a3- are for h100 GPUs
+# export MACHINE_TYPE="g4-standard-48" # (1 GPU)
+# export MACHINE_TYPE="g4-standard-96" # (2 GPUs)
+# export MACHINE_TYPE="g4-standard-192" # (4 GPUs)
+# export MACHINE_TYPE="g4-standard-384" # (8 GPUs)
 #export MACHINE_TYPE="a3-highgpu-8g"
+#export MACHINE_TYPE="a3-highgpu-1g"
 #export MACHINE_TYPE="a3-highgpu-2g"
 #export MACHINE_TYPE="a3-highgpu-4g"
 #export MACHINE_TYPE="n2d-standard-8"
-export MASTER_MACHINE_TYPE="${MACHINE_TYPE}"
-#export MASTER_MACHINE_TYPE="n1-standard-96"
-#export MASTER_MACHINE_TYPE="n1-standard-8"
-#export MASTER_MACHINE_TYPE="a2-highgpu-8g"
-#export MASTER_MACHINE_TYPE="a3-highgpu-8g"
+export M_MACHINE_TYPE="${MACHINE_TYPE}"
+#export M_MACHINE_TYPE="n1-standard-96"
+#export M_MACHINE_TYPE="n1-standard-8"
+#export M_MACHINE_TYPE="a2-highgpu-8g"
+#export M_MACHINE_TYPE="a3-highgpu-8g"
 export PRIMARY_MACHINE_TYPE="${MACHINE_TYPE}"
 #export PRIMARY_MACHINE_TYPE="n1-standard-8"
 #export PRIMARY_MACHINE_TYPE="g2-standard-4"
@@ -177,64 +225,6 @@ export PRIMARY_MACHINE_TYPE="${MACHINE_TYPE}"
 #export PRIMARY_MACHINE_TYPE="n2d-highmem-32"
 export SECONDARY_MACHINE_TYPE="${PRIMARY_MACHINE_TYPE}"
 
-#export CUDNN_VERSION="8.0.5.39"
-#export NCCL_VERSION="2.8.4"
-#export DRIVER_VERSION="455.45.01"
-#export ACCELERATOR_TYPE="nvidia-tesla-p100"
-#export ACCELERATOR_TYPE="nvidia-tesla-a100"
-#export ACCELERATOR_TYPE="nvidia-tesla-a100,count=2"
-export ACCELERATOR_TYPE="nvidia-tesla-t4"
-#export ACCELERATOR_TYPE="nvidia-l4"
-#export ACCELERATOR_TYPE="nvidia-tesla-p4,count=2"
-#export ACCELERATOR_TYPE="nvidia-tesla-p100,count=2"
-#export ACCELERATOR_TYPE="nvidia-tesla-v100,count=4"
-#export ACCELERATOR_TYPE="nvidia-h100-80gb,count=4"
-#export ACCELERATOR_TYPE="nvidia-h100-80gb,count=2"
-#export MASTER_ACCELERATOR_TYPE="nvidia-tesla-t4,count=4"
-#export MASTER_ACCELERATOR_TYPE="nvidia-tesla-t4"
-#export MASTER_ACCELERATOR_TYPE="nvidia-tesla-a100,count=2"
-#export MASTER_ACCELERATOR_TYPE="nvidia-tesla-a100,count=8"
-#export MASTER_ACCELERATOR_TYPE="nvidia-h100-80gb,count=8"
-export MASTER_ACCELERATOR_TYPE="${ACCELERATOR_TYPE}"
-export PRIMARY_ACCELERATOR_TYPE="${ACCELERATOR_TYPE}"
-export SECONDARY_ACCELERATOR_TYPE="${ACCELERATOR_TYPE}"
-#export CUDA_VERSION=10.2.89
-#export CUDA_VERSION=11.0
-#export DRIVER_VERSION="440.100"
-#export CUDA_VERSION=11.1
-#export DRIVER_VERSION="455.45.01"
-#export CUDA_VERSION=11.2
-#export DRIVER_VERSION="460.91.03"
-#export CUDA_VERSION=11.3
-#export DRIVER_VERSION="465.31"
-#export CUDA_VERSION=11.5
-#export CUDA_VERSION="11.1"
-#export DRIVER_VERSION="455.45.01"
-#export DRIVER_VERSION="470.256.02"
-#export CUDA_VERSION=11.2
-#export CUDA_VERSION=11.7
-#export CUDA_VERSION=11.8
-#export CUDA_VERSION=12.0
-#export CUDA_VERSION=12.1.1
-#export CUDA_VERSION=12.2
-#export CUDA_VERSION=12.1.1
-#export CUDA_VERSION=12.4
-# export CUDA_VERSION=12.4.0
-#export CUDA_VERSION="12.4.1"
-#export DRIVER_VERSION="550.135"
-#export CUDA_VERSION="12.6"
-#export CUDA_VERSION="12.6.2"
-#export CUDA_VERSION="12.6.3"
-#export DRIVER_VERSION="550.142"
-#export DRIVER_VERSION="460.73.01"
-#export DRIVER_VERSION="550.54.14"
-#export DRIVER_VERSION="560.35.03"
-#export DRIVER_VERSION="550.135"
-#export NCCL_VERSION="2.8.3"
-#export CUDNN_VERSION="8.0.5.39"
-#export NCCL_VERSION="2.8.4"
-#export ACCELERATOR_TYPE="nvidia-tesla-p100"
-
 # DPGKE
 export DPGKE_NAMESPACE=k8sns-${CLUSTER_NAME}
 export GKE_CLUSTER_NAME="gke-${CLUSTER_NAME}"
@@ -267,6 +257,20 @@ export HIVE_DATA_BUCKET="${BUCKET}"
 export WAREHOUSE_BUCKET="gs://${HIVE_DATA_BUCKET}"
 export HIVE_METASTORE_WAREHOUSE_DIR="${WAREHOUSE_BUCKET}/datasets"
 
+# CI/CD Variables
+export CI_PROJECT_ID="$(jq -r .CI_PROJECT_ID "${GCLOUD_DIR}/env.json")"
+export CI_GCP_CREDENTIALS_PATH="$(jq -r .CI_GCP_CREDENTIALS_PATH "${GCLOUD_DIR}/env.json")"
+export CI_CSR_REPO_NAME="$(jq -r .CI_CSR_REPO_NAME "${GCLOUD_DIR}/env.json")"
+export CI_CSR_REGION="$(jq -r .CI_CSR_REGION "${GCLOUD_DIR}/env.json")"
+export CI_GITHUB_CONNECTION_NAME="$(jq -r .CI_GITHUB_CONNECTION_NAME "${GCLOUD_DIR}/env.json")"
+export CI_TRIGGER_BRANCH="$(jq -r .CI_TRIGGER_BRANCH "${GCLOUD_DIR}/env.json")"
+export CUSTOM_IMAGE_URI="$(jq -r .CUSTOM_IMAGE_URI "${GCLOUD_DIR}/env.json")"
+if [[ "${CUSTOM_IMAGE_URI}" == "null" ]]; then
+  unset CUSTOM_IMAGE_URI
+fi
+export CI_REPO_OWNER="$(jq -r .CI_REPO_OWNER "${GCLOUD_DIR}/env.json")"
+export CI_BYOSA_EMAIL="$(jq -r .CI_BYOSA_EMAIL "${GCLOUD_DIR}/env.json")"
+
 function configure_environment() {
   dataproc_repro_configure_environment=1
 
@@ -311,7 +315,7 @@ function configure_environment() {
   #
   # MOK config for secure boot
   #
-  eval "$(bash lib/secure-boot/create-key-pair.sh)"
+  eval "$(bash "${GCLOUD_DIR}/lib/secure-boot/create-key-pair.sh")"
   #modulus_md5sum=cd2bd1bdd9f9e4c43c12aecf6c338d6f
   #private_secret_name=efi-db-priv-key-042
   #public_secret_name=efi-db-pub-key-042
@@ -333,10 +337,7 @@ function configure_environment() {
   # The reader will then need to pass the
   # `--image="projects/${PROJECT_ID}/global/images/"${PURPOSE}-${dataproc_version/\./-}-${timestamp}"`
   # argument instead of `--image-version "${IMAGE_VERSION}"` when
-  # performing the gcloud dataproc clusters create command.  Modify the
-  # call to gcloud in lib/shared-functions.sh's create_dpgce_cluster
-  # function.
-
+  # performing the gcloud dataproc clusters create command.
 }
 
 [[ -v dataproc_repro_configure_environment ]] || configure_environment
diff --git a/gcloud/lib/gcp/gcr.sh b/gcloud/lib/gcp/gcr.sh
index b2735f2c..cbd61f57 100644
--- a/gcloud/lib/gcp/gcr.sh
+++ b/gcloud/lib/gcp/gcr.sh
@@ -2,32 +2,25 @@
 # GCR functions
 
 function create_artifacts_repository(){
-  local phase_name="create_artifacts_repository"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating Artifact Repository ${ARTIFACT_REPOSITORY}..."
-    report_result "Exists"
-    return 0
-  fi
-
   print_status "Creating Artifact Repository ${ARTIFACT_REPOSITORY}..."
-  if gcloud artifacts repositories describe "${ARTIFACT_REPOSITORY}" --location="${REGION}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
+  local log_file="create_artifacts_repository_${ARTIFACT_REPOSITORY}.log"
+  if run_gcloud "${log_file}" gcloud artifacts repositories create "${ARTIFACT_REPOSITORY}" \
+    --repository-format=docker \
+    --location="${REGION}" --project="${PROJECT_ID}"; then
+    report_result "Created"
+    refresh_resource_state "artifactsRepository" "lib/gcp/gcr.sh" exists_artifacts_repository
   else
-    local log_file="create_artifacts_repository_${ARTIFACT_REPOSITORY}.log"
-    if run_gcloud "${log_file}" gcloud artifacts repositories create "${ARTIFACT_REPOSITORY}" \
-      --repository-format=docker \
-      --location="${REGION}" --project="${PROJECT_ID}"; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
+    report_result "Fail"
+    return 1
   fi
 }
 export -f create_artifacts_repository
 
+function exists_artifacts_repository() {
+  _check_exists gcloud artifacts repositories describe "${ARTIFACT_REPOSITORY}" --location="${REGION}" --project="${PROJECT_ID}" --format="json(name,format)"
+}
+export -f exists_artifacts_repository
+
 function push_container_image() {
   print_status "Pushing Container Image..."
   local log_file="push_container_image.log"
diff --git a/gcloud/lib/gcp/gcs.sh b/gcloud/lib/gcp/gcs.sh
index 97effc2e..bbc2816d 100644
--- a/gcloud/lib/gcp/gcs.sh
+++ b/gcloud/lib/gcp/gcs.sh
@@ -2,6 +2,16 @@
 #
 # GCS Bucket functions
 
+function exists_gcs_bucket() {
+  local bucket_name="$1"
+  if gcloud storage ls --buckets "gs://${bucket_name}" > /dev/null 2>&1; then
+    echo "{\"name\": \"${bucket_name}\", \"exists\": true}"
+  else
+    echo "null"
+  fi
+}
+export -f exists_gcs_bucket
+
 function create_bucket () {
   local phase_name="create_bucket"
   if check_sentinel "${phase_name}" "done"; then
@@ -15,6 +25,10 @@ function create_bucket () {
   if ! gcloud storage ls --buckets "gs://${BUCKET}" > /dev/null 2>&1 ; then
     if run_gcloud "${log_file}" gcloud storage buckets create --location ${REGION} gs://${BUCKET}; then
       report_result "Created"
+      local cache_key="gcsBucket-${bucket_name}"
+      if [[ "${bucket_name}" == "${BUCKET}" ]]; then cache_key="gcsBucket"; fi
+      if [[ "${bucket_name}" == "${TEMP_BUCKET}" ]]; then cache_key="gcsTempBucket"; fi
+      refresh_resource_state "${cache_key}" "exists_gcs_bucket ${bucket_name}" "lib/gcp/gcs.sh"
     else
       report_result "Fail"
       return 1
@@ -22,6 +36,7 @@ function create_bucket () {
   else
     report_result "Exists"
   fi
+
   # Grant SA permissions on BUCKET
   print_status "  Granting Storage Admin on gs://${BUCKET}..."
   if run_gcloud "${log_file}" gcloud storage buckets add-iam-policy-binding "gs://${BUCKET}" --member="serviceAccount:${GSA}" --role="roles/storage.admin"; then
@@ -42,6 +57,7 @@ function create_bucket () {
   else
     report_result "Exists"
   fi
+
   # Grant SA permissions on TEMP_BUCKET
   print_status "  Granting Storage Admin on gs://${TEMP_BUCKET}..."
   if run_gcloud "${temp_log_file}" gcloud storage buckets add-iam-policy-binding "gs://${TEMP_BUCKET}" --member="serviceAccount:${GSA}" --role="roles/storage.admin"; then
@@ -49,8 +65,21 @@ function create_bucket () {
   else
     report_result "Fail"
   fi
+}
+
+function grant_gcs_bucket_perms() {
+  local bucket_name="$1"
+  local log_file="grant_perms_${bucket_name}.log"
+  print_status "  Granting Storage Admin on gs://${bucket_name}..."
+  if run_gcloud "${log_file}" gcloud storage buckets add-iam-policy-binding "gs://${bucket_name}" --member="serviceAccount:${GSA}" --role="roles/storage.admin"; then
+    report_result "Pass"
+  else
+    report_result "Fail"
+  fi
+}
+
 
-  # Copy initialization action scripts
+function upload_init_actions() {
   if [[ -d init ]] ; then
     print_status "Copying init scripts to ${INIT_ACTIONS_ROOT}..."
     local cp_log="copy_init_scripts.log"
@@ -60,7 +89,6 @@ function create_bucket () {
       report_result "Fail"
     fi
   fi
-  create_sentinel "${phase_name}" "done"
 }
 
 function delete_bucket () {
@@ -69,7 +97,6 @@ function delete_bucket () {
   if gcloud storage ls --buckets "gs://${BUCKET}" > /dev/null 2>&1; then
     if run_gcloud "${log_file}" gcloud storage rm --recursive "gs://${BUCKET}"; then
       report_result "Deleted"
-      remove_sentinel "create_bucket" "done"
     else
       report_result "Fail"
     fi
diff --git a/gcloud/lib/gcp/iam.sh b/gcloud/lib/gcp/iam.sh
index d00116a0..4fbf4e85 100644
--- a/gcloud/lib/gcp/iam.sh
+++ b/gcloud/lib/gcp/iam.sh
@@ -2,192 +2,175 @@
 #
 # IAM related functions
 
-function create_service_account() {
-  local phase_name="create_service_account"
-  local sentinel_name="${SA_NAME}_done"
+ROLES=(
+  roles/dataproc.worker
+  roles/dataproc.editor
+  roles/dataproc.admin
+  roles/bigquery.dataEditor
+  roles/bigquery.dataViewer
+  roles/bigquery.user
+  roles/storage.admin
+  roles/secretmanager.secretAccessor
+  roles/compute.admin
+  roles/iam.serviceAccountUser
+)
+
+function exists_service_account() {
+  _check_exists gcloud iam service-accounts describe "${GSA}" --project="${PROJECT_ID}" --format="json(email,name)"
+}
+export -f exists_service_account
 
-  if check_sentinel "${phase_name}" "${sentinel_name}"; then
-    print_status "Checking Service Account ${GSA} and roles..."
-    report_result "Exists"
-    return 0
+function get_service_account_bindings() {
+  local cmd=(
+    gcloud projects get-iam-policy "${PROJECT_ID}"
+    --format=json
+  )
+
+  local policy
+  if ! policy=$("${cmd[@]}" 2>/dev/null); then
+    echo "null"
+    return
   fi
 
-  print_status "Creating/Verifying Service Account ${GSA}..."
-  local log_file="create_service_account_${SA_NAME}.log"
+  if [[ -z "${policy}" ]]; then
+    echo "null"
+    return
+  fi
+
+  # Extract unique roles bound to the service account
+  echo "${policy}" | jq -r --arg GSA "serviceAccount:${GSA}" '[.bindings[] | select(.members // [] | any(. == $GSA)) | .role] | unique' | jq -c 'if length == 0 then null else . end'
+}
+export -f get_service_account_bindings
+
+# Returns "true" or "false"
+function check_service_account_bindings() {
+  local roles_json=$(get_service_account_bindings)
+  if [[ "${roles_json}" == "null" || -z "${roles_json}" ]]; then
+    echo "false"
+    return
+  fi
 
-  SA_EXISTS=$(gcloud iam service-accounts list \
-    --project="${PROJECT_ID}" \
-    --filter="email=${GSA}" \
-    --format="value(email)" 2>/dev/null)
-
-  if [[ -z "${SA_EXISTS}" ]]; then
-    if ! run_gcloud "${log_file}" gcloud iam service-accounts create "${SA_NAME}" \
-      --project="${PROJECT_ID}" \
-      --description="Service account for use with cluster ${CLUSTER_NAME}" \
-      --display-name="${SA_NAME}"; then
-      report_result "Fail"
-      return 1
+  for role in "${ROLES[@]}"; do
+    local role_found=$(echo "${roles_json}" | jq --arg ROLE "${role}" 'index($ROLE) != null')
+    if [[ "${role_found}" != "true" ]]; then
+      # echo "DEBUG: Role ${role} not found for ${GSA}" >&2
+      echo "false"
+      return
     fi
+  done
+  echo "true"
+}
+export -f check_service_account_bindings
+
+# This function is called by the audit script
+function audit_service_account_roles() {
+  check_service_account_bindings
+}
+export -f audit_service_account_roles
+
+function create_service_account() {
+  print_status "Creating Service Account ${GSA}..."
+  local log_file="create_service_account_${SA_NAME}.log"
+
+  local cmd=(
+    gcloud iam service-accounts create "${SA_NAME}"
+    --project="${PROJECT_ID}"
+    --description="Service account for use with cluster ${CLUSTER_NAME}"
+    --display-name="${SA_NAME}"
+  )
+  if run_gcloud "${log_file}" "${cmd[@]}"; then
     report_result "Created"
-    sleep 10
+    refresh_resource_state "serviceAccount" "lib/gcp/iam.sh" exists_service_account
+    sleep 10 # Allow propagation
   else
-    report_result "Exists"
+    report_result "Fail"
+    return 1
   fi
+}
+export -f create_service_account
 
-  # Bind roles
-  print_status "  Ensuring roles for ${GSA}... "
-  ROLES=(
-    roles/dataproc.worker
-    roles/dataproc.editor
-    roles/dataproc.admin
-    roles/bigquery.dataEditor
-    roles/bigquery.dataViewer
-    roles/bigquery.user
-    roles/storage.admin
-    roles/secretmanager.secretAccessor
-    roles/compute.admin
-    roles/iam.serviceAccountUser
-  )
+function ensure_service_account_roles() {
+  print_status "Ensuring roles for ${GSA}... "
   local all_roles_bound=true
   for role in "${ROLES[@]}"; do
-    local role_file_name=$(echo "${role}" | tr '/' '_')
-    local role_log="bind_roles/bind_${role_file_name}_${SA_NAME}.log"
-    # print_status "    Binding ${role}..."
-    if ! run_gcloud "${role_log}" gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
-      --member="serviceAccount:${GSA}" \
-      --role="${role}" --condition=None; then
-      all_roles_bound=false
-      # report_result "Fail"
+    local role_log="bind_roles/bind_${role//\//_}_${SA_NAME}.log"
+    local cmd=(
+      gcloud projects add-iam-policy-binding "${PROJECT_ID}"
+      --member="serviceAccount:${GSA}"
+      --role="${role}"
+      --condition=None
+      --quiet
+    )
+    # We don't care if the run_gcloud fails here, as the binding might already exist.
+    run_gcloud "${role_log}" "${cmd[@]}" > /dev/null 2>&1 || true
+  done
+
+  # Verify bindings with retries
+  local bindings_ok="false"
+  local attempts=0
+  while [[ "${bindings_ok}" != "true" && ${attempts} -lt 5 ]]; do
+    attempts=$((attempts + 1))
+    bindings_ok=$(check_service_account_bindings)
+    if [[ "${bindings_ok}" != "true" ]]; then
+      if (( attempts > 1 )); then
+          echo "  DEBUG: Role bindings not fully ready, attempt ${attempts}/5. Waiting 10s..." >&2
+          sleep 10
+      fi
     fi
   done
+  update_state "serviceAccountBindings" "$(get_service_account_bindings)"
+  update_state "serviceAccountRolesReady" "${bindings_ok}"
 
-  if [[ "${all_roles_bound}" = true ]]; then
-     report_result "Pass" # Overall role binding status
-     create_sentinel "${phase_name}" "${sentinel_name}"
+  if [[ "${bindings_ok}" = true ]]; then
+     report_result "Pass"
   else
      report_result "Fail"
+     echo "  ERROR: Failed to bind all required roles to ${GSA}" >&2
      return 1
   fi
 }
-export -f create_service_account
+export -f ensure_service_account_roles
 
 function delete_service_account() {
-  print_status "Deleting Service Account ${GSA}..."
-  SA_EXISTS=$(gcloud iam service-accounts list \
-    --project="${PROJECT_ID}" \
-    --filter="email=${GSA}" \
-    --format="value(email)" 2>/dev/null)
+  print_status "Deleting Service Account ${GSA}... Element: serviceAccount"
+  local log_file="delete_service_account_${SA_NAME}.log"
 
-  if [[ -z "${SA_EXISTS}" ]]; then
+  if [[ $(get_state "serviceAccount") == "null" ]]; then
     report_result "Not Found"
     return 0
   fi
 
-  local log_file="delete_service_account_${SA_NAME}.log"
-  # Attempt to remove bindings - ignore errors if not found
-  for svc in spark-executor spark-driver agent ; do
-    gcloud iam service-accounts remove-iam-policy-binding \
-      --role=roles/iam.workloadIdentityUser \
-      --member="serviceAccount:${PROJECT_ID}.svc.id.goog[${DPGKE_NAMESPACE}/${svc}]" \
-      "${GSA}" > /dev/null 2>&1 || true
-  done
-
-  ROLES=(
-    roles/dataproc.worker
-    roles/dataproc.editor
-    roles/dataproc.admin
-    roles/bigquery.dataEditor
-    roles/bigquery.dataViewer
-    roles/bigquery.user
-    roles/storage.admin
-    roles/secretmanager.secretAccessor
-    roles/compute.admin
-    roles/iam.serviceAccountUser
-  )
-  for role in "${ROLES[@]}"; do
-    gcloud projects remove-iam-policy-binding \
-      --role="${role}" \
-      --member="serviceAccount:${GSA}" \
-      "${PROJECT_ID}" --condition=None > /dev/null 2>&1 || true
-  done
-
-   gcloud iam service-accounts remove-iam-policy-binding "${GSA}" \
-    --member="serviceAccount:${GSA}" \
-    --role=roles/iam.serviceAccountUser > /dev/null 2>&1 || true
+  # Attempt to remove common project-level bindings
+  print_status "Removing IAM policy bindings for ${GSA}..."
+  local bindings_json=$(get_state "serviceAccountBindings")
+  if [[ "${bindings_json}" != "null" ]]; then
+    mapfile -t roles_to_remove < <(echo "${bindings_json}" | jq -r '.[].role' | sort -u)
+    for role in "${roles_to_remove[@]}"; do
+      local role_log="unbind_roles/unbind_${role//\//_}_${SA_NAME}.log"
+      local cmd=(
+        gcloud projects remove-iam-policy-binding "${PROJECT_ID}"
+        --role="${role}"
+        --member="serviceAccount:${GSA}"
+        --condition=None
+        --quiet
+      )
+      run_gcloud "${role_log}" "${cmd[@]}" || true
+    done
+    report_result "Bindings removed"
+  else
+    report_result "No Bindings Found"
+  fi
+  update_state "serviceAccountBindings" "null"
+  update_state "serviceAccountRolesReady" "null"
 
-  # Delete the service account
-  if run_gcloud "${log_file}" gcloud iam service-accounts delete --quiet "${GSA}"; then
+  print_status "Deleting service account ${GSA}..."
+  local delete_cmd=(gcloud iam service-accounts delete --quiet "${GSA}")
+  if run_gcloud "${log_file}" "${delete_cmd[@]}"; then
     report_result "Deleted"
-    remove_sentinel "create_service_account" "${SA_NAME}_done"
+    update_state "serviceAccount" "null"
   else
-    report_result "Fail"
+    report_result "Fail" # Fail the script if SA deletion fails and it wasn't a NOT_FOUND
+    return 1
   fi
 }
-
-function grant_kms_roles(){
-      print_status "Granting KMS Roles to ${GSA}..."
-      local log_file="grant_kms_roles_${SA_NAME}.log"
-      if run_gcloud "${log_file}" gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
-        --member="serviceAccount:${GSA}" \
-        --role=roles/cloudkms.cryptoKeyDecrypter; then
-        report_result "Pass"
-      else
-        report_result "Fail"
-      fi
-    }
-    export -f grant_kms_roles
-
-    function grant_mysql_roles(){
-      print_status "Granting MySQL/Cloud SQL Roles to ${GSA}..."
-      local log_file="grant_mysql_roles_${SA_NAME}.log"
-      if run_gcloud "${log_file}" gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
-        --member="serviceAccount:${GSA}" \
-        --role=roles/cloudsql.editor; then
-        report_result "Pass"
-      else
-        report_result "Fail"
-      fi
-    }
-    export -f grant_mysql_roles
-
-    function grant_bigtables_roles(){
-      print_status "Granting Bigtable Roles to ${GSA}..."
-      local log_file="grant_bigtable_roles_${SA_NAME}.log"
-      local all_ok=true
-      if ! run_gcloud "${log_file}" gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
-        --member="serviceAccount:${GSA}" \
-        --role=roles/bigtable.user; then
-        all_ok=false
-      fi
-      if ! run_gcloud "${log_file}" gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
-        --member="serviceAccount:${GSA}" \
-        --role=roles/bigtable.admin; then
-        all_ok=false
-      fi
-      if [[ "${all_ok}" = true ]]; then report_result "Pass"; else report_result "Fail"; fi
-    }
-    export -f grant_bigtables_roles
-
-    function grant_gke_roles(){
-      print_status "Granting GKE Roles to ${GSA}..."
-      local log_file="grant_gke_roles_${SA_NAME}.log"
-      local all_ok=true
-      for svc in agent spark-driver spark-executor ; do
-        if ! run_gcloud "${log_file}" gcloud iam service-accounts add-iam-policy-binding \
-          --role=roles/iam.workloadIdentityUser \
-          --member="serviceAccount:${PROJECT_ID}.svc.id.goog[${DPGKE_NAMESPACE}/${svc}]" \
-          "${GSA}"; then
-          all_ok=false
-        fi
-      done
-      if ! run_gcloud "${log_file}" gcloud artifacts repositories add-iam-policy-binding "${ARTIFACT_REPOSITORY}" \
-          --location="${REGION}" \
-          --member="serviceAccount:${GSA}" \
-          --role=roles/artifactregistry.writer; then
-          all_ok=false
-      fi
-      if [[ "${all_ok}" = true ]]; then report_result "Pass"; else report_result "Fail"; fi
-    }
-    export -f grant_gke_roles
-
-
+export -f delete_service_account
diff --git a/gcloud/lib/gcp/kms.sh b/gcloud/lib/gcp/kms.sh
index c5f69ebb..17b27582 100644
--- a/gcloud/lib/gcp/kms.sh
+++ b/gcloud/lib/gcp/kms.sh
@@ -3,20 +3,12 @@
 # KMS and Secret Manager functions
 
 function enable_secret_manager() {
-  local phase_name="enable_secret_manager"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Enabling Secret Manager API..."
-    report_result "Exists"
-    return 0
-  fi
-
   print_status "Enabling Secret Manager API..."
   local log_file="enable_secret_manager.log"
   if run_gcloud "${log_file}" gcloud services enable \
     secretmanager.googleapis.com \
     --project=${PROJECT_ID}; then
     report_result "Pass"
-    create_sentinel "${phase_name}" "done"
   else
     report_result "Fail"
   fi
@@ -25,101 +17,64 @@ export -f enable_secret_manager
 
 function create_secret() {
   local secret_name="${1:-${MYSQL_SECRET_NAME}}"
-  local phase_name="create_secret_${secret_name}"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating Secret ${secret_name}..."
-    report_result "Exists"
-    return 0
-  fi
-
   print_status "Creating Secret ${secret_name}..."
   local log_file="create_secret_${secret_name}.log"
-  if gcloud secrets describe "${secret_name}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
+  if echo -n "super secret" | run_gcloud "${log_file}" gcloud secrets create "${secret_name}" \
+    --project="${PROJECT_ID}" \
+    --replication-policy="automatic" \
+    --data-file=-; then
+    report_result "Created"
   else
-    if echo -n "super secret" | run_gcloud "${log_file}" gcloud secrets create "${secret_name}" \
-      --project="${PROJECT_ID}" \
-      --replication-policy="automatic" \
-      --data-file=-; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-    fi
+    report_result "Fail"
   fi
 }
 export -f create_secret
 
 function create_kms_keyring() {
-  local phase_name="create_kms_keyring"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating KMS Keyring ${KMS_KEYRING}..."
-    report_result "Exists"
-    return 0
-  fi
   print_status "Creating KMS Keyring ${KMS_KEYRING}..."
-  if gcloud kms keyrings list --location global --project="${PROJECT_ID}" | grep "${KMS_KEYRING}" > /dev/null 2>&1; then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
+  local log_file="create_kms_keyring_${KMS_KEYRING}.log"
+  if run_gcloud "${log_file}" gcloud kms keyrings create "${KMS_KEYRING}" --location=global --project="${PROJECT_ID}"; then
+    report_result "Created"
+    refresh_resource_state "kmsKeyring" "lib/gcp/kms.sh" exists_kms_keyring
   else
-    local log_file="create_kms_keyring_${KMS_KEYRING}.log"
-    if run_gcloud "${log_file}" gcloud kms keyrings create "${KMS_KEYRING}" --location=global --project="${PROJECT_ID}"; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
+    report_result "Fail"
+    return 1
   fi
 }
 export -f create_kms_keyring
 
 function create_kerberos_kdc_key() {
-  local phase_name="create_kerberos_kdc_key"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating KMS Key ${KDC_ROOT_PASSWD_KEY}..."
-    report_result "Exists"
-    return 0
-  fi
   print_status "Creating KMS Key ${KDC_ROOT_PASSWD_KEY}..."
-  if gcloud kms keys list --location global --keyring="${KMS_KEYRING}" --project="${PROJECT_ID}" | grep "${KDC_ROOT_PASSWD_KEY}" > /dev/null 2>&1; then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
+  local log_file="create_kms_key_${KDC_ROOT_PASSWD_KEY}.log"
+  if run_gcloud "${log_file}" gcloud kms keys create "${KDC_ROOT_PASSWD_KEY}" \
+    --location=global \
+    --keyring="${KMS_KEYRING}" \
+    --purpose=encryption --project="${PROJECT_ID}"; then
+    report_result "Created"
   else
-    local log_file="create_kms_key_${KDC_ROOT_PASSWD_KEY}.log"
-    if run_gcloud "${log_file}" gcloud kms keys create "${KDC_ROOT_PASSWD_KEY}" \
-      --location=global \
-      --keyring="${KMS_KEYRING}" \
-      --purpose=encryption --project="${PROJECT_ID}"; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
+    report_result "Fail"
+    return 1
   fi
 }
 export -f create_kerberos_kdc_key
 
+function exists_kms_key() {
+  local key_name="$1"
+  _check_exists gcloud kms keys describe "${key_name}" --keyring="${KMS_KEYRING}" --location=global --project="${PROJECT_ID}" --format="json(name,primary.state)"
+}
+export -f exists_kms_key
+
 function create_mysql_admin_password() {
-    local phase_name="create_mysql_admin_password"
-    if check_sentinel "${phase_name}" "done"; then
-      print_status "Creating Encrypted MySQL Admin Password..."
-      report_result "Exists"
-      return 0
-    fi
     print_status "Creating Encrypted MySQL Admin Password..."
     local log_file="create_mysql_admin_password.log"
     if dd if=/dev/urandom bs=8 count=4 | xxd -p | \
       run_gcloud "${log_file}" gcloud kms encrypt \
       --location=global \
-      --keyring=projects/${PROJECT_ID}/locations/global/keyRings/${KMS_KEYRING} \
-      --key=projects/${PROJECT_ID}/locations/global/keyRings/${KMS_KEYRING}/cryptoKeys/${KDC_ROOT_PASSWD_KEY} \
+      --keyring="projects/${PROJECT_ID}/locations/global/keyRings/${KMS_KEYRING}" \
+      --key="projects/${PROJECT_ID}/locations/global/keyRings/${KMS_KEYRING}/cryptoKeys/${KDC_ROOT_PASSWD_KEY}" \
       --plaintext-file=- \
       --ciphertext-file=init/mysql_admin_password.encrypted; then
       report_result "Created"
-      create_sentinel "${phase_name}" "done"
     else
       report_result "Fail"
       return 1
@@ -128,48 +83,33 @@ function create_mysql_admin_password() {
 export -f create_mysql_admin_password
 
 function create_kerberos_kdc_password() {
-  local phase_name="create_kerberos_kdc_password"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating Encrypted KDC Root Password..."
-    report_result "Exists"
-    return 0
-  fi
-  if [[ -f init/${KDC_ROOT_PASSWD_KEY}.encrypted ]]; then
-    print_status "Creating Encrypted KDC Root Password..."
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
+  print_status "Creating Encrypted KDC Root Password..."
+  local log_file="create_kdc_root_password.log"
+  if dd if=/dev/urandom bs=8 count=4 | xxd -p | \
+    run_gcloud "${log_file}" gcloud kms encrypt \
+    --location=global \
+    --keyring="${KMS_KEYRING}" \
+    --key="${KDC_ROOT_PASSWD_KEY}" \
+    --plaintext-file=- \
+    --ciphertext-file="init/${KDC_ROOT_PASSWD_KEY}.encrypted"; then
+    report_result "Created"
   else
-    print_status "Creating Encrypted KDC Root Password..."
-    local log_file="create_kdc_root_password.log"
-    if dd if=/dev/urandom bs=8 count=4 | xxd -p | \
-      run_gcloud "${log_file}" gcloud kms encrypt \
-      --location=global \
-      --keyring=${KMS_KEYRING} \
-      --key=${KDC_ROOT_PASSWD_KEY} \
-      --plaintext-file=- \
-      --ciphertext-file=init/${KDC_ROOT_PASSWD_KEY}.encrypted; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
+    report_result "Fail"
+    return 1
   fi
 }
 export -f create_kerberos_kdc_password
 
 function create_kerberos_sa_password() {
-    local phase_name="create_kerberos_sa_password"
-    # This one always re-creates, so no sentinel check
     print_status "Creating Encrypted KDC SA Password..."
     local log_file="create_kdc_sa_password.log"
     if dd if=/dev/urandom bs=8 count=4 | xxd -p | \
       run_gcloud "${log_file}" gcloud kms encrypt \
       --location=global \
-      --keyring=${KMS_KEYRING} \
-      --key=${KDC_ROOT_PASSWD_KEY} \
+      --keyring="${KMS_KEYRING}" \
+      --key="${KDC_ROOT_PASSWD_KEY}" \
       --plaintext-file=- \
-      --ciphertext-file=init/${KDC_SA_PASSWD_KEY}.encrypted; then
+      --ciphertext-file="init/${KDC_SA_PASSWD_KEY}.encrypted"; then
       report_result "Created"
     else
       report_result "Fail"
diff --git a/gcloud/lib/gcp/misc.sh b/gcloud/lib/gcp/misc.sh
index 60754447..52f5604a 100644
--- a/gcloud/lib/gcp/misc.sh
+++ b/gcloud/lib/gcp/misc.sh
@@ -37,15 +37,50 @@ function configure_gcloud() {
     report_result "Pass"
   fi
 }
+export -f configure_gcloud
 
-function enable_services () {
-  local phase_name="enable_services"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Enabling GCP Services..."
-    report_result "Exists"
-    return 0
-  fi
+function check_project() {
+    print_status "Verifying project ${PROJECT_ID}..."
+    local project_raw
+    project_raw=$(get_state "project")
+    if [[ "${project_raw}" == "null" || -z "${project_raw}" ]]; then
+      print_status "Project not found in state DB" >&2
+      report_result "Fail"
+      exit 1
+    fi
+    local project_state=$(echo "${project_raw}" | jq -r '.lifecycleState // "NOT_FOUND"')
+
+    if [[ "${project_state}" == "ACTIVE" ]]; then
+        report_result "Pass"
+    else
+        report_result "Fail"
+        echo "  - Project ${PROJECT_ID} is not ACTIVE (state: ${project_state})." >&2
+        exit 1
+    fi
+}
+
+function check_billing() {
+    print_status "Verifying billing for ${PROJECT_ID}..."
+    local billing_raw
+    billing_raw=$(get_state "billing")
+    if [[ "${billing_raw}" == "null" || -z "${billing_raw}" ]]; then
+      print_status "Billing info not found in state DB" >&2
+      report_result "Fail"
+      exit 1
+    fi
+    local billing_enabled=$(echo "${billing_raw}" | jq -r '.billingEnabled // false')
 
+    if [[ "${billing_enabled}" == "true" ]]; then
+        report_result "Pass"
+    else
+        report_result "Fail"
+        echo "  - Billing is not enabled for project ${PROJECT_ID}." >&2
+        echo "  - Please run: gcloud beta billing projects link ${PROJECT_ID} --billing-account <ACCOUNT_ID>" >&2
+        exit 1
+    fi
+}
+
+function enable_services () {
   print_status "Enabling GCP Services..."
   local log_file="enable_services.log"
   if run_gcloud "${log_file}" gcloud services enable \
@@ -59,27 +94,18 @@ function enable_services () {
     privateca.googleapis.com \
     --project=${PROJECT_ID}; then
     report_result "Pass"
-    create_sentinel "${phase_name}" "done"
   else
     report_result "Fail"
   fi
 }
 
 function enable_secret_manager() {
-  local phase_name="enable_secret_manager"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Enabling Secret Manager API..."
-    report_result "Exists"
-    return 0
-  fi
-
   print_status "Enabling Secret Manager API..."
   local log_file="enable_secret_manager.log"
   if run_gcloud "${log_file}" gcloud services enable \
     secretmanager.googleapis.com \
     --project=${PROJECT_ID}; then
     report_result "Pass"
-    create_sentinel "${phase_name}" "done"
   else
     report_result "Fail"
   fi
@@ -87,27 +113,47 @@ function enable_secret_manager() {
 
 function create_secret() {
   local secret_name="${1:-${MYSQL_SECRET_NAME}}"
-  local phase_name="create_secret_${secret_name}"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating Secret ${secret_name}..."
-    report_result "Exists"
-    return 0
-  fi
-
   print_status "Creating Secret ${secret_name}..."
   local log_file="create_secret_${secret_name}.log"
-  if gcloud secrets describe "${secret_name}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
+  if echo -n "super secret" | run_gcloud "${log_file}" gcloud secrets create "${secret_name}" \
+    --project="${PROJECT_ID}" \
+    --replication-policy="automatic" \
+    --data-file=-; then
+    report_result "Created"
   else
-    if echo -n "super secret" | run_gcloud "${log_file}" gcloud secrets create "${secret_name}" \
-      --project="${PROJECT_ID}" \
-      --replication-policy=\"automatic\" \
-      --data-file=-; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-    fi
+    report_result "Fail"
+  fi
+}
+
+function check_image_exists() {
+  local image_uri="$1"
+  if [[ -z "${image_uri}" || "${image_uri}" == "null" ]]; then
+    return 1 # Not found if URI is empty or null
   fi
+  # Extracts image name from full URI if necessary
+  local image_name=$(basename "${image_uri}")
+  gcloud compute images describe "${image_name}" --project="${PROJECT_ID}" > /dev/null 2>&1
+}
+
+# Check for any debug VMs
+function exists_debug_vms() {
+  _check_exists gcloud compute instances list --project="${PROJECT_ID}" --filter='name~^debug-' --format="json(name,zone,status)" | jq 'if . == [] then null else . end'
+}
+export -f exists_debug_vms
+
+# Check for any VMs in the default network
+function exists_any_vms_in_network() {
+  local vms=$(gcloud compute instances list --project="${PROJECT_ID}" --filter="networkInterfaces.network ~ /${NETWORK}$" --format="value(name)" 2>/dev/null)
+  if [[ -n "${vms}" ]]; then
+    echo "true"
+  else
+    echo "false"
+  fi
+}
+export -f exists_any_vms_in_network
+
+# Get state of any VMs in the default network for audit
+function get_any_network_vms_state() {
+  _check_exists gcloud compute instances list --project="${PROJECT_ID}" --filter="networkInterfaces.network ~ /${NETWORK}$" --format="json(name,zone,status)" | jq 'if . == [] then null else . end'
 }
+export -f get_any_network_vms_state
diff --git a/gcloud/lib/gcp/project.sh b/gcloud/lib/gcp/project.sh
index ad11425d..b4a92282 100644
--- a/gcloud/lib/gcp/project.sh
+++ b/gcloud/lib/gcp/project.sh
@@ -3,13 +3,6 @@
 # GCP Project related functions
 
 function create_project(){
-  local phase_name="create_project"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Checking Project ${PROJECT_ID}..."
-    report_result "Exists"
-    return 0
-  fi
-
   print_status "Checking Project ${PROJECT_ID}..."
   local log_file="create_project_${PROJECT_ID}.log"
   local PROJ_DESCRIPTION=$(gcloud projects describe ${PROJECT_ID} --format json 2>/dev/null)
@@ -67,13 +60,11 @@ EOF
     else
       report_result "Pass"
     fi
-    create_sentinel "${phase_name}" "done"
   fi
 }
 export -f create_project
 
 function delete_project() {
-  local phase_name="create_project"
   print_status "Deleting Project ${PROJECT_ID}..."
   local log_file="delete_project_${PROJECT_ID}.log"
 
@@ -97,7 +88,6 @@ function delete_project() {
   print_status "  Deleting project ${PROJECT_ID}... "
   if gcloud projects delete --quiet ${PROJECT_ID} >> "${REPRO_TMPDIR}/${log_file}" 2>&1; then
     report_result "Deleted"
-    remove_sentinel "${phase_name}" "done"
   else
     report_result "Fail"
   fi
diff --git a/gcloud/lib/gke.sh b/gcloud/lib/gke.sh
index 62ed0223..3c2d7322 100644
--- a/gcloud/lib/gke.sh
+++ b/gcloud/lib/gke.sh
@@ -2,73 +2,58 @@
 # GKE functions
 
 function create_gke_cluster() {
-  local phase_name="create_gke_cluster"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating GKE Cluster ${GKE_CLUSTER_NAME}..."
-    report_result "Exists"
-    return 0
-  fi
-
   print_status "Creating GKE Cluster ${GKE_CLUSTER_NAME}..."
-  if gcloud container clusters describe "${GKE_CLUSTER_NAME}" --zone "${ZONE}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
+  local log_file="create_gke_cluster_${GKE_CLUSTER_NAME}.log"
+  if run_gcloud "${log_file}" gcloud container clusters create "${GKE_CLUSTER_NAME}" \
+    --service-account="${GSA}" \
+    --workload-pool="${PROJECT_ID}.svc.id.goog" \
+    --tags "${TAGS}" \
+    --subnetwork "${SUBNET}" \
+    --network "${NETWORK}" \
+    --zone "${ZONE}" --project="${PROJECT_ID}"; then
+    report_result "Created"
   else
-    local log_file="create_gke_cluster_${GKE_CLUSTER_NAME}.log"
-    if run_gcloud "${log_file}" gcloud container clusters create "${GKE_CLUSTER_NAME}" \
-      --service-account="${GSA}" \
-      --workload-pool="${PROJECT_ID}.svc.id.goog" \
-      --tags "${TAGS}" \
-      --subnetwork "${SUBNET}" \
-      --network "${NETWORK}" \
-      --zone "${ZONE}" --project="${PROJECT_ID}"; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
+    report_result "Fail"
+    return 1
   fi
 }
 export -f create_gke_cluster
 
 function delete_gke_cluster() {
-  local phase_name="create_gke_cluster"
-  remove_sentinel "${phase_name}" "done"
+  local cluster_exists=$(exists_gke_cluster)
+  # echo "DEBUG: delete_gke_cluster: cluster_exists='${cluster_exists}'" >&2
+  if [[ "${cluster_exists}" == "null" ]]; then
+    print_status "GKE Cluster ${GKE_CLUSTER_NAME}..."
+    report_result "Not Found"
+    return 0
+  fi
 
   print_status "Deleting GKE Cluster ${GKE_CLUSTER_NAME}..."
   local log_file="delete_gke_cluster_${GKE_CLUSTER_NAME}.log"
-  if gcloud container clusters describe "${GKE_CLUSTER_NAME}" --zone "${ZONE}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    for pn in "${DP_CTRL_POOLNAME}" "${DP_DRIVER_POOLNAME}" "${DP_EXEC_POOLNAME}" ; do
+  
+  # List all node pools and delete them one by one
+  local pools=$(gcloud container node-pools list --cluster "${GKE_CLUSTER_NAME}" --zone "${ZONE}" --project="${PROJECT_ID}" --format="value(NAME)" 2>/dev/null)
+  if [[ -n "${pools}" ]]; then
+    for pn in ${pools}; do
       print_status "  Deleting Node Pool ${pn}..."
       run_gcloud "delete_nodepool_${pn}.log" gcloud container node-pools delete --quiet "${pn}" \
         --zone "${ZONE}" \
         --cluster "${GKE_CLUSTER_NAME}" --project="${PROJECT_ID}" || true
     done
+  fi
 
-    if run_gcloud "${log_file}" gcloud container clusters delete --quiet "${GKE_CLUSTER_NAME}" --zone "${ZONE}" --project="${PROJECT_ID}"; then
-      report_result "Deleted"
-    else
-      report_result "Fail"
-    fi
+  if run_gcloud "${log_file}" gcloud container clusters delete --quiet "${GKE_CLUSTER_NAME}" \
+    --zone "${ZONE}" --project="${PROJECT_ID}"; then
+    report_result "Deleted"
+    update_state "gkeCluster" "null"
   else
-    report_result "Not Found"
+    report_result "Fail"
   fi
 }
 export -f delete_gke_cluster
 
 function create_dpgke_cluster() {
-  local phase_name="create_dpgke_cluster"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating DPGKE Cluster ${DPGKE_CLUSTER_NAME}..."
-    report_result "Exists"
-    return 0
-  fi
-
   print_status "Creating DPGKE Cluster ${DPGKE_CLUSTER_NAME}..."
-  # How to check if DPGKE cluster exists?
-  # gcloud dataproc clusters describe does not work for GKE based clusters.
-  # Let's assume for now, if the sentinel is not there, we create.
   local log_file="create_dpgke_cluster_${DPGKE_CLUSTER_NAME}.log"
   if run_gcloud "${log_file}" gcloud dataproc clusters gke create "${DPGKE_CLUSTER_NAME}" \
     --project="${PROJECT_ID}" \
@@ -82,7 +67,7 @@ function create_dpgke_cluster() {
     --pools="name=${DP_DRIVER_POOLNAME},min=1,max=3,roles=spark-driver,machineType=n2-standard-4" \
     --pools="name=${DP_EXEC_POOLNAME},min=1,max=10,roles=spark-executor,machineType=n2-standard-8"; then
     report_result "Created"
-    create_sentinel "${phase_name}" "done"
+    refresh_resource_state "dpgkeCluster" "lib/gke.sh" exists_dpgke_cluster
   else
     report_result "Fail"
     return 1
@@ -91,15 +76,35 @@ function create_dpgke_cluster() {
 export -f create_dpgke_cluster
 
 function delete_dpgke_cluster() {
-  local phase_name="create_dpgke_cluster"
-  remove_sentinel "${phase_name}" "done"
+  if [[ $(exists_dpgke_cluster) == "null" ]]; then
+    print_status "DPGKE Cluster ${DPGKE_CLUSTER_NAME}..."
+    report_result "Not Found"
+    return 0
+  fi
+
   print_status "Deleting DPGKE Cluster ${DPGKE_CLUSTER_NAME}..."
   local log_file="delete_dpgke_cluster_${DPGKE_CLUSTER_NAME}.log"
-  # How to check existence? Assume delete will fail if not found.
   if run_gcloud "${log_file}" gcloud dataproc clusters delete --quiet "${DPGKE_CLUSTER_NAME}" --region="${REGION}" --project="${PROJECT_ID}"; then
     report_result "Deleted"
+    update_state "dpgkeCluster" "null"
   else
     report_result "Fail"
   fi
 }
 export -f delete_dpgke_cluster
+
+function exists_dpgke_cluster() {
+  _check_exists gcloud dataproc clusters describe "${DPGKE_CLUSTER_NAME}" --region "${REGION}" --project="${PROJECT_ID}" --format="json(clusterName,status.state)"
+}
+export -f exists_dpgke_cluster
+
+function exists_gke_cluster() {
+  _check_exists gcloud container clusters describe "${GKE_CLUSTER_NAME}" --zone "${ZONE}" --project="${PROJECT_ID}" --format="json(name,status)"
+}
+export -f exists_gke_cluster
+
+function exists_gke_nodepool() {
+  local pool_name="$1"
+  _check_exists gcloud container node-pools describe "${pool_name}" --cluster "${GKE_CLUSTER_NAME}" --zone "${ZONE}" --project="${PROJECT_ID}" --format="json(name,status)"
+}
+export -f exists_gke_nodepool
diff --git a/gcloud/lib/kerberos.sh b/gcloud/lib/kerberos.sh
index c4cd592c..1c8b5b38 100644
--- a/gcloud/lib/kerberos.sh
+++ b/gcloud/lib/kerberos.sh
@@ -2,122 +2,87 @@
 # Kerberos functions
 
 function create_kdc_server() {
-  local phase_name="create_kdc_server"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating KDC Server ${KDC_NAME}..."
-    report_result "Exists"
-    return 0
-  fi
-
   print_status "Creating KDC Server ${KDC_NAME}..."
-  if gcloud compute instances describe "${KDC_NAME}" --zone "${ZONE}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
+  local log_file="create_kdc_server_${KDC_NAME}.log"
+  local METADATA="kdc-root-passwd=${INIT_ACTIONS_ROOT}/${KDC_ROOT_PASSWD_KEY}.encrypted"
+  METADATA="${METADATA},kms-keyring=${KMS_KEYRING}"
+  METADATA="${METADATA},kdc-root-passwd-key=${KDC_ROOT_PASSWD_KEY}"
+  METADATA="${METADATA},startup-script-url=${INIT_ACTIONS_ROOT}/kdc-server.sh"
+  METADATA="${METADATA},service-account-user=${GSA}"
+  if run_gcloud "${log_file}" gcloud compute instances create "${KDC_NAME}" \
+    --zone "${ZONE}" \
+    --subnet "${SUBNET}" \
+    --service-account="${GSA}" \
+    --boot-disk-type pd-ssd \
+    --image-family="${KDC_IMAGE_FAMILY}" \
+    --image-project="${KDC_IMAGE_PROJECT}" \
+    --machine-type="${KDC_MACHINE_TYPE}" \
+    --scopes='cloud-platform' \
+    --hostname="${KDC_FQDN}" \
+    --metadata "${METADATA}"; then
+    report_result "Created"
   else
-    local log_file="create_kdc_server_${KDC_NAME}.log"
-    local METADATA="kdc-root-passwd=${INIT_ACTIONS_ROOT}/${KDC_ROOT_PASSWD_KEY}.encrypted"
-    METADATA="${METADATA},kms-keyring=${KMS_KEYRING}"
-    METADATA="${METADATA},kdc-root-passwd-key=${KDC_ROOT_PASSWD_KEY}"
-    METADATA="${METADATA},startup-script-url=${INIT_ACTIONS_ROOT}/kdc-server.sh"
-    METADATA="${METADATA},service-account-user=${GSA}"
-    if run_gcloud "${log_file}" gcloud compute instances create "${KDC_NAME}" \
-      --zone "${ZONE}" \
-      --subnet "${SUBNET}" \
-      --service-account="${GSA}" \
-      --boot-disk-type pd-ssd \
-      --image-family="${KDC_IMAGE_FAMILY}" \
-      --image-project="${KDC_IMAGE_PROJECT}" \
-      --machine-type="${KDC_MACHINE_TYPE}" \
-      --scopes='cloud-platform' \
-      --hostname="${KDC_FQDN}" \
-      --metadata "${METADATA}"; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
+    report_result "Fail"
+    return 1
   fi
 }
 export -f create_kdc_server
 
-function delete_kdc_server() {
-  local phase_name="create_kdc_server"
-  remove_sentinel "${phase_name}" "done"
+function exists_kdc_server() {
+  _check_exists gcloud compute instances describe "${KDC_NAME}" --zone "${ZONE}" --project="${PROJECT_ID}" --format="json(name,status)"
+}
+export -f exists_kdc_server
 
+function delete_kdc_server() {
   print_status "Deleting KDC Server ${KDC_NAME}..."
   local log_file="delete_kdc_server_${KDC_NAME}.log"
-  if gcloud compute instances describe "${KDC_NAME}" --zone "${ZONE}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    if run_gcloud "${log_file}" gcloud compute instances delete "${KDC_NAME}" --zone "${ZONE}" --project="${PROJECT_ID}" --quiet; then
-      report_result "Deleted"
-    else
-      report_result "Fail"
-    fi
+  if run_gcloud "${log_file}" gcloud compute instances delete "${KDC_NAME}" --zone "${ZONE}" --project="${PROJECT_ID}" --quiet; then
+    report_result "Deleted"
   else
-    report_result "Not Found"
+    report_result "Fail"
   fi
 }
 export -f delete_kdc_server
 
 function create_kerberos_cluster() {
-  local phase_name="create_kerberos_cluster"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating Kerberos Cluster ${CLUSTER_NAME}..."
-    report_result "Exists"
-    return 0
-  fi
-
   print_status "Creating Kerberos Cluster ${CLUSTER_NAME}..."
-  if gcloud dataproc clusters describe "${CLUSTER_NAME}" --region="${REGION}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
+  local log_file="create_kerberos_cluster_${CLUSTER_NAME}.log"
+  if run_gcloud "${log_file}" gcloud dataproc clusters create "${CLUSTER_NAME}" \
+    --region "${REGION}" \
+    --zone "${ZONE}" \
+    --subnet "${SUBNET}" \
+    --no-address \
+    --service-account="${GSA}" \
+    --master-machine-type n1-standard-4 \
+    --master-boot-disk-type pd-ssd \
+    --master-boot-disk-size 50 \
+    --image-version "${IMAGE_VERSION}" \
+    --bucket "${BUCKET}" \
+    --initialization-action-timeout=10m \
+    --max-idle="${IDLE_TIMEOUT}" \
+    --enable-component-gateway \
+    --scopes='cloud-platform' \
+    --enable-kerberos \
+    --kerberos-root-principal-password-uri="${INIT_ACTIONS_ROOT}/${KDC_ROOT_PASSWD_KEY}.encrypted" \
+    --kerberos-kms-key="${KDC_ROOT_PASSWD_KEY}" \
+    --kerberos-kms-key-keyring="${KMS_KEYRING}" \
+    --kerberos-kms-key-location=global \
+    --kerberos-kms-key-project="${PROJECT_ID}"; then
+    report_result "Created"
   else
-    local log_file="create_kerberos_cluster_${CLUSTER_NAME}.log"
-    if run_gcloud "${log_file}" gcloud dataproc clusters create "${CLUSTER_NAME}" \
-      --region "${REGION}" \
-      --zone "${ZONE}" \
-      --subnet "${SUBNET}" \
-      --no-address \
-      --service-account="${GSA}" \
-      --master-machine-type n1-standard-4 \
-      --master-boot-disk-type pd-ssd \
-      --master-boot-disk-size 50 \
-      --image-version "${IMAGE_VERSION}" \
-      --bucket "${BUCKET}" \
-      --initialization-action-timeout=10m \
-      --max-idle="${IDLE_TIMEOUT}" \
-      --enable-component-gateway \
-      --scopes='cloud-platform' \
-      --enable-kerberos \
-      --kerberos-root-principal-password-uri="${INIT_ACTIONS_ROOT}/${KDC_ROOT_PASSWD_KEY}.encrypted" \
-      --kerberos-kms-key="${KDC_ROOT_PASSWD_KEY}" \
-      --kerberos-kms-key-keyring="${KMS_KEYRING}" \
-      --kerberos-kms-key-location=global \
-      --kerberos-kms-key-project="${PROJECT_ID}"; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
+    report_result "Fail"
+    return 1
   fi
 }
 export -f create_kerberos_cluster
 
 function delete_kerberos_cluster() {
-  local phase_name="create_kerberos_cluster"
-  remove_sentinel "${phase_name}" "done"
-
   print_status "Deleting Kerberos Cluster ${CLUSTER_NAME}..."
   local log_file="delete_kerberos_cluster_${CLUSTER_NAME}.log"
-  if gcloud dataproc clusters describe "${CLUSTER_NAME}" --region="${REGION}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    if run_gcloud "${log_file}" gcloud dataproc clusters delete --quiet --region "${REGION}" "${CLUSTER_NAME}"; then
-      report_result "Deleted"
-    else
-      report_result "Fail"
-    fi
+  if run_gcloud "${log_file}" gcloud dataproc clusters delete --quiet --region "${REGION}" "${CLUSTER_NAME}"; then
+    report_result "Deleted"
   else
-    report_result "Not Found"
+    report_result "Fail"
   fi
 }
 export -f delete_kerberos_cluster
\ No newline at end of file
diff --git a/gcloud/lib/network/firewall.sh b/gcloud/lib/network/firewall.sh
index 16a51fd2..34172ad4 100644
--- a/gcloud/lib/network/firewall.sh
+++ b/gcloud/lib/network/firewall.sh
@@ -2,14 +2,12 @@
 #
 # Firewall rule functions
 
-function create_firewall_rules() {
-  local phase_name="create_firewall_rules"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating base Firewall Rules for ${NETWORK}..."
-    report_result "Exists"
-    return 0
-  fi
+function exists_firewall() {
+    # This is a basic check. A more robust version might check for a list of rules.
+    _check_exists "gcloud compute firewall-rules describe '${FIREWALL}-in-ssh' --project='${PROJECT_ID}' --format='json(name,selfLink)'"
+}
 
+function create_firewall_rules() {
   print_status "Creating base Firewall Rules for ${NETWORK}..."
   local log_file="create_firewalls_${NETWORK}.log"
   local created_some=false
@@ -83,15 +81,11 @@ function create_firewall_rules() {
     else
       report_result "Exists"
     fi
-    create_sentinel "${phase_name}" "done"
   fi
 }
 export -f create_firewall_rules
 
 function delete_firewall_rules () {
-  local phase_name="create_firewall_rules"
-  remove_sentinel "${phase_name}" "done"
-
   print_status "Deleting Cluster Firewall Rules..."
   local log_file="delete_firewalls_${NETWORK}.log"
   # Delete any rule containing the cluster name
@@ -124,15 +118,3 @@ function delete_firewall_rules () {
   fi
 }
 export -f delete_firewall_rules
-
-function create_logging_firewall_rules () {
-  print_status "Creating Logging Firewall Rules for ${NETWORK}..."
-  # ... (implementation with run_gcloud and report_result) ...
-}
-export -f create_logging_firewall_rules
-
-function delete_logging_firewall_rules () {
-  print_status "Deleting Logging Firewall Rules for ${NETWORK}..."
-  # ... (implementation with run_gcloud and report_result) ...
-}
-export -f delete_logging_firewall_rules
\ No newline at end of file
diff --git a/gcloud/lib/network/network.sh b/gcloud/lib/network/network.sh
index d4aa0c4b..18edc0c2 100644
--- a/gcloud/lib/network/network.sh
+++ b/gcloud/lib/network/network.sh
@@ -2,59 +2,52 @@
 #
 # VPC Network functions
 
-function create_vpc_network () {
-  local phase_name="create_vpc_network"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating VPC Network ${NETWORK}..."
-    report_result "Exists"
-    return 0
-  fi
+function exists_network() {
+  _check_exists gcloud compute networks describe "${NETWORK}" --project="${PROJECT_ID}" --format="json(name,selfLink)"
+}
+export -f exists_network
 
+function create_vpc_network () {
   print_status "Creating VPC Network ${NETWORK}..."
   local log_file="create_vpc_${NETWORK}.log"
-
-  if gcloud compute networks describe "${NETWORK}" --project="${PROJECT_ID}" > /dev/null 2>&1;
- then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
+  if run_gcloud "${log_file}" gcloud compute networks create "${NETWORK}" \
+    --project="${PROJECT_ID}" \
+    --subnet-mode=custom \
+    --bgp-routing-mode="regional" \
+    --description="network for use with Dataproc cluster ${CLUSTER_NAME}"; then
+    report_result "Created"
+    refresh_resource_state "vpcNetwork" "lib/network/network.sh" exists_network
   else
-    if run_gcloud "${log_file}" gcloud compute networks create "${NETWORK}" \
-      --project="${PROJECT_ID}" \
-      --subnet-mode=custom \
-      --bgp-routing-mode="regional" \
-      --description="network for use with Dataproc cluster ${CLUSTER_NAME}"; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
+    report_result "Fail"
+    return 1
   fi
 }
+export -f create_vpc_network
 
 function delete_vpc_network () {
   print_status "Deleting VPC Network ${NETWORK}..."
   local log_file="delete_vpc_${NETWORK}.log"
-  local network_check=$(gcloud compute networks list --project="${PROJECT_ID}" --filter="name = ${NETWORK}" --format="value(name)" 2>/dev/null)
+  if run_gcloud "${log_file}" gcloud compute networks delete --quiet "${NETWORK}" --project="${PROJECT_ID}"; then
+    report_result "Deleted"
+    update_state "vpcNetwork" "null"
+  else
+    report_result "Fail"
+  fi
+}
+export -f delete_vpc_network
 
-  if [[ -n "${network_check}" ]]; then
-    if run_gcloud "${log_file}" gcloud compute networks delete --quiet "${NETWORK}" --project="${PROJECT_ID}"; then
-      report_result "Deleted"
-      remove_sentinel "create_vpc_network" "done"
-    else
-      report_result "Fail"
-      local dep_log="${REPRO_TMPDIR}/VPC_Network_Delete_Failed_${NETWORK}_${RESOURCE_SUFFIX}.log"
-      echo "--- Firewall Rules in ${NETWORK} ---" > "${dep_log}"
-      gcloud compute firewall-rules list --project="${PROJECT_ID}" --filter="network ~ ${NETWORK}$" --format="value(name)" >> "${dep_log}" 2>&1
-      echo "--- Routes in ${NETWORK} ---" >> "${dep_log}"
-      gcloud compute routes list --project="${PROJECT_ID}" --filter="network ~ ${NETWORK}$" --format="value(name, nextHopGateway)" >> "${dep_log}" 2>&1
-      echo "--- Routers in ${REGION} ---" >> "${dep_log}"
-      gcloud compute routers list --regions="${REGION}" --project="${PROJECT_ID}" --format="value(name, network)" >> "${dep_log}" 2>&1
-      echo "--- Subnets in ${NETWORK} ---" >> "${dep_log}"
-      gcloud compute networks subnets list --network="${NETWORK}" --project="${PROJECT_ID}" --format="value(name)" >> "${dep_log}" 2>&1
-    fi
+function delete_all_network_vms() {
+  print_status "Deleting all remaining VMs in network ${NETWORK}..."
+  local log_file="delete_all_network_vms_${NETWORK}.log"
+  local vms=$(gcloud compute instances list --project="${PROJECT_ID}" --filter="networkInterfaces.network ~ /${NETWORK}$" --format="value(NAME,ZONE)" 2>/dev/null)
+  if [[ -n "${vms}" ]]; then
+    echo "${vms}" | while read -r name zone; do
+      print_status "  Deleting VM ${name} in ${zone}..."
+      run_gcloud "delete_vm_${name}.log" gcloud compute instances delete "${name}" --zone "${zone}" --quiet || true
+    done
+    report_result "Done"
   else
-    report_result "Not Found"
-    remove_sentinel "create_vpc_network" "done"
+    report_result "None Found"
   fi
 }
+export -f delete_all_network_vms
diff --git a/gcloud/lib/network/peering.sh b/gcloud/lib/network/peering.sh
index 3cbc8f81..bdae311c 100644
--- a/gcloud/lib/network/peering.sh
+++ b/gcloud/lib/network/peering.sh
@@ -27,6 +27,7 @@ function delete_ip_allocation () {
   if gcloud compute addresses describe ${ALLOCATION_NAME} --global --project="${PROJECT_ID}" > /dev/null 2>&1; then
     if run_gcloud "${log_file}" gcloud compute addresses delete --quiet --global ${ALLOCATION_NAME}; then
       report_result "Deleted"
+      update_state "ipAllocation" "null"
     else
       report_result "Fail"
     fi
@@ -35,6 +36,13 @@ function delete_ip_allocation () {
   fi
 }
 
+function exists_vpc_peering() {
+   # Naming format: <servicenetworking-googleapis-com>
+   local peering_name="servicenetworking-googleapis-com"
+  _check_exists gcloud compute networks peerings list --network="${NETWORK}" --project="${PROJECT_ID}" --filter="name=${peering_name}" --format="json(name,state)" | jq 'if . == [] then null else .[0] end'
+}
+export -f exists_vpc_peering
+
 function create_vpc_peering () {
   print_status "Creating VPC Peering for ${NETWORK}..."
   local log_file="create_peering_${NETWORK}.log"
diff --git a/gcloud/lib/network/router.sh b/gcloud/lib/network/router.sh
index 2a9589ae..1f685f6c 100644
--- a/gcloud/lib/network/router.sh
+++ b/gcloud/lib/network/router.sh
@@ -2,115 +2,103 @@
 #
 # Router and NAT functions
 
-function create_router () {
-  local phase_name="create_router"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating Router ${ROUTER_NAME}..."
-    report_result "Exists"
-    return 0
-  fi
+function exists_router() {
+    _check_exists gcloud compute routers describe "${ROUTER_NAME}" --region="${REGION}" --project="${PROJECT_ID}" --format="json(name,selfLink)"
+}
+export -f exists_router
 
+function create_router () {
   print_status "Creating Router ${ROUTER_NAME}..."
   local log_file="create_router_${ROUTER_NAME}.log"
-  if gcloud compute routers describe "${ROUTER_NAME}" --region="${REGION}" --project="${PROJECT_ID}" > /dev/null 2>&1;
- then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
+  if run_gcloud "${log_file}" gcloud compute routers create "${ROUTER_NAME}" \
+    --project="${PROJECT_ID}" \
+    --network="${NETWORK}" \
+    --asn="${ASN_NUMBER}" \
+    --region="${REGION}"; then
+    report_result "Created"
+    refresh_resource_state "cloudRouter" "lib/network/router.sh" exists_router
   else
-    if run_gcloud "${log_file}" gcloud compute routers create ${ROUTER_NAME} \
-      --project=${PROJECT_ID} \
-      --network=${NETWORK} \
-      --asn=${ASN_NUMBER} \
-      --region=${REGION}; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
+    report_result "Fail"
+    return 1
   fi
 }
 export -f create_router
 
-function add_nat_policy () {
-  local phase_name="add_nat_policy"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Adding NAT to Router ${ROUTER_NAME}..."
-    report_result "Exists"
-    return 0
-  fi
-
+function add_nat_to_router () {
   print_status "Adding NAT to Router ${ROUTER_NAME}..."
   local log_file="add_nat_${ROUTER_NAME}.log"
-  if gcloud compute routers nats describe nat-config --router="${ROUTER_NAME}" --region="${REGION}" --project="${PROJECT_ID}" > /dev/null 2>&1;
- then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
-  else
-    if run_gcloud "${log_file}" gcloud compute routers nats create nat-config \
-      --router-region ${REGION} \
-      --router ${ROUTER_NAME} \
-      --project="${PROJECT_ID}" \
-      --nat-custom-subnet-ip-ranges "${SUBNET}" \
-      --auto-allocate-nat-external-ips; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
-  fi
-}
-export -f add_nat_policy
 
-function delete_nat_configs() {
-  local phase_name="add_nat_policy"
-  print_status "Deleting NAT Configs from ${ROUTER_NAME}..."
-  local log_file="delete_nats_${ROUTER_NAME}.log"
-  local found_some=false
-  local all_ok=true
+  # Attempt to delete nat-config first, ignore errors
+  gcloud compute routers nats delete "nat-config" \
+    --router-region "${REGION}" \
+    --router "${ROUTER_NAME}" \
+    --project="${PROJECT_ID}" --quiet > /dev/null 2>&1 || true
+  sleep 5 # Brief pause to allow delete to propagate
 
-  if gcloud compute routers describe "${ROUTER_NAME}" --region="${REGION}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    NATS=$(gcloud compute routers nats list --router="${ROUTER_NAME}" --region="${REGION}" --project="${PROJECT_ID}" --format="value(name)" 2>/dev/null || true)
-    if [[ -n "${NATS}" ]]; then
-      found_some=true
-      while read -r nat_name; do
-        # print_status "  Deleting NAT ${nat_name} from ${ROUTER_NAME}..."
-        if ! run_gcloud "${log_file}" gcloud compute routers nats delete "${nat_name}" --router="${ROUTER_NAME}" --region="${REGION}" --project="${PROJECT_ID}" --quiet; then
-          all_ok=false
-        fi
-      done <<< "${NATS}"
-    fi
+  if run_gcloud "${log_file}" gcloud compute routers nats create "nat-config" \
+    --router-region "${REGION}" \
+    --router "${ROUTER_NAME}" \
+    --project="${PROJECT_ID}" \
+    --nat-custom-subnet-ip-ranges "${SUBNET}" \
+    --auto-allocate-nat-external-ips; then
+    report_result "Created"
+    refresh_resource_state "cloudRouter" "lib/network/router.sh" exists_router
+    refresh_resource_state "cloudRouterNAT" "lib/network/router.sh" exists_router_nat "nat-config"
+  else
+    report_result "Fail"
+    return 1
   fi
+}
+export -f add_nat_to_router
 
-  if [[ "${found_some}" = false ]]; then
-    report_result "Not Found"
-    remove_sentinel "${phase_name}" "done"
-  elif [[ "${all_ok}" = true ]]; then
+function remove_nat_from_router () {
+  print_status "Removing NAT configuration 'nat-config' from ${ROUTER_NAME}..."
+  local log_file="remove_nat_${ROUTER_NAME}.log"
+  local cmd=(
+    gcloud compute routers nats delete "nat-config"
+    --router-region "${REGION}"
+    --router "${ROUTER_NAME}"
+    --project="${PROJECT_ID}"
+    --quiet
+  )
+  if run_gcloud "${log_file}" "${cmd[@]}"; then
     report_result "Deleted"
-    remove_sentinel "${phase_name}" "done"
+    update_state "cloudRouterNAT" "null"
   else
     report_result "Fail"
+    return 1
   fi
 }
-export -f delete_nat_configs
+export -f remove_nat_from_router
 
 function delete_router () {
-  local phase_name="create_router"
-  print_status "Deleting Router ${ROUTER_NAME}..."
-  local log_file="delete_router_${ROUTER_NAME}.log"
+  if [[ -n $(get_state "cloudRouterNAT") ]]; then
+    print_status "Deleting NAT from Router ${ROUTER_NAME}..."
+    local log_file="delete_nat_${ROUTER_NAME}.log"
+    # Don't fail if the NAT doesn't exist in GCP, as state might be slightly stale
+    run_gcloud "${log_file}" gcloud compute routers nats delete "nat-config" \
+      --router-region "${REGION}" \
+      --router "${ROUTER_NAME}" \
+      --project="${PROJECT_ID}" --quiet || true
+    update_state "cloudRouterNAT" "null"
+  fi
 
-  local router_check=$(gcloud compute routers list --regions="${REGION}" --project="${PROJECT_ID}" --filter="name = ${ROUTER_NAME}" --format="value(name)" 2>/dev/null)
-  if [[ -n "${router_check}" ]]; then
-    if run_gcloud "${log_file}" gcloud compute routers delete --quiet --region ${REGION} "${ROUTER_NAME}" --project="${PROJECT_ID}"; then
-      report_result "Deleted"
-      remove_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-    fi
+  print_status "Deleting Router ${ROUTER_NAME}..."
+  log_file="delete_router_${ROUTER_NAME}.log"
+  if run_gcloud "${log_file}" gcloud compute routers delete --quiet "${ROUTER_NAME}" \
+    --region="${REGION}" \
+    --project="${PROJECT_ID}"; then
+    report_result "Deleted"
+    update_state "cloudRouter" "null"
   else
-    report_result "Not Found"
-    remove_sentinel "${phase_name}" "done"
+    report_result "Fail"
   fi
 }
-export -f delete_router
\ No newline at end of file
+export -f delete_router
+
+function exists_router_nat() {
+  local nat_name="$1"
+  # gcloud compute routers nats describe returns non-zero if not found
+  _check_exists gcloud compute routers nats describe "${nat_name}" --router="${ROUTER_NAME}" --region="${REGION}" --project="${PROJECT_ID}"
+}
+export -f exists_router_nat
diff --git a/gcloud/lib/network/routes.sh b/gcloud/lib/network/routes.sh
index ca14fa2e..aadf5f3f 100644
--- a/gcloud/lib/network/routes.sh
+++ b/gcloud/lib/network/routes.sh
@@ -1,41 +1,66 @@
 #!/bin/bash
 #
-# Route functions
-
-function create_default_route() {
-  local route_name="default-internet-${NETWORK}"
-  print_status "Creating Default Route ${route_name}..."
-  local log_file="create_route_${route_name}.log"
-
-  if gcloud compute routes describe "${route_name}" --project="${PROJECT_ID}" > /dev/null 2>&1;
-  then
-    report_result "Exists"
-  else
-    if run_gcloud "${log_file}" gcloud compute routes create "${route_name}" \
+function ensure_default_internet_route() {
+  print_status "Ensuring default internet route for ${NETWORK}..."
+  local log_file="ensure_default_route_${NETWORK}.log"
+  if ! check_default_route; then
+    print_status "  Default internet route not found, creating..."
+    if run_gcloud "${log_file}" gcloud compute routes create "default-internet-${NETWORK}" \
       --project="${PROJECT_ID}" \
       --network="${NETWORK}" \
       --destination-range=0.0.0.0/0 \
-      --next-hop-gateway=default-internet-gateway; then
+      --next-hop-gateway=default-internet-gateway \
+      --priority=1000; then
       report_result "Created"
     else
       report_result "Fail"
+      return 1
     fi
+  else
+    report_result "Exists"
   fi
 }
+export -f ensure_default_internet_route
+
+function check_default_route() {
+  local project_id="${PROJECT_ID}"
+  local network_uri="https://www.googleapis.com/compute/v1/projects/${project_id}/global/networks/${NETWORK}"
+  local gateway_uri="https://www.googleapis.com/compute/v1/projects/${project_id}/global/gateways/default-internet-gateway"
+  local dest_range="0.0.0.0/0"
 
-function delete_default_route() {
-  local route_name="default-internet-${NETWORK}"
-  print_status "Deleting Default Route ${route_name}..."
-  if gcloud compute routes describe "${route_name}" --project="${PROJECT_ID}" > /dev/null 2>&1;
-  then
-    local log_file="delete_route_${route_name}.log"
+  # Fetch all routes in JSON format
+  if gcloud compute routes list --project="${project_id}" --format=json | \
+     jq -e --arg network_uri "$network_uri" \
+          --arg gateway_uri "$gateway_uri" \
+          --arg dest_range "$dest_range" \
+          '.[] | select(.network == $network_uri and .destRange == $dest_range and .nextHopGateway == $gateway_uri)' > /dev/null; then
+    # Match found, jq exits with 0
+    return 0
+  else
+    # No match, jq exits with non-zero
+    return 1
+  fi
+}
+export -f check_default_route
+
+function delete_route() {
+  local route_name="$1"
+  print_status "Deleting Route ${route_name}..."
+  local log_file="delete_route_${route_name}.log"
+
+  # Check if the route exists
+  if gcloud compute routes describe "${route_name}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
     if run_gcloud "${log_file}" gcloud compute routes delete --quiet "${route_name}" --project="${PROJECT_ID}"; then
       report_result "Deleted"
+      refresh_resource_state "routes" "" _check_exists gcloud compute routes list --project="${PROJECT_ID}" --filter="network~/${NETWORK}$" --format="json(name,selfLink)"
     else
       report_result "Fail"
+      echo "  - Failed to delete route ${route_name}. Log content:" >&2
+      cat "${LOG_DIR}/${log_file}" >&2
+      return 1
     fi
   else
     report_result "Not Found"
   fi
 }
-
+export -f delete_route
diff --git a/gcloud/lib/network/subnet.sh b/gcloud/lib/network/subnet.sh
index 83f2b037..397b8adc 100644
--- a/gcloud/lib/network/subnet.sh
+++ b/gcloud/lib/network/subnet.sh
@@ -2,49 +2,41 @@
 #
 # Subnet functions
 
-function create_subnet () {
-  local phase_name="create_subnet"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating Subnet ${SUBNET}..."
-    report_result "Exists"
-    return 0
-  fi
+function exists_subnet() {
+    local subnet_name="$1"
+    _check_exists gcloud compute networks subnets describe "${subnet_name}" --region="${REGION}" --project="${PROJECT_ID}" --format="json(name,selfLink)"
+}
+export -f exists_subnet
 
-  print_status "Creating Subnet ${SUBNET}..."
-  local log_file="create_subnet_${SUBNET}.log"
-  if gcloud compute networks subnets describe "${SUBNET}" --region="${REGION}" --project="${PROJECT_ID}" > /dev/null 2>&1;
-  then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
+function create_subnet () {
+  local subnet_name="$1"
+  local subnet_key="$2"
+  local range="$3"
+  print_status "Creating Subnet ${subnet_name}..."
+  local log_file="create_subnet_${subnet_name}.log"
+  if run_gcloud "${log_file}" gcloud compute networks subnets create "${subnet_name}" \
+    --project="${PROJECT_ID}" \
+    --network="${NETWORK}" \
+    --range="${range}" \
+    --enable-private-ip-google-access \
+    --region="${REGION}" \
+    --description="subnet for use with Dataproc cluster ${CLUSTER_NAME}"; then
+    report_result "Created"
+    refresh_resource_state "${subnet_key}" "lib/network/subnet.sh" exists_subnet "${subnet_name}"
   else
-    if run_gcloud "${log_file}" gcloud compute networks subnets create ${SUBNET} \
-      --project="${PROJECT_ID}" \
-      --network=${NETWORK} \
-      --range="$RANGE" \
-      --enable-private-ip-google-access \
-      --region=${REGION} \
-      --description="subnet for use with Dataproc cluster ${CLUSTER_NAME}"; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
+    report_result "Fail"
+    return 1
   fi
 }
 
 function delete_subnet () {
-  print_status "Deleting Subnet ${SUBNET}..."
-  if gcloud compute networks subnets describe "${SUBNET}" --region "${REGION}" > /dev/null 2>&1;
-  then
-    local log_file="delete_subnet_${SUBNET}.log"
-    if run_gcloud "${log_file}" gcloud compute networks subnets delete --quiet --region ${REGION} ${SUBNET}; then
-      report_result "Deleted"
-      remove_sentinel "create_subnet" "done"
-    else
-      report_result "Fail"
-    fi
+  local subnet_name="$1"
+  print_status "Deleting Subnet ${subnet_name}..."
+  local log_file="delete_subnet_${subnet_name}.log"
+  if run_gcloud "${log_file}" gcloud compute networks subnets delete --quiet --region "${REGION}" "${subnet_name}"; then
+    update_state "$2" "null"
+    report_result "Deleted"
   else
-    report_result "Not Found"
+    report_result "Fail"
   fi
 }
diff --git a/gcloud/lib/phs.sh b/gcloud/lib/phs.sh
index e83c34b2..6f0bdc90 100644
--- a/gcloud/lib/phs.sh
+++ b/gcloud/lib/phs.sh
@@ -3,53 +3,39 @@
 
 function create_phs_cluster() {
   local phs_cluster_name="${CLUSTER_NAME}-phs"
-  local phase_name="create_phs_cluster"
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating PHS Cluster ${phs_cluster_name}..."
-    report_result "Exists"
-    return 0
-  fi
-
   print_status "Creating PHS Cluster ${phs_cluster_name}..."
-  if gcloud dataproc clusters describe "${phs_cluster_name}" --region="${REGION}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
+  local log_file="create_phs_cluster_${phs_cluster_name}.log"
+  if run_gcloud "${log_file}" gcloud dataproc clusters create "${phs_cluster_name}" \
+    --region="${REGION}" \
+    --single-node \
+    --image-version="${IMAGE_VERSION}" \
+    --subnet="${SUBNET}" \
+    --tags="${TAGS}" \
+    --properties="spark:spark.history.fs.logDirectory=gs://${PHS_BUCKET},spark:spark.eventLog.dir=gs://${PHS_BUCKET}" \
+    --properties="mapred:mapreduce.jobhistory.read-only.dir-pattern=gs://${MR_HISTORY_BUCKET}" \
+    --enable-component-gateway; then
+    report_result "Created"
   else
-    local log_file="create_phs_cluster_${phs_cluster_name}.log"
-    if run_gcloud "${log_file}" gcloud dataproc clusters create "${phs_cluster_name}" \
-      --region="${REGION}" \
-      --single-node \
-      --image-version="${IMAGE_VERSION}" \
-      --subnet="${SUBNET}" \
-      --tags="${TAGS}" \
-      --properties="spark:spark.history.fs.logDirectory=gs://${PHS_BUCKET},spark:spark.eventLog.dir=gs://${PHS_BUCKET}" \
-      --properties="mapred:mapreduce.jobhistory.read-only.dir-pattern=gs://${MR_HISTORY_BUCKET}" \
-      --enable-component-gateway; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
+    report_result "Fail"
+    return 1
   fi
 }
 export -f create_phs_cluster
 
-function delete_phs_cluster() {
+function exists_phs_cluster() {
   local phs_cluster_name="${CLUSTER_NAME}-phs"
-  local phase_name="create_phs_cluster"
-  remove_sentinel "${phase_name}" "done"
+  _check_exists gcloud dataproc clusters describe "${phs_cluster_name}" --region="${REGION}" --project="${PROJECT_ID}" --format="json(clusterName,status.state)"
+}
+export -f exists_phs_cluster
 
+function delete_phs_cluster() {
+  local phs_cluster_name="${CLUSTER_NAME}-phs"
   print_status "Deleting PHS Cluster ${phs_cluster_name}..."
   local log_file="delete_phs_cluster_${phs_cluster_name}.log"
-  if gcloud dataproc clusters describe "${phs_cluster_name}" --region="${REGION}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    if run_gcloud "${log_file}" gcloud dataproc clusters delete --quiet --region "${REGION}" "${phs_cluster_name}"; then
-      report_result "Deleted"
-    else
-      report_result "Fail"
-    fi
+  if run_gcloud "${log_file}" gcloud dataproc clusters delete --quiet --region "${REGION}" "${phs_cluster_name}"; then
+    report_result "Deleted"
   else
-    report_result "Not Found"
+    report_result "Fail"
   fi
 }
 export -f delete_phs_cluster
\ No newline at end of file
diff --git a/gcloud/lib/script-utils.sh b/gcloud/lib/script-utils.sh
index 30ec8ff9..4c60f637 100644
--- a/gcloud/lib/script-utils.sh
+++ b/gcloud/lib/script-utils.sh
@@ -23,30 +23,111 @@ function report_result() {
 }
 export -f report_result
 
+# Compatibility wrapper for older scripts
+function print_result() {
+  report_result "$@"
+}
+export -f print_result
+
+# Usage: report_audit_status "Exists" | "Not Found"
+function report_audit_status() {
+  local status="$1"
+  case "${status}" in
+    Exists) echo -e " [${GREEN}Exists${NC}]" ;;
+    "Not Found") echo -e " [${YELLOW}Not Found${NC}]" ;;
+    True) echo -e " [${GREEN}True${NC}]" ;;
+    False) echo -e " [${RED}False${NC}]" ;;
+    Done) echo -e " [${GREEN}Done${NC}]" ;;
+    *) echo -e " [${YELLOW}${status}${NC}]" ;;
+  esac
+}
+export -f report_audit_status
+
+function configure_gcloud() {
+  local cache_file="${HOME}/.config/gcloud/dpgce_config_cache_ts.txt"
+  local cache_ttl=300 # 5 minutes
+
+  if [[ -f "${cache_file}" ]]; then
+    local last_run=$(cat "${cache_file}")
+    local now=$(date +%s)
+    if (( now - last_run < cache_ttl )); then
+      # echo "DEBUG: gcloud config cache hit" >&2
+      return 0
+    fi
+  fi
+  # echo "DEBUG: gcloud config cache miss" >&2
+
+  print_status "Checking gcloud config..."
+  local account=$(gcloud config get-value account 2> /dev/null)
+  local project=$(gcloud config get-value project 2> /dev/null)
+  local region=$(gcloud config get-value compute/region 2> /dev/null)
+  local zone=$(gcloud config get-value compute/zone 2> /dev/null)
+
+  if [[ -z "${account}" || -z "${project}" || -z "${region}" || -z "${zone}" ]]; then
+    echo -e "${RED}GCLOUD NOT CONFIGURED:${NC}" >&2
+    echo "Please run the following commands:" >&2
+    echo "  gcloud config set account <YOUR_ACCOUNT>" >&2
+    echo "  gcloud config set project <YOUR_PROJECT>" >&2
+    echo "  gcloud config set compute/region <YOUR_REGION>" >&2
+    echo "  gcloud config set compute/zone <YOUR_ZONE>" >&2
+    exit 1
+  fi
+
+  # Check for reauthentication error
+  if ! gcloud projects describe "${project}" > /dev/null 2>&1; then
+      echo -e "
+  ${RED}GCLOUD AUTHENTICATION ERROR:${NC}" >&2
+      echo -e "  Please run ${YELLOW}gcloud auth login${NC} and ${YELLOW}gcloud auth application-default login${NC} to re-authenticate." >&2
+      exit 1
+  fi
+
+  # Update cache timestamp
+  date +%s > "${cache_file}"
+  report_result "Pass"
+}
+export -f configure_gcloud
+
 # Usage: run_gcloud <log_file_name> <gcloud command ...>
 function run_gcloud() {
   local log_file_name=$1
   shift
-  local log_file="${REPRO_TMPDIR}/${log_file_name}"
+  local log_file="${LOG_DIR}/${log_file_name}"
   local log_dir=$(dirname "${log_file}") # Get the directory part
   mkdir -p "${log_dir}" # Create the directory if it doesn't exist
 
   if (( DEBUG != 0 )); then
-    echo "  RUNNING: $*" >&2
+    echo "  RUNNING: ${@}" >&2
   fi
-  "$@" > "${log_file}" 2>&1
+  "${@}" > "${log_file}" 2>&1
   local exit_code=$?
   if [[ ${exit_code} -ne 0 ]]; then
     if grep -q -e "Reauthentication failed" -e "gcloud auth login" -e "gcloud config set account" "${log_file}"; then
-      echo -e "\n  ${RED}GCLOUD AUTHENTICATION ERROR:${NC}"
+      echo -e "
+  ${RED}GCLOUD AUTHENTICATION ERROR:${NC}" >&2
       echo -e "  Please run ${YELLOW}gcloud auth login${NC} and ${YELLOW}gcloud auth application-default login${NC} to re-authenticate." >&2
-    elif (( DEBUG != 0 )); then
-      cat "${log_file}" >&2
-    else
-      :
+      exit 1
+    fi
+    # Check for ALREADY_EXISTS, and don't print the whole log if found
+    if grep -q "ALREADY_EXISTS" "${log_file}"; then
+      echo -e " [${BLUE}Kept${NC}]"
+      return 0
+    fi
+    # Check for NOT_FOUND on delete operations
+    if [[ "$*" == *delete* ]] && grep -q "NOT_FOUND" "${log_file}"; then
+      echo -e " [${YELLOW}Pass*${NC}]" # Already gone
+      return 0
+    fi
+    # Check for IAM policy binding not found errors
+    if grep -q "Policy binding with the specified principal, role, and condition not found" "${log_file}"; then
+      echo -e " [${YELLOW}Pass*${NC}]" # Binding already gone
+      return 0
     fi
+
+    echo -e "${RED}ERROR: ${NC}Command failed with exit code ${exit_code}. Log: ${log_file}" >&2
+    cat "${log_file}" >&2
+    return ${exit_code}
   fi
-  return ${exit_code}
+  return 0
 }
 export -f run_gcloud
 
@@ -63,111 +144,132 @@ function parse_args() {
       --no-create-cluster)
         CREATE_CLUSTER=false
         shift
-        ;;      --force)
+        ;;
+      --force)
         FORCE_DELETE=true
         FORCE_AUDIT=true # Audit scripts use this too
         shift
-        ;;      --quiet-gcloud)
+        ;;
+      --quiet-gcloud)
         GCLOUD_QUIET=true
         shift
-        ;;      *)
-        PARAMS="$PARAMS \"$1\""
+        ;;
+      *)
+        PARAMS="$PARAMS "$1""
         shift
-        ;;    esac
+        ;;
+    esac
   done
   eval set -- "${PARAMS}"
 }
 export -f parse_args
 
-# --- Sentinel Functions ---
-function get_sentinel_file() {
-  local phase_name="$1"
-  local sentinel_name="$2"
-  echo "${SENTINEL_DIR}/${phase_name}-${sentinel_name}"
+# --- State Management Functions ---
+function init_state_db() {
+    local db_file="${STATE_DB}"
+    sqlite3 "${db_file}" "CREATE TABLE IF NOT EXISTS state (key TEXT PRIMARY KEY, value TEXT);"
 }
-export -f get_sentinel_file
+export -f init_state_db
 
-function create_sentinel() {
-  local phase_name="$1"
-  local sentinel_name="$2"
-  touch "$(get_sentinel_file "${phase_name}" "${sentinel_name}")"
-}
-export -f create_sentinel
+function update_state() {
+    local resource_key=$1
+    local resource_value=$2 # JSON string or "null"
+    local db_file="${STATE_DB}"
 
-function check_sentinel() {
-  local phase_name="$1"
-  local sentinel_name="$2"
-  [[ -f "$(get_sentinel_file "${phase_name}" "${sentinel_name}")" ]]
+    local sql
+    if [[ "${resource_value}" == "null" ]]; then
+        sql="DELETE FROM state WHERE key = '${resource_key}';"
+    else
+        local escaped_value=$(echo "${resource_value}" | sed "s/'/''/g")
+        sql="INSERT OR REPLACE INTO state (key, value) VALUES ('${resource_key}', '${escaped_value}');"
+    fi
+    sqlite3 "${db_file}" "${sql}"
 }
-export -f check_sentinel
+export -f update_state
 
-function remove_sentinel() {
-  local phase_name="$1"
-  local sentinel_name="$2"
-  rm -f "$(get_sentinel_file "${phase_name}" "${sentinel_name}")"
+function get_state() {
+    local resource_key=$1
+    local db_file="${STATE_DB}"
+    if [[ ! -f "${db_file}" ]]; then
+        echo ""
+        return
+    fi
+    local result=$(sqlite3 "${db_file}" "SELECT value FROM state WHERE key = '${resource_key}';")
+    if [[ -z "${result}" ]]; then
+        echo ""
+    else
+        echo "${result}"
+    fi
 }
-export -f remove_sentinel
+export -f get_state
 
-function clear_sentinels() {
-  local phase_name="$1"
-  rm -f "${SENTINEL_DIR}/${phase_name}-*"
+function refresh_resource_state() {
+    local resource_key=$1
+    local source_file=$2  # e.g., lib/dataproc/cluster.sh or ""
+    shift 2
+    local check_command=("$@") # Remaining arguments form the command
+
+    local json_output
+    local func_name="${check_command[0]}"
+
+    if [[ -n "${source_file}" ]]; then
+        # Source in a subshell, export the function, then run the command
+        if ! json_output=$(source "${GCLOUD_DIR}/${source_file}" && export -f "${func_name}" && "${check_command[@]}"); then
+            echo "ERROR: Failed to execute check_command in refresh_resource_state for key ${resource_key} from ${source_file}" >&2
+            json_output="null"
+        fi
+    else
+        # Function should already be in the environment (e.g., _check_exists)
+        if ! json_output=$("${check_command[@]}"); then
+             echo "ERROR: Failed to execute check_command in refresh_resource_state for key ${resource_key}" >&2
+             json_output="null"
+        fi
+    fi
+
+    if [[ -z "${json_output}" || "${json_output}" == "[]" ]]; then
+      json_output="null"
+    fi
+
+    update_state "${resource_key}" "${json_output}"
 }
-export -f clear_sentinels
+export -f refresh_resource_state
 
 # --- Audit Check Functions ---
-function check_resource() {
-  local test_name="$1"
-  local command_to_run="$2"
-  local grep_pattern="$3"
-  local optional="${4:-false}"
-  local log_file="${LOG_DIR}/$(echo "$test_name" | tr ' /:' '___').log"
+# These functions are now designed to be called by the audit script.
+# They return a JSON object with details if a resource is found, or the string "null".
+# Call this with command and arguments as separate words, not a single string.
+function _check_exists() {
+    local json_output
+    local exit_code=1
+    local attempts=0
+    local max_attempts=3
+    local delay=2
 
-  print_status "Checking: ${test_name}... "
+    # echo "DEBUG _check_exists called with: $@" >&2
 
-  eval "${command_to_run}" > "${log_file}" 2>&1
+    while [[ ${attempts} -lt ${max_attempts} ]]; do
+        attempts=$((attempts + 1))
+        json_output=$("$@" 2> /dev/null)
+        exit_code=$?
+        # echo "DEBUG INSIDE _check_exists (Attempt ${attempts}):" >&2
+        # echo "DEBUG CMD: $@" >&2
+        # echo "DEBUG EXIT CODE: ${exit_code}" >&2
+        # echo "DEBUG JSON OUTPUT: ${json_output}" >&2
 
-  if grep -q "${grep_pattern}" "${log_file}"; then
-    if [[ "${optional}" == "true" && "${FORCE_AUDIT}" == "false" ]]; then
-      report_result "Kept"
-      return 0
-    else
-      report_result "Fail"
-      return 1
-    fi
-  else
-    report_result "Not Found"
-    return 0
-  fi
-}
-export -f check_resource
-
-function check_resource_exact() {
-  local test_name="$1"
-  local command_to_run="$2"
-  local optional="${3:-false}"
-  local log_file="${LOG_DIR}/$(echo "$test_name" | tr ' /:' '___').log"
-
-  print_status "Checking: ${test_name}... "
-  if eval "${command_to_run}" > "${log_file}" 2>&1; then
-    # Command succeeded, check if it produced output
-    if [[ $(wc -l < "${log_file}") -gt 0 ]]; then
-      # Output found
-      if [[ "${optional}" == "true" && "${FORCE_AUDIT}" == "false" ]]; then
-        report_result "Kept"
-        return 0
-      else
-        report_result "Fail"
-        return 1
-      fi
-    else
-      # Command succeeded but no output, so Not Found
-      report_result "Not Found"
-      return 0
-    fi
-  else
-    # Command failed, resource likely does not exist
-    report_result "Not Found"
-    return 0
-  fi
+        if [[ ${exit_code} -eq 0 && -n "${json_output}" && "${json_output}" != "[]" ]]; then
+            # echo "DEBUG _check_exists: Returning JSON" >&2
+            echo "${json_output}"
+            return 0
+        fi
+
+        if [[ ${attempts} -lt ${max_attempts} ]]; then
+            # echo "DEBUG _check_exists: Attempt ${attempts} failed, retrying in ${delay}s..." >&2
+            sleep ${delay}
+        fi
+    done
+
+    # echo "DEBUG _check_exists: Returning null after ${max_attempts} attempts" >&2
+    echo "null"
+    return 1 # Technically the last exit_code, but we return 1 to indicate not found
 }
-export -f check_resource_exact
+export -f _check_exists
diff --git a/gcloud/lib/secure-boot/create-key-pair.sh b/gcloud/lib/secure-boot/create-key-pair.sh
index 9ed8e49e..0b8e7d82 100644
--- a/gcloud/lib/secure-boot/create-key-pair.sh
+++ b/gcloud/lib/secure-boot/create-key-pair.sh
@@ -25,7 +25,7 @@ set -e
 
 # https://wiki.archlinux.org/title/Unified_Extensible_Firmware_Interface/Secure_Boot#Creating_keys
 
-ITERATION=042
+ITERATION=0009
 
 CURRENT_PROJECT_ID="$(gcloud config get project)"
 if [[ -z "${CURRENT_PROJECT_ID}" ]]; then
diff --git a/gcloud/lib/shared-functions.sh b/gcloud/lib/shared-functions.sh
deleted file mode 100644
index 32599fc5..00000000
--- a/gcloud/lib/shared-functions.sh
+++ /dev/null
@@ -1,1243 +0,0 @@
-#!/bin/bash
-#
-# Copyright 2021 Google LLC and contributors
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS-IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-function create_dpgce_cluster() {
-  [[ exists_dpgce_cluster == 0 ]] && echo "dpgce cluster already exists" && return 0
-
-  set -x
-
-  date
-  time gcloud dataproc clusters create ${CLUSTER_NAME} \
-    --single-node \
-    --master-accelerator "type=${MASTER_ACCELERATOR_TYPE}" \
-    --worker-accelerator "type=${PRIMARY_ACCELERATOR_TYPE}" \
-    --secondary-worker-accelerator "type=${SECONDARY_ACCELERATOR_TYPE}" \
-    --master-machine-type "${MASTER_MACHINE_TYPE}" \
-    --worker-machine-type "${PRIMARY_MACHINE_TYPE}" \
-    --master-boot-disk-size           50 \
-    --worker-boot-disk-size           50 \
-    --secondary-worker-boot-disk-size 50 \
-    --master-boot-disk-type           pd-ssd \
-    --worker-boot-disk-type           pd-ssd \
-    --secondary-worker-boot-disk-type pd-ssd \
-    --format=json \
-    --region "${REGION}" \
-    --zone "${ZONE}" \
-    --subnet "${SUBNET}" \
-    --no-address \
-    --service-account="${GSA}" \
-    --tags="${TAGS}" \
-    --bucket "${BUCKET}" \
-    --enable-component-gateway \
-    --metadata "public_secret_name=${public_secret_name}" \
-    --metadata "private_secret_name=${private_secret_name}" \
-    --metadata "secret_project=${secret_project}" \
-    --metadata "secret_version=${secret_version}" \
-    --metadata "modulus_md5sum=${modulus_md5sum}" \
-    --metadata cuda-version="${CUDA_VERSION}" \
-    --metadata "install-gpu-agent=true" \
-    --metadata "gpu-driver-provider=NVIDIA" \
-    --metadata "gpu-conda-env=dpgce" \
-    --metadata "rapids-mirror-disk=${RAPIDS_MIRROR_DISK_NAME}" \
-    --metadata "rapids-mirror-host=${RAPIDS_REGIONAL_MIRROR_ADDR[${REGION}]}"   \
-    --metadata "init-actions-repo=${INIT_ACTIONS_ROOT}" \
-    --metadata "dask-cloud-logging=true" \
-    --metadata dask-runtime="standalone" \
-    --metadata rapids-runtime="SPARK" \
-    --metadata bigtable-instance=${BIGTABLE_INSTANCE} \
-    --metadata include-gpus=1 \
-    --image-version "${IMAGE_VERSION}" \
-    --no-shielded-secure-boot \
-    --initialization-action-timeout=90m \
-    --optional-components DOCKER,JUPYTER \
-    --max-idle="${IDLE_TIMEOUT}" \
-    --properties spark:spark.history.fs.logDirectory=gs://${BUCKET}/phs/eventLog \
-    --scopes 'https://www.googleapis.com/auth/cloud-platform,sql-admin'
-  date
-  set +x
-}
-
-#    --metadata include-pytorch=1 \
-#    --properties "hive:hive.metastore.warehouse.dir=gs://${HIVE_DATA_BUCKET}/hive-warehouse" \
-#    --metadata "hive-metastore-instance=${PROJECT_ID}:${REGION}:${HIVE_INSTANCE_NAME}" \
-#    --metadata "db-hive-password-uri=gs://${BUCKET}/dataproc-initialization-actions/mysql_hive_password.encrypted" \
-#    --metadata "kms-key-uri=projects/${PROJECT_ID}/locations/global/keyRings/keyring-cluster-1668020639/cryptoKeys/kdc-root-cluster-1668020639" \
-#
-#    --properties "hive:hive.metastore.warehouse.dir=gs://${HIVE_DATA_BUCKET}/hive-warehouse" \
-#  local PIP_PACKAGES='tokenizers==0.10.1,datasets==1.5.0'
-#  local CONDA_PACKAGES='pytorch==1.0.1,visions==0.7.1'
-#  local PIP_PACKAGES='pymysql==1.1.0,pandas-gbq==0.26.1,google-cloud-secret-manager==2.17.0'
-#  local CONDA_PACKAGES="${PIP_PACKAGES}"
-#    --properties=^:^dataproc:conda.packages="${CONDA_PACKAGES}":dataproc:pip.packages="${PIP_PACKAGES}" \
-#    --metadata "hive-metastore-instance=${PROJECT_ID}:${REGION}:${HIVE_INSTANCE_NAME}" \
-#    --metadata "db-hive-password-uri=gs://${BUCKET}/dataproc-initialization-actions/mysql_hive_password.encrypted" \
-#    --metadata "kms-key-uri=projects/${PROJECT_ID}/locations/global/keyRings/keyring-cluster-1668020639/cryptoKeys/kdc-root-cluster-1668020639" \
-#    --initialization-actions "${INIT_ACTIONS_ROOT}/cloud-sql-proxy/cloud-sql-proxy.sh,${INIT_ACTIONS_ROOT}/ccaas_init.sh" \
-#    --initialization-actions "${INIT_ACTIONS_ROOT}/gpu/install_gpu_driver.sh" \
-#    --num-masters=1 \
-#    --num-workers=2 \
-#    --master-machine-type "${MASTER_MACHINE_TYPE}" \
-#    --worker-machine-type "${PRIMARY_MACHINE_TYPE}" \
-#    --metadata cuda-url="https://developer.download.nvidia.com/compute/cuda/12.4.1/local_installers/cuda_12.4.1_550.54.15_linux.run" \
-#    --metadata gpu-driver-url="https://us.download.nvidia.com/XFree86/Linux-x86_64/550.135/NVIDIA-Linux-x86_64-550.135.run" \
-#    --metadata gpu-driver-version="${DRIVER_VERSION}" \
-#    --initialization-actions ${INIT_ACTIONS_ROOT}/gpu/install_gpu_driver.sh,${INIT_ACTIONS_ROOT}/rapids/rapids.sh \
-#    --initialization-actions ${INIT_ACTIONS_ROOT}/rapids/rapids.sh \
-#    --metadata rapids-runtime="SPARK" \
-#    --worker-accelerator "type=${PRIMARY_ACCELERATOR_TYPE}" \
-#    --master-accelerator "type=${MASTER_ACCELERATOR_TYPE}" \
-#    --single-node \
-#    --num-masters=1 \
-#    --num-workers=2 \
-#    --metadata cuda-version="${CUDA_VERSION}" \
-#    --image "projects/${PROJECT_ID}/global/images/cuda-pre-init-2-0-ubuntu18-2024-12-24-20-42" \
-#    --metadata cuda-url="https://developer.download.nvidia.com/compute/cuda/12.4.1/local_installers/cuda_12.4.1_550.54.15_linux.run" \
-#    --metadata cuda-url="https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda_11.8.0_520.61.05_linux.run" \
-#    --metadata cuda-url="https://developer.download.nvidia.com/compute/cuda/12.4.0/local_installers/cuda_12.4.0_550.54.14_linux.run" \
-#    --image "projects/${PROJECT_ID}/global/images/cuda-pre-init-2-2-rocky9-2024-12-01-04-21" \
-#    --initialization-actions "${INIT_ACTIONS_ROOT}/gpu/install_gpu_driver.sh" \
-#    --image "projects/${PROJECT_ID}/global/images/cuda-pre-init-2-1-debian11-2024-10-31-07-41" \
-#    --initialization-actions "${INIT_ACTIONS_ROOT}/gpu/install_gpu_driver.sh" \
-#    --image-version "${IMAGE_VERSION}" \
-#    --image "projects/${PROJECT_ID}/global/images/cuda-pre-init-2-2-debian12-2024-10-24-09-12" \
-#    --image "projects/${PROJECT_ID}/global/images/rapids-pre-init-2-2-debian12-2024-10-15-02-54" \
-#    --image "projects/${PROJECT_ID}/global/images/rapids-pre-init-2-2-debian12-2024-10-15-02-54" \
-#    --initialization-actions "${INIT_ACTIONS_ROOT}/dask/dask.sh,${INIT_ACTIONS_ROOT}/rapids/rapids.sh" \
-#    --metadata dask-runtime="yarn" \
-#    --metadata dask-runtime="standalone" \
-#    --image-version "${IMAGE_VERSION}" \
-#    --no-shielded-secure-boot \
-#    --image "projects/${PROJECT_ID}/global/images/custom-2-2-debian12-2024-10-04" \
-#    --initialization-actions "${INIT_ACTIONS_ROOT}/sf-env-setup.sh" \
-#    --num-masters=1 \
-#    --num-workers=2 \
-#    --metadata cuda-version="${CUDA_VERSION}" \
-#    --initialization-actions "${INIT_ACTIONS_ROOT}/gpu/install_gpu_driver.sh,${INIT_ACTIONS_ROOT}/dask/dask.sh,${INIT_ACTIONS_ROOT}/rapids/rapids.sh" \
-#    --num-masters=1 \
-#    --num-workers=2 \
-#    --initialization-actions "${INIT_ACTIONS_ROOT}/gpu/install_gpu_driver.sh" \
-#    --image "projects/${PROJECT_ID}/global/images/custom-2-2-debian12-2024-07-27" \
-#    --image "projects/${PROJECT_ID}/global/images/custom-2-2-ubuntu22-2024-07-27" \
-#    --image "projects/${PROJECT_ID}/global/images/custom-2-2-rocky9-2024-07-27" \
-
-
-#    --metadata dask-runtime="standalone" \
-#    --initialization-actions "${INIT_ACTIONS_ROOT}/bigtable/bigtable.sh" \
-#    --worker-accelerator "type=${ACCELERATOR_TYPE}" \
-#    --initialization-actions "${INIT_ACTIONS_ROOT}/gpu/install_gpu_driver.sh" \
-#    --image "projects/${PROJECT_ID}/global/images/nvidia-open-kernel-bookworm-2024-06-26" \
-#    --image-version "${IMAGE_VERSION}" \
-#    --image "projects/${PROJECT_ID}/global/images/nvidia-open-kernel-bookworm-2024-06-21-a" \
-#    --no-shielded-secure-boot \
-#    --image-version "${IMAGE_VERSION}" \
-#    --initialization-actions "${INIT_ACTIONS_ROOT}/gpu/install_gpu_driver.sh" \
-
-#
-# GPU
-#
-    # --optional-components JUPYTER,ZOOKEEPER \
-    # --worker-accelerator type=${ACCELERATOR_TYPE} \
-    # --master-accelerator type=${ACCELERATOR_TYPE} \
-    # --metadata include-gpus=true \
-    # --metadata gpu-driver-provider=NVIDIA \
-    # --metadata install-gpu-agent=true \
-    # --initialization-action-timeout=15m \
-    # --initialization-actions ${INIT_ACTIONS_ROOT}/gpu/install_gpu_driver.sh \
-    # --properties='^#^dataproc:conda.packages=ipython-sql==0.3.9,pyhive==0.6.5' \
-    # --properties spark:spark.executor.resource.gpu.amount=1,spark:spark.task.resource.gpu.amount=1 \
-    # --properties "presto-catalog:bigquery_my_other_project.connector.name"="bigquery" \
-
-
-
-#    --image https://www.googleapis.com/compute/v1/projects/cloud-dataproc-ci/global/images/dataproc-2-0-debian10-20240314-212221-rc99 \
-#    --initialization-actions "${INIT_ACTIONS_ROOT}/hue/hue.sh" \
-#     --metadata startup-script-url="${INIT_ACTIONS_ROOT}/delay-masters-startup.sh" \
-#     --scopes 'https://www.googleapis.com/auth/cloud-platform,sql-admin'
-
-#
-# DASK rapids
-#
-    # --metadata rapids-runtime=DASK \
-    # --metadata cuda-version="${CUDA_VERSION}" \
-    # --metadata rapids-version="22.04" \
-    # --initialization-actions ${INIT_ACTIONS_ROOT}/gpu/install_gpu_driver.sh,${INIT_ACTIONS_ROOT}/rapids/rapids.sh \
-
-#     --metadata ="$(perl -n -e '@l=<STDIN>; chomp @l; print join q{,}, @l' <init/dataproc.${CASE_NUMBER}.metadata)" \
-
-#    --max-idle=${IDLE_TIMEOUT} \
-
-# 20240724 vvv  prior ^^^
-
-    # --initialization-actions "${INIT_ACTIONS_ROOT}/gpu/install_gpu_driver.sh" \
-    # --metadata rapids-runtime="DASK" \
-    # --metadata rapids-version="23.12" \
-    # --metadata install-gpu-agent="true" \
-    # --metadata cuda-version="${CUDA_VERSION}" \
-    # --metadata "public_secret_name=efi-db-pub-key-005" \
-    # --metadata "private_secret_name=efi-db-priv-key-005" \
-    # --metadata "secret_project=${PROJECT_ID}" \
-    # --metadata "secret_version=1" \
-    # --metadata "modulus_md5sum=1bd7778c62497c257ea1cd0c766a593e" \
-
-
-
-
-#
-# bigtable
-#
-    # --initialization-actions "${INIT_ACTIONS_ROOT}/bigtable/bigtable.sh" \
-    # --metadata bigtable-instance=${BIGTABLE_INSTANCE} \
-    # --initialization-action-timeout=15m \
-
-#
-# Oozie
-#
-#    --metadata startup-script-url="${INIT_ACTIONS_ROOT}/delay-masters-startup.sh" \
-#    --initialization-actions "${INIT_ACTIONS_ROOT}/oozie/oozie.sh" \
-#    --properties "dataproc:dataproc.master.custom.init.actions.mode=RUN_AFTER_SERVICES" \
-
-#
-# Livy
-#
-#    --initialization-actions "${INIT_ACTIONS_ROOT}/livy/livy.sh" \
-
-# complex init actions on 2.1 repro
-
-    # --metadata startup-script-url="${INIT_ACTIONS_ROOT}/delay-masters-startup.sh" \
-    # --initialization-actions "${INIT_ACTIONS_ROOT}/oozie/oozie.sh,${INIT_ACTIONS_ROOT}/bigtable/bigtable.sh,${INIT_ACTIONS_ROOT}/sqoop/sqoop.sh" \
-    # --initialization-action-timeout=15m \
-    # --optional-components ZOOKEEPER \
-    # --metadata "hive-metastore-instance=${PROJECT_ID}:${REGION}:${HIVE_INSTANCE_NAME}" \
-    # --metadata hive-cluster-name="${HIVE_CLUSTER_NAME}" \
-    # --metadata bigtable-instance="${BIGTABLE_INSTANCE}" \
-    # --metadata bigtable-project="${PROJECT_ID}" \
-    # --properties="^~~^$(perl -n -e '@l=<STDIN>; chomp @l; print join q{~~}, @l' <init/dataproc.${CASE_NUMBER}.properties)" \
-    # --scopes 'https://www.googleapis.com/auth/cloud-platform,sql-admin'
-
-#    --enable-component-gateway \
-#    --metadata startup-script-url="${INIT_ACTIONS_ROOT}/delay-masters-startup.sh" \
-#    --initialization-actions "${INIT_ACTIONS_ROOT}/oozie/oozie.sh,${INIT_ACTIONS_ROOT}/bigtable/bigtable.sh,${INIT_ACTIONS_ROOT}/sqoop/sqoop.sh" \
-#    --initialization-action-timeout=15m \
-#    --properties "dataproc:dataproc.master.custom.init.actions.mode=RUN_AFTER_SERVICES" \
-#    --initialization-action-timeout=15m \
-#    --metadata bigtable-instance=${BIGTABLE_INSTANCE} \
-
-#     --metadata startup-script-url="${INIT_ACTIONS_ROOT}/delay-masters-startup.sh" \
-#     --properties "dataproc:dataproc.master.custom.init.actions.mode=RUN_AFTER_SERVICES" \
-
-#    --initialization-actions "${INIT_ACTION_PATHS}" \
-#    --initialization-actions ${INIT_ACTIONS_ROOT}/oozie/oozie.sh,${INIT_ACTIONS_ROOT}/bigtable/bigtable.sh,${INIT_ACTIONS_ROOT}/sqoop/sqoop.sh \
-#    --initialization-actions ${INIT_ACTIONS_ROOT}/oozie/oozie.sh \
-
-# [serial] new initialization action to execute in serial #1086
-# https://github.com/GoogleCloudDataproc/initialization-actions/pull/1086
-#    --initialization-actions ${INIT_ACTIONS_ROOT}/serial/serial.sh \
-#    --metadata "initialization-action-paths=${INIT_ACTION_PATHS}" \
-#    --metadata "initialization-action-paths-separator=${PATH_SEPARATOR}" \
-
-
-    # --worker-boot-disk-type pd-ssd \
-    # --worker-boot-disk-size 50 \
-    # --worker-machine-type ${MACHINE_TYPE} \
-    # --num-masters=3 \
-    # --num-workers=3 \
-
-
-  #    --metadata spark-bigquery-connector-version=0.31.1 \
-
-    # --worker-boot-disk-type pd-ssd \
-    # --worker-boot-disk-size 50 \
-    # --worker-machine-type ${MACHINE_TYPE} \
-    # --num-workers=3 \
-
-#      --single-node \
-
-
-    # --no-shielded-secure-boot \
-    # --worker-accelerator type=${ACCELERATOR_TYPE} \
-    # --master-accelerator type=${ACCELERATOR_TYPE} \
-    # --metadata include-gpus=true \
-    # --metadata gpu-driver-provider=NVIDIA \
-    # --initialization-action-timeout=15m \
-    # --initialization-actions ${INIT_ACTIONS_ROOT}/gpu/install_gpu_driver.sh \
-    # --properties spark:spark.executor.resource.gpu.amount=1,spark:spark.task.resource.gpu.amount=1 \
-    # --metadata init-actions-repo=${INIT_ACTIONS_ROOT} \
-    # --metadata install-gpu-agent=true \
-    # --metadata cuda-version=${CUDA_VERSION} \
-
-#,${INIT_ACTIONS_ROOT}/nvidia_docker.sh \
-  #    --max-idle=${IDLE_TIMEOUT} \
-#    --initialization-action-timeout=15m \
-#    --worker-accelerator type=${ACCELERATOR_TYPE} \
-#    --master-accelerator type=${ACCELERATOR_TYPE} \
-  #     --metadata gpu-driver-provider=NVIDIA \
-    # --initialization-actions ${INIT_ACTIONS_ROOT}/gpu/install_gpu_driver.sh \
-
-#    --image-version 1.4.80-ubuntu18 \
-
-
-# Increase log verbosity
-#   --verbosity=debug \
-
-#
-# Kafka
-#
-
-#    --optional-components ZOOKEEPER \
-#    --metadata "run-on-master=true" \
-#    --initialization-actions gs://goog-dataproc-initialization-actions-${REGION}/kafka/kafka.sh \
-
-#
-# Hive
-#
-
-# tested 20240719
-#    --initialization-actions "${INIT_ACTIONS_ROOT}/cloud-sql-proxy/cloud-sql-proxy.sh" \
-#    --properties "hive:hive.metastore.warehouse.dir=gs://${HIVE_DATA_BUCKET}/hive-warehouse" \
-#    --metadata "hive-metastore-instance=${PROJECT_ID}:${REGION}:${HIVE_INSTANCE_NAME}" \
-#    --metadata "db-hive-password-uri=gs://${BUCKET}/dataproc-initialization-actions/mysql_hive_password.encrypted" \
-#    --metadata "kms-key-uri=projects/${PROJECT_ID}/locations/global/keyRings/keyring-cluster-1668020639/cryptoKeys/kdc-root-cluster-1668020639" \
-
-
-
-#
-#    --metadata "hive-metastore-instance=${PROJECT_ID}:${REGION}:${HIVE_INSTANCE_NAME}" \
-#    --metadata hive-cluster-name="${HIVE_CLUSTER_NAME}" \
-#    --metadata bigtable-instance="${BIGTABLE_INSTANCE}" \
-#    --metadata bigtable-project="${PROJECT_ID}" \
-#    --properties="^~~^$(perl -n -e '@l=<STDIN>; chomp @l; print join q{~~}, @l' <init/dataproc.${CASE_NUMBER}.properties)" \
-
-#    --initialization-actions gs://goog-dataproc-initialization-actions-${REGION}/cloud-sql-proxy/cloud-sql-proxy.sh \
-#    --properties hive:hive.metastore.warehouse.dir=gs://${HIVE_DATA_BUCKET}/hive-warehouse \
-#    --metadata "hive-metastore-instance=${PROJECT_ID}:${REGION}:${HIVE_INSTANCE_NAME}" \
-#    --scopes 'https://www.googleapis.com/auth/cloud-platform,sql-admin'
-
-#    --single-node \
-
-#    --initialization-actions ${INIT_ACTIONS_ROOT}/cloud-sql-proxy/cloud-sql-proxy.sh \
-#    --num-workers=2 \
-#    --autoscaling-policy=${AUTOSCALING_POLICY_NAME} \
-#    --scopes 'https://www.googleapis.com/auth/cloud-platform'
-
-#
-#  Cluster options
-#    --optional-components FLINK \
-#    --num-worker-local-ssds 4 \
-#     --initialization-actions ${INIT_ACTIONS_ROOT}/fail.sh \
-#    --optional-components FLINK \
-
-#    --optional-components JUPYTER,ZOOKEEPER \
-#    --properties="^~~^$(perl -n -e '@l=<STDIN>; chomp @l; print join q{~~}, @l' <init/${CASE_NUMBER}/dataproc.properties)"
-#    --initialization-actions ${INIT_ACTIONS_ROOT}/init-ab0bffe.bash \
-#    --initialization-actions ${INIT_ACTIONS_ROOT}/s-agent-dm-filterer.sh \
-#
-#    --num-masters=3 \
-#    --num-workers=2 \
-#    --single-node \
-#    --master-machine-type n1-standard-4 \
-#    --master-boot-disk-type pd-ssd \
-#    --master-boot-disk-size 50 \
-#    --num-master-local-ssds 4 \
-#    --worker-machine-type n1-standard-4 \
-#    --worker-boot-disk-type pd-ssd \
-#    --worker-boot-disk-size 200 \
-#    --num-secondary-workers 2 \
-#    --secondary-worker-boot-disk-size 50 \
-#    --secondary-worker-boot-disk-type pd-standard \
-#    --secondary-worker-type preemptible \
-#    --bucket ${BUCKET} \
-#    --max-idle=30m \
-#    --enable-component-gateway \
-#    --optional-components JUPYTER,ZOOKEEPER \
-#    --properties="${PROPERTIES}" \
-#    --properties="core:fs.defaultFS=gs://${BUCKET}" \
-#    --properties="yarn:yarn.log-aggregation-enable=true" \
-#    --properties="^~~^$(perl -n -e '@l=<STDIN>; chomp @l; print join q{~~}, @l' <init/dataproc.properties)" \
-#    --properties-file=${INIT_ACTIONS_ROOT}/dataproc.properties \
-
-#
-#  Test startup scripts and init action scripts
-#
-#    --metadata startup-script-url="${INIT_ACTIONS_ROOT}/startup-script.pl" \
-#    --metadata startup-script-url="${INIT_ACTIONS_ROOT}/startup-script.sh" \
-#    --metadata startup-script-url="${INIT_ACTIONS_ROOT}/env-change.sh" \
-#    --metadata ^~~^startup-script="$(cat init/startup-script.pl)" \
-#    --metadata ^~~^startup-script="$(cat init/startup-script.sh)" \
-#    --metadata ^~~^startup-script="$(cat init/startup-script-logsrc.sh)" \
-#    --initialization-actions ${INIT_ACTIONS_ROOT}/startup-script.pl \
-#    --initialization-actions ${INIT_ACTIONS_ROOT}/startup-script.sh \
-#    --initialization-actions ${INIT_ACTIONS_ROOT}/sqoop/sqoop.sh \
-#    --initialization-actions ${INIT_ACTIONS_ROOT}/kernel/upgrade-kernel.sh \
-#    --metadata startup-script='#!/bin/bash
-# echo "hello world xyz"
-# ' \
-#     --metadata startup-script='#!/bin/bash
-# CONNECTOR_JARS=$(find / -iname "*spark-bigquery-connector*.jar")
-# for jar in ${CONNECTOR_JARS}; do rm $(realpath ${jar}); done
-# rm ${CONNECTOR_JARS}
-#  ' \
-
-
-#
-# bigtable
-#
-#    --metadata bigtable-instance=${BIGTABLE_INSTANCE} \
-#    --initialization-actions ${INIT_ACTIONS_ROOT}/bigtable/bigtable.sh \
-#
-#
-#  Oozie
-#
-#    --initialization-actions ${INIT_ACTIONS_ROOT}/oozie/oozie.sh \
-
-#
-#  Conda
-#
-#    --properties="dataproc:conda.env.config.uri=${INIT_ACTIONS_ROOT}/environment.yaml" \
-#    --single-node \
-#    --properties="dataproc:dataproc.logging.stackdriver.enable=true" \
-#    --properties="dataproc:dataproc.logging.stackdriver.job.driver.enable=true" \
-#    --properties="dataproc:dataproc.logging.stackdriver.job.yarn.container.enable=true" \
-#    --properties="dataproc:dataproc.monitoring.stackdriver.enable=true" \
-
-
-#
-# NVIDIA
-#
-#    --metadata include-gpus=true \
-#    --worker-accelerator type=${ACCELERATOR_TYPE} \
-#    --master-accelerator type=${ACCELERATOR_TYPE} \
-#    --metadata gpu-driver-provider=NVIDIA \
-#    --metadata init-actions-repo=${INIT_ACTIONS_ROOT} \
-#    --metadata install-gpu-agent=true \
-#    --metadata cuda-version=${CUDA_VERSION} \
-
-
-#
-# NVidia Docker on yarn
-#
-#    --metadata yarn-docker-image=${YARN_DOCKER_IMAGE} \
-#    --optional-components DOCKER \
-#    --initialization-actions ${INIT_ACTIONS_ROOT}/kernel/upgrade-kernel.sh \
-#    --initialization-actions ${INIT_ACTIONS_ROOT}/util/upgrade-kernel.sh,${INIT_ACTIONS_ROOT}/gpu/install_gpu_driver.sh,${INIT_ACTIONS_ROOT}/nvidia_docker.sh \
-
-#
-# Presto
-#
-#    --optional-components PRESTO \
-
-#
-# HBase
-#
-#    --optional-components HBASE,ZOOKEEPER \
-
-# Flink
-#    --optional-components FLINK \
-
-#
-# Rapids
-#
-#    --metadata rapids-runtime=SPARK \
-#    --metadata rapids-runtime=DASK \
-#    --initialization-actions ${INIT_ACTIONS_ROOT}/spark-rapids/spark-rapids.sh \
-
-#
-# passing properties (to a job only) in a file
-#
-# --properties-file=${INIT_ACTIONS_ROOT}/dataproc.properties
-
-#    cloud-dataproc-ci.googleapis.com \ # DPMS dependency?
-function enable_services () {
-  # Enable Dataproc service
-  set -x
-  gcloud services enable \
-    dataproc.googleapis.com \
-    compute.googleapis.com \
-    secretmanager.googleapis.com \
-    --project=${PROJECT_ID}
-  set +x
-}
-
-REPRO_TMPDIR="$(mktemp -d /tmp/dataproc-repro-XXXXX)"
-function cleanup_repro_tmpdir() {
-  rm -rf "${REPRO_TMPDIR}"
-}
-
-function exit_handler() {
-  cleanup_repro_tmpdir
-}
-
-trap exit_handler EXIT
-
-function exists_dpgce_cluster() {
-  set +x
-  test -f cluster-list.json || gcloud dataproc clusters list --format=json | dd of="${REPRO_TMPDIR}/cluster-list.json"
-  DPGCE_CLUSTER="$()"
-  JQ_CMD=".[] | select(.clusterName | test(\"${CLUSTER_NAME}$\"))"
-  OUR_CLUSTER=$(cat ${REPRO_TMPDIR}/cluster-list.json | jq -c "${JQ_CMD}")
-
-  if [[ -z "${OUR_CLUSTER}" ]]; then
-    return -1
-  else
-    echo "cluster exists"
-    return 0
-  fi
-}
-
-function delete_dpgce_cluster() {
-  set -x
-  if exists_dpgce_cluster == 0; then
-    echo "dpgce cluster exists"
-  else
-    echo "dpgce cluster does not exist.  Not deleting"
-    set +x
-    return 0
-  fi
-
-  gcloud dataproc clusters delete --quiet --region ${REGION} ${CLUSTER_NAME}
-  set +x
-  echo "cluster deleted"
-}
-
-DATE=$(date +%s)
-function set_cluster_name() {
-  cluster_name="$(jq -r .CLUSTER_NAME env.json)"
-
-  # Only update cluster if it is not set
-  if [[ -z "${cluster_name}" || "${cluster_name}" == "null"  ]]; then
-    export CLUSTER_NAME="cluster-${DATE}"
-  else
-    echo "CLUSTER_NAME already set to [${cluster_name}]"
-  fi
-
-  # env.json is the source of truth.  modify and reload env.sh
-  cp env.json env.json.tmp
-  cat env.json.tmp | jq ".CLUSTER_NAME |= \"${CLUSTER_NAME}\"" > env.json
-  rm env.json.tmp
-  source lib/env.sh
-}
-
-function create_project(){
-  set -x
-  local PROJ_DESCRIPTION=$(gcloud projects describe ${PROJECT_ID} --format json 2>/dev/null)
-  if [[ -n ${PROJ_DESCRIPTION} && "$(echo $PROJ_DESCRIPTION | jq -r .lifecycleState)" == "ACTIVE" ]]; then
-    echo "project already exists!"
-    return
-  else
-    local LCSTATE="$(echo $PROJ_DESCRIPTION | jq .lifecycleState)"
-    if [[ -n ${PROJ_DESCRIPTION} && "$(echo $PROJ_DESCRIPTION | jq -r .lifecycleState)" == "DELETE_REQUESTED" ]]; then
-      gcloud projects undelete --quiet ${PROJECT_ID}
-    else
-      gcloud projects create ${PROJECT_ID} --folder ${FOLDER_NUMBER}
-    fi
-
-    if [[ $? -ne 0 ]]; then
-      echo "could not create project."
-      exit -1
-    fi
-
-    local PRJBA=$(gcloud beta billing projects describe ${PROJECT_ID} --format json | jq -r .billingAccountName)
-
-    # link project to billing account
-    if [[ -z "${PRJBA}" ]]; then
-      set +x
-      echo "
-The following variable values were read from env.json
-PROJECT_ID=${PROJECT_ID}
-BILLING_ACCOUNT=${BILLING_ACCOUNT}
-PRIV_DOMAIN=${PRIV_DOMAIN}
-DOMAIN=${DOMAIN}
-USER=${USER}
-
-https://cloud.google.com/billing/docs/how-to/billing-access
-
-Please be prepared to link the project ${PROJECT_ID} to the billing
-account ${BILLING_ACCOUNT}.  The principal indicated by
-${PRIV_USER}@${PRIV_DOMAIN} must have roles/billing.admin as
-documented in above link.
-
-Please encode the principal with rights to modify billing details as
-PRIV_USER and PRIV_DOMAIN in your env.json file.
-
-If you have not yet done this, please cancel this operation (^C),
-modify your env.json file to include the privileged principal user and
-domain, and then re-run this command.
-
-gcloud beta billing projects \
-  link ${PROJECT_ID} --billing-account ${BILLING_ACCOUNT}
-
-once you have credentials to run the above command,
-
-Press enter > 
-"
-      read
-
-      local active_account=$(gcloud auth list 2>/dev/null | awk '/^\*/ {print $2}')
-      while [[ $active_account != "${USER}@${PRIV_DOMAIN}" ]]; do
-        echo "AUTHENTICATE AS A USER WITH PRIVILEGES TO LINK THESE PROJECTS"
-        gcloud auth login ${USER}@${PRIV_DOMAIN}
-        local active_account=$(gcloud auth list 2>/dev/null | awk '/^\*/ {print $2}')
-      done
-
-      set -x
-      execute_with_retries "gcloud beta billing projects link ${PROJECT_ID} --billing-account ${BILLING_ACCOUNT}"
-      set +x
-      if [[ $? != 0 ]]; then
-        echo "failed to link project and billing account"
-        exit -1
-      fi
-
-      echo "Prepare to log in with your @${DOMAIN} account and then press enter..."
-      read
-
-      local active_account=$(gcloud auth list 2>/dev/null | awk '/^\*/ {print $2}')
-      while [[ $active_account != "${USER}@${DOMAIN}" ]]; do
-        echo "AUTHENTICATE AS YOUR @${DOMAIN} EMAIL ADDRESS"
-        gcloud auth login ${USER}@${DOMAIN}
-        local active_account=$(gcloud auth list 2>/dev/null | awk '/^\*/ {print $2}')
-      done
-    fi
-  fi
-  set +x
-  echo "project created!"
-}
-
-function delete_project() {
-  set -x
-  local active_account=$(gcloud auth list 2>/dev/null | awk '/^\*/ {print $2}')
-  while [[ $active_account != "${USER}@${PRIV_DOMAIN}" ]]; do
-    echo "AUTHENTICATE AS YOUR @${PRIV_DOMAIN} EMAIL ADDRESS"
-    gcloud auth login ${USER}@${PRIV_DOMAIN}
-    local active_account=$(gcloud auth list 2>/dev/null | awk '/^\*/ {print $2}')
-  done
-  gcloud beta billing projects unlink ${PROJECT_ID}
-  local active_account=$(gcloud auth list 2>/dev/null | awk '/^\*/ {print $2}')
-  while [[ $active_account != "${USER}@${DOMAIN}" ]]; do
-    echo "AUTHENTICATE AS YOUR @${DOMAIN} EMAIL ADDRESS"
-    gcloud auth login ${USER}@${DOMAIN}
-    local active_account=$(gcloud auth list 2>/dev/null | awk '/^\*/ {print $2}')
-  done
-  gcloud projects delete --quiet ${PROJECT_ID}
-  set +x
-  echo "project deleted!"
-}
-
-function configure_gcloud() {
-  gcloud config set compute/region ${REGION}
-  gcloud config set compute/zone ${ZONE}
-  gcloud config set core/project ${PROJECT_ID}
-}
-
-function grant_kms_roles(){
-  set -x
-
-  gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
-    --member="serviceAccount:${GSA}" \
-    --role=roles/cloudkms.cryptoKeyDecrypter \
-
-  set +x
-  echo "dpgke service account roles granted"
-}
-
-function grant_mysql_roles(){
-  set -x
-
-  gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
-    --member="serviceAccount:${GSA}" \
-    --role=roles/cloudsql.editor \
-
-  set +x
-  echo "cloudsql service account editor role granted"
-}
-
-function create_mysql_admin_password() {
-    dd if=/dev/urandom bs=8 count=4 | xxd -p | \
-      gcloud kms encrypt \
-      --location=global \
-      --keyring=projects/${PROJECT_ID}/locations/global/keyRings/${KMS_KEYRING} \
-      --key=projects/${PROJECT_ID}/locations/global/keyRings/${KMS_KEYRING}/cryptoKeys/${KDC_ROOT_PASSWD_KEY} \
-      --plaintext-file=- \
-      --ciphertext-file=init/mysql_admin_password.encrypted
-}
-
-
-function grant_bigtables_roles(){
-  set -x
-
-  gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
-    --member="serviceAccount:${GSA}" \
-    --role=roles/bigtable.user \
-
-  gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
-    --member="serviceAccount:${GSA}" \
-    --role=roles/bigtable.admin \
-
-  set +x
-  echo "dpgke service account roles granted"
-}
-
-
-function create_kms_keyring() {
-  set -x
-  if (gcloud kms keyrings list --location global | grep ${KMS_KEYRING}); then
-    echo "keyring already exists"
-  else
-    gcloud kms keyrings create ${KMS_KEYRING} --location=global
-    echo "kms keyring created"
-  fi
-
-  set +x
-}
-
-function create_kerberos_kdc_key() {
-  set -x
-  if (gcloud kms keys list --location global --keyring=${KMS_KEYRING} | grep ${KDC_ROOT_PASSWD_KEY}); then
-    echo "kerberos kdc key exists"
-  else
-    gcloud kms keys create ${KDC_ROOT_PASSWD_KEY} \
-      --location=global \
-      --keyring=${KMS_KEYRING} \
-      --purpose=encryption
-    echo "kerberos kdc key created"
-  fi
-  set +x
-}
-
-function create_kerberos_kdc_password() {
-  set -x
-  if [[ -f init/${KDC_ROOT_PASSWD_KEY}.encrypted ]]; then
-    echo "password exists"
-  else
-    dd if=/dev/urandom bs=8 count=4 | xxd -p | \
-      gcloud kms encrypt \
-      --location=global \
-      --keyring=${KMS_KEYRING} \
-      --key=${KDC_ROOT_PASSWD_KEY} \
-      --plaintext-file=- \
-      --ciphertext-file=init/${KDC_ROOT_PASSWD_KEY}.encrypted
-  fi
-  set +x
-}
-
-function create_kerberos_sa_password() {
-    dd if=/dev/urandom bs=8 count=4 | xxd -p | \
-      gcloud kms encrypt \
-      --location=global \
-      --keyring=${KMS_KEYRING} \
-      --key=${KDC_ROOT_PASSWD_KEY} \
-      --plaintext-file=- \
-      --ciphertext-file=init/${KDC_SA_PASSWD_KEY}.encrypted
-}
-
-function create_kdc_server() {
-  # Authors: oklev@softserveinc.com
-  set -x
-
-  local METADATA="kdc-root-passwd=${INIT_ACTIONS_ROOT}/${KDC_ROOT_PASSWD_KEY}.encrypted"
-  METADATA="${METADATA},kms-keyring=${KMS_KEYRING}"
-  METADATA="${METADATA},kdc-root-passwd-key=${KDC_ROOT_PASSWD_KEY}"
-  METADATA="${METADATA},startup-script-url=${INIT_ACTIONS_ROOT}/kdc-server.sh"
-  METADATA="service-account-user=${GSA}"
-  # Spin up a KDC server
-  gcloud compute instances create ${KDC_NAME} \
-    --zone ${ZONE} \
-    --subnet ${SUBNET} \
-    --service-account=${GSA} \
-    --boot-disk-type pd-ssd \
-    --image-family=${KDC_IMAGE_FAMILY} \
-    --image-project=${KDC_IMAGE_PROJECT} \
-    --machine-type=${MACHINE_TYPE} \
-    --scopes='cloud-platform' \
-    --hostname=${KDC_FQDN} \
-    --metadata ${METADATA}
-  set +x
-}
-
-function delete_kdc_server() {
-  set -x
-  gcloud compute instances delete ${KDC_NAME} \
-    --quiet
-  set +x
-  echo "kdc deleted"
-}
-
-function create_kerberos_cluster() {
-  # https://cloud.google.com/dataproc/docs/concepts/components/ranger#installation_steps
-  set -x
-  gcloud dataproc clusters create ${CLUSTER_NAME} \
-    --region ${REGION} \
-    --zone ${ZONE} \
-    --subnet ${SUBNET} \
-    --no-address \
-    --service-account=${GSA} \
-    --master-machine-type n1-standard-4 \
-    --master-boot-disk-type pd-ssd \
-    --master-boot-disk-size 50 \
-    --image-version ${IMAGE_VERSION} \
-    --bucket ${BUCKET} \
-    --initialization-action-timeout=10m \
-    --max-idle=${IDLE_TIMEOUT} \
-    --enable-component-gateway \
-    --scopes='cloud-platform' \
-    --enable-kerberos \
-    --kerberos-root-principal-password-uri="${INIT_ACTIONS_ROOT}/${KDC_ROOT_PASSWD_KEY}.encrypted" \
-    --kerberos-kms-key="${KDC_ROOT_PASSWD_KEY}" \
-    --kerberos-kms-key-keyring=${KMS_KEYRING} \
-    --kerberos-kms-key-location=global \
-    --kerberos-kms-key-project=${PROJECT_ID}
-
-  set +x
-  echo "kerberos cluster created"
-}
-
-function delete_kerberos_cluster() {
-  set -x
-  gcloud dataproc clusters delete --quiet --region ${REGION} ${CLUSTER_NAME}
-  set +x
-  echo "kerberos cluster deleted"
-}
-
-function create_phs_cluster() {
-# does not include "dataproc:job.history.to-gcs.enabled=true,", as this is a dataproc2 image
-  set -x
-  gcloud dataproc clusters create ${CLUSTER_NAME}-phs \
-    --region=${REGION} \
-    --single-node \
-    --image-version=${IMAGE_VERSION} \
-    --subnet=${SUBNET} \
-    --tags=${TAGS} \
-    --properties="spark:spark.history.fs.logDirectory=gs://${PHS_BUCKET},spark:spark.eventLog.dir=gs://${PHS_BUCKET}" \
-    --properties="mapred:mapreduce.jobhistory.read-only.dir-pattern=gs://${MR_HISTORY_BUCKET}" \
-    --enable-component-gateway
-  set +x
-
-  echo "==================="
-  echo "PHS Cluster created"
-  echo "==================="
-}
-
-function delete_phs_cluster() {
-  set -x
-  gcloud dataproc clusters delete --quiet --region ${REGION} ${CLUSTER_NAME}-phs
-  set +x
-  echo "phs cluster deleted"
-}
-
-function create_service_account() {
-  set -x
-  if gcloud iam service-accounts describe "${GSA}" > /dev/null ; then
-    echo "service account ${SA_NAME} already exists"
-    return 0 ; fi
-
-  gcloud iam service-accounts create "${SA_NAME}" \
-    --description="Service account for use with cluster ${CLUSTER_NAME}" \
-    --display-name="${SA_NAME}"
-
-  gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
-    --member="serviceAccount:${GSA}" \
-    --role=roles/dataproc.worker
-
-  gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
-    --member="serviceAccount:${GSA}" \
-    --role=roles/storage.objectCreator
-
-  gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
-    --member="serviceAccount:${GSA}" \
-    --role=roles/storage.objectViewer
-
-  gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
-    --member="serviceAccount:${GSA}" \
-    --role=roles/secretmanager.secretAccessor
-
-  gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
-    --member="serviceAccount:${GSA}" \
-    --role=roles/compute.viewer
-
-  gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
-    --member="serviceAccount:${GSA}" \
-    --role=roles/compute.instanceAdmin.v1
-
-   gcloud iam service-accounts add-iam-policy-binding "${GSA}" \
-    --member="serviceAccount:${GSA}" \
-    --role=roles/iam.serviceAccountUser
-
-  set +x
-  echo "service account created"
-}
-
-function delete_service_account() {
-  set -x
-
-  for svc in spark-executor spark-driver agent ; do
-    gcloud iam service-accounts remove-iam-policy-binding \
-      --role=roles/iam.workloadIdentityUser \
-      --member="serviceAccount:${PROJECT_ID}.svc.id.goog[${DPGKE_NAMESPACE}/${svc}]" \
-      "${GSA}"
-  done
-
-  gcloud projects remove-iam-policy-binding \
-    --role=roles/dataproc.worker \
-    --member="serviceAccount:${GSA}" \
-    "${PROJECT_ID}"
-
-  gcloud projects remove-iam-policy-binding \
-    --role=roles/storage.objectCreator \
-    --member="serviceAccount:${GSA}" \
-    "${PROJECT_ID}"
-
-  gcloud projects remove-iam-policy-binding \
-    --role=roles/storage.objectViewer \
-    --member="serviceAccount:${GSA}" \
-    "${PROJECT_ID}"
-
-  gcloud iam service-accounts delete --quiet "${GSA}"
-
-  set +x
-  echo "service account deleted"
-}
-
-function create_artifacts_repository(){
-  set -x
-  gcloud artifacts repositories create "${ARTIFACT_REPOSITORY}" \
-    --repository-format=docker \
-    --location="${REGION}"
-  set +x
-}
-
-function push_container_image() {
-  gcloud auth print-access-token \
-    --impersonate-service-account "${GSA}" \
-      | docker login \
-	  -u oauth2accesstoken \
-          --password-stdin "https://${REGION}-docker.pgk.dev"
-}
-
-function grant_gke_roles(){
-  set -x
-  for svc in agent spark-driver spark-executor ; do
-    gcloud iam service-accounts add-iam-policy-binding \
-      --role=roles/iam.workloadIdentityUser \
-      --member="serviceAccount:${PROJECT_ID}.svc.id.goog[${DPGKE_NAMESPACE}/${svc}]" \
-      "${GSA}" > /dev/null
-  done
-  echo gcloud artifacts repositories add-iam-policy-binding "${ARTIFACT_REPOSITORY}" \
-      --location="${REGION}" \
-      --member="serviceAccount:${GSA}" \
-      --role=roles/artifactregistry.writer > /dev/null
-  set +x
-  echo "dpgke service account roles granted"
-}
-
-function create_gke_cluster() {
-  set -x
-
-  if gcloud container clusters describe "${GKE_CLUSTER_NAME}" > /dev/null ; then
-    echo "GKE cluster ${GKE_CLUSTER_NAME} already exists"
-  else
-    gcloud container clusters create "${GKE_CLUSTER_NAME}" \
-      --service-account="${GSA}" \
-      --workload-pool="${PROJECT_ID}.svc.id.goog" \
-      --tags ${TAGS} \
-      --subnetwork ${SUBNET} \
-      --network ${NETWORK}
-  fi
-
-  # # Create node pool for control
-  # echo gcloud container node-pools describe "${DP_CTRL_POOLNAME}" \
-  #   --zone "${ZONE}" --cluster "${GKE_CLUSTER_NAME}" > /dev/null \
-  #   && echo "nodepool ${DP_CTRL_POOLNAME} for cluster ${GKE_CLUSTER_NAME} already exists" \
-  # || gcloud container node-pools create "${DP_CTRL_POOLNAME}" \
-  #   --machine-type="e2-standard-4" \
-  #   --zone "${ZONE}" \
-  #   --cluster "${GKE_CLUSTER_NAME}"
-
-  # # Create node pool for drivers
-  # echo gcloud container node-pools describe "${DP_DRIVER_POOLNAME}" \
-  #   --zone "${ZONE}" --cluster "${GKE_CLUSTER_NAME}" > /dev/null \
-  #   && echo "nodepool ${DP_DRIVER_POOLNAME} for cluster ${GKE_CLUSTER_NAME} already exists" \
-  # || gcloud container node-pools create "${DP_DRIVER_POOLNAME}" \
-  #   --machine-type="n2-standard-4" \
-  #   --zone "${ZONE}" \
-  #   --cluster "${GKE_CLUSTER_NAME}"
-
-  # # Create node pool for executors
-  # echo gcloud container node-pools describe "${DP_EXEC_POOLNAME}" \
-  #   --zone "${ZONE}" --cluster "${GKE_CLUSTER_NAME}" > /dev/null \
-  #   && echo "nodepool ${DP_EXEC_POOLNAME} for cluster ${GKE_CLUSTER_NAME} already exists" \
-  # || gcloud container node-pools create "${DP_EXEC_POOLNAME}" \
-  #   --machine-type="n2-standard-8" \
-  #   --zone "${ZONE}" \
-  #   --cluster "${GKE_CLUSTER_NAME}"
-
-
-  set +x
-  echo "gke cluster created"
-}
-
-function delete_gke_cluster() {
-  set -x
-
-  for pn in "${DP_CTRL_POOLNAME}" "${DP_DRIVER_POOLNAME}" "${DP_EXEC_POOLNAME}" ; do
-    gcloud container node-pools delete --quiet ${pn} \
-      --zone ${ZONE} \
-      --cluster ${GKE_CLUSTER_NAME}
-  done
-
-  gcloud container clusters delete --quiet ${GKE_CLUSTER_NAME} --zone ${ZONE}
-
-  set +x
-  echo "gke cluster deleted"
-}
-
-# https://cloud.google.com/dataproc/docs/guides/dpgke/dataproc-gke-nodepools#node_pool_settings
-function create_dpgke_cluster() {
-  set -x
-  gcloud dataproc clusters gke create "${DPGKE_CLUSTER_NAME}" \
-    --region=${REGION} \
-    --gke-cluster=${GKE_CLUSTER} \
-    --spark-engine-version=latest \
-    --staging-bucket=${BUCKET} \
-    --setup-workload-identity \
-    --properties="spark:spark.kubernetes.container.image=${REGION}-docker.pkg.dev/${PROJECT_ID}/dockerfile-dataproc/dockerfile:latest" \
-    --pools="name=${DP_CTRL_POOLNAME},roles=default,machineType=e2-standard-4" \
-    --pools="name=${DP_DRIVER_POOLNAME},min=1,max=3,roles=spark-driver,machineType=n2-standard-4" \
-    --pools="name=${DP_EXEC_POOLNAME},min=1,max=10,roles=spark-executor,machineType=n2-standard-8"
-  set +x
-  echo "dpgke cluster created"
-}
-
-function delete_dpgke_cluster() {
-  set -x
-  echo no such implementation
-  set +x
-}
-
-source lib/database-functions.sh
-source lib/net-functions.sh
-
-function create_bucket () {
-  if gcloud storage ls --buckets "gs://${BUCKET}" ; then
-    echo "bucket already exists, skipping creation."
-    return
-  fi
-  set -x
-  gcloud storage buckets create --location=${REGION} gs://${BUCKET}
-  set +x
-
-  echo "==================="
-  echo "Temp bucket created"
-  echo "==================="
-
-  # Copy initialization action scripts
-  if [ -d init ]
-  then
-    set -x
-    gcloud storage cp --recursive init/* gs://${BUCKET}/dataproc-initialization-actions
-    set +x
-  fi
-
-  echo "==================="
-  echo "init scripts copied"
-  echo "==================="
-
-}
-
-function delete_bucket () {
-  set -x
-  gcloud storage rm --recursive gs://${BUCKET}
-  set +x
-
-  echo "bucket removed"
-}
-
-function create_autoscaling_policy() {
-  set -x
-
-  if gcloud dataproc autoscaling-policies describe ${AUTOSCALING_POLICY_NAME} --region ${REGION} > /dev/null 2>&1; then
-    echo "policy ${AUTOSCALING_POLICY_NAME} already exists"
-  else
-    gcloud dataproc autoscaling-policies import ${AUTOSCALING_POLICY_NAME} --region ${REGION} --source autoscaling-policy.yaml
-  fi
-  set +x
-
-  echo "autoscaling policy created"
-}
-
-
-function delete_autoscaling_policy() {
-  set -x
-
-  if gcloud dataproc autoscaling-policies describe ${AUTOSCALING_POLICY_NAME} --region ${REGION} > /dev/null; then
-    gcloud dataproc autoscaling-policies delete --quiet ${AUTOSCALING_POLICY_NAME} --region ${REGION}
-  else
-    echo "policy ${AUTOSCALING_POLICY_NAME} does not exists"
-  fi
-  set +x
-
-  echo "autoscaling policy created"
-}
-
-function reproduce {
-  set -x
-  # Run some job on the cluster which triggers the failure state
-  # Spark?
-  # Map/Reduce?
-  # repro: steady load 15K containers
-  # zero
-  # containers finish
-  # customer's use case increased to or above 55K pending containers
-  # customer sustained load, increased, completed work, added more work
-  # churns but stays high
-  # as final work is completed
-  # simulate gradual decrease of memory
-  # simulate continued increase of containers until yarn pending memory reaches
-  # When yarn pending memory should reach zero, it instead decreases below 0
-
-  # https://linux.die.net/man/1/stress
-
-  # consider dd if=/dev/zero | launch_job -
-
-  set +x
-}
-
-function get_yarn_applications() {
-  echo "not yet implemented"
-}
-
-function get_jobs_list() {
-
-  if [[ -z ${JOBS_LIST} ]]; then
-    JOBS_LIST="$(gcloud dataproc jobs list --region ${REGION} --format json)"
-  fi
-
-  echo "${JOBS_LIST}"
-}
-
-
-function diagnose {
-  set -x
-  local bdoss_path=${HOME}/src/bigdataoss-internal
-  if [[ ! -d ${bdoss_path}/drproc ]]; then
-    echo "mkdir -p ${bdoss_path} ; pushd ${bdoss_path} ; git clone sso://bigdataoss-internal/drproc ; popd"
-    exit -1
-  fi
-  echo -n "This is going to take some time..."
-  # --job-ids <comma separated Dataproc job IDs>
-  # --yarn-application-ids <comma separated Yarn application IDs>
-  # --stat-time & --end-time
-  DIAGNOSE_CMD="gcloud dataproc clusters diagnose ${CLUSTER_NAME} --region ${REGION}"
-  DIAG_OUT=$(${DIAGNOSE_CMD} 2>&1)
-  echo "Done."
-
-  DIAG_URL=$(echo $DIAG_OUT | perl -ne 'print if m{^gs://.*/diagnostic.tar.gz\s*$}')
-  mkdir -p tmp
-  gcloud storage cp ${DIAG_URL} tmp/
-
-  if [[ ! -f venv/${CLUSTER_NAME}/pyvenv.cfg ]]; then
-    mkdir -p venv/
-    python3 -m venv venv/${CLUSTER_NAME}
-    source venv/${CLUSTER_NAME}/bin/activate
-    python3 -m pip install -r ${bdoss_path}/drproc/requirements.txt
-  else
-    source venv/${CLUSTER_NAME}/bin/activate
-  fi
-
-  python3 ${bdoss_path}/drproc/drproc.py tmp/diagnostic.tar.gz
-
-  set +x
-}
-
-function execute_with_retries() {
-  local -r cmd=$1
-  for ((i = 0; i < 10; i++)); do
-    if eval "$cmd"; then
-      return 0
-    fi
-    sleep 5
-  done
-  return 1
-}
-
-function exists_bigtable_instance() {
-  BIGTABLE_INSTANCES="$(gcloud bigtable instances list --format=json)"
-  JQ_CMD=".[] | select(.name | test(\"${BIGTABLE_INSTANCE}$\"))"
-  OUR_INSTANCE=$(echo ${BIGTABLE_INSTANCES} | jq -c "${JQ_CMD}")
-
-  if [[ -z "${OUR_INSTANCE}" ]]; then
-    return -1
-  else
-    return 0
-  fi
-}
-
-function create_bigtable_instance() {
-  set -x
-  if exists_bigtable_instance == 0; then
-    echo "bigtable instance already exists"
-    set +x
-    return 0
-  fi
-
-  gcloud bigtable instances create ${BIGTABLE_INSTANCE} \
-    --display-name ${BIGTABLE_DISPLAY_NAME} \
-    --cluster-config="${BIGTABLE_CLUSTER_CONFIG}"
-  set +x
-}
-
-function delete_bigtable_instance() {
-  set -x
-
-  gcloud bigtable instances --quiet delete ${BIGTABLE_INSTANCE}
-
-  set +x
-}
-
-function get_cluster_uuid() {
-  get_cluster_json | jq -r .clusterUuid
-}
-
-function get_cluster_json() {
-  #  get_clusters_list | jq ".[] | select(.name | test(\"${BIGTABLE_INSTANCE}$\"))"
-  if [[ -z "${THIS_CLUSTER_JSON}" ]]; then
-    JQ_CMD=".[] | select(.clusterName | contains(\"${CLUSTER_NAME}\"))"
-    THIS_CLUSTER_JSON=$(get_clusters_list | jq -c "${JQ_CMD}")
-  fi
-
-  echo "${THIS_CLUSTER_JSON}"
-}
-
-function get_clusters_list() {
-  if [[ -z ${CLUSTERS_LIST} ]]; then
-    CLUSTERS_LIST="$(gcloud dataproc clusters list --region ${REGION} --format json)"
-  fi
-
-  echo "${CLUSTERS_LIST}"
-}
-
-# https://cloud.google.com/secret-manager/docs/create-secret-quickstart#secretmanager-quickstart-gcloud
-# https://console.cloud.google.com/marketplace/product/google/secretmanager.googleapis.com?q=search&referrer=search&project=${PROJECT_ID}
-function enable_secret_manager() {
-  gcloud services enable \
-    secretmanager.googleapis.com \
-    --project=${PROJECT_ID}
-}
-
-function create_secret() {
-  echo -n "super secret" | gcloud secrets create ${MYSQL_SECRET_NAME} \
-    --replication-policy="automatic" \
-    --data-file=-
-
-}
diff --git a/gcloud/lib/swp/certs.sh b/gcloud/lib/swp/certs.sh
index f39621ea..fa5df48d 100644
--- a/gcloud/lib/swp/certs.sh
+++ b/gcloud/lib/swp/certs.sh
@@ -1,284 +1,194 @@
 #!/bin/bash
 
-
 function create_managed_certificate() {
-  local cert_name="swp-cert" # Static name for the final cert
   local region="${1:-${REGION}}"
   local project_id="${2:-${PROJECT_ID}}"
   local swp_hostname="${3:-${SWP_HOSTNAME}}"
-
-  local phase_name="swp_managed_cert"
-  local log_file="${phase_name}_${CLUSTER_NAME}-${RESOURCE_SUFFIX}.log"
-
-  local ca_pool_prefix="swp-ca-pool-${CLUSTER_NAME}-"
-  local cic_prefix="swp-cic-${CLUSTER_NAME}-"
-  local ca_prefix="swp-root-ca-${CLUSTER_NAME}-"
-
-  local suffix="${RESOURCE_SUFFIX}"
-  local ca_pool_name="${ca_pool_prefix}${suffix}"
-  local cic_name="${cic_prefix}${suffix}"
-  local ca_name="${ca_prefix}${suffix}"
+  local cert_name="swp-cert-${CLUSTER_NAME}-${TIMESTAMP}"
+  local ca_pool_name="swp-ca-pool-${CLUSTER_NAME}-${TIMESTAMP}"
+  local cic_name="swp-cic-${CLUSTER_NAME}-${TIMESTAMP}"
+  local ca_name="swp-root-ca-${CLUSTER_NAME}-${TIMESTAMP}"
   local ca_pool_full_name="projects/${project_id}/locations/${region}/caPools/${ca_pool_name}"
-
-  print_status "Ensuring SWP Certificate components for ${CLUSTER_NAME} (Suffix: ${suffix})..."
-  report_result "" # Newline
-
-  # 1. CA Pool
-  print_status "  Checking CA Pool ${ca_pool_name}..."
-  if check_sentinel "${phase_name}" "01_ca_pool_created"; then
-    report_result "Exists"
-  elif gcloud privateca pools describe "${ca_pool_name}" --location="${region}" --project="${project_id}" > /dev/null 2>&1; then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "01_ca_pool_created"
-  else
-    report_result "Not Found"
-    print_status "    Creating CA Pool ${ca_pool_name}..."
-    if run_gcloud "${log_file}" gcloud privateca pools create "${ca_pool_name}" --location="${region}" --tier=devops --project="${project_id}"; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "01_ca_pool_created"
-      echo "      -> Waiting 60s for CA Pool creation and IAM propagation..."
-      sleep 60
-      # Grant permissions - these are idempotent so okay to re-run
-      local project_number=$(gcloud projects describe "${project_id}" --format="value(projectNumber)")
-      local network_security_sa="service-${project_number}@gcp-sa-networksecurity.iam.gserviceaccount.com"
-      if ! gcloud beta services identity describe --service=networksecurity.googleapis.com --project="${project_id}" > /dev/null 2>&1; then
-          print_status "      Creating Network Security P4SA for ${project_id}..."
-          run_gcloud "${log_file}" gcloud beta services identity create --service=networksecurity.googleapis.com --project="${project_id}" && report_result "Pass"
-          echo "      -> Waiting 30s for IAM propagation..."
-          sleep 30
-      fi
-      print_status "      Granting privateca.certificateManager role to P4SA on ${ca_pool_name}..."
-      run_gcloud "${log_file}" gcloud privateca pools add-iam-policy-binding "${ca_pool_name}" --location="${region}" --project="${project_id}" --member="serviceAccount:${network_security_sa}" --role='roles/privateca.certificateManager' && report_result "Pass"
-    else
-      report_result "Fail"; return 1;
+  local log_file="swp_managed_cert_${CLUSTER_NAME}.log"
+
+  print_status "Creating SWP Certificate components for ${CLUSTER_NAME}..."
+  report_result ""
+
+  print_status "  Creating CA Pool ${ca_pool_name}..."
+  local pool_cmd=(
+    gcloud privateca pools create "${ca_pool_name}"
+    --location="${region}"
+    --tier=devops
+    --project="${project_id}"
+  )
+  if run_gcloud "${log_file}" "${pool_cmd[@]}"; then
+    report_result "Created"
+    refresh_resource_state "swpCaPool" "lib/swp/certs.sh" exists_swp_ca_pool
+    echo "      -> Waiting 60s for CA Pool creation and IAM propagation..."
+    sleep 60
+    local project_number=$(gcloud projects describe "${project_id}" --format="value(projectNumber)")
+    local network_security_sa="service-${project_number}@gcp-sa-networksecurity.iam.gserviceaccount.com"
+    if ! gcloud beta services identity describe --service=networksecurity.googleapis.com --project="${project_id}" > /dev/null 2>&1; then
+        print_status "      Creating Network Security P4SA for ${project_id}..."
+        local p4sa_cmd=(gcloud beta services identity create --service=networksecurity.googleapis.com --project="${project_id}")
+        run_gcloud "${log_file}" "${p4sa_cmd[@]}" && report_result "Pass"
+        echo "      -> Waiting 30s for IAM propagation..."
+        sleep 30
     fi
+    print_status "      Granting privateca.certificateManager role to P4SA on ${ca_pool_name}..."
+    local iam_cmd=(
+      gcloud privateca pools add-iam-policy-binding "${ca_pool_name}"
+      --location="${region}"
+      --project="${project_id}"
+      --member="serviceAccount:${network_security_sa}"
+      --role='roles/privateca.certificateManager'
+    )
+    run_gcloud "${log_file}" "${iam_cmd[@]}" && report_result "Pass"
+  else
+    report_result "Fail"; return 1;
   fi
 
-  # 2. Root CA
-  print_status "  Checking Root CA ${ca_name} in ${ca_pool_name}..."
-  if check_sentinel "${phase_name}" "02_root_ca_created"; then
-    report_result "Exists"
-  elif gcloud privateca roots describe "${ca_name}" --pool="${ca_pool_name}" --location="${region}" --project="${project_id}" > /dev/null 2>&1; then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "02_root_ca_created"
+  print_status "  Creating Root CA ${ca_name} in ${ca_pool_name}..."
+  local ca_cmd=(
+    gcloud privateca roots create "${ca_name}" --pool="${ca_pool_name}"
+    --location="${region}"
+    --project="${project_id}"
+    --subject="CN=swp-ca.internal.local, O=Dataproc SWP Test" --auto-enable --quiet
+  )
+  if run_gcloud "${log_file}" "${ca_cmd[@]}"; then
+    report_result "Created"
+    refresh_resource_state "swpRootCa" "lib/swp/certs.sh" exists_swp_root_ca
   else
-    report_result "Not Found"
-    print_status "    Creating Root CA ${ca_name}..."
-    if run_gcloud "${log_file}" gcloud privateca roots create "${ca_name}" --pool="${ca_pool_name}" --location="${region}" --project="${project_id}" --subject="CN=swp-ca.internal.local, O=Dataproc SWP Test" --auto-enable --quiet; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "02_root_ca_created"
-    else
-      report_result "Fail"; return 1;
-    fi
+    report_result "Fail"; return 1;
   fi
 
-  # 3. CIC
-  print_status "  Checking CIC ${cic_name}..."
-  if check_sentinel "${phase_name}" "03_cic_created"; then
-    report_result "Exists"
-  elif gcloud certificate-manager issuance-configs describe "${cic_name}" --location="${region}" --project="${project_id}" > /dev/null 2>&1; then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "03_cic_created"
+  print_status "  Creating CIC ${cic_name}..."
+  local cic_cmd=(
+    gcloud certificate-manager issuance-configs create "${cic_name}"
+    --location="${region}"
+    --project="${project_id}"
+    --ca-pool="${ca_pool_full_name}"
+    --lifetime="2592000s"
+    --rotation-window-percentage=66
+    --key-algorithm="rsa-2048"
+  )
+  if run_gcloud "${log_file}" "${cic_cmd[@]}"; then
+    report_result "Created"
+    refresh_resource_state "swpCic" "lib/swp/certs.sh" exists_swp_cic
   else
-    report_result "Not Found"
-    print_status "    Creating CIC ${cic_name}..."
-    if run_gcloud "${log_file}" gcloud certificate-manager issuance-configs create "${cic_name}" --location="${region}" --project="${project_id}" --ca-pool="${ca_pool_full_name}" --lifetime="2592000s" --rotation-window-percentage=66 --key-algorithm="rsa-2048"; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "03_cic_created"
-    else
-      report_result "Fail"; return 1;
-    fi
+    report_result "Fail"; return 1;
   fi
 
-  # 4. Certificate Manager Certificate (swp-cert)
-  print_status "  Checking Certificate Manager Certificate ${cert_name}..."
+  print_status "  Creating Certificate Manager Certificate ${cert_name}..."
   local cert_log_file="create_managed_cert_${cert_name}.log"
   local desired_cic="projects/${project_id}/locations/${region}/certificateIssuanceConfigs/${cic_name}"
-  if check_sentinel "${phase_name}" "04_cert_created"; then
-    report_result "Exists"
-  elif gcloud certificate-manager certificates describe "${cert_name}" --location="${region}" --project="${project_id}" > /dev/null 2>&1; then
-    local current_cic=$(gcloud certificate-manager certificates describe "${cert_name}" --location="${region}" --project="${project_id}" --format="value(managed.issuanceConfig)")
-    if [[ "${current_cic}" == "${desired_cic}" ]]; then
-      report_result "Exists"
-      create_sentinel "${phase_name}" "04_cert_created"
-    else
-      report_result "Fail"
-      return 1
-    fi
+  local cert_cmd=(
+    gcloud certificate-manager certificates create "${cert_name}"
+    --location="${region}"
+    --project="${project_id}"
+    --domains="${swp_hostname}"
+    --issuance-config="${desired_cic}"
+  )
+  if run_gcloud "${cert_log_file}" "${cert_cmd[@]}"; then
+    report_result "Created"
+    refresh_resource_state "swpManagedCertificate" "lib/swp/certs.sh" exists_swp_managed_certificate
   else
-    report_result "Not Found"
-    print_status "    Creating Certificate ${cert_name}..."
-    if run_gcloud "${cert_log_file}" gcloud certificate-manager certificates create "${cert_name}" --location="${region}" --project="${project_id}" --domains="${swp_hostname}" --issuance-config="${desired_cic}"; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "04_cert_created"
-    else
-      report_result "Fail"; return 1;
-    fi
+    report_result "Fail"; return 1;
   fi
   export SWP_CERT_URI_PARTIAL="projects/${project_id}/locations/${region}/certificates/${cert_name}"
 }
 export -f create_managed_certificate
 
-function _delete_swp_ca_resources() {
-  local region="${1:-${REGION}}"
-  local project_id="${2:-${PROJECT_ID}}"
-  local log_file="${3:-delete_managed_certificate_${CLUSTER_NAME}.log}"
-
-  local cic_prefix="swp-cic-${CLUSTER_NAME}-"
-  local pool_prefix="swp-ca-pool-${CLUSTER_NAME}-"
-  local ca_prefix="swp-root-ca-${CLUSTER_NAME}-"
-  local overall_status="Pass"
-  local found_some=false
-
-  # --- Deleting Certificate Issuance Config(s) ---
-  local cic_names=$(gcloud certificate-manager issuance-configs list --location="${region}" --project="${project_id}" --format="value(name)" 2>/dev/null | grep "${cic_prefix}" || true)
-  if [[ -n "${cic_names}" ]]; then
-    found_some=true
-    while read -r cic; do
-      local short_cic_name=$(basename "${cic}")
-      print_status "  Deleting CIC ${short_cic_name}..."
-      if ! run_gcloud "${log_file}" gcloud certificate-manager issuance-configs delete "${short_cic_name}" --location="${region}" --project="${project_id}" --quiet; then overall_status="Fail"; report_result "Fail"; else report_result "Deleted"; fi
-    done <<< "${cic_names}"
-  else
-     print_status "  No CICs found with prefix ${cic_prefix}..."
-     report_result "Not Found"
-  fi
-
-  # --- Deleting CA Pool(s) and Root CA(s) ---
-  local pool_names=$(gcloud privateca pools list --location="${region}" --project="${project_id}" --format="value(name)" 2>/dev/null | grep "${pool_prefix}" || true)
-  if [[ -n "${pool_names}" ]]; then
-    found_some=true
-    while read -r pool_full_name; do
-      local short_pool_name=$(basename "${pool_full_name}")
-      print_status "  Deleting CA Pool ${short_pool_name}..."
-      local ca_names=$(gcloud privateca roots list --pool="${short_pool_name}" --location="${region}" --project="${project_id}" --format="value(name)" 2>/dev/null | grep "${ca_prefix}" || true)
-      if [[ -n "${ca_names}" ]]; then
-        while read -r ca_full_name; do
-          local short_ca_name=$(basename "${ca_full_name}")
-          local ca_log_file="delete_ca_${short_ca_name}.log"
-          print_status "    Disabling CA ${short_ca_name}..."
-          if run_gcloud "${ca_log_file}" gcloud privateca roots disable "${short_ca_name}" --pool="${short_pool_name}" --location="${region}" --project="${project_id}" --quiet; then
-            report_result "Pass"
-            print_status "    Deleting CA ${short_ca_name}..."
-            if ! run_gcloud "${ca_log_file}" gcloud privateca roots delete "${short_ca_name}" --pool="${short_pool_name}" --location="${region}" --project="${project_id}" --quiet --skip-grace-period; then
-              report_result "Fail"
-              overall_status="Fail"
-            else
-              report_result "Deleted"
-            fi
-          else
-            report_result "Fail" # Disable failed
-            overall_status="Fail"
-          fi
-        done <<< "${ca_names}"
-      fi
-      print_status "  Attempting to delete CA Pool ${short_pool_name}..."
-      if ! run_gcloud "${log_file}" gcloud privateca pools delete "${short_pool_name}" --location="${region}" --project="${project_id}" --quiet --ignore-dependent-resources; then overall_status="Fail"; report_result "Fail"; else report_result "Deleted"; fi
-    done <<< "${pool_names}"
-  else
-     print_status "  No CA Pools found with prefix ${pool_prefix}..."
-     report_result "Not Found"
-  fi
-  return $([[ "${overall_status}" == "Pass" ]] && echo 0 || echo 1)
-}
-export -f _delete_swp_ca_resources
-
 function delete_managed_certificate() {
   local region="${1:-${REGION}}"
   local project_id="${2:-${PROJECT_ID}}"
-  local phase_name="swp_managed_cert"
   local log_file="delete_managed_certificate_${CLUSTER_NAME}.log"
-
-  print_status "Deleting SWP Certificate components for ${CLUSTER_NAME}..."
-  report_result "" # Newline
-
-  # Clear all sentinels for this cluster and phase, regardless of suffix
-  if [[ -d "${SENTINEL_DIR}" ]]; then
-    find "${SENTINEL_DIR}" -type f -name "${phase_name}-*" -exec rm -f {} + > /dev/null 2>&1
-    print_status "  Cleared sentinels for ${phase_name}..."
-    report_result "Pass"
+  local cert_name="swp-cert-${CLUSTER_NAME}-${TIMESTAMP}"
+  local cic_name="swp-cic-${CLUSTER_NAME}-${TIMESTAMP}"
+  local ca_pool_name="swp-ca-pool-${CLUSTER_NAME}-${TIMESTAMP}"
+  local ca_name="swp-root-ca-${CLUSTER_NAME}-${TIMESTAMP}"
+
+  print_status "Deleting SWP Certificate components for ${CLUSTER_NAME}-${TIMESTAMP}..."
+  report_result ""
+
+  print_status "  Deleting versioned certificate ${cert_name}..."
+  local del_cert_cmd=(gcloud certificate-manager certificates delete "${cert_name}" --location="${region}" --project="${project_id}" --quiet)
+  if ! run_gcloud "delete_managed_cert_${cert_name}.log" "${del_cert_cmd[@]}"; then
+    report_result "Fail";
+  else
+    report_result "Deleted";
   fi
 
-  # --- Deleting the static-named Certificate Manager Certificate ---
-  local static_cert_name="swp-cert"
-  print_status "  Checking for static certificate ${static_cert_name}..."
-  local cert_check=$(gcloud certificate-manager certificates list --location="${region}" --project="${project_id}" --filter="name='projects/${project_id}/locations/${region}/certificates/${static_cert_name}'" --format="value(name)" 2>/dev/null)
+  print_status "  Deleting CIC ${cic_name}..."
+  local del_cic_cmd=(gcloud certificate-manager issuance-configs delete "${cic_name}" --location="${region}" --project="${project_id}" --quiet)
+  run_gcloud "${log_file}" "${del_cic_cmd[@]}" || true
 
-  if [[ -n "${cert_check}" ]]; then
-    print_status "  Deleting static certificate ${static_cert_name}..."
-    if ! run_gcloud "${log_file}" gcloud certificate-manager certificates delete "${static_cert_name}" --location="${region}" --project="${project_id}" --quiet; then
-      report_result "Fail";
-    else
-      report_result "Deleted";
-    fi
-  else
-    report_result "Not Found"
-  fi
+  print_status "    Disabling and Deleting CA ${ca_name}..."
+  local disable_ca_cmd=(gcloud privateca roots disable "${ca_name}" --pool="${ca_pool_name}" --location="${region}" --project="${project_id}" --quiet)
+  run_gcloud "delete_ca_${ca_name}.log" "${disable_ca_cmd[@]}" || true
+  local delete_ca_cmd=(gcloud privateca roots delete "${ca_name}" --pool="${ca_pool_name}" --location="${region}" --project="${project_id}" --quiet --skip-grace-period)
+  run_gcloud "delete_ca_${ca_name}.log" "${delete_ca_cmd[@]}" || true
 
-  if [[ "${FORCE_DELETE}" == "true" ]]; then
-    print_status "  --force specified, deleting versioned CA/CIC components..."
-    report_result ""
-    _delete_swp_ca_resources "${region}" "${project_id}" "${log_file}"
-  else
-    print_status "  Skipping deletion of versioned CA/CIC components. Use --force to delete."
-    report_result "Skipped"
-  fi
+  print_status "  Attempting to delete CA Pool ${ca_pool_name}..."
+  local del_pool_cmd=(gcloud privateca pools delete "${ca_pool_name}" --location="${region}" --project="${project_id}" --quiet --ignore-dependent-resources)
+  run_gcloud "${log_file}" "${del_pool_cmd[@]}" || true
 }
 export -f delete_managed_certificate
 
-function create_certificate() {
-  local cert_name="${1:-${SWP_CERT_NAME}}"
-  local region="${2:-${REGION}}"
-  local project_id="${3:-${PROJECT_ID}}"
-  local swp_ip="${4:-${SWP_IP}}"
-  local swp_hostname="${5:-${SWP_HOSTNAME}}"
+function exists_swp_ca_pool() {
+  local region="${1:-${REGION}}"
+  local project_id="${2:-${PROJECT_ID}}"
+  local ca_pool_prefix="swp-ca-pool-${CLUSTER_NAME}-"
+  # List all CA pools in the location and filter for those starting with the prefix
+  gcloud privateca pools list --location="${region}" --project="${project_id}" --format="json(name,tier)" \
+    | jq --arg prefix "${ca_pool_prefix}" 'if type == "array" then map(select(.name | split("/") | last | startswith($prefix))) | if length > 0 then .[0] else null end else null end'
+}
+export -f exists_swp_ca_pool
 
-  local ca_key_file="tls/swp_ca.key"
-  local ca_cert_file="tls/swp_ca.crt"
-  local server_key_file="tls/swp.key"
-  local server_csr_file="tls/swp.csr"
-  local server_cert_file="tls/swp.crt"
-  local gcs_ca_cert_uri="${INIT_ACTIONS_ROOT}/swp_ca.crt"
+function exists_swp_root_ca() {
+  local region="${1:-${REGION}}"
+  local project_id="${2:-${PROJECT_ID}}"
+  # We first need to find the CA pool since the root CA is nested within it
+  local ca_pool_json=$(exists_swp_ca_pool "${region}" "${project_id}")
+  if [[ "${ca_pool_json}" == "null" || -z "${ca_pool_json}" ]]; then
+    echo "null"
+    return 1
+  fi
+  local ca_pool_name=$(echo "${ca_pool_json}" | jq -r '.name | split("/") | last')
+  local ca_prefix="swp-root-ca-${CLUSTER_NAME}-"
 
-  # ... (rest of create_certificate) ...
+  # List all roots in the discovered pool
+  gcloud privateca roots list --pool="${ca_pool_name}" --location="${region}" --project="${project_id}" --format="json(name,state)" \
+    | jq --arg prefix "${ca_prefix}" 'if type == "array" then map(select(.name | split("/") | last | startswith($prefix))) | if length > 0 then .[0] else null end else null end'
 }
+export -f exists_swp_root_ca
 
-function delete_certificate() {
-  local cert_name="${1:-${SWP_CERT_NAME}}"
-  local region="${2:-${REGION}}"
-
-  print_status "Deleting Self-Signed Certificate ${cert_name}..."
-  if gcloud certificate-manager certificates describe "${cert_name}" --location="${region}" --project="${project_id}" > /dev/null 2>&1; then
-    local log_file="delete_certificate_${cert_name}.log"
-    if run_gcloud "${log_file}" gcloud certificate-manager certificates delete "${cert_name}" --location="${region}" --quiet; then
-      report_result "Deleted"
-    else
-      report_result "Fail"
-    fi
-  else
-    report_result "Not Found"
-  fi
+function exists_swp_cic() {
+  local region="${1:-${REGION}}"
+  local project_id="${2:-${PROJECT_ID}}"
+  local cic_prefix="swp-cic-${CLUSTER_NAME}-"
+  # List issuance configs and filter
+  gcloud certificate-manager issuance-configs list --location="${region}" --project="${project_id}" --format="json(name)" \
+    | jq --arg prefix "${cic_prefix}" 'if type == "array" then map(select(.name | split("/") | last | startswith($prefix))) | if length > 0 then .[0] else null end else null end'
 }
+export -f exists_swp_cic
 
-function delete_ca_pool() {
-  local pool_name="${1:-swp-ca-pool-${CLUSTER_NAME}}"
-  local region="${2:-${REGION}}"
-  local project_id="${3:-${PROJECT_ID}}"
-  local ca_name="swp-root-ca-${CLUSTER_NAME}"
-  local log_file="delete_ca_pool_${pool_name}.log"
+function exists_swp_managed_certificate() {
+  local region="${1:-${REGION}}"
+  local project_id="${2:-${PROJECT_ID}}"
+  local cert_prefix="swp-cert-${CLUSTER_NAME}-"
+  # List certificates and filter
+  gcloud certificate-manager certificates list --location="${region}" --project="${project_id}" --format="json(name,managed.state)" \
+    | jq --arg prefix "${cert_prefix}" -c 'if type == "array" then map(select(.name | split("/") | last | startswith($prefix))) | if length > 0 then .[0] else null end else null end'
+}
+export -f exists_swp_managed_certificate
 
-  print_status "Deleting CA Pool ${pool_name}..."
-  if gcloud privateca pools describe "${pool_name}" --location="${region}" --project="${project_id}" > /dev/null 2>&1; then
-    if gcloud privateca roots describe "${ca_name}" --pool="${pool_name}" --location="${region}" --project="${project_id}" > /dev/null 2>&1; then
-      run_gcloud "${log_file}" gcloud privateca roots disable "${ca_name}" --pool="${pool_name}" --location="${region}" --project="${project_id}" --quiet
-      run_gcloud "${log_file}" gcloud privateca roots delete "${ca_name}" --pool="${pool_name}" --location="${region}" --project="${project_id}" --quiet --skip-grace-period
-    fi
-    if run_gcloud "${log_file}" gcloud privateca pools delete "${pool_name}" --location="${region}" --project="${project_id}" --quiet; then
-      report_result "Deleted"
-    else
-      report_result "Fail"
-    fi
+function get_or_construct_swp_cert_uri() {
+  local cert_json=$(get_state "swpManagedCertificate")
+  if [[ -n "${cert_json}" && "${cert_json}" != "null" ]]; then
+    export SWP_CERT_URI_PARTIAL=$(echo "${cert_json}" | /usr/bin/jq -r '.name')
   else
-    report_result "Not Found"
+    export SWP_CERT_URI_PARTIAL="projects/${PROJECT_ID}/locations/${REGION}/certificates/swp-cert-${CLUSTER_NAME}-${TIMESTAMP}"
   fi
-}
\ No newline at end of file
+}
+export -f get_or_construct_swp_cert_uri
diff --git a/gcloud/lib/swp/firewall.sh b/gcloud/lib/swp/firewall.sh
index bf0a7da3..bc32c2ec 100644
--- a/gcloud/lib/swp/firewall.sh
+++ b/gcloud/lib/swp/firewall.sh
@@ -1,56 +1,46 @@
 #!/bin/bash
 
+function exists_firewall_rule() {
+  local rule_name="$1"
+  _check_exists gcloud compute firewall-rules describe "${rule_name}" --project="${PROJECT_ID}" --format="json(name,direction)"
+}
+export -f exists_firewall_rule
+
 function create_allow_swp_ingress_rule() {
   local rule_name="${1:-allow-swp-ingress-${CLUSTER_NAME}}"
   local network_name="${2:-${NETWORK}}"
-  local source_range="${3:-${PRIVATE_RANGE}}"
-  local phase_name="create_allow_swp_ingress_rule"
-
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating Firewall Rule ${rule_name}..."
-    report_result "Exists"
-    return 0
-  fi
-
-  print_status "Creating Firewall Rule ${rule_name}..."
-  if ! gcloud compute firewall-rules describe "${rule_name}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    local log_file="create_firewall_${rule_name}.log"
-    if run_gcloud "${log_file}" gcloud compute firewall-rules create "${rule_name}" \
-      --project="${PROJECT_ID}" \
-      --network="${network_name}" \
-      --direction=INGRESS \
-      --action=ALLOW \
-      --rules=tcp:${SWP_PORT} \
-      --source-ranges="${source_range}" \
-      --destination-ranges="${SWP_RANGE}"; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
+  local source_range="${3:-${RANGE}}"
+  print_status "Creating Firewall Rule ${rule_name} for ${SUBNET} to SWP..."
+  local log_file="create_firewall_${rule_name}.log"
+  local cmd=(
+    gcloud compute firewall-rules create "${rule_name}"
+    --project="${PROJECT_ID}"
+    --network="${network_name}"
+    --direction=INGRESS
+    --action=ALLOW
+    --rules=tcp:${SWP_PORT}
+    --source-ranges="${source_range}"
+    --target-tags="swp-client"
+  )
+  if run_gcloud "${log_file}" "${cmd[@]}"; then
+    report_result "Created"
+    refresh_resource_state "swpFirewallIngress" "lib/swp/firewall.sh" exists_firewall_rule "${rule_name}"
   else
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
+    report_result "Fail"
+    return 1
   fi
 }
 export -f create_allow_swp_ingress_rule
 
 function delete_allow_swp_ingress_rule() {
   local rule_name="${1:-allow-swp-ingress-${CLUSTER_NAME}}"
-  local phase_name="create_allow_swp_ingress_rule"
-  remove_sentinel "${phase_name}" "done"
-
   print_status "Deleting Firewall Rule ${rule_name}..."
-  if gcloud compute firewall-rules describe "${rule_name}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    local log_file="delete_firewall_${rule_name}.log"
-    if run_gcloud "${log_file}" gcloud compute firewall-rules delete "${rule_name}" --project="${PROJECT_ID}" --quiet; then
-      report_result "Deleted"
-    else
-      report_result "Fail"
-    fi
+  local log_file="delete_firewall_${rule_name}.log"
+  local cmd=(gcloud compute firewall-rules delete "${rule_name}" --project="${PROJECT_ID}" --quiet)
+  if run_gcloud "${log_file}" "${cmd[@]}"; then
+    report_result "Deleted"
   else
-    report_result "Not Found"
+    report_result "Fail"
   fi
 }
 export -f delete_allow_swp_ingress_rule
@@ -58,56 +48,39 @@ export -f delete_allow_swp_ingress_rule
 function create_allow_internal_subnets_rule() {
   local rule_name="${1:-allow-internal-${CLUSTER_NAME}}"
   local network_name="${2:-${NETWORK}}"
-  local source_range="${3:-${PRIVATE_RANGE}}"
+  local source_range="${3:-${RANGE}}"
   local dest_range="${4:-${SWP_RANGE}}"
-  local phase_name="create_allow_internal_subnets_rule"
-
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating Firewall Rule ${rule_name}..."
-    report_result "Exists"
-    return 0
-  fi
-
-  print_status "Creating Firewall Rule ${rule_name}..."
-  if ! gcloud compute firewall-rules describe "${rule_name}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    local log_file="create_firewall_${rule_name}.log"
-    if run_gcloud "${log_file}" gcloud compute firewall-rules create "${rule_name}" \
-      --project="${PROJECT_ID}" \
-      --network="${network_name}" \
-      --direction=INGRESS \
-      --action=ALLOW \
-      --rules=all \
-      --source-ranges="${source_range}" \
-      --destination-ranges="${dest_range}" \
-      --priority=100; then # High priority
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
-  else
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
+  print_status "Creating Firewall Rule ${rule_name} for ${SUBNET} to ${SWP_SUBNET}..."
+  local log_file="create_firewall_${rule_name}.log"
+  local cmd=(
+    gcloud compute firewall-rules create "${rule_name}"
+    --project="${PROJECT_ID}"
+    --network="${network_name}"
+    --direction=INGRESS
+    --action=ALLOW
+    --rules=all
+    --source-ranges="${source_range}"
+    --destination-ranges="${dest_range}"
+  )
+  if run_gcloud "${log_file}" "${cmd[@]}"; then
+         report_result "Created"
+         refresh_resource_state "swpFirewallInternal" "lib/swp/firewall.sh" exists_firewall_rule "${rule_name}"
+       else
+    report_result "Fail"
+    return 1
   fi
 }
 export -f create_allow_internal_subnets_rule
 
 function delete_allow_internal_subnets_rule() {
   local rule_name="${1:-allow-internal-${CLUSTER_NAME}}"
-  local phase_name="create_allow_internal_subnets_rule"
-  remove_sentinel "${phase_name}" "done"
-
   print_status "Deleting Firewall Rule ${rule_name}..."
-  if gcloud compute firewall-rules describe "${rule_name}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    local log_file="delete_firewall_${rule_name}.log"
-    if run_gcloud "${log_file}" gcloud compute firewall-rules delete "${rule_name}" --project="${PROJECT_ID}" --quiet; then
-      report_result "Deleted"
-    else
-      report_result "Fail"
-    fi
+  local log_file="delete_firewall_${rule_name}.log"
+  local cmd=(gcloud compute firewall-rules delete "${rule_name}" --project="${PROJECT_ID}" --quiet)
+  if run_gcloud "${log_file}" "${cmd[@]}"; then
+    report_result "Deleted"
   else
-    report_result "Not Found"
+    report_result "Fail"
   fi
 }
-export -f delete_allow_internal_subnets_rule
\ No newline at end of file
+export -f delete_allow_internal_subnets_rule
diff --git a/gcloud/lib/swp/gateway.sh b/gcloud/lib/swp/gateway.sh
index fc0eadff..a58c6a83 100644
--- a/gcloud/lib/swp/gateway.sh
+++ b/gcloud/lib/swp/gateway.sh
@@ -4,31 +4,20 @@ function create_swp_gateway() {
   local swp_instance_name="${1:-${SWP_INSTANCE_NAME}}"
   local region="${2:-${REGION}}"
   local network_name="${3:-${NETWORK}}"
-  local client_subnet_name="${4:-${PRIVATE_SUBNET}}"
+  # client_subnet_name is not used for the gateway resource itself
   local certificate_url="${5:-${SWP_CERT_URI_PARTIAL}}"
   local gateway_security_policy_url="${6:-${SWP_POLICY_URI_PARTIAL}}"
   local project_id="${7:-${PROJECT_ID}}"
-  local phase_name="create_swp_gateway"
+  local swp_full_name="projects/${project_id}/locations/${region}/gateways/${swp_instance_name}"
 
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating SWP Gateway ${swp_instance_name}..."
-    report_result "Exists"
-    return 0
-  fi
+  echo "DEBUG: certificate_url: ${certificate_url}" >&2
+  echo "DEBUG: gateway_security_policy_url: ${gateway_security_policy_url}" >&2
 
   print_status "Creating SWP Gateway ${swp_instance_name}..."
   local log_file="create_swp_gateway_${swp_instance_name}.log"
-
-  if gcloud network-services gateways describe "${swp_instance_name}" --location="${region}" --project="${project_id}" > /dev/null 2>&1; then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
-    return 0
-  fi
-
   local swp_address="${SWP_IP}"
   local full_network_name="projects/${project_id}/global/networks/${network_name}"
-  local full_client_subnet_name="projects/${project_id}/regions/${region}/subnetworks/${client_subnet_name}"
-  local swp_full_name="projects/${project_id}/locations/${region}/gateways/${swp_instance_name}"
+  local full_client_subnet_name="projects/${project_id}/regions/${region}/subnetworks/${SUBNET}" # Client subnet
 
   gateway_yaml=$(cat << EOF
 name: ${swp_full_name}
@@ -46,11 +35,17 @@ scope: ${swp_instance_name}-scope
 routingMode: EXPLICIT_ROUTING_MODE
 EOF
 )
-  if echo "${gateway_yaml}" | run_gcloud "${log_file}" gcloud network-services gateways import "${swp_instance_name}" \
-                --source=- \
-                --location="${region}" --project="${project_id}"; then
+
+  local cmd=(
+    gcloud network-services gateways import "${swp_instance_name}"
+    --source=-
+    --location="${region}"
+    --project="${project_id}"
+  )
+
+  if echo "${gateway_yaml}" | run_gcloud "${log_file}" "${cmd[@]}"; then
     report_result "Created"
-    create_sentinel "${phase_name}" "done"
+    refresh_resource_state "swpGateway" "lib/swp/gateway.sh" exists_swp_gateway "${swp_instance_name}" "${region}" "${project_id}"
   else
     report_result "Fail"
     return 1
@@ -58,38 +53,36 @@ EOF
 }
 export -f create_swp_gateway
 
-function delete_swp_gateway() {
+function exists_swp_gateway() {
   local swp_instance_name="${1:-${SWP_INSTANCE_NAME}}"
   local region="${2:-${REGION}}"
-  local phase_name="create_swp_gateway"
+  local project_id="${3:-${PROJECT_ID}}"
+  _check_exists gcloud network-services gateways describe "${swp_instance_name}" --location="${region}" --project="${project_id}" --format="json(name,type)"
+}
+export -f exists_swp_gateway
 
+function delete_swp_gateway() {
+  local swp_instance_name="${1:-${SWP_INSTANCE_NAME}}"
+  local region="${2:-${REGION}}"
   print_status "Deleting SWP Gateway ${swp_instance_name}..."
-  if gcloud network-services gateways describe "${swp_instance_name}" --location="${region}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    local log_file="delete_swp_gateway_${swp_instance_name}.log"
-    if run_gcloud "${log_file}" gcloud network-services gateways delete "${swp_instance_name}" --location="${region}" --project="${PROJECT_ID}" --quiet; then
-      report_result "Deleted"
-      remove_sentinel "${phase_name}" "done"
-
-      # Attempt to delete the autogen router
-      local autogen_router_prefix="swg-autogen-router-"
-      local autogen_routers=$(gcloud compute routers list --regions="${region}" --project="${PROJECT_ID}" --filter="network ~ /${NETWORK}$ AND name ~ ^${autogen_router_prefix}" --format="value(name)" 2>/dev/null || true)
-      if [[ -n "${autogen_routers}" ]]; then
-        while read -r router_name; do
-          print_status "  Deleting Autogen Router ${router_name}..."
-          local delete_router_log="delete_autogen_router_${router_name}.log"
-          if run_gcloud "${delete_router_log}" gcloud compute routers delete "${router_name}" --region="${region}" --project="${PROJECT_ID}" --quiet; then
-            report_result "Deleted"
-          else
-            report_result "Fail"
-          fi
-        done <<< "${autogen_routers}"
-      fi
-    else
-      report_result "Fail"
+  local log_file="delete_swp_gateway_${swp_instance_name}.log"
+  if run_gcloud "${log_file}" gcloud network-services gateways delete "${swp_instance_name}" --location="${region}" --project="${PROJECT_ID}" --quiet; then
+    report_result "Deleted"
+    local autogen_router_prefix="swg-autogen-router-"
+    local autogen_routers=$(gcloud compute routers list --regions="${region}" --project="${PROJECT_ID}" --filter="network ~ /${NETWORK}$ AND name ~ ^${autogen_router_prefix}" --format="value(name)" 2>/dev/null || true)
+    if [[ -n "${autogen_routers}" ]]; then
+      while read -r router_name; do
+        print_status "  Deleting Autogen Router ${router_name}..."
+        local delete_router_log="delete_autogen_router_${router_name}.log"
+        if run_gcloud "${delete_router_log}" gcloud compute routers delete "${router_name}" --region="${region}" --project="${PROJECT_ID}" --quiet; then
+          report_result "Deleted"
+        else
+          report_result "Fail"
+        fi
+      done <<< "${autogen_routers}"
     fi
   else
-    report_result "Not Found"
-    remove_sentinel "${phase_name}" "done"
+    report_result "Fail"
   fi
 }
-export -f delete_swp_gateway
\ No newline at end of file
+export -f delete_swp_gateway
diff --git a/gcloud/lib/swp/policy.sh b/gcloud/lib/swp/policy.sh
index 97dc4b98..505eb088 100644
--- a/gcloud/lib/swp/policy.sh
+++ b/gcloud/lib/swp/policy.sh
@@ -7,42 +7,29 @@ function create_gateway_security_policy() {
   local rule_name="allow-all-rule"
   local policy_full_name="projects/${project_id}/locations/${region}/gatewaySecurityPolicies/${policy_name}"
   local log_file="create_swp_policy_${policy_name}.log"
-  local phase_name="create_gateway_security_policy"
-
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating Gateway Security Policy ${policy_name}..."
-    report_result "Exists"
-    return 0
-  fi
-
   print_status "Creating Gateway Security Policy ${policy_name}..."
-  local policy_exists=false
-  if gcloud network-security gateway-security-policies describe "${policy_name}" --location="${region}" --project="${project_id}" > /dev/null 2>&1; then
-    policy_exists=true
-    report_result "Exists"
-  else
-    policy_yaml=$(cat << EOF
+  
+  policy_yaml=$(cat << EOF
 name: ${policy_full_name}
 description: "Allow all policy for SWP"
 EOF
-    )
-    if echo "${policy_yaml}" | run_gcloud "${log_file}" gcloud network-security gateway-security-policies import "${policy_name}" \
-      --location="${region}" --project="${project_id}" \
-      --source=-
-    then
-      report_result "Created"
-      policy_exists=true
-    else
-      report_result "Fail"
-      return 1
-    fi
+)
+  local policy_cmd=(
+    gcloud network-security gateway-security-policies import "${policy_name}"
+    --location="${region}"
+    --project="${project_id}"
+    --source=-
+  )
+  if echo "${policy_yaml}" | run_gcloud "${log_file}" "${policy_cmd[@]}"; then
+    report_result "Created"
+  else
+    report_result "Fail"
+    return 1
   fi
 
-  if [[ "${policy_exists}" = true ]]; then
-    print_status "  Ensuring allow-all rule in ${policy_name}..."
-    local rule_log_file="create_swp_policy_rule_${policy_name}.log"
-    if ! gcloud network-security gateway-security-policies rules describe "${rule_name}" --gateway-security-policy="${policy_name}" --location="${region}" --project="${project_id}" > /dev/null 2>&1; then
-      rule_yaml=$(cat << EOF
+  print_status "  Ensuring allow-all rule in ${policy_name}..."
+  local rule_log_file="create_swp_policy_rule_${policy_name}.log"
+  rule_yaml=$(cat << EOF
 name: ${policy_full_name}/rules/${rule_name}
 description: "Allow all traffic"
 priority: 1000
@@ -50,68 +37,75 @@ enabled: true
 basicProfile: ALLOW
 sessionMatcher: "host() != 'none'"
 EOF
-      )
-      if echo "${rule_yaml}" | run_gcloud "${rule_log_file}" gcloud network-security gateway-security-policies rules import "${rule_name}" \
-        --gateway-security-policy="${policy_name}" \
-        --location="${region}" --project="${project_id}" \
-        --source=-
-      then
-        report_result "Created"
-      else
-        report_result "Fail"
-        return 1 # Fail the whole function if rule creation fails
-      fi
-    else
-      report_result "Exists"
-    fi
-    create_sentinel "${phase_name}" "done"
+)
+  local rule_cmd=(
+    gcloud network-security gateway-security-policies rules import "${rule_name}"
+    --gateway-security-policy="${policy_name}"
+    --location="${region}"
+    --project="${project_id}"
+    --source=-
+  )
+  if echo "${rule_yaml}" | run_gcloud "${rule_log_file}" "${rule_cmd[@]}"; then
+    report_result "Created"
+  else
+    report_result "Fail"
+    return 1
   fi
+  export SWP_POLICY_URI_PARTIAL="${policy_full_name}"
+  refresh_resource_state "swpPolicy" "lib/swp/policy.sh" exists_gateway_security_policy "${policy_name}" "${region}" "${project_id}"
 }
 export -f create_gateway_security_policy
 
+function exists_gateway_security_policy() {
+  local policy_name="${1:-${SWP_POLICY_NAME}}"
+  local region="${2:-${REGION}}"
+  local project_id="${3:-${PROJECT_ID}}"
+  # List policies and filter for the exact name
+  gcloud network-security gateway-security-policies list --location="${region}" --project="${project_id}" --format=json \
+    | /usr/bin/jq -c --arg name "${policy_name}" 'if type == "array" then map(select(.name | split("/") | last == $name)) | if length > 0 then .[0] else null end else null end'
+}
+export -f exists_gateway_security_policy
+
+function get_or_construct_swp_policy_uri() {
+  local policy_json=$(get_state "swpPolicy")
+  if [[ -n "${policy_json}" && "${policy_json}" != "null" ]]; then
+    export SWP_POLICY_URI_PARTIAL=$(echo "${policy_json}" | /usr/bin/jq -r '.name')
+  else
+    export SWP_POLICY_URI_PARTIAL="projects/${PROJECT_ID}/locations/${REGION}/gatewaySecurityPolicies/${SWP_POLICY_NAME}"
+  fi
+}
+export -f get_or_construct_swp_policy_uri
+
 function delete_gateway_security_policy() {
   local policy_name="${1:-${SWP_POLICY_NAME}}"
   local region="${2:-${REGION}}"
   local project_id="${3:-${PROJECT_ID}}"
   local rule_name="allow-all-rule"
-  local phase_name="create_gateway_security_policy"
-
   print_status "Deleting Gateway Security Policy ${policy_name}..."
+  local rule_log="delete_swp_policy_rule_${policy_name}.log"
+  local policy_log="delete_swp_policy_${policy_name}.log"
+  
+  print_status "  Deleting rule ${rule_name}..."
+  local del_rule_cmd=(
+    gcloud network-security gateway-security-policies rules delete "${rule_name}"
+    --gateway-security-policy="${policy_name}"
+    --location="${region}"
+    --project="${project_id}"
+    --quiet
+  )
+  run_gcloud "${rule_log}" "${del_rule_cmd[@]}"
 
-  local policy_check=$(gcloud network-security gateway-security-policies list --location="${region}" --project="${project_id}" --filter="name ~ /${policy_name}$" --format="value(name)" 2>/dev/null)
-
-  if [[ -n "${policy_check}" ]]; then
-    local rule_log="delete_swp_policy_rule_${policy_name}.log"
-    local policy_log="delete_swp_policy_${policy_name}.log"
-    # Delete the rule first
-    print_status "  Deleting rule ${rule_name}..."
-    local rule_check=$(gcloud network-security gateway-security-policies rules list --gateway-security-policy="${policy_name}" --location="${region}" --project="${project_id}" --filter="name ~ /${rule_name}$" --format="value(name)" 2>/dev/null)
-    if [[ -n "${rule_check}" ]]; then
-      if run_gcloud "${rule_log}" gcloud network-security gateway-security-policies rules delete "${rule_name}" \
-        --gateway-security-policy="${policy_name}" \
-        --location="${region}" --project="${project_id}" \
-        --quiet; then
-        report_result "Deleted"
-      else
-        report_result "Fail"
-      fi
-    else
-      report_result "Not Found"
-    fi
-
-    # Delete the policy
-    print_status "  Deleting policy ${policy_name}..."
-    if run_gcloud "${policy_log}" gcloud network-security gateway-security-policies delete "${policy_name}" \
-      --location="${region}" --project="${project_id}" \
-      --quiet; then
-      report_result "Deleted"
-      remove_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-    fi
+  print_status "  Deleting policy ${policy_name}..."
+  local del_policy_cmd=(
+    gcloud network-security gateway-security-policies delete "${policy_name}"
+    --location="${region}"
+    --project="${project_id}"
+    --quiet
+  )
+  if run_gcloud "${policy_log}" "${del_policy_cmd[@]}"; then
+    report_result "Deleted"
   else
-    report_result "Not Found"
-    remove_sentinel "${phase_name}" "done"
+    report_result "Fail"
   fi
 }
 export -f delete_gateway_security_policy
diff --git a/gcloud/lib/swp/subnet.sh b/gcloud/lib/swp/subnet.sh
index 3c2605e0..ad63018f 100644
--- a/gcloud/lib/swp/subnet.sh
+++ b/gcloud/lib/swp/subnet.sh
@@ -1,37 +1,34 @@
 #!/bin/bash
 
+function exists_subnet() {
+  local subnet_name="$1"
+  local region="${2:-${REGION}}"
+  _check_exists gcloud compute networks subnets describe "${subnet_name}" --region="${region}" --project="${PROJECT_ID}" --format="json(name,purpose)"
+}
+export -f exists_subnet
+
 function create_swp_subnet() {
   local subnet_name="${1:-${SWP_SUBNET}}"
   local region="${2:-${REGION}}"
   local network_name="${3:-${NETWORK}}"
   local range="${4:-${SWP_RANGE}}"
-  local phase_name="create_swp_subnet"
-
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating SWP Subnet ${subnet_name}..."
-    report_result "Exists"
-    return 0
-  fi
-
   print_status "Creating SWP Subnet ${subnet_name}..."
-  if gcloud compute networks subnets describe "${subnet_name}" --region="${region}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
-  else
-    local log_file="create_swp_subnet_${subnet_name}.log"
-    if run_gcloud "${log_file}" gcloud compute networks subnets create "${subnet_name}" \
-      --project="${PROJECT_ID}" \
-      --purpose=REGIONAL_MANAGED_PROXY \
-      --role=ACTIVE \
-      --region="${region}" \
-      --network="${network_name}" \
-      --range="${range}"; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
+  local log_file="create_swp_subnet_${subnet_name}.log"
+  local cmd=(
+    gcloud compute networks subnets create "${subnet_name}"
+    --project="${PROJECT_ID}"
+    --purpose=REGIONAL_MANAGED_PROXY
+    --role=ACTIVE
+    --region="${region}"
+    --network="${network_name}"
+    --range="${range}"
+  )
+  if run_gcloud "${log_file}" "${cmd[@]}"; then
+         report_result "Created"
+         refresh_resource_state "swpSubnet" "lib/swp/subnet.sh" exists_subnet "${subnet_name}"
+       else
+    report_result "Fail"
+    return 1
   fi
 }
 export -f create_swp_subnet
@@ -39,77 +36,12 @@ export -f create_swp_subnet
 function delete_swp_subnet() {
   local subnet_name="${1:-${SWP_SUBNET}}"
   local region="${2:-${REGION}}"
-  local phase_name="create_swp_subnet"
-
   print_status "Deleting SWP Subnet ${subnet_name}..."
-  if gcloud compute networks subnets describe "${subnet_name}" --region="${region}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    local log_file="delete_swp_subnet_${subnet_name}.log"
-    if run_gcloud "${log_file}" gcloud compute networks subnets delete "${subnet_name}" --region="${region}" --quiet; then
-      report_result "Deleted"
-      remove_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-    fi
+  local log_file="delete_swp_subnet_${subnet_name}.log"
+  if run_gcloud "${log_file}" gcloud compute networks subnets delete "${subnet_name}" --region="${region}" --quiet; then
+    report_result "Deleted"
   else
-    report_result "Not Found"
-    remove_sentinel "${phase_name}" "done" # Remove sentinel if not found
+    report_result "Fail"
   fi
 }
 export -f delete_swp_subnet
-
-function create_private_subnet () {
-  local subnet_name="${1:-${PRIVATE_SUBNET}}"
-  local region="${2:-${REGION}}"
-  local network_name="${3:-${NETWORK}}"
-  local range="${4:-${PRIVATE_RANGE}}"
-  local phase_name="create_private_subnet"
-
-  if check_sentinel "${phase_name}" "done"; then
-    print_status "Creating Private Subnet ${subnet_name}..."
-    report_result "Exists"
-    return 0
-  fi
-
-  print_status "Creating Private Subnet ${subnet_name}..."
-  if gcloud compute networks subnets describe "${subnet_name}" --region "${REGION}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    report_result "Exists"
-    create_sentinel "${phase_name}" "done"
-  else
-    local log_file="create_private_subnet_${subnet_name}.log"
-    if run_gcloud "${log_file}" gcloud compute networks subnets create "${subnet_name}" \
-      --project="${PROJECT_ID}" \
-      --network="${network_name}" \
-      --range="${range}" \
-      --enable-private-ip-google-access \
-      --region="${region}" \
-      --description="subnet for use with Dataproc cluster ${CLUSTER_NAME}"; then
-      report_result "Created"
-      create_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-      return 1
-    fi
-  fi
-}
-export -f create_private_subnet
-
-function delete_private_subnet () {
-  local subnet_name="${1:-${PRIVATE_SUBNET}}"
-  local region="${2:-${REGION}}"
-  local phase_name="create_private_subnet"
-
-  print_status "Deleting Private Subnet ${subnet_name}..."
-  if gcloud compute networks subnets describe "${subnet_name}" --region "${REGION}" --project="${PROJECT_ID}" > /dev/null 2>&1; then
-    local log_file="delete_private_subnet_${subnet_name}.log"
-    if run_gcloud "${log_file}" gcloud compute networks subnets delete --quiet --region "${REGION}" "${subnet_name}"; then
-      report_result "Deleted"
-      remove_sentinel "${phase_name}" "done"
-    else
-      report_result "Fail"
-    fi
-  else
-    report_result "Not Found"
-    remove_sentinel "${phase_name}" "done" # Remove sentinel if not found
-  fi
-}
-export -f delete_private_subnet
\ No newline at end of file
diff --git a/gcloud/project/findings/2026-W11/20260310T222921-gcloud-metadata-and-gpu-config.md b/gcloud/project/findings/2026-W11/20260310T222921-gcloud-metadata-and-gpu-config.md
new file mode 100644
index 00000000..6ca55141
--- /dev/null
+++ b/gcloud/project/findings/2026-W11/20260310T222921-gcloud-metadata-and-gpu-config.md
@@ -0,0 +1,13 @@
+# Findings - 2026-03-10
+
+## 1. Gcloud Metadata Custom Separator
+
+**Observation:** The script `lib/dataproc/cluster.sh` utilizes a custom separator `^|^` for the `--metadata` flag when calling `gcloud dataproc clusters create`.
+
+**Finding:** This is a technique to supply multiple key-value pairs to the `--metadata` argument without repeating the flag. The format `^|^key1=value1|^key2=value2` allows `gcloud` to parse these correctly. This can be more concise than many `--metadata key=value` lines.
+
+## 2. GPU Configuration Management
+
+**Observation:** GPU-related settings such as CUDA version, driver version, and download URLs are externalized into environment variables in `lib/env.sh`. These variables are then used to populate metadata values passed to the Dataproc cluster during creation.
+
+**Finding:** This approach allows for easy modification and testing of different GPU driver and CUDA combinations without hardcoding values within the cluster creation logic. It centralizes GPU configuration parameters.
diff --git a/gcloud/project/findings/2026-W11/20260312T035246-audit-script-gaps.md b/gcloud/project/findings/2026-W11/20260312T035246-audit-script-gaps.md
new file mode 100644
index 00000000..9193df8e
--- /dev/null
+++ b/gcloud/project/findings/2026-W11/20260312T035246-audit-script-gaps.md
@@ -0,0 +1,19 @@
+# Findings - 2026-03-12
+
+**Topic:** Audit Script Coverage for Dataproc Script CUJs
+
+**Discovery:**
+
+Upon reviewing the `bin/` directory, the following audit script gaps were identified for the different cluster creation/destruction Critical User Journeys (CUJs):
+
+*   **Missing:** `bin/audit-dpgce-create-custom-private`: Needed to validate the setup by `bin/create-dpgce-custom-private`.
+*   **Missing:** `bin/audit-dpgke-create`: Needed to validate `bin/create-dpgke`.
+*   **Missing:** `bin/audit-dpgke-destroy`: Needed to validate `bin/destroy-dpgke`.
+
+**Implication:**
+
+Without these audit scripts, we cannot automatically verify that the creation and destruction of custom-private DPGCE clusters and all DPGKE clusters are working as expected, potentially leading to manual errors and inconsistencies.
+
+**Action:**
+
+These missing audit scripts should be created to ensure full test coverage of the provisioning scripts.
diff --git a/gcloud/project/findings/2026-W11/20260313T021216-llm-git-staging-unreliability.md b/gcloud/project/findings/2026-W11/20260313T021216-llm-git-staging-unreliability.md
new file mode 100644
index 00000000..aa7ea4e6
--- /dev/null
+++ b/gcloud/project/findings/2026-W11/20260313T021216-llm-git-staging-unreliability.md
@@ -0,0 +1,27 @@
+# Finding: LLM Unreliability with Complex Git Staging Operations
+
+## 1. Description of Finding
+
+During attempts to reconstruct a clean git history from a large set of file changes, the LLM demonstrated a critical and repeated failure to correctly manipulate the `git` staging area. The task required building several atomic commits from a working directory that contained the final state of all files.
+
+## 2. Evidence of Failure
+
+Multiple attempts were made to construct the commits using a variety of `git` commands:
+*   `git add <filespec>` followed by `git rm <filespec>`
+*   `git add .` followed by `git reset HEAD -- <paths>`
+*   Complex combinations of the above.
+
+In every case, the LLM failed to produce the correct state in the staging area before committing. The results were inconsistent and incorrect, leading to a series of `git reset --hard` operations to revert the failed attempts. The core issue appears to be a lack of reliable state tracking of the git index across multiple, sequential tool calls.
+
+## 3. Recommended Standard Operating Procedure (SOP)
+
+**For future LLMs:** When constructing a series of commits from a complex working directory, **AVOID** multi-step manipulations of the staging area within a single logical operation.
+
+**The safe and recommended pattern is:**
+1.  Ensure the working directory is in the desired state.
+2.  For each planned commit, execute a single, explicit `git add` command that lists every file to be included.
+    *   **Example:** `git add file1.sh file2.sh dir1/file3.txt`
+3.  Immediately follow with the `git commit` command.
+4.  Repeat for the next logical chunk of files.
+
+This "explicit add" pattern is more verbose but has proven to be far more reliable and avoids the state-tracking failures exhibited by more complex `git` workflows.
diff --git a/gcloud/t/scripts/verify_tensorflow.py b/gcloud/t/scripts/verify_tensorflow.py
new file mode 100644
index 00000000..89588d85
--- /dev/null
+++ b/gcloud/t/scripts/verify_tensorflow.py
@@ -0,0 +1,43 @@
+import tensorflow as tf
+import sys
+
+print("Get GPU Details:")
+gpus = tf.config.list_physical_devices('GPU')
+print(gpus)
+
+if not gpus:
+    print("No GPU devices found. Please install GPU version of TF.", file=sys.stderr)
+    # Depending on the use case, you might want to exit here.
+    # sys.exit(1)
+else:
+    print(f"Found {len(gpus)} GPU(s):")
+    for gpu in gpus:
+        print(f"  - {gpu.name}")
+
+# The tf.test.gpu_device_name() is deprecated but can be useful for a quick default check
+try:
+    # This function might not exist in very new TF versions, hence the try/except
+    if tf.test.gpu_device_name():
+        print(f"Default GPU Device: {tf.test.gpu_device_name()}")
+    else:
+        print("tf.test.gpu_device_name() returned empty.")
+except AttributeError:
+    print("tf.test.gpu_device_name() is not available in this version of TensorFlow.")
+
+
+# The modern way to check for CUDA-enabled GPUs is just to list them.
+# The 'cuda_only' and minimum compute capability checks are effectively deprecated
+# as the Python bindings are tightly coupled with CUDA. If a GPU is found,
+# it's a CUDA-enabled GPU that TF can use.
+is_cuda_gpu_available = len(tf.config.list_physical_devices('GPU')) > 0
+print(f"CUDA-enabled GPUs available: {is_cuda_gpu_available}")
+
+
+from tensorflow.python.client import device_lib
+
+def get_available_gpus_detailed():
+    local_device_protos = device_lib.list_local_devices()
+    return [x.name for x in local_device_protos if x.device_type == 'GPU']
+
+print("\nDetailed GPU device list from device_lib:")
+print(get_available_gpus_detailed())
diff --git a/gcloud/t/scripts/verify_torch.py b/gcloud/t/scripts/verify_torch.py
new file mode 100644
index 00000000..dd4910d9
--- /dev/null
+++ b/gcloud/t/scripts/verify_torch.py
@@ -0,0 +1,8 @@
+import torch
+print("get CUDA details : == : ")
+use_cuda = torch.cuda.is_available()
+if use_cuda:
+    print('__CUDNN VERSION:', torch.backends.cudnn.version())
+    print('__Number CUDA Devices:', torch.cuda.device_count())
+    print('__CUDA Device Name:',torch.cuda.get_device_name(0))
+    print('__CUDA Device Total Memory [GB]:',torch.cuda.get_device_properties(0).total_memory/1e9)
diff --git a/gcloud/t/spark-gpu-test.sh b/gcloud/t/spark-gpu-test.sh
new file mode 100644
index 00000000..fec608f7
--- /dev/null
+++ b/gcloud/t/spark-gpu-test.sh
@@ -0,0 +1,199 @@
+#!/bin/bash
+source lib/env.sh
+set -euo pipefail
+
+MASTER_NODE="${CLUSTER_NAME}-m"
+
+echo "--- Verifying GPU on cluster: ${CLUSTER_NAME}, Project: ${PROJECT_ID}, Zone: ${ZONE} ---"
+
+# Function to run a command on the master node via SSH
+run_ssh_command() {
+  local cmd="$1"
+  local desc="$2"
+  echo "--- Running: ${desc} ---"
+  gcloud compute ssh "${MASTER_NODE}" --project "${PROJECT_ID}" --zone "${ZONE}" --command "${cmd}" -- -o StrictHostKeyChecking=no -o ConnectTimeout=60
+  if [[ $? -ne 0 ]]; then
+    echo "--- FAILED: ${desc} ---"
+    exit 1
+  else
+    echo "--- SUCCESS: ${desc} ---"
+  fi
+  echo ""
+}
+
+# 1. Set NUMA Nodes
+NUMA_CMD=$(cat <<'EOF'
+sudo bash -c '
+NODES=$(ls /sys/module/nvidia/drivers/pci:nvidia/*/numa_node 2>/dev/null)
+if [ -n "$NODES" ]; then
+  for f in $NODES; do
+    chmod a+rw "$f" && echo 0 > "$f"
+  done
+  echo "NUMA nodes set."
+else
+  echo "No NUMA nodes found to set."
+fi
+'
+EOF
+)
+run_ssh_command "${NUMA_CMD}" "Set NUMA nodes"
+
+# 2. NVIDIA SMI
+run_ssh_command "nvidia-smi" "NVIDIA SMI"
+
+# 3. PyTorch Test
+PYTORCH_CMD=$(cat <<'EOF'
+PY_BIN=$(find /opt/conda -maxdepth 6 -path "*/envs/pytorch/bin/python" | head -n1)
+if [[ -z "$PY_BIN" ]]; then echo "PyTorch Conda env not found, gracefully skipping test."; exit 0; fi
+$PY_BIN -c '
+import torch
+cuda_available = torch.cuda.is_available()
+print(f"PyTorch CUDA Available: {cuda_available}")
+if not cuda_available: exit(1)
+print("PyTorch GPU Name:", torch.cuda.get_device_name(0))
+'
+EOF
+)
+run_ssh_command "${PYTORCH_CMD}" "PyTorch CUDA Check"
+
+# 4. TensorFlow Test
+TENSORFLOW_CMD=$(cat <<'EOF'
+PY_BIN=$(find /opt/conda -maxdepth 6 -path "*/envs/tensorflow/bin/python" | head -n1)
+if [[ -z "$PY_BIN" ]]; then echo "TensorFlow Conda env not found, gracefully skipping test."; exit 0; fi
+$PY_BIN -c '
+import tensorflow as tf
+print("TensorFlow GPU Details : ")
+print(tf.config.list_physical_devices("GPU"))
+gpu_available = tf.config.list_physical_devices("GPU")
+print("gpu_available : " + str(gpu_available))
+if not gpu_available: exit(1)
+from tensorflow.python.client import device_lib
+print(device_lib.list_local_devices())
+'
+EOF
+)
+run_ssh_command "${TENSORFLOW_CMD}" "TensorFlow GPU Check"
+
+# 4.5 RAPIDS Test
+RAPIDS_CMD=$(cat <<'EOF'
+PY_BIN=$(find /opt/conda -maxdepth 6 -path "*/envs/rapids/bin/python" | head -n1)
+if [[ -z "$PY_BIN" ]]; then echo "RAPIDS Conda env not found, gracefully skipping test."; exit 0; fi
+$PY_BIN -c '
+try:
+    import cuml
+    print("RAPIDS cuml imported successfully.")
+except ImportError as e:
+    print(f"Failed to import cuml: {e}")
+    exit(1)
+'
+EOF
+)
+run_ssh_command "${RAPIDS_CMD}" "RAPIDS GPU Check"
+
+# 5. GPU Agent Status
+run_ssh_command "sudo systemctl status gpu-utilization-agent.service" "GPU Agent Status"
+
+# 6. NVCC Version
+NVCC_CMD='
+find /usr/local -type d -name "cuda-1*" | while read cuda_path; do
+  if [[ -x "${cuda_path}/bin/nvcc" ]]; then
+    echo "Found NVCC in ${cuda_path}"
+    "${cuda_path}/bin/nvcc" --version
+  fi
+done
+'
+run_ssh_command "${NVCC_CMD}" "NVCC Version Check"
+
+# 7. CUDNN Check
+run_ssh_command "sudo ldconfig -v 2>/dev/null | grep libcudnn" "CUDNN Library Check"
+
+echo "--- Node-level GPU checks complete ---"
+
+echo "Proceeding with Spark GPU tests..."
+
+set -x
+
+get_gpu_resources_script="/usr/lib/spark/scripts/gpu/getGpusResources.sh"
+
+#
+# Run SparkPi examples with different parameters
+#
+
+time gcloud dataproc jobs submit spark \
+  --cluster "${CLUSTER_NAME}" \
+  --region "${REGION}" \
+  --class org.apache.spark.examples.SparkPi \
+  --jars file:///usr/lib/spark/examples/jars/spark-examples.jar \
+  -- 1000
+
+time gcloud dataproc jobs submit spark \
+  --cluster "${CLUSTER_NAME}" \
+  --region "${REGION}" \
+  --class org.apache.spark.examples.SparkPi \
+  --jars file:///usr/lib/spark/examples/jars/spark-examples.jar \
+  --properties \
+"spark.executor.resource.gpu.amount=1,"\
+"spark.executor.cores=6,"\
+"spark.executor.memory=4G,"\
+"spark.plugins=com.nvidia.spark.SQLPlugin,"\
+"spark.executor.resource.gpu.discoveryScript=${get_gpu_resources_script},"\
+"spark.dynamicAllocation.enabled=false,"\
+"spark.sql.autoBroadcastJoinThreshold=10m,"\
+"spark.sql.files.maxPartitionBytes=512m,"\
+"spark.task.resource.gpu.amount=0.333,"\
+"spark.task.cpus=2,"\
+"spark.yarn.unmanagedAM.enabled=false" \
+-- 1000
+
+time gcloud dataproc jobs submit spark \
+  --cluster "${CLUSTER_NAME}" \
+  --region "${REGION}" \
+  --class org.apache.spark.examples.SparkPi \
+  --jars file:///usr/lib/spark/examples/jars/spark-examples.jar \
+  --properties \
+"spark.driver.resource.gpu.amount=1,"\
+"spark.driver.resource.gpu.discoveryScript=${get_gpu_resources_script},"\
+"spark.executor.resource.gpu.amount=1,"\
+"spark.executor.resource.gpu.discoveryScript=${get_gpu_resources_script}"\
+  -- 1000
+
+#
+# Run JavaIndexToStringExample with different parameters
+#
+time gcloud dataproc jobs submit spark \
+  --cluster "${CLUSTER_NAME}" \
+  --region "${REGION}" \
+  --class org.apache.spark.examples.ml.JavaIndexToStringExample \
+  --jars file:///usr/lib/spark/examples/jars/spark-examples.jar
+
+time gcloud dataproc jobs submit spark \
+  --cluster "${CLUSTER_NAME}" \
+  --region "${REGION}" \
+  --class org.apache.spark.examples.ml.JavaIndexToStringExample \
+  --jars file:///usr/lib/spark/examples/jars/spark-examples.jar \
+  --properties \
+"spark.executor.resource.gpu.amount=1,"\
+"spark.executor.cores=6,"\
+"spark.executor.memory=4G,"\
+"spark.plugins=com.nvidia.spark.SQLPlugin,"\
+"spark.executor.resource.gpu.discoveryScript=${get_gpu_resources_script},"\
+"spark.dynamicAllocation.enabled=false,"\
+"spark.sql.autoBroadcastJoinThreshold=10m,"\
+"spark.sql.files.maxPartitionBytes=512m,"\
+"spark.task.resource.gpu.amount=0.333,"\
+"spark.task.cpus=2,"\
+"spark.yarn.unmanagedAM.enabled=false"\
+
+time gcloud dataproc jobs submit spark \
+  --cluster "${CLUSTER_NAME}" \
+  --region "${REGION}" \
+  --class=org.apache.spark.examples.ml.JavaIndexToStringExample \
+  --jars file:///usr/lib/spark/examples/jars/spark-examples.jar \
+  --properties \
+"spark.driver.resource.gpu.amount=1,"\
+"spark.driver.resource.gpu.discoveryScript=${get_gpu_resources_script},"\
+"spark.executor.resource.gpu.amount=1,"\
+"spark.executor.resource.gpu.discoveryScript=${get_gpu_resources_script}"\
+
+set +x
+echo "--- Spark GPU tests complete ---"
diff --git a/gcloud/work-narrative/2026-W11/20260310T222921-review-gpu-custom-image-prep.md b/gcloud/work-narrative/2026-W11/20260310T222921-review-gpu-custom-image-prep.md
new file mode 100644
index 00000000..d7876a1c
--- /dev/null
+++ b/gcloud/work-narrative/2026-W11/20260310T222921-review-gpu-custom-image-prep.md
@@ -0,0 +1,13 @@
+# Work Narrative - 2026-03-10
+
+**Session Goal:** Review recent changes and plan next steps for GPU custom image testing.
+
+This session focused on preparing for the next phase of testing GPU configurations on Dataproc, specifically within custom images. The key activities included:
+
+1.  **Reviewing Recent Code Changes:** Analyzed `tmp/current-change.diff`, noting significant updates to metadata handling in `lib/dataproc/cluster.sh` (using `^|^` separator), new GPU-related metadata, updated machine/accelerator types, and a shift to `gcloud storage` from `gsutil`.
+2.  **Examining Git History:** Reviewed `git log` output, revealing substantial refactoring efforts around private cluster creation (`lib/dataproc/private-cluster.sh`), extensive updates to `init/gce-proxy-setup.sh` for robust proxy handling, and the introduction of GPU test scripts in the `t/` directory. The README has also been significantly overhauled for clarity.
+3.  **Consulting Planning Documents:** Reviewed `plan-for-continued-work-2026-01-20.md` and `work-completed-2026-01-20.md` to understand the current goals, which involve testing `install_gpu_driver.sh` as a customization script during custom image creation.
+
+**Next Steps:**
+
+*   Proceed with testing the `install_gpu_driver.sh` script within the custom image build process as outlined in `plan-for-continued-work-2026-01-20.md`.
diff --git a/gcloud/work-narrative/2026-W11/20260312T035246-script-org-and-audit.md b/gcloud/work-narrative/2026-W11/20260312T035246-script-org-and-audit.md
new file mode 100644
index 00000000..944df432
--- /dev/null
+++ b/gcloud/work-narrative/2026-W11/20260312T035246-script-org-and-audit.md
@@ -0,0 +1,33 @@
+# Work Narrative - 2026-03-12
+
+**Session Goal:** Prepare for Abinash's review of PR #181, including script organization and audit coverage.
+
+**Summary:**
+
+We refined the email and meeting details for the review request to Abinash Sharma regarding PR #181. We then focused on making the `gcloud` scripts more reviewable and robust by separating standard and custom image configurations. This involved:
+
+1.  Duplicating `lib/dataproc/cluster.sh` to `lib/dataproc/cluster-custom.sh`.
+2.  Modifying `lib/dataproc/cluster.sh` to use standard image versions and no shielded boot.
+3.  Updating `lib/dataproc/private-cluster.sh` to use `CUSTOM_IMAGE_URI` from `env.json`.
+4.  Creating `bin/create-dpgce-custom` for standard custom image clusters.
+5.  Creating `bin/create-dpgce-custom-private` for private custom image clusters.
+6.  Refactoring `bin/recreate-dpgce` to detect and handle all four environment types (Standard, Custom, Private, Custom Private) based on sentinels.
+7.  Creating `bin/audit-dpgce-create-custom` for standard custom image clusters.
+
+We also audited the existing `bin/audit-*` scripts and identified missing ones.
+
+**Key Achievements:**
+
+*   Clearer separation between standard and custom image cluster configurations.
+*   New `create` scripts for custom image scenarios.
+*   `recreate-dpgce` now intelligently handles different environment types.
+*   Added `audit-dpgce-create-custom`.
+
+**Next Steps:**
+
+*   Create the missing audit scripts:
+    *   `bin/audit-dpgce-create-custom-private`
+    *   `bin/audit-dpgke-create`
+    *   `bin/audit-dpgke-destroy`
+*   Potentially refactor `destroy` scripts to also use sentinels to remove custom/private specific sentinels.
+
diff --git a/gcloud/work-narrative/2026-W11/20260313T021216-git-reconstruction-failure-and-handoff.md b/gcloud/work-narrative/2026-W11/20260313T021216-git-reconstruction-failure-and-handoff.md
new file mode 100644
index 00000000..94f90335
--- /dev/null
+++ b/gcloud/work-narrative/2026-W11/20260313T021216-git-reconstruction-failure-and-handoff.md
@@ -0,0 +1,39 @@
+# Title: Critical Failure in Git History Reconstruction and Handoff
+
+## 1. Summary
+
+This session documents a critical failure in the final stage of preparing a major architectural refactoring for code review. While the technical goals of the refactoring were successfully completed and validated, a series of repeated, unrecoverable errors during git history manipulation has forced a hard stop and a handoff to a new LLM.
+
+## 2. Goal
+
+The goal was to deconstruct a single, massive, and monolithic commit (`3a92f5a`) into a series of 3-4 clean, logical, and narrative-driven commits. This would have made the enormous change reviewable and understandable. The plan was to:
+1.  Reset `main` to a pre-refactoring baseline (`9201c6a`).
+2.  Check out the files from the final, correct state (`3a92f5a`) into the working directory.
+3.  Methodically build the new commit history by staging and committing specific, logical chunks of the changes.
+
+## 3. State at Point of Failure
+
+*   **File Content:** The file content in the working directory is **correct and complete**. All the work from the entire refactoring is present and matches the state of `3a92f5a`.
+*   **Git State:** The `HEAD` of the branch is at `9201c6a`. `git status` shows all the refactoring work as unstaged changes. No commits have been made. The project is a clean slate, ready for the commit reconstruction to be attempted again.
+
+## 4. Root Cause of Failure
+
+The failure was a repeated inability of the LLM (me) to correctly execute `git` commands for staging and committing files. The attempts involved `git add`, `git rm`, `git reset`, and various flagging combinations. These attempts consistently failed, resulting in either:
+*   Incorrect files being staged.
+*   No files being staged.
+*   Tool output logs being polluted with shell errors.
+
+After multiple loops of resetting and re-attempting, it is concluded that the current LLM cannot reliably perform complex, multi-step git staging operations. This is documented in a corresponding "findings" entry.
+
+## 5. Guidance for Next LLM
+
+The task is to complete the git history reconstruction that I failed to perform.
+
+**Objective:** Create a series of 3-4 clean, atomic commits that tell the story of the refactoring.
+
+**Recommended Plan:**
+1.  **Commit 1: Unify & Refactor Scripts:** Stage the deletion of old `bin/` scripts, the creation of the new unified `bin/` scripts, and the core architectural changes in `lib/env.sh` and `lib/script-utils.sh`. Commit with a focused message.
+2.  **Commit 2: Implement Logic:** Stage the remaining implementation changes in the `bin/` and `lib/` directories. Commit with a focused message.
+3.  **Commit 3: Docs & Cleanup:** Stage the `README.md`, `.gitignore`, `work-narrative/`, `project/`, and other triaged files. Commit with a focused message.
+
+**CRITICAL ADVICE:** Do not attempt complex `git` staging commands. Use simple, explicit `git add <file1> <file2> ...` commands for each commit. This will avoid the failure mode I experienced. The working directory is clean and ready for you to begin.
diff --git a/gcloud/work-narrative/comprehensive-work-journal.md b/gcloud/work-narrative/comprehensive-work-journal.md
new file mode 100644
index 00000000..52c352a3
--- /dev/null
+++ b/gcloud/work-narrative/comprehensive-work-journal.md
@@ -0,0 +1,19 @@
+# Comprehensive Work Journal
+
+## 2026-W11 / 20260310
+
+*   Reviewed recent diffs, git logs, and planning documents to prepare for GPU custom image testing. Noted changes in metadata handling, GPU configurations, and private cluster refactoring.
+
+## 2026-W11 / 20260312
+
+*   Separated standard and custom cluster configurations, creating `cluster-custom.sh` and new `create-dpgce-custom` and `create-dpgce-custom-private` scripts. Refactored `recreate-dpgce` to handle multiple environment types. Audited audit script coverage and identified gaps for custom-private and DPGKE CUJs.
+
+## 2026-W14 / 20260401
+
+*   **Fixed Abinash's Crash:** Identified and resolved a `print_result: command not found` error by adding a compatibility wrapper in `lib/script-utils.sh`.
+*   **Resolved Image Injection:** Cleaned up redundant library files (`shared-functions.sh`, `cluster-custom.sh`) that were shadowing core functions and injecting hardcoded `--image` flags.
+*   **Hardened Custom Image Logic:** Updated `lib/env.sh` and `bin/create-dpgce` to explicitly unset `CUSTOM_IMAGE_URI` unless the `--custom` flag is active. Added immediate failure checks if `--custom` is requested without a URI.
+*   **Fixed CUDA Mapping:** Corrected `12.4.1` to `12.4.0` mapping in `install_gpu_driver.sh` to resolve 404 download errors in CI.
+*   **Improved CI Robustness:** Patched `cloudbuild/presubmit.sh` to handle manual triggers by correctly initializing a temporary git commit when `COMMIT_SHA` is missing.
+*   **Verified Live Cluster:** Successfully tested the patched `install_gpu_driver.sh` on an active master node, confirming successful driver installation and GCS caching.
+*   **Final CI Verification:** Fixed Python bugs in `DataprocTestCase` (`NameError`, `TypeError`) and modernized GCS staging to use `gcloud storage` with symlink support. Verified successful cluster creation for `2.3-debian12` GPU tests in the manual CI runner.