Depend on pyspark-client instead of the full pyspark distribution

[ajma]

What

Swaps the runtime dependency from pyspark[connect]~=4.0.0 to pyspark-client~=4.0.0.

The client only ever talks to a remote Spark Connect endpoint, so the Spark JVM jars bundled in the full distribution are dead weight. Apache Spark 4.0 publishes pyspark-client for exactly this case — the same Python tree, remote-only, with jars/ stripped.

	installed size
pyspark[connect]~=4.0.0	461 MB (442 MB of it pyspark/jars/)
pyspark-client~=4.0.0	14 MB

Every symbol this library imports is present in pyspark-client: pyspark.sql.connect.client.DefaultChannelBuilder, pyspark.sql.connect.session.SparkSession, pyspark.sql.utils.to_str, pyspark.sql.connect.shell.progress.StageInfo.

Guarding the failure mode this introduces

The two distributions have different names but provide the same pyspark module, so pip cannot see them as conflicting and installs both side by side. That turns a resolver-visible conflict into a silent file overlay.

It is easy to hit. Starting from a clean install of this library, pip install sparksql-magic — which the README instructed until this PR — yields pyspark 4.2.0 next to pyspark-client 4.0.4, and the first import then fails with:

PySparkImportError: [PACKAGE_NOT_INSTALLED] zstandard >= 0.25.0 must be installed

which names neither package. Any dependency declaring pyspark does the same.

_check_pyspark_installation() in __init__.py now detects the mismatch and points at the fix. It runs before the from .session import ... line, since that import is what fails first; the existing conflicting-package check below it would never be reached. Identical versions are left alone — every shared .py file is byte-identical at the same version, verified by comparing the two trees at 4.0.4, so that combination genuinely works in either install order.

Four unit tests cover mismatch, match, and each distribution alone.

sparksql-magic

sparksql-magic declares pyspark>=2.3.0 in its wheel metadata and setup.py, so it is now installed with --no-deps in the integration workflow, the README and DEVELOPING.md, and dropped from requirements-dev.txt with a comment.

Safe because the pin is over-broad rather than load-bearing: sparksql_magic/sparksql.py:7 imports only from pyspark.sql import SparkSession, which pyspark-client satisfies. Both sparksql-magic integration tests pass under the slim install.

Local Spark handoff

ManagedSparkSession.Builder.getOrCreate() has a is_dataproc_batch() branch (session.py:627) that hands off to a local classic Spark session. pyspark-client cannot start one, so test_create_local_spark_session — the only coverage of that branch — errored on the first CI run.

It is now guarded on pyspark.util.is_remote_only() so slim environments skip it with a clear reason, and a local-spark CI job installs the full distribution and runs it. The test needs no GCP access and takes ~7s, so it does not belong behind 80 minutes of cloud provisioning. Its dependency lives in requirements-local-spark.txt, consumed only by that job, deliberately not in requirements-test.txt — the unit and integration jobs install that file and must keep running against the slim install we ship.

The real Dataproc batch runtime is unaffected. is_remote_only() is decided by whether pyspark.core imports. Installing pyspark-client over a full pyspark leaves pyspark/core and pyspark/jars intact, so it stays False and classic mode keeps working.

Verification

• Integration suite: passed, 1h23m on the rebased branch.
• Unit tests: 169 passed plus 13 subtests. One local-only failure, test_create_session_without_application_default_credentials, reproduces identically on main and passes in CI.
• local-spark job: passes in 40s in CI, reproduced locally in an exact environment simulation with all GCP env vars unset.
• python -m build + twine check pass; wheel metadata shows Requires-Dist: pyspark-client~=4.0.0.
• pyink clean.

An earlier integration run failed on test_dpip_install_success[None] with RETRIES_EXCEEDED behind a websocket handshake timeout in proxy.py. That path uses the websockets dependency directly and the same test passed before and after, so it is flaky infrastructure rather than a regression here.

[gemini-code-assist]

Code Review

This pull request transitions the project's dependency from the full pyspark distribution to the lightweight pyspark-client package, updating setup.py, requirements-dev.txt, and documentation to reflect this change and prevent namespace shadowing. The reviewer suggests utilizing extras_require in setup.py to handle remote-only and local Spark runtime configurations more cleanly, and updating the README installation instructions accordingly.

[gemini-code-assist]

If we transition to using extras_require in setup.py to avoid dependency conflicts and namespace shadowing, we should update the installation instructions here to guide users on using the appropriate extra.

For example:

# For remote-only (lightweight, ~14 MB)
pip install google-cloud-spark-connect[client]

# For local Spark runtime support or Dataproc Serverless environments
pip install google-cloud-spark-connect[local]

[ajma]

Fixing the integration failure (cd23806)

The first integration run came back 44 passed, 1 skipped, 1 error, stopping at the error because the job runs with -x:

ERROR at setup of test_create_local_spark_session
tests/integration/test_session.py:775: in local_spark_session
    session = PySparkSession.builder.master("local").getOrCreate()
E   PySparkRuntimeError: [CONNECT_URL_NOT_SET]

Root cause

test_create_local_spark_session sets DATAPROC_WORKLOAD_TYPE=batch and asserts that ManagedSparkSession.builder.getOrCreate() returns a real local classic Spark session — it is the only test covering the batch handoff branch at session.py:599. pyspark-client ships no JVM jars and cannot start one.

The test is not wrong and the library code is not wrong. The test environment stopped providing a capability the test depends on. Everything else in the suite passed on the slim install, including both sparksql-magic tests, which confirms the --no-deps handling works.

The fix

1. Guard the test on its actual requirement. tests/integration/test_session.py now carries a requires_local_spark mark backed by pyspark.util.is_remote_only(), so a pyspark-client environment skips with a stated reason rather than erroring on an unrelated-looking CONNECT_URL_NOT_SET:

   SKIPPED [1] requires the full pyspark distribution (a local Spark runtime)
   

2. Keep the path covered in CI. A new local-spark job in tests.yaml installs pyspark[connect]~=4.0.0 over the requirements and runs that test. It needs no GCP credentials and finishes in about 90 seconds, so it does not belong in the 70-minute integration suite.

The integration suite deliberately stays on the slim install, so the 44 Connect tests keep exercising the dependency set we actually ship. Only the one test that genuinely needs a JVM gets an environment with one.

Why the real Dataproc batch runtime is unaffected

is_remote_only() is decided by whether pyspark.core imports — a file-presence test, not a flag:

try:
    from pyspark import core  # noqa: F401
    _is_remote_only = False
except ImportError:
    _is_remote_only = True

pyspark-client omits core/ and jars/; the full distribution has both. Overlaying pyspark-client onto a full pyspark install in a clean venv leaves both directories intact and is_remote_only() returns False, so classic mode keeps working. A batch image that ships its own pyspark is therefore not degraded by this dependency. I had flagged this as an open risk when opening the PR; that concern was overstated and the description has been corrected.

Verification

check	result
Failure reproduced in slim venv	errors with CONNECT_URL_NOT_SET, <1s
Same test, full pyspark venv	passes, 5.5s, all GCP env vars unset
Skip guard, slim venv	skips with reason, does not error
New job command, exact env simulation	passes, 7.4s
Run unit tests in CI	pass, 1m45s
Run local Spark tests in CI	pass, 1m27s

Integration suite is re-running.