fix(bigquery): handle 409 errors and attach to existing job

[alvarowolfx]

Towards #6717

[gemini-code-assist]

Code Review

This pull request introduces robust handling for duplicate job errors (409 Already Exists) during BigQuery query executions, allowing the client to adopt an existing job from a previous attempt instead of failing. It also refactors retry policies to ensure non-pre-RPC errors are only retried for idempotent requests, and adds comprehensive unit tests. Feedback is provided to optimize the check_job_status helper by using std::mem::take to avoid an unnecessary clone of the error list, adhering to the repository's performance guidelines.

[gemini-code-assist]

Since job is consumed by check_job_status, we can avoid cloning status.errors by taking ownership of the errors using std::mem::take on a mutable reference to the status. This avoids unnecessary allocations and clones, adhering to the repository style guide's guidance on avoiding unnecessary clones.

Suggested change

	fn check_job_status(job: Job) -> Result<Job> {
	if let Some(status) = job.status.as_ref()
	&& status.error_result.is_some()
	{
	let errors = status.errors.clone();
	return Err(QueryError::JobFailed { errors });
	}
	Ok(job)
	}
	fn check_job_status(mut job: Job) -> Result<Job> {
	if let Some(status) = job.status.as_mut()
	&& status.error_result.is_some()
	{
	let errors = std::mem::take(&mut status.errors);
	return Err(QueryError::JobFailed { errors });
	}
	Ok(job)
	}

References

1. Scrutinize expensive uses of clone() (i.e. for Strings, not for Arcs). Is it necessary to copy the data? Can we move the data instead? ^(link)

[alvarowolfx]

might need to split since handling this kind of error is tricky.

request_id only merges duplicates that are still in flight. Once the request has been answered, later duplicates collide with the job it created. 409 on jobs.query only happens if the query executes correctly, but we get a transport failure to read results, so RPC level retries try to re-execute and we get an 409.

[codecov]

Codecov Report

❌ Patch coverage is 98.05825% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.02%. Comparing base (8fb991a) to head (b99436c).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/bigquery/src/query/execution.rs	98.04%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6780      +/-   ##
==========================================
- Coverage   97.03%   97.02%   -0.01%     
==========================================
  Files         326      326              
  Lines      107511   107706     +195     
==========================================
+ Hits       104322   104507     +185     
- Misses       3189     3199      +10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[alvarowolfx]

superseded by #6801 #6799