Skip to content

feat: Add support for deleting partitions with logic operators other than equals#194

Merged
subkanthi merged 2 commits into
masterfrom
193-delete-partition-support-for-other-logical-operators
Jul 21, 2026
Merged

feat: Add support for deleting partitions with logic operators other than equals#194
subkanthi merged 2 commits into
masterfrom
193-delete-partition-support-for-other-logical-operators

Conversation

@subkanthi

Copy link
Copy Markdown
Collaborator

closes: #193

@subkanthi subkanthi linked an issue Jul 21, 2026 that may be closed by this pull request
@subkanthi
subkanthi marked this pull request as ready for review July 21, 2026 10:07
@subkanthi

Copy link
Copy Markdown
Collaborator Author
 java -jar ../../ice/target/ice-0.0.0-SNAPSHOT-shaded.jar delete nyc.taxis_p_by_day  \
  --partition '[{"name": "tpep_pickup_datetime_day", "op": "greater_than_or_equal", "values": ["2025-01-31"]}]' --dry-run=false

BEFORE

spark-sql (nyc)> select distinct date(tpep_pickup_datetime) from taxis_p_by_day order by date(tpep_pickup_datetime) desc;
2025-02-01
2025-01-31
2025-01-30

AFTER

spark-sql (nyc)> select distinct date(tpep_pickup_datetime) from taxis_p_by_day order by date(tpep_pickup_datetime) desc;
2025-01-30
2025-01-29
2025-01-28
2025-01-27

@subkanthi
subkanthi requested a review from xieandrew July 21, 2026 16:38
@subkanthi subkanthi added the ice Relates to ice label Jul 21, 2026
@xieandrew xieandrew changed the title Added support for deleting parititions with logic operators other than equals. feat: Add support for deleting partitions with logic operators other than equals Jul 21, 2026

@xieandrew xieandrew left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logic looks good, just a small issue with the example docs

Comment thread examples/scratch/README.md Outdated
Comment on lines +40 to +44
--partition '[{"name": "tpep_pickup_datetime", "values": ["2024-12-31T23:51:20"]}]' --dry-run=false

# delete a range of partitions using a comparison operator (default op is equals)
ice delete nyc.taxis_p_by_day \
--partition '[{"name": "tpep_pickup_datetime", "op": "greater_than_or_equal", "values": ["2024-12-31T23:51:20"]}]' --dry-run=false

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The value of "name" should be "tpep_pickup_datetime_day" instead of "tpep_pickup_datetime" for both delete calls. The logic was changed in an old PR (#94) and these docs weren't updated.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated.

@subkanthi
subkanthi requested a review from xieandrew July 21, 2026 19:53
@subkanthi
subkanthi merged commit f0094fc into master Jul 21, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ice Relates to ice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Delete partition support for other logical operators

2 participants