Add CI sync check between requirements.txt and system-requirements-lock.txt

[SarahAsad23]

Task Summary

Currently there is no automated validation to ensure that requirements.txt stays in sync with system-requirements-lock.txt.

A GitHub CI check should be added to verify that:

• all packages in requirements.txt are reflected in system-requirements-lock.txt
• package versions remain consistent
• updates to one file do not accidentally leave the other outdated

This would help prevent dependency mismatches and reduce environment inconsistencies during deployment and package installation.

Task Type

• Refactor / Cleanup
• DevOps / Deployment / CI
• Testing / QA
• Documentation
• Performance
• Other

[yangzhang75]

/take

[Yicong-Huang]

why do we keep two copies of the same information and require them to sync? cc @kunwp1 @SarahAsad23

[kunwp1]

#4902 (comment)

This might answer your question: system-requirements-lock.txt is a superset of requirements.txt that explicitly includes all indirect dependencies.

[Yicong-Huang]

This might answer your question: system-requirements-lock.txt is a superset of requirements.txt that explicitly includes all indirect dependencies.

I have several doubts about this approach. My understanding is that dependency resolution is both environment-sensitive and time-sensitive. The same requirements.txt can resolve to different installed versions depending on the Python version, OS, Linux distribution, hardware capabilities such as CUDA support, platform-specific wheels, and the package index state at the time of installation.

Because of that, I am not sure whether we should maintain system-requirements-lock.txt as a static checked-in file. A lock file is useful when it captures a specific environment at a specific point in time, but if this file is expected to work across many different environments, it may become misleading or stale.

A few questions I have are:

1. What specific environment is system-requirements-lock.txt generated for? For example, is it tied to a specific Python version, OS, Linux distribution, and hardware configuration?
2. Is this lock file expected to be portable across all supported environments, or only reproducible for one specific environment?
3. How often is this file expected to be regenerated, and what process ensures that it stays in sync with the actual target environments?
4. If different environments resolve dependencies differently, should we have separate generated lock files per target environment instead of one shared static file?

My suggestion is that requirements.txt should remain the source of truth for declared dependencies, while system-requirements-lock.txt should be treated as an environment-specific generated artifact. It should be generated and resolved on the fly in the target environment where it is used, such as CI, Docker builds, or release jobs. This would make the lock file accurate for that specific environment and point in time, instead of maintaining a static checked-in lock file that may not generalize across different platforms.

[SarahAsad23]

We will close this issue because we have decided not to maintain system-requirements-lock.txt as a checked-in file. Keeping the lock file synchronized with requirements.txt introduces significant maintenance overhead, and the lock file itself is inherently environment specific. Dependency resolution can vary across operating systems, Python versions, hardware configurations, and over time as package repositories evolve, making a static shared lock file difficult to keep accurate.

Instead, we will move to a runtime-generated approach. The system dependencies will be resolved in the target environment when needed, likely by installing the system requirements into a temporary environment and generating the fully resolved dependency list using pip freeze. This ensures that conflict detection is performed against the actual installed dependency set for the current environment rather than against a potentially stale or inaccurate lock file.