Conversation
2f385d4 to
7d77c41
Compare
Bypass django-storages eager file buffering when the content app proxies S3 or Azure artifacts. Preserve the generic storage fallback and HTTP range behavior. closes pulp#7806 Assisted By: Codex (GPT-5) Terra 5.6
7d77c41 to
2ed7659
Compare
| @@ -0,0 +1,166 @@ | |||
| """Temporary private streaming adapters for object-storage response bodies. | |||
There was a problem hiding this comment.
Ideally none of these wrappers are necessary, and django-storages would allow us to do something like this natively: jschneier/django-storages#1568
But for now, we cannot, and the buffering implementation is genuinely unsuitable for sufficiently large files (context: a user has reported that they are storing files ranging up to 60 or even 300 gigabytes)
There was a problem hiding this comment.
Obviously this is relatively invasive, so we need to be very careful with testing.
There was a problem hiding this comment.
Alternatively, we could temporarily fork django-storages and add better APIs directly to the implementation. It is not in a great place maintenance-wise anyway currently.
There was a problem hiding this comment.
Here is what the upstream PR looks like jschneier/django-storages#1569
|
@mdellweg I'm marking this draft because it probably needs more extensive testing, however, I would still like feedback on the direction and approach. e.g. if you want to try the "vendor/fork django-storages" approach. Both the issue itself and the upstream status of the library is a real issue we're going to need to deal with one way or the other. |
|
First, I don't know what the best approach is. Each approach has pro's and con's. |
This reverts commit 2ed7659.
Use the forked django-storages open_stream API for supported remote storage backends so Pulp can proxy large artifacts without opening their eager, seekable file objects. Preserve the existing generic storage fallback and HTTP range behavior. closes pulp#7806 Assisted By: Codex (GPT-5) Terra 5.6
cc3d9d2 to
2b3471d
Compare
Bypass django-storages eager file buffering when the content app proxies S3 or Azure artifacts. Preserve the generic storage fallback and HTTP range behavior.
closes #7806
Assisted By: Codex (GPT-5) Terra 5.6
📜 Checklist
See: Pull Request Walkthrough