Retry stream read errors in SimpleDownloader#581
Open
agoncharov-reef wants to merge 7 commits into
Open
Conversation
ppolewicz
reviewed
Jul 8, 2026
| and download_version.content_encoding | ||
| and download_version.api.api_config.decode_content | ||
| ): | ||
| if not self.SUPPORTS_DECODE_CONTENT and download_version._should_be_decoded: |
Collaborator
There was a problem hiding this comment.
_should_be_decoded is private and should not be accessed from outside. Rename to should_be_decoded?
| # or something and the server closes connection, while neither tcp or http have a problem | ||
| # with the truncated output, so we detect it here and try to continue | ||
|
|
||
| num_tries = 5 # this is hardcoded because we are going to replace the entire retry interface soon, so we'll avoid deprecation here and keep it private |
Collaborator
There was a problem hiding this comment.
I don't think you should un-hardcode it here as when this will turn into an interface that folks can rely on, it will be difficult to maintain the interface over the refactoring. Move to module scoped _MAX_RETRIES?
| return parse_http_date(self.expires) | ||
|
|
||
| @property | ||
| def _should_be_decoded(self) -> bool: |
Collaborator
There was a problem hiding this comment.
if the rest of the code expects DownloadVersion-like objects to have this method or else it's going to crash, then we have to add it to interface, meaning we have to document it so that those who inherit will know what it is
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SimpleDownloaderalready had range-based retry logic for truncated downloads (e.g. when the server closes the connection mid-stream), butChunkedEncodingError,ConnectionError, andContentDecodingErrorraised duringiter_contentbypassed that path and failed the download immediately.This PR catches those exceptions during both the initial read and follow-up range requests, logs them at debug level, and lets the existing retry loop resume. Retry is skipped when content is decoded (
content_encodingset anddecode_content=True), since a partially decoded stream cannot be resumed with byte ranges.Also fixes
v1.B2Apinot exposingapi_config, and ensures decoded downloads still get truncation checks after download (hash verification remains skipped for them).