ninjajiraiya
/
devine
forked from DRMTalks/devine
1
0
Fork 0
Code Pull Requests Activity

Only verify download size in requests downloader if possible 

Some Servers may not response with the Content-Length header, even if it's from segmented media. I.e., if it's a subtitle URL. The requests downloader required the header to be present as it downloads each URL, which is not possible.

Now it tries to get it if possible, and verifies the download size with the Content-Length value if it could be obtained.
This commit is contained in:
rlaphoenix 2023-05-16 20:49:43 +01:00
parent 2a4e9505f1
commit bdc1203514
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
devine/core/downloaders/requests.py
View File

@ -57,7 +57,7 @@ def requests(
for url, out_path in uri: for url, out_path in uri:
out_path.parent.mkdir(parents=True, exist_ok=True) out_path.parent.mkdir(parents=True, exist_ok=True)
stream = session.get(url, stream=True) stream = session.get(url, stream=True)
file_size = int(stream.headers["Content-Length"]) file_size = stream.headers.get("Content-Length")
with open(out_path, "wb") as f: with open(out_path, "wb") as f:
written = 0 written = 0
for chunk in stream.iter_content(chunk_size=1024): for chunk in stream.iter_content(chunk_size=1024):
@ -77,7 +77,7 @@ def requests(
progress(downloaded=f"{filesize.decimal(download_speed)}/s") progress(downloaded=f"{filesize.decimal(download_speed)}/s")
last_speed_refresh = now last_speed_refresh = now
download_sizes.clear() download_sizes.clear()
if written < file_size: if file_size and written < int(file_size):
raise ValueError( raise ValueError(
f"{url} finished downloading unexpectedly, got {decimal(written)}/{decimal(file_size)}") f"{url} finished downloading unexpectedly, got {decimal(written)}/{decimal(file_size)}")