mirror of https://github.com/devine-dl/devine.git
Remove unnecessary HEAD calls in requests downloader
HEAD requests were made to sum a total file size of the download operation. However, the downloader is may be used on URLs where the content is not segmented media. Therefore, the server may not support or respond with the Content-Length header which causes the requests downloader to crash before it even gets a chance to begin downloading. Even still, this total size value isn't really necessary, and would cause possibly 100s of HEAD requests (in quick succession of each other) on segmented sources. It would also add up-front delay before it actually starts to download.
This commit is contained in:
parent
e7dc138c0f
commit
2a4e9505f1
|
@ -48,13 +48,8 @@ def requests(
|
|||
if proxy:
|
||||
session.proxies.update({"all": proxy})
|
||||
|
||||
total_size = sum(
|
||||
int(session.head(url).headers["Content-Length"])
|
||||
for url, _ in uri
|
||||
)
|
||||
|
||||
if progress:
|
||||
progress(total=total_size)
|
||||
progress(total=len(uri))
|
||||
|
||||
download_sizes = []
|
||||
last_speed_refresh = time.time()
|
||||
|
@ -70,7 +65,7 @@ def requests(
|
|||
f.write(chunk)
|
||||
written += download_size
|
||||
if progress:
|
||||
progress(advance=download_size)
|
||||
progress(advance=1)
|
||||
|
||||
now = time.time()
|
||||
time_since = now - last_speed_refresh
|
||||
|
|
Loading…
Reference in New Issue