Since I'm using `futures.as_completed()`, it will never ever for loop over all tracks and segments and will forever be stuck in the primary thread of the operation. I.e., main thread for the download track threads, or the track thread for the download segment threads.
I've also removed all future cancelled checks as they will never be cancelled before they get the chance to run, because no future cancel calls are made anymore.
I've removed asyncio usage as it's generally unnecessary. If you want to run aria2c under a thread, run it under a thread. In the case for devine, this would take another thread, and would be another thread layer deep. Pointless. Would affect speed.
With this change I've been able to improve the aria2c progress capture code quite a bit.
While console.log is currently removing the need for `Padding(..., (0, 5))` as it is overwritten to do it automatically, but in terms of purpose the `console.print` function is more logical.
I hope to find a way to automate the padding within console.print in the future, but for now this will work.
It seems the commit I made to do this change initially seemed to help, it was actually pointless and issues I had were caused by other problems.
For consistency it seems best to stick with the logging module with the RichHandler applied. Using just console.log means being unable to control the log level and which level of logs appear.
This pull request adds the features that are detailed in this issue:
https://github.com/devine-dl/devine/issues/2
Also changes some debug logs to info logs, as the information would generally be wanted. Also changes some logging logs to console.logs.
I've moved log.info calls to console.log calls to reduce conflicts of logs at the same time as console refreshes, but also to reduce unnecessary log level text being printed to the console.
We don't need to know if a log is an `info` level log, but I've kept log.error's and such as we would want to know if a log is an error log and such.
This changes all logs via logging to be printed to the rich console instead. Specifically, using my custom ComfyRichHandler making the logs padded horizontally.
In this change the time is no longer printed. The log level is still printed but it's now the full log name in bold.
These add a bit of margin around all console prints/logs. It makes the logs feel a bit more comfortable rather than being crammed edge-to-edge in the terminal.
The methods used here are by no means good, but they work quite well. If you can find a better way to do it, please make a pull request.
These seem to occur when using HTTP+S proxies when connecting to a HTTPS URL. E.g., NordVPN proxies to https://google.com. While not ideal, we can't solve this solution and the warning logs are quite annoying.
It seems on Windows the pathlib.iterdir() function is always in order. However, on Linux or at least some machines this is not the case. This change fixes the order.
If you think you were affected, check your previous downloads that used DASH or HLS segmentation and make sure they dont randomly change scenes out of order.
Turns out, even if you manually set the Range header AND the server has full support, it does not work. It will act like it works, but it seems internally aria2c gets confused on what bytes it requested, what it returned, and it will either just download the full file, or the range requested (but still complain, and freeze!).
Yikes.
For some reason I had the partial removed and replaced with the direct function, meaning services using `title` and `track` during the license and service cert functions would crash.
This fixes the usage of vaults across different threads. It now makes a truly unique connection for each thread. The previous code did this as well, but put back the connection from x thread, to re-use in y thread. Now it simply creates and reuses the connection on their own thread.
Once the thread is closed, the data is now also garbage collected. This now reduces the risk of filling up memory over time.
Just like the commit for HLS multi-threading, this mimics the -j=16 system of aria2c, but manually via a ThreadPoolExecutor. Benefits of this is we still keep support for the new system, and we now get a useful progress bar via TQDM on segmented downloads, unlike aria2c which essentially fills the terminal with jumbled download progress stubs.
This mimics the -j=16 system of aria2c, but manually via a ThreadPoolExecutor. Benefits of this is we still keep support for the new system, and we now get a useful progress bar via TQDM on segmented downloads, unlike aria2c which essentially fills the terminal with jumbled download progress stubs.
This is required in segmented scenarios when multi-threaded where the same `track.path` would be get and set from possibly at the same time. It's also just better logically to do it this way.
It seems if a service sets the expiration time in seconds or by a timestamp, then this resolves as expired up to 14 hrs too early, or 14 hours too late, depending on what your timezone is relative to UTC+00:00.
However, I haven't fully confirmed if this is the right fix to make. As in, I'm not sure if changing datetime.utcnow() to datetime.now() should be done, or the other way around. However, I've had multiple people tell me changing it this way worked for them so I'm just going to roll with it.
The new system now downloads and decrypts segments individually instead of downloading all segments, merging them, and then decrypting. Overall the download system now acts more like a normal player.
This fixes#23 as the new HLS download system detects changes in keys and init segments as segments are downloaded. DASH still only supports one period, and one period only, but hopefully I can change that in the future.
Downloading code is now also moved from the Track classes to the manifest classes. Download progress is now also actually helpful for segmented downloads (all HLS, and most DASH streams). It uses TQDM to show a progress bar based on how many segments it needs to download, and how fast it downloads them.
There's only one down side currently. Downloading of segmented videos no longer have the benefit of aria2c's -j parameter. Where it can download n URLs concurrently. Aria2c is still used but only -x and -s is going to make a difference.
In the future I will make HLS and DASH download in a multi-threaded way, sort of a manual version of -j.
If EXT-X-KEY of Method=NONE is encountered, it assumes no DRM should be used, even if other supported DRM may have already been or is going to be iterated.
Path.rename() cannot move data to different drives. It can only rename the path reference on the same file system. However, shutil.move() will move the data while also changing it's name.
If the manifest is hidden behind a redirect, the url is not updated and the segments are created with the old url.
I found a French service where this situation occurred.
After this change, it was possible to correctly parse the mpd.
This happened because the WEBVTT header was segmented and appended to each other without enough newline separation so pycaption thought it was an actual caption and to be kept.
This was implemented to know the difference between services explicitly stating 5.1 channel audio, to a service explicitly stating 6.0 channel audio that is likely 5.1 channel layout.
However, since technically both are 6 channels, this is unnecessary and just causes complications.
rlaphoenix
Arias800
varyg