Related to issue #1047
After this it should be possible to build a working `packager` application from the `cmake` branch. Some further logging improvements may be needed to get full parity with the `main` branch in terms of ability to do verbose debug logging, but other than that everything is expected to work.
This is an automated sync of common workflows for this organization.
The upstream source is:
4245160a0b
Co-authored-by: Shaka Bot <shaka-bot@users.noreply.github.com>
As per the AV1 spec, the codec string may contain optional color values.
This extracts the missing color information from the mp4 `colr` atom, if
present, and generates the full AV1 codec string.
Closes#1007
This is an automated sync of common workflows for this organization.
The upstream source is:
208556c5ea
Co-authored-by: Shaka Bot <shaka-bot@users.noreply.github.com>
This fixes our use of std::filesystem to interpret all paths names as
UTF8. Before this, UTF8 paths did not work correctly in all cases.
This also adds a new unit test to cover this case.
On Windows, it is critical that a UTF8 locale be set at runtime.
Applications linking with Packager as a library should call setlocale(),
and the Packager frontends now do this automatically after converting
wide character arguments into narrow strings.
Closes#652
PR #1242 introduced some bad assertions that went unnoticed because we
are not yet building those modules in the cmake branch. This fixes those
so that ASSERT_ macros are not used in non-void functions, and so that
assumptions about a successful parse are always after an ASSERT_ that
the data was loaded properly.
All ASSERT_ macros from #1242 were audited, and the others were all
fine.
All gtest-based tests should now use add_gtest. Results from these tests
will be surfaced more prominently in GitHub Actions. Before this, it was
hard to find test failures in the log.
LOG(FATAL) aborts the executable, which means gtest can't write any
reports it might be configured to write when tests are complete. This
will interfere with reporting.
This converts the only use of LOG(FATAL), which was in
ReadTestDataFile(), to LOG(ERROR). This also updates test cases to avoid
crashing when ReadTestDataFile() returns an empty buffer.
Our workflows and Dockerfiles now use Ninja on Linux & Mac, which
enables safe parallel builds. This significantly speeds up our
workflows.
GitHub Actions typical compilation times (build step only):
- Linux 19m => 9m
- macOS 23m => 8m
- Windows 12m => 10m
- Linux arm64 (self-hosted) 72m => 29m
- Docker build 25m => 14m
Overall workflow time: 84m => 33m
Compilation time on my workstation (12 CPUs @3.3GHz): 15m => 3m
This also adds a new environment variable "PACKAGER_LOW_MEMORY_BUILD".
If defined when CMake is first run, this will configure the build to
disable parallel linking to reduce memory usage. This helps us avoid
failures on our self-hosted arm64 machines, where 6 CPUs share 4GB of
RAM.
NOTE: Parallel builds are **NOT** recommended with Unix Makefiles due to
the use of excessive RAM during parallel linking. Unix Makefiles, unlike
Ninja, cannot be configured to restrict parallel linking during a
parallel build. Anecdotally, parallel builds with Makefiles have
exhausted a system with 32GB RAM. (My workstation.)
In a follow-up, I will update the build documentation to refer to CMake
and recommend all of the flags now used in our workflows.
CentOS 8 is EOL, and the official docker images will not be updated. See
https://www.centos.org/centos-linux-eol/
This switches to the latest release (CentOS 9 Stream). Since CentOS no
longer provides official images, this uses a third-party image provider
(https://hub.docker.com/u/tgagor)
This change also exposed a warning in absl that needed to be silenced in
the default version of GCC in CentOS 9 Stream (GCC 11.4).
The `__FILE__` macro does not always get you an absolute path, so our
assumptions about it in test_data_util.cc were flawed.
If `foo.c` references `__FILE__`, something like `gcc -c foo.c`, will
define `__FILE__` as `"foo.c"`. If you use `gcc -c /path/to/foo.c`, then
`__FILE__` will be `"/path/to/foo.c"`.
The Ninja generator for CMake only generates absolute paths for source
files in certain CMake versions. (Exact range unknown.) Rather than
require newer CMake versions and depend on CMake's Ninja generator
maintaining the latest behavior forever, set the macro TEST_DATA_DIR to
point to the test data folder. This is consistent and never depends on
unspecified behavior.
This change will allow the use of the Ninja generator with older
versions of CMake, as found in most of our Docker builds.
- Update protobuf to v23.4, to fix the cstdint error on Arch Linux, and
make some related changes:
- Silence additional compiler warnings for the new protobuf
- Update absl to 20230125.3, to support the protobuf update, and make
some related changes:
- Silence additional compiler warnings for the new protobuf
- Replace GOOGLE_CHECK_OK with ABSL_CHECK_OK
- Replace GUARDED_BY with ABSL_GUARDED_BY
- Update other instances of cstdint missing in our own code
- Always pull the latest docker images, to avoid stale results when your
workstation has pulled an older image at the same label
Rebasing #1148 on top of current cmake branch
Related to #1047
---------
Co-authored-by: Bartek Zdanowski <bartek.zdanowski@gmail.com>
Co-authored-by: Joey Parrish <joeyparrish@google.com>
All HTTP-based tests now use an embedded test server instead of
httpbin.org, which makes them much faster and more reliable.
These more reliable tests also exposed some issues that began recently
with PR #1201. HttpFile's Flush() semantics were different than those
documented for files in general. Flush() used to close the file for
uploading, so that no further writes were allowed, but the documentation
stated that it would only flush data to its destination. PR #1201
brought HttpFile's Flush() in line with the docs, but gave us no way to
terminate a chunked upload.
This adds a new method to File called CloseForWriting(), which
terminates a chunked upload for HttpFile. The only other implementation
that does anything is UdpFile, which uses the socket library function
shutdown() to terminate writes while allowing reads.
This also tweaks HttpFile::CloseWithStatus() so that it will not
generate an error if the file is closed before the HTTP response is
written to the download cache.
This modifies the test HttpFileTest.MultipleWrites so that the file is
Flushed after each chunk. This adds test coverage for the changes
introduced in PR #1201.
Fixes#1224 (missing test coverage for HttpFile::Flush)
This adds Mongoose as a third-party library, and builds on top of that
an embedded HTTP server for our unit tests.
We are using a fork of Mongoose pending the merging of this PR:
https://github.com/cesanta/mongoose/pull/2301
The embedded web server will make our HTTP-based tests independent of
httpbin.org, which will make them quick and reliable.
Rebasing #1155 and applying some fixes. Had to comment out the
`mpd_notify_muxer_listener_unittest` because it depends on
`MockMpdNotifier` from `mpd/base` which has not been ported yet. Can
bring this test back once that has been ported.
Related to #1047
---------
Co-authored-by: Carlos Bentzen <carlos.bentzen@bitmovin.com>
Co-authored-by: Joey Parrish <joeyparrish@google.com>
This reverts commit 2e349845c6 from PR
#1203.
We are moving away from httpbin.org for testing, and this revert will
make subsequent changes to the tests easier to read.
Fix a bug that if the webvtt file is very short, e.g. only contains one
block
WEBVTT
00:00:00.500 --> 00:00:02.000
The Web is always changing
shaka packager will report error: "Packaging Error: 6 (END_OF_STREAM)".
Fixes#1217
Closing the upstream on flush will effectively terminate the ongoing
curl connection. This means that we would need re-establish the
connection in order to resume writing, this is not what we want. In the
spirit of the documentation of File::Flush
```c++
/// Flush the file so that recently written data will survive an
/// application crash (but not necessarily an OS crash). For
/// instance, in LocalFile the data is flushed into the OS but not
/// necessarily to disk.
```
We will instead wait for the curl thread to finish consuming what ever
might be in the upload cache, but leave the connection open for
subsequent writes.
Fixes#1196
# Low Latency DASH - `availabilityTimeComplete=false`
Low Latency DASH manifests generated by Packager were missing the
attribute `availabilityTimeComplete`. As per the [DASH
specs](https://dashif.org/docs/CR-Low-Latency-Live-r8.pdf):
**_the AdaptationSet@availabilityTimeCompleteshould be present and be
set to 'FALSE'_**
## The Issue
The missing attribute caused ULL streams from Shaka Packager to no
longer be compatible with DASH.js. Previous versions of DASH.js allowed
users to specify ULL mode when initializing the player. However, the
most recent releases of DASH.js automatically detect ULL by scanning the
manifest for ULL specific attributes. Although there are many attributes
only associated with ULL, [DASH.js only greps for
`availabilityTimeComplete` in its detection
logic](https://github.com/Dash-Industry-Forum/dash.js/blob/development/src/streaming/controllers/PlaybackController.js#L792-L805).
Because of the missing attribute in Packager and the limited ULL
verification criteria by DASH.js, Packager streams were not being
treated as low latency streams by DASH.js.
## Testing
### Unit Testing
`./mpd_unittest
--gtest_filter="SegmentTemplateTest.OneSegmentLowLatency"`
` ./mpd_unittest
--gtest_filter="LowLatencySegmentTest.LowLatencySegmentTemplate"`
### Manual Testing
- Created a low latency stream with Shaka Packager
- Observed the expected `availabilityTimeComplete=false` attribute in
the generated DASH manifest.
It appears that not all Apple implementations follow the HLS guidelines.
While the DEFAULT=NO for an audio track should be optional and default
to NO, in practice native HLS players Safari and iOS devices treat the
missing DEFAULT as a MAYBE.
Fixes#1169
Using a rolling release of OpenSUSE was unstable. But there is another
way around the old compiler in OpenSUSE 15. This upgrades the compiler
while staying on a stable release of the OS.
Cosmin Stejerean
Cosmin Stejerean
Shaka Bot
Caitlin O'Callaghan
Prakash Duggaraju
Joey Parrish
Joey Parrish
sr90
Cosmin Stejerean
Sergio Garcia Murillo
xyb
Allan Lei
Carlos Bentzen
Peter Zebühr
Dennis E. Mungai
Marcus Spangenberg
Mark Sargent