Commit Graph

1544 Commits

Author SHA1 Message Date
Joey Parrish 052fb64068
ci: Enable parallel builds (#1241)
Our workflows and Dockerfiles now use Ninja on Linux & Mac, which
enables safe parallel builds. This significantly speeds up our
workflows.

GitHub Actions typical compilation times (build step only):
 - Linux 19m => 9m
 - macOS 23m => 8m
 - Windows 12m => 10m
 - Linux arm64 (self-hosted) 72m => 29m
 - Docker build 25m => 14m

Overall workflow time: 84m => 33m

Compilation time on my workstation (12 CPUs @3.3GHz): 15m => 3m

This also adds a new environment variable "PACKAGER_LOW_MEMORY_BUILD".
If defined when CMake is first run, this will configure the build to
disable parallel linking to reduce memory usage. This helps us avoid
failures on our self-hosted arm64 machines, where 6 CPUs share 4GB of
RAM.

NOTE: Parallel builds are **NOT** recommended with Unix Makefiles due to
the use of excessive RAM during parallel linking. Unix Makefiles, unlike
Ninja, cannot be configured to restrict parallel linking during a
parallel build. Anecdotally, parallel builds with Makefiles have
exhausted a system with 32GB RAM. (My workstation.)

In a follow-up, I will update the build documentation to refer to CMake
and recommend all of the flags now used in our workflows.
2023-07-17 13:16:03 -07:00
Joey Parrish 362d98ae62
test: Move CentOS 8 tests to CentOS 9 Stream (#1240)
CentOS 8 is EOL, and the official docker images will not be updated. See
https://www.centos.org/centos-linux-eol/

This switches to the latest release (CentOS 9 Stream). Since CentOS no
longer provides official images, this uses a third-party image provider
(https://hub.docker.com/u/tgagor)

This change also exposed a warning in absl that needed to be silenced in
the default version of GCC in CentOS 9 Stream (GCC 11.4).
2023-07-17 11:10:28 -07:00
Joey Parrish 899f763dd7
test: Fix data directory computation (#1239)
The `__FILE__` macro does not always get you an absolute path, so our
assumptions about it in test_data_util.cc were flawed.

If `foo.c` references `__FILE__`, something like `gcc -c foo.c`, will
define `__FILE__` as `"foo.c"`. If you use `gcc -c /path/to/foo.c`, then
`__FILE__` will be `"/path/to/foo.c"`.

The Ninja generator for CMake only generates absolute paths for source
files in certain CMake versions. (Exact range unknown.) Rather than
require newer CMake versions and depend on CMake's Ninja generator
maintaining the latest behavior forever, set the macro TEST_DATA_DIR to
point to the test data folder. This is consistent and never depends on
unspecified behavior.

This change will allow the use of the Ninja generator with older
versions of CMake, as found in most of our Docker builds.
2023-07-17 10:06:10 -07:00
Joey Parrish 53982dbab4
chore: Split out CMake policies into package/policies.cmake (#1236) 2023-07-17 06:41:23 -07:00
Joey Parrish 4b868de9f1
chore: Revert to upstream mongoose (#1235)
Since https://github.com/cesanta/mongoose/issues/2300 was closed, revert
to upstream mongoose, specifically the most recent tag "7.11"
2023-07-16 16:49:29 -07:00
Joey Parrish 8bf2d45424
fix: Fix compilation on Arch Linux (#1233)
- Update protobuf to v23.4, to fix the cstdint error on Arch Linux, and
  make some related changes:
   - Silence additional compiler warnings for the new protobuf
- Update absl to 20230125.3, to support the protobuf update, and make
  some related changes:
   - Silence additional compiler warnings for the new protobuf
   - Replace GOOGLE_CHECK_OK with ABSL_CHECK_OK
   - Replace GUARDED_BY with ABSL_GUARDED_BY
 - Update other instances of cstdint missing in our own code
- Always pull the latest docker images, to avoid stale results when your
  workstation has pulled an older image at the same label
2023-07-14 12:40:12 -07:00
Cosmin Stejerean 86bf6cf3cd
feat: Cmake port of Media crypto (#1221)
Rebasing #1148 on top of current cmake branch

Related to #1047

---------

Co-authored-by: Bartek Zdanowski <bartek.zdanowski@gmail.com>
Co-authored-by: Joey Parrish <joeyparrish@google.com>
2023-07-14 08:19:01 -07:00
Joey Parrish af98d48726
fix: Fix flush/close semantics for HTTP files, improve testing (#1232)
All HTTP-based tests now use an embedded test server instead of
httpbin.org, which makes them much faster and more reliable.

These more reliable tests also exposed some issues that began recently
with PR #1201.  HttpFile's Flush() semantics were different than those
documented for files in general.  Flush() used to close the file for
uploading, so that no further writes were allowed, but the documentation
stated that it would only flush data to its destination.  PR #1201
brought HttpFile's Flush() in line with the docs, but gave us no way to
terminate a chunked upload.

This adds a new method to File called CloseForWriting(), which
terminates a chunked upload for HttpFile.  The only other implementation
that does anything is UdpFile, which uses the socket library function
shutdown() to terminate writes while allowing reads.

This also tweaks HttpFile::CloseWithStatus() so that it will not
generate an error if the file is closed before the HTTP response is
written to the download cache.

This modifies the test HttpFileTest.MultipleWrites so that the file is
Flushed after each chunk.  This adds test coverage for the changes
introduced in PR #1201.

Fixes #1224 (missing test coverage for HttpFile::Flush)
2023-07-13 18:55:48 -07:00
Joey Parrish d4fcfb2f4f
test: Add Mongoose embedded HTTP server (#1231)
This adds Mongoose as a third-party library, and builds on top of that
an embedded HTTP server for our unit tests.

We are using a fork of Mongoose pending the merging of this PR:
https://github.com/cesanta/mongoose/pull/2301

The embedded web server will make our HTTP-based tests independent of
httpbin.org, which will make them quick and reliable.
2023-07-13 16:36:42 -07:00
Cosmin Stejerean 8d3b2c66b6
feat: port media/event to CMake (#1222)
Rebasing #1155 and applying some fixes. Had to comment out the
`mpd_notify_muxer_listener_unittest` because it depends on
`MockMpdNotifier` from `mpd/base` which has not been ported yet. Can
bring this test back once that has been ported.

Related to #1047

---------

Co-authored-by: Carlos Bentzen <carlos.bentzen@bitmovin.com>
Co-authored-by: Joey Parrish <joeyparrish@google.com>
2023-07-13 16:25:42 -07:00
Joey Parrish 3a551f428f
test: Revert "Retry HTTP file tests on temporary httpbin failure" (#1230)
This reverts commit 2e349845c6 from PR
#1203.

We are moving away from httpbin.org for testing, and this revert will
make subsequent changes to the tests easier to read.
2023-07-13 15:55:36 -07:00
Sergio Garcia Murillo d6f28d456c
fix: Prevent crash in GetEarliestTimestamp() if periods are empty (#1173)
While I have not yet found why the periods are empty, this will prevent shaka from seg faulting

Fixes #1172
2023-07-12 14:51:00 -07:00
xyb dab165d3e5
fix: Fix failure on very short WebVTT files (#1216)
Fix a bug that if the webvtt file is very short, e.g. only contains one
block

    WEBVTT

    00:00:00.500 --> 00:00:02.000
    The Web is always changing

shaka packager will report error: "Packaging Error: 6 (END_OF_STREAM)".

Fixes #1217
2023-07-12 09:07:07 -07:00
Allan Lei d9d3c7f8be
fix: Indexing `bytes` produces `int` on python3 for `pssh-box.py` (#1228)
Fixes #1227
2023-07-12 09:03:43 -07:00
sr90 ac47e529ad
fix: Update golden files for ttml tests and failing hls unit tests. (#1226)
Updating golden files for failing ttml tests in packager_test.py related
to https://github.com/shaka-project/shaka-packager/pull/1179 and
failing HLS unit tests related to
https://github.com/shaka-project/shaka-packager/pull/1170.
2023-07-12 09:01:55 -07:00
Joey Parrish 98490edc5f Merge remote-tracking branch 'upstream/main' into cmake 2023-07-05 19:51:52 -07:00
Carlos Bentzen ee0922edf0
feat: port media/chunking to CMake (#1223)
Port media/chunking to CMake

Related to #1047
2023-07-05 15:18:36 -07:00
Peter Zebühr 53d91cd0f1
fix: Don't close upstream on HttpFile::Flush (#1201)
Closing the upstream on flush will effectively terminate the ongoing
curl connection. This means that we would need re-establish the
connection in order to resume writing, this is not what we want. In the
spirit of the documentation of File::Flush

```c++
/// Flush the file so that recently written data will survive an 
/// application crash (but not necessarily an OS crash). For 
/// instance, in LocalFile the data is flushed into the OS but not 
/// necessarily to disk.
```

We will instead wait for the curl thread to finish consuming what ever
might be in the upload cache, but leave the connection open for
subsequent writes.

Fixes #1196
2023-07-05 14:38:01 -07:00
Caitlin O'Callaghan d687ad1ed0
fix: Low Latency DASH: include the "availabilityTimeComplete=false" attribute (#1198)
# Low Latency DASH - `availabilityTimeComplete=false`

Low Latency DASH manifests generated by Packager were missing the
attribute `availabilityTimeComplete`. As per the [DASH
specs](https://dashif.org/docs/CR-Low-Latency-Live-r8.pdf):

**_the AdaptationSet@availabilityTimeCompleteshould be present and be
set to 'FALSE'_**

## The Issue
The missing attribute caused ULL streams from Shaka Packager to no
longer be compatible with DASH.js. Previous versions of DASH.js allowed
users to specify ULL mode when initializing the player. However, the
most recent releases of DASH.js automatically detect ULL by scanning the
manifest for ULL specific attributes. Although there are many attributes
only associated with ULL, [DASH.js only greps for
`availabilityTimeComplete` in its detection
logic](https://github.com/Dash-Industry-Forum/dash.js/blob/development/src/streaming/controllers/PlaybackController.js#L792-L805).
Because of the missing attribute in Packager and the limited ULL
verification criteria by DASH.js, Packager streams were not being
treated as low latency streams by DASH.js.

## Testing
### Unit Testing
`./mpd_unittest
--gtest_filter="SegmentTemplateTest.OneSegmentLowLatency"`
` ./mpd_unittest
--gtest_filter="LowLatencySegmentTest.LowLatencySegmentTemplate"`

### Manual Testing
- Created a low latency stream with Shaka Packager
- Observed the expected `availabilityTimeComplete=false` attribute in
the generated DASH manifest.
2023-07-05 14:33:51 -07:00
Dennis E. Mungai ef17cf81f2
docs: Update CONTRIBUTORS (#1190) 2023-07-05 14:32:24 -07:00
Dennis E. Mungai 998b9cf65e
docs: Update AUTHORS (#1189) 2023-07-05 14:32:03 -07:00
Dennis E. Mungai e407ec7bea
docs: Update build_instructions.md to point to the `main` branch. (#1186)
The default branch is `main`, not `master`. Indicate this on the
documentation as its' not obvious.

Fixes #1060
2023-07-05 14:31:39 -07:00
Marcus Spangenberg 494769ca86
fix: TTML generator timestamp millisecond formatting (#1179)
Fix bug where milliseconds were formatted with two digits instead of
three, resulting in incorrect timestamps in TTML cues.

Fixes #1180
2023-07-05 14:28:57 -07:00
Mark Sargent 1ab6818832
fix: hls, set the DEFAULT explicitly to NO. Supports native HLS players. (#1170)
It appears that not all Apple implementations follow the HLS guidelines.

While the DEFAULT=NO for an audio track should be optional and default
to NO, in practice native HLS players Safari and iOS devices treat the
missing DEFAULT as a MAYBE.

Fixes #1169
2023-07-05 14:18:41 -07:00
Joey Parrish bf6296c463
test: Fix OpenSUSE docker build (#1204)
Using a rolling release of OpenSUSE was unstable. But there is another
way around the old compiler in OpenSUSE 15. This upgrades the compiler
while staying on a stable release of the OS.
2023-05-01 20:03:18 -07:00
Joey Parrish 2e349845c6
test: Retry HTTP file tests on temporary httpbin failure (#1203) 2023-05-01 16:56:46 -07:00
Bartek Zdanowski 901013c34e
feat: CMake port media/trick_play (#1146)
Issue #1047 (CMake porting)

---------

Co-authored-by: Joey Parrish <joeyparrish@google.com>
2023-05-01 14:57:40 -07:00
Joey Parrish 6c1d4f3885 Merge remote-tracking branch 'upstream/main' into cmake 2023-05-01 09:31:59 -07:00
Dennis E. Mungai a4c357b992 fix: Fix type error in pssh-box.py with Python 3.10+ (#1187)
A single-line change on #L170 to `wv.protection_scheme =
struct.unpack('>L', bytes(protection_scheme, encoding='utf-8'))[0]`,
needed to work around this issue on Ubuntu 22.04LTS+ running Python
3.10+:

```sh
TypeError: a bytes-like object is required, not 'str'
```
On line 170.
2023-05-01 09:10:11 -07:00
Dennis E. Mungai 80e024013d
fix: Fix issues with `collections.abc` in Python 3.10+ (#1188)
This issue is observed on Python 3.10+ and above.
This workaround addresses a major backwards compatibility break with a
major Python release version where collections.abc isn't available.

Fixes #1192
2023-05-01 09:07:08 -07:00
Dennis E. Mungai 161947f53e
fix: Fix type error in pssh-box.py with Python 3.10+ (#1187)
A single-line change on #L170 to `wv.protection_scheme =
struct.unpack('>L', bytes(protection_scheme, encoding='utf-8'))[0]`,
needed to work around this issue on Ubuntu 22.04LTS+ running Python
3.10+:

```sh
TypeError: a bytes-like object is required, not 'str'
```
On line 170.
2023-05-01 09:06:56 -07:00
Joey Parrish 84b02926e2
fix: Fix build errors since infra upgrades (#1199) 2023-04-20 19:18:28 -07:00
Carlos Bentzen 56d3304045
feat: port media/formats/webm to CMake (#1147)
Issue #1047 (CMake port)

Co-authored-by: Joey Parrish <joeyparrish@users.noreply.github.com>
2022-12-16 21:40:00 -08:00
Carlos Bentzen e9bf0c6de4
feat: port media/codecs to CMake (#1143)
Issue #1047 (CMake port)
Issue #346 (absl port)
2022-12-15 16:53:36 -08:00
Joey Parrish 41a50962ac
fix: Fix Linux build failures (#1157)
This fixes a libxml2 build failure in release mode and a build failure
in generated proto code.
2022-12-15 13:56:52 -08:00
Bartek Zdanowski 5f435708e7
feat: Cmake port mpd/media_info_proto (#1152)
Issue #1047 (CMake porting)
2022-12-15 08:35:00 -08:00
Bartek Zdanowski 14a708d009
feat: CMake port media/replicator (#1144)
Issue #1047 (CMake porting)
2022-11-30 12:01:25 -08:00
Bartek Zdanowski 5d52d5e73c
feat: CMake port media/origin (#1145)
Issue #1047 (CMake porting)
2022-11-30 09:43:58 -08:00
Joey Parrish b1095f6bc3
feat: port media/base (#1127)
This removes all chromium dependencies from media/base/ and completes
the build system in CMake.

The ClosureThread class and its classes were removed, as they were
specific to chromium base. ClosureThread has been replaced by
std::thread.

The byte-swapping utilities in network_util.cc have been removed and
replaced with absl.

generate_unique_temp_path() was split out of file_unittest.cc into
file_test_util.cc, where other test suites could make use of it.

WARN_UNUSED_RESULT was replaced with the C++ standard attribute
[[nodiscard]].

The base::Clock interface was replaced with a typedef for a function
pointer that returns the current time.

This re-enables the tests in http_key_fetcher_unittest.cc by using
httpbin.org.

Issue #1047 (CMake porting)
Issue #346 (absl porting)
2022-11-04 15:46:41 -07:00
Joey Parrish 44afbfc866
fix: Fix protoc linking on macOS w/ update to fork (#1132)
See also:
 - https://github.com/protocolbuffers/protobuf/issues/10899
 - https://github.com/protocolbuffers/protobuf/pull/10900

Also updates flags to deal with various compile-time warnings/errors for
protobuf libs on all platforms.
2022-11-04 08:33:01 -07:00
Joey Parrish b299aa2d96
ci: Disable parallel builds (#1133)
This appears to fix some issues with crashing compilers on arm64 and
docker builds. This may be caused by resource constraints in those
environments.
2022-11-04 08:32:44 -07:00
Joey Parrish 0ee4af438d Merge branch 'main' into cmake 2022-11-03 07:58:47 -07:00
Joey Parrish afd5fec7cd
feat: Replace arraysize with std::size (#1129)
This does not depend on absl, which frees macros.h from any library deps
that would make the CMake dependency tree more complicated. This also
fixes build errors in some environments.
2022-11-03 07:21:46 -07:00
Joey Parrish dccefb1385
ci: Use multiple cores in docker tests (#1128)
This seems to bring current docker build times in the cmake branch from
4-6 minutes down to 3-4 minutes.
2022-11-03 07:21:14 -07:00
Joey Parrish 7b33f2065f
feat: Port AES and RSA crypto to mbedtls (#1119)
mbedtls works very differently from BoringSSL, and many changes had to
be made in the details of AES decryption to accomodate this.

Beyond the basic changes required for mbedtls, part of the CTS padding
implementation had to be rewritten. I believe this is because of an
assumption that held for BoringSSL, but not for mbedtls. I was unable to
determine what it was, so I rewrote the CTS decryption using reference
materials. After this, tests passed.

The deterministc PRNG I used with mbedtls in the RSA tests differs
somewhat from the old one, so the expected vectors had to be
regenerated. The old determinstic tests were also disabled, and are now
re-enabled.

Since cryptography is sensitive code, and because there were far more
changes needed here than just updating some headers and utility function
calls, this has been split into its own PR for separate review from the
rest of the media/base porting work.

Issue #1047 (CMake porting)
Issue #346 (absl porting)
2022-11-02 08:34:06 -07:00
Joey Parrish d5ca6e84e6
ci: Fix main workflows (#1122)
This brings some workflow improvements and fixes from the `cmake` branch
to `main`, as well as some unique fixes to keep gclient working, so that
we can continue to accept contributions in `main` until the `cmake`
merge is ready.

 - Fix docs build in GitHub Actions (from `cmake` branch)
 - Cancel workflow when a PR is updated (from `cmake` branch)
 - Fix docker failures caused by running as root (from `cmake` branch)
 - Work around exception in depot_tools on Windows
 - Use Windows 2019 images in GitHub Actions for compatibility with gyp
- Remove Docker build on ArchLinux, which no longer supports python2 at
all
- (NOTE: The `cmake` branch is still building on ArchLinux. Docker
builds for Arch will be restored to the `main` branch when the `cmake`
branch is finally merged to `main`.)
2022-10-28 15:46:33 -07:00
Bartek Zdanowski b221aa9caf
fix: Parse one frame mpeg-ts video (#1015)
Closes #1013

Co-authored-by: Joey Parrish <joeyparrish@users.noreply.github.com>
2022-10-27 20:22:17 -07:00
Bartek Zdanowski ab8ab12d09
fix: PTS diverge DTS when DTS close to 2pow33 and PTS more than 0 (#1050)
Fixes #1049

Co-authored-by: Joey Parrish <joeyparrish@users.noreply.github.com>
2022-10-27 14:21:03 -07:00
Shaka Bot 3d60105738
chore: Sync common workflows (#1120)
This is an automated sync of common workflows for this organization.
The upstream source is:

8bfe75f0d2

Co-authored-by: Shaka Bot <shaka-bot@users.noreply.github.com>
2022-10-27 11:20:43 -07:00
Joey Parrish c3a4951597
feat: Add CMake module for building protobufs (#1112)
Issue #1047 (CMake porting)
2022-10-26 10:07:01 -07:00