Commit Graph

1134 Commits

Author SHA1 Message Date
xyb dab165d3e5
fix: Fix failure on very short WebVTT files (#1216)
Fix a bug that if the webvtt file is very short, e.g. only contains one
block

    WEBVTT

    00:00:00.500 --> 00:00:02.000
    The Web is always changing

shaka packager will report error: "Packaging Error: 6 (END_OF_STREAM)".

Fixes #1217
2023-07-12 09:07:07 -07:00
Allan Lei d9d3c7f8be
fix: Indexing `bytes` produces `int` on python3 for `pssh-box.py` (#1228)
Fixes #1227
2023-07-12 09:03:43 -07:00
sr90 ac47e529ad
fix: Update golden files for ttml tests and failing hls unit tests. (#1226)
Updating golden files for failing ttml tests in packager_test.py related
to https://github.com/shaka-project/shaka-packager/pull/1179 and
failing HLS unit tests related to
https://github.com/shaka-project/shaka-packager/pull/1170.
2023-07-12 09:01:55 -07:00
Joey Parrish 98490edc5f Merge remote-tracking branch 'upstream/main' into cmake 2023-07-05 19:51:52 -07:00
Carlos Bentzen ee0922edf0
feat: port media/chunking to CMake (#1223)
Port media/chunking to CMake

Related to #1047
2023-07-05 15:18:36 -07:00
Peter Zebühr 53d91cd0f1
fix: Don't close upstream on HttpFile::Flush (#1201)
Closing the upstream on flush will effectively terminate the ongoing
curl connection. This means that we would need re-establish the
connection in order to resume writing, this is not what we want. In the
spirit of the documentation of File::Flush

```c++
/// Flush the file so that recently written data will survive an 
/// application crash (but not necessarily an OS crash). For 
/// instance, in LocalFile the data is flushed into the OS but not 
/// necessarily to disk.
```

We will instead wait for the curl thread to finish consuming what ever
might be in the upload cache, but leave the connection open for
subsequent writes.

Fixes #1196
2023-07-05 14:38:01 -07:00
Caitlin O'Callaghan d687ad1ed0
fix: Low Latency DASH: include the "availabilityTimeComplete=false" attribute (#1198)
# Low Latency DASH - `availabilityTimeComplete=false`

Low Latency DASH manifests generated by Packager were missing the
attribute `availabilityTimeComplete`. As per the [DASH
specs](https://dashif.org/docs/CR-Low-Latency-Live-r8.pdf):

**_the AdaptationSet@availabilityTimeCompleteshould be present and be
set to 'FALSE'_**

## The Issue
The missing attribute caused ULL streams from Shaka Packager to no
longer be compatible with DASH.js. Previous versions of DASH.js allowed
users to specify ULL mode when initializing the player. However, the
most recent releases of DASH.js automatically detect ULL by scanning the
manifest for ULL specific attributes. Although there are many attributes
only associated with ULL, [DASH.js only greps for
`availabilityTimeComplete` in its detection
logic](https://github.com/Dash-Industry-Forum/dash.js/blob/development/src/streaming/controllers/PlaybackController.js#L792-L805).
Because of the missing attribute in Packager and the limited ULL
verification criteria by DASH.js, Packager streams were not being
treated as low latency streams by DASH.js.

## Testing
### Unit Testing
`./mpd_unittest
--gtest_filter="SegmentTemplateTest.OneSegmentLowLatency"`
` ./mpd_unittest
--gtest_filter="LowLatencySegmentTest.LowLatencySegmentTemplate"`

### Manual Testing
- Created a low latency stream with Shaka Packager
- Observed the expected `availabilityTimeComplete=false` attribute in
the generated DASH manifest.
2023-07-05 14:33:51 -07:00
Marcus Spangenberg 494769ca86
fix: TTML generator timestamp millisecond formatting (#1179)
Fix bug where milliseconds were formatted with two digits instead of
three, resulting in incorrect timestamps in TTML cues.

Fixes #1180
2023-07-05 14:28:57 -07:00
Mark Sargent 1ab6818832
fix: hls, set the DEFAULT explicitly to NO. Supports native HLS players. (#1170)
It appears that not all Apple implementations follow the HLS guidelines.

While the DEFAULT=NO for an audio track should be optional and default
to NO, in practice native HLS players Safari and iOS devices treat the
missing DEFAULT as a MAYBE.

Fixes #1169
2023-07-05 14:18:41 -07:00
Joey Parrish bf6296c463
test: Fix OpenSUSE docker build (#1204)
Using a rolling release of OpenSUSE was unstable. But there is another
way around the old compiler in OpenSUSE 15. This upgrades the compiler
while staying on a stable release of the OS.
2023-05-01 20:03:18 -07:00
Joey Parrish 2e349845c6
test: Retry HTTP file tests on temporary httpbin failure (#1203) 2023-05-01 16:56:46 -07:00
Bartek Zdanowski 901013c34e
feat: CMake port media/trick_play (#1146)
Issue #1047 (CMake porting)

---------

Co-authored-by: Joey Parrish <joeyparrish@google.com>
2023-05-01 14:57:40 -07:00
Dennis E. Mungai a4c357b992 fix: Fix type error in pssh-box.py with Python 3.10+ (#1187)
A single-line change on #L170 to `wv.protection_scheme =
struct.unpack('>L', bytes(protection_scheme, encoding='utf-8'))[0]`,
needed to work around this issue on Ubuntu 22.04LTS+ running Python
3.10+:

```sh
TypeError: a bytes-like object is required, not 'str'
```
On line 170.
2023-05-01 09:10:11 -07:00
Dennis E. Mungai 80e024013d
fix: Fix issues with `collections.abc` in Python 3.10+ (#1188)
This issue is observed on Python 3.10+ and above.
This workaround addresses a major backwards compatibility break with a
major Python release version where collections.abc isn't available.

Fixes #1192
2023-05-01 09:07:08 -07:00
Dennis E. Mungai 161947f53e
fix: Fix type error in pssh-box.py with Python 3.10+ (#1187)
A single-line change on #L170 to `wv.protection_scheme =
struct.unpack('>L', bytes(protection_scheme, encoding='utf-8'))[0]`,
needed to work around this issue on Ubuntu 22.04LTS+ running Python
3.10+:

```sh
TypeError: a bytes-like object is required, not 'str'
```
On line 170.
2023-05-01 09:06:56 -07:00
Joey Parrish 84b02926e2
fix: Fix build errors since infra upgrades (#1199) 2023-04-20 19:18:28 -07:00
Carlos Bentzen 56d3304045
feat: port media/formats/webm to CMake (#1147)
Issue #1047 (CMake port)

Co-authored-by: Joey Parrish <joeyparrish@users.noreply.github.com>
2022-12-16 21:40:00 -08:00
Carlos Bentzen e9bf0c6de4
feat: port media/codecs to CMake (#1143)
Issue #1047 (CMake port)
Issue #346 (absl port)
2022-12-15 16:53:36 -08:00
Joey Parrish 41a50962ac
fix: Fix Linux build failures (#1157)
This fixes a libxml2 build failure in release mode and a build failure
in generated proto code.
2022-12-15 13:56:52 -08:00
Bartek Zdanowski 5f435708e7
feat: Cmake port mpd/media_info_proto (#1152)
Issue #1047 (CMake porting)
2022-12-15 08:35:00 -08:00
Bartek Zdanowski 14a708d009
feat: CMake port media/replicator (#1144)
Issue #1047 (CMake porting)
2022-11-30 12:01:25 -08:00
Bartek Zdanowski 5d52d5e73c
feat: CMake port media/origin (#1145)
Issue #1047 (CMake porting)
2022-11-30 09:43:58 -08:00
Joey Parrish b1095f6bc3
feat: port media/base (#1127)
This removes all chromium dependencies from media/base/ and completes
the build system in CMake.

The ClosureThread class and its classes were removed, as they were
specific to chromium base. ClosureThread has been replaced by
std::thread.

The byte-swapping utilities in network_util.cc have been removed and
replaced with absl.

generate_unique_temp_path() was split out of file_unittest.cc into
file_test_util.cc, where other test suites could make use of it.

WARN_UNUSED_RESULT was replaced with the C++ standard attribute
[[nodiscard]].

The base::Clock interface was replaced with a typedef for a function
pointer that returns the current time.

This re-enables the tests in http_key_fetcher_unittest.cc by using
httpbin.org.

Issue #1047 (CMake porting)
Issue #346 (absl porting)
2022-11-04 15:46:41 -07:00
Joey Parrish 44afbfc866
fix: Fix protoc linking on macOS w/ update to fork (#1132)
See also:
 - https://github.com/protocolbuffers/protobuf/issues/10899
 - https://github.com/protocolbuffers/protobuf/pull/10900

Also updates flags to deal with various compile-time warnings/errors for
protobuf libs on all platforms.
2022-11-04 08:33:01 -07:00
Joey Parrish b299aa2d96
ci: Disable parallel builds (#1133)
This appears to fix some issues with crashing compilers on arm64 and
docker builds. This may be caused by resource constraints in those
environments.
2022-11-04 08:32:44 -07:00
Joey Parrish 0ee4af438d Merge branch 'main' into cmake 2022-11-03 07:58:47 -07:00
Joey Parrish afd5fec7cd
feat: Replace arraysize with std::size (#1129)
This does not depend on absl, which frees macros.h from any library deps
that would make the CMake dependency tree more complicated. This also
fixes build errors in some environments.
2022-11-03 07:21:46 -07:00
Joey Parrish dccefb1385
ci: Use multiple cores in docker tests (#1128)
This seems to bring current docker build times in the cmake branch from
4-6 minutes down to 3-4 minutes.
2022-11-03 07:21:14 -07:00
Joey Parrish 7b33f2065f
feat: Port AES and RSA crypto to mbedtls (#1119)
mbedtls works very differently from BoringSSL, and many changes had to
be made in the details of AES decryption to accomodate this.

Beyond the basic changes required for mbedtls, part of the CTS padding
implementation had to be rewritten. I believe this is because of an
assumption that held for BoringSSL, but not for mbedtls. I was unable to
determine what it was, so I rewrote the CTS decryption using reference
materials. After this, tests passed.

The deterministc PRNG I used with mbedtls in the RSA tests differs
somewhat from the old one, so the expected vectors had to be
regenerated. The old determinstic tests were also disabled, and are now
re-enabled.

Since cryptography is sensitive code, and because there were far more
changes needed here than just updating some headers and utility function
calls, this has been split into its own PR for separate review from the
rest of the media/base porting work.

Issue #1047 (CMake porting)
Issue #346 (absl porting)
2022-11-02 08:34:06 -07:00
Joey Parrish d5ca6e84e6
ci: Fix main workflows (#1122)
This brings some workflow improvements and fixes from the `cmake` branch
to `main`, as well as some unique fixes to keep gclient working, so that
we can continue to accept contributions in `main` until the `cmake`
merge is ready.

 - Fix docs build in GitHub Actions (from `cmake` branch)
 - Cancel workflow when a PR is updated (from `cmake` branch)
 - Fix docker failures caused by running as root (from `cmake` branch)
 - Work around exception in depot_tools on Windows
 - Use Windows 2019 images in GitHub Actions for compatibility with gyp
- Remove Docker build on ArchLinux, which no longer supports python2 at
all
- (NOTE: The `cmake` branch is still building on ArchLinux. Docker
builds for Arch will be restored to the `main` branch when the `cmake`
branch is finally merged to `main`.)
2022-10-28 15:46:33 -07:00
Bartek Zdanowski b221aa9caf
fix: Parse one frame mpeg-ts video (#1015)
Closes #1013

Co-authored-by: Joey Parrish <joeyparrish@users.noreply.github.com>
2022-10-27 20:22:17 -07:00
Bartek Zdanowski ab8ab12d09
fix: PTS diverge DTS when DTS close to 2pow33 and PTS more than 0 (#1050)
Fixes #1049

Co-authored-by: Joey Parrish <joeyparrish@users.noreply.github.com>
2022-10-27 14:21:03 -07:00
Joey Parrish c3a4951597
feat: Add CMake module for building protobufs (#1112)
Issue #1047 (CMake porting)
2022-10-26 10:07:01 -07:00
Joey Parrish a9a4f0d52c
feat: port media/test (#1116)
Rewrite test_data_util.cc to locate files relative to the source file
itself, rather than using a service from chromium `base::`.

Issue #1047 (CMake porting)
Issue #346 (absl porting)
2022-10-25 09:15:54 -07:00
Joey Parrish 64993873c4
fix: Fix status library headers and move test (#1118)
For some reason, the status util unittest was in media/base/ instead of
status/

The broken headers in the status library were not obviously broken until
media/base/ porting started, and they were used for the first time.

Issue #1047 (CMake porting)
2022-10-24 19:09:10 -07:00
Joey Parrish 7ed49d4403
feat: port media/public/ (#1117)
Issue #1047 (CMake porting)
2022-10-24 19:08:37 -07:00
Joey Parrish f45019d478
refactor: Rename common.h to macros.h, now that it only contains macros (#1113)
Issue #1047 (CMake porting)
2022-10-24 19:08:18 -07:00
Geoff Jukes 84ba4afc51 feat: Add xHE-AAC support (#1092)
Note:
* An xHE-AAC capable encoder will auto adjust the user-specified SAP/RAP
  value to the allowed grid where SAP/RAPs can occur.
e.g.: `-rapInterval 5000` (5s) may result in actual SAPs/RAPs every
4.984s.
* To ensure SAP/RAP starts a new segment, Shaka needs to executed with a
  "--segment_duration" is less than or equal to that adjusted value.
* If every SAP/RAP should trigger a new segment, just set the segment
  length to a very low value e.g.: `--segment_duration 0.1`
2022-10-18 10:16:55 -07:00
Geoff Jukes 5d998fca7f
feat: Add xHE-AAC support (#1092)
Note:
* An xHE-AAC capable encoder will auto adjust the user-specified SAP/RAP
  value to the allowed grid where SAP/RAPs can occur.
e.g.: `-rapInterval 5000` (5s) may result in actual SAPs/RAPs every
4.984s.
* To ensure SAP/RAP starts a new segment, Shaka needs to executed with a
  "--segment_duration" is less than or equal to that adjusted value.
* If every SAP/RAP should trigger a new segment, just set the segment
  length to a very low value e.g.: `--segment_duration 0.1`
2022-10-18 10:14:31 -07:00
Joey Parrish 1131bf7eea
chore: Mass-update copyright headers (#1086)
Closes #1076
2022-08-26 08:44:59 -07:00
Joey Parrish 1ff26b4783
feat: Move all third-party deps into git submodules (#1083)
This also normalizes the structure of all submodule folders so that
there is a clear place to put configurations, outside the submodule
source, but limited in scope to that folder.

Issue #1047 (New build system)
2022-08-17 10:42:00 -07:00
Joey Parrish 31e116faec
feat: Respect the file mode for HttpFiles (#1081)
For a read mode, make a GET request.  Otherwise, make a PUT request.
2022-08-17 10:40:13 -07:00
Joey Parrish 868208198a
feat: Remove third-party source in-repo (#1082)
This removes apple_apsl and libevent (which are no longer used) and
libxml and protobuf (which will be replaced in a follow-up with
submodules).

This change only contains deletions, to make it easier to review
separately from later additions for submodules and cmake integration.

Issue #1047 (New build system)
2022-08-16 15:07:21 -07:00
Joey Parrish e6b57f72a8
feat: First phase of CMake build system implementation (#1072)
There are a lot of changes in this first phase, because there was a
lot of infrastructure required to get some meaningful amount of
porting done.  Future PRs should be simpler.

<b>Summary of changes:</b><details>

 - Remove old deps:
   - boringssl (replaced with mbedtls, lighter, easier to build)
   - gflags (replaced with absl::flags)
   - Chromium build tools
 - New deps to replace parts of Chromium base:
   - abseil-cpp
   - glog
   - nlohmann::json (for tests only)
 - Submodules, updates, and CMake build rules for third-party
   libraries:
   - curl
   - gmock/gtest
 - Ported internal libraries and their tests by removing Chromium deps
   and adding CMake build rules:
   - file (now using C++17 filesystem APIs)
   - license_notice
   - status
   - version
 - Test improvements
   - Removed file tests that can never be re-enabled
   - Re-enabled all other disabled file tests
   - Debug JSON values when HTTP tests fail
   - Fixed chunked-encoding issues in HTTP tests
 - Updated and refactored Dockerfiles testing
   - All docker files working, with OS versions updated to meet the
     new tool requirements
   - Local docker builds no longer write files to your working
     directory as root
   - Local docker builds can now be run in parallel without clobbering
     each others' build outputs
   - DEBUG=1 can drop you into an interactive shell when a docker
     build fails
 - Updated and heavily refactored workflows and Dockerfiles
   - All docker files now tested in parallel on GitHub, speeding up CI
   - All common workflow components broken out and using workflow_call
     instead of custom actions
   - Self-hosted runners now optional, to make testing easier on forks
   - CMake porting works-in-process can now be fully tested on GitHub
   - Building ported libraries and passing ported tests on all three
     platforms!
 - CI hacks for macOS removed, now testing on macos-latest!
 - Python2 no longer required!  (Only Python3)
 - Using strict build flags, treating all warnings as errors.

</details>

<b>Required to build:</b>

 - CMake >= 3.16
 - Python 3
 - A compiler supporting C++ >= 17
   - g++ >= 9 if using GCC (Clang also fine)
   - MSVC for Windows

<b>Still needs work:</b><details>

 - Moving other dependencies into submodules (if we keep them):
   - apple_apsl
   - icu
   - libevent
   - libpng
   - libwebm
   - libxml
   - modp_b64
   - protobuf
   - zlib
 - Port remaining internal libraries:
   - app
   - hls
   - media/base
   - media/chunking
   - media/codecs
   - media/crypto
   - media/demuxer
   - media/event
   - media/formats/dvb
   - media/formats/mp2t
   - media/formats/mp4
   - media/formats/packed_audio
   - media/formats/ttml
   - media/formats/webm
   - media/formats/webvtt
   - media/formats/wvm
   - media/origin
   - media/public
   - media/replicator
   - media/trick_play
   - mpd
 - Port main application
   - Add logging flags in absl and connect them to glog (which expects
     gflags)
 - Port pssh-box.py
 - Port main test targets (packager_test.py and packager_app.py)
 - Updating all requirement and build documentation
 - Remove any remaining refs to gclient, depot_tools, ninja
 - Update and complete release workflows using release-please
</details>

Issue #346 (Switch to abseil)
Issue #1047 (New build system)
2022-08-16 11:34:51 -07:00
Vishal Shah dc0395291a
fix: dash_roles add role=description for DVS audio per DASH-IF-IOP-v4.3 (#1054)
Fixes support for description role for audio DVS tracks as per spec.
2022-06-02 09:40:34 -07:00
Vishal Shah b9d477b969
fix: webvtt single cue do not fail on EOS (#1061)
While Parsing cue body check for the block size. 
If it's the last block do not error if it doesn't have a newline.

Fixes #1018
2022-06-02 09:27:47 -07:00
Joey Parrish 3fd538a587
fix: Roll back depot_tools, bypass vpython (#1045)
Using the latest depot_tools no longer works.  depot_tools also wants
to auto-update itself, which must now be disabled.

We also need to disable the copy of python (vpython) included in
depot_tools, since for some distros, it has dependencies on system
libraries that no longer exist.

Finally, we need to force some distros to use python 2, because our
build system is ancient and needs to be ripped out and replaced some
day soon.

This fixes build issues in our CI, our Dockerfiles, and in general on
certain platforms or distros.

Closes #1023
2022-03-08 16:46:18 -08:00
Joey Parrish f577e2a0cf
chore: Update URLs after moving projects (#1042)
Since a project URL is encoded into outputs, this means also updating
the golden output files.

Closes #1043
2022-03-07 11:56:34 -08:00
Vishal Shah e1b0c7c454
Fix WEBVTT Region parse 100 precent (#1006) 2021-11-15 21:28:15 -08:00
Joey Parrish e2d66b33fa fix: Fix crash in static-linked linux builds
The official, static-linked linux builds were crashing in their use of
getaddrinfo, which libcurl was configured to use.  Both getaddrinfo
and all of its alternatives available in glibc fail with static
linking.

We can fix this by configuring libcurl to use libc-ares on Linux
instead.  This allows us to keep the benefits of a statically-linked
Linux binary.

Closes #996

Change-Id: Ib4a9eb939813fd165727788726459ef4adf3fc4d
2021-10-14 10:35:49 -07:00
Joey Parrish 1b96427983 test: Improve debugging ability for docker tests
The script in packager/testing/dockers/test_dockers.sh now outputs
more useful info for debugging, uses unique container names per OS so
that the containers can be debugged, and allows filtering to re-run
specific OSes if a build fails.

Change-Id: I0cace282549c093a643009f5e60e7545a039168c
2021-10-14 10:16:22 -07:00
Joey Parrish 0afda1725c test: Update Dockerfiles
This updates the main Dockerfile and all the docker-based
distro-specific tests.  The base OS versions have been updated to
versions that have not reached end-of-life status yet, and the list of
dependencies required has been updated and pruned.

Change-Id: Ibcff2f60e739fd5d999af100af76c40aa91a75bc
2021-10-13 12:37:16 -07:00
Joey Parrish efbca399c0 fix: Add missing limits header
In many places, we used std::numeric_limits without including the
proper header.  This would build on some Linux distributions, but not
others.

This adds the missing includes, fixing the build on Fedora, among
other distros.

Change-Id: I63e9e37e5973fe23bbdf9868552db51062b1dae4
2021-10-13 12:25:34 -07:00
Joey Parrish 7392d8003e fix: Fix failure fetching encryption keys
In one of the low-latency changes, a change was made to HttpFile that
caused responses to HTTP POST requests to go missing.  This resulted
in failures to fetch encryption keys.

The breaking change was recommended by me in a PR review, and was not
caught by any unit tests.  New tests would be ideal, but I chose to
fix the bug first, rather than leave the repo broken.

This bug was brought to my attention in google/shaka-streamer#87 and
has not appeared in any release versions.

Change-Id: I9eca73d187a8a30f16c4a920fcdb7b4872253858
2021-09-06 21:54:34 -07:00
Caitlin O'Callaghan c87c5bcdef
Fix for gap size warning in Low Latency mode (#985)
## The issue
- With LL-DASH mode enabled, the gap size warning was hit and printed to the console every time a new segment was registered to the manifest.
- This occurred because the first chunk's size and duration were being stored for each segment, rather than the full segment size and duration. Note, only the first chunk's metrics are known at first because in low latency mode, the segment is registered to the manifest before it is finished being processed and written.
- Because of this, the gap size check was comparing the end time of the first chunk in the previous segment to the beginning time of the current segment, causing the check to fail every time.

## The Fix
- Update a low latency segment's duration and size once the segment file has been fully written.
- The full segment size and duration will be used to update the bandwidth estimator and the segment info list. 
- Updating the segment info list to hold the full duration is necessary for satisfying [the gap size check found in Represenation.cc](https://github.com/google/shaka-packager/blob/master/packager/mpd/base/representation.cc#L391).
- NOTE: bandwidth estimation is currently only used in HLS
2021-09-03 09:57:43 -07:00
Joey Parrish 249c83f14b build: Fix additional python2/3 issues
The generate_version_string script was only producing correct results
in python 2, not python 3.  The gyp file that references it explicitly
runs it in python3.  The shebang line of the script has been updated
to match.  The script itself has been updated such that it now works
correctly in both python2 and python3.

Scripts that are only used as modules (not executed directly) have had
their shebang lines removed.

This fixes CI failures on GitHub Actions.

Change-Id: I309bafd2fb05e8fb33f5e092ead179c8c42ea5d3
2021-08-26 12:22:20 -07:00
Caitlin O'Callaghan cd018a71c3
Low latency DASH support (#979)
# LL-DASH Support
These changes add support for LL-DASH streaming. 

**NOTE:** LL-HLS support is still in progress, but it's coming. :) 

## Testing
`./chunking_unittest --gtest_filter="ChunkingHandlerTest.LowLatencyDash"`

`./media_event_unittest --gtest_filter="MpdNotifyMuxerListenerTest.LowLatencyDash"`

`./mpd_unittest --gtest_filter="PeriodTest.LowLatencyDashMpdGetXml"`
`./mpd_unittest --gtest_filter="SimpleMpdNotifierTest.NotifyAvailabilityTimeOffset"`
`./mpd_unittest --gtest_filter="SimpleMpdNotifierTest.NotifySegmentDuration"`
`./mpd_unittest --gtest_filter="LowLatencySegmentTest.LowLatencySegmentTemplate"`

Note, packager_test must be run from the main project directory
`./out/Release/packager_test --gtest_filter="PackagerTest.LowLatencyDashEnabledAndUtcTimingNotSet"`
`./out/Release/packager_test --gtest_filter="PackagerTest.LowLatencyDashEnabledAndUtcTimingNotSet"`
2021-08-25 08:38:05 -07:00
Joey Parrish ac125564b9 build: Fix pylint 2.10 issues, use python3 where possible
The newest pylint release complained about several issues that the
older release did not.  This resolves those issues:

 - removes unneeded "u" prefix from strings
 - adds "encoding" parameter for all open() calls
 - because "encoding" is a python3-only parameter, use python3 in all
   the scripts that we control

Unfortunately, python2 is required for any scripts that import modules
from the ancient Chromium build system we're using (referenced by
DEPS), as well as kokoro scripts.

Change-Id: I2e9f97af508efe58b5a71de21740e59b1528affd
2021-08-24 12:27:03 -07:00
Joey Parrish b411af7ed9
ci: Produce static release executables on Linux (#978)
We never produced static release executables on Linux before, but the dynamic libraries they depended on were universal enough that nobody noticed. Now that we have released v2.5 and switched to GitHub Actions for CI builds, the Linux executables depend on libatomic, which is causing issues for some users.

Although we can't create fully-static executables on macOS or Windows, we can at least do so on Linux.

This adds a GYP variable static_link_binaries which can be set to request full-static binaries on Linux. This also exposes the Chromium build variable disable_fatal_linker_warnings, which is necessary when static linking on Linux due to static-link-related warnings generated by libcurl for its use of getaddrinfo. Finally, this enforces the definition of __UCLIBC__ with static linking on Linux, which is the only way to disable malloc hooks in Chromium base. Those hooks cause linker failures when linking statically on Linux.

A new check has been added to the release workflow to ensure that the builds we create are statically linked on Linux.

Closes #965
2021-08-12 20:14:43 -07:00
Joey Parrish cfbe5c08c2 cleanup: Convert all time parameters to signed
This converts all time parameters to signed, finishing a cleanup that
was started in 2018 in b4256bf0.  This changes the type of:
  - timestamps
    - PTS specifically
  - timestamp offsets
  - timescales
  - durations

This excludes:
  - MP4 box definitions
  - DTS specifically

This is meant to address signed/unsigned conversion issues on arm64
that caused some test cases to fail.

Change-Id: Ic752a20cbc6e31fea6bc0894d1771833171e7cbe
2021-08-05 18:24:15 +00:00
Joey Parrish eba176f032 build: Disable CPU-specific optimizations in libpng
This fixes the Debug build of libpng on arm64 by avoiding CPU-specific
optimizations that are not in our sources list.  The Release build
appears to have been unaffected, possibly due to link-time
optimizations or dead code stripping.

Change-Id: I900e00fe30b9f3748f2587cfea89a636b3a19811
2021-08-04 12:27:10 -07:00
Joey Parrish 68b50f656d build: Stop using hermetic clang, libc++, etc
This brings our default build config more in line with what is
necessary for some platforms anyway: using the system-installed
toolchain and sysroot to build everything.

We will no longer fetch source or binaries for any specific build
tools, such as libc++, clang, gold, binutils, or valgrind.

The main part of this change is the changing of default gyp settings
in gyp_packager.py.  For this, a bug in gyp_packager.py had to be
fixed, in which similar GYP_DEFINE key names (such as clang and
host_clang) would conflict, causing some defaults not to be installed
properly.

In order to enable clang=0 by default, some changes had to be made in
common.gypi:
  - compiler macros added to fix a compatibility issue between
    Chromium's base/mac/ folder and the actual OSX SDK
	- replaced clang_warning_flags variables with standard cflags
	  settings, plus xcode_settings for OSX
  - turned off warnings-as-errors for non-shaka code, rather than
		allow-listing specific warning types, since we can't actually fix
    those warnings on any platform
  - disabled two specific warnings in shaka code, both of which are
    caused by headers from our non-shaka dependencies

Also, one warning (missing "override" keyword) has been fixed in
vod_media_info_dump_muxer_listener.h.

Although these changes were done to make building simpler on a wider
array of platforms (arm64, for example), it seems to make the build a
bit faster, too.  For me, at least, on my main Linux workstation:
  - "gclient sync" now runs 20-30% faster
  - "ninja -C out/Release" now runs 5-13% faster

The following environment variables are no longer required:
  - DEPOT_TOOLS_WIN_TOOLCHAIN
  - MACOSX_DEPLOYMENT_TARGET
Documentation, Dockerfiles, and GitHub Actions workflows have been
updated to reflect this.

The following GYP_DEFINES are no longer required for anyone:
  - clang=0
  - host_clang=0
  - clang_xcode=1
  - use_allocator=none
  - use_experimental_allocator_shim=0
Documentation, Dockerfiles, and GitHub Actions workflows have been
updated to reflect this.

The following repos are no longer dependencies in gclient:
  - binutils
  - clang
  - gold
  - libc++
  - libc++abi
  - valgrind

The following gclient hooks have been removed:
  - clang
  - mac_toolchain
  - sysroot

Change-Id: Ie94ccbeec722ab73c291cb7df897d20761a09a70
2021-07-29 13:54:44 -07:00
nvincen f018c9a9bf
Added MPEG-H support (mha1, mhm1)
Implemented according to `Audio Amendment to Guidelines for Implementation:
DASH-IF Interoperability Points, Version 4.3`
(https://dashif.org/docs/Audio%20Amendment%20to%20DASH%20IOP%204.3.pdf).

Closes #930.
2021-06-29 23:10:53 -07:00
Daniel Cantarín b7ef11fa70
Workaround warning spam using http_file
Fixes #948.
2021-06-22 11:57:51 -07:00
Joey Parrish 56e227267c Fix python linter errors and add linter checks to CI
Internal CI systems and the new GitHub CI system were out of sync,
with the external system not doing any linting.  Further, the internal
system was using an internal-only linter for Python.

This creates a script for Python linting based on the open-source
pylint tool, checks in the Google Style Guide's pylintrc file, creates
a custom action for linting and adds it to the existing workflows,
fixes pre-existing linter errors in Python scripts, and updates pylint
overrides.

b/190743862

Change-Id: Iff1f5d4690b32479af777ded0834c31c2161bd10
2021-06-21 21:46:48 +00:00
Joey Parrish e32b35f3e1 Fix clang-format output during linting
Instead of printing a binary object, treat the output of clang-format
as a utf-8 string.

b/190743862

Change-Id: I596d223792597f8157fdee2d75773131cc858c9a
2021-06-21 21:46:48 +00:00
KongQun Yang c0d68083ac Fix playready_extra_header_data
It should be enclosed in the <DATA> element.

https://docs.microsoft.com/en-us/playready/specifications/playready-header-specification

Fixes #961.
Fixes b/173146719.

Change-Id: I7d235a44afda4a0a301e7231dc8a4e87cce124be
2021-06-18 18:34:24 +00:00
Joey Parrish a2e07a901e Refactor actions and workflows
It turns out that workflows were the wrong way to abstract reusable
pieces of work.  This turns common steps into custom actions (build
docs, build packager, test packager) which can be used as encapsulated
steps in multiple workflows.

This is a much more natural way to avoid duplication compared to the
previous approach of triggering one workflow from another.  This also
has the benefit of all of the steps of a release being represented on
GitHub as a single workflow, making it easier to understand what is
happening and what event triggered those steps.

Change-Id: Ife156d60069a39594c7b3bb3bc32080e6453b544
2021-06-17 10:32:24 -07:00
Joey Parrish 0f8749a211 CI overhaul based on GitHub Actions
This replaces Travis (for Linux & Mac) and Appveyor (for Windows) with
GitHub Actions.  In addition to using GitHub Actions to test PRs, this
also expands the automation of releases so that the only manual steps
are:

 1. Create a new CHANGELOG.md entry
 2. Create a release tag

Workflows have been create for building and testing PRs and releases,
for publishing releases to GitHub, NPM, and Docker Hub, and for
updating documentation on GitHub Pages.

When a new PR is created, GitHub Actions will:
 - Build and test on all combinations of OS, release type, and library
   type

Appveyor's workflow took ~2 hours, whereas the new GitHub Actions
workflow takes ~30 minutes.

When a new release tag is created, GitHub Actions will:
 - Create a draft release on GitHub
 - Extract release notes from CHANGELOG.md & attach them to the
   draft release
 - Build and test on all combinations of OS, release type, and library
   type, aborting if any build or test fails
 - Attach release artifacts to the draft release, aborting if any
   one artifact can't be prepared
 - Fully publish the draft release on GitHub
 - Publish the same release to NPM (triggered by GitHub release)
 - Publish the same release to Docker Hub (triggered by GitHub release)
 - Update the docs on GitHub pages

Closes #336 (GitHub Actions workflow to replace Travis and Appveyor)

b/190743862 (internal; tracking replacement of Travis)

Change-Id: Ic53eef60a8587c5d1487769a0cefaa16eb9b46e7
2021-06-16 11:52:02 -07:00
Joey Parrish 032cf2a345 Fix libpackager_type variable undefined by default
In e2efb5d4, I fixed shared_library builds on Windows, but I
introduced another issue in which the libpackager_type variable was
not correctly defined by default.  This meant that the build only
worked with this variable explicitly-defined in GYP_DEFINES when
gclient sync was run.

This fixes the default definition so that libpackager_type does not
need to be defined explicity.

Related to #318 (shared_library builds on Windows)

Issue #336 (progress toward GitHub Actions workflow to replace Travis
  and Appveyor, where we need to build and test shared_library on all
  platforms)

b/190743862 (internal; tracking replacement of Travis)

Change-Id: If353e1d3c312ab0c568d4d4d2b789e922d7216e1
2021-06-16 10:14:04 -07:00
Joey Parrish 098f58a143 Fix shared library test runs on Windows
Shared library builds worked, but failed tests because they were made
with the wrong CRT linker settings.  Strings allocated within the
library could not be freed outside the library because the dynamic CRT
was not used.

This sets necessary gyp variables to link with a dynamic CRT on
Windows, thereby fixing tests running in shared library mode that
otherwise hung in a GitHub Actions environment.

Related to #318 (shared_library builds on Windows)

Issue #336 (progress toward GitHub Actions workflow to replace Travis
  and Appveyor, where we need to build and test shared_library on all
  platforms)

b/190743862 (internal; tracking replacement of Travis)

Change-Id: Iffefd27c2aa4ec479ce1d10b099483e417d2231f
2021-06-15 19:37:32 -07:00
Joey Parrish e2efb5d41c Fix Windows shared_library builds
To make shared_library builds work on Windows with MSVS 2019, this
commit:
 - Silences a useless warning about a private member in dll-exported
   Status class.
 - Exports the File class used by packager.exe
 - Removes the explicit File dependency in packager.exe in favor of
   libpackager, now that File is exported
 - Add missing defines in packager.exe and packager_test.exe that
   instruct the linker to import Status and File from the library

Closes #318 (shared_library builds on Windows)
Issue #336 (progress toward GitHub Actions workflow to replace Travis
  and Appveyor, where we need to build and test shared_library on all
  platforms)
b/190743862 (internal; tracking replacement of Travis)

Change-Id: I091f1655d88d36f353f7df497101eef17729eefe
2021-06-15 13:16:47 -07:00
Joey Parrish 2526c61e9e
Merge pull request #955 from joeyparrish/master
At this point static_library builds are working in MSVS 2019. shared_library builds are still not working.

Closes #867 (MSVS 2019)
Issue #318 (progress toward shared_library support on Windows)
Issue #336 (progress toward replacing Travis & Appveyor with GitHub Actions, which uses MSVS 2019)
b/190743862 (internal; tracking replacement of Travis)
2021-06-14 15:25:39 -07:00
Joey Parrish ae89145015 Fix vector+unique_ptr issues with MSVS 2019
Because a StreamState object contains a unique_ptr, it is not
copyable.  A vector of StreamStates, therefore, causes a compile error
on resize or push_back, both of which invoke the copy constructor.

I don't know why MSVS complains, but clang does not.

To fix this, I'm changing vector<StreamState> into deque<StreamState>.

At this point static_library builds are working in MSVS 2019.
shared_library builds are still not working.

Issue #867 (MSVS 2019)
Issue #336 (progress toward replacing Travis & Appveyor with GitHub
  Actions, which uses MSVS 2019)
b/190743862 (internal; tracking replacement of Travis)

Change-Id: Iaa9d5fc357102d15eac96c29ebeee7c7236e976b
2021-06-14 12:17:24 -07:00
KongQun Yang ffb4d771ca Update Dockerfile due to depot_tools update
Change-Id: Iaf2415a2fb954a3f2473233eb0522edc211b4a9b
2021-06-11 11:07:41 -07:00
KongQun Yang 72ea5d30f0 Fix gpylint errors
Change-Id: I861cfe98d50867ab9ee9be69cd4ffc3a7157fbd8
2021-06-09 12:17:48 -07:00
Samidh 4686454a73
Add segment_list support for DASH on-demand profile
Configurable under flag --dash_force_segment_list, default to false.

Note that DASH live profile is not supported right now.
2021-05-25 12:08:58 -07:00
Daniel Cantarín fe9e26d5d9
Remove "atomic" warning for HttpFile
Fixes #945.
2021-05-25 11:59:53 -07:00
Mattias Wadman 056140ae83
Update git-clang-format URL
The previous svn link is broken.
2021-05-07 10:21:40 -07:00
Daniel Cantarín 8515a66031
Changed default HTTP UserAgent to ShakaPackager/<version>
Closes #939.
2021-05-07 10:15:44 -07:00
Mattias Wadman 62f37eb3b7
Ignore matroska projection metadata
Warn instead of fail parsing.

Closes #932.
2021-05-07 10:13:02 -07:00
KongQun Yang 2e521c8413 Remove another use of regex library
It is not working correctly in gcc 4.8 or earlier, which is still
popular (bundled by default in CentOS 7).

Issue #865, #929.

Change-Id: I136446a70831bd0237cd29646dd349fe7558176b
2021-05-05 18:01:27 +00:00
Vishal Shah d9124d6aaa
[WEBVTT] Fix missing text alignment tags from output
Legacy players, e.g. older versions of ExoPlayer, do not handle default webvtt text alignment correctly. Need to specify `align:center` explicitly cues without text alignment for backwards compatibility.

Fixes #925.
2021-05-04 22:57:43 -07:00
Daniel Cantarín 37b16b4091
HTTP File upload fixes and tweaks
- Do not write the HTTP PUT response to cache which can potentially overflow the cache buffer as it is not consumed.
- VLOG(1) instead of LOG(ERROR) on HttpFile::Size() as it can be called during normal code execution.
- Add a command line flag `--user_agent` to allow users to specify their custom user agent string.

Fixes #939.
2021-05-04 22:53:23 -07:00
KongQun Yang 4528bdb330 Remove the use of regex library
It is not working correctly in gcc 4.8 or earlier, which is still
popular (e.g. bundled by default in CentOS 7).

Fixes #865, #929.

Change-Id: I55a42428dbd2a12fc2c3b1e6a49fdd662a295dca
2021-05-04 02:09:08 +00:00
KongQun Yang 23953bf931 Update check_formatting.py to work with python3
Change-Id: Iff3ba89f3b3e6fe0991e48482546084d1ba66ac5
2021-05-04 02:07:49 +00:00
Daniel Cantarín f6c02e629d
Generate object type properly for MPEG-1 audio
Fix #905.
2021-04-04 22:47:31 -07:00
Joey Parrish d48bf0f48e fix: Explicitly signal the lack of CEA captions in HLS
Closes #922

Change-Id: If097a66642ca40c355a476655ff1394b7b937b17
2021-03-30 18:26:37 +00:00
Jacob Trimble c1f64e5350 Fix transparency case in DVB-SUB.
This fixes some math errors in the color conversions and handles the
case of Y=0.

Fixes #903

Change-Id: I796246e4d62a3161b44916f97e9e98f9203ad338
2021-03-29 16:34:39 +00:00
Daniel Cantarín dd935f6dc3
TTML: change "imagetype" attribute to camel case
Fixes #908
2021-03-09 10:10:32 -08:00
Sergio Garcia Murillo f9908362f8
Prevent seg fault if webm fragment is not initialized or last frame is EOS
Fixes #900
2021-03-07 15:01:08 -08:00
Sergio Garcia Murillo b8ce44aba0
Prevent seg fault if mp4 fragment is not initialized
Related to #900.
2021-03-02 23:53:58 -08:00
Jacob Trimble 00af192626 Cleanup HttpFile and related PR.
This implements many of the comments made on the PR and cleans up those
files.

Closes #149

Change-Id: Ice73fe3c04a6f595da6986a4c070e50cb20f9435
2021-03-02 17:43:47 +00:00
Jacob Trimble f0a52cbbf2 Fix TTML codec string in HLS.
Issue #903

Change-Id: I1f73d16ad79c2593996d7d50f9556984302fe21e
2021-02-25 16:17:45 -08:00
Jacob Trimble 025e10de69 Add better detection for text streams.
Change-Id: I7f44026c223b868cd73f52bf700e29372bb9461e
2021-02-17 10:40:14 -08:00
Jacob Trimble a0f3f2cd3a Add cc_index to stream descriptor.
This also allows setting the language of different text streams from
the same input.  Multiple streams can use the same input stream
using different cc_index values and can each use a different language.

This also will try to pull the language from the input if not
specified.

Change-Id: I7078710b509b7d77dad8cb4299a82f954af7e9e7
2021-02-17 18:33:53 +00:00
Jacob Trimble 78be14c092 Add DVB-sub parser
Note that this only supports a single page within the DVB-sub stream.
Multiple pages will be merged together.  A follow-up will allow
selecting a specific page.

This only supports outputting using TTML or MP4+TTML; you cannot have
DVB-sub output nor can you output it in WebVTT.  Since DVB-sub
uses images, it is hard to impossible to do this with WebVTT.

This also only supports interlaced images, not progressive images
nor text.

Closes #832

Change-Id: Id6dbb6393c7b9a05722e61c6bd255bef5e69a7d8
2021-02-17 18:32:03 +00:00
Jacob Trimble 95089593fc Don't re-open WebVTT file to determine size.
Change-Id: Id92226adce813b7d0c4c741e47e36dbf8f208797
2021-02-08 20:31:13 +00:00
JPeMu 36ef7ec945
[MPEG-TS] Fix PCR reserved bits not being set correctly
Fixes #893.
2021-02-03 12:09:07 -08:00
Ole Andre Birkedal aa17521268
HTTP PUT output support (#737)
Issue #149

Co-authored-by: Andreas Motl <andreas.motl@elmyra.de>
Co-authored-by: Rintaro Kuroiwa <rkuroiwa@google.com>
Co-authored-by: Ole Andre Birkedal <o.birkedal@sportradar.com>
2021-02-02 10:51:50 -08:00