Commit Graph

1094 Commits

Author SHA1 Message Date
Joey Parrish 1ff26b4783
feat: Move all third-party deps into git submodules (#1083)
This also normalizes the structure of all submodule folders so that
there is a clear place to put configurations, outside the submodule
source, but limited in scope to that folder.

Issue #1047 (New build system)
2022-08-17 10:42:00 -07:00
Joey Parrish 31e116faec
feat: Respect the file mode for HttpFiles (#1081)
For a read mode, make a GET request.  Otherwise, make a PUT request.
2022-08-17 10:40:13 -07:00
Joey Parrish 868208198a
feat: Remove third-party source in-repo (#1082)
This removes apple_apsl and libevent (which are no longer used) and
libxml and protobuf (which will be replaced in a follow-up with
submodules).

This change only contains deletions, to make it easier to review
separately from later additions for submodules and cmake integration.

Issue #1047 (New build system)
2022-08-16 15:07:21 -07:00
Joey Parrish e6b57f72a8
feat: First phase of CMake build system implementation (#1072)
There are a lot of changes in this first phase, because there was a
lot of infrastructure required to get some meaningful amount of
porting done.  Future PRs should be simpler.

<b>Summary of changes:</b><details>

 - Remove old deps:
   - boringssl (replaced with mbedtls, lighter, easier to build)
   - gflags (replaced with absl::flags)
   - Chromium build tools
 - New deps to replace parts of Chromium base:
   - abseil-cpp
   - glog
   - nlohmann::json (for tests only)
 - Submodules, updates, and CMake build rules for third-party
   libraries:
   - curl
   - gmock/gtest
 - Ported internal libraries and their tests by removing Chromium deps
   and adding CMake build rules:
   - file (now using C++17 filesystem APIs)
   - license_notice
   - status
   - version
 - Test improvements
   - Removed file tests that can never be re-enabled
   - Re-enabled all other disabled file tests
   - Debug JSON values when HTTP tests fail
   - Fixed chunked-encoding issues in HTTP tests
 - Updated and refactored Dockerfiles testing
   - All docker files working, with OS versions updated to meet the
     new tool requirements
   - Local docker builds no longer write files to your working
     directory as root
   - Local docker builds can now be run in parallel without clobbering
     each others' build outputs
   - DEBUG=1 can drop you into an interactive shell when a docker
     build fails
 - Updated and heavily refactored workflows and Dockerfiles
   - All docker files now tested in parallel on GitHub, speeding up CI
   - All common workflow components broken out and using workflow_call
     instead of custom actions
   - Self-hosted runners now optional, to make testing easier on forks
   - CMake porting works-in-process can now be fully tested on GitHub
   - Building ported libraries and passing ported tests on all three
     platforms!
 - CI hacks for macOS removed, now testing on macos-latest!
 - Python2 no longer required!  (Only Python3)
 - Using strict build flags, treating all warnings as errors.

</details>

<b>Required to build:</b>

 - CMake >= 3.16
 - Python 3
 - A compiler supporting C++ >= 17
   - g++ >= 9 if using GCC (Clang also fine)
   - MSVC for Windows

<b>Still needs work:</b><details>

 - Moving other dependencies into submodules (if we keep them):
   - apple_apsl
   - icu
   - libevent
   - libpng
   - libwebm
   - libxml
   - modp_b64
   - protobuf
   - zlib
 - Port remaining internal libraries:
   - app
   - hls
   - media/base
   - media/chunking
   - media/codecs
   - media/crypto
   - media/demuxer
   - media/event
   - media/formats/dvb
   - media/formats/mp2t
   - media/formats/mp4
   - media/formats/packed_audio
   - media/formats/ttml
   - media/formats/webm
   - media/formats/webvtt
   - media/formats/wvm
   - media/origin
   - media/public
   - media/replicator
   - media/trick_play
   - mpd
 - Port main application
   - Add logging flags in absl and connect them to glog (which expects
     gflags)
 - Port pssh-box.py
 - Port main test targets (packager_test.py and packager_app.py)
 - Updating all requirement and build documentation
 - Remove any remaining refs to gclient, depot_tools, ninja
 - Update and complete release workflows using release-please
</details>

Issue #346 (Switch to abseil)
Issue #1047 (New build system)
2022-08-16 11:34:51 -07:00
Vishal Shah dc0395291a
fix: dash_roles add role=description for DVS audio per DASH-IF-IOP-v4.3 (#1054)
Fixes support for description role for audio DVS tracks as per spec.
2022-06-02 09:40:34 -07:00
Vishal Shah b9d477b969
fix: webvtt single cue do not fail on EOS (#1061)
While Parsing cue body check for the block size. 
If it's the last block do not error if it doesn't have a newline.

Fixes #1018
2022-06-02 09:27:47 -07:00
Joey Parrish 3fd538a587
fix: Roll back depot_tools, bypass vpython (#1045)
Using the latest depot_tools no longer works.  depot_tools also wants
to auto-update itself, which must now be disabled.

We also need to disable the copy of python (vpython) included in
depot_tools, since for some distros, it has dependencies on system
libraries that no longer exist.

Finally, we need to force some distros to use python 2, because our
build system is ancient and needs to be ripped out and replaced some
day soon.

This fixes build issues in our CI, our Dockerfiles, and in general on
certain platforms or distros.

Closes #1023
2022-03-08 16:46:18 -08:00
Joey Parrish f577e2a0cf
chore: Update URLs after moving projects (#1042)
Since a project URL is encoded into outputs, this means also updating
the golden output files.

Closes #1043
2022-03-07 11:56:34 -08:00
Vishal Shah e1b0c7c454
Fix WEBVTT Region parse 100 precent (#1006) 2021-11-15 21:28:15 -08:00
Joey Parrish e2d66b33fa fix: Fix crash in static-linked linux builds
The official, static-linked linux builds were crashing in their use of
getaddrinfo, which libcurl was configured to use.  Both getaddrinfo
and all of its alternatives available in glibc fail with static
linking.

We can fix this by configuring libcurl to use libc-ares on Linux
instead.  This allows us to keep the benefits of a statically-linked
Linux binary.

Closes #996

Change-Id: Ib4a9eb939813fd165727788726459ef4adf3fc4d
2021-10-14 10:35:49 -07:00
Joey Parrish 1b96427983 test: Improve debugging ability for docker tests
The script in packager/testing/dockers/test_dockers.sh now outputs
more useful info for debugging, uses unique container names per OS so
that the containers can be debugged, and allows filtering to re-run
specific OSes if a build fails.

Change-Id: I0cace282549c093a643009f5e60e7545a039168c
2021-10-14 10:16:22 -07:00
Joey Parrish 0afda1725c test: Update Dockerfiles
This updates the main Dockerfile and all the docker-based
distro-specific tests.  The base OS versions have been updated to
versions that have not reached end-of-life status yet, and the list of
dependencies required has been updated and pruned.

Change-Id: Ibcff2f60e739fd5d999af100af76c40aa91a75bc
2021-10-13 12:37:16 -07:00
Joey Parrish efbca399c0 fix: Add missing limits header
In many places, we used std::numeric_limits without including the
proper header.  This would build on some Linux distributions, but not
others.

This adds the missing includes, fixing the build on Fedora, among
other distros.

Change-Id: I63e9e37e5973fe23bbdf9868552db51062b1dae4
2021-10-13 12:25:34 -07:00
Joey Parrish 7392d8003e fix: Fix failure fetching encryption keys
In one of the low-latency changes, a change was made to HttpFile that
caused responses to HTTP POST requests to go missing.  This resulted
in failures to fetch encryption keys.

The breaking change was recommended by me in a PR review, and was not
caught by any unit tests.  New tests would be ideal, but I chose to
fix the bug first, rather than leave the repo broken.

This bug was brought to my attention in google/shaka-streamer#87 and
has not appeared in any release versions.

Change-Id: I9eca73d187a8a30f16c4a920fcdb7b4872253858
2021-09-06 21:54:34 -07:00
Caitlin O'Callaghan c87c5bcdef
Fix for gap size warning in Low Latency mode (#985)
## The issue
- With LL-DASH mode enabled, the gap size warning was hit and printed to the console every time a new segment was registered to the manifest.
- This occurred because the first chunk's size and duration were being stored for each segment, rather than the full segment size and duration. Note, only the first chunk's metrics are known at first because in low latency mode, the segment is registered to the manifest before it is finished being processed and written.
- Because of this, the gap size check was comparing the end time of the first chunk in the previous segment to the beginning time of the current segment, causing the check to fail every time.

## The Fix
- Update a low latency segment's duration and size once the segment file has been fully written.
- The full segment size and duration will be used to update the bandwidth estimator and the segment info list. 
- Updating the segment info list to hold the full duration is necessary for satisfying [the gap size check found in Represenation.cc](https://github.com/google/shaka-packager/blob/master/packager/mpd/base/representation.cc#L391).
- NOTE: bandwidth estimation is currently only used in HLS
2021-09-03 09:57:43 -07:00
Joey Parrish 249c83f14b build: Fix additional python2/3 issues
The generate_version_string script was only producing correct results
in python 2, not python 3.  The gyp file that references it explicitly
runs it in python3.  The shebang line of the script has been updated
to match.  The script itself has been updated such that it now works
correctly in both python2 and python3.

Scripts that are only used as modules (not executed directly) have had
their shebang lines removed.

This fixes CI failures on GitHub Actions.

Change-Id: I309bafd2fb05e8fb33f5e092ead179c8c42ea5d3
2021-08-26 12:22:20 -07:00
Caitlin O'Callaghan cd018a71c3
Low latency DASH support (#979)
# LL-DASH Support
These changes add support for LL-DASH streaming. 

**NOTE:** LL-HLS support is still in progress, but it's coming. :) 

## Testing
`./chunking_unittest --gtest_filter="ChunkingHandlerTest.LowLatencyDash"`

`./media_event_unittest --gtest_filter="MpdNotifyMuxerListenerTest.LowLatencyDash"`

`./mpd_unittest --gtest_filter="PeriodTest.LowLatencyDashMpdGetXml"`
`./mpd_unittest --gtest_filter="SimpleMpdNotifierTest.NotifyAvailabilityTimeOffset"`
`./mpd_unittest --gtest_filter="SimpleMpdNotifierTest.NotifySegmentDuration"`
`./mpd_unittest --gtest_filter="LowLatencySegmentTest.LowLatencySegmentTemplate"`

Note, packager_test must be run from the main project directory
`./out/Release/packager_test --gtest_filter="PackagerTest.LowLatencyDashEnabledAndUtcTimingNotSet"`
`./out/Release/packager_test --gtest_filter="PackagerTest.LowLatencyDashEnabledAndUtcTimingNotSet"`
2021-08-25 08:38:05 -07:00
Joey Parrish ac125564b9 build: Fix pylint 2.10 issues, use python3 where possible
The newest pylint release complained about several issues that the
older release did not.  This resolves those issues:

 - removes unneeded "u" prefix from strings
 - adds "encoding" parameter for all open() calls
 - because "encoding" is a python3-only parameter, use python3 in all
   the scripts that we control

Unfortunately, python2 is required for any scripts that import modules
from the ancient Chromium build system we're using (referenced by
DEPS), as well as kokoro scripts.

Change-Id: I2e9f97af508efe58b5a71de21740e59b1528affd
2021-08-24 12:27:03 -07:00
Joey Parrish b411af7ed9
ci: Produce static release executables on Linux (#978)
We never produced static release executables on Linux before, but the dynamic libraries they depended on were universal enough that nobody noticed. Now that we have released v2.5 and switched to GitHub Actions for CI builds, the Linux executables depend on libatomic, which is causing issues for some users.

Although we can't create fully-static executables on macOS or Windows, we can at least do so on Linux.

This adds a GYP variable static_link_binaries which can be set to request full-static binaries on Linux. This also exposes the Chromium build variable disable_fatal_linker_warnings, which is necessary when static linking on Linux due to static-link-related warnings generated by libcurl for its use of getaddrinfo. Finally, this enforces the definition of __UCLIBC__ with static linking on Linux, which is the only way to disable malloc hooks in Chromium base. Those hooks cause linker failures when linking statically on Linux.

A new check has been added to the release workflow to ensure that the builds we create are statically linked on Linux.

Closes #965
2021-08-12 20:14:43 -07:00
Joey Parrish cfbe5c08c2 cleanup: Convert all time parameters to signed
This converts all time parameters to signed, finishing a cleanup that
was started in 2018 in b4256bf0.  This changes the type of:
  - timestamps
    - PTS specifically
  - timestamp offsets
  - timescales
  - durations

This excludes:
  - MP4 box definitions
  - DTS specifically

This is meant to address signed/unsigned conversion issues on arm64
that caused some test cases to fail.

Change-Id: Ic752a20cbc6e31fea6bc0894d1771833171e7cbe
2021-08-05 18:24:15 +00:00
Joey Parrish eba176f032 build: Disable CPU-specific optimizations in libpng
This fixes the Debug build of libpng on arm64 by avoiding CPU-specific
optimizations that are not in our sources list.  The Release build
appears to have been unaffected, possibly due to link-time
optimizations or dead code stripping.

Change-Id: I900e00fe30b9f3748f2587cfea89a636b3a19811
2021-08-04 12:27:10 -07:00
Joey Parrish 68b50f656d build: Stop using hermetic clang, libc++, etc
This brings our default build config more in line with what is
necessary for some platforms anyway: using the system-installed
toolchain and sysroot to build everything.

We will no longer fetch source or binaries for any specific build
tools, such as libc++, clang, gold, binutils, or valgrind.

The main part of this change is the changing of default gyp settings
in gyp_packager.py.  For this, a bug in gyp_packager.py had to be
fixed, in which similar GYP_DEFINE key names (such as clang and
host_clang) would conflict, causing some defaults not to be installed
properly.

In order to enable clang=0 by default, some changes had to be made in
common.gypi:
  - compiler macros added to fix a compatibility issue between
    Chromium's base/mac/ folder and the actual OSX SDK
	- replaced clang_warning_flags variables with standard cflags
	  settings, plus xcode_settings for OSX
  - turned off warnings-as-errors for non-shaka code, rather than
		allow-listing specific warning types, since we can't actually fix
    those warnings on any platform
  - disabled two specific warnings in shaka code, both of which are
    caused by headers from our non-shaka dependencies

Also, one warning (missing "override" keyword) has been fixed in
vod_media_info_dump_muxer_listener.h.

Although these changes were done to make building simpler on a wider
array of platforms (arm64, for example), it seems to make the build a
bit faster, too.  For me, at least, on my main Linux workstation:
  - "gclient sync" now runs 20-30% faster
  - "ninja -C out/Release" now runs 5-13% faster

The following environment variables are no longer required:
  - DEPOT_TOOLS_WIN_TOOLCHAIN
  - MACOSX_DEPLOYMENT_TARGET
Documentation, Dockerfiles, and GitHub Actions workflows have been
updated to reflect this.

The following GYP_DEFINES are no longer required for anyone:
  - clang=0
  - host_clang=0
  - clang_xcode=1
  - use_allocator=none
  - use_experimental_allocator_shim=0
Documentation, Dockerfiles, and GitHub Actions workflows have been
updated to reflect this.

The following repos are no longer dependencies in gclient:
  - binutils
  - clang
  - gold
  - libc++
  - libc++abi
  - valgrind

The following gclient hooks have been removed:
  - clang
  - mac_toolchain
  - sysroot

Change-Id: Ie94ccbeec722ab73c291cb7df897d20761a09a70
2021-07-29 13:54:44 -07:00
nvincen f018c9a9bf
Added MPEG-H support (mha1, mhm1)
Implemented according to `Audio Amendment to Guidelines for Implementation:
DASH-IF Interoperability Points, Version 4.3`
(https://dashif.org/docs/Audio%20Amendment%20to%20DASH%20IOP%204.3.pdf).

Closes #930.
2021-06-29 23:10:53 -07:00
Daniel Cantarín b7ef11fa70
Workaround warning spam using http_file
Fixes #948.
2021-06-22 11:57:51 -07:00
Joey Parrish 56e227267c Fix python linter errors and add linter checks to CI
Internal CI systems and the new GitHub CI system were out of sync,
with the external system not doing any linting.  Further, the internal
system was using an internal-only linter for Python.

This creates a script for Python linting based on the open-source
pylint tool, checks in the Google Style Guide's pylintrc file, creates
a custom action for linting and adds it to the existing workflows,
fixes pre-existing linter errors in Python scripts, and updates pylint
overrides.

b/190743862

Change-Id: Iff1f5d4690b32479af777ded0834c31c2161bd10
2021-06-21 21:46:48 +00:00
Joey Parrish e32b35f3e1 Fix clang-format output during linting
Instead of printing a binary object, treat the output of clang-format
as a utf-8 string.

b/190743862

Change-Id: I596d223792597f8157fdee2d75773131cc858c9a
2021-06-21 21:46:48 +00:00
KongQun Yang c0d68083ac Fix playready_extra_header_data
It should be enclosed in the <DATA> element.

https://docs.microsoft.com/en-us/playready/specifications/playready-header-specification

Fixes #961.
Fixes b/173146719.

Change-Id: I7d235a44afda4a0a301e7231dc8a4e87cce124be
2021-06-18 18:34:24 +00:00
Joey Parrish a2e07a901e Refactor actions and workflows
It turns out that workflows were the wrong way to abstract reusable
pieces of work.  This turns common steps into custom actions (build
docs, build packager, test packager) which can be used as encapsulated
steps in multiple workflows.

This is a much more natural way to avoid duplication compared to the
previous approach of triggering one workflow from another.  This also
has the benefit of all of the steps of a release being represented on
GitHub as a single workflow, making it easier to understand what is
happening and what event triggered those steps.

Change-Id: Ife156d60069a39594c7b3bb3bc32080e6453b544
2021-06-17 10:32:24 -07:00
Joey Parrish 0f8749a211 CI overhaul based on GitHub Actions
This replaces Travis (for Linux & Mac) and Appveyor (for Windows) with
GitHub Actions.  In addition to using GitHub Actions to test PRs, this
also expands the automation of releases so that the only manual steps
are:

 1. Create a new CHANGELOG.md entry
 2. Create a release tag

Workflows have been create for building and testing PRs and releases,
for publishing releases to GitHub, NPM, and Docker Hub, and for
updating documentation on GitHub Pages.

When a new PR is created, GitHub Actions will:
 - Build and test on all combinations of OS, release type, and library
   type

Appveyor's workflow took ~2 hours, whereas the new GitHub Actions
workflow takes ~30 minutes.

When a new release tag is created, GitHub Actions will:
 - Create a draft release on GitHub
 - Extract release notes from CHANGELOG.md & attach them to the
   draft release
 - Build and test on all combinations of OS, release type, and library
   type, aborting if any build or test fails
 - Attach release artifacts to the draft release, aborting if any
   one artifact can't be prepared
 - Fully publish the draft release on GitHub
 - Publish the same release to NPM (triggered by GitHub release)
 - Publish the same release to Docker Hub (triggered by GitHub release)
 - Update the docs on GitHub pages

Closes #336 (GitHub Actions workflow to replace Travis and Appveyor)

b/190743862 (internal; tracking replacement of Travis)

Change-Id: Ic53eef60a8587c5d1487769a0cefaa16eb9b46e7
2021-06-16 11:52:02 -07:00
Joey Parrish 032cf2a345 Fix libpackager_type variable undefined by default
In e2efb5d4, I fixed shared_library builds on Windows, but I
introduced another issue in which the libpackager_type variable was
not correctly defined by default.  This meant that the build only
worked with this variable explicitly-defined in GYP_DEFINES when
gclient sync was run.

This fixes the default definition so that libpackager_type does not
need to be defined explicity.

Related to #318 (shared_library builds on Windows)

Issue #336 (progress toward GitHub Actions workflow to replace Travis
  and Appveyor, where we need to build and test shared_library on all
  platforms)

b/190743862 (internal; tracking replacement of Travis)

Change-Id: If353e1d3c312ab0c568d4d4d2b789e922d7216e1
2021-06-16 10:14:04 -07:00
Joey Parrish 098f58a143 Fix shared library test runs on Windows
Shared library builds worked, but failed tests because they were made
with the wrong CRT linker settings.  Strings allocated within the
library could not be freed outside the library because the dynamic CRT
was not used.

This sets necessary gyp variables to link with a dynamic CRT on
Windows, thereby fixing tests running in shared library mode that
otherwise hung in a GitHub Actions environment.

Related to #318 (shared_library builds on Windows)

Issue #336 (progress toward GitHub Actions workflow to replace Travis
  and Appveyor, where we need to build and test shared_library on all
  platforms)

b/190743862 (internal; tracking replacement of Travis)

Change-Id: Iffefd27c2aa4ec479ce1d10b099483e417d2231f
2021-06-15 19:37:32 -07:00
Joey Parrish e2efb5d41c Fix Windows shared_library builds
To make shared_library builds work on Windows with MSVS 2019, this
commit:
 - Silences a useless warning about a private member in dll-exported
   Status class.
 - Exports the File class used by packager.exe
 - Removes the explicit File dependency in packager.exe in favor of
   libpackager, now that File is exported
 - Add missing defines in packager.exe and packager_test.exe that
   instruct the linker to import Status and File from the library

Closes #318 (shared_library builds on Windows)
Issue #336 (progress toward GitHub Actions workflow to replace Travis
  and Appveyor, where we need to build and test shared_library on all
  platforms)
b/190743862 (internal; tracking replacement of Travis)

Change-Id: I091f1655d88d36f353f7df497101eef17729eefe
2021-06-15 13:16:47 -07:00
Joey Parrish 2526c61e9e
Merge pull request #955 from joeyparrish/master
At this point static_library builds are working in MSVS 2019. shared_library builds are still not working.

Closes #867 (MSVS 2019)
Issue #318 (progress toward shared_library support on Windows)
Issue #336 (progress toward replacing Travis & Appveyor with GitHub Actions, which uses MSVS 2019)
b/190743862 (internal; tracking replacement of Travis)
2021-06-14 15:25:39 -07:00
Joey Parrish ae89145015 Fix vector+unique_ptr issues with MSVS 2019
Because a StreamState object contains a unique_ptr, it is not
copyable.  A vector of StreamStates, therefore, causes a compile error
on resize or push_back, both of which invoke the copy constructor.

I don't know why MSVS complains, but clang does not.

To fix this, I'm changing vector<StreamState> into deque<StreamState>.

At this point static_library builds are working in MSVS 2019.
shared_library builds are still not working.

Issue #867 (MSVS 2019)
Issue #336 (progress toward replacing Travis & Appveyor with GitHub
  Actions, which uses MSVS 2019)
b/190743862 (internal; tracking replacement of Travis)

Change-Id: Iaa9d5fc357102d15eac96c29ebeee7c7236e976b
2021-06-14 12:17:24 -07:00
KongQun Yang ffb4d771ca Update Dockerfile due to depot_tools update
Change-Id: Iaf2415a2fb954a3f2473233eb0522edc211b4a9b
2021-06-11 11:07:41 -07:00
KongQun Yang 72ea5d30f0 Fix gpylint errors
Change-Id: I861cfe98d50867ab9ee9be69cd4ffc3a7157fbd8
2021-06-09 12:17:48 -07:00
Samidh 4686454a73
Add segment_list support for DASH on-demand profile
Configurable under flag --dash_force_segment_list, default to false.

Note that DASH live profile is not supported right now.
2021-05-25 12:08:58 -07:00
Daniel Cantarín fe9e26d5d9
Remove "atomic" warning for HttpFile
Fixes #945.
2021-05-25 11:59:53 -07:00
Mattias Wadman 056140ae83
Update git-clang-format URL
The previous svn link is broken.
2021-05-07 10:21:40 -07:00
Daniel Cantarín 8515a66031
Changed default HTTP UserAgent to ShakaPackager/<version>
Closes #939.
2021-05-07 10:15:44 -07:00
Mattias Wadman 62f37eb3b7
Ignore matroska projection metadata
Warn instead of fail parsing.

Closes #932.
2021-05-07 10:13:02 -07:00
KongQun Yang 2e521c8413 Remove another use of regex library
It is not working correctly in gcc 4.8 or earlier, which is still
popular (bundled by default in CentOS 7).

Issue #865, #929.

Change-Id: I136446a70831bd0237cd29646dd349fe7558176b
2021-05-05 18:01:27 +00:00
Vishal Shah d9124d6aaa
[WEBVTT] Fix missing text alignment tags from output
Legacy players, e.g. older versions of ExoPlayer, do not handle default webvtt text alignment correctly. Need to specify `align:center` explicitly cues without text alignment for backwards compatibility.

Fixes #925.
2021-05-04 22:57:43 -07:00
Daniel Cantarín 37b16b4091
HTTP File upload fixes and tweaks
- Do not write the HTTP PUT response to cache which can potentially overflow the cache buffer as it is not consumed.
- VLOG(1) instead of LOG(ERROR) on HttpFile::Size() as it can be called during normal code execution.
- Add a command line flag `--user_agent` to allow users to specify their custom user agent string.

Fixes #939.
2021-05-04 22:53:23 -07:00
KongQun Yang 4528bdb330 Remove the use of regex library
It is not working correctly in gcc 4.8 or earlier, which is still
popular (e.g. bundled by default in CentOS 7).

Fixes #865, #929.

Change-Id: I55a42428dbd2a12fc2c3b1e6a49fdd662a295dca
2021-05-04 02:09:08 +00:00
KongQun Yang 23953bf931 Update check_formatting.py to work with python3
Change-Id: Iff3ba89f3b3e6fe0991e48482546084d1ba66ac5
2021-05-04 02:07:49 +00:00
Daniel Cantarín f6c02e629d
Generate object type properly for MPEG-1 audio
Fix #905.
2021-04-04 22:47:31 -07:00
Joey Parrish d48bf0f48e fix: Explicitly signal the lack of CEA captions in HLS
Closes #922

Change-Id: If097a66642ca40c355a476655ff1394b7b937b17
2021-03-30 18:26:37 +00:00
Jacob Trimble c1f64e5350 Fix transparency case in DVB-SUB.
This fixes some math errors in the color conversions and handles the
case of Y=0.

Fixes #903

Change-Id: I796246e4d62a3161b44916f97e9e98f9203ad338
2021-03-29 16:34:39 +00:00
Daniel Cantarín dd935f6dc3
TTML: change "imagetype" attribute to camel case
Fixes #908
2021-03-09 10:10:32 -08:00
Sergio Garcia Murillo f9908362f8
Prevent seg fault if webm fragment is not initialized or last frame is EOS
Fixes #900
2021-03-07 15:01:08 -08:00
Sergio Garcia Murillo b8ce44aba0
Prevent seg fault if mp4 fragment is not initialized
Related to #900.
2021-03-02 23:53:58 -08:00
Jacob Trimble 00af192626 Cleanup HttpFile and related PR.
This implements many of the comments made on the PR and cleans up those
files.

Closes #149

Change-Id: Ice73fe3c04a6f595da6986a4c070e50cb20f9435
2021-03-02 17:43:47 +00:00
Jacob Trimble f0a52cbbf2 Fix TTML codec string in HLS.
Issue #903

Change-Id: I1f73d16ad79c2593996d7d50f9556984302fe21e
2021-02-25 16:17:45 -08:00
Jacob Trimble 025e10de69 Add better detection for text streams.
Change-Id: I7f44026c223b868cd73f52bf700e29372bb9461e
2021-02-17 10:40:14 -08:00
Jacob Trimble a0f3f2cd3a Add cc_index to stream descriptor.
This also allows setting the language of different text streams from
the same input.  Multiple streams can use the same input stream
using different cc_index values and can each use a different language.

This also will try to pull the language from the input if not
specified.

Change-Id: I7078710b509b7d77dad8cb4299a82f954af7e9e7
2021-02-17 18:33:53 +00:00
Jacob Trimble 78be14c092 Add DVB-sub parser
Note that this only supports a single page within the DVB-sub stream.
Multiple pages will be merged together.  A follow-up will allow
selecting a specific page.

This only supports outputting using TTML or MP4+TTML; you cannot have
DVB-sub output nor can you output it in WebVTT.  Since DVB-sub
uses images, it is hard to impossible to do this with WebVTT.

This also only supports interlaced images, not progressive images
nor text.

Closes #832

Change-Id: Id6dbb6393c7b9a05722e61c6bd255bef5e69a7d8
2021-02-17 18:32:03 +00:00
Jacob Trimble 95089593fc Don't re-open WebVTT file to determine size.
Change-Id: Id92226adce813b7d0c4c741e47e36dbf8f208797
2021-02-08 20:31:13 +00:00
JPeMu 36ef7ec945
[MPEG-TS] Fix PCR reserved bits not being set correctly
Fixes #893.
2021-02-03 12:09:07 -08:00
Ole Andre Birkedal aa17521268
HTTP PUT output support (#737)
Issue #149

Co-authored-by: Andreas Motl <andreas.motl@elmyra.de>
Co-authored-by: Rintaro Kuroiwa <rkuroiwa@google.com>
Co-authored-by: Ole Andre Birkedal <o.birkedal@sportradar.com>
2021-02-02 10:51:50 -08:00
Jacob Trimble 5bcda6b88b Use TsStreamType for MP2T parser.
This also changes some of the logs to error so the user can see why
the parsing failed.

Change-Id: Ib8b7a5076462bccc718e17ef9e0a57d172d1f7b4
2021-02-01 20:13:13 +00:00
Qingquan(Q.Q.) Wang a6cce5e611
Fix wrong _create_bin_int implementation in pssh-box.py
#885.
2021-01-30 11:45:50 -08:00
Jacob Trimble 2eb32ee177 Propagate Flush errors in MP2T parser.
Issue #832

Change-Id: I59f31ff491437b81ffc22ab5760ad0c059e9933e
2021-01-20 18:27:31 +00:00
Jacob Trimble 89d407f9ae Add subtitle composition to DVB-sub parser.
Issue #832

Change-Id: Iababe884619e1e48f1abe0806e8b863c95a3c1ef
2021-01-20 18:26:28 +00:00
Jacob Trimble 32c5393fba Add helpers for DVB-sub colors.
Issue #832

Change-Id: I6350306c7d9a6450d82994bbd9a9a239986bc3fa
2021-01-20 18:25:43 +00:00
Vishal Shah 427d264b46
Do not create encryption handler for text streams
The text stream should not be encrypted.

Fixes #883.
2021-01-19 11:57:24 -08:00
Vishal Shah 8e3e8d3e8e
[WEBVTT] Support both center and middle text alignments
Fixes #882.
2021-01-19 11:45:20 -08:00
KongQun Yang b231c36539 Properly handle SkipBytes with num_bytes as 0
Previously if there are no bytes remaining, SkipBytes(0) would fail,
which results in parsing error in
AACAudioSpecificConfig::ParseProgramConfigElement.

Fixes #875.

Change-Id: I271899a37303d0d3fa0cf1bf90f99227058b82df
2020-12-23 11:08:33 -08:00
KongQun Yang d90cf9a0fd Add end to end tests for docker tests
Change-Id: I5aae2122db1294494df9e90b120bdfee96988203
2020-12-15 19:10:29 +00:00
KongQun Yang 3b04881a04 Update docker file for Arch Linux and OpenSUSE
Change-Id: Ia573a6ff84ff9a9789061813ef47137b086bd597
2020-12-14 15:15:05 -08:00
KongQun Yang 10daa39901 [MP4] Allow not to generate 'sidx' box for single-segment too
I.e. the flag --generate_sidx_in_media_segments,
--nogenerate_sidx_in_media_segments work for both single-segment
and multi-segment mode with this change.

Related to #862.

Change-Id: Icd27fd00e8e036ba0c4709b48650372429cc0351
2020-12-11 19:08:37 +00:00
KongQun Yang 516430bde1 [MP4] Truncate segment references in 'sidx' if necessary
The reference count in 'sidx' box is a uint16 field, which allows at
most 0xFFFF entries, i.e. at most 0xFFFF subsegments, which is roughly
18 hours for one second segments.

Do not fail packaging when it happens. Instead, generate a warning and
truncate the number of references to 0xFFFF instead.

Note that the actual number of mp4 fragments in the mp4 file can still
be more than 0xFFFF. The stream will not play to the end in DASH, but
it will play successfully in HLS.

Workarounds #862.

Change-Id: Ib3930418d1528df1f9ea64cda0d0ebaa78d26abb
2020-12-11 19:07:56 +00:00
pszemus c6b01f90ae
Fix python3 in pssh-box
The script now works with both python 2 and python 3 (tested with python 2.7 and python 3.8.6).

Fixes #833.
2020-12-11 09:43:29 -08:00
Jacob Trimble 346c844e42 Output SegmentList for some text streams.
This allows us to have presentationTimeOffset for text streams since
this needs to appear on a SegmentList/SegmentBase/SegmentTemplate.

Issue #832

Change-Id: Id3ab3308029185815d50a6cb1142e6a97f744d4f
2020-12-10 22:21:25 +00:00
Jacob Trimble c4a11aaed4 Add support for text cue heights.
Issue #832

Change-Id: Ifccbd6c6c46916d3d28ac4afaba01fc158c9c361
2020-12-10 13:26:34 -08:00
Jacob Trimble fe44b5e13c Add background image to TextSample and TTML output
Issue #832

Change-Id: I50f23223fa4362559087ada9b40488c089594450
2020-12-08 19:24:31 +00:00
Jacob Trimble 1ca89edec2 Add libpng to third-party.
This will be used by DVB-sub subtitles, which use embedded images.

Issue #832

Change-Id: I0169e2b612f4662b31eec18e7652a7d473c8f987
2020-12-08 19:23:39 +00:00
Jacob Trimble 9b036b764b Add TextSample handling to MP2T parser.
This also changes the callbacks a bit to (a) avoid passing references
for already ref-counted types, and (b) don't pass PID since the
parent knows this and gives it to the child parser.

Issue #832

Change-Id: I7dd44436c8d1ad81d42a813d16f850175b85ad1a
2020-12-08 10:30:29 -08:00
Jacob Trimble a93eeca5db Add TTML-in-MP4 output support.
This changes the default MP4 output to use TTML and adds a way to
choose which one is used.  This is done with 'format=ttml+mp4' or
'format=vtt+mp4'.

This also fixes the boxes output in WebVTT in MP4.

Change-Id: Ieaa7fc44fbf4dc020a5bb70cfa3578ec10e088ce
2020-11-20 15:10:33 -08:00
Jacob Trimble 4766654b4d Add TTML text output.
This only supports TTML output; meaning the user can convert WebVTT
into TTML, but not the other way around.  This will be useful for
DVB-sub subtitles that would be better supported within TTML.

This only adds text-based output; a follow-up will add MP4 support.

Change-Id: I0944b7df95d7765e55f203fc5e9a644f5c455dd8
2020-11-19 18:03:39 +00:00
Jacob Trimble 26334f2808 Refactor XmlNode and libxml usages.
- Use std::move to transfer ownership.
- Avoid libxml primitives outside XmlNode.
- Have attribute setting return errors.
- Mark bool returns with WARN_UNUSED_RESULTS and use return value.
- Use std::string over char*.

Change-Id: Ia00bb29c690025275e270f10e5c065722481c0c5
2020-11-16 14:35:37 -08:00
Jacob Trimble 9a34b9c3f3 Fix trick-mode property values.
Change-Id: Idc59c0002cd999e8a39862615949a339a2fe0780
2020-11-10 14:01:23 -08:00
KongQun Yang 741ae7f545 Fix docker build
Alpine does not support python3 yet, but depot_tools enabled python3
by default recently.

Disable python3 for now.

Fixes #763.

Change-Id: I57cd414702e89cafbe1b8beee810f89760129d10
2020-11-09 18:13:51 +00:00
Jacob Trimble 8e85862bda Ignore unsupported H26x streams.
This adds a new path when parsing MPEG2-TS streams to ignore unsupported
streams.  This allows extracting supported streams when some of the
streams are unsupported.  For example, you can extract audio from a
file that has unsupported video.

Change-Id: I608fcb19d0a573bfd35e9272f60b0b69346ae11a
2020-11-09 18:11:07 +00:00
Jacob Trimble 10e71680a1 Parse WebVTT regions and styles.
This adds more generic settings for regions and CSS styles.  These are
global settings, so they go on the StreamInfo object.

Change-Id: Ibb76c060206152ccf8e9a067c09877226f67c927
2020-11-09 18:08:42 +00:00
KongQun Yang 4a777b6207 [HLS] Fixes attributes for DVS tracks
Always set AUTOSELECT=YES for DVS tracks and also exclude it from
standard AUTOSELECT and DEFAULT configuration logic.

Fixes #857.

Change-Id: Ie179d02390ff09f6e8c0b54892c1d6d32a3c38d6
2020-11-05 02:56:53 +00:00
Jacob Trimble 1f21cc78cd Add style support for cue fragments.
Now text cues are composed of nested fragments that can be individually
styled.  This allows portions of the cue to be bold, etc.  The
WebVTT parser doesn't parse the inputs, but the original tags are
preserved in WebVTT output.  The WebVTT output will add tags if the
style elements are present in the cue object.

Change-Id: I6abba4175e376e4f753193f7d8cac63e958d3c89
2020-10-21 10:53:59 -07:00
Jacob Trimble f4c07b9ce0 Work around non-deterministic tests.
We currently have a bug about non-deterministic output in the MPD
generator.  This works around that bug by optionally doing everything
in a single thread.  This allows us to run manifest comparisons without
making the major changes needed to add that feature.

Issue #177

Change-Id: I10e1084dac77841220161fbd2575cdcb5c13c00e
2020-10-20 22:00:39 +00:00
Jacob Trimble 414f4589c8 Parse TextSettings from WebVTT.
Now the Cue settings are a generic object that is parsed in WebVTT.
This will allow setting the settings in different parsers without having
to use WebVTT-specifics.

Change-Id: I36689bec725bd2e515af962b7174fc5977f96fa2
2020-10-20 21:50:06 +00:00
Jacob Trimble c169c83613 Add integration test for WebVTT in MP4.
Change-Id: Ic8cd785d1fefde4004f27cfce346df2b2a8f643e
2020-10-19 10:55:35 -07:00
Jacob Trimble b2220eb0c6 Add text fragment and setting types.
This sets the groundwork for more generic text cues by having a more
generic object for the settings and the body.  This also changes the
TextSample to be immutable and accepts the fields in the constructor
instead of using setters.

Change-Id: I76b09ce8e8471a49e6bf447e8c187f867728a4bf
2020-10-19 10:39:17 -07:00
Jacob Trimble 56908a83a7 Convert text WebVTT output to muxer.
Now text-based WebVTT also uses the generic media pipeline.  This
converts the WebVttTextOutputHandler to a WebVttMuxer to be more
consistent with the other muxer types.

This also allows choosing between single-segment text and multi-segment.
Before, we would generate both and use single-segment for DASH and
multi-segment for HLS; but now you can choose between either and either
are supported in both DASH and HLS.

Change-Id: I6f7edda09e01b5f40e819290d3fe6e88677018d9
2020-10-19 17:24:38 +00:00
Jacob Trimble 2909ca0c77 Handle segmented text same as audio/video.
Now the same pipeline for handling the audio/videos streams will handle
the segmented text streams too.  This doesn't apply to the text output,
only to the MP4 variants.  This also fixes a bug where we added the
X-TIMESTAMP-MAP tag even when there wasn't TS streams; this doesn't
otherwise change the behavior around that tag.

Change-Id: I03f7cea56efa42e96311c00841330629a14aa053
2020-10-07 17:49:03 +00:00
Jacob Trimble 9babfb883b Fix segmented WebVTT test.
The test added in the previous CL was broken due to a rebase on another
change.  This subtly changed some of the byte offsets that broke the
test.  This wasn't caught since I didn't rebase and re-run the tests
before merging.

Change-Id: Id7e4c7688278eae37da1a14f1648263b4dda98cd
2020-09-16 13:24:16 -07:00
Jacob Trimble e3bc85f12d Make WebVttParser a MediaParser.
This changes it from an OriginHandler to a MediaParser and moves the
handling of it to the Demuxer.  This will allow more generic handling
of text by giving it the same abstractions as video/audio handling.

Change-Id: Ibbde3c84d228ec8e83af1ed266ea97dbc9589c24
2020-09-16 20:12:48 +00:00
Jacob Trimble ba33a63693 Add TextSample to MediaParser.
In addition to the MediaSample handling of the MediaParser, this now
adds callbacks for TextSample.  This allows reading text streams from
the media files.

Change-Id: I6c00e286e98bc9aafe05b99cf2f7ce6f89d167a9
2020-09-16 20:09:28 +00:00
Jacob Trimble 748e7e0056 Make the text readers use streams.
Instead of having the text readers reading from the file directly, they
now accept the data as a stream.

Change-Id: Id1b32c867a8058a68ae7aab5c568f77672a4401d
2020-09-16 20:06:51 +00:00
koln67 5b9fd409a5
[MP4] Change major brand from isom to mp41
This fixes warnings that 'isom' can only be a compatible instead of major brand.

Issue #755.
2020-09-08 15:45:21 -07:00
koln67 562040e000
[TS] Calculate frame rate using the second sample
Related to #751.
2020-09-02 14:27:02 -07:00
Jacob Trimble 4c0d38bde3 Don't use Chromium sysroot.
The Chromium sysroot has a broken C++11 implementation that doesn't
allow us to use <condition_variable> or <chrono>.  So we need to use the
system sysroot to use all C++11 features.

Change-Id: I41bec7ebddf598e1bf0494ee8bb2844e67bc8acf
2020-09-01 17:59:52 +00:00