The docs build in GitHub Actions broke with the release of jinja 3.1,
which introduced a breaking change that broke the cloud_sptheme sphinx
extension in use by our docs. That extension is not maintained any more,
so there will be no upstream fix. See also
https://foss.heptapod.net/doc-utils/cloud_sptheme/-/issues/47
That extension adds support for table styling (see
https://cloud-sptheme.readthedocs.io/en/latest/lib/cloud_sptheme.ext.table_styling.html),
but we only appear to have one table in the docs currently
(docs/source/options/segment_template_formatting.rst) and it does not
use the options added by the cloud_sptheme extension.
Therefore the best fix for GitHub Actions and any other system running
jinja 3.1 is to remove that extension from our config.
There are a lot of changes in this first phase, because there was a
lot of infrastructure required to get some meaningful amount of
porting done. Future PRs should be simpler.
<b>Summary of changes:</b><details>
- Remove old deps:
- boringssl (replaced with mbedtls, lighter, easier to build)
- gflags (replaced with absl::flags)
- Chromium build tools
- New deps to replace parts of Chromium base:
- abseil-cpp
- glog
- nlohmann::json (for tests only)
- Submodules, updates, and CMake build rules for third-party
libraries:
- curl
- gmock/gtest
- Ported internal libraries and their tests by removing Chromium deps
and adding CMake build rules:
- file (now using C++17 filesystem APIs)
- license_notice
- status
- version
- Test improvements
- Removed file tests that can never be re-enabled
- Re-enabled all other disabled file tests
- Debug JSON values when HTTP tests fail
- Fixed chunked-encoding issues in HTTP tests
- Updated and refactored Dockerfiles testing
- All docker files working, with OS versions updated to meet the
new tool requirements
- Local docker builds no longer write files to your working
directory as root
- Local docker builds can now be run in parallel without clobbering
each others' build outputs
- DEBUG=1 can drop you into an interactive shell when a docker
build fails
- Updated and heavily refactored workflows and Dockerfiles
- All docker files now tested in parallel on GitHub, speeding up CI
- All common workflow components broken out and using workflow_call
instead of custom actions
- Self-hosted runners now optional, to make testing easier on forks
- CMake porting works-in-process can now be fully tested on GitHub
- Building ported libraries and passing ported tests on all three
platforms!
- CI hacks for macOS removed, now testing on macos-latest!
- Python2 no longer required! (Only Python3)
- Using strict build flags, treating all warnings as errors.
</details>
<b>Required to build:</b>
- CMake >= 3.16
- Python 3
- A compiler supporting C++ >= 17
- g++ >= 9 if using GCC (Clang also fine)
- MSVC for Windows
<b>Still needs work:</b><details>
- Moving other dependencies into submodules (if we keep them):
- apple_apsl
- icu
- libevent
- libpng
- libwebm
- libxml
- modp_b64
- protobuf
- zlib
- Port remaining internal libraries:
- app
- hls
- media/base
- media/chunking
- media/codecs
- media/crypto
- media/demuxer
- media/event
- media/formats/dvb
- media/formats/mp2t
- media/formats/mp4
- media/formats/packed_audio
- media/formats/ttml
- media/formats/webm
- media/formats/webvtt
- media/formats/wvm
- media/origin
- media/public
- media/replicator
- media/trick_play
- mpd
- Port main application
- Add logging flags in absl and connect them to glog (which expects
gflags)
- Port pssh-box.py
- Port main test targets (packager_test.py and packager_app.py)
- Updating all requirement and build documentation
- Remove any remaining refs to gclient, depot_tools, ninja
- Update and complete release workflows using release-please
</details>
Issue #346 (Switch to abseil)
Issue #1047 (New build system)
Using the latest depot_tools no longer works. depot_tools also wants
to auto-update itself, which must now be disabled.
We also need to disable the copy of python (vpython) included in
depot_tools, since for some distros, it has dependencies on system
libraries that no longer exist.
Finally, we need to force some distros to use python 2, because our
build system is ancient and needs to be ripped out and replaced some
day soon.
This fixes build issues in our CI, our Dockerfiles, and in general on
certain platforms or distros.
Closes#1023
The official, static-linked linux builds were crashing in their use of
getaddrinfo, which libcurl was configured to use. Both getaddrinfo
and all of its alternatives available in glibc fail with static
linking.
We can fix this by configuring libcurl to use libc-ares on Linux
instead. This allows us to keep the benefits of a statically-linked
Linux binary.
Closes#996
Change-Id: Ib4a9eb939813fd165727788726459ef4adf3fc4d
This updates the main Dockerfile and all the docker-based
distro-specific tests. The base OS versions have been updated to
versions that have not reached end-of-life status yet, and the list of
dependencies required has been updated and pruned.
Change-Id: Ibcff2f60e739fd5d999af100af76c40aa91a75bc
# LL-DASH Support
These changes add support for LL-DASH streaming.
**NOTE:** LL-HLS support is still in progress, but it's coming. :)
## Testing
`./chunking_unittest --gtest_filter="ChunkingHandlerTest.LowLatencyDash"`
`./media_event_unittest --gtest_filter="MpdNotifyMuxerListenerTest.LowLatencyDash"`
`./mpd_unittest --gtest_filter="PeriodTest.LowLatencyDashMpdGetXml"`
`./mpd_unittest --gtest_filter="SimpleMpdNotifierTest.NotifyAvailabilityTimeOffset"`
`./mpd_unittest --gtest_filter="SimpleMpdNotifierTest.NotifySegmentDuration"`
`./mpd_unittest --gtest_filter="LowLatencySegmentTest.LowLatencySegmentTemplate"`
Note, packager_test must be run from the main project directory
`./out/Release/packager_test --gtest_filter="PackagerTest.LowLatencyDashEnabledAndUtcTimingNotSet"`
`./out/Release/packager_test --gtest_filter="PackagerTest.LowLatencyDashEnabledAndUtcTimingNotSet"`
The newest pylint release complained about several issues that the
older release did not. This resolves those issues:
- removes unneeded "u" prefix from strings
- adds "encoding" parameter for all open() calls
- because "encoding" is a python3-only parameter, use python3 in all
the scripts that we control
Unfortunately, python2 is required for any scripts that import modules
from the ancient Chromium build system we're using (referenced by
DEPS), as well as kokoro scripts.
Change-Id: I2e9f97af508efe58b5a71de21740e59b1528affd
This brings our default build config more in line with what is
necessary for some platforms anyway: using the system-installed
toolchain and sysroot to build everything.
We will no longer fetch source or binaries for any specific build
tools, such as libc++, clang, gold, binutils, or valgrind.
The main part of this change is the changing of default gyp settings
in gyp_packager.py. For this, a bug in gyp_packager.py had to be
fixed, in which similar GYP_DEFINE key names (such as clang and
host_clang) would conflict, causing some defaults not to be installed
properly.
In order to enable clang=0 by default, some changes had to be made in
common.gypi:
- compiler macros added to fix a compatibility issue between
Chromium's base/mac/ folder and the actual OSX SDK
- replaced clang_warning_flags variables with standard cflags
settings, plus xcode_settings for OSX
- turned off warnings-as-errors for non-shaka code, rather than
allow-listing specific warning types, since we can't actually fix
those warnings on any platform
- disabled two specific warnings in shaka code, both of which are
caused by headers from our non-shaka dependencies
Also, one warning (missing "override" keyword) has been fixed in
vod_media_info_dump_muxer_listener.h.
Although these changes were done to make building simpler on a wider
array of platforms (arm64, for example), it seems to make the build a
bit faster, too. For me, at least, on my main Linux workstation:
- "gclient sync" now runs 20-30% faster
- "ninja -C out/Release" now runs 5-13% faster
The following environment variables are no longer required:
- DEPOT_TOOLS_WIN_TOOLCHAIN
- MACOSX_DEPLOYMENT_TARGET
Documentation, Dockerfiles, and GitHub Actions workflows have been
updated to reflect this.
The following GYP_DEFINES are no longer required for anyone:
- clang=0
- host_clang=0
- clang_xcode=1
- use_allocator=none
- use_experimental_allocator_shim=0
Documentation, Dockerfiles, and GitHub Actions workflows have been
updated to reflect this.
The following repos are no longer dependencies in gclient:
- binutils
- clang
- gold
- libc++
- libc++abi
- valgrind
The following gclient hooks have been removed:
- clang
- mac_toolchain
- sysroot
Change-Id: Ie94ccbeec722ab73c291cb7df897d20761a09a70
This replaces Travis (for Linux & Mac) and Appveyor (for Windows) with
GitHub Actions. In addition to using GitHub Actions to test PRs, this
also expands the automation of releases so that the only manual steps
are:
1. Create a new CHANGELOG.md entry
2. Create a release tag
Workflows have been create for building and testing PRs and releases,
for publishing releases to GitHub, NPM, and Docker Hub, and for
updating documentation on GitHub Pages.
When a new PR is created, GitHub Actions will:
- Build and test on all combinations of OS, release type, and library
type
Appveyor's workflow took ~2 hours, whereas the new GitHub Actions
workflow takes ~30 minutes.
When a new release tag is created, GitHub Actions will:
- Create a draft release on GitHub
- Extract release notes from CHANGELOG.md & attach them to the
draft release
- Build and test on all combinations of OS, release type, and library
type, aborting if any build or test fails
- Attach release artifacts to the draft release, aborting if any
one artifact can't be prepared
- Fully publish the draft release on GitHub
- Publish the same release to NPM (triggered by GitHub release)
- Publish the same release to Docker Hub (triggered by GitHub release)
- Update the docs on GitHub pages
Closes#336 (GitHub Actions workflow to replace Travis and Appveyor)
b/190743862 (internal; tracking replacement of Travis)
Change-Id: Ic53eef60a8587c5d1487769a0cefaa16eb9b46e7
At this point static_library builds are working in MSVS 2019. shared_library builds are still not working.
Closes#867 (MSVS 2019)
Issue #318 (progress toward shared_library support on Windows)
Issue #336 (progress toward replacing Travis & Appveyor with GitHub Actions, which uses MSVS 2019)
b/190743862 (internal; tracking replacement of Travis)
For windows running `GYP_DEFINES='target_arch=X64'` shows an error GYP_DEFINES is
not recognized as an internal or external command.
The actual command is to set the value of GYP_DEFINES. so it should be
`SET GYP_DEFINES='target_arch=X64'`.
This also allows setting the language of different text streams from
the same input. Multiple streams can use the same input stream
using different cc_index values and can each use a different language.
This also will try to pull the language from the input if not
specified.
Change-Id: I7078710b509b7d77dad8cb4299a82f954af7e9e7
Issue #149
Co-authored-by: Andreas Motl <andreas.motl@elmyra.de>
Co-authored-by: Rintaro Kuroiwa <rkuroiwa@google.com>
Co-authored-by: Ole Andre Birkedal <o.birkedal@sportradar.com>
I.e. the flag --generate_sidx_in_media_segments,
--nogenerate_sidx_in_media_segments work for both single-segment
and multi-segment mode with this change.
Related to #862.
Change-Id: Icd27fd00e8e036ba0c4709b48650372429cc0351
Added --crypt_byte_block, --skip_byte_block to allow users to specify
protection pattern other than 1:9 for pattern based encryption scheme,
e.g. cbcs and cens.
Closes#710.
b/147307451
Change-Id: I9f64a7639170c737f138572689b28d17286325c7
HLS uses the EXT-X-MEDIA-SEQUENCE tag at the start of a live playlist in
order to specify the first segment sequence number. This is because any
live playlist have a limited number of segments, and they also keep
updating with new segments while removing old ones. When a player refreshes
the playlist, this information is important for keeping track of segments
positions.
When the packager starts, it naturally starts this count from zero. However,
there are many situations where the packager may be restarted, without this
meaning starting this value from zero (but continuing a previous sequence).
The most common situations are problems in the encoder feeding the packager.
With those cases in mind, this parameter allows to set the initial
EXT-X-MEDIA-SEQUENCE value. This way, it's possible to continue the sequence
number from previous packager run.
Closes#691.
Add dash_accessibilities stream descriptor, which is a semi-colon
separated list of accessibility_scheme_id_uri=value. It is optional.
Add dash_roles stream descriptor, which is a semi-colon separated
list of strings. It is optional.
Closes#565.
Change-Id: Idb1c20bb410fdd016db07e11fe507c102a3dd8ea
VLC seems to generate access units with extra AUDs. In #526, the below
sequence is seen:
AUD | SPS | PPS | SPS | PPS | AUD | SEI | SEI | SEI | IDR_SLICE
Previously we exit early when seeing AUD, which results in delayed
processing of the access unit.
The behavior is changed to continue processing the following NAL units
to workaround the content issue.
Closes#526.
Change-Id: I80f571c0711c6db1337eb393fce36fae5432b6c5
It allows users to override the default language for text tracks.
If not specified, --default_language applies to both audio and text
tracks.
Issue #430.
Change-Id: I86a9baba2072be27b6661fa7b65a8bc8b6adb3cc
Add hls_characteristics stream descriptor, which is a colon or semi-colon
separated list of strings. It is optional.
Fixes#430.
Change-Id: Ifcf79316e68768ff065891933de565cd0ff32ec4
Only Marlin Adaptive Streaming Specification – Simple Profile is
supported.
Two additional updates:
- Remove FairPlay ContentProtection element from DASH mpd as FairPlay
does not define a signaling in DASH.
- Updated end to end test to include all DRMs we support.
Closes#381.
Change-Id: Id12269b471ea34983b782cbd92f687332292ef59
Keep bundled binutil scripts but not downloading actual binaries by default.
Automatic downloading of binutils has been causing problems for some users:
see #164, #412, #440.
Using bundled binutils helps reduce linking time, but packager codebase is
relatively small, so the gain is not significant.
Local testing shows that the full build time increases from 1m42s to 1m50s, i.e.
an increase of about 8s, or ~8%.
User can still enable the usage of bundled binutils by running
'python src/packager/third_party/binutils/download.py' and set
'linux_use_bundled_binutils' and 'linux_use_bundled_gold' to 1 in GYP_DEFINES.
Change-Id: I7ecae2333861f41054f957d0a6081dae948167bd
Note that STYLE and REGION are not supported in mp4 container due to
spec limitation as 14496-30:2014 does not specify a way to signal
styles/regions inside mp4.
Closes#344.
Change-Id: I05c14df916f7b2c7ca4364ee9407e0eda4dc7a3f
- Also fixed compilations in Alpine Linux and other flavors of Linux.
- Added container versions in docker files to always use a verified
version.
Closes#164.
Change-Id: I949a8709e4d70c49129c9c2e8608dd78193d964c
Joey Parrish
Vishal Shah
Joey Parrish
Caitlin O'Callaghan
sr90
KongQun Yang
Suraj kumar (sk)
Jacob Trimble
Alen Vrečko
Ole Andre Birkedal
Joe Foraci
Widevine Buildbot
Daniel Cantarín
Tomohiro IKEDA
Herlon Aguiar