Rewrite test_data_util.cc to locate files relative to the source file
itself, rather than using a service from chromium `base::`.
Issue #1047 (CMake porting)
Issue #346 (absl porting)
The hack of using "secrets" to store per-repo settings was not working.
The main reason is that pull_request workflows don't have access to
secrets no matter what you do. So it was impossible to make this work
for settings like "ENABLE_SELF_HOSTED" for PR tests.
This change replaces that old hack with a new one. Now a repo owner must
create a "GitHub Environment" with the name of the setting they want to
enable. Currently supported values are "self_hosted", to add self-hosted
runners to the build/test matrix, and "debug", to start an SSH server
for debugging when a workflow fails.
Issue #1047 (CMake porting)
For some reason, the status util unittest was in media/base/ instead of
status/
The broken headers in the status library were not obviously broken until
media/base/ porting started, and they were used for the first time.
Issue #1047 (CMake porting)
If another instance of the PR workflow is started for the same PR,
cancel the old one. If a PR is updated and a new test run is started,
the old test run will be cancelled automatically to conserve
resources.
Build in parallel with however many cores are available on the system.
This might not affect build times in every environment (for example, if
GitHub Actions VMs are single core).
From the initial test run of this PR, we're seeing roughly 1x build
speed on Linux, 3x on macOS, and 1.5x on Windows, compared to the same
build step on a contemporaneous PR.
The docs build in GitHub Actions broke with the release of jinja 3.1,
which introduced a breaking change that broke the cloud_sptheme sphinx
extension in use by our docs. That extension is not maintained any more,
so there will be no upstream fix. See also
https://foss.heptapod.net/doc-utils/cloud_sptheme/-/issues/47
That extension adds support for table styling (see
https://cloud-sptheme.readthedocs.io/en/latest/lib/cloud_sptheme.ext.table_styling.html),
but we only appear to have one table in the docs currently
(docs/source/options/segment_template_formatting.rst) and it does not
use the options added by the cloud_sptheme extension.
Therefore the best fix for GitHub Actions and any other system running
jinja 3.1 is to remove that extension from our config.
Note:
* An xHE-AAC capable encoder will auto adjust the user-specified SAP/RAP
value to the allowed grid where SAP/RAPs can occur.
e.g.: `-rapInterval 5000` (5s) may result in actual SAPs/RAPs every
4.984s.
* To ensure SAP/RAP starts a new segment, Shaka needs to executed with a
"--segment_duration" is less than or equal to that adjusted value.
* If every SAP/RAP should trigger a new segment, just set the segment
length to a very low value e.g.: `--segment_duration 0.1`
Note:
* An xHE-AAC capable encoder will auto adjust the user-specified SAP/RAP
value to the allowed grid where SAP/RAPs can occur.
e.g.: `-rapInterval 5000` (5s) may result in actual SAPs/RAPs every
4.984s.
* To ensure SAP/RAP starts a new segment, Shaka needs to executed with a
"--segment_duration" is less than or equal to that adjusted value.
* If every SAP/RAP should trigger a new segment, just set the segment
length to a very low value e.g.: `--segment_duration 0.1`
This also normalizes the structure of all submodule folders so that
there is a clear place to put configurations, outside the submodule
source, but limited in scope to that folder.
Issue #1047 (New build system)
This removes apple_apsl and libevent (which are no longer used) and
libxml and protobuf (which will be replaced in a follow-up with
submodules).
This change only contains deletions, to make it easier to review
separately from later additions for submodules and cmake integration.
Issue #1047 (New build system)
There are a lot of changes in this first phase, because there was a
lot of infrastructure required to get some meaningful amount of
porting done. Future PRs should be simpler.
<b>Summary of changes:</b><details>
- Remove old deps:
- boringssl (replaced with mbedtls, lighter, easier to build)
- gflags (replaced with absl::flags)
- Chromium build tools
- New deps to replace parts of Chromium base:
- abseil-cpp
- glog
- nlohmann::json (for tests only)
- Submodules, updates, and CMake build rules for third-party
libraries:
- curl
- gmock/gtest
- Ported internal libraries and their tests by removing Chromium deps
and adding CMake build rules:
- file (now using C++17 filesystem APIs)
- license_notice
- status
- version
- Test improvements
- Removed file tests that can never be re-enabled
- Re-enabled all other disabled file tests
- Debug JSON values when HTTP tests fail
- Fixed chunked-encoding issues in HTTP tests
- Updated and refactored Dockerfiles testing
- All docker files working, with OS versions updated to meet the
new tool requirements
- Local docker builds no longer write files to your working
directory as root
- Local docker builds can now be run in parallel without clobbering
each others' build outputs
- DEBUG=1 can drop you into an interactive shell when a docker
build fails
- Updated and heavily refactored workflows and Dockerfiles
- All docker files now tested in parallel on GitHub, speeding up CI
- All common workflow components broken out and using workflow_call
instead of custom actions
- Self-hosted runners now optional, to make testing easier on forks
- CMake porting works-in-process can now be fully tested on GitHub
- Building ported libraries and passing ported tests on all three
platforms!
- CI hacks for macOS removed, now testing on macos-latest!
- Python2 no longer required! (Only Python3)
- Using strict build flags, treating all warnings as errors.
</details>
<b>Required to build:</b>
- CMake >= 3.16
- Python 3
- A compiler supporting C++ >= 17
- g++ >= 9 if using GCC (Clang also fine)
- MSVC for Windows
<b>Still needs work:</b><details>
- Moving other dependencies into submodules (if we keep them):
- apple_apsl
- icu
- libevent
- libpng
- libwebm
- libxml
- modp_b64
- protobuf
- zlib
- Port remaining internal libraries:
- app
- hls
- media/base
- media/chunking
- media/codecs
- media/crypto
- media/demuxer
- media/event
- media/formats/dvb
- media/formats/mp2t
- media/formats/mp4
- media/formats/packed_audio
- media/formats/ttml
- media/formats/webm
- media/formats/webvtt
- media/formats/wvm
- media/origin
- media/public
- media/replicator
- media/trick_play
- mpd
- Port main application
- Add logging flags in absl and connect them to glog (which expects
gflags)
- Port pssh-box.py
- Port main test targets (packager_test.py and packager_app.py)
- Updating all requirement and build documentation
- Remove any remaining refs to gclient, depot_tools, ninja
- Update and complete release workflows using release-please
</details>
Issue #346 (Switch to abseil)
Issue #1047 (New build system)
Using the latest depot_tools no longer works. depot_tools also wants
to auto-update itself, which must now be disabled.
We also need to disable the copy of python (vpython) included in
depot_tools, since for some distros, it has dependencies on system
libraries that no longer exist.
Finally, we need to force some distros to use python 2, because our
build system is ancient and needs to be ripped out and replaced some
day soon.
This fixes build issues in our CI, our Dockerfiles, and in general on
certain platforms or distros.
Closes#1023
This is an automated sync of common workflows for this organization.
The upstream source is:
b39597e92d
Co-authored-by: Shaka Bot <shaka-bot@users.noreply.github.com>
This is an automated sync of common workflows for this organization.
The upstream source is:
9517df8f73
Co-authored-by: Shaka Bot <shaka-bot@users.noreply.github.com>
This is an automated sync of common workflows for this organization.
The upstream source is:
aa5e38eb86
Co-authored-by: Shaka Bot <shaka-bot@users.noreply.github.com>
The official, static-linked linux builds were crashing in their use of
getaddrinfo, which libcurl was configured to use. Both getaddrinfo
and all of its alternatives available in glibc fail with static
linking.
We can fix this by configuring libcurl to use libc-ares on Linux
instead. This allows us to keep the benefits of a statically-linked
Linux binary.
Closes#996
Change-Id: Ib4a9eb939813fd165727788726459ef4adf3fc4d
The script in packager/testing/dockers/test_dockers.sh now outputs
more useful info for debugging, uses unique container names per OS so
that the containers can be debugged, and allows filtering to re-run
specific OSes if a build fails.
Change-Id: I0cace282549c093a643009f5e60e7545a039168c
This updates the main Dockerfile and all the docker-based
distro-specific tests. The base OS versions have been updated to
versions that have not reached end-of-life status yet, and the list of
dependencies required has been updated and pruned.
Change-Id: Ibcff2f60e739fd5d999af100af76c40aa91a75bc
In many places, we used std::numeric_limits without including the
proper header. This would build on some Linux distributions, but not
others.
This adds the missing includes, fixing the build on Fedora, among
other distros.
Change-Id: I63e9e37e5973fe23bbdf9868552db51062b1dae4
In one of the low-latency changes, a change was made to HttpFile that
caused responses to HTTP POST requests to go missing. This resulted
in failures to fetch encryption keys.
The breaking change was recommended by me in a PR review, and was not
caught by any unit tests. New tests would be ideal, but I chose to
fix the bug first, rather than leave the repo broken.
This bug was brought to my attention in google/shaka-streamer#87 and
has not appeared in any release versions.
Change-Id: I9eca73d187a8a30f16c4a920fcdb7b4872253858
## The issue
- With LL-DASH mode enabled, the gap size warning was hit and printed to the console every time a new segment was registered to the manifest.
- This occurred because the first chunk's size and duration were being stored for each segment, rather than the full segment size and duration. Note, only the first chunk's metrics are known at first because in low latency mode, the segment is registered to the manifest before it is finished being processed and written.
- Because of this, the gap size check was comparing the end time of the first chunk in the previous segment to the beginning time of the current segment, causing the check to fail every time.
## The Fix
- Update a low latency segment's duration and size once the segment file has been fully written.
- The full segment size and duration will be used to update the bandwidth estimator and the segment info list.
- Updating the segment info list to hold the full duration is necessary for satisfying [the gap size check found in Represenation.cc](https://github.com/google/shaka-packager/blob/master/packager/mpd/base/representation.cc#L391).
- NOTE: bandwidth estimation is currently only used in HLS
It was suggested in code review for another project that we update the
runner labels for clarity. This brings Packager in line with that, so
that we are using the same labels across projects.
The runners have already been updated to register with the new label.
Change-Id: I30b22530225b5bd22b965ba98d276bcd74ade6cf
Now that we have multiple architectures, we should factor both OS and
architecture into the names of release binaries. This makes the names
more formulaic, as well as consistent with the static-ffmpeg-binaries
repository. Shaka Streamer will pull binaries from both this repo and
that one, so consistent names would be helpful.
The pssh-box release is actually OS and architecture independent, so
remove the suffix from that and only release one copy of it.
Change-Id: Ief3de49fae267c5267647a8dd4377023777ead37
The generate_version_string script was only producing correct results
in python 2, not python 3. The gyp file that references it explicitly
runs it in python3. The shebang line of the script has been updated
to match. The script itself has been updated such that it now works
correctly in both python2 and python3.
Scripts that are only used as modules (not executed directly) have had
their shebang lines removed.
This fixes CI failures on GitHub Actions.
Change-Id: I309bafd2fb05e8fb33f5e092ead179c8c42ea5d3
# LL-DASH Support
These changes add support for LL-DASH streaming.
**NOTE:** LL-HLS support is still in progress, but it's coming. :)
## Testing
`./chunking_unittest --gtest_filter="ChunkingHandlerTest.LowLatencyDash"`
`./media_event_unittest --gtest_filter="MpdNotifyMuxerListenerTest.LowLatencyDash"`
`./mpd_unittest --gtest_filter="PeriodTest.LowLatencyDashMpdGetXml"`
`./mpd_unittest --gtest_filter="SimpleMpdNotifierTest.NotifyAvailabilityTimeOffset"`
`./mpd_unittest --gtest_filter="SimpleMpdNotifierTest.NotifySegmentDuration"`
`./mpd_unittest --gtest_filter="LowLatencySegmentTest.LowLatencySegmentTemplate"`
Note, packager_test must be run from the main project directory
`./out/Release/packager_test --gtest_filter="PackagerTest.LowLatencyDashEnabledAndUtcTimingNotSet"`
`./out/Release/packager_test --gtest_filter="PackagerTest.LowLatencyDashEnabledAndUtcTimingNotSet"`
The newest pylint release complained about several issues that the
older release did not. This resolves those issues:
- removes unneeded "u" prefix from strings
- adds "encoding" parameter for all open() calls
- because "encoding" is a python3-only parameter, use python3 in all
the scripts that we control
Unfortunately, python2 is required for any scripts that import modules
from the ancient Chromium build system we're using (referenced by
DEPS), as well as kokoro scripts.
Change-Id: I2e9f97af508efe58b5a71de21740e59b1528affd
This was causing failures on arm64, where the build action had an
arm-specific clause that was skipped due to the missing parameter.
Change-Id: I71b7fb15120855c444749dc2216b5f19f0561f6e
When using CC=clang CXX=clang++, there is a binutils version check
that does not work correctly in common.gypi. Since we are stuck with
a very old version of chromium/src/build, there is nothing to do but
patch it to remove the check. Thankfully, this version number does
not control anything critical in the build settings as far as we can
tell.
Change-Id: Id749d97c5898917592f66136538ee0fa5ca78767
We never produced static release executables on Linux before, but the dynamic libraries they depended on were universal enough that nobody noticed. Now that we have released v2.5 and switched to GitHub Actions for CI builds, the Linux executables depend on libatomic, which is causing issues for some users.
Although we can't create fully-static executables on macOS or Windows, we can at least do so on Linux.
This adds a GYP variable static_link_binaries which can be set to request full-static binaries on Linux. This also exposes the Chromium build variable disable_fatal_linker_warnings, which is necessary when static linking on Linux due to static-link-related warnings generated by libcurl for its use of getaddrinfo. Finally, this enforces the definition of __UCLIBC__ with static linking on Linux, which is the only way to disable malloc hooks in Chromium base. Those hooks cause linker failures when linking statically on Linux.
A new check has been added to the release workflow to ensure that the builds we create are statically linked on Linux.
Closes#965
There is not a good reason to use a long-lived token attached to
shaka-bot. Instead, use a short-lived, automatic token generated by
GitHub Actions for the workflow run.
This converts all time parameters to signed, finishing a cleanup that
was started in 2018 in b4256bf0. This changes the type of:
- timestamps
- PTS specifically
- timestamp offsets
- timescales
- durations
This excludes:
- MP4 box definitions
- DTS specifically
This is meant to address signed/unsigned conversion issues on arm64
that caused some test cases to fail.
Change-Id: Ic752a20cbc6e31fea6bc0894d1771833171e7cbe
Joey Parrish
Geoff Jukes
Marcus Wichelmann
Vishal Shah
Shaka Bot
Joey Parrish
Caitlin O'Callaghan