This also normalizes the structure of all submodule folders so that
there is a clear place to put configurations, outside the submodule
source, but limited in scope to that folder.
Issue #1047 (New build system)
This removes apple_apsl and libevent (which are no longer used) and
libxml and protobuf (which will be replaced in a follow-up with
submodules).
This change only contains deletions, to make it easier to review
separately from later additions for submodules and cmake integration.
Issue #1047 (New build system)
There are a lot of changes in this first phase, because there was a
lot of infrastructure required to get some meaningful amount of
porting done. Future PRs should be simpler.
<b>Summary of changes:</b><details>
- Remove old deps:
- boringssl (replaced with mbedtls, lighter, easier to build)
- gflags (replaced with absl::flags)
- Chromium build tools
- New deps to replace parts of Chromium base:
- abseil-cpp
- glog
- nlohmann::json (for tests only)
- Submodules, updates, and CMake build rules for third-party
libraries:
- curl
- gmock/gtest
- Ported internal libraries and their tests by removing Chromium deps
and adding CMake build rules:
- file (now using C++17 filesystem APIs)
- license_notice
- status
- version
- Test improvements
- Removed file tests that can never be re-enabled
- Re-enabled all other disabled file tests
- Debug JSON values when HTTP tests fail
- Fixed chunked-encoding issues in HTTP tests
- Updated and refactored Dockerfiles testing
- All docker files working, with OS versions updated to meet the
new tool requirements
- Local docker builds no longer write files to your working
directory as root
- Local docker builds can now be run in parallel without clobbering
each others' build outputs
- DEBUG=1 can drop you into an interactive shell when a docker
build fails
- Updated and heavily refactored workflows and Dockerfiles
- All docker files now tested in parallel on GitHub, speeding up CI
- All common workflow components broken out and using workflow_call
instead of custom actions
- Self-hosted runners now optional, to make testing easier on forks
- CMake porting works-in-process can now be fully tested on GitHub
- Building ported libraries and passing ported tests on all three
platforms!
- CI hacks for macOS removed, now testing on macos-latest!
- Python2 no longer required! (Only Python3)
- Using strict build flags, treating all warnings as errors.
</details>
<b>Required to build:</b>
- CMake >= 3.16
- Python 3
- A compiler supporting C++ >= 17
- g++ >= 9 if using GCC (Clang also fine)
- MSVC for Windows
<b>Still needs work:</b><details>
- Moving other dependencies into submodules (if we keep them):
- apple_apsl
- icu
- libevent
- libpng
- libwebm
- libxml
- modp_b64
- protobuf
- zlib
- Port remaining internal libraries:
- app
- hls
- media/base
- media/chunking
- media/codecs
- media/crypto
- media/demuxer
- media/event
- media/formats/dvb
- media/formats/mp2t
- media/formats/mp4
- media/formats/packed_audio
- media/formats/ttml
- media/formats/webm
- media/formats/webvtt
- media/formats/wvm
- media/origin
- media/public
- media/replicator
- media/trick_play
- mpd
- Port main application
- Add logging flags in absl and connect them to glog (which expects
gflags)
- Port pssh-box.py
- Port main test targets (packager_test.py and packager_app.py)
- Updating all requirement and build documentation
- Remove any remaining refs to gclient, depot_tools, ninja
- Update and complete release workflows using release-please
</details>
Issue #346 (Switch to abseil)
Issue #1047 (New build system)
The official, static-linked linux builds were crashing in their use of
getaddrinfo, which libcurl was configured to use. Both getaddrinfo
and all of its alternatives available in glibc fail with static
linking.
We can fix this by configuring libcurl to use libc-ares on Linux
instead. This allows us to keep the benefits of a statically-linked
Linux binary.
Closes#996
Change-Id: Ib4a9eb939813fd165727788726459ef4adf3fc4d
The newest pylint release complained about several issues that the
older release did not. This resolves those issues:
- removes unneeded "u" prefix from strings
- adds "encoding" parameter for all open() calls
- because "encoding" is a python3-only parameter, use python3 in all
the scripts that we control
Unfortunately, python2 is required for any scripts that import modules
from the ancient Chromium build system we're using (referenced by
DEPS), as well as kokoro scripts.
Change-Id: I2e9f97af508efe58b5a71de21740e59b1528affd
This fixes the Debug build of libpng on arm64 by avoiding CPU-specific
optimizations that are not in our sources list. The Release build
appears to have been unaffected, possibly due to link-time
optimizations or dead code stripping.
Change-Id: I900e00fe30b9f3748f2587cfea89a636b3a19811
This brings our default build config more in line with what is
necessary for some platforms anyway: using the system-installed
toolchain and sysroot to build everything.
We will no longer fetch source or binaries for any specific build
tools, such as libc++, clang, gold, binutils, or valgrind.
The main part of this change is the changing of default gyp settings
in gyp_packager.py. For this, a bug in gyp_packager.py had to be
fixed, in which similar GYP_DEFINE key names (such as clang and
host_clang) would conflict, causing some defaults not to be installed
properly.
In order to enable clang=0 by default, some changes had to be made in
common.gypi:
- compiler macros added to fix a compatibility issue between
Chromium's base/mac/ folder and the actual OSX SDK
- replaced clang_warning_flags variables with standard cflags
settings, plus xcode_settings for OSX
- turned off warnings-as-errors for non-shaka code, rather than
allow-listing specific warning types, since we can't actually fix
those warnings on any platform
- disabled two specific warnings in shaka code, both of which are
caused by headers from our non-shaka dependencies
Also, one warning (missing "override" keyword) has been fixed in
vod_media_info_dump_muxer_listener.h.
Although these changes were done to make building simpler on a wider
array of platforms (arm64, for example), it seems to make the build a
bit faster, too. For me, at least, on my main Linux workstation:
- "gclient sync" now runs 20-30% faster
- "ninja -C out/Release" now runs 5-13% faster
The following environment variables are no longer required:
- DEPOT_TOOLS_WIN_TOOLCHAIN
- MACOSX_DEPLOYMENT_TARGET
Documentation, Dockerfiles, and GitHub Actions workflows have been
updated to reflect this.
The following GYP_DEFINES are no longer required for anyone:
- clang=0
- host_clang=0
- clang_xcode=1
- use_allocator=none
- use_experimental_allocator_shim=0
Documentation, Dockerfiles, and GitHub Actions workflows have been
updated to reflect this.
The following repos are no longer dependencies in gclient:
- binutils
- clang
- gold
- libc++
- libc++abi
- valgrind
The following gclient hooks have been removed:
- clang
- mac_toolchain
- sysroot
Change-Id: Ie94ccbeec722ab73c291cb7df897d20761a09a70
Also roll_boringssl.py script is also adjusted to work with latest
boringssl.
Disable ASM compilation on Windows x64 which fails to compile. It also
means there are no HW AES on Windows x64.
Issue #198.
Change-Id: Ib7e8ff506f014c8c733f1882eeeddbe34fa28511
With the update, the script needs to be launched from
out/{Debug,Release} directory.
Also updated docs to make it clear how to use the tool.
Fixes#538.
Change-Id: I37b71afc33e3b73ff5232e43d79c52ffa5859d57
- Update curl to 7.57.0
- Roll clang, which is needed due to MacOS / XCode update
- Fixes and suppress compilation errors due to clang update
Fixes#285
Change-Id: Ibac3288c641861605c3c0500d34d27373e6eecfe
Seeing some failures on some platforms when compiled with clang
disabled:
GYP_DEFINES="clang=0" gclient runhooks
Several changes to make it work:
1. Mark packager code with packager_code=1 in GYP definitions.
2. Disable a few checks in non-packager code, which we do not have
direct control: dangling-else, deprecated-declarations,
unused-function
3. Fix the relevant errors in packager code.
4. Revert HAVE_STROPTS_H in curl config which is not available in
all linux distributions.
Fixes#286Fixes#293
Change-Id: I729b41f99403c5ad9487c6cc4a7dc06f6323cef8
Cues used to be generated in the end of the file; if http range
request is not supported, clients have to download the whole
file to get to the cues.
This CL updated TwoPassSingleSegmentSegmenter to write cues
after writing webm header.
This CL also updates libwebm dependency to latest.
Closes#159
Change-Id: Ic73548e1b872e6b13a37627707e7d0ff3e556877
WebRTC portion (gyp build) now checked in because separate repository is no longer available due to transition to git.
Change-Id: I56436cbbc3f02a5f77112917d6229aea69f21d97
Create a new MacOSX configuration for curl:
- Add a curl_config.h created on MacOSX
- Add a find_curl_ca_bundle.sh that searches default folder from Homebrew and MacPorts for a ca bundle.
To find the files either install OpenSSL from Homebrew or MacPorts, or Curl from MacPorts.
When generating ninja files we end up with two libwebm targets.
packager/media/formats/webm/webm.gyp:webm -> becomes target libwebm
packager/third_party/libwebm/libwebm.gyp:libwebm -> becomes target libwebm
Solve this conflict by renaming libwebm.gyp:libwebm to mkvmuxer.
Find curl_ca_bundle script is used by linux only. Move it out of
gyp and make sure it runs only in linux.
Also make zlib deps generic for all systems, not for linux only.
This is the first step of getting edash_packager builds on Windows
and Mac.
Change-Id: I82d3e5872fd0dd6f57127890bc062490332a1b69
Fix "Problem with the SSL CA cert" issue @
https://github.com/google/edash-packager/issues/6. On Ubuntu, CURL_CA_BUNDLE
is located at "/etc/ssl/certs/ca-certificates.crt" while on CentOs, it is
located at "/etc/ssl/certs/ca-bundle.crt".
With this change, GYP scans the common appearing locations for SSL CA cert.
Change-Id: I838bae3954ec51f1430c900d1e82596d45a4db84
Joey Parrish
Joey Parrish
Jacob Trimble
KongQun Yang
Haoming Chen
Thomas Inskip
Anders Hasselqvist