feat: First phase of CMake build system implementation (#1072)
There are a lot of changes in this first phase, because there was a
lot of infrastructure required to get some meaningful amount of
porting done. Future PRs should be simpler.
<b>Summary of changes:</b><details>
- Remove old deps:
- boringssl (replaced with mbedtls, lighter, easier to build)
- gflags (replaced with absl::flags)
- Chromium build tools
- New deps to replace parts of Chromium base:
- abseil-cpp
- glog
- nlohmann::json (for tests only)
- Submodules, updates, and CMake build rules for third-party
libraries:
- curl
- gmock/gtest
- Ported internal libraries and their tests by removing Chromium deps
and adding CMake build rules:
- file (now using C++17 filesystem APIs)
- license_notice
- status
- version
- Test improvements
- Removed file tests that can never be re-enabled
- Re-enabled all other disabled file tests
- Debug JSON values when HTTP tests fail
- Fixed chunked-encoding issues in HTTP tests
- Updated and refactored Dockerfiles testing
- All docker files working, with OS versions updated to meet the
new tool requirements
- Local docker builds no longer write files to your working
directory as root
- Local docker builds can now be run in parallel without clobbering
each others' build outputs
- DEBUG=1 can drop you into an interactive shell when a docker
build fails
- Updated and heavily refactored workflows and Dockerfiles
- All docker files now tested in parallel on GitHub, speeding up CI
- All common workflow components broken out and using workflow_call
instead of custom actions
- Self-hosted runners now optional, to make testing easier on forks
- CMake porting works-in-process can now be fully tested on GitHub
- Building ported libraries and passing ported tests on all three
platforms!
- CI hacks for macOS removed, now testing on macos-latest!
- Python2 no longer required! (Only Python3)
- Using strict build flags, treating all warnings as errors.
</details>
<b>Required to build:</b>
- CMake >= 3.16
- Python 3
- A compiler supporting C++ >= 17
- g++ >= 9 if using GCC (Clang also fine)
- MSVC for Windows
<b>Still needs work:</b><details>
- Moving other dependencies into submodules (if we keep them):
- apple_apsl
- icu
- libevent
- libpng
- libwebm
- libxml
- modp_b64
- protobuf
- zlib
- Port remaining internal libraries:
- app
- hls
- media/base
- media/chunking
- media/codecs
- media/crypto
- media/demuxer
- media/event
- media/formats/dvb
- media/formats/mp2t
- media/formats/mp4
- media/formats/packed_audio
- media/formats/ttml
- media/formats/webm
- media/formats/webvtt
- media/formats/wvm
- media/origin
- media/public
- media/replicator
- media/trick_play
- mpd
- Port main application
- Add logging flags in absl and connect them to glog (which expects
gflags)
- Port pssh-box.py
- Port main test targets (packager_test.py and packager_app.py)
- Updating all requirement and build documentation
- Remove any remaining refs to gclient, depot_tools, ninja
- Update and complete release workflows using release-please
</details>
Issue #346 (Switch to abseil)
Issue #1047 (New build system)
2022-08-16 18:34:51 +00:00
|
|
|
#!/bin/bash
|
|
|
|
|
|
|
|
# Exit on first error.
|
|
|
|
set -e
|
|
|
|
|
|
|
|
# To debug a failure, run with the variable DEBUG=1. For example:
|
|
|
|
# DEBUG=1 ./packager/testing/test_dockers.sh
|
|
|
|
|
|
|
|
SCRIPT_DIR="$(dirname "$0")"
|
|
|
|
PACKAGER_DIR="$(realpath "$SCRIPT_DIR/../..")"
|
|
|
|
TEMP_BUILD_DIR="$(mktemp -d)"
|
|
|
|
|
|
|
|
function docker_run_internal() {
|
|
|
|
if [[ "$DEBUG" == "1" ]]; then
|
|
|
|
# For debugging, allocate an interactive terminal in docker.
|
|
|
|
INTERACTIVE_ARG="-it"
|
|
|
|
else
|
|
|
|
INTERACTIVE_ARG=""
|
|
|
|
fi
|
|
|
|
|
|
|
|
docker run \
|
|
|
|
${INTERACTIVE_ARG} \
|
|
|
|
-v ${PACKAGER_DIR}:/shaka-packager \
|
|
|
|
-v ${TEMP_BUILD_DIR}:/shaka-packager/build \
|
|
|
|
-w /shaka-packager \
|
|
|
|
-e HOME=/tmp \
|
|
|
|
--user $(id -u):$(id -g) \
|
|
|
|
${CONTAINER} "$@"
|
|
|
|
}
|
|
|
|
|
|
|
|
function docker_run() {
|
|
|
|
if ! docker_run_internal "$@"; then
|
|
|
|
echo "Command failed in ${CONTAINER}: $@"
|
|
|
|
if [[ "$DEBUG" == "1" ]]; then
|
|
|
|
echo "Launching interactive shell to debug."
|
|
|
|
docker_run_internal /bin/bash
|
|
|
|
exit 1
|
|
|
|
else
|
|
|
|
echo "Run with DEBUG=1 to debug!"
|
|
|
|
exit 1
|
|
|
|
fi
|
|
|
|
fi
|
|
|
|
}
|
|
|
|
|
|
|
|
# Command line arguments will be taken as an allowlist of OSes to run.
|
|
|
|
# By default, a regex that matches everything.
|
|
|
|
FILTER=".*"
|
|
|
|
if [[ $# != 0 ]]; then
|
|
|
|
# Join arguments with a pipe, to make a regex alternation to match any of
|
|
|
|
# them. The syntax is a mess, but that's bash. Set IFS (the separator
|
|
|
|
# variable) in a subshell and print the array. This has the effect of joining
|
|
|
|
# them by the character in IFS. Then add parentheses to make a complete regex
|
|
|
|
# to match all the arguments.
|
|
|
|
FILTER=$(IFS="|"; echo "$*")
|
|
|
|
FILTER="($FILTER)"
|
|
|
|
fi
|
|
|
|
|
|
|
|
function on_exit() {
|
|
|
|
# On exit, print the name of the OS we were on. This helps identify what to
|
|
|
|
# debug when the start of a test run scrolls off-screen.
|
|
|
|
echo "Failed on $OS_NAME!"
|
|
|
|
rm -rf "${TEMP_BUILD_DIR}"
|
|
|
|
}
|
|
|
|
trap 'on_exit' exit
|
|
|
|
|
|
|
|
echo "Using OS filter: $FILTER"
|
|
|
|
RAN_SOMETHING=0
|
|
|
|
for DOCKER_FILE in ${SCRIPT_DIR}/dockers/*; do
|
|
|
|
# Take the basename of the dockerfile path, then remove the trailing
|
|
|
|
# "_Dockerfile" from the file name. This is the OS name.
|
|
|
|
OS_NAME="$( basename "$DOCKER_FILE" | sed -e 's/_Dockerfile//' )"
|
|
|
|
|
|
|
|
if echo "$OS_NAME" | grep -Eqi "$FILTER"; then
|
|
|
|
echo "Testing $OS_NAME."
|
|
|
|
# Fall through.
|
|
|
|
else
|
|
|
|
echo "Skipping $OS_NAME."
|
|
|
|
continue
|
|
|
|
fi
|
|
|
|
|
|
|
|
# Build a unique container name per OS for debugging purposes and to improve
|
|
|
|
# caching. Containers names must be in lowercase.
|
|
|
|
CONTAINER="$( echo "packager_test_${OS_NAME}" | tr A-Z a-z )"
|
|
|
|
|
|
|
|
RAN_SOMETHING=1
|
2023-11-30 21:17:01 +00:00
|
|
|
docker buildx build --pull -t ${CONTAINER} -f ${DOCKER_FILE} ${SCRIPT_DIR}/dockers/
|
feat: First phase of CMake build system implementation (#1072)
There are a lot of changes in this first phase, because there was a
lot of infrastructure required to get some meaningful amount of
porting done. Future PRs should be simpler.
<b>Summary of changes:</b><details>
- Remove old deps:
- boringssl (replaced with mbedtls, lighter, easier to build)
- gflags (replaced with absl::flags)
- Chromium build tools
- New deps to replace parts of Chromium base:
- abseil-cpp
- glog
- nlohmann::json (for tests only)
- Submodules, updates, and CMake build rules for third-party
libraries:
- curl
- gmock/gtest
- Ported internal libraries and their tests by removing Chromium deps
and adding CMake build rules:
- file (now using C++17 filesystem APIs)
- license_notice
- status
- version
- Test improvements
- Removed file tests that can never be re-enabled
- Re-enabled all other disabled file tests
- Debug JSON values when HTTP tests fail
- Fixed chunked-encoding issues in HTTP tests
- Updated and refactored Dockerfiles testing
- All docker files working, with OS versions updated to meet the
new tool requirements
- Local docker builds no longer write files to your working
directory as root
- Local docker builds can now be run in parallel without clobbering
each others' build outputs
- DEBUG=1 can drop you into an interactive shell when a docker
build fails
- Updated and heavily refactored workflows and Dockerfiles
- All docker files now tested in parallel on GitHub, speeding up CI
- All common workflow components broken out and using workflow_call
instead of custom actions
- Self-hosted runners now optional, to make testing easier on forks
- CMake porting works-in-process can now be fully tested on GitHub
- Building ported libraries and passing ported tests on all three
platforms!
- CI hacks for macOS removed, now testing on macos-latest!
- Python2 no longer required! (Only Python3)
- Using strict build flags, treating all warnings as errors.
</details>
<b>Required to build:</b>
- CMake >= 3.16
- Python 3
- A compiler supporting C++ >= 17
- g++ >= 9 if using GCC (Clang also fine)
- MSVC for Windows
<b>Still needs work:</b><details>
- Moving other dependencies into submodules (if we keep them):
- apple_apsl
- icu
- libevent
- libpng
- libwebm
- libxml
- modp_b64
- protobuf
- zlib
- Port remaining internal libraries:
- app
- hls
- media/base
- media/chunking
- media/codecs
- media/crypto
- media/demuxer
- media/event
- media/formats/dvb
- media/formats/mp2t
- media/formats/mp4
- media/formats/packed_audio
- media/formats/ttml
- media/formats/webm
- media/formats/webvtt
- media/formats/wvm
- media/origin
- media/public
- media/replicator
- media/trick_play
- mpd
- Port main application
- Add logging flags in absl and connect them to glog (which expects
gflags)
- Port pssh-box.py
- Port main test targets (packager_test.py and packager_app.py)
- Updating all requirement and build documentation
- Remove any remaining refs to gclient, depot_tools, ninja
- Update and complete release workflows using release-please
</details>
Issue #346 (Switch to abseil)
Issue #1047 (New build system)
2022-08-16 18:34:51 +00:00
|
|
|
mkdir -p "${TEMP_BUILD_DIR}"
|
2023-07-17 20:16:03 +00:00
|
|
|
docker_run cmake -S . -B build/ -DCMAKE_BUILD_TYPE=Debug -G Ninja
|
|
|
|
docker_run cmake --build build/ --config Debug --parallel
|
2022-11-03 14:21:14 +00:00
|
|
|
docker_run bash -c "cd build && ctest -C Debug -V"
|
feat: First phase of CMake build system implementation (#1072)
There are a lot of changes in this first phase, because there was a
lot of infrastructure required to get some meaningful amount of
porting done. Future PRs should be simpler.
<b>Summary of changes:</b><details>
- Remove old deps:
- boringssl (replaced with mbedtls, lighter, easier to build)
- gflags (replaced with absl::flags)
- Chromium build tools
- New deps to replace parts of Chromium base:
- abseil-cpp
- glog
- nlohmann::json (for tests only)
- Submodules, updates, and CMake build rules for third-party
libraries:
- curl
- gmock/gtest
- Ported internal libraries and their tests by removing Chromium deps
and adding CMake build rules:
- file (now using C++17 filesystem APIs)
- license_notice
- status
- version
- Test improvements
- Removed file tests that can never be re-enabled
- Re-enabled all other disabled file tests
- Debug JSON values when HTTP tests fail
- Fixed chunked-encoding issues in HTTP tests
- Updated and refactored Dockerfiles testing
- All docker files working, with OS versions updated to meet the
new tool requirements
- Local docker builds no longer write files to your working
directory as root
- Local docker builds can now be run in parallel without clobbering
each others' build outputs
- DEBUG=1 can drop you into an interactive shell when a docker
build fails
- Updated and heavily refactored workflows and Dockerfiles
- All docker files now tested in parallel on GitHub, speeding up CI
- All common workflow components broken out and using workflow_call
instead of custom actions
- Self-hosted runners now optional, to make testing easier on forks
- CMake porting works-in-process can now be fully tested on GitHub
- Building ported libraries and passing ported tests on all three
platforms!
- CI hacks for macOS removed, now testing on macos-latest!
- Python2 no longer required! (Only Python3)
- Using strict build flags, treating all warnings as errors.
</details>
<b>Required to build:</b>
- CMake >= 3.16
- Python 3
- A compiler supporting C++ >= 17
- g++ >= 9 if using GCC (Clang also fine)
- MSVC for Windows
<b>Still needs work:</b><details>
- Moving other dependencies into submodules (if we keep them):
- apple_apsl
- icu
- libevent
- libpng
- libwebm
- libxml
- modp_b64
- protobuf
- zlib
- Port remaining internal libraries:
- app
- hls
- media/base
- media/chunking
- media/codecs
- media/crypto
- media/demuxer
- media/event
- media/formats/dvb
- media/formats/mp2t
- media/formats/mp4
- media/formats/packed_audio
- media/formats/ttml
- media/formats/webm
- media/formats/webvtt
- media/formats/wvm
- media/origin
- media/public
- media/replicator
- media/trick_play
- mpd
- Port main application
- Add logging flags in absl and connect them to glog (which expects
gflags)
- Port pssh-box.py
- Port main test targets (packager_test.py and packager_app.py)
- Updating all requirement and build documentation
- Remove any remaining refs to gclient, depot_tools, ninja
- Update and complete release workflows using release-please
</details>
Issue #346 (Switch to abseil)
Issue #1047 (New build system)
2022-08-16 18:34:51 +00:00
|
|
|
rm -rf "${TEMP_BUILD_DIR}"
|
|
|
|
done
|
|
|
|
|
|
|
|
# Clear the exit trap from above.
|
|
|
|
trap - exit
|
|
|
|
|
|
|
|
if [[ "$RAN_SOMETHING" == "0" ]]; then
|
|
|
|
echo "No tests were run! The filter $FILTER did not match any OSes." 1>&2
|
|
|
|
exit 1
|
|
|
|
fi
|