feat: First phase of CMake build system implementation (#1072)
There are a lot of changes in this first phase, because there was a
lot of infrastructure required to get some meaningful amount of
porting done. Future PRs should be simpler.
<b>Summary of changes:</b><details>
- Remove old deps:
- boringssl (replaced with mbedtls, lighter, easier to build)
- gflags (replaced with absl::flags)
- Chromium build tools
- New deps to replace parts of Chromium base:
- abseil-cpp
- glog
- nlohmann::json (for tests only)
- Submodules, updates, and CMake build rules for third-party
libraries:
- curl
- gmock/gtest
- Ported internal libraries and their tests by removing Chromium deps
and adding CMake build rules:
- file (now using C++17 filesystem APIs)
- license_notice
- status
- version
- Test improvements
- Removed file tests that can never be re-enabled
- Re-enabled all other disabled file tests
- Debug JSON values when HTTP tests fail
- Fixed chunked-encoding issues in HTTP tests
- Updated and refactored Dockerfiles testing
- All docker files working, with OS versions updated to meet the
new tool requirements
- Local docker builds no longer write files to your working
directory as root
- Local docker builds can now be run in parallel without clobbering
each others' build outputs
- DEBUG=1 can drop you into an interactive shell when a docker
build fails
- Updated and heavily refactored workflows and Dockerfiles
- All docker files now tested in parallel on GitHub, speeding up CI
- All common workflow components broken out and using workflow_call
instead of custom actions
- Self-hosted runners now optional, to make testing easier on forks
- CMake porting works-in-process can now be fully tested on GitHub
- Building ported libraries and passing ported tests on all three
platforms!
- CI hacks for macOS removed, now testing on macos-latest!
- Python2 no longer required! (Only Python3)
- Using strict build flags, treating all warnings as errors.
</details>
<b>Required to build:</b>
- CMake >= 3.16
- Python 3
- A compiler supporting C++ >= 17
- g++ >= 9 if using GCC (Clang also fine)
- MSVC for Windows
<b>Still needs work:</b><details>
- Moving other dependencies into submodules (if we keep them):
- apple_apsl
- icu
- libevent
- libpng
- libwebm
- libxml
- modp_b64
- protobuf
- zlib
- Port remaining internal libraries:
- app
- hls
- media/base
- media/chunking
- media/codecs
- media/crypto
- media/demuxer
- media/event
- media/formats/dvb
- media/formats/mp2t
- media/formats/mp4
- media/formats/packed_audio
- media/formats/ttml
- media/formats/webm
- media/formats/webvtt
- media/formats/wvm
- media/origin
- media/public
- media/replicator
- media/trick_play
- mpd
- Port main application
- Add logging flags in absl and connect them to glog (which expects
gflags)
- Port pssh-box.py
- Port main test targets (packager_test.py and packager_app.py)
- Updating all requirement and build documentation
- Remove any remaining refs to gclient, depot_tools, ninja
- Update and complete release workflows using release-please
</details>
Issue #346 (Switch to abseil)
Issue #1047 (New build system)
2022-08-16 18:34:51 +00:00
|
|
|
# Copyright 2022 Google LLC
|
|
|
|
#
|
2022-08-26 15:44:59 +00:00
|
|
|
# Use of this source code is governed by a BSD-style
|
|
|
|
# license that can be found in the LICENSE file or at
|
|
|
|
# https://developers.google.com/open-source/licenses/bsd
|
feat: First phase of CMake build system implementation (#1072)
There are a lot of changes in this first phase, because there was a
lot of infrastructure required to get some meaningful amount of
porting done. Future PRs should be simpler.
<b>Summary of changes:</b><details>
- Remove old deps:
- boringssl (replaced with mbedtls, lighter, easier to build)
- gflags (replaced with absl::flags)
- Chromium build tools
- New deps to replace parts of Chromium base:
- abseil-cpp
- glog
- nlohmann::json (for tests only)
- Submodules, updates, and CMake build rules for third-party
libraries:
- curl
- gmock/gtest
- Ported internal libraries and their tests by removing Chromium deps
and adding CMake build rules:
- file (now using C++17 filesystem APIs)
- license_notice
- status
- version
- Test improvements
- Removed file tests that can never be re-enabled
- Re-enabled all other disabled file tests
- Debug JSON values when HTTP tests fail
- Fixed chunked-encoding issues in HTTP tests
- Updated and refactored Dockerfiles testing
- All docker files working, with OS versions updated to meet the
new tool requirements
- Local docker builds no longer write files to your working
directory as root
- Local docker builds can now be run in parallel without clobbering
each others' build outputs
- DEBUG=1 can drop you into an interactive shell when a docker
build fails
- Updated and heavily refactored workflows and Dockerfiles
- All docker files now tested in parallel on GitHub, speeding up CI
- All common workflow components broken out and using workflow_call
instead of custom actions
- Self-hosted runners now optional, to make testing easier on forks
- CMake porting works-in-process can now be fully tested on GitHub
- Building ported libraries and passing ported tests on all three
platforms!
- CI hacks for macOS removed, now testing on macos-latest!
- Python2 no longer required! (Only Python3)
- Using strict build flags, treating all warnings as errors.
</details>
<b>Required to build:</b>
- CMake >= 3.16
- Python 3
- A compiler supporting C++ >= 17
- g++ >= 9 if using GCC (Clang also fine)
- MSVC for Windows
<b>Still needs work:</b><details>
- Moving other dependencies into submodules (if we keep them):
- apple_apsl
- icu
- libevent
- libpng
- libwebm
- libxml
- modp_b64
- protobuf
- zlib
- Port remaining internal libraries:
- app
- hls
- media/base
- media/chunking
- media/codecs
- media/crypto
- media/demuxer
- media/event
- media/formats/dvb
- media/formats/mp2t
- media/formats/mp4
- media/formats/packed_audio
- media/formats/ttml
- media/formats/webm
- media/formats/webvtt
- media/formats/wvm
- media/origin
- media/public
- media/replicator
- media/trick_play
- mpd
- Port main application
- Add logging flags in absl and connect them to glog (which expects
gflags)
- Port pssh-box.py
- Port main test targets (packager_test.py and packager_app.py)
- Updating all requirement and build documentation
- Remove any remaining refs to gclient, depot_tools, ninja
- Update and complete release workflows using release-please
</details>
Issue #346 (Switch to abseil)
Issue #1047 (New build system)
2022-08-16 18:34:51 +00:00
|
|
|
|
|
|
|
# A reusable workflow to build and test Packager on every supported OS and
|
|
|
|
# architecture.
|
|
|
|
name: Build
|
|
|
|
|
|
|
|
# Runs when called from another workflow.
|
|
|
|
on:
|
|
|
|
workflow_call:
|
|
|
|
inputs:
|
|
|
|
ref:
|
|
|
|
required: true
|
|
|
|
type: string
|
|
|
|
secrets:
|
|
|
|
# The GITHUB_TOKEN name is reserved, but not passed through implicitly.
|
|
|
|
# So we call our secret parameter simply TOKEN.
|
|
|
|
TOKEN:
|
|
|
|
required: false
|
|
|
|
|
|
|
|
# These below are not actual secrets, but secrets are the only place to
|
|
|
|
# keep repo-specific configs that make this project friendlier to forks
|
|
|
|
# and easier to debug.
|
|
|
|
|
|
|
|
# If non-empty, start a debug SSH server on failures.
|
|
|
|
ENABLE_DEBUG:
|
|
|
|
required: false
|
|
|
|
# If non-empty, enable self-hosted runners in the build matrix.
|
|
|
|
ENABLE_SELF_HOSTED:
|
|
|
|
required: false
|
|
|
|
|
|
|
|
# By default, run all commands in a bash shell. On Windows, the default would
|
|
|
|
# otherwise be powershell.
|
|
|
|
defaults:
|
|
|
|
run:
|
|
|
|
shell: bash
|
|
|
|
|
|
|
|
jobs:
|
|
|
|
# Configure the build matrix based on inputs. The list of objects in the
|
|
|
|
# build matrix contents can't be changed by conditionals, but it can be
|
|
|
|
# computed by another job and deserialized. This uses
|
|
|
|
# secrets.ENABLE_SELF_HOSTED to determine the build matrix, based on the
|
|
|
|
# metadata in build-matrix.json.
|
|
|
|
matrix_config:
|
|
|
|
runs-on: ubuntu-latest
|
|
|
|
outputs:
|
|
|
|
INCLUDE: ${{ steps.configure.outputs.INCLUDE }}
|
|
|
|
OS: ${{ steps.configure.outputs.OS }}
|
|
|
|
ENABLE_DEBUG: ${{ steps.configure.outputs.ENABLE_DEBUG }}
|
|
|
|
steps:
|
|
|
|
- uses: actions/checkout@v2
|
|
|
|
with:
|
|
|
|
ref: ${{ inputs.ref }}
|
|
|
|
|
|
|
|
- name: Configure Build Matrix
|
|
|
|
id: configure
|
|
|
|
shell: node {0}
|
|
|
|
run: |
|
|
|
|
const enableDebug = "${{ secrets.ENABLE_DEBUG }}" != '';
|
|
|
|
const enableSelfHosted = "${{ secrets.ENABLE_SELF_HOSTED }}" != '';
|
|
|
|
|
|
|
|
// Use ENABLE_SELF_HOSTED to decide what the build matrix below
|
|
|
|
// should include.
|
|
|
|
const {hosted, selfHosted} = require("${{ github.workspace }}/.github/workflows/build-matrix.json");
|
|
|
|
const include = enableSelfHosted ? hosted.concat(selfHosted) : hosted;
|
|
|
|
const os = include.map((config) => config.os);
|
|
|
|
|
|
|
|
// Output JSON objects consumed by the build matrix below.
|
|
|
|
console.log(`::set-output name=INCLUDE::${ JSON.stringify(include) }`);
|
|
|
|
console.log(`::set-output name=OS::${ JSON.stringify(os) }`);
|
|
|
|
|
|
|
|
// Output the debug flag directly.
|
|
|
|
console.log(`::set-output name=ENABLE_DEBUG::${ enableDebug }`);
|
|
|
|
|
|
|
|
// Log the outputs, for the sake of debugging this script.
|
|
|
|
console.log({enableDebug, include, os});
|
|
|
|
|
|
|
|
build:
|
|
|
|
needs: matrix_config
|
|
|
|
strategy:
|
|
|
|
fail-fast: false
|
|
|
|
matrix:
|
|
|
|
include: ${{ fromJSON(needs.matrix_config.outputs.INCLUDE) }}
|
|
|
|
os: ${{ fromJSON(needs.matrix_config.outputs.OS) }}
|
|
|
|
build_type: ["Debug", "Release"]
|
|
|
|
lib_type: ["static", "shared"]
|
|
|
|
|
|
|
|
name: ${{ matrix.os_name }} ${{ matrix.target_arch }} ${{ matrix.build_type }} ${{ matrix.lib_type }}
|
|
|
|
runs-on: ${{ matrix.os }}
|
|
|
|
|
|
|
|
steps:
|
|
|
|
- name: Configure git to preserve line endings
|
|
|
|
# Otherwise, tests fail on Windows because "golden" test outputs will not
|
|
|
|
# have the correct line endings.
|
|
|
|
run: git config --global core.autocrlf false
|
|
|
|
|
|
|
|
- name: Checkout code
|
|
|
|
uses: actions/checkout@v2
|
|
|
|
with:
|
|
|
|
ref: ${{ inputs.ref }}
|
|
|
|
submodules: true
|
|
|
|
|
|
|
|
- name: Install Linux deps
|
|
|
|
if: runner.os == 'Linux'
|
|
|
|
run: sudo apt install -y libc-ares-dev
|
|
|
|
|
|
|
|
- name: Generate build files
|
|
|
|
run: |
|
|
|
|
mkdir -p build/
|
|
|
|
|
|
|
|
if [[ "${{ matrix.lib_type }}" == "shared" ]]; then
|
|
|
|
LIBPACKAGER_SHARED="ON"
|
|
|
|
else
|
|
|
|
LIBPACKAGER_SHARED="OFF"
|
|
|
|
fi
|
|
|
|
|
|
|
|
cmake \
|
|
|
|
-DCMAKE_BUILD_TYPE="${{ matrix.build_type }}" \
|
|
|
|
-DLIBPACKAGER_SHARED="$LIBPACKAGER_SHARED" \
|
|
|
|
-S . \
|
|
|
|
-B build/
|
|
|
|
|
|
|
|
- name: Build
|
|
|
|
# This is a universal build command, which will call make on Linux and
|
|
|
|
# Visual Studio on Windows. Note that the VS generator is what cmake
|
|
|
|
# calls a "multi-configuration" generator, and so the desired build
|
|
|
|
# type must be specified for Windows.
|
2022-10-21 18:28:15 +00:00
|
|
|
run: cmake --build build/ --config "${{ matrix.build_type }}" --parallel
|
feat: First phase of CMake build system implementation (#1072)
There are a lot of changes in this first phase, because there was a
lot of infrastructure required to get some meaningful amount of
porting done. Future PRs should be simpler.
<b>Summary of changes:</b><details>
- Remove old deps:
- boringssl (replaced with mbedtls, lighter, easier to build)
- gflags (replaced with absl::flags)
- Chromium build tools
- New deps to replace parts of Chromium base:
- abseil-cpp
- glog
- nlohmann::json (for tests only)
- Submodules, updates, and CMake build rules for third-party
libraries:
- curl
- gmock/gtest
- Ported internal libraries and their tests by removing Chromium deps
and adding CMake build rules:
- file (now using C++17 filesystem APIs)
- license_notice
- status
- version
- Test improvements
- Removed file tests that can never be re-enabled
- Re-enabled all other disabled file tests
- Debug JSON values when HTTP tests fail
- Fixed chunked-encoding issues in HTTP tests
- Updated and refactored Dockerfiles testing
- All docker files working, with OS versions updated to meet the
new tool requirements
- Local docker builds no longer write files to your working
directory as root
- Local docker builds can now be run in parallel without clobbering
each others' build outputs
- DEBUG=1 can drop you into an interactive shell when a docker
build fails
- Updated and heavily refactored workflows and Dockerfiles
- All docker files now tested in parallel on GitHub, speeding up CI
- All common workflow components broken out and using workflow_call
instead of custom actions
- Self-hosted runners now optional, to make testing easier on forks
- CMake porting works-in-process can now be fully tested on GitHub
- Building ported libraries and passing ported tests on all three
platforms!
- CI hacks for macOS removed, now testing on macos-latest!
- Python2 no longer required! (Only Python3)
- Using strict build flags, treating all warnings as errors.
</details>
<b>Required to build:</b>
- CMake >= 3.16
- Python 3
- A compiler supporting C++ >= 17
- g++ >= 9 if using GCC (Clang also fine)
- MSVC for Windows
<b>Still needs work:</b><details>
- Moving other dependencies into submodules (if we keep them):
- apple_apsl
- icu
- libevent
- libpng
- libwebm
- libxml
- modp_b64
- protobuf
- zlib
- Port remaining internal libraries:
- app
- hls
- media/base
- media/chunking
- media/codecs
- media/crypto
- media/demuxer
- media/event
- media/formats/dvb
- media/formats/mp2t
- media/formats/mp4
- media/formats/packed_audio
- media/formats/ttml
- media/formats/webm
- media/formats/webvtt
- media/formats/wvm
- media/origin
- media/public
- media/replicator
- media/trick_play
- mpd
- Port main application
- Add logging flags in absl and connect them to glog (which expects
gflags)
- Port pssh-box.py
- Port main test targets (packager_test.py and packager_app.py)
- Updating all requirement and build documentation
- Remove any remaining refs to gclient, depot_tools, ninja
- Update and complete release workflows using release-please
</details>
Issue #346 (Switch to abseil)
Issue #1047 (New build system)
2022-08-16 18:34:51 +00:00
|
|
|
|
|
|
|
- name: Test
|
2022-10-24 16:31:56 +00:00
|
|
|
run: ctest -C "${{ matrix.build_type }}" -V --test-dir build/
|
feat: First phase of CMake build system implementation (#1072)
There are a lot of changes in this first phase, because there was a
lot of infrastructure required to get some meaningful amount of
porting done. Future PRs should be simpler.
<b>Summary of changes:</b><details>
- Remove old deps:
- boringssl (replaced with mbedtls, lighter, easier to build)
- gflags (replaced with absl::flags)
- Chromium build tools
- New deps to replace parts of Chromium base:
- abseil-cpp
- glog
- nlohmann::json (for tests only)
- Submodules, updates, and CMake build rules for third-party
libraries:
- curl
- gmock/gtest
- Ported internal libraries and their tests by removing Chromium deps
and adding CMake build rules:
- file (now using C++17 filesystem APIs)
- license_notice
- status
- version
- Test improvements
- Removed file tests that can never be re-enabled
- Re-enabled all other disabled file tests
- Debug JSON values when HTTP tests fail
- Fixed chunked-encoding issues in HTTP tests
- Updated and refactored Dockerfiles testing
- All docker files working, with OS versions updated to meet the
new tool requirements
- Local docker builds no longer write files to your working
directory as root
- Local docker builds can now be run in parallel without clobbering
each others' build outputs
- DEBUG=1 can drop you into an interactive shell when a docker
build fails
- Updated and heavily refactored workflows and Dockerfiles
- All docker files now tested in parallel on GitHub, speeding up CI
- All common workflow components broken out and using workflow_call
instead of custom actions
- Self-hosted runners now optional, to make testing easier on forks
- CMake porting works-in-process can now be fully tested on GitHub
- Building ported libraries and passing ported tests on all three
platforms!
- CI hacks for macOS removed, now testing on macos-latest!
- Python2 no longer required! (Only Python3)
- Using strict build flags, treating all warnings as errors.
</details>
<b>Required to build:</b>
- CMake >= 3.16
- Python 3
- A compiler supporting C++ >= 17
- g++ >= 9 if using GCC (Clang also fine)
- MSVC for Windows
<b>Still needs work:</b><details>
- Moving other dependencies into submodules (if we keep them):
- apple_apsl
- icu
- libevent
- libpng
- libwebm
- libxml
- modp_b64
- protobuf
- zlib
- Port remaining internal libraries:
- app
- hls
- media/base
- media/chunking
- media/codecs
- media/crypto
- media/demuxer
- media/event
- media/formats/dvb
- media/formats/mp2t
- media/formats/mp4
- media/formats/packed_audio
- media/formats/ttml
- media/formats/webm
- media/formats/webvtt
- media/formats/wvm
- media/origin
- media/public
- media/replicator
- media/trick_play
- mpd
- Port main application
- Add logging flags in absl and connect them to glog (which expects
gflags)
- Port pssh-box.py
- Port main test targets (packager_test.py and packager_app.py)
- Updating all requirement and build documentation
- Remove any remaining refs to gclient, depot_tools, ninja
- Update and complete release workflows using release-please
</details>
Issue #346 (Switch to abseil)
Issue #1047 (New build system)
2022-08-16 18:34:51 +00:00
|
|
|
|
|
|
|
# TODO(joeyparrish): Prepare artifacts when build system is complete again
|
|
|
|
# - name: Prepare artifacts (static release only)
|
|
|
|
# run: |
|
|
|
|
# BUILD_CONFIG="${{ matrix.build_type }}-${{ matrix.lib_type }}"
|
|
|
|
# if [[ "$BUILD_CONFIG" != "Release-static" ]]; then
|
|
|
|
# echo "Skipping artifacts for $BUILD_CONFIG."
|
|
|
|
# exit 0
|
|
|
|
# fi
|
|
|
|
# if [[ "${{ runner.os }}" == "Linux" ]]; then
|
|
|
|
# echo "::group::Check for static executables"
|
|
|
|
# (
|
|
|
|
# cd build/Release
|
|
|
|
# # Prove that we built static executables on Linux. First, check that
|
|
|
|
# # the executables exist, and fail if they do not. Then check "ldd",
|
|
|
|
# # which will fail if the executable is not dynamically linked. If
|
|
|
|
# # "ldd" succeeds, we fail the workflow. Finally, we call "true" so
|
|
|
|
# # that the last executed statement will be a success, and the step
|
|
|
|
# # won't be failed if we get that far.
|
|
|
|
# ls packager mpd_generator >/dev/null || exit 1
|
|
|
|
# ldd packager 2>&1 && exit 1
|
|
|
|
# ldd mpd_generator 2>&1 && exit 1
|
|
|
|
# true
|
|
|
|
# )
|
|
|
|
# echo "::endgroup::"
|
|
|
|
# fi
|
|
|
|
# echo "::group::Prepare artifacts folder"
|
|
|
|
# mkdir artifacts
|
|
|
|
# ARTIFACTS="$GITHUB_WORKSPACE/artifacts"
|
|
|
|
# cd build/Release
|
|
|
|
# echo "::endgroup::"
|
|
|
|
# echo "::group::Strip executables"
|
|
|
|
# strip packager${{ matrix.exe_ext }}
|
|
|
|
# strip mpd_generator${{ matrix.exe_ext }}
|
|
|
|
# echo "::endgroup::"
|
|
|
|
# SUFFIX="-${{ matrix.os_name }}-${{ matrix.target_arch }}"
|
|
|
|
# EXE_SUFFIX="$SUFFIX${{ matrix.exe_ext}}"
|
|
|
|
# echo "::group::Copy packager"
|
|
|
|
# cp packager${{ matrix.exe_ext }} $ARTIFACTS/packager$EXE_SUFFIX
|
|
|
|
# echo "::endgroup::"
|
|
|
|
# echo "::group::Copy mpd_generator"
|
|
|
|
# cp mpd_generator${{ matrix.exe_ext }} $ARTIFACTS/mpd_generator$EXE_SUFFIX
|
|
|
|
# echo "::endgroup::"
|
|
|
|
# # The pssh-box bundle is OS and architecture independent. So only do
|
|
|
|
# # it on this one OS and architecture, and give it a more generic
|
|
|
|
# # filename.
|
|
|
|
# if [[ '${{ matrix.os_name }}' == 'linux' && '${{ matrix.target_arch }}' == 'x64' ]]; then
|
|
|
|
# echo "::group::Tar pssh-box"
|
|
|
|
# tar -czf $ARTIFACTS/pssh-box.py.tar.gz pyproto pssh-box.py
|
|
|
|
# echo "::endgroup::"
|
|
|
|
# fi
|
|
|
|
|
|
|
|
# TODO(joeyparrish): Attach artifacts when build system is complete again
|
|
|
|
# - name: Attach artifacts to release
|
|
|
|
# if: matrix.build_type == 'Release' && matrix.lib_type == 'static'
|
|
|
|
# uses: dwenegar/upload-release-assets@v1
|
|
|
|
# env:
|
|
|
|
# GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
|
|
# with:
|
|
|
|
# release_id: ${{ needs.draft_release.outputs.release_id }}
|
|
|
|
# assets_path: artifacts
|
|
|
|
|
|
|
|
- name: Debug
|
|
|
|
uses: mxschmitt/action-tmate@v3.6
|
|
|
|
with:
|
|
|
|
limit-access-to-actor: true
|
|
|
|
if: failure() && needs.matrix_config.outputs.ENABLE_DEBUG != ''
|