shaka-packager/.github/workflows/build.yaml

202 lines
7.6 KiB
YAML
Raw Normal View History

feat: First phase of CMake build system implementation (#1072) There are a lot of changes in this first phase, because there was a lot of infrastructure required to get some meaningful amount of porting done. Future PRs should be simpler. <b>Summary of changes:</b><details> - Remove old deps: - boringssl (replaced with mbedtls, lighter, easier to build) - gflags (replaced with absl::flags) - Chromium build tools - New deps to replace parts of Chromium base: - abseil-cpp - glog - nlohmann::json (for tests only) - Submodules, updates, and CMake build rules for third-party libraries: - curl - gmock/gtest - Ported internal libraries and their tests by removing Chromium deps and adding CMake build rules: - file (now using C++17 filesystem APIs) - license_notice - status - version - Test improvements - Removed file tests that can never be re-enabled - Re-enabled all other disabled file tests - Debug JSON values when HTTP tests fail - Fixed chunked-encoding issues in HTTP tests - Updated and refactored Dockerfiles testing - All docker files working, with OS versions updated to meet the new tool requirements - Local docker builds no longer write files to your working directory as root - Local docker builds can now be run in parallel without clobbering each others' build outputs - DEBUG=1 can drop you into an interactive shell when a docker build fails - Updated and heavily refactored workflows and Dockerfiles - All docker files now tested in parallel on GitHub, speeding up CI - All common workflow components broken out and using workflow_call instead of custom actions - Self-hosted runners now optional, to make testing easier on forks - CMake porting works-in-process can now be fully tested on GitHub - Building ported libraries and passing ported tests on all three platforms! - CI hacks for macOS removed, now testing on macos-latest! - Python2 no longer required! (Only Python3) - Using strict build flags, treating all warnings as errors. </details> <b>Required to build:</b> - CMake >= 3.16 - Python 3 - A compiler supporting C++ >= 17 - g++ >= 9 if using GCC (Clang also fine) - MSVC for Windows <b>Still needs work:</b><details> - Moving other dependencies into submodules (if we keep them): - apple_apsl - icu - libevent - libpng - libwebm - libxml - modp_b64 - protobuf - zlib - Port remaining internal libraries: - app - hls - media/base - media/chunking - media/codecs - media/crypto - media/demuxer - media/event - media/formats/dvb - media/formats/mp2t - media/formats/mp4 - media/formats/packed_audio - media/formats/ttml - media/formats/webm - media/formats/webvtt - media/formats/wvm - media/origin - media/public - media/replicator - media/trick_play - mpd - Port main application - Add logging flags in absl and connect them to glog (which expects gflags) - Port pssh-box.py - Port main test targets (packager_test.py and packager_app.py) - Updating all requirement and build documentation - Remove any remaining refs to gclient, depot_tools, ninja - Update and complete release workflows using release-please </details> Issue #346 (Switch to abseil) Issue #1047 (New build system)
2022-08-16 18:34:51 +00:00
# Copyright 2022 Google LLC
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
feat: First phase of CMake build system implementation (#1072) There are a lot of changes in this first phase, because there was a lot of infrastructure required to get some meaningful amount of porting done. Future PRs should be simpler. <b>Summary of changes:</b><details> - Remove old deps: - boringssl (replaced with mbedtls, lighter, easier to build) - gflags (replaced with absl::flags) - Chromium build tools - New deps to replace parts of Chromium base: - abseil-cpp - glog - nlohmann::json (for tests only) - Submodules, updates, and CMake build rules for third-party libraries: - curl - gmock/gtest - Ported internal libraries and their tests by removing Chromium deps and adding CMake build rules: - file (now using C++17 filesystem APIs) - license_notice - status - version - Test improvements - Removed file tests that can never be re-enabled - Re-enabled all other disabled file tests - Debug JSON values when HTTP tests fail - Fixed chunked-encoding issues in HTTP tests - Updated and refactored Dockerfiles testing - All docker files working, with OS versions updated to meet the new tool requirements - Local docker builds no longer write files to your working directory as root - Local docker builds can now be run in parallel without clobbering each others' build outputs - DEBUG=1 can drop you into an interactive shell when a docker build fails - Updated and heavily refactored workflows and Dockerfiles - All docker files now tested in parallel on GitHub, speeding up CI - All common workflow components broken out and using workflow_call instead of custom actions - Self-hosted runners now optional, to make testing easier on forks - CMake porting works-in-process can now be fully tested on GitHub - Building ported libraries and passing ported tests on all three platforms! - CI hacks for macOS removed, now testing on macos-latest! - Python2 no longer required! (Only Python3) - Using strict build flags, treating all warnings as errors. </details> <b>Required to build:</b> - CMake >= 3.16 - Python 3 - A compiler supporting C++ >= 17 - g++ >= 9 if using GCC (Clang also fine) - MSVC for Windows <b>Still needs work:</b><details> - Moving other dependencies into submodules (if we keep them): - apple_apsl - icu - libevent - libpng - libwebm - libxml - modp_b64 - protobuf - zlib - Port remaining internal libraries: - app - hls - media/base - media/chunking - media/codecs - media/crypto - media/demuxer - media/event - media/formats/dvb - media/formats/mp2t - media/formats/mp4 - media/formats/packed_audio - media/formats/ttml - media/formats/webm - media/formats/webvtt - media/formats/wvm - media/origin - media/public - media/replicator - media/trick_play - mpd - Port main application - Add logging flags in absl and connect them to glog (which expects gflags) - Port pssh-box.py - Port main test targets (packager_test.py and packager_app.py) - Updating all requirement and build documentation - Remove any remaining refs to gclient, depot_tools, ninja - Update and complete release workflows using release-please </details> Issue #346 (Switch to abseil) Issue #1047 (New build system)
2022-08-16 18:34:51 +00:00
# A reusable workflow to build and test Packager on every supported OS and
# architecture.
name: Build
# Runs when called from another workflow.
on:
workflow_call:
inputs:
ref:
required: true
type: string
secrets:
# The GITHUB_TOKEN name is reserved, but not passed through implicitly.
# So we call our secret parameter simply TOKEN.
TOKEN:
required: false
# These below are not actual secrets, but secrets are the only place to
# keep repo-specific configs that make this project friendlier to forks
# and easier to debug.
# If non-empty, start a debug SSH server on failures.
ENABLE_DEBUG:
required: false
# If non-empty, enable self-hosted runners in the build matrix.
ENABLE_SELF_HOSTED:
required: false
# By default, run all commands in a bash shell. On Windows, the default would
# otherwise be powershell.
defaults:
run:
shell: bash
jobs:
# Configure the build matrix based on inputs. The list of objects in the
# build matrix contents can't be changed by conditionals, but it can be
# computed by another job and deserialized. This uses
# secrets.ENABLE_SELF_HOSTED to determine the build matrix, based on the
# metadata in build-matrix.json.
matrix_config:
runs-on: ubuntu-latest
outputs:
INCLUDE: ${{ steps.configure.outputs.INCLUDE }}
OS: ${{ steps.configure.outputs.OS }}
ENABLE_DEBUG: ${{ steps.configure.outputs.ENABLE_DEBUG }}
steps:
- uses: actions/checkout@v2
with:
ref: ${{ inputs.ref }}
- name: Configure Build Matrix
id: configure
shell: node {0}
run: |
const enableDebug = "${{ secrets.ENABLE_DEBUG }}" != '';
const enableSelfHosted = "${{ secrets.ENABLE_SELF_HOSTED }}" != '';
// Use ENABLE_SELF_HOSTED to decide what the build matrix below
// should include.
const {hosted, selfHosted} = require("${{ github.workspace }}/.github/workflows/build-matrix.json");
const include = enableSelfHosted ? hosted.concat(selfHosted) : hosted;
const os = include.map((config) => config.os);
// Output JSON objects consumed by the build matrix below.
console.log(`::set-output name=INCLUDE::${ JSON.stringify(include) }`);
console.log(`::set-output name=OS::${ JSON.stringify(os) }`);
// Output the debug flag directly.
console.log(`::set-output name=ENABLE_DEBUG::${ enableDebug }`);
// Log the outputs, for the sake of debugging this script.
console.log({enableDebug, include, os});
build:
needs: matrix_config
strategy:
fail-fast: false
matrix:
include: ${{ fromJSON(needs.matrix_config.outputs.INCLUDE) }}
os: ${{ fromJSON(needs.matrix_config.outputs.OS) }}
build_type: ["Debug", "Release"]
lib_type: ["static", "shared"]
name: ${{ matrix.os_name }} ${{ matrix.target_arch }} ${{ matrix.build_type }} ${{ matrix.lib_type }}
runs-on: ${{ matrix.os }}
steps:
- name: Configure git to preserve line endings
# Otherwise, tests fail on Windows because "golden" test outputs will not
# have the correct line endings.
run: git config --global core.autocrlf false
- name: Checkout code
uses: actions/checkout@v2
with:
ref: ${{ inputs.ref }}
submodules: true
- name: Install Linux deps
if: runner.os == 'Linux'
run: sudo apt install -y libc-ares-dev
- name: Generate build files
run: |
mkdir -p build/
if [[ "${{ matrix.lib_type }}" == "shared" ]]; then
LIBPACKAGER_SHARED="ON"
else
LIBPACKAGER_SHARED="OFF"
fi
cmake \
-DCMAKE_BUILD_TYPE="${{ matrix.build_type }}" \
-DLIBPACKAGER_SHARED="$LIBPACKAGER_SHARED" \
-S . \
-B build/
- name: Build
# This is a universal build command, which will call make on Linux and
# Visual Studio on Windows. Note that the VS generator is what cmake
# calls a "multi-configuration" generator, and so the desired build
# type must be specified for Windows.
run: cmake --build build/ --config "${{ matrix.build_type }}"
- name: Test
run: cd build; ctest -C "${{ matrix.build_type }}" -V
# TODO(joeyparrish): Prepare artifacts when build system is complete again
# - name: Prepare artifacts (static release only)
# run: |
# BUILD_CONFIG="${{ matrix.build_type }}-${{ matrix.lib_type }}"
# if [[ "$BUILD_CONFIG" != "Release-static" ]]; then
# echo "Skipping artifacts for $BUILD_CONFIG."
# exit 0
# fi
# if [[ "${{ runner.os }}" == "Linux" ]]; then
# echo "::group::Check for static executables"
# (
# cd build/Release
# # Prove that we built static executables on Linux. First, check that
# # the executables exist, and fail if they do not. Then check "ldd",
# # which will fail if the executable is not dynamically linked. If
# # "ldd" succeeds, we fail the workflow. Finally, we call "true" so
# # that the last executed statement will be a success, and the step
# # won't be failed if we get that far.
# ls packager mpd_generator >/dev/null || exit 1
# ldd packager 2>&1 && exit 1
# ldd mpd_generator 2>&1 && exit 1
# true
# )
# echo "::endgroup::"
# fi
# echo "::group::Prepare artifacts folder"
# mkdir artifacts
# ARTIFACTS="$GITHUB_WORKSPACE/artifacts"
# cd build/Release
# echo "::endgroup::"
# echo "::group::Strip executables"
# strip packager${{ matrix.exe_ext }}
# strip mpd_generator${{ matrix.exe_ext }}
# echo "::endgroup::"
# SUFFIX="-${{ matrix.os_name }}-${{ matrix.target_arch }}"
# EXE_SUFFIX="$SUFFIX${{ matrix.exe_ext}}"
# echo "::group::Copy packager"
# cp packager${{ matrix.exe_ext }} $ARTIFACTS/packager$EXE_SUFFIX
# echo "::endgroup::"
# echo "::group::Copy mpd_generator"
# cp mpd_generator${{ matrix.exe_ext }} $ARTIFACTS/mpd_generator$EXE_SUFFIX
# echo "::endgroup::"
# # The pssh-box bundle is OS and architecture independent. So only do
# # it on this one OS and architecture, and give it a more generic
# # filename.
# if [[ '${{ matrix.os_name }}' == 'linux' && '${{ matrix.target_arch }}' == 'x64' ]]; then
# echo "::group::Tar pssh-box"
# tar -czf $ARTIFACTS/pssh-box.py.tar.gz pyproto pssh-box.py
# echo "::endgroup::"
# fi
# TODO(joeyparrish): Attach artifacts when build system is complete again
# - name: Attach artifacts to release
# if: matrix.build_type == 'Release' && matrix.lib_type == 'static'
# uses: dwenegar/upload-release-assets@v1
# env:
# GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# with:
# release_id: ${{ needs.draft_release.outputs.release_id }}
# assets_path: artifacts
- name: Debug
uses: mxschmitt/action-tmate@v3.6
with:
limit-access-to-actor: true
if: failure() && needs.matrix_config.outputs.ENABLE_DEBUG != ''