Mirror-Tools
/
shaka-packager
mirror of https://github.com/shaka-project/shaka-packager.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
shaka-packager/.github/workflows/README.md

67 lines
2.2 KiB
Markdown
Raw Normal View History

Address PR feedback - Document necessary repo secrets - Compress build artifacts directly to the arifacts folder - Log test commands as they are executed - Add comments Change-Id: I1cc150995d339e2e93bee4570d80263dae362bb9
2021-06-16 21:15:08 +00:00
# GitHub Actions CI
feat: First phase of CMake build system implementation (#1072) There are a lot of changes in this first phase, because there was a lot of infrastructure required to get some meaningful amount of porting done. Future PRs should be simpler. <b>Summary of changes:</b><details> - Remove old deps: - boringssl (replaced with mbedtls, lighter, easier to build) - gflags (replaced with absl::flags) - Chromium build tools - New deps to replace parts of Chromium base: - abseil-cpp - glog - nlohmann::json (for tests only) - Submodules, updates, and CMake build rules for third-party libraries: - curl - gmock/gtest - Ported internal libraries and their tests by removing Chromium deps and adding CMake build rules: - file (now using C++17 filesystem APIs) - license_notice - status - version - Test improvements - Removed file tests that can never be re-enabled - Re-enabled all other disabled file tests - Debug JSON values when HTTP tests fail - Fixed chunked-encoding issues in HTTP tests - Updated and refactored Dockerfiles testing - All docker files working, with OS versions updated to meet the new tool requirements - Local docker builds no longer write files to your working directory as root - Local docker builds can now be run in parallel without clobbering each others' build outputs - DEBUG=1 can drop you into an interactive shell when a docker build fails - Updated and heavily refactored workflows and Dockerfiles - All docker files now tested in parallel on GitHub, speeding up CI - All common workflow components broken out and using workflow_call instead of custom actions - Self-hosted runners now optional, to make testing easier on forks - CMake porting works-in-process can now be fully tested on GitHub - Building ported libraries and passing ported tests on all three platforms! - CI hacks for macOS removed, now testing on macos-latest! - Python2 no longer required! (Only Python3) - Using strict build flags, treating all warnings as errors. </details> <b>Required to build:</b> - CMake >= 3.16 - Python 3 - A compiler supporting C++ >= 17 - g++ >= 9 if using GCC (Clang also fine) - MSVC for Windows <b>Still needs work:</b><details> - Moving other dependencies into submodules (if we keep them): - apple_apsl - icu - libevent - libpng - libwebm - libxml - modp_b64 - protobuf - zlib - Port remaining internal libraries: - app - hls - media/base - media/chunking - media/codecs - media/crypto - media/demuxer - media/event - media/formats/dvb - media/formats/mp2t - media/formats/mp4 - media/formats/packed_audio - media/formats/ttml - media/formats/webm - media/formats/webvtt - media/formats/wvm - media/origin - media/public - media/replicator - media/trick_play - mpd - Port main application - Add logging flags in absl and connect them to glog (which expects gflags) - Port pssh-box.py - Port main test targets (packager_test.py and packager_app.py) - Updating all requirement and build documentation - Remove any remaining refs to gclient, depot_tools, ninja - Update and complete release workflows using release-please </details> Issue #346 (Switch to abseil) Issue #1047 (New build system)
2022-08-16 18:34:51 +00:00
## Reusable workflows
- `build.yaml`:
Build and test all combinations of OS & build settings. Also builds docs on
Linux.
- `build-docs.yaml`:
Build Packager docs. Runs only on Linux.
- `docker-image.yaml`:
Build the official Docker image.
- `lint.yaml`:
Lint Shaka Packager.
- `test-linux-distros.yaml`:
Test the build on all Linux distros via docker.
## Composed workflows
- On PR (`pr.yaml`), invoke:
- `lint.yaml`
- `build.yaml`
- `build-docs.yaml`
- `docker-image.yaml`
- `test-linux-distros.yaml`
- On release tag (`github-release.yaml`):
- Create a draft release
- Invoke:
- `lint.yaml`
- `build.yaml`
- `test-linux-distros.yaml`
- Publish the release with binaries from `build.yaml` attached
Refactor actions and workflows It turns out that workflows were the wrong way to abstract reusable pieces of work. This turns common steps into custom actions (build docs, build packager, test packager) which can be used as encapsulated steps in multiple workflows. This is a much more natural way to avoid duplication compared to the previous approach of triggering one workflow from another. This also has the benefit of all of the steps of a release being represented on GitHub as a single workflow, making it easier to understand what is happening and what event triggered those steps. Change-Id: Ife156d60069a39594c7b3bb3bc32080e6453b544
2021-06-17 04:32:05 +00:00
- On release published:
feat: First phase of CMake build system implementation (#1072) There are a lot of changes in this first phase, because there was a lot of infrastructure required to get some meaningful amount of porting done. Future PRs should be simpler. <b>Summary of changes:</b><details> - Remove old deps: - boringssl (replaced with mbedtls, lighter, easier to build) - gflags (replaced with absl::flags) - Chromium build tools - New deps to replace parts of Chromium base: - abseil-cpp - glog - nlohmann::json (for tests only) - Submodules, updates, and CMake build rules for third-party libraries: - curl - gmock/gtest - Ported internal libraries and their tests by removing Chromium deps and adding CMake build rules: - file (now using C++17 filesystem APIs) - license_notice - status - version - Test improvements - Removed file tests that can never be re-enabled - Re-enabled all other disabled file tests - Debug JSON values when HTTP tests fail - Fixed chunked-encoding issues in HTTP tests - Updated and refactored Dockerfiles testing - All docker files working, with OS versions updated to meet the new tool requirements - Local docker builds no longer write files to your working directory as root - Local docker builds can now be run in parallel without clobbering each others' build outputs - DEBUG=1 can drop you into an interactive shell when a docker build fails - Updated and heavily refactored workflows and Dockerfiles - All docker files now tested in parallel on GitHub, speeding up CI - All common workflow components broken out and using workflow_call instead of custom actions - Self-hosted runners now optional, to make testing easier on forks - CMake porting works-in-process can now be fully tested on GitHub - Building ported libraries and passing ported tests on all three platforms! - CI hacks for macOS removed, now testing on macos-latest! - Python2 no longer required! (Only Python3) - Using strict build flags, treating all warnings as errors. </details> <b>Required to build:</b> - CMake >= 3.16 - Python 3 - A compiler supporting C++ >= 17 - g++ >= 9 if using GCC (Clang also fine) - MSVC for Windows <b>Still needs work:</b><details> - Moving other dependencies into submodules (if we keep them): - apple_apsl - icu - libevent - libpng - libwebm - libxml - modp_b64 - protobuf - zlib - Port remaining internal libraries: - app - hls - media/base - media/chunking - media/codecs - media/crypto - media/demuxer - media/event - media/formats/dvb - media/formats/mp2t - media/formats/mp4 - media/formats/packed_audio - media/formats/ttml - media/formats/webm - media/formats/webvtt - media/formats/wvm - media/origin - media/public - media/replicator - media/trick_play - mpd - Port main application - Add logging flags in absl and connect them to glog (which expects gflags) - Port pssh-box.py - Port main test targets (packager_test.py and packager_app.py) - Updating all requirement and build documentation - Remove any remaining refs to gclient, depot_tools, ninja - Update and complete release workflows using release-please </details> Issue #346 (Switch to abseil) Issue #1047 (New build system)
2022-08-16 18:34:51 +00:00
- `docker-hub-release.yaml`, publishes the official Docker image
- `npm-release.yaml`, publishes the official NPM package
- `update-docs.yaml`:
- Invoke `build-docs.yaml`
- Push the output to the `gh-pages` branch
## Common workflows from shaka-project
- `sync-labels.yaml`
- `update-issues.yaml`
- `validate-pr-title.yaml`
Address PR feedback - Document necessary repo secrets - Compress build artifacts directly to the arifacts folder - Log test commands as they are executed - Add comments Change-Id: I1cc150995d339e2e93bee4570d80263dae362bb9
2021-06-16 21:15:08 +00:00
## Required Repo Secrets
- `DOCKERHUB_CI_USERNAME`: The username of the Docker Hub CI account
- `DOCKERHUB_CI_TOKEN`: An access token for Docker Hub
- To generate, visit https://hub.docker.com/settings/security
Make CI workflows safe when testing in a fork Testing CI workflows is a pain. This usually involves forking the main repo and testing various operations there, where the results will not break the main repo. However, some things like NPM and Docker package names were initially hard-coded. This meant that a fork would need to customize those in the workflows to avoid pushing official-looking packages during CI testing. This change moves those hard-coded names to GitHub Secrets. Though the names are not actually secret, the secret store is per-repo, and will be independent in a fork. This makes it easier to avoid accidentally pushing official-looking releases during testing, even if the fork has access to the same auth tokens. Change-Id: Ide8f7aa92a028dd217200fca60881333bf8ae579
2021-06-17 20:36:56 +00:00
- `DOCKERHUB_PACKAGE_NAME`: Not a true "secret", but stored here to avoid
someone pushing bogus packages to Docker Hub during CI testing from a fork
- In a fork, set to a private name which differs from the production one
Address PR feedback - Document necessary repo secrets - Compress build artifacts directly to the arifacts folder - Log test commands as they are executed - Add comments Change-Id: I1cc150995d339e2e93bee4570d80263dae362bb9
2021-06-16 21:15:08 +00:00
- `NPM_CI_TOKEN`: An "Automation"-type access token for NPM for the `shaka-bot`
account
- To generate, visit https://www.npmjs.com/settings/shaka-bot/tokens and
select the "Automation" type
Make CI workflows safe when testing in a fork Testing CI workflows is a pain. This usually involves forking the main repo and testing various operations there, where the results will not break the main repo. However, some things like NPM and Docker package names were initially hard-coded. This meant that a fork would need to customize those in the workflows to avoid pushing official-looking packages during CI testing. This change moves those hard-coded names to GitHub Secrets. Though the names are not actually secret, the secret store is per-repo, and will be independent in a fork. This makes it easier to avoid accidentally pushing official-looking releases during testing, even if the fork has access to the same auth tokens. Change-Id: Ide8f7aa92a028dd217200fca60881333bf8ae579
2021-06-17 20:36:56 +00:00
- `NPM_PACKAGE_NAME`: Not a true "secret", but stored here to avoid someone
pushing bogus packages to NPM during CI testing from a fork
- In a fork, set to a private name which differs from the production one
feat: First phase of CMake build system implementation (#1072) There are a lot of changes in this first phase, because there was a lot of infrastructure required to get some meaningful amount of porting done. Future PRs should be simpler. <b>Summary of changes:</b><details> - Remove old deps: - boringssl (replaced with mbedtls, lighter, easier to build) - gflags (replaced with absl::flags) - Chromium build tools - New deps to replace parts of Chromium base: - abseil-cpp - glog - nlohmann::json (for tests only) - Submodules, updates, and CMake build rules for third-party libraries: - curl - gmock/gtest - Ported internal libraries and their tests by removing Chromium deps and adding CMake build rules: - file (now using C++17 filesystem APIs) - license_notice - status - version - Test improvements - Removed file tests that can never be re-enabled - Re-enabled all other disabled file tests - Debug JSON values when HTTP tests fail - Fixed chunked-encoding issues in HTTP tests - Updated and refactored Dockerfiles testing - All docker files working, with OS versions updated to meet the new tool requirements - Local docker builds no longer write files to your working directory as root - Local docker builds can now be run in parallel without clobbering each others' build outputs - DEBUG=1 can drop you into an interactive shell when a docker build fails - Updated and heavily refactored workflows and Dockerfiles - All docker files now tested in parallel on GitHub, speeding up CI - All common workflow components broken out and using workflow_call instead of custom actions - Self-hosted runners now optional, to make testing easier on forks - CMake porting works-in-process can now be fully tested on GitHub - Building ported libraries and passing ported tests on all three platforms! - CI hacks for macOS removed, now testing on macos-latest! - Python2 no longer required! (Only Python3) - Using strict build flags, treating all warnings as errors. </details> <b>Required to build:</b> - CMake >= 3.16 - Python 3 - A compiler supporting C++ >= 17 - g++ >= 9 if using GCC (Clang also fine) - MSVC for Windows <b>Still needs work:</b><details> - Moving other dependencies into submodules (if we keep them): - apple_apsl - icu - libevent - libpng - libwebm - libxml - modp_b64 - protobuf - zlib - Port remaining internal libraries: - app - hls - media/base - media/chunking - media/codecs - media/crypto - media/demuxer - media/event - media/formats/dvb - media/formats/mp2t - media/formats/mp4 - media/formats/packed_audio - media/formats/ttml - media/formats/webm - media/formats/webvtt - media/formats/wvm - media/origin - media/public - media/replicator - media/trick_play - mpd - Port main application - Add logging flags in absl and connect them to glog (which expects gflags) - Port pssh-box.py - Port main test targets (packager_test.py and packager_app.py) - Updating all requirement and build documentation - Remove any remaining refs to gclient, depot_tools, ninja - Update and complete release workflows using release-please </details> Issue #346 (Switch to abseil) Issue #1047 (New build system)
2022-08-16 18:34:51 +00:00
## Optional Repo Secrets
- `ENABLE_DEBUG`: Set to non-empty to enable debugging via SSH after a failure
- `ENABLE_SELF_HOSTED`: Set to non-empty to enable self-hosted runners in the
build matrix