052fb64068
Our workflows and Dockerfiles now use Ninja on Linux & Mac, which enables safe parallel builds. This significantly speeds up our workflows. GitHub Actions typical compilation times (build step only): - Linux 19m => 9m - macOS 23m => 8m - Windows 12m => 10m - Linux arm64 (self-hosted) 72m => 29m - Docker build 25m => 14m Overall workflow time: 84m => 33m Compilation time on my workstation (12 CPUs @3.3GHz): 15m => 3m This also adds a new environment variable "PACKAGER_LOW_MEMORY_BUILD". If defined when CMake is first run, this will configure the build to disable parallel linking to reduce memory usage. This helps us avoid failures on our self-hosted arm64 machines, where 6 CPUs share 4GB of RAM. NOTE: Parallel builds are **NOT** recommended with Unix Makefiles due to the use of excessive RAM during parallel linking. Unix Makefiles, unlike Ninja, cannot be configured to restrict parallel linking during a parallel build. Anecdotally, parallel builds with Makefiles have exhausted a system with 32GB RAM. (My workstation.) In a follow-up, I will update the build documentation to refer to CMake and recommend all of the flags now used in our workflows. |
||
|---|---|---|
| .. | ||
| README.md | ||
| build-docs.yaml | ||
| build-matrix.json | ||
| build.yaml | ||
| docker-hub-release.yaml | ||
| docker-image.yaml | ||
| github-release.yaml | ||
| lint.yaml | ||
| npm-release.yaml | ||
| pr.yaml | ||
| settings.yaml | ||
| sync-labels.yaml | ||
| test-linux-distros.yaml | ||
| update-docs.yaml | ||
| update-issues.yaml | ||
| validate-pr-title.yaml | ||
README.md
GitHub Actions CI
Reusable workflows
-
build.yaml: Build and test all combinations of OS & build settings. Also builds docs on Linux. -
build-docs.yaml: Build Packager docs. Runs only on Linux. -
docker-image.yaml: Build the official Docker image. -
lint.yaml: Lint Shaka Packager. -
test-linux-distros.yaml: Test the build on all Linux distros via docker.
Composed workflows
-
On PR (
pr.yaml), invoke:lint.yamlbuild.yamlbuild-docs.yamldocker-image.yamltest-linux-distros.yaml
-
On release tag (
github-release.yaml):- Create a draft release
- Invoke:
lint.yamlbuild.yamltest-linux-distros.yaml
- Publish the release with binaries from
build.yamlattached
-
On release published:
docker-hub-release.yaml, publishes the official Docker imagenpm-release.yaml, publishes the official NPM packageupdate-docs.yaml:- Invoke
build-docs.yaml - Push the output to the
gh-pagesbranch
- Invoke
Common workflows from shaka-project
sync-labels.yamlupdate-issues.yamlvalidate-pr-title.yaml
Required Repo Secrets
DOCKERHUB_CI_USERNAME: The username of the Docker Hub CI accountDOCKERHUB_CI_TOKEN: An access token for Docker Hub- To generate, visit https://hub.docker.com/settings/security
DOCKERHUB_PACKAGE_NAME: Not a true "secret", but stored here to avoid someone pushing bogus packages to Docker Hub during CI testing from a fork- In a fork, set to a private name which differs from the production one
NPM_CI_TOKEN: An "Automation"-type access token for NPM for theshaka-botaccount- To generate, visit https://www.npmjs.com/settings/shaka-bot/tokens and select the "Automation" type
NPM_PACKAGE_NAME: Not a true "secret", but stored here to avoid someone pushing bogus packages to NPM during CI testing from a fork- In a fork, set to a private name which differs from the production one
Repo Settings
Each of these workflow features can be enabled by creating a "GitHub Environment" with the same name in your repo settings. Forks will not have these enabled by default.
debug: enable debugging via SSH after a failureself_hosted: enable self-hosted runners in the build matrix