This adopts release-please to manage releases and changelogs, similar to
other shaka-project repos.
All release workflows can be run by forks by configuring repo secrets.
See docs in .github/workflows/ for details.
- Use release-please for releases, changelogs
- Convert publication jobs (docs, docker, npm) into reusable workflows
- Update workflow documentation
- Modernize docker commands
- Fix doc permissions for publication
- Update artifact handling in build workflow
- Fix paths in Dockerfile
- Fix paths and arm64 support in NPM package
- Fix install paths for PSSH tools
- Fix warnings in NPM & Docker actions
- Delete custom changelog tooling
The hack of using "secrets" to store per-repo settings was not working.
The main reason is that pull_request workflows don't have access to
secrets no matter what you do. So it was impossible to make this work
for settings like "ENABLE_SELF_HOSTED" for PR tests.
This change replaces that old hack with a new one. Now a repo owner must
create a "GitHub Environment" with the name of the setting they want to
enable. Currently supported values are "self_hosted", to add self-hosted
runners to the build/test matrix, and "debug", to start an SSH server
for debugging when a workflow fails.
Issue #1047 (CMake porting)
There are a lot of changes in this first phase, because there was a
lot of infrastructure required to get some meaningful amount of
porting done. Future PRs should be simpler.
<b>Summary of changes:</b><details>
- Remove old deps:
- boringssl (replaced with mbedtls, lighter, easier to build)
- gflags (replaced with absl::flags)
- Chromium build tools
- New deps to replace parts of Chromium base:
- abseil-cpp
- glog
- nlohmann::json (for tests only)
- Submodules, updates, and CMake build rules for third-party
libraries:
- curl
- gmock/gtest
- Ported internal libraries and their tests by removing Chromium deps
and adding CMake build rules:
- file (now using C++17 filesystem APIs)
- license_notice
- status
- version
- Test improvements
- Removed file tests that can never be re-enabled
- Re-enabled all other disabled file tests
- Debug JSON values when HTTP tests fail
- Fixed chunked-encoding issues in HTTP tests
- Updated and refactored Dockerfiles testing
- All docker files working, with OS versions updated to meet the
new tool requirements
- Local docker builds no longer write files to your working
directory as root
- Local docker builds can now be run in parallel without clobbering
each others' build outputs
- DEBUG=1 can drop you into an interactive shell when a docker
build fails
- Updated and heavily refactored workflows and Dockerfiles
- All docker files now tested in parallel on GitHub, speeding up CI
- All common workflow components broken out and using workflow_call
instead of custom actions
- Self-hosted runners now optional, to make testing easier on forks
- CMake porting works-in-process can now be fully tested on GitHub
- Building ported libraries and passing ported tests on all three
platforms!
- CI hacks for macOS removed, now testing on macos-latest!
- Python2 no longer required! (Only Python3)
- Using strict build flags, treating all warnings as errors.
</details>
<b>Required to build:</b>
- CMake >= 3.16
- Python 3
- A compiler supporting C++ >= 17
- g++ >= 9 if using GCC (Clang also fine)
- MSVC for Windows
<b>Still needs work:</b><details>
- Moving other dependencies into submodules (if we keep them):
- apple_apsl
- icu
- libevent
- libpng
- libwebm
- libxml
- modp_b64
- protobuf
- zlib
- Port remaining internal libraries:
- app
- hls
- media/base
- media/chunking
- media/codecs
- media/crypto
- media/demuxer
- media/event
- media/formats/dvb
- media/formats/mp2t
- media/formats/mp4
- media/formats/packed_audio
- media/formats/ttml
- media/formats/webm
- media/formats/webvtt
- media/formats/wvm
- media/origin
- media/public
- media/replicator
- media/trick_play
- mpd
- Port main application
- Add logging flags in absl and connect them to glog (which expects
gflags)
- Port pssh-box.py
- Port main test targets (packager_test.py and packager_app.py)
- Updating all requirement and build documentation
- Remove any remaining refs to gclient, depot_tools, ninja
- Update and complete release workflows using release-please
</details>
Issue #346 (Switch to abseil)
Issue #1047 (New build system)
There is not a good reason to use a long-lived token attached to
shaka-bot. Instead, use a short-lived, automatic token generated by
GitHub Actions for the workflow run.
Internal CI systems and the new GitHub CI system were out of sync,
with the external system not doing any linting. Further, the internal
system was using an internal-only linter for Python.
This creates a script for Python linting based on the open-source
pylint tool, checks in the Google Style Guide's pylintrc file, creates
a custom action for linting and adds it to the existing workflows,
fixes pre-existing linter errors in Python scripts, and updates pylint
overrides.
b/190743862
Change-Id: Iff1f5d4690b32479af777ded0834c31c2161bd10
Testing CI workflows is a pain. This usually involves forking the
main repo and testing various operations there, where the results will
not break the main repo.
However, some things like NPM and Docker package names were initially
hard-coded. This meant that a fork would need to customize those in
the workflows to avoid pushing official-looking packages during CI
testing.
This change moves those hard-coded names to GitHub Secrets. Though
the names are not actually secret, the secret store is per-repo, and
will be independent in a fork. This makes it easier to avoid
accidentally pushing official-looking releases during testing, even if
the fork has access to the same auth tokens.
Change-Id: Ide8f7aa92a028dd217200fca60881333bf8ae579
It turns out that workflows were the wrong way to abstract reusable
pieces of work. This turns common steps into custom actions (build
docs, build packager, test packager) which can be used as encapsulated
steps in multiple workflows.
This is a much more natural way to avoid duplication compared to the
previous approach of triggering one workflow from another. This also
has the benefit of all of the steps of a release being represented on
GitHub as a single workflow, making it easier to understand what is
happening and what event triggered those steps.
Change-Id: Ife156d60069a39594c7b3bb3bc32080e6453b544
- Document necessary repo secrets
- Compress build artifacts directly to the arifacts folder
- Log test commands as they are executed
- Add comments
Change-Id: I1cc150995d339e2e93bee4570d80263dae362bb9
Joey Parrish
Joey Parrish