The default .npmignore would disallow bin/, which contains binaries for
our NPM release. Every release before v3.0.3 was missing .npmignore, and
therefore had no binaries. These were unusable in NPM, and have been
marked as deprecated because of it.
Closes#1369
The release workflows did not run checkout with `fetch-tags: true`, so
the builds were unable to compute the correct release version number. I
audited all instances of `actions/checkout` to add `fetch-tags` where
needed and clean up unneeded options.
I also had to fix options to `docker/build-push-action`, which by
default ignores `actions/checkout` and tries to pull from git itself.
This led to the Docker build running in a context without the new tag.
Finally, to make verification easier and provide version info in the
build logs, this adds debugging info to the version-generation script
via stderr.
Closes#1366
Support for this came out in January, 2024. Explicit macos versions seem
to be necessary for now, until GitHub offers "latest" labels targeting
specific architectures.
This adds the option FULLY_STATIC to create fully-static executables.
To create portable, fully-static release executables on Linux, we need
to use musl instead of glibc. Static executables from glibc are not
portable.
The popular musl-gcc wrapper does not support C++, so instead we use a
full musl cross-compiler toolchain in the build workflow.
To build FULLY_STATIC, the user must point to the appropriate
cross-compiler, as we do in the workflow. On systems where musl is the
native libc (such as Alpine Linux), this is not necessary.
I have also read that musl's allocator is not very fast in
multi-threaded applications. So when FULLY_STATIC is enabled, we will
also enable mimalloc, a replacement allocator that is very fast.
I tested a very basic packaging command to compare speeds of dynamic
glibc, static musl, and static musl+mimalloc:
dynamic glibc:
runs: 2.527, 2.798, 2.703, 2.756, 2.959
avg = 2.749, std dev = 0.156s
static musl:
runs: 2.813, 2.920, 3.129, 3.003, 2.738
avg = 2.921s, std dev = 0.154s
static musl+mimalloc:
runs: 2.291, 2.034, 2.415, 2.303, 2.265
avg = 2.262s, std dev = 0.140s
The mimalloc build is 82% faster than musl default allocator, 77% faster
than glibc, and has more consistent runtime characteristics (lower
standard deviation).
According to a comment in
packager/third_party/abseil-cpp/source/absl/log/CMakeLists.txt, many
linkers will strip the contents of absl::log_flags because its symbols
symbols are only used in a global constructor, and that for now, clients
should link using
$<LINK_LIBRARY:WHOLE_ARCHIVE,absl::log_flags>.
Closes#1325
This work was done over ~80 individual commits in the `cmake` branch,
which are now being merged back into `main`. As a roll-up commit, it is
too big to be reviewable, but each change was reviewed individually in
context of the `cmake` branch. After this, the `cmake` branch will be
renamed `cmake-porting-history` and preserved.
---------
Co-authored-by: Geoff Jukes <geoffjukes@users.noreply.github.com>
Co-authored-by: Bartek Zdanowski <bartek.zdanowski@gmail.com>
Co-authored-by: Carlos Bentzen <cadubentzen@gmail.com>
Co-authored-by: Dennis E. Mungai <2356871+Brainiarc7@users.noreply.github.com>
Co-authored-by: Cosmin Stejerean <cstejerean@gmail.com>
Co-authored-by: Carlos Bentzen <carlos.bentzen@bitmovin.com>
Co-authored-by: Cosmin Stejerean <cstejerean@meta.com>
Co-authored-by: Cosmin Stejerean <cosmin@offbytwo.com>
This is an automated sync of common workflows for this organization.
The upstream source is:
4245160a0b
Co-authored-by: Shaka Bot <shaka-bot@users.noreply.github.com>
This is an automated sync of common workflows for this organization.
The upstream source is:
208556c5ea
Co-authored-by: Shaka Bot <shaka-bot@users.noreply.github.com>
This brings some workflow improvements and fixes from the `cmake` branch
to `main`, as well as some unique fixes to keep gclient working, so that
we can continue to accept contributions in `main` until the `cmake`
merge is ready.
- Fix docs build in GitHub Actions (from `cmake` branch)
- Cancel workflow when a PR is updated (from `cmake` branch)
- Fix docker failures caused by running as root (from `cmake` branch)
- Work around exception in depot_tools on Windows
- Use Windows 2019 images in GitHub Actions for compatibility with gyp
- Remove Docker build on ArchLinux, which no longer supports python2 at
all
- (NOTE: The `cmake` branch is still building on ArchLinux. Docker
builds for Arch will be restored to the `main` branch when the `cmake`
branch is finally merged to `main`.)
This is an automated sync of common workflows for this organization.
The upstream source is:
8bfe75f0d2
Co-authored-by: Shaka Bot <shaka-bot@users.noreply.github.com>
Using the latest depot_tools no longer works. depot_tools also wants
to auto-update itself, which must now be disabled.
We also need to disable the copy of python (vpython) included in
depot_tools, since for some distros, it has dependencies on system
libraries that no longer exist.
Finally, we need to force some distros to use python 2, because our
build system is ancient and needs to be ripped out and replaced some
day soon.
This fixes build issues in our CI, our Dockerfiles, and in general on
certain platforms or distros.
Closes#1023
This is an automated sync of common workflows for this organization.
The upstream source is:
b39597e92d
Co-authored-by: Shaka Bot <shaka-bot@users.noreply.github.com>
This is an automated sync of common workflows for this organization.
The upstream source is:
9517df8f73
Co-authored-by: Shaka Bot <shaka-bot@users.noreply.github.com>
This is an automated sync of common workflows for this organization.
The upstream source is:
aa5e38eb86
Co-authored-by: Shaka Bot <shaka-bot@users.noreply.github.com>
The official, static-linked linux builds were crashing in their use of
getaddrinfo, which libcurl was configured to use. Both getaddrinfo
and all of its alternatives available in glibc fail with static
linking.
We can fix this by configuring libcurl to use libc-ares on Linux
instead. This allows us to keep the benefits of a statically-linked
Linux binary.
Closes#996
Change-Id: Ib4a9eb939813fd165727788726459ef4adf3fc4d
It was suggested in code review for another project that we update the
runner labels for clarity. This brings Packager in line with that, so
that we are using the same labels across projects.
The runners have already been updated to register with the new label.
Change-Id: I30b22530225b5bd22b965ba98d276bcd74ade6cf
Now that we have multiple architectures, we should factor both OS and
architecture into the names of release binaries. This makes the names
more formulaic, as well as consistent with the static-ffmpeg-binaries
repository. Shaka Streamer will pull binaries from both this repo and
that one, so consistent names would be helpful.
The pssh-box release is actually OS and architecture independent, so
remove the suffix from that and only release one copy of it.
Change-Id: Ief3de49fae267c5267647a8dd4377023777ead37
This was causing failures on arm64, where the build action had an
arm-specific clause that was skipped due to the missing parameter.
Change-Id: I71b7fb15120855c444749dc2216b5f19f0561f6e
We never produced static release executables on Linux before, but the dynamic libraries they depended on were universal enough that nobody noticed. Now that we have released v2.5 and switched to GitHub Actions for CI builds, the Linux executables depend on libatomic, which is causing issues for some users.
Although we can't create fully-static executables on macOS or Windows, we can at least do so on Linux.
This adds a GYP variable static_link_binaries which can be set to request full-static binaries on Linux. This also exposes the Chromium build variable disable_fatal_linker_warnings, which is necessary when static linking on Linux due to static-link-related warnings generated by libcurl for its use of getaddrinfo. Finally, this enforces the definition of __UCLIBC__ with static linking on Linux, which is the only way to disable malloc hooks in Chromium base. Those hooks cause linker failures when linking statically on Linux.
A new check has been added to the release workflow to ensure that the builds we create are statically linked on Linux.
Closes#965
There is not a good reason to use a long-lived token attached to
shaka-bot. Instead, use a short-lived, automatic token generated by
GitHub Actions for the workflow run.
This brings our default build config more in line with what is
necessary for some platforms anyway: using the system-installed
toolchain and sysroot to build everything.
We will no longer fetch source or binaries for any specific build
tools, such as libc++, clang, gold, binutils, or valgrind.
The main part of this change is the changing of default gyp settings
in gyp_packager.py. For this, a bug in gyp_packager.py had to be
fixed, in which similar GYP_DEFINE key names (such as clang and
host_clang) would conflict, causing some defaults not to be installed
properly.
In order to enable clang=0 by default, some changes had to be made in
common.gypi:
- compiler macros added to fix a compatibility issue between
Chromium's base/mac/ folder and the actual OSX SDK
- replaced clang_warning_flags variables with standard cflags
settings, plus xcode_settings for OSX
- turned off warnings-as-errors for non-shaka code, rather than
allow-listing specific warning types, since we can't actually fix
those warnings on any platform
- disabled two specific warnings in shaka code, both of which are
caused by headers from our non-shaka dependencies
Also, one warning (missing "override" keyword) has been fixed in
vod_media_info_dump_muxer_listener.h.
Although these changes were done to make building simpler on a wider
array of platforms (arm64, for example), it seems to make the build a
bit faster, too. For me, at least, on my main Linux workstation:
- "gclient sync" now runs 20-30% faster
- "ninja -C out/Release" now runs 5-13% faster
The following environment variables are no longer required:
- DEPOT_TOOLS_WIN_TOOLCHAIN
- MACOSX_DEPLOYMENT_TARGET
Documentation, Dockerfiles, and GitHub Actions workflows have been
updated to reflect this.
The following GYP_DEFINES are no longer required for anyone:
- clang=0
- host_clang=0
- clang_xcode=1
- use_allocator=none
- use_experimental_allocator_shim=0
Documentation, Dockerfiles, and GitHub Actions workflows have been
updated to reflect this.
The following repos are no longer dependencies in gclient:
- binutils
- clang
- gold
- libc++
- libc++abi
- valgrind
The following gclient hooks have been removed:
- clang
- mac_toolchain
- sysroot
Change-Id: Ie94ccbeec722ab73c291cb7df897d20761a09a70
Internal CI systems and the new GitHub CI system were out of sync,
with the external system not doing any linting. Further, the internal
system was using an internal-only linter for Python.
This creates a script for Python linting based on the open-source
pylint tool, checks in the Google Style Guide's pylintrc file, creates
a custom action for linting and adds it to the existing workflows,
fixes pre-existing linter errors in Python scripts, and updates pylint
overrides.
b/190743862
Change-Id: Iff1f5d4690b32479af777ded0834c31c2161bd10
Testing CI workflows is a pain. This usually involves forking the
main repo and testing various operations there, where the results will
not break the main repo.
However, some things like NPM and Docker package names were initially
hard-coded. This meant that a fork would need to customize those in
the workflows to avoid pushing official-looking packages during CI
testing.
This change moves those hard-coded names to GitHub Secrets. Though
the names are not actually secret, the secret store is per-repo, and
will be independent in a fork. This makes it easier to avoid
accidentally pushing official-looking releases during testing, even if
the fork has access to the same auth tokens.
Change-Id: Ide8f7aa92a028dd217200fca60881333bf8ae579
It turns out that workflows were the wrong way to abstract reusable
pieces of work. This turns common steps into custom actions (build
docs, build packager, test packager) which can be used as encapsulated
steps in multiple workflows.
This is a much more natural way to avoid duplication compared to the
previous approach of triggering one workflow from another. This also
has the benefit of all of the steps of a release being represented on
GitHub as a single workflow, making it easier to understand what is
happening and what event triggered those steps.
Change-Id: Ife156d60069a39594c7b3bb3bc32080e6453b544
- Document necessary repo secrets
- Compress build artifacts directly to the arifacts folder
- Log test commands as they are executed
- Add comments
Change-Id: I1cc150995d339e2e93bee4570d80263dae362bb9
This replaces Travis (for Linux & Mac) and Appveyor (for Windows) with
GitHub Actions. In addition to using GitHub Actions to test PRs, this
also expands the automation of releases so that the only manual steps
are:
1. Create a new CHANGELOG.md entry
2. Create a release tag
Workflows have been create for building and testing PRs and releases,
for publishing releases to GitHub, NPM, and Docker Hub, and for
updating documentation on GitHub Pages.
When a new PR is created, GitHub Actions will:
- Build and test on all combinations of OS, release type, and library
type
Appveyor's workflow took ~2 hours, whereas the new GitHub Actions
workflow takes ~30 minutes.
When a new release tag is created, GitHub Actions will:
- Create a draft release on GitHub
- Extract release notes from CHANGELOG.md & attach them to the
draft release
- Build and test on all combinations of OS, release type, and library
type, aborting if any build or test fails
- Attach release artifacts to the draft release, aborting if any
one artifact can't be prepared
- Fully publish the draft release on GitHub
- Publish the same release to NPM (triggered by GitHub release)
- Publish the same release to Docker Hub (triggered by GitHub release)
- Update the docs on GitHub pages
Closes#336 (GitHub Actions workflow to replace Travis and Appveyor)
b/190743862 (internal; tracking replacement of Travis)
Change-Id: Ic53eef60a8587c5d1487769a0cefaa16eb9b46e7
Cosmin Stejerean
Joey Parrish
Shaka Bot
Joey Parrish
KongQun Yang