c-ares (used on Linux only) was an exception to the rule of only linking
against internally-built libraries. This fixes that, to support a truly
static build of packager.
This is used to run a fairly large number of integrations tests (which are exposing some failures on the CMake branch). Let's copy these over first to enable the integration tests workflow.
Actually running integration tests in CTest is off by default (gated by `SKIP_INTEGRATION_TESTS` while we fix the tests).
pssh-box.py needs Python protos, which we were not building. It also
needs to have those protos installed properly.
This fixes generation of Python proto interfaces and fixes the install
targets to put those protos where they are needed. This also updates the
documentation to match.
The link-test targets rebuilt every time you built the project because
that is the behavior of commands attached to add_custom_target. This is
fixed by combining add_custom_command and add_custom_target.
Fixing that led to a situation where the link-test targets built on the
first build, but didn't properly rebuild when their dependencies
changed. This is fixed by using set_source_file_properties to treat
test.cc as dirty when the test-install command runs.
Updated the build docs for Linux, Mac and Windows.
Updated the Linux profiling docs to remove references to things that don't work anymore.
---------
Co-authored-by: Joey Parrish <joeyparrish@google.com>
this adds an install target which will install binaries, libraries, pkg-config and headers, along with a simple link-test program to verify the installed library and headers work.
---------
Co-authored-by: Joey Parrish <joeyparrish@google.com>
This organizes all public headers for the library into
include/ and makes sure then don't rely on any headers from
other folders.
To accomplish this, this change also refactors macros.h,
media/base/macros.h, and status/status_macros.h into macros/classes.h,
macros/compiler.h, macros/crypto.h, macros/logging.h, macros/status.h,
and public/export.h. Now the export macros from macros.h live in
include/ to keep include/ from requiring anything else.
This refactor enables an install target that includes public headers
only.
---------
Co-authored-by: Cosmin Stejerean <cstejerean@meta.com>
This deprecates --vmodule (not available in absl, mapping it to --v for
a very minimal sort of compatibility) and adds the absl::log flags
--minloglevel, --stderrthreshold, --log_backtrace_at, and --log_prefix.
Reorder headers to follow the Google C++ Style Guide:
> In dir/foo.cc or dir/foo_test.cc:
>
> 1. dir2/foo2.h.
> 2. A blank line
> 3. C system headers (more precisely: headers in angle brackets with
the .h extension), e.g., <unistd.h>, <stdlib.h>.
> 4. A blank line
> 5. C++ standard library headers (without file extension), e.g.,
<algorithm>, <cstddef>.
> 6. A blank line
> 7. Other libraries' .h files.
> 8. A blank line
> 9. Your project's .h files.
https://google.github.io/styleguide/cppguide.html#Names_and_Order_of_Includes
This feeds into efforts to create a working install target.
The order of headers is still funky, and was "fixed" by clang-format,
but in a way that doesn't exactly align with the style guide. Further
cleanup of header order is coming in a follow-up PR.
Related to issue #1047
After this it should be possible to build a working `packager` application from the `cmake` branch. Some further logging improvements may be needed to get full parity with the `main` branch in terms of ability to do verbose debug logging, but other than that everything is expected to work.
This is an automated sync of common workflows for this organization.
The upstream source is:
4245160a0b
Co-authored-by: Shaka Bot <shaka-bot@users.noreply.github.com>
As per the AV1 spec, the codec string may contain optional color values.
This extracts the missing color information from the mp4 `colr` atom, if
present, and generates the full AV1 codec string.
Closes#1007
This is an automated sync of common workflows for this organization.
The upstream source is:
208556c5ea
Co-authored-by: Shaka Bot <shaka-bot@users.noreply.github.com>
This fixes our use of std::filesystem to interpret all paths names as
UTF8. Before this, UTF8 paths did not work correctly in all cases.
This also adds a new unit test to cover this case.
On Windows, it is critical that a UTF8 locale be set at runtime.
Applications linking with Packager as a library should call setlocale(),
and the Packager frontends now do this automatically after converting
wide character arguments into narrow strings.
Closes#652
PR #1242 introduced some bad assertions that went unnoticed because we
are not yet building those modules in the cmake branch. This fixes those
so that ASSERT_ macros are not used in non-void functions, and so that
assumptions about a successful parse are always after an ASSERT_ that
the data was loaded properly.
All ASSERT_ macros from #1242 were audited, and the others were all
fine.
All gtest-based tests should now use add_gtest. Results from these tests
will be surfaced more prominently in GitHub Actions. Before this, it was
hard to find test failures in the log.
LOG(FATAL) aborts the executable, which means gtest can't write any
reports it might be configured to write when tests are complete. This
will interfere with reporting.
This converts the only use of LOG(FATAL), which was in
ReadTestDataFile(), to LOG(ERROR). This also updates test cases to avoid
crashing when ReadTestDataFile() returns an empty buffer.
Our workflows and Dockerfiles now use Ninja on Linux & Mac, which
enables safe parallel builds. This significantly speeds up our
workflows.
GitHub Actions typical compilation times (build step only):
- Linux 19m => 9m
- macOS 23m => 8m
- Windows 12m => 10m
- Linux arm64 (self-hosted) 72m => 29m
- Docker build 25m => 14m
Overall workflow time: 84m => 33m
Compilation time on my workstation (12 CPUs @3.3GHz): 15m => 3m
This also adds a new environment variable "PACKAGER_LOW_MEMORY_BUILD".
If defined when CMake is first run, this will configure the build to
disable parallel linking to reduce memory usage. This helps us avoid
failures on our self-hosted arm64 machines, where 6 CPUs share 4GB of
RAM.
NOTE: Parallel builds are **NOT** recommended with Unix Makefiles due to
the use of excessive RAM during parallel linking. Unix Makefiles, unlike
Ninja, cannot be configured to restrict parallel linking during a
parallel build. Anecdotally, parallel builds with Makefiles have
exhausted a system with 32GB RAM. (My workstation.)
In a follow-up, I will update the build documentation to refer to CMake
and recommend all of the flags now used in our workflows.
CentOS 8 is EOL, and the official docker images will not be updated. See
https://www.centos.org/centos-linux-eol/
This switches to the latest release (CentOS 9 Stream). Since CentOS no
longer provides official images, this uses a third-party image provider
(https://hub.docker.com/u/tgagor)
This change also exposed a warning in absl that needed to be silenced in
the default version of GCC in CentOS 9 Stream (GCC 11.4).
The `__FILE__` macro does not always get you an absolute path, so our
assumptions about it in test_data_util.cc were flawed.
If `foo.c` references `__FILE__`, something like `gcc -c foo.c`, will
define `__FILE__` as `"foo.c"`. If you use `gcc -c /path/to/foo.c`, then
`__FILE__` will be `"/path/to/foo.c"`.
The Ninja generator for CMake only generates absolute paths for source
files in certain CMake versions. (Exact range unknown.) Rather than
require newer CMake versions and depend on CMake's Ninja generator
maintaining the latest behavior forever, set the macro TEST_DATA_DIR to
point to the test data folder. This is consistent and never depends on
unspecified behavior.
This change will allow the use of the Ninja generator with older
versions of CMake, as found in most of our Docker builds.
- Update protobuf to v23.4, to fix the cstdint error on Arch Linux, and
make some related changes:
- Silence additional compiler warnings for the new protobuf
- Update absl to 20230125.3, to support the protobuf update, and make
some related changes:
- Silence additional compiler warnings for the new protobuf
- Replace GOOGLE_CHECK_OK with ABSL_CHECK_OK
- Replace GUARDED_BY with ABSL_GUARDED_BY
- Update other instances of cstdint missing in our own code
- Always pull the latest docker images, to avoid stale results when your
workstation has pulled an older image at the same label
Rebasing #1148 on top of current cmake branch
Related to #1047
---------
Co-authored-by: Bartek Zdanowski <bartek.zdanowski@gmail.com>
Co-authored-by: Joey Parrish <joeyparrish@google.com>
All HTTP-based tests now use an embedded test server instead of
httpbin.org, which makes them much faster and more reliable.
These more reliable tests also exposed some issues that began recently
with PR #1201. HttpFile's Flush() semantics were different than those
documented for files in general. Flush() used to close the file for
uploading, so that no further writes were allowed, but the documentation
stated that it would only flush data to its destination. PR #1201
brought HttpFile's Flush() in line with the docs, but gave us no way to
terminate a chunked upload.
This adds a new method to File called CloseForWriting(), which
terminates a chunked upload for HttpFile. The only other implementation
that does anything is UdpFile, which uses the socket library function
shutdown() to terminate writes while allowing reads.
This also tweaks HttpFile::CloseWithStatus() so that it will not
generate an error if the file is closed before the HTTP response is
written to the download cache.
This modifies the test HttpFileTest.MultipleWrites so that the file is
Flushed after each chunk. This adds test coverage for the changes
introduced in PR #1201.
Fixes#1224 (missing test coverage for HttpFile::Flush)
Joey Parrish
Cosmin Stejerean
Joey Parrish
sr90
Cosmin Stejerean
Shaka Bot
Caitlin O'Callaghan
Prakash Duggaraju
Cosmin Stejerean