This fixes our use of std::filesystem to interpret all paths names as
UTF8. Before this, UTF8 paths did not work correctly in all cases.
This also adds a new unit test to cover this case.
On Windows, it is critical that a UTF8 locale be set at runtime.
Applications linking with Packager as a library should call setlocale(),
and the Packager frontends now do this automatically after converting
wide character arguments into narrow strings.
Closes#652
All HTTP-based tests now use an embedded test server instead of
httpbin.org, which makes them much faster and more reliable.
These more reliable tests also exposed some issues that began recently
with PR #1201. HttpFile's Flush() semantics were different than those
documented for files in general. Flush() used to close the file for
uploading, so that no further writes were allowed, but the documentation
stated that it would only flush data to its destination. PR #1201
brought HttpFile's Flush() in line with the docs, but gave us no way to
terminate a chunked upload.
This adds a new method to File called CloseForWriting(), which
terminates a chunked upload for HttpFile. The only other implementation
that does anything is UdpFile, which uses the socket library function
shutdown() to terminate writes while allowing reads.
This also tweaks HttpFile::CloseWithStatus() so that it will not
generate an error if the file is closed before the HTTP response is
written to the download cache.
This modifies the test HttpFileTest.MultipleWrites so that the file is
Flushed after each chunk. This adds test coverage for the changes
introduced in PR #1201.
Fixes#1224 (missing test coverage for HttpFile::Flush)
There are a lot of changes in this first phase, because there was a
lot of infrastructure required to get some meaningful amount of
porting done. Future PRs should be simpler.
<b>Summary of changes:</b><details>
- Remove old deps:
- boringssl (replaced with mbedtls, lighter, easier to build)
- gflags (replaced with absl::flags)
- Chromium build tools
- New deps to replace parts of Chromium base:
- abseil-cpp
- glog
- nlohmann::json (for tests only)
- Submodules, updates, and CMake build rules for third-party
libraries:
- curl
- gmock/gtest
- Ported internal libraries and their tests by removing Chromium deps
and adding CMake build rules:
- file (now using C++17 filesystem APIs)
- license_notice
- status
- version
- Test improvements
- Removed file tests that can never be re-enabled
- Re-enabled all other disabled file tests
- Debug JSON values when HTTP tests fail
- Fixed chunked-encoding issues in HTTP tests
- Updated and refactored Dockerfiles testing
- All docker files working, with OS versions updated to meet the
new tool requirements
- Local docker builds no longer write files to your working
directory as root
- Local docker builds can now be run in parallel without clobbering
each others' build outputs
- DEBUG=1 can drop you into an interactive shell when a docker
build fails
- Updated and heavily refactored workflows and Dockerfiles
- All docker files now tested in parallel on GitHub, speeding up CI
- All common workflow components broken out and using workflow_call
instead of custom actions
- Self-hosted runners now optional, to make testing easier on forks
- CMake porting works-in-process can now be fully tested on GitHub
- Building ported libraries and passing ported tests on all three
platforms!
- CI hacks for macOS removed, now testing on macos-latest!
- Python2 no longer required! (Only Python3)
- Using strict build flags, treating all warnings as errors.
</details>
<b>Required to build:</b>
- CMake >= 3.16
- Python 3
- A compiler supporting C++ >= 17
- g++ >= 9 if using GCC (Clang also fine)
- MSVC for Windows
<b>Still needs work:</b><details>
- Moving other dependencies into submodules (if we keep them):
- apple_apsl
- icu
- libevent
- libpng
- libwebm
- libxml
- modp_b64
- protobuf
- zlib
- Port remaining internal libraries:
- app
- hls
- media/base
- media/chunking
- media/codecs
- media/crypto
- media/demuxer
- media/event
- media/formats/dvb
- media/formats/mp2t
- media/formats/mp4
- media/formats/packed_audio
- media/formats/ttml
- media/formats/webm
- media/formats/webvtt
- media/formats/wvm
- media/origin
- media/public
- media/replicator
- media/trick_play
- mpd
- Port main application
- Add logging flags in absl and connect them to glog (which expects
gflags)
- Port pssh-box.py
- Port main test targets (packager_test.py and packager_app.py)
- Updating all requirement and build documentation
- Remove any remaining refs to gclient, depot_tools, ninja
- Update and complete release workflows using release-please
</details>
Issue #346 (Switch to abseil)
Issue #1047 (New build system)
Issue #149
Co-authored-by: Andreas Motl <andreas.motl@elmyra.de>
Co-authored-by: Rintaro Kuroiwa <rkuroiwa@google.com>
Co-authored-by: Ole Andre Birkedal <o.birkedal@sportradar.com>
The new directory permission is set to the permission of the last
existing directory in the file path. If none of the directory exists,
it is default to 0755.
Previously we use base::CreateDirectoryAndGetError(), which always sets
the new directory permission to 0700, which is not what we want.
Fixes#499.
Change-Id: Iee9429d5e63ada9588f74ff20d3cce28a1a6437b
Previously packaging completes successfully without any error
or warning.
With the fix, an error will be reported if write fails. It may
appear as "Cannot close file error" as we use threaded IO, which
could delay the error reporting until Close() call, so the user
of the File API needs to make sure Close() returns successfully.
Also fixed a deadlock in threaded_io_file if internal_file->Write
fails.
Fixes#160
Change-Id: I17f945150fb4021d2dcdbe784e557673f53ca583
Cosmin Stejerean
Joey Parrish
Jacob Trimble
Ole Andre Birkedal
KongQun Yang