Commit Graph

25 Commits

Author SHA1 Message Date
Daniel Cantarín 37b16b4091
HTTP File upload fixes and tweaks
- Do not write the HTTP PUT response to cache which can potentially overflow the cache buffer as it is not consumed.
- VLOG(1) instead of LOG(ERROR) on HttpFile::Size() as it can be called during normal code execution.
- Add a command line flag `--user_agent` to allow users to specify their custom user agent string.

Fixes #939.
2021-05-04 22:53:23 -07:00
Jacob Trimble 00af192626 Cleanup HttpFile and related PR.
This implements many of the comments made on the PR and cleans up those
files.

Closes #149

Change-Id: Ice73fe3c04a6f595da6986a4c070e50cb20f9435
2021-03-02 17:43:47 +00:00
Ole Andre Birkedal aa17521268
HTTP PUT output support (#737)
Issue #149

Co-authored-by: Andreas Motl <andreas.motl@elmyra.de>
Co-authored-by: Rintaro Kuroiwa <rkuroiwa@google.com>
Co-authored-by: Ole Andre Birkedal <o.birkedal@sportradar.com>
2021-02-02 10:51:50 -08:00
KongQun Yang ff5f3f3abc Call WSAStartup once for every new socket
Some users are seeing problems with only one WSAStartup call.

WSAStartup can be called more than once:
https://docs.microsoft.com/en-us/windows/win32/api/winsock/nf-winsock-wsastartup

Fixes #643.

Change-Id: I81e0e7979d6a586da452984a96a8557b7b3ce7f6
2019-11-20 14:00:51 -08:00
KongQun Yang 0342adb132 Fix possible packager hangs when reading mp4 from FIFO
For MP4 files, mp4 parser tries to open and read the same file again
to handle trailing MOOV box.

Open FIFO again may result in packager hang.

Update the code to only attempt to load MOOV for local regular files,
as FIFO and remote files are unseekable anyway.

Fixes #664.

Change-Id: Ib286d2876d202cd5a248ffe70b13589b3cc74bc9
2019-10-21 18:49:52 -07:00
KongQun Yang 3ec2975e36 Log socket error code on failure
Issue #643.

Change-Id: I8adfb9910e1e43294738abb7783a64645a4f42c3
2019-09-10 21:44:28 +00:00
Tomohiro IKEDA f810fea0ef Cleanup condition on `bind` error (#624) 2019-08-01 17:32:28 -07:00
KongQun Yang d97e531465 Set output directory permission appropriately
The new directory permission is set to the permission of the last
existing directory in the file path. If none of the directory exists,
it is default to 0755.

Previously we use base::CreateDirectoryAndGetError(), which always sets
the new directory permission to 0700, which is not what we want.

Fixes #499.

Change-Id: Iee9429d5e63ada9588f74ff20d3cce28a1a6437b
2018-12-11 23:17:15 +00:00
KongQun Yang 715ed939f1 Add instructions to build on Alpine Linux
- Also fixed compilations in Alpine Linux and other flavors of Linux.
- Added container versions in docker files to always use a verified
  version.

Closes #164.

Change-Id: I949a8709e4d70c49129c9c2e8608dd78193d964c
2018-08-17 20:46:11 +00:00
KongQun Yang 2fee1e673b Set default --io_block_size to 64K
--io_block_size determines the buffer size for each read/write in
ThreadedIO.

A big io_block_size does not help much as long as --io_cache_size
is big enough.

Instead, it may cause problems. For example, Linux pipes block until
reading io_block_size number of bytes, which causes a large delay
when reading audio streams.

Change-Id: I5bdd3d61388579c7e8647cdab3152135a40a752b
2018-08-09 22:24:43 +00:00
KongQun Yang b4e827e01b Make sure TempFilePath always generate unique file path
Previously it is possible that the same file path is generated when
the function is called consecutively in the same thread.

The problem can be re-produced in Windows. Does not seem to be
re-producible in Linux and Mac.

Fixes #448, #449.

Change-Id: Ia0163492e3494eba00f56f4d356aa1010b9660cc
2018-08-08 17:33:01 +00:00
KongQun Yang d932153dd7 Make MemoryFile thread-safe
- Also make it explicit that MemoryFile does not support opening an
  already open file. An error will be returned when trying to open an
  already open file. Previously the code may crash with memory problems.
- Also updated packager_test to use different test directories for
  different tests.

Fixes Appveyor crash due to memory corruption: #449.

Appveyor may still fail but will contain a meaningful error logging.

Change-Id: Ibc9346ef7f301e416a4a09f120bca56504c939d8
2018-08-08 17:32:46 +00:00
Yohann Connell ec63c53ed5 Removing dependence on base file atomicops.h
This part of issue #346.

Change-Id: Iab685145d10d6e6c99ff666e08bca3c36b6d5a66
2018-07-23 22:19:29 +00:00
KongQun Yang d3903fad19 Add buffer_size UDP file options
This allows UDP receive buffer size to be increased to avoid potential
packet loss result from receive buffer overrun.

Another related flag is --io_cache_size. buffer_size in UDP options
defines the UDP buffer size of the underlying system while
io_cache_size defines the size of the internal circular buffer managed
by Shaka Packager.

Closes #411

Change-Id: I57c843a88f13da546417dadc2a78df2bee0a00f3
2018-06-29 00:19:30 +00:00
KongQun Yang 7bc90004e0 Add IO Cache circular buffer full warning
Under VLOG(1).

IO Cache circular buffer full could result in UDP packet loss with UDP
inputs.

Related to #390 and #411.

Change-Id: Ia5636a70827978b5c2f71f6495256d758766632f
2018-06-29 00:19:14 +00:00
fatpelt 6a8d2aa443 Add Source Specific Multicast (SSM) support
SSM can be enabled using "source" option in UDP options. Previously, "source" is considered the same as "interface", which is incorrect. 

Fixes #332.
2018-03-06 10:09:28 -08:00
Aaron Vaage 9e7b5c1ca9 Change CharReader to FileReader
Instead of having multiple char readers, one for strings and
one for files, just have one for files and use memory files
when testing.

Change-Id: Id1a2230046ba540ddf69ca10edb3edc74d2419b6
2018-01-04 14:36:52 -08:00
KongQun Yang 8e96dd3b37 Address gcc build failures in some platforms
Seeing some failures on some platforms when compiled with clang
disabled:
  GYP_DEFINES="clang=0" gclient runhooks

Several changes to make it work:
1. Mark packager code with packager_code=1 in GYP definitions.
2. Disable a few checks in non-packager code, which we do not have
   direct control: dangling-else, deprecated-declarations,
                   unused-function
3. Fix the relevant errors in packager code.
4. Revert HAVE_STROPTS_H in curl config which is not available in
   all linux distributions.

Fixes #286
Fixes #293

Change-Id: I729b41f99403c5ad9487c6cc4a7dc06f6323cef8
2017-12-07 11:45:26 -08:00
KongQun Yang 82e0fd2095 Default multicast interface address to "0.0.0.0" if absent
It aligns with what a lot of other apps are doing, e.g. tsplay.

Also added notes in docker to adjust network setting if wants to use
multicast in docker container.

Change-Id: I666a8979cc041c904f7ff8e3e022f800c0830d5d
2017-12-04 19:21:52 +00:00
KongQun Yang e889578a2d Creates directories in the output path if not exist
Also added end to end tests for single segment HLS and multi-segment
HLS.

Fixes #276

Change-Id: I27e5d57cdc3ab79a8e5ae0c3a8da46fcf2c467b6
2017-10-11 20:19:51 +00:00
KongQun Yang 9c861d03f7 Report an error when disk is full
Previously packaging completes successfully without any error
or warning.

With the fix, an error will be reported if write fails. It may
appear as "Cannot close file error" as we use threaded IO, which
could delay the error reporting until Close() call, so the user
of the File API needs to make sure Close() returns successfully.

Also fixed a deadlock in threaded_io_file if internal_file->Write
fails.

Fixes #160

Change-Id: I17f945150fb4021d2dcdbe784e557673f53ca583
2017-09-15 22:22:12 +00:00
KongQun Yang ea45ce3158 Add support for callback file
Change-Id: Ieb116bf3f645a35601f1182ed139c59ddaab8ad8
2017-09-07 19:26:26 +00:00
KongQun Yang 9e3dc06d20 Fix file permission issue with manifests
ImportantFileWriter::WriteFileAtomically uses mkstemp internally,
which set file permission to 0600, which is not what we want.
Update the code to not use mkstemp instead.

Also updated temporary file name logic to include process id in
the name so it can be unique across processes.

Fixes #259

Change-Id: I2d5a375925cf552bc0db5269f409d7522e63fca5
2017-08-01 15:42:55 -07:00
KongQun Yang 723079310f Disable IP_MULTICAST_ALL when setting up UDP multicast
This avoids interference caused when two sockets are bound to the
same port in different multicast groups.

Fixes #241

Change-Id: Iba1b3300e3850f1f268886b7a49f5a3315d95b80
2017-07-27 18:02:51 -07:00
KongQun Yang eb98a87f7d Move packager/media/file to packager/file
Change-Id: Idc0d75a7ceab7c8ce460ef896fdefdec088124bb
2017-07-17 12:08:28 -07:00