Commit Graph

1539 Commits

Author SHA1 Message Date
Joey Parrish 9f11077768 Install pylint explicitly in custom linter action
Change-Id: I61891ef6682af209ee94b187b247f5b779e83c8a
2021-06-21 16:37:24 -07:00
Joey Parrish f8816030a1 Release v2.5.1
Change-Id: Id4e1d8f57c3413ae7eba6c3498fa71077c0d9a10
2021-06-21 21:47:08 +00:00
Joey Parrish 56e227267c Fix python linter errors and add linter checks to CI
Internal CI systems and the new GitHub CI system were out of sync,
with the external system not doing any linting.  Further, the internal
system was using an internal-only linter for Python.

This creates a script for Python linting based on the open-source
pylint tool, checks in the Google Style Guide's pylintrc file, creates
a custom action for linting and adds it to the existing workflows,
fixes pre-existing linter errors in Python scripts, and updates pylint
overrides.

b/190743862

Change-Id: Iff1f5d4690b32479af777ded0834c31c2161bd10
2021-06-21 21:46:48 +00:00
Joey Parrish e32b35f3e1 Fix clang-format output during linting
Instead of printing a binary object, treat the output of clang-format
as a utf-8 string.

b/190743862

Change-Id: I596d223792597f8157fdee2d75773131cc858c9a
2021-06-21 21:46:48 +00:00
KongQun Yang c0d68083ac Fix playready_extra_header_data
It should be enclosed in the <DATA> element.

https://docs.microsoft.com/en-us/playready/specifications/playready-header-specification

Fixes #961.
Fixes b/173146719.

Change-Id: I7d235a44afda4a0a301e7231dc8a4e87cce124be
2021-06-18 18:34:24 +00:00
Joey Parrish b07988e4f9
Merge pull request #959 from joeyparrish/master
CI overhaul based on GitHub Actions
2021-06-18 10:40:32 -07:00
Joey Parrish acafc0fd93 Make CI workflows safe when testing in a fork
Testing CI workflows is a pain.  This usually involves forking the
main repo and testing various operations there, where the results will
not break the main repo.

However, some things like NPM and Docker package names were initially
hard-coded.  This meant that a fork would need to customize those in
the workflows to avoid pushing official-looking packages during CI
testing.

This change moves those hard-coded names to GitHub Secrets.  Though
the names are not actually secret, the secret store is per-repo, and
will be independent in a fork.  This makes it easier to avoid
accidentally pushing official-looking releases during testing, even if
the fork has access to the same auth tokens.

Change-Id: Ide8f7aa92a028dd217200fca60881333bf8ae579
2021-06-17 13:36:59 -07:00
Joey Parrish a2e07a901e Refactor actions and workflows
It turns out that workflows were the wrong way to abstract reusable
pieces of work.  This turns common steps into custom actions (build
docs, build packager, test packager) which can be used as encapsulated
steps in multiple workflows.

This is a much more natural way to avoid duplication compared to the
previous approach of triggering one workflow from another.  This also
has the benefit of all of the steps of a release being represented on
GitHub as a single workflow, making it easier to understand what is
happening and what event triggered those steps.

Change-Id: Ife156d60069a39594c7b3bb3bc32080e6453b544
2021-06-17 10:32:24 -07:00
Joey Parrish 53c39ef9ac Address PR feedback
- Document necessary repo secrets
 - Compress build artifacts directly to the arifacts folder
 - Log test commands as they are executed
 - Add comments

Change-Id: I1cc150995d339e2e93bee4570d80263dae362bb9
2021-06-16 14:24:24 -07:00
Joey Parrish 0fdd046a14 Tweak job name in Update Docs workflow
Change-Id: Ifdf4343613921aa4e34a91a76fa241ab85862760
2021-06-16 12:18:01 -07:00
Joey Parrish 0f8749a211 CI overhaul based on GitHub Actions
This replaces Travis (for Linux & Mac) and Appveyor (for Windows) with
GitHub Actions.  In addition to using GitHub Actions to test PRs, this
also expands the automation of releases so that the only manual steps
are:

 1. Create a new CHANGELOG.md entry
 2. Create a release tag

Workflows have been create for building and testing PRs and releases,
for publishing releases to GitHub, NPM, and Docker Hub, and for
updating documentation on GitHub Pages.

When a new PR is created, GitHub Actions will:
 - Build and test on all combinations of OS, release type, and library
   type

Appveyor's workflow took ~2 hours, whereas the new GitHub Actions
workflow takes ~30 minutes.

When a new release tag is created, GitHub Actions will:
 - Create a draft release on GitHub
 - Extract release notes from CHANGELOG.md & attach them to the
   draft release
 - Build and test on all combinations of OS, release type, and library
   type, aborting if any build or test fails
 - Attach release artifacts to the draft release, aborting if any
   one artifact can't be prepared
 - Fully publish the draft release on GitHub
 - Publish the same release to NPM (triggered by GitHub release)
 - Publish the same release to Docker Hub (triggered by GitHub release)
 - Update the docs on GitHub pages

Closes #336 (GitHub Actions workflow to replace Travis and Appveyor)

b/190743862 (internal; tracking replacement of Travis)

Change-Id: Ic53eef60a8587c5d1487769a0cefaa16eb9b46e7
2021-06-16 11:52:02 -07:00
Joey Parrish 032cf2a345 Fix libpackager_type variable undefined by default
In e2efb5d4, I fixed shared_library builds on Windows, but I
introduced another issue in which the libpackager_type variable was
not correctly defined by default.  This meant that the build only
worked with this variable explicitly-defined in GYP_DEFINES when
gclient sync was run.

This fixes the default definition so that libpackager_type does not
need to be defined explicity.

Related to #318 (shared_library builds on Windows)

Issue #336 (progress toward GitHub Actions workflow to replace Travis
  and Appveyor, where we need to build and test shared_library on all
  platforms)

b/190743862 (internal; tracking replacement of Travis)

Change-Id: If353e1d3c312ab0c568d4d4d2b789e922d7216e1
2021-06-16 10:14:04 -07:00
Joey Parrish 098f58a143 Fix shared library test runs on Windows
Shared library builds worked, but failed tests because they were made
with the wrong CRT linker settings.  Strings allocated within the
library could not be freed outside the library because the dynamic CRT
was not used.

This sets necessary gyp variables to link with a dynamic CRT on
Windows, thereby fixing tests running in shared library mode that
otherwise hung in a GitHub Actions environment.

Related to #318 (shared_library builds on Windows)

Issue #336 (progress toward GitHub Actions workflow to replace Travis
  and Appveyor, where we need to build and test shared_library on all
  platforms)

b/190743862 (internal; tracking replacement of Travis)

Change-Id: Iffefd27c2aa4ec479ce1d10b099483e417d2231f
2021-06-15 19:37:32 -07:00
Joey Parrish e2efb5d41c Fix Windows shared_library builds
To make shared_library builds work on Windows with MSVS 2019, this
commit:
 - Silences a useless warning about a private member in dll-exported
   Status class.
 - Exports the File class used by packager.exe
 - Removes the explicit File dependency in packager.exe in favor of
   libpackager, now that File is exported
 - Add missing defines in packager.exe and packager_test.exe that
   instruct the linker to import Status and File from the library

Closes #318 (shared_library builds on Windows)
Issue #336 (progress toward GitHub Actions workflow to replace Travis
  and Appveyor, where we need to build and test shared_library on all
  platforms)
b/190743862 (internal; tracking replacement of Travis)

Change-Id: I091f1655d88d36f353f7df497101eef17729eefe
2021-06-15 13:16:47 -07:00
Joey Parrish 2526c61e9e
Merge pull request #955 from joeyparrish/master
At this point static_library builds are working in MSVS 2019. shared_library builds are still not working.

Closes #867 (MSVS 2019)
Issue #318 (progress toward shared_library support on Windows)
Issue #336 (progress toward replacing Travis & Appveyor with GitHub Actions, which uses MSVS 2019)
b/190743862 (internal; tracking replacement of Travis)
2021-06-14 15:25:39 -07:00
Joey Parrish 60464334c1 Update build instructions for Windows
The build instructions have been updated with necessary environment
variables for newer MSVS versions.

Issue #867 (MSVS 2019)
Issue #336 (progress toward replacing Travis & Appveyor with GitHub
  Actions, which uses MSVS 2019)
b/190743862 (internal; tracking replacement of Travis)

Change-Id: Ic80ba22a750946a508803c52af6b7869964d595b
2021-06-14 12:17:31 -07:00
Joey Parrish 98782c051e Skip clang updates on Windows
We do not use clang to build on Windows, so we should skip trying to
update clang.  This is critical to MSVS 2019 support, since the older
clang tools we depend on do not recognize that newer environment.
Since we don't need clang anyway, skipping the update avoids useless
errors about finding a matching clang version for the environment.

Issue #867 (MSVS 2019)
Issue #336 (progress toward replacing Travis & Appveyor with GitHub
  Actions, which uses MSVS 2019)
b/190743862 (internal; tracking replacement of Travis)

Change-Id: I5600ed809b11e68444034a06cc891403e6bfb5cc
2021-06-14 12:17:28 -07:00
Joey Parrish ae89145015 Fix vector+unique_ptr issues with MSVS 2019
Because a StreamState object contains a unique_ptr, it is not
copyable.  A vector of StreamStates, therefore, causes a compile error
on resize or push_back, both of which invoke the copy constructor.

I don't know why MSVS complains, but clang does not.

To fix this, I'm changing vector<StreamState> into deque<StreamState>.

At this point static_library builds are working in MSVS 2019.
shared_library builds are still not working.

Issue #867 (MSVS 2019)
Issue #336 (progress toward replacing Travis & Appveyor with GitHub
  Actions, which uses MSVS 2019)
b/190743862 (internal; tracking replacement of Travis)

Change-Id: Iaa9d5fc357102d15eac96c29ebeee7c7236e976b
2021-06-14 12:17:24 -07:00
Joey Parrish 3d211440a6 Roll src/packager/tools/gyp/ e7079f0e0..caa60026e (69 commits)
e7079f0e0e..caa60026e2

In order to use MSVS 2019, gyp must be upgraded.  The old version
didn't recognize MSVS 2019, but the newer version does.

This isn't everything that's needed, but it's a first step.

Created with:
  roll-dep src/packager/tools/gyp

Issue #867 (MSVS 2019 support)
Issue #336 (progress toward replacing Travis & Appveyor with GitHub
  Actions, which uses MSVS 2019)
b/190743862 (internal; tracking replacement of Travis)

Change-Id: I26c433682b1bfd584bf3af0bb4e0bd04646535c1
2021-06-14 12:17:09 -07:00
KongQun Yang ffb4d771ca Update Dockerfile due to depot_tools update
Change-Id: Iaf2415a2fb954a3f2473233eb0522edc211b4a9b
2021-06-11 11:07:41 -07:00
sr90 d84260ca78
Properly format the documentation for dash_only and hls_only 2021-06-10 14:21:23 -07:00
KongQun Yang 98b97709d7 Update CHANGELOG for v2.5.0
Change-Id: I20717c787024f7fbace2f7735f4950529d3a17e7
2021-06-09 17:08:43 -07:00
KongQun Yang 72ea5d30f0 Fix gpylint errors
Change-Id: I861cfe98d50867ab9ee9be69cd4ffc3a7157fbd8
2021-06-09 12:17:48 -07:00
Samidh 4686454a73
Add segment_list support for DASH on-demand profile
Configurable under flag --dash_force_segment_list, default to false.

Note that DASH live profile is not supported right now.
2021-05-25 12:08:58 -07:00
Daniel Cantarín fe9e26d5d9
Remove "atomic" warning for HttpFile
Fixes #945.
2021-05-25 11:59:53 -07:00
Mattias Wadman 056140ae83
Update git-clang-format URL
The previous svn link is broken.
2021-05-07 10:21:40 -07:00
Daniel Cantarín 8515a66031
Changed default HTTP UserAgent to ShakaPackager/<version>
Closes #939.
2021-05-07 10:15:44 -07:00
Mattias Wadman 62f37eb3b7
Ignore matroska projection metadata
Warn instead of fail parsing.

Closes #932.
2021-05-07 10:13:02 -07:00
KongQun Yang 2e521c8413 Remove another use of regex library
It is not working correctly in gcc 4.8 or earlier, which is still
popular (bundled by default in CentOS 7).

Issue #865, #929.

Change-Id: I136446a70831bd0237cd29646dd349fe7558176b
2021-05-05 18:01:27 +00:00
Vishal Shah d9124d6aaa
[WEBVTT] Fix missing text alignment tags from output
Legacy players, e.g. older versions of ExoPlayer, do not handle default webvtt text alignment correctly. Need to specify `align:center` explicitly cues without text alignment for backwards compatibility.

Fixes #925.
2021-05-04 22:57:43 -07:00
Daniel Cantarín 37b16b4091
HTTP File upload fixes and tweaks
- Do not write the HTTP PUT response to cache which can potentially overflow the cache buffer as it is not consumed.
- VLOG(1) instead of LOG(ERROR) on HttpFile::Size() as it can be called during normal code execution.
- Add a command line flag `--user_agent` to allow users to specify their custom user agent string.

Fixes #939.
2021-05-04 22:53:23 -07:00
KongQun Yang 4528bdb330 Remove the use of regex library
It is not working correctly in gcc 4.8 or earlier, which is still
popular (e.g. bundled by default in CentOS 7).

Fixes #865, #929.

Change-Id: I55a42428dbd2a12fc2c3b1e6a49fdd662a295dca
2021-05-04 02:09:08 +00:00
KongQun Yang 23953bf931 Update check_formatting.py to work with python3
Change-Id: Iff3ba89f3b3e6fe0991e48482546084d1ba66ac5
2021-05-04 02:07:49 +00:00
KongQun Yang 6c8ad30217 Update Sphinx config for markdown handling
The config format has changed in v3.0 in the handling of markdown.

See https://www.sphinx-doc.org/en/master/usage/markdown.html.

Change-Id: I3f2dad962824e1d5f410f8420795c68715b03635
2021-04-09 21:18:06 -07:00
Daniel Cantarín f6c02e629d
Generate object type properly for MPEG-1 audio
Fix #905.
2021-04-04 22:47:31 -07:00
Joey Parrish d48bf0f48e fix: Explicitly signal the lack of CEA captions in HLS
Closes #922

Change-Id: If097a66642ca40c355a476655ff1394b7b937b17
2021-03-30 18:26:37 +00:00
Jacob Trimble c1f64e5350 Fix transparency case in DVB-SUB.
This fixes some math errors in the color conversions and handles the
case of Y=0.

Fixes #903

Change-Id: I796246e4d62a3161b44916f97e9e98f9203ad338
2021-03-29 16:34:39 +00:00
Suraj kumar (sk) aa046a8d9e
Fix the build instructions on Windows
For windows running `GYP_DEFINES='target_arch=X64'` shows an error GYP_DEFINES is
not recognized as an internal or external command.

The actual command is to set the value of GYP_DEFINES. so it should be 
`SET GYP_DEFINES='target_arch=X64'`.
2021-03-22 11:32:26 -07:00
Daniel Cantarín dd935f6dc3
TTML: change "imagetype" attribute to camel case
Fixes #908
2021-03-09 10:10:32 -08:00
Sergio Garcia Murillo f9908362f8
Prevent seg fault if webm fragment is not initialized or last frame is EOS
Fixes #900
2021-03-07 15:01:08 -08:00
Sergio Garcia Murillo b8ce44aba0
Prevent seg fault if mp4 fragment is not initialized
Related to #900.
2021-03-02 23:53:58 -08:00
Jacob Trimble 00af192626 Cleanup HttpFile and related PR.
This implements many of the comments made on the PR and cleans up those
files.

Closes #149

Change-Id: Ice73fe3c04a6f595da6986a4c070e50cb20f9435
2021-03-02 17:43:47 +00:00
Jacob Trimble f0a52cbbf2 Fix TTML codec string in HLS.
Issue #903

Change-Id: I1f73d16ad79c2593996d7d50f9556984302fe21e
2021-02-25 16:17:45 -08:00
Jacob Trimble 025e10de69 Add better detection for text streams.
Change-Id: I7f44026c223b868cd73f52bf700e29372bb9461e
2021-02-17 10:40:14 -08:00
Jacob Trimble c4ea028d11 Update documentation for text support.
Change-Id: I43acad9ae62f89b06491a3b56fb846da1edd4002
2021-02-17 18:39:07 +00:00
Jacob Trimble a0f3f2cd3a Add cc_index to stream descriptor.
This also allows setting the language of different text streams from
the same input.  Multiple streams can use the same input stream
using different cc_index values and can each use a different language.

This also will try to pull the language from the input if not
specified.

Change-Id: I7078710b509b7d77dad8cb4299a82f954af7e9e7
2021-02-17 18:33:53 +00:00
Jacob Trimble 78be14c092 Add DVB-sub parser
Note that this only supports a single page within the DVB-sub stream.
Multiple pages will be merged together.  A follow-up will allow
selecting a specific page.

This only supports outputting using TTML or MP4+TTML; you cannot have
DVB-sub output nor can you output it in WebVTT.  Since DVB-sub
uses images, it is hard to impossible to do this with WebVTT.

This also only supports interlaced images, not progressive images
nor text.

Closes #832

Change-Id: Id6dbb6393c7b9a05722e61c6bd255bef5e69a7d8
2021-02-17 18:32:03 +00:00
Jacob Trimble 95089593fc Don't re-open WebVTT file to determine size.
Change-Id: Id92226adce813b7d0c4c741e47e36dbf8f208797
2021-02-08 20:31:13 +00:00
JPeMu 36ef7ec945
[MPEG-TS] Fix PCR reserved bits not being set correctly
Fixes #893.
2021-02-03 12:09:07 -08:00
Alen Vrečko 449945dfba
Added IDE support in build instructions
Fixes #772.
2021-02-03 11:02:13 -08:00