Commit Graph

848 Commits

Author SHA1 Message Date
KongQun Yang a7463f60c1 Enable segment auto removal for live packaging
The number of preserved segments outside live window can be
configured using flag --preserved_segments_outside_live_window,
which is default to 50, i.e. 5 minutes for 6s segment.

Note that the segment removal will be disabled if it is set to 0.

Only HLS live playlist and DASH dynamic MPD are affected by this flag.

- Also add end to end tests.

Fixes #223.

Change-Id: I8a566efebe2f1552c7d9509ab017bade5a4a1c98
2018-05-02 00:18:42 +00:00
KongQun Yang b6f0da246f Add support for FLAC codec
Implemented according to spec:
  https://github.com/xiph/flac/blob/master/doc/isoflac.txt

Closes #345.

Change-Id: If2e277d8eac4baff5965faa0e13c44c334f6184d
2018-05-02 00:16:51 +00:00
Aaron Vaage cb2ec22a06 Set Text Time Scale In Tests
Problem: The time scale for text was being set to zero in our tests
         because it was never needed.

Change: This changes it to use MS as that is what's most common and
        requires the least amount of changes.

Change-Id: Ia046ac1994b4cede079d2f801275c7f058d5bdd3
2018-05-01 22:43:13 +00:00
Aaron Vaage 03afb16c0c Use Map To Track Demuxers and Aligners
To move us toward no longer need to ensure order when building
our pipeline, use a map to share demuxers between stream descriptors.
This will even allow use to use the same demuxers in the text pipelines
while still building them separately from the audio and video streams.

Change-Id: I4d4dbddbc06adee36cbe7f4aa1f6769f7bb2a3f6
2018-05-01 22:31:04 +00:00
KongQun Yang 37d4ff017f Support removing segments outside of live window in HLS
Issue: #223

Change-Id: Ib91c60268d8df9adbaf5f6cac77eaebd6a3edb6e
2018-04-30 17:25:39 -07:00
KongQun Yang 224b597b48 Support approximate SegmentTimeline
It is not always possible to align segment duration to target duration
exactly. For example, for AAC with sampling rate of 44100, there are
always 1024 audio frames per sample, so the sample duration is
1024/44100. For a target duration of 2 seconds, the closest segment
duration would be 1.984 or 2.00533.

This feature allows MPD generator to treat these segments as having
the same duration, thus allows MPD generator to generate less
SegmentTimeline entries and potentially no SegmentTimeline entries
(replaced with SegmentTemplate@duration instead if
  --segment_template_constant_duration flag is enabled).

Under flag --allow_approximate_segment_timeline. Disabled by default.

Fixes #330.

Change-Id: I5044eaa348ebbf45bf792a2af53fc95a115ae21b
2018-04-30 17:15:47 -07:00
Aaron Vaage 902dfb905a Create Tests For Bandwidth
Created HLS and DASH tests that include overriding the
computed bandwidth value.

Issue #389

Change-Id: I6063fe780f6a68e2e7a7ad4440c9d76bacbbffe3
2018-04-30 15:44:55 -07:00
Haoming Chen dddbb19701 Rename Playready to PlayReady.
Change-Id: I11e0b0a0cc32aec574804e7cb02291dc3174357c
2018-04-26 11:38:31 -07:00
Haoming Chen aa4b573342 Implement multi DRM support. (Part 3)
- Allow including Widevine and Common SystemID PSSH boxes
  for PlayReadyKeySource.
- --playready_key_id and --playready_key flags are deprecated.
- --enable_raw_key_encryption already supports playready PSSH generation.

Addresses issue #245

Change-Id: I072d4f43a3239875959e4c5b1eb6854415d7367e
2018-04-26 18:14:22 +00:00
Aaron Vaage 2563ae7fe9 Add Missing Header Guards To Cue Aligner
Change-Id: I0f9e892172932dc560101865df6539736388721a
2018-04-26 17:51:02 +00:00
Aaron Vaage 56c2f227ff Skip Style and Region Blocks
To ensure that we can parse content with style and region blocks,
this change updates the parser to skip those blocks so that we
can still parse the cues from a file.

Full style and region support will be added later this year.

Issue #380

Change-Id: I11b8fd862a108c27a5c67b15d4703532b44a1214
2018-04-25 20:41:10 +00:00
KongQun Yang b2932d5a33 Fix bitrate for DASH on-demand profile too
Removed the logic in MuxerListener to estimate bandwidth from file
size and duration, since it is not compliant to the spec.

MpdBuilder will estimate bandwidth from segment size and duration
if bandwidth is not specified in MediaInfo.

Here is the statement from DASH spec (23009-1:2014):

Consider a hypothetical constant bitrate channel of
bandwidth with the value of this attribute in bits per second
(bps). Then, if the Representation is continuously delivered
at this bitrate, starting at any SAP that is indicated either by
@startwithsap or by any Segment Index box, a client can
be assured of having enough data for continuous playout
providing playout begins after @minbuffertime *
@bandwidth bits have been received (i.e. at time
@minbuffertime after the first bit is received).
For dependent Representations this value specifies the
bandwidth according to the above definition for the
aggregation of this Representation and all complementary
Representations.

Fixes #376.

Change-Id: I0fddce39e709d0cded0a4c9ae59adbbcc97ec5ea
2018-04-24 23:25:02 +00:00
KongQun Yang 60d7a4b2d5 Make MediaPlaylist accept HlsParams as contructor param
And for HlsNotifier and SimpleHlsNotifier as well.

This will make it easier to add preserve_segments_outside_live_window
param in a later CL.

Change-Id: I86d464fe247e04574158a0a76e39d8a122960ae4
2018-04-23 18:29:38 -07:00
KongQun Yang adc9549e2d Add new URL fields in MediaInfo
The file_name fields will be used to solely indicate file paths on the
designated file system, and they are used to do normal file operations,
including file creation, file updating and file removal if needed;
added new xxx_url fields, for the URLs that should appear on DASH
manifest or HLS playlists.

xxx_url are the URIs of the media in the manifest. The fields are
converted from file_name fields but adjusted to be relative to DASH
manifest path or HLS playlist path, optionally with base_url prepended.

Previously the file_name fields are converted in place to indicate
URLs when passing to manifest / playlist builders. The original file
names were lost, which made it difficult to remove files outside of
live window.

Now that the input file names are preserved. File system APIs can
operate on the original file names while manifest / playlist generation
functions can operate on URLs.

Issue: #233

Change-Id: I36a64f16e3d1261ce91783a86588f24ad1371662
2018-04-23 22:49:47 +00:00
KongQun Yang 43be3c8e8b Use max bitrate in Representation@bandwidth instead of average bitrate
According to DASH spec (23009-1:2014):

Consider a hypothetical constant bitrate channel of
bandwidth with the value of this attribute in bits per second
(bps). Then, if the Representation is continuously delivered
at this bitrate, starting at any SAP that is indicated either by
@startwithsap or by any Segment Index box, a client can
be assured of having enough data for continuous playout
providing playout begins after @minbuffertime *
@bandwidth bits have been received (i.e. at time
@minbuffertime after the first bit is received).
For dependent Representations this value specifies the
bandwidth according to the above definition for the
aggregation of this Representation and all complementary
Representations.

This suggests that max bitrate should be used instead of average
bitrate.

Also cleaned up BandwidthEstimator code.

Fixes #376.

Change-Id: Ibf5896394c5c6bb820849771a2129c59202d2273
2018-04-17 22:39:58 +00:00
KongQun Yang 64c2ad7d6e Support removing segments outside of live window in DASH
Issue: #223

Change-Id: Ie2d98a8f1553862532f1bdd074ca022d8904f0ea
2018-04-13 14:15:58 -07:00
KongQun Yang b6e1e5ee8b Set Widevine key request content-type to JSON
Content-Type for Widevine key request was incorrectly set to text/xml,
but it should be application/json.

Also added VLOGS for curl calls.

Fixes #372.

Change-Id: I4230795a582112c6d9c12883b5e61481b63284aa
2018-04-13 21:13:32 +00:00
KongQun Yang 0ef078a23b Fix default_language not effective with 2-char code
Two-character ISO-639 code in --default_language was ignored due to
a bug in language code matching as the language code in stream is
always converted to 3-character code.

Fixes #371.

Change-Id: I8618938af583a417446636ff9efe1c72ce822c33
2018-04-13 20:08:12 +00:00
KongQun Yang 4549b1d569 Remove pto_adjustment flag
This flag was introduced to workaround a rounding error in Chrome
(probably in other browsers too).

Also although this flag avoids the first frame of a Period to be
dropped due to rounding error but it could cause the last frame of a
Period to be dropped.

Now that we use a high precision Period@duration, we do not expect to
see rounding errors any more. The player would be a better place for
the workaround even if it is still needed.

Related issue: #368.

Change-Id: I3bd517ecc6d548ff62e0c13394edb49d4bc68e8f
2018-04-13 20:07:47 +00:00
KongQun Yang 09dc0a1907 Collect and generate license notice
Under flag --licenses

Bug: 74447460

Change-Id: If99c4e927431ce08158e3734428e3b4d23d62c75
2018-04-13 00:29:55 +00:00
KongQun Yang 2453c93f91 Update mac curl ca bundle: add /etc/ssl/cert.pem
Change-Id: I0ae6cf093b6f72672f3c918063806b8c95c0e924
2018-04-10 17:30:49 +00:00
KongQun Yang 772aa7c93f Do not force earliest_presentation_time to 0 for VOD
Instead, the actual earliest presentation time is used except for
the first segment if there is an offset between presentation time
(pts) and decoding time (dts).

Chrome (as of v66) reports dts instead of pts in buffered ranges in
MSE API. To avoid breaking Chrome, the earliest_presentation_time
of the first segment is set to its dts as Chrome does not like negative
values for
  adjusted dts = dts + Period@start (0 for the first period)
                 - presentationTimeOffset (earliest_presentation_time).

Fixes #303.

Change-Id: I5ca80e05d5570961400499436f2bcc01f06e69e0
2018-04-09 18:39:15 +00:00
Aaron Vaage 061777db9b Add CueEvent Support To WebVtt Output
The WebVtt Output Handler did not recognize cue events. This change
allows the handler to accept the events and tell muxer listener
about them.

Issue #362

Change-Id: I7c3318b72e539adc19af587c8e213fdb0af8290b
2018-04-07 02:45:52 +00:00
KongQun Yang ad75b9bd42 LOG an error on unsupported streams
So user knows where the problems, e.g. in #349.

Change-Id: Ie5310345d2f2c8cd83f2b6c7f96d49da94a78f4d
2018-04-06 21:45:45 +00:00
Aaron Vaage d5e820c744 Fix Spelling Mistake in Cue Alignment Handler
"bufferring" -> "buffering"

Change-Id: Ib516460295a52f0e5aa09583432868cf0efb3fe3
2018-04-06 13:16:55 -07:00
KongQun Yang 222737d5b5 Support UTCTiming
UTCTiming schemeIdUri and value pairs can be provided to packager using
--utc_timings flag. It should be comma separated list of
schemeIdUri=value pairs.

Note that urn:mpeg:dash:utc:direct:2014 scheme is not supported as it
requires the MPD to be dynamically generated on the fly when MPD is
served to client.

Fixes #311.

Change-Id: Ibc07af8a6d8b2b6261ba3ecd2c02f23809f96614
2018-04-04 01:37:01 +00:00
KongQun Yang 634ee65663 Generate a more accurate time in Period@duration
Chrome internally uses time accurate to microseconds, which is
implemented per MSE spec (https://www.w3.org/TR/media-source/).

Generate Period@duration with better precision to avoid possible
buffered range gaps in Chrome (possibly other browsers too), which
may lead to other problems like playback stall.

b/74238961
Fixes #368.

Change-Id: I357a0f62b67f75c7ca044bb99ea4e3c8bbb6fecd
2018-04-03 21:42:33 +00:00
Aaron Vaage 113a7d123b Don't Output Empty Text Samples
Only output text samples to webvtt text output when they have a payload.

Change-Id: I958d2b4f087209f540fa32cef002cd2cef37c65c
2018-04-03 18:29:41 +00:00
Aaron Vaage e1df680392 Clean-up Connecting Handlers
Created "Chain" function to connect a series of handlers together so
that the connection is easier to read.

Changed other calls to use RETURN_IF_ERROR to help make it easier to
read as we no longer need to track the status variable anymore.

Change-Id: Iffb76ca395b6d62f8feb054c09470c77718e8feb
2018-04-02 16:25:05 -07:00
Aaron Vaage e5fe2a76d2 Created Text Padder Handler To Fill Gaps
Created a media handler to come after parsers that will handle filling
in gaps between text samples. The padder takes a min duration, and if
the samples do not cover the min duration when flushed, one last empty
sample will be injected so that the samples will go up to the min duration.

Change-Id: I88605059664d09279676edac418ff3d4990d7556
2018-04-02 23:11:26 +00:00
KongQun Yang 4e9aadcb15 Write @startNumber for SegmentTemplate with $Time$ too
Change-Id: I8d55d81e7114e02b511656e73ecc0ad3bb1aa74f
2018-03-30 20:35:27 +00:00
Aaron Vaage 901d37a2a7 Correct some spelling mistakes in media_handler.h
Corrected some minor spelling mistakes in media_handler.h.

Change-Id: Ia120e96c9d5c29dba14919a505fb86c3dc1c15b2
2018-03-29 22:36:36 +00:00
Aaron Vaage 7dd1574b04 Add Cue Alignment For Segmented Text
Added the cue alignment handler to the segmented text pipeline.

Issue: #362

Change-Id: If105e5310fa82d433c84692b967a93c1cdf148b0
2018-03-29 20:29:23 +00:00
Aaron Vaage a11cbf93a7 Handle CueEvent in TextChunker
Updated TextChunker to handle incoming CueEvents. Connecting the
text chunker with the cue alignment handler will happen in a later
CL.

Issue: #362

Change-Id: Ib1fa9f457cf4ec0ce413dadcfa7eed5895ecd628
2018-03-29 11:03:15 -07:00
KongQun Yang 14aab56474 Allow SyncPointQueue to be cancelled
Change-Id: Idbf6ee7a5d9721681811189fca4190e7a0286c83
2018-03-27 19:14:37 +00:00
KongQun Yang db45e0868a Generate full segment after cue point
Also added RETURN_IF_ERROR status macro.

Change-Id: I04643b6252ea5623128f9a16fa744a255c91be17
2018-03-27 19:14:26 +00:00
Aaron Vaage 8f565bf388 Change Text Sample to Use int64_t
Changed Text Sample to use int64_t so that it will use the same
type for time as Media Sample.

Change-Id: I4cfbfdc60c37bb511517993976cd1a459bdf6667
2018-03-26 20:46:58 +00:00
Haoming Chen fac16cbf0a Implement multi DRM support. (Part 2)
- Allow including  Widevine and Playready PSSH boxes for RawKeySource.

Partially addresses issue #245

Change-Id: I8a48d6dbbbc21c76cf2c44384dd70286ccf80363
2018-03-26 20:18:22 +00:00
Aaron Vaage 56caecd073 Rename WebVttSegmenter To TextChunker
Renamed the webvtt segmenter to text chunker as there is nothing
webvtt specific in the code.

Change-Id: I0111c50d545fa168d945b39bc8a78c46c62678be
2018-03-26 17:58:56 +00:00
Aaron Vaage 2dd198a9d2 Move WebVtt Segmenter To Chunking Directory
Move the webvtt segmenter to the chunking directory so that it
can be converted to a general purpose text chunker.

Change-Id: I9ecd7ee39cb73070dab07b64f65ef24af1404813
2018-03-26 17:58:38 +00:00
KongQun Yang 4e57259d2a Integrate CueAlignmentHandler
Also changed ChunkingHandler to be one-one handler.

Issue: #355

Change-Id: Ie98a96bcc0ddded347699c9f333f604826976d11
2018-03-26 17:58:30 +00:00
Aaron Vaage 2b337e29f6 Removed WebVTT Pipeline Tests
Now that we have the end-to-end tests, we no longer need the webvtt pipeline
tests to verify that it is working.

Change-Id: I4ebec34e66eda67c40999d8802b447e2551e1fa6
2018-03-26 17:57:47 +00:00
Aaron Vaage a191c25f57 Clean Up WebVtt Parser
There were a couple of bad practices in WebVtt that needed to
be cleaned-up.

Change-Id: I0fe21e26f11141709d8d855077805fc625e6dad5
2018-03-23 14:19:52 -07:00
KongQun Yang 6ffa344553 Moves AdaptationSet@id management out of Period class
It is now managed in SimpleMpdNotifier.

This avoids unnecessary increment in AdaptationSet id counter.

Also makes sure the AdaptationSet is sorted by id in XML output.

Change-Id: Ibcd0b047a71c19cd30ad7d8af9a2ed0bb05e043e
2018-03-22 12:15:43 -07:00
Aaron Vaage 3b5b2bccca Implements SyncPointQueue and CueAlignmentHandler
SyncPointQueue manages all cue points and returns aligned cue
points to the callers (CueAlignmentHandlers).

CueAlignmentHandler is responsible for aligning cues from different
streams. It uses SyncPointQueue internally to align / synchronize the
cue points.

Issue: #355

Change-Id: I281fecb46a3ca7172d71e7495bdd07b8efdeb283
2018-03-22 18:21:40 +00:00
Aaron Vaage 833b8165f1 Change Scte35 to use double for time
Change-Id: I2588d3c86d48dc2d9f3692aa0aab28b9de63d109
2018-03-15 18:11:00 -07:00
Aaron Vaage 834ceffd18 Change CueEvent to use time in seconds
Change-Id: I03527db76905f25f0a142940f5fcf73e2a92d42f
2018-03-16 01:10:10 +00:00
Aaron Vaage 631bbdc883 Update Encrypted Tests to Use DiffDir
This changes almost all encrypted tests to use DiffDir. The remaining
un-updated tests are tests that won't match every time.

Change-Id: I858bbaa0beea34887cc3100eb784aa2aecd5a0a5
2018-03-13 16:37:10 +00:00
Aaron Vaage fb2dfe129c Update CheckTestResults to sanitize media info
Change MediaInfo media paths in test output to only use the filename so
that they can be compared between test runs.

Change-Id: I26bb3f5ce79a83400b8a162c61cf5b54e41f9c70
2018-03-13 16:37:10 +00:00
Aaron Vaage 5fee0391fb Change all HLS tests to generate iframe playlists
iframe playlists will be given a name based on the output name of
the main stream. This is done to ensure unique filenames between
different video output streams.

Change-Id: Id0de9c5834599e9d896243e30569c55f34e7cd68
2018-03-07 17:29:47 -08:00
Aaron Vaage b2ce6322b6 Remove Test File Index in packager_test.py
Instead of using the test file's index in a list to create the output
name, use the original filename and the descriptor.

This caused some problems with file name collisions when some tests
were using the same name. That was fixed by changing the names. This
will go away once they are transitioned to use DiffDir like the other
tests.

Change-Id: I0a4c480406705ca63fcea61c86c67d4a5f739295
2018-03-06 21:38:04 +00:00
fatpelt 6a8d2aa443 Add Source Specific Multicast (SSM) support
SSM can be enabled using "source" option in UDP options. Previously, "source" is considered the same as "interface", which is incorrect. 

Fixes #332.
2018-03-06 10:09:28 -08:00
KongQun Yang b24a95b1aa Rewrite init segment in MultiSegmentSegmenter::DoFinalize
This is called when reaching end of the file of the media. The duration
of the media is updated.

Fixes #340.

Change-Id: I446f2d341b02125d4a7d8c958bda269b5403cb9c
2018-03-05 14:25:08 -08:00
Haoming Chen 76d68de6c1 Implement multi DRM support. (Part 1)
- Refactoring current clearkey DRM support for Widevine key source.
- Add playready pssh box in Widevine key source encrypted content.

Partially addresses issue #245

Change-Id: Id30ef02ff9f8dc0e2c58d62479e07896452919cc
2018-03-03 01:39:16 +00:00
KongQun Yang 62bfc1c6c1 Recognize m4s as a valid extension for mp4
Closes #331.

Change-Id: Ic4aaa02245b31a0aa4fdfe08212fa3b242cd9262
2018-03-02 23:22:45 +00:00
Aaron Vaage 7dd88ca5b2 Create CheckLiveTestResults
Created a method to compare test output for live tests. This wraps
CheckTestResults but first replaces the times that would diff between
manifests.

Change-Id: Iafb51a28a9bcb9f32b210c6d76bf23e2b9a3e0d6
2018-03-02 21:00:03 +00:00
Aaron Vaage 033fa65105 Removed Old WebVtt Code
Now that we have the new webvtt code in place, we don't need to keep the
old solution.

Change-Id: I20540ba3adf93f535f0ed011acb8e2555653522a
2018-03-02 20:59:23 +00:00
Aaron Vaage 74c53239dd Updated Nested Directories Test
This require a change to the update gold method as it would not
create nested directories. Instead it deletes the old directory
and copies the whole new directory over. This made the code simplier
and far-less error prone.

Change-Id: I1da5d9bda171b0f106d1425b204139dfbcb39b42
2018-03-02 12:21:39 -08:00
Aaron Vaage 4ace2754fd Avoid Full Path until Needed in packager_test.py
To make working with file names easier in packager_test.py _GetStream
the full path is not created until needed so that multiple paths can
use the same root filename.

Change-Id: I23a9231433b22ab5cef9b88bd6359f351da56eda
2018-03-02 10:27:35 -08:00
Aaron Vaage cd16e95d79 Make All Output Filenames Similar
Changed all output file names (not paths) to follow the pattern that
any qualifier (e.g. trick play) will be join the current name using '-'
but will use '_' within itself (e.g. trick_play_1).

Change-Id: Ib0247bf1ca6d94815fedaaf73d3a400d31c20c40
2018-03-02 10:27:34 -08:00
Aaron Vaage 7dd80884c3 WebVtt Ignore Zero Duration Cues
We have an assert that ensures that the end time is greater than
the start time for any cue. However we never checked that cues
had a non-zero duration when parsing them.

We will throw away cues with a duration of zero (and print a
warning message) as they are not spec compliant.

Closes: #335
Change-Id: I404e8f3a5a8d43eff75a2554db3e38e8d340f421
2018-03-02 01:25:41 +00:00
KongQun Yang de716a6544 Skip reading meta box as it may not be well formed
In the video captured by Android's default camera app, meta box is written as
as Box instead of FullBox specified in the spec.

Closes #319.

Change-Id: I526492fdd505d5929c5161cb1ed1503b724de7e9
2018-03-01 20:21:37 +00:00
KongQun Yang ee9b7f6392 Calculate presentationTimeOffset and Period@duration from segments
Prefer timestamps from Video AdaptationSets if available - this avoids
possible video playback jitters due to gaps.

presentationTimeOffset is not applied to the first period as it may in
negative dts which Chrome does not like: https://crbug.com/398141.

It is safe to apply to subsequent periods as the actual offset applied
takes Period@start into consideration:

    offset = Period@start - presentationTimeOffset

The result timestamp with offset applied is close to Period@start, so
it is unlikely to result in a negative dts value.

Closes b/73899306.

Change-Id: If8361f5469610093b3aac6675754536ad7e83c4c
2018-03-01 08:55:10 -08:00
Aaron Vaage 30b5fdf19f Add Text Codec To Codec String
This change has the hls master playlist add the text codec string
to the list of codecs in a variant.

Bug: 72942756

Change-Id: Ib25bb2064a291d10d7b1d261a4307991df62220c
2018-02-28 19:23:36 +00:00
Aaron Vaage c7929b1505 Remove "with" from test names
Remove the leading "with" from test directories.

Bug: 73830478
Change-Id: I45a0914b5a0e56ec9d3340ec81eb5048c1d653a3
2018-02-27 17:43:16 -08:00
Aaron Vaage bbdcf5ee91 Fix Wrong Remove Function in Packager Test
Change-Id: I58350c2041abfd3b9c533003d408d447b9213123
2018-02-27 22:30:30 +00:00
Aaron Vaage a097aa4d3b Update Many Tests to Use CheckTestResults
Updated as many tests as we easily could. The tests that were not updated
all use live manifests or encryption that all require some "help", either
by dealing with times differing or with verifying decryption.

Bug: 73830478
Change-Id: I6803c2d960b71b459eb57b7a5e562164bb713e2a
2018-02-27 13:32:32 -08:00
Aaron Vaage 60160fe602 Change Old Test Names to Current Standard
Change-Id: Ib74b1726f6515ec562fdb82a2af390e43c36b94f
2018-02-27 13:32:32 -08:00
Aaron Vaage 6e0088c4c0 Change "AudioVideo" test to use DiffDir
Changed the end-to-end test "AudioVideo" to use diff dir
instead of manually checking each file.

Change-Id: Ic1ce1e3664cde7b2b2db262a9fde0cbe2f5b400a
2018-02-27 17:12:50 +00:00
KongQun Yang b81b832522 Create GetStream
Created GetStream, a function used to get the stream descriptor for a
single stream. This allows for finer control over the stream compared
to the batch GetStreams.

Added "trick_play_factor" to GetStream so that the trick play factor
did not have to be added the stream descriptor value and "injected"
into the stream.

Change-Id: Ifbde64b961a673aafa5f6d21f33a68d02fb46610
2018-02-27 17:12:36 +00:00
Aaron Vaage 9547000eb6 Fix Update Golden With DiffDir
Change-Id: I30c79a2acaae2d3066982d84776aaa922e183515
2018-02-27 17:12:28 +00:00
Aaron Vaage 387fe8e33b Isolated OnMediaSample in Chunker
Change-Id: I3af45543a1da1d1cf0b0741dd732dc71b2c30abe
2018-02-23 23:16:35 +00:00
Aaron Vaage 21acb41e63 Isolate OnScte35Event in ChunkerHandler
Change-Id: I9b34f959c22df498a338eb49c87b67003617ead2
2018-02-23 23:12:11 +00:00
Aaron Vaage d9d2cb3e1f Isolate OnStreamInfo in Chunking Handler
Change-Id: If35b9f545f96237095177438a5f95f964cbbfe3d
2018-02-23 23:12:00 +00:00
Aaron Vaage a82ac20dbb Updated "testFirstStream" to use DiffDir
Change-Id: I8c279ba0c208f1e56d53460ae134b3ed8c5f552c
2018-02-23 10:10:34 -08:00
Aaron Vaage 7096ca8998 HLS WebVtt End-to-End Tests
Created end-to-end tests for the segmented webvtt text in HLS. The
text stream is longer than both audio and video, so there are more
text segments than video/audio.

Change-Id: Ib860ccd3f2ac0048983fdeebe3847a0b9dcbc160
2018-02-23 16:55:50 +00:00
Aaron Vaage 393f989839 Create DiffDir in packager_test.py
Created a method to diff the output dir against the golden copy
of the output dir. The goal with this is to simplify our end-to-end
test structure by removing the need to track the output files within
the test runner.

With all the golden output in one directory, this makes it easier to
setup end-to-end tests and even allows the golden files to be playable.

This should not affect the repo size too much as duplicate files will
only be stored in the git object store once.

Tests will migrate to this in separate CLs.

Change-Id: Id1be9f8f60c9e362b9b615445dd8ca7da996af4c
2018-02-20 11:09:06 -08:00
KongQun Yang 74f6cefb82 Change "live" setting to be "segmented"
Changed the "live" setting in _GetStreams to be "segmented" so that it
make sense in a more general sense.

Change-Id: I2ceaad67653e1e2237533b68fccbc9fe3fabfc9b
2018-02-17 00:24:24 +00:00
KongQun Yang 594d1f0de4 Copy over parameter sets in the sample if different to sample entry
When converting from NAL unit stream to byte stream.

The packager should not assume the parameter sets in the sample is the
same as the parameter sets in sample entry (decoder configuration).

Fixes #327.

Change-Id: I7e84d28a296f4b33db0523cca5eabd62f623e852
2018-02-09 15:43:00 -08:00
KongQun Yang 11cbbd86fb Fix fMP4 'cbcs' playback issue in Safari
Looks like Safari does not like v0 tenc box.

- Use v1 tenc box for cbcs and cens protection_scheme as required by
  CENCv3 spec.
- Set crypt_byte_block and skip_byte_block to 0 for full sample
  encryption cbcs and cens.

Fixes #326

Change-Id: I5581cd856fffc4ff104d950f3ca19b9337d57a78
2018-02-08 23:40:46 -08:00
KongQun Yang aa4bb49b2f Fix language option not honored on text streams
Change-Id: I9034b779e9b193c39201a5f2d142ec390927e5f7
2018-02-08 03:48:46 +00:00
KongQun Yang 7dcab4c799 Fix '--temp_dir' option does not work problem
Fixes #322

Change-Id: Ia0ea32029c81fd8ac5661f8002154e2a54d3a84d
2018-02-07 13:46:13 -08:00
Aaron Vaage a82edb5228 Make StreamDescriptor Object
Create a StreamDescriptor class that can be used to build the stream
descriptor command line argument.

Change-Id: Ifcba4f37d69a1a36c4add51a2a10c3f0c8b2ccd7
2018-02-07 19:22:05 +00:00
KongQun Yang 221ac81772 Prefer Period@duration for static MPD with >1 periods
It is easier to insert Ad Periods with Period@duration compared to
Period@start.

Change-Id: Ib52e81612562bf60b0e0513a09d9a31c42b09604
2018-02-07 01:13:28 +00:00
Aaron Vaage 48cb55c8d4 Correct WEBVTT Header In Sample Test Files
The old vtt sample data has the header as "WEBVTT FILE" which does
not reflect what the spec says it should be.

This changes it to "WEBVTT" as described in the WebVTT spec.

Change-Id: Ib54396a8b937501a835dafe966bf05b95b0b22c4
2018-02-07 00:07:16 +00:00
KongQun Yang 2d5e2b0903 Fix DCHECK of SimpleThread on invalid command line options
- Construct SimpleThreads only after all workers have been successfully
  initialized

b/72999680

Change-Id: I4016321a4bdd27b1de746c50c5f1d7a60d7ae9f1
2018-02-06 22:41:38 +00:00
Aaron Vaage c7eb2b4a7e Output Empty WebVTT Text Segments
Have the text webvtt pipeline output empty segments rather than skipping
them.

Bug: #324
Change-Id: I175cf22b65b241f6bf566aba2440518f3df95277
2018-02-06 13:32:42 -08:00
Aaron Vaage f35410deb1 Output Empty MP4 WebVtt Cues
When a gap is found in the text stream, the WebVtt to Mp4 converter will
now output the special empty vtt cue.

Change-Id: I8be88c6b7589aa120a2215e1e4b8e98031fe326d
Closes: #324
2018-02-06 16:45:04 +00:00
KongQun Yang 8257eea804 Bug fixes and clean-ups for I-Frame playlists
- Add empty lines between different types of renditions to improve
  readability.
- Group variants with the same audio/text group together, as it is
  where the Adaptation occurs.
- Write master playlist after writing media playlists. This makes
  more sense and it is also necessary to have the bandwidth of
  the last iframe playist segment correctly computed.
- For fMP4, I-Frame segment must include the 'moof' header.
- Fix a problem that hls_iframe_playlist_name is not passed to
  MuxerListenerFactory.

Issue: #287

Change-Id: Icf37c5de1dc29f85ae3f419cbc3264d04ca491a4
2018-02-05 17:36:28 -08:00
KongQun Yang 1750357024 Support EXT-X-I-FRAME-STREAM-INF in master playlist
Issue: #287

Change-Id: I07fdfa095fe1797b3aa091d48798a2b5fbb4dfbe
2018-02-06 01:11:25 +00:00
Aaron Vaage c991490e82 Prepare WebVTT To MP4 to handle gaps
Adding all the code needed to allow the webvtt to mp4 converter to
send empty vtt cues. This change does everything except writing the
mp4 box.

Bug: #324
Change-Id: I16188e6357632b2ed06f7e9bab7844f093266696
2018-02-06 00:55:32 +00:00
Aaron Vaage 9452a1bb1a Make Segments Their Own Struct
Instead of always tracking segments with an index, this change makes a struct
that will act as the segment and track which segments are in it. Each
segment will store all the samples in the order they were given, it will
avoid any sorting.

Closes: 72867775
Change-Id: Ic5829161510fe8f3320d960c3bc4a276c26ff3be
2018-02-06 00:55:00 +00:00
Aaron Vaage 80ed014cf8 Create Preserve Order Test
Create WebVtt test that shows that the current implementation of the
webvtt segmenter does not respect the input order of cues.

Bug: 72867775
Change-Id: I811b3a93c10650e0cf9290c9c0c1680f562deb30
2018-02-06 00:54:15 +00:00
KongQun Yang 04577b9638 Refactor MasterPlaylist::WriteMasterPlaylist
Change-Id: Ida7453761748087cd13aab640c400381696578e1
2018-02-02 14:13:12 -08:00
KongQun Yang b647c6731c Support KeyFrame events in TS and MP4
Issue: #287

Change-Id: I7c50853c7cc61c5fadbb620f44f7574a27dc2b68
2018-02-02 14:05:50 -08:00
KongQun Yang 570a2d1a15 Support 'iframe_playlist_name' stream descriptor
Issue: #287

Change-Id: I484761dfacadce05175d16c4c12454f0ed932579
2018-02-02 22:00:18 +00:00
Aaron Vaage e98a150d62 Added DEFAULT and AUTOSELECT for Text
Took the same logic for DEFAULT and AUTOSELECT used by audio and
applied it to text. Combined the build tag logic for audio and
text as they were the same expect for a couple fields.

Bug: #205
Change-Id: I75ecbf4b25cd559b826982d12a5b132e70b83b69
2018-02-02 11:07:38 -08:00
Aaron Vaage e2dae2d960 Add Subtitle Support to HLS Playlists
The master playlist and media playlist did not have implementations
for handling text streams. This change adds support for both.

Bug: #205

Change-Id: I1329b8cc2585f15b89959071db9dd16d35847cba
2018-02-02 18:33:13 +00:00
KongQun Yang 82735be58d Support key frame events in MuxerListener and HlsNotifier
Issue: #287

Change-Id: I33b91b73988fc40e113bd2e627d08f549a7e3dc5
2018-02-02 00:00:53 +00:00
Aaron Vaage 8104628f48 Use Tag in Media Playlist
Instead of formatting the tag manually in MediaPlaylist, use the tag
class from MasterPlaylist.

Change-Id: I4099169a80a6d8595ab7f49e21cac4e7e0614832
2018-02-01 22:59:45 +00:00
Aaron Vaage 44847a0737 Changed Group Name Fallback
If no hls group name is given, we would default to "audio" but as
we are working to support text, this won't make sense to see
GROUP="audio" on a text stream.

Instead default to "default-audio-group" for audio streams. A specific
text default will be added later.

Bug: #205

Change-Id: Iefc8bfd35708286619d0004348294d98f2c38482
2018-02-01 10:24:58 -08:00
Aaron Vaage 1731fef14d Move Tag to Own File
Moved HLS Tag to its own file so that it can be used in other
HLS classes.

Change-Id: Ie3c2668fde28e43784661fc789d2edc80fc4a8fa
2018-02-01 17:02:46 +00:00
Aaron Vaage 0f17631017 Use Variants To Build Audio and Video Tags
In the master play list, use a class called variant to build
the video tags.

The variant list will always have one entry. In the case that there
are no audio streams, the variant list will have a null entry.

Bug: #205

Change-Id: I6e4acd83a31cd267c173e4f4c910f93781fe6efd
2018-02-01 17:02:31 +00:00
Aaron Vaage 771944a3f5 Output Empty WebVtt Segments
According to the HLS+WebVTT spec, if there is no text in a segment
a webvtt file with no cues can be added to the manifest. By outputting
empty segments it allows the accumulated duration in the Master Playlist
to better represent the duration of the text stream.

Spec Reference: //tools.ietf.org/html/draft-pantos-http-live-streaming-23
  Section 3.5. WebVTT

Bug: #205

Change-Id: I5de01200fd9fa99c57949c773e8ee926b0f6ba8a
2018-02-01 17:02:19 +00:00
Aaron Vaage ea96b72d6a Consolidated Making MockMediaPlaylists
It was the same code to make the media playlists for each test, so this
change moves that logic into common functions.

Change-Id: Ifb8680247ad4335332251815a2db5d56d28151aa
2018-02-01 02:21:30 +00:00
Aaron Vaage 48b7282257 Creating HLS Tag Class
Made a tag class to make it easier to build tags for hls
content.

Bug: #205

Change-Id: Ifad6a690c3ade90c089f96ace870a62fd2ef420f
2018-02-01 02:03:37 +00:00
KongQun Yang 322337a958 Order Representations and AdaptationSets by id()
Change-Id: I04509819c1f8fa78e4826d53966531bf98e90849
2018-01-31 21:56:21 +00:00
KongQun Yang 6d0a6bb120 Support I-Frames only playlist in MediaPlaylist
Issue: #287

Change-Id: I204774fd87cf9a159a4448ba3457db5ab17f7889
2018-01-31 21:37:34 +00:00
KongQun Yang 4194f0746e [DASH][WebM] Fix subsegmentAlignment not set problem
MuxerListener::OnNewSegment was not sent for WebM SingleSegmentSegmenter.

Change-Id: Iecae84e7a7c1a8a4d3f577158fa286b9c9959934
2018-01-31 19:58:15 +00:00
KongQun Yang 610e112d16 Cleans up Status class
- Removed SetError, Swap, Matches functions, which are not used
- Added moveable version of Update function
- Added VLOG(1) warning in Status for non-ok status
- Replaced OS_WIN with _WIN32 as OS_WIN may not have been defined.

Change-Id: Ib6b7aab6e6fee270937b150f1e4bf993e914a568
2018-01-31 03:32:07 +00:00
KongQun Yang 84f1f96f05 Rename SAMPLE-AES-CENC to SAMPLE-AES-CTR
Per the definition in the latest specification:
https://storage.googleapis.com/wvdocs/Widevine_DRM_HLS.pdf

Change-Id: Ic9b222c72ca8fbc4ae011e6557f2c4daaad2e624
2018-01-29 15:45:16 -08:00
Aaron Vaage c520d85b96 Move File Copy Out Of StreamInfoToTextMediaInfo
Moved the file copy out of StreamInfoToTextMediaInfo as the file
copy is only needed by the MPD output but the Media Info is needed
by MPD output and media info dump.

Bug: 36138902
Change-Id: Id233f2041b3e72345b8f709791c6b6070484222f
2018-01-29 17:22:43 +00:00
KongQun Yang 9ecee46658 Fix SAMPLE-AES EC3 encryption
The IV was incorrectly updated across samples.

Created a new cryptor SampleAesEc3Cryptor specially for SAMPLE-AES
EC3 encryption / decryption. The new cryptor uses constant-iv, and
makes sure the IV is reset to the initial value at the beginning
of each audio frame and chained across syncframes within the the
audio frame.

Also added E-AC3 end to end test.

Fixes #279

Change-Id: I0aa60c17836daeef5ba433a05e5ff0906191d9ac
2018-01-24 03:02:49 +00:00
KongQun Yang 1455f43c02 Share chunkers from the same input except for WVM
Updated test files and added end to end tests for AdCues.

Change-Id: I28f57c2f0d15f65fa04f599fae446a5c1373bf47
2018-01-22 17:43:25 -08:00
KongQun Yang b74e511f29 Workaround for possible rounding errors in MSE
There could be rounding errors in MSE which could cut the first key
frame of the representation and thus cut all the frames until the next
key frame, which then leads to a big gap in presentation
timeline which stalls playback.

Adjusting presentationTimeOffset (PTO) by -1, i.e. backing off PTO by 1
to compensate for the possible rounding error. It should not cause any
playback issues as it is small enough. The workaround can be removed
once the problem is handled in all players.

The PTO adjustment is configurable with command line flag:
 --pto_adjustment, with value set to -1 by default.

b/71808910

Change-Id: I9c4d1e2ee84008b859a3638a8146c910cead1f15
2018-01-20 01:18:45 +00:00
KongQun Yang c9fb2b4a85 Use start_time from CueEvent in NotifyCueEvent
Instead of using next segment start time, as CueEvent time may not
align with segment start time exactly.

Also remove the incorrect DCHECK in ChunkingHandler when processing
kScte35Event.

Change-Id: I4987740c99c8d0d25c9b99bddc5e557e45d308e0
2018-01-20 01:18:38 +00:00
KongQun Yang d9871330e7 Move common manifest flags out of mpd_flags.cc
Also updated documentations.

Fixes #304

Change-Id: I5e8e87b005ebfa9dc78d47e2501585b083120f6c
2018-01-19 21:02:54 +00:00
KongQun Yang 3c30ec3a11 Fix test failures in previous CL
Change-Id: I54dea7fe62f30a9a9648559922c02d5bad0236e3
2018-01-18 18:26:10 -08:00
KongQun Yang 32d26094ba [HLS] Support AUTOSELECT and DEFAULT in EXT-X-MEDIA
According to HLS spec: https://goo.gl/MiqjNd 4.3.4.1.1. Rendition Groups
- A Group MUST NOT have more than one member with a DEFAULT attribute of
  YES.
- Each EXT-X-MEDIA tag with an AUTOSELECT=YES attribute SHOULD have a
  combination of LANGUAGE[RFC5646], ASSOC-LANGUAGE, FORCED, and
  CHARACTERISTICS attributes that is distinct from those of other
  AUTOSELECT=YES members of its Group.

We tag the first rendition with a particular language in an audio group
with 'AUTOSELECT'; it is tagged with 'DEFAULT' too if the language
matches --default_language.

Fixes #315

Change-Id: Iacc0bc8c89ebffce8717fa65e82d6daf5a1f6adc
2018-01-19 02:10:07 +00:00
KongQun Yang 53892c7467 Add end to end test for ac3
Also updated bear-640x360.ts test files.

Change-Id: Iff53800b7b4706fb2f6d94682b9446406a94cd6f
2018-01-18 17:23:20 -08:00
KongQun Yang 435d69c3a6 Insert EXT-X-PLACEMENT-OPPORTUNITY on CueEvent
See https://support.google.com/dfp_premium/answer/7295798?hl=en

Change-Id: I931427512cb695dc68d5c8c2ec214fe743161bcb
2018-01-18 17:07:55 +00:00
KongQun Yang feabf770bf Sync streams coming to ChunkingHandler
Change-Id: I50d1a64f2ec3c50dfb1b7282241b47f3a55e26ce
2018-01-18 16:23:59 +00:00
KongQun Yang d76ccea46f [DASH] Support multiple period
Change-Id: Ifd17bf0eabbd61ec7a1d35f0b864b5aa6666aa87
2018-01-11 21:44:18 +00:00
KongQun Yang 2ba23e1075 Fix incorrect CHANNELS reporting for AC3/EC3
Fixes #313

Change-Id: I0b8d8081cda8aaa4db2ca4916bfc6721bfa3bd22
2018-01-11 03:50:56 +00:00
KongQun Yang 858fcd8698 Fix incorrect HEVC decoder configuration data
When transmuxing from TS to MP4 if there is emulation prevention
bytes in general_profile_tier_level_data.

Fixes #312

Change-Id: I48857046c1ea4bed4f455d03d060551e3ef4bec8
2018-01-10 23:53:35 +00:00
Aaron Vaage 5d5cb8678d Support WebVtt Text to WebVtt Mp4
Added new path to allow webvtt text input to be written as
webvtt mp4. This will happen when the input is a vtt file and
the output path is a mp4 path or the output format is mp4.

Missing test content for this.

Close: 70990714

Change-Id: Idbb4023db4a126a560528e73801f762cc45f4685
2018-01-10 12:21:56 -08:00
Aaron Vaage c57da2b42d Create Muxer Listener Factory
Moved muxer listener creation into a factory so that it can be
initialized once and then the factory only needs to be passed
around rather than all the parameters.

Change-Id: Ibf7df66c08debddbe6f6ff8995e7910b9ad17f3f
2018-01-10 18:59:11 +00:00
Aaron Vaage c877af9f3b Use Mock Muxer Listener in WebVttOutput Tests
Use a mock muxer listener so that we can verify that the webvtt
output handlers will write to a manifest correctly.

Change-Id: I0294a998bfaf06a6d8f7d4c287fb014839b38f73
2018-01-10 17:51:39 +00:00
Aaron Vaage 49933737cd Moved MuxerFactory to own file
To remove the direct dependencies between packager.cc and the different
muxer implementations, this change remove the muxer factory to its
own file.

Change-Id: I4a1503c493f0f4d10a67e78461f96da3f4238944
2018-01-10 17:51:26 +00:00
Aaron Vaage 1e9e5d1f58 Integration Test for WebVtt Pipelines
Connected the WebVtt Pipeline together to test that the parser,
segmenter, and output all work together as expected.

Change-Id: I07138f7b5b1318f84c27c5b607d8df207c57ddb3
2018-01-09 22:14:04 +00:00
KongQun Yang ef2c424876 Implement Representation::GetDurationSeconds
Also added Period::GetAdaptationSets and
AdaptationSet::GetRepresentations.

Instead of implementing GetDurationSeconds in all MpdBuilder classes
(MpdBuilder/Period/Adaptation/Representation) like what we used to do
with GetEarliestTimestamp, the two new functions allows MpdBuilder to
iterate through the Representations to get durations.

Also updates GetEarliestTimestamp functions to use the same iteration
method.

Change-Id: I682b70c07c248c0f6511ec3d9019086f986ee10e
2018-01-09 01:03:05 +00:00
Aaron Vaage 509963d60f Create HLS Text Pipeline
Linked together all the handlers to create the HLS text pipeline.
This change does not include tests for the pipeline.

Bug: 36138902
Change-Id: I32e7a7cefae1e7c74ec6d0bdc6335c0d65e5e79c
2018-01-09 00:03:29 +00:00
Aaron Vaage 4ecb567daf Changed muxer listener to use "default"
Changed muxer listener to use "default" instead of {}
when defining its constructor and destructor in its
header file.

Change-Id: I3c80e808aa624293c3d7720ac6a47b470a663312
2018-01-08 17:52:04 +00:00
Aaron Vaage 37b5f401d1 Support MP4 to Mp4 Text
This change adds the missing code for passing mp4 text through
the pipeline.

This introduces the MuxerFactory to simplify creating muxers.

Change-Id: Ic7be1a2e0ff4b720a01061edb43aded946d05a47
2018-01-08 17:41:49 +00:00
KongQun Yang 5f48cbb0c1 Fix build failures in MacOS 10.12.6 with XCode 9.0
- Update curl to 7.57.0
- Roll clang, which is needed due to MacOS / XCode update
- Fixes and suppress compilation errors due to clang update

Fixes #285

Change-Id: Ibac3288c641861605c3c0500d34d27373e6eecfe
2018-01-05 12:21:41 -08:00
Aaron Vaage c88fe54553 WebVTT Output
This change introduces handlers to output WebVtt text files. There is
only one output but there is a common base to support others.

WebVttOutputHandler which handles all communication with other handlers
and WebVttSegmentedOutputHandler is responsible for listening for events
and choosing when and where to write cues and headers.

Bug: 36138902
Change-Id: I2b13a94262554398e66fee8cf024aa21041ddbab
2018-01-05 00:05:48 +00:00
Aaron Vaage 9e7b5c1ca9 Change CharReader to FileReader
Instead of having multiple char readers, one for strings and
one for files, just have one for files and use memory files
when testing.

Change-Id: Id1a2230046ba540ddf69ca10edb3edc74d2419b6
2018-01-04 14:36:52 -08:00
KongQun Yang db74d6756e Propage CueEvent to MpdNotifier/HlsNotifier
Change-Id: I9828af11a28300d20cc8742251bbe1b4ebfdbad1
2018-01-04 18:25:03 +00:00
Aaron Vaage b0edec8c40 Changed Text Segmenter to output segment info last
To better align with the Chunking Handler, the Text Segmenter
now outputs the SegmentInfo at the end of the segment rather
than the start.

Change-Id: If69ab951947d00779b4b63a52c4b6662bbdc4c0d
2018-01-03 23:40:37 +00:00
KongQun Yang 66f713fd23 Merge SimpleMpdNotifier and DashIopMpdNotifier
Also removed MpdNotifier::AddContentProtectionElement which is not
used anywhere.

Change-Id: I69eb596ec377c601bb3e45bbef30c248c1d03e86
2017-12-22 15:56:14 -08:00
KongQun Yang 82842407cb Add content_protection_in_adaptation_set_=false tests
Also make AdaptationSet::AddAdaptationSetSwitching virtual to make it
easier to mock.

Change-Id: I7f66fa0c4824607f77559a61227aaef6cd153f9f
2017-12-22 15:53:35 -08:00
KongQun Yang 718fce068d Refactor DashIopMpdNotifier Part 2
Move AdaptationSet related functions to the new Period class, which
maps to <Period> element and provides methods to add AdaptationSets.

Change-Id: I0fee290769fbe9a6355cc1b8c86baec8fbc4b4fd
2017-12-22 13:24:38 -08:00
KongQun Yang 99469834e8 Refactor DashIopMpdNotifier Part 1
Clean up DashIopMpdNotifier to make it easier to add Period class.
- Consolidate protected AdaptationSet grouping related logic to the
  internal ProtectedAdaptationSetMap class
- Clean up AdaptationSet creation logic and updating logic

Change-Id: Idd5b39bb89d38e490ca4f561a6b840ccc9f1e40a
2017-12-22 21:20:13 +00:00
Aaron Vaage ab19082c20 WebVtt To MP4 Handler
Implemented a MediaHandler that takes text samples and creates
media samples. The data in each media sample is the MP4 box for
non-overlapping cues.

As per WebVtt in Mp4, all cues must be non-overlapping. This handler
takes care of grouping and dividing cues.

Bug: 36138902

Change-Id: I0c1d27964180c14a22cb200591f70e46e04a651f
2017-12-22 21:08:17 +00:00
KongQun Yang fcc334d652 [Doc] Fix a few problems
- Add --time_shift_buffer_depth to HLS doc.
- Fix a problem in HLS tutorial

Partially address #304, fixes #305

Change-Id: I60bc1eb659a30beb0a6ab73433f4cc99a7deab07
2017-12-21 23:07:59 +00:00
Aaron Vaage c03c447e0b Created WebVtt Segmenter
This change creates the webvtt segmenter which will be used for
segmented text webvtt for HLS. It takes a stream of text samples
and injects segment info messages based on the segment duration.

It is possible for samples to extend between segments, if so the
sample will appear twice (once in each segment).

Change-Id: Iae0134ee61cf269948026086520b6d3f8ce3785b
2017-12-21 21:12:34 +00:00
Aaron Vaage 4dcfe413f5 WebVTT Parser
Took the WebVTT Media Parser and created the WebVTT Parser
that will take in a file and output a stream of cues that
will later be passed to another Media Handler that takes in
cues and chunks them.

Bug: 36138902

Change-Id: Ic77813fe19678e85d500269e69f46917510ab7ec
2017-12-21 21:12:18 +00:00
Michael Stillwell a8f46fcd2e Update playlist_name docs (#307)
Make it clear that shaka packager will create the playlist file.

#305
2017-12-20 10:25:58 -08:00
Haoming Chen feeb7361ff Clean up #ifndef in packager/media. (#310) 2017-12-19 16:56:36 -08:00
Haoming Chen f22320bfa7 Remove non-exist class MediaStream. (#308) 2017-12-18 13:28:19 -08:00
KongQun Yang 55edaf0bea Move SegmentTemplateTest and TimeShiftBufferDepthTest too
Moved from mpd_unittest.cc to representation_unittest.cc.

Also removed unused test data files in packager/mpd/test/data.

Change-Id: I23811390b4b1ffbfac3e69c1ffcf92544099a53d
2017-12-15 20:05:32 +00:00
KongQun Yang cacdab2209 Split mpd_unittest.cc into three files
Move AdaptationSet related tests to adaptation_set_unittest.cc and
Representation related tests to representation_unittest.cc.

Change-Id: I360e9aea3ba08a32e5ba3b0e8b34345f2b3f0b20
2017-12-15 20:05:15 +00:00
KongQun Yang 327537df36 Clean up mpd_unittest
Added gMock matchers for matching xmlNodePtr and its attributes.

Clean up mpd_unittests to make it easier to separate adaptation set
tests and representation tests.

Change-Id: I31816b06e9c76f92d4a82656c659f3b9acae8cb5
2017-12-15 20:04:47 +00:00
KongQun Yang 86d960bea6 Move AdaptationSet, Representation out of mpd_builder.h
Change-Id: I61fffb4d956f189b44c7537ddcf183bbf4129840
2017-12-14 21:28:06 +00:00
Aaron Vaage dedbd8b724 Isolate Job Management
Moved all code dealing with jobs into its own class so that the
packager code does not need to worry about initializing, running,
or stopping jobs.

Change-Id: I3e9ef1f22bd93d671f77d59ad15f23d1239078cf
2017-12-13 18:35:51 +00:00
KongQun Yang 2d025dae2d Include shared_library and mpd_generator in ci and releases
Change-Id: Ie9449cbca711324831cb15e83d0efaf7a43cf9ef
2017-12-12 23:11:01 +00:00
KongQun Yang feb7951f33 Add docker files for testing on various linux distros
Change-Id: Ic05e2dc326a2b70c43132118d78f6d496d8b7e18
2017-12-12 22:51:05 +00:00
Kamesh Devarakonda 4e3a24fb41 Determine chunking point
Change-Id: Ic5690f803c1a2c22b80785aefbc1f1cef2f11c38
2017-12-12 14:24:28 -05:00
Kamesh Devarakonda 7e46ad1021 Capture SCTE35 events
Change-Id: Id06239ed4820104fecbca08a4103a47ecbed5eda
2017-12-12 01:20:52 +00:00
KongQun Yang b3194866a6 [HLS] Add CHANNELS attribute to #EXT-X-MEDIA
As specified in HLS spec (https://tools.ietf.org/html/rfc8216):

All audio EXT-X-MEDIA tags SHOULD have a CHANNELS attribute.  If a
Master Playlist contains two Renditions encoded with the same
codec but a different number of channels, then the CHANNELS
attribute is REQUIRED; otherwise, it is OPTIONAL.

Fixes #299

Change-Id: Ic2308c39b170178b11cb0d94c3a8083c8b5f3353
2017-12-12 01:04:21 +00:00
KongQun Yang 8e96dd3b37 Address gcc build failures in some platforms
Seeing some failures on some platforms when compiled with clang
disabled:
  GYP_DEFINES="clang=0" gclient runhooks

Several changes to make it work:
1. Mark packager code with packager_code=1 in GYP definitions.
2. Disable a few checks in non-packager code, which we do not have
   direct control: dangling-else, deprecated-declarations,
                   unused-function
3. Fix the relevant errors in packager code.
4. Revert HAVE_STROPTS_H in curl config which is not available in
   all linux distributions.

Fixes #286
Fixes #293

Change-Id: I729b41f99403c5ad9487c6cc4a7dc06f6323cef8
2017-12-07 11:45:26 -08:00
KongQun Yang e97c89391a Address flakiness in ProducerConsumerQueueTest
1. Expect the time elapsed to be greater than kTimeout instead of
   approximately equal to kTimeout if there is a "wait";
2. Expect the time elapsed to be smaller than kTimeout instead of
   approximately equal to 0 if there is no "wait".

Also rename a variable in packager.cc to better reflect what it is.

Change-Id: I67975a6263b8dbc1124b78feae0f8e0d112bda50
2017-12-05 22:56:22 +00:00
KongQun Yang 9631ec71de Roll curl to 7.56.1
Fixes #285

Change-Id: I3cda80974e8cc66f11763c41c36d832565077374
2017-12-04 19:28:27 +00:00
KongQun Yang 7097270f13 Roll boringssl to fc9c67599d9bdeb2e0467085133b81a8e28f77a4
Change-Id: I8ff75ede6c5b6a51719163d208c96d6ec0be61bc
2017-12-04 19:27:34 +00:00
KongQun Yang 82e0fd2095 Default multicast interface address to "0.0.0.0" if absent
It aligns with what a lot of other apps are doing, e.g. tsplay.

Also added notes in docker to adjust network setting if wants to use
multicast in docker container.

Change-Id: I666a8979cc041c904f7ff8e3e022f800c0830d5d
2017-12-04 19:21:52 +00:00
KongQun Yang 0fdb0d02aa Log an error when seeing unsupported stream type in MPEG2TS
The error is logged only once to avoid log spamming.

Added definitions for the unsupported stream types for easy look up.

Change-Id: I097e2f05759bc84ef03f264cfabd2fb20da7c711
2017-12-01 14:56:22 -08:00
Kamesh Devarakonda e7540bf303 Dispatch Scte35Events.
This patch also fixes a bug which was introduced in
https://widevine-internal-review.googlesource.com/c/packager/+/38040
where it was inconsistently assumed that AdCueGenerator would be MIMO
while connecting with other handlers and at the same time it was assumed
SISO in the AdCueGenerator class. Now we assume SISO everywhere.

Change-Id: Icd8c40b5ccfe0d98f47f09a60ea1635f74dceef9
2017-12-01 02:23:36 +00:00
KongQun Yang 068e220ac7 Handle additional unused 'mdat' properly
The spec allows having more than one 'mdat' boxes even if it is not
used.

This is happening on some mp4 files in the wild:
http://www.sample-videos.com/.

Fixes #298.

Change-Id: I729cc94bee095560b5c4b3ee8511323f25f7ad5a
2017-12-01 00:28:30 +00:00
l-law 22c758ed06 Fix order of VPS, SPS, PPS in hvcC box of HEVC/MP4 (#300)
Fix order of VPS, SPS, PPS in hvcC box of HEVC/MP4

Fixes #297
2017-11-30 14:34:42 -08:00
Kamesh Devarakonda cc778f60ab Defined CueEvent struct.
Change-Id: If4ba55a7ff403248de9d0411c2ba8553b88c491b
2017-11-28 14:26:32 -05:00
Kamesh Devarakonda c4e7d02b80 Connect AdCueGenerator in the pipeline.
Change-Id: Iaae73661bc57ddecd2cc103dbd34d0c3102ffdaa
2017-11-27 21:09:26 +00:00
Kamesh Devarakonda 154e309f3f Remove unused argument.
Change-Id: I1ee3ca6527ec4ecce7dd57e18eee70a8c6a11eb0
2017-11-21 15:04:36 -05:00
Kamesh Devarakonda dac0328b47 Parse ad_cues flag.
Change-Id: I4f5897dcde20e73a01468b80a04d3f3142bc5e42
2017-11-20 16:39:16 -05:00
Kamesh Devarakonda 622ff64595 Add skeleton AdCueGenerator handler
Change-Id: I53ba982eb76c7f61122e226d295e6989fe425ae5
2017-11-17 00:06:27 +00:00
Kamesh Devarakonda dcf2d2fc0b Created a build target for files in media/public/
This new build target is explicitly listed in the packager/packager.gyp
file.

Change-Id: I49e68ea64ae9a9a30204a269a1af17744f9d25d7
2017-11-16 00:24:17 +00:00
Kamesh Devarakonda 8c877379b6 Defined Scte35Event struct and out of band cue marker flags
Change-Id: I5a4db1a691544cbedf2ccfbaf6f71169a8eed6e7
2017-11-16 00:13:29 +00:00
KongQun Yang 5cf2b17ade Add --hls_key_uri to allow setting fairplay/identity key uri
Change-Id: I52e0f56cd10390faf515170f407ea488a6b3c9fc
2017-11-14 15:49:27 +00:00
KongQun Yang 61e36d7d21 Support encrypted E-AC3 in TS
Implemented E-AC3 MPEG-2 Stream Encryption Format for HTTP Live
Streaming specified in https://goo.gl/1sgcwY

Issue #279

Change-Id: I36c1a05e3d0529ff810eaf52bdca45414baa93eb
2017-11-13 22:54:07 +00:00
Kamesh Devarakonda 09cdb2bbec Removed unused structs PeriodInfo and MediaEvent.
Change-Id: I0504853ec555fd161df21e5680d26aab50648052
2017-11-09 17:28:31 +00:00
KongQun Yang 0a69779f7c Fix a parsing problem with AC3 in TS
Audio samples per frame for AC3 was not specified correctly earlier.

Also the number of channels is not correct if LFE channel is on.

Issue #165

Change-Id: Ibf20aa4c7aec43c07ec7cd394d631c537cb387dd
2017-11-07 18:47:09 -08:00
KongQun Yang 72df5af150 Support encrypted AC3 in TS
Implemented MPEG-2 Stream Encryption Format for HTTP Live Streaming
specified in https://goo.gl/N7Tvqi.

This change also moved ProgramMapTableWriter creation from TsWriter
to TsSegment.

Issue #165

Change-Id: Ia89dd16a5e6405706dd3d85ec6b6de580f7b13a7
2017-10-30 15:22:22 -07:00
KongQun Yang d7f531fe10 Add AC3 support in TS
EC3 and encrypted AC3 in TS are not handled yet.

Partially address #165

Change-Id: If4839ee7801eac902e64e9c677dd37709ec8e88b
2017-10-30 15:11:43 -07:00
KongQun Yang ad836a5cf1 Rename EsParserAdts to EsParserAudio
Change-Id: I8d9fc265a4cf740970256e1a46b172349f1bf794
2017-10-27 23:58:50 +00:00
KongQun Yang 79a54d7748 Cleanup ProgramMapTableWriter
Change-Id: I08093a0fc35dc15a20f6e72166ed46d3ab94b334
2017-10-27 23:58:32 +00:00
KongQun Yang 2a2493ec2e Refactor EsParserAdts to handle other audio
Rename will happen in next CL

Change-Id: I28df29bb264eea7a6225b0a4a2eb38de7968f9d6
2017-10-25 23:21:35 +00:00
KongQun Yang 867244645a Implement BitWriter
Change-Id: I5b7dfc2bc07227c2fe05f6de8857042bcab944c4
2017-10-25 16:19:12 -07:00
KongQun Yang b5a8185543 Refactor ProgramMapTableWriter
To make it easier to support more codecs.

Change-Id: Ifcde0f3d7d6f74015d723a0b74401a86c8e65a72
2017-10-25 20:25:56 +00:00
KongQun Yang d66d307fc2 Rename fixed_key to raw_key
Change-Id: I1ee9398387b54aa685a5bef5b8704849e62ebae3
2017-10-24 21:52:01 +00:00
KongQun Yang 7c38963c29 [doc] Update raw key tutorial
Change-Id: Ife841975c036cf8042c9775583c409e0daa089c7
2017-10-24 21:51:39 +00:00
KongQun Yang 1bf1ec2445 Add multi-key support in fixed/raw keys
The keys can be specified in --keys option, with the form of
  label=<label>:key_id=<32-digit hex string>:key=<32-digit hex string>,label=...

There can be multiple "label=..." block. The DRM label can be one
of "AUDIO, SD, HD, UHD1, UHD2" or a custom label.

Mark --key and --key_id as deprecated. We can achieve the same result
with --keys=label=:key_id=<key_id>:key=<key>.

Also add a new optional field in stream descriptor: drm_label, which
is used to overwrite the internally generated DRM label for the stream.
If not provided, the DRM label is generated automatically based on
audio/video information, e.g. resolution.

Code changes:
- Merged RawKeyEncryptionParams and RawKeyDecryptionParams into
  RawKeyParams.
- Make FixedKeySource accepts RawKeyParams as input.

Change-Id: Ic8c2f071cc71188e13f14bc6396fc2b3ffa5cac6
2017-10-24 21:04:49 +00:00
KongQun Yang 27c1900f12 [pssh-box] Support generating Widevine PSSH without content-id
Widevine PSSH with key-id and without content-id is now be supported.

Change-Id: I356590b79f79ff4c0310b22e5dc19b6d24ec4cee
2017-10-18 21:38:23 +00:00
KongQun Yang c9cc68ba82 [Playready] Allow unencrypted client cert private key
Change-Id: I3244b428f8e3e32787520d9dd0f015e6d9301fe0
2017-10-18 21:37:36 +00:00
Joey Parrish e2bb22c633 Update docs for HLS descriptor fields
These fields are no longer required, and have sensible defaults.

Change-Id: I35bd489c8e6058f3272210cb6f3b0f504819e7c0
2017-10-17 11:37:44 -07:00
KongQun Yang e889578a2d Creates directories in the output path if not exist
Also added end to end tests for single segment HLS and multi-segment
HLS.

Fixes #276

Change-Id: I27e5d57cdc3ab79a8e5ae0c3a8da46fcf2c467b6
2017-10-11 20:19:51 +00:00
KongQun Yang c87c4c2327 [HLS] Fix single segment ULR isn't respecting folder location
Fixes #253

Change-Id: I61de1b28807ff8be9a881edd3adaa36612713dd8
2017-10-11 00:14:09 +00:00
KongQun Yang 0861452932 [Test] Rotate key_ids in PSSH as well for key rotation
Note that fixed key / raw key rotation should not be used in production.

Change-Id: I87fca41ebd00f3cadbf157ff129b8e6a9ca04e7c
2017-10-09 16:44:14 -07:00
KongQun Yang ae17159b73 [HLS] Fix init segment URL isn't respecting folder locations
Problem occurs when the media playlists are in a sub-directory under
master playlist. If base_url is not set, segment URL should be
relative to the media playlist.

This only affects fMP4 as TS does not have init segment.

Fixes #253

Change-Id: Icddd9ed500d0a705e8b3260bfd4e916ecbba3f28
2017-10-06 17:19:13 -07:00
KongQun Yang 94a64bef03 Fix WebM parsing crash when the source content contains color elements
Also updates outdated comment in packager.h related to multi-manifests.

Fixes #275

Change-Id: Ia2eb235607860f92a3c19bd791d711209da85c08
2017-10-06 22:48:04 +00:00
Aaron Vaage 65d5ecc3cf Separated Text Processing From Audio/Video
To help isolate the differences between audio/video and text in
preparation for adding the better text support, this change moves
all audio/video job creation and text job creation into their
own functions.

Change-Id: Ia71cf82ac147672b70c69123eeabdfc6b41c4a3d
2017-10-02 11:31:20 -07:00
KongQun Yang 937ecfbdcd Move language tag convertion to packager.cc
This fixes shared_library not able to compile problem.

Also removed StreamDescriptorList and replaced with expicit
comparison to make the code clearer.

Also removed an out-dated DCHECK that mpd_notifier and hls_notifier
does not co-exist.

Change-Id: I3d04b8880741fecf0931a764cc12e50aa5e392b9
2017-09-29 18:05:03 -07:00