Commit Graph

72 Commits

Author SHA1 Message Date
KongQun Yang 74df8d30cc Exclude short segments from peak bandwidth computation
https://tools.ietf.org/html/rfc8216#section-4.1
The peak segment bit rate of a Media Playlist is the largest bit rate
of any contiguous set of segments whose total duration is between 0.5
and 1.5 times the target duration.

Fixes #498.

Change-Id: I1f28972b9cc5977735e47906bdcd88ba3942db5a
2018-11-26 20:02:09 +00:00
KongQun Yang 42083d205d [DASH] Fix TTML text input passthrough (regression)
Note that TTML in ISO-BMFF is not supported yet.

Also updated packager_test.py:
- Added a test using TTML passthrough.
- Computed output extension from input extension unless output_format
  is specified.

Fixes #478.

Change-Id: Ia917fc4ed3c326782791ed67601fba02ea28b11d
2018-09-19 23:27:48 +00:00
KongQun Yang ca810e06d5 Order trickplay outputs in trickplay factor ASC order
This also ensures that it does not violate std::sort() requirement on
strict ordering, which is enforced in gcc/g++ though not in clang.

std::sort() strict ordering requirement:
std::sort() need a comparator that return true iff the first argument is
strictly lower than the second one. That is: must return false when they
are equal.

Change-Id: I781cf4ed4125fcad212eba5430a264f3a3d71c16
2018-08-01 15:06:21 -07:00
KongQun Yang 51d39d96a1 Add offset to tranport streams (MPEG2-TS, HLS Packed Audio)
Configurable with --transport_stream_offset_ms.

This is needed to compensate for possible negative timestamps in
inputs, which could happen on ISO-BMFF with EditLists.

Issue #112.

Change-Id: I0fce8766c9df2911b9bb859c1e54052a8ed2abfb
2018-07-26 23:20:21 +00:00
KongQun Yang 86a86ce954 Return an error with '$' in stream output for WebVTT
WebVTT output with one file per Representatin per Period is not
supported yet.

Bug: 111359775.

Change-Id: I63f8dbbd65e6e286778474175d33f337f2a758ae
2018-07-24 18:45:44 +00:00
Aaron Vaage 1600909c4b Don't Always Start Text At Time Zero
Previously, the text padder media handler would assume that text always
started at time zero. This would work for VOD but would result with a
large pad at the start of LIVE content.

To avoid this, the text padder will use a bias to test whether or not it
thinks the content starts at zero. Right now the bias is set to be 10
minutes, but will later be configurable with a command line flag.

10 minutes was used as LIVE content will have much larger values and VOD
content should have much lower values.

Issue: #416

Change-Id: I07af15a577392fb030e36f052085cd4e667700e8
2018-06-29 23:02:22 +00:00
Aaron Vaage 0734ba998b Remove Old Cue Insertion Code
When we built the cue alignment media handler, we did not remove
the old solution.

This change removes all the code for the old solution.

Change-Id: I851b284c449c7d25aaabc2f55df5579ba7b5aad1
2018-06-21 18:47:04 +00:00
Aaron Vaage 4408718b5b Move ChainHandlers to MediaHandler
Made ChainHandlers to MediaHandler::Chain so that it can be used in our
test code as well. After all it is a pretty helpful function.

Change-Id: I8d83ee184052cd9fa9b37f2741c96f3223d5ab48
2018-06-21 09:52:04 -07:00
Aaron Vaage dbce84f3ea Remove Time Scale Assumptions From Text Chunker
Before, the text chunker would assume that all text streams were in
MS, which is not a safe assumption to make.

This changes it to take the time scale from StreamInfo and work
natively with scaled time units.

This required updating the tests. While doing so the tests were
rewritten with the goal to make them easier to read.

Closes: #399

Change-Id: Ib792ad306f40d749763418cde645337913a6046b
2018-06-13 17:43:49 +00:00
KongQun Yang e5863f1e0e Warns if HLS type is not set to LIVE for UDP inputs
- Also fixed documentation missing --hls_playlist_type LIVE for HLS
  live examples.
- Also updated packager.cc to use RETURN_IF_ERROR macro for
  consistency.

Fixes #347
Fixes #403

Change-Id: Idbccd7137b873170cd54e2c780bd554d25031a0b
2018-06-07 20:47:57 +00:00
Aaron Vaage 9df2825f3d Remove WebVTT Mp4 To Mp4 Path
Since the WebVtt mp4 to mp4 path did not include all the features
we needed and we know of no-one who is asking for it. We are opted
to remove the path so that there will be an error rather than
incorrect output if they try to use it.

Issue #405

Change-Id: Id2c37bb385c514dd8e31f7d3bd75fb3904b70d78
2018-06-07 11:13:54 -07:00
Aaron Vaage b9dee56294 Rebuild Text WebVtt to MP4 WebVtt
The previous text to mp4 webvtt pipeline was incomplete. It
did not insert ad cues and it could only insert a segment
after a sample ended.

Now the pipeline supports ad cue insert and segment insertion
mid text sample. This required the pipeline to use the text
chunker (to split samples and insert segments) and required
a major overhaul of the text to mp4 converter.

Before the converter came before the chunker. This meant that
the converter only expected to see stream info and text samples.
Moving the converter after the cue aligner and chunker means
that the convert had to be aware of segments and cues.

The general approach is the same, however the converter will
convert the samples per-segment as the chunker will introduce
duplicate samples if a sample spans across segments.

Closes #362
Closes #382

Change-Id: I0f54a40524c36a602ad3804a0da26e80851c92fd
2018-06-01 19:49:56 +00:00
KongQun Yang 6e4820eedc [Ad Insertion] Support one file per Representation per Period
Allow the output file to contain template identifiers like $Number$,
$Time$ etc. Unlike the actual template used in SegmentTemplate, the
identifiers in output will be populated before pushing to the DASH
manifest.

Issue: #384

Change-Id: Ife1caadb6fccd32167fa1bc83fe2afcb2d2ad087
2018-05-31 23:59:21 +00:00
Aaron Vaage 388d49cf98 Rename webvtt_output_handler files
Renamed all the files called "webvtt_output_handler*" to
"webvtt_text_output_handler*" to better reflect the class name
in them.

Change-Id: I977bab362076974a124f263bcefff716ed8b6a0f
2018-05-25 18:56:54 +00:00
Aaron Vaage b3c148607f Simplify WebVtt Output
Before, the webvtt output handler was written so that it could
share code between a segmented and non-segmented handler. As
we are not worried about that right now, this change simplifies
the handler to just be about segmented output.

Change-Id: I29dbc4e3a4ffbeb7ea10e23db489ee74b398a6c4
2018-05-24 18:09:48 +00:00
KongQun Yang 7f7843d241 Clean up output format determination
Requiring output format determined from 'output' to be consistent with
output format determined from 'segment_template'.

Change-Id: I32cbd63fcd6e2a4272dd0db531c1d5b385315445
2018-05-17 17:33:47 +00:00
KongQun Yang 0af2c5cdcf Integrate PackedAudioWriter
Closes #342.

Change-Id: I2fb4a651ad90448ab226b386c92c94e11ff9f9a8
2018-05-16 18:12:42 -07:00
Aaron Vaage 03afb16c0c Use Map To Track Demuxers and Aligners
To move us toward no longer need to ensure order when building
our pipeline, use a map to share demuxers between stream descriptors.
This will even allow use to use the same demuxers in the text pipelines
while still building them separately from the audio and video streams.

Change-Id: I4d4dbddbc06adee36cbe7f4aa1f6769f7bb2a3f6
2018-05-01 22:31:04 +00:00
KongQun Yang 224b597b48 Support approximate SegmentTimeline
It is not always possible to align segment duration to target duration
exactly. For example, for AAC with sampling rate of 44100, there are
always 1024 audio frames per sample, so the sample duration is
1024/44100. For a target duration of 2 seconds, the closest segment
duration would be 1.984 or 2.00533.

This feature allows MPD generator to treat these segments as having
the same duration, thus allows MPD generator to generate less
SegmentTimeline entries and potentially no SegmentTimeline entries
(replaced with SegmentTemplate@duration instead if
  --segment_template_constant_duration flag is enabled).

Under flag --allow_approximate_segment_timeline. Disabled by default.

Fixes #330.

Change-Id: I5044eaa348ebbf45bf792a2af53fc95a115ae21b
2018-04-30 17:15:47 -07:00
KongQun Yang 0ef078a23b Fix default_language not effective with 2-char code
Two-character ISO-639 code in --default_language was ignored due to
a bug in language code matching as the language code in stream is
always converted to 3-character code.

Fixes #371.

Change-Id: I8618938af583a417446636ff9efe1c72ce822c33
2018-04-13 20:08:12 +00:00
Aaron Vaage e1df680392 Clean-up Connecting Handlers
Created "Chain" function to connect a series of handlers together so
that the connection is easier to read.

Changed other calls to use RETURN_IF_ERROR to help make it easier to
read as we no longer need to track the status variable anymore.

Change-Id: Iffb76ca395b6d62f8feb054c09470c77718e8feb
2018-04-02 16:25:05 -07:00
Aaron Vaage e5fe2a76d2 Created Text Padder Handler To Fill Gaps
Created a media handler to come after parsers that will handle filling
in gaps between text samples. The padder takes a min duration, and if
the samples do not cover the min duration when flushed, one last empty
sample will be injected so that the samples will go up to the min duration.

Change-Id: I88605059664d09279676edac418ff3d4990d7556
2018-04-02 23:11:26 +00:00
Aaron Vaage 7dd1574b04 Add Cue Alignment For Segmented Text
Added the cue alignment handler to the segmented text pipeline.

Issue: #362

Change-Id: If105e5310fa82d433c84692b967a93c1cdf148b0
2018-03-29 20:29:23 +00:00
KongQun Yang 14aab56474 Allow SyncPointQueue to be cancelled
Change-Id: Idbf6ee7a5d9721681811189fca4190e7a0286c83
2018-03-27 19:14:37 +00:00
Aaron Vaage 56caecd073 Rename WebVttSegmenter To TextChunker
Renamed the webvtt segmenter to text chunker as there is nothing
webvtt specific in the code.

Change-Id: I0111c50d545fa168d945b39bc8a78c46c62678be
2018-03-26 17:58:56 +00:00
Aaron Vaage 2dd198a9d2 Move WebVtt Segmenter To Chunking Directory
Move the webvtt segmenter to the chunking directory so that it
can be converted to a general purpose text chunker.

Change-Id: I9ecd7ee39cb73070dab07b64f65ef24af1404813
2018-03-26 17:58:38 +00:00
KongQun Yang 4e57259d2a Integrate CueAlignmentHandler
Also changed ChunkingHandler to be one-one handler.

Issue: #355

Change-Id: Ie98a96bcc0ddded347699c9f333f604826976d11
2018-03-26 17:58:30 +00:00
Aaron Vaage 30b5fdf19f Add Text Codec To Codec String
This change has the hls master playlist add the text codec string
to the list of codecs in a variant.

Bug: 72942756

Change-Id: Ib25bb2064a291d10d7b1d261a4307991df62220c
2018-02-28 19:23:36 +00:00
KongQun Yang aa4bb49b2f Fix language option not honored on text streams
Change-Id: I9034b779e9b193c39201a5f2d142ec390927e5f7
2018-02-08 03:48:46 +00:00
KongQun Yang 8257eea804 Bug fixes and clean-ups for I-Frame playlists
- Add empty lines between different types of renditions to improve
  readability.
- Group variants with the same audio/text group together, as it is
  where the Adaptation occurs.
- Write master playlist after writing media playlists. This makes
  more sense and it is also necessary to have the bandwidth of
  the last iframe playist segment correctly computed.
- For fMP4, I-Frame segment must include the 'moof' header.
- Fix a problem that hls_iframe_playlist_name is not passed to
  MuxerListenerFactory.

Issue: #287

Change-Id: Icf37c5de1dc29f85ae3f419cbc3264d04ca491a4
2018-02-05 17:36:28 -08:00
Aaron Vaage c520d85b96 Move File Copy Out Of StreamInfoToTextMediaInfo
Moved the file copy out of StreamInfoToTextMediaInfo as the file
copy is only needed by the MPD output but the Media Info is needed
by MPD output and media info dump.

Bug: 36138902
Change-Id: Id233f2041b3e72345b8f709791c6b6070484222f
2018-01-29 17:22:43 +00:00
KongQun Yang 1455f43c02 Share chunkers from the same input except for WVM
Updated test files and added end to end tests for AdCues.

Change-Id: I28f57c2f0d15f65fa04f599fae446a5c1373bf47
2018-01-22 17:43:25 -08:00
KongQun Yang 32d26094ba [HLS] Support AUTOSELECT and DEFAULT in EXT-X-MEDIA
According to HLS spec: https://goo.gl/MiqjNd 4.3.4.1.1. Rendition Groups
- A Group MUST NOT have more than one member with a DEFAULT attribute of
  YES.
- Each EXT-X-MEDIA tag with an AUTOSELECT=YES attribute SHOULD have a
  combination of LANGUAGE[RFC5646], ASSOC-LANGUAGE, FORCED, and
  CHARACTERISTICS attributes that is distinct from those of other
  AUTOSELECT=YES members of its Group.

We tag the first rendition with a particular language in an audio group
with 'AUTOSELECT'; it is tagged with 'DEFAULT' too if the language
matches --default_language.

Fixes #315

Change-Id: Iacc0bc8c89ebffce8717fa65e82d6daf5a1f6adc
2018-01-19 02:10:07 +00:00
Aaron Vaage 5d5cb8678d Support WebVtt Text to WebVtt Mp4
Added new path to allow webvtt text input to be written as
webvtt mp4. This will happen when the input is a vtt file and
the output path is a mp4 path or the output format is mp4.

Missing test content for this.

Close: 70990714

Change-Id: Idbb4023db4a126a560528e73801f762cc45f4685
2018-01-10 12:21:56 -08:00
Aaron Vaage c57da2b42d Create Muxer Listener Factory
Moved muxer listener creation into a factory so that it can be
initialized once and then the factory only needs to be passed
around rather than all the parameters.

Change-Id: Ibf7df66c08debddbe6f6ff8995e7910b9ad17f3f
2018-01-10 18:59:11 +00:00
Aaron Vaage 49933737cd Moved MuxerFactory to own file
To remove the direct dependencies between packager.cc and the different
muxer implementations, this change remove the muxer factory to its
own file.

Change-Id: I4a1503c493f0f4d10a67e78461f96da3f4238944
2018-01-10 17:51:26 +00:00
Aaron Vaage 509963d60f Create HLS Text Pipeline
Linked together all the handlers to create the HLS text pipeline.
This change does not include tests for the pipeline.

Bug: 36138902
Change-Id: I32e7a7cefae1e7c74ec6d0bdc6335c0d65e5e79c
2018-01-09 00:03:29 +00:00
Aaron Vaage 37b5f401d1 Support MP4 to Mp4 Text
This change adds the missing code for passing mp4 text through
the pipeline.

This introduces the MuxerFactory to simplify creating muxers.

Change-Id: Ic7be1a2e0ff4b720a01061edb43aded946d05a47
2018-01-08 17:41:49 +00:00
KongQun Yang 66f713fd23 Merge SimpleMpdNotifier and DashIopMpdNotifier
Also removed MpdNotifier::AddContentProtectionElement which is not
used anywhere.

Change-Id: I69eb596ec377c601bb3e45bbef30c248c1d03e86
2017-12-22 15:56:14 -08:00
Aaron Vaage dedbd8b724 Isolate Job Management
Moved all code dealing with jobs into its own class so that the
packager code does not need to worry about initializing, running,
or stopping jobs.

Change-Id: I3e9ef1f22bd93d671f77d59ad15f23d1239078cf
2017-12-13 18:35:51 +00:00
KongQun Yang e97c89391a Address flakiness in ProducerConsumerQueueTest
1. Expect the time elapsed to be greater than kTimeout instead of
   approximately equal to kTimeout if there is a "wait";
2. Expect the time elapsed to be smaller than kTimeout instead of
   approximately equal to 0 if there is no "wait".

Also rename a variable in packager.cc to better reflect what it is.

Change-Id: I67975a6263b8dbc1124b78feae0f8e0d112bda50
2017-12-05 22:56:22 +00:00
Kamesh Devarakonda e7540bf303 Dispatch Scte35Events.
This patch also fixes a bug which was introduced in
https://widevine-internal-review.googlesource.com/c/packager/+/38040
where it was inconsistently assumed that AdCueGenerator would be MIMO
while connecting with other handlers and at the same time it was assumed
SISO in the AdCueGenerator class. Now we assume SISO everywhere.

Change-Id: Icd8c40b5ccfe0d98f47f09a60ea1635f74dceef9
2017-12-01 02:23:36 +00:00
Kamesh Devarakonda c4e7d02b80 Connect AdCueGenerator in the pipeline.
Change-Id: Iaae73661bc57ddecd2cc103dbd34d0c3102ffdaa
2017-11-27 21:09:26 +00:00
Kamesh Devarakonda 154e309f3f Remove unused argument.
Change-Id: I1ee3ca6527ec4ecce7dd57e18eee70a8c6a11eb0
2017-11-21 15:04:36 -05:00
KongQun Yang 5cf2b17ade Add --hls_key_uri to allow setting fairplay/identity key uri
Change-Id: I52e0f56cd10390faf515170f407ea488a6b3c9fc
2017-11-14 15:49:27 +00:00
KongQun Yang 1bf1ec2445 Add multi-key support in fixed/raw keys
The keys can be specified in --keys option, with the form of
  label=<label>:key_id=<32-digit hex string>:key=<32-digit hex string>,label=...

There can be multiple "label=..." block. The DRM label can be one
of "AUDIO, SD, HD, UHD1, UHD2" or a custom label.

Mark --key and --key_id as deprecated. We can achieve the same result
with --keys=label=:key_id=<key_id>:key=<key>.

Also add a new optional field in stream descriptor: drm_label, which
is used to overwrite the internally generated DRM label for the stream.
If not provided, the DRM label is generated automatically based on
audio/video information, e.g. resolution.

Code changes:
- Merged RawKeyEncryptionParams and RawKeyDecryptionParams into
  RawKeyParams.
- Make FixedKeySource accepts RawKeyParams as input.

Change-Id: Ic8c2f071cc71188e13f14bc6396fc2b3ffa5cac6
2017-10-24 21:04:49 +00:00
Aaron Vaage 65d5ecc3cf Separated Text Processing From Audio/Video
To help isolate the differences between audio/video and text in
preparation for adding the better text support, this change moves
all audio/video job creation and text job creation into their
own functions.

Change-Id: Ia71cf82ac147672b70c69123eeabdfc6b41c4a3d
2017-10-02 11:31:20 -07:00
KongQun Yang 937ecfbdcd Move language tag convertion to packager.cc
This fixes shared_library not able to compile problem.

Also removed StreamDescriptorList and replaced with expicit
comparison to make the code clearer.

Also removed an out-dated DCHECK that mpd_notifier and hls_notifier
does not co-exist.

Change-Id: I3d04b8880741fecf0931a764cc12e50aa5e392b9
2017-09-29 18:05:03 -07:00
Aaron Vaage 8067fd4818 Made Trick Play A 1:1 Handler
Before the trick play handler would have one input and multiple
outputs. The normal media handler API for adding handlers had to
be ignored when setting-up the trick play handler.

This changes the trick play handler to have one input and one output.
It uses the standard AddHandler method calls. When initialized, the
trick play factor must be provided.

This included a run of 'clang-format' over all edited files.

Change-Id: I7b3cdf0a2472e2f89ab194867c4b984e26f18f24
2017-09-26 08:25:04 -07:00
Aaron Vaage 304f7d2a88 Rename "crypto_handler" to "encryption_handler"
Change-Id: I44307fad6f4f0c4c2f8b61c3c6b07867859c9867
2017-09-20 19:13:01 +00:00