Support delivery methods other than progressive HTTP

[AudricV]

_{Previous too-long title: Add support of other delivery methods than progressive HTTP (DASH, HLS and SmoothStreaming), support non-URLs contents and more}

• I carefully read the contribution guidelines and agree to them.
• I have tested the API against NewPipe.
• I agree to create a pull request for NewPipe as soon as possible to make it compatible with the changed API. Support delivery methods other than progressive HTTP NewPipe#8153

This PR superseeds #663 with more and less changes and improvements.
@wb9688 has been the initiator of this feature/enhancement, thank you again for your work!

This PR mainly adds support of other delivery methods than progressive HTTP (DASH, HLS and SmoothStreaming) and of non-URLs content in the extractor but also:

• fixes the extraction of PeerTube audio streams as video streams;
• returns all delivery methods available in services (except SoundCloud, at least for now);
• extracts the downloadable file of SoundCloud tracks (on tracks on which download is available);
• removes (and initially fixes) parsing of YouTube DASH manifests;
• adds the ability to generate DASH manifests for YouTube progressive, OTF and post-live streams, by using the data returned in the first sequence for the last two;
• adds more data to ItagItem, to generate better and more types of DASH manifests using the extractor;
• changes the way of how Stream objects (classes which are extended the Stream class) are built (⚠ it's a breaking change for clients which build Stream objects! ⚠️);
• deprecates access to some fields in Stream classes (replaced by getters and/or setters);
• deprecates the getUrl() method of the Stream class (use the getContent() method instead and the isUrl() one to check if the content is a URL or not), which returns now null if the stream's content is not a URL;
• updates tests to consider the changes made;
• and more.

Summary of (important) code changes (check out the code changes for more details):

New (sub)packages:

• dashmanifestcreators (in the youtube service package), which contains the YouTube DASH manifest creators of YouTube streams (see below).

New enums:

• DeliveryMethod (in the stream package), to represent the delivery method of contents extracted. You can get it in the Stream class, by using the getDeliveryMethod() method;
• DeliveryType (in the youtube service package), which contains the stream delivery types of YouTube streams (which are used by manifest creators). This enum is not representing the delivery method of streams, which is the job of the enum above;

New classes:

• ItagInfo, (in the youtube service package) a class which contains everything needed to build YouTube streams (intended to be only used internally): the content, whether the content is a URL and an ItagItem;
• ManifestCreatorCache (in the utils package), a class to cache results of a manifest creator (types of keys and values must be serializable), which allow setting a cache limit, a clear factor and more;
• Pair (in the utils package), a class to store and access a pair of serializable objects (this class is intended to be only used internally);
• YoutubeProgressiveDashManifestCreator, YoutubeOtfDashManifestCreator and YoutubePostLiveStreamDvrDashManifestCreator (in the dashmanifestcreators YouTube service package), classes to generate respectively DASH manifests of YouTube progressive, OTF, and post-live streams. Common methods are present in a separate class, YoutubeDashManifestCreatorsUtils (in the dashmanifestcreators YouTube service package too). YouTube DASH manifest creators cache results and allow setting limits to the cache (by using the methods of ManifestCreatorCache instance used);
• CreationException (in the dashmanifestcreators YouTube service package), a RuntimeException which is thrown when an exception occured in YouTube DASH manifest creators;
• YoutubeDashManifestCreatorsTest (in the youtube service test package), to test what we can test of testable YouTube DASH manifest creators (see the code of the test class for more information).

Changes in the StreamType enum:

• Add POST_LIVE_STREAM and POST_LIVE_AUDIO_STREAM values to the end of the enum and document this enum.
• Remove FILE value.

Changes in the ItagItem class:

• Add new constants: AVERAGE_BITRATE_UNKNOWN, SAMPLE_RATE_UNKNOWN, FPS_NOT_APPLICABLE_OR_UNKNOWN, TARGET_DURATION_SEC_UNKNOWN, APPROX_DURATION_MS_UNKNOWN, AUDIO_CHANNELS_NOT_APPLICABLE_OR_UNKNOWN and CONTENT_LENGTH_UNKNOWN;
• Add new methods: getAverageBitrate(), getFps(), setFps(int), getResolutionString(), getSampleRate(), setSampleRate(int), getAudioChannels(), setAudioChannels(int), getTargetDurationSec(), setTargetDurationSec(int), getApproxDurationMs(), setApproxDurationMs(long), getContentLength() and setContentLength(long);
• Deprecate the use of avgBitrate, fps and resolutionString. Use getAverageBitrate(), getFps(), setFps(int) and getResolutionString() instead.

Changes in the Stream classes:

• In the Stream abstract class:

  • Add new constants: FORMAT_ID_UNKNOWN, ID_UNKNOWN and ITAG_NOT_AVAILABLE_OR_NOT_APPLICABLE;
  • Add new methods: getId(), getContent(), isUrl(), getDeliveryMethod(), getBaseUrl() and getItagItem() (abstract method which is implemented in the classes which extend this class). Check out the corresponding documentations to know and understand what they return;
  • Deprecate getUrl(), which may return now null. The extractor is able to extract non-URL contents, so if you call this method on a non-URL content, it will return null. Use getContent() instead and isUrl() to know whether the content is the URL to the content or the content itself.
  • Remove equals(Stream). Use equalStats(Stream) to compare statistics of two streams and the Java standard equals(Object) method to compare the equality of two streams instead.

• In classes which extends the Stream class:

⚠️ WARNING: These changes break your projet if you build objects of classes which extends the Stream class! ⚠️

• Add new classes to build objects of these classes. See the documentations of each class and their methods to know what you need to set and what you can set to build these Builders;

• Remove the ability to build objects of theses classes with constructors. Use the Builder classes instead, which may also prevent you to set values that are default.

• In the AudioStream class:

  • Add a new constant: UNKNOWN_BITRATE.

• In the VideoStream class:

  • Add a new constant: RESOLUTION_UNKNOWN;
  • Deprecate the use of isVideoOnly and resolution fields. Use the corresponding getters instead, isVideoOnly() and getResolution().

Changes in StreamExtractors:

• Apply changes in StreamExtractor classes and improve some of their code (check out the code changes by yourself for more details).

Changes in StreamExtractor test classes:

• Update DefaultStreamExtractorTest and SoundcloudStreamExtractorTest to consider the changes made in Stream classes (check out the code changes for more details).

Changes in other test classes:

• Test the ManifestCreatorCache class in UtilsTest (check out the code changes for more details);
• Remove DashMpdParser, in which the parsing of YouTube DASH manifest was initially fixed (and is still available in Git history).

Closes #223, closes #273, closes #461, fixes #648.

[TobiGr]

I reviewed the first batch. Note that I did not check for spelling or grammar.

[opusforlife2]

Note that I did not check for spelling or grammar.

Don't worry, I've got your back, @TobiGr. ;)

[TobiGr]

Well done!

[opusforlife2]

Some more.

[litetex]

Thank you for the PR.

A lot better than the old one but still some problems (and too big IMHO).

What is the advantage of this exactly?
Because currently I see none - despite having 6k new lines of code to maintain...

I wouldn't merge this PR without having Sonarcloud generate a report for it.

[opusforlife2]

What is the advantage of this exactly?

It fixes Peertube seeking. Currently, if you try to seek in a video, it just takes you to the tip of the buffer, like a livestream. You can only seek properly in the DASH APK.

Edit: Oh, and it also restores streams on certain music (and other) videos on YT. OTF streams were disabled on an earlier 0.20.x build, so currently, some videos only have 720p MP4, 360p MP4, and 144p 3GPP streams. The DASH APK shows and plays all MP4 and WebM streams.

[Stypox]

Overriding hashCode and equals is always a problem because usually things are in other ways "equal" or at least considered to be so.
Why are these methods overridden?

@litetex those are needed to distinguish between different streams in NewPipe, see TeamNewPipe/NewPipe#6537 (comment)

[Stypox]

Partial review, will finish later/tomorrow/(the day after)

[litetex]

@litetex those are needed to distinguish between different streams in NewPipe, see TeamNewPipe/NewPipe#6537 (comment)

@Stypox
Sorry but I still don't see the reason why these are overridden (the source code from that discussion no longer exists - at least what GitHub says):

Stypox:
I'm not so sure about this way of creating a key: the first 20% of the content might be equal for some streams, wouldn't it? E.g. for videos with normal urls, the first 20% characters might always be the same. What about using the hash of the string? It wouldn't ensure uniqueness, but the probability of clashes should be pretty low.

TiA4f8R:
I didn't have an idea to how use the content, but your idea of using the hash code is really great! Why not using the hash code of the stream object itself instead of using the hashcode of the content?

Stypox:
Yeah, that makes sense, though the Stream class does not seem to have hashCode overridden, and the default implementation obviously does not return something useful. So either you make changes in the extractor and override the hashCode() method in the various *Stream classes, or you just use getContent().hashCode() and call it a day. (obviously only use hashes if resolution.equals(RESOLUTION_UNKNOWN) && mediaFormat == null)

The argument also doesn't work for me because there is also currently no overridden equals and hashCode method so NewPipe works without one...

I'm so nitpicking about these because I had similar cases (in other software) in the past: E.g. someone created an object slightly different and the persistence framework failed to save these objects correctly because equals and hashcode always returned the same for "different" objects. The problem then was that part A of the app assumed that the current equals method was correct and part B required another implementation. How do you fix this? You can't because there is only one equals and hashCode.
→ Don't add such stuff if we don't need it.

[AudricV]

The argument also doesn't work for me because there is also currently no overridden equals and hashCode method

@litetex What? There are overridden in all Stream classes! I don't understand why you said this.

Note that I added more checks in the methods locally in the Stream abstract class, so here is how looks the code of these methods:

@Override
public boolean equals(final Object obj) {
    if (this == obj) {
        return true;
    }

    if (obj == null || getClass() != obj.getClass()) {
        return false;
    }

    final Stream stream = (Stream) obj;
    return id.equals(stream.id) && mediaFormat == stream.mediaFormat
            && deliveryMethod == stream.deliveryMethod
            && content.equals(stream.content)
            && isUrl == stream.isUrl
            && Objects.equals(baseUrl, stream.baseUrl);
}

@Override
public int hashCode() {
    return Objects.hash(id, mediaFormat, deliveryMethod, content, isUrl, baseUrl);
}

[litetex]

@litetex What? There are overridden in all Stream classes! I don't understand why you said this.

Note that I added more checks in the methods locally in the Stream abstract class, so here is how looks the code of these methods: ...

https://github.com/TeamNewPipe/NewPipeExtractor/blob/7f2ea133f0f5888adee896eb5594eaf0030b0817/extractor/src/main/java/org/schabi/newpipe/extractor/stream/Stream.java

No.

[AudricV]

@litetex You're looking on the dev branch, I am implementing these methods in the PR 😅

[AudricV]

My knowledge in unit tests is like nothing, so I am not able to write good tests, I'm afraid for this.

Due to my lack of time, I am not really able to maintain this PR for now (I will push most of the requested changes and rebase the PR however, so please wait before doing something until it's done).

So someone else will have to apply the final changes in this PR and the app part of the changes.

Note that when #780 is merged, you will have to add, in YoutubeDashManifestCreator, the same piece of code added to the Android client for the iOS one, with the relevant methods.

[Stypox]

I reviewed everything, though I can't really confirm that the service-specific code work well (which this PR is mostly about). Tests should come in handy for that purpose, so I would be ok with merging this even if I didn't try every possible service request myself but just let the tests do their job.
The code structure looks good, though maybe sometimes the javadocs are not much to-the-point, and there are many methods being added and never used (such as all of the different ways to clear cache, change clear factor, ... in YoutubeDashManifestCreator). But these two problems are arguably just my opinion and not really big problems anyway.

[litetex]

@litetex You're looking on the dev branch, I am implementing these methods in the PR 😅

I'm not talking about that these methods are overridden in a single class and not in another.

Read #810 (comment)

[Stypox]

@litetex please merge unless you can find other critical problems. Even if you can still possibly find some minor things to improve, I would merge this anyway and you can directly create a small PR improving such small things (which takes less effort than making a review).

Solved

[opusforlife2]

Damn, DASH support has finally been added after almost 2 years (Mauricio's PR was on 21 June). Mad props to everyone involved for all the effort!