Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for channel tabs and channel tags #1082

Merged
merged 19 commits into from
Aug 6, 2023

Conversation

AudricV
Copy link
Member

@AudricV AudricV commented Jul 19, 2023

This PR supersedes #951 by reusing the changes from @Theta-Dev and @Stypox and applying the latest requested changes where relevant. Thanks a lot for your original work!

It introduces support of channel tabs and tags. The channel tab changes are breaking changes, as ChannelExtractor is now only an Extractor abstract implementation instead of a ListExtractorone: this means that ChannelExtractor do not provide StreamInfoItem anymore. They can be extracted from channel tabs instead.

Channel tabs (which are strings to be compliant with the current LinkHandler filtering and sorting system) need to be fetched separately from channels. A dynamic list of available tabs is returned by ChannelInfo and ChannelExtractor with the method getTabs. The tab list is a list of ListLinkHandlers, which can be used to fetch tabs using the getChannelTabExtractor method.
In some cases, tab data is sent with the channel data, so fetching tabs would require an additional and duplicate request. To avoid this issue, a new custom LinkHandler can be used to create a ChannelTabExtractor instance with the data already fetched in it: ReadyChannelTabListLinkHandler.

Alternatively, a specific tab can be fetched using the corresponding ChannelExtractor methods taking a specific channel tab name (which should be one of the ChannelTabs constants) and returning channel tab extractors. In this case, ReadyChannelTabListLinkHandlers cannot be used, so duplicate requests could happen with this process; passing an invalid or unsupported tab name may return an exception indicating that the given tab name is invalid or unsupported.

Services' channel tab extractors, which so have ChannelTabExtractor as a base abstract class, are ListExtractor implementations allowing any InfoItem type. This results in the ability to provide streams, playlists and channels in channel tabs, and would allow to implement a posts item type in the future to support contents of YouTube's community tab for instance.

Channel tabs could be also fetched with their Info equivalent, ChannelTabInfo.

Note: Contrary to the original PR, channel tabs have their own package, tabs, a package of the channel one, in which ChannelTabExtractor, ChannelTabInfo and ChannelTabs classes are located. The ReadyChannelTabListLinkHandler class, which may need to be renamed, is still in the linkhandler package.

The following channel tabs are supported with this PR:

  • for YouTube: Videos (tab's parameters have been updated to use the one provided and used by the desktop website), Shorts (by using only the channel tab except for age-restricted channels, see below), Live, Playlists and Channels;
  • for SoundCloud: Tracks, Playlists and Albums;
  • for MediaCCC: Videos;
  • for PeerTube: Videos, Playlists and Channels;
  • for Bandcamp: Tracks and Albums.

What is done with #951 but has been not highlighted is the support of channel tags: a list of strings which can be get using the getTags method of ChannelInfo and ChannelExtractor. Only YouTube seem to provide this information, so support has been added for this service only, other ones return an empty list, the default behavior.

I decided to do a little bit more than copying and improving changes of the original PR in this one: I also added support of detecting age-restricted YouTube channels and getting StreamInfoItems of them using YouTube's system playlists. These playlists may contain geo-restricted (and so unavailable) videos. This only works when getting tabs from a channel, due to the usage of a ReadyChannelTabListLinkHandler and a ChannelTabExtractor implementation wrapping a PlaylistExtractor in this case. This support should be added when getting directly tabs, by moving the corresponding code as a fallback for stream items to YouTube's standard ChannelTabExtractor.

So, for these channels, the Videos, Shorts and Live tabs will be always returned, even if a tab or multiple tabs do or does not exist (it/they would return no items in this case, even if system playlists don't exist when a type of a content has been not created by a channel).
Note that support of the Shorts tab using system playlists is currently broken, as YouTube return a short UI for playlists with only shorts in most cases; this UI is currently not supported by the extractor. See #1077 for more details.

Tests have been of course updated to reflect the changes made in this PR, and the YouTube channels mocks had to be updated.

Closes #227, closes #501 (only support for topic channels which have regular tabs has been added, so system channels such as Music or Gaming return no tabs, this is tracked by #611), closes #585.

Note for reviewers: I recommend to review the changes commit by commit, as they are huge.

Debug APK built by @Stypox: app-debug.zip

Copy link
Member

@Stypox Stypox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partial review; I had some time to look at the first few commits (up until the YouTube one, where I stopped at YoutubeChannelExtractor)

Copy link
Member

@Stypox Stypox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a few small things to change, but overall everything is fine. If you don't have objections, I can fix everything myself and then merge. Thank you again to you and Theta-Dev for the effort on this PR!

Btw, for the future try to keep the commit subject under 50 characters: https://stackoverflow.com/q/2290016

* </pre>
* </p>
*/
public class ReadyChannelTabListLinkHandler extends ListLinkHandler {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did we decide on a better name or not in the end?

AudricV and others added 19 commits August 6, 2023 12:11
This commit introduces the following breaking changes:

- Three new classes have been added:
  - ChannelTabExtractor, class extending ListExtractor<InfoItem>, which
  extracts InfoItems from a channel tab;
  - ChannelTabInfo extending ListInfo<InfoItem>, which extracts InfoItems from
  a ChannelTabExtractor and returns them as a ChannelTabInfo;
  - ChannelTabs, an immutable class containing all supported channel tabs.
- StreamingService implementations must implement new methods returning a
channel tab LinkHandlerFactory (getChannelTabsLHFactory) and a
ChannelTabExtractor (getChannelTabExtractor);
- ChannelExtractor inherits Extractor instead of ListExtractor<StreamInfoItem>
and ChannelInfo inherits Info instead of ListInfo<StreamInfoItem>;
- ChannelExtractor and ChannelInfo have now getters and/or setters of tabs.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
Tags' getters and/or setters have been added in ChannelExtractor and
ChannelInfo to do so.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
…y getId and getUrl methods of LinkHandlerFactory and its base implementations

This change advertise to clients that channel tabs' link handler factories can
return an UnsupportedOperationException when a tab provided to them is
unsupported.
This class makes easier for LinkHandlerFactory implementations to declare an
UnsupportedOperationException.
…fo.getTabs() when a specific tab's data has already been fetched

This new ListLinkHandler, ReadyChannelTabListLinkHandler, should help saving
clients data, energy and time by helping to reduce duplicate requests.

Co-authored-by: Stypox <stypox@pm.me>
Support of channels and videos has been added for accounts and support of
videos and playlists has been added for video channels.

The following changes have been also done:
- collectStreamsFrom method in PeertubeParsingHelper has been renamed to
collectItemsFrom;
- PeertubeChannelInfoItemExtractor.getStreamCount method has been fixed due to
ChannelExtractor's new inheritance;
- the declaration of the UnsupportedOperationException exception thrown has
been added to the service's LinkHandlers;
- a channel tab LinkHandlerFactory has been added,
PeertubeChannelTabLinkHandlerFactory;
- all service's LinkHandlers are now using properly the singleton pattern.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
Support of tracks and albums has been added for artists.

Also use the singleton pattern and add the declaration of the
UnsupportedOperationException exception to the service's LinkHandlers and
improved some code in the files changed.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
Support of tracks, playlists and albums has been added for users.

Also add the declaration of the UnsupportedOperationException exception to the
service's LinkHandlers.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
…atterns

This is required to parse duration of YouTube's reelItemRenderers, returned
only inside accessibility data.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
…annels

Support of tags and videos, shorts, live, playlists and channels tabs has been
added for non-age restricted channels.

Age-restricted channels are now also supported and always returned the videos,
shorts and live tabs, accessible using system playlists. These tabs are the
only ones which can be accessed using YouTube's desktop website without being
logged-in.

The videos channel tab parameter has been updated to the one used by the
desktop website and when a channel extraction is fetched, this tab is returned
in the list of tabs as a cached one in the corresponding link handler.

Visitor data support per request has been added, as a valid visitor data is
required to fetch continuations with contents on the shorts tab. It is only
used in this case to enhance privacy.

A dedicated shorts UI elements (reelItemRenderers) extractor has been added,
YoutubeReelInfoItemExtractor. These elements do not provide the exact view
count, any uploader info (name, URL, avatar, verified status) and the upload
date.

All service's LinkHandlers are now using the singleton pattern and some code
has been also improved on the files changed.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
…s with the Test JUnit annotation

These changes should help to detect tests as tests, when running a subset of
tests or all tests.
They should be also implemented in these interfaces' implementations (new and
existing ones).

Co-authored-by: ThetaDev <t.testboy@gmail.com>
…returned by a channel extractor

Only the first content filter of the ListLinkHandler instances provided is
used when collecting all channel tabs of the ListLinkHandler list, as channel
tabs implementations only use one content filter per ListLinkHandler instance.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
… on tests

Tests in BandcampChannelExtractorTest and BandcampChannelLinkHandlerFactoryTest
have been also fixed.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
… on tests

Co-authored-by: ThetaDev <t.testboy@gmail.com>
… on tests

Co-authored-by: ThetaDev <t.testboy@gmail.com>
…es on tests

Co-authored-by: ThetaDev <t.testboy@gmail.com>
…d channels tabs and tags changes on tests

Co-authored-by: ThetaDev <t.testboy@gmail.com>
@Stypox Stypox force-pushed the channel-tabs-and-tags-support branch from 53df9d6 to e7d6409 Compare August 6, 2023 10:19
Copy link
Member

@Stypox Stypox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aha! I took back ownership of commits! I rebased and tested locally, everything works fine. I will open a PR for the small suggestions I proposed above, as they are not blocking.

@Stypox Stypox merged commit 35f3a4a into TeamNewPipe:dev Aug 6, 2023
3 of 4 checks passed
@AudricV AudricV deleted the channel-tabs-and-tags-support branch August 6, 2023 11:32
@Stypox Stypox mentioned this pull request Aug 6, 2023
1 task
TobiGr added a commit that referenced this pull request Aug 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement multiservice youtube service, https://www.youtube.com/
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

[Bandcamp] Support albums in artists' profile [FR] YouTube Topic Channels Playlist in channels
3 participants