-
Couldn't load subscription status.
- Fork 17
YouTube Music provider #123
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
feathecutie
wants to merge
10
commits into
kellnerd:main
Choose a base branch
from
feathecutie:youtube-music-provider
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
b08fea2
feat: YouTube Music provider
feathecutie 3e0ff20
chore(YouTube): Factor out track conversion
feathecutie 884c61c
chore(YouTube): Clean up code
feathecutie b09330d
feat(YouTube): Remove dependency on YouTube.js
feathecutie a8da1a2
feat(YouTube): Warn on alternate release versions
feathecutie f84c352
chore(YouTube): Add some documentation comments
feathecutie fcdea86
chore(YouTube): Add `releaseLookup` tests
feathecutie 95f47f7
fix(YouTube): Fix GTIN lookup for GTINs with leading 0s
feathecutie b18e07c
chore(YouTube): Clean up request body
feathecutie 670acf6
fix(YouTube): Only warn for multiple releases if GTIN lookup was used
feathecutie File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,227 @@ | ||
| type TrackingParams = { trackingParams: string }; | ||
|
|
||
| type APIResponse = { | ||
| responseContext?: { | ||
| serviceTrackingParams: unknown[]; | ||
| }; | ||
| } & TrackingParams; | ||
|
|
||
| export type Album = { | ||
| contents: Renderer<'TwoColumnBrowseResults'>; | ||
| microformat: Renderer<'MicroformatData'>; | ||
| background?: Renderer<'MusicThumbnail'>; | ||
| } & APIResponse; | ||
|
|
||
| export type Playlist = { | ||
| contents: Renderer<'TwoColumnBrowseResults'>; | ||
| } & APIResponse; | ||
|
|
||
| export type Credits = { | ||
| onResponseReceivedActions: { | ||
| clickTrackingParams: string; | ||
| openPopupAction: { | ||
| popup: Renderer<'DismissableDialog'>; | ||
| popupType: string; | ||
| }; | ||
| }[]; | ||
| } & APIResponse; | ||
|
|
||
| export type SearchResult = { | ||
| contents: Renderer<'TabbedSearchResults'>; | ||
| } & APIResponse; | ||
|
|
||
| type Icon = { iconType: string }; | ||
|
|
||
| type Thumbnail = { | ||
| thumbnails: { url: string; width: number; height: number }[]; | ||
| thumbnailCrop: string; | ||
| thumbnailScale: string; | ||
| } & TrackingParams; | ||
|
|
||
| export type BrowseEndpoint = { | ||
| browseEndpoint: { | ||
| browseId: string; | ||
| params: string; | ||
| browseEndpointContextSupportedConfigs: { | ||
| browseEndpointContextMusicConfig: { pageType: string }; | ||
| }; | ||
| }; | ||
| }; | ||
|
|
||
| export type WatchEndpoint = { | ||
| watchEndpoint: { | ||
| videoId: string; | ||
| playlistId?: string; | ||
| watchEndpointMusicSupportedConfigs?: { | ||
| musicVideoType?: string; | ||
| }; | ||
| }; | ||
| }; | ||
|
|
||
| export type QueueAddEndpoint = { | ||
| queueAddEndpoint: unknown; | ||
| }; | ||
|
|
||
| export type ModalEndpoint = { | ||
| modalEndpoint: unknown; | ||
| }; | ||
|
|
||
| export type Nodes = { | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/MicroformatData.ts */ | ||
| MicroformatData: { | ||
| urlCanonical: string; | ||
| }; | ||
| DismissableDialog: unknown; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/Tab.ts */ | ||
| Tab: { | ||
| title: string; | ||
| selected: boolean; | ||
| content: Renderer<'SectionList'>; | ||
| tabIdentifier: string; | ||
| } & TrackingParams; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/SectionList.ts */ | ||
| SectionList: { | ||
| contents: ( | ||
| | Renderer<'ItemSection'> | ||
| | Renderer<'MusicCardShelf'> | ||
| | Renderer<'MusicShelf'> | ||
| | Renderer<'MusicPlaylistShelf'> | ||
| | Renderer<'MusicResponsiveHeader'> | ||
| | Renderer<'MusicCarouselShelf'> | ||
| )[]; | ||
| }; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/ItemSection.ts */ | ||
| ItemSection: { | ||
| contents: unknown[]; | ||
| } & TrackingParams; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/MusicShelf.ts */ | ||
| MusicShelf: { | ||
| title: string; | ||
| contents: Renderer<'MusicResponsiveListItem'>[]; | ||
| bottomText: YTText; | ||
| bottomEndpoint: unknown; | ||
| }; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/MusicPlaylistShelf.ts */ | ||
| MusicPlaylistShelf: { | ||
| contents: Renderer<'MusicResponsiveListItem'>[]; | ||
| collapsedItemCount: number; | ||
| targetId: string; | ||
| } & TrackingParams; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/MusicCardShelf.ts */ | ||
| MusicCardShelf: { | ||
| thumbnail: Renderer<'MusicThumbnail'>; | ||
| title: YTText; | ||
| subtitle: YTText; | ||
| buttons: unknown[]; | ||
| menu: Renderer<'Menu'>; | ||
| onTap: unknown; | ||
| header: Renderer<'MusicCardShelfHeaderBasic'>; | ||
| endIcon: Icon; | ||
| thumbnailOverlay: Renderer<'MusicItemThumbnailOverlay'>; | ||
| }; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/MusicThumbnail.ts */ | ||
| MusicThumbnail: { | ||
| thumbnail: Thumbnail; | ||
| }; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/menus/Menu.ts */ | ||
| Menu: { | ||
| items: (Renderer<'MenuNavigationItem'> | Renderer<'MenuServiceItem'>)[]; | ||
| }; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/menus/MenuNavigationItem.ts */ | ||
| MenuNavigationItem: { | ||
| text: YTText; | ||
| icon: Icon; | ||
| navigationEndpoint: BrowseEndpoint | ModalEndpoint | QueueAddEndpoint | WatchEndpoint; | ||
| }; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/menus/MenuServiceItem.ts */ | ||
| MenuServiceItem: unknown; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/MusicCardShelfHeaderBasic.ts */ | ||
| MusicCardShelfHeaderBasic: unknown; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/MusicItemThumbnailOverlay.ts */ | ||
| MusicItemThumbnailOverlay: unknown; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/MusicCarouselShelf.ts */ | ||
| MusicCarouselShelf: { | ||
| header: Renderer<'MusicCarouselShelfBasicHeader'>; | ||
| contents: Renderer<'MusicTwoRowItem'>[]; | ||
| }; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/MusicCarouselShelfBasicHeader.ts */ | ||
| MusicCarouselShelfBasicHeader: { | ||
| title: YTText; | ||
| accessibilityData?: { | ||
| accessibilityData: { | ||
| label?: string; | ||
| }; | ||
| }; | ||
| }; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/MusicTwoRowItem.ts */ | ||
| MusicTwoRowItem: { | ||
| thumbnailRenderer?: Renderer<'MusicThumbnail'>; | ||
| aspectRatio?: string; | ||
| title: YTText; | ||
| subtitle?: YTText; | ||
| navigationEndpoint: BrowseEndpoint; | ||
| menu?: Renderer<'Menu'>; | ||
| thumbnailOverlay: Renderer<'MusicItemThumbnailOverlay'>; | ||
| }; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/MusicResponsiveListItem.ts */ | ||
| MusicResponsiveListItem: { | ||
| thumbnail: Renderer<'MusicThumbnail'>; | ||
| overlay: Renderer<'MusicItemThumbnailOverlay'>; | ||
| flexColumns: Renderer<'MusicResponsiveListItemFlexColumn'>[]; | ||
| fixedColumns?: Renderer<'MusicResponsiveListItemFixedColumn'>[]; | ||
| menu: Renderer<'Menu'>; | ||
| badges: Renderer<'MusicInlineBadge'>[]; | ||
| flexColumnDisplayStyle: string; | ||
| navigationEndpoint: BrowseEndpoint; | ||
| index?: YTText; | ||
| } & TrackingParams; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/MusicResponsiveHeader.ts */ | ||
| MusicResponsiveHeader: { | ||
| thumbnail?: Renderer<'MusicThumbnail'>; | ||
| buttons?: unknown[]; | ||
| title: YTText; | ||
| subtitle: YTText; | ||
| straplineTextOne: YTText; | ||
| straplineThumbnail: Renderer<'MusicThumbnail'>; | ||
| subtitleBadge: Renderer<'MusicInlineBadge'>[]; | ||
| description: Renderer<'MusicDescriptionShelf'>; | ||
| secondSubtitle: YTText; | ||
| }; | ||
| MusicResponsiveListItemFixedColumn: { | ||
| text: YTText; | ||
| displayPriority?: string; | ||
| size?: string; | ||
| }; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/MusicResponsiveListItemFlexColumn.ts */ | ||
| MusicResponsiveListItemFlexColumn: { | ||
| text: YTText; | ||
| displayPriority: string; | ||
| }; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/MusicDescriptionShelf.ts */ | ||
| MusicDescriptionShelf: { | ||
| description: YTText; | ||
| straplineBadge?: Renderer<'MusicInlineBadge'>[]; | ||
| }; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/MusicInlineBadge.ts */ | ||
| MusicInlineBadge: { | ||
| icon: Icon; | ||
| accessibilityData: unknown[]; | ||
| }; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/TabbedSearchResults.ts */ | ||
| TabbedSearchResults: { | ||
| tabs: Renderer<'Tab'>[]; | ||
| }; | ||
| /** @see https://github.com/LuanRT/YouTube.js/blob/v14.0.0/src/parser/classes/TwoColumnBrowseResults.ts */ | ||
| TwoColumnBrowseResults: { | ||
| secondaryContents: Renderer<'SectionList'>; | ||
| tabs: Renderer<'Tab'>[]; | ||
| }; | ||
| }; | ||
|
|
||
| type RendererName<T extends string> = `${Uncapitalize<T>}Renderer`; | ||
|
|
||
| export type Renderer<Node extends keyof Nodes> = { [K in RendererName<Node>]: Nodes[Node] }; | ||
|
|
||
| type YTText = { runs: TextRun[]; accessibility?: { accessibilityData?: unknown } }; | ||
|
|
||
| export type TextRun = { text: string; navigationEndpoint?: BrowseEndpoint | WatchEndpoint }; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| export const BROWSE_URL = new URL('https://www.youtube.com/youtubei/v1/browse?prettyPrint=false&alt=json'); | ||
| export const SEARCH_URL = new URL('https://www.youtube.com/youtubei/v1/search?prettyPrint=false&alt=json'); | ||
|
|
||
| export const USER_AGENT = | ||
| 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36'; | ||
|
|
||
| const CLIENT_VERSION = '1.20250219.01.00'; | ||
|
|
||
| export const YOUTUBEI_HEADERS = { | ||
| accept: '*/*', | ||
| 'accept-language': '*', | ||
| 'content-type': 'application/json', | ||
| origin: 'https://www.youtube.com', | ||
| 'user-agent': USER_AGENT, | ||
| 'x-youtube-client-name': '67', | ||
| 'x-youtube-client-version': CLIENT_VERSION, | ||
| }; | ||
|
|
||
| export const YOUTUBEI_BODY = { | ||
| context: { | ||
| client: { | ||
| hl: 'en', | ||
| gl: 'US', | ||
| clientName: 'WEB_REMIX', | ||
| clientVersion: CLIENT_VERSION, | ||
| }, | ||
| }, | ||
| }; | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| import { describeProvider, makeProviderOptions } from '@/providers/test_spec.ts'; | ||
| import { stubProviderLookups } from '@/providers/test_stubs.ts'; | ||
| import { afterAll, describe } from '@std/testing/bdd'; | ||
|
|
||
| import YouTubeMusicProvider from './mod.ts'; | ||
| import { assertStrictEquals } from 'std/assert/assert_strict_equals.ts'; | ||
| import type { ArtistCreditName } from '../../harmonizer/types.ts'; | ||
|
|
||
| describe('YouTube Music provider', () => { | ||
| const youtubeMusic = new YouTubeMusicProvider(makeProviderOptions()); | ||
| const stub = stubProviderLookups(youtubeMusic); | ||
|
|
||
| describeProvider(youtubeMusic, { | ||
| urls: [ | ||
| { | ||
| description: 'channel page', | ||
| url: new URL('https://music.youtube.com/channel/UCxgN32UVVztKAQd2HkXzBtw'), | ||
| id: { type: 'channel', id: 'UCxgN32UVVztKAQd2HkXzBtw' }, | ||
| isCanonical: true, | ||
| }, | ||
| { | ||
| description: 'playlist page', | ||
| url: new URL('https://music.youtube.com/playlist?list=OLAK5uy_ncbxWnjKunOOgJ7N1XELrneNgiaMMPXxA'), | ||
| id: { type: 'playlist', id: 'OLAK5uy_ncbxWnjKunOOgJ7N1XELrneNgiaMMPXxA' }, | ||
| isCanonical: true, | ||
| serializedId: 'OLAK5uy_ncbxWnjKunOOgJ7N1XELrneNgiaMMPXxA', | ||
| }, | ||
| { | ||
| description: 'playlist page with additional query parameters', | ||
| url: new URL( | ||
| 'https://music.youtube.com/playlist?list=OLAK5uy_ncbxWnjKunOOgJ7N1XELrneNgiaMMPXxA&feature=shared', | ||
| ), | ||
| id: { type: 'playlist', id: 'OLAK5uy_ncbxWnjKunOOgJ7N1XELrneNgiaMMPXxA' }, | ||
| serializedId: 'OLAK5uy_ncbxWnjKunOOgJ7N1XELrneNgiaMMPXxA', | ||
| }, | ||
| { | ||
| description: 'album (browse) page', | ||
| url: new URL('https://music.youtube.com/browse/MPREb_q16Gzaa1WK8'), | ||
| id: { type: 'browse', id: 'MPREb_q16Gzaa1WK8' }, | ||
| isCanonical: true, | ||
| serializedId: 'MPREb_q16Gzaa1WK8', | ||
| }, | ||
| { | ||
| description: 'track page', | ||
| url: new URL('https://music.youtube.com/watch?v=-C_rvt0SwLE'), | ||
| id: { type: 'watch', id: '-C_rvt0SwLE' }, | ||
| isCanonical: true, | ||
| }, | ||
| ], | ||
| releaseLookup: [ | ||
| { | ||
| description: 'Lookup by playlist URL', | ||
| release: new URL('https://music.youtube.com/playlist?list=OLAK5uy_nMjlCmokT89b9UhrFkht6X-2cWdS4nYNo'), | ||
| assert: (release) => { | ||
| assertStrictEquals(release.media.length, 1); | ||
| const medium = release.media[0]; | ||
| assertStrictEquals(medium.tracklist.length, 28); | ||
|
|
||
| const assertArtist = (artistCredits: ArtistCreditName[] | undefined) => { | ||
| assertStrictEquals(artistCredits?.length, 1); | ||
| assertStrictEquals(artistCredits[0].externalIds?.at(0)?.id, 'UCC2AOoHt1RS4Xk0JexgeJZA'); | ||
| }; | ||
| assertArtist(release.artists); | ||
| medium.tracklist.every((track) => assertArtist(track.artists)); | ||
| }, | ||
| }, | ||
| { | ||
| description: 'Lookup by browse URL', | ||
| release: new URL('https://music.youtube.com/browse/MPREb_WvqEoZqND4g'), | ||
| assert: (release) => { | ||
| assertStrictEquals(release.media.at(0)?.tracklist.length, 1); | ||
| }, | ||
| }, | ||
| { | ||
| description: 'GTIN lookup with multiple results', | ||
| release: 60270082120, | ||
| assert: (release) => { | ||
| // Release as associated alternate version. | ||
| // Searching for either releases GTIN (60270082120 and 634164416317) | ||
| // incorrectly returns the version with GTIN 60270082120 | ||
| // | ||
| // Because of this, the provider gives a warning message stating that YouTube returned multiple releases | ||
| assertStrictEquals(release.info.messages.filter((message) => message.type === 'warning').length, 1); | ||
| }, | ||
| }, | ||
| ], | ||
| }); | ||
|
|
||
| afterAll(() => { | ||
| stub.restore(); | ||
| }); | ||
| }); |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we might be want to decide those values based of region parameter
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Setting the
glparameter based on the region would probably be a good idea (though it doesn't seem to affect YouTube's behaviour), but I'm not sure how to select one region to use if multiple regions are provided.As far as I can tell, other providers solve this by using
queryAllRegionsto query each region separately, but this doesn't make sense in this case since the region parameter doesn't affect anything anywaysThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn't know harmony can set multiple regions...
by the way, my main focus is hl parameter, since YTM (sometimes) have a localized tracklists and will be changed by user's language settings. i didn't check it is depending to this parameters, but I think it will.
(sidenote: but I also think change language parameter by region parameter is good or not? i don't know. Apple Music behave like that, but actually some Apple Music region supports multiple language in one regions and shows based of your device's language settings. e.g. JP region have a japanese and english tracklist)
e.g. https://music.youtube.com/playlist?list=OLAK5uy_nXbgxu51sJwZQz69UwVWVvu-j22_IYvA8 this album have a Japanese tracklist and English tracklist, and if your language settings aren't japanese, it should show english tracklist
I think different tracklist for en-US and en-GB are possibly exists, but I don't know about it actually exists or not.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yup okay, setting
hltojaseems to work in order to get localized track lists.I could try to set this dynamically, but I'm still not sure which language to use when.
I think I've read that the MusicBrainz policy is that release and recording titles should always be submitted in their original localisation (so, japanese localisation for Japanese releases), but that would require knowing what the "original" localisation is.
The problem with this is that for example, I've noticed that some releases and tracks tend to have Japanese localisations even if they aren't originally Japanese (like this album by the German band Rammstein)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When the user specifies multiple preferred regions, they will only be used by providers which support the
queryAllRegionsfallback. But if YouTube doesn't "fail" to return data for a certain unavailable region (in some way), this concept doesn't make sense and the first "supported" region of the preferred regions should be used.So unless there is a meaningful list of supported regions for YouTube, I would simply use the first region.
Regarding the language /
hlparameter, I wouldn't try to derive this from the region as these aren't always one-to-one mappings. If we arrive at the conclusion that the language is a useful parameter (for YouTube and potentially other providers), Harmony should allow the user the set a preferred language as well. We can even extract the language from the release URL for some providers (namely Deezer, Spotify, Beatport, maybe Qobuz in the future), which would override the preference input just like a region from a release URL does (for Apple).