Add support for Media Segment API and implement Skip Intro button

[noranraskin]

Implements #1525

This is a work in progress, but I wanted to open this PR already for people to comment on how to do things better.
To test this (and to use it) the Jellyfin server requires a plugin that scans the library for segments (intros, ads, etc.)
https://jellyfin.org/docs/general/server/metadata/media-segments/#plugin-support

This code uses types from JellyfinAPI but implement getMediaSegments since that doesn't exist there.

Goals:

• Fetch Media Segments from Jellyfin server after starting playback
• Store segments in MediaPlayerItem
• Update currentSegment in MediaPlayerManager
• Implement skipCurrentSegment function
• Create SkipSegmentButton view
• Display SkipSegmentButton in PlaybackControls
• Get SkipSegmentButton to work for Native player
• Add option in settings to define behaviour for different types of segments
• Add localisation
• Fix TvOS views
• Fix autoskip for Swiftfin player

If possible I would really like to get this to work for the native player too, I tried a few different things. In its current state there's just an overlay on the NativeVideoPlayerView, however this is not working (and I don't understand why not).

I unfortunately don't have access to an AppleTV at the moment so I'd very gladly accept some help polishing the views for TvOS especially since it seems to not be working in general atm. This is what it looks like for me:

On an iPad simulator it looks like this:

[JPKribs]

This code uses types from JellyfinAPI but implement getMediaSegments since that doesn't exist there.

Please use the SDK for this item and please call this via a ViewModel. We can likely call this from ItemViewModel but please note that this will likely need to come after #1752 as there are some extensive reworks to ItemViewModel. Alternatively, this can be called when the MediaPlayer is generated which might make more sense.

Please find the correct SDK usage below:

Paths.getItemSegments(itemID: String, includeSegmentTypes: [MediaSegmentType]?)

For reference:
https://github.com/jellyfin/jellyfin-sdk-swift/blob/main/Sources/Paths/GetItemSegmentsAPI.swift

[noranraskin]

Ahh thanks was looking for getMediaSegments so I didn't find the appropriate function. Will change that. However the way I understand it is that MediaPlayerManager keeps state of playback und should therefore handle segment skips and updates. The MediaPlayerManager only has a reference to a MediaPlayerItem. I agree it would be preferable to have the fetching logic in ItemViewModel but I don't see an elegant way to pass the mediaSegments. Any idea or should I just wait for #1752 ?

[JPKribs]

I think the manager should be fine? Full disclosure, I haven't touched a lot of playback since the rework. I think it makes sense to proceed with it in the manager. Taking a quick look, that appears correct but I'd have to look at this more after Thanksgiving to confirm. I think you're good to proceed in the manager since that other PR is more ItemViewModel.

For tvOS, no need to touch the player View since that's being redone. I've personally taken a look at the tvOS player recently and SwiftUI differences on tvOS make building a player UI absolutely miserable IMO haha. That will all come in a later PR but I imagine I'm going to have to lean on someone with more UIKit experience since remote control functionality appears to be severely limited in SwiftUI. For this PR, I would make sure the shared components and settings exist but you can leave a TODO in the existing player view to add this for now.

I think for this PR:

• Settings/Config on iOS and tvOS
• Swiftfin Player UI element
• Make the skip generic to allow for auto-skipping

You can skip NativePlayer UI. If you are feeling adventurous, this would be a cool time for automatic skipping as well as a setting.

Last note is this button itself right now relies on padding for placement. I would prefer we figure out a cleaner was to do this as to prevent issue with screen sizing. Even just placing this in the bottom right with insets but moving it up when the standard overlay exists?

I might also ask we use the TintedMaterial button for styling to stay more consistent with other elements.

Let me know if there is anything I can help answer or assist with!

[JPKribs]

I think we'd want, for settings we'd want a Default for:

MediaSegments: Bool

Then, if enabled, we'd have a secondary setting under for SkipMediaSegment which contains the types using:

https://github.com/jellyfin/Swiftfin/blob/main/Swiftfin/Components/OrderedSectionSelectorView.swift

You can see how I did customizing enabled filters for that usage. Default that setting to []

In practice, show media segments of enabled. Automatically skip if the bool is true and the media segment type exists in that list.

[noranraskin]

I went ahead and implemented the settings how they would be most logical to me. Feel free to take a look and give me an opinion. I didn't quite get your explanation about customised enable filters, but I think my solution does that.

Here's what it looks like atm:

On iPad:

On apple TV:

In both cases the configure media segments is only visible when the media segments toggle is enabled.

All strings are localised but with english translations.

A few kinks to work out still:

• Skipping via button (when setting is ask) works well for TV and iOS under Swiftfin player
• Autoskipping does not work for Swiftfin player (log shows VLC state change to buffering and it doesn't continue playing) I don't really know what the problem could be since it should be doing the same as a a button press (I tried calling skipSegment from main thread but that didn't change anything). Autoskipping does however work for the native player.

I could use some help for the autoskip not working properly.

Next I'm gonna try to work improving the styling/placement of the button like you mentioned.

[JPKribs]

Made a quick change to the settings like this:

Simulator.Screen.Recording.-.iPhone.17.-.2025-11-27.at.11.01.28.mov

This should make this more dynamic so as these types are added in the future this will automatically be added to our settings

[noranraskin]

Ahh okay now I understand. I was more following the functionality of jellyfin web with my design

With your change how would you make it that ads are always auto skipped, intros and outros make a "Skip Intro/Outro" button appear and recaps and previews don't make a button appear?

[JPKribs]

That's definitely a fair point! Let me think about that and get back to you. I agree, it would be nice to set these on a per segment type basis, but, where possible, we want to try and keep the settings manageable.

I think the correct solution might be a ForEach loop over MediaSegmentType.allCases. Then, three settings in total. Enable media segments, skip media segments, and ask media segments.

The first one would be a Boolean, and the other two would be arrays of items.

But, for now, definitely don't worry about this part of it! I can make a commit with what I'm thinking about when I have time this weekend. If you wanna focus on the functional part of auto skip, that would be perfect! I don't personally have media segments set up on my server so I will need to do some set up before I can help test this out, but I can help you test probably in a couple days after I've had a chance to set up that feature

[JPKribs]

Okay, reworked the settings. This should be dynamic, in the sense that any future additions will be accounted for without needing to change the settings. This also accounts for the Off/Ask/Skip logic to allow more granular settings for this. Let me know what you think!

Simulator.Screen.Recording.-.iPhone.17.-.2025-11-28.at.00.13.57.mov

[noranraskin]

This looks very good! Will be trying to work out the remaining problems now

[JPKribs]

I found a plugin for media segments so I'm starting that scan now. It might be a bit but I can assist with testing this whenever this wraps up!

[LePips]

Thank you for your contribution!

For the segment handling, this should be implemented as a MediaPlayerObserver conforming object instead of having logic directly in the MediaPlayerManager. I understand this was probably following the PreviewImageProvider implementation, but that should eventually be changed as well.

Here is an example of what that object would look like, quickly spun up for testing:

class ItemSegmentObserver: ViewModel, MediaPlayerObserver {

    @Published
    private(set) var currentSegments: [MediaSegmentDto] = []

    private(set) var skippedSegments: Set<String> = []

    private var segmentTask: Task<[MediaSegmentDto]?, Never>!

    weak var manager: MediaPlayerManager? {
        willSet {
            guard let newValue else { return }
            setup(with: newValue)
        }
    }

    init(itemID: String) {
        super.init()

        self.segmentTask = Task { [weak self] in
            let request = Paths.getItemSegments(itemID: itemID)
            let response = try? await self?.userSession.client.send(request)
//            return response?.value.items

            return [
                .init(
                    endTicks: 100_000_000,
                    id: "test",
                    itemID: nil,
                    startTicks: 100,
                    type: .intro
                ),
            ]
        }
    }

    private func setup(with manager: MediaPlayerManager) {
        manager.secondsBox.$value
            .sink { [weak self] newValue in Task { await self?.secondsDidChange(newValue) } }
            .store(in: &cancellables)
    }

    private func secondsDidChange(_ newSeconds: Duration) async {
        guard let segments = await segmentTask.value else { return }

        currentSegments = segments.filter { segment in
            guard let startSeconds = segment.startSeconds, let endSeconds = segment.endSeconds else { return false }
            return startSeconds ..< endSeconds ~= newSeconds
        }

        if let introSegment = currentSegments.first(where: { $0.type == .intro }), !skippedSegments.contains(introSegment.id!),
           let endSeconds = introSegment.endSeconds
        {
            skippedSegments.insert(introSegment.id!)
            manager?.proxy?.setSeconds(endSeconds)
        }
    }
}

extension MediaSegmentDto {

    var endSeconds: Duration? {
        guard let endTicks else { return nil }
        return .ticks(endTicks)
    }

    var startSeconds: Duration? {
        guard let startTicks else { return nil }
        return .ticks(startTicks)
    }
}

For now, this is good enough to be initialized and added to the MediaPlayerItem.observers array, similar to MediaProgressObserver.

For the button, you've already put it in PlaybackControls like it should, but here's how it would be made with the provider above:

NavigationBar()

Spacer()

if let itemSegmentProvider = manager.playbackItem?.observers
    .first(where: { $0 is ItemSegmentObserver }) as? ItemSegmentObserver
{
    SegmentButton(segmentProvider: itemSegmentProvider)
        .edgePadding()
}

PlaybackProgress()

SegmentButton can just be a plain white button. We should extend MediaSegmentDto to have a display message for each skippable type: Skip Intro, Skip Recap, Skip Outro, etc. We can later take a look at implementing a Play Next Item/Episode later, where the outro segment would look at its end ticks, the media's runtime, and the queue to determine what message to present.

You'll notice that my example object has a set on what segments have already been skipped. If the segment is to be automatically skipped, we should still provide functionality to scrub back through media without skipping - and should instead present the skip button during that segment.

You'll also notice the example segment I have doesn't start at 0 ticks. Right now, the VLCUI wrapper will first play the media and then set the correct starting seconds. This could be problematic if there are intro segments that start at or are close to 0 ticks. (Edit: don't implement logic to consider this. I should have a change in VLCUI to this behavior)

We should also not worry about the native player, as it will be removed. Also don't worry about tvOS for now, as its implementation will require more focus considerations.

[LePips]

Don't sort based on the display title. We should also conform MediaSegmentDto to SupportedCaseIterable, so we don't use unknown.

[JPKribs]

Sorry that was me. For unknown, that's part of the DTO and essentially anything that isn't intro, outro, commercial, preview, or recap.

When I was messing around with this, things like "Trialer" where I typo'd would be in there. For our usage, maybe this is L10n.other instead of unknown and we have that string be "Skip "+"name". So, for my example, it's "Skip Trialer" instead of "Skip Unknown" and managed via an "Other" toggle.

All else we'd keep localized but unknown we'd just pull from server.

That's IF we want to build in a way to capture non-standard configs.

[LePips]

I would prefer a [MediaSegmentType: MediaSegmentBehavior] dictionary instead of 2 arrays to try and keep mutually exclusive.

[noranraskin]

Thanks for the feedback and the code examples! Will try to update everything accordingly. Free time is a bit limited atm. Is the native player going to be removed before the next TvOS release? I rely on that since I'm using airplay speakers. Any updates on the audio issues with VLC or is mpv going to replace native? I don't want to capture this pr but I would like to learn more

[LePips]

Unsure if it would be removed on next release, but the usage of AVPlayer won't ago away. Instead, the video player is able to swap between VLC and AVPlayer, but the AVPlayer layer isn't complete yet along with proper PiP support in the video player. I can't say anything about what mpv would replace for now.