Applying path clips to their respective images.

[benmagos]

Q: How can I apply accurate clipping regions to Pixmaps that are not Rects?

My question is essentially a duplicate of: #1312
Which got me in to the wonderful world of bboxlog and seqno, but it was still not providing angled (non-rectangular) clips, or at least, I struggled with applying them.

So, given that my question will center more around the newer (v1.22.0+) get_drawings(extended=True), I thought a new discussion was merited.

Here's my best attempt so far at applying the hard clipping masks to the images, but I am highly hopeful there is a more reasonable solution. I don't believe Pixmap's have these kinds of functions so I am using Pillow for the compositing.
What I am really looking for is Pixmap.clip(Shape) or similar.

Steps:

1. Use Page.get_drawings(extended=True) to get clips.
2. Attempt to accurately link clips with their respective images.
3. Draw clip as black Shape and save as Pixmap.
4. Use Page.get_image_info(xrefs=True) or equivalent to get image.
5. Apply black clip Pixmap as alpha composite of original image Pixmap.
6. Profit.

A couple nested questions from the above steps:

2. Attempt to accurately link clips with their respective images.
In my testing, clips do not have seqno, and their bbox/scissor will not 100% match the image bbox.
However, the items of the clip path match the immediately preceding path, which has a seqno and can then be applied to the image at seqno + 1.

Is this reliable? 🤔

Example:

items = None
index = 0 # would be set to previous drawings `seqno` in below loop as well.
for i, path in enumerate(page.get_drawings(extended=True)):
        if path["type"] == "group":
            continue

        is_clip = path["type"].startswith("clip")  # could be clip or clip-rect
        prevItems = items
        items = path.get("items")
        if is_clip and items == prevItems:
            # save the clip region to be used later
            index = index + 1  # clip applies to image at the "would-be" `seqno` of the clip
            image_clips[index] = (rect, pix)
        else: # ...

5. Apply black clip Pixmap as alpha composite of original image Pixmap.
This does not seem like the best approach at all, but it's returning an image that at least has the correct portion of the image cropped out.

    f = io.BytesIO()
    image = Image.open(io.BytesIO(pix.pil_tobytes(format="png")))
    clipped_region_image = Image.open(io.BytesIO(
        clipped_region_pix.pil_tobytes(format="png"))).resize(image.size)

    if image.mode == "RGBA":
        image = Image.composite(
            image, clipped_region_image, clipped_region_image)
    image.save(f, format="png")
    pix = fitz.Pixmap(f.getvalue()) # clipped final image

Original Image:

Clipped Image (note, I have not applied the transform to the image yet):

Not quite right yet.

Please tear the approach apart and let me know where better solutions may be available!
Thank you.

Sample file:
samplefile.pdf

[JorjMcKie]

The vector graphics extraction that includes clips and groups follows the logic as explained in the documentation here.
These additional items establish a hierarchy among the extracted dictionaries, expressed by the level dict key.
Groups and clips dictionaries carry no sequence number, because they do not draw or write anything.

The scissor rectangle of a clip is defined as the {union of all points of its items} intersected with the scissor of its parent clip - it will thus always be a subset of the parent scissor.
It may happen, that not all items are fully contained in the scissor.

There is no effort to somehow compute non-rectangular clip areas: The items in the clip may define pretty much arbitrary shapes like rhombuses, triangles, circle segments, etc. Evaluating this is one of the tasks of any PDF viewer. Whatever a clips shape is: the only thing that can be deducted directly is that it will be contained in the scissor.

There is a high chance that regular (= stroke or fill) paths have items that are at least partly outside the parent's clip scissor. After all, achieving this is the purpose in the life of a cip.

Please be aware, that all information in page.get_drawings() is taken directly from what MuPDF found in the page's /Contents objects. There is no effort to validate anything, or resolve discrepancies. It is a protocol of what was encountered during sequentially parsing the contents source.

The sequence numbers are ascending because they originate from counting the encounters of fill and stroke paths. Any gaps are caused by other commands: text, images etc.

[more-strive]

Is there any progress