Expand/Collapse Fields

[pmdevita]

Is your feature request related to a problem? Please describe.

One of the more recent API patterns I've seen is expand/collapse, where you can request an API to return a relation on a Schema that it normally would exclude for query performance.

GET /book/123

{
  "id": 123,
  "name": "Pride and Prejudice"
}

GET /book/123?expand=author

{
  "id": 123,
  "name": "Pride and Prejudice",
  "author": {
    "id": 456
     "name": "Jane Austen"
  }
}

It would be nice if Shinobi provided assistance for expanding/collapsing fields in Schema responses. Manually writing the logic for this on each endpoint that uses such a schema would make the pattern difficult to adopt and maintain.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Generally, if we're going to fetch across a relation in the first place, the user already has to specify what schema it is that will be attached.

class BookSchema(ModelSchema):
    author: AuthorSchema

    class Meta:
        model = Book
        fields = ["id", "name"]

I think it may be possible to create a type annotation that can mark this as something to expand.

class BookSchema(ModelSchema):
    author: Expand[AuthorSchema]

    class Meta:
        model = Book
        fields = ["id", "name"]

During boot and setup of the API endpoints, Shinobi can then inspect a response schema for it's expand fields (and the children of those fields as well for nested expansion) and attach those as query params to the endpoint. We already have to process all of a Schema's fields in its Metaclass so I figure we just solve it there too.

Finally, during validation from ORM to Schema, Shinobi would take the given expand params and either include or exclude those fields in the response. I think this can maybe just get passed through with context but we'll have to see, I'm hoping it doesn't slow down validation in large lists/pagination.

Alternatively, if this is on pagination, Shinobi could manipulate the query to block the loading of non-included fields (.prefetch_related(Prefetch("author", Author.objects.none())), this is a bit more risky depending on use but it would perform better. Shinobi should not do the opposite and prefetch for you though.

Collapse would work just opposite of Expand, it would be a field that would be normally included but could be optionally excluded.

[maxhollmann]

Would be amazing if this was a native feature. We implemented something like this ourselves like this:

def resolve_nested(fn):
    fn_name: str = fn.__name__
    prefix, resource_name = fn_name.split("_", maxsplit=1)
    assert prefix == "resolve", "@resolve_nested should only be used on resolve_* functions"
    include_attr = f"include_{resource_name}"

    @wraps(fn)
    def wrapper(obj, *, context):
        if not hasattr(context["request"], "include"):
            return None

        if getattr(context["request"].include, include_attr):
            return fn(obj, context=context)
        else:
            return None

    wrapper._nested_resource_name = resource_name

    return staticmethod(wrapper)


class ProjectSchema(ProjectNestedSchema):
    provider: ProviderSchema | None

    @staticmethod
    @resolve_nested
    def resolve_provider(obj, *, context):
        return obj.provider


@validate_include_query_schema(ProjectSchema)
class ProjectIncludesSchema(Schema):
    include_provider: bool = False


@router.get("/", response=list[ProjectSchema])
def list_projects(request, include: Query[ProjectIncludesSchema]):
    request.include = include
    return Project.objects.all()

Issues we have with this:

• We can only specify includes in the first level
• The response type in the OpenAPI spec doesn't know about the include logic, so our frontend will need to do extra work to tell Typescript that provider definitely exists when we included it. Probably not fixable with plain OpenAPI, but maybe a specification extension could help allow frontend code generators generate better types.