Skip to content

Unable to fetch all data from Apis that implement pagination mechanisms #302

Open
@dpsenner

Description

@dpsenner

There are many api's out there that implement pagination mechanisms. To mention some:

  1. Atlassian Jira
  2. Teamviewer Api

It would be great if the plugin would support these apis better by leveraging the pagination and fetching more data. This is not as easy as it might sound. I can't propose a sound technical spec on how to implement this. Therefore I'm highlighting two concrete examples:

Teamviewer Api endpoint https://webapi.teamviewer.com/api/v1/reports/devices

This api sends an optional next_offset item in the json response that should be sent as http argument offset_id to fetch more data. For example the first response to /reports/devices?from_date=2021-08-01T05:14:00Z json may look like:

{
   ...,
   "next_offset": "eyJTdGFydCI6IjIwMjEtMDgtMDFUMDU6MTQ6MDArMDA6MDAiLCJUYWJsZUNvbnRpbnVhdGlvblRva2VuIjoiOGVmYjQyNjctMjM1NS00YzMxLWIyYmYtNDgzMTlkMzIzOTRhIn0="
}

And the following request to fetch more data should now be sent with the parameter /reports/devices?offset_id=eyJTdGFydCI6IjIwMjEtMDgtMDFUMDU6MTQ6MDArMDA6MDAiLCJUYWJsZUNvbnRpbnVhdGlvblRva2VuIjoiOGVmYjQyNjctMjM1NS00YzMxLWIyYmYtNDgzMTlkMzIzOTRhIn0=.

A smart guy would be able to base64 decode eyJTdGFydCI6IjIwMjEtMDgtMDFUMDU6MTQ6MDArMDA6MDAiLCJUYWJsZUNvbnRpbnVhdGlvblRva2VuIjoiOGVmYjQyNjctMjM1NS00YzMxLWIyYmYtNDgzMTlkMzIzOTRhIn0= to:

{"Start":"2021-08-01T05:14:00+00:00","TableContinuationToken":"8efb4267-2355-4c31-b2bf-48319d32394a"}

Atlassian Jira Api endpoint https://jira.example.com/rest/api/2/search

This api endpoint expects numeric http parameters maxResults and startAt while returning startAt, maxResults, total in the json response. For example requesting ?maxResults=2000&startAt=100 the following is returned:

{
    "startAt": 100,
    "maxResults": 1000,
    "total": 1521,
    ...
}

Note that maxResults is capped at 1000.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Status

    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions