Instances list endpoint pagination

[james-owen]

• Adds paginate() to postgres query functions.
• Adds PAGE_SIZE env var, that when defined, sets up pagination by default
• No functional change if PAGE_SIZE is not defined

Pagination will work by fetching the first page, then doing another request with the last item that was returned prev=<uri>. Paginating in this way using the "seek" method isn't obviously better than using offset at first glance, especially considering that it requires an order by clause, but in local testing it is significantly faster especially at high offsets.

Here are the results from querying a table of clay-paragraph components with a page size of 10:

0th item

• no difference

10,000

• offset: 9 ms
• seek: 5 ms

100,000

• offset: 3189 ms
• seek: 5 ms

500,000

• offset: 17375 ms
• seek: 9 ms

No paginating, get all instances

• 8285 ms

This approach closes a current DoS vector and also avoids creating an even worse vector.

[mattoberle]

This looks really good.
Appreciate the additional work that went into performance testing too!

I left feedback on a couple of documentation-related items, nothing critical.

One case that is not covered in the tests is explicitly setting the CLAY_STORAGE_PAGE_SIZE environment variable. That could be a little tricky to mock depending on where your test is importing postgres/client.js, but might be worth covering since a future change to the parsing logic or default value could constitute a breaking change. What do you think?

[mattoberle]

I think you should note in the docs here that a page size of 0 will explicitly disable pagination.

[mattoberle]

This is referred to everywhere else as CLAY_STORAGE_PAGE_SIZE.
It seems that we already have an inconsistent naming scheme for the environment variables in this project, but maybe stick with the more generic CLAY_STORAGE_PAGE_SIZE (something that other storage back-ends could adhere to).

[jjpaulino]

🍕