Add pagination to getPapers endpoint to prevent slowness and improve performance on fetching data

[gfviegas]

Hello there! First of all I'd like to congratulate you all for this work and the paper. Really impressive stuff and I hope it to be useful for me as a graduate computer science student.

As I was checking out the demo I couldn't help but notice it was way too slow. I've checked my browsers dev tools and found out the main reason is the frontends call to getPapers endpoint.

The demo kept loading for a long while until I realize the reason: it's really a full data fetch.

I'd like to propose a improvement to tackle this performance issue and introduce pagination parameters to the getPapers endpoint, so you can fetch papers as they are required/a small set at a time.
Of course eventually it would require a change in frontend to consume this feature, but I think the tool is completely inviable without it. I mean, the demo still haven't load for me (20 minutes downloading the response and counting) in a macbook pro, imagine what would look like in low specs computers and mobiles.

I mean, you are requiring the browser to download around 90MB of compressed data and then parse it.

What are your thoughts? I could help implement it myself as I'm a bit experienced in this kind of problem.

[arpitnarechania]

Hi @gfviegas, thanks for reinforcing what my team has determined as a must-have enhancement in the next release! You're right, it may take some time but I did not expect it to be 20 minutes -- that is really good to know as it takes about half a minute on my computer.

If we move to a paginated API, as you rightly pointed out, the UI will also have to update in more than one way. Below are three implications that I had noted a while back:

1. We currently have an infinite virtual scrolling datatable that creates DOM elements on-the-fly (but all data is still stored in memory). Once we implement pagination, we could EITHER make goToNextPage() or goToPage(pageID)-like API calls when the user swipes/scrolls down to request more records OR move to a traditional data table with buttons for pagination. What do you think?

2. Another implication due to the above search is on the SEARCH and FILTER functionalities currently in the datatable, specifically in the COLUMN HEADERs. These will also have to move server side, e.g., when a user looks up a paper by "keyword"-based query or applies a filter to show papers only in the year 2020 will have to be done server-side through an API.

3. The UMAP visualization will have to be loaded from a new API (e.g., getUMAPData()) because it does require ALL the paper coordinates on load. In the same API, we can pass the paper's ID with the coordinates so that when the user hovers/clicks on the points, we can call a new getPaperInfo(paperID) API to fetch additional details (that can be shown in tooltips) OR even to add it to the SELECT/SAVE lists.

All other aspects should remain as-is, I think. Would you like to work with me to implement some of the above aspects?

[gfviegas]

Overall sounds good! It's not that much of a change but it will significantly impact the application.

We currently have an infinite virtual scrolling datatable that creates DOM elements on-the-fly (but all data is still stored in memory). Once we implement pagination, we could EITHER make goToNextPage() or goToPage(pageID)-like API calls when the user swipes/scrolls down to request more records OR move to a traditional data table with buttons for pagination. What do you think?

Definitely I'd pick traditional pagination with next / previous page and random access page. It's way easier, friendly and prevents really annoying bugs on mobile, specially in safari. But I think that's the way to go. A async call to API to fetch the data when the page changes.

Another implication due to the above search is on the SEARCH and FILTER functionalities currently in the datatable, specifically in the COLUMN HEADERs. These will also have to move server side, e.g., when a user looks up a paper by "keyword"-based query or applies a filter to show papers only in the year 2020 will have to be done server-side through an API.

Yes, and that's how supposed to be in any web-based application with that amount of data. Leave the hard work to the server.

The UMAP visualization will have to be loaded from a new API (e.g., getUMAPData()) because it does require ALL the paper coordinates on load. In the same API, we can pass the paper's ID with the coordinates so that when the user hovers/clicks on the points, we can call a new getPaperInfo(paperID) API to fetch additional details (that can be shown in tooltips) OR even to add it to the SELECT/SAVE lists.

A hash-map with the key being the coordinate and the value the paper id fulfills the need to build the UMAP visualization. You can transverse all coordinates with a simple Object.keys(map) to draw the map and also quick-fetch the related ID to that point in a simple map[id] access. You are absolutely right.

---

PS: Since there will be changes all-around the API itself, I would also suggest to rename the endpoints to substantive-named routes, as it is the REST pattern. Besides following the pattern, you'll also be clear what your resource actually is. I'd say something like:

• /api/papers
• /api/papers?page=2&itemsPerPage=20&sort=name:asc
• /api/papers/:id
• /api/umap

Etc. Some reference if you may. I've also written some stuff about REST APIs, in portuguese though.

[arpitnarechania]

Thank you for the pointers @gfviegas. Let me spend some time this weekend on this and I'll get back to you. There's another open issue that I'm planning to start working on in-parallel. :)