Support backends other than furaffinity.net

[featherbutt]

The idea

FALocalRepo is an incredible tool for downloading and locally organizing art galleries, but it's currently limited in only being able to scrape FurAffinity (or sites with identical HTML). Other gallery sites exist that have an identical or nearly identical data model, and being able to organize content from those galleries in the same local repo would useful.

I'm willing to take on the necessary work for this and submit it as a pull-request.

Implementation ideas

The biggest obstacle is that the FA-specific code in FAAPI is closely coupled to backend-agnostic code, but this can be fixed in two ways.

1. Create a separate module for each backend site, which implements a standard interface. Allow FALocalRepo invocations to specify a module to load, with FAAPI as the default. This requires no additional changes to FAAPI, but would necessitate some amount of code duplication in each module.

2. Attempt to decouple FAAPI into its FA-specific and generic parts. The user can inject a dependency into FAAPI's constructor which overrides backend specific behavior.

Neither option would require bloating FAAPI with code for other backends. (2) seems cleaner but would require more work.

Assuming that we want to allow content from multiple backends to coexist in the same local repo, we would also need a "source" field on submissions/users/journals, both to differentiate reused IDs across backends, and as an additional field in searches.

[MatteoCampinoti94]

I like the idea!

FALocalRepo would indeed be able to handle it just by adding a source field to the tables, although I think it would be cleaner to "lock" a database to a specific site when initialized, then the user would only have to choose which database to use and the program would select the library based on the database itself.

[MatteoCampinoti94]

Alas it would require a lot of work to make the necessary libraries.

Most of these sites do not have a proper API, so they'd need a scraper built for each one of them, and most of the code could not be ported over from FAAPI. The classes would probably require the least amount of work, but the parse module would have to be built from scratch with a suite of tests to ensure it keeps working.

[MatteoCampinoti94]

For example, So Furry does not have a "view all" option for the galleries, requiring branching into the sub-folders to navigate everything. It also doesn't have a previous/next button on the actual art/story pages (it has a series navigator for the latter, but it's based on the folder, not the entire gallery). Furthermore, the way user pages work (subdomains) would necessitate a rewrite of the connection module too.

[MatteoCampinoti94]

As a starting point we may want to focus on a site that has a proper API, like Weasyl.

[featherbutt]

Starting with a site that has a proper API like Weasyl makes sense.

Ironically, for SoFurry at least I already wrote a SoFurry scraper that I used to create a local backup, so converting it to a FAAPI-compatible parse module should be doable: the test suite will probably be the most laborious part.

[MatteoCampinoti94]

So, I've made a library for Weasyl :3

There are already was one called pyweasyl, but it hasn't been updated in 7 years and I wanted to rewrite with proper typing and objects. It only needs a bit of testing, but otherwise it's ready and has all the API endpoints implemented with pydantic models for validation and decoding.

The problem is with FALocalRepo. I've started laying out the structure of a backend-based downloader, but it's going to require a LOT of work. The current version relies on specific functions and classes from FAAPI, so we'd have to write some generalised objects, a generalised downloader, and classes that tie the two together using the different libraries for each website.

[Deer-Spangle]

Hey there, spotted this, and had considered the same in the past for some other stuff. But there's a lot of difference between sites that would make structures tricky, I should think?
(Some sites have multiple images per submission, as well, for instance)
A structure flexible enough to support each site, might just be messy, or oversimplistic, I dunno?
It might be okay for FALocalRepo, as it's just downloading a gallery or whatever, but yeah, I think the generalised objects would depend heavily on your usecase, and hence be a bit specific

I've got a spreadsheet comparing a bunch of different furry sites and libs by featureset btw, which might help? But it needs some updates, but it might be helpful?
https://docs.google.com/spreadsheets/d/e/2PACX-1vS9rYfh38L9FaroN31n-ghBX8n4SfpwTch5wD_vBLPoXrsH0BlVTYzfmJ0oPoi-hgCLUuiB8B6KQ_EA/pubhtml?gid=0&single=true

[MatteoCampinoti94]

That's a great resource @Deer-Spangle, thanks for sharing!

Actually, feature-wise the only thing that would need adding is the ability to have multiple media files for each submission, which is something I had already considered implementing and would require only minor modifications to the database library.

I have just finished writing a generalised base class to handle download operations and it's now working fine with both FA and Weasyl. The only issue is that Weasyl has character entries, not just submissions and journals, and they have their own separate ID's. Currently I'm trying to find a way for them to coexist in the same table without resorting to multiple primary keys.

Another extra feature are SoFurry's story series, which could be generalised with grouping to support FA's folders as well, though I'm a bit reluctant to implement something so dynamic and not easily updated (if a folder changes name, then newly downloaded submissions would not be linked to older ones).

[MatteoCampinoti94]

I'm closing this issue and moving all discussions to the Alternative FALocalRepo API Backends project :)