Skip to content

zailleh/aus_pet_crawler

Repository files navigation

😸 Australian Pet Adoption API

🐕Mission: to be an API for a database that contains pets for adoption from all over Australia!

Use the API at: https://au-pet-api.herokuapp.com/

🐰 How to use the API

The API returns application/json data only. The root for the API returns a list of all pets. You can parse this data into an object in the language of your choice.

🐶 API URLS

  • .../ returns all pets
  • .../pets/ returns all pets
  • .../pets/:id returns urls for pictures for the pet :id
  • .../shelters/ returns all shelters
  • .../shelters/:id returns shelter for :id

Features to add:

  • .../pets/:type to return pets of type :type, eg 'cat'
  • .../pets/:state to return pets in a given :state, eg 'NSW'
  • .../pets/search?s=:state&t=:type... search url to return pets matching given criteria

Technical Information

This API is run using:

  • Ruby 2.5.1
  • Rails 5.2.0
  • Postgresql 10.4

Dependencies

  • chromedriver ( install with brew cask install chromedriver )
  • Chrome Binary (ie, Google Chrome browser must be installed)

Buildpacks for Heroku:

Heroku requires the chromedriver and google-chrome buildpacks to run the scraper portion of the API:

heroku buildpacks:add https://github.com/heroku/heroku-buildpack-google-chrome
heroku buildpacks:add https://github.com/heroku/heroku-buildpack-chromedriver

Database Setup

As mentioned above, the database in use is PostgreSQL. After cloning the repo, assuming you have PostgreSQL already running, simply run the below:

rails db:create && rails db:migrate && rails db:seed

db:create can be left off for Heroku deployment.

Services

This API currently uses a single ActiveJob to gather data, called CrawlJob. This job will run automatically 1 minute after startup and every hour thereafter.

To run the job manually, in the Rails concole, run:

CrawlJob.perform_now

This job should be further simplified and broken down as more sites and crawlers are added.

About

A crawler/scraper and API that will contain pets for adoption from all over Australia

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published