Documentation

Infrastructure

The website runs in Docker on a virtual server on the VU campus running Ubuntu. The application runs using 3 docker containers:

• searchblocks-db a PostgreSQL 14 database, https://hub.docker.com/_/postgres. This contains the tables used by Blacklight and the ActiveRecord store for the admin pages.

• solr a SOLR 8 instance, https://hub.docker.com/_/solr. Note: to keep memory usage down this also runs the cores of 2 other Blacklight sites running on the same server. The SOLR instance is not, and should not be, directly exposed to the internet.

• searchblocks-app runs the Ruby on  Rails app. Built from https://github.com/peer35/searchblocks/blob/master/Dockerfile. searchblocks-app is exposed to the internet by proxying via nginx

Relevant docker-compose:

version: '3'

networks:
  default:
    external:
      name: bl-network # create with docker network create bl-network
  
services:
  solr:
    container_name: ${SOLR_HOST}
    restart: always
    image: solr:8
    environment:
      SOLR_JAVA_MEM: "-Xms1024m -Xmx1024m"
    volumes:
      - /docker/solr/data:/var/solr/data
      - /docker/solr/configsets:/opt/solr/server/solr/configsets
    entrypoint:
      - bash
      - "-c"
      - "precreate-core ${SOLR_CORE_SEARCHBLOCKS} /opt/solr/server/solr/configsets/${SOLR_CORE_SEARCHBLOCKS}; exec solr -f"

  
  searchblocks-app:
    container_name: searchblocks-app
    restart: unless-stopped
    build:
      context: ../searchblocks
      args:
        - APP_USER_UID=${SEARCHBLOCKS_USER_UID}
        - APP_USER_GID=${SEARCHBLOCKS_USER_GID}
    user: ${SEARCHBLOCKS_USER_UID}:${SEARCHBLOCKS_USER_GID}
    depends_on:
      - searchblocks-db
      - solr
    environment:
      DB_USER: ${DB_USER_SEARCHBLOCKS}
      DB_PW: ${DB_PW_SEARCHBLOCKS}
      DB_HOST: ${DB_HOST_SEARCHBLOCKS}
      DB_NAME: ${DB_NAME_SEARCHBLOCKS}
      RAILS_ENV: production
      SITE_NAME: blocks.bmi-online.nl
      SECRET_KEY_BASE: ${SECRET_KEY_BASE_SEARCHBLOCKS}
      SOLR_URL: http://${SOLR_HOST}:8983/solr/${SOLR_CORE_SEARCHBLOCKS}
      WEB_CONCURRENCY: 1
      RAILS_MAX_THREADS: 5
    volumes:
      - /docker/searchblocks/log:/usr/src/app/log
    ports:
      - "3012:3000"
 

  searchblocks-db:
    container_name: ${DB_HOST_SEARCHBLOCKS}
    image: postgres:14
    restart: always
    environment:
      POSTGRES_PASSWORD: ${DB_PW_SEARCHBLOCKS}
      POSTGRES_USER: ${DB_USER_SEARCHBLOCKS}
      POSTGRES_DB: ${DB_NAME_SEARCHBLOCKS}
      PGDATA: /var/lib/postgresql/data/pgdata
    volumes:
      - /docker/searchblocks/db:/var/lib/postgresql/data

Structure

Front-end

Blacklight

• Base is the open source app "blacklight", https://github.com/projectblacklight/blacklight. This basically is a user-friendly faceted search on a SOLR index. Note we are running on Blacklight version 7.
• The basic mapping between the SOLR index and the Search interface can be found in: controllers/catalog_controller.rb

Back-end

• Custom rails active records application for record management.

We have a single records table called admin, defined in app/models/admin.rb

• All records are stored in the postgres database. On create/update/delete of a record it is immediately indexed in SOLR.
• It's also possible to reindex all records from the database.

Database diagram

!searchblocks_schema 2.png Definition in db/schema.rb

Record management

• admins
  • keywords, creators and searchblocks are stored as a json string.
• versions   - admins versioning using the  paper trail gem

Blacklight

• bookmarks Bookmarks created by visitors
• searches Searches stored by visitors

Rails

• ar_internal_metadata
• users
• schema_migrations

SOLR index

For the config see solr_config.xml and schema.xml in the source

The solr records are basically the same as the database records. The "also see" id numbers are stored as . See this code snippet from models/admin.rb:

...
    keyword_sm = JSON::parse(admin.keywords)
    names_sm = JSON::parse(admin.creators)
    also_sm = []

    unless admin.also.nil?
      also_ids = JSON::parse(admin.also)
      also_sm = []
      # (also store the id??)
      also_ids.each do |id|
        also_sm.push(Admin.find(id).title)
      end
    end

	@@solr.add :title_s => admin.title, :keyword_sm => keyword_sm, :names_sm => names_sm, :also_sm => also_sm, :notes_s => admin.notes, :searchblock_s => admin.searchblocks, :date_s => admin.creationdate.to_s[0, 10], :date_dt => admin.creationdate, :id => admin.id
    @@solr.commit

  end
...

Note there are a few helper functions in https://github.com/peer35/searchblocks/blob/master/app/helpers/application_helper.rb to correctly format the searchblocks in the front end.

Functional flow

Every visitor can register as a user on the site, this allows only the standard blacklight functionality: storing bookmarks and searches.

By adding the email address to a config file config/initializers/admins.rb the user gains privileges to access the backend. Make sure to add this file to the source code before you build the docker container:

Rails.application.config.x.admin_users_email=['someuser@gmail.com','otheruser@vu.nl']

• Edit and delete links are shown next to the record:

• Clicking Edit takes you to the form:

• Blocks can be edited, added or removed:

• Other multi-valued fields are implemented as a special select box:

Versions are shown at the bottom of the page:

"Open version" shows the form with the previous values, click submit to store it.

The overview page https://blocks.bmi-online.nl/admins/ shows recently changed and deleted records.

Open https://blocks.bmi-online.nl/admins/new to start a new record

Note the form is implemented in https://github.com/peer35/searchblocks/tree/master/app/views/admins and https://github.com/peer35/searchblocks/blob/master/app/controllers/admins_controller.rb