Janky Flask Personal Website™

Setup if I brick another machine

1. Set up Docker, MySQL, and Nginx on host machine
   • MySQL data directory should be in the bind-mounted directory specified in deployment/docker/compose.yaml
     • Recover database data from backups
   • Make sure default key for SSH and for GitHub pushing has no passcode if planning to use automatic db/image backup scripts. No hack pls
2. git clone
   • Make sure all files and folders in the entire project are owned by the user that Docker runs its containers as (see deployment/systemd_reference/personal_website.service)
3. Install packages:
   • Install Python modules from requirements.txt by running pip install -r requirements.txt (ideally within a virtualenv)
   • Install JS modules from app/static/package.json by running npm install in the app/static/ directory
4. Add back gitignored files:
   • .env: randomly generated SECRET_KEY and SQLAlchemy DATABASE_URL for connecting to MySQL from host
     • DATABASE_URL should be something like mysql+pymysql://[db username]:[db password]@[hostname]:[port]/[database name]?charset=utf8mb4
   • deployment/docker/flask/envs/.env: same as .env but with DATABASE_URL modified to connect to the MySQL Docker container (i.e. the hostname part of the URL is the name of the MySQL container service in deployment/docker/compose.yaml, in this case mysql)
   • deployment/docker/mysql/envs/.mysqlenv: nothing yet (no environment variables if bind-mounting existing MySQL data directory)
   • deployment/backup_scripts/db_backup_config.sh: set the variables referenced in deployment/backup_scripts/db_backup.sh
   • app/static/css/custom_bootstrap.css and app/static/css/custom_bootstrap.css.map: run npm compile_bootstrap from within the app/static/ folder
5. Navigate to deployment/docker/ and run deploy.sh (or use a systemd service, for example deployment/systemd_reference/personal_website.service)

Developer notes to compensate for possibly scuffed code

IMPORTANT:

Keep up-to-date:

• .gitignore
• config.py
• Server-side access control
• README
• Backup scripts in deployment/backup_scripts/
• Cloudflare WAF rules etc.

Deployment maintenance:

• Sync/keep up-to-date according to comments and common sense:
  • deployment/docker/compose.yaml
  • Dockerfiles
  • Docker entrypoint scripts
  • Docker environment variables
  • deployment/backup_scripts/db_backup_config.sh configs
  • Backup scripts
  • systemd services
• To connect to the MySQL instance running in Docker from the host:
  • Make sure the MySQL port (default 3306) is exposed from Docker and there is a .env file on the host with DATABASE_URL pointing to localhost
  • Use mysql --protocol=tcp to connect so it doesn't try to use a Unix socket; make sure to use the MySQL user that has % as its host (because that means it can connect from any host, whereas localhost would mean that it can only connect from within the Docker container)
• To edit database schema:
  • Edit app/models.py on the host
    • IMPORTANT: currently MySQL-specific!!!
  • Run flask db migrate on the host in the Python venv; this requires MySQL connectivity from the host
  • CHECK MIGRATION SCRIPT IN migrations/versions/!!!

    • If renaming columns, you will probably have to edit the Alembic script in migrations/versions/ to use alter_column()! existing_type is a required argument:

      batch_op.alter_column(column_name='[]', new_column_name='[]', existing_type=[])
      

      Reference migrations/versions/79665802aa08_rename_blogpage_title_and_subtitle_to_.py for examples.

    • If changing unique constraint, you will need

      batch_op.create_unique_constraint("[constraint_name]", ['[column_name]'])
      

      and

      batch_op.drop_constraint("[constraint_name]", type_='unique')
      

  • Run flask db upgrade on the host in the Python venv or restart the Docker containers

Access control notes:

• Assume the user can reach all endpoints, so access-control must be perfect server-side
  • Use the functions defined in app/utils.py for access control
• It doesn't matter as much if client-side is lax on updating hidden HTML links etc. on session expiry. This is good because my client-side is an absolute dumpster fire.

Adding new blogpages:

• Add to database (reference current database entries)
  • Add a developer/backrooms blogpage too with its blogpage_id being the negative of the public one
  • blogpage_id is always an integer except for the commented cases in config.py, where they must be strings to avoid confusion with negative values and list/dictionary accessing
• Update config.py:
  • Update BLOGPAGE_ID_TO_PATH with the same paths that you gave the new blogpage and its developer blogpage in the database; this is used for blueprint initialization (we can't access database before app context is fully created)
  • Update URLS_LOGIN_REQUIRED with the backrooms blogpage
• Create new static directories for it in app/blog/static/blogpage/ from the template, and update other static directory names if necessary
  • Remember that since HTML templates are the same for every blogpage, things like font or background image customizations must be done through static files like CSS, which are imported individually per blogpage
  • If overriding default background image, change backgroundImgOverrideName in a file app/blog/static/blogpage/[blueprint]/js/override_background_img.js

Changing blogpage IDs/blogpage static paths:

• Change the static directories, obviously
• Update paths in config.py
• Update Markdown expansion/collapse regex in app/models.py
• Update image paths in app/admin/routes.py. This is important to make sure we don't accidentally delete/move important files!
• Update static paths for all linked JS/CSS in templates
• Update image paths for all existing images in db

Adding new forms:

• GET forms:
  • These should not modify server-side state!
  • Usage guidelines:
    • Do not implement a CSRF Token hidden field to avoid leaking token in the URL (per OWASP guidelines). This means that we shouldn't use the boostrap_wtf.quick_form() macro for GET forms!
  • Refer to app/blog/static/blogpage/js/goto_page_form.js and its associated app/blog/templates/blog/blogpage/index.html for an example of a GET form
• POST forms:
  • All other forms
  • Usage guidelines:
    • Must be Ajax, using fetchWrapper() in app/static/js/util_ajax.js and sending FormData (since the CSRF error handling is designed only for FormData). See doAjaxResponseBase() in the same file for documentation on the basic, always-supported JSON keys that the backend can return.
  • Refer to app/static/js/util_session.js, app/static/js/main_form_submit.js, and app/blog/static/blogpage/js/post_comments.js for examples of POST forms
• Always add HTML classes auth-true/auth-false (for showing/hiding elements) when needed

Updating HTML custom errors:

• Update CUSTOM_ERRORS in config.py
• Update fetchWrapper() in app/static/js/util_ajax.js
• Update app/routes.py error handlers if necessary

Other notes:

• url_for() to a blueprint (trusted destination!) should always be used with _external=True in both HTML templates and Flask to simplify the cross-origin nature of having a blog subdomain
• Try not to modify any of the forms.pys, as some JS might rely on hardcoded values of the form fields. I don't do frontend, ok?

Blog writer notes

Markdown syntax and custom syntax:

• Make sure to check out the documentation for Python-Markdown's official extensions
  • Check code in app/blog/blogpage/routes.py to see which ones are used
  • Attribute Lists allows you to take advantage of the many util CSS classes in app/static/css/util.css
• Custom Markdown syntax:
  • Check source code for detailed documentation and usages
  • Inline:
    • ~~<text>~~: strikethrough TODO: update once finalized
  • Blocks (all delimiters must be surrounded by a blank line on both sides; not allowed in comments due to potential bugs):
    • \begin{<block type>} and end{<block type>}, surrounded by a blank line on both sides, puts everything in between in the specified <block type>
    • Available <block type>s:
      • captioned_figure: a figure with a caption underneath
        • Requires nested caption block inside
      • cited_blockquote: a blockquote with a citation underneath
        • Requires nested citation block inside
      • dropdown: an expandable/collapsible dropdown
        • Alternative <block type>s:
          • exer: exercise
          • pf: proof
          • rmk: remark
        • Requires nested summary block inside, except for the following <block type>s that get defaults:
          • pf
          • rmk
      • textbox: a textbox
        • Alternative <block type>s:
          • coro: corollary
          • defn: definition
          • impt: important
          • notat: notation
          • prop: proposition
          • thm: theorem
  • Images:
    • Only give the filename for images in Markdown; the full path will be automatically expanded (won't work if you put in full path because I'm bad at regex!!!)

Other syntax notes:

• Raw HTML (including with attributes!) will be rendered, which is useful for additional styling or in environments where Markdown equivalents may not always work (footnotes, tables, blockquotes etc.). Examples:
  • <span></span> with pretty much any custom CSS styling you want (or with existing styling classes, once CSP is able to block inline style attributes)
  • <pre><code></code></pre> with <br> newlines for multiline code blocks in a table, as raw newlines would interfere with the table syntax
  • <small></small> for small text
  • <p></p> for paragraphs and line breaks (note: not supported in footnotes; use <br><br> instead)
    • E.g. lists, which have had the space between it and the previous paragraph removed by default
  • <br> for line breaks that aren't new paragraphs and don't leave extra space, like between lines in a stanza, and <br> surrounded by two empty lines for more space than a normal paragraph, like between stanzas

Debug:

• Use debugTestSelfLinks() in the browser console to test for dead self-links on the current page

Tables:

• Uses Markdown tables with "Compact mode" and "Line breaks as <br>" checked
• For merged cells, use the Attribute Lists extension to set colspan. To keep valid table syntax, put <span></span> {: hidden } in cells that have been merged into other ones.
• To specify column width attributes (in HTML, not CSS) for example with Attribute Lists, either specify in pixels or percentages. Pixels are absolute while percentages are relative to the width of the table. If percentages are used, or if no width specified at all, table will have min-width: 100% of parent div.

Cookie explanation from empirical observations and devtools

Comparing Flask's built-in session cookie with PERMANENT_SESSION_LIFETIME config vs. Flask-Login's remember me cookie with REMEMBER_COOKIE_DURATION config:

• session.permanent does not actually affect if a cookie is invalidated by PERMANENT_SESSION_LIFETIME; cookies will always adhere to this lifetime (including the non-signed-in, default cookie for storing Flask's session): session.permanent=False means the session cookie is invalidated by Flask but not deleted when this lifetime is up, while session.permanent=True actually gives it an expiration time.
• remember from Flask-Login only affects how the cookies are handled when the browser is closed (although it seems many browsers nowadays will persist even session (non-remembered) cookies as well on close).

	Session cookie stored in:	Remember cookie stored in:	PERMANENT_SESSION_LIFETIME effect on session cookie	REMEMBER_COOKIE_DURATION effect on remember cookie	User experience when PERMANENT_SESSION_LIFETIME reached	User experience when REMEMBER_COOKIE_DURATION reached
session.permanent=False, remember=False	Memory (non-persistent)	-	Invalidated by Flask (docs)	-	Logged out	-
session.permanent=False, remember=True	Memory (non-persistent)	Disk (persistent)	Invalidated by Flask	Expires & is deleted	Logged out	Logged out if browser closed
session.permanent=True, remember=False	Disk (persistent)	-	Expires & is deleted	-	Logged out	-
session.permanent=True, remember=True	Disk (persistent)	Disk (persistent)	Expires & is deleted	Expires & is deleted	Logged out	Logged out if browser closed