Avoid unnecessary CMS DB hits for pages that will 404

[stevejalim]

This changeset adds a lookahead check before we call wagtail.views.serve to know whether it's worth asking the CMS to serve a page.

The idea is that if the page isn't in the lookahead's data source, we can avoid touching the DB just to ultimately return a 404 earlier.

This will save us DB load, particularly when we get drive-by scans that pepper the site with irrelevant URLs.

• I used an AI to write some of this code - I bounced some ideas around with ChatGPT, and reworked some of the code suggestions to fit with Bedrock

Significant changes and points to review

Please be sceptical about this, particularly around cache invalidation/cache updating - e.g. when a page is published, unpublished or moved in the page tree.

Important: this change will need to go hand in hand with an infra update that does give us a networked cache (Redis or Memcached) - if we stick with LocMemCache, then while the pods can and will build their own lookahead in local cache, that cache will not be invalidated when a page is published, unpublished or moved.

Issue / Bugzilla link

#15505 #14742

Testing

Details to come

[codecov]

Codecov Report

Attention: Patch coverage is 80.41237% with 19 lines in your changes missing coverage. Please review.

Project coverage is 79.30%. Comparing base (aa2f6a0) to head (2af4211).

Files with missing lines	Patch %	Lines
bedrock/cms/signal_handlers.py	51.35%	18 Missing ⚠️
bedrock/cms/apps.py	87.50%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #15628      +/-   ##
==========================================
+ Coverage   79.29%   79.30%   +0.01%     
==========================================
  Files         159      161       +2     
  Lines        8347     8443      +96     
==========================================
+ Hits         6619     6696      +77     
- Misses       1728     1747      +19     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[stevejalim]

Talking with Pmac, we're gonna try using postgres as a DB-backed networked cache, possibly/probably with a locmem cache on each pod.

[stevejalim]

So the whole db-based cache thing didn't work out, because getting from the cache also triggers the invalidation check, which then can result in an error when called on a readonly postgres DB (which is the situation for the web pods).

Instead, new-new plan:

• simple new model that holds the latest tree info in the the DB - available to web and CMS pods.
• web pods will only ever read from the DB table and cache the info in their locmem caches, so no networked-db-cache-invalidation pain should happen.
• we'll retain the signal-based approach to updating the tree info when CMS changes occur, and put it in the DB table instead.
• we'll add a cron job to keep that tree up to date, just in case we have strangeness around the signals

[maribedran]

@stevejalim This makes sense to me. The only situation I can think of right now that could break this is if we ever decide to use the RoutablePageMixin: https://docs.wagtail.org/en/stable/reference/contrib/routablepage.html#routable-page-mixin

Maybe we could check if that app is installed and, if a page inherits from that class, we never raise a 404 for paths that start with that page's URL.

[stevejalim]

@stevejalim This makes sense to me. The only situation I can think of right now that could break this is if we ever decide to use the RoutablePageMixin: https://docs.wagtail.org/en/stable/reference/contrib/routablepage.html#routable-page-mixin

Yeah, I remember discussing this with someone else and we agreed routable pages would be a problem with this approach - but because we (currently) don't use them, it wasn't a big concern.

Maybe we could check if that app is installed and, if a page inherits from that class, we never raise a 404 for paths that start with that page's URL.

I'm thinking along similar lines - there must be a way to introspect the page and get all the routes that defined via the @path decorator. I'll have a rummage through the Wagtail source when I get a moment