Skip to content

Stop using the vae_index#1334

Merged
dwwoelfel merged 7 commits intomainfrom
stop-using-index
Jul 7, 2025
Merged

Stop using the vae_index#1334
dwwoelfel merged 7 commits intomainfrom
stop-using-index

Conversation

@dwwoelfel
Copy link
Contributor

@dwwoelfel dwwoelfel commented Jul 3, 2025

Follow-up to #1326 and prerequisite for #1330

Modifies some of the datalog queries to replace usage of the vae_index with the new vae_uuid_index. Once we're no longer using the vae_index at all, we can drop it and save ourselves some disk space and cpu on the db instance.

Three main changes:

  1. Prefer the :vae index over the :av index in attr-pat when both are applicable (i.e. querying a unique ref attr). This saves us converting an entity_id to JSON in a lot of cases--we can just directly compare the entity_id to json_uuid_to_uuid(value), which requires no calculation because it's already in the vae_uuid_index.
  2. Update the join-cols to use the json_uuid_to_uuid(value) = entity_id instead of value = to_jsonb(entity_id) for better use of the vae index.
  3. Update in-or-eq to use [:json_uuid_to_uuid :value] when it encounters the vae index, instead of using [:to_jsonb :value]

The structure of the symbol map changed. Instead of being a two-element array, it's now a map with some extra details. [0 1] => {:pattern-idx 0 :triples-idx 1, :ctype :a}. I did that so we could pass down whether the value was a ref into the join cols. It also has the benefit of making it easier to write the join-cond-for-or. The ors need to fully qualify the

Minor changes:

  1. Renamed :value-blob to just :value. It's no longer needed now that we stopped tracking the :value-uuid and makes the code a bit cleaner.
  2. Replaced the :match-0- prefix with just :m. It makes the queries easier to read and it's less data to and from the db.

Did a backtest and didn't find any queries that got faster or slower.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 4, 2025

View Vercel preview at instant-www-js-stop-using-index-jsv.vercel.app.

@dwwoelfel dwwoelfel changed the title [wip] Stop using the vae_index Stop using the vae_index Jul 7, 2025
@dwwoelfel dwwoelfel marked this pull request as ready for review July 7, 2025 17:41
Copy link
Contributor

@stopachka stopachka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Biyutiful!

(testing "join conditions matches values to coerced entities"
(is (= '(([:= :value [:to_jsonb :match-0-entity-id]]))
(raw-pats->join-conds '[[:ea ?a] [:eav _ _ ?a]]))))
(testing "join conditions matches values to coerced attrs"
Copy link
Contributor Author

@dwwoelfel dwwoelfel Jul 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the test cases where we compare on attr ids because we never actually compare attr ids to anything.

@dwwoelfel dwwoelfel merged commit acf6f9d into main Jul 7, 2025
33 checks passed
@dwwoelfel dwwoelfel deleted the stop-using-index branch July 7, 2025 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants