fetch_and_import_range_from_sequence log exceptions instead of raise
Fixes #1234
Raising the exception will kill all Tasks, the sequence and the fetcher
because of the linking from Task.async_stream. This leads to when 1
Task times out in a DB transaction, they all are affected, leading to
slowdowns when the sequence has to be rebuilt when the fetcher restarts.#5
Conversation
Ecto 3.0 gets us easy access to partitions and the windowing function
`lag`. `lag` is able to keep track of the previous row without a self
join. Importantly, the new query with `lag` and no more
`generate_series` completes in ~2.5 seconds on Ethereum mainnet
using `:timer.tc(Explorer.Chain, :missing_block_number_ranges, [6889126..0])`
on ETH Mainnet Test DB 40 while the old version doesn't return after 5+
minutes.
I've heavily commented the new version of the query since this took
awhile to figure out and most importantly finding the proper terminology:
"gaps and islands" was key to find the starting query using `lag`.
The query in SQL is as follows:
```sql
SELECT island.last_number + 1,
island.next_number - 1
FROM (
SELECT lag(land.number) OVER "w" AS last_number,
land.number AS next_number
FROM (
SELECT blocks.number AS number
FROM blocks
WHERE blocks.consensus = TRUE
UNION ALL (
SELECT LEAST(min_blocks.number, before_range_min)
FROM (
SELECT min(blocks.number) AS number
FROM blocks
WHERE blocks.consensus = TRUE)
) AS min_blocks
WHERE min_blocks.number IS NULL OR
min_blocks.number != before_range_min
)
UNION ALL (
SELECT GREATEST(max_blocks.number, after_range_max)
FROM (
SELECT max(blocks.number) AS number
FROM blocks
WHERE blocks.consensus = TRUE
) AS max_blocks
WHERE max_blocks.number IS NULL OR
max_blocks.number != after_range_max
) AS land
WINDOW "w" AS (ORDER BY land.number)
) AS islands
WHERE islands.last_number != islands.next_number - 1
ORDER BY islands.last_number
```
Fixes blockscout#1232 `{:error, :timeout}` should have been caught, but additionally, it should never have been allowed to propagate out unassigned to a step as it is not possible to track the timeout to a specific fetch.
Fixes blockscout#1234 Raising the exception will kill all Tasks, the sequence and the fetcher because of the linking from `Task.async_stream`. This leads to when 1 Task times out in a DB transaction, they all are affected, leading to slowdowns when the sequence has to be rebuilt when the fetcher restarts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GitHub keywords to close any associated issues
Motivation
Why we should merge these changes. If using GitHub keywords to close issues, this is optional as the motivation can be read on the issue page.
Changelog
Enhancements
Things you added that don't break anything. Regression tests for Bug Fixes count as Enhancements.
Bug Fixes
Things you changed that fix bugs. If it fixes a bug but, in so doing, adds a new requirement, removes code, or requires a database reset and reindex, the breaking part of the change should also be added to "Incompatible Changes" below.
Incompatible Changes
Things you broke while doing Enhancements and Bug Fixes. Breaking changes include (1) adding new requirements and (2) removing code. Renaming counts as (2) because a rename is a removal followed by an add.
Upgrading
If you have any Incompatible Changes in the above Changelog, outline how users of prior versions can upgrade once this PR lands or when reviewers are testing locally. A common upgrading step is "Database reset and re-index required".
Checklist for your Pull Request (PR)
master.