Fix TikTok artifacts SQL parsing and add robust error handling#1396
Open
AdityaS207 wants to merge 2 commits intoabrignoni:mainfrom
Open
Fix TikTok artifacts SQL parsing and add robust error handling#1396AdityaS207 wants to merge 2 commits intoabrignoni:mainfrom
AdityaS207 wants to merge 2 commits intoabrignoni:mainfrom
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This pull request fixes runtime failures when parsing TikTok artifacts by ensuring constructed SQL is syntactically valid and by adding robust exception handling around SQLite operations. The change prevents artifact crashes caused by missing contact tables or schema differences across TikTok database samples.
Previously observed errors
sqlite3.OperationalError: near ")": syntax error (caused by empty/invalid subquery e.g., LEFT JOIN ())
sqlite3.OperationalError: no such table: TIMMessageKVORM
These raised unhandled exceptions that stopped artifact processing and sometimes terminated the overall run.
Changes Included
1. tikTok Artifact
Use ' UNION ALL '.join(...) when assembling contact subqueries so multiple AwemeContactsV% tables are combined correctly.
Add a safe fallback subquery when no contact tables are found (a valid SELECT that returns zero rows) to prevent LEFT JOIN () syntax errors.
Wrap main SQL executions in try/except sqlite3.OperationalError and log errors via logfunc(...) rather than letting exceptions crash the parser.
2. tikTokReplied Artifact
General improvements
Standardized, non-intrusive error handling for SQLite operations: Error messages now follow the pattern: Reading had SQL error:
All fixes preserve original parsing logic and output formats where possible; changes prioritize stability over failing the entire run.
Testing Performed
1. Local verification:
ran ileapp.py with the provided profile limiting to tikTok and tiktok_replied modules (example command used during testing was the same command the user runs with 13-4-1.tar).
2. Observed behavior:
Prior crashes caused by malformed subqueries or missing tables no longer abort artifact processing.
When contact tables exist, artifacts still return results as before.
When contact tables are absent or queries fail due to schema differences, the artifacts log the SQL error and continue; the run finishes and report generation completes.
Results
1. No unhandled sqlite3 OperationalErrors for the tikTok and tiktok_replied artifacts in tested samples.
2. Artifacts are resilient to:
missing AwemeContactsV% tables,
missing TIMMessageKVORM table,
SQL differences that would previously crash the module.
3. Report generation completes without interruption; missing data is handled gracefully with logged diagnostics.
Notes / Caveats
The fallback subquery intentionally returns zero rows. That means contact-related columns will be empty when contact tables are genuinely absent; this is a deliberate tradeoff to maintain run continuity.
If the SQLite build used in a DB does not support window functions (e.g., ROW_NUMBER() in the CTE), the try/except will catch the error and log it. If preferred, we can provide a compatibility fallback (group-by/aggregate-based deduplication) instead of relying on CTE/window functions.
These changes are defensive and do not add new external dependencies.