update retl bigquery setup with storage-api #7945
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
stayseesong
reviewed
Sep 18, 2025
Comment on lines
+84
to
+88
| When setting up a BigQuery Reverse ETL source, you can choose which API Segment uses to read from BigQuery. You can make this selection during the initial setup, or later from Sources > Reverse ETL > BigQuery Source > Settings > Connection Settings. | ||
| - **REST API** | ||
| Recommended for most tables. REST is generally more cost-efficient, but syncs may be slower for very large datasets. | ||
| - **Storage API** | ||
| Recommended for large tables. The Storage API can significantly improve throughput and reduce sync times, but may incur higher costs. See [BigQuery Storage API client libraries](https://cloud.google.com/bigquery/docs/reference/storage/libraries) and [BigQuery pricing for data extraction](https://cloud.google.com/bigquery/pricing?hl=en#data_extraction_pricing) for more information |
There was a problem hiding this comment.
@gilsegment is there a reason why you want this info added after step 10? the placement of this info is very weird as step 10 is about the data location, unless you're saying that choosing which API Segment uses is related to the data location?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ticket
https://twilio-engineering.atlassian.net/browse/CLAS-1069
Proposed changes
Updating Reverse-ETL Setup with BigQuery Storage-API
Merge timing
TBD