Report sync and bugfixes (#13) * Fix metrics field exclusion bug * Enable report discovery and sync * Remove the prefix from fields names in the resource schema and json schema * Fix bug caused by shared references * Use a tuple literal * Add table-foreign-key-properties to stream metadata * Add function to create nested JSON schema for core streams * Add tests for `create_nested_resource_schema` * Refactor function to allow reuse with reports * Fix tests after refactor * Working sync for core streams * Fix field exclusion and metadata bugs * Add ReportStream class, update PK names * Remove flatten because schema is no longer flat * Fix primary key bug * Require date as a segment * Fix ads stream bugs * Add remapping for ads in report * WIP Incremental sync for report streams * Make state global * Add hash of attributes and segments as PK for report streams * Change reports to use the ReportStream class * Change adgroupperformancereport to use report stream, don't write state as often * Make pylint happy * Change fields-to-sync to tap-google-ads.api-field-names Co-authored-by: leslie vandemark lvandemark@stitchdata.com * Fix record hash and add tests for record hash * Make inclusion automatic on foreign keys * Add docstring to create_resource_schema * Change metadate fields_to_sync to tap-google-ads.api-field-names, don't look for api-field-names on record hash * Add conversion window, Don't store the whole response in memory since paging happens automatically * Enable `campaign_criteria_report` * Fix field exclusion bug where one resource has the other in its `selectable_with` list but not the other way around * Change multiple resource reports to use one resource for alpha * Change core sync to not load whole response in memory * Change query range to 1 day, Add logs, begin refactoring * Separate functionality into separate files * Remove unused functions and variables * WIP discovery refactor * Move Required Config to init * remove reports.py after rename * Remove unused import * Qa/reports 1 (#14) * create spikes folder for discovery and add protobuff to setup * add more fields to campaign schema * Attempt at getting some kind of auto schema * Clean up, added comments to links where to look up missing classes, refactored out giant elif blocks * use refs * Add note to inspect for enum types * WIP on handling enums * updated disco test * wip syncing report streams * Manger Segment BUG squashed * wip on canary * updated discovery and documented discrepancies within test * auto fields bug mostly squarshed * WIP canarying all streams * sync canary passing with partial coverage * Updates to dicovery, start date, and canary * start date test wip Co-authored-by: dylan-stitch <28106103+dylan-stitch@users.noreply.github.com> Co-authored-by: Dan Mosora <30501696+dmosorast@users.noreply.github.com> Co-authored-by: Bryant Gray Co-authored-by: kspeer * Use the pylint disable env var (#15) Co-authored-by: dylan-stitch <28106103+dylan-stitch@users.noreply.github.com> * Change single quotes to double quotes * Remove `format_field_names` We made this change to get a simple set of unit tests passing before we refactor all of this discovery code. There seems to be a bug in the current implementation of `format_field_names` * Ensure segmenting resources are accounted for in metadata * Pulled `format_field_names` into ReportStream class * Pulled `format_field_names` into BaseStream class * Refactor WIP; complete format_field_name moves in both base and report classes * Refactor WIP; move metadata creation to base and report classes * Refactor WIP: Complete discovery refactoring * Make pylint as happy as possible knowing it will never be pleased * First performance report update * Remove extra category assignment We ran tests locally and found that every `field_schema` has a category of segment already. And because we weren't setting the category for attributes and metrics as well, this code is just redundant. * disco test update, auto fields test added * Ran black * Add user_view_performance_report * Update config with parallelized jobs and steps * trust the workspace attach to persist a file * just copy env vars where needed * testing a test counter * Remove multiple-resource related code (#16) * Use `singer.utils.should_sync_field` to filter for selected fields * Add the user_view_performance_report to tests * Write the json schema for date fields as datetimes * Compare datetime objects instead of strings (#17) Co-authored-by: dylan-stitch <28106103+dylan-stitch@users.noreply.github.com> * updates to auto, disoc, start tests * Add workarounds to get tests passing * Sort excluded streams * Update user_view to user_location_performation_report * auto field test passing with workaround * Adds failing tests for metadata bugs. Add failing tests for TDL-17848 and TDL-17845 * Fix failing metadata tests Adds forced replication method and valid replication key. * Ensure transformer filters out non-selected fields * Update test expectations to allow null foreign keys * tests partial refactor for selecting streams/fields, add first report to bookmarks test * Update reports test to check each report for automatic fields * Raise exception when report only has automatic fields selected * Clean up imports * increased test coverage for start date * updates to auto fields test, simplify field selection in start date * fix automatic fields for core stream foreign keys * WIP start date test split out streams to test * Finalized start date test for Alpha * fix the parallelism 9->6 * just install required dependencies * don't count test * fix pylint with variable reference * Install pylint * Show the tap installed correctly * Revert "Show the tap installed correctly" This reverts commit 5b066f4acd97c228278417bde5ab91a6f46a21e8. * Revert "Install pylint" This reverts commit 06d196920b789fd50a67226b4c003cc4670dcecf. * Install tap-google-ads without `-e`, install dev dependencies * Bugs squashed: 17839, 17827 * Bug squashed: TDL-17840 * Failing assertions for TDL-17887 * Regroup functions for readability, rename sync_* to sync, set currently syncing in core streams * Revert to conversion_window per discussed naming conventions * Clear currently_syncing after stream completion * Fix bug TDL-17887, whitespace clean up * Fix import for create_nested_resource_schema * Remove tests for deleted functions * Update import for generate_hash * Add get_query_date, add unit tests for get_query_date * Add unit tests to circle runs * Finish refactor - call the new function name * Fix imports in unit test * Fix bug in refactor: update arg name * Remove default value in get_bookmark * Pass in a datetime string to get_query_date * Return datetime objects from get_query_date * Clean up TODO * Fix unit tests * Bookmark updates, test more streams, remove lookback from start_date * minor cleanup bookmarks test * Delete campaigns.json * Delete schema_gen_protobuf.py * Test Cleanup * comments * fix typo in disco test * fix typo in discovery test Co-authored-by: Bryant Gray Co-authored-by: dylan-stitch <28106103+dylan-stitch@users.noreply.github.com> Co-authored-by: Andy Lu Co-authored-by: Dan Mosora <30501696+dmosorast@users.noreply.github.com> Co-authored-by: kspeer Co-authored-by: btowles Co-authored-by: Andy Lu

Add a license #17

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions