:tada: New Source: Salesforce Marketing Cloud (Singer) by schlattk · Pull Request #9080 · airbytehq/airbyte

Konrad Schlatte (schlattk) · 2021-12-23T11:08:53Z

What

adding a new source connector for salesforce marketing cloud (tap-exacttarget)

How

using a singer tap

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

This change is

CLAassistant · 2021-12-23T11:09:08Z

All committers have signed the CLA.

Marcos Marx (marcosmarxm) · 2021-12-23T19:04:20Z

thanks for this contribution Konrad Schlatte (@schlattk). I'm waiting #4933 to get credentials and test your contribution.

…irbytehq#8294) * SAT: Add assert that output catalog does not have any in it * SAT: update code after review, add test parametrizes

…rbytehq#7783)

…st one data point (airbytehq#8272) * Implement vlidatation if each field in a stream has appeared it least once in some record. * Add unit tests for `_validate_empty_streams` TestBasicRead method. * Add validate_data_points basic read input option. * Update `_validate_field_appears_at_least_once_in_stream` to support oneOf/anyOf choices. Add oneOf/anyOf unit tests. * Bump docker version. Update CHANGELOG.md * Fix test_core.py imports.

…ehq#8342)

…q#8329) * ignored fields added * remove 'skip_dynamic_fields' Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

…8354) * Change synchronisationProgress to synchronizationProgress * Change locale key

…irbytehq#8140) * make spec field required * remove spec backfill logic * remove usages of specFetcher.getSpec() * remove unused code and the caching scheduler clients * fix tests to work with fetching specs from definitions * fetch spec from definition in config repository and fix RunMigrationTest * remove unused SpecFetcher methods/tests * run gw format * run gw format * undo change to main method * add back newlines * set additional properties to true on destination definition * remove now-unused VERSION_0_32_0_FORCE_UPGRADE env var

…iant date string (airbytehq#5814) * added format_reference.txt, edited edge_case_messages.txt, edge_case_catalog.json Co-authored-by: Yurii Bidiuk <yura.bidyuk@gmail.com>

* 'Boards' stream returns only specific (board_ids) records * github secret -> google secret Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* Remove unique key on exploded nested tables * un-nest hint * Regen SQL

* allow retrieving deleted sources and destinations * fix test

Co-authored-by: sherifnada <sherifnada@users.noreply.github.com>

…airbytehq#8381)

Co-authored-by: jrhizor <jrhizor@users.noreply.github.com>

* Fix incremental with empty final tables * upgrade docker images * Regen SQL * Bumpversion & format

…ally_large_values (airbytehq#8264) * Commented incremental tests to skip them in SAT * limit the slice range with end_date * added unit tests for stream_slices * changed the connector version * changed reason of the comment yml * use freezgun in unit tests * formatted code * updated version in spec yaml Co-authored-by: Auganbay <auganenu@gmail.com>

* Make all these public so Cloud can reuse these helpers. * Reorganise the env var page for better readability. * Format. * Reorganise the EnvConfigs class.

* get date-time format form json schema * created universal date-time converter * implemented jsonnode transformation for avro and parquet * removed unneeded dependency from build.gradle * fix checkstyle * add DateTimeUtilsTest * add AvroRecordHelperTest * resolve merge conflicts | fix checkstyle * update LocalTime parsing * added String type to avro schema for Logical Types, removed date-time conversion * fix checkstyle * fix checkstyle * added static String schema, added comments * bump version * Bump GCS version with avro/parquet timestamp conversion * update docs * update docs Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>

) Co-authored-by: ChristopheDuong <ChristopheDuong@users.noreply.github.com>

airbytehq#7913)

…sts (airbytehq#8840) * fix for jdk 17 * Destination MariaDbColumnstore updated Docker image * update Docker image for ssh tests Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>

…irbytehq#8699) * Add incremental sync support for companies, contact_lists, contacts, deals, line_items, products, quotes, tickets streams * Updated PR number * Fix formating * Fix typo * Commented unsupported tests * Updated to review * Updated abnormal state file * Deleted comment * Updated version in docker * Updated version in spec yaml and seed

…ytehq#8742) * fix schema for 'breakdowns' and 'action_breakdowns fields Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* fix logs slicing * add unit test * add more tests * bumb version * send last line if there are not trackback or container error

Signed-off-by: Serhii Chvaliuk <grubberr@gmail.com>

* Prepend all workspaces resources with /workspaces

… data (airbytehq#8855)

* update changelog files * remove mentions to oauth

* feat(zendesk): Add Brands and CustomRoles * feat(zendesk): add incremental unsorted cursor stream implement IncrementalUnsortedCursorStream to ticket_metrics * feat(zendesk): use sorted cursor pagination for ticket comments and macros * feat(zendesk): use unsorted cursor stream for groups, group memberships and satisfaction ratings * fix(zendesk): use safe method to get value from nested dict * style(zendesk): reformat using gradlew * fix(zendesk): format created_at and updated_at to date-time format * feat(zendesk): add business hours schedule * bump connector version * bump dockerfile version Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>

…data. (airbytehq#8649) * feat(zendesk): Add Brands and CustomRoles * feat(zendesk): add incremental unsorted cursor stream implement IncrementalUnsortedCursorStream to ticket_metrics * feat(zendesk): use sorted cursor pagination for ticket comments and macros * feat(zendesk): use unsorted cursor stream for groups, group memberships and satisfaction ratings * fix(zendesk): use safe method to get value from nested dict * style(zendesk): reformat using gradlew * fix(zendesk): format created_at and updated_at to date-time format * feat(zendesk): add business hours schedule * bump connector version * bump dockerfile version * reset * resolve webapp files Co-authored-by: asyarif93 <asyarif93@gmail.com> Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>

* it's working * clean up process factory * fix and add tests * clarify airbyte_default * fix build * fix kube acceptance test (maybe) * oops * fix output prop issue * fix output propagation regression * fix kube singleton problem * sync passing on kube but getting wrong exit code of 7 * misc * fix port usage * remove host port that causes conflicts * eliminate envconfigs static usage * this took way too long to figure out * get rid of annoying ==== this is new log messages * finally successfully completing syncs * stop using magic strings and clean up logging * misc minor cleanups * fmt * misc * correct * misc fixes * rename + misc * better logs * logging fix 1 * logging fix 1 -- fixed * finally get logging working nicely * add comment for simplification * fmt * misc * fmt * break into separate class * remove comment * remove flaky multi-node testing * try to fix connector build * remove separate node check * switch to new configs type * fix regression from logging config changes * only log path one time * remove misleading setting terminology * fix connector build * fix earlier merge conflict * fix runtime kubernetes bug * fix connector build (again) * greatly simplify logging config by forcing the container-orchestrator to use default (non-job) logging * add secret insertion for orchestrator on kube * fix k8s ports * add four ports * fix logging test regression * temporarily disable kube tests to check logging * improve comments * make Docker run more secure by limiting env vars transferred * re-enable kubernetes tests * fix conflict * fix docker launching * revert temporal hacks * match master * fix typo * remove completed todo * fix conflict * increase memory requirement to something reasonable * Update airbyte-container-orchestrator/Dockerfile Co-authored-by: Davin Chia <davinchia@gmail.com> * Update airbyte-container-orchestrator/Dockerfile Co-authored-by: Davin Chia <davinchia@gmail.com> * see if this stabilitizes tests * address review comments * bump new container version * revert temporary addition * change port back to 9000 * make re-initialization actually a no-op * add feature flag * fix version from merging earlier * fix dockerfile * fix connector build * fix * bump node version * fix dockerfile Co-authored-by: Davin Chia <davinchia@gmail.com>

explore all the nodes to mask secrets, even if they are arrays or nested at a given deep level. It makes what has been made in airbytehq#8859 more generic

* Implement destination null * Update existing testing destinations * Merge in logging consumer * Remove old destination null * Add documentation * Add destination to build and summary * Fix test * Update acceptance test * Log state message * Remove unused variable * Remove extra statement * Remove old null doc * Add dev null destination * Update doc to include changelog for dev null * Format code * Fix doc * Register e2e test destination in seed

* Extend bootloader for Cloud. * Add test that postLoad hook executes.

* Bump Airbyte version from 0.34.1-alpha to 0.34.2-alpha Co-authored-by: Davin Chia <davinchia@gmail.com>

* Fix documentation url * Update pr url * Update doc * Bump version

* Add skeleton to support all postgres types * Consolidate type tests * Fix corner cases * Bump postgres version * Add tests for time and timetz * Format code * Revert date to timestamp * Update comment * Fix unit tests * 🐛 Jdbc sources: switch from "string" to "array" schema type for columns with JDBCType.ARRAY (airbytehq#8749) * support array for jdbc sources * fixed PR comments, added test cases * added more elements for test case * Fix test case * add array test case for JdbcSourceOperations Co-authored-by: Liren Tu <tuliren.git@outlook.com> * Revert changes to support special number values Postgres source cannot handle these special values yet See https://github.com/airbytehq/airbyte/issues/8902 * Revert infinity and nan assertion in unit tests This reverts commit 3bee7d1. * Update documentation * Bump postgres source version in seed Co-authored-by: Yurii Bidiuk <35812734+yurii-bidiuk@users.noreply.github.com>

* move webapp tests module

add line rem line

…c_singer_source

Harshith Mullapudi (harshithmullapudi) · 2022-01-06T19:21:54Z

Hey, Konrad Schlatte (@schlattk) looks something went wrong can you fix it.

Konrad Schlatte (schlattk) · 2022-02-02T14:45:12Z

hi Harshith Mullapudi (@harshithmullapudi) why is that PR blocked? Is it because of the missing test account or is there another issue?

Harshith Mullapudi (harshithmullapudi) · 2022-02-02T15:39:43Z

Hey I could see 2103 commits and I was not able to understand the PR so I couldn't move forward is it possible to fix it ?

Konrad Schlatte (schlattk) · 2022-02-02T17:56:59Z

Harshith Mullapudi (@harshithmullapudi) I have created a new clean PR, hopefully that works now
#10026

github-actions bot added the area/connectors Connector related issues label Dec 23, 2021

Octavia Squidington III (octavia-squidington-iii) added the community label Dec 23, 2021

Marcos Marx (marcosmarxm) changed the title ~~Mc singer source~~ 🎉 New Source: Salesforce Marketing Cloud (Singer) Dec 23, 2021

Marcos Marx (marcosmarxm) assigned Harshith Mullapudi (harshithmullapudi) Dec 23, 2021

Yevhenii (yevhenii-ldv) and others added 24 commits January 4, 2022 10:10

🎉 SAT: Add assert that output catalog does not have any $ref in it (a…

875efd7

…irbytehq#8294) * SAT: Add assert that output catalog does not have any in it * SAT: update code after review, add test parametrizes

Source Shopify: fix for missing properties & items in schemas (ai…

1226b6b

…rbytehq#7783)

Update SAT for validate_data_points Test Basic Read option. (airbyt…

8c4899f

…ehq#8342)

Added temporal build script (airbytehq#8325)

f835fd9

🐛 Source Hubspot: remove 'skip_dynamic_fields' config param (airbyteh…

9a5b6e4

…q#8329) * ignored fields added * remove 'skip_dynamic_fields' Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Change synchronisationProgress to synchronizationProgress (airbytehq#…

8cf4b2b

…8354) * Change synchronisationProgress to synchronizationProgress * Change locale key

🐛 📝 DAT: verify that a destination is able to write any ISO8601-compl…

e767aa6

…iant date string (airbytehq#5814) * added format_reference.txt, edited edge_case_messages.txt, edge_case_catalog.json Co-authored-by: Yurii Bidiuk <yura.bidyuk@gmail.com>

Source Trello: Pull data only for configured board_ids (airbytehq#8338)

fd3ea39

* 'Boards' stream returns only specific (board_ids) records * github secret -> google secret Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Fix normalization when un-nesting (airbytehq#8378)

2934566

* Remove unique key on exploded nested tables * un-nest hint * Regen SQL

fix off by one month (airbytehq#8350)

c4613e7

allow retrieving deleted sources and destinations (airbytehq#8355)

e2266d7

* allow retrieving deleted sources and destinations * fix test

Bump Airbyte version from 0.32.8-alpha to 0.32.9-alpha (airbytehq#8391)

27da34b

Co-authored-by: sherifnada <sherifnada@users.noreply.github.com>

🐛 Source Mixpanel: fix for discovery command takes a very long time (…

e7f252a

…airbytehq#8381)

Bump Airbyte version from 0.32.9-alpha to 0.32.10-alpha (airbytehq#8403)

5e26705

Co-authored-by: jrhizor <jrhizor@users.noreply.github.com>

🐛 Fix incremental normalization with empty tables (airbytehq#8394)

a1f1d34

* Fix incremental with empty final tables * upgrade docker images * Regen SQL * Bumpversion & format

Better Env Var Organisation. (airbytehq#8390)

5dcca20

* Make all these public so Cloud can reuse these helpers. * Reorganise the env var page for better readability. * Format. * Reorganise the EnvConfigs class.

Better breaking changes instructions.

7813b96

Bump Airbyte version from 0.32.10-alpha to 0.32.11-alpha (airbytehq#8426

51b8a99

) Co-authored-by: ChristopheDuong <ChristopheDuong@users.noreply.github.com>

🐛 Source Zenloop: re-add missing seed files (airbytehq#8299)

eb7778a

airbytehq#7727 source postgres: add several types to integration tests (

60babfd

airbytehq#7913)

VitaliiMaltsev and others added 23 commits January 4, 2022 10:10

Destination MariaDbColumnstore: update Docker image for acceptance te…

f5cb7bf

…sts (airbytehq#8840) * fix for jdk 17 * Destination MariaDbColumnstore updated Docker image * update Docker image for ssh tests Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>

🎉 Source Facebook Marketing: fix schema for 'breakdowns' fields (airb…

87765f6

…ytehq#8742) * fix schema for 'breakdowns' and 'action_breakdowns fields Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

🐛 SAT: fix logs slicing (airbytehq#8848)

67fb47d

* fix logs slicing * add unit test * add more tests * bumb version * send last line if there are not trackback or container error

Bumps lxml from 4.6.3 to 4.6.5. (airbytehq#8845)

9b87c8a

Signed-off-by: Serhii Chvaliuk <grubberr@gmail.com>

Jamakase/add workspaces word to url (airbytehq#8878)

3ab3cd2

* Prepend all workspaces resources with /workspaces

🎉 Destination Redshift: add option to enable/disable deleting staging…

391668d

… data (airbytehq#8855)

Update changelog 2021-25-11 (airbytehq#8247)

2bad693

* update changelog files * remove mentions to oauth

make sure that all the secrets are remove (airbytehq#8881)

f087f9f

explore all the nodes to mask secrets, even if they are arrays or nested at a given deep level. It makes what has been made in airbytehq#8859 more generic

Extend bootloader for Cloud. (airbytehq#8879)

ef96d30

* Extend bootloader for Cloud. * Add test that postLoad hook executes.

Bump Airbyte version from 0.34.1-alpha to 0.34.2-alpha (airbytehq#8901)

a31d4fb

* Bump Airbyte version from 0.34.1-alpha to 0.34.2-alpha Co-authored-by: Davin Chia <davinchia@gmail.com>

allow serializing to yaml without quoting strings (airbytehq#8896)

d26e314

🐞 Destination e2e test: fix documentation url (airbytehq#8905)

b3762d6

* Fix documentation url * Update pr url * Update doc * Bump version

Remove jobs and attempts tables from exported archive (airbytehq#8646)

c91d6a4

airbyte-e2e-testing => airbyte-webapp-e2e-tests (airbytehq#8892)

4ed2d60

* move webapp tests module

tests pass

dde7c34

add line rem line

Merge branch 'mc_singer_source' of github.com:schlattk/airbyte into m…

7d1c7da

…c_singer_source

dockerfile

e5fcce2

Harshith Mullapudi (harshithmullapudi) added the blocked label Jan 11, 2022

Konrad Schlatte (schlattk) mentioned this pull request Feb 3, 2022

New Source Salesforce Marketingcloud Singer #10026

Closed

40 tasks

Harshith Mullapudi (harshithmullapudi) closed this Feb 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Source: Salesforce Marketing Cloud (Singer)#9080

🎉 New Source: Salesforce Marketing Cloud (Singer)#9080
Konrad Schlatte (schlattk) wants to merge 2103 commits intoairbytehq:masterfrom
schlattk:mc_singer_source

Konrad Schlatte (schlattk) commented Dec 23, 2021 •

edited by edgao

Loading

Uh oh!

CLAassistant commented Dec 23, 2021 •

edited

Loading

Uh oh!

Marcos Marx (marcosmarxm) commented Dec 23, 2021

Uh oh!

Harshith Mullapudi (harshithmullapudi) commented Jan 6, 2022

Uh oh!

Konrad Schlatte (schlattk) commented Feb 2, 2022

Uh oh!

Harshith Mullapudi (harshithmullapudi) commented Feb 2, 2022

Uh oh!

Konrad Schlatte (schlattk) commented Feb 2, 2022 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

Konrad Schlatte (schlattk) commented Dec 23, 2021 • edited by edgao Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Uh oh!

CLAassistant commented Dec 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Marcos Marx (marcosmarxm) commented Dec 23, 2021

Uh oh!

Harshith Mullapudi (harshithmullapudi) commented Jan 6, 2022

Uh oh!

Konrad Schlatte (schlattk) commented Feb 2, 2022

Uh oh!

Harshith Mullapudi (harshithmullapudi) commented Feb 2, 2022

Uh oh!

Konrad Schlatte (schlattk) commented Feb 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Konrad Schlatte (schlattk) commented Dec 23, 2021 •

edited by edgao

Loading

CLAassistant commented Dec 23, 2021 •

edited

Loading

Konrad Schlatte (schlattk) commented Feb 2, 2022 •

edited

Loading