v1.19.4

What's Changed

• Bump axios to 1.15.2 by @radovanjorgic in #191

Full Changelog: v1.19.3...v1.19.4

v1.19.3

What's Changed

• chore: Fix incorrect SDK log classification by @gasperzgonec in #188
• Chore(deps): Bump protobufjs from 7.5.4 to 7.5.5 by @dependabot[bot] in #189

Full Changelog: v1.19.2...v1.19.3

v1.19.2

What's Changed

• [Snyk] Security upgrade axios from 1.14.0 to 1.15.0 by @gasperzgonec in #184
• chore: Update extraction time calculation to correct phase by @gasperzgonec in #185
• chore: Update follow-redirects dependency by @gasperzgonec in #187

Full Changelog: v1.19.1...v1.19.2

v1.19.1

What's Changed

• Emit AttachmentLoadingDelayed instead of DataLoadingDelayed by @radovanjorgic in #183

Full Changelog: v1.19.0...v1.19.1

v1.19.0

What's Changed

• Update REFERENCE.md file by @gasperzgonec in #176
• Align edm schema and types to the latest recipe changes by @radovanjorgic in #177
• Implement createMockEvent and export it for fixture testing by @gasperzgonec in #178
• Add proper timeout handling in loadItemTypes and loadAttachments functions by @radovanjorgic in #179
• Chore(deps): Bump lodash and @microsoft/api-extractor by @dependabot[bot] in #180
• Fix the way '--local' flag is parsed by @gasperzgonec in #181

Full Changelog: v1.18.0...v1.19.0

v1.18.0

What's Changed

• Emit report and processed files if loading event by @radovanjorgic in #165
• Make EventContext and MockServer public by @gasperzgonec in #168
• Add support for selective object extraction by @radovanjorgic in #166
• Update express dependency to prevent a CVE by @gasperzgonec in #173
• Stop processing attachments and emit progress in case of soft timeout by @radovanjorgic in #171
• Fix EventContext and MockServer exports by @gasperzgonec in #172
• Rewrite mock server with native node:http module by @radovanjorgic in #174
• Add support for Control Protocol V2 by @gasperzgonec in #144
• Don't resolve timeValue if field is undefined by @radovanjorgic in #175

Full Changelog: v1.17.0...v1.18.0

---

Migration guide — Automatic reports and processed_files in loading events (v1.18.0)

The SDK now automatically includes reports and processed_files in the event payload when emitting loading events. Previously, connector developers had to manually pass these fields in the data parameter of adapter.emit() — this was error-prone and inconsistent with how extraction events already handled artifacts automatically.

Passing reports and processed_files manually still works but is no longer necessary. The SDK tracks loading reports and processed files internally and attaches them to the event payload automatically.

Before

import { LoaderEventType, processTask } from '@devrev/adaas-sdk';

processTask({
  task: async ({ adapter }) => {
    const { reports, processed_files } = await adapter.loadItemTypes({
      itemTypesToLoad: [
        /* ... */
      ],
    });

    await adapter.emit(LoaderEventType.DataLoadingDone, {
      reports,
      processed_files,
    });
  },
  onTimeout: async ({ adapter }) => {
    const reports = adapter.reports;
    const processed_files = adapter.processedFiles;

    await adapter.emit(LoaderEventType.DataLoadingError, {
      reports,
      processed_files,
      error: { message: 'Failed to load data. Lambda timeout.' },
    });
  },
});

After

import { LoaderEventType, processTask } from '@devrev/adaas-sdk';

processTask({
  task: async ({ adapter }) => {
    await adapter.loadItemTypes({
      itemTypesToLoad: [
        /* ... */
      ],
    });

    await adapter.emit(LoaderEventType.DataLoadingDone);
  },
  onTimeout: async ({ adapter }) => {
    await adapter.emit(LoaderEventType.DataLoadingError, {
      error: { message: 'Failed to load data. Lambda timeout.' },
    });
  },
});

What changed

Aspect	Before	After
reports in emit	Manually passed in event data	Automatically included by the SDK
processed_files in emit	Manually passed in event data	Automatically included by the SDK
loadItemTypes return value	Had to capture and forward reports and processed_files	Return value can be ignored (SDK tracks internally)
Timeout / error handlers	Had to read adapter.reports and adapter.processedFiles and pass them manually	SDK attaches them automatically

Key points

• No action required — existing connectors that pass reports and processed_files manually will continue to work. The SDK merges any manually provided values with its internal state.
• Applies to all loading events — DataLoadingDone, DataLoadingProgress, DataLoadingError, AttachmentLoadingDone, etc. all automatically include reports and processed_files.
• Extraction events are unaffected — extraction events continue to automatically include artifacts as before.

v1.17.0

What's Changed

• Clean up of not needed files, removed some dependencies by @radovanjorgic in #154
• Migrate External Sync Units to file upload and add confirmation skip by @gasperzgonec in #156
• Add retry mechanism for ECONNABORTED errors on slow platform responses by @gasperzgonec in #160
• Expose content_type field in NormalizedAttachment interface by @radovanjorgic in #161
• Improve logging in "Worker has exited without emitting an event" by @gasperzgonec in #158

Full Changelog: v1.16.0...v1.17.0

---

Migration guide — External sync unit extraction (v1.17.0)

The SDK now supports uploading external sync units as artifacts instead of sending them inline in the callback event payload. Previously, the external_sync_units array was included directly in the event data sent to ADaaS, which could hit the SQS message size limit (~250KB) for connectors with many sync units. Now, you should initialize a repo for external sync units, push to it, and let the SDK upload them as gzipped JSONL artifacts — the same way data extraction already works.

Passing external_sync_units inline still works but is deprecated. The SDK will handle it behind the scenes for now, but you should migrate to the repo-based approach.

Before

import {
  ExternalSyncUnit,
  ExtractorEventType,
  processTask,
} from '@devrev/adaas-sdk';

processTask({
  task: async ({ adapter }) => {
    const externalSyncUnits: ExternalSyncUnit[] = await fetchSyncUnits();

    await adapter.emit(ExtractorEventType.ExternalSyncUnitExtractionDone, {
      external_sync_units: externalSyncUnits,
    });
  },
  onTimeout: async ({ adapter }) => {
    await adapter.emit(ExtractorEventType.ExternalSyncUnitExtractionError, {
      error: {
        message: 'Failed to extract external sync units. Lambda timeout.',
      },
    });
  },
});

After

import {
  AirSyncDefaultItemTypes,
  ExternalSyncUnit,
  ExtractorEventType,
  processTask,
} from '@devrev/adaas-sdk';

processTask({
  task: async ({ adapter }) => {
    adapter.initializeRepos([
      {
        itemType: AirSyncDefaultItemTypes.EXTERNAL_SYNC_UNITS,
        overridenOptions: {
          batchSize: 25000,
          skipConfirmation: true,
        },
      },
    ]);

    const externalSyncUnits: ExternalSyncUnit[] = await fetchSyncUnits();

    await adapter
      .getRepo(AirSyncDefaultItemTypes.EXTERNAL_SYNC_UNITS)
      ?.push(externalSyncUnits);

    await adapter.emit(ExtractorEventType.ExternalSyncUnitExtractionDone);
  },
  onTimeout: async ({ adapter }) => {
    await adapter.emit(ExtractorEventType.ExternalSyncUnitExtractionError, {
      error: {
        message: 'Failed to extract external sync units. Lambda timeout.',
      },
    });
  },
});

What changed

Aspect	Before	After
How sync units are sent	Inline in event data (external_sync_units array)	Uploaded as artifact(s) via a repo
Size limit	Bounded by SQS message size (~250KB)	Unbounded (artifact upload)
Batch size	N/A	25,000 sync units per artifact
skipConfirmation	N/A	true — the sync run doesn't exist yet at this phase, so artifact confirmation is skipped

Key points

• initializeRepos — use AirSyncDefaultItemTypes.EXTERNAL_SYNC_UNITS ('external_sync_units') as the item type.
• skipConfirmation: true — required because external sync unit extraction happens before a sync run is created, so the normal artifact confirmation step (which attaches the artifact to a sync) would fail.
• batchSize: 25000 — external sync units are small objects, so the batch size is much larger than the default 2000 used for data extraction.
• Don't pass external_sync_units in emit — the repos are uploaded automatically before the event is sent, and the artifact references are attached to the callback payload.

v1.16.0

What's Changed

• Improve test performance by @gasperzgonec in #143
• Prevent task and onTimeout running concurrently by wait for task to resolve before executing onTimeout by @radovanjorgic in #147
• Update CI coverage action by @gasperzgonec in #149
• Fix issue with the race-condition in spawn.ts by @gasperzgonec in #150
• Add external domain metadata interface by @radovanjorgic in #146
• Remove npm tokens in release workflows for the sake of trusted publishing by @radovanjorgic in #148
• Fix attachment streaming errors and add observability for failed attachments by @gasperzgonec in #136
• Fix sending large messages to SQS queue by @gasperzgonec in #86
• Fix minimatch vulnerability coming from @microsoft/api-extractor by @radovanjorgic in #152
• Revert minimatch override by @radovanjorgic in #153

Full Changelog: v1.15.2...v1.16.0

---

Migration guide — Timeout handling (v1.16.0)

SDK now prevents task and onTimeout from running concurrently. Previously, when a timeout occurred, the SDK would execute onTimeout immediately while task was still running - meaning both could access state, repos, and emit events at the same time, leading to potential bugs. Now, onTimeout only runs after task has finished.

This means you must check adapter.isTimeout in your extraction loops and return from task when it's true.

Before (task and onTimeout ran concurrently)

processTask({
  task: async ({ adapter }) => {
    adapter.initializeRepos(repos);
    const projects = await httpClient.getProjects();

    for (const project of projects) {
      const todos = await httpClient.getTodos(project.id);
      await adapter.getRepo('todos')?.push(todos);

      for (const todo of todos) {
        const comments = await httpClient.getComments(todo.id);
        await adapter.getRepo('comments')?.push(comments);
      }
    }

    await adapter.emit(ExtractorEventType.DataExtractionDone);
  },
  onTimeout: async ({ adapter }) => {
    await adapter.emit(ExtractorEventType.DataExtractionProgress);
  },
});

After (task must return before onTimeout runs)

processTask({
  task: async ({ adapter }) => {
    adapter.initializeRepos(repos);
    const projects = await httpClient.getProjects();

    for (const project of projects) {
      if (adapter.isTimeout) return;

      const todos = await httpClient.getTodos(project.id);
      await adapter.getRepo('todos')?.push(todos);

      for (const todo of todos) {
        if (adapter.isTimeout) return;

        const comments = await httpClient.getComments(todo.id);
        await adapter.getRepo('comments')?.push(comments);
      }
    }

    await adapter.emit(ExtractorEventType.DataExtractionDone);
  },
  onTimeout: async ({ adapter }) => {
    await adapter.emit(ExtractorEventType.DataExtractionProgress);
  },
});

Timeout flow

Step	Duration	What happens
Task starts	0 min	Your extraction logic runs normally
Soft timeout	10 min	adapter.isTimeout is set to true
Grace period	10–13 min	Task should detect isTimeout, return, then onTimeout runs
Hard timeout	13 min	If the worker hasn't exited, it is forcefully killed and extraction fails

v1.15.2

What's Changed

• Add attachment deduplication checks for both ID and Parent ID by @gasperzgonec in #131
• Fix brace-expansion vulnerability by @radovanjorgic in #141
• Fix vulnerability introduced through qs transitive dependency by @radovanjorgic in #142

Full Changelog: v1.15.1...v1.15.2

v1.15.1

What's Changed

• Add v2 release workflow for visibility in Actions UI by @radovanjorgic in #137
• [Snyk] Security upgrade axios from 1.12.1 to 1.13.5 by @gasperzgonec in #139

Full Changelog: v1.15.0...v1.15.1