-
Notifications
You must be signed in to change notification settings - Fork 31
feat: add FDv2 FileDataSource initializer #1010
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
this commit will introduce the implementation of FDv2 File initiator
|
@launchdarkly/browser size report |
|
@launchdarkly/js-sdk-common size report |
|
@launchdarkly/js-client-sdk-common size report |
|
@launchdarkly/js-client-sdk size report |
packages/shared/sdk-server/src/data_sources/fileDataInitilizerFDv2.ts
Outdated
Show resolved
Hide resolved
5d8cadd to
cf28ade
Compare
| | DeleteObject | ||
| | PayloadTransferred | ||
| | GoodbyeObject | ||
| | ErrorObject; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If new values are added to the protocol, is the code well behaved? Same for nearby types.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yea payload changes will potentially break this, but that is probably true even if we didn't declare these types?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Double check logic in the processFDv1FlagsAndSegments in PollingProcessorFDv2 matches this and/or consider commonizing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ah yea the logic is the same! Let me think of a way to merge the logic.
Now that I am thinking about this more, I don't think sdk-common is the right place to create this builder? I am not sure how FDv2 is going to show up for client side implementations so probably move this out to sdk-server then, if we find out that client implementation is similar, move it back out to sdk-common? @abarker-launchdarkly @kinyoklion thoughts?
Otherwise, I think the plan will be change this to a FDv1PayloadAdaptor instead of a builder.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the data inside the objects is opaque, then the the code itself would likely be the same for both of them.
I am fine with it not being common until we need it though.
A note specifically for internal/common code, but code we intend to use client-side we likely want to avoid classes and do this with functions, as generally speaking they end up smaller when minified. (Or old school function that returns object).
packages/shared/sdk-server/src/data_sources/fileDataInitilizerFDv2.ts
Outdated
Show resolved
Hide resolved
| 'DataSourceOptions', | ||
| typeof options.dataSource, | ||
| ), | ||
| ]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should a customer be able to configure to use only a file data source? @kinyoklion thoughts?
I think they should be able to configure just the file data source with polling and streaming disabled, but you'll have to confirm with others on team as I haven't dealt with file data source implementations.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we discussed sync, but yeah it should be possible. This would be a typical use case with FDv1. Often times people use file data sources for testing purposes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay I think we can have a separate story to handle file synchronizer? I think it would be a fairly trivial implementation on top of this PR.
packages/shared/sdk-server/src/data_sources/fileDataInitilizerFDv2.ts
Outdated
Show resolved
Hide resolved
cf28ade to
15d65d7
Compare
24bee72 to
10363a9
Compare
packages/shared/sdk-server/src/data_sources/fileDataInitilizerFDv2.ts
Outdated
Show resolved
Hide resolved
this commit adds a fdv1 payload adaptor to process fdv1 data sets
b1b1883 to
e1b1349
Compare
this enables users to specify file data source initializer
e1b1349 to
718b11a
Compare
this commit will change the payload adaptor from a class to function so it could be minified better
dca2863 to
c42e4f4
Compare
This PR will introduce file data source initializer for FDv2 that could be specified by adding in an optional
intializerOptionto the FDv2 datasource option. If this option is not specified then the behavior should not change from before this change.Additionally:
customdata source options mode that allows users to specify the ordering of their initializers and synchronizersNOTE: will need to look into a problem where while the initializers are running in expected order, the client initialized event seems to be prematurely firing. Example with the following config:
STDOUT with the
sample-featureflag set tofalsein LD:I think this issue could be addressed in a separate change.
Note
Introduces a file-based FDv2 initializer and configurable custom data source pipeline, adds an adaptor to convert FDv1 payloads to FDv2 events, tightens FDv2 proto/types, and wires everything into LDClient with tests.
FDv1PayloadAdaptor(packages/shared/common/src/internal/fdv2/FDv1PayloadAdaptor.ts): converts FDv1 flags/segments into FDv2 event sequences; exported viainternal/fdv2/index.PollingProcessorFDv2: replaces inline FDv1 conversion withFDv1PayloadAdaptorand sets selector; unchanged FDv2 path.payloadProcessor: relaxespayload-transferredvalidation to allow emptystateand makes checks null/undefined-safe.proto: adds strictEventType,ObjectKind,GoodbyeObject,ErrorObject; refines eventdataunion; makesPayloadTransferred.idoptional.LDDataSystemOptions: newcustomoption with orderedinitializers(file|polling) andsynchronizers(streaming|polling).LDClientImpl: buildsCompositeDataSourcefrom custom configs; supports file initializer, streaming/polling synchronizers, and FDv1 fallback polling; exportsDEFAULT_STREAM_RECONNECT_DELAY.Configuration: validatescustomdata source options; exportsDEFAULT_STREAM_RECONNECT_DELAY.FileDataInitializerFDv2: loads JSON/YAML from disk, merges multiple files, emits FDv2 payload viaFDv1PayloadAdaptor; no auto-update; error handling for invalid/missing FS.FDv1PayloadAdaptor.CompositeDataSourcetests for multi-initializer fallback/short-circuit.FileDataInitializerFDv2(JSON/YAML parsing, multi-file merge/overwrite, errors).Written by Cursor Bugbot for commit c42e4f4. This will update automatically on new commits. Configure here.