[ELI-688] - Adding Kinesis Data Stream as a buffer for Firehose#609
[ELI-688] - Adding Kinesis Data Stream as a buffer for Firehose#609
Conversation
…ient and mock stream
|
Turns out that moto is a little bit shallow on firehose and wont actually pass records from a fake kinesis client to a fake firehose client - which in fairness would be a lot to ask! One option would be to create some bridging code in the test itself to do that passing of the record. Thinking about this in a slightly different frame - we are sort of wading into the territory of "not the lambda's responsibility anymore" to ensure that audit records reach the s3 bucket, rather in the new world that is a task for firehose to handle. So from that perspective maybe we shouldn't be directly testing for that in our code and we have have to wait until a deployment to dev perhaps to see that working. For now I think some bridging code achieves the aim of proving that we are writing to the kinesis stream, while preserving existing tests so we'll go with that, happy to discuss. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
The dismissed checkov warnings were about being overly permissive with a KMS key where it sees "*" but in relation to the key there is only one of them - apparently this is a known issue |
Description
Adds this new service in-between our lambda and Firehose to act as a queue of sorts, which will allow us to get around Firehose service limits.
This does not cover tuning of the new stream as the scope of this is already big enough, we should have that as a follow up ticket. For now I'm suggesting we use the on demand setting as this will scale to whatever we need for the time being.
Context
Due to the size of our audit records, when we scale up to 400rps we hit Firehose service limits. It was decided that we cannot condense the audit records any further without losing key information, so this workaround allows us to keep the full record and scale up to the SLA response requirements.
Type of changes
Checklist
Sensitive Information Declaration
To ensure the utmost confidentiality and protect your and others privacy, we kindly ask you to NOT including PII (Personal Identifiable Information) / PID (Personal Identifiable Data) or any other sensitive data in this PR (Pull Request) and the codebase changes. We will remove any PR that do contain any sensitive information. We really appreciate your cooperation in this matter.