AWS-firehose-s3-producer is an AWS Lambda function in Javascript that is designed to ingest files uploaded to a S3 bucket into AWS Kinesis Firehose.

Files are uploaded to a S3 bucket which is configured to publish notifications to SQS. SQS then triggers this lambda, which will read the file and stream it to Firehose.

• streams files rather than load them to memory, therefore large files can be processed with minimal resources
• gzipped files are supported

The Lambda requires the following environment variable to be set:

• FIREHOSE_STREAM_NAME: the Kinesis firehose datastream to stream to.

Edit, then attach the provided policy to the Role your AWS Lambda is configured to use.