How to trigger a Lambda Function from S3

July 27th, 2021

Reading Time: 2 minutes

If your application requires you to be able to trigger a Lambda function from S3, it is possible to do so using the Amazon Simple Storage Service (Amazon S3). With Amazon S3, you can set up an event notification on an Amazon S3 bucket that will send a notification to an AWS Lambda function when an object is created or deleted. In this post, we will walk through the steps to trigger a Lambda function from S3.

Sometimes we need to let a Lambda know when a new file was created in an S3 bucket. That’s can be the case when we have a data lake and we want to process those files. But beware, when we talk about a data lake, we mean a huge amount of data so we shouldn’t start processing immediately a file is written since our capacity can be limited. In that case, we should use a queue mechanism but that is out of the scope of this post so Let’s concentrate on our specific problem: trigger a Lambda from S3.

We could do this from the console using point and click but as a good practice, let’s automate this provisioning with Cloudformation. This way we will be able to move our code across different accounts and deal with environments easily.

The Cloudformation approach can be slow at the beginning but once you get familiar with it, you will accelerate your dev process. Another big advantage of Cloudfromation is versioning. Since our output is a yml file, it can be easily add it to a github repo. Thus, tracking changes becomes a piece of cake. However, you may resolve your problem first from the console and then translate it to Cloudformation

Let’s do it step by step, building small pieces of Cloudformation components, deploy them and then put all together in one file.

Have in mind this is not a production-ready solution. Probably you will have to make your own customizations so take this as a starting point.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  lambda-s3-trigger

  Sample SAM Template for lambda-s3-trigger

Globals:
  Function:
    Timeout: 3

Parameters:
  Environment:
    Type: String
    Description: Environment name. Example, staging, dev, prod, etc.

Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello-world/
      Handler: app.lambdaHandler
      Runtime: nodejs12.x
      Role:
        Fn::GetAtt:
          - "MyRole"
          - "Arn"
    Tags:
      Name: !Sub "${Environment}-my-function"


  MyRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: AccessToS3Notifications
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 's3:GetBucketNotification'
                  - 's3:PutBucketNotification'
                  - "s3:GetObject"
                Resource: !Sub 'arn:aws:s3:::${AWS::AccountId}-${Environment}-my-bucket'

  MyBucket:
    Type: AWS::S3::Bucket
    DependsOn:
      - MyFunction
    Properties:
      # the bucket name has the account as a prefix since it has to be unique globally at AWS level
      BucketName: !Sub "${AWS::AccountId}-${Environment}-my-bucket"
      NotificationConfiguration:
        LambdaConfigurations:
          - Event: 's3:ObjectCreated:*'
            Function: !GetAtt MyFunction.Arn

  PermissionForEventsToInvokeLambda:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !GetAtt MyFunction.Arn
      Action: "lambda:InvokeFunction"
      Principal: "s3.amazonaws.com"
      SourceAccount: !Ref 'AWS::AccountId'

Build and deploy: