Tag: cloudformation

How to write to S3 bucket from Lambda function

January 29th, 2023

Reading Time: 2 minutes

AWS Lambda functions are a great way to automate certain tasks and processes in the cloud. They can be triggered by events, such as a file upload to an S3 bucket or a message sent to an SNS topic, allowing you to execute some code in response.

In this post, we’ll show you how to write data to an S3 bucket from a Lambda function. This can be useful for a variety of tasks, such as archiving log files or uploading data to a data lake.

S3 (Simple Storage Service) is Amazon’s cloud storage solution that allows you to store and access data from anywhere in the world. Writing to an S3 bucket from a Lambda function is a simple way to store and access data in the cloud.

In this post, I’ll walk you through how to write to an S3 bucket from a Lambda function. We’ll use the AWS SDK for Node.js to access S3.

Take this example as a starting point. This is not a production-ready code, probably some tweaks for permissions will be necessary to meet your requirements.

AWS resources we need

Lambda Function
S3 Bucket
Lambda Role
Bucket Policy

The Lambda function

const AWS = require("aws-sdk");
const s3 = new AWS.S3({
    region: "us-east-1",
});
let response;

exports.lambdaHandler = async (event, context) => {
    try {
        console.log(event);
        const params = {
            Bucket: process.env.MY_BUCKET,
            Key: 'test-file.txt',
            Body: "test content"
        }
        console.log('writing to s3', params);
        const result =  await s3.putObject(params).promise();
        console.log(result);

        response = {
            'statusCode': 200,
        }
    } catch (err) {
        console.log(err);
        return err;
    }

    return response
};

The SAM template

For the sake of simplicity, we are going to use Principal: “*” which makes your bucket 100% public!

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  Sample SAM Template for Lambda and S3
Parameters:
  Environment:
    Type: String
    Description: Environment name. Example, staging
Resources:
  MyLambda:
    Type: AWS::Serverless::Function
    DependsOn:
      - "MyBucket"
    Properties:
      CodeUri: hello-world/
      Handler: app.lambdaHandler
      Runtime: nodejs12.x
      Environment:
        Variables:
          MY_BUCKET:
            Ref: "MyBucket"
      Role:
        Fn::GetAtt:
          - "MyLambdaRole"
          - "Arn"
    Tags:
      Name: !Sub "${Environment}-my-test-lambda"

  MyBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub "${Environment}-my-test-bucket"

  MyBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      PolicyDocument:
        Id: BucketPolicy
        Version: 2012-10-17
        Statement:
          - Sid: AccessAll
            Action: s3:*
            Effect: Allow
            # Beware: this makes your bucket public!
            Principal: "*"
            Resource: !Join
              - ''
              - - 'arn:aws:s3:::'
                - !Ref MyBucket
                - /*
      Bucket: !Ref MyBucket

  MyLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Action: "sts:AssumeRole"
            Principal:
              Service:
                - "lambda.amazonaws.com"
        Version: "2012-10-17"
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: MyLambdaPolicy
          PolicyDocument:
            Statement:
              - Effect: Allow
                Action: "s3:*"
                Resource:
                  - Fn::GetAtt:
                      - "MyBucket"
                      - "Arn"
            Version: "2012-10-17"

As a good practice, always receive the environment as a parameter. This way, all your resources will be easier to identify.

Deploy

Build our lambda and template. This will check also the syntax of your template

$ sam build

Only for the first time run and follow the steps:

$ sam deploy --guided

The second time and so on you can execute:

$ sam deploy --no-confirm-changeset

Clean up your test AWS resources. Delete unused lambdas, buckets, etc to keep your account organized and the most important: no extra costs

Resources

Salesforce integration with AWS AppFlow, S3, Lambda and SQS

January 29th, 2023

Reading Time: 5 minutes

Salesforce and Amazon’s Web Services (AWS) are two powerful software development and cloud computing tools. In this post, we’ll discuss how these two tools can be integrated for an optimized and efficient workflow.

The integration of Salesforce and AWS allows businesses to take advantage of the scalability, reliability, and security of both platforms. The integration enables businesses to quickly and efficiently move key data and applications between the cloud platforms and reduces the complexity of integration.

There are many ways to sync up our Salesforce data with third parties in realtime. One option is a mix of Salesforce and AWS services, specifically Change Data Capture from Salesforce and AppFlow from AWS. We are going to build a Cloudformation yml file with all that we need to deploy our integration on any AWS environment. However can be a good option to do it first by point and click through the AWS console and then translate it into a Cloudformation template.

If you are using Heroku and Postgres, Heroku Connect is a good option too

About Salesforce Change Data Capture

Receive near-real-time changes of Salesforce records, and synchronize corresponding records in an external data store.

Change Data Capture
publishes change events, which represent changes to Salesforce records. Changes include creation of a new record, updates to an existing record, deletion of a record, and undeletion of a record.

Important:
Change Data Capture does not support relationships at the time this post was written (08/2021). This means you will only be able to sync up beyond your object unless you implement some tricks using Process Builder and Apex. That’s out of the scope of this post and we are going to see it in a different one because requires some extra steps and knowledge.

To start listening on specific object go to Setup -> Integrations -> Change Data Capture. Move the object you want to the right.

Advantages of using AppFlow approach

Data being transferred securely
Credentials are managed by Oauth process
No coding required unless you want to run some specific logic for every sync up
100% serverless, pay as use

Disadvantages of using AppFlow approach

The connection must exist before deploying the infrastructure. This is a manual step
This approach can take some time to learn and configure, specially if you are already familiar with callouts from Salesforce

Requirements for Salesforce

Your Salesforce account must be enabled for API access. API access is enabled by default for the Enterprise, Unlimited, Developer, and Performance editions.
Your Salesforce account must allow you to install connected apps. If this functionality is disabled, contact your Salesforce administrator. After you create a Salesforce connection in Amazon AppFlow, verify that the connected app named Amazon AppFlow Embedded Login App is installed in your Salesforce account.
The refresh token policy for the Amazon AppFlow Embedded Login App must be set to Refresh token is valid until revoked. Otherwise, your flows will fail when your refresh token expires.
You must enable change data capture in Salesforce to use event-driven flow triggers.
If your Salesforce app enforces IP address restrictions, you must grant access to the addresses used by Amazon AppFlow.
To create private connections using AWS PrivateLink, you must enable both Manager Metadata and Manage External Connections user permissions in your Salesforce account. Private connections are currently available in the us-east-1 and us-west-2 AWS Regions.

Architecture for the solution

Let say we want to listen to changes on Account object. Every time a new Account is created or updated there will be an event to AppFlow through Salesforce Data Capture.

We could add some logic in the Lambda function to decide if we are interested in that change or not.

How to create the Salesforce Oauth Connection

As we said, an Oauth connection must exist before deploying our stack to AWS. This is something we have to create by hand. If we deal with different environments in AWS, we can create as many connection as we want pointing to our different Salesforce instances.

Open your AWS console and go to Amazon App Flow
Go to View Flows and click on Connections
Click on Create Connection. Select production in case you have a dev org. Provide a connection name
Once you click on Continue, a Salesforce popup will be open. Put your Salesforce credentials to login
After that your connection will be created and available to use

The Cloudformation template

      
# Commands to deploy this through SAM CLI
#  sam build
#  sam deploy --no-confirm-changeset

AWSTemplateFormatVersion: 2010-09-09
Description: >-
  app flow lambda + s3 + SQS

Transform:
  - AWS::Serverless-2016-10-31

Parameters:
  Environment:
    Type: String
    Description: Environment name. Example, dev,staging,testing, etc

Globals:
  Function:
    Runtime: nodejs12.x
    Timeout: 30
    MemorySize: 128


Resources:
  MyLambda:
    Type: AWS::Serverless::Function
    DependsOn:
      - "MyQueue"
    Properties:
      Handler: src/handlers/my.handler
      Description: Sync up lambda
      Environment:
        Variables:
          QueueURL:
            Ref: "MyQueue"
          MyBucket: !Sub "${AWS::AccountId}-${Environment}-my-bucket"
      Role:
        Fn::GetAtt:
          - "MyLambdaRole"
          - "Arn"
    Tags:
      Name: !Sub "${Environment}-my-lambda"

  MyLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Action: "sts:AssumeRole"
            Principal:
              Service:
                - "lambda.amazonaws.com"
        Version: "2012-10-17"
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: AccessOnMyQueue
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action: "sqs:SendMessage"
                Resource:
                  - Fn::GetAtt:
                      - "MyQueue"
                      - "Arn"
        - PolicyName: AccessToS3Notifications
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 's3:GetBucketNotification'
                Resource: !Sub 'arn:aws:s3:::${AWS::AccountId}-${Environment}-my-bucket'
        - PolicyName: AccessOnS3Objects
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - "s3:GetObject"
                Resource: !Sub 'arn:aws:s3:::${AWS::AccountId}-${Environment}-my-bucket/*'


  MyBucket:
    Type: AWS::S3::Bucket
    DependsOn:
      - MyLambda
    Properties:
      BucketName: !Sub "${AWS::AccountId}-${Environment}-my-bucket"
      NotificationConfiguration:
        LambdaConfigurations:
          - Event: 's3:ObjectCreated:*'
            Function: !GetAtt MyLambda.Arn
      LifecycleConfiguration:
        Rules:
          - Id: ExpirationInDays
            Status: 'Enabled'
            ExpirationInDays: 3
          - Id: NoncurrentVersionExpirationInDays
            Status: 'Enabled'
            NoncurrentVersionExpirationInDays: 3

  MyBucketPolicy:
    Type: AWS::S3::BucketPolicy
    DependsOn: MyBucket
    Properties:
      Bucket: !Ref MyBucket
      PolicyDocument:
        Version: '2008-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: appflow.amazonaws.com
            Action:
              - s3:PutObject
              - s3:AbortMultipartUpload
              - s3:ListMultipartUploadParts
              - s3:ListBucketMultipartUploads
              - s3:GetBucketAcl
              - s3:PutObjectAcl
            Resource:
              - !Sub "arn:aws:s3:::${AWS::AccountId}-${Environment}-my-bucket"
              - !Sub "arn:aws:s3:::${AWS::AccountId}-${Environment}-my-bucket/*"

  MyQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: !Sub "${Environment}-my-queue.fifo"
      FifoQueue: true
      ContentBasedDeduplication: true
      RedrivePolicy:
        deadLetterTargetArn:
          Fn::GetAtt:
            - "MyDeadLetterQueue"
            - "Arn"
        maxReceiveCount: 2

  MyDeadLetterQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: !Sub "${Environment}-my-queue-dlq.fifo"
      FifoQueue: true
      MessageRetentionPeriod: 1209600 # 14 days (the max supported)

  MyQueuePolicy:
    DependsOn:
      - "MyQueue"
    Type: AWS::SQS::QueuePolicy
    Properties:
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - "events.amazonaws.com"
                - "sqs.amazonaws.com"
            Action:
              - "sqs:SendMessage"
              - "sqs:GetQueueUrl"
              - "sqs:DeleteMessage"
              - "sqs:ReceiveMessage"
            Resource:
              Fn::GetAtt:
                - "MyQueue"
                - "Arn"
      Queues:
        - Ref: "MyQueue"

  # AppFlow flow to connect SFDC and AWS
  MyAppFlow:
    Type: AWS::AppFlow::Flow
    Properties:
      FlowName: !Sub "${Environment}-my-app-flow"
      Description: Flow to sync up with Salesforce
      TriggerConfig:
        TriggerType: Event
      SourceFlowConfig:
        ConnectorType: Salesforce
        ConnectorProfileName: !Sub "${Environment}-my-connection" # the name of the Oauth connection created in AWS console
        SourceConnectorProperties:
          Salesforce:
            Object: Account__ChangeEvent
            EnableDynamicFieldUpdate: false
            IncludeDeletedRecords: true
      DestinationFlowConfigList:
        - ConnectorType: S3
          DestinationConnectorProperties:
            S3:
              BucketName: !Ref MyBucket
              S3OutputFormatConfig:
                AggregationConfig:
                  AggregationType: None
                PrefixConfig:
                  PrefixFormat: MINUTE
                  PrefixType: FILENAME
                FileType: JSON
      Tasks:
        - TaskType: Filter
          ConnectorOperator:
            Salesforce: PROJECTION
          SourceFields:
            - Name
        - TaskType: Map
          SourceFields:
            - Name
          TaskProperties:
            - Key: SOURCE_DATA_TYPE
              Value: Name
            - Key: DESTINATION_DATA_TYPE
              Value: Name
          DestinationField: Name

Debugging

It’s important we have a way to troubleshoot in case things go wrong. Since this integration deals with different AWS services, we have to see what we have available in each one.

AppFlow run history
CloudWatch for our Lambda
Spy on S3 to see objects created
Spy on SQS messages created (monitor tab)

Resources

How to publish a message in SQS from SNS

May 11th, 2021
Reading Time: 2 minutes
Overview

Let say you want to process images in background or any other task that requires heavy processing and you don’t want to tie this operation to any other core operation or service.

Suppose service A deals with users:
- Sign in
- Sign up
- Forgot password
- Profile update
- etc
Every time a user uploads an image to his profile you want to resize it and generate multiple thumbnails for multiple platforms or devices. You can do this operation under the same umbrella of service A but soon this business logic will grow up making our microservice a larger service.
So, you decide to separate this kind of operations to a new microservice (service B) but how you communicate with each other?

One option is to call the service directly but sounds you are tying bot services. Another option (here we go with this post) is to broadcast an event from service A called “profile updated”.

Now we have to see how service B is notified to start processing the image and here is where SNS and SQS come.

In the following example, I show you have to deploy an SNS topic that writes a message in an SQS queue. After that, you could have a Lambda function that is triggered by this queue but this is out of this scope.

I strongly recommend that you start playing in a brand new project instead of trying to add more stuff to an existing one. Thus, will be easier to narrow down errors

The Cloudformation template
```
      
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  Sample SAM Template SNS and SQS

Parameters:
  Environment:
    Type: String
    Description: example, staging
Resources:
  MyQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: !Sub "${Environment}-my-queue.fifo"
      FifoQueue: true
  MyTopic:
    Type: AWS::SNS::Topic
    Properties:
      ContentBasedDeduplication: true
      FifoTopic: true
      Subscription:
        - Endpoint:
            Fn::GetAtt:
              - MyQueue
              - Arn
          Protocol: sqs
      TopicName: !Sub "${Environment}-my-topic.fifo"
  MyQueuePolicy:
    Type: AWS::SQS::QueuePolicy
    DependsOn:
      - MyQueue
    Properties:
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - "events.amazonaws.com"
                - "sqs.amazonaws.com"
                - "sns.amazonaws.com"
            Action:
              - "sqs:SendMessage"
              - "sqs:ReceiveMessage"
            Resource:
              Fn::GetAtt:
                - MyQueue
                - Arn
            Condition:
              ArnEquals:
                aws:SourceArn:
                  Ref: MyTopic
      Queues:
        - Ref: MyQueue
Outputs:
  MyTopicTopicARN:
    Value:
      Ref: MyTopic
  MyQueue:
    Value:
      Fn::Join:
        - " "
        - - 'ARN:'
          - Fn::GetAtt:
              - MyQueue
              - Arn
          - 'URL:'
          - Ref: MyQueue

      
    
```
Beware with indentations. Use a code formatted such as Webstorm to auto-format the document. Otherwise, you will get crazy looking at misleading errors

Deploying using Sam
```
      
sam build
      
    
```
```
      
sam deploy --guided
      
    
```
Resources

Tag: cloudformation

How to write to S3 bucket from Lambda function

AWS resources we need

The Lambda function

The SAM template

Deploy

Resources

Salesforce integration with AWS AppFlow, S3, Lambda and SQS

About Salesforce Change Data Capture

Advantages of using AppFlow approach

Disadvantages of using AppFlow approach

Requirements for Salesforce

Architecture for the solution

How to create the Salesforce Oauth Connection

The Cloudformation template

Debugging

Resources

How to publish a message in SQS from SNS

Overview

The Cloudformation template

Deploying using Sam

Resources