Integrating API Gateway with other AWS Services can be pretty important to increase the scope of an API into other services.

What I wanted to achieve was a cheaper upload mechanism for S3. The easiest way to allow upload through API gateway is to call a Lambda for every API call and then upload the payload into an S3 bucket. But this is rather costly if you increase the throughput from a few single call to a few hundred calls a second.

So what I came up with, was to directly invoke the Kinesis Firehose from an API Gateway. That way I could avoid the cost of the Lambda and the cost of a S3 request per API call.

So here an overview picture of what I am about to build. API Gateway to Kinesis Firehose

The easiest part in CloudFormation is the S3 bucket. This one does not require any specific configuration for this to work.

AWSTemplateFormatVersion: '2010-09-09'
Description: Firehose sample
Resources:
  DataBucket:
    Type: AWS::S3::Bucket

Now I configured a Firehose. As a prerequisite I needed a role which would allow the Firehose to actually write into the S3 bucket.

KinesisRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
      - Effect: Allow
        Principal:
          Service:
          - firehose.amazonaws.com
        Action:
        - sts:AssumeRole
    Path: "/"
    Policies:
      - PolicyName: KinesisRolePolicy
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
              - s3:AbortMultipartUpload
              - s3:GetBucketLocation
              - s3:GetObject
              - s3:ListBucket
              - s3:ListBucketMultipartUploads
              - s3:PutObject
            Resource:
              - Fn::GetAtt: [ DataBucket , Arn ]
              - Fn::Join:
                - ""
                - - Fn::GetAtt: [ DataBucket , Arn ]
                  - "/*"

Now the Firehose configuration.

EventFirehose:
  Type: AWS::KinesisFirehose::DeliveryStream
  Properties:
    S3DestinationConfiguration:
      BucketARN:
        Fn::GetAtt: [ DataBucket, Arn ]
      BufferingHints:
        IntervalInSeconds: 60
        SizeInMBs: 10
      CompressionFormat: GZIP
      Prefix: events/
      RoleARN:
        Fn::GetAtt: [ KinesisRole, Arn ]

The API Gateway only consists of one method (without any resources), to keep the sample as simple as possible. Here the API Gateway definition.

ApiGatewayRestApi:
  Type: AWS::ApiGateway::RestApi
  Properties:
    Name:
      Fn::Join:
        - ""
        - - Ref: AWS::StackName
          - "-api"

Now the tricky part for all of this lies in the method definition (specifically the input mapping).

ApiGatewayPostMethod:
  Type: AWS::ApiGateway::Method
  Properties:
    ApiKeyRequired: true #to secure my API I used a simple API key. Otherwise my Firehose would be open to the internet.
    AuthorizationType: NONE
    HttpMethod: POST
    Integration:
      Type: AWS #signal that we want to use an internal AWS service
      Credentials:
        Fn::GetAtt: [ GatewayRole, Arn ] #role for the API to actually invoke the firehose
      Uri:
        Fn::Join:
          - ""
          - - "arn:aws:apigateway:"
            - Ref: AWS::Region
            - ":firehose:action/PutRecord" #this URI basically describes the service and action I want to invoke.
      IntegrationHttpMethod: POST #for kinesis using POST is required
      RequestTemplates:
        application/json: #now the mapping template for an incoming JSON
          Fn::Join:
            - ""
            - - "#set( $key = $context.identity.apiKey )\n" #assign the API key to local variable
              - "#set( $keyname = \"apiKey\" )\n"
              - "#set( $traceidval = $input.params().get(\"header\").get(\"X-Amzn-Trace-Id\"))" #get the trace id to later extract a timestamp of the incoming request
              - "#set( $bodyname = \"body\" )\n"
              - "#set( $traceid = \"traceid\")\n"
              - "#set( $body = $input.body )\n" #assign the request payload to variable
              - "#set( $quote = '\"' )\n"
              - "#set( $b64 = $util.base64Encode(\"{$quote$keyname$quote:$quote$key$quote,$quote$traceid$quote:$quote$traceidval$quote,$quote$bodyname$quote:$body}\") )\n"
              #now encode the payload in base64 to form a valid Firehose request
              - "{\n" #begin of the Firehose request json
              - "\"DeliveryStreamName\": \""
              - Ref: EventFirehose
              - "\",\n"
              - " \"Record\": { \"Data\": \"$b64\" }\n}" #end of the Firehose request json
      RequestParameters: #Firehose requires the content type to not be json, but amz-json
        integration.request.header.Content-Type: "'application/x-amz-json-1.1'"
      IntegrationResponses:
        - StatusCode: 200 #create a default response for the caller
          ResponseTemplates:
            application/json: '{"status":"OK"}'
    MethodResponses:
      - StatusCode: 200
    ResourceId:
      Fn::GetAtt: [ ApiGatewayRestApi , RootResourceId ]
    RestApiId: !Ref ApiGatewayRestApi

Now we still need the Gateway role to be defined.

GatewayRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
      - Effect: Allow
        Principal:
          Service:
          - apigateway.amazonaws.com
        Action:
        - sts:AssumeRole
    Path: "/"
    Policies:
      - PolicyName: GatewayRolePolicy
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
              - firehose:PutRecord
            Resource: "*"

I added a Deployment, so the API would automatically get deployed after it is created.

ApiGatewayDeployment:
  Type: AWS::ApiGateway::Deployment
  DependsOn:
    - ApiGatewayPostMethod
  Properties:
    RestApiId: !Ref ApiGatewayRestApi
    StageName: prod

Here is the overall CloudFormation template:

[api-to-firehose-cf.yml template][1]

[1]:{{ site.url }}/assets/api-to-firehose-cf.yml