Development

CloudFormation to secure and manage apps across multiple clusters and clouds

About CloudFormation

If you are not familiar with CloudFormation, an easy way to define it is AWS CloudFormation is an AWS service that uses template files to automate the setup of resources.

It can also be described as an infrastructure automation or Infrastructure-as-Code (IaC) tool. You can use CloudFormation to automate the configuration of workloads that run on the most popular AWS services, including their managed Kubernetes offering, the Elastic Kubernetes Service, or EKS.

Current Challenges

If your company started on AWS, it is not unusual to use CloudFormation to manage your EKS (Kubernetes) infrastructure, but as you scale, you may need to address a few common issues:

  • How do you define application policies in a consistent and simple way that allows you to scale clusters, applications, and users?
  • How do you make it easy for developers to consume Kubernetes without getting them into the underlying infrastructure complexities?
  • As you grow, deploying clusters across multiple providers, self-hosted, or different cloud providers may become a reality, so how can you still use CloudFormation for that?

Addressing the points above may be challenging because:

  • CloudFormation is tied to your AWS infrastructure. Managing and securing applications on other cloud providers and clusters is not supported. 
  • Deploying and managing applications requires your DevOps team to create and manage Helm chart templates for developers, which is ineffective and time-consuming.
  • A Kubernetes cluster by itself is not an application platform. You need security, monitoring, application observability, and more. Managing those with CloudFormation can be time-consuming and complex. 

Overall, cluster management is becoming a commodity, and you can quickly spin up a cluster using any cloud provider and IaC tool. As more teams adopt a microservice-based architecture, the challenge and big win are in delivering a structured application layer to manage and secure applications effectively anywhere, using your existing stack, including CloudFormation.

Extending CloudFormation

This post will discuss how an application platform with an application-level API, Shipa, can help you quickly and efficiently address the items above.

Here is the high-level architecture of what we will implement:

Using the architecture above, we will connect CloudFormation to Shipa, which will in turn:

  • Allow us to use CloudFormation to implement application policies across multiple clusters in different cloud providers
  • Give developers an easy to create application definition, so we (DevOps) don’t need to keep creating and managing Helm charts and Kustomize templates.
  • Provide application observability, dependency maps, service to service communication, and more, which will allow DevOps, SREs, and Developers to better support applications across clusters.

CloudFormation Provider

You can use the Shipa provider for CloudFormation to have those application-level APIs and platform available to CloudFormation. Here is how you can get started:

Requirement:
You will need the CloudFormation Command Line (cfn) installed in your local machine to complete the steps below. 
You can find details of how to install the CloudFormation CLI here.

Start by cloning the following GitHub repository:

git clone https://github.com/shipa-corp/cloudformation-provider

Then use python to create a virtual environment and install the required dependencies:

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Once you install the dependency requirements, you can build the provider locally:

make build

With the build complete, you can then generate and submit/upload the resource types to CloudFormation:

make submit

With the provider built and the resource types uploaded to your AWS account, we can now break the architecture above down to its different components.

Defining Policies

Most of the current solutions operate at the Kubernetes infrastructure layer, which forces you to develop cluster-level YAML scripts, learn new programming languages, and essentially be tied to the underlying cluster provider and version.

By connecting CloudFormation and Shipa, you can define policies at the application layer without worrying about the underlying cloud provider, cluster version, and others. Because it is a standard definition, you can scale infrastructure, applications, and teams without scaling complexity around enforcing policies.

Below is an example of defining an application policy using CloudFormation and Shipa:

Resources:
  CreateProdPolicy:
    Type: Shipa::Framework::Item
    Properties:
      Name: cf-prod-policy
      ShipaHost: '{{resolve:secretsmanager:ShipaHost}}'
      ShipaToken: '{{resolve:secretsmanager:ShipaToken}}'
      Resources:
        General:
          Setup:
            Provisioner: kubernetes
          Access:
            Append: ["shipa-team"]
          Plan:
            Name: shipa-plan
          Router: traefik
          Security:
            DisableScan: false
            IgnoreCVES: ["CVE-2019-16782", "CVE-2018-16471", "CVE-2018-16470", "CVE-2020-8161"]
            IgnoreComponents: ["glibc", "django"]
          AppQuota:
            Limit: 5
          NetworkPolicy:
            Ingress:
              PolicyMode: deny-all
            Egress:
              PolicyMode: deny-all
            DisableAppPolicies: true
          ContainerPolicy:
            AllowedHosts: ["docker.io/shipasoftware"]

The example above defines policies across RBAC, networking, container registry control, and more. As you can see, the policies defined above are not tied to the underlying cluster provider, cluster version, or ingress controller but instead defined at a scalable level across a hybrid infrastructure stack.

That allows you to easily define policies to address scenarios like multiple environments, different projects, applications, or teams, and have those tied to EKS or different Kubernetes clusters.

Each policy that you create and later bind to a cluster creates a namespace in your cluster:

To run the stack above, you can use the command below:

aws cloudformation create-stack --region us-east-1 
  --template-body "file://policy.yaml" 
  --stack-name "cf-shipa-stack"

Note that the policy stack definition uses 2 variables:

  • ShipaHost
  • ShipaToken

You can store both as secrets using the command below as an example:

aws secretsmanager create-secret --name ShipaHost  
--description "Shipa Cloud URL"            
--secret-string https://target.shipa.cloud:8081

If you are using Shipa Cloud, the value for the ShipaHost variable will always be https://target.shipa.cloud:8081

For the ShipaToken, you can find your token by logging into Shipa using Shipa’s CLI and using the following command:

shipa token show

Connecting Clusters

As mentioned above, you can connect one or multiple policies to a single cluster, and Shipa will create a namespace for each policy.

Later, each application definition will point to a specific policy. Shipa will use this to enforce the policy’s rule and know which namespace and cluster should be used for deployment.

Below is an example of connecting a policy to a cluster using CloudFormation:

Resources:
 AddCluster:
    Type: Shipa::Cluster::Item
    DependsOn:
    - CreateProdPolicy
    Properties:
      Name: gke-dev
      Endpoint:
        Addresses:
          - https://35.230.124.10
        Certificate: '{{resolve:secretsmanager:GKEDevCert}}'
        Token: '{{resolve:secretsmanager:GKEDevToken}}'
      Resources:
        Frameworks:
          - Name: cf-prod-policy
      ShipaHost: '{{resolve:secretsmanager:ShipaHost}}'
      ShipaToken: '{{resolve:secretsmanager:ShipaToken}}'

Note that the definition above requires the same two variables entered before.

It also requires you to enter a cluster certificate and service account token that Shipa can use to connect to your cluster and manage application resources. You can find detailed information about creating a service account and retrieving its token and certificate here.

Deploying Applications

While creating policies and binding them to clusters is mainly owned by the DevOps or Platform Engineering team, developers need an easy way to define and deploy their applications.

Today, most of the time, DevOps or Platform teams create and manage templates for developers because those templates are defined at an infrastructure level. Giving developers an application-level definition enables self-service and faster application delivery.

Here is an example of how developers can use CloudFormation and Shipa to define and deploy applications:

Resources:
 CreateApp:
    Type: Shipa::Application::Item
    Properties:
      Name: aws-prod-app
      Teamowner: shipa-team
      Framework: cf-prod-policy
      ShipaHost: '{{resolve:secretsmanager:ShipaHost}}'
      ShipaToken: '{{resolve:secretsmanager:ShipaToken}}'

  DeployDevApp:
    Type: Shipa::AppDeploy::Item
    DependsOn:
    - CreateApp
    Properties:
      App: aws-prod-app
      Image: docker.io/shipasoftware/bulletinboard:1.0
      ShipaHost: '{{resolve:secretsmanager:ShipaHost}}'
      ShipaToken: '{{resolve:secretsmanager:ShipaToken}}'

The example above will:

  • Create an application called aws-prod-app and bind it to the cf-prod-policy, the one you created in the previous step
  • Deploy the application by passing the Docker image

As part of the deployment, Shipa will automatically enforce all rules defined in the policy this application is bound to. If all checks are acceptable, Shipa will create and deploy all the required Kubernetes objects for that application to run in the destination cluster.

Once deployed, SREs, DevOps, and developers can use Shipa’s application portal to manage their applications, understand dependencies, service-to-service communication, and more.

Conclusion

Hopefully, this introduction of CloudFormation and Shipa can paint an example of how an application platform and API can extend different IaC providers and unlock you from being tied to specific cloud providers.

We will soon be publishing detailed documentation and tutorials for the CloudFormation provider and uploading a publicly available resource type that you can use without the need to build the provider and upload it individually to your account and take your CloudFormation stack multi-cloud and multi-cluster 🙂