2 posts tagged with "security"

Okay, so you have decided to write your amazing application in Wing. You have enjoyed the benefits of Wing's cloud-oriented programming model and high level abstractions. Everything works great, the queues are queuing, the functions are functional, and the buckets are filling. You are ready to hand this application off to the ops team for deployment when suddenly, you are told there is a problem: the infrastructure doesn't comply with your organizations cloud excellence requirements.

Susan, who is an underappreciated, and sleep deprived platform engineer, tells you that your taggable infra resources must adhere to a rigorous tagging convention. She continues to tell you that all buckets must have versioning and replication enabled. You also gather that she was probably going out for drinks later, since she kept going on about her security group and ciders.

Before you take to Twitter and post a long thread about how Wing is not enterprise ready, you recall the tech lead (we will call him Greg) who gave a presentation about Wing at your organization's last Cloud Center of Excellence (CCoE) meeting. Greg assured everyone that they would be able to use Wing and only focus on the functional aspects of their cloud applications. He said this would be made possible by leveraging the organization's custom Wing plugins. So now all that remains is, to figure out what a Wing plugin is.

Welcome to the Wing Plugin System​

The Wing SDK is hard at work abstracting away the non-functional concerns of your cloud application. Which is great, now you can focus on the business logic of your application and not even care about what cloud this code will run on. However, these abstractions only solve a piece of the puzzle that is the cloud compiler. Inevitably with any production grade deployment, we will need a way to customize the compilation output to meet business requirements. Whether they be security, compliance, or cost optimizations these scenarios will require drilling down bellow the abstractions and into the compiler.

This is where the Wing plugin system comes in as the first steps to opening up hooks into the Wing compilation process. By using these plugin hooks Wing is still able to decouple the functional and non-functional concerns of our applications. Think of it as the SDK handles all functional concerns such as queues, functions, and buckets, while the plugin system handles the non-functional concerns such as encryption, versioning, and security groups.

The plugin system is boosting the Wing toolchain into the next level of cloud development. Unlocking the ability for teams to solve complex real world problems in Wing without compromising their organizations cloud principles. Actually, the plugin system enables organizations to double down and enforce their cloud principles without slowing down innovation.

But Why?...​

I think the "why" is important to talk about for a moment. Why should developers not care about the non-functional requirements of their application when writing code? The answer in my opinion is not that developers should not care about it, it's that they don't want to care about it in most cases. Developers want to focus on innovations and pushing boundaries, not be shackled by the low level details of the cloud. This has been true through the history of software development, that we build abstraction layers on-top of implementation details. Most developers don't want to understand the inner workings of file systems and how they differ between operating systems. We just want to be able to read and write files, thus we have file system abstractions. Then if we need to handle special cases based on operating systems, or CPU architectures we expect the abstraction to give us a way to do that, without rewriting our entire code base.

Though I think it's worth noting that the purpose of the abstraction is not to hide implementation details and make cloud application development more vague, but rather unlock new mental models that drive innovation.

"Being abstract is something profoundly different from being vague … The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise" - Edsger Dijkstra

This is the "why" for our plugin system in Wing, it's intended to be a mechanism that supports teams to be more precise in their cloud applications. If the SDK unlocks this semantic level of thought, then the plugin system protects it.

The Basics of Compiler Plugins​

The plugin system is a simple and powerful way to customize the compilation output of your Wing application. It's comprised of a series of hooks that are called at various stages of the compilation process. In our initial release of the plugin system we have made available 3 hooks: preSynth, postSynth and validate. There are additional hooks that are currently in the think tank, but we will save those for another blog post and focus on what we have available today.

To write a plugin all you need is to implement a JavaScript file, in which you can export one or all of the compiler hooks. Once your plugin is written, you can use the --plugin flag in the wing cli to include it in the compilation process.

wing compile -t tf-aws my-app.w --plugin my-plugin.js

The preSynth hook is executed after the construct tree has been initialized but before the code has been synthesized to produce deployment artifacts. In which our plugins have the opportunity to add and mutate resources in the construct tree.

The postSynth hook's execution is right after synthesis has been completed, and provides a means in which we can manipulate the deployment artifacts (Terraform Config, CloudFormation template, etc).

The validate hook is only executed after all compilation and synthesis has been completed. This is important as this hook is meant to serve as a way to examine and validate deployment artifacts without concerns that some later process will mutate them.

Plugins In Action​

No blog post on a new feature would be complete without a walk-through :) So lets walk through the process of writing our own plugin. Not just any plugin though, but one that will help our favorite underappreciated platform engineer, Susan. As more teams have started adopting Wing in her organization she has realized that she can make use of plugins to help teams meet requirements for deploying applications into the cloud, without asking them to rewrite their code.

She has identified a common use case where her organization has deployed an IAM role into every AWS Account using nested stacks. The existence of this role as a permission boundary for all IAM roles is enforced through AWS organization SCPs (Service Control Policies). Thus, without it no IAM role can be created in the account, this is a cause of friction for teams that want to get their Wing applications deployed quickly. (whew thats a whole lot of things a developer should not have to care about)

She has decided to write a plugin that implements 2 hooks preSynth and validate. During the preSynth hook, she wants to add the required permission boundary to all IAM roles in the construct tree. Then she intends to validate the existence of the permission boundary on all roles during the validate hook. This way teams can know their app will fail to deploy at compile time rather than deploy, making for faster feedback loops.

Susan starts with writing the bare necessities of a plugin. She creates a file named permission-boundary-compliance.js and adds the following code:

// Add permission boundary to all IAM roles
exports.preSynth = function (app) { }

// Validate that all IAM roles have a permission boundary
exports.validate = function (config) { }

She plans to make use of a concept from CDKTF known as Aspects to traverse the construct tree and add the permission boundary. She can safely use this since she knows her intended target will be Terraform for AWS.

const iam_role = require("@cdktf/provider-aws/lib/iam-role");
const cdktf = require("cdktf");

class PermissionBoundaryAspect {
  constructor(permissionBoundaryArn) {
    this.permissionBoundaryArn = permissionBoundaryArn;
  }

  visit(node) {
    if (node instanceof iam_role.IamRole) {
      node.permissionsBoundary = this.permissionBoundaryArn;
    }
  }
}

// Add permission boundary to all IAM roles
exports.preSynth = function (app) {
  if (!process.env.PERMISSION_BOUNDARY_ARN) {throw new Error("env var PERMISSION_BOUNDARY_ARN not set")}
  cdktf.Aspects.of(app).add(new PermissionBoundaryAspect(process.env.PERMISSION_BOUNDARY_ARN))
}

So above we can see she created a new Aspect class that implements the visit method. Each node the aspect visits will be checked to determine if it is an IAM role and if so, set the permission boundary which was passed into the plugin through an environment variable PERMISSION_BOUNDARY_ARN.

Finally for her validate step, she will simply traverse the Terraform config for all IAM roles and check if the permission boundary is set. Even though her preSynth hook will have already done the job, she knows that preSynth is a mutable hook and that another plugin may have altered things after.

// Validate that all IAM roles have a permission boundary
exports.validate = function (config) {
  for (const iamRole of Object.keys(config.resource.aws_iam_role)) {
    const role = config.resource.aws_iam_role[iamRole];
    if (!role.permission_boundary) {
      throw new Error(`Role ${iamRole} does not have a permission boundary`);
    }

    if (role.permission_boundary !== process.env.PERMISSION_BOUNDARY_ARN) {
      throw new Error(`Role ${iamRole} has incorrect permission boundary. Expected: ${process.env.PERMISSION_BOUNDARY_ARN} but got: ${role.permission_boundary}}`);
    }
  }
}

Now Susan can use the plugin in her CD pipelines to ensure that all IAM roles have the correct permission boundary set, without imposing this non-functional requirement on the application developers. Susan will go on to write more plugins to help her organization meet their security and compliance requirements. She is no longer the underappreciated platform engineer we know from the beginning of our blog post, but rather a hero with her own corner office, private parking spot, and an on call pager that never goes off.

Ask Not What Your Plugin Can Do For You...​

This new plugin system is very exciting, and has a lot of possibilities. However, if it is to ever reach its full potential we need your help! If you have some ideas for useful plugins, or thoughts on additional hooks, or even just questions about how to make use of the plugin system, we want to hear from you! Open a pull request or an issue on our GitHub also join our community slack and let us know what you think.

Want to read more about Wing plugins? Check out our plugin documentation for more information on the plugin system. For more code examples visit our plugin code examples

There are two ways to create resources in the cloud: in preflight, or in inflight. In this post, I'll explore what these terms mean, and why I think most cloud applications should avoid dynamically creating resources in inflight and instead stick to managing resources in preflight using tools like IaC.

Today, the cloud computing revolution has made it easier than ever to build applications that scale to meet the demands of users. However, as the cloud has become more prevalent, it has also become more complex.

One of the important questions you'll have to answer in order to build an application with AWS, Azure, or Google Cloud is: how should I create the cloud resources for my application?

For simple applications, you can get away with creating resources by clicking around in the cloud console. But as your application grows, a more structured approach is necessary. Infrastructure as code (IaC) tools like Terraform and CloudFormation have become popular for this purpose.

In general, there are two ways to create cloud resources for an application: before the application starts running, as part of the deployment process, and while the application is running, as part of the data path. We refer to these two phases of the application's lifecycle as preflight and inflight. Clever, ha?

In the cloud ecosystem, many cloud services do not make a hard distinction between APIs that manage resources and APIs that use those resources. For example, in AWS's documentation for SQS, operations like CreateQueue and SendMessage are listed side by side. The same goes for Google Cloud's Pub/Sub service.

However, there are significant differences between these two types of APIs in practice. This post will explore why I believe most cloud applications should avoid dynamically creating resources in inflight and, instead, focus on managing resources in preflight using tools like IaC.

Resource management is hard​

First, dynamic resource creation introduces enormous complexity from a resource management perspective. This is the main reason why the IaC tools were created. Not only is it too cumbersome and error-prone to create large numbers of cloud resources by clicking buttons in your web browser, but it also becomes difficult to reliably maintain, update, and track the infrastructure. This is especially true as you start to pay attention to the cost of your application.

When you use tools like Terraform or CloudFormation, you typically create a YAML file or JSON file that describes resources in a declarative format. These solutions have several benefits:

• By using version control, it's easier to identify where resources came from or when they were changed among different versions of your app (especially across apps and teams).
• Provisioning tools can detect and fix "resource drift" (when the actual configuration of a resource differs from the desired configuration).
• You can estimate the cost of your workload based on the list of resources using tools like infracost.
• It's more straightforward to clean up / spin down your application, since all of the resources in your app are tracked in the file.

When resources are created, updated, and deleted dynamically as part of an application's data path, we lose many of these benefits. I’ve heard of many cases where an application was designed around creating resources dynamically, and entire projects and teams had to be dedicated just to writing code that garbage collects these resources.

There are a few kinds of applications that require dynamic resource creation of course (like applications that provision cloud resources on behalf of other users), but these tend to be the exception to the rule.

Static app architectures are more resilient​

Second, dynamic resource creation can make your application more likely to encounter runtime errors in production. Resource creation and deletion typically requires performing control plane operations on the underlying cloud provider, while most inflight operations only require data plane operations.

Cloud services are more fault tolerant when they only depend on data plane operations as part of the business logic's critical path. This is because even if the control plane of a cloud service has a partial outage (for example, if AWS Lambda functions could not be updated with new code), the data plane can continue running with the last known configuration, even as servers come in and out of service. This property, called static stability, is a desirable attribute in distributed systems, and most cloud platforms are designed around these tradeoffs.

Dynamic resource creation requires broader security permissions​

Lastly, dynamic resource creation means your code needs to have admin-like permissions, which dramatically increases the attack surface for bad actors.

In the cloud, most machines ultimately need some form of network access - whether it’s to connect with other VMs in a cluster, or to connect to other cloud services (like automatically scaling databases and messaging queues).

When resources are statically defined, you can narrowly scope these permissions to define which resources are exposed to the public, which resources can call which endpoints, and even which teams can view sensitive data (and how data accesses are logged and audited).

How to follow best practices... in practice?​

I believe the best way to write applications for the cloud is to define your resources in preflight, and then use them in inflight. That's why Wing, the programming language my team and I are building, encourages developers to create resources in preflight as the easiest path to follow. We think the distinction between preflight and inflight is critical, which is why we've built it into the language itself. For example, if you try to create a resource in a block of code that is labeled with an inflight scope, Wing will produce a compiler error:

bring cloud;

let queue = new cloud.Queue();
queue.on_message(inflight (message: str) => {
  // error: Cannot create the resource "Bucket" in inflight phase.
  new cloud.Bucket();
});

Wing is intended to be a general purpose language, so you'll still be able to make API calls to a cloud providers (through network requests or JavaScript/TypeScript libraries) to dynamically create resources if you really want to. But in these scenarios, Wing won't provide resource management capabilities or generate resource permissions for you, so it would be your responsibility to manage the resource and ensure they get cleaned up.

If you're curious to learn more, check out our getting started guide or join us on our community slack and share what kinds of applications you're building in the cloud! We would love to hear your feedback about this design -- and if you have use case where dynamically creating resources would be helpful, please share it with us through a GitHub issue or on this blog's discussion post! ❤️