programming | FLRNKS

My first scala app

Sat, 10 Oct 2020 11:11:00 +0000

Motivation

In this post I wanted to write about a personal project I started some time ago, with the goal of learning more about Scala. At work, we use Scala quite often to run big data jobs on AWS using Apache Spark. I’ve never used Scala before I joined my current team, and its syntax was very alien to me. However, recently I had the chance to work on a task, where I had to modify a component to use AWS Secrets Manager instead of HashiCorp’s Vault for fetching some secret value at runtime. To my surprise I could complete this work without much struggle with Scala, and afterwards I became eager to learn more. Based on a colleague’s recommendation I started reading a book from Cay S. Horstmann titled Scala for the impatient (2nd edition). I’m making slow but steady progress.

Shortly after starting with the book, I had the idea to start a small project so that I can practice Scala by doing.

The Idea

The idea, like many others before, came while fixing a bug at work. The bug was found within a component written in Scala to interact with the AWS Athena service. It had some neatly written functionality for making queries and waiting for their completion before trying to fetch the results. I thought I would try to write something similar for AWS Systems Manager (SSM). It is a service with few different components, so I decided to focus on Automation Documents that can carry out actions in an automated fashion. For example, the AWS provided SSM document AWS-StartEC2Instance can run any EC2 instance when invoked with the below 2 input parameters:

InstanceId: to specify which EC2 instance you want to start
AutomationAssumeRole: to specify an IAM role which can be assumed by SSM to carry out this action

I realized quite early on, that if I wanted to implement this capability in my Scala app, it needed to be quite generic, so that it could support any Automation Document with an arbitrary number of input parameters. I also wanted it to be able to wait for the execution and report whether it failed or succeeded. Here are the final requirements I came up with:

create 2 separate git repos for:
- a module that’s home for the AWS utility/helper classes
- a module for implementing the CLI App
support extra AWS services such as KMS, Secrets Manager and CloudFormation
utilize localstack for integration testing (when possible)

Initial setup

Firstly, I had to figure out which third-party packages I needed to implement the app according to these simple requirements. To interact with AWS from Scala code, I decided to go with v2 of the official Java SDK for AWS. To implement the CLI app I mainly relied on the picocli Java package, which was a bit less straightforward, but eventually it proved to be a good choice.

Secondly, I have to admit that creating a re-usable scala package from scratch was a rather non-trivial task for me. Most of my programming experience comes from working with in non-JVM based environments so that’s probably no surprise. I initially started out with sbt for build & dependency management, but I was running into issues that I couldn’t solve on my own, so I decided to swap it with maven which was a bit more familiar to me.

Finally, separating the project into two distinct git repositories allowed me to practice versioning and dependency management which I also found very useful:

AWS Scala Utils: https://github.com/florianakos/aws-utils-scala
AWS SSM CLI App: https://github.com/florianakos/aws-ssm-scala-app

The utils module

Creating the utils module that would serve as a kind of glue between the scala CLI app and AWS Systems Manager was actually not as difficult as I thought. This is mostly thanks to the example I’ve seen at work for a similar project with the AWS Athena service.

The core functionality of the utils module when it comes to SSM, is captured in the below functions:

private def executeAutomation(documentName: String, parameters: java.util.Map[String,java.util.List[String]]): Future[String] = {
val startAutomationRequest = StartAutomationExecutionRequest.builder()
.documentName(documentName)
.parameters(parameters)
.build()
Future {
val executionResponse = ssmClient.startAutomationExecution(startAutomationRequest)
logger.info(s"Execution id: ${executionResponse.automationExecutionId()}")
executionResponse.automationExecutionId()
}
}
private def waitForAutomationToFinish(executionId: String): Future[String] = {
val getExecutionRequest = GetAutomationExecutionRequest.builder().automationExecutionId(executionId).build()
var status = AutomationExecutionStatus.IN_PROGRESS
Future {
var retries = 0
while (status != AutomationExecutionStatus.SUCCESS) {
val automationExecutionResponse = ssmClient.getAutomationExecution(getExecutionRequest)
status = automationExecutionResponse.automationExecution.automationExecutionStatus()
status match {
case AutomationExecutionStatus.CANCELLED | AutomationExecutionStatus.FAILED | AutomationExecutionStatus.TIMED_OUT =>
throw SsmAutomationExecutionException(status, automationExecutionResponse.automationExecution.failureMessage)
case AutomationExecutionStatus.SUCCESS =>
logger.info(s"Query finished with status: $status")
case status: AutomationExecutionStatus =>
logger.info(s"SSM Automation execution status: $status, check #$retries.")
Thread.sleep(if (retries <= 3) 2500 else if (retries <= 10) 5000 else 15000)
}
retries += 1
}
}.map(_ => executionId)
}

The first one executeAutomation crafts an execution request and then submits it to AWS, returning its execution ID. This ID can be passed to the waitForAutomationToFinish function that periodically checks in with AWS until the execution is complete. Between subsequent API requests it uses an increasing timeout to prevent API rate-limiting caused by excessive polling.

Testing the utils module

Once I had the core functionality ready I wanted to write integration tests to ensure it works as expected. Instead of having hard-coded AWS credentials or an AWS profile for a real account I wanted to use Localstack that mocks the real AWS API so that you can interact with it. For this reason I slightly tweaked the SsmAutomationHelper class to accept an Optional second argument which can be used while building the SSM API client:

class SsmAutomationHelper(profile: String, apiEndpoint: Option[String]) extends LazyLogging {
private val ssmClient = apiEndpoint match {
case None => SsmClient.builder()
.credentialsProvider(ProfileCredentialsProvider.create(profile))
.region(Region.EU_WEST_1)
.build()
case Some(localstackEndpoint) => SsmClient.builder()
.credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create("foo", "bar")))
.endpointOverride(URI.create(localstackEndpoint))
.build()
}
}

This allowed me to pass http://localhost:4566 when running the integration tests against localstack and have the API calls directed to those mocked endpoints. Previously each mocked service had its own dedicated port, but thanks to a recent change in localstack, now all AWS services can be run on a single port, they call EDGE port.

According to the documentation, SSM is supported in localstack, however I’ve found out that running Automation Documents is feature that is still missing. As a result, I had to run the integration tests against a real AWS account that I set up for such scenarios. I was okay with doing this since there are plenty of built-in Automation Documents provided by AWS that I could safely use for this purpose.

Eventually I decided to encode in the tests AWS-StartEC2Instance & AWS-StopEC2Instance which only required me to set up a dummy EC2 instance which would be the target of these requests. I also added a special Tag to these integration tests so that they are excluded from running when invoked via mvn test but still available to run manually whenever necessary.

CLI App implementation

After running the tests, I was confident that the AWS utils worked correctly, so I started putting together the CLI app. For this, I’ve searched on the web for a third party package and found that it’s not as simple as it is when using Python’s argparse package. I eventually settled with picocli, which is written in Java but can also be used from Scala via the below annotations:

@Command(name = "SsmHelper", version = Array("v0.0.1"), mixinStandardHelpOptions = true, description = Array("CLI app for running automation documents in AWS SSM"))
class SsmCliParser extends Callable[Unit] with LazyLogging {
@Option(names = Array("-D", "--document"), description = Array("Name of the SSM Automation document to execute"))
private var documentName = new String
@Parameters(index = "0..*", arity = "0..*", paramLabel = "<param1=val1> <param2=val2> ...", description = Array("Key=Value parameters to use as Input Params"))
private val parameters: util.ArrayList[String] = null
[...]

According to the original idea, there had to be one constant CLI flag which controlled the name of the AWS Automation Document (--document) and then there had to be a variable number of additional arguments for specifying the Input Parameters required by the given document. The picocli package supported this workflow via the @Option and the @Parameters annotations.

The only thing left was a custom function that would carry out the needed transformation of Input Parameters. The values received in the parameters were in the form of an ArrayList: [<param1=val1>, <param2=val2>, ...] which had to be transformed into a Map: [param1 -> [val1], param2 -> [val2]] by splitting each String on the = character. The desired format was a requirement of the AWS SDK for SSM. After some iterations I ended up with the below function that could do this transformation:

private def process(params: util.ArrayList[String]): util.Map[String, util.List[String]] = {
params.asScala
.map(_.split('='))
.collect { case Array(key, value) => key -> value }
.groupBy(_._1)
.mapValues(_.map(_._2).asJava).asJava
}

Finally, I constructed the below method which utilized the SsmAutomationHelper class from the utils module and passed the two variables provided by picocli to it so it would invoke the necessary Automation Document and wait to retrieve its result via the Await mechanism of Scala:

def call(): Unit = {
val conf = ConfigFactory.load()
val inputParams = process(parameters)
Await.result(SsmAutomationHelper.newInstance(conf).runDocumentWithParameters(documentName, inputParams), 10.minutes)
}

Packaging the CLI app

At this point I was ready with the CLI app and wanted to run it to see how it would function. Before I could run it, I needed to figure out how to package it all into a fat JAR file with all needed dependencies, so that it could be invoked with CLI arguments. I googled around a bit and quickly found the spring-boot-maven-plugin which has the repackage goal that’s just what I needed:

Repackages existing JAR and WAR archives so that they can be executed from the command line using java -jar. With layout=NONE can also be used simply to package a JAR with nested dependencies (and no main class, so not executable).

I only had to add the below lines to my project’s pom.xml:

<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>2.3.2.RELEASE</version>
<configuration>
<layout>JAR</layout>
</configuration>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
</plugin>

Next I just had to run the mvn package command, which invokes the plugin to builds the fat JAR.

Running the CLI app

Once the JAR is available, it can be used via the java -jar ... command with extra arguments to run the any Automation Document such as AWS-StartEC2Instance:

$ ▶ java -jar ./target/scala-cli-app-1.0.0.jar --document=AWS-StartEC2Instance InstanceId=i-0ed4574c5ba94c877 AutomationAssumeRole=arn:aws:iam::{{global:ACCOUNT_ID}}:role/AutomationServiceRole
15:24:41.998 [main] INFO c.f.utils.ssm.SsmAutomationHelper :: Going to kick off SSM orchestration document: AWS-StartEC2Instance
15:24:42.773 [ForkJoinPool-1-worker-29] INFO c.f.utils.ssm.SsmAutomationHelper :: Execution id: <...>
15:24:42.882 [ForkJoinPool-1-worker-11] INFO c.f.utils.ssm.SsmAutomationHelper :: Current status: [InProgress], retry counter: #0
[...]
15:28:01.226 [ForkJoinPool-1-worker-11] INFO c.f.utils.ssm.SsmAutomationHelper :: Current status: [InProgress], retry counter: #21
15:28:16.442 [ForkJoinPool-1-worker-11] INFO c.f.utils.ssm.SsmAutomationHelper :: Execution finished with final status: [Success]
15:28:16.444 [main] INFO com.flrnks.app.SsmCliParser :: SSM execution run took 215 seconds

Seems to be working quite well!

Bonus: running in a container

I thought I would take the above one step further and package the JAR into a java based docker container. This would allow me to forget about the syntax of the java command that I previously used to run the app. Instead, I can hide it in a very minimal Dockerfile:

FROM openjdk:8-jdk-alpine
MAINTAINER flrnks <flrnks@flrnks.netlify.com>
ADD target/scala-cli-app-1.0.0.jar /usr/share/backend/app.jar
ENTRYPOINT [ "/usr/bin/java", "-jar", "/usr/share/backend/app.jar"]

The mvn package command which is used to build the fat JAR will save it into the /target subdirectory, so one can put this Dockerfile into the project’s root and then manually build the docker image by running docker build -t ssmcli .. This will create an image called ssmcli without issues, however I’ve found an awesome plugin called dockerfile-maven-plugin built by Spotify which can automagically take this Dockerfile and turn it into an image based on the plugin configuration:

<plugin>
<groupId>com.spotify</groupId>
<artifactId>dockerfile-maven-plugin</artifactId>
<version>1.4.10</version>
<executions>
<execution>
<id>default</id>
<goals>
<goal>build</goal>
</goals>
<configuration>
<repository>flrnks/ssmcli</repository>
<tag>latest</tag>
</configuration>
</execution>
</executions>
</plugin>

This plugin hooks into the mvn package goal and when it’s executed it will automatically create the docker image:

[INFO] --- spring-boot-maven-plugin:2.3.2.RELEASE:repackage (default) @ scala-cli-app ---
[INFO] Layout: JAR
[INFO] Replacing main artifact with repackaged archive
[INFO]
[INFO] --- dockerfile-maven-plugin:1.4.10:build (default) @ scala-cli-app ---
[INFO] dockerfile: null
[INFO] contextDirectory: /Users/flszabo/Desktop/personal-wrkspc/scala/scala-cli-app
[INFO] Building Docker context /Users/flszabo/Desktop/personal-wrkspc/scala/scala-cli-app
[INFO] Path(dockerfile): null
[INFO] Path(contextDirectory): /Users/flszabo/Desktop/personal-wrkspc/scala/scala-cli-app
[INFO]
[INFO] Image will be built as flrnks/ssmcli:latest
[INFO] Step 1/4 : FROM openjdk:8-jdk-alpine
[INFO] Pulling from library/openjdk
[INFO] Digest: sha256:94792824df2df33402f201713f932b58cb9de94a0cd524164a0f2283343547b3
[INFO] Status: Image is up to date for openjdk:8-jdk-alpine
[INFO] ---> a3562aa0b991
[INFO] Step 2/4 : MAINTAINER flrnks <flrnks@flrnks.netlify.com>
[INFO] ---> Using cache
[INFO] ---> efcc673b4f35
[INFO] Step 3/4 : ADD target/scala-cli-app-1.0.0.jar /usr/share/backend/app.jar
[INFO] ---> 8b2cf76f03c2
[INFO] Step 4/4 : ENTRYPOINT [ "/usr/bin/java", "-jar", "/usr/share/backend/app.jar"]
[INFO] ---> Running in c9633237f9fa
[INFO] Removing intermediate container c9633237f9fa
[INFO] ---> 6db69aa30fb1
[INFO] Successfully built 6db69aa30fb1
[INFO] Successfully tagged flrnks/ssmcli:latest

To test this new docker image I ran the AWS-StopEC2Instance Automation Document and specified the same CLI arguments as before, thanks to the ENTRYPOINT configuration in the Dockerfile. As an extra step I needed to share the AWS profile with the docker container at runtime by using the flag -v ~/.aws:/root/.aws:

$ ▶ ddocker run --rm -v ~/.aws:/root/.aws flrnks/ssmcli --document=AWS-StopEC2Instance InstanceId=i-0ed4574c5ba94c877 AutomationAssumeRole=arn:aws:iam::{{global:ACCOUNT_ID}}:role/AutomationServiceRole
17:18:59.541 [main] INFO c.f.utils.ssm.SsmAutomationHelper :: Going to kick off SSM orchestration document: AWS-StopEC2Instance
17:19:00.789 [ForkJoinPool-1-worker-13] INFO c.f.utils.ssm.SsmAutomationHelper :: Execution id: <...>
17:19:00.966 [ForkJoinPool-1-worker-11] INFO c.f.utils.ssm.SsmAutomationHelper :: Current status: [InProgress], retry counter: #0
17:19:03.564 [ForkJoinPool-1-worker-11] INFO c.f.utils.ssm.SsmAutomationHelper :: Execution finished with final status: [Success]
17:19:03.568 [main] INFO com.flrnks.app.SsmCliParser :: SSM execution run took 5 seconds

One may say that typing that long docker run ... command above takes longer than typing java -jar ./target/scala-cli-app-1.0.0.jar ... but I would argue that running it inside a docker container has its valid use-cases as well. It allows for controlled setup of the runtime environment and prevents dependency issues too!

Conclusion

This project has allowed me to learn much more than I initially expected. I learnt a lot about Scala, which was the original goal, but I also gained valuable experience with Maven, its plugin ecosystem and of course with Java as well. I hope whoever reads this post will find something useful in it too!

Testing Terraform Modules

Sun, 12 Jul 2020 11:11:00 +0000

Intro

I first head of Terraform about 1 year ago while working on an assignment for a job interview. The learning curve was steep, and I still remember how confused I was about the syntax of HCL that resembled JSON but was not exactly the same. I also remember hearing about the concept of Terraform Modules, but for the assignment it was not needed, so I skipped it for the time being.

Fast forward to present day, I’ve had a good amount of exposure to Terraform Modules at work, where we use them to provision resources on AWS in a standardized and rapid fashion. In order to broaden my knowledge on Terraform Modules, I decided to create an exercise in which I created two TF Modules with using version 0.12 of Terraform. In this post I wanted to describe these two Terraform Modules and how I went about testing them to ensure they did what they were meant to.

What is a Terraform Module

According to official documentation a Terraform module is simply a container for multiple resources that are defined and used together. Terraform Modules can be embedded in each other to create a hierarchical structure of dependent resources. To define a Terraform Module one needs to create one or more Terraform files that define some input variables, some resources and some outputs. The input variabls are used to control properties of the resources, while the outputs are used to reveal information about the created resources. These are often organized into such structure as follows:

variables.tf defining the Terraform variables
main.tf creating the Terraform resources
output.tf listing the Terraform outputs

Note that the above is just an un-enforced convention, it simply makes it easier to get a quick understanding about a Terraform Module. As an example, if an organization needs to have their AWS S3 buckets secured with the same policies to protect their data, they can embed these security policies in a TF Module and then prescribe its use within the organization to enable those security policies automatically. Next up is an example of just that.

The Secure-Bucket TF Module

The first of the 2 Terraform Modules is tf-module-s3-bucket which can be used to create an S3 bucket in AWS that is secured to a higher degree, so that it may be suitable for storing highly sensitive data. The security features of the bucket consists of:

filtering on Source IPs that can access its contents
enforcing encryption at rest (KMS) and in transit
object-level and server access logging enabled
filtering on IAM principals based on official docs

When using this module, one can define a list of IPs, and a list of IAM Principals to control who and from which networks can access the contents of the bucket. These restrictions are written into the Bucket Policy, which is considered a resource-based policy that always takes precendence over Identity based policies, so it does not matter if an IAM Role has specific permission granted to access the bucket, if the bucket’s own Bucket Policy denies the same access. Below is a good overview of the whole evaluation logic of AWS IAM:

In addition, server-access and object-level logging can be enabled as well to improve the bucket’s level of auditability. Altogether, these settings can greatly elevate the security of data in the S3 bucket that was created by this module.

The S3-AuthZ TF Module

This 2nd Terraform Module is called tf-module-s3-auth and it was written to in part to complement the other one used to create an S3 bucket. The aim of this module is to help with the creation of a single IAM policy that can cover the S3 and KMS permissions needed for a given IAM Principal. The motivation behind this module comes from some difficulties I’ve faced at work which meant that some IAM Roles we used had too many policies attached. For further reference see the AWS docs on this.

The Bucket Policy that is crafted by the first TF Module allows the definition of list of IAM Principals that are allowed to interact with the bucket. With this TF module one can actually define the particular S3 actions that those IAM Principals CAN carry out on the data in the bucket. Additionally, this TF module can also be used allow KMS actions on the KMS keys that are protecting the data at rest in the bucket.

Untested code is broken code

With infrastructure-as-code, just as with normal code, testing is often an afterthought. However, it seems to be catching on more and more nowadays. Nothing shows this better than the amount of search results in Google for Infrastructure as Code testing: 235.000.000 as of today (15.8.2020). While Infrastructure as Code is a much broader topic with many other interesting projects, this post will have a sole focus on Terraform. With Terraform, a good step in the right direction is as simple as running terraform validate that can catch silly mistakes and syntax errors and provide feedback such as below:

Error: Missing required argument
on main.tf line 107, in output "s3_bucket_name":
107: output "s3_bucket_name" {
The argument "value" is required, but no definition was found.

In addition to the terraform validate option, many IDEs such as IntelliJ, already have plugins that can alert to such issues, so I find myself not using it so often. However, it’s still nice to have this feature built into the terraform executable!

Once all syntax errors are fixed, the next stage of testing can continue with the terraform plan command. This command uses terraform state information (local or remote) to figure out what changes are needed if the configuration is applied. This is truly very useful in showing in advance what will be created or destroyed. However, a successful terraform plan can still result in a failed deployment because some constraints cannot be verified without making the actual API calls to the Cloud Service Provider. The terraform plan command does not make any actual API calls, it only computes the difference that exist between the Terraform Code vs. the Terraform State (local or remote). The failures are usually very provider specific.

data "aws_iam_policy_document" "Deny-Non-CiscoCidr-S3-Access" {
statement {
sid = "Deny-All-S3-Actions-If-Not-In-IP-PrefixList"
effect = "Deny"
actions = [ "s3:*" ]
resources = [ "*" ]
condition {
test = "NotIpAddress"
variable = "aws:SourceIp"
values = local.ip_prefix_list
}
}
}

This Terraform Code is syntactically correct nd passes the terraform validate, and terraform plan produces a valid plan. However, it still fails at the terraform apply stage because AWS has a restriction on the sid: For IAM policies, basic alphanumeric characters (A-Z,a-z,0-9) are the only allowed characters in the Sid value. This constraint is never checked before terraform apply is called, at which point it is going to fail the whole action with the below error:

An error occurred: Statement IDs (SID) must be alpha-numeric. Check that your input satisfies the regular expression [0-9A-Za-z]*

Such types of errors can only be caught when making real API calls to the Cloud Service Provider (or to a truly identical mock of the real API) which will validate the calls and return errors if any are found. Next I will go into some details on how I went about testing the 2 Terraform Modules I wrote.

Manual Testing via AWS

This most rudimentary form of testing can be done by setting up a real project that imports and uses the two Terraform modules. This test can be found in my repository’s test/terraform/aws/ directory. For this to work properly the AWS provider has to be set up with real credentials, which is beyond the scope of this post. I also opted to use S3 as TF state backend storage but this is optional, it can just ass well store the state locally in a .tfstate file.

First, terraform has to be initialized which will trigger the download of the AWS Terraform Provider via terraform init. Next, the changes can be planned and applied via terraform plan & apply respectively. It’s interesting to note that a complete terraform apply takes close to 1 minute to complete:

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.
Outputs: [...]
real 0m49.090s
user 0m3.532s
sys 0m1.929s

Once the terraform apply is complete one can make manual assertions whether it went as expected based on the outputs (if any) and by manually inspecting the resources that were created. While this can be good enough for new setups, it may be not so good when an already deployed project has to be modified and one needs to make sure the changes will not have any undesired side effects.

Manual Testing via localstack

In order to save time (and some costs), one may also consider using localstack which replicates most of the AWS API and its features to enable faster and easier development and testing. It’s important to note that it only works if one is an AWS customer. In an earlier post I’ve already written on how to set it up, so I will not repeat it here. The most important thing is to enable S3, IAM and KMS services in the docker-compose.yaml by setting this environment variable: SERVICES=s3,kms,iam so the corresponding API endpoints are turned on.

The Terraform files I wrote for testing with on real AWS can be re-used for testing with localstack with some tweaks, for more detail look to test/terraform/localstack/ folder in my repository. Then it’s just a matter of running terraform init followed by a terraform plan & apply to create the fake resources in Localstack.

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.
Outputs: [ ... ]
real 0m11.649s
user 0m3.589s
sys 0m1.580s

Notice that this time the terraform apply took only about 10 seconds, which is considerably faster than using the real AWS API.

Automating tests via Terratest

As I’ve shown, running tests via Localstack can be much faster on average, but sometimes a project may require the use of some AWS services that are not supported by Localstack. In this case it becomes necessary to run tests against the real AWS API. For such situations I recommend terratest from Gruntwork.io, which is a Go library that provides capabilities to automate tests.

It still requires a terraform project to be set up, as described in Manual Testing via AWS, however having the ability to formally define and verify tests can greatly increase the confidence that the code being tested will function the way it’s supposed to. In the test I implemented some assertions on the output values of the terraform apply as well as about the existence of the S3 bucket just created. In addition, the Go library also provides ways to verify the AWS infrastructure setup, by making HTTP calls or SSH connections. This can be a pretty powerful tool.

This terratest setup can be found in my repo under test/go/terraform_test.go.

Running this test takes considerably longer than either of the two previous ones, but the advantage is that this can be easily automated and integrated into a CI/CD build where it can verify on-demand that the TF code still works as intended, even if there were some changes.

▶ go test
TestTerraform 2020-08-09T21:46:22+02:00 logger.go:66: Terraform has been successfully initialized!
...
TestTerraform 2020-08-09T21:47:30+02:00 logger.go:66: Apply complete! Resources: 7 added, 0 changed, 0 destroyed.
...
TestTerraform 2020-08-09T21:48:08+02:00 logger.go:66: Destroy complete! Resources: 7 destroyed.
...
PASS
ok github.com/florianakos/terraform-testing/tests 116.670s

The basic idea of terratest is to automate the process or creation and cleanup of resources for the purposes of tests. To avoid name clashes with existing AWS resources, it’s a good practice to append some random strings to resource names as part of the test, so they are not going to fail due to unique name constraints.

Conclusion

In this post I have shown what options are available for testing a Terraform Module in local or remote settings. If one only works with AWS services then Localstack can be a great tool for quick local tests during development, while terratest from Gruntwork can be a great help with codifying and automating such tests that run against the real AWS Cloud from your favourite CI/CD setup.

Identity & Access Management

Mon, 03 Feb 2020 11:11:00 +0000

INTRODUCTION

In this post I show how the Identity and Access Management service in the AWS Public Cloud works to secure resources and workloads. It is a very important topic, because it underpins all of the security that is needed for hosting one’s resources in the public cloud.

At the end of the day, the cloud is just a concept that offers a convenient illusion of dedicated resources, but in reality it’s just some process that runs on someone else’s hardware, so one has to be absolutely sure about security before trusting it and running their business-critical workloads on it.

It is enough to do a quick google search for unsecured s3 bucket to see plenty of examples of administrators failing to properly harden and configure their AWS resources, and falling victim to accidental disclosure of often business-critical information.

IAM exists in the realm of AWS Cloud as a standalone service, providing various ways in which access to resources and workloads can be restricted. For example, if someone has an S3 bucket for storing arbitrary data, one can use IAM policies to restrict access to data stored in the bucket based on various criteria such as user identity, connection source IP, VPC environment and so on. S3 is a convenient service to demonstrate IAM capabilities, because it is very easy to grasp the result of restrictions: access to files in an S3 bucket is either granted or denied.

HOW IT WORKS

In order to illustrate how IAM works, I decided to create a Python Lambda function, which is just an AWS service offering server-less functions, and implemented a routine that tries to access some data stored in a particular S3 bucket. By default the Lambda starts running with an IAM role that has only read-only permission to the bucket. This is verified by making an API call with the boto3 package, which returns without any error. Next the Lambda tries to write some new data to the bucket, but this fails because the IAM role is not equipped with Write permission to the S3 bucket.

To mitigate this problem, I use boto3 to make an AWS Secure Token Service ( STS) call and assume a new role which is equipped with the necessary read-write access. Using this new role the program demonstrates that it can write to the bucket as expected. Below is a sample output of the Lambda Function in action:

=== Checking IAM Identity ===
ARN: arn:aws:sts::ACCOUNT_ID:assumed-role/Base-Lambda-Custom-Role/lambda

=== Testing Read access to S3 file in bucket ===
{
 "field1": true,
 "field2": 1.4107917E7
}

=== Testing Write access to S3 bucket ===
Error: AccessDenied!

=== Assumed New IAM Identity ===
ARN: arn:aws:sts::ACCOUNT_ID:assumed-role/S3-RW-Role/lambda

=== Testing Write access to S3 bucket (using new role) ===
... file was written successfully!

To get a better understanding how this all worked in code, feel free to check out the source code repository in Github ( link). Because I am a big fan of Terraform, I defined all resources (S3, IAM, Lambda) in code which makes it very simple and straightforward to deploy and test the code if you feel like!

ADVANCED IAM

Besides providing the basic functionality to restrict access to resources base on user identity, there are some cool and more advanced features of AWS IAM that I wanted to touch upon. For example, to show how simple it is to give read-only permissions to a bucket for an IAM role:

data "aws_iam_policy_document" "s3_ro_access_policy_document" {
statement {
effect = "Allow"
actions = [
"s3:GetObject",
"s3:ListBucket",
]
resources = [
"arn:aws:s3:::my-bucket",
"arn:aws:s3:::my-bucket/*"
]
}
}
resource "aws_iam_policy" "s3_ro_access_policy" {
name = "S3-ReadOnly-Access"
policy = data.aws_iam_policy_document.s3_ro_access_policy_document.json
}
resource "aws_iam_role_policy_attachment" "Allow_S3_ReadOnly_Access" {
role = aws_iam_role.aws_custom_role_for_lambda.name
policy_arn = aws_iam_policy.s3_ro_access_policy.arn
}
resource "aws_iam_role" "aws_s3_readwrite_role" {
name = "S3-RW-Role"
description = "Role to allow full RW to bucket"
}

Full source code on GitHub.

With this short Terraform code, I created a role, and assigned an IAM policy to it, which has RO access to my-bucket resource in S3. To spice this up a bit, it is possible to add extra restrictions based on various elements of the request context to restrict access based on Source IP for example:

data "aws_iam_policy_document" "s3_ro_access_policy_document" {
statement {
effect = "Deny"
actions = [
"s3:*"
]
resources = [ "*"]
condition {
test = "IpAddress"
variable = "aws:SourceIp"
values = [ "192.168.2.0/24" ]
}
}
}

All of a sudden, even if the user who makes the request to S3 has correct credentials, but is connecting from a subnet which is outside the one specified above, the request will be denied! This can be very useful for example, when trying restricting access to resources to be possible only from within a corporate network with specific CIDR range.

One small issue with this source IP restriction is that it can cause issues for certain AWS services that run on behalf of a principal/user. When using the AWS Athena service for example, triggering a query on data stored in S3 means Athena will make S3 API requests on behalf of the user who initiated the Athena query, but will have a source IP address from some Amazon AWS CIDR range and the request will fail. For this purpose, there is an extra condition that can be added to remediate this issue:

data "aws_iam_policy_document" "s3_ro_access_policy_document" {
statement {
effect = "Deny"
actions = [
"s3:*"
]
resources = [ "*"]
condition {
test = "IpAddress"
variable = "aws:SourceIp"
values = [ "192.168.2.0/24" ]
}
condition {
test = "Bool"
variable = "aws:ViaAWSService"
values = [ "false" ]
}
}
}

The aws:viaAWSService = false condition will ensure that this Deny will only take effect when the request context does not come from an AWS Service Endpoint. For additional info on what other possibilities exist that can be used to grant or deny access, please consult the AWS documentation.

CONCLUSION

In this post I demonstrated how to use the boto3 python package to make AWS IAM and STS calls to access resources in the AWS cloud protected by IAM policies. I also discussed some advanced features of AWS IAM that can help you implement more granular IAM policies and access rights. The linked repository also contains an example which may be run locally and does not need the Lambda function to be created (it still, however, requires the Terraform resources to be deployed).

Cloud Service Testing

Fri, 17 Jan 2020 11:11:00 +0000

In this blog post I discuss a recent project I worked on to practice my skills related to AWS, Python and Datadog. It includes topics such as integration testing using pytest and localstack; running Continuous Integration via Travis-CI and infrastructure as code using Terraform.

Intro

For the sake of this blog post, let’s assume that a periodic job runs somewhere in the Cloud, outside the context of this application, which generates a file with some meta-data about the job itself. This data includes mostly numerical values, such as the number of images used to train an ML model, or the number of files processed, etc. This part is depicted on the below diagram as a dummy Lambda function that periodically uploads this metadata file to an S3 bucket with random numerical values.

When this file is uploaded, an event notification is sent to the message queue. The goal of the Python application is to periodically drain these messages from the queue. When the application runs, it fetches the S3 file referenced in each SQS message, parses the file’s contents and submits the numerical metrics to DataDog for the purpose of visualisation and alerting.

Testing

Since the application interacts with two different APIs (AWS & Datadog), I figured it was a good idea to create integration tests that can be run easily via some free CI service (e.g.: Travis-CI.org). When writing the integration tests, I opted to create a simple mock class for testing the interaction with the Datadog API, and chose to rely on localstack for testing the interaction with the AWS API.

Thanks to localstack I could skip creating real resources in AWS and instead use free fake resources in a docker container, that mimic the real AWS API close to 100%. The AWS SDK called boto3 is very easy to reconfigure to connect to the fake resources in localstack with the endpoint_url= parameter.

In the following sections I go through different phases of the project:

coding the python app
mocking Datadog statsd client
setting up AWS resources in localstack
creating integration tests
Travis-CI integration
running the datadog-agent locally
setting up real AWS resources
live testing

~ Coding the python app ~

The code is mainly composed of two Python classes with methods to interact with AWS and DataDog. The CloudResourceHandler class has methods to interact with S3 and SQS, which can be replaced in integration-tests with preconfigured boto3 clients for localstack.

The MetricSubmitter class uses the CloudResourceHandler internally and offers some additional methods for sending metrics to DataDog. Internally it uses statsd from the datadog python package, which can be replaced via dependency injection in integration tests with a mock statsd class that I created to test its interaction with the Datadog API.

To connect to the real AWS & Datadog APIs (via a preconfigured local datadog-agent) there needs to be two environment variables specified at run-time:

STATSD_HOST set to localhost
SQS_QUEUE_URL set to the URL of the Queue

os.environ['STATSD_HOST'] = 'localhost'
os.environ['SQS_QUEUE_URL'] = 'https://sqs.eu-central-1.amazonaws.com/????????????/cloud-job-results-queue'
session = boto3.Session(profile_name='profile-name')
MetricSubmitter(statsd=datadog_statsd,
sqs_client=session.client('sqs'),
s3_client=session.client('s3')).run()

In addition, it also requires a preconfigured AWS profile in ~/.aws/credentials which is necessary for boto3 to authenticate to AWS:

[profile-name]
aws_access_key_id = XXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
region = eu-central-1

But before running it, let’s set up some integration tests!

~ Mocking Datadog statsd client ~

In truth, the application does not interact directly with the Datadog API, but rather it uses statsd from the datadog python package, which interacts with the local datadog-agent, which in turn forwards metrics and events to the Datadog API.

To test this flow that relies on statsd, I created a class called DataDogStatsDHelper. This class has 2 functions (gauge/event) with identical signatures to the real functions from the official datadog-statsd package. However, the mock functions do not send anything to the datadog-agent. Instead, they accumulate the values they were passed in local class variables:

class DataDogStatsDHelper:
event_title = None
event_text = None
event_alert_type = None
event_tags = None
event_counter = 0
gauge_metric_name = None
gauge_metric_value = None
gauge_tags = None
gauge_counter = 0
def event(self, title, text, alert_type=None, aggregation_key=None, source_type_name=None,
date_happened=None, priority=None, tags=None, hostname=None):
...
def gauge(self, metric, value, tags=None, sample_rate=None):
...

When the MetricSubmitter class is tested, this mock class is injected instead of the real statsd class, which enables assertions to be made and compare expectations with reality.

~ AWS resources in localstack ~

To test how the python app integrates with S3 and SQS, I decided to use loalstack, running in a Docker container. To make it simple and repeatable, I created a docker-compose.yaml file that allows the configuration parameters to be defined in YAML:

version: '3.2'
services:
 localstack:
 image: localstack/localstack:latest
 container_name: localstack
 ports:
 - '4563-4599:4563-4599'
 - '8080:8080'
 environment:
 - SERVICES=s3,sqs
 - AWS_ACCESS_KEY_ID=foo
 - AWS_SECRET_ACCESS_KEY=bar

The resulting fake AWS resources are accessible via different ports on localhost. In this case, S3 runs on port 4572 and SQS on port 4576. Refer to the docs on GitHub for more details on ports used by other AWS services in localstack.

It is important to note that when localstack starts up, it is completely empty. Thus, before the integration tests can run, it is necessary to provision the S3 bucket and SQS queue in localstack, just as one would normally do it when using real AWS resources.

For this purpose, it’s possible to write a simple bash script that can be called from the localstack container as part of an automatic init script:

aws --endpoint-url=http://localhost:4572 s3api create-bucket --bucket "bucket-name" --region "eu-central-1"
aws --endpoint-url=http://localhost:4576 sqs create-queue --queue-name "queue-name" --region "eu-central-1" --attributes "MaximumMessageSize=4096,MessageRetentionPeriod=345600,VisibilityTimeout=30"

However, for the sake of making the integration-tests self-contained, I opted to integrate this into the tests as part of a class setup phase that runs before any tests and sets up the required S3 bucket and SQS queue:

@classmethod
def setUpClass(cls):
cls.ls = LocalStackHelper()
cls.ls.get_s3_client().create_bucket(Bucket=cls.s3_bucket_name)
cls.ls.get_sqs_client().create_queue(QueueName=cls.sqs_queue_name)

~ Creating integration tests ~

As a next step I created the integration tests which use the fake AWS resources in localstack, as well as the mock statsd class for DataDog. I used two popular python packages to create these:

unittest which is a built-in package
pytest which is a 3rd party package

Actually, the test cases only use unittest, while pytest is used for the simple collection and execution of those tests. To get started with the unittest framework, I created a python class and implemented the test cases within this class:

import unittest
from app.utils.datadog_fake_statsd import DataDogStatsDHelper
from app.utils.localstack_helper import LocalStackHelper
from app.submitter import MetricSubmitter
class ProjectIntegrationTesting(unittest.TestCase):
@classmethod
def setUpClass(cls):
...
def setUp(self):
...
def test_ddg_submitter_valid_payload(self):
...
def test_ddg_submitter_invalid_payload(self):
...
def test_aws_handler_invalid_s3key(self):
...
def test_aws_handler_valid_s3key(self):
...

In the setUpClass method, a few things are taken care of before tests can be executed:

define class variables for the bucket & the queue
create SQS & S3 clients using localstack endpoint url
provision needed resources (Queue/Bucket) in localstack

To test the interaction with DataDog via the statsd client, the submitter app is executed, which stores some values in the mock statsd class’s internal variables, which are then used in assertions to compare values with expectations.

The other tests inspect the behaviour of the CloudResourceHandler class. For example, one of the assertions tests whether the .has_available_messages() function returns false when there are no more messages in the queue.

A nice feature of unittest is that it’s easy to define tasks that need to be executed before each test, to ensure a clean slate for each test. For example, the code in the setUp method ensures two things:

the fake SQS queue is emptied before each test
class variables of the mock DataDog class are reset before each test

Theoretically, it would be possible to run the test by running pytest -s -v in the python project’s root directory, however the tests rely on localstack, so they would fail…

~ Travis-CI integration ~

So now that the integration tests are created, I thought it would be really nice to have them automatically run in a CI service, whenever someone pushes changes to the Git repo. To this end, I created a free account on travis-ci.org and integrated it with my github rep by creating a .travis.yaml file with the below initial content:

os: linux
language: python
python:
 - "3.8"
services:
 - docker
script:
 - {...}

However, I still needed a way to run localstack and then execute the integration tests within the CI environment. Luckily I found docker-compose to be a perfect fit for this purpose. I had already created a yaml file to describe how to run localstack, so now I could just simply add an extra container that would run my tests. Here is how I created a docker image to run the tests via docker-compose:

FROM python:3.8-alpine
WORKDIR /app
COPY ./requirements-test.txt ./
RUN apk add --no-cache --virtual .pynacl_deps build-base gcc make python3 python3-dev libffi-dev \
 && pip3 install --upgrade setuptools pip \
 && pip3 install --no-cache-dir -r requirements-test.txt \
 && rm requirements-test.txt
COPY ./utils/*.py ./utils/
COPY ./*.py ./
ENV LOCALSTACK_HOST localstack
ENTRYPOINT ["pytest", "-s", "-v"]

It installs the necessary dependencies to an alpine based python 3.8 image; adds the necessary source code, and finally executes pytest to collect & run the tests. Here are the updates I had to make to the docker-compose.yaml file:

version: '3.2'
services:
 localstack:
 {...}
 integration-tests:
 container_name: cloud-job-it
 build:
 context: .
 dockerfile: Dockerfile-tests
 depends_on:
 - "localstack"

Docker Compose auto-magically creates a shared network to enable connectivity between the defined services, which can call one-another by name. So when the tests are running in the cloud-job-it container, they can use the hostname localstack to create the boto3 session via the endpoint url to reach the fake AWS resources.

For easier to creation of AWS clients via localstack, I used a package called localstack-python-client, so I don’t have to deal with port numbers and low level details. However, this client by default tries to use localhost as the hostname, which wouldn’t work in my setup using docker-compose. After digging through the source-code of this python package, I found a way to change this by setting an environment variable named LOCALSTACK_HOST.

As a final step, I just had to add two lines to complete to the .travis.yaml file:

script:
 - docker-compose up --build --abort-on-container-exit
 - docker-compose down -v --rmi all --remove-orphans

Thanks to the --abort-on-container-exit flag, docker-compose will return the same exit code which is returned from the container that first exits, which first this use-case perfectly, as the cloud-job-it container only runs until the tests finish. This way the whole setup will gracefully shut down, while preserving the exit code from the container, allowing the CI system to generate an alert if it’s not 0 (meaning some test failed).

~ Running the datadog-agent locally ~

Note: while Datadog is a paid service, it’s possible to create a trial account that’s free for 2 weeks, without the need to enter credit card details. This is pretty amazing!

Now that the integration tests are automated and passing, I wanted to run the datadog-agent locally, so that I can test the python application with some real data that was to he submitted to Datadog via the agent. Here is an article that was particularly useful to me, with instructions on how the agent should be set up.

While the option of running it in docker-compose was initially appealing, I eventually decided to just start it manually as a long-lived detached container. Here is how I went about doing that:

DOCKER_CONTENT_TRUST=1 docker run -d \
 --name dd-agent \
 -v /var/run/docker.sock:/var/run/docker.sock:ro \
 -v /proc/:/host/proc/:ro \
 -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
 -e DD_API_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX \
 -e DD_SITE="datadoghq.eu" \
 -e DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true \
 -p 8125:8125/udp \
 datadog/agent:7

Most notable of these lines is the DD_API_KEY environment variable which ensures that whatever data I send to the agent is associated with my own account. In addition, since I am closest to the EU region, I had to specify the endpoint via the DD_SITE variable. Also, because I want the agent to accept metrics from the python app, I need to turn on a feature via the environment variable DD_DOGSTATSD_NON_LOCAL_TRAFFIC, as well as expose port 8125 from the docker container to the host machine:

 ▶ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
477cb2ea74b2 datadog/agent "/init" 3 days ago Up 3 days (healthy) 0.0.0.0:8125->8125/udp, 8126/tcp dd-agent

All seems to be well!

~ Deploying real AWS resources ~

Here I briefly discuss how I deployed some real resources in AWS to see my application running live. In a nutshell, I set the infra up as code in Terraform, which greatly simplified the whole process. All the necessary files are collected in a directory of my repository:

variables.tf defines some variables used in multiple places
init.tf initialisation of the AWS provider and definition of AWS resources
outputs.tf defines some values that are reported when deployment finishes

The first and last files are not very interesting. Most of the interesting stuff happens in the init.tf, which defines the necessary resources and permissions. One extra resource not mentioned before, is an AWS Lambda function, which gets executed every minute and is used to upload a JSON file to the S3 bucket. This acts as a random source of data, so that the python app has some work to do without manual intervention.

~ Live testing ~

Now that all parts seem to be ready, it’s time to run the main python app using the real S3 bucket and SQS queue, as well as the local datadog-agent. The console output provides some hints whether it’s able to pump the metrics from AWS to a DataDog:

▶ python3 submitter.py
Initializing new Cloud Resource Handler with SQS URL - https://.../cloud-job-results-queue
Processing available messages in SQS queue:
- sending data to DataDog via statsd/datadog-agent.
- removing message from SQS (AQEBO37smPPHg6OIqbh3HMu3g...)
- ...
- sending data to DataDog via statsd/datadog-agent.
- removing message from SQS (AQEBV0/JzMVEP6k5kBmx2kvGn...)
No more messages visible in the queue, shutting down ...
Process finished with exit code 0

Next, I checked my DataDog account to see whether the metric data arrived. For this I created a custom Notebook with graphs to display them:

All seems to be well! The deployed AWS Lambda function has already run a few times, providing input data for the python app, which were successfully processed and forwarded to Datadog. As seen on the Notebook above, it is really easy to display metric data over time about any recurring workload, which can provide pretty useful insights into those jobs.

Furthermore, since DataDog also submission of events it becomes possible to design dashboards and create alerts which trigger based on mor complex criteria, such as the presence or lack of events over certain periods of time. One such example can be seen below:

This is a so-called screen-board which I created to display the status of a Monitor that I set up previously. This Monitor tracks incoming events with the tag cloud_job_metric and generates an alert, if there is not at least one such event of type success in the last 30 minutes. The screen-board can be exported via a public URL if needed, or just simply displayed on a big screen somewhere in the office.

Conclusions

In this post I discussed a relatively complex project with lots of exciting technology working together in the realm of Cloud Computing. In the end, I was able to create DashBoards and Monitors in DataDog, which can ingest and display telemetry about AWS workloads, in a way that makes it useful to track and monitor the workloads themselves.

RunCode.ninja Challenges

Sat, 11 Jan 2020 11:11:00 +0000

This post was born on a misty saturday morning, while slowly sipping some good quality coffe in a Prague café. The last several days after work was over I spent solving programming challenges on runcode.ninja and I thought it would be nice to share my experience and spread the word about it.

RunCode.ninja

I can’t really recall how I discovered this website in the first place… All I remember is that I was really into the simplistic idea of it all. The basic idea for most of the challenges goes something like this:

check problem description
inspect any sample input (if any)
write your program locally
test on sample input (if any)
submit source code to the evaluation platform

If all went well, you will get feedback within a few seconds whether the submitted code worked correctly for the given task at hand. If it didn’t, then you can turn to their FAQ for some advice. It definitely has some useful info, however if all else fails, you can also contact the team behind the platform on their slack channel. They are really friendly people so be sure to respond to their effort in kind!

Another nice thing about their platform is that they categorized all their challenges (119 in total as of now) into nice categories such as binary, encoding, encryption, forensics, etc. which allows you to select what you are interested in. When I started out, I was first aiming to complete the challenges in Easy which offers a combination of relatively easy challenges from math, text-parsing, encoding and other categories.

As it currently stands, I rank 155 our of around ~2400 registered users, which seems quite impressive at first, but I suspect there may be quite a few inactive accounts in their database. Also, there are some hardcore people who have already completed all their challenges that seems quite impressive. If only a few rainy and cold weekends I could spend working on these, I would probably catch up soon!

Last but not least, their platform is set up to interpret a several different programming languages, so you can choose to solve them in the language you are most comfortable with. Once you solve a challenge, you can access its write-ups which provide some very useful inspiration on how others have solved the same problem. This can provide some very valuable lessons, like that one time when I wrote a Go program that was 20 lines long to solve a challenge that took only 1 line into solve in Bash…

If you are interested to check out my solutions for some of the challenges, you can find them in my GitHub repository. For some of them I even created two different solutions, one in Python and another Go, just to compare and practice working with both languages.

Oh and I almost forgot to mention, they have some really cool stickers that they are not shy to send half-way across the world by post, so that’s another big plus for sticker fans :)

That’s all for now, thank you for tuning in! :)

Docker with Ansible

Fri, 13 Dec 2019 11:11:00 +0000

Introduction

This post was written as a kind of learning diary for my most recent venture into the world of automation through Ansible. The project I implemented uses Docker to package 2 services into a micro-services architecture and Ansible to build and deploy those services on remote hosts (with the help of Dockr Compose).

The Idea

The service implements a file processing utility which monitors the file-system (a particular folder) and grabs any newly created files and stores them in another folder while compressing it. Interacting with the service is possible through a web interface which offers a way to upload files, simple statistics and the possibility to request email summaries.

The Approach

The first idea was to write it all in Go, because I am quite comfortable with the language. However, after a few searches on the interweb, I discovered that a handy UNIX utility already exists for my exat use-case: inotify. While Go has some packages that offer wrappers around this utility, I eventually decided to just write a bash script for using the inotify tool, instead of relying on Go for implementing all parts of this service. This also allowed me a convenient excuse to make the service into a 2 piece set, both of which can be deployed and scaled independently, in the spirit of micro-service architecture. Next, I set out to learn enough of Ansible that can be used to deploy a packaged in Docker containers.

ANSIBLE 101

Before this project, I never had the chance to use Ansible, but I wanted to learn about it for quite a while, so here I would describe it briefly for those who are also on the start of their journey with Ansible.

At the basic level, it is a tool for provisioning and configuring applications on remote systems in an automated fashion. To achieve the automation it uses so-called playbooks, which define what steps are necessary to reach a desired state for remote systems. It runs mainly on UNIX systems, but is able to provision and configure both UNIX and Windows based systems.

It is an agentless tool, which means it does not require any special software to be included in the remote hosts. Instead it relies on an SSH connection to remote hosts, through which bash or PowerShell utilities are used to carry out the necessary steps.

Ansible uses an inventory that describes the remote systems that can be provisioned through the playbooks. Inventories can be defined statically in local filesystem on the Ansible master node, or pulled dynamically from remote systems as well.

ANSIBLE MEETS DOCKER

For the purpose of this project, them main use of Ansible lies in its ability to build and run Docker containers. While Docker is not strictly needed to deploy this service on multiple remote hosts, it becomes much easier when all the necessary dependencies and the source code are packaged neatly in a container that can be easily shipped. Within the Docker container, all dependencies are set up and the service is configured in a reliable and consistent manner, while Ansible takes care of deploying and running the service.

It is worth mentioning that other tools exist, such as Kubernetes, Docker Swarm and others, which focus more on shipping containerised applications. This blog post, however will not deal with those, but focus entirely on Ansible and Docker instead. Future posts may discuss those alternatives in more detail.

Below is a brief summary of the proposed architecture that depicts how Ansible and Docker are used together to achieve the desired state of deploying the containerised service on each Ansible host.

Detailed instructions are out of scope for this post as well, but briefly: the above shows a snapshot of my local environment using virtual machines in VirtualBox. First, I created a master VM with Ubuntu Desktop and then two slave VMs with Ubuntu server (no GUI necessary). Ansible was installed on the Master node and proper SSH access was configured for both slave VMs from the master VM. In the Ansible playbook used to deploy the service on remote systems, the first few tasks were about installing necessary dependencies and setting up a local docker environment, which can later build and run containerised applications.

MONOLITHIC VS MICROSERVICE

Before discussing how Ansible was used to deploy the service on remote machines using Docker, it is worth going through the building blocks of the service itself. The set of features needed for the service:

file monitoring service that grabs and compresses files
web interface for file uploads, email sending and service stats

These features could be implemented in one application that runs all the necessary functions in parallel. In fact, on my first iteration, I opted to solve it this way, packaging all features into a single container. The below figure shows how it worked.

However, for the sake of learning, it is worth to consider using a microservice approach. This essentially means breaking up big monolithic applications to smaller sub-components. Docker is a perfect tool for this. For our purposes, such an architecture could mean deploying 2 separate containers: one for the Web UI backend (for uploads, statistics and email) and another that implements the monitoring and compression service. Below is an updated figure showing the breakup of our previously monolithic approach.

Breaking up the one container from the first iteration into two separate containers enables us to reap some benefits of microservice architecture. Our application components can fail independently, for example, a bug in the email sending service will not bring down the monitoring service. Also, such an architecture means in the future we can scale better with demand, in case there would be a huge surge in requests to the web frontend, we could just deploy more instances of the container and use a load-balancer to distribute requests among those instances.

IMPLEMENTATION

To implement the web component, I used simmple static HTML being served from a GO backend, that also handled file-uploads, sending email notifications and extracting statistical data from a shared SQLite3 database. In order to implement the file monitoring service, I used the inofity-tools available on UNIX systems, and wrapped it in a bash script that took care of the zipping, and generating of logs and statistics into the SQLite3 database.

Docker-Compose

Docker Compose was used to enable easier testing and deployment. The definitions in the docker-compose.yml describe what docker containers should be started with what parameters. The two services defined in the docker-compose correspond to the two containers defined above using the micro-service architecture.

The webserver running the GO backend uses a few mounted folders plus an exposed port to let inbound communication reach the server. The monitor uses 4 folders mounted from the host FS, which enable its core functionality (listening for files and zipping them to a different folder).

Ansible

Thanks to Docker Compose, it was relatively simple to deploy and run the service with Ansible, once the necessary packages and dependencies are installed on Ansible hosts. All it took was a simple Ansible Task using the docker_compose module:

- name: Docker-Compose UP
 docker_compose:
 project_src: path_to_docker_compose_yml
 build: yes

While testing the sevice a few issues were discovered that could be considered as bugs, but instead let’s call them features!

Feature #1

Since the service lets users upload files, sometimes, if the file is large enough, the processing may kick in faster than the upload can be completed. In this case, the file may be corrupted and would not be possible to recover after unzipping. To mitigae this to a certain extent, a 5 second processing delay has been added to the monitor_service.sh script, which will, as a result, delay the processing and hope that during those 5 seconds, the upload has finished.

Feature #2

While creating the two Docker files describing each component of the service, I wanted to take an extra step and created a non-root user, so that the main process of the service starts as some user which does not have full root access to everything. This worked well while developing and testing on a local system using manual execution via docker-compose up/down commands. However, once Ansible has been updated to use DC via the docker_compose module, certain functionalities would be broken due to file/folder permission issues. Basically the mounted folders would belong to root, whereas the running process was non-root, so it could not save uploaded files for example. Further investigations will be done to solve this, until then, the Dockerfiles have been reverted to use root when starting the main processes.

CONCLUSION

All in all, working on this project has been a great opportunity to practice such tools as Docker, Docker Compose and Ansible. While I have used Docker briefly before, I have never once used Ansible, and I learnt a great deal about it during this project. I can definitely see how it enables large organisations to streamline their processes when it comes to deploying and configuring various systems and services in their infrastructure. While this project is rather rudimentary, it gave me a good entry to this realm of IT.

Performance tuning GO

Mon, 11 Nov 2019 11:11:00 +0000

Introduction

This post is going to contain a short story on how I managed to optimize the execution of a simple program, written for a coding challenge on the site runcode.ninja.

Short description of the task:

There is a text file which is given as argument to your program.This text
file contains lines, each of which is an encoded englishword. Recover them
and print them out to the standard output lineby line. Hint: the UNIX
built-in dictionary may come in handy at "/usr/share/dict/american-english".

To attack problem, I used the GO language to write a program which used the built-in encoding and os/exec packages to decode the lines and to call grep to search in the file-based dictionary. It was not very difficult to figure out that the encoding in use was base64.

However, to make each line valid either a single = or double equation == characters had to be added to each line. The below code takes care of this addition of extra characters at the end of each line.

func decode(encodedStr string) string {
decoded, err := base64.StdEncoding.DecodeString(encodedStr)
for err != nil {
encodedStr += "="
decoded, err = base64.StdEncoding.DecodeString(encodedStr)
}
return string(decoded)
}

In order to test if the result of a decode operation is a valid word, a helper function was written, which is passed a string as an argument and performed the call to grep via os/exec.

func dictLookup(word string) bool {
dictLocation := "/usr/share/dict/american-english"
_, err := exec.Command("grep", "-w", word, dictLocation).Output()
if err != nil {
return false
}
return true
}

Finally, putting these pieces together, there is a function which reads in the txt file, iterates over the lines and calls decode and dict lookup until a valid word comes out, then prints it to standard output. Below is the sample code.

scanner := bufio.NewScanner(file)
var line string
for scanner.Scan() {
line = decode(scanner.Text())
for !(dictLookup(line)) {
line = decode(line)
}
fmt.Println(line)
}

Initial results

The sample code worked well enough and running it on the test / sample data provided yielded correct output, so all seemed to be fine!

flrnks@t460:~/drop_the_bass (master) ▶ go run main.go input.txt
interpretation
sanctioned
lawn
electives
unifying

Then came the idea to try to test this code on both of my laptops because it did not seem to run very quickly, even though it only had to decode 5 lines. So one of the machines I have is a ThinkPad T460 with an i5 and 16GB of RAM, while the other is a 15” MacBook Pro with i9 CPU and 32GB of RAM. I initially developed the code on the ThinkPad, and was quite surprised how much slower it was to execute on the MacBook. I would have expected that it would be the opposite, since the ThinkPad is around 3-4 years old already with a less powerful CPU. Initial test results from both machine:

 [MacBook] [ThinkPad]
interpretation 285.76ms 32.61ms
lawn 425.63ms 59.31ms
unifying 1.10s 93.60ms
electives 1.20s 91.10ms
sanctioned 6.18s 141.28ms

Overall the MacBook took on average 9 seconds to finish, while the ThinkPad took around 0.5 to 1 second to finish. This was not normal, so I had to investigate! 👀 😄

Performance Tuning 1.0

Seeing the results and the difference in performance, I was quite interested what could be the cause for such a performance drop on the MacBook. My first idea was to implement concurrency into the processing, so that instead of reading lines sequentially, they get processed in parallel by getting assigned to a worker using channels, which will return it to the main routine waiting for the results.

The above figure contains the basic idea for this concurrent processing model and the below code snippet shows some parts of the code that are most important:

// define the channels for distributing work and collecting the results
jobs := make(chan string)
results := make(chan string)
// use the waitgroup for syncing up between the workers
wg := new(sync.WaitGroup)
// start up some workers that will block and wait
for w := 1; w <= 5; w++ {
wg.Add(1)
go workerFunc(jobs, results, wg)
}
// interate over the file line by line and queue them up in the jobs channel
go func() {
scanner := bufio.NewScanner(file)
for scanner.Scan() {
jobs <- scanner.Text()
}
close(jobs)
}()
// In parallel routine wait for WG to finish and close channel for results
go func() {
wg.Wait()
close(results)
}()
// Print out the results from the results channel.
for v := range results {
fmt.Println(v)
}

This parallel processing has noticeable improved the performance, but still did not eliminate the substantial difference between the two platforms.

Note: implementing the concurrent model means the words on the standard output will appear in a random order, and so the submission to the grading system might fail.

Performance Tuning 2.0

Next, I was looking around on the internet (StackOverFlow.com in particular) where I got the idea to stop calling grep via the os/exec package, and instead read the contents of the dictionary into memory and perform lookups that way. Essentially this was trading memory footprint for speed. So then I create a global dictionary {‘map[string]bool’} which was loaded once at the start of the program and used as often as needed by the various go-routines. And this was perfectly fine because the worker routines called read-only operations on this map so there was no issue with concurrent access to the global map variable.

var wordDict = make(map[string]bool)
func loadDictionary() {
dict, _ := os.Open("/usr/share/dict/american-english")
defer dict.Close()
ds := bufio.NewScanner(dict)
for ds.Scan() {
wordDict[ds.Text()] = true
}
}

This way the lookups in the dictionary cannot be a bottleneck of the I/O system of the particular OS the program is running on. Executing the same timing test this time yielded much improved results. It became clear that the issue on the MacBook was slow execution of the external grep call from the GO program. Why this is the reason I am not sure, but the results speak for themselves:

 [MacBook] [ThinkPad]
interpretation 54.691µs 24.17µs
lawn 65.922µs 9.176µs
unifying 155.726µs 71.785µs
electives 113.074µs 47.478µs
sanctioned 286.94µs 464.20µs

Somehow the older and less powerful ThinkPad still seems considerably faster, but at least the difference is not so substantial anymore… 😌

Results

The below picture briefly summarizes the observed results when it comes to performance, which was measured by execution time. In order to mitigate transient effects on execution time, there were 10 measurements taken for each variant.

Explanation for the different variants (Seq vs. Con and Grep vs Map):

Seq: each line is decoded one after the other in sequence.
Con: each line is processed concurrently on a pool of workers.
Grep: dictionary lookup done via exec call to GREP.
Map: dictionary is loaded into a string map in memory.

Quite frankly, the results speak for themselves. The most notable thing is that, compared to the most basic version (Seq-Grep), the biggest improvement is achieved not by using concurrency, but by eliminating the repeated calls to Grep.

This is not to say that enabling concurrency did not have an impact on the execution time, on average it decreased from 9 to 6 seconds, which is quite good already!

However, I/O latency seems to have a higher cost on the performance than lack of parallel processing. At least at the scale of input for this example this is the case. This difference is less pronounced when tests were run using a file which had 500 lines of encoded words (instead of just 5).

Conclusion

Never underestimate the power of I/O delay and the effect it can have on your program. Even if you have a very powerful machine, this can bog your performance down considerably! Also, it may help your program’s performance further, if you implement proper concurrent processing whenever possible.