java | FLRNKS

My first scala app

Sat, 10 Oct 2020 11:11:00 +0000

Motivation

In this post I wanted to write about a personal project I started some time ago, with the goal of learning more about Scala. At work, we use Scala quite often to run big data jobs on AWS using Apache Spark. I’ve never used Scala before I joined my current team, and its syntax was very alien to me. However, recently I had the chance to work on a task, where I had to modify a component to use AWS Secrets Manager instead of HashiCorp’s Vault for fetching some secret value at runtime. To my surprise I could complete this work without much struggle with Scala, and afterwards I became eager to learn more. Based on a colleague’s recommendation I started reading a book from Cay S. Horstmann titled Scala for the impatient (2nd edition). I’m making slow but steady progress.

Shortly after starting with the book, I had the idea to start a small project so that I can practice Scala by doing.

The Idea

The idea, like many others before, came while fixing a bug at work. The bug was found within a component written in Scala to interact with the AWS Athena service. It had some neatly written functionality for making queries and waiting for their completion before trying to fetch the results. I thought I would try to write something similar for AWS Systems Manager (SSM). It is a service with few different components, so I decided to focus on Automation Documents that can carry out actions in an automated fashion. For example, the AWS provided SSM document AWS-StartEC2Instance can run any EC2 instance when invoked with the below 2 input parameters:

InstanceId: to specify which EC2 instance you want to start
AutomationAssumeRole: to specify an IAM role which can be assumed by SSM to carry out this action

I realized quite early on, that if I wanted to implement this capability in my Scala app, it needed to be quite generic, so that it could support any Automation Document with an arbitrary number of input parameters. I also wanted it to be able to wait for the execution and report whether it failed or succeeded. Here are the final requirements I came up with:

create 2 separate git repos for:
- a module that’s home for the AWS utility/helper classes
- a module for implementing the CLI App
support extra AWS services such as KMS, Secrets Manager and CloudFormation
utilize localstack for integration testing (when possible)

Initial setup

Firstly, I had to figure out which third-party packages I needed to implement the app according to these simple requirements. To interact with AWS from Scala code, I decided to go with v2 of the official Java SDK for AWS. To implement the CLI app I mainly relied on the picocli Java package, which was a bit less straightforward, but eventually it proved to be a good choice.

Secondly, I have to admit that creating a re-usable scala package from scratch was a rather non-trivial task for me. Most of my programming experience comes from working with in non-JVM based environments so that’s probably no surprise. I initially started out with sbt for build & dependency management, but I was running into issues that I couldn’t solve on my own, so I decided to swap it with maven which was a bit more familiar to me.

Finally, separating the project into two distinct git repositories allowed me to practice versioning and dependency management which I also found very useful:

AWS Scala Utils: https://github.com/florianakos/aws-utils-scala
AWS SSM CLI App: https://github.com/florianakos/aws-ssm-scala-app

The utils module

Creating the utils module that would serve as a kind of glue between the scala CLI app and AWS Systems Manager was actually not as difficult as I thought. This is mostly thanks to the example I’ve seen at work for a similar project with the AWS Athena service.

The core functionality of the utils module when it comes to SSM, is captured in the below functions:

private def executeAutomation(documentName: String, parameters: java.util.Map[String,java.util.List[String]]): Future[String] = {
val startAutomationRequest = StartAutomationExecutionRequest.builder()
.documentName(documentName)
.parameters(parameters)
.build()
Future {
val executionResponse = ssmClient.startAutomationExecution(startAutomationRequest)
logger.info(s"Execution id: ${executionResponse.automationExecutionId()}")
executionResponse.automationExecutionId()
}
}
private def waitForAutomationToFinish(executionId: String): Future[String] = {
val getExecutionRequest = GetAutomationExecutionRequest.builder().automationExecutionId(executionId).build()
var status = AutomationExecutionStatus.IN_PROGRESS
Future {
var retries = 0
while (status != AutomationExecutionStatus.SUCCESS) {
val automationExecutionResponse = ssmClient.getAutomationExecution(getExecutionRequest)
status = automationExecutionResponse.automationExecution.automationExecutionStatus()
status match {
case AutomationExecutionStatus.CANCELLED | AutomationExecutionStatus.FAILED | AutomationExecutionStatus.TIMED_OUT =>
throw SsmAutomationExecutionException(status, automationExecutionResponse.automationExecution.failureMessage)
case AutomationExecutionStatus.SUCCESS =>
logger.info(s"Query finished with status: $status")
case status: AutomationExecutionStatus =>
logger.info(s"SSM Automation execution status: $status, check #$retries.")
Thread.sleep(if (retries <= 3) 2500 else if (retries <= 10) 5000 else 15000)
}
retries += 1
}
}.map(_ => executionId)
}

The first one executeAutomation crafts an execution request and then submits it to AWS, returning its execution ID. This ID can be passed to the waitForAutomationToFinish function that periodically checks in with AWS until the execution is complete. Between subsequent API requests it uses an increasing timeout to prevent API rate-limiting caused by excessive polling.

Testing the utils module

Once I had the core functionality ready I wanted to write integration tests to ensure it works as expected. Instead of having hard-coded AWS credentials or an AWS profile for a real account I wanted to use Localstack that mocks the real AWS API so that you can interact with it. For this reason I slightly tweaked the SsmAutomationHelper class to accept an Optional second argument which can be used while building the SSM API client:

class SsmAutomationHelper(profile: String, apiEndpoint: Option[String]) extends LazyLogging {
private val ssmClient = apiEndpoint match {
case None => SsmClient.builder()
.credentialsProvider(ProfileCredentialsProvider.create(profile))
.region(Region.EU_WEST_1)
.build()
case Some(localstackEndpoint) => SsmClient.builder()
.credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create("foo", "bar")))
.endpointOverride(URI.create(localstackEndpoint))
.build()
}
}

This allowed me to pass http://localhost:4566 when running the integration tests against localstack and have the API calls directed to those mocked endpoints. Previously each mocked service had its own dedicated port, but thanks to a recent change in localstack, now all AWS services can be run on a single port, they call EDGE port.

According to the documentation, SSM is supported in localstack, however I’ve found out that running Automation Documents is feature that is still missing. As a result, I had to run the integration tests against a real AWS account that I set up for such scenarios. I was okay with doing this since there are plenty of built-in Automation Documents provided by AWS that I could safely use for this purpose.

Eventually I decided to encode in the tests AWS-StartEC2Instance & AWS-StopEC2Instance which only required me to set up a dummy EC2 instance which would be the target of these requests. I also added a special Tag to these integration tests so that they are excluded from running when invoked via mvn test but still available to run manually whenever necessary.

CLI App implementation

After running the tests, I was confident that the AWS utils worked correctly, so I started putting together the CLI app. For this, I’ve searched on the web for a third party package and found that it’s not as simple as it is when using Python’s argparse package. I eventually settled with picocli, which is written in Java but can also be used from Scala via the below annotations:

@Command(name = "SsmHelper", version = Array("v0.0.1"), mixinStandardHelpOptions = true, description = Array("CLI app for running automation documents in AWS SSM"))
class SsmCliParser extends Callable[Unit] with LazyLogging {
@Option(names = Array("-D", "--document"), description = Array("Name of the SSM Automation document to execute"))
private var documentName = new String
@Parameters(index = "0..*", arity = "0..*", paramLabel = "<param1=val1> <param2=val2> ...", description = Array("Key=Value parameters to use as Input Params"))
private val parameters: util.ArrayList[String] = null
[...]

According to the original idea, there had to be one constant CLI flag which controlled the name of the AWS Automation Document (--document) and then there had to be a variable number of additional arguments for specifying the Input Parameters required by the given document. The picocli package supported this workflow via the @Option and the @Parameters annotations.

The only thing left was a custom function that would carry out the needed transformation of Input Parameters. The values received in the parameters were in the form of an ArrayList: [<param1=val1>, <param2=val2>, ...] which had to be transformed into a Map: [param1 -> [val1], param2 -> [val2]] by splitting each String on the = character. The desired format was a requirement of the AWS SDK for SSM. After some iterations I ended up with the below function that could do this transformation:

private def process(params: util.ArrayList[String]): util.Map[String, util.List[String]] = {
params.asScala
.map(_.split('='))
.collect { case Array(key, value) => key -> value }
.groupBy(_._1)
.mapValues(_.map(_._2).asJava).asJava
}

Finally, I constructed the below method which utilized the SsmAutomationHelper class from the utils module and passed the two variables provided by picocli to it so it would invoke the necessary Automation Document and wait to retrieve its result via the Await mechanism of Scala:

def call(): Unit = {
val conf = ConfigFactory.load()
val inputParams = process(parameters)
Await.result(SsmAutomationHelper.newInstance(conf).runDocumentWithParameters(documentName, inputParams), 10.minutes)
}

Packaging the CLI app

At this point I was ready with the CLI app and wanted to run it to see how it would function. Before I could run it, I needed to figure out how to package it all into a fat JAR file with all needed dependencies, so that it could be invoked with CLI arguments. I googled around a bit and quickly found the spring-boot-maven-plugin which has the repackage goal that’s just what I needed:

Repackages existing JAR and WAR archives so that they can be executed from the command line using java -jar. With layout=NONE can also be used simply to package a JAR with nested dependencies (and no main class, so not executable).

I only had to add the below lines to my project’s pom.xml:

<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>2.3.2.RELEASE</version>
<configuration>
<layout>JAR</layout>
</configuration>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
</plugin>

Next I just had to run the mvn package command, which invokes the plugin to builds the fat JAR.

Running the CLI app

Once the JAR is available, it can be used via the java -jar ... command with extra arguments to run the any Automation Document such as AWS-StartEC2Instance:

$ ▶ java -jar ./target/scala-cli-app-1.0.0.jar --document=AWS-StartEC2Instance InstanceId=i-0ed4574c5ba94c877 AutomationAssumeRole=arn:aws:iam::{{global:ACCOUNT_ID}}:role/AutomationServiceRole
15:24:41.998 [main] INFO c.f.utils.ssm.SsmAutomationHelper :: Going to kick off SSM orchestration document: AWS-StartEC2Instance
15:24:42.773 [ForkJoinPool-1-worker-29] INFO c.f.utils.ssm.SsmAutomationHelper :: Execution id: <...>
15:24:42.882 [ForkJoinPool-1-worker-11] INFO c.f.utils.ssm.SsmAutomationHelper :: Current status: [InProgress], retry counter: #0
[...]
15:28:01.226 [ForkJoinPool-1-worker-11] INFO c.f.utils.ssm.SsmAutomationHelper :: Current status: [InProgress], retry counter: #21
15:28:16.442 [ForkJoinPool-1-worker-11] INFO c.f.utils.ssm.SsmAutomationHelper :: Execution finished with final status: [Success]
15:28:16.444 [main] INFO com.flrnks.app.SsmCliParser :: SSM execution run took 215 seconds

Seems to be working quite well!

Bonus: running in a container

I thought I would take the above one step further and package the JAR into a java based docker container. This would allow me to forget about the syntax of the java command that I previously used to run the app. Instead, I can hide it in a very minimal Dockerfile:

FROM openjdk:8-jdk-alpine
MAINTAINER flrnks <flrnks@flrnks.netlify.com>
ADD target/scala-cli-app-1.0.0.jar /usr/share/backend/app.jar
ENTRYPOINT [ "/usr/bin/java", "-jar", "/usr/share/backend/app.jar"]

The mvn package command which is used to build the fat JAR will save it into the /target subdirectory, so one can put this Dockerfile into the project’s root and then manually build the docker image by running docker build -t ssmcli .. This will create an image called ssmcli without issues, however I’ve found an awesome plugin called dockerfile-maven-plugin built by Spotify which can automagically take this Dockerfile and turn it into an image based on the plugin configuration:

<plugin>
<groupId>com.spotify</groupId>
<artifactId>dockerfile-maven-plugin</artifactId>
<version>1.4.10</version>
<executions>
<execution>
<id>default</id>
<goals>
<goal>build</goal>
</goals>
<configuration>
<repository>flrnks/ssmcli</repository>
<tag>latest</tag>
</configuration>
</execution>
</executions>
</plugin>

This plugin hooks into the mvn package goal and when it’s executed it will automatically create the docker image:

[INFO] --- spring-boot-maven-plugin:2.3.2.RELEASE:repackage (default) @ scala-cli-app ---
[INFO] Layout: JAR
[INFO] Replacing main artifact with repackaged archive
[INFO]
[INFO] --- dockerfile-maven-plugin:1.4.10:build (default) @ scala-cli-app ---
[INFO] dockerfile: null
[INFO] contextDirectory: /Users/flszabo/Desktop/personal-wrkspc/scala/scala-cli-app
[INFO] Building Docker context /Users/flszabo/Desktop/personal-wrkspc/scala/scala-cli-app
[INFO] Path(dockerfile): null
[INFO] Path(contextDirectory): /Users/flszabo/Desktop/personal-wrkspc/scala/scala-cli-app
[INFO]
[INFO] Image will be built as flrnks/ssmcli:latest
[INFO] Step 1/4 : FROM openjdk:8-jdk-alpine
[INFO] Pulling from library/openjdk
[INFO] Digest: sha256:94792824df2df33402f201713f932b58cb9de94a0cd524164a0f2283343547b3
[INFO] Status: Image is up to date for openjdk:8-jdk-alpine
[INFO] ---> a3562aa0b991
[INFO] Step 2/4 : MAINTAINER flrnks <flrnks@flrnks.netlify.com>
[INFO] ---> Using cache
[INFO] ---> efcc673b4f35
[INFO] Step 3/4 : ADD target/scala-cli-app-1.0.0.jar /usr/share/backend/app.jar
[INFO] ---> 8b2cf76f03c2
[INFO] Step 4/4 : ENTRYPOINT [ "/usr/bin/java", "-jar", "/usr/share/backend/app.jar"]
[INFO] ---> Running in c9633237f9fa
[INFO] Removing intermediate container c9633237f9fa
[INFO] ---> 6db69aa30fb1
[INFO] Successfully built 6db69aa30fb1
[INFO] Successfully tagged flrnks/ssmcli:latest

To test this new docker image I ran the AWS-StopEC2Instance Automation Document and specified the same CLI arguments as before, thanks to the ENTRYPOINT configuration in the Dockerfile. As an extra step I needed to share the AWS profile with the docker container at runtime by using the flag -v ~/.aws:/root/.aws:

$ ▶ ddocker run --rm -v ~/.aws:/root/.aws flrnks/ssmcli --document=AWS-StopEC2Instance InstanceId=i-0ed4574c5ba94c877 AutomationAssumeRole=arn:aws:iam::{{global:ACCOUNT_ID}}:role/AutomationServiceRole
17:18:59.541 [main] INFO c.f.utils.ssm.SsmAutomationHelper :: Going to kick off SSM orchestration document: AWS-StopEC2Instance
17:19:00.789 [ForkJoinPool-1-worker-13] INFO c.f.utils.ssm.SsmAutomationHelper :: Execution id: <...>
17:19:00.966 [ForkJoinPool-1-worker-11] INFO c.f.utils.ssm.SsmAutomationHelper :: Current status: [InProgress], retry counter: #0
17:19:03.564 [ForkJoinPool-1-worker-11] INFO c.f.utils.ssm.SsmAutomationHelper :: Execution finished with final status: [Success]
17:19:03.568 [main] INFO com.flrnks.app.SsmCliParser :: SSM execution run took 5 seconds

One may say that typing that long docker run ... command above takes longer than typing java -jar ./target/scala-cli-app-1.0.0.jar ... but I would argue that running it inside a docker container has its valid use-cases as well. It allows for controlled setup of the runtime environment and prevents dependency issues too!

Conclusion

This project has allowed me to learn much more than I initially expected. I learnt a lot about Scala, which was the original goal, but I also gained valuable experience with Maven, its plugin ecosystem and of course with Java as well. I hope whoever reads this post will find something useful in it too!

Monitoring Flink on AWS EMR

Sun, 16 Aug 2020 11:11:00 +0000

Brief intro

This is going to be a somewhat unusual post on this blog. It is about a problem I recently encountered while trying to improve the monitoring of a long-running Flink cluster we have on AWS EMR, following the official documentation from Datadog.

The EMR setup

Our EMR cluster consumes 4 Kinesis Data Streams which are used to send s3 files in AVRO format for processing. When a new file arrives, the Flink job will fetch it from S3, do some validation and filtering and then convert it to ORC format and save it to a new location on s3. In early June we experienced a failure in one of the Flink jobs consuming a production stream. Sadly we did not have adequate monitoring set up to detect this on time. We only learnt about it when we noticed that data in the output bucket was missing for certain dates. Our streams were configured with the maximum retention period of 7 days. By the time we noticed the missing data in the stream was already piling up, and the oldest was close to half of this retention period. By the time we managed to find the root cause and deploy the fix to the Flink job, it was too late, and some data had already expired from the stream.

The existing monitoring solution was implemented via AWS Lambda functions running every 8 hours. These functions were making Athena queries to check if any data arrived to the S3 bucket during the last 48 hours. The problem with this was approach was that we do not get alerts about missing data for up to 2 days because of the way our query used a sliding window of 2 days.

The Flink cluster runs in a private VPC, so reaching the Flink Web UI to check the status of the jobs was quite difficult to say the least. We either had to set up an SSH port forwarding session and use a FoxyProxy setup in Firefox, or set up a personal VM the same private VPC via the AWS WorkSpaces managed service and then connect from that VM’s browser to the cluster’s Flink UI. Either way it was quite cumbersome and still a manual process to connect to the Flink UI to check the cluster health. I wanted an automated way of gathering metrics and alerting if something went wrong, so I looked into how Flink could be monitored by Datadog.

Datadog ❤️ Flink

A quick Google search threw up the official documentation from Datadog where I found really straightforward instructions on enabling the submission of Flink metrics to Datadog, which could be instantly visualized in their default Flink dashboard. These main steps are:

adding some new parameters to the flink-conf.yaml, such as the Datadog API/APP keys and custom tags
copying the flink-datadog-metrics.jar to the active flink installation path

The first step was quite easy. Our cluster was defined in Cloudformation where we used AWS::EMR::Cluster which allows specifying the flink-conf.yaml content as below:

Cluster:
 Type: AWS::EMR::Cluster
 Properties:
 Name: Flink-Cluster
 Configurations:
 - Classification: flink-conf
 ConfigurationProperties:
 metrics.reporter.dghttp.class: org.apache.flink.metrics.datadog.DatadogHttpReporter
 metrics.reporter.dghttp.apikey: '{{resolve:secretsmanager:datadog/api_key:SecretString}}'
 metrics.reporter.dghttp.tags: name:flink-cluster, app:flink-cluster, region:eu-central-1, env:prod
 [...]

The above CF snippet shows just the 3 most important lines of the flink-conf.yaml: (1) the full package name of the java class which implements the metric submission, (2) the Datadog API key loaded from AWS Secrets Manager and (3) a few custom tags which will be added to metrics sent to Datadog.

To copy the necessary datadog-metrics JAR where it would be loaded from (/usr/lib/flink/lib), I added a new AWS::EMR::Step to in CloudFormation which is executed only on the EMR Master Node in order to activate Datadog monitoring on the cluster via the supplied Java class and API key in the flink-conf.yaml.

To test that it was working properly I just needed to redeploy the cluster which was surprisingly easy thanks to the Cloudformation setup we had in place. But something was still not right.

Know your continent

After redeploying the cluster I waited and waited and waited a bit more but metrics were not showing up in the Flink dashboard. So I got in touch with Datadog support who were very helpful in figuring out what the issue was. After a few rounds of emails back and forth we quickly discovered why the metrics were not showing up.

The reason was that we had our Datadog account set up in the EU region and not in the USA. Thus, all our metrics were supposed to flow to the EU endpoint at app.datadoghq.eu/api/ instead of the USA endpoint at app.datadoghq.com/api/. The difference is quite subtle, only a simple change in the TLD from .com to .eu. The catch was that our EMR cluster was running Flink 1.9.1 (provided by the EMR release 5.29.0) which had this API endpoint hardcoded, pointing to the USA data centre. The Datadog Support Engineer uncovered some extra instructions on how this can be solved by adding an extra line to the flink-conf.yaml to change the default US region to the EU instead:

Cluster:
 Type: AWS::EMR::Cluster
 Properties:
 Name: Flink-Cluster
 [...]
 Configurations:
 - Classification: flink-conf
 ConfigurationProperties:
 [...]
 metrics.reporter.dghttp.class: org.apache.flink.metrics.datadog.DatadogHttpReporter
 metrics.reporter.dghttp.apikey: '{{resolve:secretsmanager:datadog/api_key:SecretString}}'
 metrics.reporter.dghttp.tags: name:flink-cluster, app:flink-cluster, region:eu-central-1, env:prod
 metrics.reporter.dghttp.dataCenter: EU # << points the metrics reported to the EU region
 [...]

The problem was that this was only available in Flink v1.11.0 while the highest version offered by EMR through the latest EMR Release was only v1.10.0, so this was not going to work for me. I almost gave up on the idea of monitoring Flink via Datadog when I had the idea to clone the official Flink repository from Github and tweak the code in v1.9.1 which we were running to change the hardcoded API endpoint from .com to .eu. It was much easier than I expected, I just needed to tweak this class slightly ./src/main/java/org/apache/flink/metrics/datadog/DatadogHttpClient.java:

/**
 * Http client talking to Datadog.
 */
public class DatadogHttpClient {
/* Changed endpoint for metric submission to use .eu instead of .com */
private static final String SERIES_URL_FORMAT = "https://app.datadoghq.eu/api/v1/series?api_key=%s";
/* Changed endpoint for API key validation to use .eu instead of .com */
private static final String VALIDATE_URL_FORMAT = "https://app.datadoghq.eu/api/v1/validate?api_key=%s";
...
}

Once I made the above code changes, I built a new JAR via mvn clean package. The new JAR was made available at ./flink-metrics/flink-metrics-datadog/target/flink-metrics-datadog-1.9.1.jar which I then uploaded to an S3 bucket where we store such files in my team. Next I slightly tweaked the AWS EMR step to load this JAR from S3 redeployed the cluster once more. Finally, metrics started flowing! And it looked so nice, I was especially happy to see the TaskManager heap distribution, because the issue which sparked this whole endeavor was showing symptoms of Heap Memory issues.

Unfortunately this default dashboard was not perfect, as it had some graphs that were failing to show some data. Maybe it was because of using v1.9.1 of Flink instead of v1.11.0, not sure. In any case, I ended up cloning the dashboard and fixing the graphs manually, while also adding a few extras to show data about the AWS Kinesis streams which were feeding into the Flink cluster.

Now it shows very nicely the age of each Flink job, which was not visible at all on the default dashboard. The end result is much better in my opinion.

Conclusion

All in all, I am quite happy with how this whole story turned out in the end. Despite the issue with the hardcoded API endpoints to the USA region in v1.9.1 of Flink, I managed to implement a simple workaround thanks to the Open Source nature of the project. The result is that we have much better visibility and monitoring implemented for our Flink cluster which makes our lives in the DevOps world much better. I did not write much about it in this post, but once these metrics became available in our Datadog account it was trivial to set up a few Monitors which would alert us if for example one of the 4 Flink jobs were failing. I will leave it up to the reader to imagine how that’s done.