go | FLRNKS

Testing Terraform Modules

Sun, 12 Jul 2020 11:11:00 +0000

Intro

I first head of Terraform about 1 year ago while working on an assignment for a job interview. The learning curve was steep, and I still remember how confused I was about the syntax of HCL that resembled JSON but was not exactly the same. I also remember hearing about the concept of Terraform Modules, but for the assignment it was not needed, so I skipped it for the time being.

Fast forward to present day, I’ve had a good amount of exposure to Terraform Modules at work, where we use them to provision resources on AWS in a standardized and rapid fashion. In order to broaden my knowledge on Terraform Modules, I decided to create an exercise in which I created two TF Modules with using version 0.12 of Terraform. In this post I wanted to describe these two Terraform Modules and how I went about testing them to ensure they did what they were meant to.

What is a Terraform Module

According to official documentation a Terraform module is simply a container for multiple resources that are defined and used together. Terraform Modules can be embedded in each other to create a hierarchical structure of dependent resources. To define a Terraform Module one needs to create one or more Terraform files that define some input variables, some resources and some outputs. The input variabls are used to control properties of the resources, while the outputs are used to reveal information about the created resources. These are often organized into such structure as follows:

variables.tf defining the Terraform variables
main.tf creating the Terraform resources
output.tf listing the Terraform outputs

Note that the above is just an un-enforced convention, it simply makes it easier to get a quick understanding about a Terraform Module. As an example, if an organization needs to have their AWS S3 buckets secured with the same policies to protect their data, they can embed these security policies in a TF Module and then prescribe its use within the organization to enable those security policies automatically. Next up is an example of just that.

The Secure-Bucket TF Module

The first of the 2 Terraform Modules is tf-module-s3-bucket which can be used to create an S3 bucket in AWS that is secured to a higher degree, so that it may be suitable for storing highly sensitive data. The security features of the bucket consists of:

filtering on Source IPs that can access its contents
enforcing encryption at rest (KMS) and in transit
object-level and server access logging enabled
filtering on IAM principals based on official docs

When using this module, one can define a list of IPs, and a list of IAM Principals to control who and from which networks can access the contents of the bucket. These restrictions are written into the Bucket Policy, which is considered a resource-based policy that always takes precendence over Identity based policies, so it does not matter if an IAM Role has specific permission granted to access the bucket, if the bucket’s own Bucket Policy denies the same access. Below is a good overview of the whole evaluation logic of AWS IAM:

In addition, server-access and object-level logging can be enabled as well to improve the bucket’s level of auditability. Altogether, these settings can greatly elevate the security of data in the S3 bucket that was created by this module.

The S3-AuthZ TF Module

This 2nd Terraform Module is called tf-module-s3-auth and it was written to in part to complement the other one used to create an S3 bucket. The aim of this module is to help with the creation of a single IAM policy that can cover the S3 and KMS permissions needed for a given IAM Principal. The motivation behind this module comes from some difficulties I’ve faced at work which meant that some IAM Roles we used had too many policies attached. For further reference see the AWS docs on this.

The Bucket Policy that is crafted by the first TF Module allows the definition of list of IAM Principals that are allowed to interact with the bucket. With this TF module one can actually define the particular S3 actions that those IAM Principals CAN carry out on the data in the bucket. Additionally, this TF module can also be used allow KMS actions on the KMS keys that are protecting the data at rest in the bucket.

Untested code is broken code

With infrastructure-as-code, just as with normal code, testing is often an afterthought. However, it seems to be catching on more and more nowadays. Nothing shows this better than the amount of search results in Google for Infrastructure as Code testing: 235.000.000 as of today (15.8.2020). While Infrastructure as Code is a much broader topic with many other interesting projects, this post will have a sole focus on Terraform. With Terraform, a good step in the right direction is as simple as running terraform validate that can catch silly mistakes and syntax errors and provide feedback such as below:

Error: Missing required argument
on main.tf line 107, in output "s3_bucket_name":
107: output "s3_bucket_name" {
The argument "value" is required, but no definition was found.

In addition to the terraform validate option, many IDEs such as IntelliJ, already have plugins that can alert to such issues, so I find myself not using it so often. However, it’s still nice to have this feature built into the terraform executable!

Once all syntax errors are fixed, the next stage of testing can continue with the terraform plan command. This command uses terraform state information (local or remote) to figure out what changes are needed if the configuration is applied. This is truly very useful in showing in advance what will be created or destroyed. However, a successful terraform plan can still result in a failed deployment because some constraints cannot be verified without making the actual API calls to the Cloud Service Provider. The terraform plan command does not make any actual API calls, it only computes the difference that exist between the Terraform Code vs. the Terraform State (local or remote). The failures are usually very provider specific.

data "aws_iam_policy_document" "Deny-Non-CiscoCidr-S3-Access" {
statement {
sid = "Deny-All-S3-Actions-If-Not-In-IP-PrefixList"
effect = "Deny"
actions = [ "s3:*" ]
resources = [ "*" ]
condition {
test = "NotIpAddress"
variable = "aws:SourceIp"
values = local.ip_prefix_list
}
}
}

This Terraform Code is syntactically correct nd passes the terraform validate, and terraform plan produces a valid plan. However, it still fails at the terraform apply stage because AWS has a restriction on the sid: For IAM policies, basic alphanumeric characters (A-Z,a-z,0-9) are the only allowed characters in the Sid value. This constraint is never checked before terraform apply is called, at which point it is going to fail the whole action with the below error:

An error occurred: Statement IDs (SID) must be alpha-numeric. Check that your input satisfies the regular expression [0-9A-Za-z]*

Such types of errors can only be caught when making real API calls to the Cloud Service Provider (or to a truly identical mock of the real API) which will validate the calls and return errors if any are found. Next I will go into some details on how I went about testing the 2 Terraform Modules I wrote.

Manual Testing via AWS

This most rudimentary form of testing can be done by setting up a real project that imports and uses the two Terraform modules. This test can be found in my repository’s test/terraform/aws/ directory. For this to work properly the AWS provider has to be set up with real credentials, which is beyond the scope of this post. I also opted to use S3 as TF state backend storage but this is optional, it can just ass well store the state locally in a .tfstate file.

First, terraform has to be initialized which will trigger the download of the AWS Terraform Provider via terraform init. Next, the changes can be planned and applied via terraform plan & apply respectively. It’s interesting to note that a complete terraform apply takes close to 1 minute to complete:

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.
Outputs: [...]
real 0m49.090s
user 0m3.532s
sys 0m1.929s

Once the terraform apply is complete one can make manual assertions whether it went as expected based on the outputs (if any) and by manually inspecting the resources that were created. While this can be good enough for new setups, it may be not so good when an already deployed project has to be modified and one needs to make sure the changes will not have any undesired side effects.

Manual Testing via localstack

In order to save time (and some costs), one may also consider using localstack which replicates most of the AWS API and its features to enable faster and easier development and testing. It’s important to note that it only works if one is an AWS customer. In an earlier post I’ve already written on how to set it up, so I will not repeat it here. The most important thing is to enable S3, IAM and KMS services in the docker-compose.yaml by setting this environment variable: SERVICES=s3,kms,iam so the corresponding API endpoints are turned on.

The Terraform files I wrote for testing with on real AWS can be re-used for testing with localstack with some tweaks, for more detail look to test/terraform/localstack/ folder in my repository. Then it’s just a matter of running terraform init followed by a terraform plan & apply to create the fake resources in Localstack.

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.
Outputs: [ ... ]
real 0m11.649s
user 0m3.589s
sys 0m1.580s

Notice that this time the terraform apply took only about 10 seconds, which is considerably faster than using the real AWS API.

Automating tests via Terratest

As I’ve shown, running tests via Localstack can be much faster on average, but sometimes a project may require the use of some AWS services that are not supported by Localstack. In this case it becomes necessary to run tests against the real AWS API. For such situations I recommend terratest from Gruntwork.io, which is a Go library that provides capabilities to automate tests.

It still requires a terraform project to be set up, as described in Manual Testing via AWS, however having the ability to formally define and verify tests can greatly increase the confidence that the code being tested will function the way it’s supposed to. In the test I implemented some assertions on the output values of the terraform apply as well as about the existence of the S3 bucket just created. In addition, the Go library also provides ways to verify the AWS infrastructure setup, by making HTTP calls or SSH connections. This can be a pretty powerful tool.

This terratest setup can be found in my repo under test/go/terraform_test.go.

Running this test takes considerably longer than either of the two previous ones, but the advantage is that this can be easily automated and integrated into a CI/CD build where it can verify on-demand that the TF code still works as intended, even if there were some changes.

▶ go test
TestTerraform 2020-08-09T21:46:22+02:00 logger.go:66: Terraform has been successfully initialized!
...
TestTerraform 2020-08-09T21:47:30+02:00 logger.go:66: Apply complete! Resources: 7 added, 0 changed, 0 destroyed.
...
TestTerraform 2020-08-09T21:48:08+02:00 logger.go:66: Destroy complete! Resources: 7 destroyed.
...
PASS
ok github.com/florianakos/terraform-testing/tests 116.670s

The basic idea of terratest is to automate the process or creation and cleanup of resources for the purposes of tests. To avoid name clashes with existing AWS resources, it’s a good practice to append some random strings to resource names as part of the test, so they are not going to fail due to unique name constraints.

Conclusion

In this post I have shown what options are available for testing a Terraform Module in local or remote settings. If one only works with AWS services then Localstack can be a great tool for quick local tests during development, while terratest from Gruntwork can be a great help with codifying and automating such tests that run against the real AWS Cloud from your favourite CI/CD setup.

RunCode.ninja Challenges

Sat, 11 Jan 2020 11:11:00 +0000

This post was born on a misty saturday morning, while slowly sipping some good quality coffe in a Prague café. The last several days after work was over I spent solving programming challenges on runcode.ninja and I thought it would be nice to share my experience and spread the word about it.

RunCode.ninja

I can’t really recall how I discovered this website in the first place… All I remember is that I was really into the simplistic idea of it all. The basic idea for most of the challenges goes something like this:

check problem description
inspect any sample input (if any)
write your program locally
test on sample input (if any)
submit source code to the evaluation platform

If all went well, you will get feedback within a few seconds whether the submitted code worked correctly for the given task at hand. If it didn’t, then you can turn to their FAQ for some advice. It definitely has some useful info, however if all else fails, you can also contact the team behind the platform on their slack channel. They are really friendly people so be sure to respond to their effort in kind!

Another nice thing about their platform is that they categorized all their challenges (119 in total as of now) into nice categories such as binary, encoding, encryption, forensics, etc. which allows you to select what you are interested in. When I started out, I was first aiming to complete the challenges in Easy which offers a combination of relatively easy challenges from math, text-parsing, encoding and other categories.

As it currently stands, I rank 155 our of around ~2400 registered users, which seems quite impressive at first, but I suspect there may be quite a few inactive accounts in their database. Also, there are some hardcore people who have already completed all their challenges that seems quite impressive. If only a few rainy and cold weekends I could spend working on these, I would probably catch up soon!

Last but not least, their platform is set up to interpret a several different programming languages, so you can choose to solve them in the language you are most comfortable with. Once you solve a challenge, you can access its write-ups which provide some very useful inspiration on how others have solved the same problem. This can provide some very valuable lessons, like that one time when I wrote a Go program that was 20 lines long to solve a challenge that took only 1 line into solve in Bash…

If you are interested to check out my solutions for some of the challenges, you can find them in my GitHub repository. For some of them I even created two different solutions, one in Python and another Go, just to compare and practice working with both languages.

Oh and I almost forgot to mention, they have some really cool stickers that they are not shy to send half-way across the world by post, so that’s another big plus for sticker fans :)

That’s all for now, thank you for tuning in! :)

Docker with Ansible

Fri, 13 Dec 2019 11:11:00 +0000

Introduction

This post was written as a kind of learning diary for my most recent venture into the world of automation through Ansible. The project I implemented uses Docker to package 2 services into a micro-services architecture and Ansible to build and deploy those services on remote hosts (with the help of Dockr Compose).

The Idea

The service implements a file processing utility which monitors the file-system (a particular folder) and grabs any newly created files and stores them in another folder while compressing it. Interacting with the service is possible through a web interface which offers a way to upload files, simple statistics and the possibility to request email summaries.

The Approach

The first idea was to write it all in Go, because I am quite comfortable with the language. However, after a few searches on the interweb, I discovered that a handy UNIX utility already exists for my exat use-case: inotify. While Go has some packages that offer wrappers around this utility, I eventually decided to just write a bash script for using the inotify tool, instead of relying on Go for implementing all parts of this service. This also allowed me a convenient excuse to make the service into a 2 piece set, both of which can be deployed and scaled independently, in the spirit of micro-service architecture. Next, I set out to learn enough of Ansible that can be used to deploy a packaged in Docker containers.

ANSIBLE 101

Before this project, I never had the chance to use Ansible, but I wanted to learn about it for quite a while, so here I would describe it briefly for those who are also on the start of their journey with Ansible.

At the basic level, it is a tool for provisioning and configuring applications on remote systems in an automated fashion. To achieve the automation it uses so-called playbooks, which define what steps are necessary to reach a desired state for remote systems. It runs mainly on UNIX systems, but is able to provision and configure both UNIX and Windows based systems.

It is an agentless tool, which means it does not require any special software to be included in the remote hosts. Instead it relies on an SSH connection to remote hosts, through which bash or PowerShell utilities are used to carry out the necessary steps.

Ansible uses an inventory that describes the remote systems that can be provisioned through the playbooks. Inventories can be defined statically in local filesystem on the Ansible master node, or pulled dynamically from remote systems as well.

ANSIBLE MEETS DOCKER

For the purpose of this project, them main use of Ansible lies in its ability to build and run Docker containers. While Docker is not strictly needed to deploy this service on multiple remote hosts, it becomes much easier when all the necessary dependencies and the source code are packaged neatly in a container that can be easily shipped. Within the Docker container, all dependencies are set up and the service is configured in a reliable and consistent manner, while Ansible takes care of deploying and running the service.

It is worth mentioning that other tools exist, such as Kubernetes, Docker Swarm and others, which focus more on shipping containerised applications. This blog post, however will not deal with those, but focus entirely on Ansible and Docker instead. Future posts may discuss those alternatives in more detail.

Below is a brief summary of the proposed architecture that depicts how Ansible and Docker are used together to achieve the desired state of deploying the containerised service on each Ansible host.

Detailed instructions are out of scope for this post as well, but briefly: the above shows a snapshot of my local environment using virtual machines in VirtualBox. First, I created a master VM with Ubuntu Desktop and then two slave VMs with Ubuntu server (no GUI necessary). Ansible was installed on the Master node and proper SSH access was configured for both slave VMs from the master VM. In the Ansible playbook used to deploy the service on remote systems, the first few tasks were about installing necessary dependencies and setting up a local docker environment, which can later build and run containerised applications.

MONOLITHIC VS MICROSERVICE

Before discussing how Ansible was used to deploy the service on remote machines using Docker, it is worth going through the building blocks of the service itself. The set of features needed for the service:

file monitoring service that grabs and compresses files
web interface for file uploads, email sending and service stats

These features could be implemented in one application that runs all the necessary functions in parallel. In fact, on my first iteration, I opted to solve it this way, packaging all features into a single container. The below figure shows how it worked.

However, for the sake of learning, it is worth to consider using a microservice approach. This essentially means breaking up big monolithic applications to smaller sub-components. Docker is a perfect tool for this. For our purposes, such an architecture could mean deploying 2 separate containers: one for the Web UI backend (for uploads, statistics and email) and another that implements the monitoring and compression service. Below is an updated figure showing the breakup of our previously monolithic approach.

Breaking up the one container from the first iteration into two separate containers enables us to reap some benefits of microservice architecture. Our application components can fail independently, for example, a bug in the email sending service will not bring down the monitoring service. Also, such an architecture means in the future we can scale better with demand, in case there would be a huge surge in requests to the web frontend, we could just deploy more instances of the container and use a load-balancer to distribute requests among those instances.

IMPLEMENTATION

To implement the web component, I used simmple static HTML being served from a GO backend, that also handled file-uploads, sending email notifications and extracting statistical data from a shared SQLite3 database. In order to implement the file monitoring service, I used the inofity-tools available on UNIX systems, and wrapped it in a bash script that took care of the zipping, and generating of logs and statistics into the SQLite3 database.

Docker-Compose

Docker Compose was used to enable easier testing and deployment. The definitions in the docker-compose.yml describe what docker containers should be started with what parameters. The two services defined in the docker-compose correspond to the two containers defined above using the micro-service architecture.

The webserver running the GO backend uses a few mounted folders plus an exposed port to let inbound communication reach the server. The monitor uses 4 folders mounted from the host FS, which enable its core functionality (listening for files and zipping them to a different folder).

Ansible

Thanks to Docker Compose, it was relatively simple to deploy and run the service with Ansible, once the necessary packages and dependencies are installed on Ansible hosts. All it took was a simple Ansible Task using the docker_compose module:

- name: Docker-Compose UP
 docker_compose:
 project_src: path_to_docker_compose_yml
 build: yes

While testing the sevice a few issues were discovered that could be considered as bugs, but instead let’s call them features!

Feature #1

Since the service lets users upload files, sometimes, if the file is large enough, the processing may kick in faster than the upload can be completed. In this case, the file may be corrupted and would not be possible to recover after unzipping. To mitigae this to a certain extent, a 5 second processing delay has been added to the monitor_service.sh script, which will, as a result, delay the processing and hope that during those 5 seconds, the upload has finished.

Feature #2

While creating the two Docker files describing each component of the service, I wanted to take an extra step and created a non-root user, so that the main process of the service starts as some user which does not have full root access to everything. This worked well while developing and testing on a local system using manual execution via docker-compose up/down commands. However, once Ansible has been updated to use DC via the docker_compose module, certain functionalities would be broken due to file/folder permission issues. Basically the mounted folders would belong to root, whereas the running process was non-root, so it could not save uploaded files for example. Further investigations will be done to solve this, until then, the Dockerfiles have been reverted to use root when starting the main processes.

CONCLUSION

All in all, working on this project has been a great opportunity to practice such tools as Docker, Docker Compose and Ansible. While I have used Docker briefly before, I have never once used Ansible, and I learnt a great deal about it during this project. I can definitely see how it enables large organisations to streamline their processes when it comes to deploying and configuring various systems and services in their infrastructure. While this project is rather rudimentary, it gave me a good entry to this realm of IT.

Performance tuning GO

Mon, 11 Nov 2019 11:11:00 +0000

Introduction

This post is going to contain a short story on how I managed to optimize the execution of a simple program, written for a coding challenge on the site runcode.ninja.

Short description of the task:

There is a text file which is given as argument to your program.This text
file contains lines, each of which is an encoded englishword. Recover them
and print them out to the standard output lineby line. Hint: the UNIX
built-in dictionary may come in handy at "/usr/share/dict/american-english".

To attack problem, I used the GO language to write a program which used the built-in encoding and os/exec packages to decode the lines and to call grep to search in the file-based dictionary. It was not very difficult to figure out that the encoding in use was base64.

However, to make each line valid either a single = or double equation == characters had to be added to each line. The below code takes care of this addition of extra characters at the end of each line.

func decode(encodedStr string) string {
decoded, err := base64.StdEncoding.DecodeString(encodedStr)
for err != nil {
encodedStr += "="
decoded, err = base64.StdEncoding.DecodeString(encodedStr)
}
return string(decoded)
}

In order to test if the result of a decode operation is a valid word, a helper function was written, which is passed a string as an argument and performed the call to grep via os/exec.

func dictLookup(word string) bool {
dictLocation := "/usr/share/dict/american-english"
_, err := exec.Command("grep", "-w", word, dictLocation).Output()
if err != nil {
return false
}
return true
}

Finally, putting these pieces together, there is a function which reads in the txt file, iterates over the lines and calls decode and dict lookup until a valid word comes out, then prints it to standard output. Below is the sample code.

scanner := bufio.NewScanner(file)
var line string
for scanner.Scan() {
line = decode(scanner.Text())
for !(dictLookup(line)) {
line = decode(line)
}
fmt.Println(line)
}

Initial results

The sample code worked well enough and running it on the test / sample data provided yielded correct output, so all seemed to be fine!

flrnks@t460:~/drop_the_bass (master) ▶ go run main.go input.txt
interpretation
sanctioned
lawn
electives
unifying

Then came the idea to try to test this code on both of my laptops because it did not seem to run very quickly, even though it only had to decode 5 lines. So one of the machines I have is a ThinkPad T460 with an i5 and 16GB of RAM, while the other is a 15” MacBook Pro with i9 CPU and 32GB of RAM. I initially developed the code on the ThinkPad, and was quite surprised how much slower it was to execute on the MacBook. I would have expected that it would be the opposite, since the ThinkPad is around 3-4 years old already with a less powerful CPU. Initial test results from both machine:

 [MacBook] [ThinkPad]
interpretation 285.76ms 32.61ms
lawn 425.63ms 59.31ms
unifying 1.10s 93.60ms
electives 1.20s 91.10ms
sanctioned 6.18s 141.28ms

Overall the MacBook took on average 9 seconds to finish, while the ThinkPad took around 0.5 to 1 second to finish. This was not normal, so I had to investigate! 👀 😄

Performance Tuning 1.0

Seeing the results and the difference in performance, I was quite interested what could be the cause for such a performance drop on the MacBook. My first idea was to implement concurrency into the processing, so that instead of reading lines sequentially, they get processed in parallel by getting assigned to a worker using channels, which will return it to the main routine waiting for the results.

The above figure contains the basic idea for this concurrent processing model and the below code snippet shows some parts of the code that are most important:

// define the channels for distributing work and collecting the results
jobs := make(chan string)
results := make(chan string)
// use the waitgroup for syncing up between the workers
wg := new(sync.WaitGroup)
// start up some workers that will block and wait
for w := 1; w <= 5; w++ {
wg.Add(1)
go workerFunc(jobs, results, wg)
}
// interate over the file line by line and queue them up in the jobs channel
go func() {
scanner := bufio.NewScanner(file)
for scanner.Scan() {
jobs <- scanner.Text()
}
close(jobs)
}()
// In parallel routine wait for WG to finish and close channel for results
go func() {
wg.Wait()
close(results)
}()
// Print out the results from the results channel.
for v := range results {
fmt.Println(v)
}

This parallel processing has noticeable improved the performance, but still did not eliminate the substantial difference between the two platforms.

Note: implementing the concurrent model means the words on the standard output will appear in a random order, and so the submission to the grading system might fail.

Performance Tuning 2.0

Next, I was looking around on the internet (StackOverFlow.com in particular) where I got the idea to stop calling grep via the os/exec package, and instead read the contents of the dictionary into memory and perform lookups that way. Essentially this was trading memory footprint for speed. So then I create a global dictionary {‘map[string]bool’} which was loaded once at the start of the program and used as often as needed by the various go-routines. And this was perfectly fine because the worker routines called read-only operations on this map so there was no issue with concurrent access to the global map variable.

var wordDict = make(map[string]bool)
func loadDictionary() {
dict, _ := os.Open("/usr/share/dict/american-english")
defer dict.Close()
ds := bufio.NewScanner(dict)
for ds.Scan() {
wordDict[ds.Text()] = true
}
}

This way the lookups in the dictionary cannot be a bottleneck of the I/O system of the particular OS the program is running on. Executing the same timing test this time yielded much improved results. It became clear that the issue on the MacBook was slow execution of the external grep call from the GO program. Why this is the reason I am not sure, but the results speak for themselves:

 [MacBook] [ThinkPad]
interpretation 54.691µs 24.17µs
lawn 65.922µs 9.176µs
unifying 155.726µs 71.785µs
electives 113.074µs 47.478µs
sanctioned 286.94µs 464.20µs

Somehow the older and less powerful ThinkPad still seems considerably faster, but at least the difference is not so substantial anymore… 😌

Results

The below picture briefly summarizes the observed results when it comes to performance, which was measured by execution time. In order to mitigate transient effects on execution time, there were 10 measurements taken for each variant.

Explanation for the different variants (Seq vs. Con and Grep vs Map):

Seq: each line is decoded one after the other in sequence.
Con: each line is processed concurrently on a pool of workers.
Grep: dictionary lookup done via exec call to GREP.
Map: dictionary is loaded into a string map in memory.

Quite frankly, the results speak for themselves. The most notable thing is that, compared to the most basic version (Seq-Grep), the biggest improvement is achieved not by using concurrency, but by eliminating the repeated calls to Grep.

This is not to say that enabling concurrency did not have an impact on the execution time, on average it decreased from 9 to 6 seconds, which is quite good already!

However, I/O latency seems to have a higher cost on the performance than lack of parallel processing. At least at the scale of input for this example this is the case. This difference is less pronounced when tests were run using a file which had 500 lines of encoded words (instead of just 5).

Conclusion

Never underestimate the power of I/O delay and the effect it can have on your program. Even if you have a very powerful machine, this can bog your performance down considerably! Also, it may help your program’s performance further, if you implement proper concurrent processing whenever possible.