Debugging Continued

Advanced dynamic analysis using GDB debugger with 8 challenging CTF exercises covering XOR encryption, memory analysis, register examination, and advanced debugging techniques.

Lab Overview

Building upon the skills acquired in the previous lab, this hands-on debugging session takes your expertise to the next level with a fresh set of challenges. These new exercises further enhance your dynamic analysis capabilities using the GNU Debugger (GDB). In the world of reverse engineering and cybersecurity, debugging is an indispensable skill, particularly when the source code remains elusive. This lab provides the ideal platform to fortify your skills by introducing unique challenges, each designed to push the boundaries of your GDB proficiency.

You’ll set breakpoints, scrutinize register values, and decipher assembly instructions to uncover concealed passwords and flags. For example, in the “XorStr” challenge, you’ll leverage GDB to identify an XOR mask and use it to decrypt a string, while “StaticInt” will have you focusing on EAX values and local variables to print the flag. By the conclusion of this lab, you will have cemented your debugging abilities and effectively surmounted eight new challenges, solidifying your expertise in dynamic analysis for situations where source code accessibility is restricted.

In your home directory you will find some binaries that you need to reverse engineer in order to determine the password that the program expects. Once you have found the password, run the program and enter the password to receive the flag.

Authors: Thalita Vergilio, Tom Shaw, and Z. Cliffe Schreuders
License: CC BY-SA 4.0
Difficulty: intermediate
CyBOK Knowledge Areas: MAT: Malware Analysis analysis techniques analysis environments DYNAMIC ANALYSIS
Tags: gdb dynamic-analysis debugging advanced ctf xor memory-analysis register-analysis

Contents

    Introduction

    Debugging is a practical skill, so the best way for you to become proficient is by getting plenty of hands-on practice within realistic reverse-engineering contexts (i.e. where the source code is not available to you). This week, we continue to focus on dynamic analysis using the GNU Debugger (GDB) whilst solving eight more challenges.

    Advanced GDB Techniques

    Before diving into the advanced CTF challenges, let’s cover some essential GDB techniques that you’ll need for the more complex debugging scenarios ahead.

    Backtrace and Stack Analysis

    When a program crashes or you want to understand the call chain, use the backtrace command:

    bt

    This shows you the sequence of function calls that led to the current point in execution.

    To examine the values pushed to the stack before a function call:

    x/10x $rsp

    This shows the top 10 values on the stack, which often include function parameters.

    Function Call Analysis

    Understanding how function parameters are passed is crucial for advanced debugging:

    Note: In x86-64 Linux, function parameters are passed in registers:

    • 1st parameter: RDI
    • 2nd parameter: RSI
    • 3rd parameter: RDX
    • 4th parameter: RCX

    To examine the first parameter to a function:

    x/s $rdi

    To examine the second parameter:

    x/s $rsi

    XOR Operations and Encryption

    XOR is commonly used in encryption and obfuscation. In assembly, you’ll see instructions like:

    xor eax, 0x12345678

    Tip: When you see XOR operations, one operand is often the data and the other is the encryption key.

    To examine XOR operations, look for the xor instruction in the disassembly and note the values being XORed together.

    Advanced Memory Examination

    Here are different format specifiers for the x command:

    x/s $rdi          # Read as string
x/x $rax          # Read as hexadecimal
x/d $rax          # Read as decimal
x/c $rax          # Read as character
x/10i $rip        # Read as 10 instructions

    To examine strings in memory, use:

    x/s $rdi

    The /s format specifier reads the data as a null-terminated string.

    Setting Breakpoints in Function Calls

    To set a breakpoint just before a function call:

    break *main+0x123

    This allows you to examine the parameters being passed to the function.

    Use the fin command to step out of a function and return to the caller:

    fin

    Tip: The fin command is particularly useful when you’ve stepped into a function and want to return to the calling function.

    CTF Challenges

    There are eight challenges this week and, as you would expect, there is more than one way to solve them - feel free to explore different approaches.

    General tips

    Tip: Don’t forget to change your GDB configuration settings so programs are disassembled using the Intel notation (GDB defaults to AT&T).

    ==action: Run:==

    vi ~/.gdbinit

    ==action: Add the following line:==

    set disassembly-flavor intel

    ==action: Save and close the file.==

    Flag: For all the challenges, once you have found the password, run the program again outside of GDB to get your flag.

    Tip: If at any point your console layout looks jumbled, with the text you type overwriting the text on the screen, hit Ctrl + L to fix it.

    Tip: Remember to use these essential GDB commands:

    Command Description
    run execute the program from within GDB
    si “step into” the next assembly instruction
    ni “step over” the next assembly instruction (don’t step into functions)
    fin “step out” of the function
    c continue, or “play”

    XorLong

    Hint: There are different ways to solve this challenge. One of them is to find the cmp in main() that controls whether printflag() will be called. Put a breakpoint there. Print the values of the registers being compared (as hexadecimal numbers). One of them will be the password.

    XorStr

    Hint: One of the ways to solve this challenge is to put a breakpoint before the call to strlen(), then examine the value of rdi (as a string) when the execution stops there.

    Hint: Next, look for the XOR line in the disassembled code. See if you can find the mask.

    Hint: Write a simple C program that goes through every character of the string (char array) and XORs it against the mask.

    SegvBacktrace

    Hint: Run the program in gdb. When the execution fails following a wrong password, run “bt”.

    Hint: Put a break in each of the functions, just before they call the next function.

    Hint: When execution stops, examine the value of the rdi register (as a string) in each of them.

    2DArrays {#2darrays}

    Hint: Using GDB to obtain this flag could almost be considered cheating… but since our focus this week is to get as much practice with GDB as possible, you are allowed (and indeed encouraged) to use it.

    Hint: Disassemble main() and find the cmp instruction that compares a register with a hardcoded value.

    InputFormat

    Hint: The password is made up of three three-digit values, separated by a space: an integer, a character and a hexadecimal number. When testing, enter something like 111 aaa 222. For this challenge, you will need to work on one value at a time.

    Hint: First, set the breakpoints:

    Hint: 1. Find the call to scanf() that takes user input. Then find the cmp instruction that follows it. Put a breakpoint there.

    Hint: 2. Find the call to strcmp() a little further down and put a breakpoint there.

    Hint: 3. Finally, find the next cmp instruction and put a breakpoint there.

    Hint: Run the program in GDB. When it stops at the first cmp instruction, examine the values being compared. One of them is your input, the other is the answer.

    Hint: Run the program again, this time replacing the integer/character/decimal part of the answer with the value you found. When the execution breaks, examine the values being pushed onto the stack before strcmp() is called. You should be able to find your second value.

    Hint: Run the program one more time, replacing the two parts of the answer you now know with the correct values. When the execution stops, once again, examine the values being compared to find your third and final value.

    StaticInt

    Hint: Disassemble main(). See that EAX needs to be set to not zero for it to print the flag, and the code that sets the value of EAX is inside the check_code() function.

    Hint: Use breakpoints in check_code() to read the value of a local variable just before the cmp instruction is executed. Remember to examine it as a double-word decimal.

    StaticRE

    Hint: This one is fairly straightforward. Disassemble main(). Put a breakpoint on the cmp based on which the flag will be printed.

    Hint: Run the program and check the values compared. Remember to examine the local variable as a double-word decimal.

    StaticStrcmp

    Hint: Find the call to strcmp (it doesn’t show the name, just the function address). Put a breakpoint just before the function is called and examine the values pushed to the stack.

    Conclusion

    At this point you have:

    • Become more confident in using GDB to step through a program’s execution at assembly level, without having access to the source code;
    • Solved practical challenges using MetaCTF and found 8 more flags!

    Well done!

    By now, you will have had plenty of exposure to GDB and should have started to feel confident performing advanced dynamic analysis on malware for which the source code is not available.