---
title: 'CTF Reverse Engineering: DownUnderCTF Flag Checker — REA'
url: https://rea.tools/showcase/ctf/
---

[Home](https://rea.tools/) / [Showcases](https://rea.tools/showcase/) / CTF

Case study · DownUnderCTF 2023

# Find the flag in a CTF binary.

Use REA to inspect a small flag checker, turn its rules into equations
and find an input that makes the original program print
`Correct!`.

masked squares flag checker · Easy · Linux x86-64

[View the original challenge](https://github.com/DownUnderCTF/Challenges_2023_Public/tree/b48f83db2984bc7fa2ec7879a32d7efb1be8d333/rev/masked-squares-flag-checker)

From a binary to a checked answer

1. 01 · Find the check

   **Strings → checking functions**

   REA finds the prompt, its references and the code that accepts or
   rejects an input.
2. 02 · Extract the rules

   **36 characters · 26 sums**

   REA returns the instructions and data behind each check. The
   agent turns them into equations.
3. 03 · Solve and run

   **Recovered flag → Correct!**

   A Python script solves the equations. REA captures the original
   program accepting the result.

The challenge is by joseph, published in DownUnderCTF's official
repository.

## Find the checking code

The handout is a 15 KB executable. It asks for a flag, then prints
`Correct!` or `Incorrect!`. Start with a
simple request to your agent.

Your coding agent

Use REA to analyze ms_flag_checker and find the flag. Explain
how the checks work, then verify the answer with the original
program.

An example prompt, with the downloaded challenge available locally
and a native analysis provider connected.

1. #### Follow the input prompt to its function

   Agent → REA

   ```
   open_binary
   {"path": "ms_flag_checker"}

   search_strings
   {"pattern": "flag|correct|wrong|input",
    "mode": "regex", "case_sensitive": false}

   xrefs
   {"address": "0x102004"}

   resolve_containing_procedure
   {"address": "0x10126d"}
   ```

   REA → agent · selected results

   ```
   0x102004  "What is the flag? "
   0x10201c  "Incorrect!"
   0x102027  "Correct!"

   Prompt referenced at: 0x10126d
   Containing function: 0x10124d
   ```

   The reference gives the agent a place to investigate, even
   though the executable's original function names have been
   removed.
2. #### Read the checker and its two helpers

   Agent → REA

   ```
   analyze_function
   {"procedure": "0x10124d"}

   analyze_function
   {"procedure": "0x101189"}

   analyze_function
   {"procedure": "0x101217"}
   ```

   REA → agent · selected instructions

   ```
   0x1012c1: CMP EDI, 0x24
   0x1012c6: LEA RBX, [0x1040e0]
   0x1012cd: LEA RBP, [0x104060]
   0x1012d4: LEA R13, [RBX + 0x270]
   0x1012e1: CALL 0x00101189
   0x1012ec: CALL 0x00101217
   0x1012f1: CMP dword ptr [RBP], EAX
   0x1012f4: JNZ 0x00101335
   0x1012f6: ADD RBX, 0x18
   ```

   The first loop copies 36 character codes. The checking loop
   decodes a mask, sums the selected codes and compares the
   result with a stored number. Its `0x18`-byte steps
   across `0x270` bytes give 26 masks.

These are selected results from a fresh REA analysis. The local
target path is shortened for display; addresses refer to REA's
imported image.

## One check reveals one character

The checker arranges the input in a 6×6 grid. A mask chooses which
cells contribute to a sum. Its compact encoding uses negative
numbers to skip cells and positive numbers to select them.

![The seventh mask skips 21 cells, selects position 21, then skips 14. Only that cell contributes to the sum. Its required character code is 55, which means the character is 7.](https://rea.tools/assets/figures/ctf-mask-equation.svg)

The seventh check selects just one position.
[Open figure](https://rea.tools/assets/figures/ctf-mask-equation.svg).

### Read the mask and its target from the binary

Agent → REA · read_bytes

```
{"address": "0x1040e0", "length": 624}
{"address": "0x104060", "length": 144}
```

REA → agent · seventh entries

```
Mask at 0x104170:
eb 01 f2 00  →  -21, +1, -14, stop

Target at 0x104078:
37 00 00 00  →  55
```

The instruction `MOVSX` reads mask bytes as signed
values. This is why `eb` means −21. The target is a
little-endian integer. Together, these bytes tell the agent that
`code[21] = 55`, so that character is `7`.

## From instructions to a readable check

Select a step to connect the summing helper and its caller to the
same rule in C.

REA · selected original instructions

```
0x10121c: MOV ECX, 0x0

0x10122c: CMP dword ptr [RSI + RAX*0x1], 0x0
0x101230: JZ 0x00101223

0x101232: ADD ECX, dword ptr [RDI + RAX*0x1]

0x101223: ADD RAX, 0x4

0x1012f1: CMP dword ptr [RBP], EAX
0x1012f4: JNZ 0x00101335
```

Readable C · one-check summary

```
int check_one_mask(const int codes[36],
                   const int mask[36],
                   int target) {
    int total = 0;
    for (int i = 0; i < 36; ++i) {
        if (mask[i]) {
            total += codes[i];
        }
    }
    return total == target;
}
```

**01 · Start at zero.** `ECX` holds
the running sum.

**02 · Check the mask cell.** A zero value skips
the character; a nonzero value includes it.

**03 · Add a character code.** The input's
integer at the same offset is added to the sum.

**04 · Move to the next cell.** The original loop
advances four bytes per integer and visits six rows of six
cells.

**05 · Compare with the stored target.** The
caller receives the sum in `EAX`. A mismatch jumps
to `Incorrect!`.

The instructions come from the summing helper and its caller. The
C is an explanatory summary with descriptive names and a flattened
36-cell loop.

## Solve the 26 equations

Most masks select several cells. For example, the first check adds
16 character codes and requires a total of 1441. The overlapping
selections give equations that a small Python script can solve
together.

Python · solver excerpt

```
from z3 import Int, Or, Solver, Sum, sat

codes = [Int(f"code_{i}") for i in range(36)]
solver = Solver()
alphabet = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_{}"
for code in codes:
    solver.add(Or(*[code == ord(c) for c in alphabet]))
for code, char in zip(codes, "DUCTF{"):
    solver.add(code == ord(char))
solver.add(codes[-1] == ord("}"))

for selected, target in checks:
    solver.add(Sum([codes[i] for i in selected]) == target)

assert solver.check() == sat
model = solver.model()
print("".join(chr(model.eval(code).as_long()) for code in codes))
```

Z3 is a solver that finds values satisfying these equations. This
excerpt uses the extracted checks and assumes a
`DUCTF{…}` flag with letters, digits, underscores and
braces. The complete download includes all 26 masks and targets.

[Download the complete solver](https://rea.tools/showcase/ctf/solve.py)

## Check the answer with the original program

REA's process capture records the original executable accepting
the solved flag. Changing one selected character from
`z` to `y` makes the first sum 1440 instead
of 1441, and the program rejects it.

| Input | Observed output | Exit code |
| --- | --- | --- |
| Solver's flag | `Correct!` | 0 |
| One character changed | `Incorrect!` | 255 |

Show the recovered flag and captured output

REA · original program output

```
What is the flag? DUCTF{ezzzpzzz_07bcda7bfe81faf43caa}
Correct!
```

The recovered value also matches the flag in the organizers'
published challenge metadata, checked after solving the binary.


Source, target identity and analysis details

Fresh analysis and two process captures on 8 October 2026, using
REA 4.1.0 with Ghidra 12.1.4 on Linux x64. The binary is a
stripped x86-64 ELF, 15,248 bytes.

Original executable · SHA-256

```
dcd3bec4f608e11f3a12ce461100aaf5f621851ee63a1f870d7c90d4e3dd51ae
```

The string references, functions and constants were inspected
before reading the organizers' source or solution. The Python
solver and the diagram were built from those REA results.

[Compare with the organizers' source](https://github.com/DownUnderCTF/Challenges_2023_Public/blob/b48f83db2984bc7fa2ec7879a32d7efb1be8d333/rev/masked-squares-flag-checker/src/ms_flag_checker.c)
·
[Challenge metadata and flag](https://github.com/DownUnderCTF/Challenges_2023_Public/blob/b48f83db2984bc7fa2ec7879a32d7efb1be8d333/rev/masked-squares-flag-checker/ctfcli.yaml)
·
[REA evidence notes](https://github.com/morluto/rea/blob/main/website/evidence/ctf-masked-squares.md)

## Try it yourself

Download the original challenge and the solver into one folder.
Use the prompt above to investigate with your agent, or run the
included script.

[Download the original challenge](https://raw.githubusercontent.com/DownUnderCTF/Challenges_2023_Public/b48f83db2984bc7fa2ec7879a32d7efb1be8d333/rev/masked-squares-flag-checker/publish/ms_flag_checker)[Download solver](https://rea.tools/showcase/ctf/solve.py)

Terminal · solve and run

```
python3 -m venv .venv
.venv/bin/python -m pip install z3-solver
.venv/bin/python solve.py

chmod +x ms_flag_checker
./ms_flag_checker
```

The script prints a candidate flag; paste it at the program's
prompt. Solving needs Python and `z3-solver`. Running
the original executable needs Linux x86-64.

[Set up REA with your coding agent](https://rea.tools/get-started/)