When an Incident Response team finds a suspicious binary, the question is simple: "What does this do?" The answer requires Malware Analysis. It is the art of dissecting compiled code safely to understand its origin, payload, and communication channels (C2).
Safety First: The Lab
NEVER analyze malware on your host machine. One double-click and you are encrypted.
Requirement: An isolated Virtual Machine (VM) with NO network adapter (Host-Only or properly configured NAT).
Tools: VMWare Workstation/VirtualBox, Remnux (Linux distro), FlareVM (Windows distro).
1. The Two Schools of Thought
| Type | Method | Tools | Risk |
|---|---|---|---|
| Static Analysis | Reading content without execution. | Strings, PEStudio, Ghidra, IDA Pro | Low |
| Dynamic Analysis | Running it and watching behavior. | Procmon, Wireshark, RegShot, Cuckoo | High |
2. Static Analysis: Reading the Blueprint
Before touching the grenade, we look at the markings on the shell.
2.1. Hashing & VirusTotal
First, get the MD5/SHA256 signature and check VirusTotal. If 50 antiviruses already flag it as "WannaCry", you save yourself hours of work.
2.2. Strings Analysis
Compiled programs contain ASCII strings (error messages, URLs, file paths). Running the `strings` command often reveals secrets.
If you see a URL, you know where the C2 server is. If you see weird garbled text, it's likely Packed.
2.3. Packing & Obfuscation
Malware authors compress their executable to hide the code from AV scanners.
Common Packers: UPX, Themida.
Deobfuscation: You must "unpack" it in memory before analyzing. upx -d evil.exe handles the easy cases.
3. Dynamic Analysis: Detonation
Static analysis is hard if the code is obfuscated. Dynamic analysis is easier: Run it and see what it breaks.
3.1. Sysinternals Suite
- Procmon (Process Monitor): The god tool. It records every file system, registry, and network activity. Filter by "Process Name = evil.exe".
- Process Explorer: Tells you if the malware spawned child processes or injected DLLs into legitimate processes (DLL Injection).
3.2. RegShot
Take a "snapshot" of your registry before infection. Run malware. Take a 2nd snapshot. RegShot compares them.
3.3. Network Traffic (Faking the Internet)
Malware will try to phone home. If your VM is offline, it fails. Use INetSim to fake the internet. It answers ALL DNS requests with your own IP and accepts ALL traffic.
Run Wireshark listening on the loopback. You will see the HTTP POST request sending your passwords to the C2 server.
4. Advanced Static: Disassembly (The Matrix)
When you need to know exactly how the encryption algorithm works, you need a Disassembler.
4.1. IDA Pro & Ghidra
These tools turn machine code (0s and 1s) into Assembly (ASM) and sometimes pseudo-C code.
This reveals the malware is using a simple XOR cipher with key 0x42. Now you can write a decryption script in Python to unlock the victim's files without paying the ransom.
5. Automated Sandboxing (Cuckoo)
For scale, enterprise teams use Cuckoo Sandbox or Hybrid Analysis. You upload the file, the sandbox spins up a VM, runs it for 2 minutes, records everything, kills the VM, and generates a PDF report.
Evading Sandboxes: Modern malware checks: "Is the mouse moving?" "Is the hard drive only 20GB?" "Is the user named 'Cuckoo'?" If yes, it does nothing (goes to sleep) to trick the analyst.
Malware Behavior Patterns
Dropper: Extract a file from itself and write it to disk.
Downloader: Reach out to URL and download the 2nd stage.
Keylogger: Hook SetWindowsHookEx to capture typing.
Ransomware: Enumerate all drives (A-Z), look for .doc/.pdf, encrypt, delete Shadow Copies (vssadmin delete shadows /all).