Hunting Fileless Malware in Memory

There’s nothing on disk to hash. That’s the whole point, and it’s exactly why fileless malware keeps working as well as it does. Reflective DLL injection, process hollowing, PowerShell running entirely in memory off a downloaded script that never touches a file—none of it leaves the artifact your standard file-scanning tools are built to find. If your hunting program stops at “scan the filesystem,” you’re missing a whole category of modern tradecraft.

Memory hunting is harder than file-based analysis, no question. It requires different tools, different mental models, and honestly a fair bit more patience. But it’s not some exotic specialty anymore—it’s close to table stakes for a hunter who wants to catch anything past the entry-level stuff.

Know the Shape of the Common Techniques Before You Go Looking

Process hollowing creates a legitimate process in a suspended state, then replaces its memory contents with malicious code before resuming execution—the process name and PID look completely normal, but the actual code running inside doesn’t match what should be there for that binary. Reflective DLL injection loads a DLL directly into a process’s memory space without ever writing the DLL to disk or registering it normally with the OS loader, which means standard “what DLLs are loaded” checks that rely on the loader’s own bookkeeping can miss it entirely.

Process doppelgänging and process ghosting are newer variations on the same theme—using NTFS transaction tricks or delete-pending file states to get malicious code executed while leaving essentially nothing behind for forensic recovery. You don’t need to memorize every variant’s exact mechanics, but understanding the general category—legitimate process shell, illegitimate contents—helps you know what you’re actually looking for when you start pulling memory.

Memory Analysis Tooling: What Actually Gets Used

Volatility remains the workhorse for offline memory forensics—pull a memory image (via a tool like WinPmem or a commercial EDR’s memory acquisition feature) and run it through Volatility’s plugin set. malfind is the plugin most hunters reach for first because it specifically looks for memory regions with characteristics common to injected code—executable permissions on a memory page that shouldn’t normally have them, or a PE header sitting somewhere it doesn’t belong within a process’s address space.

pslist versus psscan comparison is a classic technique worth knowing: pslist walks the OS’s own linked list of active processes, while psscan scans memory directly for process structure signatures regardless of whether the OS’s list still references them. A process that shows up in psscan but not pslist has been deliberately unlinked from the process list—a classic rootkit or advanced hiding technique—and that discrepancy alone is worth escalating immediately.

For live, at-scale hunting rather than single-machine forensics, modern EDR platforms increasingly bake in memory-scanning capability directly—the ability to push a YARA rule and scan live process memory across your entire fleet without needing to acquire and transfer full memory images machine by machine. That’s usually the more practical starting point for hunting broadly, saving full offline Volatility analysis for the specific hosts that scan flags as interesting.

What Actually Looks Wrong in a Memory Region

The core thing you’re hunting for across most of these techniques: memory regions with execute permissions that don’t correspond to a normally-loaded, properly-mapped module. Legitimate code execution almost always ties back to a DLL or EXE that the OS loader mapped in through the normal process—injected code frequently exists in memory regions allocated directly (via VirtualAllocEx or similar) that have execute permissions but no backing file on disk that the OS can point to.

A concrete pattern: a process showing a memory region marked RWX (read-write-execute) that isn’t associated with any loaded module. Legitimate software very rarely needs RWX memory—most legitimate use cases separate write and execute permissions for security reasons (this is literally what DEP/NX exists to enforce), so RWX regions specifically are a decent, if imperfect, signal worth building a hunt around.

Don’t Skip Network Correlation Just Because the Malware Is “Fileless”

Fileless doesn’t mean traceless. Injected code still has to communicate somehow in most cases—beacon out, receive commands, exfiltrate data. This is where tying your memory hunting back into the network telemetry from your Zeek or Suricata deployment pays off. If malfind on a specific host turns up a suspicious memory region in a legitimate-looking process, pull that host’s network connections for the same timeframe. A previously-quiet browser process suddenly making regular outbound connections to an unfamiliar IP, right around when the suspicious memory region first appeared, ties the investigation together in a way memory analysis alone can’t.

I’ll be honest about the limitation here too: memory analysis is resource and time-intensive, and it doesn’t scale the way file-based scanning does. You’re not going to run full Volatility analysis across a 10,000-endpoint fleet weekly. The practical approach is layering—use lightweight, EDR-driven memory scanning broadly for known patterns, and reserve deep offline forensic analysis for the specific hosts your broader hunting (network anomalies, process tree oddities, frequency-based LOLBAS flags) has already flagged as worth the deeper look.

Memory hunting is where a lot of hunters level up from “good at reading logs” to actually understanding what’s happening at the OS level during an attack. It’s slower work, and it demands more from you technically. It’s also where you find the stuff that never shows up anywhere else.