A fundamental weakness in the way almost all endpoint detection and response (EDR) systems work gives attackers an opening to sneak malware past them.

Fixing the issue is not going to be easy, requiring a substantial overhaul of most current EDR systems on the market, Optiv said in a report this week.

EDR products are designed to detect and respond to suspicious behaviors and attacks on endpoint devices. Most combine signature-based malware detection with heuristic analysis, sandboxing, and other techniques to spot and block threats. The technology allows security teams to quickly isolate compromised systems and collect endpoint logs and other threat indicators to facilitate remediation.

One technique many EDR products use to detect suspicious activity and gather information for behavior-based analytics is called "hooking." Matthew Eidelberg, technical manager at Optiv, describes hooking as a technique for monitoring computer programs as they run. The hooks are placed at a System Call (syscall) interface, which allows a running process to interact with the operating system to request services, such as allocating memory, or to create a file.

"Many EDR products place these hooks at a point of program execution that users have access to so they have permissions to remove or completely bypass them," Eidelberg says.

The hooks give EDR agents on endpoint devices a way to monitor all running processes and look for any changes to those processes. The EDR agent passes data gathered via hooking to the EDR vendor's platform for further analysis.

The problem is that because the hooks are placed in user space, everything in a process' memory space when the process is created has the same permissions as the user running it.

"This means that malicious code has the same permissions as the system Dynamic Link Libraries," Eidelberg says.

As a result, attackers can modify the hooks in these system DLLs to allow malicious code to bypass the EDR product's detection and remediation mechanisms. Because of where the hooks exist, attackers could also write their own malicious system call functions into a process and have the operating system execute the functions.

"EDR products don't know that these exist or where they are located, so it makes it impossible for them to hook,'' Eidelberg said.

Optiv's research into weaknesses around EDR and memory hooks builds on previous work by other researchers. But rather than focusing on techniques that work only on individual products, the security vendor looked to see whether it could find systemic issues with hooking across all EDR products that attackers could exploit, Optiv said in its report. As part of its effort, the company developed exploits showing how it could sneak a malicious payload past products from four leading EDR product vendors.

An attacker would only need access to a remote endpoint to execute these attacks, Eidelberg noted. However, EDRs that operate in the kernel space shouldn't be as affected.

"This is because the kernel space is one area where these hooks are pretty reliable," he says. "It's hard for an attacker to do anything in the kernel given security controls around loading code."

Eidelberg says Optiv has been helping EDR vendors understand how attackers can exploit these issues in the wild. Several of them have begun to revamp their products and augment the telemetry collected via hooking to see whether they can detect tampering of system DLLs, he says. Some vendors are even moving their hooks into protected kernel space.

"All of these are great steps but will take some time to implement," he noted.

In the meantime, organizations should continue ensuring they have strong controls and detection mechanisms for preventing attackers from getting to the point where they can execute malicious code on an endpoint system in the first place, Eidelberg says.

"For an attacker to exploit this, they would need to get their malicious code onto an organization's systems and execute these actions," he says.