Attackers are increasingly automated, with credential stuffing and automated application attacks surging, making the ability to detect, manage and mitigate bot-based requests increasingly important.

The sector focused on managing bots gained a new player last week, when Edgio — a company formed through a merger of Limelight Networks, Yahoo Edgecast and Layer0 — announced its own bot management offering. The service focuses on using machine learning and the company's Web security capability to allow granular policy controls, and joins other offerings from Web application firewall (WAF) providers and Internet infrastructure providers, such as Akamai, Cloudflare, HUMAN, and Imperva.

The service is not just about warding off increasingly automated attacks: Gaining the ability to see and manage so-called "good" bots — such as search bots and performance monitoring services — is just as important as preventing attacks from "bad" bots, such as credential stuffing attacks and application service attacks, says Richard Yew, senior director of product management for security at Edgio.

"Bot management means not just identifying and mitigating the bad bots, but also identifying and helping customers monitor good bots," Yew says. "You definitely need the security solution ... but you also want visibility to be able to monitor good bot traffic."

While human-originated requests continue to dominate overall Internet traffic, accounting for an increasing share of bandwidth, the total volume of bot requests have increased over time. In 2022, for example, the number of application and API attacks more than doubled, growing by 137%, according to Internet infrastructure firm Akamai. Another major source of bad bot traffic, credential stuffing attacks, accounted for a third (34%) of authentication events, according to authentication service provider Okta.

Requests from automated systems — and not humans — accounted for approximately 42% of all Internet traffic in 2021, the latest data available, according to data from Imperva. (Click here for a larger view of the above chart.)

Are You a Good Bot or Bad Bot?

The impact of bots on consumers can be obvious, but the impact of bots on businesses is not always so clear. While bots can hoard inventory, such as causing a shortage in Taylor Swift tickets or the Sony Playstation 5, other arguably legitimate uses include scraping data for competitive advantage or training machine learning systems. Still others are welcomed by most businesses, including search bots for which many companies try to optimize.

Dealing with bots often falls to security, but the threats extend to all facets of a business, says Sandy Carielli, principal analyst at Forrester Research.

"Bot management is not just about security being the decision-makers," she says. "If you're dealing with a lot of inventory-hoarding attacks, your e-commerce team is going to want to say in [the response]. If you're dealing with a lot of ad fraud, your marketing team will want to be in the room."

Bot management systems typically boil down to identifying the source of Web or API requests and then using policies to determine what to allow, what to deny, and which requests represent potentially interesting events or anomalies.

Bots Focus More on APIs

While the most common type of bot-based attack is credential stuffing, where the bots are used to attempt to use stolen credentials to log into a cloud service, the rise of API traffic has led to a variety of new attacks. Resource consumption attacks, which are a common form of denial-of-service attack, are now being ported to Web application programming interfaces (APIs).

By sending seemingly legitimate or outright fraudulent requests to an API server, an attacker could cause that server to process the request, which could be resource-intensive as more computational power is being made accessible through APIs. Because many of today's most expensive API calls are machine interfaces to machine learning systems, for example, attackers could run up a company's bill by using bots and APIs.

In a way, the bots — and defenses against them — are creating a circular economy, says Patrick Donahue, vice president of product at Cloudflare.

"It's interesting because essentially we're creating AI and machine learning systems to help other AI companies to keep their costs in check, so it's really like machine learning being used to stop machines from abusing other machine learning systems," he says.

Close to the Edge

Increasingly, bot management firms are pushing their technologies closer to the edge of the network, not just to protect clients but to catch attacks as soon as possible.

Edgio, as its name indicates, inspects traffic at the edge of the network and only allows "clean" traffic through its network. Requests enter through a point of presence (POP) close to the user — or the attacker — and exit through a POP close to the destination server or cloud service.

The benefit of such an architecture is that it stops attacks before they can impact other parts of the network, says Edgio's Yew.

"The advantage of us being close to the end user is that we're also close to the attacker because, let's face it, a lot of the attackers, they're just compromised devices," he says. "The same computer you use to watch TV and browse the Internet [is] probably the same device that's part of the zombie botnet launching any sort of bot attack. So it's very important to have a distributed infrastructure that allows us to run any workload with high performance and low latency."

The model mirrors what other vendors have done as well. Content delivery networks (CDNs), such as Akamai, Cloudflare and Fastly, also use edge servers to provide fast access to content and have adopted bot management features as well.