Inside IBM's Patent Applications For Airport Security

A dozen little-known IBM patent applications lay out a sophisticated computer-analysis-based approach to airport security. The technology has the potential to apply profiling of passengers, based on attributes such as age and type of clothing worn. One of the patents IBM is seeking even appears to go Israeli-style security one better, using analysis of furtive glances in the application entitled "Detecting Behavioral Deviations By Measuring Eye Movements."

The objective of the technology in the passel of patent applications is to alert officials to potential terminal and tarmac threats using a network of video, motion, chemical, and biometric sensors arrayed throughout the airport. The sensors feed into a grid of networked computers, which provide high-powered processing to get results to officials in so-called real time, yet the systems are compact enough to be located on-site.

The "secret sauce" in the set up is a software "inference engine," which crunches the data fed in by the multitude of sensors, separating the high-risk wheat from the false-alarm chaff. That engine uses heuristics and rules developed by the three co-inventors behind the patent applications--Robert Angell, Robert Friedlander and James Kraemer.

"These patents are built on the inference engine, which has the ability to calculate very large data sets in real time," Angell told me last Friday.

He called me because he was surprised I had uncovered one of the patents, which I wrote about recently in my blog post, "Obama Security Push Spurring Scanner Patents (IBM's Seeking One)." That post focused on the patent application "Risk assessment in a pre/post security area within an airport."

Angell told me he believed the patents were under seal. That piqued my interest, because it indicated that this technology is probably more important -- in the sense of being proprietary and cutting edge -- than I had initially realized. As well, I knew of only the one patent and hadn't realized that, according to Angell, there were eight. (Since our conversation, I've uncovered 12 unique applications; the discrepancy might be due to the presence of duplicates--patent lawyers often revise and resubmit applications--or spin-offs.)

It turns out that, in fact, the patent applications are not under seal; that's something I don't think you can do, because the patent process is by definition open. Companies which want to shield proprietary technology go the trade-secret route, which means you keep your cutting-edge technology out of the public eye and hope no one will reverse-engineer it.

I have now tracked down all the applications, and will go into the technology details, below. [Update, Jan. 26: A paragraph in the original story stating that IBM didn't put down the company name as the assignee on three of its patent applications, which was based on failure to find that name on three applications viewed on the main patent search site, has been removed. The company name is present on the applications, when they've viewed via a different USPTO search. "We don't purposely withhold IBM's name from patent applications," as IBM spokesman said, and I accept that statement as fact.]

Angell also said that he's no longer with IBM. "I was laid off last year along with thousands of other people," he told me. Angell is currently teaching a computer science course at a community college in Salt Lake City, Utah, where he lives. I was flabbergasted, wondering how Big Blue could let go a guy like this, who obviously has heavy duty data-analysis chops and is behind such seemingly important technology.

Angell called me, he said, because he's concerned that the technology be applied effectively. "If it's done right, we could do passive profiling [and] passive detection and do it without a whole lot of fanfare," he said.

This profiling of potentially dangerous passengers, as outlined in the applications, appears in many ways to be more neutral than the profiling currently the subject of widespread public debate, because it's software-based and runs off of pre-programmed rules which, in general, are intended to identify suspicious behavior. (On the other hand, this wouldn't necessarily always apply, since markers such as a person's apparent age are listed in the patent applications as potential data points.) Let's look at the patents in more detail. The profiling, off of sensor input, is described in patent application number 20090204695, filed last September. It's entitled "Unique Cohort Discovery From Multimodal Sensory Devices."

This patent application describes the use of a large number of sensors of all types -- chemical, biometric, etc -- around the airport perimeter, so data can be fed into a computer for analysis to detect threats.

Here's the relevant wording from the patent application:

"[Data processing parses the data to form attributes.] Attributes may include an individual's age, make and/or model of a vehicle, color of a hat, breed of a dog, sound of an engine, a medical diagnosis, a date of birth, a color, item of clothing, walking, talking, running, a type of food eaten, an identification of an item purchased.

An attribute that is an event may include eating, smoking, walking, jogging, walking a dog, carrying bags, carrying a baby, riding a bicycle, an engine running, a baby crying, or any other event.

Sensory data processing categorizes the events. . . For example, a type of event may include a pace of walking, a companion of the cohort, a time of day a cohort eats a meal, a brand of soda purchased by the cohort, a pet purchased by the cohort, a type of medication taken by the cohort, or any other event."

In terms of the sensors themselves, the system uses lots of diverse data-gatherers. From the patent application:

"Multimodal sensors comprises at least one of a set of global positioning satellite receivers, a set of infrared sensors, a set of microphones, a set of motion detectors, a set of chemical sensors, a set of biometric sensors, a set of pressure sensors, a set of temperature sensors, a set of metal detectors, a set of radar detectors, a set of photosensors, a set of seismographs, and a set of anemometers."

Angell told me that the system can even use olfactory sensors, which means they'll smell the environment. The patent application also variously mentions license plate recognition technology, face recognition software, and retina scanners. Data captured from video streams from airport cameras is also analyzed.

How does one computer process all this data fast enough to deliver a threat assessment quickly enough to airport security officials? Remember, the idea is to do the analysis in real time, as passengers are streaming through the terminal to board their flights. For a single box, this would be a processing challenge. However, the inventors envision using a small grid of computers connected over a network. This'd deliver ample power to do the real-time data crunching.

"Computers aren't fast enough to do real-time modeling unless the paradigm shifts," Angell told me. "That's why this inference engine is a pretty big deal."

That shift is embedded in how inference engine is formulated. It uses rule sets, designed by Angell, Friedlander, and Kraemer, which enable it to fairly efficiently query 5 million or 10 million data cohorts, in a very short period of time. There is another patent application in the group which takes the analysis of potential passenger threats to a whole 'nother level. It's entitled "Detecting Behavioral Deviations By Measuring Eye Movements." (Patent application number 2009232357, filed September 2009.) (Friedlander is not involved in this patent; it's Angell and Kraemer only.) From the filing:

"The ocular metadata [patterns of eye movement] is analyzed. . .In response to the patterns of ocular movements indicating behavioral deviations in the member of the cohort group, the member of the cohort group is identified as a person of interest."

Specifically, eye movement characteristics which are monitored and analyzed include: change in pupil size (dilation); direction of gaze; visual line of gaze (where someone is looking); and rate of blinking; and furtive glances.

Profiling is specifically addressed in this patent application, as follows:

"The profiled past comprises data that may be used, in whole or in part, for identifying the person, determining whether to monitor the person, and/or determining whether the person is a person of interest. Global profile data may be retrieved from a file, database, data warehouse, or any other data storage device. Multiple storage devices and software may also be used to store identification data 506. Some or all of the data may be retrieved from the point of contact device, as well. The profiled past may comprise an imposed profile, global profile, individual profile, and demographic profile. The profiles may be combined or layered to define the customer for specific promotions and marketing offers."

However, analysis of eye movements aren't the final word in indentifying passengers with potential ill intent. Patent application targets "Detecting Behavioral Deviations by Measuring Respiratory Patterns in Cohort Groups."

Click here for images from IBM's patent applications.

Have a comment on this story? Please click "Discuss" below. If you'd like to contact Dark Reading's editors directly, send us a message.