Data mining: Darpa bets

The R&D arm of the U.S. Department of Defense laid the groundwork for that breakthrough with experiments in high-reliability telecommunications called Arpanet and Darpanet. Today's World Wide Web is the improbable legacy of an agency focused mainly on futuristic weaponry and intelligence-gathering techniques.

Darpa's more recent adventures in information technology have met with much less acclaim. Fifteen months ago the agency came under fire for trying to use market principles to gauge the probabilities of future catastrophes. Critics derided the idea as a "terrorism futures market" that invited investors to profit from such unsavory events as a nuclear explosion or political assassination.

Officially titled FutureMAP (Futures Markets Applied to Prediction), the program was canned, and the IT side of Darpa is still reeling from the fiasco. The agency would much rather publicize its investments in unmanned combat aircraft or sensors to detect bioterrorism. But Darpa continues to channel a portion of its $3 billion budget toward advances in data mining and text analysis, and it is even building a commercial track record.

Take, for instance, MetaCarta, founded in 1999 by thenMassachusetts Institute of Technology physics graduate student John Frank. The Cambridge, Massachusetts, company got a $500,000 Darpa grant in 2001 to adapt its text- and geography-analysis system to military mapping and intelligence requirements. Venture capital money followed, accelerating MetaCarta's move into such commercial applications as insurance underwriting and oil exploration.

马萨诸塞州康科德附近披露Mining Technology hopes to repeat that exercise. With $3 million from Darpa, the company's principals spent much of the past decade developing a sophisticated military-data analysis tool. They formed DMT last year to bring the system into the financial industry. "With properly trained artificial intelligence, traders or regulators can search through vast quantities of data -- say, all 10-K and 10-Q reports -- and find anomalies on the fly that are not visible to the naked eye," explains DMT chairman and co-founder Dennis Caulfield. "There's no question about system capacity -- it has been proved by the military."

New Brunswick, New Jersey's Connotate similarly got its start in the late 1990s with a Darpa grant to develop data-mining agents to comb the Internet. Cincinnati-based Cinergy Corp. began using the technology in its Houston energy-trading operations last year. A "deep search" of the Internet "can give traders a jump on mainstream news sources," says Connotate CEO Bruce Molloy, who is now focusing on sales to financial institutions.

Though Darpa isn't talking about it, the concept underlying FutureMAP remains very much alive. The technology developer, California Institute of Technology spin-out Net Exchange, has supplied trading software to clients, including State Street Corp., since 1993. And the predictive power of "political futures" has long been evident in the Iowa Electronic Markets, a University of Iowa program that accepts bets on presidential candidates -- and that has often predicted elections more accurately than have opinion polls. (As of late September, George W. Bush was the projected popular-vote winner, 51 percent to 49 percent.)

San Diego's Net Exchange -- which obtained $750,000 from Darpa in 2001 -- is still "bringing market efficiencies to types of commerce that haven't traditionally had them," such as trucking, reports founder Charles Polk. He recently left the company and moved to New York to launch Common Knowledge Markets, which aims to embed predictive market mechanisms in the corporate world.

Polk believes the approach is ideal for such decision support tasks as risk management, which requires aggregating and filtering objective data and subjective judgments from numerous sources, and for market research on such questions as, "What will be the effect on car sales if gasoline prices top $2 a gallon?" Polk's reasoning also applies to FutureMAP: Results will be most useful if survey participants have relevant expertise and a tangible stake in the process -- a payment for participation in a survey or a reward (read: investment return) for accuracy.

Polk says he has lined up some blue-chip clients, although he's not ready to name them. "When we got caught up in the drama last year, we got a couple of bites," he says. "It just goes to show that no publicity is bad publicity."