Dr. Thomas Boghardt, Historian
Intelligence agencies are facing two principal challenges—the acquisition of secret information, and its twin, the peril of information overload. The difficulty of finding the proverbial needle in the haystack has exponentially grown since the introduction of the Internet and the explosion—and availability—of human knowledge. Every day, intelligence agencies must sift through literally millions of pieces of information—satellite images, blogs, intercepted phone calls, emails, blogs, etc.—to determine what is going on in the world. Evidently, human beings cannot accomplish this task alone.
Several years ago, the CIA commissioned its investment arm, In-Q-Tel, to devise a scheme to mine and evaluate information on the web. In-Q-Tell, in turn, hired the software company Visible Technologies to monitor and analyze social media. Now, the company crawls over half a million web sites a day, mining more than a million posts and discussions taking place on blogs, online forums, Flickr, YouTube, Twitter and Amazon. Every hit is categorized, thus making the mass of online information easier to digest for human end users at Langley and elsewhere in the intelligence community.
At this point, Visible Technologies limits its monitoring to openly available sources. The process is designed to provide U.S. intelligence an “early-warning detection on how issues are playing internationally,” says an In-Q-Tel spokesman. In fact… if you are using any of the common online tools—and since you are reading this blog, you probably are—chances are, you’ve already been mined.
Nothing is What It Seems.