Recommind used a technique called Probabilistic Latent Semantic Analysis, which are statistical models that the system builds from your documents.
"It will look at the language and derive meanings, themes and concepts from within content, then relate them to similar concepts in a different batch of documents," says Carpenter.
Autonomy uses Bayesian statistical models, similar to those used to filter spam, to determine the categories of documents. FAST uses a semantic index that can restrict the scope of a concept to a sentence or paragraph to get a more accurate answer. For example, a document might talk about both orange (the fruit) and orange (the colour) but a paragraph is more likely to be about one or the other. It also extracts entities' like names, phone numbers, addresses and companies.
Microsoft does face the challenge of integrating FAST with SharePoint, but the core Enterprise Search Platform remains popular. Future Microsoft products may also use technology from another acquisition, Powerset. This startup uses natural language processing to extract concepts from documents so it can build a semantic index and a conceptual graph of relationships between them; the Powerset site (and iPhone app) searches Wikipedia but the concept should translate to documents.
Although Google is experimenting with machine learning with the Google Sets feature, the results you can see from this on Google's public site today are not sophisticated. A list of animals is currently likely to include terms like wood and plastic because they're found on web pages dealing with toy animals. The web is a rich source of training data for machine learning, so Google may well improve its classification in the future.
More than the web
If you run the same query on the public web via Google and on your internal corporate systems, you should get very different results, because you want what your company knows about the topic, not just what's common knowledge.
Enterprise search needs to integrate with document repositories, corporate databases, ERP and CRM systems, email, call centre and customer support systems, directories like LDAP and Active Directory, your HR and accounting systems and everywhere else you store business information. And it needs to find experts as well as the documents they've written.
InQuira offers both enterprise search and customer support systems that are used by Avis, Honda and Apple.
Serena software is another customer. Chief executive Jeremy Burton is also on InQuira's board of directors and what he values is not just the natural language search engine but the ability to correlate information from different systems.
