Search gets serious

Like it or not, your enterprise is drowning in its own growth. The explosion of corporate content — both in the physical form of documents, records, and data, and in the human form of personal knowledge — has pressed companies into a crisis: Find a way to tap into and effectively leverage that knowledge, or watch your company’s most vital assets wither on the vine.

Now, more and more enterprises are turning to search technology for answers.

Alongside this rapid accumulation of data, enterprise search has matured, growing more sophisticated to keep pace with the deluge. From the bud of indexing and retrieval, search tools are blossoming to include capabilities such as taxonomy development, metadata extraction, classification, and personalization.

Because it is increasingly viewed as a way to link users to content and to bolster the ROI of existing applications and systems, search is emerging as a critical link in the IT infrastructure.

“If you look inside a large corporation, anywhere you swing a stick you can find something that can be made better by search,” says Matthew Berk, research director at Jupiter Research.

“We are dependent on information technology for everything employees do. Within an explosion of information, the ability of workers to find

the information they need to do their jobs is vital,” he says.

Enterprise search tools fall essentially into two camps: external-facing, Web site search tools for customers and field workers; and tools designed to scour repositories that reside behind the firewall, such as file systems, databases, and business applications.

Natural language technologies in combination with linguistic processing and guided navigation capabilities are being put to use in search for external-facing self-service applications. Vendors focusing on this space include iPhrase, Kanisa, and InQuira.

Meanwhile, the internally focused enterprise search efforts, lead by vendors such as Verity, Autonomy, Convera, and Fast (Fast Search and Transfer), employ a variety of techniques including concept-based searching, auto categorization, taxonomy development, summarization, and personalization, all in an effort to improve the reach and effectiveness of search. Categorization groups content into related groups, whereas taxonomies help structure content by linking similar terms and concepts together. In addition to the larger vendors, a smattering of smaller companies specializes in specific areas such as taxonomy creation and categorization.

The mixture of all these technologies and techniques is helping drive search toward more of a discovery process that enables workers to unearth content they didn’t necessarily know existed.

Specifically, automating the corporate taxonomy development and categorization processes is helping propel search toward this looser mode of discovery.

Taking search to the next level is “when you don’t know the exact terms to query on, or you don’t know exactly what is on the network. We want to expose that information,” says Andy Feit, Verity’s senior vice president of marketing. “Taxonomies can uncover content you may not have known about before looking at it. It lets you expose content to more people.”

For external-facing site search, the job of search is not only to solve the query or problem, but also act like a customer service rep who can tell the user about additional services that could prevent the problem in the future, according to Andre Pino, vice president of marketing at iPhrase.

“We are finding people want to shift away from a search-and-find mentality and move toward discovery and managed dialog, especially for external search,” Pino says.

In addition, search tools are now employing more than just keywords to unlock information.

Autonomy, for instance, emphasizes the use of pattern-matching techniques that can identify concepts in queries and results. Autonomy’s system performs standard Boolean text searches, as well.

Another player in the search space, Endeca, uses a combination of text search and guided navigation to manage the relationships between pieces of content. The Endeca Navigation Engine suggests where users should go next by generating follow-up questions designed to broaden and refine a specific query.

Fast offers Live Analytics technology designed to give on-the-fly data and statistical analysis of content, which enables business performance monitoring, according to Fast officials.

Search Spreads its Wings

Search is less and less a standalone engine targeting a contained problem. Most large search infrastructure vendors have a healthy OEM strategy designed to push their technology under the hood of a variety of applications, namely content management, portals, CRM, and collaboration.

In addition to leveraging these embedded search capabilities, organizations should also consider a larger search strategy, Berk says. “Search is not just a problem to make individual apps searchable. [Enterprises] need to think about a shared services architecture that can be deployed enterprisewide, and have different line-of-business applications take advantage of [the architecture],” he says.

Aided by the use of XML, open APIs, and Web services in search platforms from large vendors, enterprises can standardize a search offering and stitch the technology throughout the business. “Enterprise search has been around for a while. Large companies may have 40 different implementations of search. Many are saying, ‘Why are we doing this over and over again? Let’s standardize on a platform that can be used [in] a variety of ways,’ ” Berk says.

Getting Specific

One of the growing ways to put search to use is through search-derivative applications, in which core search functionality is pressed into service for specific processes such as knowledge management, marketing, SFA, help desk, and training.

“The next generation of search beyond find is taking core language processing technologies — the engines, the neural network, and algorithms — and applying them

to different and existing business processes like supply chain and self-service as a way to enhance those processes” says Rob Lancaster, a senior analyst at The Yankee Group.

Furthermore, many enterprises are looking to search tools to solve emerging pain points, such as compliance. In fact, vendors are rolling out specialized toolsets designed for specific applications of search technology, and vendors are working with customers on custom deployments.

“You sell what is hot. Right now compliance around information management, Sarbanes-Oxley, records management, and e-mail management is a big issue. Finding that critical piece of information or document can make or break an audit, particularly as enterprise content grows exponentially,” Lancaster says.

Many search vendors have developed specific modules or added capabilities to existing products to address compliance or to target other applications for search.

Autonomy, for instance, created two divisions within its business to focus on call center and compliance applications. Audentify, for call centers, and Aungate, for communications compliance, are both based on Autonomy’s intelligent data operating layer, but they use search and indexing in different ways. Fast, for its part, plans next year to introduce a series of search-based applications targeting compliance and fraud detection.

Some enterprises, such as independent brokerage firm LPL (Linsco Private Ledger), initially eschewed early search technologies due to doubts about accuracy, but came back around as search has proven itself as a means to lower TCO and raise the ROI for existing systems.

LPL tapped natural language search vendor iPhrase to add customer-facing search to its password-protected Web site for LPL brokers/dealers in the field. LPL hoped search technology could alleviate pressure on its call center, in which 50 people fielded about 50,000 calls per month.

“One big reason we wanted search is that nearly half of the calls from brokers could be answered if they knew where to look on our site. We are wasting our time publishing those documents if people can’t find them,” says Mike Hamm, an assistant vice president for special projects at LPL. “We wanted to provide quick access to answers. Content was buried two to five clicks deep in our intranet site. We wanted to bring that information to the forefront.”

Rather than build a new knowledge base from scratch, LPL’s focus was to leverage content and technology investments already in place, Hamm says.

To ensure accurate results in the specialized financial industry, iPhrase uses an English language dictionary with a financial module on top to ensure that queries for stocks and bonds did not return information about chicken soup and glue. The iPhrase engine also allowed LPL to tailor the dictionary to custom terms such as “static asset management,” which is defined differently throughout the financial industry.

Further out, LPL hopes to leverage the iPhrase search product to enhance LPL’s investment research efforts, using cross-selling modules to deliver research related to specific topics, according to Hamm.

Looking toward the future, enterprise search technology will continue to expand beyond its seek-and-find roots, blurring the lines between efforts such as business intelligence and knowledge management in an effort to present a full view of information assets within a company.

With the increased use of technologies that can slice through structured and unstructured data, then identify and analyze patterns in that data, search is bleeding into business intelligence, according to Berk.

“Search is the next business intelligence. Search will replace that layer of OLAP and BI that used to fit on top of the database,” Berk says. “Search is breaking out of search.”