Williams-Sonoma uses big data to zero in on customers

What happens when you combine big data, statistical modeling, and marketing analytics? If you’re Williams-Sonoma, you gain the ability to process 50 million rows of data daily, enabling you to tailor marketing to individual consumers at huge scale.

Mohan Namboodiri, vice president of customer analytics at Williams-Sonoma, a public company with nearly $4 billion in revenue and 30,000 employees, sought to improve the retailer’s marketing analytics. The company suspected that online ads and emails were more effective than catalogs for certain customers. And it wanted to find a way to achieve marketing attribution at scale — that is, to understand the effect of each campaign that led to a sale for each customer. This in turn could enable marketing campaign budgets to be reallocated to target individual customers, not just segments of customers.

[ Download the Big Data Analytics Deep Dive by InfoWorld’s David Linthicum for a comprehensive, practical overview of this booming field. | Harness the power of Hadoop with InfoWorld’s 7 top tools for taming big data. ]

Marketing analytics used to be simple enough: Trace your campaign code to your buyer, and you understand your campaign’s effectiveness. The marketing campaign data was relatively manageable, relating to catalogs and other direct mailings, email blasts, ads, telemarketing, and not much more.

But the marketing tools, data sources, and data size kept growing. Today there are mobile messages, banner ads, online search campaigns, in-store promotions, loyalty cards, recordings of every mouse click, and more. Augmented data sources keep expanding, with third-party data covering customer demographics, credit scores, and so on. Plus, there are factors beyond marketing’s control, such as seasonal buying habits and customer buying histories. Finally, there are the challenges of managing the marketing systems; many marketing strategies rely on disparate applications or organizations.

The latest marketing systems make it easier to tailor offers to customers for a new product, discount promotion, or loyalty points. However, to be most effective, marketers need to thoroughly understand what offers drive which customers, while being careful not to fatigue their customers — for example, with constant email blasts. All this makes it difficult to have both one view of the customer’s activity and the ability to act on it in the most precisely effective way.

The problem is multifold. Due to the volume, velocity, and variety, it’s a big data problem. It’s a data silo problem due to the variety of disconnected systems, and trying to understand the effectiveness of marketing campaigns on individual consumers for given factors is a big statistical analytics problem. For instance, some customers might react to an email campaign only during the Christmas season, while others might click on email campaigns and engage to buy throughout the year.

Moreover, in order to get a clear view of the information, you need intuitive dashboards that show the relevancy of campaigns on purchasing behavior.

The marketing attribution SaaS solution that Williams-Sonoma selected was created by UpStream. The UpStream development team employs a multidisciplinary approach combining backgrounds in business, marketing, computer science, math, physics and statistics to solve complex business problems.

To handle both the big data and data silo challenges, UpStream’s hosted service uses Hadoop as both ETL (extract-transform-load) middleware and as a distributed processing data store. Hadoop is used to prepare the data from marketing programs and score consumers’ behavior: Consumer X clicked on an ad and purchased a product, and so on. Williams-Sonoma provides its in-house marketing data (website visits, mobile sites, call center data, and more) to UpStream, which aggregates it with third-party consumer data from brokers like Experian.

The data aggregation enables UpStream to tackle a number of tasks. First, it can score the integrated data with Hadoop to instantly drive the right marketing campaigns to individual consumers at a massive scale, processing more than 50 million scores per day per client. Second, it enables Williams-Sonoma to have a single dashboard of all campaign touches, interactions with the retail stores, online behaviors, and purchasing.

The data aggregation enables statistical analytics as well. UpStream employs a novel approach, having created survival regression models (also known as hazard models, or time-to-event models) in the R language. Those models have been used successfully in the health care industry, but for much smaller data sets covering a few hundred patients.

UpStream retrofitted the models to handle attributive marketing for retail to analyze the weighted effect of each campaign on a consumer’s purchase. With this understanding, budget can be allocated much more effectively. To make this part of the solution more scalable, UpStream uses a commercial edition of R from Revolution Analytics. Finally, the models can be used to predict the likelihood for a given customer to buy based upon a marketing campaign.

UpStream and Williams-Sonoma continue to work together to create customized, targeted campaigns to individual consumers. Their models let them determine which consumers are left in which kind of marketing stream (email versus regular mail), as well as which are taken out and to only be targeted with online campaigns such as targeted banner ads.

Although Williams-Sonoma will not disclose detailed results, Namboodiri indicates they are encouraging, seeing improvements on scale and qualitative levels that have not been available until now.

This article, “Williams-Sonoma uses big data to zero in on customers,” was originally published at InfoWorld.com. Read more of Andrew Lampitt’s Think Big Data blog, and keep up on the latest developments in big data at InfoWorld.com For the latest business technology news, follow InfoWorld.com on Twitter.