Sometimes it’s OK to treat people like numbers

One of the most hackneyed complaints in the modern world is the notion that being “treated like a number” is always a bad thing.

This isn’t to say I’m disparaging the importance of treating customers as individuals, personalizing engagement with each of them, maintaining a 360-degree view of each, and keeping the human touch when possible. But given the focus of big data and its myriad applications on statistical analysis, most modern organizations have bet the business on treating every customer and everything else “like a number” (aka leveraging statistical data analysis).

[ Download InfoWorld’s Big Data Analytics Deep Dive for a comprehensive, practical overview of this booming field. | For a quick, smart take on the news you’ll be talking about, check out InfoWorld TechBrief — subscribe today. ]

The concern has always been that big institutions are supposedly hiding behind opaque, abstract, impersonal “numbers” that prevent them from seeing each customer as a unique individual. However, today’s irony is that the business world increasingly leverages the numbers (aka powerful statistical models and other advanced analytics) to drive fine-grained personalization of every customer recommendation, interaction, transaction, and experience.

Nonetheless, the “impersonal numbers” concern still retains a kernel of validity in the era of next-best actions powered by big data analytics. Inline recommendation engines (aka next-best action or decision automation infrastructure), which drive much of the personalization, often rely on extraordinarily complex analytical models, business rules, and other embedded logic.

As I noted in a recent LinkedIn post, the increasingly intricate yarn ball of logic that drives personalization can make it difficult to produce a full, transparent accounting of all the context, data, rules of thumb, and assumptions upon which specific automated decisions were made.

Just as problematic, personal responsibility for those decisions grows blurrier as the roster of data scientists and subject-matter experts who build and tune that logic grows more crowded.

Furthermore, these smart people collaborate to produce logic that drives highly unique, personalized, and situation-specific recommendations and guidance on across one or more engagement channels: portal, call center, smartphone interface, and so on. The specific recommendations and next best actions that their models drive cannot always be anticipated by any one human in advance.

My feeling is that it’s OK to treat people “like numbers” as long as they are the correct numbers, drive desirable outcomes, and are available for deep introspection and clear explanation. These thoughts came back to me in spades as I read an excellent recent article on “deconstructing recommender systems.” The piece, authored by Joseph A. Konstan and John Riedl, discusses a statistical modeling approach — singular value decomposition (SVD) — used by Amazon, Netflix, and others in their personalized recommendation engines. It discusses these and other modern-day recommendation engines within the context of industry innovations dating back to the early 1990s.

In particular, this passage jumped out at me:

Have you ever wondered what you look like to Amazon? Here is the cold, hard truth: You are a very long row of numbers in a very, very large table. This row describes everything you’ve looked at, everything you’ve clicked on, and everything you’ve purchased on the site; the rest of the table represents the millions of other Amazon shoppers. Your row changes every time you enter the site, and it changes again with every action you take while you’re there. That information in turn affects what you see on each page you visit and what e-mail and special offers you receive from the company.

As did this one, describing how data scientists use SVD in modeling the dimensions of personal preference/taste from these tables of numbers:

The technique involves factoring the original giant matrix into two “taste matrices” — one that includes all the users and the 100 taste dimensions and another that includes all the foods and the 100 taste dimensions — plus a third matrix that, when multiplied by either of the other two, re-creates the original matrix….the dimensions that get computed are neither describable nor intuitive; they are pure abstract values, and try as you might, you’ll never identify one that represents, say, “salty.” And that’s okay, as long as those values ultimately yield accurate recommendations.

Let me net that out for the layperson: The more dimensions of individual preference/taste that your data scientist attempts to capture in their decision-automation model, the more complex the model grows. As it grows more complex, the model becomes more opaque, in terms of any human (including the data scientists and subject matter experts who built it) being able to specifically attribute any resultant recommendation, decision, or action that it drives to any particular variable. Therefore, in spite of the fact those “values” (aka numbers) ultimately yield accurate recommendations, the number-driven outcomes become more difficult to understand or explain.

Hence, an increasingly complex subject-domain relationship may cause the associated data model to become less comprehensible, which in turn, makes the decision-automation process less transparent. This can occur even in those business scenarios where all the numbers and all the recommendation-engine logic that leverages them is on the table and available for anybody to inspect.

If nobody, not even your data scientists, can explain or justify the numbers, then you have a problem. That’s a potential Achilles’ heel in any complex data-driven application, not just recommendation engines.

This story, “Sometimes it’s OK to treat people like numbers,” was originally published at InfoWorld.com. Read more of Extreme Analytics and follow the latest developments in big data at InfoWorld.com. For the latest developments in business technology news, follow InfoWorld.com on Twitter.