Why we lose out if we leave everything to algorithms

“Bad Romance”—an amazing piece of journalism by Sarah Jeong at the Verge—implicitly answers this question. It’s about the romance genre on Kindle Unlimited, and the royal rumble that’s been happening this year. It’s a story about how “rampant algorithmic tricks” have ripped apart an author community. Deep down, the cause of the controversy is about data, automation, and how people try to game systems that are too focused on KPIs.

The story is about over-reliance on algorithms and KPIs. Royalties and high ranking on the Amazon stories are everything, and authors do whatever they can to get there. When Amazon paid according to books read, authors published shorter and shorter books. Amazon’s fix was to pay based on number of pages read. The outcome? Books got longer and longer, and authors jumped people to the last page on opening the book.

“Gaming best-seller charts was already a practice in the world of print, but technology enabled that kind of behavior on an entirely new level of sophistication,” says Jeong. Here’s the rub. In all walks of life, technology, data, and automation provide enormous benefits, but sometimes, they encourage the worst in us to come to the fore. As competition has intensified, so have the tricks authors will perform to outdo one another. When the algorithms rule the system, humans find ways to exploit it. Who knew romance writing could be so cutthroat?

As a result, romance author Margaret Bates notes that some authors are moving to publish on other platforms (like Apple, Barnes & Noble, and Kobo) that use human curation as well as automation.

The problem goes far beyond Kindle Unlimited, of course. People love league tables, and governments play to this demand. In the United Kingdom, school league tables are, theoretically, a good measure of a school’s quality. Unfortunately, chasing high league table positions leads to schools gaming the system. Amanda Spielman, chief inspector at Ofsted, complained in December 2017 about the rise of “offrolling”—where pupils are expelled before they sit their GCSEs at the end of Year 11. This helps schools improve their performance goals, but hits children with special needs particularly hard.

The corporate world suffers, too. Take, for example, measuring employee performance. If you judge based entirely on metrics, your employees will work to that system. “You can’t prevent people from gaming numbers, no matter how outstanding your organization,” says by Sir Andrew Likierman in the Harvard Business Review. “The moment you choose to manage by a metric, you invite your managers to manipulate it. Metrics are only proxies for performance.”

Liz Ryan, CEO of Human Workplace, writes on Forbes that “you’ll never cultivate Team Mojo by measuring more of your team’s activities, but you can kill your culture by measuring too much and evaluating people based on their numbers more than other factors.”

Solutions lie in the same solutions Apple, Barnes & Noble, and Kobo have for the digital bookstores: keep people involved. While numbers are excellent guides, the human knowledge of the immeasurable is vital too.

In any league table, be it in education, health care, or transport, the devil is always in the details. More sensitivity around this from all sides would greatly help: Politicians and the media have a role to play in how they speak on these topics. They must take care to consider the nuance behind any number.

In business, any KPI needs to be understood correctly, too. Likierman recommends increasing the number of KPIs being measured and varying how the numbers are sourced. “Metrics should have varying sources (colleagues, bosses, customers) and time frames,” he says. Measurement is fine, but it requires excellent human interaction and understanding to complete the measurement of the success of any metric.

Algorithms are here to stay. The fighting among the Kindle Unlimited romance writers is an indication of the risks of overreliance on automated systems. People can spot when a metric is being exploited. We can also grasp the unquantifiable—monitoring and judging success. Human oversight and intervention are more important than ever to understand the nuances an automated system cannot yet deal with.