Where’s the value in Retail big data?

Written by David

Retailers have never had it so tough. Disruptive competitors have forced a choice between a race to the bottom on price and quality, or getting smart and winning on the data. The traditional levers of broad-brush discounts, promotions, BOGOFs and bundles are now often an insidious source of profit decline. Winning companies know that not all customers are created equal and they use data to truly understand their customers needs at a deep level.

The advent of smartphones, widespread personal computing and cheap digital sensors produce a pervasive and overwhelming amount of data that has already revolutionised the retail sector. On both sides of the checkout, information has become a source of power. Savvy shoppers showroom by checking online prices from the shop floor, while retailers tap predictive technology and analytics to uncover shopping patterns.

Even though reports of the Death of the British High Street and the consolidation of the global retail sector may be exaggerated, there is no question that companies such as Amazon present a significant threat. To compete with these game-changing companies, retailers need to enhance their ways of working. Those that leverage their data insights start to level the playing field.

Many retailers actually have an advantage over Bezos when it comes to data. They know their niche market and customer needs at high resolution. Those who are thriving are finding new ways to embed deep customer insights into their business.

Below we share some of those methods and what information retailers must uncover to outperform their competition.


Data comes in many shapes and sizes

First we start with the various sources of data. For retailers, these include:

  • Ecommerce transaction data – both archives, and real-time feeds.
  • Electronic point of sale (EPOS) data identifying spending patterns and trends by extracting raw data from the systems that process retail transactions from across vast inventories and hundreds of millions of transactions.
  • In-store sensor data the decreasing cost of digital sensors and network equipment, along with the increasing promotion of the Internet of Things (IoT), allows physical stores to be embedded with sensors. These allow retailers to compare trends such as in-store foot traffic vs planograms, dwell time vs discount levels, customer Wi-Fi and network usage. Anecdotally we have also even seen some proactive management co-opting checkout employees to estimate customer demographics on a per transaction basis.
  • Cameras use of dedicated infrared cameras to monitor queue length or optimise planograms.
  • Supply chain systems extracting data from supply chain systems can augment EPOS data and provide predictive analytics for seasonal inventory management.
  • Social media metrics beyond setting up social media pages on Facebook, Twitter or other platforms more advanced retailers use social media to gain real-time insights into drivers of customer behavior.
  • Customer online engagement with the advent of machine learning models that can process natural language data, smart retailers are now sifting through their reviews and post sale interactions with customers to gauge satisfaction or changes in sentiment on a systematic basis.


Big data is an engineering challenge

Once the initial excitement of tapping the potential of so many new data sources has passed, the collection, preparation and analysis of structured and unstructured datasets is no small task. Data volumes are growing exponentially and, because this data is both complex and often siloed in many different areas of a company (often controlled by different sets of stakeholders and management presenting problems for integration and ownership), making sense of it all is a formidable task. Many retailers are either unsure of where to start, or how to progress to the next level.

Make no mistake, this challenge is primarily an engineering one that requires several steps:

  1. Gathering the data from a number of different sources like different database architectures server logs, custom data sources, geo-location data and even publicly available data.
  2. Integrating diverse data sources, whilst at the same time validating, and cleaning or discarding, dubious and erroneous data points.
  3. Picking the right analytical tools and machine learning methods to extract predictive and valuable insights.
  4. Often overlooked is the presentation layer, where meaning is communicated visually to stakeholders in the final step. This enables them to view the insights in a clear way in order to make faster and better decisions, with more confidence.


Better profits with Data Scientists

Thankfully IT and Marketing leaders are not left to figure all this out using existing competencies in their business. Data scientists specialise in analysing large volumes of diverse data and can help retailers gain a deeper understanding of customer demand. By applying their big data skills, retailers can make shopping more personalised, relevant and convenient for their customers.

But since adoption of data science has increased so rapidly, how does one assess the accuracy of any analysis? How can leaders screen for the important skills and traits in their data science function?

  1. Look for a strong mathematical and scientific background

This should be a pre-requisite but in the current market, with top talent hunted almost to extinction, it may be tempting to reach for a candidate with a limited grasp of the scientific fundamentals. Maybe your current first choice can use a few R packages and is handy with vlookups in Excel. Maybe they tick the boxes for the immediate task at hand, and you have been under pressure to fill this role for several months. But hiring a short term stop-gap could send the reliability of your analysis and business information downhill fast. Recovering from that can take a long time, especially once other departments have lost trust in your team’s output.

  1. Seek out commercial experience

Just as no battle plan survives contact with the enemy, no simple model survives contact with the noisy world of retail. And the most ornate and elegant retail model may fail for an entirely unpredictable (and entirely human) reason such as a renegade store manager, a sudden stock-out, or a group of POS terminals re-set to the wrong time zone. Candidates who cannot adapt their approach to the dynamic situation at hand may become overwhelmed and make a hasty exit, sending you back to square one.

Similarly, candidates who are all brains and no common sense could overlook a crucial factor which invalidates their analysis. Did a negative value get classed as a refund, a discount or an error? Did a geographic analysis overlook competitor locations? Commercial experience is a valuable asset and worth probing for in the interview.

  1. Grit

In big organisations, the left hand frequently doesn’t know what the right hand is doing. Did customer loyalty shift because of marketing or because the core product is now manufactured at a new plant and there are new QA issues? A data scientist must be able to expect the unexpected, and build robust models and tests which still deliver confident results in spite of all the moving parts in retail companies.

  1. Maturity

Look for a mature approach to failure. A candidate who is uncomfortable returning negative profit results, or who talks up a correlation at low statistical confidence should set alarm bells ringing. Failure of a hypothesis is just as useful as proving it is correct. What’s important is surfacing the truth, so that accurate decisions can be made to inform future strategy.


What are the benefits from all this effort?

Cracking this data-engineering challenge can turn in-store and e-commerce, customer data sources into a major competitive advantage for retailers. Having access to concrete results, at high statistical confidence can drive advertising and promotional effectiveness, and quantify ROI to a high degree of precision. Marketing leaders are constantly looking for better ways to attribute their activities to increased sales and ultimately demonstrate the value they bring to the business. Data holds the key to achieving this with clarity, extracting clear ROI from the marketing campaign portfolio, and reallocating resources to campaigns with the biggest growth potential.

The reward for smart data analysis is to drastically increase sales while lowering cost of sales. Insights from data analytics grant the ability to cross-sell to high-value customers with impeccable timing and drive a customer retention policy that predicts churn on an individual basis. Then for an encore, retailers can protect their profits by better management of their inventory. By better predicting seasonality in demand you will not have cash tied up for storing, and managing, stock that will be hard to sell.

Thriving retailers use data scientists to answer critical questions like:

  • Who are our most profitable and loyal customers?
  • What, How, When and Where do they like to buy from us?
  • How do I reach them in a cost-effective way?
  • How do I improve profits with the right promotions?
  • How do I improve our cash flow through smarter inventory management?

The retail press is filled with stories of bellwether businesses who have not survived, so the pressure on business decisions has never been higher. As a business becomes more data-driven, its leaders will make decisions based on validity, rather than intuition. They will continuously seek ways to use data analytics to measure, understand and review the impact of every decision. This ultimately reduces risk, shortens the time to profitability, and keeps them one step ahead of competitors.

    Download Form Title

    You're nearly there! Just enter your contact details and you will get instance access to the whitepaper