Retail faces multiple challenges – A key one is fixing the data layer

As the UK consumer battles rising inflation, interest rates, scary heating bills and even considering back-up generators to avoid potential power cuts, retailers have needed to consistently up their games to maintain customer loyalty. On the supply side, the latest uncertainties are an unwelcome addition to the disruption from COVID, Brexit, and even a chip shortage that took PS5s off the shelves, and is still-disrupting supply chains. Retail CIOs have reacted by attempting to digitally transform their data stack. This has largely been marked by a serious commitment to new IT architecture and a full embrace of cloud. Unfortunately, all that important, necessary, and beneficial work, while setting you up well for the short and even mid-term, isn’t finished yet. The retail CIO has one last, critical job: fix the back end database to access all the available power, flexibility, and agility needed to make cloud a true enterprise business platform. Don’t shoot the messenger. I know you think you already did that when you updated your IT during the pandemic. But as we’re entering an extended period of austerity and uncertainty, despite the boost you’ll get from peak trading at next month’s Black Friday, and the Christmas period, Database modernisation is the final step you need to take. Why the data layer has to be your next big retail IT project I’m not the only one who thinks this. McKinsey recently flagged the need for a retail sector “tech transformation imperative,” so retail can keep pace with the changing landscape--using tech to improve profitability. The management consultancy argues that only a few retailers have built “true omnichannel offerings, harnessed data at scale and implemented agile ways of working throughout their organisations”. To get there, you need to bring all your attention to the data layer. Consider why retail started thinking about cloud in the first place. What was once a US-only phenomenon of the Black Friday-Cyber Monday weekend now typically accounts for 20%+ of all online sales. Years ago we realised that it was unrealistic to prepare for that online ecommerce (and so business systems) spike by adding more on-prem resource. We knew from grim experience that all that extra capacity would just sit idle for the other 10 months of the year. For sure, some did avoid this by building out a temporary cloud presence just for peak days. But—quite rightly--the retail sector likes to have sound options available, so enabling infrastructure and processing capacity through cloud became standard. As a result, cloud is now embedded into retail and not just via IaaS or PaaS. But are you making the most of it? In reality, you’ll need to become even more nimble, and more able to dynamically price and source. The way database works in the real cloud—not the marketing PPT version This all means you need to get a lot more out of the cloud to help. My experiences working with two different North American retailers have shown me that properly managing and optimising the data layer is not a trivial challenge. The problem comes down to the way databases work in the real world. The customers I’m thinking of were still using old style corporate databases for the heavy lifting. These did a pretty good job, allowing the teams to concentrate on their applications and their new shiny front end. The retailers fully expected their old-style monolithic databases would carry on running the required numbers. Unfortunately, it turned out they couldn’t. The reality is that sticking an Oracle database on a virtual machine in the cloud just doesn’t work; they aren’t built for the cloud's distributed architecture. To cope, one of these organisations bought lots of small, old style databases and duplicated them to protect its data (the idea was that if one falls over, the other one doesn't)… which worked well enough--but it was too expensive and too complex to be practical for day-to-day ops. A big problem was outages; if these databases went down, they had to be manually hand-cranked back to life. The other firm—one of its biggest rivals--tried to solve this central cloud database problem by going NoSQL. On paper, again, this made sense, as databases like this are undeniably good at running in the cloud, scaling out, and so on. NoSQL was written for the cloud, after all. The retailer also only paid for what they actually used. However, this sort of software doesn't handle a form of data work which is very important to retailers: transactions. Lack of transactional capability immediately became an issue the second the business saw it was getting back inconsistent data. This meant that customers and vendors on its new marketplaces were getting unacceptable service. Resources had to be immediately thrown at the problem to keep these complicated new systems running. This was costly, and the brand was losing velocity, as so much IT effort was going into just keeping the system running rather than adding value. This is clearly not ideal for retailers. To make transactions work in the cloud, the major brand’s developer squad had to write lots (and lots) of extra code—all of which was done for them automatically in the old style SQL database, but was a bit of a mystery to shiny new NoSQL. Both these cases show that while it is possible to make traditional databases work in cloud retail, it can result in additional complexity and cost. It reduces the businesses’ ability to respond to change and mires the IT team in seemingly endless technical debt. But if Oracle won’t work in the cloud but neither will NoSQL… what can you do? The last mile to true digital retail transformation The last mile in your digital transformation journey is to avoid compromising your data layer. You need to make cloud a truly strategic weapon in the coming battle for survival in the next ‘seven years of lean’. More and more retailers are discovering that distributed SQL--which offers the best of NoSQL scalability and availability, plus the transactional consistency and familiarity of regular SQL—gives you a properly cloud-native data layer. An example is YugabyteDB, which was in fact the way both the retailers mentioned above escaped their data layer retail problems. Using a form of open source SQL like this allows your engineering team to focus on adding value to the business, not just continually patching up a compromised data layer that doesn’t work properly. So, retailers gearing up for Black Friday, Christmas and the complex world of UK retail in 2023 should be looking to adopt distributed SQL as the final piece in their digital and cloud journey. Why? Because it will allow you to feel confident that you are properly set up to meet whatever challenges the world has in store for the retail sector in the next few months. Martin Gaffney is Area Vice President – EMEA for Yugabyte