This automatically-generated transcript is taken from the IT Pro Podcast episode ‘Building the supermarket of the future'. To listen to the full episode, click here. We apologise for any errors.
Adam Shepherd
Hi, I'm Adam Shepherd.
Bobby Hellard
And I'm Bobby Hellard.
Adam
And on this week's episode of the IT Pro Podcast, we're talking about the future of shopping.
Bobby
Now, if you live in the UK, you'll probably recognise Ocado's delivery vans. These colourful vehicles roam the streets, dropping customers' online shopping direct to their doors. What you may not know, however, is that these deliveries are powered by cutting edge technology such as AI, robotics, and distributed computing.
Adam
All of this technology is enabling deliveries to be made faster, more cheaply, and with fewer errors, with better results for both supermarkets and their customers. Joining us this week to talk about the innovation behind your weekly shop is James Donkin, CTO of Ocado Technology. James, thanks for joining us on the podcast.
James Donkin
Yeah. Hey, Adam. It's great to be on.
Adam
So James, before we kick off, could you give the listeners a little bit of background on what Ocado Technology does, and its relationship with the grocery business?
James
Yeah, for sure. So, first and foremost, Ocado is a client focused technology business. So we offer technology services, not only to the Ocado retail business in the UK, but also a range of other retailers across the globe.
Bobby
I remember, one of the first stories I covered for IT Pro was Ocado. And the day after someone messaged to say you've got it confused, there are two different companies: Ocado Technology and the Ocado Group. It is very specific what you guys do.
James
Yeah, so we're all part of a wider Ocado family. But there's a retail business that's focused in the UK on on the consumer offering, and getting great service to people. There's a logistics business. So you see the vans driving around, we have a warehouse to distribute groceries in, we employ a lot of people in the UK doing that. And the tech business supports both those operations, but also our retail clients globally.
Adam
So if listeners are looking for someone to blame for their shopping turning up late, it's not going to be James. So James, what kind of scale are you operating at within Ocado Technology?
James
Yes, there's lots of ways of measuring scale. I think also, first of all, the kind of scale of our clients. So we're basically operating in all the major regions of the world; we're in North America, supporting Kroger and Sobeys. We're in Europe, we're setting up in Australia and Japan. So there's a lot of global scope of what we do. And then actually the bit I find most interesting, I guess, is looking internally, our software and hardware suite of products internally, it's called the Ocado Smart Platform, or OSP, consists of more than 500 individually developed applications, as well as custom hardware. We have more than 200 development teams kind of building those systems. And then to get technical, we see roughly between 200 and 300 merge requests go for a day. About 1,000 completed rollouts of changes a month. Yeah, gosh, 7,000 servers or CPUs with a service running, our servers live, on average, less than an hour. So we're cloud based and cloud native. So we're very much in the kind of immutable server, a deployment is a killing the old servers and starting new ones, I guess a lot going on altogether.
Bobby
It's quite a lot to sort of keep up with; is it a struggle to keep up with the pace of technology and implement that across your business all the time?
James
It's a real balance. So obviously, we're doing a lot of activity, and we're delivering a lot of value. And then when we're looking at the technology stack and the tools we're using, we're trying to get a sweet spot where we don't want to be using the latest thing that's been released yesterday. But we don't want to be using technology that's too old. So there's a real art in judging which tech, when we should we migrate to new technology? How do we keep things as simple as possible? And how do we focus on automating jobs so our development and engineers can kind of get on with the actual problem, solving real problems rather than overly focusing on technology?
Adam
So speaking of automation, one of the I think most impressive elements of Ocado's kind of technological offering, is the automated picking robots that operate in some of the warehouses, I believe it's known as the Hive?
James
Yes, that's one of the more common names.
Adam
And that is, that's very impressive. If you've not, if listeners haven't seen them in action, we will post a link to a video in the show notes. And it is incredibly impressive watching them all kind of whizz about the, whizz about the grid. It's, it's almost hypnotic.
James
Yeah, it brings together a really interesting range of different disciplines as well. You've got kind of mechanical and electrical engineering, you've got custom silicon. Custom wireless technology, there's a whole bunch of software; software on the individual bots, but also kind of the hive mind that controls them is a really interesting piece of engineering. And then that that that hive mind, runs on the cloud So you've got kind of networking, cloud technology, all coming together to get a business outcome that's really revolutionary.
Adam
Yeah, we all thought Skynet was gonna come from military automation. Turns out it might actually come from Ocado.
James
Well, there's always, one thing about those bots is they live on their grid, you're not likely to see one whizzing down the high street anytime soon, at least not those type of robots.
Adam
Well, on that subject, as with any system, it's of course not flawless. Last summer, there was an incident where deliveries had to be cancelled due to some of those robots colliding in the warehouse. How do you solve for these kinds of challenges, you know, particularly when you're working with so much kind of cutting edge technology and so much custom technology, more to the point?
James
I think that brings in some of kind of Ocado's real cultural strengths, right back from the really early days of when you get events like that, and challenges, people come together from all these different disciplines and look at how can you improve the situation? How can you improve the product? How can you bring in solutions quickly, and that there's no one thing that solves that kind of area, but there's a lot of work, you can do interdisciplinary to help. And that's, that's Ocado's culture to me really, is we really, people at all levels of the organisation will look at a problem and look for great innovative solutions. And we're never afraid to step beyond the tech we've used in the past. So that the whole approach with the hive and the grid and the bots was a massive step forward from our older facilities, which are more conveyor based. And one of the great things actually working in Ocado, is we're always willing to challenge and move on to the next thing, we're never going to be too attached to what we did in the past, what's the right thing for us now? What's the way to kind of get that next 10x improvement?
Bobby
With that sort of constant upgrading to new technologies and searching for new things, does it create a problem with in house training and recruitment, finding the people that can keep up with those skills and deliver them?
James
Yeah, it's a tough market out there for hiring generally. But I think the biggest, one of the biggest attractions to work at Ocado is the actual range of problems that you can work on. There's very few companies where you can go all the way from control systems, firmware, hardware development, right through to kind of pure web development, mobile apps and consumer facing systems and everything in between. So actually, I think that's a massive strength, as is the really interesting people you'll find around the organisation, I never get tired of talking to people in their particular discipline, who are super deep experts in the thing that they're passionate about. And that's, that's one thing that makes Ocado great, and how do we cope with that? Well actually, we have development centres in lots of countries now. And you can kind of draw on the local talents to give a real diversity of approach.
Adam
And I'd imagine that's a strategy that's been kind of quite heavily supported by hybrid and remote working.
James
Yeah, actually it put us in a really good situation with moving to a hybrid or a remote working state, because actually, we were already used to working across locations with people who aren't physically present. We've got very used to that through our gradual expansion of different development centres in different countries, it was still a bit of a shock to everyone just kind of march home and set up. But I do think another Ocado trait is that ability to adapt and adjust. We've all got really good at this over, I'm sure you know this as well. But over the last couple of years, it's we've kind of been forced to get good at it. And people now are really good at that way of working. And in some senses, it's a great equaliser because when everyone's at home working on on their own setup, it doesn't matter whether you're in Barcelona or Sofia, or Poland or the UK, actually, we're all peers in that sense. So it's a, there's less dependency on where you are physically, although obviously if you're in a team building hardware, you do need to be near the machinery.
Adam
Yes.
Bobby
With all that great history of technology and new technology coming in, is there a particular element of your IT stack you're most proud of?
James
Well, that's a tough question. There's lots of good stuff. There's a really good, I think, the modern car platform that we built for the Ocado Smart Platform and the way we've architected the microservices for that, I'm definitely really proud of. We took a lot of lessons from the, I guess the older kind of year 2000-era systems that have been built more around kind of, you typically go for kind of a Sun box running Oracle, and you'd have like a stack in front of it, like N-Tier, maybe you'd run Apache at the front and then Tomcat; that was kind of old school databases. And actually, we learned from the weaknesses around that and the problems we've had scaling it to an approach which is cloud native, microservices; There's some really strong rules around not sharing data through databases, but sharing through API's. So it's a very clever approach to API's, small services, load balanced and scaling from the start. Including chaos engineering from day one was a big boon as well. So we've kind of grown up building the new platform. Initially, we used those Netflix tools, we've actually built our own in house ones as our platforms evolved, so we're still constantly terminating instances and servers across all of our environments, right through to production to prove we've got that resilience across availability zones. So that's worked out really well for us. And it's allowed us to scale internationally quickly, I.E we're bringing up, rather than having to build a data centre in every territory, cloud partners have got a lot of resources. And we can kind of roll out easily without needing to hire a lot of teams in the sites we're operating. So our development centres are not necessarily the same sites our customers are in. So we go where the talent is to hire. And that's not necessarily the people who are at the sites where we're picking groceries in. Yeah, cloud's great, development tools, strong. We had a very good experience early on with Google Cloud, back before it was Google Cloud. And we've grown up very much alongside Google and their stack, the Google kind of workplace tools, we went very early on to that and away from kind of on prem email and collaboration tools; that served us very well along the way.
Adam
Yeah. So you mentioned the kind of microservices and API led architecture that you've gone with. I'm just curious, when did you kind of adopt that as a, as a strategy? Because I'm getting the sense that you were probably fairly early to that in, you know, in the context of the wider business community, if you like.
James
Yeah, so the term microservices wasn't really very current at the time we headed down this route of small, independent deployable services. So this was way back back. I can't get the exact date for you. But yeah, way more than five years ago, so this was back at the inception of OSP. At the time, we'd launched our second, our first external client in the UK, and we'd reused our old platform for that launch. And we'd learned a lot from that. And we'd learned that we couldn't scale to 10 or 100 clients using that tech stack. So actually we got an amazing opportunity at that point, to basically start fresh, and build on firm foundations. But we put a lot of thought into standards and technical quality, and architecture, so we could build it from firm foundations. And the ability to effectively rewrite or change one small component without breaking other components using API's was a massive part of what we wanted to do. Because one thing we knew was, whatever we built on day one won't be what we want in five years or 10 years. So that ability to change and adapt without needing to do major rewrites is really important to us.
Adam
Yeah, that flexibility is absolutely crucial when you're kind of building at scale. And particularly when you're using kind of as many, let's say, actively developed kind of components and technologies, as as you are, you know, this, this kind of stuff is advancing on, you know, what seems like a daily basis. And being locked into a particular technology or particular way of working, can really bite organisations in the bum.
James
Yeah, it also enables one of our core cultural principles were team autonomy is really important to us. And effectively, as long as you're staying behind your API, as a team, you can then change which database you're using or adapt your, your service or change how its operating or even rewrite it without effectively disrupting the other teams who are consuming your API. So it enables, the the organisational culture enables the tech and vice versa, in harmony.
Adam
Yeah, absolutely. And having that kind of, let's say operational agility, is something that developers and engineers tend to really, really appreciate.
James
I think the other element I'd highlight is that is, effectively by providing cloud services and a high level of abstraction, it lets people get things done in a more self service sense. So problems we've had, in the in the past were waiting for infrastructure teams to deliver services. So waiting for an MQ server or similar for potentially months, rather than a sort of click to service approach and getting results immediately. And effectively, we've channelled on a golden path where the kind of supported main services that we recommend are very easy to get, and the team can just get access to SQS or, or a Postgres database. If you need something that's unusual, then there's a conversation to have. But we don't want barriers in the way of getting a typical service live.
Bobby
As you said, you don't want to be doing the technology now you want to be doing the technology of the future. So with that in mind, is there any new technology we can expect rolling out?
James
Well I guess first of all, I'd highlight. We had a an event called Re:Imagined very recently. And I definitely recommend checking the video out online, there. And there's a set of really exciting key technologies coming out there. And I'd highlight as well as our, I guess, our current generation bots, the 500 series, have a look at the particular photos and videos of the 600 series. It uses a whole bunch of really exciting tech that, I'm actually glad that event's happened now so I can talk about it more, but we did a lot of work getting in some really advanced additive manufacturing and 3D printing technology at short notice; some really interesting kit from HP. It's got amazing capabilities. And when you see that kind of almost organic look of that bot, and the lightness of it, it's the lightest bot of its kind. It really changes a lot of our other stack, which is where these initiatives, like the new new grid technology, you have a lighter bot, you can do more things with the grid, it can go into existing buildings rather than purpose built ones, and it's much faster to deploy. And then on-grid robotic pick is another super exciting tech, seeing the arm next to the bots moving around in harmony, it really is kind of the future of where we're heading towards; more and more products pickable through automation. Faster, more accurate orders, low price to consumers, more immediate. I've I've done a lot of picking; again, old school Ocado, I should add. I've been at Ocado 16 years, pre IPO. So in the early days, we spent a lot of time actually picking in our fulfilment centres, and it's less common now. But in those days, it was really all hands to the pump, get the job done. The ability to, to pick an order end to end so quickly and so precisely and dispatch it opens up huge possibilities in terms of different lead times, product range, freshness. So yeah, it's really exciting seeing that rolling out, super proud of that video. And then this is actually a personal interest of mine, because it's one of the toughest jobs in the warehouse. There's a bit of video in that announcement around a system called automatic frame loading. So frame loading, I've done this task, but I couldn't do this for eight hours. But I've done it for short periods. The full delivery tote, so the containers that have groceries in get loaded into metal frames, and those frames go into delivery vans, and that is a very, it's a labour intensive, they're heavy totes. And people do it amazingly fast. That that machinery for automating that process is really interesting to me, because the physicality of those of those totes loading into a frame. And the frames are not perfect. They have a long lifespan, they're steel and they get deformed. And the ability for the system to look and understand what's going on and get those things loaded is really, really amazing. Yeah, I've got I've got quite a long list like this as well. The hardware elements are definitely the ones you can see the motion when you look at it. And some of the software announcements are also very close to my heart. So Swift Router; I worked, one of the first things I worked on Ocado is our route optimization and slot booking systems, for a decent amount of time. The ability to kind of blend immediacy, sort of like sub-two hour orders, with kind of batch next day orders onto a delivery route in real time and get out there, it kind of gives the best of all worlds, you get the operational efficiency, but you also get people able to get those last minute top-up orders. So that's another I guess another highlight I'd pull out from that list.
Adam
Yeah, and that is something kind of as as an Ocado customer, that is something that I've always found very impressive. The kind of the, the the length of time, kind of within, you know, before the order gets dispatched that you have to, to actually amend the order and kind of thinking about the the tech that you would need behind that in terms of the infrastructure and the kind of the, the agility of the response times in the in the warehouse is very impressive. So on that topic, then, we've touched on it a little bit, but what impacts can deploying technology like this in warehouses, whether that's grocery warehouses, or, you know, retail more generally, what kind of impacts can that have on the actual consumers, you know, the shoppers, the people ordering this stuff for delivery.
James
Really, it's about taking away compromise between different elements, we want to give consumers everything they want with no compromise. So people don't like walking around a supermarket for hours pushing a trolley themselves and loading their own trolley. It's very stressful, particularly in COVID times, it's not a fun activity. So actually going back right to my early days in Ocado, giving people some of their time back so they can do more interesting things is a key part of it. So having enough smarts to make it easy to build your shop and get you the products you want. And also add the fun factor of interesting products. People like discovery but they don't want to have to worry about their day to day essentials so you want to really help with that. Getting people their groceries when they want it on their terms. So do you want to order on your way home and have it delivered really quickly? Is it top-up shops every day, is it a large grocery shop once a week? We want to support all those different elements. And then for operational efficiency, giving people the best prices. It's a very price sensitive market in groceries; margins are low. It's really, people, people want to, don't want to have to pay more to get good service, you want great service, great value, and on your terms. So that's where we're driving towards. Your example of like more time editing your shopping is a great example of, we don't do technology for its own sake. I worked on a bunch of work years ago to try and give customers more time to shop. And it was all about how can we give people more time. At that point, we were cutting orders off maybe four o'clock in the afternoon. And we were like, how can we give people all night to shop and edit their order? And then it was how can we use tech to enable it? So we start with the customer, we start with the outcome, and we use tech to to deliver it.
Adam
So looking at that, kind of, particularly that kind of price point question that that you mentioned, how is technology kind of really enabling that for you guys?
James
Yeah, so price point wise, this is about making the whole process of bringing the goods into a picking facility, getting them picked for a customer and get them delivered to the doorstep as efficient as possible. So driving costs out through automation of that process means that those benefits can be accrued to the actual customer who's receiving the goods. So yeah, that's, I guess you'd say half of what we do is trying to have an amazing proposition. And the other half is trying to get the the actual act of logistics of getting people there shopping as efficient as we possibly can.
Bobby
So it's just interesting, what you said there about getting the discovery element of shopping into the Ocado experience, because I mean, I personally hate going around the supermarket, it's like an hour and a half of time you can spend doing something more useful, but it does have that benefit that you'll you'll just find things like ooh, try that. I was wondering how you replicate that online?
James
Yeah, it's, it's a really big challenge, because you think about 50,000 product range, which is what you might typically expect in our online solutions. And if you're shopping on your smartphone, maybe you can see four or five items at once. So you're effectively, a customer has got a window into this enormous range. And this is where using kind of ML techniques and AI to present people options that we think they're going to really love. So you don't have to do the work of exploring every product to kind of front up interesting products. So looking at what people are buying already. And what they might love based on that experience is a really key part of this. And you get more, you can see more on a web browser than on a mobile app. But it's still limited. It's a very limited view of the of the entire range. And then yeah, there's all the frustrations you don't get with ecommerce though, because of course, I've been in the store before and thought what really is a search function, but I'm standing in the store, how am I going to search the store, well I've got to walk around in some way. So and search is so important as well, it's people's natural way of entering into this this world. And so improving search, improving product range. And then what the large range lets us do is have a lot more smaller manufacturers, niche products, you can find those, those things that you might not find in a typical size store. And you need a big range because everyone's got their own, unique sets of favourites, you will have base items and and things you add on but no one else is quite adding on in that combination. So you need to have a big range to cover everybody's unique needs.
Adam
Yeah, and I would imagine that there's a lot of data science and kind of data processing that that goes into that in terms of figuring out not just kind of what, what to recommend to each individual customer based on their purchasing habits, but also analysing purchasing trends on a kind of macro scale in order to determine what stock you're getting in, right?
James
Yeah, so this is a hugely different challenge when you're looking at a large basket that's edited. So 30 or 40 more items, it's a big shop. It's not the same as recommending you've bought one thing. So you're buying one more, it's more about based on your overall shopping habits, which things are interesting. So yeah, it's a really interesting data science problem that we've, we've addressed in multiple iterations over the years. But that's super interesting. But yeah, and then the whole supply chain forecasting efficiency, the other side of operational efficiency is food waste. Our typical food waste is about 0.4% versus an industry average of two or 3%. So there's a lot less wastage. And there's a few reasons for that; forecasting is important. But also, people ordering in advance, and then having a very efficient just in time supply chain, we effectively can buy some things in based on orders that are already placed. So we're more confident they're actually going to get used. So yeah, that driving, driving down food waste, not just at our supply chain, but also delivering things fresher to people so they have more of a life of the product is also important. That way, they're more likely to actually consume the goods as well. So it's a really, it's a really exciting area, again, a lot of data science and machine learning applied in this space. And actually, really interesting outcome because a lot of people internally are really passionate about environmental issues and areas like food waste, it's a great area to work on.
Adam
Yeah, I can imagine. So you mentioned kind of AI and machine learning a couple of times, how much of a challenge has that been to develop and rollout? Because that's a technology that is still, arguably in its infancy in terms of its maturity in the market. Have you had any kind of teething troubles with that at all?
James
Yeah, deploying it, there's definitely a lot of challenges. One advantage we've had is we've had people working on this kind of element longer than the term data scientist has existed. So if I roll back the clock, we had teams called analytics and optimization. And actually a lot more was statistical methods and like optimization routines, more so than machine learning then; machine learning has definitely grown up. But we've always had a strong, I guess, statistical, data-led teams. But it is, growing those teams in the environment is tough. There's also, it's a very fast moving area. So when we think about frameworks and ways of kind of deploying machine learning, kind of ML ops, that's a fast moving area. So we, it's important to stay on top of it. And I guess, I guess, look at where the industry is going with it. And my prediction over the next few years, is we're gonna see more, I guess, high level services, more frameworks, and more sort of standardisation in how machine learning is applied and deployed. And more, I guess, higher level services offered by people like cloud vendors. But at the moment, it's, it's, again, it's a tool in the toolbox, that the goal is to solve people's problems, whether it's clients and operating efficiency or customer problems. And actually, we won't, we don't use ML for the sake of it, if we can solve a problem with an optimization theory, where we're just as happy to run simulated annealing around colony as we are to do a kind of a neural net style approach. It's all about the outcome.
Adam
So James, do you have any predictions on how the kinds of technology that you're using and exploring at the moment will kind of bleed over into the world beyond grocery logistics?
James
Yeah, and I mentioned additive manufacturing earlier, but I think that's something not just at Ocado, but I think we're gonna see that come in more and more over the next five to 10 years and really change manufacturing and society in lots of ways; it's a really important up and coming element. If I get my crystal ball out a little, I think the driver of of lockdowns is going to improve tooling around remote collaboration and coding. And I also guess I'd point out there isn't really a silver bullet to complexity in software development, to kind of paraphrase Fred Brooks, it's, nothing is going to completely solve that, that building software is hard. But actually, we continue to improve as an industry in that space. And I think more and more harmonious development, hardware and software that works well together. When you look at companies like Apple and Tesla, that is something that I'm hoping we see more of as an industry, that really well integrated solutions that kind of work together well.
Adam
Well, I'm afraid that's all the time we have for this week. But thanks once again to Ocado Technology CTO, James Donkin for being with us.
Bobby
You can find links to all the topics we've spoken about today in the show notes and even more on our website itpro.co.uk.
Adam
You can also follow us on social media as well as subscribe to our daily newsletter.
Bobby
Don't forget to subscribe to the IT Pro Podcast wherever you find your podcast. And if you're enjoying the show, leave us a rating and a review.
Adam
We'll be back next week with more analysis from the world of IT but until then, goodbye.
Bobby
Goodbye.
