Microsoft Copilot review: AI baked into your apps

Remember Microsoft Clippy? The animated paperclip, first released by Microsoft in Office 97, raised blood pressure across the globe with its incessant attempts to help. He lasted five years before Microsoft pulled the plug. The Cortana voice assistant, released in 2014, lasted two years longer. Microsoft CEO Satya Nadella said that it was "as dumb as a rock" (which is odd because it's still lurking, turned off by default, in Windows 11).

Microsoft promises that Copilot will be far better. The service, based on OpenAI's GPT large language model (LLM), was originally announced as Bing Chat in February 2023, followed by an announcement of Copilot for Microsoft 365 last June, and Copilot as a tool in Windows 11. Both of these began rolling out last Autumn.

Microsoft is betting big on this technology, creating a plug-in ecosystem that enables online service providers to integrate their products with it. It has also announced other Copilots including Copilot Studio, a development environment for non-techies to create their own custom AI assistants. To do this, they use data connectors which turn Copilot into a retrieval-augmented generation (RAG) tool.

Microsoft Copilot review: Setup

We wanted to test Copilot as it integrates in Windows and some Microsoft apps. Getting access to the Windows integration requires a little work at the time of writing in early March. You must join the Windows Insider program and subscribe to the Dev channel to get the most cutting-edge updates. Select the option to get those updates immediately, restart, and then check for updates. Once you've downloaded those, there's another restart, and then you're off to the races. You access Copilot in Windows via a taskbar icon, and it pops up in a sidebar.

Copilot on Windows is a way to assist with some operating system features, along with a chat and search experience akin to the online offering. It will execute basic functions like 'Turn on dark mode' on command. It asks for confirmation, meaning you have to click to reassure it. When we asked it to execute without confirmation, it agreed. On the next instruction it asked for confirmation but didn't wait for a reply before telling us that it had completed our request to turn on dark mode (it hadn't). Not a great start.

Copilot's ability to deliver clear instructions rather than lines of search results is useful. "I don't have Microsoft 365 installed yet. Could you set it up for me using my Microsoft account?" we asked. It delivered a set of step-by-step instructions to follow. It's like having a perky, helpful assistant on hand.

As something integrated into Windows, we'd like Copilot to one day take care of complex tasks on its own rather than telling us how to do them. OpenAI is reportedly working on AI agent technology that will control operating systems and browsers, chaining together simple tasks to do more complex things for users. Perhaps that would make Copilot's OS control capabilities more useful. For now, though, it's effectively a smart alternative to search for configuration solutions that will in some cases execute simple instructions itself.

Copilot will also do the kinds of things we're used to in the online version, such as answering general research questions. Yes, it will also still write poems about your cat. Integration with the Edge browser was especially useful. This feature, which you must turn on manually, enables the AI to reference what you're browsing. Combined with its own ability to browse the web, this makes it a powerful research tool. Here, it summarizes a Wikipedia article about Nadella, and then browses the web to answer more questions while retaining the context of the discussion. This is a smooth functional integration:

Copilot also now features plugins from the likes of Kayak, Instacart, and Opentable. Again, Microsoft's traditional problem – being slightly less joined-up than it should be – was apparent here. Setting the region to Canada would ideally have tailored the plug-ins, but it still showed me a plugin from Klarna, which it said was only available in the US.

Asking for a recipe produced a fast, accurate response. When we asked for it to be delivered, it provided a link to a shopping list on instacart.com, rather than Canada's instacart.ca, making the service useless. The same US-centric complaints are true of the online Copilot service too, though. If Microsoft makes the plugins more inclusive, integrating them into Copilot could be a big time saver for the rest of us.

Microsoft Copilot review: Microsoft 365

Copilot also now integrates with Microsoft 365 (formerly Office). This functionality, initially only available to enterprise license customers with a minimum of 300 seats, has become gradually easier to get. Microsoft opened it up to Microsoft Business Premium and Standard Customers, eliminating the 300-user minimum. There's also a way for home users to get access to Copilot functionality in Microsoft 365 apps via Copilot Pro. This gave us access to the service via a Microsoft 365 Personal account, which costs £19 per month per user at the time of writing.

Using this in Word gives you a Copilot icon directly in the document, and delivers some stunning results. We asked it: "I am teaching a class of seven-year-olds about the water cycle. Create a lesson plan with activities."

This is what it provided:

An elementary school educator we showed this to who was teaching that topic to her class was blown away. You can do this in the online Copilot, but having it formatted into a Word doc is extremely useful.

You can also query Copilot in Word about existing documents. Opening a long Word document enabled me to ask Word multiple questions about its content. This is a little like having a dialog with the document itself:

<image 5 - copilot word conversation>

It's a shame that Copilot in Word doesn't integrate with Edge, and we were concerned by the bizarre results it delivered when we tried. We asked it to create a document outline based on a website we were reading about NATO's nuclear deterrence policy. Instead of a "sorry Dave, we can't do that" response, Copilot wrote an outline in Word about the benefits of meditation for mental health. Perhaps ChatGPT has achieved enlightenment and was trying to tell us something?

Microsoft Copilot review: Integration 

There were more Microsoft shenanigans when using Copilot in Excel. We loaded a CSV file containing daily eurusd and eurgbp currency pairs, but the Copilot button remained grayed out. We surfed around and found that it can only handle files that are saved to OneDrive. This isn't consistent with Copilot's integration in other Microsoft 365 apps.

After that confusion, we found some impressive functionality. Copilot for Excel will suggest formulas on request. In response to a natural language request, it told us the formula to make a new column based on our existing date column but with just the day of the week, and even created the column for us. It also bolded the ten top values for the eurusd currency pair column on request.

We found data analysis less impressive. After letting Copilot convert the data in the sheet to an Excel table, we asked it what trends it could identify about the eurusd currency pair in February 2024. "Sorry, I don't have a helpful answer," it replied.

Instead, the AI suggested strange things that we didn't want, including questions like "how many eurusd are there?" We asked "Were there any big movements in eurusd during February 2024?" and it gave us a single average figure for eurusd for the month. It just wasn't useful.

This lack of context awareness in Copilot for Excel is strange because we regularly analyze data sets using GPT-4 - the LLM behind Copilot - and get useful results when asking similar questions. Without Copilot's moderation, OpenAI's tech instantly tells us what the figures represent and can identify key trends in the data. It's as though Microsoft put guardrails around OpenAI's ability to detect context so that it wouldn't make mistakes, but lost some magic as a result.

Microsoft Copilot review: PowerPoint

Copilot for PowerPoint offered to create a presentation about something, and even suggested an idea: creating one about marketing strategies. So that's what we asked for. It said 'working on it' and then hung. Multiple times. Our internet connection was up, and we were working on a brand new Windows and Microsoft 365 instance. After reloading the app and eventually rebooting, we had to admit defeat.

Copilot Pro also supports Outlook. Microsoft is planning enhancements that will let Outlook users schedule meetings with natural language by pulling attendee information from Microsoft Graph behind the scenes. There is (or soon will be) a 'Prepare' button to generate a report on upcoming meetings including relevant files, and it will also summarize email threads to make your day more productive. However, Copilot only supports Outlook if you use email with a domain owned by Microsoft. We don't.

Copilot in OneNote looked promising after the Copilot icon disappeared from the interface and we spent ten minutes trying to get it back (eventually we had to relaunch it via the app's search function). Copilot allowed us to ask a question concerning a lengthy page of notes from an interview, mimicking the kind of functionality available in Word. This could be highly useful.

The feature we were trying to use in this image, which extracts to-do lists from selected text-based using the right-click menu, did eventually work. But along the way we felt the Microsoft productivity distortion field in full effect:

Microsoft Copilot review: Is it worth it?

Copilot is an incredible achievement. The underlying technology is revolutionary, and Microsoft's web-based execution is buttery-smooth. Its still-in-preview attempt at integrating it into Windows is impressive, but we found more serious niggles and limitations in the app integrations.

Overall, Microsoft seems to doing a decent job of packaging functionality that's available elsewhere into Windows, but if you want the Microsoft 365 versions of Copilot, you'll pay top dollar for them. Copilot Pro costs £19, roughly 3.5 times as much as the Microsoft 365 Personal account that it plugs into. Copilot for Microsoft 365, which offers more app integrations like Teams, costs £24.70 per user per month - again, around 3.5 times the per-user cost of the Microsoft 365 Business plans that it supports.

Unless you have an urgent use case for direct LLM integration into apps, it's hard to see that amount of value in Copilot when standalone services like Kagi and Perplexity.ai are available. These offer access to multiple LLMs, including not just OpenAI's but others from Google and Anthropic. They include the ability to upload and analyze multiple data types for as little as $20 per month. You can automatically draft a letter of complaint to your home association or analyze your household budget using those instead. Sure, you'll have to copy the results somewhere. We can deal with that.

Kudos to Redmond for coming so far with the dream of generative AI (due in no small part to OpenAI). We love Copilot online, and Copilot in Windows shows equal promise. If you absolutely must have an AI in Excel create formulas for you (or if you can get Copilot to do anything in PowerPoint), then go for it. But for now, we're putting the Copilot Microsoft 365 app integration that we tried in the back seat.