Windows AI feature that screenshots everything labeled a security ‘disaster’

A security researcher is concerned about Windows 11’s new Recall feature after testing it.

Microsoft is about to launch a new AI-powered Recall feature that screenshots everything you do on your PC. Recall is part of the new Copilot Plus PCs that are debuting on June 18th, but experts who have tested the feature are already warning that Recall could be a “disaster” for cybersecurity.

Recall is designed to use local AI models to screenshot everything you see or do on your computer and then give you the ability to search and retrieve anything in seconds. There’s even an explorable timeline you can scroll through. Everything in Recall is designed to remain local and private on-device, so no data is used to train Microsoft’s AI models.

Despite Microsoft’s promises of a secure and encrypted Recall experience, cybersecurity expert Kevin Beaumont has found that the AI-powered feature has some potential security flaws. Beaumont, who briefly worked at Microsoft in 2020, has been testing out Recall over the past week and discovered that the feature stores data in a database in plain text. That could make it trivial for an attacker to use malware to extract the database and its contents.

“Every few seconds, screenshots are taken. These are automatically OCR’d by Azure AI, running on your device, and written into an SQLite database in the user’s folder,” explains Beaumont in a detailed blog post. “This database file has a record of everything you’ve ever viewed on your PC in plain text.”

Beaumont shared an example of the plain text database on X, scolding Microsoft for telling media outlets that a hacker cannot exfiltrate Recall activity remotely. The database is stored locally on a PC, but it’s accessible from the AppData folder if you’re an admin on a PC. Two Microsoft engineers demonstrated this at Build recently, and Beaumont claims the database is accessible even if you’re not an admin.

The fear is that Recall makes it easier for malware and attackers to steal information. InfoStealer trojans already exist to steal credentials and information from PCs, and hackers currently distribute this type of malware to steal and sell information. “Recall enables threat actors to automate scraping everything you’ve ever looked at within seconds,” says Beaumont.

Beaumont has exfiltrated his own Recall database and created a website where you can upload a database and instantly search it. “I am deliberately holding back technical details until Microsoft ship the feature as I want to give them time to do something,” he says.

Microsoft is currently planning to enable Recall by default on Copilot Plus PCs. In my own testing on a prerelease version of Recall, the feature is enabled by default when you set up a new Copilot Plus PC, and there is no option to disable it during the setup process unless you tick an option that then opens the Settings panel. Microsoft is reportedly discussing whether to change this setup process, though.

Reaction to Microsoft’s Recall announcement has been swift, with privacy campaigners calling it a potential “privacy nightmare” and the UK’s Information Commissioner’s Office stepping in to make inquiries with Microsoft over its use of the AI-powered feature.

Microsoft maintains Recall is an optional experience and that it has built privacy controls into the feature. You can disable certain URLs and apps, and Recall won’t store any material that’s protected with digital rights management tools. “Recall also does not take snapshots of certain kinds of content, including InPrivate web browsing sessions in Microsoft Edge, Firefox, Opera, Google Chrome, or other Chromium-based browsers,” says Microsoft on its explainer FAQ page.

However, Recall doesn’t perform content moderation, so it won’t hide information like passwords or financial account numbers in its screenshots. “That data may be in snapshots that are stored on your device, especially when sites do not follow standard internet protocols like cloaking password entry,” warns Microsoft.
Microsoft’s FAQ page doesn’t address the potential for malware to try and steal the Recall database, though. “Recall snapshots are kept on Copilot Plus PCs themselves, on the local hard disk, and are protected using data encryption on your device and (if you have Windows 11 Pro or an enterprise Windows 11 SKU) BitLocker,” says Microsoft.

As Beaumont points out, disk encryption is only good for certain scenarios. “When you’re logged into a PC and run software, things are decrypted for you,” explains Beaumont. “Encryption at rest only helps if somebody comes to your house and physically steals your laptop — that isn’t what criminal hackers do.”

Microsoft may well find itself needing to rework Recall, or recall it, if you like. There are clearly some obvious holes in the way data is stored here that need to be addressed, and making this an opt-out experience has privacy campaigners concerned. Recall’s launch comes just weeks after Microsoft CEO Satya Nadella called on employees to make security Microsoft’s “top priority,” even if that means prioritizing it over new features.

“If you’re faced with the tradeoff between security and another priority, your answer is clear: Do security,” said Nadella (emphasis his) in an internal memo obtained by The Verge. “In some cases, this will mean prioritizing security above other things we do, such as releasing new features or providing ongoing support for legacy systems.”

The Verge reached out to Microsoft to comment on the security and privacy concerns with Recall, but the company did not reply in time for publication.

Quoted from:
https://www.theverge.com/2024/6/3/24170305/microsoft-windows-recall-ai-screenshots-security-privacy-issues