On-Device AI: What It Actually Means for Your Files

Ask an app whether its AI runs "on-device" and most will say yes, because the phrase sells well. The honest answer is usually mixed: some of what an app does runs locally, and some of it quietly reaches a server the moment you have a connection. Knowing the real distinction, and what you give up by choosing one over the other, is more useful than trusting the badge.

This matters most for anything holding your own material: photos, screenshots, notes, saved links. Send that to the wrong kind of service and it might sit on someone else's disk for as long as they choose to keep it. Keep it local and there is no copy anywhere else to worry about. Neither option is automatically the right one, and the rest of this page sets out why.

What does on-device AI actually mean?

Cloud AI processes your request somewhere else. A photo, a question or a document travels over the internet to a company's server, a large model running on serious hardware does the work, and the result comes back to your screen. Every well-known chatbot works this way by default, ChatGPT and Gemini included: the model itself is never on your machine, so your data has to make the trip before anything can happen.

On-device AI runs the model itself on your own machine. The calculations that would otherwise happen on a server happen on your Mac's own chip instead, using a version of the model small enough to fit there. The request does not need to reach a server, because a server was never part of the process. That is the whole distinction: where the model lives, and so where your data has to go to get an answer.

How did ordinary Macs get powerful enough for this?

Three changes moved AI from "only possible on a server" to "possible on the machine in front of you" within a few years. The first is dedicated hardware. Every Apple Silicon Mac has a Neural Engine, a chip built specifically for this kind of maths. Apple's own machine learning research describes it as "the energy-efficient and high-throughput engine for ML inference on Apple silicon", first shipped in the iPhone in 2017 and now standard across the Mac line too, with roughly twenty-six times the processing power of that first version.

The second is unified memory. On an Apple Silicon Mac, the CPU, the GPU and the Neural Engine all draw from one shared pool of memory instead of separate ones, so a model does not need to be copied between pools before anything can use it. The third is that models themselves have shrunk. Techniques like quantisation compress a model's size while keeping most of its accuracy, so something useful can now fit in a few gigabytes rather than needing a data centre behind it. Dedicated hardware, shared memory and smaller models: none of the three would be enough alone, but together they are why an ordinary laptop can now do work that needed a server room a decade ago.

What do you actually give up by running AI on your own Mac?

On-device is not simply better. A model built to fit on a Mac is a fraction of the size of the ones running in a data centre, and size still tracks with capability: a small local model makes more mistakes on unusual questions, handles long or messy input less reliably, and is more likely to misread a cluttered photo than a frontier cloud model would. A recent academic survey of on-device AI is direct about the shape of the trade: local models operate under real resource constraints, in exchange for speed and privacy a remote server does not offer.

Running a model locally also costs something a cloud request does not. It uses a share of your Mac's battery and memory while it works, and a genuinely heavy job, tagging thousands of images in one pass, say, will usually take longer on a laptop chip than on a server built for exactly that load. And a local model is frozen at whatever version shipped with the app. A cloud provider can swap its model for a better one overnight without you doing anything; your on-device model only improves when you update the app yourself.

What does on-device AI genuinely buy you, then?

It works with no connection at all. Turn off Wi-Fi and a genuinely local feature keeps working exactly as before, because there was never a server in the loop to lose. It also has no per-request cost: running a local model does not meter you the way an API-billed cloud tool does, so using it a thousand times costs the same as using it once. Nothing about what you process sits on a server waiting to be retained, misused or breached, because it was never sent anywhere to begin with. And it keeps working if the company behind it changes its pricing, gets acquired or shuts its service down tomorrow, because the model was already installed on your machine rather than rented from something that can disappear.

How can you tell what an app is actually doing?

Marketing copy rarely settles this on its own, but three checks usually do.

1. Turn off Wi-Fi and try the feature. If it still works, the processing genuinely happens on your machine. If it stalls or errors, it was reaching a server the whole time.
2. Look for a plain statement of where processing happens. A privacy policy or settings screen that names the mechanism, not just the word "AI", is a good sign the vendor is being straight with you.
3. Check whether the feature needs an account. A feature that only works once you have signed in is very rarely running locally. There is usually a server on the other end of that login.

Where does Muse fit into this?

Muse is a Mac app I build, so it is worth being precise here rather than reassuring. The AI Muse ships with, the tagging it applies automatically, the text it reads out of a screenshot, the colours it pulls from a picture, and the optional Pro AI pack's meaning-based search, all run on-device, using models installed with the app itself. That is only possible because Muse requires an Apple Silicon Mac, the exact hardware this piece has been describing.

Two features are the honest exceptions, and both are opt-in. Muse Vision is a reverse image search: point it at one picture and it sends that single image to a web visual-search service to find similar ones. The optional Claude features send data to Anthropic through an API key you supply yourself, and only run if you turn them on. Neither happens in the background, and Muse works fully without either.

The pattern generalises past Muse. "On-device" is not a single fact about an app, it is a fact about one feature at a time, and the only way to know which category a given feature falls into is to check, not to read the tagline.