What is semantic image search? And why it can find a photo nobody labelled

Type "beach" into an old photo app and it finds files named beach, or ones you tagged beach by hand. Type the same word into a newer one and it finds a photo of sand and water that has never been tagged, named or described by anyone. That second behaviour is semantic image search, and the shift it represents is worth understanding properly, because it changes what you can reasonably expect a search box to do.

What is the actual difference from keyword search?

Keyword search matches text against text. It looks at what you typed and compares it, exactly or with light fuzzing, against a filename, a tag, a caption, or words that appear inside the image itself. If nobody ever attached the word "beach" to a photo, in a filename or a tag or text printed in the shot, a keyword search has nothing to match, no matter how obviously the picture shows a beach.

Semantic search matches meaning. It does not care whether the word "beach" appears anywhere near the file. It has learned, from being shown a very large number of images paired with their captions, what a beach generally looks like, and it can recognise that quality in a picture even when no one wrote the word down. The practical result is a search box that answers a description rather than a label: "quiet room, morning light" can return a photo that fits that description perfectly and was never captioned at all.

How does a computer compare a sentence with a picture?

The mechanism is not magic, and it is worth explaining plainly, because the explanation is what makes the result trustworthy rather than mysterious. A model is trained on a huge number of image and caption pairs until it learns to convert both a picture and a sentence into the same kind of thing: a list of numbers, typically a few hundred of them, called an embedding. The training process arranges those numbers so that inputs with similar meaning end up with similar lists, regardless of whether the input was a picture or a sentence.

A useful way to picture this is a map. Imagine every photo and every possible search phrase plotted as a point in space. The training process is what decides where each point goes, and it is trained to place "a photo of a beach at sunset" and the sentence "orange sky over sand and sea" close together on that map, even though one arrived as pixels and the other as words. Searching is then just this: turn your typed phrase into a point on the same map, and hand back whichever photos sit nearest to it. Nothing is read or interpreted at search time. The position was decided when the model was trained, and the search step is pure distance measurement, which is why it can run instantly even over a large library.

The model most commonly credited with proving this works well at scale is OpenAI's CLIP, short for Contrastive Language-Image Pre-training, trained on 400 million image and caption pairs collected from the internet. The original CLIP paper describes training the model by having it predict which caption actually belongs with which image out of a large batch of options, over and over, until the two halves, image and text, land in the same shared space. That same approach, or a close variant of it, sits behind most of the semantic image search you will meet in an app today.

Is find similar images the same thing?

Yes, pointed at a different kind of input. "Find similar" hands the model a picture instead of a sentence, and asks the same question: which other points on the map sit nearest to this one? Because photos and phrases share the same space once they are converted into embeddings, the exact machinery that answers "quiet room, morning light" also answers "more like this one," just starting from a picture instead of typed words. It is the same nearness measurement, run from a different starting point.

Where does semantic search genuinely fall short?

This is the part worth being honest about, because a search feature that only ever gets praised is one you should not trust. Semantic search has real, structural weaknesses, not just rough edges.

It is vague about specifics. The model has learned general visual concepts, not facts. It knows what a mug generally looks like far better than it knows that the mug in your third photo is the same mug as the one in your ninth. Ask it for "the blue mug" and it can genuinely help; ask it to find one particular person's face among thousands of strangers, or one exact product model number, and it will typically fail, because that is closer to precise identification than to general meaning matching.

It inherits the biases of what it was trained on. A model trained on 400 million images and captions scraped from the internet reflects that internet: which subjects were photographed often, which words were used to describe them, and whose. That shows up as search results that skew toward whatever was common in the training data, quietly and without any warning label attached to a given query.

And it cannot read your intent, only your words. If you search "dog" while thinking specifically of your own dog, the model has no way to know that; it will return anything that generally matches the visual concept of a dog. It answers the query you typed, not the memory you had in mind when you typed it.

That is exactly why keyword and text search still win for exact things. A file name, a serial number, an exact phrase printed on a receipt: none of that benefits from a model estimating meaning, and a fuzzy semantic match can actively get in the way when what you actually want is a precise text hit. A search tool that only does one or the other is picking a fight it does not need to have. A tool that fuses both, keyword for the exact and semantic for the remembered, gives you the right one automatically depending on what you typed.

Does this run on your Mac, or somewhere else?

This distinction matters more than it first appears to. Some apps run the embedding model in the cloud: your search phrase, and often your images too, travel to a server somewhere to be converted into those number lists and compared. Other apps run the same model locally, on your own device, so the comparison happens without your photos or your search terms ever leaving the machine they are stored on.

For a personal photo or reference library, that is not a minor implementation detail. A library of your own photos is often a genuinely private thing, and whether searching it means uploading it to search it is a real difference, not a technicality.

Muse's built-in AI, including its meaning-based search, runs on-device: once the optional AI pack is installed, describing an image to find it happens locally, without your photos being uploaded to search them. Two separate features in Muse are opt-in and do use the network: Muse Vision, which searches for visually similar images out on the web rather than in your own library, and the optional Claude features, which use your own Anthropic API key if you choose to add one. Both are switched off unless you turn them on, and neither is what powers everyday on-device search of your own library.

So when is semantic search actually the right tool?

Reach for it when you remember what something looked like but not what it was called: a mood, a colour, a composition, a general subject. That is exactly the gap keyword search cannot close, because there was never a word attached to close it with. Reach for plain keyword or text search instead when you know precisely what you are after: an exact filename, a specific phrase, one particular person or product. The honest answer is not that one approach replaces the other. It is that they solve different problems, and a search box that only offers one of them is always going to disappoint you on the other half of your questions. That same split is why Spotlight and a dedicated visual library end up doing different jobs on a Mac, rather than one making the other redundant.