You already have a free, private LLM, and you don't know it

It started with a tiny need. On a side project (a price tracker that scrapes a few shops), I had product titles to filter: “does this listing really match a 12V water pump, or is it an accessory?”. A keyword filter handled it poorly. The modern reflex is to call a cloud LLM.

And there, a slight unease. For a minor feature, I was about to: send data to a third party, create an API key, pay per token, and depend on a remote service, all for a task any small model could handle. Except my Mac already ships an AI model. Why not use it?

That question, trivial on a side project, is exactly the one we ask in a company the moment AI comes up: where does the data go, to whom, and what stays under our control?

Privacy, concretely

Let’s be precise about words. One might be tempted to say “sovereignty”, but that would be a stretch: it’s an American model (Apple), on an American chip. So much for technological independence. 😄

What we do gain, however, is very real and boils down to one thing: the data doesn’t move.

• No data leaves the machine. The processed text is never sent to a third party. No GDPR question about a sub-processor, no possible leak, no clause to audit.
• No key, no quota, no marginal cost. Call the model as much as you want, including in a loop.
• It works offline. On a train, on a plane, behind a dead network.
• No dependency on a remote service that can change its pricing, its terms, or vanish.

It’s obviously not the answer for every use case (we’ll come back to that, honestly). But for a broad class of tasks (summarize, classify, extract, rephrase), a small model running on your own machine is not just sufficient: for sensitive data, it’s preferable.

Apple’s model, and its lock

Since macOS 26, Apple exposes its built-in model through the FoundationModels framework: a model of roughly 3 billion parameters, optimized to run on the chip, with no network. It’s what powers part of Apple Intelligence.

The catch: this model is accessible only through a Swift / Objective-C API. No Python or JavaScript binding, no native HTTP endpoint. In other words, to call it you need a Swift process linked against the framework. This constraint shapes the whole architecture.

The architecture: an HTTP bridge

The idea fits in one sentence: write the bare minimum in Swift to talk to the model, and expose it over HTTP in the OpenAI API format. From there, everything else in the ecosystem (any chat UI, the openai SDK in Python, a plain fetch in JavaScript) already knows how to talk to it, without a single extra line of Swift.

flowchart LR
    subgraph mac["Your Mac: nothing leaves the machine"]
        direction LR
        UI["UI / app / script"] -->|"OpenAI-compatible HTTP"| Shim["Swift bridge"]
        Shim -->|"Swift FoundationModels API"| Model["On-device model (~3B)"]
        Model -.->|"response"| Shim
        Shim -.-> UI
    end

This little bridge (a shim) turns the model into a drop-in: the same codebase that called OpenAI keeps working, just by changing the URL.

Reproducible installation

Prerequisites

• An Apple Silicon Mac (M1 or newer).
• macOS 26 or later.
• Apple Intelligence enabled: Settings → Apple Intelligence & Siri → enable (the model then downloads, a few GB).
• The Command Line Tools (xcode-select --install) to have swiftc.

To check the model is ready, this tiny Swift script is enough:

import FoundationModels

if case .available = SystemLanguageModel.default.availability {
    print("Model available ✅")
} else {
    print("Unavailable: is Apple Intelligence enabled?")
}

swift check.swift

The Swift bridge

Here is a minimal but working version of the bridge: an HTTP server that implements POST /v1/chat/completions and delegates to the model. It has no dependencies (the system’s Network framework).

import Foundation
import Network
import FoundationModels

let port = NWEndpoint.Port(rawValue: 11435)!

// Calls the on-device model and returns the generated text.
func generate(_ prompt: String) async -> String {
    let session = LanguageModelSession()
    let opts = GenerationOptions(temperature: 0)
    let response = try? await session.respond(to: prompt, options: opts)
    return response?.content ?? ""
}

// Extracts the prompt from an OpenAI body {messages:[{role,content}]}.
func prompt(from body: Data) -> String {
    let obj = try? JSONSerialization.jsonObject(with: body) as? [String: Any]
    let messages = obj?["messages"] as? [[String: Any]] ?? []
    return messages.compactMap { $0["content"] as? String }.joined(separator: "\n\n")
}

final class Conn {
    let c: NWConnection
    var buf = Data()
    init(_ c: NWConnection) { self.c = c }
    func start() { c.start(queue: .global()); read() }

    func read() {
        // STRONG capture of self: otherwise the connection is freed and never replies.
        c.receive(minimumIncompleteLength: 1, maximumLength: 65536) { data, _, done, err in
            if let data { self.buf.append(data) }
            if let r = self.buf.range(of: Data("\r\n\r\n".utf8)) {
                let body = self.buf.subdata(in: r.upperBound..<self.buf.endIndex)
                Task { await self.reply(body) }
                return
            }
            if err != nil || done { self.c.cancel(); return }
            self.read()
        }
    }

    func reply(_ body: Data) async {
        let text = await generate(prompt(from: body))
        let json = try! JSONSerialization.data(withJSONObject: [
            "choices": [["message": ["role": "assistant", "content": text]]]
        ])
        var out = Data("HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: \(json.count)\r\nConnection: close\r\n\r\n".utf8)
        out.append(json)
        c.send(content: out, completion: .contentProcessed { _ in self.c.cancel() })
    }
}

let listener = try NWListener(using: .tcp, on: port)
listener.newConnectionHandler = { Conn($0).start() }
listener.start(queue: .global())
print("Local LLM → http://127.0.0.1:11435/v1")
dispatchMain()

A subtlety that cost me a while: if the connection isn’t held by a strong reference, it’s freed immediately and the client waits for a response that never comes. The compiler flags it with a “weak reference will always be nil” warning, so don’t ignore it.

Compile a real binary (interpreted mode recompiles on every launch):

swiftc -O fm-shim.swift -o ~/fm-shim/fm-shim

Always available: a launchd service

So the bridge starts at login and restarts on its own if it dies, we make it a LaunchAgent. File ~/Library/LaunchAgents/com.gluendo.fm-shim.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>            <string>com.gluendo.fm-shim</string>
    <key>ProgramArguments</key> <array><string>/Users/YOU/fm-shim/fm-shim</string></array>
    <key>RunAtLoad</key>        <true/>
    <key>KeepAlive</key>        <true/>
    <key>StandardErrorPath</key><string>/Users/YOU/fm-shim/fm-shim.log</string>
</dict>
</plist>

# load the service (modern launchd API)
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.gluendo.fm-shim.plist
# restart it / stop it
launchctl kickstart -k gui/$(id -u)/com.gluendo.fm-shim
launchctl bootout      gui/$(id -u)/com.gluendo.fm-shim

Important detail: a LaunchAgent (not a LaunchDaemon) runs in the user’s graphical session, which is required to access Apple Intelligence, unavailable to system processes.

A chat UI

Since the bridge speaks the OpenAI API, any compatible client plugs into it. Jan, an open-source native chat app, does the job nicely:

brew install --cask jan

Then in Jan → Settings → Model Providers → Add Provider (OpenAI-compatible type):

• Base URL: http://127.0.0.1:11435/v1
• API Key: local (anything works, the bridge ignores it)

And there you have a full, private chat interface wired to the machine’s model.

Use it from anywhere

That’s the whole point of the OpenAI format: the same endpoint serves your scripts and your applications alike.

curl http://127.0.0.1:11435/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Summarize this text in one sentence: ..."}]}'

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:11435/v1", api_key="local")
resp = client.chat.completions.create(
    model="apple-foundation",
    messages=[{"role": "user", "content": "Classify this ticket: bug, feature or question?"}],
)
print(resp.choices[0].message.content)

In a Node app, it’s the same endpoint via a plain fetch. A codebase that talked to OpenAI switches to local by changing one URL.

What this model can (and can’t) do

Let’s be honest: a 3-billion-parameter model is not GPT-4. Getting this clear before productionizing a use case avoids disappointment. Here’s how a small test battery went, in French:

Task	Verdict	Time
Summarizing a paragraph	✅ Faithful and clean	~1.7 s
Structured extraction (JSON)	✅ Correct values	~0.6 s
Classification (category)	✅ Correct	~0.2 s
Rephrasing with an ambiguous instruction	⚠️ Sometimes misread	~0.9 s
Lexical nuance (synonyms)	⚠️ Approximate	~1.5 s
Multi-step numerical reasoning	❌ Wrong	~3.5 s

The lesson is clear. This model is fast (often under a second) and reliable on the “mechanical” language tasks: summarize, classify, extract, tag, produce JSON. It is to be avoided for arithmetic, multi-step reasoning, or cutting-edge knowledge. It’s a paring knife, not a Swiss army knife, and a well-sharpened paring knife is enormously useful.

In conclusion

AI privacy isn’t a binary choice between “all cloud” and “all local”. It’s a slider. Many everyday tasks (summarizing an email, classifying a ticket, extracting fields from a client document) have no reason to leave the machine. Reserving them for a local model wins on privacy, cost and autonomy, while keeping the cloud for the tasks that truly demand it.

No, this isn’t sovereignty, the model and the chip still belong to an American giant. But it’s a concrete win on the one point that’s negotiated daily: your data stays with you. And the most striking part is the effort required: a few dozen lines of Swift, a service file, and the model already present in our machines becomes a reusable infrastructure building block. Privacy, sometimes, is mostly a matter of bothering to plug in what we already have at hand.