Contact
by Anaïs Dupont
AI

How to run a privacy-preserving llm on a raspberry pi 5 for offline note-taking

Mar 17, 2026 • by Anaïs Dupont
How to run a privacy-preserving llm on a raspberry pi 5 for offline note-taking

I wanted a private, offline note-taking assistant that I could carry around on a cheap, low-power device. The Raspberry Pi 5—when paired with the right model and software—lets you do exactly that: run a local language model that summarizes, tags, and searches your notes without ever sending data to the cloud. I built one, iterated a few times, and in this article I’ll walk you through the practical choices, performance trade-offs, and step-by-step setup so you can do the same.

Why run an LLM on a Raspberry Pi 5?

There are three reasons I care about this setup:

  • Privacy: All text lives on your hardware. No third-party inference, no telemetry to worry about.
  • Offline reliability: The assistant works without internet—great for travel, secure environments, or when you want guaranteed latency.
  • Low cost + low power: A Pi 5 with an SSD uses far less energy than a laptop or cloud VM and is inexpensive to duplicate for different rooms or desks.

    • What you’ll need (hardware and software)

    Here are the components I recommend. I tested with the 8GB Pi 5; 4GB is possible but you’ll be very constrained, and 16GB is ideal if you want larger models.

    Component Recommendation Why
    Raspberry Pi 5 8GB or 16GB Enough RAM for 7B-class quantized models and decent caching
    Storage NVMe SSD via USB4/PCIe adapter Models are large; SSD makes startup and swapping acceptable
    Power Official USB-C PSU Stable power for consistent performance
    OS Ubuntu 24.04 64-bit or Raspberry Pi OS 64-bit Better toolchain and 64-bit address space for models

    Choosing a model: size, license, and quantization

    On-device LLMs rely on two levers: model size and quantization. The sweet spot for the Pi 5 is typically 6–7B models in a 4-bit quantized format (ggml/q4_*). These strike a good balance between capability and memory footprint.

  • Models to consider: Llama 2 7B (if license permits your use), Mistral-small (open-weight variants), OpenLLaMA-family, and community quantized models like Alpaca or Vicuna ports. Always check the license for commercial or derivative use.
  • Quantization: Tools like llama.cpp and its ggml/gguf formats let you run models in q4_0/q4_1 or even q8_0. q4 models typically load in ~4–6GB RAM for 7B weights; q8 uses more but can be faster in some cases.

    • Software stack I used

    My stack is intentionally minimal and well-supported:

  • llama.cpp — small, portable C++ runtime with ARM support and ggml/gguf formats
  • llama-cpp-python — Python bindings that make integration with note apps simple
  • Obsidian or Joplin for notes — local-first note storage, optionally encrypted
  • FastAPI + Uvicorn (optional) — to expose a local-only REST API for my note-taking front end

    • Step-by-step setup (high level)

    This is the condensed path I followed. Exact commands depend on the OS choice but this is representative for Ubuntu 24.04 ARM64.

  • Install OS and update:

    sudo apt update && sudo apt upgrade -y

  • Install build tools and dependencies:

    sudo apt install -y git build-essential cmake python3-pip python3-venv libssl-dev

  • Clone and build llama.cpp optimized for ARM:

    git clone https://github.com/ggerganov/llama.cpp.gitcd llama.cppmake clean && make -j4

    Note: the makefile auto-detects ARM and will build with NEON where possible.
  • Get a quantized model:

    Either download a community pre-quantized ggml/gguf model (search "ggml-7b-q4") or convert a PTH/ggml file using the conversion tools included in the repo. Keep the model on the SSD to avoid SD-card I/O limits.

  • Install Python bindings (optional but useful for integration):

    python3 -m venv venvsource venv/bin/activatepip install wheelpip install llama-cpp-python

    Then point the binding to the model path in your code.
  • Run a quick inference to validate:

    python -c "from llama_cpp import Llama; m=Llama(model_path='path/to/ggml-model-q4.bin'); print(m('Summarize: I like apples', max_tokens=64))"

    • Integrating with your note app

    I chose Obsidian for daily notes because it’s local-first and supports community plugins. For automation I run a small FastAPI service on the Pi that accepts a note, calls the LLM to summarize or tag it, and returns metadata to be saved alongside the note. This keeps the UI decoupled from the model code and makes it easy to swap models later.

  • Example workflow:
  • User writes a note in Obsidian and triggers "Summarize" (via a custom plugin or template).
  • Obsidian POSTs note text to Pi’s local API (http://localhost:8000/summarize).
  • The FastAPI service uses llama-cpp-python to get a summary and suggested tags, returns them to Obsidian to append to the note.

    • Performance tips and gotchas

  • RAM swap and zram: Enable zram to avoid thrashing if the model pushes near RAM limits. Swapping to SD is slow; SSD swap is better.
  • Use SSE/NEON builds: Ensure the llama.cpp build uses ARM NEON optimizations—this dramatically improves throughput.
  • Reduce context when possible: Large context windows increase memory use. For note summarization, a rolling context or chunking works well.
  • Model warm-up: The first request often includes JIT/warm CPU paths; expect the first inference to be slower.

    • Security and privacy considerations

    Running everything locally solves many privacy risks, but there are still important steps to harden the box:

  • Disable unnecessary network services and block inbound traffic via ufw/firewall unless you explicitly need local network access.
  • Encrypt the SSD (LUKS) if the device may be physically stolen. My notes are personal—encryption is non-negotiable for me.
  • Keep OS and llama.cpp updated. Community builds fix vulnerabilities and performance bugs.

    • Limitations and trade-offs

    On-device LLMs on tiny hardware are a set of trade-offs:

  • Capability: You won’t get the reasoning power of a 70B or cloud-hosted model. For summarization, classification, and short-answer tasks, 7B quantized works very well.
  • Latency: Local is fast for small prompts; larger generations and chains of thought will be slower than cloud GPUs.
  • Maintenance: You manage updates and model licenses yourself.

    • Running a privacy-preserving LLM on a Raspberry Pi 5 for offline note-taking is practical today. It’s not for every workflow—if you need state-of-the-art creative writing or massive context windows, cloud models still win—but for private, reliable note summarization and tagging, a Pi-based assistant works surprisingly well. I’ve used mine for weeks, iterated on quantization and caching, and it now sits in my home office making notes searchable and useful without touching the cloud.

    You should also check the following news:

    Can your team replace pagerduty with cheaper alternatives? a cost, reliability and security playbook
    Software

    Can your team replace pagerduty with cheaper alternatives? a cost, reliability and security playbook

    I’ve helped teams evaluate incident management tooling from both an engineering and product...

    Mar 15 Read more... →