Building an AI brain of your own

Every new chat starts from zero. The model does not know how you write, what you decided last month, or that you have corrected the same mistake four times. People feel this and conclude the tools are overrated. The tools are fine. What is missing is a memory, and you can build one in an afternoon with plain text files. This is the system I run daily, reduced to its load-bearing parts.

A working personal brain is a loop with four stages. Capture what mattered from the work. Distill it into something short enough to be useful. Index it so it can be found. Recall it into the model's context at the moment it is needed. Most setups fail by skipping distillation, hoarding transcripts, and then wondering why nothing useful comes back.

The cheapest memory is a file of instructions loaded into every session. Claude Code reads CLAUDE.md, Cursor reads rules files, and most serious tools now respect AGENTS.md as a vendor-neutral convention. Whatever the file is called, the content discipline is the same: who you are, how you like to work, the standing facts of your environment, and the corrections you are tired of repeating.

---
name: prefers-pr-summaries-short
description: Matthew wants PR descriptions under 10 lines, outcomes first
type: feedback
---

PR descriptions: lead with the outcome in one sentence,
then bullet the changes. No restating the diff.
Why: long PR prose hides the decision that matters.

Instructions cover the permanent. The second layer holds what accumulates: decisions and their reasons, project state, preferences discovered along the way. The format that has survived contact with real work is unglamorous: a directory of small markdown files with structured front matter, plus one index file listing each entry on a single line. The index is what loads into context; the files load only when relevant. Cheap to store, cheap to recall.

Tooling can take over from there. Claude Code maintains a persistent memory directory natively. Frameworks like mem0 and Letta manage extraction and retrieval as a service, and ChatGPT's built-in memory does a lighter version of the same thing. The principle does not change with the vendor: distilled facts, stored outside the model, loaded just-in-time.

A memory you cannot find is a memory you do not have.

Recall works when memory is searchable in two ways at once: plain keyword search for the things you can name, and semantic search for the things you can only describe. Local embedding databases make the semantic half trivial now, and for the keyword half nothing has dethroned plain text and a fast grep. The test of the whole system is one question: when I ask about a decision from six weeks ago, does the right paragraph land in context, or does the whole archive?

A brain that cannot see your calendar, your repositories, or your documents still makes you the courier between systems. The Model Context Protocol has become the standard plug here: one open protocol that lets any assistant query the systems you already use, with the same server working across Claude, Cursor and a growing list of others.

claude mcp add github -- npx -y @modelcontextprotocol/server-github

The pieces, all real and available today

Claude CodeAnthropicAgentic coding with CLAUDE.md instructions and a persistent file-based memory directory built in.Model Context ProtocolOpen standardThe open protocol for connecting assistants to tools and data. Write a server once, use it from any client.ObsidianLocal-first notesPlain markdown files with linking. A natural home for the distilled layer, and AI tools read it directly.MEmem0Open sourceA memory layer that extracts, stores and retrieves facts from conversations automatically, with APIs for any stack.LELettaFrom the MemGPT researchAgents with self-editing long-term memory, built on the MemGPT pattern of paging context in and out.CHChromaOpen sourceAn embedding database that runs locally, for the semantic half of recall across your own notes.

• Corrections you have made more than once. The second time is a signal; write it down with the why.
• Decisions and their reasoning, dated. The decision without the why ages into noise.
• Preferences about taste: tone, formatting, structure. These compound fastest because they touch everything.
• Stable facts about your environment: paths, systems, names, constraints.
• Kept out: secrets and credentials, raw transcripts, anything the source system already records better, and anything you would not want re-surfaced verbatim later.

None of this is exotic. Text files, a search index, a protocol. The compounding comes from discipline: every session that ends with a distilled note makes the next session start further ahead. After a few months the assistant stops feeling like a tool you operate and starts feeling like a colleague who was in the room last quarter. That is the entire trick, and it is available to anyone willing to file things properly.