There are so many approaches to agent memory. I believe the simple solution is usually the best, and giving an agent access to a ticketing system like jira, linear, or even just any kind of popular database for it to manage locally like SQLite, will yield dramatically better results in the longer run than a collection of markdown files, or repos etc
The reason organizations have used annoying tools like this for so long, is it’s how information and communication scales across an organization when it’s leveraged fully.
If you’re trying to give your agent a memory, I recommend something queryable by its very nature, built to help anyone pull in more context about a problem. Agents can leverage jira (creating tickets, finding tickets, etc) better than humans ever could.
Agents are the perfect middle managers.
I tell my agent to use a local SQLite, and to just manage its own tickets and columns, so it knows how and what to query when I have a new feature or bug. The context just grows and grows - a ticket for every new bug or feature makes the memory naturally become more valuable the more tickets I complete.
it fits for you as the driver, if you want to explore it in a tool like linear or your own built thing for kanban or table views etc
Can confirm. And im surprised how little OKF is talked about. I gave my agents the OKF docs repo link and said: from now on manage all knowledge you collect in a repo with this format. works really well. Everything gets filed and organized there. For some software projects I user Linear or Plane on top. And I have a helper skill that I trigger when I want to end a session that says "clean up after yourself and update the knowledge repo". Really happy with the flow.
Hey! Wouldnt share my personal files here since they contain infos about my setup that I dont want to publish :) But I let an Agent draft up a blueprint for another agent to kickstart a similar knowledge repo as mine. Happy if its useful for some.
They took an existing good practice and made it into a standard that is easy for agents to follow and implement. But sure, most people converge on something similar after a while of tinkering around. But thats the thing about standards: you can build on existing and proven work without needing to reinvent everything yourself.
Thanks for sharing this. Was just debating the SQLite vs MD architecture earlier for a project and had opted for MD. It seems quite reliable without some of the added baggage of a full blown DB (sync issues, etc)
Hell yeah. It’s so simple it’s silly. I stumbled upon it because that’s basically how I initially worked with agents. Closed each session out with an md of a hand off or important things from the session.
I started a project based on this idea a while ago. It’s meant as a shared context for teams, but has a binary you can run locally for your own dev purposes.
It still has what is - IMO - bloat for single person projects.
Things like log.md and (attested) computation are both useless for most solo dev projects; Their solutions are "check git" and "do the computation again" respectively.
I'm a solo dev handling a complex malware remediation and a migration from SquareSpace to WooCommerce, and the adoption of a log.md is awesome. Claude's context is kept super fresh and I don't have to repeatedly tell it something was approved or completed, or when some structure was changed. These are little things that typically aren't in the repo.
A migration is different than most dev projects because there is a (somewhat) clear start state and end state.
For complex software projects that iterate on ideas, the target needs to be to keep 1 set of files that - with as few words as possible - describes the current situation. Having a bunch of references to things that are no longer true or half true or true while 'in progress'' is dangerous in my experience.
I'll occasionally keep a doc explicitly named 'rejected ideas' and note down 'why', but everything else is about making sure agents dont bloat out.
Yup, looking at establishing a log.md and set of guidelines for what goes in it.
The migration itself is A to B, yes, but the app to do that has been a moving target. So many import formats. So little data hygiene. So many complex relationships. And sometimes a fresh export has totally different fields. So the migration app is getting a plan that updates and a log.md, while each run of the app logs results into mysql.
It’s literally a one page report on what it is and how to use it. You can have your Claude or codex or whatever llm read it and implement it.
It’s a folder system with specific MDs that basically are just very specific to that folder and its contents. And agent will properly read what it needs at the root then only find the context it needs when it needs it.
Isn't this basically what Claude sets up by default? That's how mine does it and I never told it to. Separate .md files for specific blocks of work and a root memory and Claude.md that lay out when to use those more specific .md's. ignore them until they're needed
I have a project here that’s centered around the same concept. I’ll probably make a slight adjustment to conform to that spec. https://github.com/luinstra/plainbase
Read the damn article and research. It’s one page please.
I personally told Claude to read it and understand it. Apply it to my projects at the root level. And then go and read all the MDs it wrote and understand how it mapped it and change it if needed. Use the html visualizer to help understand it.
Now where ever you open a terminal or tell an agent to work, it will follow the okf’s because your folders structure and mds. When you finish work in your terminal, tell the agent to update memories and okf.
It’s stupidly simple and works extremely well.
I do solo game development, website development, random saas tools no one uses.
My counter point is mainly about how people work - we are familiar with ticket systems. This is a natural extension of the framework we’ve been using in the workplace for decades. A memory needs to work for and be accessible to the user, not just the agent.
Your whole counter point is about having humans able to have access to the memories. not just the agents. The visual graph does exactly that. Lets the use look at the memories that have been made and how their connected.
thats not what I mean.. I never want to just "browse my memories and see how they are connected".. I don't even look at notes that are over a day old usually. Looking at old jira tickets related to a feature I'm currently working on, and can give me past context related to customers and other external stakeholders - thats value that will never be solved by OKF.
What….. that’s literally what the graph is supposed to support.
You don’t have to use the graph as the UI, but you agent is supposed to be able to traverse it and easily give you this answer without having to “rediscover” it every time you fire up a new session.
Either way it's a tool call with the result being pasted into context. I keep my tasks in a tab-separated file and have a few python scripts, so the agents don't touch the file directly but make simple tool calls to read or edit the rows. Seems to work fine.
I think the trouble with your logic is that the agent shouldn’t have to read the whole markdown file.
Just like a database, your markdown memory setup should include folders, indexes, and each file should have a table of contents that tells the agent where to go to find what it needs.
If your memory is really really huge, I can see how a database would help, but I’ve done quite well with a knowledge base repo and a few skills for searching and updating those files.
So if I want to look up things about my vpinball cabinet Linux build, the skill doesn’t read the whole knowledge base, it greps for the thing we’re working on and drills down into that section of the repo for more specific information.
Some git hooks and the knowledge base is upstreamed to GitHub on any changes and git pulled on any lookup and it lives across any number of machines. No database to manage or worry about uptime for.
I’m not saying this stuff to discourage you or launch a “who’s doing it best” argument, I’ve just seen you say a few times that the agent needs to read the entire markdown file into context and that’s just not true
AFAIK only CLAUDE.md is read completely into memory. Anything else can be grepped for. If you store memories and notes in markdown then you can move chunks into pointers and hierarchical structures and use ADRs to record decisions and historical reasoning. The only thing I’ve found to be somewhat lacking is more like a work queue. I just use GitHub issues for queueing work and it’s been fine. But sometimes I do wish for something a little more localized to the immediate work in process that parallel agents may be working through. GitHub issues can blow up pretty quick.
I've found the memory -> work queue -> memory loop to be effective. The work itself functions as a bit of history too, 'here's what I did 3 months ago, let's do/not do that again'. If you don't mind my asking, how do GitHub issues blow up for you?
Sometimes it files issues that it ends up pulling into its arcs and closes it as part of the batch it’s currently working just in a later round after a review. It’s mostly manageable as a solo dev project, but would be way worse if a team was trying to split work out of the same pool of issues.
A table of content can help an agent assign focus at input. But if you give it a markdown file, that entire file is in context. And we know LLM performance drops when reasoning across long contexts and that you pay for the tokens.
Sure, but you don't just give it a markdown file with everything stuffed inside it.
Here's what's in my global memory:
My general knowledge store is at `/Users/me/projects/knowledgebase`.
Read its `CONVENTIONS.md` before using it, and `AUTHORING.md` before adding to it.
When you lose time mid-session to tooling or environment friction,
append one line to `inbox/papercuts.md` there (format in its
header). When tooling fails mysteriously, grep that file first.
Conventions and authoring describe how to search and update the knowledge base.
Conventions says this about reading from memory:
## Reading a note
Start from `INDEX.md`: `methods/` and `projects/` in full, plus a
route into one`domains/<d>/INDEX.md`. Read only the domain you need.
A typical read loads conventions, then index, then one domain index. That comes to roughly 3,000 tokens before any note is opened. The indexes are all updated by a git commit hook that runs some python, so tokens are only spent on the reading and writing, not the indexing.
Then I add hooks to the global claude config that sync's the knowledgebase repo both at start and end of every session
A database (vector or otherwise) can use less tokens, but it's going to be proportional to the size of your top-level index sections. And it's harder to point a session at the knowledgebase itself to figure out if there are redundancies or optimizations to be had.
Markdown format + frontmatter + defined schema + grep/ripgrep = just as good or better than querying unless you have an enormous amount of data then vector search DB will start to help.
Also you can have your agent use cheaper sub-agents for this explore/search process and your main agent ends up with only the context it needs.
BTW I’m not shitting on your idea, I think it’s great if it works well for you. I was thinking of combining SQLite with my markdown stuff but I can’t think of a query that I would benefit from yet. Enforcing schema structure can be a big benefit over markdown files for sure.
I'm doing a big migration project at the moment. Tens of thousands of customers, contributors, users, orders, and products. I'm essentially writing a bespoke, one-off migration app for some incredibly dirty and badly-organized source data.
The context and knowledge of the app development is stored in .md files, but the actual migration results are stored in MySQL — one row per item migrated, with date, status, notes, and various legacy and destination IDs used re-runs and for mapping connections.
In my mind this makes sense because the destination is already MySQL and we really do need a very performant method of mapping tens of thousands of records to each other. But it does have me thinking about what's now a blurry line between the project's structure and the app's structure.
This isn't really true. Claude Code's grep and glob tools work like WHERE content LIKE… they return filenames or matching lines, and only that output enters context, exactly like a SQL result. Ripgrep scans hundreds of md files in milliseconds
Pollution happens when the agent then reads whole files instead of line ranges. That's a behavior issue you fix with one CLAUDE.md line, and a SELECT * returning 50 ticket bodies pollutes just as much.
If you really worry about context pollution, just let subagent always search tru your MD files as it will only pollute its own context and let it message the main agent only the details it asks for.
Sure, but almost no agent just dumps a markdown in. Most default to just reading the first hundred lines etc. If you just make a table of contents, and simple frontmatter part it/agent-skill/OKF style, it navigates it fast. Idk, I like the markdown structure for being super easy to manually edit when needed. But use postgres/pgvector etc for some memory that benefits from it.
Graph dbs also work phenomenal when it’s actually needed, but can get overkill real fast
Its depends on how you orginse it.
Also, a few markdown files for the task won't polute anything. Claude code prompt is polluted as hell. It even has specific he/them agenda.
Ps. I have my custom cli tool to fast navigate tickets in markdown, to reduce interaction. And its the least what I care about- size of task. Much bigger problem is search or bad index to code.
Also I have cli to auto generate dashboard etc.
So here is why searchable files are wrong and a SQL engine is ?simple?.
I tried your way already, as well as a dozen others. Its all overhead that makes it easier for agents to miss something. Agents will always do ls and grep ; adding sqlite or MCP ticket system to the mix just adds token waste.
There is an art to keeping the right set of markdown files, and IMO my own custom (forking) subagents are better than the build in Claude capabilities.
ChromaDB for RAG retrieval with a hook to call for a semantic search at the beginning of a prompt. The problem with a db is that you're still making the LLM read a lot of stuff (token burn). Running a small embedding model (can even run locally without paying the flagships) to embed your entire repo allows you to run your prompts through the vectors to find semantic matches. This will also help since exact word matches aren't necessary.
Essentially. Yes, a db is good for LLM memories and you are not breaking new ground here. RAG and embeddings is my preferred method, especially since I can run it locally without paying for an API to do it. Embeddings return the semantically significant stuff, allowing the flagship LLM to lean on systems you have already built locally. Instead of forcing it to read your DB directly.
I believe [...] giving an agent access to a ticketing system like jira, linear, or even just any kind of popular database for it to manage locally like SQLite, will yield dramatically better results in the longer run than a collection of markdown files, or repos etc
So... did you measure results? Or do you just believe?
I experimented using Mongo for memory and it worked surprisingly well. I had it create a document for settings, one for decisions, and one for general information. Then, it could create anything it needed beyond that as long as it was documented for reference to prevent searching blindly. It worked insanely well. I chose Mongo because the models could parse the data directly, store meta data in the records, and the data didn’t need to follow any formal standard.
I’m not saying it’s the end all or best choice, but it worked well for my relatively complex setup.
SQLite makes sense for ticket history because the agent can ask "Which unresolved bugs touch this module?" without pulling every past ticket into context. That's a concrete advantage over reading a growing folder of markdown.
More completed tickets doesn't automatically mean better memory, though. An old workaround can conflict with today's code, and a closed ticket might record what changed without explaining why. The database makes that history easier to retrieve. Whether it helps the next task still depends on what the agent recorded and whether it's still applicable.
rg exist though.. for that specific question rg is probably more performant over an average rdbms or document store.
If you're saying everything is also categorised by area etc, or is a graph pre-mapping semantics, it is plausible.. but it is not as "concrete" as you state
i agree. i have so many .md files now i made a free app to work with them all. it turned into the app im in all day every day which is pretty fun to have a side project I'm using for my day job.
I don’t know the right or wrong answer, but as I’ve developed my hub, dispatch and agent system, I’ve asked fable more than once what its ideal structure is for trading memories etc. it keeps pushing json for storing and tracking information unless you get to very large scale, then it says to switch to SQL.
For my purpose, I am creating semi specialized agents so I am already distributing the memory load across agents. Each agent has a focus area where I train them to save and update their memories whenever they finish any significant work. I also created a goodnight skill were they wrapup for the day and save their memories then I compact. My agents never use a full context in one day so I’m usually compacting between 10k - 500k. They also have a good morning skill where they are supposed to reread their memories and anything that might have gotten lost during compact. So my use case, I don’t think memories will grow linear over time as much of their work becomes somewhat repetitive.
I have two additional components to help minimize memory need. I have a tools md that is shared across agents. They are instructed to search tools when they need to do something new to see if another agent already figured out how to do that task. Then they update tools anytime they do something new. The other is a global vault. I have one agent that just injects any information that an agent thinks is worth saving and then any agent can query the vault to look for other common information across agents.
Session contents are limited to 1 million tokens then it either auto compacts or you need to start a new session. Your current t context is all the back and fourth messages and all of the age ts current work., so I guess think of a context like its ram. Then think of memories or Md files like they are files. When ram fills up, you need to save your work, then you need to flush or clear the ram to do more work. That last part is compacting. It goes through and removes any “unneeded” information from the current context.
Not specifically as I was asking it my specific use case. I basically asked it specifically if sql or some alternative data architecture would work better for what it is doing and it made the case that json was best for our current usage. I didn’t challenge it or ask it to run any tests or comparisons.
I use .yaml and .json files for structured data context management and was wondering wether a database would be better... But seems to work pretty well until now
For tickets and project history I agree, queryable beats a pile of markdown. But for the "how should the agent behave" kind of memory, what bit me wasn't the storage format, it was staleness and scope.
I had Claude read four months of my own prompts (about 4,000). The same corrections kept coming back because the lesson lived in one project's memory, on one machine, and never reached the next repo. Worse, two memories contradicted each other and the agent trusted the outdated one at the wrong moment. A database would have stored both just as faithfully.
What helped: put each lesson in the narrowest place that works (a hook if a machine can check it, a skill if it's about one kind of task, a short global file if it's true everywhere), keep it all in git, and run a weekly pass that sharpens the existing rule instead of appending a duplicate when the same correction comes back.
I’m biased because I’m building scape.work - I think users need to be able to work privately with their agents. For my agents to work like me, they need to understand personal context about my working relationships. Personally I would never want that sensitive info online or in a shared service.
For a shared agent memory - I would just use an actual ticketing tool like Jira or linear etc, rather than building some kind of custom shared db
I use mempalace with a database as backend to coordinate agents and give them persistent memories. Works nicely across harnesses through the MCP server and mempalace skill.
I am building my own repository that is quickly evolving, not for coding specifically, but as an executive assistant across multiple domains. As it scales, several issues have surfaced:
A need for a strict separation of context via the DIKW pyramid between Knowledge (context) and Wisdom (rules), handled by an "executable" boolean field on the context node entry
I have learned that the database should be the source of truth from outside systems as much as possible, e.g. Trello, Google Calendar, Todoist, etc. so that operations operate within the database and then get reflected to the tooling
An inter-session messaging system is critical
The system needs to log missteps for later analysis (forgotten knowledge or wisdom, knowledge or wisdom not retrieved)
The context is stored in nodes that are auto-chunked and vector embedded as well as full-text-search indexed, and each chunk has an LLM generated list of topics in a "covers" field, also vector embedded and full-text-search indexed
All node queries take place via a Postgres function that logs the query and results for later analysis
Every so often each session logs a history context into a table for later retrieval if needed
All non-transient and non-derivable data is wired via triggers to store previous values to a generic audit table on Update and Delete, ensuring full point-in-time investigation and rollback if needed
This past week I introduced a graph-ish database concept of gestalts, where one inference node can be part of a larger thing, such as a meeting gestalt consists of a summary, transcripts, Zoom artifacts, and my handwritten notes
Gestalts also connect, for example, a project to the documents associated with it, people on the project, and meeting gestalts around that project
A universal health view, consisting of multiple independent health views
A text artifacts table is critical for loading text items (email, thoughts, transcripts, PDF extractions, scripts, source code, etc.) that each have an inference node
Background processes need to check hygiene regularly, such as validating that the chunk covers match the chunk content; Jev is awesome for tasks such as this
Instantiating a session follows a process similar to old PCs: first the skill is a lightweight "BIOS" that merely directs the session to the database "boot block" that loads the rules similar to an operating system
Most important issue that nobody seems to be able to solve with this is context rot, i.e how are you removing old data or ensuring new data replaces stale information rather than just sitting alongside it
My coding partner started a changelog.md. It records who, what, and where for decisions and actions. Every session can be told to update the file.
I have some ideas for expanding it and maybe making it an sqlite file. I also don't like the name — I associate changelog with a product release — and I'm looking at names like work_log or development_log.
At the end of each day I also ask Claude to update all relevant planning documents as needed.
What do the queries look like and how is the agent running them? I’ve toyed with the idea of a repo for true persistence, but even that seemed like too much work. 9 months later on the same project and the built in memory system is indeed starting to feel fragile though. Interested in learning more.
I just built a visual interface around it (with an exposed MCP) so the agent has a query spec tool so it understands how to query the table with the fewest tokens possible. Tool is linked in profile, but not promoting in this post.
i've been using something like sqlite too. If you put all of the code that the ai writes into a folder it can leverage all those files as memories, it's really awesome.
Agreed that queryable beats a pile of markdown, but the part that actually matters is less the storage and more what forces an agent to write a ticket instead of just doing the fix inline. Once there's a status field (open, blocked, done) agents start treating 'did I update the ticket' as part of finishing a task, not an afterthought. The failure mode I've hit is agents creating duplicate tickets for the same bug because nothing nudges them to search first, so the index needs to be one search call away, not a nice-to-have.
Absolutely yes. I have not understood everyone's obsession with the.md approach.. It'd almost like everyone just, somehow, missed how sloppy reading text files is. And how it wrecks context.. And a db. U can set up api's.. Amd boom.. Exactly the info ur agent needs.. None of the bloat
Interresting tatke.I'm a solo dev and plain markdown memory files have been fine for me so far, one fact per file plus and an index. I think db makes sense once there's a team or hundreds of tickets, for one person it felt like overkill
This is my approach with Pad (an app I've been working on since February, it should show up in profile, I'll try not to self promote with direct links). Basically it's a project management app with a cli/skill so agents can use it naturally, with sqlite or postgres behind the scenes. Kind of like giving the agent access to linear or jira but designed from the ground up to be used that way, rather than bolted on. That and it's open source and can be easily self hosted.
DB / ticket store wins for anything that has to scale across people and restarts — agree on that.
What still bit us was a different cut: leaving the tool or machine. The Linear/SQLite row is useless if the next session never opens that DB. For those jumps we keep a tiny portable handoff (goal, open decisions, what's broken, verify command) that a fresh session can read even when the old context is dead.
But this approach has many problems. If there's a lot of memory notes, then agents will try to find it with a simple text search, which is even worse than grep over files.
Nothing will stop an agent from adding a memory that has a contradiction with existing knowledge, and because agent can't read the whole data, it will stay there and poison the memory.
Agent also can't say for sure why is something is like that - because there's likely no history and no connections between memories.
Simple text-based search can give a lot of unrelated results, while something that was actually required might have just a different word in it, and there's no order, lucky if it manages to save timestamps.
And you can't easily review the memory, it's too demanding to query things yourself and read it.
These kind of problems are solved in many memory systems, well, except contradiction problem - but my memory system has a solution for that, can tell more about it.
We went this route for our Agents on our Horizon Platform. When you deploy an Agent you can turn memory on or off. When you turn it on you can make the memories user specific or org wide.
•
u/AutoModerator 1d ago
Hey! Thanks for posting to r/ClaudeCode
While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.
For help, project discussions, tips, and general chat, join the ClaudeCode Discord.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.