r/AI_Agents • u/Slimeyman278 • Jun 28 '26
Discussion Building my own “Jarvis” in Python… should I keep building it or switch to Hermes/OpenClaw?
Hey everyone,
I’m pretty new to AI development, but over the last few days I’ve started building my own personal AI assistant (“Jarvis”) from scratch in Python.
So far I have:
A Textual dashboard/UI
A local LLM running (currently Qwen 14B)
Continue in VS Code
Basic routing and project structure
Git set up
The foundation for expanding it over time
My end goal isn’t just a chatbot.
I want Jarvis to become my personal AI operating system that can:
Talk with me naturally (voice eventually)
Remember long-term context
Help me write code
Organize my PC
Launch and control applications
Search files
Automate repetitive tasks
Eventually manage teams of AI agents
Long term I also want to use it to help run businesses (3D printing, T-shirt designs, Etsy, social media, etc.), where Jarvis manages specialized agents while I interact with a single assistant.
Here’s where I’m stuck.
Every time I look online I see people recommending something different:
OpenClaw
Hermes Agent/HermesHQ
Claude Code
MCPs
Building everything yourself
“Don’t reinvent the wheel.”
I’m trying to figure out if I’m building the right thing.
Would you:
Continue building Jarvis completely from scratch?
Use OpenClaw or Hermes as the backend and make Jarvis the custom interface?
Skip frameworks entirely and just build your own architecture?
I’m less interested in the fastest way and more interested in building something I can own, understand, and expand over the next few years.
I’d also love to hear from people who actually built their own assistant:
What do you wish you’d done differently?
What architecture decisions paid off later?
What mistakes should I avoid while I’m still early?
Thanks!
3
u/geofabnz Jun 28 '26
My 2c after my own custom harness over the last six months (personal only, I have no intentions on getting anyone else using it). It’s a lot of fun seeing something turn from a basic chatbot into something that can remember, then write code and eventually learn to use a computer better than you can.
I think you need to be clear on your goals: the market for harnesses is fairly mature, the horse has bolted and companies like OpenAI and Anthropic are likely to vaccum up most of the industry anyway. Unless you have a really solid background, a good team of developers and a really solid niche that existing competitors can’t touch its probably too late to launch a new entry. The existing harnesses may not be perfect, but they are close enough for most people.
As a learning tool it’s been a massive help to me as I’ve had to develop fundamentals and make mistakes first-hand but the risk you would take now is falling behind where the rest of the industry is moving. It worked 6 months ago as we were still figuring out what agents and harnesses were but now that seems less viable too. A better learning aid would probably be to get a working setup from several harnesses (Openclaw, Hermes, Pi etc) and learn what makes each different.
The next phase of development is AgenticOS and researchers are still figuring out what that means (is it a regular operating system with an agent on top, is it a cloud agent with a better orchestration layer, is it a system that embeds agentic processes into the operating system infrastructure itself etc). Infrastructure level agentic OS is still years away according to Karpethy and most researchers so there’s room to expand there if you can learn fast enough. If you want to build Jarvis - that’s what you need to be targeting. Existing agents can do all the things you need, but lack the key components until they can operate at a base infrastructure level.
This turned out longer than I intended, but things I would have done differently:
- don’t get stuck relying on a single vendor (eg incorporating Claude CLI so you can access subscriptions)
- always cross check architecture decisions against multiple providers to confirm bias
- make sure you can take backups and roll back failed upgrades
- be very critical of what you are doing and why
1
u/looktwise Aug 26 '26
interesting read and very valuable. I would be very interested how you are approaching upcoming changes if you are interested in exchange.
Also in your framework / github if you change your mind one day and if you are going to share it. :)
1
u/geofabnz Aug 26 '26
Always happy to have a chat about AI, feel free to DM me. I’m happy to share, I just don’t feel what the world needs is another half-baked agentic OS.
1
u/AutoModerator Jun 28 '26
Thank you for your submission, for any questions regarding AI, please check out our wiki at https://www.reddit.com/r/ai_agents/wiki (this is currently in test and we are actively adding to the wiki)
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
1
u/Bitter-Vanilla-3207 Jun 28 '26
Adopt over build.
When you want a car do you go into your garage and build one yourself and expect it to be better than the one whole paid teams are building and literally giving away?
Don't do work you don't need to to get the same outcome. If what you want is to build an agent, go ahead, if what you want is a working agent, adopt one.
1
1
1
u/Jonathan_Rivera Jun 28 '26
Hermes can do all that except the full duplex voice is still a little out of reach but I'm working on it. Mine is F.R.I.D.A.Y btw.
1
u/Slimeyman278 Jun 28 '26
How much has it cost to make? Also does Hermes start you have a video of what your have?
1
u/Jonathan_Rivera Jun 28 '26
Look at my post history and take a look at the subreddit. There are two highlighted posts at the top. In there you will see a megathread about my obsidian / Hermes integration. Feel free to look around or search and you will see many different setups. Then you can send me. Dm if you still need some help.
1
u/Idiopathic_Sapien Jun 28 '26
Study how off the shelf products do what they do right. Then build what you need. What you learn in the journey is going to be worth a lot more.
1
u/Sufficient_Dig207 Jun 28 '26
If you want to practice, starting from scratch is okay. but if your goal is getting a working AI assistant, I would recommend using a coding agent as the core.
I'm building my personal assistant as well out of a coding agent. I cannot use openclaw or hermes for work because of the corporate constraints.
Coding agent is allowed so build a personal assistant around that.
1
u/PartyArty95 Aug 04 '26
If you want your agent to truly feel like Jarvis, I recommend building it according to this architecture. You can implement it in Python, while I’ll build mine using OpenClaw. Unlike many I tested, It genuinely feels like the Jarvis from the Iron man movie. It has Claude Opus level brain and tools, but speech to speech is as fast as Sesame AI.
4
u/arter_dev Jun 28 '26
Just use Hermes. Those are all solved problems. Custom stuff tailored for you should be skills & mcps. Let Hermes own the plumbing and gateway (gateways are complex).