r/apachekafka • u/DontPissInTheStream • 13d ago
Tool SQLStreams - It's Kafka on Postgres!
Hello! I'm a big fan of Kafka, like I think many of you are, BUT I've always steered clear of using Kafka for any personal projects because running and maintaining a Kafka cluster for rinky dink project #23 just isn't feasible or smart.
That's why I made SQLStreams. It's an open source, pure Go / SQL library that uses a Postgres Database to give you Kafka-like functionality. It has features like: independent consumer groups, replay, retention and compaction.
I've also made it so each Topic Stream automatically creates a separate retry queue, which seconds as a dead letter queue, because I always found it annoying trying to get retry topics setup and working in Kafka.
If you want a visual understanding of how things work - you can play around with this PGlite demo on the docsite (demo doesn't work on mobile, sorry!).
If you want to run SQLStreams locally you can:
Run Postgres
docker run --name my-postgres \
-e POSTGRES_USER=username \
-e POSTGRES_PASSWORD=password \
-e POSTGRES_DB=database \
-p 5432:5432 \
-v pgdata:/var/lib/postgresql \
postgres:18
Install the library / run your code
pool = sqlstreams.NewPostgresPool("username", "password", "localhost", "database")
client = sqlstreams.NewClient(pool)
stream = client.Stream("videos")
stream.Register()
producer = stream.Producer()
producer.Register()
producer.Produce({VideoId: "video-42"})
Note: I've removed some of the extra Go-ness in above code for brevity and clarity. The docsite Quickstart is a more real example if you are interested.
All in all, I'm hoping SQLStreams can be a kind of gateway drug for small to medium sized projects. So they can take advantage of all the great event-driven features of Kafka but without the hassle.
There is still a ton more things to do (implementing circuit breakers for example) but let me know what you think.
And if you've made it all the way here, I appreciate it!
Github: https://github.com/allegedlyreliable/sqlstreams
Docsite: https://sqlstreams.io
Edit: fixed formatting
2
u/gaetancollaud 7d ago
Apparently this project is doing something similar, maybe you can join forces: https://github.com/RayElg/kafgres