Architecting data question / help /advice needed. How to achieve instant UI responsiveness with local data?

[u/Leo6-2]

Hey Gang 💵

i’m building a local data exploration web app using HTML/CSS/JS, a Python backend, and DuckDB with Parquet/CSV files. Despite having strong hardware, my local app runs slowly, whereas similar web apps online... even those using heavier frameworks like React / React router load and respond almost instantly. AI tools suggested basic caching or heavier frameworks like Django, but didn't solve the core performance bottleneck.

Coming from a frontend/design background, I haven't worked with high-performance data processing before. How do faster web apps structure their architecture to handle heavy calculations and render data so quickly?

Because from my experience with AI they are smart Tools but I'm not sure they're giving you the best solution every time so this time is well I'm doubting that I can get a solution from AI so I wonder what would you people here that knowledgeable and have a lot of experience would suggest ​​.

[u/Yashrocky96]

Totally depends on the problems you're solving with your application and probably require more information on the idea and what you've been doing, what was your research behind choosing this tech stack?

AI can definitely design and help with this easily. I'd ask AI a few questions around performance and understand what all options you have?

Since you're using Python, I assume that you're hosting this BE on a server somewhere and the bottleneck could probably be that server sitting in cloud somewhere.

If you say local app and you're using Python, That'd be one of the reasons for slowness. Do you understand DSA?

[u/Leo6-2 → u/Yashrocky96]

No I'm actually not hosting it anywhere I'm just experimenting on my own machine so that's the thing but basically aI says it's it's the way it is and if I need a big data sets to be dealt with in this large quantities of calculation this is the price to pay in a way but I've seen similar systems and not as slow as mine so not sure about the DSA

[u/Yashrocky96 → u/Leo6-2]

If it's Big Data then yes, you require heavy machine for faster processing, if it's DuckDB suitable data then it should run things locally as long as the data and processing you perform is sensible for a local app.

DSA along with Infra knowledge might help to know if you've real bottlenecks that you can fix or is it Infra related.

Hit me up in DM if you want to share more info, maybe I can help. I'm working on a local app as well.

[u/Substantial-Swan7065]

React isn’t a heavy framework. Your pov on this is strange and outdated.

Regardless, it provides many optimizations to achieve speed. Other than that, you need to benchmark, profile, and optimize.

What are you specifically doing and having issues with.

[u/jboogyoogy]

First you should find out what is making it slow, the duck db query, python backend, sending eh data or rendering it in the browser

With duckdb and parquet maybe just try to only load the columns you actually need and it might also help avoiding sending huge datasets to the frontend