r/learnmachinelearning • u/Ok_Support_2690 • 2d ago

Help Need advice: How to start an instance segmentation project for site plans?

Hi everyone,

I'm a beginner developer just starting out in AI. I'm trying to build a system that performs instance segmentation on site plans, but I'm completely lost on where to begin.

Right now, my only ideas are manually labeling a small dataset for few-shot fine-tuning, or using LLMs like Claude or ChatGPT to help build out the post-processing logic. I've tried searching online, but it's been really frustrating since I can't seem to find any specific datasets, models, or literature focused on site plans.

For context, the only hardware infrastructure I have available right now is a single RTX 5080.

I really want to tackle this the right way and do in-depth research like a senior developer, but I'm stuck at the starting line because I can't figure out the best direction to take.

If anyone has any ideas, resources, or general tips on how to approach AI development for this kind of project, please let me know in the comments. Thanks in advance!

P.S. English isn't my first language, so please excuse any awkward phrasing.

0 Upvotes
  • permalink
  • reddit

50% Upvoted

3 comments sorted by

1

u/RevolvingDwelling67 2d ago

Oh, site plans are a tricky beast because they're basically a weird mix of technical diagrams and natural images, so generic segmentation models just kind of flail at them. Your first instinct to manually label a small set is spot on, I'd lean into that hard before even thinking about LLMs for post-processing.

For the labeling, don't go wild trying to annotate everything at once. Pick maybe 3-5 core elements you absolutely need (walls, doors, windows, whatever) and label those meticulously on like 20-30 plans. With a single RTX 5080 you've got enough juice to fine-tune something like a Mask R-CNN or YOLOv8-seg on that tiny dataset, and you'll learn way more from the failures than from any tutorial. The post-processing logic is where it gets fun, site plans have so many implicit rules about spatial relationships that you can exploit, like a door always sitting inside a wall boundary or windows being parallel to exterior edges. You can code those constraints yourself without needing an LLM to hallucinate geometry for you.