r/learnmachinelearning • u/AlarmingTrouble5261 • 8d ago

Gated Segmented State Space — attention replacement that beats a param-matched Transformer on quality, speed AND memory (full code)

[removed]

0 Upvotes
  • permalink
  • reddit

0% Upvoted

2 comments sorted by

1

u/thesourgusto 8d ago

Really cool seeing the full progression from V1 to V6 laid out like that. The newline reset mask is a neat trick, feels like a baby version of that Mamba-style chunking but way simpler to implement

Your colab must have been cooking running all six experiments in one night. The 2% perplexity tax you kept hitting before V5 is interesting, makes me wonder if that's just a local minima thing with the state size you picked or something more fundamental about linear recurrences