A lab seminar on Mamba
While DRIFT was under review — that stretch where you don't yet know whether a paper is in or out — we started thinking about what could come next. One idea was to bring the Mamba architecture into the work, as a way to push the research further.
So that the team and I would actually understand it before building on it, I gave a lab seminar walking through the paper: "Mamba: Linear-Time Sequence Modeling with Selective State Spaces."
The appeal is simple to state and hard to achieve: Transformers are powerful but their attention costs O(L²), while RNNs infer fast but struggle to hold long context. Mamba goes after both at once — RNN-speed inference with Transformer-level quality. What makes that work is selection: letting the state-space model's parameters depend on the input, so it can decide what to remember and what to forget instead of treating every token the same.
Preparing the talk was the real lesson. It was only while building the slides that the equations — and the selection mechanism itself — finally clicked for me.