I’m lucky enough to have an M5 Max MacBook with 128gb of Ram, and I’ve been spending a lot of sweat this summer trying to understand the ins and outs of running local AI. And the ins and outs are a lot more detailed than I originally thought. There are lots of new little terms to learn, that are connected to big, complex concepts with major implications on how effectively a local model can run. I hope to put up more blog posts later with some details on the different setups I’ve been exploring. But this one is a quick note that the brand new Qwen Flash Preview model is way better, and way more accessible, than I expected it to be.
This is a big model. It’s about 100GB of download. So you’ll need a larger computer to even have it sitting around. On top of that, it takes a lot of RAM to run (around 100GB minimum if you’re going to do any real work with it). If you have space and RAM though, this model is great.
This model is super new though. Just a few days old, and support in the apps that run it is hot off the presses (new as of this morning). So expect bugs if you are going to try it yourself. Here’s the setup I have working well for me this morning.
- MTPLX, Download the latest version (2.10.0)
- This quantization of the model – Youssofal’s optimized speed build. It takes a big part of the model (the ngram layers) and keeps it on your SSD instead of in RAM. This frees up a bit of space for the actual context you’ll be processing.
- A coding harness – Most coding harnesses work well with local AI runners, but I’m using Pi, because it’s lightweight by default, and tries to keep its initial context small. As I’ve done more local AI I’ve quickly realized that context takes up a lot of RAM and takes a lot of time to parse. Efficiency matters.
This new model uses a new, more efficient architecture that is more clever with the way it computes the attention math between the weights of the model and the conversation context. This means that, even though the model is huge in memory, I’m getting between 30-60 tokens per second when generating, and 388 tokens per second when parsing on real context (61k on my current task). With Qwen3.8 27b, a much smaller model, my numbers were about 1/3 of that as context grew.
That said, I’m still trying to feel out overall memory growth. According to other blog posts, because of the way this model structures its attention mechanism, a full 256k of context should only cost about 6GB of RAM:
But my current task at 71k context keeps spiking into the warning range, close to 100GB of usage. I’m interested to see whether the thread I’m currently working on ends up exceeding memory limits and crashing like many of my previous long-running threads have done on oMLX with this (and other) models.
Have fun! And I’d love it if you let me know how this goes for you if you try it out.
Caveat
I’m excited about the speed of this model at its size, but I have yet to see if the quality of its work measures up to what I’ve been getting over the last week with the smaller (but much heavier to run) Qwen3.8-27b. Crossing my fingers.