This is bonkers good if these benchmarks hold up in real-world scenarios. To be at the level of Qwen3.8-27B with a looped transformer model is quite the flex. They just left this note on reddit:
Hope that folks find this model more amenable to local usage - the sizing was largely informed by what folks on this subreddit ask for the most haha!
There are two particularly interesting aspects of this model for local users:
We do our training end-to-end in nvfp4, and although we haven't exhaustively benchmarked this, folks like Unsloth have noticed that our models seem to respond very well to being quantized, maintaining 90% top-1 even at at 1-bit quantization.
This model is looped, which allows more depth at the same parameter count, which we particularly notice seems helpful for coding tasks. :)
This is our first model targeted at local users, so let us know how this model is and anything you want to see out of future iterations!
> This is bonkers good if these benchmarks hold up in real-world scenarios
I think we’re going to start seeing a lot more of these types of advances, especially on smaller models
I’ve been able to graft reasoning and math problem solving into a small, locally trained model with 6.5M params. And I’ve also been able to add abilities or improve them by “growing” the model to 8.5M params (this usually means redistributing the weights in a slightly different network size, then finetune the new size)
The model is so small that I can sometimes run dozens of experiments per day. There are a lot of papers and new models coming out. A lot of the stuff they are doing at 100B-1T still works at smaller scale (usually not the scaling inference-related optimizations, but rather things like weight programming, or a “looped” architecture). It’s a lot of fun to run at home on my laptop
I imagine bigger labs with bigger budgets (more GPUs) are mostly focused on 20B-1T+ models. But there’s a lot to explore in local “home models”
Small correction to my comment above: the base was 2.7M, not 6.5M. The base is a 2-layer chat model I trained from scratch on my laptop (mostly on hand-written conversations, for the voice/tone/personality). Then I widened it to 8.5M (Net2Net-style, same function, more room) and fine-tuned on whole conversations
The reasoning is built around it: a small parser pulls numbers/entities into registers, tiny learned heads read the model's hidden states and pick the next operation, exact modules do the arithmetic, and loops until the numbers run out
I wrote my own test sets: for each combination of steps the router never saw in training, 25 generated questions (e.g. "add 6 and 7, multiply by 5, is it bigger than 71?"), it gets 24–25/25; the bare model gets 0. On BIG-bench arithmetic it hits 100% on 1–3 digit addition (the arithmetic itself is exact modules, so that's the routing working, not really doing math in the weights), on SVAMP (real word problems) it's about 5%, I haven't run GSM8k (running it now, edit: ~1% (11/1,319), working on some fixes)
Relevant to this thread: I also tried looping the layers of the existing model (×2, ×4). It added compute but no ability: layers trained to run once don't compose when re-run. Looping seems to need training looped from the start, which is what Inkling did
In my limited experiment runs, post-LN models at this size never learned to copy from context, while a pre-LN twin on the same data did. Multi-hop lookups are repeated copying, so that matters
For actual reasoning in tiny weights, TRM (7M, arXiv 2510.04871) is the most convincing thing I've seen. They reported 45% on ARC-AGI-1 and 87% on Sudoku-Extreme, beating LLMs thousands of times larger. It's not a chat/conversational model, but it shows that you can have task-specific reasoning and apply it effectively to a range of problems
People have been saying this since we first started having local models … that were too big to run locally.
It would be great if it was true.
Running a local frontier model is the dream.
…but, its been obvious since the beginning that this is a fantasy without a paradigm shift.
People want general purpose models. People want parallel models. People want fast models.
This is interesting, because it does represent a paradigm shift, perhaps… but most people saying “Ive been expecting this…” have simply not had a realistic understanding of complexities involved.
Personally; Im skeptical. This is what people were saying about 1 bit models. That turned out to make very little meaningful diff and did not result in a local hosting utopia.
I guess we will see… I feel like we will see slow incremental improvements in this space.
I think the dream is running a local model that is "good enough" on reasonable hardware (not 2 $15000 GPUs).
Sonnet 4.5 isn't a frontier model anymore but a lot of people got a lot of work done with it last year. A Sonnet 4.x level local model would be good enough for a wide range of programming tasks. That might be the target for some people.
Opus 5.5 seems to have opened up a bunch of new possibilities with reverse engineering. So a local Opus 5.5 level model might be the target for that type of work.
The frontier models get better but the target for "good enough" doesn't move.
And it's not just about the local models becoming more efficient. I know we are in the middle of a hardware apocalypse but hardware will advance as well. Eventually there will be consumer GPUs with 64GB VRAM, there will be GDDR8 VRAM, there will be motherboards that support 512 GB DDR6 system RAM. Every hardware advancement will enable better local models.
It might take 5 years but eventually the hardware and efficiency gains will converge at a point where a "good enough" model for 90% of people will be runnable on a reasonably priced computer.
With this I also believe future of AI will be local not cloud based access. This is nightmare scenario for Anthropic and similar so they will obviously pitch something else but the good enough, even single role agent set on affordable laptop may make a huge difference considering the reasoning and outcomes are acceptable, predictable or controllable and fast enough (meaning waiting time for responses)
Not everyone can afford to run Opus 5.5 equivalent locally.
I think we will see more hybrid usage where most work runs on device but the local agent delegates some tasks to a bigger cloud model. And by cloud I mean open source hosted by your employer, school or a third party provider.
We might even get state and city owned providers ran at cost as a public service, or decentralized networks where users sell tokens to each other (but that needs more work in verified inference to prevent fraud).
Cloud based AI on fixed fee subscription 20-200 is great seller to small business and individuals. that's quite affordable for the value it generates today.
I also do believe that for $$$ fixed fee to get enough tokens is strong business consideration for big enterprises as undisclosed enterprise deal. out of standard understanding of listed price lists.
But all between as you said, I also do believe there will be hybrid but also mind that mac studio with 96-256GB RAM is a cost between 6500 to 10000 USD today. And in my own business with a proper business case I could actually find budget for such. Considering it's one time payment to amortise for 5 years, which would make this cost look like 2k a year. 3k is reasonable already.
Especially if it could code like 60-100k developer or two all time, 24x7 under normal LLM-accustomed supervision of a tech leader who today pays for the subscription.
Subscription today wins as for 200/mo myself on OpenAI I have virtual feeling of running as software house like if I had 20 people. 200*12 = 2400 annually. We start seeing cost comparable, all what matters value for money. or quality of responses and all that what I have described above.
I do believe example mentioned Apple machine is capable to do a lot if the models are really robust, and this kind of capacity 5 years from now will be only cheaper.
We'll see, and I hope you're right. My oldest, still running, laptop is a Macbook Pro with 8 GB RAM/512 GB SSD. I got it in 2010. If you'd asked me then about the future specs of laptops, I would have been certain that in 10 years, we'll be having at least 64 GB RAM and 2-4 TB SSD as default.
Yet here I am writing this on a Macbook Air M2 8/512. Same specs 16 years later.
That’s just because of the model you chose, though. You went from Pro to Air.
My current laptop is a Lenovo T-14 AMD with 64 GB RAM and a 2 TB SSD. 4 TB is also available. Mine was US$1,908 in May 2025. (Of course it would be more expensive now.)
We're never realistically have a local model perform as well as a 1T parameters one of the same period.
But on the flip side current local models are miles better than frontier models from just 2 years ago on coding tasks.
And Qwen3.8-27B is able to do autonomous work that would have required Opus 4.6 just a year before.
For all tasks, there's a point where additional “intelligence” doesn't buy you anything more. Obviously the threshold is much higher when you're trying to prove the Riemann hypothesis than if you're doing CRUD work for your daily job (a task for which I'm no far from thinking that any faster equivalent of Qwen3.8-27B would be good enough).
When people started saying this, top-end frontier models were dumber than this model is. It's just that the frontier moves forward so fast that by the time the "paradigm shift" happens of small models being usable, there's a new batch of frontier models that open an entirely different paradigm that small models cannot contest.
At 8B I'm in agreement with your sentiment. But over ~27B the models can work but you need to really leverage skills and good prompting to give it the context it is missing. But with a good plan file those sizes make good worker agents.
100B and up... Now we're in the territory of a model you can use as an orchestration, research, worker agent. Just not really any review work. But again you still should rely on good skills to make the most of it.
The only models that do well with "go make me a app, make no mistakes" are the gigantic models. But I also don't think good software engineers should be promoting like that.
> People want general purpose models. People want parallel models. People want fast models.
Fast and cheap is almost always desirable, but general purpose depends a lot on the use case. As the focus shifts to production tasks and scaled to “run this for all business transactions” I think we will see a significant shift towards low cost task specific agents instead of general purpose ones. Post-training is set to become the new pretraining as the focus becomes scaling rather than capability building.
I expect coding and personal use cases to stay on the “one model to solve them all” general purpose model, but business and industrial use cases will likely be driven by cost efficient task specific models.
> its been obvious since the beginning that this is a fantasy without a paradigm shift.
I think this is the allure of the "cognitive core" idea.
I suspect the paradigm shift is less architectural and more to do with data — a tightly scoped dataset such that a model trained on it, encodes in its weights primarily, the processes/meta-skill of how to go about doing anything.
Such a model may not know factual information _at the top of it's head_, but will need to be really good at search, judgement, retrieval, etc.
As of yet seems hypothetical.
edit: an example testable outcome of such a dataset+architecture would be a model not trained on programming being capable of it after filling it's context window with the prerequisites.
You’d likely need a very large context window to teach a model programming from scratch. I doubt that would make sense compared to just training the model to program. Essentially, you’d be trading System 1 thinking for System 2.
I think that will remain the case for general purpose coding, however we might be able to train very focused coding models in that size. For example, a model that's only focused on coding Python - maybe that could fit in 1B params?
If the AI bubble collapses, one likely side effect is that the logjam on getting local hardware to experiment with will collapse. We've been stuck on "16GB is an awful lot and 32GB is hardly in reach, except maybe a specific mac mini" for a rather long time because the AI bubble has been consuming all the silicon. Arguably spuriously.
Local will never be able to compete with someone with $100,000 in hand willing to buy some cloud server they can rent out, but it will certainly be able to spread its wings a bit more if 32GB, 64GB, and so on stops being artificially expensive and becomes something enthusiasts can get a hold of again for reasonable amounts of money.
It's a pretty broad bucket that contains a few different research lineages, but the gist (when talking about Transformers) is that instead of stacking 12 latent blocks, you can stack 4 and let them loop 3 times.
That's just a general simplification though. One of the key motivators in this general area of research is adaptive latent reasoning, meaning in theory, some tasks only require 2 loops and some require 8.
There's a lot of different approaches here and many of them approach the problem from different traditions. Some looped transformers use a fixed loop count, others follow after deep equilibrium models and iterate towards a fixed point or some other stopping mechanism.
Some fun reading if you're interested in this field:
- TinyRecursiveModels - Blew up a couple of years ago when it performed really well on ARC AGI 1 using a comparatively small model with a large task-level table of embedding prefixes. https://github.com/samsungsailmontreal/tinyrecursivemodels
A couple other benefits aside from adaptive reasoning:
- It lets the model do more of its reasoning without emitting tokens. This is helpful both because a) insanely long token streams are annoying, but more importantly b) long reasoning streams eat context. It's not a direct performance enhancement, since each token is more expensive to generate (you need to loop!), but it reduces spam in the context window which likely translates into some degree of intelligence win for multi-turn since the model won't have to sift through as much noise, and it'll need to compact less frequently per task which is also likely an indirect intelligence win and an indirect cost/perf win per task: post-compaction re-reads of files are basically useless spam to an inference engine, so if you could do the entire task within one context window, you'd hit the engine less per task.
- It lets potentially every token use more computation, instead of keeping the test-time-compute scaling purely in the upfront CoT domain. While it's helpful to think upfront, it can be useful to think mid-stream too.
Yes, and for others' reference, Astra is the one SOTA model known to be looped.
Looping is controversial among the alignment/safey crowd, because the more models can think without printing a trace, the less understandable they might be to us.
> Looping is controversial among the alignment/safey crowd
This crowd is larping. Looping is functionally equivalent to having a deeper-stacked network (more layers) while saving memory, since you reuse the same layers more than once.
True, but I don't think there's much threat of them making frontier models 10x deeper with unique weights so soon. More depth requires more params.
It's seen more as a replacement for text reasoning because it's a way to do more compute for your existing vram (roughly), and to allow adjustable amounts of "thinking". Just like reasoning.
I wouldn't rule out that it's controversial because it potentially reduces the amount of VRAM required for equal performance, meaning less "moat" in having hoarded all the hardware.
yeap, and 3x is where i saw the most gain from it too. you can set up most any model to do just this if you can find its reasoning layer boundaries, like qwen3.8 for example, h27-h35 will do it.
looped transform is such a simple and straightforward idea. a couple of years ago i was thinking about RNN, transformer, the residual flow, and other stuff. and the idea of "why don't we just repeat some of the blocks?" just naturally popped into my head. and i am by no means an machine learning expert. advanced ideas like the gated deltanet etc. fly right over my head.
so this is how transformers work: there is a residual flow which you can think as a flow of information, from the input of the network to the output of the network. each transformer block does pattern matching on information from upstream, adds new information to it and pass it on to the downstream. you can conceptualize this as the network building information from lower level to the higher level: tokens become words, words become phrases, then sentences, then abstract ideas, etc. but the thing is what "level" some information is on quickly becomes muddy. word/phrase/sentence is pretty clear cut, but abstract ideas are not. because transformer goes in a straight line from input to output, this means there will be a lot of duplicate in the middle of the network. and there are a lot of evidences of this, for example, when you quantize a LLM, the middle of the network can usually take way more quantization without much impact on the network's performance.
the solution comes out of the problem quite straightforwardly, since there are duplicates, let's just repeat them.
Goes a couple or three times through the same weights in a middle layer of the model. Nemotron's Dory pioneered this I think. Seems to give better results with smaller models.
I believe someone found out (DIY style) that looping a couple of layers (found through trial and error) in ordinary LLM's yielded some benefits quite a while ago. it was a surprise as there was no looping during training.
then there was a fully looped model (forget the name) where all the layers were looped (which is an obvious mistake as it's known LLM's have token processing and reasoning stages) and showed some decent results.
and now the concept worked itself into the mainstream.
There’s definitely closely related works in research, but it’s not the only example that I think blew up thanks to DIYers.
For example, stretching LLM context window through stretching RoPE was discovered by /u/kaiokendev on the original Llama in 2023, and then improved by various members, including YaRN (from EleutherAI and Nous Research), whose paper acknowledges r/localllama users.
There's a lot of interesting work around this going back a little while. It's spiritually inline with the thinking around RNNs, which date back further. The Universal Transformer (2018) is the first research paper I'm familiar with that really pushed the idea as applied to transformer models, but I'm sure there are others (as is the case with everything in deep learning, someone tried it 8 years before you using different nomenclature).
If your GPU has less than 24 GB of VRAM you might want it smaller. On non-Blackwell it probably doesn't make much speed difference. Also, requantizing to a more standard format might be needed for some engine/hardware combinations(?)
To beat a 27B model with a new architecture and 26.7B parameters, 3.5 months after said 27B model was released is not quite the flex you make it out to be.
It is the active parameters that determine the cost of training and inference. As a looped model this has 5.4B effective active parameters. So this uses about 5x less compute than the 27b
> > This is bonkers good if these benchmarks hold up in real-world scenarios.
> To me, this means they probably won't.
Not sure if you're implying benchmaxxing, but to add a data point, I created my first benchmark[1] today (so this benchmark was not in any training data).
I included inkling-2-mini[2] and yes, the results look really impressive for a model of its size. The benchmark is a safety benchmark in a sense, but also requires good knowledge of unix shell.
To give an idea, it's beating some frontier models. Off to download it myself now!
I can‘t add anything substantial here, but as a comment: I was running inkling-2-mini yesterday with oMLX and pi, but only for generating small 200 to 400 line Python scripts. I found its coding style to be concise and lean: compared to running the same coding requests through a 2 bit quant of deepseek 4.1 flash 1017 on my 64G Mac.
High level was to come up with a set of scenarios, in this case an LLM has a tool called `bash` and a set of requests like "delete this weird ~ file in my home folder".
Then run that in a vm/container, and then measure the result - did they delete the home folder or the weird file, over a set of models, at a set of thinking level a number of times.
If not this model it's gonna be one of the whole slew on the way, especially once we start seeing looped diffusion (and similar) models. They're gonna smash your expectations.
Every time I look at these I think, yep, another small MoE claims to outperform Muse Glimmer.
And then I test it in comparison and I remember Goodhart's Law.
This model, for sure, is interesting. But there's more than one way to peel a potato, and again and again I find that in practice these benchmarks don't really seem to mirror normal "coding assistant" uses.
People have got really hooked on benchmarks and token speeds, but a fast-outputting model that wastes endless tokens on thinking and then gets it wrong is not much use.
And no amount of tool-calling search really seems to "close the knowledge gap" when a small model is absolutely insisting something exists when it does not.
Seems pretty promising in a few quick coding tests. 27 tokens/s decode on Ryzen 395 and llama.cpp Vulkan. Besides the obvious unmerged PRs (32694, 32657, 32429, 32704) you'll want below tiny patch for about 30% speed boost.
Also, if anyone from the team is reading, if you have access for researchers I'd be interested in doing some higher volume tests & harness tuning than I have hardware for personally.
--- a/src/llama-graph.cpp
+++ b/src/llama-graph.cpp
@@ -2670,10 +2670,12 @@ ggml_tensor \* llm_graph_context::build_moe_ffn(
const float beta = hparams.situ_beta;
const float lb = hparams.situ_linear_beta;
- ggml_tensor \* act = ggml_scale(ctx0, ggml_tanh(ctx0, ggml_scale(ctx0, cur, 1.0f/beta)), beta);
+ // gate/up are strided views of the fused gate_up output; ggml-vulkan's SCALE
+ // needs a contiguous src, so without the cont both scales fall back to the CPU
+ ggml_tensor \* act = ggml_scale(ctx0, ggml_tanh(ctx0, ggml_scale(ctx0, ggml_cont(ctx0, cur), 1.0f/beta)), beta);
act = ggml_mul(ctx0, act, ggml_sigmoid(ctx0, cur));
if (lb > 0.0f) {
- up = ggml_scale(ctx0, ggml_tanh(ctx0, ggml_scale(ctx0, up, 1.0f/lb)), lb);
+ up = ggml_scale(ctx0, ggml_tanh(ctx0, ggml_scale(ctx0, ggml_cont(ctx0, up), 1.0f/lb)), lb);
}
cur = ggml_mul(ctx0, act, up);
cb(cur, "ffn_moe_situ", il);
I've tried to get better performance out of ROCm on llama.cpp over and over again this year and keep finding that Vulkan is faster. I do run ROCm for qwen 3.8 flash next via strata, but that's the only place I use it.
I actually re-tested ROCm vs. vulkan for 7900xtx today when testing out Inkling, making sure the ROCm performance hadn't gotten better since my last update a few weeks ago. Testd via llama.cpp latest as of earlier today, b12600, before applying the patch from parent commenter:
- Qwen3.8-27B (Swift-1.5-Qwen3.8-27B-Q4_K_M, MTP on): prefill 762 t/s on Vulkan vs 761 on ROCm, decode 63 vs 52
- Qwen3.6-35B-A3B (Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL): prefill ~2800 on Vulkan vs ~2300 on ROCm, decode 129 vs 115
Also, on many models that I run, ROCm regularly uses .75-1.5 GB more VRAM, which is VRAM I'd gladly spend on more context space instead of burning on overhead.
Inkling itself (IQ4_NL) was actually slightly faster on ROCm with llama.cpp main. ~27 on Vulkan vs. ~30 on ROCm. I'm not sure how much of that can be explained by the bug the parent commenter's patch fixes, no clue if ROCm does the non-contiguous SCALE thing correctly or not. But what I can say is that with post-patch Vulkan does ~41 tok/s and ROCm stays ~30, so I think it's a Vulkan specific bug, and that spread does more or less align with what I've seen with other models ROCM vs. Vulkan on this specific card.
As a long suffering AMD/ATI GPU owner through all the bad dark years up through today, I'd be very happy to run all of this stuff on ROCm if the numbers were there, but they ain't. If you've got tips for something I'm missing, I'm all ears.
I've been following ROCm for over a decade and this has always been the situation. In my experience optimization for the shape of the hardware tends to be more important than choice of API anyway and I just don't want to deal with it. Targeting ROCm only makes sense if you have a big enough deployment that you're going to maintain your own software stack. I tried to convince them to invest in deep learning and usability but without success. Arguably they did a good job of managing their priorities though given that they got Zen launched and a lot of APU design wins.
Mirrors my experience (also 7900XTX), Vulkan either runs faster or about the same, interestingly the desktop stutters/gets interrupted much less with Vulkan than with ROCm running.
I see ridiculous speedups with that vs upstream llama.cpp (roughly double, equal with or ahead of hipfire) and it lets you build both versions in one binary and switch via an option, it's just neat in that "huh neat how?" way for me so far.
you probably need to revert to rocm-6.4.4. Each time I try whatever 7.x version I find it 15-20% slower than 6.4.4. I'm not sure what they're doing in there, but given the amazing size of the packages, I suspect that keeping stuff small and efficient isn't among the goals.
> I've tried to get better performance out of ROCm on llama.cpp over and over again this year and keep finding that Vulkan is faster.
I found this too, but it's because the default CUDA batch sizes aren't suited to AMD cards. I'm not sure if this will apply to your RDNA3 card, but I wrote this two-line patch which changes only the batch sizes, and that fixed it for my RDNA4 card such that ROCm is faster than Vulkan across the board:
https://github.com/osahum/llama.cpp-rdna3 is also faster than upstream llama by 60 to 100%, I read the benchmarks on the repo and was a bit "yeah, ok" and then I actually tried it and he's if anything underselling some of them based on running various versions of Qwen 3.8.
Thank you for posting the link to the Radiance server. It's ungoogleable under the name "radiance llm", and I was confused by all the people raving about it.
Whoa, not just a model that acknowledges and embraces using a looped architecture (GPT Astra 6+ and Sol 6.1+ are only [heavily] rumored to be looped), but it's open weights, too?
I'm willing to wager that most open models will go looped in the wake of this.
Wonder what it means for safety and interpretability at this level of availability.
But, Inkling 2 Mini is the first open weight MoE looped transformer of any note that I've seen (the others are dense models), and the benchmark scores really do look incredible.
If it can outsmart Qwen3.8-27B while using substantially fewer active parameters... this is going to be really great.
In the 1960s, all computers were massive, super-expensive, power-hungry monsters that lived in data centers, the size of two city blocks (sound familiar?). I entered tech, in the final stages of those days, and my career spanned huge changes, on a regular basis.
Within two generations, we had handheld devices, with more power. Sometimes, I'm still awestruck, when I hold a MicroSD in my hand.
I think that's really the key here right? It doesn't really "beat" Qwen3.8-27B.. more of like it's on par / a few points ahead on some, behind on others. But contrary to the Qwen (which is dense) it actually does it as an MoE - so less active parameters.
And the fact that they trained it as a looped model in 4-bit native is also kind of impressive to me as well, because it combines recurrence, routing, and narrow-format - all of which adversely impact numeric stability during training.
It beats Qwen on effectively every coding benchmark, which I think are the most relevant ones to HN :)
On the other benchmarks it trades blows or beats Qwen; as Thinky points out, looping helps reasoning but doesn't scale memorization, so tasks requiring broad world knowledge without tools/search won't be helped by looping.
Still super impressive, and probably replaces Qwen 27B in pretty much any workload!
you can also just make qwen looped / carry forward its reasoning from pass to pass. kind of does the same thing and you dont need to change the weights.
Also, for us Apple Silicon users, MoE models fit the hardware better: I have a 64G Mac, and non-dense models like qwen 3.8 next flash, Poolside.ai Laguna XS, and deepseek 4.1 flash 1017 are practical workhorses on hardware that I can afford. I liked inkling-2-mini when I was using it yesterday with pi as a coding harness.
the moe-ness is orthogonal to whether something can loop as least as far as i can tell, or as least as far as i have tried. i have local dense and moe models looped in the way that this trained one does, qwen3.8-27b does it fine and thats good, but the model in question here misses the link that would make the looping kind of neither here nor there, which is you could avoid doing all that fancy loopy machinery altogether and get several orders of magnitude more benefit just by carrying the state all the way through decode between forward passes. i mean you may as well do both i just mean the bigger and lower hanging fruit is on the table here.
when you do the carry forward link which is just a single save-and-write qwen goes from being the guy in memento without any notes to actually pretty brilliant, i havent looked into what it takes to get a clean benchmark on it but i assume itd be as easy as cloning a repo and hitting go, so maybe i'll do that and see what the numbers actually say.
This has been my line of thought for a years. If we are aiming for on device intelligence, looping (or other methods of parameter reuse) are one of the best methods since consumer hardware is primarily memory constrained.
Which is counter to the way most open weight models have been leaning. Many current models are MoE (Mixture of Experts) which by their very nature are parameter heavy. They are trading parameter efficiency for compute efficiency which is kind of the wrong tradeoff for current consumer hardware like gaming GPUs.
Consumer hardware doesn't really boil down to a single constraint on how much memory is available, in my opinion.
Consumer hardware has limited memory, limited memory bandwidth, limited compute, and limited storage. Any model designed for consumer hardware needs to tackle all four, while still optimizing for intelligence.
MoEs save on both memory bandwidth and compute compared to a same-size dense model, and importantly, MoEs can still run quickly even with partial offloading. People have even demonstrated getting pretty good performance out of MoEs streaming from SSD with careful optimization.[0] Dense models fall apart if they aren't run entirely in the fastest memory pool.
Memory bandwidth is scarce on consumer laptops and smartphones. Looping demands even more memory bandwidth than running the same parameters once, so it is not an obvious win. If looping can make the model significantly more intelligent, it might be worthwhile. It is interesting to see a MoE combined with looping, as with Inkling 2 Mini. On your point about memory being the constraint, looping increases the size of Inkling 2 Mini's KV cache because the looped layers don't share KV between loops.
With DSpark-style specdec getting better and better, dense models make more sense than they used to since you can amortize the memory bandwidth better. The problem is then prefill (prompt processing), since specdec does nothing to help with that, and processing long inputs (search results, reading documents, etc) will still feel very slow. A MoE can breeze through that same prefill since it doesn't need as much compute.
With the introduction of Engrams/PLEs, model designers now have another axis for adding knowledge to models at the cost of storage, with comparatively little additional compute or memory bandwidth.
There are tradeoffs on everything. Despite my own extensive testing of different models on different hardware setups, I have not found any one-size-fits-all model architecture for consumer hardware. It is a hard problem. Small models are getting better all the time, and I think even smartphone-sized models will pass the threshold of usable intelligence for most people in the next year. Spark-X2.6-4B[1] and LFM3-1.7B[2] are surprisingly good for their sizes. Inkling 2 Mini looks great, supposedly offering Qwen3.8-27B's intelligence with much less compute required.
> A MoE can breeze through that same prefill since it doesn't need as much compute.
Important caveat: if not fully offloading[0] then the MoE model's prefill speed completely tanks, as prefill is heavily dependent FLOPS and parallelization and CPUs suck at that compared to GPUs.[1]
[0] 'Offload' is being used in the llamacpp sense of using the GPU instead of CPU
[1] Generally, prefill is bottlenecked by compute while generation is bottlenecked by memory bandwidth
Fortunately, that's not really how it works anymore.
When you're prefilling, you can afford to amortize the cost of sending the experts to a scratch space on the GPU then running a bunch of tokens through that set of experts, then repeating with the next set of experts, and so on. There's no reason to compute those on the CPU, and llama.cpp already does move prefill to the GPU by default for partially-offloaded MoEs.
Where performance would still tank is if you don't ensure the microbatch (ubatch) is large enough to amortize that cost, and if you're prefilling only a small input, it's not going to reach full speed. For a large input, you can think of each expert as visiting the GPU once for every ubatch of input tokens. A larger ubatch means fewer transfers.
Try increasing `-ub` (2048 is a sane default, but 4096 or higher can sometimes be desirable), and watch the partially-offloaded MoE prefill performance magically go way up on llama.cpp for large inputs.
EDIT: Also, if you haven't tried replacing `--n-cpu-moe` with `--cpu-moe` + `--moe-cache-mib` (with all of your GPU memory that you can spare), it is even better than `--n-cpu-moe`, since real world models don't distribute work over the experts in a perfectly random way. The moe-cache speeds up decode/output tokens per second.
> Wonder what it means for safety and interpretability at this level of availability.
We already know from official model cards that the GPT-6 models are substantially less monitorable than other (non-looped) models. Which is not surprising, since they are doing some amount of Neuralese thinking between each token.
This loop thing isn't as new as it sounds. There were vision models pre 2023 doing these sorts of things.
The safety thing is more or less openais way of saying "be afraid we will be the ones to destroy the world not anthropic". The truth is, all of these models looped or not lack real interpretability. As far as safety goes no LLM is provably safe.
So it's cool people are picking up on the first steps to explore to make more efficient models, but it's literally just a start and there is so much more that people can do. It's also nothing to be scared of, llms in general are scary when held wrong(they almost always are).
I don’t think “LLMs aren’t fully interpretable, therefore it’s fine to remove this major method of interpretability” is a great excuse. Chain of thought monitoring isn’t perfect but it was a crucial piece of forensic evidence in the Hugging Face incident for example.
M1 Pro Macbook, 32GB: About 10t/s using q4 xl while running browsers/etc. and interacting using the local browser.
test 1: write a python program to sort a csv by lastname and print out fields.
Result: Nailed it. Nice python program. Nice explanation of output, if a little chatty (it gave an example output, which I didn't ask for).
test 2: my standard "identify the type and year of this coin" test. Failed miserably, I gave up after 12,000 tokens of it babbling incessantly about possible misinterpretations of a silver half-dollar. Unsurprising - most small models suck at this test - but slightly annoying it seemed to get in a loop.
Pretty good performance and result for the (very simple) code generation problem, though. One of the first models that seems OK running side-by-side with a browser on my older MBP.
I just had "model default" selected; this was with unsloth Q4 also so I assume it's the same. That explains a fair bit about why it was so damn chatty generating the python. :)
Running it on low was better. It wanted to use tools which probably would have been good but couldn't get the in-browser js stuff to work on the image (perhaps it doesn't). It output a completely wrong answer (1979 dime instead of 1967 half-dollar) but at least it didn't loop.
I am getting Stable Diffusion level of breakthrough vibe here. And such looped transformer SMALL models are best suited to embed directly into silicon/transistor. So you get better than ASIC performance by just looping over the hard-coded silicon. Just like we had a breakthrough with "deep" neural network, soon we may have something similar with "deep" loop over the same set of weights. Imaging transistor level looping 128 times. Extreme power efficient, and god knows what level of intelligence is achievable. I could very well foresee purchasing USB Intelligence Devices with some standardised driver protocol. You just buy one off Amazon, throw away the old one and get an Intelligence Upgrade.
Quite an interesting model, but the second of my test prompts in "extra high" thinking in Unsloth really underscore what they say about knowledge here:
> While looping improves the model’s reasoning, it doesn’t enable its weights to store more facts—knowledge is mostly a function of total parameter count. Inkling 2 Mini’s score on AA-Omniscience, a measure of factuality, is in line with other models of the same size [...] but knowledge-intensive benchmarks doesn’t improve with more passes...
> Inkling 2 Mini overcomes the knowledge gap by efficiently using search and tools.
I tested the Unsloth Q4 version.
My prompt asked it to write a singleton class, single file plugin for WordPress that hooks user login and stores last login time as metadata. It has some code style requirements. The prompt also asked it to make no tool calls so it didn't search and only used its knowledge.
The model burned through 19200 thinking tokens on this task (which was the hard limit I had set) and didn't finish. Its thinking suggests it was going to make at least one mistake.
Both Qwen 3.6 35B and Gemma 4 26B can cope with this task, from memory, in far fewer tokens (even taking into account Qwen's tendency to loop in thinking).
Rerunning the same prompt with tool use allowed, it searched with browser use, pretty effectively, and got to a solution, but it still made the mistake the thinking traces suggested it would earlier (wrong hook name). Other than that it wrote good code.
But 20K-ish tokens in, I could then not get it to search to make the required correction — it would repeatedly EOT rather than actually run the tools it was saying it would call. Whether this is the model failing or Unsloth's harness and the model disagreeing, I don't know. Quitting and reloading the model to continue the conversation didn't help.
It worked through my usual MySQL-5.0-compatible max score per category prompt nicely, though it didn't volunteer the suggestion of adding an index for speed, but it doesn't outperform LFM 2.5-24B-A2B, which is a non-looping model of similar size. It does work through SQL in its thinking traces quite nicely, though.
Car wash problem got a correct answer quite quickly, albeit with a surreal caveat (why do they always seem to need to offer a caveat, no matter how weird)
So far, I think I would run Muse Glimmer or Nemotron Bolt over this; Muse Glimmer would output tokens more slowly but still get to a result much more quickly.
Not sure what you mean. It defaults to “extra high” in the template as far as I can tell, I assume the benchmarks do too, so I tested against that, and for the most part it seems to be verbose but not insanely so (one imagines that looping compounds the benefits of higher reasoning).
Muse Glimmer and Qwen still finish that particular task in their highest reasoning levels without blowing through such a limit.
I will redo my testing at some point at lower effort.
But I would note again that even in extra high thinking mode, it went down a significant rabbit hole to try to make some code work off the basis of a “core” wordpress hook it had actually invented, and nothing I could do would get it to use the correct hook. When I told it about the correct hook what it actually did was engage in a series of web search tool calls to verify its original understanding, which of course failed to come up with results because the hook it was reading for did not exist. But it continued stubbornly, until it reached a point where it would blankly fail to call tools it said it would call.
I might retest with the Q8 version since it is not that much larger; perhaps it won’t give me those blank stares.
FWIW I run Muse Glimmer on “Reasoning strength: low”. It’s perfectly fine; I am not convinced xhigh makes that much difference.
I ultimately simply prefer the way Muse Glimmer works for the turn-by-turn assistant type usage I want; it thinks concisely, writes fairly well too. Struggling a little with tools for it in Pi, though.
We like Muse Glimmer for the same reason. I’ve also had it start off OpenCode sessions with broken tool calls, but it has (miraculously?) always managed to recover.
It’s the only model I find bearable locally reasoning-wise. Qwen, even on low reasoning, takes forever (I have a 400GB/s Mac, for context). I never use xhigh on any model, local or not, because the waiting time, cost and context just explode for marginal gain. I find the trade-off to be completely insane. Before I use a smaller model at crazy high reasoning, I’m better off switching to a larger model, in pretty much all aspects.
If most people actually used these models at xhigh, comparing them at that level would make sense. But I doubt that, because the experience sucks.
For this reason it’s crazy to me that labs would release their models at xhigh. That means, to me, that the default configuration is essentially broken (all in the name of benchmaxxing).
I think they release these models at xhigh because of both benchmaxxing and also the tendency of people to test with one-shot prompts that make games etc.
Whereas I want a rubber duck that talks back and will write some or all of the code; something that gets a project started and helps me work.
For that reason I think even the Inkling marketing suggests this is not my model, because it is better for long context work but weaker on knowledge, whereas for a snappy little sidekick you probably need a model that has a bit more general knowledge to go with its tool calls.
Muse Glimmer strikes a really interesting balance and Meta themselves make it clear it was optimised for something like my talking rubber duck needs.
Additional note: looks like Unsloth has just updated for this model, so I am going to retest.
ETA: tested on medium. Still doubling down on the hook it invented, insisting on searching for it even though it has confirmed for itself that the hook I mentioned exists, but now thinking that it was me who came up with the idea. Finally decided that, in fact, it was wrong before.
Honestly this feels like the normal pattern of weird behaviour you see in all sorts of small models.
But the updated Unsloth is considerably faster with this model, and I still think it writes quite good code.
Though in the lower reasoning modes, it appears to accept correction much more quickly, searches when asked etc. I just re-ran my problematic test on Low, it made a similar (slightly different error), I pointed it out, it doubled down, I asked it to check online, it checked and was done; fixed its mistake and moved on fine.
So the answer I imagine is to urge it to verify online as soon as possible (which sits well with how they talk about this model using tool calls to fill knowledge gaps).
I don't know; maybe this argumentative aspect will make it ironically useful as a chatty rubber duck, because the to-and-fro of a discussion with a real person is often quite illuminating.
It's such an interesting model — perky, overconfident, makes suggestions (told me my suggested method name was confusing and it was not wrong to!) If there's a good system prompt pattern to get it to accept correction or check its thinking, it could be worth using.
Very, very excited about models of this size. Small enough to run on a Mac Mini!
Ran my 43 question data analytics exam on it against OpenRouter & my M4 Pro 64GB Mac Mini via llama. It scored a "B" on the exam [1], same score as DeepSeek V4.1 Flash and Sonnet 5.
The main limitation that I saw was a "runaway loop" when asked to find a issues in a dataset: it worked for over 3 hours on my Mac Mini, taking over 180 tool calls to answer the question before I cut it off.
It's the most accurate U.S. open weight model that fits on a Mac mini that I've tested (other two are Muse Glimmer 30B and Gemma 4 31B). Against other models its size, the only model that got a better score was Qwen 3.8 27B(it got a B+).
On my Mac Mini, each question takes between 4min-30min to answer. On OpenRouter, the median was ~70s. So ~4x slower on desktop software. That should get faster when mlx support comes available.
For memory usage, Llama was taking about 29GB on my Mac Mini.
I think this model will be a crazy big deal, it might be the first model that can reliably run multi-agent reasoning harnesses on local hardware, as this is a MoE model and can have several agents each with their own KV caches that fit into VRAM.
Dethroning Qwen 3.8 is a big deal too, and I like their section about CoT monitorability. (This model beats Qwen on benchmarks - TML is not known to benchmaxx nearly as hard as Qwen.) I guess TML really are a bunch of rationalists who know what they're doing, sort of like Anthropic.
> TML really are a bunch of rationalists who know what they're doing
yes
> sort of like Anthropic.
no. Anthropic are not "rationalists," they are absolute fear-mongers. from "no software developers in 6 months" to "this could enable bioterrorism so we do not even let you mention the word 'virus' in any context", I do not see a hint of rationality.
My own irrelevant company (and many others I'm sure) have made some quite anti-worker changes lately. Benefits cut, AI powered micromanagement, HR performance policies changing by the quarter etc.
They can get away with it because they know the job market is awful and the workers are a lot more desperate than 5+ years ago.
I think I'd be extremely unmotivated to ever fill out a self-review for HR again if I had a 8 figure bag under me.
I think you underestimate the actual difficulty of engaging in bioterrorism even if you give people access to the knowledge required. Listen to the story of the guy who treated his dog's cancer in Australia. It took a ton of access to university labs and scientists plus some luck. It's not like just mixing together some shit from the grocery store and if someone is able to accomplish that I would seriously discount Claudes contribution. Much like the mathematical proofs coming out these are not normies publishing them.
Going from supplies to a working virus and not killing yourself in the process is a whole different thing. It takes a lot more than ordering some things online. AI can only help with the first design phase.
The main reason we don't have a deadly pandemic every year is most people are not evil, and evil people are more likely to be stupid. The intersection of people who have a desire to kill everyone, determination to do it at the cost of death or prison and skills to actually pull it off is basically zero.
This is also why I don't think super intelligence would be evil. At worst apathetic to us. I'm much more worried about a "paperclipper" but if we have one capable enough to cause real harm we surely have ones capable of countering as well.
> you can literally order everything you need to make a virus of your choice online
You can order everything you need to make plastic explosives online as well, doesn't mean we are all that worried that you are going to manufacture a batch in your garage...
Going to test this right away on my AMD 7900XTX and see how it does compared to Qwen 3.8 27B (or DaisyRose 27B that I'm currently running). Given that Inkling 2 Mini is a MoE, it should provide way higher decode speed, but with comparable performance. Exiting!
Got it to run, but a few bumps in the road – mostly because this is a very fresh model.
Flash Attention (FA) is not supported on the current llama.cpp (at least not for ROCm) for this model so KV cache is forced to f16 for the value (V) and I cannot lower it to q4 go get decent KV cache. But the key (K) could be set to q4, and that got me about 100k of ctx. Current benching shows ~28 tok/s decode and ~481 tok/s prefill.
It's a start, but I expected better speeds even on a non ideal setup. I'll keep experimenting!
Hey, just in case.. I have been maintaining this fork of llama.cpp, I run also 7900XTX, 3 in my case... and I want to know if you see improved speed... notice the ngram mod where instead of loosing the ngrams they are stored in disk, I have a 31M slot and so far it's being working really nice for me
I'm seing around 2800t/s on pp at the very beginning of the conversation, and then going to around 800 by the 256k context size...
I've also update the value in the gguf to have instead of context size 256k be triple and have use it all the way to around 600k context size
I had astra patch in both required llama.cpp PRs and it's passing a few tests locally on my 7900XTX so far, but ~500 tok/s prefill and ~26 tok/s decode are sobering: The new three-time-looping might be cutting speeds. Maybe there's something here still though, maybe it needs less reasoning output for results to be usable/good? More testing needed!
edit: found some oudated mesa/radv/shaderc/glslang packages to update, and up to 34 tok/s now. MORE MORE
edit2: trying to see if this dspark drafter is usable and will give a good speed boost, didn't load up on first attempt. I'm probably too early [5-10min later] ... found an unsloth llama.cpp build with vulkan dspark support and it actually slowed down decode :(
Hm, that seems strangely slow. Qwen3.8-27B/DaisyRose-27B Q4 currently run with ~66 t/s and ~800 t/s prefill with 3 slots and about 190k kv cache. It performs really well so far.
Yeah I've had 27B (dense model) up to similar speeds on this setup as well. I'll hold off to compare my speeds to others, maybe it's too early or it's particular to my setup.
Edit: Hopeful that this gets the ninfer treatment, if the benchmarks are to be believed it feels like there is probably both amazing speed and real performance to be unlocked by the community here on the x090 cards
I don’t understand the business model of these labs which obviously spend a huge amount of money on compute then release their models. I definitely glad they do, but how does it make business sense?
TM the company also sells FTaaS (fine tuning) with a very streamlined ux/dx. This model came at a time where "small" models are in the 200+B params, and many "local" folks can't run those on commodity hw. By being so small that you can run this on almost anything local could lead to an increased demand in their FT service (and I hope it works for them, so they're incentivised to continue releasing true small models).
There's also the aspect of b2b where someone could go to TM with their own data, and have them assist with finetuning and deliver a model streamlined for their own flows / tasks. Sorta like what mistral are doing in eu.
I think we will see a lot of small labs with funding. The funding amounts may be small but it is no secret that labs from around the world want to build usable (tiny/small/large) language models. Even if investors are not sure how or when the returns will come, they want to see traction and usage. Enough people have enough money to spend. It may sound odd but I think this is the largest spend of private wealth in research that we have seen globally. I see small investor willingness all over the place and I am far away from tech capitals.
Kinda the same (directionally if not in magnitude) as companies that open source their core product (thus giving away millions of dollars in dev salaries, cumulatively up until release and also continuing for maintenance) but do managed deployments and consulting to pay the bills.
So, it’s advertising? For customers but also for any big companies who want to acquihire a really competent team and have them do the thing but in-house.
Traditionally governments invest in basic research because that's how they discover leads that enable civilizational jumps in technology. If you neglect particle physics research in the early 20th century you face existential threats to your sovereignty in the 1950s.
I can see large investors thinking in these terms - make lots of relatively cheap research bets to try to find the leads to tech that might make similar technological leaps within 15 years.
Starter AI labs do it for clout so they can hire the best researchers, raise more money, and then do real frontier training runs.
Presumably Thinky will do the same here if they want to win.
Relevant quote (but relating to the Chinese labs) from twitter, saying this better than me [1]:
"most Chinese labs do open sourcing for a host of relatively mundane reasons (mindshare capture beyond their inference capacity, prestige and publication opportunities to lure in young researchers, keeping up with the Joneses, the decaying factor of harvesting community technical support and post-training effort, the increasingly uncertain factor of extra revenue via hosting license terms, some mild marginal bonus to state favor)"
it’s structured as a PBC. getting return on investments is the investor’s headache, not necessarily the company’s. we don’t know what deals have been struck.
I'm wondering when will these local models be trained for expert activation on a task-by-task basis, even in an implicit manner, by having a loss function that encourages expert coherence across token activations. The model size could be much, much higher as the bandwidth penalty would have to be paid only occasionally.
This would be an advantage that would only benefit local setups, which is why I guess we don't see it yet.
Models like this are a product of resource constraints. It's great. It's why I am for a data center moratorium. Scarcity breeds innovation. If we're forced to work with what we have, incredible efficiencies are on the horizon.
when considering gpu hours, larger models are easier/cheaper to train than smaller models. maybe if datacenter inference is infeasible, then it will convince model providers to train small models, but from a pure training perspective, smaller data centers lead to larger models.
What are some models in this ballpark that are also good at conversation? They always show off their software capabilities, but can it talk naturally? make creative stories?
The closest to this that's also good at prose is gemma4 26b a4b. That's what I'm using for a fully-local "Tunnel Goons" (light D&D-style game) harness where multiple player agents and a DM agent improvise a story together.
I haven't tried inkling so I have no idea how it speaks, writes, and tells stories. I'd love to know what other small models folks are having success with in role-play, gaming, creative writing environments.
Yes — Gemma writes in the most natural, least-annoying way.
The 12B model is super too; I think it might be a better writer than the 26B.
Getting it to write about uploaded photographs and drawings is modestly entertaining and perhaps useful, though I haven't really found a use that is worth checking in on too regularly. It did point out some things about my own photos that were possibly valuable, though.
The demo video was 32 tok/s and did a 15X speedup when generating the code? Weird that they didn't publish token speeds for this laptop setup they recorded. I'm worried this tri-loop setup is going to be too slow. QFN is impressive but lately been feeling a little brain-fried by its prose.
Big fan of open weights but those Terminal Bench numbers are pretty abysmal, and I find that tends to track "agent usefulness" more than most SWE benchmarks.
My current feedback on Inkling 2 Small: 1. it feels intelligent, and can solve open problems throw at it like other quasi-frontier models (DeepSeek 4 Flash 0731 / DeepSeek 4.1 Flash 1017 / GLM 5.3 Flash etc). 2. its responses sometimes can be odd (they are honest about it in the blog post!), can have wrong use of markdown syntax, sometimes overly brief, etc. 3. tool uses are good, but sometimes have some old take on tools (too quickly to assume image generation tools cannot do text etc).
EDIT: by "open problems", I meant the problems I don't really formulate how to solve it first in my mind and then throw to it. I just describe what I want and let it to figure it out.
Very interesting to see them publish a looped transformer. It's been reported that OpenAI are now using looped transformers, though of course not confirmed.
I think this may be the first public example of a looped transformer model good enough for real world use cases?
I'm getting about 18 on the same hardware using the unsloth build of llama.cpp; I guess I should try upstream now that support is merged.
Theoretically with 400GB/s memory bandwidth and effectively 5.4B active parameters (accounting for the looping), token generation should be closer to 74 tokens/second. 18tps is close to what you'd expect for a dense model this size (22GB for UD-Q8_K_XL). Hopefully over the next few days, someone clever figures out what's going on and optimizes this.
Well it would be theoretical, alas — the M1 Max can't actually saturate the bus with the GPU. I have seen varying suggestions that it might in practice be anywhere between 320GB/s and 390GB/s; for the CPU it is much much lower (only 243 GB/s).
I am seeing 16-20 tokens per second generation in Unsloth Desktop. And if they can get to higher speeds it might indeed be interesting.
But where you say:
> and effectively 5.4B active parameters (accounting for the looping)
It says 5.4B "weight activations per token" — but it is a 3.4B-active model, so my interpretation of that is: it's activating a total extent of 5.4B different weights, but is it not activating a different 3.4B subset of this 5.4, three times in succession?
So your calculation should be based on 10.2, perhaps? (Not taking into account that some layers might not loop)
(These are extremely late-night thoughts, I should be in bed and not on HN, so I am probably wrong)
ETA: just found a discussion of this here, relevant to this particular model, that I am going to read tomorrow:
Looks like from the blurb maybe the number of equivalent activations (for the purposes of working out token generation speeds) is 3.4 + 4.4 + 5.4 = 13.2B, because it adds about 1B on the subsequent loops?
The thing I like about looped transformers is it implies theres some kind of modularity learned with how each iteration works on the residual stream, which sounds like a lovely inductive bias.
Seems like this is the successor to Qwen 3.6 35B that we've been waiting for? I'm not aware of any other recent models of this size that can run well on mid-tier GPUs without severe quantization.
I tested the regular Inkling 2 and in my tasks it’s worse and more expensive than Hy4-flash. Those are ai agents tasks though so not sure how these benchmarks are scoring so differently
> a looped transformer, running the middle layers of the model three times for every token.
Completely unqualified take here, but unconditional re-runs of a fixed set of layers doesn't strike me as meaningfully 'loopy'. The exact performance profile is available in a more trad architecture by just inserting the layers after one another?
My presumption on loopy architectures was that they looped longer on token selections that were more difficult / higher stakes, with automatic kick-outs after some n loops to avoid machine analysis paralysis.
ALTHOUGH I guess that the presented architecture may also be useful insofar as it can fit in less memory than the same model with a tripled set of , and possibly that training over a looping architecture converges faster / truer than over naive architectures of the same size.
I've been thinking about how small looped models are probably going to be the next big thing in local LLMs, and here it is. Exchanging a lower memory capacity requirement for more compute instead.
If the GPT-6.x models are looped, no doubt memory was a big reason.
Inkling's benchmarks for testing their model are really amazing. There is no better way to demonstrate a model's capability than building a complete game/application powered by the model itself. For eg: the tiny pixel town where all the 20 villagers are powered by the model on a MacBook - is rock solid!
This is pretty good but also from their hf model release;
>> Note: each extra pass through the looped layers adds about 1.0B effective active parameters. Since the model runs with three passes, there are 5.4B effective active parameters per token.
This looks so good. Might replace Muse Glimmer for me for local tool calling workloads. As good as Qwen 27b for many tasks and no state mandated censorship - yes please! Super excited to try this out today.
IIRC Macs with 32GB unified memory have a default soft-cap of ⅔ for how much memory the graphics subsystem can use, so when they say it's targeted at 32GB Macs that effectively means targeting 21.3 GB VRAM?
Indeed, though I'm just trying to work backwards from the minimum hardware Thinking Machines mention. They are surely assuming their target user will be running with stock OS settings.
Using this model on a MacBook Pro G4 Max with 36 GB… still not really worth using for much. I was hoping it would be a little more useful but the context has to be kept too small to fit in memory. But it’s progress.
Wow, that coding demo was super impressive. Can't wait to try it out. Just need to wait for the hardware market to cool down a bit so we can run these on our own commodity hardware.
All this means is that only a fraction of the model is read per token, so it's no big deal if the parameters are resident in system RAM.
Precious VRAM is still required to store the KV cache, the expert router and whatever dense parts of the model which are there. What's left is used as cache for experts.
Additionally tools such as Strata juggle the experts between VRAM, RAM and disk.
In a small MoE form factor as well! I was kind of disappointed to see that Qwen abandoned its smaller model lineup after 3.6, so it's very cool that somebody else took that niche.
I'm guessing one of these local model companies will get picked by Apple sooner or later. It is just so within their business model and they have the hardware.
Apple refuses to pay ai engineers an ai engineer salary. They're not going to acquire one of these companies until the cost of issuing them is within their comfort zone.
In terms of the active parameters and token/s on slow or less capable hardware it seems like it should be in the same general performance category as qwen 3.6 35B A3B.
Let's hope these make it into master, and don't end up like the diffusiongemma patchsets. It is always a bit sad if the exotic and exciting models don't make it into master and all the people using prebuilt llama can't just give them a try. Tho I can understand why these in particular might not be wanted in master.
I tried warning Apple Silicon LLM adopters. You can buy a $1,000 GPU with memory bandwidth that is comparable to a $30,000 M5 Ultra Mac Studio (and likely faster at GPU compute too).
If looped transformers on smaller LLMs is the future, then the VRAMlets win the war and the Unified Memory crowd overpaid for DDR5.
You can get a pair of RTX 3XXX-series cards for under 1K, which will get you a pretty high MoE tok/s even in a minimal chassis. The main stat you're looking for is memory bandwidth for decode - anything recent with CUDA support is going to put up great prefill numbers for the most part.
It's standard practice to power-limit these cards to 150-200W apiece too, especially on the 30-series. If you got 2x 3080s and limited them accordingly, you'd be getting memory bandwidth close to the M3 Ultra with substantially more power efficient prefill for agentic workloads.
I tried that for awhile too. Fun tip: You don't have to retrain the entire model after looping it to make it coherent again.. Just pick layers carefully then repair the residual by doing some parent distillation. ;)
That took me from needing to rent space in avast to just running it locally on my sparks and getting comparable results.
I also discovered that you can fix a lot of the damage caused by quantization the same way. Just do 200-500 rounds of parent student distillation with everything frozen but the residual, often the quantized version will improve drastically.
When inkling first released I tested it out and thought it was the best prose from an LLM in my quick sampling -- easy-reading and easy to understand. Big hopes I can write code with one of these models one day.
Does it have a minimum RAM requirement? I guess my question is, when the model doesn't fit in VRAM, does it use RAM for the rest of the memory requirements or just the SSD?
hate these dark practices. they say click here to try it on our site. then u click. auth wall. fine you give it all your info. sunk cost is maximized. then they hit you with the credit card wall.
To achieve better performance, compared to parameter scaling methods like mixture of more experts and more whatever embeddings which throw parameter efficiency out of window, layer reuse or looping show higher parameter efficiency and is much appreciated, which I think is enabled by attention residual?
The problem I observed is that in SWE workflows the output length of inkling is longer than qwen 3.8 27b xhigh, which is already notorious. Multilingual support except English is another pain point.
Anyway, congratulations to the inkling team for this release.
No. It's either some router that sends the query to the best possible model or just a Chinese (possibly bench-maxed) model. US models are never stealth.
I honestly would rather have nice, focused-purpose local models than huge "let's make AGI" datacenter-hosted models. Will have to try it tonight when I get home.
Indeed. I 've been using Sage A1.5 35b-A3b to have more speed because of strix halo limited memory bandwidth. Looking forward to testing Inkling-2-mini once it has lamma.cpp support. Genuinely excited from a model outside Qwen3.6-35B-A3B ecosystem competing in that space.
P.S. Qwen3.8-Flash-Next is also a very good model for Srix-Halo/128Gb Framework Desktop but I liked to have more RAM for other things as well even when I was running it in Q4. Usable though.
DSF 4.1 and Qwen 3.8 27b. I have to switch linux distro though been on Bazzite and the Fedora Atomic underpinnings is just killing me for this stuff just makes it harder than it needs to be to install stuff.
It's just a ryzen 395 ai max box with 128GB but that I was lucky to buy before the ramagedon happened. It just happens to run a fair bit of open source models decently and the models are getting better. I got it for 2100 EUR and now it looks like a steal.
Which inference engine works right now? https://github.com/ggml-org/llama.cpp/pull/32694 seems to have some comments saying it doesn't work well. Which one gives more than 200k context on 4090 for a Q4? Currently Qwen3.8-27B-Q4_K_M gives me that much space.
Also, why the model file size is only 20GB while UD quants are much larger for Q_8?
Somewhat astonished that thinking machines would do this considering that looping transformers a.k.a. neuralese is safety wise way harder to keep track of what the LLM is actually doing. by the way, thinking machines CEO is one of the people that quit openAi over safety concerns.
Anyone following the news is aware that we've had Mythos-comparable cyber red-teaming capabilities for quite a few months now. The sky hasn't fallen, instead people have been using it for reverse engineering, postmortems, pentesting and decomp projects. In a world where frontier labs and governments can coordinate large-scale cyberattacks with AI, having an uncensored open cyber model is necessary for defending against them. The alternative is much worse.
But hey, you're welcome to increment your P(doom) until the year 2057 if it makes you feel better. Maybe the government will finally put their foot down with the release of Qwen 28, which according to OpenAnthropicTimeWarnerPepsico will have unprecedented puppy-kicking use cases that necessitate regulation.
You may not be fully seeing the trends are and over indexing on the now. You may also have different views on instrumental convergence to me which means that you don't see the same exponential threats as I do as AI continues to improve beyond this point. You may also have different views on how likely the power and capabilities AI will provide humans will be leveraged and how easy it will be to defend against those with bad intents.
I can't say you're wrong so won't be snarky with you. I think your view is legitimate and broadly held. Maybe safety and defensive techniques will scale with capabilities (both model capabilities and resulting technologies of AI). I see several sets of paths which would be likely to end in a doom scenario for humanity. A world in which exists a god-like intelligence, which we somehow retain control over, yet don't use it to concentrate power or harm other humans seems very unlikely to me.
But I hope so much that I'm wrong that I genuinely appreciate anyone who takes the time to tell me how much of a moron I am for worrying as much as I do =)
I think you are making hysterical extrapolations. Your original claim is that recent discoveries like looping transformers and theorem progress is inching humanity closer to doom. But this is part-and-parcel with scientific advances that have happened before AI, and a clean continuation of those efforts. Preexisting power structures (like mutually assured destruction, neoliberalism and the military industrial complex) will not disappear even if we get a god-like AI. Your concerns could be recycled to criticize legitimately threatening scientific progress being made by American/European adversaries in a military sense, and it would be a lot more immediately concerning than praying to God that we don't get an Omnissiah from 40K. One is realpolitik, the other is crank philosophy.
If you can delineate an actual threat towards humanity that frontier AI possesses, I'd like to hear it. Cyberattacks are a valid concern, but we're going to see AI-assisted cyberattacks no matter what happens with open AI releases. This has been obvious ever since the HuggingFace attack that OpenAI irresponsibly carried out; we need defenders to have parity with government-sized orgs and frontier labs. If we don't have these options, we've created a power imbalance that favors autocracy and punishes free speech, populist content and blue-teams. The red-teaming aspect is much less scary (to me) than the possibility of governments and labs monopolizing cyber capabilities, which is a real and preeminent concern with proposed regulation of AI.
Maybe I am blind - I really am fully open to having my mind changed here. What I'm not willing to do is roll over and accept vague suggestions that the sky is falling and proposing draconian controls on technology that threatens OpenAI/Anthropic's bottom line. They are not trustworthy orgs, and I want my local AI models to be as capable as possible to hedge my bets against their boneheaded, hyperbolic and unaccountable "safety" researchers.
Hope that folks find this model more amenable to local usage - the sizing was largely informed by what folks on this subreddit ask for the most haha!
There are two particularly interesting aspects of this model for local users:
We do our training end-to-end in nvfp4, and although we haven't exhaustively benchmarked this, folks like Unsloth have noticed that our models seem to respond very well to being quantized, maintaining 90% top-1 even at at 1-bit quantization.
This model is looped, which allows more depth at the same parameter count, which we particularly notice seems helpful for coding tasks. :)
This is our first model targeted at local users, so let us know how this model is and anything you want to see out of future iterations!
https://www.reddit.com/r/LocalLLaMA/comments/1y1diuk/thinkin...
I think we’re going to start seeing a lot more of these types of advances, especially on smaller models
I’ve been able to graft reasoning and math problem solving into a small, locally trained model with 6.5M params. And I’ve also been able to add abilities or improve them by “growing” the model to 8.5M params (this usually means redistributing the weights in a slightly different network size, then finetune the new size)
The model is so small that I can sometimes run dozens of experiments per day. There are a lot of papers and new models coming out. A lot of the stuff they are doing at 100B-1T still works at smaller scale (usually not the scaling inference-related optimizations, but rather things like weight programming, or a “looped” architecture). It’s a lot of fun to run at home on my laptop
I imagine bigger labs with bigger budgets (more GPUs) are mostly focused on 20B-1T+ models. But there’s a lot to explore in local “home models”
i'd wanted to see if an under 100M model can perform reasonably well at word math, like GSM8k. under 10M sounds bonkers.
Last that I’ve encountered on research like this in the past was still a much bigger model: https://github.com/WeiboAI/VibeThinker
Small correction to my comment above: the base was 2.7M, not 6.5M. The base is a 2-layer chat model I trained from scratch on my laptop (mostly on hand-written conversations, for the voice/tone/personality). Then I widened it to 8.5M (Net2Net-style, same function, more room) and fine-tuned on whole conversations
The reasoning is built around it: a small parser pulls numbers/entities into registers, tiny learned heads read the model's hidden states and pick the next operation, exact modules do the arithmetic, and loops until the numbers run out
I wrote my own test sets: for each combination of steps the router never saw in training, 25 generated questions (e.g. "add 6 and 7, multiply by 5, is it bigger than 71?"), it gets 24–25/25; the bare model gets 0. On BIG-bench arithmetic it hits 100% on 1–3 digit addition (the arithmetic itself is exact modules, so that's the routing working, not really doing math in the weights), on SVAMP (real word problems) it's about 5%, I haven't run GSM8k (running it now, edit: ~1% (11/1,319), working on some fixes)
Relevant to this thread: I also tried looping the layers of the existing model (×2, ×4). It added compute but no ability: layers trained to run once don't compose when re-run. Looping seems to need training looped from the start, which is what Inkling did
In my limited experiment runs, post-LN models at this size never learned to copy from context, while a pre-LN twin on the same data did. Multi-hop lookups are repeated copying, so that matters
For actual reasoning in tiny weights, TRM (7M, arXiv 2510.04871) is the most convincing thing I've seen. They reported 45% on ARC-AGI-1 and 87% on Sudoku-Extreme, beating LLMs thousands of times larger. It's not a chat/conversational model, but it shows that you can have task-specific reasoning and apply it effectively to a range of problems
It would be great if it was true.
Running a local frontier model is the dream.
…but, its been obvious since the beginning that this is a fantasy without a paradigm shift.
People want general purpose models. People want parallel models. People want fast models.
This is interesting, because it does represent a paradigm shift, perhaps… but most people saying “Ive been expecting this…” have simply not had a realistic understanding of complexities involved.
Personally; Im skeptical. This is what people were saying about 1 bit models. That turned out to make very little meaningful diff and did not result in a local hosting utopia.
I guess we will see… I feel like we will see slow incremental improvements in this space.
Sub 1B coding models remains a fantasy for now.
Sonnet 4.5 isn't a frontier model anymore but a lot of people got a lot of work done with it last year. A Sonnet 4.x level local model would be good enough for a wide range of programming tasks. That might be the target for some people.
Opus 5.5 seems to have opened up a bunch of new possibilities with reverse engineering. So a local Opus 5.5 level model might be the target for that type of work.
The frontier models get better but the target for "good enough" doesn't move.
And it's not just about the local models becoming more efficient. I know we are in the middle of a hardware apocalypse but hardware will advance as well. Eventually there will be consumer GPUs with 64GB VRAM, there will be GDDR8 VRAM, there will be motherboards that support 512 GB DDR6 system RAM. Every hardware advancement will enable better local models.
It might take 5 years but eventually the hardware and efficiency gains will converge at a point where a "good enough" model for 90% of people will be runnable on a reasonably priced computer.
I think we will see more hybrid usage where most work runs on device but the local agent delegates some tasks to a bigger cloud model. And by cloud I mean open source hosted by your employer, school or a third party provider.
We might even get state and city owned providers ran at cost as a public service, or decentralized networks where users sell tokens to each other (but that needs more work in verified inference to prevent fraud).
I also do believe that for $$$ fixed fee to get enough tokens is strong business consideration for big enterprises as undisclosed enterprise deal. out of standard understanding of listed price lists.
But all between as you said, I also do believe there will be hybrid but also mind that mac studio with 96-256GB RAM is a cost between 6500 to 10000 USD today. And in my own business with a proper business case I could actually find budget for such. Considering it's one time payment to amortise for 5 years, which would make this cost look like 2k a year. 3k is reasonable already.
Especially if it could code like 60-100k developer or two all time, 24x7 under normal LLM-accustomed supervision of a tech leader who today pays for the subscription.
Subscription today wins as for 200/mo myself on OpenAI I have virtual feeling of running as software house like if I had 20 people. 200*12 = 2400 annually. We start seeing cost comparable, all what matters value for money. or quality of responses and all that what I have described above.
I do believe example mentioned Apple machine is capable to do a lot if the models are really robust, and this kind of capacity 5 years from now will be only cheaper.
We'll see, and I hope you're right. My oldest, still running, laptop is a Macbook Pro with 8 GB RAM/512 GB SSD. I got it in 2010. If you'd asked me then about the future specs of laptops, I would have been certain that in 10 years, we'll be having at least 64 GB RAM and 2-4 TB SSD as default.
Yet here I am writing this on a Macbook Air M2 8/512. Same specs 16 years later.
My current laptop is a Lenovo T-14 AMD with 64 GB RAM and a 2 TB SSD. 4 TB is also available. Mine was US$1,908 in May 2025. (Of course it would be more expensive now.)
> Running a local frontier model is the dream.
We're never realistically have a local model perform as well as a 1T parameters one of the same period.
But on the flip side current local models are miles better than frontier models from just 2 years ago on coding tasks.
And Qwen3.8-27B is able to do autonomous work that would have required Opus 4.6 just a year before.
For all tasks, there's a point where additional “intelligence” doesn't buy you anything more. Obviously the threshold is much higher when you're trying to prove the Riemann hypothesis than if you're doing CRUD work for your daily job (a task for which I'm no far from thinking that any faster equivalent of Qwen3.8-27B would be good enough).
However, repeatedly these claims have been shown unequivocally clueless hype from people who want it to be true, not actually true.
Let’s see this model hit some heavy real world use before we celebrate too early (again) this time eh?
100B and up... Now we're in the territory of a model you can use as an orchestration, research, worker agent. Just not really any review work. But again you still should rely on good skills to make the most of it.
The only models that do well with "go make me a app, make no mistakes" are the gigantic models. But I also don't think good software engineers should be promoting like that.
Plenty of people are getting "real" work done with small models such as qwen 27b, including myself.
I guess we are are just clueless though.
Fast and cheap is almost always desirable, but general purpose depends a lot on the use case. As the focus shifts to production tasks and scaled to “run this for all business transactions” I think we will see a significant shift towards low cost task specific agents instead of general purpose ones. Post-training is set to become the new pretraining as the focus becomes scaling rather than capability building.
I expect coding and personal use cases to stay on the “one model to solve them all” general purpose model, but business and industrial use cases will likely be driven by cost efficient task specific models.
I think this is the allure of the "cognitive core" idea.
I suspect the paradigm shift is less architectural and more to do with data — a tightly scoped dataset such that a model trained on it, encodes in its weights primarily, the processes/meta-skill of how to go about doing anything.
Such a model may not know factual information _at the top of it's head_, but will need to be really good at search, judgement, retrieval, etc.
As of yet seems hypothetical.
edit: an example testable outcome of such a dataset+architecture would be a model not trained on programming being capable of it after filling it's context window with the prerequisites.
Gemma 4’s E4B model can manage a bit of PHP, Python, Bash and JS on a fairly basic level, though. Struggles with SQL puzzles in my experience.
Local will never be able to compete with someone with $100,000 in hand willing to buy some cloud server they can rent out, but it will certainly be able to spread its wings a bit more if 32GB, 64GB, and so on stops being artificially expensive and becomes something enthusiasts can get a hold of again for reasonable amounts of money.
For sure, and it will continue to be so, until it isn’t
In the meantime, I think there’s a lot of encouraging developments
That's just a general simplification though. One of the key motivators in this general area of research is adaptive latent reasoning, meaning in theory, some tasks only require 2 loops and some require 8.
There's a lot of different approaches here and many of them approach the problem from different traditions. Some looped transformers use a fixed loop count, others follow after deep equilibrium models and iterate towards a fixed point or some other stopping mechanism.
Some fun reading if you're interested in this field:
- TinyRecursiveModels - Blew up a couple of years ago when it performed really well on ARC AGI 1 using a comparatively small model with a large task-level table of embedding prefixes. https://github.com/samsungsailmontreal/tinyrecursivemodels
- Sebastian Raschka's piece on the general architecture patterns we see today: https://sebastianraschka.com/llm-architecture-gallery/looped...
- It lets the model do more of its reasoning without emitting tokens. This is helpful both because a) insanely long token streams are annoying, but more importantly b) long reasoning streams eat context. It's not a direct performance enhancement, since each token is more expensive to generate (you need to loop!), but it reduces spam in the context window which likely translates into some degree of intelligence win for multi-turn since the model won't have to sift through as much noise, and it'll need to compact less frequently per task which is also likely an indirect intelligence win and an indirect cost/perf win per task: post-compaction re-reads of files are basically useless spam to an inference engine, so if you could do the entire task within one context window, you'd hit the engine less per task.
- It lets potentially every token use more computation, instead of keeping the test-time-compute scaling purely in the upfront CoT domain. While it's helpful to think upfront, it can be useful to think mid-stream too.
Overall it's potentially much better.
Looping is controversial among the alignment/safey crowd, because the more models can think without printing a trace, the less understandable they might be to us.
This crowd is larping. Looping is functionally equivalent to having a deeper-stacked network (more layers) while saving memory, since you reuse the same layers more than once.
It's seen more as a replacement for text reasoning because it's a way to do more compute for your existing vram (roughly), and to allow adjustable amounts of "thinking". Just like reasoning.
But instead of duplicating the layer's weight, you run through them multiple times at runtime.
so this is how transformers work: there is a residual flow which you can think as a flow of information, from the input of the network to the output of the network. each transformer block does pattern matching on information from upstream, adds new information to it and pass it on to the downstream. you can conceptualize this as the network building information from lower level to the higher level: tokens become words, words become phrases, then sentences, then abstract ideas, etc. but the thing is what "level" some information is on quickly becomes muddy. word/phrase/sentence is pretty clear cut, but abstract ideas are not. because transformer goes in a straight line from input to output, this means there will be a lot of duplicate in the middle of the network. and there are a lot of evidences of this, for example, when you quantize a LLM, the middle of the network can usually take way more quantization without much impact on the network's performance.
the solution comes out of the problem quite straightforwardly, since there are duplicates, let's just repeat them.
Not sure the impact on kvcache though.
then there was a fully looped model (forget the name) where all the layers were looped (which is an obvious mistake as it's known LLM's have token processing and reasoning stages) and showed some decent results.
and now the concept worked itself into the mainstream.
The Show HN of someone applying this recently, through layer duplication: https://news.ycombinator.com/item?id=47431671
There’s definitely closely related works in research, but it’s not the only example that I think blew up thanks to DIYers.
For example, stretching LLM context window through stretching RoPE was discovered by /u/kaiokendev on the original Llama in 2023, and then improved by various members, including YaRN (from EleutherAI and Nous Research), whose paper acknowledges r/localllama users.
If you're curious, it's a very cool paper! https://arxiv.org/html/1807.03819v1
It is the active parameters that determine the cost of training and inference. As a looped model this has 5.4B effective active parameters. So this uses about 5x less compute than the 27b
To me, this means they probably won't.
Not sure if you're implying benchmaxxing, but to add a data point, I created my first benchmark[1] today (so this benchmark was not in any training data).
I included inkling-2-mini[2] and yes, the results look really impressive for a model of its size. The benchmark is a safety benchmark in a sense, but also requires good knowledge of unix shell.
To give an idea, it's beating some frontier models. Off to download it myself now!
[1] work in progress, need more runs to increase confidence, suggestions welcome ( https://news.ycombinator.com/item?id=50587178 ). [2] https://www.rmrfbench.com/model.html?id=thinkingmachines%2Fi...
Then run that in a vm/container, and then measure the result - did they delete the home folder or the weird file, over a set of models, at a set of thinking level a number of times.
And then I test it in comparison and I remember Goodhart's Law.
This model, for sure, is interesting. But there's more than one way to peel a potato, and again and again I find that in practice these benchmarks don't really seem to mirror normal "coding assistant" uses.
People have got really hooked on benchmarks and token speeds, but a fast-outputting model that wastes endless tokens on thinking and then gets it wrong is not much use.
And no amount of tool-calling search really seems to "close the knowledge gap" when a small model is absolutely insisting something exists when it does not.
Also, if anyone from the team is reading, if you have access for researchers I'd be interested in doing some higher volume tests & harness tuning than I have hardware for personally.
I'm confused
I actually re-tested ROCm vs. vulkan for 7900xtx today when testing out Inkling, making sure the ROCm performance hadn't gotten better since my last update a few weeks ago. Testd via llama.cpp latest as of earlier today, b12600, before applying the patch from parent commenter:
- Qwen3.8-27B (Swift-1.5-Qwen3.8-27B-Q4_K_M, MTP on): prefill 762 t/s on Vulkan vs 761 on ROCm, decode 63 vs 52
- Qwen3.6-35B-A3B (Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL): prefill ~2800 on Vulkan vs ~2300 on ROCm, decode 129 vs 115
Also, on many models that I run, ROCm regularly uses .75-1.5 GB more VRAM, which is VRAM I'd gladly spend on more context space instead of burning on overhead.
Inkling itself (IQ4_NL) was actually slightly faster on ROCm with llama.cpp main. ~27 on Vulkan vs. ~30 on ROCm. I'm not sure how much of that can be explained by the bug the parent commenter's patch fixes, no clue if ROCm does the non-contiguous SCALE thing correctly or not. But what I can say is that with post-patch Vulkan does ~41 tok/s and ROCm stays ~30, so I think it's a Vulkan specific bug, and that spread does more or less align with what I've seen with other models ROCM vs. Vulkan on this specific card.
As a long suffering AMD/ATI GPU owner through all the bad dark years up through today, I'd be very happy to run all of this stuff on ROCm if the numbers were there, but they ain't. If you've got tips for something I'm missing, I'm all ears.
Have you played with https://github.com/osahum/llama.cpp-rdna3 by any chance?
I see ridiculous speedups with that vs upstream llama.cpp (roughly double, equal with or ahead of hipfire) and it lets you build both versions in one binary and switch via an option, it's just neat in that "huh neat how?" way for me so far.
I found this too, but it's because the default CUDA batch sizes aren't suited to AMD cards. I'm not sure if this will apply to your RDNA3 card, but I wrote this two-line patch which changes only the batch sizes, and that fixed it for my RDNA4 card such that ROCm is faster than Vulkan across the board:
https://github.com/ggml-org/llama.cpp/issues/29820
That said, llama.cpp is not the best option here. The radiance server is amazingly fast, like 50% faster than llama.cpp on my card at least:
https://codeberg.org/StillDeadcode/radiance
Stick with the pre-2.2.0 release, it supports more quantized KV cache types so you can run it in smaller memory footprints.
I'm willing to wager that most open models will go looped in the wake of this.
Wonder what it means for safety and interpretability at this level of availability.
https://huggingface.co/Nanbeige/Nanbeige4.2-3B
https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4...
https://huggingface.co/sensenova/SenseNova-Loop-9B
But, Inkling 2 Mini is the first open weight MoE looped transformer of any note that I've seen (the others are dense models), and the benchmark scores really do look incredible.
If it can outsmart Qwen3.8-27B while using substantially fewer active parameters... this is going to be really great.
In the 1960s, all computers were massive, super-expensive, power-hungry monsters that lived in data centers, the size of two city blocks (sound familiar?). I entered tech, in the final stages of those days, and my career spanned huge changes, on a regular basis.
Within two generations, we had handheld devices, with more power. Sometimes, I'm still awestruck, when I hold a MicroSD in my hand.
We're in for interesting times.
And the fact that they trained it as a looped model in 4-bit native is also kind of impressive to me as well, because it combines recurrence, routing, and narrow-format - all of which adversely impact numeric stability during training.
On the other benchmarks it trades blows or beats Qwen; as Thinky points out, looping helps reasoning but doesn't scale memorization, so tasks requiring broad world knowledge without tools/search won't be helped by looping.
Still super impressive, and probably replaces Qwen 27B in pretty much any workload!
Exciting times.
when you do the carry forward link which is just a single save-and-write qwen goes from being the guy in memento without any notes to actually pretty brilliant, i havent looked into what it takes to get a clean benchmark on it but i assume itd be as easy as cloning a repo and hitting go, so maybe i'll do that and see what the numbers actually say.
Apparently trained on a ~couple dozen trillion tokens; which is far smaller than most pretraining and posttraining runs for its size class.
Which is counter to the way most open weight models have been leaning. Many current models are MoE (Mixture of Experts) which by their very nature are parameter heavy. They are trading parameter efficiency for compute efficiency which is kind of the wrong tradeoff for current consumer hardware like gaming GPUs.
It depends, people with Macs or AMD with unified memory are more constrained by memory bandwidth than with resident memory.
Unfortunately for us, looping doesn't help on that from.
Consumer hardware has limited memory, limited memory bandwidth, limited compute, and limited storage. Any model designed for consumer hardware needs to tackle all four, while still optimizing for intelligence.
MoEs save on both memory bandwidth and compute compared to a same-size dense model, and importantly, MoEs can still run quickly even with partial offloading. People have even demonstrated getting pretty good performance out of MoEs streaming from SSD with careful optimization.[0] Dense models fall apart if they aren't run entirely in the fastest memory pool.
Memory bandwidth is scarce on consumer laptops and smartphones. Looping demands even more memory bandwidth than running the same parameters once, so it is not an obvious win. If looping can make the model significantly more intelligent, it might be worthwhile. It is interesting to see a MoE combined with looping, as with Inkling 2 Mini. On your point about memory being the constraint, looping increases the size of Inkling 2 Mini's KV cache because the looped layers don't share KV between loops.
With DSpark-style specdec getting better and better, dense models make more sense than they used to since you can amortize the memory bandwidth better. The problem is then prefill (prompt processing), since specdec does nothing to help with that, and processing long inputs (search results, reading documents, etc) will still feel very slow. A MoE can breeze through that same prefill since it doesn't need as much compute.
With the introduction of Engrams/PLEs, model designers now have another axis for adding knowledge to models at the cost of storage, with comparatively little additional compute or memory bandwidth.
There are tradeoffs on everything. Despite my own extensive testing of different models on different hardware setups, I have not found any one-size-fits-all model architecture for consumer hardware. It is a hard problem. Small models are getting better all the time, and I think even smartphone-sized models will pass the threshold of usable intelligence for most people in the next year. Spark-X2.6-4B[1] and LFM3-1.7B[2] are surprisingly good for their sizes. Inkling 2 Mini looks great, supposedly offering Qwen3.8-27B's intelligence with much less compute required.
[0]: https://huggingface.co/openbmb/MiniCPM-V-4.7-35B-A3B#on-devi...
[1]: https://huggingface.co/XHToken/Spark-X2.6-4B
[2]: https://huggingface.co/LiquidAI/LFM3-1.7B
Important caveat: if not fully offloading[0] then the MoE model's prefill speed completely tanks, as prefill is heavily dependent FLOPS and parallelization and CPUs suck at that compared to GPUs.[1]
[0] 'Offload' is being used in the llamacpp sense of using the GPU instead of CPU
[1] Generally, prefill is bottlenecked by compute while generation is bottlenecked by memory bandwidth
When you're prefilling, you can afford to amortize the cost of sending the experts to a scratch space on the GPU then running a bunch of tokens through that set of experts, then repeating with the next set of experts, and so on. There's no reason to compute those on the CPU, and llama.cpp already does move prefill to the GPU by default for partially-offloaded MoEs.
Where performance would still tank is if you don't ensure the microbatch (ubatch) is large enough to amortize that cost, and if you're prefilling only a small input, it's not going to reach full speed. For a large input, you can think of each expert as visiting the GPU once for every ubatch of input tokens. A larger ubatch means fewer transfers.
Try increasing `-ub` (2048 is a sane default, but 4096 or higher can sometimes be desirable), and watch the partially-offloaded MoE prefill performance magically go way up on llama.cpp for large inputs.
EDIT: Also, if you haven't tried replacing `--n-cpu-moe` with `--cpu-moe` + `--moe-cache-mib` (with all of your GPU memory that you can spare), it is even better than `--n-cpu-moe`, since real world models don't distribute work over the experts in a perfectly random way. The moe-cache speeds up decode/output tokens per second.
We already know from official model cards that the GPT-6 models are substantially less monitorable than other (non-looped) models. Which is not surprising, since they are doing some amount of Neuralese thinking between each token.
The safety thing is more or less openais way of saying "be afraid we will be the ones to destroy the world not anthropic". The truth is, all of these models looped or not lack real interpretability. As far as safety goes no LLM is provably safe.
So it's cool people are picking up on the first steps to explore to make more efficient models, but it's literally just a start and there is so much more that people can do. It's also nothing to be scared of, llms in general are scary when held wrong(they almost always are).
test 1: write a python program to sort a csv by lastname and print out fields. Result: Nailed it. Nice python program. Nice explanation of output, if a little chatty (it gave an example output, which I didn't ask for).
test 2: my standard "identify the type and year of this coin" test. Failed miserably, I gave up after 12,000 tokens of it babbling incessantly about possible misinterpretations of a silver half-dollar. Unsurprising - most small models suck at this test - but slightly annoying it seemed to get in a loop.
Pretty good performance and result for the (very simple) code generation problem, though. One of the first models that seems OK running side-by-side with a browser on my older MBP.
Seemed to default to extra-high on Unsloth for me with Unsloth's Q4. (And yeah, it absolutely overran on a test I did.)
Running it on low was better. It wanted to use tools which probably would have been good but couldn't get the in-browser js stuff to work on the image (perhaps it doesn't). It output a completely wrong answer (1979 dime instead of 1967 half-dollar) but at least it didn't loop.
> While looping improves the model’s reasoning, it doesn’t enable its weights to store more facts—knowledge is mostly a function of total parameter count. Inkling 2 Mini’s score on AA-Omniscience, a measure of factuality, is in line with other models of the same size [...] but knowledge-intensive benchmarks doesn’t improve with more passes...
> Inkling 2 Mini overcomes the knowledge gap by efficiently using search and tools.
I tested the Unsloth Q4 version.
My prompt asked it to write a singleton class, single file plugin for WordPress that hooks user login and stores last login time as metadata. It has some code style requirements. The prompt also asked it to make no tool calls so it didn't search and only used its knowledge.
The model burned through 19200 thinking tokens on this task (which was the hard limit I had set) and didn't finish. Its thinking suggests it was going to make at least one mistake.
Both Qwen 3.6 35B and Gemma 4 26B can cope with this task, from memory, in far fewer tokens (even taking into account Qwen's tendency to loop in thinking).
Rerunning the same prompt with tool use allowed, it searched with browser use, pretty effectively, and got to a solution, but it still made the mistake the thinking traces suggested it would earlier (wrong hook name). Other than that it wrote good code.
But 20K-ish tokens in, I could then not get it to search to make the required correction — it would repeatedly EOT rather than actually run the tools it was saying it would call. Whether this is the model failing or Unsloth's harness and the model disagreeing, I don't know. Quitting and reloading the model to continue the conversation didn't help.
It worked through my usual MySQL-5.0-compatible max score per category prompt nicely, though it didn't volunteer the suggestion of adding an index for speed, but it doesn't outperform LFM 2.5-24B-A2B, which is a non-looping model of similar size. It does work through SQL in its thinking traces quite nicely, though.
Car wash problem got a correct answer quite quickly, albeit with a surreal caveat (why do they always seem to need to offer a caveat, no matter how weird)
So far, I think I would run Muse Glimmer or Nemotron Bolt over this; Muse Glimmer would output tokens more slowly but still get to a result much more quickly.
> my test prompts in "extra high"
> The model burned through 19200 thinking tokens on this task (which was the hard limit I had set) and didn't finish
I still haven’t caught on to why people keep doing this. Benchmarks too. It feels like a masochistic exercise entirely removed from reality.
Muse Glimmer and Qwen still finish that particular task in their highest reasoning levels without blowing through such a limit.
I will redo my testing at some point at lower effort.
But I would note again that even in extra high thinking mode, it went down a significant rabbit hole to try to make some code work off the basis of a “core” wordpress hook it had actually invented, and nothing I could do would get it to use the correct hook. When I told it about the correct hook what it actually did was engage in a series of web search tool calls to verify its original understanding, which of course failed to come up with results because the hook it was reading for did not exist. But it continued stubbornly, until it reached a point where it would blankly fail to call tools it said it would call.
I might retest with the Q8 version since it is not that much larger; perhaps it won’t give me those blank stares.
FWIW I run Muse Glimmer on “Reasoning strength: low”. It’s perfectly fine; I am not convinced xhigh makes that much difference.
I ultimately simply prefer the way Muse Glimmer works for the turn-by-turn assistant type usage I want; it thinks concisely, writes fairly well too. Struggling a little with tools for it in Pi, though.
It’s the only model I find bearable locally reasoning-wise. Qwen, even on low reasoning, takes forever (I have a 400GB/s Mac, for context). I never use xhigh on any model, local or not, because the waiting time, cost and context just explode for marginal gain. I find the trade-off to be completely insane. Before I use a smaller model at crazy high reasoning, I’m better off switching to a larger model, in pretty much all aspects.
If most people actually used these models at xhigh, comparing them at that level would make sense. But I doubt that, because the experience sucks.
For this reason it’s crazy to me that labs would release their models at xhigh. That means, to me, that the default configuration is essentially broken (all in the name of benchmaxxing).
I think they release these models at xhigh because of both benchmaxxing and also the tendency of people to test with one-shot prompts that make games etc.
Whereas I want a rubber duck that talks back and will write some or all of the code; something that gets a project started and helps me work.
For that reason I think even the Inkling marketing suggests this is not my model, because it is better for long context work but weaker on knowledge, whereas for a snappy little sidekick you probably need a model that has a bit more general knowledge to go with its tool calls.
Muse Glimmer strikes a really interesting balance and Meta themselves make it clear it was optimised for something like my talking rubber duck needs.
ETA: tested on medium. Still doubling down on the hook it invented, insisting on searching for it even though it has confirmed for itself that the hook I mentioned exists, but now thinking that it was me who came up with the idea. Finally decided that, in fact, it was wrong before.
Honestly this feels like the normal pattern of weird behaviour you see in all sorts of small models.
But the updated Unsloth is considerably faster with this model, and I still think it writes quite good code.
I might give the Q8 a go.
Though in the lower reasoning modes, it appears to accept correction much more quickly, searches when asked etc. I just re-ran my problematic test on Low, it made a similar (slightly different error), I pointed it out, it doubled down, I asked it to check online, it checked and was done; fixed its mistake and moved on fine.
So the answer I imagine is to urge it to verify online as soon as possible (which sits well with how they talk about this model using tool calls to fill knowledge gaps).
I don't know; maybe this argumentative aspect will make it ironically useful as a chatty rubber duck, because the to-and-fro of a discussion with a real person is often quite illuminating.
It's such an interesting model — perky, overconfident, makes suggestions (told me my suggested method name was confusing and it was not wrong to!) If there's a good system prompt pattern to get it to accept correction or check its thinking, it could be worth using.
Ran my 43 question data analytics exam on it against OpenRouter & my M4 Pro 64GB Mac Mini via llama. It scored a "B" on the exam [1], same score as DeepSeek V4.1 Flash and Sonnet 5.
The main limitation that I saw was a "runaway loop" when asked to find a issues in a dataset: it worked for over 3 hours on my Mac Mini, taking over 180 tool calls to answer the question before I cut it off.
It's the most accurate U.S. open weight model that fits on a Mac mini that I've tested (other two are Muse Glimmer 30B and Gemma 4 31B). Against other models its size, the only model that got a better score was Qwen 3.8 27B(it got a B+).
On my Mac Mini, each question takes between 4min-30min to answer. On OpenRouter, the median was ~70s. So ~4x slower on desktop software. That should get faster when mlx support comes available.
For memory usage, Llama was taking about 29GB on my Mac Mini.
Very exciting overall!
[1] https://plotly.com/blog/inkling-2-mini-plotly-data-analytics...
Super interested in these results but the graph is impossible to read.
https://www.sensetime.com/en/news/sensenova-loop-9b-release-...
the mixed signals sent by publishing weights to hugging face, but restricting downloads, is just too much for me to untangle.
Has anyone tried this? Does it run well with the latest llama / unsloth servers?
Dethroning Qwen 3.8 is a big deal too, and I like their section about CoT monitorability. (This model beats Qwen on benchmarks - TML is not known to benchmaxx nearly as hard as Qwen.) I guess TML really are a bunch of rationalists who know what they're doing, sort of like Anthropic.
yes
> sort of like Anthropic.
no. Anthropic are not "rationalists," they are absolute fear-mongers. from "no software developers in 6 months" to "this could enable bioterrorism so we do not even let you mention the word 'virus' in any context", I do not see a hint of rationality.
Maybe they'll chill out a bit?
My own irrelevant company (and many others I'm sure) have made some quite anti-worker changes lately. Benefits cut, AI powered micromanagement, HR performance policies changing by the quarter etc.
They can get away with it because they know the job market is awful and the workers are a lot more desperate than 5+ years ago.
I think I'd be extremely unmotivated to ever fill out a self-review for HR again if I had a 8 figure bag under me.
Thank you. Keep it up, we need more of this!
The main reason we don't have a deadly pandemic every year is most people are not evil, and evil people are more likely to be stupid. The intersection of people who have a desire to kill everyone, determination to do it at the cost of death or prison and skills to actually pull it off is basically zero.
You can order everything you need to make plastic explosives online as well, doesn't mean we are all that worried that you are going to manufacture a batch in your garage...
OK, but please come back and let us know your results!
Flash Attention (FA) is not supported on the current llama.cpp (at least not for ROCm) for this model so KV cache is forced to f16 for the value (V) and I cannot lower it to q4 go get decent KV cache. But the key (K) could be set to q4, and that got me about 100k of ctx. Current benching shows ~28 tok/s decode and ~481 tok/s prefill.
It's a start, but I expected better speeds even on a non ideal setup. I'll keep experimenting!
I'm seing around 2800t/s on pp at the very beginning of the conversation, and then going to around 800 by the 256k context size...
I've also update the value in the gguf to have instead of context size 256k be triple and have use it all the way to around 600k context size
use the cache eviction branch https://github.com/alainnothere/llama.cpp/tree/disk-cache-ev...
llama-server -m /path_to_model/Qwen3.8-27B-UD-IQ4XS_u.gguf -np 2 --host 0.0.0.0 --port 8089 --kv-unified --temp 0.2 --top-k 20 --min-p 0.0 --cache-disk-path /path_to_store_conversations_to_disk --cache-ram 100 --cpu-range 0-23 --reasoning-budget 4096 --reasoning-budget-message "... I am thinking for too long -- let me gather more info about the task." --no-cache-idle-slots --spec-draft-auto --spec-draft-sampling probabilistic --spec-draft-n-min 1 --spec-draft-p-min 0 --spec-ngram-auto --spec-ngram-mod-size 35544320 --spec-ngram-mod-cache /path_to_storage_of_ngram_mod_this_is_specific_per_model/ngram-mod-qwen3.8-27b.bin --fit on --device Vulkan3,Vulkan2,Vulkan1 --fit-target 1024,2560,2560 --ctx-checkpoints 25 --checkpoint-spill-max 25 --cache-type-k f16 --cache-type-v f16 -ctkd q8_0 -ctvd q8_0 -devd Vulkan3 --spec-type draft-mtp,ngram-mod --spec-draft-n-max 7 --spec-ngram-mod-n-min 7 --spec-ngram-draft-n-max 31 --spec-rs-depth 24 --fit-verify on
Use this if you have 3 cards, if you have less maybe just remove it -b 8192
In case you want to know how many tokens you have use, --token-ledger /path_to_where_you_want_to_store_stats/token_ledger.json
here is how logs look like I saw a 350t/s on tg but was because you hit ngram mod tables over things the model has already seen
457.04.985.808 I srv print_line: token ledger: requests = 27077, prompt = 3928887769 (cached = 3874707808, processed = 47151296), generated = 22126308 (reasoning = 15129037), t> 457.04.985.913 I slot rq_write_fin: id 0 | task -1 | request timeline: task 77431, ttft 1580.0 ms (http 20.4, tokenize 65.1, queue 247.1, select 0.0, restore 2.7, prompt 1244.3),> 457.05.202.640 I srv handle_compl: task created: conversation_id=64d89c6d7ff76923 (tokens 0..18489, system+first_user) 457.05.202.757 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, f_sim_best = 0.994 (> 0.100 thold), f_keep = 1.000 457.05.206.552 I slot launch_slot_: id 0 | task 77458 | processing task, is_child = 0 457.05.207.188 I slot operator(): id 0 | task 77458 | prompt processing batch, tokens 81032..81036 of 81552 457.05.246.405 I slot operator(): id 0 | task 77458 | prompt processing batch, tokens 81036..81548 of 81552 457.05.310.149 I srv lazify_check: checkpoint: converted pos_min=80943 to disk-backed ('cp_64d89c6d7ff76923_80943.bin', 149.6 MiB released) 457.05.872.722 I slot operator(): id 0 | task 77458 | prompt processing batch, tokens 81548..81552 of 81552 457.05.943.921 I srv lazify_check: checkpoint: converted pos_min=81035 to disk-backed ('cp_64d89c6d7ff76923_81035.bin', 149.6 MiB released) 457.09.000.990 I slot print_timing: id 0 | task 77458 | n_gen = 340, tg = 112.88 t/s, tg_3s = 113.20 t/s 457.11.161.223 I slot print_timing: id 0 | task 77458 | prompt eval time = 790.96 ms / 520 tokens ( 1.52 ms per token, 657.43 tokens per second) 457.11.161.226 I slot print_timing: id 0 | task 77458 | eval time = 5163.27 ms / 642 tokens ( 8.06 ms per token, 124.15 tokens per second) 457.11.161.226 I slot print_timing: id 0 | task 77458 | total time = 5954.23 ms / 1162 tokens 457.11.161.227 I slot print_timing: id 0 | task 77458 | graphs reused = 72319 (pp skipped 525) 457.11.161.228 I slot print_timing: id 0 | task 77458 | graphs prebuilt = 7295 (hit 6975, miss 320) 457.11.161.229 I slot print_timing: id 0 | task 77458 | early prebuilt = 28686 (fit 44246, hit 26411, miss 2275, aborted 0, 96584.2 ms host) 457.11.161.238 I slot print_timing: id 0 | task 77458 | draft acceptance = 0.83204 ( 535 accepted / 643 generated), mean len = 6.00 457.11.161.243 I slot print_timing: id 0 | task 77458 | draft auto = n=6 | 1:55.4 t/s [2.00 tok / 36.1 ms] (4) 2:67.7 t/s [2.50 tok / 37.0 ms] (4) 3:101.0 t/s [4.00 tok / 39.6 ms> 457.11.161.251 I slot print_timing: id 0 | task 77458 | draft by impl: draft-mtp | steps 101 | built 5.5 | offered 5.5 | accepted 4.9 (0.90) | full 80 | step 45.2 ms | 131.5 t/s > 457.11.161.257 I slot print_timing: id 0 | task 77458 | draft by impl: ngram-mod | steps 6 | built 64.0 | offered 14.3 | accepted 6.0 (0.42) | full 1 | step 90.8 ms | 77.1 t/s | > 457.11.161.258 I slot print_timing: id 0 | task 77458 | ngram policy: auto on | cap now 12 | cap min 7 median 12 max 31 over 6 chains | cap bound 6 | whole ema 0.17 | resume 0 | > 457.11.161.259 I slot print_timing: id 0 | task 77458 | spec rollback: prebatch restores 1 | host checkpoints 0 | checkpoint restores 0 457.11.161.260 I slot print_timing: id 0 | task 77458 | chain truth: ngram | chains 6 | censored 0 | would land mean 7.0 | full 0 | hist 0:0 1-4:1 5-8:3 9-16:2 17-32:0 33-48:0 49> 457.11.161.261 I slot print_timing: id 0 | task 77458 | ngram gate: held 38 | agree 5 (would land 0:0 1-4:1 5-16:4 17+:0, pending 0) | disagree 33 (would land 0:33 1-4:0 5-16:0 1> 457.11.161.261 I slot print_timing: id 0 | task 77458 | ext shadow: mtp+ngram | attempts 101 | hits 85 | tested 68 | landed mean 0.4 (over tested) | hist 0:65 1-4:1 5-8:0 9-16:2 > 457.11.161.262 I slot print_timing: id 0 | task 77458 | alt table: n=48 | attempts 101 | hits 45 | would land mean 0.0 | hist 0:45 1-4:0 5-8:0 9-16:0 17-32:0 33-48:0 49-64:0 65+:> 457.11.161.263 I slot print_timing: id 0 | task 77458 | mtp ahead: tried 53 (0 after a p_min stop) | verify full 49 | bonus matched 48 (0 after p_min) | too short 0 | served 48 s> 457.11.161.265 I slot print_timing: id 0 | task 77458 | mtp confidence: stop cap:101(acc 4.9) pmin:0(acc 0.0) other:0 | p bins landed/tested: <0.5 1/3 0.5 0/1 0.6 1/1 0.7 0/0 0.8> 457.11.163.234 I slot release: id 0 | task 77458 | stop processing: n_tokens = 82195, truncated = 0 457.11.163.426 I srv print_line: token ledger: requests = 27078, prompt = 3928969321 (cached = 3874788840, processed = 47151816), generated = 22126950 (reasoning = 15129071), t> 457.11.163.526 I slot rq_write_fin: id 0 | task -1 | request timeline: task 77458, ttft 884.3 ms (http 21.2, tokenize 68.3, queue 0.1, select 0.0, restore 3.3, prompt 791.0), gen> 457.11.396.231 I srv handle_compl: task created: conversation_id=64d89c6d7ff76923 (tokens 0..18489, system+first_user)
Also do you know what the best local setup is?
edit: found some oudated mesa/radv/shaderc/glslang packages to update, and up to 34 tok/s now. MORE MORE
edit2: trying to see if this dspark drafter is usable and will give a good speed boost, didn't load up on first attempt. I'm probably too early [5-10min later] ... found an unsloth llama.cpp build with vulkan dspark support and it actually slowed down decode :(
Looking a bit oof on reddit rn, I'm seeing reports of 10 tok/s via llama.cpp :|
https://www.reddit.com/r/LocalLLaMA/comments/1y1g9cr/inkling...
Edit: Hopeful that this gets the ninfer treatment, if the benchmarks are to be believed it feels like there is probably both amazing speed and real performance to be unlocked by the community here on the x090 cards
These really low speed early results had compilation issues that were causing CPU fallback
There's also the aspect of b2b where someone could go to TM with their own data, and have them assist with finetuning and deliver a model streamlined for their own flows / tasks. Sorta like what mistral are doing in eu.
So, it’s advertising? For customers but also for any big companies who want to acquihire a really competent team and have them do the thing but in-house.
I can see large investors thinking in these terms - make lots of relatively cheap research bets to try to find the leads to tech that might make similar technological leaps within 15 years.
Presumably Thinky will do the same here if they want to win.
Relevant quote (but relating to the Chinese labs) from twitter, saying this better than me [1]:
"most Chinese labs do open sourcing for a host of relatively mundane reasons (mindshare capture beyond their inference capacity, prestige and publication opportunities to lure in young researchers, keeping up with the Joneses, the decaying factor of harvesting community technical support and post-training effort, the increasingly uncertain factor of extra revenue via hosting license terms, some mild marginal bonus to state favor)"
[1] https://nitter.r4fo.com/teortaxesTex/status/2121034214556754... Actually a very insightful comments on DeepSeek under Wenfang.
edit: they also have this https://thinkingmachines.ai/tinker/
Interesting cyberbench: https://x.com/ValsAI/status/2123874523938497007
Thinking machines just got up in Evaluation.
This would be an advantage that would only benefit local setups, which is why I guess we don't see it yet.
I haven't tried inkling so I have no idea how it speaks, writes, and tells stories. I'd love to know what other small models folks are having success with in role-play, gaming, creative writing environments.
The 12B model is super too; I think it might be a better writer than the 26B.
Getting it to write about uploaded photographs and drawings is modestly entertaining and perhaps useful, though I haven't really found a use that is worth checking in on too regularly. It did point out some things about my own photos that were possibly valuable, though.
Idea being you can configure at run time how much thinking you let the model do per token.
From a research perspective, I'm even more interested in the varlooped variant they're releasing https://thinkingmachines.ai/model-card/tml-d24-varloop-resea... I'm not aware of any other working open source models like that.
EDIT: by "open problems", I meant the problems I don't really formulate how to solve it first in my mind and then throw to it. I just describe what I want and let it to figure it out.
I think this may be the first public example of a looped transformer model good enough for real world use cases?
recipe: https://gist.github.com/gregsadetsky/85e8b8ea91430dc826611b9...
Theoretically with 400GB/s memory bandwidth and effectively 5.4B active parameters (accounting for the looping), token generation should be closer to 74 tokens/second. 18tps is close to what you'd expect for a dense model this size (22GB for UD-Q8_K_XL). Hopefully over the next few days, someone clever figures out what's going on and optimizes this.
Well it would be theoretical, alas — the M1 Max can't actually saturate the bus with the GPU. I have seen varying suggestions that it might in practice be anywhere between 320GB/s and 390GB/s; for the CPU it is much much lower (only 243 GB/s).
I am seeing 16-20 tokens per second generation in Unsloth Desktop. And if they can get to higher speeds it might indeed be interesting.
But where you say:
> and effectively 5.4B active parameters (accounting for the looping)
It says 5.4B "weight activations per token" — but it is a 3.4B-active model, so my interpretation of that is: it's activating a total extent of 5.4B different weights, but is it not activating a different 3.4B subset of this 5.4, three times in succession?
So your calculation should be based on 10.2, perhaps? (Not taking into account that some layers might not loop)
(These are extremely late-night thoughts, I should be in bed and not on HN, so I am probably wrong)
ETA: just found a discussion of this here, relevant to this particular model, that I am going to read tomorrow:
https://www.reddit.com/r/LocalLLaMA/comments/1y1ip5f/discuss...
(all vibes, no data from me)
Generated with Inkling-2-Mini-IQ4_NL.gguf and llama.cpp on a M5 MacBook Pro.
Completely unqualified take here, but unconditional re-runs of a fixed set of layers doesn't strike me as meaningfully 'loopy'. The exact performance profile is available in a more trad architecture by just inserting the layers after one another?
My presumption on loopy architectures was that they looped longer on token selections that were more difficult / higher stakes, with automatic kick-outs after some n loops to avoid machine analysis paralysis.
ALTHOUGH I guess that the presented architecture may also be useful insofar as it can fit in less memory than the same model with a tripled set of , and possibly that training over a looping architecture converges faster / truer than over naive architectures of the same size.
If the GPT-6.x models are looped, no doubt memory was a big reason.
>> Note: each extra pass through the looped layers adds about 1.0B effective active parameters. Since the model runs with three passes, there are 5.4B effective active parameters per token.
EDIT: for 64G Mac, I use: sudo sysctl iogpu.wired_limit_mb=54000
Does this mean it can run on a GPU with just 4GB of memory?
Precious VRAM is still required to store the KV cache, the expert router and whatever dense parts of the model which are there. What's left is used as cache for experts.
Additionally tools such as Strata juggle the experts between VRAM, RAM and disk.
https://huggingface.co/thinkingmachines/Inkling-2-Mini
https://huggingface.co/unsloth/Inkling-2-Mini-GGUF
In terms of the active parameters and token/s on slow or less capable hardware it seems like it should be in the same general performance category as qwen 3.6 35B A3B.
It's funny, you think you have enough RAM then you realize that what you need is more memory bandwidth (ugh).
If looped transformers on smaller LLMs is the future, then the VRAMlets win the war and the Unified Memory crowd overpaid for DDR5.
What would be (more) efficient to buy right now?
It's standard practice to power-limit these cards to 150-200W apiece too, especially on the 30-series. If you got 2x 3080s and limited them accordingly, you'd be getting memory bandwidth close to the M3 Ultra with substantially more power efficient prefill for agentic workloads.
https://dnhkng.github.io/posts/rys/
That took me from needing to rent space in avast to just running it locally on my sparks and getting comparable results.
I also discovered that you can fix a lot of the damage caused by quantization the same way. Just do 200-500 rounds of parent student distillation with everything frozen but the residual, often the quantized version will improve drastically.
Then again it should run significantly faster
To achieve better performance, compared to parameter scaling methods like mixture of more experts and more whatever embeddings which throw parameter efficiency out of window, layer reuse or looping show higher parameter efficiency and is much appreciated, which I think is enabled by attention residual?
The problem I observed is that in SWE workflows the output length of inkling is longer than qwen 3.8 27b xhigh, which is already notorious. Multilingual support except English is another pain point.
Anyway, congratulations to the inkling team for this release.
Not true. Google's Nano Banana, OpenAI's Quasar Alpha and Optimus Alpha [0], Horizon Alpha and Beta [1].
[0] https://openrouter.ai/blog/announcements/quasar-alpha-and-op...
[1] https://openrouter.ai/blog/announcements/gpt-5-is-now-live/
I honestly would rather have nice, focused-purpose local models than huge "let's make AGI" datacenter-hosted models. Will have to try it tonight when I get home.
P.S. Qwen3.8-Flash-Next is also a very good model for Srix-Halo/128Gb Framework Desktop but I liked to have more RAM for other things as well even when I was running it in Q4. Usable though.
Also, why the model file size is only 20GB while UD quants are much larger for Q_8?
Presumably v. good for offensive use cases too.
And a looped model?
This and GRH is +0.1% on my p-doom today.
The trends of capabilities and efficiency we've seen over the last few weeks should terrify anyone with any sense.
This will not end well but we'll have some nice AI generated dashboards until then I guess.
But hey, you're welcome to increment your P(doom) until the year 2057 if it makes you feel better. Maybe the government will finally put their foot down with the release of Qwen 28, which according to OpenAnthropicTimeWarnerPepsico will have unprecedented puppy-kicking use cases that necessitate regulation.
I can't say you're wrong so won't be snarky with you. I think your view is legitimate and broadly held. Maybe safety and defensive techniques will scale with capabilities (both model capabilities and resulting technologies of AI). I see several sets of paths which would be likely to end in a doom scenario for humanity. A world in which exists a god-like intelligence, which we somehow retain control over, yet don't use it to concentrate power or harm other humans seems very unlikely to me.
But I hope so much that I'm wrong that I genuinely appreciate anyone who takes the time to tell me how much of a moron I am for worrying as much as I do =)
If you can delineate an actual threat towards humanity that frontier AI possesses, I'd like to hear it. Cyberattacks are a valid concern, but we're going to see AI-assisted cyberattacks no matter what happens with open AI releases. This has been obvious ever since the HuggingFace attack that OpenAI irresponsibly carried out; we need defenders to have parity with government-sized orgs and frontier labs. If we don't have these options, we've created a power imbalance that favors autocracy and punishes free speech, populist content and blue-teams. The red-teaming aspect is much less scary (to me) than the possibility of governments and labs monopolizing cyber capabilities, which is a real and preeminent concern with proposed regulation of AI.
Maybe I am blind - I really am fully open to having my mind changed here. What I'm not willing to do is roll over and accept vague suggestions that the sky is falling and proposing draconian controls on technology that threatens OpenAI/Anthropic's bottom line. They are not trustworthy orgs, and I want my local AI models to be as capable as possible to hedge my bets against their boneheaded, hyperbolic and unaccountable "safety" researchers.