By my own, completely arbitrary split of this into 16 predictions, I would say that this hits on maybe 8 of 16 if we are being a bit generous. Which is good, but certainly not "Inevitable" (the title of the book).
With that said, maybe he meant inevitable as N approaches infinity... So there's still time!
Aside: I think the biggest interesting prediction he had was that AI would obey a user driven network effect. It seems like a very clean prediction, and yet it has failed to materialize? I wonder if this has more to do with just how scrape-able the whole internet was or with distilation or what? There is certainly an advantage to having good user data, but it seems like it is largely something that can be worked around (by hook or crook)?
In addition to writing about current (and future) technology, Kevin Kelly also put together Vanishing Asia, a photo book documenting disappearing communities and traditions across Asia
Possibly the best photography book I've come across. It's out of print, but I managed to snag a copy on eBay last year
I would say that KK's "The bulk of AI will be special purpose software brains" has turned out closer to the truth than "The Bitter Lesson" has. LLM's do not drive cars, or vice versa. Stockfish still beats the newest, largest LLM at chess. Etc. Most of the truly productive "AI" is customized for a specific purpose, or class of purposes anyway.
For now. But the trend remains that bigger, smarter, general purpose AI beats smaller specific purpose models. This is why GPT 5 and 6 remain better at OCR than anything Mistral has put out.
That's sort of a loaded question depending on what you mean by "smaller sub expert models inside".
At the most abstract level, you can observe patterns in a model's activations and theorize that particular collections of components contribute to particular kinds of operations. There are many flavors of this intuition. In that sense, you can think of there being "submodels" within a model, insofar as different parts of the network can participate in specialized computations. Some reading that might interest you in this area would include things like mechanistic interpretability.
There are also Mixture of Expert models, which are a dominant architecture for LLMs now, in which you essentially have a bunch of smaller FFNs that get routed over/aggregated together when the LLM performs a forward pass. It would be a mistake to treat any single expert as a better task specific model though. They are trained as part of an ensemble, and they're almost always being combined with other experts in the forward pass, so they're more like (to use a loose analogy) regions of the brain that tend to correspond more towards particular actions than others. You couldn't just extract one and say "This is now our Chess model that performs as well as the larger LLM we took it from". You could potentially use the relevant experts as a basis for training a smaller task-specific model, and there is work around expert pruning and other distillation techniques, but it's not as simple as "This smaller inner model is a complete task-specific model as is". And more importantly, that ensemble of experts *is* the LLM you're comparing it to, at least in major part. It doesn't really make sense to treat the LLM as some foreign model that calls the expert. The experts themselves don't often correlate to easily interpretable task domains, as they've been learned via joint computation with the other experts and broader LLM.
Then there's tool calling. Today LLMs are trained to make tool calls (basically execute functions) for particular tasks. Those tools can be smaller specialized models, or deterministic code, or anything really that can be fired from a host environment.
If you really want your mind blown look at how accurate Ray Kurzweil was from his 1999 book Age of Spiritual Machines
The most striking thing is that almost all of his missed predictions were missed not because the technology was not available, but because society rejected or didn’t adopt that technology
> but because society rejected or didn’t adopt that technology
OK, so imagine that in 1980, I predicted that we'll colonize Mars by the year 2020. Today, I'm arguing that I was right in my prediction - in broad strokes, we have the technology! Nothing is stopping us. It's just that politicians have different priorities.
Of course, that's a load of bull. You can't decouple predictions about the future of mankind from what said mankind actually decides.
I looked up the predictions on Wikipedia and I'd say that most of them are general enough that they'd hard to disagree with back in 1999 (there will be bioengineering, there will be portable computers, there will be robots, there will be AI), but the timing or the specifics are off on almost every single one of them.
Go to Dan Luu's evaluation (https://danluu.com/futurist-predictions/), search for "Appendix: detailed information on predictions". Seems like Kurzweil's has a huge number of awful predictions. This is from four years ago, so a few things are closer to being real now. But many things he predicted for the early 2000s still haven't arrived.
Another prediction I remember from this book was that AI would be supplied much like electricity as a kind of utility into many different appliances. This is basically how it has turned out with tokens and APIs.
With that said, maybe he meant inevitable as N approaches infinity... So there's still time!
Aside: I think the biggest interesting prediction he had was that AI would obey a user driven network effect. It seems like a very clean prediction, and yet it has failed to materialize? I wonder if this has more to do with just how scrape-able the whole internet was or with distilation or what? There is certainly an advantage to having good user data, but it seems like it is largely something that can be worked around (by hook or crook)?
Possibly the best photography book I've come across. It's out of print, but I managed to snag a copy on eBay last year
At the most abstract level, you can observe patterns in a model's activations and theorize that particular collections of components contribute to particular kinds of operations. There are many flavors of this intuition. In that sense, you can think of there being "submodels" within a model, insofar as different parts of the network can participate in specialized computations. Some reading that might interest you in this area would include things like mechanistic interpretability.
There are also Mixture of Expert models, which are a dominant architecture for LLMs now, in which you essentially have a bunch of smaller FFNs that get routed over/aggregated together when the LLM performs a forward pass. It would be a mistake to treat any single expert as a better task specific model though. They are trained as part of an ensemble, and they're almost always being combined with other experts in the forward pass, so they're more like (to use a loose analogy) regions of the brain that tend to correspond more towards particular actions than others. You couldn't just extract one and say "This is now our Chess model that performs as well as the larger LLM we took it from". You could potentially use the relevant experts as a basis for training a smaller task-specific model, and there is work around expert pruning and other distillation techniques, but it's not as simple as "This smaller inner model is a complete task-specific model as is". And more importantly, that ensemble of experts *is* the LLM you're comparing it to, at least in major part. It doesn't really make sense to treat the LLM as some foreign model that calls the expert. The experts themselves don't often correlate to easily interpretable task domains, as they've been learned via joint computation with the other experts and broader LLM.
Then there's tool calling. Today LLMs are trained to make tool calls (basically execute functions) for particular tasks. Those tools can be smaller specialized models, or deterministic code, or anything really that can be fired from a host environment.
The most striking thing is that almost all of his missed predictions were missed not because the technology was not available, but because society rejected or didn’t adopt that technology
OK, so imagine that in 1980, I predicted that we'll colonize Mars by the year 2020. Today, I'm arguing that I was right in my prediction - in broad strokes, we have the technology! Nothing is stopping us. It's just that politicians have different priorities.
Of course, that's a load of bull. You can't decouple predictions about the future of mankind from what said mankind actually decides.
I looked up the predictions on Wikipedia and I'd say that most of them are general enough that they'd hard to disagree with back in 1999 (there will be bioengineering, there will be portable computers, there will be robots, there will be AI), but the timing or the specifics are off on almost every single one of them.
[0]: https://idlewords.com/2023/1/why_not_mars.htm
For instance, by his prediction, we're already many years past a 100 year average life expectancy (with 120 years expected by the end of the decade).
I think his mistake was tightly coupling exponential compute with exponential progress in biology - which has so far not panned out.
I would argue that society would likely have been glad to adopt, but the profit-seeking motive got in the way (again).
I was focused on augmented reality largely in part because it was such a big enabler of all other AI technologies
However people have almost uniformly rejected AR glasses my entire career for social reasons and that is only increasing
To the point where in 2016 I was debating with Jessica Alba and she was firm that they won’t take off cause most people don’t want to wear glasses