When I first moved to the Valley to join Cisco, my second job in tech, they handed out a book at orientation. Geoffrey Moore’s Crossing the Chasm. I was working in QA and had not begun to understand marketing yet, but one thing stuck with me early: I liked being the guy who got in on the tech before the chasm. It’s probably why I still tend to work in it today. Always first to roll the dice on early iOS builds that bricked your phone, first to try new hardware that now sits in a drawer somewhere, and when GPT dropped in late 2022 I was hooked fairly early and haven’t looked back.
So when custom GPTs started I was right on board, and several of those early ones still sit in my library. Claude projects have since become my poison of choice. They are how a one-man practice turns around value as fast as clients expect.
As an avid user, I noticed something happening across my projects. They were getting slower, less accurate, and I was doing way more corrections and deeper rewrites than before. A few times I gave up and did it old school because I could do it better on my own.
How, in a world where new models are supposed to be taking over and ending humanity in a few years, could my AI be getting Dumb and Dumber? So I dug in.
With models changing this often, and some of us jumping between tiers with different access, we are all running a hodgepodge of the latest and greatest. The tools encourage it. Retiring old models, giving away free use of newer ones for short periods. When you build a project or a GPT, you build it around whatever model you chose at the time. If that project lives for a while, you switch, maybe a few times. Some of mine were built to be evergreen.
So the old project is giving the new model instructions in the old way which is total something I would assume a new model would fix, apparently not. The new model follows instructions more literally, so every workaround I wrote to force the old one to do things the way I wanted now gets obeyed to the letter and fights defaults that might be better than my original rules. The instruction files and reference docs grew for months to cover things the old model could not figure out on its own, and all of that now sits in context on every turn, which is where some of the slow comes from. And the examples I pasted in to show it what good looked like are now a ceiling, because the new model might handle everything in a new way.
The engine got “smarter” but it could not adapt to scaffolding I built around the old one aged. If the project is old enough and complex enough, the thing that used to be kick ass at its job is now slow and less accurate than the day you tuned it, and using the latest, greatest and more expensive model is making it worse.
Kind of a bummer this isn’t just handled for you. The fix- I spent a few hours last weekend retuning three or four of my worst offenders, mostly by deleting, a little bit of rewriting and they are humming again. A new step for projects that are more than a generation or two old. A small price for all this AI wonder, but I am not so sure the AI apocalypse is five years out anymore. Maybe we will get ten before Skynet becomes self-aware.