Judging something while it’s massively improving

I have a toddler and he plays basketball. But I can beat him really easily. I can steal the ball from him or block his shot whenever I want (relax, I don’t). He can’t guard me. I can score at will. So, I guess he’s not a very good basketball player. 

Now everything I just said is true, but it’s also completely ridiculous, because it’s the wrong standard to judge him by. Why? Because he’s so young, and he’s massively improving. Given his current trajectory, it won’t be long before he’s a much better player than me.
 

Actually, he’s pretty good already.

This strikes me as similar to the criticisms I hear about large language models (LLMs) like ChatGPT. Most of them go like this: “They’re not accurate all the time” and “They make things up” and “The guy I know who’s been coding for 20 years is a much better coder than ChatGPT.” This sounds a lot like judging the toddler for not being good enough at basketball. 

And even though my son is making tremendous strides at a great pace, large language models are improving even more rapidly. If you compare the coding abilities of these models to just a year ago, they’re drastically better. They now implement multiple chains of thought and reasoning, and they’ve been trained on many more examples of code. 

It’s not that the critics are wrong per se. Each point they make is true. But they’re missing the point—it’s about the pace of the progress. And it isn’t hard to extrapolate the trajectory of this improvement, even just a year or two into the future, and to be wildly impressed by the capabilities of these models. 

So the next time you hear something negative about what ChatGPT can’t do, or doesn’t do all that well—remember that my toddler isn’t ready for the NBA either. But that says absolutely nothing about the basketball player he’s going to become.

Best regards,

Evan McGoff