While Alice in Wonderland may have been able to believe several impossible things before breakfast, I personally have a hard time reconciling the hype around AI coding.
- Major companies like Microsoft, Google and Meta (a.k.a. Facebook) have 30% or more of their code written by AI, according to their CEOs. (Of course, it should be noted that these companies have a vested interested in getting funding and customers for their AI.
- AI is good at automating generic, repetitive tasks – I have found this to be true.
- AI can generate mediocre code that 80% works. It is as good as the bottom 10% of programmers – I have found this to be true.
- Apps can be created by “vibe coding” – just type a few words and the AI will spit out the code for a complete app. I have found this to be completely false in all but pretty basic cases.
- A famous study from MIT using a randomized control trial found that using AI made expert developers SLOWER.
Now, wait a minute! We all spent decades hearing about how Google, Microsoft, Facebook etc. hired the best and brightest. There was a whole industry devoted to teaching people how to pass their coding interviews. So, did all these Google and Microsoft engineers suck? Actually, the ones I have heard speak at events were pretty smart. I don’t think they spent their days creating web pages to collect your email, credit card number and address and write which pair of overpriced yoga pants you added to your shopping cart.
What do you mean by “AI writes 30% of the code” ?
Before AI, we had Stack Overflow, which I am sure is part of the data that most models were trained on. New and experienced developers alike would go to the site and search for answers to questions from “How do you define a dictionary in Python?” to “Why is the JavaScript code I posted here not working?”
Often, you didn’t even need to post a question, you could just search for when someone asked the same question you had and read the answer, along with several snarky comments.
Does that mean Stack Overflow wrote 50% of the code out there? I certainly saw a lot of spaghetti code “copy pasta” that had clearly been pasted from the web. I’d read it and think – wait a minute, this variable / function isn’t needed here at all. It does nothing. That’s because the code had been copied from some larger project.
Before we had Stack Overflow (yes, I was programming before the Internet), if I needed to, say, do an inventory control system to track the progress of parts on a production line, I didn’t start with a blank screen. I’d walk down the hall, stick my head into someone’s office and say, “Hey, Bob, I have to do this inventory control thing. Have you done anything like that?”
Bob might, or he might say, “No, but I think Helen might, over at the Kearny Mesa office.” We didn’t have email yet, either, so I’d drive over to where Helen was, look at her program, or at Bob’s, and modify it to fit the needs of the missile line.
My point, which you may have despaired of me having, is that we never sat down at a keyboard and wrote one line of code after another. If you count me asking copilot, “I want to pop the last element off the array named back_array” and copying the code – then yes, AI “writes” 15% of my code, just in the same way that Stack Overflow, Google, intellisense and Helen used to write my code.
The Economics Don’t Add Up
Partly because I believed the hype and also because I like to learn new things, I spent six months learning about large language models, Python, Vertex AI, Gemini AI, retrieval augmented generation and multi-agent systems. I read books, took courses and spent many, many hours prompting, using Google Colab and much more. I didn’t start from ground zero. I already had decades of experience programming and teaching statistics.
After many frustrating hours, I was able to create an app that would write code for educational games using our code base and many pages of documentation to augment the LLM’s knowledge of Python, JavaScript and HTML. It worked – most of the time. Then, I ran out of credits and was being charged to use the LLM and host our database.
It would give an answer, say, “The problem is that a second event listener is added to this same element later in the code.” When I responded that no, that was not correct, it would give another incorrect answer. I could totally see that MIT finding that developers take longer using AI.
Let me take a brief interlude and tell you about vibe coding – though I’m not sure I can without far exceeding the amount of profanity our Board of Directors will accept, and they’re a pretty liberal bunch. Suffice it to say that AI produced the wrong code over and over. I primarily used Gemini’s pro model which is supposedly optimized for code. About 80% of the time it gave functions that worked and the other 20% it failed repeatedly. This was true for both Javascript and Python.
The Two Reasons I Noped Right Out of AI and Went Back to Writing My Own Code
The repeated incorrect answers were reason enough, but you could argue that ChatGPT 11.6, Claude or whatever you want to suggest would be better. Maybe you’re right. The number one issue for me was that I would need to pay whatever company it was to use their model, Not only would that increase the price I need to charge but I’d be totally tied to them in the future. If their prices went up, which they inevitably will, I’d have no choice but to pay it. I’ve noticed that my Adobe subscription goes up next year and that I am now using credits for AI tasks that used to be unlimited. My cost for each user for Google apps has also gone up, they say it is to pay for the AI features but I don’t want most of those. The free Colab option for creating agents times out much more quickly, too. My Office 365 subscription is also going up. These are all costs the companies can force me to pay unless I want to quit using their products.
I have to wonder, if so much code is being written by AI now, why their prices are going up instead of down. Shouldn’t it be cheaper?
Anyway …. we went back to the drawing board and our rewriting the game builder ourselves. Besides cost, the second reason is we have total control over the code. We know what everything does and as we write it, on a regular basis, we see room for improvement. We ask ourselves what would be most useful for users and what would be annoying. For example, if I realize, yes, I do want a title page after all, I can go back several screens, change my decision about a title and then create the title screen but everything else up to that point will be saved.
Oh, and yes, it does take less time programming without AI. Those MIT folks were right.
If you’d like to get the free demo of our game builder, it will be available in April. Sign up here to get on the list. If you’re not at a school district, never fear, we will have a few sessions open to the public.
