AI coding helpers get FAILing grade

An academic study says ChatGPT is wrong more than half the time, when asked the sort of programming questions you’d find on Stack Overflow. The “comprehensive analysis” concludes that GitHub Copilot’s LLM engine will make many conceptual errors, couching its output in a wordy, confident and authoritative tone.

So, it’s hard to spot the errors, say the researchers. In this week’s Secure Software Blogwatch, we can’t say we’re totally surprised.

Your humble blogwatcher curated these bloggy bits for your …

academic analysis chatgpt coding copilot engine errors find github github copilot hard llm overflow programming questions sort stack stack overflow study wrong