Thinking over Typing 2.0

As a developer, I’ve found it possible to describe to ChatGPT my general programming style and have it generate a good bulk of my code.

When I do this, I spend less time typing and a much higher percentage of my time thinking.

I think this will force a significant shift in developers who want to stay relevant. For years I’ve been urging developers to stop treating their IDE like a text editor and more like an engineering tool (eg use the refactorings!). This is much deeper.

Workflows

A Zero-Shot workflow involves setting up a context, and prompting the LLM with a question so that it can “answer your question” by generating more content.

A Chain-of-Thought workflow progressively chains together user-provided and LLM provided content so that it looks more like a conversation. It provides opportunities for you to guide the conversation linearly through phases like creative elaboration, critical analysis, and convergence.

A Tree-of-Thought workflow expands to a series of branches providing much wider creative elaboration and an opportunity to selectively include from different branches to cut the noise you’re converging through. This is a new technique, and the research paper claims a jump from 4% solution success with Chain-of-Thought to 74% solution success with Tree-of-Thought.

More Realistic Problem Domains

There are a ton of examples of using LLMs in simplistic problem domains that don’t stress the LLM’s context size. The context size is the number of tokens the model can use while generating, and this is split between context you provide and the amount of tokens it can generate before it hits the upper limit.

The move from GPT-3.5 to GPT-4 expanded my usable context size from 4K to 8K.

While 8K handles trivial code generating tasks well, it is very quickly stretched to support larger problem domains.

Expressing Context

The context you provide to prompt the generative LLM makes the difference between code you can use, and code you must rewrite.

Trying to find the minimum number of tokens in which to effectively express context has become an interesting exercise for me.

I’m not using tricks, rather I’m trying to use it as an opportunity to practice expressive precision.

eg:

You are an expert software craftsman that applies SOLID principles and the Four Rules of Simple Design in order to create clean, well organized, and easy to read code. Use typescript with parameter property constructors where possible.

It’s especially interesting for me because I must expose my intuitive knowledge. A skill and practice I use for coaching, but that doesn’t get exercised nearly enough as it is during this work.

Buggy Output

I also have to say though, ChatGPT comes up with some pretty insidious bugs.

Watches a generated test suite fail when run as a suite, but every individual test passes

“Ooooooooh, you’re newing up an instance of this object in your export statement!”

… and then I’m trying to cram one more style point in my system context that I’d never have violated intuitively…

I Gotta Wear Shades

I am incredibly excited for a software development future that involves more thinking and less typing code.

I feel that mindlessly typing code demeans the software development practice, and the humans doing it.

I think that software organizations that operate as feature factories will get buried under what’s possible when we combine real software engineering skill with these kinds of generative techniques.