A Simplistic Example

Let’s start from a simplistic example so that you can get a sense of where I’m thinking.

The way I work as a developer, is I think about the problem domain, think about thing I’d like my code to do, and write my first test that should prove whether it’s doing it.

Classic TDD cycle.

My first prompt:

You are an expert software craftsman that applies SOLID principles and the Four Rules of Simple Design in order to create clean, well organized, and easy to read code. Use typescript with parameter property constructors where possible.

I run a coaching practice that consists of both full-time (FTE) and contract coaches. I have to recover the costs of the coaches across a number of funding sources within our organization that are tied to initiatives or value-streams.

Let’s call this software project CoachingPracticeFinanceTracker.

Write me a first test that verifies that the implementation code is capable of adding a new funding source to the system. Do not write the implementation, only the test.

Now, here is the code that it wrote:

import { FundingSource } from '../src/funding-source'; // Adjust path as needed
import { CoachingPracticeFinanceTracker } from '../src/coaching-practice-finance-tracker'; // Adjust path as needed

describe('CoachingPracticeFinanceTracker', () => {
    let financeTracker: CoachingPracticeFinanceTracker;

    beforeEach(() => {
        financeTracker = new CoachingPracticeFinanceTracker();
    });

    test('should add new funding source', () => {
        // Arrange
        const fundingSource: FundingSource = new FundingSource('Initiative1', 10000);

        // Act
        financeTracker.addFundingSource(fundingSource);

        // Assert
        const addedFundingSource = financeTracker.getFundingSource('Initiative1');
        expect(addedFundingSource).not.toBeNull();
        expect(addedFundingSource?.name).toEqual(fundingSource.name);
        expect(addedFundingSource?.budget).toEqual(fundingSource.budget);
    });
});

Now notice a few things. Were I actually TDD’ing this through, I wouldn’t have written nearly this much code.

Rethinking

So let’s rethink what TDD is for, and how it helps us humans.

TDD forces us to build in very small batches. This is for a number of reasons, we want a very fast cycle time between starting a cycle and finishing a cycle. Well, what’s fast for a human vs an LLM?

Also, TDD forces us to take on minimal cognitive overhead so that we don’t get stuck juggling too many ideas at the same time through each cycle. Well, what’s minimal for an LLM?

So with this, I’m thinking two things.

1. I want to use a development style where I can work incrementally through collaborating with the LLM
2. I want to focus on the specification for the system so that I can be sure that I’m travelling with what I actually want it to do versus getting lost in the itty bitty details of the code.

How do I apply this?

I use the specification for the system as context, described by my prompts to the LLM.

Because the LLM can elaborate a small implementation and test together, perhaps I just let it do that.

New First Step

I start a new chat from scratch.

Here’s my new first prompt:

You are an expert software craftsman that applies SOLID principles and the Four Rules of Simple Design in order to create clean, well organized, and easy to read code. Use typescript with parameter property constructors where possible.

I run a coaching practice that consists of both full-time (FTE) and contract coaches. I have to recover the costs of the coaches across a number of funding sources within our organization that are tied to initiatives or value-streams.

Let’s call this software project CoachingPracticeFinanceTracker.

Write me a first test that verifies that the implementation code is capable of adding a new funding source to the system.

All I did was remove the constraint that it not write the implementation.

Here’s ChatGPT-4’s full response:

In the context of the requirement, I’d suggest using a testing library like Jest along with TypeScript to create your test.

Firstly, let’s define a simple FundingSource class, which we will use to represent a funding source in our tests and implementation.

class FundingSource {
  constructor(public id: string, public name: string, public description: string) {}
}

Now, let’s create a FinanceTracker class, which will represent our main system. In this class, we’ll define a addFundingSource method. This method will take a FundingSource object as its only parameter.

class FinanceTracker {
  private fundingSources: FundingSource[] = [];

  addFundingSource(source: FundingSource): void {
    // Implementation code here
  }

  getFundingSources(): FundingSource[] {
    return this.fundingSources;
  }
}

Now that we have a basic structure, let’s write the first test.

import { FinanceTracker, FundingSource } from './FinanceTracker'; // assuming both classes are in the same file
import { describe, expect, it } from '@jest/globals';

describe('FinanceTracker', () => {
  it('adds a new funding source to the system', () => {
    // Arrange
    const tracker = new FinanceTracker();
    const source = new FundingSource('id1', 'name1', 'description1');

    // Act
    tracker.addFundingSource(source);

    // Assert
    expect(tracker.getFundingSources()).toContain(source);
  });
});

This test first creates a FinanceTracker object and a FundingSource object. Then, it calls the addFundingSource method of the FinanceTracker with the FundingSource as its argument. Finally, it verifies that the FundingSource object is now present in the FinanceTracker’s funding sources by using Jest’s toContain function.

This is a basic unit test that verifies that the addFundingSource method of the FinanceTracker is working as expected.

Keep in mind, this test won’t pass until you implement the addFundingSource method in the FinanceTracker class. Here’s a possible simple implementation:

class FinanceTracker {
  private fundingSources: FundingSource[] = [];

  addFundingSource(source: FundingSource): void {
    this.fundingSources.push(source);
  }

  getFundingSources(): FundingSource[] {
    return this.fundingSources;
  }
}

Please note that this is a rudimentary implementation and the actual code might need to be more complex depending on other requirements.

So GPT-4 created a simple data object with a parameter property constructor, an initial FinanceTracker that doesn’t have any code for adding a funding source, and a single test that adds a new funding source through the implementation. This test will fail because the implementation is missing.

It doesn’t always do the same thing every time, but notice how this time it noticed that the test can’t pass until the implementation is added so it writes a new version of the FinanceTracker right away that provides that implementation so that the test can pass.

After creating a new empty typescript project and adding these files to the right places within it, the tests pass and I have a working implementation.

Now the TDD’ers among you will notice again how it wrote so much code at once, and that it completed the get function perhaps out of order.

Does that matter though? A single transaction (Zero-Shot) with the LLM has produced useful implementation code that’s fully covered by the tests.

Batch size: small. Cycle time: seconds (after writing the prompt).

Throw Away Responses

My time as a developer was all in crafting my prompt to the LLM.

And the majority of that time spent was thinking about what my first vertical slice of functionality should be, and how to express that clearly.

The knowledge I required to write these prompts was a healthy intersection of domain knowledge of the problem I’m trying to solve, as well as technical knowledge, how to prompt the LLM to craft code that I’d consider usable.

Domain knowledge:

…

I run a coaching practice that consists of both full-time (FTE) and contract coaches. I have to recover the costs of the coaches across a number of funding sources within our organization that are tied to initiatives or value-streams.

…

Write me a first test that verifies that the implementation code is capable of adding a new funding source to the system.

Developer knowledge:

You are an expert software craftsman that applies SOLID principles and the Four Rules of Simple Design in order to create clean, well organized, and easy to read code. Use typescript with parameter property constructors where possible.

…

Let’s call this software project CoachingPracticeFinanceTracker.

Ensemble Programming 2.0

Ensemble programming involves putting all of the cross-functional knowledge on a team together so that they can fully implement and deploy a small new capability to stakeholders within hours.

One challenge in this technique is that you can run the risk of team members disengaging from the work if less experienced practitioners fumble a little in their craft. Developer struggles with unfamiliar syntax or API, eyes glaze, people lean back, start checking their phones.

Now, there are ways to manage that, but look at our cycle time above. We’re all active in crafting a prompt that includes the bits that it needs from each of our experience. Likely we make mistakes, watch what the LLM generates, a team member points out the gap or error, and everyone comes back together to adjust the prompt and try again.

The LLM doesn’t fumble in elaborating the code. It does so quickly, and assertively, and often incorrectly, providing ample engagement opportunities for the team members to engage and scrutinize.

I haven’t tried it in this context yet, but I’m curious to do it.