So, I get the idea of having a layer of abstraction between your tests and the system under test. You decouple _what_ is being done from _how_ it is done, which leads to robust, easily maintained, and even more re-usable test suites. I've seen it in action, it's a great idea, I've reaped the benefits myself. Great stuff.
What I've never really understood is why some people choose natural language for the abstraction. It just seems like such an odd and expensive choice. Most of the layers of abstraction we add in our codebase are written in the programming language of choice itself, they're functions, classes, what have you. But for this one specific flavor of abstraction, folks reach for natural language.
The one potential benefit I could think of for this over a more standard DSL is that non-technical colleagues would be able to read or even write these specs. But in practice I've only seen this pan out once in my life, and that was with a more traditional DSL rather than natural language.
Because the intent has always been that Product delivers the BDD statements not Engineering, but Product people have (for the most part in my experience) refused to deliver what BDD flows need to be tested. That’s why most of the time the test suites aren’t as resilient because the engineer is guessing at which flows to test.
In that context it makes more sense that Llms are a push to replace Product people not Engineering people.
I also think BDD is a really powerful tool for UI acceptance tests! I've now worked on adding it to two AI codebases (one engineered, another vibecoded), and so far it has been mostly positive. Having non-engineers be able to see what effects their code changes have in the UI is nice; and I find having high level API and UI tests is forcing the first layer of cleanup (standardize access to DB through models/services, and a single API SDK).
I am also really interested in standardizing the REST interface with typespec, and the database schema with DBML. I think creating these closed-loops with high-level specs is one of the best ways to make an AI project ownable by a team.
The example of “transcibe the initial project discussion, turn that into BDD and that makes the rails to guide the AI is … either brilliant and insightful or a snake eating it’s own tail.
I've been doing that for almost a year and I feel like I'm taking crazy pills when I talk about it, because it works so well.
I transcribe most product meetings, turn them into a transcript, run that through an accuracy pass with a bunch of context (mostly vocabulary definitions etc), and use that as a starting doc for generating issues and tests. There are apps that do it fully end to end now I think.
Yeah, basically I would just start transcribing everything, if you aren't already, and get a bunch of context. It's best to start from the very beginning but the reality of enterprise is that nothing is truly greenfield.
Maybe start with a project kickoff or brainstorm, some kind of planning meeting, and select one task out of it to start. I always start with just the meeting transcript as a context, and expand context from there as needed with things like internal standards docs, libraries, references from other projects, etc, and only as little as is necessary. That generates the project plan, and then I feed that into claude code or whatever.
I think the biggest thing I've done recently is start to be extremely critical about the output, basically using promptfoo to try to figure out what the right level of context and instructions are, not just for external facing e.g. chatbots or whatever, but for my own internal development process, planning, everything. The more you can pin down and calibrate the better - it's easy to run a given prompt for a month, so spending a day (or even a week!) out of the month to refine your prompts beats just grinding ad hoc.
Edit: another thing I do is generate tests bdd-style separately from the code that they're testing (for e2e and high level at least), that derives out of the project plan too.
This is probably HN's tech background speaking. In the enterprise dev world, everyone knows what BDD means just like every dev out there knows what OOP means.
Thanks. (That’s what it suggested for me as well. Clarifying that I meant in the context of software, it suggested a misspelling of SOAD - Service-Oriented Analysis and Design.)
Yadda 3.0.0 is out. The release modernises the JavaScript BDD library, but more interestingly, it was largely built by Claude Code and points to why executable specifications may become even more valuable in an agentic development world.
It may be an age thing. 15-ish years ago, TDD, BDD, Red/Green Testing, and a whole bunch of others were hot topics. They’re still useful approaches to know about, even if some (like, IMO, BDD) didn’t really stand the test of time.
What I've never really understood is why some people choose natural language for the abstraction. It just seems like such an odd and expensive choice. Most of the layers of abstraction we add in our codebase are written in the programming language of choice itself, they're functions, classes, what have you. But for this one specific flavor of abstraction, folks reach for natural language.
The one potential benefit I could think of for this over a more standard DSL is that non-technical colleagues would be able to read or even write these specs. But in practice I've only seen this pan out once in my life, and that was with a more traditional DSL rather than natural language.
Is there a hidden benefit I'm overlooking?
In that context it makes more sense that Llms are a push to replace Product people not Engineering people.
I am also really interested in standardizing the REST interface with typespec, and the database schema with DBML. I think creating these closed-loops with high-level specs is one of the best ways to make an AI project ownable by a team.
I think it has to be worth a try though…
I transcribe most product meetings, turn them into a transcript, run that through an accuracy pass with a bunch of context (mostly vocabulary definitions etc), and use that as a starting doc for generating issues and tests. There are apps that do it fully end to end now I think.
Maybe start with a project kickoff or brainstorm, some kind of planning meeting, and select one task out of it to start. I always start with just the meeting transcript as a context, and expand context from there as needed with things like internal standards docs, libraries, references from other projects, etc, and only as little as is necessary. That generates the project plan, and then I feed that into claude code or whatever.
I think the biggest thing I've done recently is start to be extremely critical about the output, basically using promptfoo to try to figure out what the right level of context and instructions are, not just for external facing e.g. chatbots or whatever, but for my own internal development process, planning, everything. The more you can pin down and calibrate the better - it's easy to run a given prompt for a month, so spending a day (or even a week!) out of the month to refine your prompts beats just grinding ad hoc.
Edit: another thing I do is generate tests bdd-style separately from the code that they're testing (for e2e and high level at least), that derives out of the project plan too.
I was expecting something on Binary Decision Diagrams. SAOD is worst with TLAs.
Funny enough, Google's overview box also suggests soad: "SAOD (or SOAD) most commonly refers to the rock band System of a Down"
I'd consider explaining what BBD is at the start of the front page of your project.
That's disconcerting. Software engineers ought to know.
My experiences with Opus have been such that I always max out my Fable allotment but rarely exceed 40% of the remaining limit on Claude.