Lessons Learned Writing a Book Collaboratively with LLMs

(Note: I'm not linking the resulting book. This post focuses solely on the process and practical lessons learned collaborating with LLMs on a large writing project.)

Hey HN, I recently finished a months-long project collaborating intensively with various LLMs (ChatGPT, Claude, Gemini) to write a book about using AI in management. The process became a meta-experiment, revealing practical workflows and pitfalls that felt worth sharing.

This post breaks down the workflow, quirks, and lessons learned.

Getting Started: Used ChatGPT as a sounding board for messy notes. One morning, stuck in traffic, tried voice dictation directly into the chat app. Expected chaos, got usable (if rambling) text. Lesson 1: Capture raw ideas immediately. Use voice/text to get sparks down, then refine. Key for overcoming the blank page.

My Workflow evolved organically: Conversational Brainstorming: "Talk" ideas through with the AI. Ask for analogies, counterarguments, structure. Treat it like an always-available (but weird) partner. Partnership Drafting: Let AI generate first passes when stuck ("Explain X simply for Y"), but treat as raw material needing heavy human editing/fact-checking. Or, write first, have AI polish. Often alternated. Iterative Refinement: The core loop. Paste draft > ask for specific feedback ("Is this logic clear?") -> integrate selectively > repeat. (Lesson 2: Vague prompts = vague results; give granular instructions. Often requires breaking down tasks: logic first, then style). Practice Safe Context Management: LLMs forget (context windows). (Lesson 3: You are the AI's external memory. Constantly re-paste context/style guides; use system prompts. Assume zero persistence across time). Read-Aloud Reviews: Use TTS or read drafts aloud. (Lesson 4: Ears catch awkwardness eyes miss. Crucial for natural flow).

The "AI A-Team": Different models have distinct strengths: ChatGPT: Creative "liberal arts" type; great for analogies/prose, but verbose/flattery-prone. Claude: Analytical "engineer"; excels at logic/accuracy/code, but maybe don't invite for drinks. Gemini: The "copyeditor"; good for large-context consistency. Can push back constructively. (Lessons 5 & 6: Use the right tool for the job; learn strengths via experimentation & use models to check each other. Feeding output between them often revealed flaws - Gemini calling out ChatGPT's tells was useful).

Stuff I Did Not Do Well:

Biggest hurdles:

AI Flattery is Real: Helpfulness optimization means praise for bad work. (Lesson 7: Prompt for critical feedback. 'Critique harshly'. Don't trust praise; human review vital). The "AI Voice" is Pervasive: Understand why it sounds robotic (training bias, RLHF). (Lesson 8: Combat AI-isms. Prompt specific tones; edit out filler/hedging/repetition/'delve'; kill em dashes unless formal). Verification Burden is HUGE: AI hallucinates/facts wrong. (Lesson 9: Assume nothing correct without verification. You are the fact-checker. Non-negotiable despite workload. Ground claims; be careful with nuance/lived experience). Perfectionism is a Trap: AI enables endless iteration. (Lesson 10: Set limits; trust judgment. Know 'good enough'. Don't let AI erode voice. Kill your darlings).

My Personal Role in This fiasco:

Deep AI collaboration elevates the human role to: Manager (goals/context), Arbitrator (evaluating conflicts), Integrator (synthesizing), Quality Control (verification/ethics), and Voice (infusing personality/nuance).

Conclusion: This wasn't push-button magic; it was intensive, iterative partnership needing constant human guidance, judgment, and effort. It accelerated things dramatically and sparked ideas, but final quality depended entirely on active human management.

Key takeaway: Embrace the mess. Capture fast. Iterate hard. Know your tools. Verify everything. Never abdicate your role as the human mind in charge. Would love to hear thoughts on others' experiences.

4 points | by scottfalconer 8 hours ago

1 comments

  • robotbikes 8 hours ago
    Nice. I leverage the strengths of AI in a way that affirms the human element in the collaboration. AI as it exists in LLMs is a powerful source of potentially meaningful language but at this point LLMs don't have a consistent conscious mind that exists over time like humans do. So it's more like summoning a djinn to perform some task and then it disappears back into the ether. We of course can interweave these disparate tasks into a meaningful structure and it sounds like you have some good strategies for how to do this.

    I have found that using an LLM to critique your writing is a helpful way of getting free generic but specific feedback. I find this route more interesting than the copy pasta AI voiced stuff. Suggesting that AI embodys a specific type of character such as a pirate can make the answers more interesting than just finding the median answer, add some flavor to the white bread.

    [-]
    • scottfalconer 8 hours ago
      One of the things I found helpful about getting out of the specific / formulaic feedback was asking the LLM to ask me questions. At one point I asked a fresh LLM to read the book and then ask me questions. It showed me where there were narrative gaps / confusing elements that a reader would run into, but didn't realy on the specific "answer" from the LLM itself.

      I also had a bunch of personal stories interwoven in and it told me I was being "indulgent" which was harsh but ultimately accurate.