I've had the same happen with a production run, except the crystals were from the same manufacturer and the same part number, just different batch. It was fun to track down.
I wonder if the author is right; using a slightly wrong capacitor for the clock (keeping in mind that they're usually +/- 10% or worse from the factory) may produce a different oscillation speed, but usually not an outright failure. These chips accept a pretty wide range of clock frequencies.
And in the comments, they note:
"I replaced the 18pF capacitors on one of the non-working boards with 10pF capacitors, but it still doesn’t boot or respond to the debugger. That surprised me – I thought that was the answer! It was a rush rework job and I made a bit of a mess of it, including accidentally desoldering and reinstalling the crystal, so I’ll try again later with another board. But it appears that the capacitor value may not have been the issue after all. Either 10pF is a bad value, or it’s the crystal itself that’s at fault, or I’ve failed somewhere in my troubleshooting reasoning. Hmm."
In my experience, mystery stability problems are often caused by capacitors, but of a different kind: decoupling capacitors on the power supply pins. If there's not enough of them to keep up with the noise originating from motors or digital switching, I'd expect that exact issue. Intermittent "impossible" CPU states on some boards, no rhyme or reason (because sometimes, that +/- 10% saves you and sometimes it does not). I'd try more caps and possibly some ferrite beads.
I'm really surprised that nowhere in the article does the author mention putting an oscilloscope on either the power rails or the clock signal. Normally when I'm troubleshooting broken devices those are the first things I check.
A multimeter can make it look like a chip is receiving the correct voltages, but there can be all sorts of noise and fluctuations that will cause really strange and inconsistent issues. You don't need a super fancy scope, even a 100mHz cheap handheld is more than good enough for checking power rails.
Also, for anyone who has to troubleshoot or fix boards, those smd resistor and capacitor kits are absolutely invaluable to have on the bench. After a decent scope, meter, and soldering station I would say that should be the next purchase for setting up an electronics lab. Nothing more annoying than trying to debug an issue and having to wait for the right value cap to be shipped. Especially since capacitors specifically often seem to have some trial and error to finding the right value for something like a decoupling cap. Fully modeling the noise generated on a supply rail often isn't completely possible, at least in my experience.
Generally you shouldn't put a probe on crystals because the capacitive loading changes your circuit. I've had mixed success but especially if you think that's the issue, it's an iffy debug step. Some micros support a clock output mode that you can use, though. Another method is to form a basic loop antenna with the ground clip and hover near it.
But you'd need a few more than 100 millihertz for that lol
20 MHz is well within the range of any random scope, and you should have at least one "any random scope" on your desk if you're doing electronics seriously enough to reach the "diagnosing QA fail boards" stage. So probe the damn thing - and see if the crystal ever reaches a stable operating frequency once the board is powered. That would tell a lot.
And yes, the "holy book of SMD passives" is a must too. They cost you what, $40 each? And can save you days of waiting for the "right" passives to arrive when you need to test a circuit change. You don't need to have every size - footprints are negotiable when you're hand-soldering - but you should have at least one Book of Many Resistors and one Book of Many Capacitors.
Not an electrical engineer, but I was 2 sentences in when I declared to myself "its the external clock source" and then a physical shudder as I remembered how many times a contract manufacturer has used some aging crystal or third party component src and not updated the matching capacitors, causing very similar issues on projects I've worked on in the past. The very very worst is the aging xtals - a warm wind can make them work for an undetermined amount of time, which is bad if the factory is running warm around 26C or so and you end up shipping the batch to Europe where its early winter...
When I got to my current workplace I was surprised to see we always, invariably (unless it's really impossible with the chip) use an oscillator not a crystal. But every time I see one of these posts I think about the difference in cost for us relative to the math of capacitance and debugging, it's just not worth it. And of course the first board I worked that we basically built to print and had a crystal integral to the design... They got the capacitance wrong and it didn't go.
“Some exhibited “haunted” behavior, seemingly jumping to random sections of the mcu program code, outputting messages on the display that made no sense given the context.
One of them appeared to work in slow motion, with LED blinking and display updates noticeably more sluggish than normal,”
my first thought was that it smelled like a clock issue.
Some of the nastier issues I have had the pleasure to debug included (a) traces that had microcracks which affected analog readings when the PCB heated up after prolonged usage (QC issue from the PCB fab) and (b) a (suspected) ESD strike that gradually took out several components in the weeks following as I was investigating the device while new problems kept popping up. Marginally stable composite amplifiers have also caused some headaches over the years.
And in the comments, they note:
"I replaced the 18pF capacitors on one of the non-working boards with 10pF capacitors, but it still doesn’t boot or respond to the debugger. That surprised me – I thought that was the answer! It was a rush rework job and I made a bit of a mess of it, including accidentally desoldering and reinstalling the crystal, so I’ll try again later with another board. But it appears that the capacitor value may not have been the issue after all. Either 10pF is a bad value, or it’s the crystal itself that’s at fault, or I’ve failed somewhere in my troubleshooting reasoning. Hmm."
In my experience, mystery stability problems are often caused by capacitors, but of a different kind: decoupling capacitors on the power supply pins. If there's not enough of them to keep up with the noise originating from motors or digital switching, I'd expect that exact issue. Intermittent "impossible" CPU states on some boards, no rhyme or reason (because sometimes, that +/- 10% saves you and sometimes it does not). I'd try more caps and possibly some ferrite beads.
A multimeter can make it look like a chip is receiving the correct voltages, but there can be all sorts of noise and fluctuations that will cause really strange and inconsistent issues. You don't need a super fancy scope, even a 100mHz cheap handheld is more than good enough for checking power rails.
Also, for anyone who has to troubleshoot or fix boards, those smd resistor and capacitor kits are absolutely invaluable to have on the bench. After a decent scope, meter, and soldering station I would say that should be the next purchase for setting up an electronics lab. Nothing more annoying than trying to debug an issue and having to wait for the right value cap to be shipped. Especially since capacitors specifically often seem to have some trial and error to finding the right value for something like a decoupling cap. Fully modeling the noise generated on a supply rail often isn't completely possible, at least in my experience.
But you'd need a few more than 100 millihertz for that lol
20 MHz is well within the range of any random scope, and you should have at least one "any random scope" on your desk if you're doing electronics seriously enough to reach the "diagnosing QA fail boards" stage. So probe the damn thing - and see if the crystal ever reaches a stable operating frequency once the board is powered. That would tell a lot.
And yes, the "holy book of SMD passives" is a must too. They cost you what, $40 each? And can save you days of waiting for the "right" passives to arrive when you need to test a circuit change. You don't need to have every size - footprints are negotiable when you're hand-soldering - but you should have at least one Book of Many Resistors and one Book of Many Capacitors.
“Some exhibited “haunted” behavior, seemingly jumping to random sections of the mcu program code, outputting messages on the display that made no sense given the context. One of them appeared to work in slow motion, with LED blinking and display updates noticeably more sluggish than normal,”
my first thought was that it smelled like a clock issue.
Some of the nastier issues I have had the pleasure to debug included (a) traces that had microcracks which affected analog readings when the PCB heated up after prolonged usage (QC issue from the PCB fab) and (b) a (suspected) ESD strike that gradually took out several components in the weeks following as I was investigating the device while new problems kept popping up. Marginally stable composite amplifiers have also caused some headaches over the years.
Hardware really is hard.