27 comments

  • jpollock 4 hours ago
    The design of the system is very interesting, particularly how it expects to handle errors.

    In 90's Telco, you used to have a pair of systems and if they disagreed, they would decide which side was bad and disable it.

    In modern cloud, you accept there are errors. There's another request in ~10+ms. You only look when the error rate becomes commercially important.

    My understanding of spacecraft is that there would be 3 independent implementations and they would vote.

    The plane has a matrix of sensors and systems, allowing faults to be bubbled up and bad elements disabled independently.

    The ADIRU does compare values to detect failures (median of 3 sensors), but they could only detect errors that last >1s. The flight computer used the raw data - because the sensors aren't interchangeable (they won't have consistent readings in all flight modes)!

    Very nifty.

    One thing, they say "memorisation period", I don't think it's a memorisation period? From my reading of the algorithm, it should be more "last value retention period"? Or "sensor spurious fault reading delay"?

    Section 2.1 A330/A340 flight control system design "AOA computation logic"

    https://www.atsb.gov.au/sites/default/files/media/3532398/ao...

    • jpollock 4 hours ago
      For example....

      "Preliminary A330/A340 FCPC algorithm"

      "The algorithm did not effectively manage a specific situation where AOA 2 and AOA 3 on one side of the aircraft were temporarily incorrect and AOA 1 on the other side of the aircraft was correct, resulting in ADR 1 being rejected."

      So, you've got a system where _two_ of the three sensors are bad, and you need to deal with it.

      • Loudergood 1 hour ago
        I'm in awe of the fact that two sensors can be wrong AND agree with each other.
        • Nextgrid 33 minutes ago
          Those being analog sensors measuring analog, physical things, they will never exactly agree with each other; so there's a plausibility window. As long as the fault causes the sensors to remain within said window they will be considered as valid.
        • UltraSane 14 minutes ago
          It is just like having range of values considered to be equal for floating point numbers.
    • rubatuga 3 hours ago
      Space computers are generally in 3 with a hot spare
  • addaon 19 hours ago
    I’d really, really like to know what microcontroller family this was found on. Assuming that this is a safety processor (lockstep, ECC, etc) it suggests that ECC was insufficient for the level of bit flips they’re seeing — and if the concern is data corruption, not unintended restart, it means it’s enough flips in one word to be undetectable. The environment they’re operating in isn’t that different from everyone else, so unless they ate some margin elsewhere (bad voltage corner or something), this can definitely be relevant to others. Also would be interesting to know if it’s NVM or SRAM that’s effected.
    • RealityVoid 15 hours ago
      See my other comments in the other threads. This does not have EDAC. I was as surprised as you but it doesn't seems to be an MCU but a composition of several distinct chips. That flight computer was designed in the 90's and updated in 2002 with a new hw variant that does have edac. So yes, for this kind of thing, I can buy that a bit flip happened.

      You can see much more data in the report:

      https://www.atsb.gov.au/sites/default/files/media/3532398/ao...

      • lxgr 11 hours ago
        > This does not have EDAC. I was as surprised as you but it doesn't seems to be an MCU but a composition of several distinct chips.

        Wasn't the philosophy back then to run multiple independent (and often even designed and manufactured by different teams) computers and run a quorum algorithm at a very high level?

        Maybe ECC was seen as redundant in that model?

        • RealityVoid 9 hours ago
          > Wasn't the philosophy back then to run multiple independent (and often even designed and manufactured by different teams) computers and run a quorum algorithm at a very high level?

          It was, and they did (well, same design, but they were independent). I quote from the report:

          "To provide redundancy, the ADIRS included three air data inertial reference units (ADIRU 1, ADIRU 2, and ADIRU 3). Each was of the same design, provided the same information, and operated independently of the other two"

          > Maybe ECC was seen as redundant in that model?

          I personally would not eschew any level of redundancy when it can improve safety, even in remote cases. It seems at the moment of the module's creation, EDAC was not required, and it probably was quite more expensive. The new variant apparently has EDAC. They retrofitted all units with the newer variants whenever one broke down. Overall, ECC is an extra layer of protection. The _presumably_ bit flip would be plausible to blame for data spikes. But even so, the data spikes should not have caused the controls issue. The controls issue is a separate problem, and it's highly likely THAT is what they are going to address, in another compute unit.

          "There was a limitation in the algorithm used by the A330/A340 flight control primary computers for processing angle of attack (AOA) data. This limitation meant that, in a very specific situation, multiple AOA spikes from only one of the three air data inertial reference units could result in a nose-down elevator command. [Significant safety issue]"

          This is most likely what they will address. The other reports confirm that the fix will be in the ELAC produced by Thales and the issue with the spikes detailed in the report was in an ADIRU module produced by Northrop Gruman.

        • JorgeGT 10 hours ago
          I don't know about the A320 but this was certainly the model for the Eurofighter. One of my university professors was in one of the teams, they were given the specs and not allowed to communicate with the other teams in any way during the hw and sw development.
          • RealityVoid 9 hours ago
            > they were given the specs and not allowed to communicate with the other teams in any way during the hw and sw development.

            Jeez, it would drive me _up the wall_. Let's say I could somewhat justify the security concerns, but this seems like it severely hampers the ability to design the system. And it seems like a safety concern.

            • lambdaone 9 hours ago
              What you are trying to minimize here is the error rate of the composite system, not the error rate of the individual modules. You take it as a given that all the teams are doing their human best to eliminate mistakes from their design. The idea of this is to make it likely that the mistakes that remain are different mistakes from those made by the other teams.

              Providing errors are independent, it's better to have three subsystems with 99% reliability in a voting arrangement than one system with 99.9% reliability.

              • saltcured 3 hours ago
                This seems like it would need some referees who watch over the teams and intrude with, "no, that method is already claimed by the other team, do something else"!

                Otherwise, I can easily see teams doing parallel construction of the same techniques. So many developments seem to happen like this, due to everyone being primed by the same socio-technical environment...

            • K0balt 9 hours ago
              The idea was to build three completely different systems to produce the same data, so that an error or problem in one could not be reasonably replicated in the others. In a case such as this, even ideas about algorithms could result in undesirable similarities that could end up propagating an attack surface, a logic error, or a hardware weakness or vulnerability. The desired result is that the teas solve the problem separately using distinct approaches and resources.
              • Filligree 6 hours ago
                And did they?

                Sometimes the solution is obvious, such that if you ask three engineers to solve it you’ll get three copies of the same solution, whereas that might not happen if they’re able to communicate.

                I’m sure they knew what they were doing, but I wonder how they avoided that scenario.

                • addaon 4 hours ago
                  I can (and have in past) written a long explanation on my experience with this, but…

                  Redundancy is a tool for reducing the probability of encountering statistical errors, which come from things like SEUs.

                  Dissimilarity is a tool for reducing the “probability” of encountering non-statistical errors — aka defects, bugs — but it’s a bit of a category error to discuss the probability of a non-probabilistic event; either the bug exists or it does not, at best you can talk about the state coverage that corresponds to its observability, but we don’t sample state space uniformly.

                  There has been a trend in the past few decades, somewhat informed by NASA studies, to favor redundancy as the (only, effective) tool for mitigating statistical errors, but to lean against heavy use of dissimilarity for software development in particular. This is because of a belief that (a) independent software teams implement the same bugs anyway and (b) an hour spent on duplication is better spent on testing. But at the absolute highest level of safety, where development hours are a relatively low cost compared to verification hours, I know it’s still used; and I don’t know how the hardware folks’ philosophy has evolved.

                • cmckn 5 hours ago
                  Even with the same approach, I imagine the implementation could differ enough to still meet the goal. But I’m also curious if the differences were actually quantified after the fact, it seems an important step.
                • sandworm101 4 hours ago
                  Not at airbus. Ask a german, french and british engineer the same question and you will never, ever get the same answer from each.
            • lxgr 5 hours ago
              How so? It’s a safety measure at least as much as a security one.

              It’s essentially a very intentional trade-off between groupthink and the wisdom of crowds, but it lands on a very different point on that scale than most other systems.

              Arguably the track record of Airbus’s fly-by-wire does them some justice for that decision.

      • Reason077 13 hours ago
        The recalled aircraft include the latest A320neo model, some of which are basically brand new. Why would they be using flight computers from before 2002? Why is an old report from 2008, relating to a completely different aircraft type (A330), relevant to the A320 issue today?
        • t0mas88 12 hours ago
          > Why would they be using flight computers from before 2002?

          Because getting a new one certified is extremely expensive. And designing an aircraft with a new type certificate is unpopular with the airlines. Since pilots are locked into a single type at a time, a mixed fleet is less efficient.

          Having a pilot switch type is very expensive, in the 50-100k per pilot range. And it comes with operational restrictions, you can't pair a newly trained (on type) captain with a newly trained first officer, so you need to manage all of this.

          • Reason077 12 hours ago
            I think you're confusing a type certificate (certifying the airworthiness of the aircraft type) with a type rating, which certifies the pilot is qualified to operate that type.

            Significant internal hardware changes might indeed require re-certification, but it generally wouldn't mean that pilots need to re-qualify or get a new type rating.

            • t0mas88 11 hours ago
              No I meant designing a new aircraft with a new type certificate instead of creating the A320neo generation on the same type certificate. The parent comment wondered why Airbus would keep the old computers around, I tried to explain why they keep a lot of things the same and only incrementally add variants. Adding a variant allows them to be flown with the same type rating or with only differences training (that's what EASA calls it, not sure about the US term) which is much less costly.
              • darkwater 10 hours ago
                Asking from ignorance: shouldn't the computer design be an implementation detail to the captain, while the interface used by who pilots stays the same for that type of airplane? I understand physical changes in the design need a retraining but the computer?
                • t0mas88 9 hours ago
                  Ideally you would not change the computer at all so your type certificate doesn't change. If you have to (or for commercial reasons really want to) make a change you would try very hard to keep that the same type certificate or at most a variant of the same type certificate. If you can do that then it will be flown with the same type rating and you avoid all the crew training cost issues.

                  But to do that you'll still have to prove that the changes don't change any of the aircraft characteristics. And that's not just the normal handling but also any failure modes. Which is an expensive thing to do, so Airbus would normally not do this unless there is a strong reason to do it.

                  The crew is also trained on a lot of knowledge about the systems behind the interface, so they can figure out what might be wrong in case of problems. That doesn't include the software architecture itself but it does include a lot of information on how redundancy between the systems work and what happens in case one system output is invalid. For example how the fail over logic works in case of a flight control computer failure, or how it responds to loosing certain inputs. And how that affects automation capabilities, like: no autoland when X fails, no autopilot and degradation to alternate contol law when Y fails, further degradation if X and Z fail at the same time. Sometimes also per "side", not all computers are connected to all sensors.

                  The computer change can't change any of that without requiring retraining.

                  • mlyle 4 hours ago
                    1. I don't think adding robustness necessarily requires changing how systems are presented to the flight crew.

                    2. Bigger changes than this are made all the time under the same type certificate. Many planes went from steam gauges to glass cockpits. A320 added a new fuel tank with transfer valves and transfer logic and new failure modes, and has completely changed control law over the type. etc.

          • RealityVoid 6 hours ago
            Since the new versions of the same ADIRU have EDAC, they have been using it on planes since 2002 and they have been putting the EDAC variant in whenever an old one was being returned for repairs, I don't think this is the reason. I think the reason is that they had 3 ADIRU's and even if one got wonky, the algorithm on the ELAC flight computer would have to take the correct decision. It did not take the correct decision. The ELAC is the one being updated in this case.
        • LiamPowell 12 hours ago
          > Why would they be using flight computers from before 2002?

          Why would you assume they're not? I don't know about aircraft specifically, but there's plenty of hardware that uses components older than that. Microchip still makes 8051 clones 45 years after the 8051 was released.

          • K0balt 8 hours ago
            That’s just wild to think about. We should all strive to build solutions that plague our descendants with their persistent utility.
            • hylaride 4 hours ago
              From a pure safety point of view, it's easier to deal with older, but well-understood products, only updating them if it's an actual safety issue. The alternative is having to deal with many generations of tech, as well as permutations with other components, that could get infinitely complicated. On top of that, it's extremely time consuming and expensive to certify new components.

              There's a reason the airlines and manufacturers hem and haw about new models until the economics overwhelmingly make it worthwhile, and even then it can still be a shitshow. The MCAS issue is case in point of how introducing new tech can cause unexpected issues (made worse by Boeing's internal culture).

              The 787 dreamliner is also a good example of how hard it is. By all accounts is a success, but it had some serious teething problems and still has some concerns about the long term wear and tear of the composite materials (though a lot of it's problems wasn't necessarily the application of new tech, but Boeing's simultaneous desire to overcomplicate the manufacturing pipeline via outsourcing and spreading out manufacturing).

        • RealityVoid 9 hours ago
          The issue detailed in the linked report details why the spike happened in the first place on the ADIRU (produced by Northrop Gruman). The recalled controller is the ELAC that comes from Thales. The problem chain was that despite the ADIRU spiking up, the ELAC should not have taken the reactions it took. So they are fixing it in the ELAC.
        • 4ndrewl 12 hours ago
          The neo is not brand new - it's an incremental update to the 320. neo refers to New Engine Option
          • rkomorn 12 hours ago
            They wrote "some of which are basically brand new", which is technically correct.

            They didn't say the design was brand new.

        • Havoc 13 hours ago
          > Why would they be using flight computers from before 2002?

          Guessing that using previously certified stuff is an advantage

        • RealityVoid 13 hours ago
          Because the problem isn't just this. It's that the flight controller did not properly decide what to do when the data spiked because of this issue as well.
      • Liftyee 10 hours ago
        What does EDAC mean here? I wasn't able to find a definition. My guess is "error detection and correction"?

        Difference between it and ECC?

        • K0balt 9 hours ago
          EDAC is the concept, ECC is a family of algorithmic solutions in the service of the concept. Specific implementations of ECC are the engineering solution that implement the specific form of ECC in specific devices at the hardware or software level.

          It’s confusing because EDAC and ECC seem to mean the same thing, but ECC is a term primarily used in memory integrity, where EDAC is a system level concept.

        • RealityVoid 9 hours ago
          That was my initial confusion as well. It means exactly what you guessed, "Error detection and correction". The term is also spelled out in the report. I asked Claude about it (caveat emptor) and it said EDAC is the correct name for the circuitry and implementation itself whereas ECC is the algorithm. Gemini said that EDAC is the general technique and ECC is one implementation variant. So, at this point, I'm not sure. They are used interchangeably (maybe wrongly so), and in this case, we're referring to, essentially, the same thing, with maybe some small differences in the details. In my professional life, almost always I referred to ECC. In the report, they were only using EDAC. I thought I'd maintain consistency with the report so I tried using EDAC as well.
          • Normal_gaussian 9 hours ago
            Large portions of this comment provides zero to negative value. You've quoted two LLMs and couched it in "caveat emptor" and "so I'm not sure". The rest of your comment has then mused over this data you do not trust using generalities ("my profession" are you a JS S/W eng? A chip design specialist at ARM? A security researcher?).

            All of the value of your comment comes from the first sentence and the last two.

            • RealityVoid 6 hours ago
              Sheesh, tough crowd.
              • ImPostingOnHN 5 hours ago
                Feel free to consult LLMs, with all their downsides (like you needing to verify what they say, because it could be totally wrong).

                What you're doing here is half the job: consulting an LLM and sharing the output without verifying whether it is true. You're then saying 'okay everyone else, finish my job for me, specifically the hard part of it (the verification), while I did the easy part (asking a magic 8 ball)'.

                From this perspective, your comment could be viewed as disrespectful of others by asking them to finish your job, and of negative value because it could be totally hallucinated and false, and you didn't care enough about others to find out before posting it.

                tl;dr: 'I asked an LLM and it said X' will likely, for the near future, be downvoted just like 'I flipped a coin and it said X'. You should be pretty confident that what you post is not false before posting it, regardless of how you came up with it.

          • dgacmu 7 hours ago
            The more correct and general answer is that:

            - EDAC is a term that encompasses anything used to detect and correct errors. While this almost always involves redundancy of some sort, _how_ it is done is unspecified.

            - The term ECC used stand-alone refers specifically to adding redundancy to data in the form of an error correcting code. But it is not a single algorithm - there are many ECC / FEC codes, from hamming codes used on small chunks of data such as data stored in RAM, to block codes like reed-solomon more commonly used on file storage data.

            - The term ECC memory could really just mean "EDAC" memory, but in practice, error correcting codes are _the_ way you'd do this from a cost perspective, so it works out. I don't think most systems would do triple redundancy on just the RAM -- at that point you'd run an independent microcontroller with the RAM to get higher-level TMR.

          • Yokolos 9 hours ago
            EDAC is a general term for an error detection and correction system. It can encompass ECC memory or other solutions.

            https://www.sciencedirect.com/science/article/abs/pii/S01419...

    • TehCorwiz 18 hours ago
      • mlyle 15 hours ago
        Yah, but that's a case of the package not being opaque enough.
      • russdill 15 hours ago
        Completely unrelated and due to a design failure by the rpi folks.
        • hughw 5 hours ago
          Is it really so unrelated? Isn't it a case where a similar phenomenon -- radiation impacting a computer calculation -- happened and it's one we can all relate to more easily, and reproduce if we cared to, than high altitude avionics? Not necessarily disputing but it just seems like a relatable case that helps me understand the issue better. If it's a radically different case somehow I'm interested to learn.
          • russdill 4 hours ago
            No, because it's a completely different kind of radiation.
            • hughw 3 hours ago
              Different band, sure. Same principle, right?
              • russdill 53 minutes ago
                The difference between ionizing and non-ionizing radiation is quite different. But for much of the radiation effecting electronics at high altitudes it's largely subatomic particles.

                And of course you can block the type radiation that caused problems for the rpi with a good piece of paper.

                • hughw 13 minutes ago
                  Got it. Ty.
    • anonymousiam 15 hours ago
      proper SEU mitigation goes far beyond ECC. Satellites fly higher than the A320, and they (at least the ones I know about) use Triple Modular Redundancy: https://en.wikipedia.org/wiki/Triple_modular_redundancy

      https://en.wikipedia.org/wiki/Single-event_upset

      For manned spaceflight, NASA ups N from 3 to 5.

      Other mitigations include completely disabling all CPU caches (with a big performance hit), and continuously refreshing the ECC RAM in background.

      There are also a bunch of hardware mitigations to prevent "latch up" of the digital circuits.

      • rkagerer 14 hours ago
        In redundant systems like these, how do you avoid the voting circuit becoming a single point of failure?

        Eg. I could understand if each subsystem had its own actuators and they were designed so any 3 could aerodynamically override the other 2, but I don't think that's how it works in practice.

        • simne 7 hours ago
          > how do you avoid the voting circuit becoming a single point of failure

          They do not. Just make voting circuit much more reliable than computing blocks.

          As example, computing block could be CMOS, but voting circuit made from discrete components, which are just too large to be sensitive to particles.

          Unfortunately, discrete components are sensitive to overall exposure (more than nm scale transistors), because large square gather more events and suffered by diffusion.

          Other example from aviation world - many planes still have mechanic connection of steering wheel to control surfaces, because mechanic connection considered ideally reliable. Unfortunately, at least one catastrophe happen because one pilot blocked his wheel and other cannot overcome this block.

          BTW weird fact, modern planes don't have rod physically connected to engine, because engine have it's own computer, which emulate behavior of old piston carburetor, and on Boeing emulating stick have electronic actuator, so it automatically placed in position, corresponding to actual engine mode, but Airbus don't have such actuator.

          I want to say - especially big planes (and planes overall), are weird mix of very conservative inherited mechanisms and new technologies.

          • anonymousiam 2 hours ago
            Electronics in high-radiation environments benefit from a large feature size with regard to SEU reduction, but you're correct that the larger parts degrade faster in such environments, so they've created "rad-hard" components to mitigate that issue.

            https://en.wikipedia.org/wiki/Radiation_hardening

            It's interesting to me that triple-voting wasn't as necessary on the older (rad-hard) processors. Every foundry in the world is steering toward CPUs with smaller and smaller feature sizes, because they are faster and consume less power, but the (very small) market for space-based processors wants large feature sizes. Because those aren't available anymore, TMR is the work-around.

            https://en.wikipedia.org/wiki/IBM_RAD6000

            https://en.wikipedia.org/wiki/RAD750

            Most modern space processing systems use a combination of rad-hard CPUs and TMR.

        • cpgxiii 5 hours ago
          In some cases, it is exactly the case of multiple independent actuators, such that the "voting" is effectively performed by the physical mechanism of the control surface.

          In other cases all of the subsystems implement the comparison logic and "vote themselves out" if their outputs diverge from the others. A lot of aircraft control systems are structured more as primary/secondary/backup where there is a defined order of reversion in case of disagreement, rather than voting between equals.

          But, more generally, it is very hard to eliminate all possible single points of failure in complex control systems, and there are many cases of previously unknown failure points appearing years or decades into service. Any sort of multi-drop shared data bus is very vulnerable to common failures, and this is a big part of the switch to ethernet-derived switched avionics systems (e.g. Afdx) from older multi-drop serial busses.

        • AlphaSite 13 hours ago
          Voting can be coordinated between the N cpus rather than an external arbiter (even making that redundant eventually required the CPUs to decide what to do if they disagree so may as well handle it internally).
        • V__ 7 hours ago
          Can't this be solved by having a "high refresh-rate"? Even if the voting circuit gets hit, if it updates 60 times a second it won't really affect any mechanical parts since the next signal will quickly override the error?
        • jasonwatkinspdx 10 hours ago
          My understanding is you're roughly right: the actuators will have their own microcontroller. It receives commands from the say 3 flight computers, then decides locally how respond if they mismatch. Ie for 2 out of 3 matching it may continue as commanded, but with only 1 out of 3 it may shift into a fail safe strategy for whatever that actuator is doing.
        • exe34 13 hours ago
          if the issue is radiation bit flipping, you could make that part overly shielded?
          • baq 12 hours ago
            Define ‘overly’. You can submerge it in a sphere of water, but that’s going to be expensive to launch.
            • exe34 1 hour ago
              I suspect a couple millimeters of lead in the right place would do it. cheaper to shield the voting mechanism than the whole thing.
      • aborsy 12 hours ago
        TMR and co are basically repetition codes, simplest performant least efficient ECC.
    • jayanmn 18 hours ago
      I am worried about a software fix for what looks like hardware problem.
      • afavour 17 hours ago
        It could be as simple as storing multiple copies of the relevant data and adding a checksum, something like that.

        Hardware fix is the ultimate solution but it might be possible to paper over with software.

      • themerone 17 hours ago
        Gracefully handling hardware faults is a software problem. The Air France Flight 447 crash was the result of bad software and bad hardware.
        • f1shy 14 hours ago
          And bad pilot training, if I recall correctly.
          • amelius 11 hours ago
            I suppose because they were not instructed to work around the software and hardware flaws.
            • f1shy 7 hours ago
              No. The behavior was wrong, no matter what you are flying. Same problem with proper training would have result in a simple report.
            • janeric1 10 hours ago
              Because they kept pitching up in a stall
              • f1shy 7 hours ago
                And two pilots were trying to fly the plane without talking to each other. One learned “if something happens just pull back, this plane cannot stall”. No pilot learned to say “my plane” when flying. Way too many errors.
        • foldr 11 hours ago
          Crashes caused by pilots failing to execute proper stall recovery procedures are surprisingly common, and similar accidents have happened before in aircraft with traditional control schemes, so I’m skeptical that there are any hardware changes that would have made much difference. The official report doesn’t identify the hardware or software as significant factors.

          The moment to avoid the accident was probably the very first moment when Bonin entered a steep climb when the plane was already at 35,000 feet, only 2000 feet below the maximum altitude for its configuration. This was already a sufficiently insane thing to do that the other less senior pilot should have taken control, had CRM been functioning effectively. What actually happened is that both of the pilots in the cockpit at the start of the incident failed to identify that the plane was stalled despite the fact that (i) several stall warnings had sounded and (ii) the plane had climbed above its maximum altitude (where it would inevitably either stall or overspeed) and was now descending. It’s never very satisfying to blame pilots, but this was a monumental fuck up.

          If the pilots genuinely disagree about control inputs there is not much that hardware or software can do to help. Even on aircraft with traditional mechanically linked control columns like the 737, the linkage will break if enough pressure is applied in opposite directions by each pilot (a protection against jamming).

          • julik 8 hours ago
            True. I would say, however, that every "concept" of airliner flight deck has its own gimmicks that can kill. The Airbus "dual input" is such a gimmick. Even though there was, for example, an AF accident with a 777 where there was hardware linkage between yokes and the two pilots were fighting... each other. Physically.
            • foldr 7 hours ago
              The official report doesn't identify the lack of sidestick linkage as a factor in the accident. Neither of the two pilots who were at the controls had any idea what was happening. Both pulled back on their sticks repeatedly right up to the moment of impact. The captain, who eventually realized (too late) that the plane was stalled, was standing behind them, and so would not have benefited from linked sticks.

              There's a detailed breakdown here: https://admiralcloudberg.medium.com/the-long-way-down-the-cr...

        • vel0city 17 hours ago
          I'm reminded of the Apollo moon landing where the computer was rapidly rebooting and being in an OK-ish state to continue to be useful almost immediately
          • CrossVR 15 hours ago
            It wasn't rebooting, it ran out of memory and started aborting lower priority tasks. It was a excellent example of robust programming in the face of unexpected usage scenarios.
            • f1shy 14 hours ago
              Of topic for the thread, but on for the comment: I was working in an automotive project 3 years ago. It was all about safety, and one hypothesis was the processor could get overloaded. I was astonished no one in a grouo of 20 “senior sw architecs” had any idea about the concept of load shedding. The proposed solution was “in that case, reboot”.

              Mind you whatever came out of that project is rolling on the street today.

              • concinds 12 hours ago
                We really should mandate all that stuff to be open-source, so we can be aware of how defective everything is.
            • K0balt 8 hours ago
              Fail safe/fail soft

              I still design this into many of the things I work on, especially if I’m working close to the metal on controller systems. At some point it becomes ridiculous / impossible but I’m often thinking about how a system would handle memory corruption, bit flips, invalid sensor data, etc. These days, somebody should design a triple redundant microcontroller that runs quorum on the gpio at the hardware level. It could be a 0.30 part instead of 0.10 one, but I would specify it just about everywhere. Add $3 to BOM cost to categorically eliminate an entire class of failure would be ramrodded by legal into just about every medical device, PLC, critical automotive system, etc one would think. Seems like a good gambit for a riscV startup, but what do I know.

        • idkfasayer 15 hours ago
          [dead]
      • kachapopopow 17 hours ago
        software fixes are totally fine since the chance of two redundant pairs failing within the time it takes to correct these errors is more zero's than there are atoms in the universe. (each pilot has a redundant computer and because there's two pilots there's two redundant pairs)
      • willis936 8 hours ago
        It's a system problem. The system is being fixed.
  • rossjudson 47 minutes ago
    My armchair guess is that they had a new control pathway not properly participating in their integrity hand-off protocols, doing some kind of transformation outside of that protection.

    I once saw some HW engineers go nuts trying to find out why a storage device had an error rate several orders of magnitude higher than the extremely low error rate they expected (and triggering data corruption errors). It turns out to be one extremely deep VHDL-based control area for an FPGA that didn't properly do integrity. You'd have to flip a bit at an incredibly precise point in time for error to occur, but that's what was happening. When all the math was said and done, that FPGA control path integrity miss exactly accounted for the the higher error rate.

  • rene_d 12 hours ago
    The Aviation Herald has more technical details:

    https://avherald.com/h?article=52f1ffc3&opt=0

    • loxodrome 12 hours ago
      Thanks for the link. This line in particular is concerning.

      "This identified vulnerability could lead in the worst case scenario to an uncommanded elevator movement that may result in exceeding the aircraft structural capability."

      • isodev 6 hours ago
        Well, I think in the grand scheme of things (including on the ground), the range of safety faults that can be triggered by a simple bitflip at the wrong moment range from inconvenient to absolute disaster. So in that sense, I'm very happy that Airbus has managed to identify opportunities to improve their design to be even more resilient.
  • nickdothutton 11 hours ago
    I’d just like to point out that if you are in the computing industry long enough, you will get to see a few such incidents under different circumstances, not only in industries like aerospace. Mostly things like ECC save your a*, sometimes your software will be able to recognise a temporary spurious reading and disregard it because you had enough alternative checking logic, or in the case of realtime and safety critical maybe even your systems can take a vote between them. Got caught out by (cpu cache line) bit flips in the 90s, months of pain trying to track it down. Some of your will know :-)
    • Theodores 11 hours ago
      Is that you, Julian?

      I jest, but, once upon a time I worked with an infallible developer. When my projects crashed and burned, I would assume that it was my lack of competence and take that as my starting point. However, my colleague would assume that it was a stray neutrino that had flipped a bit to trigger the failure, even if it was a reproducible error.

      He would then work backwards from 93 million miles away to blame the client, blame the linux kernel, blame the device drivers and finally, once all of that and the 'three letter agencies' were eliminated, perhaps consider the problem was between his keyboard and his chair.

      In all fairness, he was a genius, and, regarding the A320 situation, he would have been spot on!

    • LadyCailin 11 hours ago
      We noticed this in our logs once! We service a huge amount of traffic, and as part of that, we log what is effectively an enum. We did a summarization of this field once, and noticed that there were a couple of “impossible” values being logged. One of my coworkers realized that the string that actually got logged was exactly one bit off from a valid string, and we came to the conclusion that we were probably seeing cosmic rays in action, either in our service, or in the logging service.
      • Philip-J-Fry 1 hour ago
        I also saw a similar thing. I also naively pointed at "cosmic rays". It wasn't until someone found the actual bug that I realised how unlikely that was.

        The actual bug was unsafe code somewhere else in the application corrupting the memory. The application worked fine, but the log message strings were being slightly corrupted. Just a random letter here and there being something it shouldn't be.

        The question really should have been, if this was truly cosmic interference, why only this service and why was the problem appearing more than once over multiple versions of the application?

        Cosmic rays are a great excuse to problems you don't yet understand. But the reality of them is extremely rare and it's like 99% a memory corruption bug caused by application code.

      • tuetuopay 6 hours ago
        I had a similar story on my NAS that got one btrfs path corrupt. Plopped in on the btrfs IRC, one of the devs noticed the inconsistency was one bitflip away from the right value. Incredibly they were able to give me the right commands to fix it! Got to give credit where it is due, btrfs took the safe path and refused to touch the affected directory until fixed, and has enough tooling to fix this.

        I won’t blame cosmic rays but more likely dying RAM. The NAS now runs ECC memory.

  • pyb 15 hours ago
    The aerospace industry has had countermeasures in place against bit-flips for a long time, oftentimes thanks to redudancy

    Airbus/Thales's fix in this case appears to add more error checking, and to restart the misbehaving component. https://bea.aero/fileadmin/user_upload/BEA2024-0404-BEA2025-...

    ("une supervision interne du composant à l’origine de la défaillance ; - un mécanisme de redémarrage automatique de ce composant dès lors que la défaillance est détectée)

    • nolist_policy 8 hours ago
      The linked document is not related to this incident.
  • supernova87a 10 hours ago
    I wonder how the incident was diagnosed? Does the FDR record low level errors that might've contributed to this? I thought that it only recorded certain input parameters and high-level flight metrics but I'm no expert.

    If a radiation event caused some bit-flip, how would you realize that's what triggered an error? Or maybe the FDR does record when certain things go wrong? I'm thinking like, voting errors of the main flight computers?

    Anyway, would be very interested to know!

    • yread 8 hours ago
      From a comment on avherald:

      "Had the same problem with low power CMOS 3 transistor memory cells used in implantable defibrillators in the 1990s. Needed software detection and correction upgrade for implanted devices, and radiation hardening for new devices. Issue was confirmed to be caused by solar radiation by flying devices between Sydney and Buenos Aires over the south pole multiple times, accumulating a statistically significant different error rate to control sample in Sydney."

  • qaq 19 hours ago
    Has BoFesc vibes "It's friday, so I get into work early, before lunch even. The phone rings. Shit!

    I turn the page on the excuse sheet. "SOLAR FLARES" stares out at me. I'd better read up on that..."

  • 65a 15 hours ago
    There's a great postmortem here about what might have been a similar SEU (single event upset--bitflip) here: https://www.atsb.gov.au/sites/default/files/media/3532398/ao...
  • skx001 1 hour ago
    This video shows the the A320 computer and how the computer cooling system works

    https://www.youtube.com/watch?v=HQuc_HhW6VA

  • minitoar 16 hours ago
    We flew too close to the sun
  • joelthelion 16 hours ago
    Do they really need to ground the entire fleet for that? One incident for ten thousand planes in the air for years. I'd think that giving airlines two months to fix it would be sufficient.
    • mrpippy 15 hours ago
      I don’t believe it’s been years, only the latest firmware version for the ELAC is affected. The fix is to downgrade (or replace hardware with a unit running earlier firmware)
    • jfoster 14 hours ago
      I wonder who eats the cost of this? I presume it's the airlines.

      So the immediate cost to Airbus of grounding the fleet is quite low, whilst the downside of not grounding the fleet (risk of incident, lawsuits, reputation, etc.) could be substantial.

      • Havoc 12 hours ago
        Yeah should be airlines

        It sounds like the fix is fairly quick so probably not as expensive as the max multi month groundings

        I doubt anyone is going to sue. Repairs etc are a part of life when owning aircraft. So as long as Airbus makes this happen fast and smooth they’re probably ok

    • miyuru 12 hours ago
      this is Airbus, not Boeing
    • kijin 15 hours ago
      I imagine it could help with Airbus marketing.

      "We take proactive measures, whereas our competitor only takes action after multiple fatal crashes!"

      • probably_wrong 8 hours ago
        I know someone who is stranded in another continent thanks to this. Trust me, all the understanding I could have as a technical user has been offset by the MASSIVE pain in the ass that is rebooking an international flight. And non-technical users have heard "the plane will not travel because it requires a software update", which does not inspire confidence.

        As far as I'm concerned it has not helped with their marketing.

      • brabel 12 hours ago
        Imagine an airplane crashed in these 2 months. I bet you would join the chorus and blame them for gross negligence.
        • kijin 10 hours ago
          There's a huge difference between "manufacturer recommended updates, but airline waited until the last week to apply them" and "manufacturer didn't even acknowledge the issue" in terms of who the chorus is going to blame.
    • f1shy 14 hours ago
      I would personally not want to seat in those planes in those 2 months.
      • upcoming-sesame 9 hours ago
        nothing worse than rushing a fix in production - only to find out the fix has caused more damage than the original bug
    • pyb 11 hours ago
      I get the feeling that they are doing this partly for marketing purposes.
    • refulgentis 4 hours ago
      Yeah, because the alternative is knowing you might kill people due to a mundane engineering known issue.
    • Bud 15 hours ago
      From their viewpoint, you have to think about what happens if, after they became aware of this vulnerability, there was then a crash because they weren't prompt and aggressive enough in addressing it. That's the kind of thing that ruins your entire company forever.
      • Esophagus4 8 hours ago
        Yep - Boeing is still dealing with it years later.

        (As they should - I’m still very mad at them.)

  • 1970-01-01 5 hours ago
    They said the same thing at Toyota when the unintended accel problem was in the news, but never found a real world example. There are a lot more old Toyotas still on the road than Airbuses in the air, so distance to the sun makes all the difference here? I wonder if they only see issues when flying near the north pole?
  • nubinetwork 5 hours ago
    Why would a CME disrupt a single brand and model of aircraft, when the entire planet is covered in computers that almost never have bitflip issues when a CME rolls through every few months?
    • squarefoot 5 hours ago
      I would guess barely enough cable shielding paired with long enough paths along the aircraft so that the signals there would be more likely affected by EM induced currents.
    • 1970-01-01 5 hours ago
      I'm guessing EM shielding flaw or something electronic. See my comment on the Toyotas. It doesn't make sense from a raw probability perspective.
  • rishabhaiover 2 hours ago
    I hope Airbus only uses Honeywell or Collins in their newer planes.
  • owenthejumper 18 hours ago
    A friend works at Jetblue. They are scrambling hard to do the updates.
  • jfoster 17 hours ago
    I've noticed that some carriers seem to be suggesting that there might be no impact to flights, but isn't this an immediate grounding for each aircraft until the update is made?

    How is it possible that this wouldn't impact upon flight schedules?

    • icegreentea2 17 hours ago
      The grounding is for 6000 of 11000 A320 series. I believe it's some combination of software and hardware configuration that is at risk.
      • jfoster 15 hours ago
        Thank you; that makes sense. I had the impression it was the entire fleet.
        • julik 8 hours ago
          It depends on whether the ELAC is an LRU (line-replaceable unit, i.e. a box with ports that can be swapped at an airport) and whether a software update can be uploaded into a unit that is installed (not all aircraft have a "firmware update via cable or floppy", so to speak)
    • simne 5 hours ago
      If possible for exact this plane, could make software update just as routine procedure.

      But as I hear, air transporters could buy planes in different configurations, so for example, Emirates airlines, or Lufthansa always buy planes with all features included, but small Asian airlines could buy limited configuration (even without some safety indicators).

      So for Emirates or Lufthansa, will need one empty flight to home airport, but for small airline will need to flight to some large maintenance base (or to factory base) and wait in queue there (you could find in internet images of Boeing factory base with lot of grounded 737-MAXes few years ago).

      So for Emirates or Lufthansa will be minimal impact to flights (just like replacement of bus), but for small airlines things could be much worse.

    • arrel 17 hours ago
      N of 1, but I’m stuck in phoenix overnight because our flight was delayed an hour and a half by airbus maintenance and we missed our connection.
  • oofbey 1 hour ago
    This is in response to JetBlue flight 1230 from Cancun to Newark on October 30, 2025, where a cosmic ray of some kind flipped a bit and caused a dangerous situation. At the time there was a minor (G1) geomagnetic storm - meaning more cosmic rays than normal. The Planetary K-index was at 5. These are somewhat elevated numbers - enough to produce a visible Aurora in Canada, but probably not even the northernmost US. But also this level of space weather is also very common. We hit G1 or higher about once a week. That's the really damning part. If it had happened in a G4 or G5 storm, then the engineers might have responded "we can't fix everything", but this level of reliability is clearly unacceptable.
  • jakub_g 10 hours ago
    From newspaper reporting on this, they are rolling back a software update. I wonder what was the original cause or the update? How often are flight computers software updated and why?
    • julik 8 hours ago
      This ELAC version is 100-something, and the A320 first flew around 1988. Why the updates - for example, there are updates to flight control law transitions, like after 1991 where the aircraft would limit flight control inputs during landing, thinking it would be preventing a stall - because it would not go into the flare law appropriately. See https://en.wikipedia.org/wiki/Iberia_Flight_1456

      The cause could have also been an extra check introduced in one of the routines - which backfired in this particular failure scenario.

  • ChrisArchitect 23 hours ago
  • op00to 19 hours ago
    Solar radiation like solar wind, or sunlight? They don’t say.
    • tyingq 5 minutes ago
      Would guess "cosmic rays". Sun Microsystems had a batch of UltraSparc servers that were very sensitive to it, and it was a big issue.

      https://docs.oracle.com/cd/E19095-01/sf4810.srvr/816-5053-10...

      https://en.wikipedia.org/wiki/Cosmic_ray

    • mr_toad 19 hours ago
      “Analysis of a recent event”

      I presume they mean a Coronal Mass Ejection.

      • bparsons 19 hours ago
        There was a very large CME ten days ago. The NOAA scale had predicted a high likelihood of disruptions, and had specifically suggested that spacecraft and high altitude aircraft could be impacted.

        https://www.swpc.noaa.gov/noaa-scales-explanation

        https://kauai.ccmc.gsfc.nasa.gov/CMEscoreboard/prediction/de...

      • fwip 19 hours ago
        I feel like the event was something that happened to a plane. That said, I wouldn't think sunlight would be penetrating to the chips running the plane.
        • dtagames 19 hours ago
          Gamma rays penetrate everything and have definitely been known to disrupt computer circuits.
          • fwip 15 hours ago
            Yes, which is why the solar flare scenario makes more sense.
        • awesome_dude 18 hours ago
          > The grounding of Airbus A320neo aircraft around the world can be traced back to an incident on a JetBlue flight operating a Cancun to New Jersey service on 30 October.

          > At least 15 passengers were injured and taken to the hospital after a sudden drop in altitude on the flight from Mexico was forced to make an emergency landing in Florida, US aviation officials said at the time.

          > The Thursday flight from Cancun was headed to Newark, New Jersey, when the altitude dropped, leading to the diversion to Tampa International Airport, the US Federal Aviation Administration said in a statement.

          > Pilots reported “a flight control issue” and described injuries including a possible “laceration in the head,” according to air traffic audio recorded by LiveATC.net.

          > Medical personnel met the passengers and crew on the ground at the airport. Between 15 and 20 people were taken to hospitals with non-life-threatening injuries, said Vivian Shedd, a spokesperson for Tampa Fire Rescue.

          > Pablo Rojas, a Miami-based attorney who specialises in aviation law, said a “flight control issue” indicated that the aircraft wasn't responding to the pilots.

          https://www.stuff.co.nz/travel/360903363/what-happened-fligh...

          • lostlogin 18 hours ago
            > At least 15 passengers were injured and taken to the hospital after a sudden drop in altitude on the flight from Mexico was forced to make an emergency landing in Florida, US aviation officials said at the time.

            I’m surprised passengers are allowed to unbuckle for so much of each flight. You can get injured while buckled it, but that seems less common.

            • MaxfordAndSons 17 hours ago
              The flight attendants/safety card will tell you to stay buckled whenever seated, even if the seat belt sign is off, but many (most?) people will ignore that guidance and stay unbuckled for as long as they are technically allowed.

              Only aviation professionals or recovering flight phobics like me who have watched every episode of Air Crash Investigation will take proactive safety measure of their own accord. To normies it's all just a pointless hassle.

              • sailfast 15 hours ago
                I stay buckled and I’m just a “normie” not afraid of flying that understands turbulence doesn’t always happen in a bell curve with some notice. Not sure if that makes you feel any better? :)
                • MaxfordAndSons 14 minutes ago
                  Having an understanding of the bell curve of turbulence makes you a bit more advanced than a normie on the aviation knowledge bell curve imo :)
              • rainsford 7 hours ago
                I'm amazed how many grown ass adults on airplanes act like little kids when it comes to seat belts and basically everything else.

                Not just ignoring flight crew advice and common sense to generally stay buckled in order to gain maybe a minor amount of comfort and convenience being unbuckled, but unbuckling even when the seat belt sign is on and again common sense says being buckled in is the smart move. On my most recent flight I heard quite a few people unbuckling their seat belts while the plane was still rolling down the runway after landing. You couldn't wait 5 more minutes until the plane is at the gate?

                • MaxfordAndSons 10 minutes ago
                  lol yep. It's like they have the same mentality as being a schoolbus (which, it's similarly wild to me that kids are just implicitly allowed to not wear their seatbelts on them but I guess thats an even more intractable enforcement problem).

                  Also: people clapping the second the back wheels touch on landing is particularly hilarious to me because it implies an acknowledgement of the precariousness of flying, but a complete ignorance of the fact that you're just entering the second most dangerous 30 seconds of the entire flight.

              • danmaz74 11 hours ago
                I'm not flight phobic but I still stay buckled all the time when I don't need to move. It's a very little nuisance.
              • baq 12 hours ago
                People have different priors for bad things that can happen on a plane. If you’ve experienced turbulence you’ll probably buckle up.
              • seg_lol 11 hours ago
                No reason to not buckle, I keep the belt a little looser, but buckled the entire time. Esp on Boeing planes, I want to get sucked out with the seat.
      • nQQKTz7dm27oZ 10 hours ago
        [dead]
  • raverbashing 13 hours ago
    Apparently the fix is reverting to a previous version of the SW (see https://avherald.com/h?article=52f1ffc3&opt=0 )

    Curious what a sw change might have done in terms of resiliency. Maybe an incorrect memory setting or some code path that is not calculating things redundantly maybe?

    • rootusrootus 5 hours ago
      So it's not just Boeing that can screw up software on an airplane. I guess now I have to be a little afraid of all the airliners.
  • kappi 17 hours ago
    Following the Airbus A320 emergency airworthiness action, everyone will be talking about the ELAC (Elevator Aileron Computer) manufactured by Thales, which caused a sudden pitch-down without pilot input on JetBlue 1230 back in October.

    So here’s everything you need to know about ELAC.

    The ELAC System in the Airbus A320: The Brains Behind Pitch and Roll Control https://x.com/Turbinetraveler/status/1994498724513345637

  • jMyles 23 hours ago
    This is one of the rare cases where, IMO, it makes sense to use a modified title as you've done here.
  • rvz 16 hours ago
    [flagged]
  • viiralvx 17 hours ago
    I was traveling during this entire ordeal. My flight got delayed by 7 hours. Insane day, just now boarding my flight. American Airlines was in shambles today.