What’s new with our in-house load balancing service

(dropbox.tech)

48 points | by ABS 5 days ago

4 comments

  • leetrout 2 days ago
    I am surprised to see zookeeper being used with etcd.

    Why use both?

    Why even consider ZK in 2024? Is it just popular in java shops?

    Maybe I am misunderstanding?

    • aftbit 19 hours ago
      Dropbox has been using Zookeeper since the beginning. I assume they are just familiar with running it and understand its guarantees better.
    • bittermandel 1 day ago
      Since it's written ZooKeeper/etcd-based, I assume they use ZK and are going over to Etcd.
    • pgwhalen 1 day ago
      > Why even consider ZK in 2024?

      I'm no expert on these consensus-stores, but I guess my question would be "why not?" ZK is proven and works.

  • nepthar 1 day ago
    We used a deterministic aperture load balancing to solve this at twitter - https://blog.x.com/engineering/en_us/topics/infrastructure/2....

    Cool to see some other approaches, although I believe they've complicated theirs a bit more than necessary.

    • jauntywundrkind 1 day ago
      Ah nice, a "Power of 2 (random) choices" strategy. Article credits "The power of two choices in randomized load balancing" (1996) by Michael Mitzenmacher. PDF link: https://www.eecs.harvard.edu/~michaelm/postscripts/mythesis....

      I heard about someone using PO2C for cache eviction recently. Maybe Oxide talking about their storage work? Seems obviously smart. No need to keep LRU or carefully track & decide, just use statistics to compare two random elements and drop the least used one.

    • renewiltord 1 day ago
      Very cool. Low-state approaches are always more interesting.
  • renewiltord 1 day ago
    I see: load-aware routing. Why skip the weight update phase on low weight reports? Isn’t that precisely the period when those nodes are likely overloaded?
  • dang 1 day ago
    [stub for offtopicness]
    • aftbit 19 hours ago
      Bees gather pollen. In the fall, as temperatures get lower, they still want to keep gathering pollen, but sometimes they run out of warmth and energy to fly all the way back to the nest. Thus you will find bees sleeping on flowers through the night. When morning comes and the sun warms them, they'll pick up and fly off to the nest to drop off the last night's load of pollen.
    • ryandvm 1 day ago
      [flagged]
    • biddendidden 1 day ago
      [dead]
      • edm0nd 1 day ago
        Robinhood != Robin Hood != Robbin' The Hood
    • terramauthe 2 days ago
      [flagged]
      • yawnxyz 2 days ago
        "load balancing" is a cold way to describe a layoff...
        • blitzar 2 days ago
          "load balancing" is apt description of the role of HR
          • toomuchtodo 1 day ago
            More like container orchestration and bin packing, amirite? Faulty container? Terminate and launch a new container. Cattle, not pets.
        • Simon_ORourke 2 days ago
          "fully automated market-driven load balancing" more like.
      • guessmyname 2 days ago
        He does → https://www.linkedin.com/in/yi-shu-tai-13160154/

        According to his LinkedIn profile: 7 years 6 months into the job.

        Although, it looks like he transfered to a new team a year ago to do ML Ops.

        • apocalyptic0n3 1 day ago
          Dropbox just laid off 20% of the company yesterday. In many cases, people don't rush to update their LinkedIn so he may or may not still be there.
      • dylan604 1 day ago
        [flagged]
    • xyzzy4747 1 day ago
      [flagged]
      • bayindirh 1 day ago
        There's a tool called Robinhood which started its life around 2004 [0], In turn, Robinhood Markets are founded in 2013.

        Imagine knowing that Robinhood exists as one of the most popular (or the only) policy engines for large (EB sized) filesystems for but still naming your new company Robinhood.

        I think Robinhood Markets should C&D immediately and find themselves a new name.

        [0]: https://github.com/cea-hpc/robinhood

      • pgwhalen 1 day ago
        The Robinhood being discussed in this post is not a product, it's internal technology. The name clashing doesn't really matter.
        • xyzzy4747 1 day ago
          There are near-infinite things they could name their software, but instead they copied an existing name of another software project.

          Quite insufferable if you ask me.

          • bayindirh 1 day ago
            Software developers who love their job generally are into fantasy worlds, RPG games, astronomy and whatnot. If we nitpick every name choice, then life will have no fun.

            I have computers named after stars and galaxies. Some codenames of the applications I have developed myself has stupid connections to characters because of what they (the software) do are similar to characters themselves. I named a module "Scrat" because it was a state machine which changed thousands of states per second according to input it received from a file I/O module called Manfred (which carries things on his back or with its trunk).

            I have my names clash with others, and others' names clash with mine. Who cares, it's fun and makes things enjoyable for everyone, plus it allows one to inspire themselves by setting a vision for their software.

            Or, maybe we should start naming our software with UUID strings. That won't clash in a long time. :D

          • sophacles 1 day ago
            Why does every single thing need a unique name? Even trademarked names for businesses and products are scoped to business sectors - its why apple the computer company who took the name of apple the record label was just fine (until apple invented itunes and had to buy the record label).

            Heck even code does scoped naming... the compiler can't really handle punctuation that is slightly wrong and even it allows me to reuse variable names in the same program.

            I don't know why we'd need a univerally unique name for each piece of software ever created. The existing software called robinhood is unrelated to this load balancer, so even if I could download and use it, I doubt I'd have much of a problem distinguishing the two.

          • edm0nd 1 day ago
            imo its your mindset that is quite insufferable, not the companies decisions of naming internal tooling.
      • bityard 1 day ago
        Were Facebook, Google, and Reddit also the names of centuries-old fictional storybook characters?
        • xyzzy4747 1 day ago
          Were centuries-old fictional characters software projects?
    • satvikpendem 2 days ago
      [flagged]
      • guessmyname 2 days ago
        Round Robin [1] is a popular algorithm for load balancing.

        That’s the only reason they named the software “Robinhood”.

        [1] https://en.wikipedia.org/wiki/Round-robin_scheduling

        • Dalewyn 2 days ago
          Why specifically "Robinhood", though? Robinhood is a fairly well known brokerage.

          Why not Robin Hood or some other unique or generic name?

          • CogitoCogito 2 days ago
            I don't think there are many people who will confuse an in-house load balancing service used at dropbox with a brokerage even if both are called Robinhood. Similarly, I don't think that people will confuse said brokerage with an old character of English folklore (and no I don't think it's the space that's important).

            Context matters. There simply isn't a risk for meaningful confusion here. So what's the issue exactly?

            • donpott 1 day ago
              I think it's quite easy to confuse the two if one is diagonally reading the HN frontpage and sees a headline that begins with "What's new with Robinhood [...]".
              • ajoseps 1 day ago
                maybe dropbox didn’t optimize naming their in-house software for HN readers
            • dylan604 1 day ago
              > I don't think that people will confuse said brokerage with an old character of English folklore

              I don't know about that. Why do you think people would not link a financial company to the folklore character famous for redistributing money? That's kind of disingenuous claim to me.

              When I first heard about it, I actually assumed it was a reverse Robin Hood by taking money from the poor to give back to the rich type of scheme.

            • Dalewyn 1 day ago
              >(and no I don't think it's the space that's important).

              That space or the lack thereof is important.

              To date I only know of two Robinhoods, the brokerage and the load balancer, and I only learned about the second with this thread.

          • rsynnott 1 day ago
            I think Robinhood the brokerage was _fairly_ obscure til 2020 or so, no? This appears to have gone into production in 2020, so is likely older than that.
          • bayindirh 1 day ago
            Robinhood is a large filesystem policy engine and maintenance tool since 2004, too. Soo Robinhood Markets are already late to the game. :)
          • Veen 1 day ago
            Many things are named after Robin Hood. It's not as though the brokerage invented the term or has any right of exclusive use. You'd have to be quite dim-witted to muddle up a brokerage and a load balancer.
            • Dalewyn 1 day ago
              Many things are named after Robin Hood, but I only know of two things specifically calling themselves Robinhood and one of them I only learned from this thread.

              In fairness the load balancer presumably came first, but the point remains.

      • kergonath 2 days ago
        You cannot avoid all names that were or will be ever used by anyone. Collisions will happen. And Robin Hood is not particularly obscure…
      • stavros 2 days ago
        Probably because it takes from the rich and gives to the poor.
      • almost 1 day ago
        Weird that a gambling app would take thename of a popular character from legend!

        Or maybe not, try actually naming something sometime and you'll see. It's not easy to come up with something that fits all the requirements AND has never been used before.

        And it's not like anyone is ever going to get confused and accidentally try and use Dropbox's internal load balancer to buy options on a stock or something. Or try and make a gambling app the star of their next hit movie for that matter.

        • tgv 1 day ago
          A gambling app that --like all-- takes from the poor and gives to the rich. Sarcasm, or bitter misantropy, I suppose.
      • whalesalad 1 day ago
        because it is an internal tool? why does that cause an issue?
    • say_it_as_it_is 1 day ago
      [flagged]
      • qeternity 1 day ago
        Totally. They should just continue to carry a workforce they don't need, risking the viability of the business and the other 80% of the jobs.