What’s new with our in-house load balancing service

(dropbox.tech)

49 points | by ABS 260 days ago

4 comments

  • leetrout 257 days ago
    I am surprised to see zookeeper being used with etcd.

    Why use both?

    Why even consider ZK in 2024? Is it just popular in java shops?

    Maybe I am misunderstanding?

    • pgwhalen 257 days ago
      > Why even consider ZK in 2024?

      I'm no expert on these consensus-stores, but I guess my question would be "why not?" ZK is proven and works.

    • bittermandel 257 days ago
      Since it's written ZooKeeper/etcd-based, I assume they use ZK and are going over to Etcd.
    • aftbit 256 days ago
      Dropbox has been using Zookeeper since the beginning. I assume they are just familiar with running it and understand its guarantees better.
  • nepthar 257 days ago
    We used a deterministic aperture load balancing to solve this at twitter - https://blog.x.com/engineering/en_us/topics/infrastructure/2....

    Cool to see some other approaches, although I believe they've complicated theirs a bit more than necessary.

    • jauntywundrkind 257 days ago
      Ah nice, a "Power of 2 (random) choices" strategy. Article credits "The power of two choices in randomized load balancing" (1996) by Michael Mitzenmacher. PDF link: https://www.eecs.harvard.edu/~michaelm/postscripts/mythesis....

      I heard about someone using PO2C for cache eviction recently. Maybe Oxide talking about their storage work? Seems obviously smart. No need to keep LRU or carefully track & decide, just use statistics to compare two random elements and drop the least used one.

    • renewiltord 257 days ago
      Very cool. Low-state approaches are always more interesting.
  • renewiltord 257 days ago
    I see: load-aware routing. Why skip the weight update phase on low weight reports? Isn’t that precisely the period when those nodes are likely overloaded?
  • dang 256 days ago
    [stub for offtopicness]
    • aftbit 256 days ago
      Bees gather pollen. In the fall, as temperatures get lower, they still want to keep gathering pollen, but sometimes they run out of warmth and energy to fly all the way back to the nest. Thus you will find bees sleeping on flowers through the night. When morning comes and the sun warms them, they'll pick up and fly off to the nest to drop off the last night's load of pollen.
    • ryandvm 257 days ago
      [flagged]
    • biddendidden 257 days ago
      [dead]
      • edm0nd 257 days ago
        Robinhood != Robin Hood != Robbin' The Hood
    • terramauthe 257 days ago
      [flagged]
      • yawnxyz 257 days ago
        "load balancing" is a cold way to describe a layoff...
        • Simon_ORourke 257 days ago
          "fully automated market-driven load balancing" more like.
        • blitzar 257 days ago
          "load balancing" is apt description of the role of HR
          • toomuchtodo 257 days ago
            More like container orchestration and bin packing, amirite? Faulty container? Terminate and launch a new container. Cattle, not pets.
      • guessmyname 257 days ago
        He does → https://www.linkedin.com/in/yi-shu-tai-13160154/

        According to his LinkedIn profile: 7 years 6 months into the job.

        Although, it looks like he transfered to a new team a year ago to do ML Ops.

        • apocalyptic0n3 257 days ago
          Dropbox just laid off 20% of the company yesterday. In many cases, people don't rush to update their LinkedIn so he may or may not still be there.
      • dylan604 257 days ago
        [flagged]
    • satvikpendem 257 days ago
      [flagged]
      • guessmyname 257 days ago
        Round Robin [1] is a popular algorithm for load balancing.

        That’s the only reason they named the software “Robinhood”.

        [1] https://en.wikipedia.org/wiki/Round-robin_scheduling

        • Dalewyn 257 days ago
          Why specifically "Robinhood", though? Robinhood is a fairly well known brokerage.

          Why not Robin Hood or some other unique or generic name?

          • CogitoCogito 257 days ago
            I don't think there are many people who will confuse an in-house load balancing service used at dropbox with a brokerage even if both are called Robinhood. Similarly, I don't think that people will confuse said brokerage with an old character of English folklore (and no I don't think it's the space that's important).

            Context matters. There simply isn't a risk for meaningful confusion here. So what's the issue exactly?

            • donpott 257 days ago
              I think it's quite easy to confuse the two if one is diagonally reading the HN frontpage and sees a headline that begins with "What's new with Robinhood [...]".
              • ajoseps 257 days ago
                maybe dropbox didn’t optimize naming their in-house software for HN readers
            • Dalewyn 257 days ago
              >(and no I don't think it's the space that's important).

              That space or the lack thereof is important.

              To date I only know of two Robinhoods, the brokerage and the load balancer, and I only learned about the second with this thread.

            • dylan604 257 days ago
              > I don't think that people will confuse said brokerage with an old character of English folklore

              I don't know about that. Why do you think people would not link a financial company to the folklore character famous for redistributing money? That's kind of disingenuous claim to me.

              When I first heard about it, I actually assumed it was a reverse Robin Hood by taking money from the poor to give back to the rich type of scheme.

          • Veen 257 days ago
            Many things are named after Robin Hood. It's not as though the brokerage invented the term or has any right of exclusive use. You'd have to be quite dim-witted to muddle up a brokerage and a load balancer.
            • Dalewyn 257 days ago
              Many things are named after Robin Hood, but I only know of two things specifically calling themselves Robinhood and one of them I only learned from this thread.

              In fairness the load balancer presumably came first, but the point remains.

          • rsynnott 257 days ago
            I think Robinhood the brokerage was _fairly_ obscure til 2020 or so, no? This appears to have gone into production in 2020, so is likely older than that.
          • bayindirh 257 days ago
            Robinhood is a large filesystem policy engine and maintenance tool since 2004, too. Soo Robinhood Markets are already late to the game. :)
      • kergonath 257 days ago
        You cannot avoid all names that were or will be ever used by anyone. Collisions will happen. And Robin Hood is not particularly obscure…
      • stavros 257 days ago
        Probably because it takes from the rich and gives to the poor.
      • almost 257 days ago
        Weird that a gambling app would take thename of a popular character from legend!

        Or maybe not, try actually naming something sometime and you'll see. It's not easy to come up with something that fits all the requirements AND has never been used before.

        And it's not like anyone is ever going to get confused and accidentally try and use Dropbox's internal load balancer to buy options on a stock or something. Or try and make a gambling app the star of their next hit movie for that matter.

        • tgv 257 days ago
          A gambling app that --like all-- takes from the poor and gives to the rich. Sarcasm, or bitter misantropy, I suppose.
      • whalesalad 257 days ago
        because it is an internal tool? why does that cause an issue?
    • xyzzy4747 257 days ago
      [flagged]
      • bayindirh 257 days ago
        There's a tool called Robinhood which started its life around 2004 [0], In turn, Robinhood Markets are founded in 2013.

        Imagine knowing that Robinhood exists as one of the most popular (or the only) policy engines for large (EB sized) filesystems for but still naming your new company Robinhood.

        I think Robinhood Markets should C&D immediately and find themselves a new name.

        [0]: https://github.com/cea-hpc/robinhood

      • pgwhalen 257 days ago
        The Robinhood being discussed in this post is not a product, it's internal technology. The name clashing doesn't really matter.
        • xyzzy4747 257 days ago
          There are near-infinite things they could name their software, but instead they copied an existing name of another software project.

          Quite insufferable if you ask me.

          • bayindirh 257 days ago
            Software developers who love their job generally are into fantasy worlds, RPG games, astronomy and whatnot. If we nitpick every name choice, then life will have no fun.

            I have computers named after stars and galaxies. Some codenames of the applications I have developed myself has stupid connections to characters because of what they (the software) do are similar to characters themselves. I named a module "Scrat" because it was a state machine which changed thousands of states per second according to input it received from a file I/O module called Manfred (which carries things on his back or with its trunk).

            I have my names clash with others, and others' names clash with mine. Who cares, it's fun and makes things enjoyable for everyone, plus it allows one to inspire themselves by setting a vision for their software.

            Or, maybe we should start naming our software with UUID strings. That won't clash in a long time. :D

          • sophacles 257 days ago
            Why does every single thing need a unique name? Even trademarked names for businesses and products are scoped to business sectors - its why apple the computer company who took the name of apple the record label was just fine (until apple invented itunes and had to buy the record label).

            Heck even code does scoped naming... the compiler can't really handle punctuation that is slightly wrong and even it allows me to reuse variable names in the same program.

            I don't know why we'd need a univerally unique name for each piece of software ever created. The existing software called robinhood is unrelated to this load balancer, so even if I could download and use it, I doubt I'd have much of a problem distinguishing the two.

          • edm0nd 257 days ago
            imo its your mindset that is quite insufferable, not the companies decisions of naming internal tooling.
      • bityard 257 days ago
        Were Facebook, Google, and Reddit also the names of centuries-old fictional storybook characters?
        • xyzzy4747 257 days ago
          Were centuries-old fictional characters software projects?
    • say_it_as_it_is 257 days ago
      [flagged]
      • qeternity 257 days ago
        Totally. They should just continue to carry a workforce they don't need, risking the viability of the business and the other 80% of the jobs.