3 comments

  • noduerme 6 hours ago
    I love the idea, somehow, that these systems had to be built with domain knowledge integrated into the choices of layers and the order of algorithms, using rather brilliant discoveries for optimizing lines and edges, or whatever was relevant to the problem. That's an order of specificity that's lacking from our modern reliance on "one shot" solutions. Letting a one shot solver grind its brains against creating a custom solution to a specific set of transformations is kind if a closing of the circle when it comes to performance and resources for a certain task.

    I'm looking forward to the day we can have an optical device that runs only on light input, that looks like a piece of quartz, that you can place over a handwritten number (0-9) and on the top it displays the number in a clean font. Where the neural network to handle this can be translated into physical transformations once it's discovered, no inference processing power required.

    • andrewbutts 5 hours ago
      Here’s a 2024 project that built a denoiser out of passive optics! https://nips.cc/virtual/2024/poster/95165
    • imtringued 1 hour ago
      How to defeat the bitter lesson 101: Use AI's domain knowledge as a substitute for human domain knowledge.

      The mistake was thinking that domain knowledge does not scale alongside the generalized scaling AI.

      • consensus1 1 hour ago
        Sounds like that is just another bitter lesson, not the refutation of it.
  • agnosticmantis 7 hours ago
    Thanks for putting this together. Very nicely demystifies the weight-space programming phenomenon which I otherwise found quite magical and mysterious.

    It's not that the models magically "intuit" meaning from a blob of weights, but they methodically iterate on setting the weights with reasoning similar to a human engineer's reasoning.

    • consensus1 1 hour ago
      But how did it find the right neurons to iterate on in the massive multi terabyte weight space?
  • robrenaud 9 hours ago
    Would PPO learn quickly from the hand engineered solution? Is PPO getting a terrible init?

    It's interesting to see how tolerant of tedium the models are.

    • AshotSanamyan 4 hours ago
      Interesting question. My guess is it would learn faster at first, then drift away from the hand-built solution, unless you somehow hold it close for the first part of training. Would love to see someone try it.