Ask HN: Web scraping in production?

Are any of you maintaining any web scrapers in production?

I've done some for side projects, automated testing, and personal scripts (checking personal bank balances, getting a Global Entry interview slot, etc.), but it always feels very brittle.

Curious what applications people have in industry and what sorts of techniques people use for reliability.

3 points | by arkmm 2 hours ago

2 comments

  • 9d 2 hours ago
    Scraping is inherently brittle, but it can be very useful for short-term scraping in very specific circumstances. I haven't had any in maybe 10 years.
    [-]
    • sargstuff 2 hours ago
      IMHO, "untyped" format/delimited file yes. Directly placing/'compiling' in appropriate topological construct/environment works wonders. aka environment of database, spreadsheet, "reports" with information beyond raw data, etc
  • sargstuff 2 hours ago
    excel web scraping[0] (vs. using python[1] and/or odbc/delimited files)

    A few 2025 use cases [2],[3]:

       Use publically available database information (construction, taxes, sales, traffic report, proposed building/zone changes, etc) to find out what's going on withing an area aka. zip code, housing area, 'vacation spot', etc
    ----

       creative take on topic:

      modern looming / static 'threaded' approach : https://news.ycombinator.com/item?id=43977384

      Structurally reprogrammable magnetic maetamaterials hold promise for biomedicine, soft robotics. ("web" support formed via scraping material in relevant patterns) : https://techxplore.com/news/2025-05-reprogrammable-magnetic-metamaterials-biomedicine-soft.html

      3d printed smart-fabrics : https://techxplore.com/news/2025-05-d-smart-fabrics-flexibility-ability.html
    ----

    [0] : excel scraping : https://www.youtube.com/watch?app=desktop&v=6coVzIt93vk

    [1] : python scraping : https://www.youtube.com/watch?v=Oo8-nEuDBkk

    [2] : https://dataforest.ai/blog/top-web-scraping-use-cases

    [3] : https://www.parsehub.com/blog/web-scraping-examples/

    [-]
    • arkmm 2 hours ago
      Neat - didn't realize there were affordances for scraping in Excel (but in hindsight I shouldn't be surprised).

      I didn't follow the connection between modern looming and scraping though?

      [-]