I made https://newsagents.app and I ended up using the extract API from kagi and falling back to cloudflares browser API for problem pages. That looks to be more cost efficient than firecrawl, but I could run this and get more extraction. Search APIs dominate the expense though.
It being vibe coded aside, does it support screenshots? I noticed the daemon mode has a lot of weird limitations too like not being able to return html.
I think the biggest issue is that cloud providers turned something that's a cheap commodity (bandwidth) into something that needs to be nickel and dimed. There's a reason AWS is Amazon's most profitable division and it's certainly not because they offer the best value towards customers.
The colo pricing is a fraction of a fraction of what AWS is charging. There's a reason that piracy video and file hosting sites can survive on just ads alone. If you try building something like YouTube on AWS you will go bankrupt in two days. The big cloud bandwidth pricing is the biggest scam ever. They don't want to charge up front for the human cost of running servers so they bake it an exorbitant overhead into the variable consumption cost hoping most web devs are too stupid too worry about it (it's usually Somebody Else's Problem).
It seems the scraping itself is much of the current internet traffic problem.
On mobile and satellite you still have bandwidth caps. Sometimes on cable.
great work!