This is painful, I was building for fly.io because of their dynamic request routing but I'm thinking I'm just going to go with hetzner. The cost difference is so substantial. If I want high availability I can just get 5 servers and have that much more compute available and I'd still end up paying less.
> If I want high availability I can just get 5 servers and have that much more compute available and I'd still end up paying less.
Please give us an update here when you are done building your hetzner beowulf cluster and let us know how it fares. I think you are grossly underestimating the system you describe.
Whether this is completely out of the box or a multi-year engineering project to get equivalent levels of utility in the long term is completely up to how one specifically uses Fly.io.
E.g. a load balancer in front of some relatively stateless http microservices is really not going to tie you to Fly.io because of complexity. A globally distributed edge compute environment with database, RPC, varying scaling patterns, and environment certifications is probably not going to have the value prop impacted by this kind of minor pricing change though.
True, but managing 5 servers requires overhead. Where does the load balancer sit? What happens when it goes down? What if the entire datacenter loses comms? That is why cloud providers have AZ's that are physically separated from one another at different sites. When your boxes are spread across different physical networks, how do you connect them securely? For true HA you need a system that can self heal. Just throwing servers at the problem does not make things better. Reminds me of the expression 9 women can't make a baby in 1 month.
None of this is impossible, but when you are a small startup it makes sense to offload this to someone else and not waste time reinventing the wheel.
Don’t host anything important on Hetzner, as they’re cheap they get a lot of questionable users, and their systems are tuned to null route anything that looks remotely suspicious or high-volume to their hair-trigger network monitoring and DDoS protection.
And then you’re at the mercy of whatever support agent decides to look at your ticket and that can take days.
I disagree, at my company (elest.io) we have several thousands VMs with Hetzner and reliability is very comparable to AWS from our internal stats.
In case of DDOS attack we get notified and can discuss with their team.
Also support is competent and quick. We usually get an answer in few hours most of the time.
I’m glad you’ve had a positive experience, maybe it’s the usage level that grants you that treatment. As a one-time new client I had the worst possible experience with them: null routed my game server on launch day, very slow to respond support team, and effectively held my server hostage for 3 days. I guess I needed to inform them I was intending to use the server I paid for?
This same argument applies to most cloud hosting, non-cloud (or less cloudy, Hetzner is a little cloudy) will always be cheaper because the service is more basic in a number of ways.
Fly’s pitch is not just on demand scalable compute, but also that compute being close to the edge, and with “ops-less” deployments. All of those are factors you pay for.
You could pay a little less with a big cloud provider’s serverless platform, a bit less again with regular cloud VMs, a lot less with colo hardware, and a ton less running your own DC (at scale). Obviously not all of these work for all businesses, but saying Hetzner is cheaper than Fly misses the point of those two services on the spectrum of service levels.
This has me so confused. Should the cloud abstractions not clearly make the service more optimizable at scale and thus cheaper to offer? Sandboxing things, limiting options, not having to deal with users wanting to provision VMs and do all kinds of unexpected stuff? The marginal cost of the beautiful software to make all of that happen smoothly, and all the dx it affords, that will just go towards 0, no?
I understand that people ask for what they can get away with. But does the market fail or am I missing a piece of the cost/price puzzle?
At one side, the big cloud providers have their very fair profit margins. But I've always assumed that comes mostly from that they can, and do, charge extra for the beautiful software (not seldomly through ridiculous egress bandwidth pricing).
I'm skeptical of how much flexibility and optimisation the big cloud providers really get compared to a very large VPS provider?
All the compute products (different form of container hosting, serverless) are essentially still priced at CPU+RAM + a premium. Some additional abstraction for nouveau databases ("capacity" or "billing" units). Many of the highly abstracted offerings also have the scale-to-zero concept, while the provider still have to have physical capacity ready (to some extent).
Sure, you can probably crank out some percentage more utilisation, but that doesn't really matter for the 100+% price-add on the market bears.
But I of course have no data to back up my claims, just speculation.
The tooling for managing the "lower" layers is progressively worse / slower / harder the closer to the metal you get,
so you usually are paying a heavy premium for convenience / DX.
Only very specific serverless products where the provider can bin-pack you with other tenants allow them to possibly offer lower prices.
For a lot of people (most...?), Hetzner gives them exactly what they're looking for at an excellent price point. As an ecosystem, we never really moved past VMs, we just went up the stack more.
I'm not missing the point of the service, I'm against the idea of infrastructure being an 84% margin business. But I guess this is what the VCs demand and some are willing to pay it.
How has Flys reliability been as of late? From past discussions I got the impression that despite it being quite polished on the surface, behind the scenes their operations are a bit of a mess.
From personal experience, still not great. Sometimes things just don't deploy.
I host my little mobile braai simulator on it (braaisim.com) because it's an art project and easy to add new domains / certificates. Convenient to be able to scale up easily if it went viral in ZA. Nice that they support Johannesburg region (jnb), though, which is close to me.
You were downvoted, but as a user, I’ve experienced the same inconsistency with deploys.
There’s also a hard container size limit I’ve run into multiple times. If you add a dependency and go over the size limit, your app won’t deploy unless you switch to a GPU instance, which is substantially more expensive.
A bunch of mini outages and the main site + dashboard was out for maybe an hour last week on Thursday.
my production app has had 1 outage where stripe webhooks weren’t delivering for 45 minutes which is a huge issue for sure.
I mitigated by checking in addition to the webhook event and also it hasn’t had a problem since. Their explanation was a 3rd party networking issue at or near one of their data centers
I see two sibling comments with reports of patchy reliability, so I'll note that it's been rock-solid for me (multiple daily deploys every weekday) since I last commented on it (about a year ago?)
I get the impression that the operations are a bit of a mess too, but that support is empowered to actually make people whole, which is a nice change of pace.
Bad enough that we had to migrate our key customer-facing platform off of it because of the impact.
I love the idea and will still use it for many less-business-critical apps but after multiple late-night weird un-debuggable database outages we had to switch.
Don't know if anyone noticed, but Ben Johnson recently became head of product over there. I'm fortunate to have called Ben a friend for a long ass time now and I gotta say, I'm extremely bullish on fly as a result of this. Ben is not just a developers developer, he's genius level intelligence. https://github.com/benbjohnson
(We fumbled the ball hard at DigitalOcean ngl, fingers crossed Ben can build what I dreamed of building one day!)
There's also physical limits though, and structural economic lag in terms of investment. If you look at any CDN costs globally and compare it with the number of intercontinental fiber connections between those places you generally start having a good idea that the two are related. In some places there's also just less qualified talent (due to having less ability to train people in enough universities, or just not having them for as long) and datacenter space built and extra import costs that delay everything.
At some point everything will saturate and probably energy production costs will dominate (largely for cooling), but I think we're still a bit far from that.
Do you have numbers to support it? If you mean import of hardware that do makes sense, but what about hardware that's already here? (I'm also Brazilian)
Looking at AWS site it explains that PIS/COFINS and ISS are charged, excerpt below:
"AWS SBL can issue an NFS-e to a Bra zilian customer only after receiving a valid Tax Registration Number ("TRN") (i.e., CPF/CNPJ). The relevant taxes applicable to AWS cloud services offered locally are the following: PIS and COFINS (9.25%, combined rate) and ISS (2.9%, charged by São Paulo municipality)."
> but what about hardware that's already here? (I'm also Brazilian)
It's ridiculously expensive.
Very little hardware is produced domestically, and the little that is, you don't want to be running your switches on what is in essence a no-name "intelbras" brand.
Great!, then I'd rather my app be a littler slower for those people if it saves me money. If the latency is not noticeable, why pay more to have that edge hosted.
Recently evaluated Fly.io for a company project but decided against it. I get the vibe that it fits the hobby/side project market more and that they're still in startup mode.
So, hosting in Japan is more expensive than regions in America? JPY is in a horrible place compared to USD. That seems like a good way to push away Japanese users away -- make it more expensive in countries where the local currency is weak.
The nice thing about Fly is it’s very docker oriented, so depending on your app migrating it off may be fairly straightforward?
My apps are all on Fly but in the regions without big hikes thankfully, so I’m sticking around for now, but it sucks some people are being hit by this…
Technically you are right. However my app is using litefs. In theory you could set this up on your own servers. Which is part of the reason I was so interested in it.
However with fly it’s just one command to add a new region and my data is auto replicated there.
That’s super nice compared to manually managing this.
As long as bandwidth costs do not go up drastically I think it’ll be okay.
We looked at both and chose Render based on various posts here along with our real world testing and bake off parameters. We use the Pro Ultra config and handle ~20rps on a Node / Express stack w/ 36 workers using clustering. Our bandwidth is around 50Gb / month.
Render has been quite solid and the support has been on point when we have found issues or run into unexpected edge cases. I have been impressed that despite a big raise and subsequent scaling up, they continue to ship a solid product and the platform improvements have been useful as well.
Never used Fly.io but I have a website running on Render (2.4M unique users per month) and the service is getting quite nice. Occasional hiccups, but they are improving. Not as stable as Heroku used to be 7-8 years ago yet, but it's also waaaaay cheaper.
I really like their Docker support and infra as code, makes it very easy to spin up a whole thing while not being too far conceptually from Kubernetes for example.
> If you were on the free Hobby plan at the time that the paid Hobby plan became the default for new organizations, your plan is now called the Legacy Hobby plan. Your costs stay the same as they were, with no monthly subscription fee, and no included usage beyond the free resource allowances.
so i get to use the legacy hobby plan because i'm grandfathered in, nice. only use it for a hobby project with one instance anyways(see bio), if they really decide to start charging me i'll gladly leave.
Heya, we deprecated pretty loudly! We made a post on the community forum [0] and we sent an email to all active accounts saying the following:
> The first improvement we're excited to announce is that the $5 Hobby plan is no more. We're replacing it with a very simple Pay As You Go plan. On this plan there's no more upfront $5 charge and no minimum monthly commitment. You only pay for what you use. If you don't use anything for a given month you pay $0. You still need a credit card on file to prevent abuse. But your card is only charged if you use the service.
Sorry, didn’t intend for throwaway to come across as not standing behind the product.
I wrote the community forum announcement and was one of the people that worked on implementing our PAYG plan. I just wanted to clarify that we didn’t deprecate silently, so I quickly created an account to do so.
random side note, it would be nice if the region selector for regional pricing would display human readable region labels in addition to 3 letter acronyms.
> suspicious “fees” to get servers through customs
Is that an admission of paying bribes? Corporate training always tells me to never pay anyone unofficially, do the people at fly have no problem with compliance? I wouldn’t write that in a public blog post.
Compliance with what exactly? When you bring servers down to Brazil it gets weird, outside just not supporting that region, not much you can't do about it. We never did anything in Brazil because it was such a huge pain in the ass, it's hard to get stuff in and expensive to rack once you're there. imo Good for fly for doing the work, even if they did have to slip someone a 5 on the way in.
Compliance with ISO 37001 for example. As a big company, you can’t just go around and bribe random border guards to get your stuff through.
I’m also not sure if it is illegal to pay bribes in other countries when you’re an American company. So compliance with laws is another problem when paying bribes.
Looks like it is explicitly forbidden to bribe foreign officials as a US entity.
Not very smart to suggest you’re bribing people in other countries in official communications.
Grease payments are allowed. You can’t bribe border guards to sneak things through, but if they demand a bribe to do their job or your servers get mysteriously blocked from import, paying them off is allowed.
It's not overt, as they mentioned in their post, mystical fees just show up on your invoice from the government. I do agree tho, it's a really silly detail to include. I'm a bit passionate about this because I ran a long disability study at DigitalOcean to see if we could do Brazil.
To be fair, Brazil does have weird and expensive customs/import duties on technology products, which makes a lot of hardware expensive. Maybe fly.io just didn't know about it and that's why they say "suspicious"; But yeah, it definetely sounds like admitting to bribery, which as an American-registered company is illegal even in other countries.
Hetzner gives you bare (or virtual) metal that you have to install stuff on, and maintain it.
Fly.io and similar give you a higher abstraction layer where you throw a Docker container and the stuff below and around it is managed for you. Basic suff like OS patches, but also autoscaling, load balancing, request routing, etc.
Does the 4x difference make it up? It will depend entirely on you and how good you are with the things fly.io abstract for you, and how much you value your time spent on it.
Please give us an update here when you are done building your hetzner beowulf cluster and let us know how it fares. I think you are grossly underestimating the system you describe.
E.g. a load balancer in front of some relatively stateless http microservices is really not going to tie you to Fly.io because of complexity. A globally distributed edge compute environment with database, RPC, varying scaling patterns, and environment certifications is probably not going to have the value prop impacted by this kind of minor pricing change though.
None of this is impossible, but when you are a small startup it makes sense to offload this to someone else and not waste time reinventing the wheel.
And then you’re at the mercy of whatever support agent decides to look at your ticket and that can take days.
Use literally anyone but Hetzner is my advice.
Also support is competent and quick. We usually get an answer in few hours most of the time.
Took my data and never returned.
Fly’s pitch is not just on demand scalable compute, but also that compute being close to the edge, and with “ops-less” deployments. All of those are factors you pay for.
You could pay a little less with a big cloud provider’s serverless platform, a bit less again with regular cloud VMs, a lot less with colo hardware, and a ton less running your own DC (at scale). Obviously not all of these work for all businesses, but saying Hetzner is cheaper than Fly misses the point of those two services on the spectrum of service levels.
This has me so confused. Should the cloud abstractions not clearly make the service more optimizable at scale and thus cheaper to offer? Sandboxing things, limiting options, not having to deal with users wanting to provision VMs and do all kinds of unexpected stuff? The marginal cost of the beautiful software to make all of that happen smoothly, and all the dx it affords, that will just go towards 0, no?
I understand that people ask for what they can get away with. But does the market fail or am I missing a piece of the cost/price puzzle?
At one side, the big cloud providers have their very fair profit margins. But I've always assumed that comes mostly from that they can, and do, charge extra for the beautiful software (not seldomly through ridiculous egress bandwidth pricing).
I'm skeptical of how much flexibility and optimisation the big cloud providers really get compared to a very large VPS provider?
All the compute products (different form of container hosting, serverless) are essentially still priced at CPU+RAM + a premium. Some additional abstraction for nouveau databases ("capacity" or "billing" units). Many of the highly abstracted offerings also have the scale-to-zero concept, while the provider still have to have physical capacity ready (to some extent).
Sure, you can probably crank out some percentage more utilisation, but that doesn't really matter for the 100+% price-add on the market bears.
But I of course have no data to back up my claims, just speculation.
Only very specific serverless products where the provider can bin-pack you with other tenants allow them to possibly offer lower prices.
e.g. https://news.ycombinator.com/item?id=36808296
https://news.ycombinator.com/item?id=39365735
I host my little mobile braai simulator on it (braaisim.com) because it's an art project and easy to add new domains / certificates. Convenient to be able to scale up easily if it went viral in ZA. Nice that they support Johannesburg region (jnb), though, which is close to me.
There’s also a hard container size limit I’ve run into multiple times. If you add a dependency and go over the size limit, your app won’t deploy unless you switch to a GPU instance, which is substantially more expensive.
my production app has had 1 outage where stripe webhooks weren’t delivering for 45 minutes which is a huge issue for sure.
I mitigated by checking in addition to the webhook event and also it hasn’t had a problem since. Their explanation was a 3rd party networking issue at or near one of their data centers
That’s about every early stage company, though.
I love the idea and will still use it for many less-business-critical apps but after multiple late-night weird un-debuggable database outages we had to switch.
(We fumbled the ball hard at DigitalOcean ngl, fingers crossed Ben can build what I dreamed of building one day!)
There is an interesting dynamic of server prices and customer spend in different parts of the world.
Usually where the server prices are high, customers spend less.
At some point everything will saturate and probably energy production costs will dominate (largely for cooling), but I think we're still a bit far from that.
Looking at AWS site it explains that PIS/COFINS and ISS are charged, excerpt below:
"AWS SBL can issue an NFS-e to a Bra zilian customer only after receiving a valid Tax Registration Number ("TRN") (i.e., CPF/CNPJ). The relevant taxes applicable to AWS cloud services offered locally are the following: PIS and COFINS (9.25%, combined rate) and ISS (2.9%, charged by São Paulo municipality)."
https://aws.amazon.com/tax-help/Brazil/
This is below 15% which many countries do tax for cloud services.
I'm not saying that Brazilian taxes are really high, they are for hardware imports, but the taxes into the service itself does seem "fair".
It's ridiculously expensive.
Very little hardware is produced domestically, and the little that is, you don't want to be running your switches on what is in essence a no-name "intelbras" brand.
It would be nice to have stable pricing and not have a fear of different features costing more out of the blue.
Other than this, fly has been a great service, I love the system, the CLI and litefs.
It scares me that internal bandwidth charges will start being added, the application I’m using has a lot of internal transfer which is currently free.
"Any traffic exiting a Fly.io Machine is considered outbound data transfer, including:
- traffic destined for the internet - traffic destined for other Machines or apps in your network"
Edit: maybe you are right actually. I have to check on this closer. App is just about to launch so I hadn’t paid so much attention
My apps are all on Fly but in the regions without big hikes thankfully, so I’m sticking around for now, but it sucks some people are being hit by this…
However with fly it’s just one command to add a new region and my data is auto replicated there.
That’s super nice compared to manually managing this.
As long as bandwidth costs do not go up drastically I think it’ll be okay.
Render has been quite solid and the support has been on point when we have found issues or run into unexpected edge cases. I have been impressed that despite a big raise and subsequent scaling up, they continue to ship a solid product and the platform improvements have been useful as well.
I really like their Docker support and infra as code, makes it very easy to spin up a whole thing while not being too far conceptually from Kubernetes for example.
I hope they don't cancel this plan, currently on it :D
EDIT: Ohh, it's only visible when I'm logged in, weird...
EDIT2: ...indeed
> The paid Hobby plan and the Legacy Hobby plan are not available for new sign-ups.
https://fly.io/docs/about/pricing/
damn :(
> If you were on the free Hobby plan at the time that the paid Hobby plan became the default for new organizations, your plan is now called the Legacy Hobby plan. Your costs stay the same as they were, with no monthly subscription fee, and no included usage beyond the free resource allowances.
so i get to use the legacy hobby plan because i'm grandfathered in, nice. only use it for a hobby project with one instance anyways(see bio), if they really decide to start charging me i'll gladly leave.
> The first improvement we're excited to announce is that the $5 Hobby plan is no more. We're replacing it with a very simple Pay As You Go plan. On this plan there's no more upfront $5 charge and no minimum monthly commitment. You only pay for what you use. If you don't use anything for a given month you pay $0. You still need a credit card on file to prevent abuse. But your card is only charged if you use the service.
[0]: https://community.fly.io/t/fresh-produce-pay-as-you-go-plan/...
I wrote the community forum announcement and was one of the people that worked on implementing our PAYG plan. I just wanted to clarify that we didn’t deprecate silently, so I quickly created an account to do so.
Apologies for any confusion this caused!
Is that an admission of paying bribes? Corporate training always tells me to never pay anyone unofficially, do the people at fly have no problem with compliance? I wouldn’t write that in a public blog post.
I’m also not sure if it is illegal to pay bribes in other countries when you’re an American company. So compliance with laws is another problem when paying bribes.
Edit: https://en.wikipedia.org/wiki/Foreign_Corrupt_Practices_Act
Looks like it is explicitly forbidden to bribe foreign officials as a US entity. Not very smart to suggest you’re bribing people in other countries in official communications.
Grease payments are allowed. You can’t bribe border guards to sneak things through, but if they demand a bribe to do their job or your servers get mysteriously blocked from import, paying them off is allowed.
I’ve heard the same thing about their core product.
Oh wait, they even acknowledged it: https://community.fly.io/t/reliability-its-not-great/11253
Fly.io and similar give you a higher abstraction layer where you throw a Docker container and the stuff below and around it is managed for you. Basic suff like OS patches, but also autoscaling, load balancing, request routing, etc.
Does the 4x difference make it up? It will depend entirely on you and how good you are with the things fly.io abstract for you, and how much you value your time spent on it.