TF Grid: Billing Discussion

[Mik]

Hi there!

There have been some great discussions on the Threefold Grid Tester Community channel on Telegram.
In short, there are some questions and concerns about billing.
Many things have been said in the chats, so I will try and share some of those ideas.

If you have some more things to say, or if I forgot something, let us know and we will update this post.

---

Some Information on TF Billing

First, users need to know that there has been a change in the Threefold billing process for utilization.
You can read more here: https://github.com/threefoldtech/zos/blob/main/docs/architecture/decisions/0001-nu-reporting-for-priv-traffic.md

• From now on, traffic to the public internet is counted, and reported as NU consumption. Not only the traffic over public ips.
• All traffic generated by the VM to public internet will be billed by the user. Even if the VM has no public IP.

---

Some Questions and Concerns on TF Billing and Related Notions

• It is hard to know the cost of deployments
• We’d need more details about hour rating, e.g. make it easier to track by users (bandwidth + basic deployment cost)
• Using mUSD in the TF website can be tricky as it is not common to use
• Users would benefit from having clear comparison tables to see the price of deploying workloads on the TF Grid, compared to other service providers
• Could locked TFT value during deployment be possible? Or some variations. See this post.
  • this would be easier to track workload costs, it would also avoid major price changes (see the +/- 100% price change in the last days)
• Explain clearly how rewards for utilization work, how farmers get rewards
  • if the farmers have blocks of IPv4 public IP vs IPv6 and Planetary Network
  • farmers get TFT rewards from utilization based on network unit (if they have public IPv4 addresses available), not compute or storage unit
• Can a part of the Proof-of-Utilization rewards go to farmers?

---

This really is a kind of draft to get the Telegram discussions into the TF Forum.

Please speak your mind if you have something to share

[azmy]

Thanks @Mik for summarizing the discussion. And indeed all traffic now is being reported and monitored as NU units. This makes it very hard to predict the cost of a deployment in advance.

Normally it’s easy to calculate the cost of the reserved capacity since these are fixed (Cpu, Memory, and Storage) unlike network. May be someone can provide a price calculator where you can input your expect network capacity and then give an estimate price?

Anyway, I wanted to add one small thing to the discussion, which is what is a fair price? Before adding this billing it was unfair for the farmer because users on the grid can utilize his entire network, leaving nothing to other users and can cause his monthly internet bill to increase. So adding this was necessary to grantee fair usage of the network. But may be the price can go lower?

Also we can price traffic over reserved public IP at different rate than other network traffic (which include yggdrasil and traffic over the user private network).

I think this should be left to the community to vote on and decide.

By the way, the problem with private network traffic is that we can’t be sure if this traffic stays within the same farm or leave to the public internet hence billing at a lower price makes more sense to me

[nooba]

Thanks @Mik for this post. I will give my comments bellow:

" It is hard to know the cost of deployments"

• We’d need more details about hour rating, e.g. make it easier to track by users (bandwidth + basic deployment cost)

I would say it’s impossible because the NU don’t show up seperatly on playground. I am not also sure what is the sourcing of price of TFT in any given period… to add to this it’s not clear to me if the billing period is aligned per deployment or it depends on the hour:minute that the deployment was made. Sometimes I see little differences in some VMs, and I assume that maybe the billing update period was not the same or the network traffic is causing the differences. I would appreciate there was more information, after all we are all beta testers here and you are not really sure if you have an issue to report or not.

[ParkerS]

In the telegram discussion last night, a user reported having three exact copies of the same vm and 2 had the same price but one didn’t.

My question, when is the price of determined in the billing cycle currently, is it checked each hour?

[nooba]

“Using mUSD in the TF website can be tricky as it is not common to use”

Yes. after some math i realized the actual pricing ( if I get it right) is 50USD/TB! I am not sure if users/farmers realize the impact this could have on deployments total pricing.
If this is good to farmers? depends, on your perspective. If pricing is too high probably you will get no deployments (most nodes are in this situation right now). I would prefer a model with a way cheaper USD/TB ratio where you actually could farm something.
I am not sure what happens all over the world. Traffic in the country I live is unlimited so it doesn’t represent additional cost to me… but not sure other parts of the world tbh.

Also, in the recent calculator provided on portal there in no single mention, or form field that you can simulate traffic, IPV4 addresses, all things related to networking. i know this information is available else where, but for simplicity, Why isn’t this info here?

[nooba]

"Could locked TFT value during deployment be possible? Or some variations. See this post.

• this would be easier to track workload costs, it would also avoid major price changes (see the +/- 100% price change in the last days)"

I have made my contribution to this post already. I would love that forum users could get involved on this improvement ( in my opinion) and the importance it could have to the grid.

[FLnelson]

This is another layer of complexity, but farmers could set their cost per TB since is varies so much. I have no data cap for example. Meanwhile Comcast has a cap of 1.2 TB. “After that, blocks of 50 GB will automatically be added to your account for an additional fee of $10. Charges will not exceed $100 each month, no matter how much data you use.”

[nooba]

" Explain clearly how rewards for utilization work, how farmers get rewards

• if the farmers have blocks of IPv4 public IP vs IPv6 and Planetary Network
• farmers get TFT rewards from utilization based on network unit (if they have public IPv4 addresses available), not compute or storage unit"

I’ve had some conversations before with farmers that are providing IPv4 public addresses to the Grid, I was trying to decide if I should also provide some public IPs to the grid. For what I understood (again correct me if I am wrong) you’ll get no rewards (although in the farmer simulator you have a little reward) for providing a public IPv4 address to the grid, you just earn for it’s utilization. I really don’t understand why comparing it to the current farming model (I mean CPU, RAM, SSD and HDD).
For comparison, you add a node to the grid (probably some already owned hardware) and you get farming rewards even if your node is with zero workloads - in that scenario the only cost you’ll have with that node is electricity. But if you provide public IPv4 addresses, you will be billed extra for that IP address each month, and you could still have no workloads, so zero rewards from it!
This lead us to another topic, the pricing of IP addresses on the grid. An IP address costs about 3USD a month to a user on the grid, and if I am right, all traffic is billed extra. What is the reward from that IP address to the farmer assuming that this node VM has minimal traffic?
I believe this is the main reason of the lack of public addresses on the grid - there isn’t a fair reward for it and no true incentive for getting them.

[nooba]

I apologize in advance if some of the facts in my posts are not 100% correct. If there is something not right I would appreciate your correction.

[ParkerS]

I fully agree with this, as of now the reward structuring for providing public ipv4 and above and beyond network is pretty small.

I have a 128 address block. The way I recieve reward is

• 100 tft per month per ipv4 address assigned to a node as a gateway address. This is the kind that is in use all the time

• For my ipv4 addresses that are deployable with workloads

  • My understanding is that I receive rewards for each hour it’s reserved and for each gb of use. I’m not sure how much of the cost of these things the farmers get.

I’m not sure why we don’t capacity rewards for ip addresses, I am unable to use those addresses as soon as they are added to the grid, and they are quite expensive, not mention the incredible amount of work it takes to build the network infrastructure and get everything setup.

No rewards for link speed so my 5 gig would be the same price as 1gig. Whitch it isn’t working properly so they are all 1gig each anyways.

[ParkerS]

I really like the idea of a farmer being able to turn off bandwidth cost, that would be a huge way to make a farm competitive and I’m unmetered to true unlimited so I have no need of those rewards if not having them would encourage workloads.

I did some in depth reading on how most cps handle with found an incredible resource on the topic on serve the home

It’s seems most only bill for outbound traffic, which seems reasonable. And most have a included amount of all data. I think 50$ per tb is too much, but there definitely needs to be a way for a farmer to ensure his bill doesn’t get sent to the moon with no rewards.

@azmy can you speak to what prompted the change in billing, was there an incident where a problem occurred or was this a change based off foresight of possible problems?

[nooba]

From the farming simulator i cannot get any conclusion about farming rewards for public IPs. No 100 TFT, and I don’t have any option to input the number of IPS here. Or does “Public Ip” here means gateway to the internet and not public IP assigned/rented to specific VM?

[ParkerS]

Yea that check mark is for gateway address and I thought I remembered it adding 100tft,

The rewards for ip that are rented with vms belong to the farm and not any specific node within the farm, so I imagine that is why that is not included as you wouldn’t add ips for each node and there isn’t rewards without use.

[azmy]

@ParkerS
The change started because of an internal discussion in the office with @delandtj I was wondering why we bill only public ip traffic and not any traffic generate by the VM? and the answer i got was that it’s technically harder to know which traffic to monitor and count.

So to me this was a completely technical problem, i didn’t look or consider the effect on the billing. ZOS is not concerned with billing, it is concerned only on providing real and correct consumption reports, then it’s up to the chain later to decide what to do with this.

While changing the system to also report the NU consumption of traffic I added fine-tuning knobs to control the following:

• ratio of inbound traffic and outbound traffic
• ratio of (number of packages) also in both direction
• ratio of public to private traffic

It means the following:

• I can control if only inbound or outbound traffic is reported, or if they are reported at different rates. Say full outbound traffic, and only 50% of inbound traffic
• I can control if only public IP traffic is reported, or Private (ygg+wg) or both and also at what rate

So reverting back, or using a different rates is matter of few knobs i can tune. I just need to know a decision what to do.

[ParkerS]

Ahh I see, this provides a much better understanding. If the change was unintended, then we should probably made sure we’re only booking for what is documented currently, then open a vote on setting some new rules with the above information.

I’m reading I found most providers primarily bill for outbound traffic, and there is a usually a sizeable data allowance per month,

In a perfect world I would love to see us bill for outbound traffic that is leaving the lan over a certain amount say 1.3gb per hour (1tb per month) for example

What we bill is the next question so we need to determine what an average gb/tb of bandwidth should cost.

[azmy]

Does this mean i need to revert now to the previous state, then agree later on the correct way?

[ParkerS]

I think what you did was an improvement to the accounting of zos, but had an unintended consequence of affecting billing.

I think the important part is that we make sure our billing is as documented, the trouble becomes, I don’t know of anywhere this was/is documented as to what ips bill.

I think the best route would be to ensure as of now we are only billing for ipv4 which I feel like was the general understanding.

Then we should open a dao proposal for a formal bandwidth billing model

If we can do all of that while keeping your changes, that would be even better because this a more accurate accounting system technically.

[azmy]

yes, i am creating and deploying a mainnet patch to disable the reporting of private traffic for now (only keeping public IPv4) so in couple of hours this should be effective on mainnet

[nooba]

I am not sure if there was some update to this page https://library.threefold.me/info/threefold#/cloud/threefold__pricing
or maybe today this seems more clear to me!
Even though I would approve a reduction on transfer billing over IPv4 and propose something symbolic for traffic over planetary (YGG), I don’t think it’s clear in the “pricing page” that transfer is only billed over IPv4. For my understanding their is no distinction, only “bandwidth as used by TF grid users”
@ParkerS can you share (again) the link saying that YGG traffic would be charged starting in october or november ?

[nooba]

Yeah, you’re right. Your idea, makes sense, but would probably add some more complexity to the system… not sure if it should be a main concern at the moment, as there are more urgent stuff to work right now.
But I would probably do it by country or region and not based on node - this would be more in line of what other cloud providers do (in same casea the pricing varies based on datacenter and/or region) and this also would make simpler for users. I can imagine the pricing calculator adding a region combo box, and depending on you region of choice show up the pricing / or include bandwith. I don’t like the ideia of a calculator showing up pricing per node - mostly because it would be adding another layer of complexity IMO.