Looks like some solid methodology on estimating bandwidth requirements. Getting those constants right on mbps for HDD, SDD and RAM is super important. Somebody out there must have a strong answer. I’ll do some research on it as well. 1mbps for a titan seems awfully low for normal needs, but I suppose there are different levels of service to be expected across geographical regions.
Storage / Bandwidth Ratio
I suppose there are different levels of service to be expected across geographical regions
Yes good point. Also we could have a range of bandwidth that would be tolerable for the Grid. Say you exceed the bandwidth, it’s just even better but maybe useless. If you go below, you would be penalized in terms of farming.
Here’s something I wrote to @scott that could be a template of a general answer for the community:
"
Message concerning the Internet Bandwidth for Farming (Titan/DIY)
-
A 3node Titan needs a mimimum of 1mbps internet bandwidth.
-
For many Titans (2 to 5) or a DIY 3node with up to 30 TB of storage/8 vcores/64GB ram, a high speed internet is sufficient. (Between 10 to 100 mbps).
-
A 1 gbps connection is necessary for farms with multiple high capacity 3nodes such as 16-32vcores/128-256 GB ram servers (from 1 to 20 servers) or multiple titans (6 to 40).
NB: Details on the necessary bandwidth for cultivation use cases are yet to be determined with further precisions.
A farm with a specific setup of 3nodes having an obviously too low bandwidth compared to the 3 examples above would be considered as an abuse of the farming system that would be limiting the farm efficiency in period of cultivation (on Grid demands). If such case of abuses were to arrived, the DAO could penalize the abusive farmer by slashing/reducing the farming rewards to a ratio proportional to the bandwidth capacity.
"
Last sentence is just a proposition but the details would need to be discussed. Clearly these 3 examples have gaps and do not covert smoothly the spectrum. With the help of the community, we could rapidly come up with a general answer that will guarantee good Grid capacity in the future. And we’ll have time to settle the details.
All - thank you all for the contributions. Let’s collect all the input and start making a model based on the information discusses that works like a simulator. I am quite fluent in spreadsheets (not so much in coding) so I’ll start with a sheet to share. Please allow me to spend some time over the weekend on this - my rest of the week is pretty busy. 
4 Likes
That’s great. So meanwhile I added a general question + answer in the FAQ for Farmers. This will serve as a general statement until we have more details.
Here it is if someone wants to change anything:
Q: What is the bandwidth needed when it comes to running 3nodes on the Grid?
[This answer is temporary and aim to give a general idea. The community will soon have clearer and more official documentation.]
- A 3node Titan needs a mimimum of 1mbps internet bandwidth.
- For many Titans (2 to 5) or a DIY 3node with up to 30 TB of storage/8 vcores/64GB ram, a high speed internet is sufficient. (Between 10 to 100 mbps).
- A connection of 1 gbps is necessary for farms with multiple high capacity 3nodes such as 16-or 32 vcores and 128 or 256 GB ram servers (from 1 to 20 servers) or multiple Titans (6 to 40).
NB: Notice the gray areas between the three propositions. This goes to show that prices numbers are not yet determined. Using those numbers as a guide will set you in the good direction.
3 Likes
Hey guys,
Do we have any news on this subject?
The bandwidth needed per farm will be an important notion when utilization is there and farmers must be ready and be sure that they are not oversizing their farm for their available ISP bandwidth.
The synthesis above I drafted is too basic of course. Many people brought the subject in the TG channels.
A kind of simple ratio to be scaled could be more useful for the time being.
What about saying
1TB SSD + 64 GB + 8 threads --> needs an ideal minimum of 10 mbps
*if you have another number in mind, instead of 10 mbps, please share your thoughts.
The left side is the basic unit based on the most profitable farming ratio. This is like saying each 16 CU needs 10 mbps and we already know that each 16 CU will need a minimum of 1 TB SSD. We can suggest that farmers will most likely build 3nodes configurations that are closest to this ratio, as it gives more rewards. Per Game Theory logic, it would be sound to conclude that farmers will in general aim at this ratio.
But then, why 10 mbps? Because it’s 10 times more than the minimum we ask for a Titan, and that a Titan is 8 threads and 32 GB. Here we have 64GB of RAM. If someone has another idea for the quantity of mbps, speak your mind.
Then we take this ratio and take the maximum result with the example of a 5 TB, 384 GB ram, 48 threads server, we get this:
I use this example as it’s @Valentine969’s really nice build that is kind of on the upper spectrum of big server.
A big farm of 4 servers would need here 240 mbps.
An big farm case scenario with 16 servers would need a 1 gbps.
The Bandwidth Minimum per TF Farm Equation
min Bandwidth per Farm (mbps) = (# of 3nodes) * k * max_per_3node((Total SSD TB / 1 Tb),(Total Threads / 8 Threads),(Total GB / 64 GB))
with k = 10 mbps, we have:
min bandwidth = (# of 3nodes) * 10 * max_per_3node((Total SSD TB / 1 Tb),(Total Threads / 8 Threads),(Total GB / 64 GB))
Here’s the case with 16 serves:
16 * 10 * max(5, 6, 6) = 960 mbps
960 < 1000
With 17 servers in this case, you’d need more than 1 gbps bandwidth.
In the equation, the function max(…) ensures that you do not under scale.
What do you guys think?
We can have a range of this variable. Instead of 10, it’s between 8 to 16 mbps (for example).
Of course, with empirical data, we’ll know more of the bandwidth needed in real-case scenario.
Let’s New Internet!
EDIT: We’d also have to add the HDD storage in there. What do you suggest as a basic unit ratio?
OK and then what about this to take into consideration the HDD storage:
min Bandwidth per node (mbps) = k * max_per_3node((Total SSD TB / 1 Tb),(Total Threads / 8 Threads),(Total GB / 64 GB)) + k * (Total HDD TB / 2)
With k = 10 and a 1 servers with 1 TB SSD, 64GB ram, 8 Threads and an additional 8TB HDD:
min bandwidth = 10 * (max (1,1,1)) * + 10 * ( 8 / 2 ) = 10 * 1 + 10 * 5 = 50 mbps.
More concise equation:
min. Bandwidth per 3node (mbps) = k * [ max((Total SSD TB / 1 Tb),(Total Threads / 8 Threads),(Total GB / 64 GB)) + (Total HDD TB / 2) ]
where k is the mbps factor. So if k = 4, it’s 4 mbps * […]
I think 10 mbps could be realistic, but it can be modified.
Let me know what you guys think!
Hi @Mik. Thank you for your effort to get some consensus around this topic. I have not found the time to collect all the presented materials in (this) thread(s). Let try to do this collectively. Reading up and down a thread is not the easiest, and it’s hard to retain the essence of the discussion and what we have learned. So let’s review all the WHY
- Bandwidth is a source of income for farmers. Usage of bandwidth generates farming rewards for farmers. Please check here. So when a farmer provides good usable bandwidth he will attract usage and this results in higher farming rewards, for bandwidth usage and IPv4 (if applicable) usage.
- the farming bandwidth requirement should prevent for farms from having too little bandwidth to render them useful and still be rewarded for farming CU and SU.
So the why for having bandwidth requirements is to create usable farms. Capacity consumers will at some point see the amount of bandwidth available to a farm and can therefore decide where to process or store their data. For farmers that provide good bandwidth, this will increase their farming rewards. With regards to “what is the minimally acceptable bandwidth” to be able to have a farm getting farming rewards, we have to also take into account the geography. Some regions in the world have affordable Gbps into the home, others should be thankful of they get a few Mbps to the home and then there are the underserved parts (in terms of infrastructure / in particular network access).
So before talking hard numbers, how do we get an idea of what is an acceptable bandwidth level based on geography? Is anyone familiar with a sight that tracks network speed/penetration per capita or something similar? Let’s try to find a good source for that first.
1 Like
Thanks @weynandkuijpers that helps to clarify. Clearly I didn’t get everything correctly.
Good idea to have “acceptable bandwidth level based on geography”.
I’ll try and do some research on this. Here’s some idea:
So with the equation:
min Bandwidth per node (mbps) = k * max_per_3node((Total SSD TB / 1 Tb),(Total Threads / 8 Threads),(Total GB / 64 GB)) + k * (Total HDD TB / 2)
We could adjust the parameter k depending on the “acceptable bandwidth level based on geography”.
In the simulator, we could have a parameter : geographical region, with main regions to select. Then selecting one region, the simulator would adjust the k parameter and it would tell the bandwidth needed for the 3node’s specs in the simulator.
This could be a good starter.
Also just to shoot ideas: having k = 10 could be for high bandwidth region, and k = 1 for low bandwidth region (as 1mbps for 1 titan is the minimum requirement as stated by Threefold). Then there could be steps in between.
Of course, the equation can change. It’s just to brainstorm.
Amazing discussion. Thanks for reading and let’s NEW INTERNET!
EDIT: I’ll start some data collecting and post some here so we get a general idea of the situation.
Objective: find acceptable bandwidth level based on geography
Documentation and links for research (big picture)
- Wikipedia : List of countries by Internet connection speeds link
- Visual Capitalist : Mapped: The Fastest (and Slowest) Internet Speeds in the World link
- Speedtest Global Index link
1 Like
Thank you - very helpful. This created a rabbit hole for me (which I stuck my head in for a little bit). So from the wiki, you can find out that they get this information from: https://www.speedtest.net/. This well known authority in terms of internet speed checking also provides command-line interface to their test software: https://www.speedtest.net/apps/cli
Installed it, Linux binary, and execute it:
{
"download": 93509230.46298379,
"upload": 95016876.45176126,
"ping": 5.024,
"server": {
"url": "http://speedtest1.etisalat.ae:8080/speedtest/upload.php",
"lat": "25.2500",
"lon": "55.3000",
"name": "Dubai",
"country": "United Arab Emirates",
"cc": "AE",
"sponsor": "ETISALAT-UAE",
"id": "17336",
"host": "speedtest1.etisalat.ae:8080",
"d": 1.7183145989433646,
"latency": 5.024
},
"timestamp": "2022-05-12T04:23:32.177313Z",
"bytes_sent": 118841344,
"bytes_received": 117319962,
"share": null,
"client": {
"ip": "2.51.127.183",
"lat": "25.2633",
"lon": "55.3087",
"isp": "Etisalat",
"isprating": "3.7",
"rating": "0",
"ispdlavg": "0",
"ispulavg": "0",
"loggedin": "0",
"country": "AE"
}
}
It creates a very nicely structured json dataset. This was from my laptop: what if we run this in a z-machine on the grid… :-). From freefarm in Belgium:
{
"type": "result",
"timestamp": "2022-05-12T04:41:36Z",
"ping": {
"jitter": 0.078,
"latency": 4.675
},
"download": {
"bandwidth": 76093440,
"bytes": 537762760,
"elapsed": 7207
},
"upload": {
"bandwidth": 109309476,
"bytes": 427469170,
"elapsed": 3904
},
"isp": "Telenet",
"interface": {
"internalIp": "185.69.166.156",
"name": "eth1",
"macAddr": "0E:14:94:61:B7:4C",
"isVpn": false,
"externalIp": "185.69.166.156"
},
"server": {
"id": 15212,
"host": "netgauge.telenet.be",
"port": 8080,
"name": "Telenet BVBA/SPRL",
"location": "Mechelen",
"country": "Belgium",
"ip": "195.162.192.137"
},
"result": {
"id": "e2bdfa7f-e89f-4884-834f-54c139e95b47",
"url": "https://www.speedtest.net/result/c/e2bdfa7f-e89f-4884-834f-54c139e95b47",
"persisted": true
}
}
Nice bandwidth @delandtj (10Gbps up and down) . Cool so this works out of the box (weird difference in the speedtest binary that on ubuntu I had to accept a license fist, on Archlinux it just did what I asked it to do… anyway).
The real test if we have a 3node that does not have a public IP address… I choose node 67 in the US. Spec-wise it looks like a titan, so it should exist somewhere in a home in the US. I choose to have access over the planetary networks and specify a really small Z-machine (1 core, 512 MB memory).
After deploying the z-machine and installing the software this is the result (deleted specifics of the Titan and Z-machine):
{
"type": "result",
"timestamp": "2022-05-12T04:56:00Z",
"ping": {
"jitter": 2.737,
"latency": 14.062
},
"download": {
"bandwidth": 28531416,
"bytes": 242930960,
"elapsed": 8604
},
"upload": {
"bandwidth": 1405157,
"bytes": 8803184,
"elapsed": 6305
},
"packetLoss": 0,
"isp": "**********",
"interface": {
"internalIp": "***.***.***.***",
"name": "eth0",
"macAddr": "32:AE:DB:11:AE:E6",
"isVpn": false,
"externalIp": "***.***.***.***"
},
"server": {
"id": 16974,
"host": "speedtest.west.rr.com",
"port": 8080,
"name": "Spectrum",
"location": "Los Angeles, CA",
"country": "United States",
"ip": "***.***.***.***"
}
}
So we can find out per installed machine that the download and upload speeds are by which the machine is connected.
Just some playful hacking
2 Likes
Could you run this test on 3081 for me?
1 Like
{
> "type": "result",
> "timestamp": "2022-05-13T03:11:09Z",
> "ping": {
> "jitter": 1.052,
> "latency": 17.976
> },
> "download": {
> "bandwidth": 117736312,
> "bytes": 1048621576,
> "elapsed": 9000
> },
> "upload": {
> "bandwidth": 117139141,
> "bytes": 1107368316,
> "elapsed": 9609
> },
> "packetLoss": 0,
> "isp": "AT&T U-verse",
left the rest out
1 Like
Thank you! Been trying to figure out of the nics we’re acting up, or it was just a weird status light abnormality.
That answers it!
1 Like
Impressive BTW - 1Gbps down and UP.
1 Like
If you could hit all three nodes we could test the whole link… I actually don’t have any user devices capable of doing that
2 of my nodes have 10 gig nics but they are only grabbing links not sure if it’s my structure cables or a device/software issue kinda a gremlin
This is amazing.
This is starting to get interesting.
Thus we have (1) a way to check available bandwidth speed. And we are (2) currently working on finding proper bandwidth per specific 3nodes depending on geographical region.
Proposition: Region parameter + Bandwidth needed per 3node
We could have, on the simulator, an added parameter to enter the geographical region (via a dropdown menu we can select a region). When this data is entered, through a precise equation yet to be settled by the TF community, the simulator could state: you need x mbps download and y mbps upload bandwidth for your region for this precise 3node configuration. Make sure you have the correct bandwidth from your Internet service provider (ISP). You can also verify your 3node bandwidth with the TF one click bandwidth app (see next section).
TF One click bandwidth reader app:
A one-click app on the TF Grid that can test your 3node bandwidth based on speedtest software as @weynandkuijpers just showed above. (Maybe it could be in the form of a telegram bot as @scott did with the Farmer Status Bot.)
With this, we have the basic to make sure the Grid has a bandwidth level that is of highest standard possible based on available resources for each geographical region.
Main points
- Collect data of bandwidth available per region
- Build equation that takes 1. as an input and gives the bandwidth needed per 3node as an output
- Optional*: a one-click app/simple bot to verify that the 3node has the correct bandwidth
*The basic option is for the farmer to get the proper bandwidth package from the Internet Service Provider.
Proposition for the next steps of this project
-
Find the bandwidth equation per region based on the speedtest bandwidth per region database.
-
Add the region + bandwidth parameters on the simulator.grid.tf
-
Build an easy to use app (or bot) to test the bandwidth of a 3node
Note for 1.: With the basic equation I proposed before, we could simply adjust the k parameter based on regions’ available bandwidth, with a method/data partitioning to be determine that would give sound k parameter per region. I’m currently playing with some speedtest’s data and spreadsheets to find a proper method. If anyone wants to explore equation possibilities, it would be great.
Note for 3.: Of course 3. is not essential, as people can simply get the proper package from their ISP to have the correct bandwidth. That being said, it would be a precise tool for TFarmers to make sure they have the proper speed per 3node (and it would show how cool and polyvalent TF Grid is). @ParkerS 3node is a good example of an amazing bandwidth speed for the TF Grid.
What do you guys think?
Very nice discussion.
1 Like
This could be very interesting when it comes to clustering peer groups as the network grows. Obviously geography will have impact on inter-node links, but, for instance each of my nodes should be able to do that at the same time. With ygg, and a readily sortable database of node location and link speed we can sort coaster so the best link supports the other, kind like they take there stuff to the post office and it gets airmailed the rest of the way…
1 Like
Hey guys,
Continuing on this discussion, I recall the equation:
min Bandwidth per node (mbps) = k * max_per_3node((Total SSD TB / 1 Tb),(Total Threads / 8 Threads),(Total GB / 64 GB RAM)) + k * (Total HDD TB / 2)
Using this website as a guide:
With median - fixed broadband
Here is a sample of the data:
Here in this next table, you can see different k factor depending on the Mbps (median) available.
So anything more than 90 mbps would need a k = 10.
I realize that this approach is clearly arbitrary.
The equation is based on how farmers would build their 3nodes, with HDD being apart from SSD storage.
The k parameter gives a more or less accurate scaling of bandwidth.
What do you think?
Here’s the complete list from speedtest so you can check if the k parameter makes sense:
Here’s some examples with different countries using the Titan as a base:
In Sudan, they have 4.44 mbps in media from speedtest, thus it’s between 0 and 10. With the table above, they would need 1 mbps per Titan.
In Laos, they have 27.81, they are in the 20-30 bracket. They would need 3 mbps per Titan.
In Brazil, with 91.82, they are >90, then they would need 10 mbps per Titan.
2 Likes
I think I’ll spend some time on creating an flist that when booted on a node it does this speed test of a 3nodes connection and posts it somewhere for anyone to see. This way, if farmers want to “showcase” their connectivity there is a simple way of doing this.
Weekend project!
6 Likes
Yep, this works I did something very similar a while back out of curiosity while investigating a potential network issue for a farmer. The status bot was occasionally having trouble pinging the node, so I deployed a VM on it that periodically checked and logged connectivity. While logged in, I thought, why not run a speed test?
While we’re discussing link quality, it’s worth noting that my finding was the node in question was actually offline whenever the bot couldn’t reach it (not just an Yggdrasil issue). Currently, we have no way of penalizing nodes that have unreliable connections. As long as they submit an uptime report at some point before the end of the minting cycle, the uptime is counted. Even adjusting the uptime logic isn’t enough to solve this. A node that flips from connected to disconnected every minute, for example, would be able to pass an uptime check while only spending half of its life online.
I think using Terraform and some scripting, it could be possible to automate the process of gathering this data from all nodes/farms. It would also provide a basic test that the node is actually functional and capable of deploying workloads.
Ehrm… storage to bandwidth ratio…?
Anyway, returning to the main topic here, thanks @Mik for sourcing some data on bandwidth availability per region. Thinking about outcomes here, I wonder if it might be the case that farm size is naturally limited in regions where bandwidth is limited. Accommodating farmers in regions where bandwidth is limited is the right idea, I think, but there’s still a natural limit to how much capacity a given link can serve, no matter the location.
As for actually settling on a formula, I was drawn in my research back to offerings from incumbent hosting providers. This time, I looked at bare metal boxes you can rent from Vultr and others. The quick takeaway is that minimum bandwidth allocations for dedicated servers with similar specs to a Titan is 5TB/month (or about 17mbps saturated). Vultr tops out at 25TB for their biggest box, but other providers offer as much as 100TB or even unmetered connections at 300-1000mbps.
Now, as Weynand pointed out before, these offers are certainly “oversubscribed” (most people don’t use their limit, so the host doesn’t actually need as much bandwidth as they sell). If that’s a 5x oversubscription, then our Titan sized server only needs 1TB monthly and can comfortably fit on a 10mbps link with plenty to spare for home use. For “fat” nodes we might allocate 5-10TB after accounting for oversubscription. This means a 1gbps link can handle 15-30 such nodes at half saturation.
Note that, coming full circle in a way, we’ve determined closer to a 1:1 or 1:2 ratio for the small box, that is SSD:monthly throughput. Vultr’s largest bare metal machine has about 65TB of SSD, which actually shifts the ratio to 6.5:1 at 10TB bandwidth allocation. If we assume the 25TB allocated by Vultr is not really oversubscribed at this end of their offerings, that’s 2.6:1.
Whooptydoo. But what does it all mean Basil?
Our goal here is to give farmers an answer to their question about how much bandwidth they’ll need. Ideally this is a simple answer that’s close enough to “right” to avoid problems later. Simultaneously we have to acknowledge that this is guess work until we see data from the Grid under load.
After evaluating this from several directions, I think the best answer is that our farmers are like colocation providers, in a sense, and they should provide (or at least be able to provide) a comparable level of service.
Let’s recall our speed versus monthly throughput figures:
10 mbps = ~3 TB / month
100 mbps = ~30 TB / month
1 gbps = ~300 TB / month
Using this methodology, we’ve determined ratios of SSD to monthly throughput in the range of 5:1 to 1:5. Let’s try it with SSD in the middle and HDD at the low end, with X being TBs of SSD and Y being TBs of HDD:
Required monthly throughput = X + Y/5
Let’s look at a node with a lot of storage, like node 299 on mainnet with 7.68TB SSD and 672TB HDD:
7.68 + 672/5 = 142
The farm contains five nodes with these specs, meaning this farmer needs over 2gbps to satisfy the requirement based on this formula.
I’d love to pull data on total storage in all the farms, then compute their bandwidth requirements based on tweaks to this formula, but that will have to wait for another day
2 Likes