I agree that this is a good way to see the situation and it is very pragmatic. The worst case scenario is someone using your farm all of a sudden and needing much bandwidth, not having it on time then moving on to another farm.
This is why it’s kind of tricky to know in advance what will be the requirements in terms of bandwidth.
For someone who doesn’t want to go into a DC, it is essential to know how much 3nodes one can install with the most ISP bandwidth they can get.
I feel like the purpose of threefold is spreading up the node to be people internet. If at the end everyone puts their server in a data center that destroys the purpose of threefold. Just a thought,
I partially disagree. I think the project is largely about the people hosting the services that run the internet instead of Amazon and Microsoft. They have proven themselves to pull things down due to popular opinion.
Having a decentralized network of nodes is an important part of the project as well and I do operate a network of small, at home, servers with friends and family as well. But even us having servers in a datacenter is more decentralized than you would think. Even a moderately sized city can have many datacenters. Jacksonville, FL has at least 5 available, and that is not a high tech place. While I couldn’t easily find how many DC’s Amazon runs, I think its only about 50 in the US. That’s pretty centralized given its market share.
I think you guys, @noretreat and @FLnelson, are in an interesting discussion.
I would agree that I thought at first Threefold would be against data center in general but lately I did read and heard the team explain how it does have great advantages.
Indeed there’s a world of difference between one giant Amazon DC and many smaller DC run by the people.
I choose personally the route of having servers elsewhere than DC. Maybe I’ll change my mind when cultivation is kicking!
I’ll be glad to hear what you and others have to say on this topic.
Interesting indeed. ThreeFold is pushing (a part of) the internet to go back to it’s roots: by people for people. This means that capacity “production” is not in the hands of a few large providers but can be done by anyone who wishes to be involved. How they deliver their capacity depends on what they can and cannot do.
we truly believe that local produce (like everything else, food being a good example) in general is better for the world and therefore want this internet capacity to happen everywhere, close to use cases and IT capacity consumption.
How this is done is entirely up to the people that opt in to produce. A rack in the basement or a rack in a datacenter, servers in the attic, or a server in a mobile telephone mast or windfarm. Altogether this presents a much more sustainable and equal way to create IT (internet) capacity than a few hyperscale datacenters servicing customers globally (thus far away from consumption and putting a lot of string on networks everywhere).
So capacity production local to consumer needs and by anyone. 
Hey guys,
Just to complement our discussion here, I’ll add some things.
In another post, @kristof made some good points and we came up with a draft of an equation to know the bandwidth needed:
The general equation is:
min_bandwidth = bandwidth_per_TB-HDD * Qty_TB-HDD + bandwidth_per_TB-SSD * Qty_TB-SSD +
bandwidth_per_GB-RAM * Qty_GB-RAM
Taking into account what @FLnelson and @weynandkuijpers said with HDD and SSD, we could have
750GB HDD storage per 10 mbps --> 13 mbps per 1TB HDD
200GB SSD storage per 10 mpbs --> 50 mbps per 1TB SSD
This would give:
min_bandwidth = 13 * Qty_TB-HDD + 50 * Qty_TB-SSD +
bandwidth_per_GB-RAM * Qty_GB-RAM
For the sake of exploration, let’s say a 32 GB ram titan needs a 10 mbps connection, we’d have 0.3125 mbps per 1 GB of ram. The equation becomes:
Equation (1): min_bandwidth = 13 * Qty_TB-HDD + 50 * Qty_TB-SSD +
0.3125 * Qty_GB-RAM
Say we have a 3node with 128GB of ram, 10TB HDD and 2 TB SSD. We’d have:
min_bandwidth = 13 * 10 + 50 * 2 +
0.3125 * 128 = 273 mbps
Doesn’t it seem too much bandwidth for a 128GB/10TB HDD/2TB SSD 3node?
Let’s say we cut in half the mbps needed for HDD and SSD. We’d have 156.5 mbps needed. Does this make more sense?
It’s clearly not rocket science here but this kind of equation can give us a good idea of what could be done.
I remember that we said 1 mbps would be the minimum for a Titan which is 32GB ram/0TB HDD/1TB SSD.
With the above equation (1), a Titan would need 60 mbps. Clearly 60 > 1. So we are oversizing bandwidth in this regard.
Let me know what you guys think!
With equation (1), basically we just need to find fitting ratios and it could work pretty well.
Do you know if DC’s let you use your own hardware or will you have to use theirs or upgrade?
They can offer either. Some DC’s will rent servers out, but this isn’t going to be very economical. Many data centers offer colocation aka colo, where you can put your server in the datacenter.
Looks like some solid methodology on estimating bandwidth requirements. Getting those constants right on mbps for HDD, SDD and RAM is super important. Somebody out there must have a strong answer. I’ll do some research on it as well. 1mbps for a titan seems awfully low for normal needs, but I suppose there are different levels of service to be expected across geographical regions.
I suppose there are different levels of service to be expected across geographical regions
Yes good point. Also we could have a range of bandwidth that would be tolerable for the Grid. Say you exceed the bandwidth, it’s just even better but maybe useless. If you go below, you would be penalized in terms of farming.
Here’s something I wrote to @scott that could be a template of a general answer for the community:
"
Message concerning the Internet Bandwidth for Farming (Titan/DIY)
A 3node Titan needs a mimimum of 1mbps internet bandwidth.
For many Titans (2 to 5) or a DIY 3node with up to 30 TB of storage/8 vcores/64GB ram, a high speed internet is sufficient. (Between 10 to 100 mbps).
A 1 gbps connection is necessary for farms with multiple high capacity 3nodes such as 16-32vcores/128-256 GB ram servers (from 1 to 20 servers) or multiple titans (6 to 40).
NB: Details on the necessary bandwidth for cultivation use cases are yet to be determined with further precisions.
A farm with a specific setup of 3nodes having an obviously too low bandwidth compared to the 3 examples above would be considered as an abuse of the farming system that would be limiting the farm efficiency in period of cultivation (on Grid demands). If such case of abuses were to arrived, the DAO could penalize the abusive farmer by slashing/reducing the farming rewards to a ratio proportional to the bandwidth capacity.
"
Last sentence is just a proposition but the details would need to be discussed. Clearly these 3 examples have gaps and do not covert smoothly the spectrum. With the help of the community, we could rapidly come up with a general answer that will guarantee good Grid capacity in the future. And we’ll have time to settle the details.
All - thank you all for the contributions. Let’s collect all the input and start making a model based on the information discusses that works like a simulator. I am quite fluent in spreadsheets (not so much in coding) so I’ll start with a sheet to share. Please allow me to spend some time over the weekend on this - my rest of the week is pretty busy. 
That’s great. So meanwhile I added a general question + answer in the FAQ for Farmers. This will serve as a general statement until we have more details.
Here it is if someone wants to change anything:
Q: What is the bandwidth needed when it comes to running 3nodes on the Grid?
[This answer is temporary and aim to give a general idea. The community will soon have clearer and more official documentation.]
NB: Notice the gray areas between the three propositions. This goes to show that prices numbers are not yet determined. Using those numbers as a guide will set you in the good direction.
Hey guys,
Do we have any news on this subject?
The bandwidth needed per farm will be an important notion when utilization is there and farmers must be ready and be sure that they are not oversizing their farm for their available ISP bandwidth.
The synthesis above I drafted is too basic of course. Many people brought the subject in the TG channels.
A kind of simple ratio to be scaled could be more useful for the time being.
What about saying
1TB SSD + 64 GB + 8 threads --> needs an ideal minimum of 10 mbps
*if you have another number in mind, instead of 10 mbps, please share your thoughts.
The left side is the basic unit based on the most profitable farming ratio. This is like saying each 16 CU needs 10 mbps and we already know that each 16 CU will need a minimum of 1 TB SSD. We can suggest that farmers will most likely build 3nodes configurations that are closest to this ratio, as it gives more rewards. Per Game Theory logic, it would be sound to conclude that farmers will in general aim at this ratio.
But then, why 10 mbps? Because it’s 10 times more than the minimum we ask for a Titan, and that a Titan is 8 threads and 32 GB. Here we have 64GB of RAM. If someone has another idea for the quantity of mbps, speak your mind.
Then we take this ratio and take the maximum result with the example of a 5 TB, 384 GB ram, 48 threads server, we get this:
I use this example as it’s @Valentine969’s really nice build that is kind of on the upper spectrum of big server.
A big farm of 4 servers would need here 240 mbps.
An big farm case scenario with 16 servers would need a 1 gbps.
min Bandwidth per Farm (mbps) = (# of 3nodes) * k * max_per_3node((Total SSD TB / 1 Tb),(Total Threads / 8 Threads),(Total GB / 64 GB))
with k = 10 mbps, we have:
min bandwidth = (# of 3nodes) * 10 * max_per_3node((Total SSD TB / 1 Tb),(Total Threads / 8 Threads),(Total GB / 64 GB))
Here’s the case with 16 serves:
16 * 10 * max(5, 6, 6) = 960 mbps
960 < 1000
With 17 servers in this case, you’d need more than 1 gbps bandwidth.
In the equation, the function max(…) ensures that you do not under scale.
What do you guys think?
We can have a range of this variable. Instead of 10, it’s between 8 to 16 mbps (for example).
Of course, with empirical data, we’ll know more of the bandwidth needed in real-case scenario.
Let’s New Internet!
EDIT: We’d also have to add the HDD storage in there. What do you suggest as a basic unit ratio?
OK and then what about this to take into consideration the HDD storage:
min Bandwidth per node (mbps) = k * max_per_3node((Total SSD TB / 1 Tb),(Total Threads / 8 Threads),(Total GB / 64 GB)) + k * (Total HDD TB / 2)
With k = 10 and a 1 servers with 1 TB SSD, 64GB ram, 8 Threads and an additional 8TB HDD:
min bandwidth = 10 * (max (1,1,1)) * + 10 * ( 8 / 2 ) = 10 * 1 + 10 * 5 = 50 mbps.
More concise equation:
min. Bandwidth per 3node (mbps) = k * [ max((Total SSD TB / 1 Tb),(Total Threads / 8 Threads),(Total GB / 64 GB)) + (Total HDD TB / 2) ]
where k is the mbps factor. So if k = 4, it’s 4 mbps * […]
I think 10 mbps could be realistic, but it can be modified.
Let me know what you guys think!
Hi @Mik. Thank you for your effort to get some consensus around this topic. I have not found the time to collect all the presented materials in (this) thread(s). Let try to do this collectively. Reading up and down a thread is not the easiest, and it’s hard to retain the essence of the discussion and what we have learned. So let’s review all the WHY
So the why for having bandwidth requirements is to create usable farms. Capacity consumers will at some point see the amount of bandwidth available to a farm and can therefore decide where to process or store their data. For farmers that provide good bandwidth, this will increase their farming rewards. With regards to “what is the minimally acceptable bandwidth” to be able to have a farm getting farming rewards, we have to also take into account the geography. Some regions in the world have affordable Gbps into the home, others should be thankful of they get a few Mbps to the home and then there are the underserved parts (in terms of infrastructure / in particular network access).
So before talking hard numbers, how do we get an idea of what is an acceptable bandwidth level based on geography? Is anyone familiar with a sight that tracks network speed/penetration per capita or something similar? Let’s try to find a good source for that first.
Thanks @weynandkuijpers that helps to clarify. Clearly I didn’t get everything correctly.
Good idea to have “acceptable bandwidth level based on geography”.
I’ll try and do some research on this. Here’s some idea:
So with the equation:
min Bandwidth per node (mbps) = k * max_per_3node((Total SSD TB / 1 Tb),(Total Threads / 8 Threads),(Total GB / 64 GB)) + k * (Total HDD TB / 2)
We could adjust the parameter k depending on the “acceptable bandwidth level based on geography”.
In the simulator, we could have a parameter : geographical region, with main regions to select. Then selecting one region, the simulator would adjust the k parameter and it would tell the bandwidth needed for the 3node’s specs in the simulator.
This could be a good starter.
Also just to shoot ideas: having k = 10 could be for high bandwidth region, and k = 1 for low bandwidth region (as 1mbps for 1 titan is the minimum requirement as stated by Threefold). Then there could be steps in between.
Of course, the equation can change. It’s just to brainstorm.
Amazing discussion. Thanks for reading and let’s NEW INTERNET!
EDIT: I’ll start some data collecting and post some here so we get a general idea of the situation.
Thank you - very helpful. This created a rabbit hole for me (which I stuck my head in for a little bit). So from the wiki, you can find out that they get this information from: https://www.speedtest.net/. This well known authority in terms of internet speed checking also provides command-line interface to their test software: https://www.speedtest.net/apps/cli
Installed it, Linux binary, and execute it:
{
"download": 93509230.46298379,
"upload": 95016876.45176126,
"ping": 5.024,
"server": {
"url": "http://speedtest1.etisalat.ae:8080/speedtest/upload.php",
"lat": "25.2500",
"lon": "55.3000",
"name": "Dubai",
"country": "United Arab Emirates",
"cc": "AE",
"sponsor": "ETISALAT-UAE",
"id": "17336",
"host": "speedtest1.etisalat.ae:8080",
"d": 1.7183145989433646,
"latency": 5.024
},
"timestamp": "2022-05-12T04:23:32.177313Z",
"bytes_sent": 118841344,
"bytes_received": 117319962,
"share": null,
"client": {
"ip": "2.51.127.183",
"lat": "25.2633",
"lon": "55.3087",
"isp": "Etisalat",
"isprating": "3.7",
"rating": "0",
"ispdlavg": "0",
"ispulavg": "0",
"loggedin": "0",
"country": "AE"
}
}
It creates a very nicely structured json dataset. This was from my laptop: what if we run this in a z-machine on the grid… :-). From freefarm in Belgium:
{
"type": "result",
"timestamp": "2022-05-12T04:41:36Z",
"ping": {
"jitter": 0.078,
"latency": 4.675
},
"download": {
"bandwidth": 76093440,
"bytes": 537762760,
"elapsed": 7207
},
"upload": {
"bandwidth": 109309476,
"bytes": 427469170,
"elapsed": 3904
},
"isp": "Telenet",
"interface": {
"internalIp": "185.69.166.156",
"name": "eth1",
"macAddr": "0E:14:94:61:B7:4C",
"isVpn": false,
"externalIp": "185.69.166.156"
},
"server": {
"id": 15212,
"host": "netgauge.telenet.be",
"port": 8080,
"name": "Telenet BVBA/SPRL",
"location": "Mechelen",
"country": "Belgium",
"ip": "195.162.192.137"
},
"result": {
"id": "e2bdfa7f-e89f-4884-834f-54c139e95b47",
"url": "https://www.speedtest.net/result/c/e2bdfa7f-e89f-4884-834f-54c139e95b47",
"persisted": true
}
}
Nice bandwidth @delandtj (10Gbps up and down) . Cool so this works out of the box (weird difference in the speedtest binary that on ubuntu I had to accept a license fist, on Archlinux it just did what I asked it to do… anyway).
The real test if we have a 3node that does not have a public IP address… I choose node 67 in the US. Spec-wise it looks like a titan, so it should exist somewhere in a home in the US. I choose to have access over the planetary networks and specify a really small Z-machine (1 core, 512 MB memory).
After deploying the z-machine and installing the software this is the result (deleted specifics of the Titan and Z-machine):
{
"type": "result",
"timestamp": "2022-05-12T04:56:00Z",
"ping": {
"jitter": 2.737,
"latency": 14.062
},
"download": {
"bandwidth": 28531416,
"bytes": 242930960,
"elapsed": 8604
},
"upload": {
"bandwidth": 1405157,
"bytes": 8803184,
"elapsed": 6305
},
"packetLoss": 0,
"isp": "**********",
"interface": {
"internalIp": "***.***.***.***",
"name": "eth0",
"macAddr": "32:AE:DB:11:AE:E6",
"isVpn": false,
"externalIp": "***.***.***.***"
},
"server": {
"id": 16974,
"host": "speedtest.west.rr.com",
"port": 8080,
"name": "Spectrum",
"location": "Los Angeles, CA",
"country": "United States",
"ip": "***.***.***.***"
}
}
So we can find out per installed machine that the download and upload speeds are by which the machine is connected.
Just some playful hacking
Could you run this test on 3081 for me?
{
> "type": "result",
> "timestamp": "2022-05-13T03:11:09Z",
> "ping": {
> "jitter": 1.052,
> "latency": 17.976
> },
> "download": {
> "bandwidth": 117736312,
> "bytes": 1048621576,
> "elapsed": 9000
> },
> "upload": {
> "bandwidth": 117139141,
> "bytes": 1107368316,
> "elapsed": 9609
> },
> "packetLoss": 0,
> "isp": "AT&T U-verse",
left the rest out