Wa aleikum assalam…
That sounds great! I’ll send you a message on Telegram.
Thanks!
Distributed Farming with unsteady Renewable Energy
2 Likes
So I have been in contact with our fibre optics provider. They are working on getting our wind field connected (Up to 100 Gigabit, public IPs, optional redundancy). Some of the infrastructutre still has to be built up, but if eversthing goes well we may be ready some time later this year.
From all the data I have analysed we should be able to reach uptimes of 90% (for a 100 kW operation, a little more for 50 kW). Though 95% or even 98% percent is not realistic with wind power only. We are still planning on getting solar up and running in the area, but it will take another 1-2 years probably. And even then we most likely won’t reach 95% consistently.
If we could get a special kind of green certification with lower uptime requirements going we’d be able to (instead of or in addition to building a more centralized data center next to our windfield) to put 3Nodes with about 2-7 kW directly into the tower bases of each of our 17 wind turbines at this location.
Once we have this successful POC at our initial wind field we then may be able to reproduce this in a very decentralized way on wind fields all over Germany.
The main thing thats really holding us back from doing it are the high uptime requirements. So if we
could get the discussion going again on how we may be able to lower these requirements for farmers using exclusively renewables that would be great!
2 Likes
Hi @jakubprogramming. Good to hear and I am personally very motivated to get this moving forward. I will drive the internal and forum (GEP) discussion to get a “green” cert created for 100% renewable energy farmers.
Let me take this and come forward with a proposal in the next week or so for you to have a look at. Very excited! 
2 Likes
Incredible! Looking forward to this! 
Hey guys,
this is @Dany writing. I have to use this account since the forum only allows login with the 3Fold connect app (or I just don’t find alternativ login ways). My @Dany account was created without the app and I only have access from my office PC where I currently have no access to. So please don’t wonder! Maybe one of the forum administrators can hook up with me to fix this.
I would highly appreciate!
First of all… great progress @jakubprogramming. Having fiber connection (ideally redundant) at the wind/solar-park sites is one major requirement for serious farming setups (comparable to datacenter environmets). So this is a giant step forward. Great job!!
Another major requirement is having a stable and reliable energy supply. From my point of view trying to handle with downtime due to unsteady renewable energy supply is the wrong way to deal with this issue. Servers have to be up all time, apart from maintenance downtime (should be less then <1%)… but that’s what servers are supposed to do! So we have to find a solution to ensure this without giving up the green farming prerequisite. I asume that your (@jakubprogramming) sites are connected to the energy grid both ways. Meaning that you can provide energy as well as beeing able to consume energy. So…in case these farms need energy because the wind/solar-park can’t provide it… the energy consumtion should be provided by a supplier that provides you with power created by 100% renewable energy ressources so that we would still fullfill prerequisites for “super” green farming. The true benefit of the wind/solar-powered energy sites/grids comes up when there is superfluous/excessive energy available that could be used for high energy consuming tasks. That’s what it makes “super” green farming. And this is what can only be addressed from the grid side where we need the cooperation between each other (farmers/solution-providers or users). There has to be a machanism established where the particular farm/nodes can shout out “Hey grid, I have plenty of energy that would be wasted if I don’t get heavy workload to handle with. Please give me input!”. From the wind/solar-park operators point of view this migth don’t make sense in the first place because you want to sell energy and don’t want to path it trough but I am sure there will be ways to make this possible form a business side of view. It only makes sense when you get the whole picture.
Beside these two major requirements we have to consider the last parameter which is environmental conditions like temperature, humidity, security and so on. Of course this depends on the indivduell preconditions. For now… lets assume that this is the easiest to achieve.
So lets get serios now. How can we get this done? Apart from the above mentioned requirements we need the farms to be set up. So in the last couple of weeks we tried to build up a rig that could be a setup at one of those wind/solar-parks as a module that can be multiplied. Therefore we designed a rack containing 38 servers (1 rack-units) together with the additonal equipment (router/firewall, switches, UPS etc.). I tried to hook up with pertinet hardware suppliers (HPE, DELL and others). But to be honest… what they have come up with is out of my financial possibilities. So i got back to my salesman were we got the hardware for our first tiny farm and he provided us with 45 refurbished servers with pretty competetive prices. It is still a serious amount of money but we were able to get it done. In the end we rented a rack in a DC which gives us the opportunity to get the necessary power supply for our 3nodes provided by 100% renewable energy created sources. It was pretty hard to find a DC but finally we did it. It’s certified by the german so called “TÜV” (comparable to NIST in USA). You won’t believe me when I tell you that we are paying 0,39€ (including tax) for one kW/h…but it’s a fact. The above mentioned design takes about 4-5 kW/h in idle (!!!). You can do the math yourself. In addition to this a proper bandwith takes another 2k EUR/month (I’m not kidding)… for now we are running at 3 Gbit/s . And this is propaply not enough. We are going for 5 Gbit/s within the next months. I want to point out that farming rewards at the moment are barely covering operating costs (in Germany, for this particular setup) considering the actual token price. To be fair I have to mention that this includes interest and payment for the investment in hardware (scheduled for the next 36 months). But…despite of this beeing quite challenging…there is a huge potential for green or even super green farming sites which migth be given additional rewards. Competetive price per kW/h and bandwith is what gives the true financial benefits so we could make a business modell out of it in order to attract wind/solar park operators. When these become solution providers and gain another additional benefits therefor it should be an easy decission from a business side of view to participate in ThreeFold.
Finally I contracted our hardware provider to supply us with up to 200-300 servers for a fixed price for the next 6 month. It’s plenty of work to preconfigure the nodes and do BIOS and LOM and components updates and prepare them for running ZOS. But we can come up with fully piled up racks with Router/Firewall, Switches, AC, UPS and whatever is neccessary, It takes us 2-3 weeks to get it done.
I would love to have a call with you guys soon (@jakubprogramming, @weynandkuijpers, @andreaspow) and figure out the next steps. Maybe we can schedule a meeting short-term.
Cheers
Dany
4 Likes
Let’s do that - it seems that we have two great initiatives meeting and manifesting new ways of farming. Kudos 
. I’ll try to setup a meeting for the coming week!
Happy Saturday!
3 Likes
I will be glad to hop on a call/meeting with you guys as soon as possible.
I am currently on a family trip and won’t really be available until Wednesday.
Anything after that will definitely work for me though.
If I find the opportunity I will try and address some of the important points @Dany (@sigzag) has brought up as soon as possible, ideally before we have our meeting.
There is just one thing I need to address right away. Since I made my post yesterday I came to an important realization. So I feel the urge to make a correction right away. Since ThreeFold payouts are calculated monthly and not yearly, I have to calculate the power availability of each given month in the year, instead of overall availability. Since wind and weather can vary a lot throughout the year the results of course are less steady and more fluctuant. So I went through all the raw data and aggregated it into groups where I was able to see the actual power availability within each month. The results can be viewed in the graph below.
As you can see, especially in the summer months the power availability for 100 kW can fall far below 90%. For 25 kW on the other hand it looks pretty good.
Luckily those summer months are exactly those where we will have plenty of solar power coming up. At the moment we are planning to deploy 2 MW of solar power right next to our wind field.
In the meantime we could maybe start off with only 25 kW or less of ThreeFold Farming power, unless we can find a way to integrate nodes into the grid with even less availability (eg. 85%) without harming overall grid performance and health too much. Unfortunately, even though we theoretically are connected to the public grid and could power our hardware with electricity from other green energy suppliers (as @Dany suggested) it is not as easy. It may be possible though to leave the servers running idle in these time periods. I will investigate this further and come back to you.
2 Likes
Maybe you already mentioned this cause I didn’t read every sentence, but there could be certain workloads that are more suitable to less than perfect uptime. Backup storage or compute for non critical sytlstems for example.
2 Likes
@jakubprogramming and @sigzag: What time would be a good time to organize a online meet? I have availability tomorrow.
For the forum readers, I made the mistake to try to arrange the meeting here: apologies. It’s been done were meeting next week Tuesday and we will report back here! 
3 Likes
Before we have our next meeting I wanted to address some of the important points @Dany has brought up in his previous longer post. In general each of the following “topics” potentially plays out a bit differently, depending on whether we’re speaking about deploying a farm next to the wind field (inside a separate building) or right into the tower base of a wind turbine. But I will try to keep it as general and concise as possible.
Fiber Connection
Once we got it up and running it will be available next to the wind field as well as inside the wind turbines themselves. I’ve been told 100 Gigabits are possible, but we will have to evaluate if this will be true for all wind turbines.
Stable and reliable energy supply
This is the one thing thats requires the most work to realize. As I have already hinted in my last post, when there is no electricity being produced due to lack of wind, we can’t simply buy green energy from other providers.
How does it work when we purchase electricity on the wind field and what does it cost?
Each year the peak power draw on the wind field is being tracked. Let’s say for whatever reason at time X during the year we are drawing 25 kW that we have to purchase from another provider. Even if we only drew those 25 kW for a duration of 1 minute, this will be our peak power draw for that given year. Now we will have to pay 25 kW * 48€/kW = 1200€ on top of all the kWh (electrical work) that we may have purchased. That makes purchasing large amounts of power, even for a very short time, disproportionately expensive. This becomes especially true, since we won’t really need those peak power draws for more than 90% of the year, but still we will have to pay for them in full.
So let’s say during the entire year we purchase additional 5000 kWh for an average price of 0.4€ per kWh, we will pay 2000€ for the electrical work. But having peaked at 25 kW, even for just a minute, we will have to pay 1200€ on top of that. Nonetheless, purchasing energy that is actually from green sources may not be possible.
Can electricity that we purchase next to our wind field actually be green?
Since we ourselves are not producing green energy, it is highly unlikely that the energy that we would be able to purchase will physically be from green sources. Some of it may potentially be solar power, but we won’t really be able to tell. Instead we would have to fall back on providers using what we in Germany call “Herkunftsnachweise” (guarantee of origin for electricity). That means we’d potentially buy non-green electricity, where the provider guarantees that he will compensate some green energy provider (probably from some other European country) for our purchased electricity. In the end the money we paid will theoretically indirectly go to a green energy producer, but technically its more green washing than actually green. This is not what we have in mind when thinking about Super-Green Farming. And also does this not really contribute to more green energy being produced or consumed.
What else could we do to reduce downtime?
@Dany has stated that servers have to be up all the time, apart from maintenance downtime. In an ideal scenario we would prefer the servers to be turned off whenever downtime is more than lets say 4 hours (optimal duration for this will have to be determined). By making sure that smaller supply gaps of less than X hours can be bridged, we could significantly reduce downtime over the course of a month. But how could we achieve this?
Let’s consider batteries… depending on the size of our farm(s) and the amount of hours that have to be bridged before a farm goes offline, batteries may be a viable option for those smaller gaps where green energy is not available. Whether or not this is economically reasonable depends on a lot of factors. But we should be able to calculate this quite accurately. It may not be cheaper than paying for those short “peak power draws” that come with purchasing electricity from other sources, but at least it will be 100% green. The costs of charging those batteries could be reduced by charging them is some “smart” schedule where they wait to be charged until excess or cheap energy is available (unless predictions don’t see cheap energy coming in within the next day or so).
Since most batteries come with some constraints in regards to their ambient temperature, it may or may not be practicable to deploy them directly inside our wind turbines though (read my paragraph on temperature below). For farms that are deployed in a separate building this could be handled.
Reducing the negative impact of downtimes
Maybe we could furthermore consider limiting the workloads that “Super Green Farms” can accept to use cases where availability is less significant.
As it has already been proposed by myself and others in this thread this may include backup storage & archiving, machine learning or 3D-Rendering.
Hosting a web server for example is probably less suitable.
Some other ideas that come to my mind:
- by providing forecasts about future predicted energy production and thus availability, the grid could potentially be informed in advance, making proper preparations such as moving workloads to another node
- Setting minimum bandwidth requirements (at least some level of fibre) so that data can quickly be transferred in case workloads need to move to a different location where more power is available
Temperature & Humidity
As we are not allowed to make structural changes to the architecture of our wind turbines, we are unable to install active cooling components. As a result, each wind turbines can only host a limited amount of hardware, before too much heat is being emitted. Luckily the towers of the wind turbines are able to absorb a lot of heat being produced further down. Still, deploying more than 10 kW per wind turbine is probably not a good idea. At the moment we have some ASICs running inside one of our wind turbines. Their total power is about 23 kW. Last year in the summer we already reached ambient temperatures of more than 38°C. At this temperature components and backup batteries are not in their ideal working environment. In the winter it is also possible for temperatures to drop below 0°C (unclear whether this would also happen if hardware was running non stop). The following table shows some aggregated temperature and humidity data from our testing wind mill from the last 2 years (Sensor being inside the wind turbine).
Magnetic field
Electrical equipment such as transformers create magnetic fields inside wind turbines. We may have to measure and evaluate whether this may be a problem for server hardware. Our ASICs that are currently mining inside one of our wind turbines don’t seem to be affected though (not sure how I’d figure out if they were though).
Security
Since data on ThreeFold and ZOS is strongly encrypted and distributed by default, I believe data security is already taken care of.
The security of hardware is another issue. Since our wind turbines are already protected quite well from intruders and access to the machines is greatly limited, I don’t see much risk. We haven’t really had many problems with this in the past. Depending on the total worth of hardware being deployed additional measurements may be taken.
Excess Energy
I agree 100% with @Dany on this. We should definitely think about how we can let the grid know, when very cheap or even excess energy is available to some of its nodes. When it is, workloads that are not time sensitive, but very power hungry could start running. Smart contracts could potentially be used to configure the parameters under which a given workload should execute (eg. avg. price per kWh, only excess energy, deadline etc.) Obviously our energy selling business would not directly profit from this, but as Dany pointed out we’re looking at the greater picture here.
As @Dany also pointed out, turning a profit with ThreeFold farming in Germany is not easy, given the high cost for electricity and bandwidth.
The bandwidth costs can to some degree be lowered with scale. As far as electricity prices are concerned, we do have a market advantage when using energy from our own production. And thats the key. We want to use as much energy from our own production as possible, without relying on other sources, which may potentially not really be green. When all of this is potentially combined with additional rewards for super green farmers, it could eventually become a viable business model that could hopefully even be an example for others to follow.
Just some things to think about before our meeting on Wednesday…
Talk to you guys soon!
3 Likes
@jakubprogramming and @Dany. We had our call yesterday and I must say I am very excited to see your two projects going forward. I admire both of you ambitions and strengths to make this world a better place, one step at a time. Also proud to be involved and seeing that initiatives in the community are growing and make impact.
As a short update (I’ll leave the details to you two to add later):
- Agreed to start creating a GEP for a “supergreen farming” category which is 100% renewable energy powered (with proof) and to keep it 100% guaranteed renewable based there has to be an allowance to have a lower uptime requirement for the “proof of capacity”.
- Dicsussed and agreed that @Dany will start a concept installation of “everything” in one rack (cooling, switching, UPS, PDU and 3nodes) to create an “autonomous” install that can exist is any “real estate” with power and connectivity, not having any further requirements for the property owner. Also - there are a serious amount of servers being reserved for when this MVP install is siccessfull.
Super excited, 
5 Likes
Based on this ongoing discussion and some private talks with @andreaspow, @weynandkuijpers and @Dany I am formulating a GEP draft for Super Green Farming with 100% Renewable Energy. Once we have talked about my initial draft here and potentially made some changes to it, I will make sure to post it to the according GEP section of the forum.
Uptime Requirements
Since renewable energy is not available 100% of the time, lower uptime requirements are needed for 100% green farming. Based on my analysis the following uptimes could be realistically achieved:
- 90% Uptime using a combination of wind & solar power, or a combination of wind & backup batteries
- 85% Uptime can be achieved using wind power alone
Since wind power availability can fluctuate greatly for different months of the year, it could be considered to have uptime requirements be calculated on a yearly basis, instead of monthly basis. This way more windy months could potentially compensate for less windy months.
Minting Rewards
In order to incentivise farmers to use renewables to power their 3Nodes, minting rewards should be higher than for conventional farmers. This way the grid will eventually become more sustainable itself, as more farmers go green. I propose the following minting rewards:
- 125% for super green farmers on DIY hardware
- 150% for super green farmers on certified hardware
Suitable Workloads
Lower uptime requirements mean that those green servers are not very practical for use cases that require more or less constant availability (eg. hosting a website or running a gaming server). Instead I can imagine that those 3Nodes flagged as Super Green could be constrained to accept only certain workloads, including but not limited to:
- Backup & Archiving
- 2D/3D Rendering
- Machine Learning
- Other less time sensitive high performance computing tasks
Bandwidth Requirements
Without a constant uptime, and considering the suitable workloads, it may be necessary to send and receive larger amounts of data from and to the 3Nodes and in a higher frequency than usual. Therefore it may be beneficial to set higher bandwidth requirements for super green farmers. My proposal would be:
- 1 Gbps as a general minimum
- For larger super green farms 10 Gbps may be required
Use of cheap Excess Energy
Many times of the year (when it is very windy) more energy is being produced than the electricity grids can handle. Huge amounts of energy are being wasted this way. If super green nodes were able to let the grid know, that they currently have plenty of excess energy available, certain less time sensitive but power hungry workloads could be started. This would improve the grid both in an economical and a sustainable way.
Smart Contracts for IT
By using smart contracts on the TF Grid we could potentially make this entire thing even greater and more efficient. When a workload definition is sent out by a capacity consumer (user) the following things could optionally be specified:
- super_green: deploy workload on super green node(s)
- deadline: The grid will make sure that the task at hand will be completed by then (eg. rendering a 3D scene). Super green nodes (if available) can be preferred for being assigned the workload. If no super green nodes are available, conventional nodes may be used. If no deadline is specified, workload will not run until super green nodes are ready to process.
- only_excess_energy: If somebody wants to be as sustainable as possible, he or she could let the workload run only on excess energy that would have been wasted otherwise (may take a while until workload is handled).
contract = {
version: contractVersion,
contract_id: contractID
…
super_green: true, //run workload on super green nodes
deadline: “TimeStampOfDeadLine”, // workload must finish by this time,
only_excess_energy: false // if true, workload will only run on excess energy
}
All of this could be further enhanced by providing predictions about power availability of our 3Nodes to the grid. We are able to predict (to a certain degree) what our power production in the near future will be. This data could be valuable for the grid to decide how and when to run workloads or move them over to a different location (in case a power drop is expected soon). In the future it may also be thinkable to provide energy prices of our nodes to the grid. On the german energy exchange the price per kWh changes every hour, so it may be beneficial for users (and the environment) to have their workloads run when energy prices are lower (by giving them as small discount on their used capacity).
Proof of origin of electricity
In order to prevent farmers using conventional energy to power their super green nodes, and thus cheat the system, a proof of origin of electricity for super green farmers shall be required. As a green energy company we have all the necessary electric meters for this already in place. If required, a certified public accountant can confirm this for any farms joining the grid in the future.
Lets talk about this!
Feel free to discuss the proposed changes and let us know of your opinions.
I may not have addressed all relevant points that make up a proper GEP, so if you have anything to add please go ahead. Any constructive criticism or further viewing angles to this are highly appreciated!
2 Likes
Looks good.
About this:
- 1 Gbps as a general minimum
I am not sure if this is optimal. Is it possible that some geographical regions where 100% green farming is happening come with sub 1 gbps ISP bandwidth?
In this case, would the 1 gbps cap limit 100% green farm growth?
Also it depends on the 3node’s resources. So it could be scalable instead of an absolute value (e.g. X gbps bandwidth per Y 3node resources).
Nice work @jakubprogramming !
2 Likes
@Mik You are right. A minimum bandwidth of 1Gbps could limit 100% green farm growth. I agree that a more flexible and scalable solution, depending on the number of nodes, would probably be a better overall solution. The exact bandwidth requirements are definitely up to debate. I just think that, depending on the workload, a lower uptime requirement may partially be compensated for by having higher bandwidth.
1 Like
I think the required bandwidth is related to amount of storage space. The more information can be stored, the more bandwidth is required to get data in and out. I this case, since the uptime will more likely be lower than other grid nodes, the bandwidth requirement might need to be higher than other 3nodes… 
2 Likes
That was exactly my thinking. Whatever the bandwidth requirements for regular nodes with a given storage volume are, it should probably be a bit higher for 100% green nodes…
If we assume that the uptime of those green nodes is 10% less on average than of regular nodes, how much more data would have to be transferred (on average, considering suitable workloads) from and to the nodes? I think this is rather hard to determine beforehand, as it depends on a lot of different factors and especially the type of workload that is deployed.
Maybe we could just start with twice the minimum bandwidth initially to keep it simple?
I like your entheusiasm! Byut maybe 2x is a little over the top. The aim is to have the nodes reach an uptime of 90% or more, so in theory you need to compensate for loss of “time” to up / download by increasing speed by 10% or less to come to a same up / download “capability” than other nodes. So IMHO I think a bandwidth uplift of 25% should cover it… @jakubprogramming what do you think?
You are right that the uptime will only be about 10% less compared to other nodes. The downtime on the other hand will be 2-3 times as high. I definitely wouldn’t want to put higher bandwidth requirements onto this than needed. If an increase of 25% seems reasonable I will be happy with that.
@weynandkuijpers I could update the bandwidth section my proposal accordingly and publish it to the GEP Thread. What do you think?
Please update the GEP thread and then publish it. Looking forward to it!