What is the uptime requirement for farmers?

Update 1.28.2022: For minting v2, farming rewards will now be paid out on a prorated basis for uptime in a given month. This means that a node which is online for half the month will farm half of the tokens. Farmers who migrate to Grid 3, therefore, will not lose out on any tokens. Discussion below should still be relevant to how we handle uptime on minting v3 going forward.

~

The ThreeFold Grid is designed with an uptime requirement for farmers to receive their farmed tokens. This is meant to encourage farmers to provide a certain service level for those utilizing the Grid. In the traditional cloud computing world, service levels are often measured by “how many 9s”, meaning that uptime commitments look like 99.X% where X is some number of 9s following the decimal. Of course, maintaining such levels in a decentralized network is challenging, and so far farmers have only been held to a 98% uptime requirement.

Farming rewards have two components: farmed and cultivated tokens. Farming happens all the time, while cultivation happens when a farmer’s node is being utilized. Uptime requirements only apply to farming, in that only farmers who meet the minimum uptime for a month will receive farmed tokens. Utilization, on the other hand, is only tied to uptime in the sense that a node must be online in order to be utilized. But under the current model, farmers are actually paid as soon as capacity is reserved via a capacity pool which provides credit for the user to draw from as their workloads run.

We’ve seen much concern from the community around the idea that certified farming will require farmers to maintain 99.8% uptime. This figure was proposed as the service level for certified farming, but was only ever a proposal. At such a service level, certified farms would be in the same league as corporate cloud providers. Without at least some farms providing this kind of uptime, it would be very difficult to drive adoption among the segment of the market which needs very reliable infrastructure.

To be clear, there is currently no expectation that owners of certified Titan nods will need to achieve 99.8% uptime. The current 98% requirement means that a node can be offline for ~14.5 hours per month before losing out on a month’s tokens. In practice, the vast majority of 3Nodes meet this level of uptime without issue. You can review the figures on explorer.grid.tf to see that months or even years of continuous uptime is possible. Personally, I’ve had a node running at home for a full year without ever losing out on a month of farmed tokens.

Finally, let’s also address the fact that even 98% uptime can be infeasible in some parts of the world where electricity and internet are less reliable. ThreeFold is well aware of this, and geographical provisions for infrastructure variability have always been a part of the farming reward model. If you think this applies to you and are having issues with not receiving farmed tokens due to downtime, please reach out to the team to discuss by leaving a reply here or using the chat button in the bottom right corner of this page.

My intention here is to gather and address concerns being expressed by the community via chats, in order to gather the conversation in a more permanent space. Please don’t hesitate to provide your input or questions below. We’ll also have an update on this from the team coming soon.

2 Likes

This is a great topic.

I would like to share what happened to me this month and show one of the needs that could be addressed. It is not 100% related to the certified question of 99.8% uptime but it has to do with farming uptime in general.

My current set-up benefits from reliable Internet connection and energy supply. But what happened this month is that for X reason, my DIY 3node has been unplugged from the wall socket. I only realized this 20 hours later. So I will not benefit from the farming reward this month.

If there was a way to send an email linked to the farm (or a text sent to a cellphone) to tell the farmer when the 3node goes offline, the downtime of my farm would have been very little as I was able to replug but simply didn’t notice on time.

So, to have a system that sends a warning as soon as the node goes offline could drastically help the overall uptime of farmers as it would reduce such avoidable errors. From what I understand, the grid scans farms’ status every few minutes or so. The system could at this moment send the warning if the farm goes offline.

I know this idea has been talked about before in the chat. I thought it could be good to bring it up here. I haven’t read about it in the forum so far.

I’ll be happy to read opinions of others on this topic and on the general uptime requirement for farmers.

This community is great! Thanks @scott for this discussion. :smiley:

4 Likes

There are some corncerns for me as a newcoming DIY farmer.

  1. Uptime
    Last month was my first month. Rewards transferred into my wallet for 8 days uptime, made me content.
    This month I bought a high performance UPS to strengthen the reliability of my 3nodes, then after 3 weeks of uptime, my internet went off for appr. 20 hours. It made me sick because I had nothing to do about it, my nodes are placed in my family home in a small village in Hungary, with only one internet provider. I turned it back on, but from my understanding, this month’s 96-97% uptime worths zero tokens.
    I think the soulution for this would be that every year the uptime should be checked, and if on average it would be above let’s say 98%, then one would get the rewards one couldn’t get during the year (as in my case: this month).

  2. V3 vesting
    For me this is the reason I haven’t yet encouraged anyone in my environment to participate, because this adds huge risk factor in my opinion:
    Let’s say one would be up for 23 months without utilization, then his nodes completely get wrecked, meaning he wouldn’t get any tokens because one’s tokens would still be vested and locked

Also, vesting without incentives is from an economic standpoint is a waste of resources. Providing liquidity for pancakeswap gets you 0.17% apy, with vesting you lose the time value of money.
I think the soulution for this would be to create benefits for farmers whose coins are locked (vested) so that they can earn passively, just like staking.

Also, in the V3 monthly farming email would be nice to show:

  1. How many tokens earned and staked that month
  2. How many tokens earned and staked cumulatively
  3. The staking incentives (X% apy)

Getting notified when you’re offline is also a must have feature, like idrnd wrote.

These are my ideas and concerns moving forward, hopefully a very nice and meaningful journey awaits us, so I’m happy I can participate after all. :slight_smile:

3 Likes

I think this idea is good and should be worked on. Here’s some thoughts around this.

This idea might be altered with some tweaking. The concept is great. So first of all, I think the proper number is 99,8% uptime minimum per month, so it’s 14,4 hours per month of maximum downtime.

Let’s say you can have 99,8% uptime year-round, but one month you get some trouble for X reason and you bust the 14,4 hours window, then you would still get reward of the month (once the year is done). That sounds interesting and comprehensive on the farmer’s side of the equation. But is it acceptable on the customer’s side of the equation?

If we think about it, then maybe it would be too much, since it means you could be off for 0,02 * 24 * 365 = 174,72 hours straight ([1 - 98%]*24 hours * 365 days). Which would be very un-attractive for potential customers who want to store on your cloud. Imagine if you need your cloud and it’s off for seven days straight. It would clearly be a rare phenomenon, but extreme cases must be taken into account in this domain. So overall the idea is great. I simply think out-loud and share some thoughts on what would be the pros and cons and what parameters could be tested.

Maybe the ThreeFold team could ponder on this and find a middle ground. For example, let’s say you need 99,8% uptime for two months in a row, instead of one month. This would give you 28,8 hours of downtime straight maximum until you lose reward, and as you said, it could be only adjusted at the end of the year so it still encourages consistency on the farmer’s end. Then it could be still fairly attractive for the customers and it would give room for some of the farmer’s occasional downtime.

Let’s see what ThreeFolders think of all this!

Thanks, @Scott for starting this one.

I have not given a thought about the uptime requirements before until browsing the last forum e-mail summary and up until recenlty I did not pay much attention to it. But was always eager to find a way to monitor my nodes - asked a couple of times on telegram. Am most of us are busy with other stuff I assume I needed a solution that won’t take much of my time and I finally got to do one. It is not very elegant - a bash script ran by cron on one third party cloud server to monitor and send an e-mail alert if any of my farming nodes goes down.

I like @Mik and @tributto ideas for alerting the farmer and total yearly uptime threshold.

To add to this I will share some of my own trials/errors when I was assuming that all nodes are working and online.

Site 1 setup: Dual ISP - fibre symmetric 100Mbps International and a 1Gig domestic peering, second ISP: cable 50Mbps with automatic failover - pfSense GW firewall/router, small managed Cisco Switch, Node1: HP MicroServer with iLO, Node 2 : PLUTO V2, Small UPS it can hold on the rack power for about 15min, Tasmota energy monitor with temperature and humidity sensors all in small wall attached rack with passive and active cooling. Dedicated setup for ThreeFold farming.

Site 2 setup: domestic ISP & router, Node 3 patched to it in another geolocation: Intel NUC with 12V power adapter.

I live and work on Site 3 in a location that is a few hours away by plane from Site 1 ( the site is mainly unmanned in eastern EU ) and an hour from Site 2 ( my friend live in EU )

Site 1 so far has suffered from a couple of issues:

  • A neighbour trimming the vine in the back yard and cutting my primary ISP fibre it took a few days before I release I’m running on failover ISP; then the primary ISP was notified and it took another week as it was weekend until splice and fix and restore the primary line - in this case only speed was lower.
  • A heavy thunder storm ( we have a few of those every summer ) took out the town power grid. Power was restored after 16 hours, Node 1 come back online, Node 2 needed to be powered on manually ( realised what happened after a few days ) and it took a week to arrange with my mum to go on-site and push power on! )
  • Occasional power grid outages some of them are mitigated by the UPS as they are short blips; longer lasting than 20min would bring Site 1 down, all will come back okay usually.
  • A few months ago my secondary ISP suffered a week-long blackout due to a fire in one of its central POPs, this did not affect my setup just nocked down my Internet failover, at the same time there was a brownout there was flip-flop on the primary ISP.

Site 2: As my friend was moving houses it took quite a long time before he had everything Internet& house to settle and plug in our joint node 3.
One day he calls me and asks me if Node 3 is online - and I confidently respond - oh yes of course it is - I did a check on the https://explorer.grid.tf a week ago, then he goes on - I do not see any lights on the Intel NUC, it turns out that some of his kids unplug the Node as there was no power for his PS3 and then left it unplugged behind the cabinet, so we were unaware for how long this was going on? A few days or week?

I’m trying to visit and check https://explorer.grid.tf once a week but that’s highly inefficient and insufficient as I sometimes forget to do it, as you can see from the above live examples.

We do need alerting build in either by farmers e-mail or via the push-notifications that could be in the threefold connect app if the TFT address is registered for farming. Or a third-party solution if anyone has the time to invest and create one that lives on the grid that leverages explorer.grid.tf APIs?

Cheesrs,
Martin

1 Like