April 2023 Minting [Closed]

[scott]

Hi Everyone,

We are tracking reports from some farmers that their nodes did not mint in the way that they expected for the April period that was paid out last week. First, let me just say that we appreciate your patience while we investigate what happened and determine a course of action to provide missing rewards to farmers if and where appropriate. If you think there was an issue with how your nodes minting for this period, please contact our support team with details. To do that, you can visit this site and click “Chat with us”.

So far, what I can say about the root cause is that some updates were made to the minting code for this round of minting, including but not limited to how minting is handled for nodes managed by the farmerbot. Part of the changes were additional checks that nodes are reporting uptime in a way that’s consistent with the expected behavior of Zero OS. We will need to look further into what exactly happened with the subset of nodes that are affected.

Again, thanks for your patience and understanding. If you have any general questions, feel free to ask below. For any specific concerns regarding your nodes, please contact the support team as described above. We will be updating farmers individually through the support system when there’s more information about their situation, and we’ll also provide a general update as we know more.

[scott]

Thanks again for your patience if you’ve been waiting on minting rewards from April. We are moving ahead with a plan to resolve this, as follows:

1. For nodes that were found to be online and otherwise eligible for minting, we will mint again according to the minting parameters used in the previous minting cycles before April

2. Consider any remaining nodes for payment from a treasury wallet on a case by case basis

This two part approach is necessary because in addition to exclusions for nodes that would have minted in the past, we also had a smaller number of cases related to changes with the recent 3.9 release that need to be considered separately.

What happened?

3Nodes are considered for minting based on the uptime reports that they submit to TF Chain. Under perfect circumstances, nodes submit consistent uptime reports every 40 minutes, with a small amount of deviation due to the need for TF Chain to achieve consensus.

In practice, nodes can submit uptime reports that don’t adhere to the expected pattern for a variety of reasons, including unexpected behavior in Zero OS and issues with the nodes’ hardware. Therefore, the minting code accommodates this, within a certain range, but nodes that behave outside of those limits are excluded from minting.

For the majority of nodes that didn’t mint in April, the cause is that they produced irregular uptime reports due to some aspects of Zos operation that have since been optimized. Specifically, this means better handling of how uptime is reported when Zos has trouble connecting to TF Chain nodes. Along with updates for 3.9, the minting code executed for April’s minting included some tighter requirements for uptime reporting that excluded a large number of nodes.

Why did it take so long to fix?

There are a few reasons we didn’t reach a resolution sooner. The first is this was a complex situation with several changes that could affect minting happening in close proximity, including the 3.9 release and the introduction of the farmerbot. We also had some debate about whether to pay all of the farmers from treasury funds or to mint again. And as always, we are a relatively small team working on a huge project, so while this has been a high priority, there have been other items demanding attention too.

Finally, and most importantly, minting is of critical importance and we wanted to be sure that we took the correct approach.

Looking forward

We are further responding to this incident by evaluating the minting process and looking for ways to improve. As always, TFT are only minted for nodes contributing capacity to the ThreeFold Grid according to blockchain data and and our proof of capacity rules. In this case, we were initially just a bit too cautious about which nodes to include.