Nodes stopped working

#1

I have two DIY farms, farm1 has one node and farm2 has four nodes. After a power outage I noticed none of my nodes are working. They boot the OS and shortly after they connect the screen turns black with a blinking cursor.

Using the threefold connect app I upgraded my farms to V3 and downloaded the v3 OS from bootstrap.grid.tf. Now one node from 1stfarm and 1stfarm01 says it is connected from within the OS but I do not see my node online on https://explorer.threefold.io/all. I did wipe the disks and reboot all of the nodes.

Also I have three additional nodes with 1stfarm01 and when it boots up the OS says “node is not registered”

#2

All my nodes are down, and there was no power outage…
rpviewer

#3

Hi @techafterhours. Not sure what happened. Please contact support and create an issue. I have made them aware of this post and they should have a look at this post as well.

#4

I sent a message to support, I will wait for their response. Thanks.

#5

Strange things going on, when I check my farm, I see 4 nodes up and running, I go to capacity and search for the nodes, they show all down, when I go back to my farm details, nodes also show as down…

#6

Hi - please have a look here as well: https://gridproxy.grid.tf/swagger/index.html#/. This also provides feedback that the node is down.

âžś  tmp curl -X GET "https://gridproxy.grid.tf/nodes/486" -H "accept: application/json"
{"version":316,"id":"gc6zUhQRJh","nodeId":486,"farmId":25,"twinId":693,"country":"Belgium","gridVersion":3,"city":"Merchtem","uptime":534595,"created":1643039850,"farmingPolicyId":1,"updatedAt":"2022-02-21T14:12:08.608573Z","capacity":{"total_resources":{"cru":40,"sru":1999307276288,"hru":112006304628736,"mru":269509197824,"ipv4u":0},"used_resources":{"cru":0,"sru":3998614552576,"hru":112006304628736,"mru":269509197824,"ipv4u":0}},"location":{"country":"Belgium","city":"Merchtem"},"publicConfig":{"domain":"","gw4":"","gw6":"","ipv4":"","ipv6":""},"status":"down","certificationType":"Diy","hypervisor":"","zosVersion":"","proxyUpdatedAt":0}
âžś  tmp

Nodes still show the same screen as you posted at the start of this thread? Also pushed this to support.

#7

rpviewer
This is the screen after the restart this morning

#8

The unified explorer isn’t pulling in farm names for v3 farms in all cases yet. If you search by your farm ids (1336,1337), you’ll see one node online in each farm. I’m working with the dev team to update so experience is the same as for v2 nodes.

There’s a known issue where SSDs connected through server storage adapters are being seen as HDD by Zos. When Zos sees no SSD, it boots but doesn’t fully come online. Devs are working on this and we’ll be issuing some compensation for missed uptime. I’ll make a note to follow up with you once we have resolution to see how many TFT you’re missing.

#9

I see my nodes by their ID, thanks. Hopefully the devs can make the v3 experience the same as v2.

I rebooted my other nodes but they all say “node is not registered” I have replied back to the support ticket. Screenshot below.

threefold

#10

OK Scott, is there any use in keeping the servers online during that time?
The strange thing is that I did not reboot them at any point, and the SSD’s used to be recognized…
In fact, when looking at the capacity explorer, the servers are showing the correct setup regarding HDD and SSD
These kind of bugs are really detrimental to the whole farming idea, especially looking at the current electricity prices!

#11

Is there anything that I can do on my end to work around or resolve this issue?

#12

Hi Erwin, I’ll see if there is anything needed from your side. The issue is tracked here. @scott is on it as you can see.

#13

Problem has been resolved thx to Jan & crew.

1 Like
#14

:+1: good to hear and great community interaction. :pray:

#15

I power cycled my nodes and have waited 12 hours but they are still experiencing the same issue.