There was an issue affecting a small number of recently delivered Titan nodes which prevented them from booting successfully. We have resolved the majority of cases and are working with the remaining farmers to get their nodes online. We have also identified the root cause and have taken steps to prevent further problems.
If your Titan is not booting, please contact our support team for assistance using the blue chat icon in the bottom right hand of this page or within the ThreeFold Connect App. We will be compensating affected farmers for their inconvenience. Thanks for your patience and understanding.
Background
This year ThreeFold has received a huge influx of orders for Titan nodes. We are super grateful for all of our new farmers who are growing the ThreeFold Grid, which currently consists of more than 80,000,000 GB and 20,000 cores of autonomous storage and compute capacity. Ramping up production capacity in the midst of a global computing hardware shortage has been challenging, and this issue arose as we were moving as quickly as possible to get Titan nodes to everyone who placed an order.
Technical Details
The affected nodes were shipped with an incorrectly configured BIOS, which prevented the secure booting of Zero OS over network via IPXE. Certified Titan nodes have their BIOS password locked as a security feature for the certification process. Before rolling out a fix, we needed to discuss internally whether the best course of action was to provide the passwords to the farmers or to have the nodes returned for service. Based on the small number of affected nodes and the inconvenience of asking farmers to ship nodes back, we decided to provide the passwords and instructions for farmers to carry out the change on their own.
Moving Forward
We have taken this opportunity to review the Titan setup and preshipping check process with our partner who produced these nodes. Specifically, we have clarified that all nodes should be allowed to fully complete the Zero OS boot sequence after all configuration has been completed.
Our support team is now also better equipped to handle issues with newly delivered Titans, which can also include issues with farmers’ networks not caused by the Titan itself. This will expedite the process of identifying and resolving any problems that arise in the future.
Finally, we are also providing better instructions for farmers to set up and troubleshoot their nodes when they arrive. This too will expedite the support process whenever issues arise.
Thanks everyone for your support of ThreeFold, and if you were affected by this issue, thanks again for your patience and understanding.