TF Product Updates: February 2023
My name is Sasha from ThreeFold’s Product Management, and I would like to give my monthly product update, this time - ThreeFold Product Update February 2023.
After completing our TFGrid v3.8.0 release (testnet) earlier this month, the team is moving forward in developing the next version of the grid - TFGrid v3.9.0. Meanwhile, bugs fixes from v3.8.0 and urgent updates will be simultaneously released under TFGrid v3.8.1 and so on.
Let’s take a close look on what the development team is working on for this version, and what have been completed so far. Please keep in mind that this is a product update highlight. The content of this post will highlight some of the most awaited features and component improvements of TFGrid v3.9.0, and by all means, won’t cover all of the updates that are going to take place within the version.
TFGrid V3.9.0 - Progress Updates and Feature Highlights
Add Docker Compose Reproducible Builds for TFGrid
On this release, we are adding Docker Compose for TFGrid. Docker Compose is a tool for defining and running multi-container Docker applications. We are adding this in order to allow everyone with two physical computers to build and deploy a full TFGrid stack, deploy their local zeros-nodes and workloads.
Some of the key features of using Docker Compose:
- Have multiple isolated environments on a single host
- Preserves volume data when containers are created
- Only recreate containers that have changed
- Supports variables and moving a composition between environments
This will make it easier for everyone to have their own TFGrid deployment. In order to accomplish this, the development team is creating and applying both the Docker Builders and the Docker Compose to TFGrid Components (TFChain, GraphQL, ZOS, and others.).
FarmerBot v1.0.0
The Farmerbot is an opt-in feature that can be run seperatly from the chain and zos. Click here for more details and progress update forum post around Farmerbot and Capacity Planning.
Brief Description
On TFChain, power state and power target of the node will be added in order to let the farmerbot control the power state of it’s nodes. The farmerbot can power down, power on nodes as it sees fit. The farmerbot should expose a method on its actor to allow a user or tool to query the most optimal node in its farm. It should try and match the workload with the current online / offline nodes and power on a node if it needs to accomodate a workload.
Farmerbot Features v1.0.0
Configuration and shutting down (power saving)
- Farmers can define their configuration in an easy markdown format
- Through WOL (Wake on Lan) mechanism, machines can be turned on and off
- The Farmerbot should contact ZOS nodes through RMB (ask for used resources etc)
- Based on the info it will shut down nodes if needed. It will change the powertarget of the node on TFchain which will emit an event that the node will catch and handle upon.
- The provisioning scripts need to be adjusted to wake a node when required
- The farming manager checks which nodes can be turned off
Handling new jobs and Reactivate offline Nodes
- When the TSClient wants to deploy on a node it will create a job configuration and send it to the farmerbot via via RMB
- The jobs will tell the Farmerbot that a client wants to deploy a job on a node
- The farmerbot will execute the jobs by bringing up the node if it is offline: it will change the powertarget of the node on tfchain, which will emit an event that the other nodes in the farm will catch. They will then send the WOL packet to that node.
- The Node is reactivated and ready to receive new job.
Waking up nodes periodically
- The farmerbot will periodically have to wake nodes who are down to send uptime reports.
Quick Q&A on Farmerbot (v1.0.0)
Why do farmerbot needs to wake nodes up for uptime report, if its unused and shut down?
Because we want to know that the node is still available (is still there to be used). A farmer can unplug a node at any time so we can’t distinct a node that has been unplugged from a node that is off.
Can farmer opt in for farmerbot but also make exemption for certain nodes to never be shutdown regardless (for v1)?
We could do that, but as we are in the testing phase right now, it is not a good idea. But since The farmer is already able to configure the capacity planning rule: the percentage of how much unused resources he/she wants to be available, this deemed to be unnacessary. Whenever that percentage is reached a new node is powered on.
Will this feature require a Master node that will never be shut down?
Yes, that is a requirement. Nodes are powered on by other nodes (WOL packet comes from one of the nodes in the farm). So if you want to be able to power them back on you have to have at least one node powered on.
Turning on off nodes - at the end, how does a farmer decide what ‘almost full’ means? what do we decide?
Nodes are powered on in two cases: when the total resource usage reaches a certain percentage (defined by the farmer) or when there is an incoming find_node request and that the selected node is off.
Regarding the definition of full nodes: The farmerbot will know the amount of resources (used and unused) of each node. Every time a find_node request is sent the farmerbot will reserve the required capacity (for some amount of time). It basically substracts the resources from the total amount.
We periodically ask the resource status to the nodes and we then decide if we want to shutdown a node. So full just means the total amount of resources has been reached. Almost full means we reached a certain percentage.
Distributed RMB Relays
To remind everyone again of what RMB is, RMB (reliable message bus) is a set of tools (client and daemon) that aims to abstract inter-process communication between multiple processes running over multiple nodes. They are like service messengers for the nodes
Using RMB as our way of as our process communication tool is beneficial for user privacy because it allows process communications without the need to reveal network addresses or the identity of the calls, unlike HTTP(S), where the caller must know exactly the address (or DNS name) and endpoints of the calls. Instead, RMB requires you to only know about the Twin ID (numeric ID) of where the service can be found, and the call request itself.
Twins are stored on TFChain. By storing the twins on TFChain, the identity of the Twins is granted not to be spoofed or phished. When a twin is created, it needs to define 2 things: its RMB Relay and Elliptic Curve public key.
On this release, our team is planning to move away from the need to connect to Yggdrasil (private) network to deliver and receive process messages, to instead, running distributed RMB Relays and connecting it to RMB-peers.
RMB-peers will establish a connection to a nearby RMB relay, and update its information on TFChain. Once the connection is established, RMB relays then can receive messages from their clients, to forward them to either a directly connected client or to another RMB relay.
For more information on distributed RMB, you can read its specifications here.
NEW Freeflow Twin Mobile App (Beta)
Good news for Freeflow Twin’s Users! Soon on this release, you will be able to start using your Freeflow Twin’s account on both your android and apple devices. Our team is implementing the mobile version of our Freeflow App (beta). The app itself will be downloadable via Google play store and Apple store. Since Apple takes a longer time to approve applications onto its market, there is a possibility that the android version of freeflow app will roll out earlier than its iOS counterparts.
FreeFlow Twin is a digital experience that presents alternatives to centralized social media, messenger, file storage, docs, web browser, and video conferencing.
Mobile App Preview
Freeflow Mobile App will have the same functionalities and technical specifications as the one currently live on its desktop version (Profile Dashboard, Instant Messaging, Video Calls, File Storage, and Social Media Post). Click here to try and start using Freeflow App (Beta) the desktop version.
Weblets
NEW Freeflow Weblet (TF Playground)
On this release, we are going to add Freeflow to our TFPlayground weblet list, so that everyone could deploy their own Freeflow on the TFGrid.
Enable TFConnect Login on Discourse Weblet
From the previous release, users could already deploy their own discourse instance on the TFGrid via TFPlayground. On this release, we are making it even easier for everyone to login to their own discourse instance, by simply eliminating the old-fashioned email signup process that are usually required on the centralized discourse, and replacing it with a TFConnect login mechanism. New users would simply just need to click on ‘Login using TFConnect’ button on your discourse to start using the platform.
NEW Wordpress Weblet
On this release, we are going to add Wordpress to our TFPlayground Weblet list, so that everyone could deploy their own wordpress-based website on the TFGrid. WordPress is a content management system (CMS) that allows you to host and build websites. Hosting your own wordpress website via TFGrid would be beneficial to users for its decentralized nature, compared to hosting it on centralized servers.
NEW Jitsi Meet Weblet
On this release, we are going to add Jitsi Meet to our TFPlayground Weblet list. Jitsi Meet is a fully encrypted, 100% open source video conferencing solution. Fully-encrypted by nature, and decentralized.
WebWG
On this release, our team has created a set of networking tools to be used on the TFGrid, which we called “WebWG”. The main idea is to create an application offering a variety of networking functions to the users.
How it Works
Nodes which have public configs will run a “server” binary, while users/hidden nodes run a “client” binary. Configuration (when needed) is intended to be done through a to-be-developed tendermint-based blockchain. For the first version of WebWG, we support the following functionality:
TCP-Proxy with connection sniffing such that we can offer multi tenant proxying from public nodes to hidden nodes (similar to tcprouter). Note that we don’t intend to terminate TLS.
TF Dashboard
Some of the exciting new features for TF Dashboard on this release are:
-
Adding TFConnect App link to TFDashboard so users can easily navigate back and forth from/ to their TFConnect App from the dashboard.
-
Add New Node Filtering feature onto the the dashboard. This feature was previously requested by our community member on the forum, and on this release we’re implementing it. Users will be able to sort nodes by countries, node types (dedicated/non-dedicated) nodes, by network types (IPv4, IPv6) and by availability.
-
Currently we are supporting farm sorting by farm name on the TF Dashboard. On this release we are working on allowing users to sort farm based on its public IPs availability (available/used public IPs).
What’s Next?
Within the next weeks I will come back to announce more updates from this upcoming release, hopefully with some feature previews and more visual representations. Feel free to comment below if you have any question about TFGrid v3.9.0.
Again, I would like to remind the community that changes in priorities can happen within February 2023 as announced, thus the next updates about TFGrid v3.9.0 progress might differ from the content of this update.
We’d like to also invite you to join our TFGrid Tester Community telegram channel to meet other community members, test our products, receive new product updates and announcements, and start some conversations on our new improvements, and many more.
Love and Gratitude,
Sasha Astiadi
ThreeFold Product Management