DIY 3node Server Guide (Dell R620-720)

In the following DIY guide, you will learn how to turn a Dell server (R620, R720) into a 3node farming on the Threefold Grid 3.0.

Screen Shot 2022-02-02 at 9.18.26 PM

Dell R620 1U server

image

Some will recommend to wear anti-static gloves as shown here. If you don’t have anti-static gloves, you can simply never forget to touch the metal side of the server before manipulating the hardware. Your hands will discharge the static on the outside of the box, which is secure.

image

Here is one of the two 2TB SSD NVME m.2 that we will install on the server. Above the SSD is the PCIe Gen 3, x4 that we will use to connect the SSD to the server.

image
You can see at the left of the adaptor that there is a metal piece that can be used to hold more firmly the PCIe adaptor and the SSD. We will remove it for this DIY build. Why? Because it is not necessary as the adaptor can hold the weight of the SSD. Also, this metal piece is full while the brackets in the server have holes in it. This will ensure a better airflow and thus less heat.

image

We remove the screws with a star screwdriver.

image

image

This SSD already has a heatsink. There is no need to use the heatsink included in the PCIe adaptor kit. If you remove the heatsink or the sticker under the SSD, you will lose your 5 years warranty.

image

When you put the SSD in the adaptor, make sure you have the opening in line with the adaptor.

image

image

Fitting in the SSD takes some force. Do not overdo it and take your time!

image

It’s normal that the unscrewed part is lifting in the air before you screw the SSD on the adaptor.

image

To screw the SSD in place, use the screwdriver included in the PCIe adaptor kit.

image

image

Now that’s a steady SSD!

image

It’s now time to get under the hood! Make sure the case is at the unlocked position. If you need to turn it to unlocked position, use a flathead screwdriver or the like.

image

image

Lift up the lock and the top server plate should glide to the back. You can remove the top of the server.

image

Here’s the full story! R620 and all!

image

image

image

To remove this plastic piece, simply lift with your fingers at the designated spot (follow the blue line!).

image

Here’s the RAMs! This R620 came already equipped with 256GB of rams dispersed in 16x16GB sticks.

image

To remove a stick, push on the clips on both sides. You can do it one at a time if you want. Make sure it doesn’t pop out and fall on a hardware piece! Once the clips are opened, pull out the RAM stick by holding it on the sides. This will ensure it does not get damaged.

image

Here’s the RAM in it’s purest form!

image

Here you can see that the gap is not in the middle of the RAM stick. You must be careful when inserting the RAM. Make sure you have the gap aligned with the RAM holder.

image

When you want to put a RAM stick in its slot, make sure the plastic holders on the sides are opened and insert the RAM stick. Make sure you align the RAM stick properly. You can then push on one side at a time until the RAM stick clicks in. You can do it both sides at once if you are at ease.

image

To put back the plastic protector, simply align the plastic piece with the two nudges in the metal case.

image

image

We will now remove this PCIe riser in order to connect the SSDs.

image

Optional step: put the SSDs and the PCIe riser next to each other so they can talk and break the ice. They will get to learn one another before going into the server to farm TFT.

image

Just like with RAM sticks, you want to make sure you are aligned with the slot.

image

Next, push the adaptor inside the riser’s opening. This takes some force too. If you are well aligned, it should be done with ease.

image

This is what the riser looks like with the two SSDs installed. Now you simply need to put the riser back inside the server.

image

Push down on the riser to insert it properly.

image

It’s good to notice that the inside of the top plate of the server has great pictures showing how to manipulate the hardware.

image

Now you will want to plug in the power cable in the PSU. Here we show two 495W PSUs. With 256GB of RAM and two SSDs NVME, it is better to use two 750W PSUs. Note that this server will only use around 100W at idle. There are two power cables for redundancy. The unit does not need more than one to function.

image

Plugging in the power cable is pretty straight forward. Just make sure you have the 3 pins oriented properly!

image

Then you plug the power cable in a surge protector. If you have unsteady electricity at your location, it might be good to use a UPS, uninterrupted power supply. A surge protector is essential to avoid overpowering and damaging the server.

image

image

image

Before starting the server, you can plug in the monitor and the keyboard as well as the ethernet cable. Make sure you plug the ethernet cable in one of the four NIC ports.

image

Now, power it on!

image

The server is booting.

EXTRA:

image

image

If you want to change the DVD optical drive, push where indicated and remove the power and SATA cables.

The hardware part is done. Now you will want to set the BIOS properly as well as get the bootstrap image of Zero-OS.

Zero-OS Bootstrap Image

With R620 and R720 Dell servers, UEFI does not work well. You will want to use either a DVD or a USB in BIOS mode.

Go on https://bootstrap.grid.tf/ and download the appropriate image.

Write your farmer ID and make sure you select production mode. This is grid 3.0. Things are getting serious.

image

image

Use the ISO image for DVD boot and the USB image for USB BIOS boot (not UEFI). We use the farm ID 1 here as an example. Put your own farm ID.

For the ISO image, download the file and burn it on a DVD.

For the USB image, with Linux, you will want to do:

sudo dd status=progress if=FILELOCATION.ISO(or .IMG) of=/dev/sd*. Here the * is to indicate that you must adjust according to your disk. To see your disks, write lsblk in the command window. Make sure you select the proper disk!

For Windows, you can use Rufus and Balena-Etcher, two free and open-source software that will let you create a bootstrap image on a USB key.

BIOS Time

image

Before starting the server, plug in the USB bootstrap image. You can also insert the DVD once the server is on.

When you start the server, press F2 to get into System Setup.

image

image

Then, select System BIOS. In System BIOS settings, select Processor Settings.

image

image

There, make sure you have enabled the Logical Processor (Hyper Threading with HP). This turns 8 cores into 16 virtual cores!

image

It is also good to take a look at the Processors and make sure you have the right ones. Here we have two E5-2640 v2 at 2.00GHz. This is thus 16 cores, or 32 vcores, and a very lower power consumption.

image

image

Go to System BIOS Settings and select Boot Settings. In Boot Settings, choose BIOS and not UEFI as the Boot Mode. You need to save your preferences and comeback to select BIOS Boot Settings.

Once back in BIOS Boot Settings, go to Boot Sequence. Depending on your bootstrap image of Zero-OS, select either the USB key or the Optical Drive CD-DVD option. The name of the USB key can be Drive C or else depending on where you plugged it and your server model.

You can also disable the booting options you do not need. It can be good to have a DVD and a USB key with the bootstrap images for redundancy. If one boot fails, it would try with the other options of the boot sequence. This can be done with 2 USB keys too.

Boot Sequence Retry enabled will simply redo the booting sequence if the last time did not work.

You can then save your preferences and exit. Your server should restart and load the bootstrap image.

image

When you see this, be happy and wait for Zero-OS to boot.

The first time you boot a 3node, it will be written: “This node is not registered (farmer ***: NameOfFarm). This is normal. The Grid will create a node ID and you will be able to see it on screen. This can take a couple of minutes.

Once you have your node ID, you can also go on the Threefold explorer to see your 3node and verify that all is good.


Note that the main difference between the R620 and the R720 is that the former is a 1U and the latter a 2U. 2U servers are usually less noisy and generate less heat than 1U servers since they have a greater volume. In the R720, fans are bigger and thus less noisy. This can be an important factor to consider. Both offer great performances and work well with Zero-OS.

Note: For Dell servers, if you want to use the onboard storage, you will have to either cross-flash the RAID controller with an “IT-mode-Firmware” (for more info, read this documentation and watch this video) or get a DELL H310-controller (which has the non-RAID option). Otherwise, you can install a NVME SSD with a PCIe adaptor as shown in this DIY guide.


If you have questions or comments, please share your thoughts in the comment section.
I hope you learned something here and that it helped you build the New Internet.

8 Likes

Killer guide.

Note on HP servers “logical processors” is called “hyper threading”.

Its also a good idea to set it to automatically boot up when power is applied in case of an outage, this is under various headings depending on the BIOS.

2 Likes

Thanks. I added hyper threading.

If you have the steps to automatically boot up when power goes out, I could add it.
Otherwise, I will check on the servers next time I do maintenance! Good idea.

Thanks for the amazing guide.

I’m thinking of building one, but I wanted to ask something. You used normal NVME SSD (consumer grade) or they need to be different?

Thanks!

1 Like

I use both. Nothing wrong with consumer grade, but I try to keep it at least 1000TBW for a 2tb. My goto is the 2tb Inland Premium M.2 (not platinum).

2 Likes

@FLnelson answered very well!

TBW is good to look at, and if the seller has a warranty that lasts many years, it’s a good sign that the NVME SSD is made of high quality material. For example, I have a NVME SSD with 5 years warranty and 1300+ TBW. This should last!

2 Likes

Thanks so much for you answers @Mik and @FLnelson. Now I know what to look for :smiley:

Edit: Just to check and be sure everything looks good, my planned server is:

  • HP DL360P gen8 with 32 vCPU
  • 256 GB of RAM
  • (x2) Sabrent 2TB Rocket NVMe PCIe M.2 (with 3115 TBW and 5 years of warranty - just has you thought me ahah)
  • (x2) Sabrent NVMe M.2 SSD to PCIe X16 / X8 / X4 Card with Aluminium Heatsink.
1 Like

I just got a HP-DL360P-Gen8 up yesterday.

Just to let you know: there is no need for PCIe M2 Storage. you can easily put the raid-controller (probably a P420i) in hba-mode and put on 2.5" SSDs. I got 8x 800GB attached to the backplane…works perfectly fine.

see: https://forums.unraid.net/topic/82007-solved-unraid-with-hp-p420i-raid-card-in-hp-proliant-dl380p-g8/

@FLnelson published a video tutorial on the exact same issue.

3 Likes

Thanks for sharing. That’s a great information to know.

1 Like

I also got another 3node up on a DELL R620 where the 2,5" SSDs are attached to the regular frontpanel backplane. Also no need for PCIe connected storage just to avoid some configs on the RAID-Controller. And especialy no crazy workarounds necessary like unplugging/detaching the DVD-drive in order to attach a single SSD there (but yes… it’s possible).

Those DELL server models most likely come along with a H710/H810 or similar RAID-Controller. What HP called “hba-mode” for their RAID-controllers is called “non-raid” by DELL. You need to know that those controllers mentioned above do not have the ability of this “non-raid” mode out-of-the-box. Therefore you need to crossflash the RAID-Controller with a so called “IT-mode-Firmware” from the controller chipset manufacturer (which is LSI Corporation btw). That might sound tricky but isn’t that hard at all. @FLNelson put out a video tutorial on this too.

But… if you are afraid of this and you really wan’t to avoid that “crossflashing” thing you can just get yourself a DELL H310-controller (these are really cheap, like 20-30€). This is the only DELL controller (for this server series) that can handle drives in non-raid mode out-of-the-box. It’s just plug and play!

If you’re running into trouble with this feel free to ask for support. I’ll give you a rope!

1 Like

Thanks for this information once again.

I’ll add this to the FAQ so others can use this.

Have a good day @Dany !

1 Like

ISSUE SOLVED
Hi I’m struggling with my HP DL380p G8 2xE5-2680v2 (40Vcores total) with 320gb RAM and 2x2tb NVMe, (2x 4.400TBW) on 100/100 fiber optical connection.

First - Little history with some instructions that I hope will save time for someone:

I’ve updated the ILO to 2.79 and System ROM to P70 05/24/2019

in order to UnRAID you have 2 options:

  1. install this 5GB .iso service pack:P03093_001_spp-Gen8.1-SPPGen81.4.iso in order to get the p420i to v.8 then the intelligent provisioning would not start when hitting F10, and i had to reinstall it. it work fine afterwards. or,

  2. load ubuntu or linux mint, with ILO remote console or bootable USB media done with balenaEtcher or similar. - don’t install it just try or load it up you’ll need it just to install this file: “hpssacli-2.10-14.0_amd64.deb” open console and type in following commands:

cd /opt/hp/hpssacli/bld
sudo hpssacli controller slot=0 modify hbamode=on

if error
sudo hpssacli ctrl slot=0 ld 1 delete

as alternative if you use linux mint:
cd /opt/hp/hpssacli/bld and run this command:

controller slot=0 modify hbamode=on

Took me a while to figure it out, since this is my first time tinkering with a rack server or linux, and I think I made a mistake to try to do both options, thinking option 1 was to update the system and option 2 to unRAID. Nevertheless i was UNable to install that “hpssacli-2.10-14.0_amd64.deb” file in any OS, because it gave me errors.

hpssa error ubuntu

So don’t be like me. :slight_smile:

That’s it. Hope it works for you guys.

At this point i did not have caddies for front HDD so i said F*** it I’m going with the NVMEs and do the upgrades later. (Upgrades are disassembling 2 to 4 Seagate and WD 4TB 2.5" USB3.0 drives that I have practically unused and plugging them in.)

Now THE PROBLEM I have:

  1. used balenaEtcher to make my two USB drives one with “Grml Live Linux” to wipe the drives, and the second one with my ZeroOS from https://v3.bootstrap.grid.tf/ with my farm ID
  2. GMRL loaded fine and i wiped the drives
  3. Second USB with ZeroOS loaded fine, it downloaded the image and started installing:
    pprob11

and I started getting following errors:
prob2
after few minutes I noticed that the time changes, i don’t know why, maybe this is the issue!? >
time change

Left it overnight and still blank screen.
blank screen
Done the same thing few times, same result. Checked in the TF explorer just in case, node is not listed. Any suggestions? …I hope I’m posting this in the right place.

UPDATE: Well, 15 minutes after sending my post, the node “magically” started working. So all things considered I guess “Patience is a virtue!” :grin:

Hope some of you will find use of my post and save time.

Fruitful farming everyone!!! :smiley:

2 Likes

OK that’s good to know that the problem was solved. Do you have any idea what helped to get it solved? You just rebooted the 3node with the bootstrap image and it worked the last time?

It looks it had problem connecting with the DHCI or the internet overall, then finally found a proper gateway.

Thanks for sharing your adventure, that’s how we get stronger as a community. Knowledge is Power, sharing knowledge is powering on the New Internet!

Hey 3bolt,

at first…congrats on getting it done! Please see comments below…

I’ve updated the ILO to 2.79 and System ROM to P70 05/24/2019

good job!

in order to UnRAID you have 2 options:

  1. install this 5GB .iso service pack:P03093_001_spp-Gen8.1-SPPGen81.4.iso in order to get the p420i to v.8 then the intelligent provisioning would not start when hitting F10, and i had to reinstall it. it work fine afterwards. or,

  2. load ubuntu or linux mint, with ILO remote console or bootable USB media done with balenaEtcher or similar. - don’t install it just try or load it up you’ll need it just to install this file: “hpssacli-2.10-14.0_amd64.deb” open console and type in following commands:

It is correct that you have these two possibilities to unRAID the controller. But… from my point of view option 2 is much more easy. First of all… in order to download the SPP (from it’s original source) you need to have an account to the HPE Support Center. The file itself is >5GB and you only need a couple of MB contained in the ISO. Since these particular files are separately available I avoid downloading/installing the SPP.

Took me a while to figure it out, since this is my first time tinkering with a rack server or linux, and I think I made a mistake to try to do both options, thinking option 1 was to update the system and option 2 to unRAID. Nevertheless i was UNable to install that “hpssacli-2.10-14.0_amd64.deb” file in any OS, because it gave me errors.

Well… installing the SPP does update the server components and tools…but… for zos it’s just not necessary (f. e. intelligent provisioning, Smart Storage Administrator…). iLo and BIOS upgrade ist important of course . Upgrading RAID-Controller, NIC and Power Management Controller Firmware can be done manually. So no need for SPP.

The reason why you have been unable to install “hpssacli-2.10-14.0_amd64.deb” is pretty simple. You just downloaded the wrong file. According to the screenshot you were trying to install the
“HP smart storage administrator Utility” (purple highlighted). But you need to install the HP smart storage administrator command line interface (green highlighted). Give it another try with the correct package…and follow the instructions. I’m sure you’ll get it done.

image

Let us know if you could fix it.

Cheers!!

PS: make sure that you have deleted all logical arrays on the controller before you try to activate hba-mode.

3 Likes

Have you checked time and date in BIOS? Did zos change the maschines time to real time? If so… it looks like the server time is set to 2h in the future.

For the error installing the package, I had trouble with the GUI installer once or twice. Go into terminal and use a dpkg command to install.

1 Like

Sharp eye noticing the wrong package. I would have missed that.

2 Likes

Mik i got to guess one of the answer is somewhere here**:intelligent provisioning would not start when hitting F10, and i had to reinstall it. it work fine afterwards**. and then I just rebooted, but while i was writing my original post i flashed the USB drive’s again, wiped the drives again, booted the system and waited. It all went well apparently- 30minutes later giver or take everything was working as it should.

1 Like

I don’t know the reason here why the time changed. From my time zone 19:20 was the correct time, not the 17:25 so i don’t know.
And also, in BIOS the time was correct, I know because I checked to see if everything is set before booting.

Gents (and ladies if any :blush:) I see names as Samsung, WD, Sabrent and whatever, but I found and bought this gems:

Transcend 220S - the 2 TB Nvme has - 4,400 TBW and I bet it costs less then any other options out there for less the TBW. Look it up.

Cheers!