Monthly Archives: August 2015

FreeNAS 10.1 as a VM in vSphere 6.0

I wanted to write a blog about my FreeNAS installation. I’ve been testing out FreeNAS 9.3 and find it to be well suited for my home-lab.

Having read through several posts about not to run FreeNAS as a VM, and others blogs saying, “you can, but shouldn’t”, and some “yes you can, just make sure…” I wanted to try out for myself and find out if I could make a stable setup in my home-lab.

To start with, I have been running FreeNAS 9.3 on one of my physical hosts, booting from a USB. The setup was pretty stable, but I believe my cheap USB stick that I used for boot died as after a reboot yesterday the bios did not find the USB drive to boot from.

That gave me a reason to make some changes. I want to test out FreeNAS 10.1 that is available as a nightly build. I also didn’t want to run the FreeNAS setup on one of my physical hosts as the host is a Dell R710 server, Dual X5675 Xeon 3Ghz 6 Core CPU’s, having 288GB of ram, 6 x 2TB disks and a Dell H700 Controller. – The machine was a total overkill just to run FreeNAS for my other Dell R710 VMware host with same specs.

So, the main issue I have in regard of running FreeNAS that uses ZFS, is the fact that my server has the Dell H700 controller (LSI 2108 based), and that controller is unable to work in IT mode (IT Mode is a “non-Raid” mode, where each HDD is visible to the OS without creating Disk Volumes on the RAID controller)

ZFS wants to see the pure disks without a Raid controller, and some controllers can be installed with an IT mode firmware or have this natively as the Dell H200 controller. – I didn’t want to experiment with cross-flashing the controller I have with an original LSI firmware, and I’m not sure if that would enable me to run IT mode on the LSI 2108 chip anyway.

I decided to go ahead, and find a solution I could use, and what I found that was recommended was to present the SAS controller via PCI Pass-through to the FreeNAS VM, and as I wanted to use all me 6 2TB Drives for the ZFS system, and the R710 server I have has 6 3.5″ HDD bays, I had to find a way to create a datastore for the FreeNAS VM configuration files and boot disk. This option turned out to be a no-go for me as I only have one controller in my server and I can’t use PCI pass-through as well as have a datastore for the FreeNAS VM. I carried on though and went with the option to use RDM for the disks to the FreeNAS vm.

What I did was to carve out a 50GB volume of one of the 2TB disks when I created the Virtual Disk in the H700 controller.

I then created a secondary volume for the remaining space on the disk. What you have to note here, that you have to create the other VHD’s with the same size as the first VHD, as FreeNAS won’t be able to put different sized disks into the same ZFS raid volume. For each VHD, I set Read Policy to “no read ahead” and Write policy to “Write Though”

I still boot the ESXi from a USB so this 50GB volume should be free for running the FreeNAS VM and host the ESXi system logs

The next step is to install ESXi 6.0 on the new USB Stick, and that process is straight forward and I don’t want to spend this blog post on the whole ESXi installation process, but I wanted to share a screenshot to show how ESXi sees the volumes and the USB stick.

When the ESXi installation is finished and basic settings have be set for the host, I create a datastore on the 50GB volume and name it “FreeNAS-Boot”. I install FreeNAS 10.1 on this datastore like a normal VM.

I give this VM 2 vCPU’s, 64GB or RAM and a 20GB HDD. I select to reserve all guest memory for this VM, as the datastore does not have space to hold the RAM disk file.

At this stage, the FreeNAS VM has only the boot disk, and I install the system on this device.

When I initial installation is done and network settings, DNS and such has been set, I shut down the FreeNAS VM and add the storage network adapters.

For the storage network, I have prepared on the host 4 x iSCSI enabled VMkernel ports.

Each switch has a VMkernel port and a standard VM port group.

In my hosts I have 4x 1Gbit On-board Broadcom QLogic 5709 based NICs for iSCSI, and a PCI express dual port Intel I350 based adapter for management and VM traffic

Now I add 4 network adapters to the VM, each on the separate iSCSI portgroup

And on the FreeNAS side I set the IP address and subnet mask accordingly.

Now I have the network connections set up, and next step is to get those Virtual disk from the H700 Controller up to the FreeNAS VM. Before I start I shut down the FreeNAS vm.

There are 3 ways you can go about this

  1. Create a datastore for each of the volumes , and present a HDD to the FreeNAS vm
  2. Raw device map each volume up to the FreeNAS vm
  3. Use DirectPath IO and present the H700 controller up to the FreeNAS vm

I would like to go with option 3 on this, but as I have the 50GB datastore on the controller, it’s not free for DirectPath. To have a look how this option is, I went to the DirectPath I/O settings.

When you have selected OK, I get a warning:

And even though I select “Yes” and reboot, the setting defaults back to not having the H700 controller in Pass though mode. If I want to experiment with this option, I have to have a separate controller for the FreeNAS VM but the R710 server I have has a fixed backplane for 6 drives. I would have to add an SAS controller with external connections and have some external SATA or SAS disks.

Next option I wanted to try was to add the volumes as raw device mapped disk to the VM. This is by default disabled for local disks.

VMware’s KB 1017530 shows how you prepare the disks so they can be added as RDM disks using vmkfstools.

In my case this was the list of devices and commands to create the correct .vmdk pointer files.

The command “ls -l /vmfs/devices/disks/ ” gave me a list of devices:

Resulting in those commands to create the .vmdk pointer files.

Now I could add the disks to the FreeNAS vm as “Existing Hard Disk”

Now a big portion of my coworkers yell at me, “you told us never to use RDM’s!” – And from what I read on the FreeNAS forum, I would be hanged for this. – But remember this is a lab installation in my home and for most part I want to try things here and see if they are working or if they give me trouble…

Anyhow, – RDM’s are set up and the FreeNAS now has 6 drives to play with

I decided to create a Raid 10 volume. The FreeNAS GUI is not so clear about how to create a Raid 10, but the process is that you select “Mirror”, and then select 3 stripes of mirrors

This gives me the best performance and 5.3TB disk space for my VMs.

Next thing is to set up iSCSI and have it listen to the 4 interfaces I have set up. I’ll create a Part 2 of this entry for this at a later time.

Hope you find this post useful and if so, share.

VSAN 6.0 in a nested ESXi 6.0 lab

I wanted to test VSAN in my lab without having to go out and buy SSD’s or invest in more hardware in my lab

The obvious path was to spin up several ESXi VM’s and do the settings in regard of networking and set normal HDD based volume as SSD. And to make things easy for you I took screenshots and wrote down every setting and step I made in this blog post.




To prepare networking for the ESXi VM’s you have to set Promiscuous mode to “Accept” in the security settings for the portgroup you place your ESXi VM’s on. You should not do this in a production installation on your whole vSwitch. In my lab I created a “NestedESXi” portgroup, where I enabled promiscuous mode by overriding the vSwitch default setting of “Reject” a VMware KB article explains this a bit more

This allows packets to travel from your physical nic on your ESXi host, up to the virtual nic of your virtual ESXi host, and up to its virtual VM’s virtual nic. Think inception + communications between each state.

Next thing to do is to create the ESXi VM’s. Select “Other” in “Guest OS Family”, and select “VMware ESXi 6.x” under “Guest OS Version.

This is pretty straight forward, but there is one setting in the “customize hardware” tab, and that is the option to set “Expose Hardware assisted virtualization to the guest OS” under CPU section.

Other settings on the VM’ is 2 cpu and 16GB of RAM (VSAN 6.0 memory requirements) state that each host should have a minimum of 32GB memory to accommodate the maximum number of 5 disk groups and a maximum of 7 capacity devices per disk group, – but in this lab test where I will only present 1 SSD and 1 HDD to the VSAN cluster, 16GB for the ESXi VM should work fine.

For the disks, I add one 4 GB disk for ESXi Installation, one 50GB disk to act as a simulated SSD disk, and one 150GB disk to act as a capacity device

I also in this step I select the “NestedESXi” network port group I prepared earlier.

I created 3 identical ESXi vm’s like this, and on more that had no extra hard disks, to test out the remote storage access of the VSAN cluster. VSAN requires 3 hosts as a minimum, with minimum 1 flash device and 1 spinning disk.

Next thing is to add the VM’s to vCenter as ESXi hosts.

I had earlier assigned IP addresses and created DNS records for the ESXi vm’s and I added the new hosts into a folder just for housekeeping reasons.

Before I create the VSAN cluster, I have to prepare the 50GB Hard disks and mark them as flash disk. In vSphere 6.0 this is really simple, just select the disk device and click the “F” button

This gives me a confirmation dialog to mark the selected disk as flash disk, and there you hit “yes”

This will mark the drive type to Flash

I also have to prepare a VMkernel port for VSAN SAN traffic. In this lab I’ll use the default vmk0 adapter for both management, vmotion and VSAN traffic. In production you should separate this though.

I do this for the other 2 ESXi VM’s and now everything is set up to create a VSAN Cluster.

To enable VSAN, select the “Turn on” checkbox under “Virtual SAN”

And then add your nested ESXi Hosts to the cluster.

After a minute or two, all the disks for the nested ESXi hosts automatically joined the VSAN and created a vsanDatastore.

And that’s it! – Now I have a VSAN datastore in my nested ESXi cluster.

As this is nested, using “fake” flash devices, I don’t expect to get much performance out of this, but for testing the process of creating a VSAN cluster this setup works great.

I hope you like this post, and send me your thoughs in the comments or on twitter.