Tag Archives: esxi

How to run nested Hyper-V in vSphere 6.0

Few days ago I was testing a virtual –to- virtual converter software from StarWind, and as a by-product of an upcoming blog on the matter, I took some screenshot and wrote down settings that are relevant to running a nested Hyper-V in an ESXi environment.

The problem

When Installing the Hyper-V role on a windows 2012R2 machine that is a virtual machine, you get an error message saying “Hyper-V cannot be installed: A hypervisor is already running”, and to “fool” the Windows OS to believe it is running on a native x86 machine, you have to set several options for the VM

The steps when installing hyper-V on a Windows 2012R2

You select the “Hyper-V” role and hit “Next”

And you hit next, and automatically get the option to install the management tools as well, and in most cases you would want that, so you click “Add Features” to continue

And soon as you hit the add feature button you get a validation result popup stating that you can’t install hyper-V on this windows machine.

Sure, – you have a hypervisor running on your ESXi Host, – but you want to have a hypervisor running on this virtual machine as well. –

The solution

First you have to shut down the VM and remove it from vCenter inventory

Then enable SSH on the ESXi host if you have not already

Then you edit the .vmx file for the VM and add 2 lines at the bottom of the file.

vhv.enable = “TRUE”
hypervisor.cpuid.v0 = “FALSE”

Then using the datastore browser find the same .vmx file, and right click it and select add to inventory

 

 

In the web client, edit the settings for the VM and expand the CPU settings. There under “CPU/MMU virtualization (*) section, select “hardware CPU and MMU”

Also notice that the checkbox is set for “hardware virtualization”, and that is due to the hypervisor.cpuid.v0 = “FALSE” setting in the .vmx file.

When finished, boot up the VM and go back to the Add roles and Features Wizard on the VM

Now the wizard runs and installs Hyper-V on a nested VM

 

When you have this set up, and you have created a Hyper-V virtual machine, and you need network connectivity to the VM, you need to set your ESXi vSwitch security policy to “Promiscuous mode: Accept”. – This is the same setting that you have to do when you run nested ESXi and you have vm’s that need network connectivity.

When this was ready, I tried to run the Hyper-V integration services tools on the nested hyper-v VM, (this sounds like something from inception…)

I get and error “The Hyper-V Integration service can only be installed inside of a virtual machine running under Hyper-V”

I did not find a workaround for this, but if you have one, please let me know in the comments or by sending me an e-mail and I’ll update the post.

Conclusion

You can run a nested Hyper-V installation under ESXi if you need to do some basic testing. In my case I was testing a Virtual to Virtual Converter software from StarWind Software, and need create a hyper-V virtual machine, convert it to a .vmdk file and boot it up in my vSphere based lab (and via versa), but don’t expect to have a good performance on the nested VM 😉

 

Share if you feel this post to be useful.

Storage in the home lab.

Home Labs in general

When asking my colleagues what to run as a storage platform in my home-lab, I got an honest question from a fellow blogger and vExpert Rasmus Haslund (@haslund)

What are your requirements, challenges and constrains??
My answer: “Well, I want all the features and best performance, but I have limited or no budget!”

This could easily be applied to your production setup where you have the challenge of providing a stable service level, while having limited budget on external storage. So if you work for a small/medium company looking for a storage solution for your virtual workloads, read on and hopefully you can apply the solution described in this blogpost to your installation.

The challenge

As a vExpert, blogger and enthusiast for all sorts of storage and virtualization solutions, I find it necessary to have a lab at home to do tests and evaluate different solutions. I also run several vm’s for my home network that I have to take care of and have to answer to my son and wife if I screw up!

For quite some time I had a limited flexibility in regard of the lab and to maintain some level of service for my home network I had to find a better solution.

My son has a Minecraft server running that need to be up in the evenings specially, and my wife’s ideas about SLA for her e-mail and picture library in this regard is that a 100% uptime is “normal”!! So it’s tough ground to maintain and also have flexibility when it comes to testing and running some ad-hoc workloads.

In my basement there is a storage space and after I got a networking cable down there from my apartment on 2th floor, I could start up more hardware without my family being disturbed by noise and cables running all over my desk. Down there I can maintain a stable setup for my home network and have some extra hardware to play around with when I need to try out something.

When I got the chance to repurpose some servers from work I decided to redesign the home lab. It had been running from a one ESXi white-box host with 1 x Intel I7 3770K CPU and 32GB RAM and surly could befit of more CPU and RAM resources.

To set out some requirements and figure out the challenges.

The goals

  1. Maintain reasonable level of uptime and performance of my home network.
  2. Have available disk space and resources to set up a nested ESXi environment for testing different setups and solutions without exposing the home network to risk.
  3. Have a storage solution to be accessible by my 2 ESXi hosts.
  4. Minimizing heat generation and electricity cost for running the home network, but still have the ability to spin up more workloads for testing in the lab when needed.

The hardware

The servers I got for the lab are pretty massive!

3 x Dell PowerEdge R710, each having dual X5675 3,0Ghz CPU’s and 288GB of RAM. Each server has 4 x 1Gbit network cards onboard, and 1 x dual port 1Gbit NIC. Each server has the Dell H700 SAS controller (LSI based controller)

The solution

When looking for a storage solution I decided to use one of the R710 machines as an iSCSI target device as it had 6 x 3.5” drive bays. There I could place my 6 x 2TB SATA drives I previously had in my white-box server. This R710 server would become the shared storage for the 2 ESXi hosts as well as being a proxy server for my Veeam Backup installation, a Minecraft server for my son and a PLEX media server for my home entertainment system. (All those workloads that had been running on my wife’s desktop for some time, much to her enjoyment as you can believe) On one of the ESXi hosts I would run my home network workloads, but have the option to turn on ESXi host 2, and for lab testing.

I looked at several options, both Linux and windows based, virtual and non-virtual, that would enable me to run both the NAS iSCSI workload, but also the Veeam proxy, PLEX and Minecraft service. The setup I found most appealing for testing the different RAID levels and was a non-virtual windows based Starwind Virtual SAN solution

The main reason for running the workload in a non-virtualized Windows installation, was the fact that this enabled me test different IO and cache policies on the physical volume used as an iSCSI target. On native windows I could use the LSI MegaRaid Storage Manager to create and destroy volumes without having to reboot the server.

At a later stage I might run ESXi on this host, reducing the footprint down to 2 physical R710 machines using Starwind 2 node cluster setup.

Features of the Starwind SAN solution that I found interesting

Main Product page and Free Product Page

There are several features in the Starwind software that I found extremely cool. Also the simple setup and configuration process of the solution is truly remarkable. It makes testing the different configurations fast and easy.

To name a few features that got my attention while testing the software, that other users could benefit of both in regard of lab testing and for production workloads.

  • Use of defined amount of RAM for cache for each defined iSCSI device.

This allows me to define the amount of RAM assigned for the NAS storage role, keeping RAM available to other workloads on the server. This also allows me to define different devices and iSCSI target with different amount of RAM depending on workload types. Keep in mind that if you assign many GB’s of RAM for cache in a production setup, make sure you have a UPS to be able to commit all cached writes to disk!

  • Create a RAM based disk device.

Using this super-fast iSCSI target is great for testing and deploying temporary workloads in the lab. I plan to experiment with this feature more, but keep in mind this in in memory, so data is not written to any persistence storage! Non-persistence VDI disks (linked-clones) come in mind or classroom VM’S could use this feature to give great end-user experience.

  • Log-Structured File system while thin-provisioning the storage device.
    This feature turns otherwise “all writes are random” situation while running mixed virtual workloads, into sequential write on the underling storage. A whitepaper (https://www.starwindsoftware.com/whitepapers/eliminating-the-io-blender-by-jon-toigo.pdf) by Jon Toigo explains this in great detail, but this features boosts the benefits of thin-provisioning to a whole new level!
  • Publish a physical disk directly as an iSCSI target.
    This feature caught my eye, and I still have to investigate the pros and cons in this regard.

 The Network design

To give out a clear picture of my setup, I made the following diagrams.

Layer 1 Diagram

Picture 1: Cabling layout

  • 2 x 1Gbit network interfaces are connected from each ESXi host to the iSCSI NAS host.
  • 2 x 1Gbit network interfaces are used for vMotion and replication.

Layer 2-3 Diagram

Picture 2: Layer 2-3 diagram

The diagram shows the networking layout of the 2 iSCSI networks. Different subnets are used for each physical adapter assigned to iSCSI to provide active-active paths to the iSCSI target machine.
Path selection Policy is set to “Round Robin” for link load balancing

vMotion network between the hosts are bound to 2 physical network adapters, on a single subnet.

Storage design

For testing purposes, I decided to install Windows 2012 directly on a 2 disk mirror, and have the 4 extra drive slots to test different RAID levels and drive types. This allowed me to run the LSI MegaRaid Manager software and set different settings on the volumes and save me the reboot time when changing raid levels or drive types.

I had 4 x 2TB, 7.4K SATA drivers and 4 x 600GB, 15K SAS drives to test.

 Different Raid Levels and drive types.

First I tested out different RAID levels and on both types of drives, and ran FIO tests locally on the volume created.

Different Raid Levels

It caught my eye that when using the SATA drives, performance gain from Raid 10 to Raid 0 was minimal, while the SAS drives had huge performance gain while running Raid 0 vs Raid 10. Later I plan to do a 6 x 2TB SATA drive Raid 10, and that’s most likely the configuration I’ll end up using for my lab setup.

For the remaining of the performance tests, I ran the Raid 10 setup on the 4 x 15K drives, and the main goal was to find out if the different deployment options on the Starwind SAN software made any measurable difference, and also to see how it performed against the native Windows 2012R2 iSCSI target.

CrystalDiskMark tests

First test was done by using CrystalDiskMark measuring MB/Sec

CrystalDiskMark MB/secCrystalDiskMark IOPS

The tests show that in any configuration, the Starwind SAN software outperforms the Windows 2012R2 Built in iSCSI target solution by far. The only tests where the Windows iSCSI target was close was while testing sequential reads or writes, and I believe the limiting factor was the single threaded process and use of one network connection between the 2 physical machines.

All the random reads and writes tests showed huge benefits while using the Starwind solution. The CrystalDiskMark is a simple tool to test disk performance and it does not allow you to change from the fixed 4K block size, or go beyond the queue depth of 32.
The H700 controller on the iSCSI target machine has queue depth of 975 and to utilize the 2x 1GB network connection I moved from the CrystalDiskMark to more customable test tool, FIO.

To create a baseline and to get the maximum performance without the limitation of my 2 x 1GB network connections between hosts, I ran all tests both locally on the iSCSI target machine and on a remove VM. To test the performance running locally, I mapped a set of iSCSI targets as drives on the windows iSCSI target machine and an identical set of targets to my ESXi host.

The FIO test setup.

Each Starwind iSCSI target configured with 10GB Memory Cache

VM runs on a ESXi 6.0 Hosts, connected by 2 x 1GB Network cards, each configured on separate Subnets, – Round Robin PSP selected

FIO WindowsIO Engine settings:
Random Read/Write:    33/66
Block Size:                          64K
Queue Depth:                  975
4 x 15GB Jobs, 4 files each

FIO MB/sec

FIO IOPS

Direct = FIO Run directly on iSCSI target machine disk volume
Flat = Starwind iSCSI Target with Flat provisioned Image file
LSFS = Starwind iSCSI Target with Thin provisioned disk using LSFS
LSFS Dedup = Starwind iSCSI Target with Thin provisioned disk using LSFS and Deduplication enabled
Physical Disk = Starwind iSCSI Target from physical disk

The direct testing showed how much performance I could get from direct disk access. As I ran those tests, I got a clear picture of the different deployment options in the Starwind SAN software and my findings showed that the Thin Provisioned disk utilizing the LSFS was the fastest option.

While testing deduplication, performance dropped to some degree compared to the LSFS option in regard of IOPS. I also noticed some (5-7%) increased CPU load on the iSCSI target machine while I was running the tests. Also keep in mind that each 1 x TB of deduplicated storage requires 3.5GB of RAM. In my setup this was not an issue but if you have limited amount of RAM you should take note of this fact.

Future plans and few points.

Later, when I have finished the performance tests, I plan to create a target device, for the system drives for my home network VM’s, using deduplication, and save space there, but I’ll leave that option disabled for the PLEX media library and also the photo library as those media files are unlikely to be good candidates for deduplication.

When rebooting the iSCSI target machine, I noticed that the FLAT file and Physical DISK targets were active soon after boot time, but the thin provisioned LSFS and LSFS Dedup targets took some time to become active. After some investigation I saw the LSFS files were all read though, most likely due to file-checking and verification. My test targets were all 100GB in size and it took some time (5-10 minutes) to become active. When evaluating the benefits of FLAT or Physical targets, I guess if you have large targets (3TB as in my case for PLEX media library) you would prefer to use the FLAT file option there to have the targets online soon after reboot.

Conclusion

For a 2-3 hosts setup like mine, or even 1 host installation, it is clearly beneficial to use the Starwind SAN iSCSI software rather than direct disk access or native Windows iSCSI target software.

My findings on different deployment options will hopefully help you decide on what to go with both in your lab or production installations.

A colleague of mine pointed out that my home lab had more performance than many of his client’s production setups, and told me that if I was happy with the performance of the Starwind SAN software, he could recommend it to his clients for production!

PernixData FVP and Citrix Netscaler, a killer combo.

PernixData FVP and Citrix Netscaler, a killer combo.

Those 2 technologies are playing in different playgrounds, but they have a lot in common in regard of their purpose and ideology. They both use smart software technologies to save you money by moving workload from the classic components of your datacenter. Storage, network and compute all are saved from load and operational risk by the two products. Here is an overview of my thoughts on this matter.

First I wanted to write a brief overview of PernixData FVP, the idea, installation and settings options.

PernixData FVP is a software based storage acceleration platform, which can utilize both flash media and RAM to cache both read and write IO to the storage you have under VMware’s ESXi Hosts. FVP software is a 3 part system. Kernel module installed at each ESXi host, a management service and the vCenter plugin (available both for the legacy client and the web client). The software has a small footprint, and you can easily install the FVP management service on your vCenter Windows server, or if you prefer on a separate Windows machine. The installation is plain and simple, you need a database for the configuration and performance graphs, but otherwise the install is straight forward. You connect it to your vCenter and installation is pretty much done. Configuration is done though the vCenter client, through it you install the license, but you have to be on the same system as the FVP is installed to activate as the license is host based. It’s recommended that you install a valid SSL certificate for the FVP service, ether from your domain’s Certificate Store, or by using your public SSL vendor’s certificate. How to do this is explained here: https://pernixdatainc.force.com/articles/KB_Article/Creating-custom-SSL-certificate-for-FVP

When that is done you can connect as usual from your workstation or terminal service and create your FVP cluster.

What the software allows you to do then, is to create an accelerated cluster (FVP Cluster), which you assign ether RAM, Flash or both. If you have the standard license you can choose either one, but if you have the Enterprise license you can mix hosts with RAM assigned as cache medium and hosts with Flash assigned. You can mix those in the cluster, but only one type at each host per cluster. You can however create a new FVP cluster with the same hosts, one with RAM and other with Flash, and then move VM’s between FVP clusters to utilize ether one of the options. (The same VM can’t use the two types at the same time, but the host can service multiple FVP clusters. Please note that the FVP cluster in not the same as you normal ESXi host cluster.

When you have created the FVP cluster, assigned acceleration media to it, and moved some vm’s over to the hosts, you have the option to select ether the VM’s to accelerate on that FVP cluster, or to make things easy for you, you can select the datastore, and then all vm’s on that datastore gets the acceleration methods you select.

You can choice from “write though” and “write back”, or function wise, ether you accelerate read requests, or you accelerate both read and write requests. – A good read on the subject is found here: http://frankdenneman.nl/2013/07/19/write-back-and-write-through-policies-in-fvp/ .

When you have set things up, you can start to look at the performance data, and soon after you see the software accelerate you storage IO, you can then look at your storage system, and see how it gets a huge reduction in IO.

Give it a few hours to utilize the cache media, and on day 2 I can promise you that you want to license more hosts!

The idea and business case for this is of course to give you better performance, but not less important is the fact that you can save IOPS from the storage array and therefore save money on expensive SAN upgrades.

 

And where does Citrix Netscaler fit in all this? – Surely that is not in play in this respect of ESXi hosts and SAN storage. Netscaler gives you a lot of features in regard of networking and application functions, load balance, content switching, application firewall to name a few.

What I’m going to write about in context of this blog post, are the acceleration features.

There are few functions to mention in this regard and they are SSL Offloading, Integrated caching, Compression and TCP optimization.

All serve to offload your backend services from workloads, hence save IO on your datacenter network, compute and storage.

SSL offloading works by installing the SSL certificate you would normally install on your webserver, on to your Netscaler appliance (Netscaler comes as an appliance, or as a VM). The Appliance has dedicated SSL cards that take care of the otherwise CPU intensive process of encrypting and decrypting, and if you high SSL traffic load on your services, this offload function can save you a lot of CPU power on the backend, up to 30% of webservers CPU workload can be SSL related workload so there is a lot to save here. It also gives you a single entry point in managing your SSL certificates, where you see their expiry date and you don’t have to have your webservers with multiple IP addresses for each SSL service as your webservers are not service SSL content any more.

Integrated caching is my favorite function. It uses cache-control headers from your web service to determine if the objects requested from the client may be stored in cache, and if so, it uses RAM and flash (optional) to store the content for the next request. You can also set up your own caching rules if the webserver/application admin is unable to control the cache-control headers at the backend. Once the content is in the Netscaler cache, the client’s connecting get the content served from the cache store. When you have a high traffic website, this can save you enormous amounts of network, CPU and storage load in the backend. You can have the Netscaler cache the objects for a very short time, like for some ticketing system data, or for longer time for static content.

Compression can also be moved from the web service to the Netscaler appliance, so your webserver’s CPU can be offloaded from that workload. This feature is also saving you outbound network traffic as your clients receive more content compressed than your web service might be set to compress.

TCP optimization also save your resources. It work by having your clients connecting to the Netscaler appliance, and the Netscaler creates a new TCP session to your backend. Let’s say you have 10.000 concurrent client connections to your website. If Netscaler were not used, your webserver would be overwhelmed by the amount of TCP session as its CPU would be busy just handling all the session and actual data traffic from the web service would be suffocated in TCP control packets. This can easily bring a good performance webserver to its knees even though the actual data served is minimal. What the Netscaler does in this regard, is to multiplex the data traffic into few TCP sessions to the backed services. With this, the backend service can use the server’s CPU to serve actual content instead of spending its time on session handling.

Those 4 functions of the Netscaler appliance all save you load on your backend. One of my customers at work moved their website from a previous 13 physical server’s web farm that was load balanced by DNS round robin method, to a 3 server web farm load balanced and accelerated with our Netscaler appliance. To test the system after installation, I even had at one point 1 backend server active, and the website performance was still ok for a normal day operation.

So with those 2 technologies, Citrix Netscaler on the frontend, and PernixData FVP on the ESXi level, you can save huge amounts of money on both Capex and Opex throughout your datacenter.

I hope this was a useful read and interesting for you.
Cheers.

Regarding home labs

Regarding home labs.

I wanted to share my experience from two years ago, when i decided make use of some old servers from work, to make a home lab.
The short story is and thoughts. Don’t do it!
Why?

Then the long story…

I decided to bring from work 2 old dell PowerEdge 6950 that had been decommissioned and not been used for quite some time. Those 6950 servers are huge rack mounted servers and really heavy 4U units. Each server had 32 GB ram, and 4x dual core AMD CPU’s. So there was plenty of cores and ram to play around with. Somehow I managed to put the units into the trunk in my Volvo S60 and get them home to my basement. I live in a small apartment building where each apartment in the building has a small private storage room, and also there are shared room for washing machines and a dry room. To prepare I set up a small table to put the servers on, installed some power sockets from the light switch socket. I made two 4 inch holes at the top of the wall out to the hallway in front of the drying room. I then created a funnel from the back of the server with 2 outlets. From the outlets I installed two 4 inch dryer hoses that went up to the 2 holes in the wall. I also had an old UPS from work installed on top of the 2 servers and had the heat from it also in the funnel.

Before I had this project started, I had one old home-pc with some 2TB SATA drives installed, and that one I decided to use as an iSCSI storage box for the 2 ESXi hosts. To create the iSCSI network I installed a 2 port Intel nic in all 3 servers and connected one port from the two hosts directly to the nic on the storage server. On the ESXi hosts the other nic were connected to a small home 5 port gig switch I had at hand. From the switch I installed a cat5e cable up to my study room in my apartment on second floor where I have my workstation and an additional ESXi host where I installed my monowall router vm, Symantec NetBackup vm, an AD server vm and vCenter vm.

Everything was awesome at this point. I installed windows 2012 on the old pc in the basement and set up storage spaces on those 2TB disks to provide iSCSI to the 2 ESXi hosts. I created some vm’s to serve my home domain, a secondary AD server vm, a web server, exchange 2010 vm, a pair of windows 2012 fileservers with DFS, Observium monitoring server on a Linux vm for performance and traffic logging, an xymon Linux machine for monitoring and alerting vm, an Citrix Netscaler VPX vm and so on.. The 2 hosts could easily handle the load and I played around with nested ESXi also.

When everything was ready, I decided to write down the status of the electric meter for my apartment and report it back to the power company. My thoughts were to get an accurate report of the usage before and after 1 month of usage with this setup running.

After few days I got a few questions from people in the building regarding what was making all the noise down in the basement, and when I told them what I was doing, they didn’t mind the noise so much, but they were happy that the heat from the servers blew directly into the drying room, and cloths were drying twice as fast than before… I went down to investigate, and surely there was some heat blown from the servers out to the hallway, but nothing was overheating. But then again I got worried that those old servers were generating to much heat than they should be doing, and it might hurt to see the electricity bill for next month. I decided to let the system run until the beginning of next month though. I had pretty good monitoring on the setup, and set up some alerts, and just in case I installed a smoke detector in the storage room. I continued to play around with some vm’s, and I was quite happy with the setup in terms of performance and as this was basically a free installation for me, I thought that even if I had to pay a little extra for electricity, this could work out ok for me.

After a month I reported back to the power company the status of the electricity meter. I saw right away that the bill doubled from previous month before installation! Much more than I had imagined or was willing to pay to have a home lab running.
I calculated based on the usage, that after 6 months I would have spent more money on electricity than the cost of a new setup made up of a new motherboard, 32gb of ddr3 ram and a new intel i7 3770K 4 core CPU. I quickly decided to cancel this lab setup with those 2 old hosts and upgrade the old windows machine I had used as an iSCSI box instead. I went out and bought the new CPU, ram and memory, I also refurbished from work an old dell perc5 raid controller, and installed ESXi on the new box. (I had to modify the perc5 though to run it on a normal desktop Intel chipset pc) I put a large CPU fan in the box, overclocked the 3.5 GHz CPU to 4.5 GHz, and the setup has been running my home lab since. I cut down the numbers of vm’s though as now I only have one 32GB host versus 2 x 32GB hosts, but the CPU performance on this single CPU, 4 core with hyper threading, is so much faster than before.
The power usage of the new host ended up at 1/7 of the old lab setup, and added a reasonable amount to the household’s electricity bill.

Later I bought a hardware Mikrotik router, moved the vCenter vm down to the lab ESXi host, installed a pair of 2 GB disk on my wife’s pc, and I run the backups to those drives. I also got rid of the ESXi host that were running in my study room. I sold off the motherboard, CPU and memory from that host. So after almost two years I think I’m pretty well off in regard of total cost of ownership on my home lab.

Hopefully this has been an interesting blog post for you, and a warning for those who plan to bring old servers to life for a home lab project.