Last month I did an initial post on this topic, and here is the follow up.
Our environment consists of dual 3PAR P10000 SANs and all storage at the controller level is thin provisioned. All 3PAR volumes are configured for Zero Detect. We run approximately 40 physical servers and 450 VMs on ESXi 5.1 U1 hosts. All storage at the VM layer is thin provisioned also, thus we have Thin-On-Thin provisioning. We have scheduled CompactCPG tasks that run weekly.
Most of the fleet is Windows 2008 R2 SP1, with some legacy 2003 servers in the mix.
There can be a disconnect between what the servers file system (NTFS) sees as used/free space, and what the hypervisor or storage controller see. For example you may thin provision a volume on the SAN and present this to a server, the server uses space from this volume and then deletes data from it, NTFS sees this space change but the storage controller and/or the hypervisor do not, therefore the free space is unable to be reclaimed without some form of intervention. Being the fussy Systems Engineer I am I wanted this space reclaimed on the back end.
For some time I was manually reclaiming this space by running either SDELETE or FSUTIL on Windows, or VMKFSTOOLS on the hypervisor. The process works, but it is manual and time consuming. I had partially automated with scripts but it still consumed too much of my time to administer. I was looking for an automated solution and I found it with Raxco Software PerfectStorage.
UnitingCare was very fortunate to obtain some NFR licences from Raxco Software for their PerfectStorage solution so we could carry out testing. PerfectStorage has a central Enterprise Console and from that you simply deploy an agent to your Windows OS. Any system you push the agent to will show in the Enterprise Console, so easy to keep track of systems. You then configure some schedules on when you want the agents to check in to the console, when you want the agent to scan/analyse the system, and when you want the space reclamation to take place.
PerfectStorage utilises 2 technologies to reclaim the space: SCSI UNMAP and Smart Zero Fill™. SCSI UNMAP is from the SCSI T10 command set and reclaims space from blocks that have been deleted by sending SCSI UNMAP commands. Smart Zero Fill™ writes zeros to NTFS free clusters. Both options signal to either the storage controller or the hypervisor that space can be reclaimed. You have the option of what method/s to use.
In our environment I am using Smart Zero Fill™ for physical servers and SCSI UNMAP for the VMs. Within the Enterprise Console you can easily segregate virtual or physical servers by groups, and you can configure “Configuration Profiles” for each.
PerfectStorage has some pretty cool features but there are 2 that I specifically want to mention.
1. StealthPatrol: once you have deployed the agent to your servers you want to set a schedule on when reclamation will occur. This would normally be outside of business hours or off-peak periods. StealthPatrol allows the reclamation to take place at any time the server is idle (can set exclusion periods) and allows you to set CPU/Memory and Disk I/O thresholds which kick in and pause the reclaim process on the server should they elevate above a set level. You can also specify application/s, that if running will pause the reclamation process.
2. Virtual Environment: PerfectStorage has a feature for virtual environments called Virtual Awareness. As we use VMware, I will discuss its benefits related to VMware. Virtual Awareness gives PerfectStorage the ability to monitor individual ESXi Host or vCenter. This allows PerfectStorage to see the utilisation of the underlying host that the VM resides on and can dynamically adjust reclaim activity based on host resource usage.
My initial testing was carried out on 5 physical servers, and 59 development and integration environment VMs. To enable “on-the-fly” space reclamation for the VMs, I needed to upgrade the VMs to hardware version 9. I achieved this by VMware Update Manager.
Prior to starting I recorded details of the backend storage provisioning (consumed per volume), VMware Data store utilisation (used/free) and individual VM disk consumption.
I deployed the agents without issue to the servers, and then went about configuring the schedules for check-in, analysing and reclaiming. I then let the reclaim process kick off as scheduled.
On the first initial run, PerfectStorage reclaimed 3.1TB, with zero downtime, zero impact and it was all automated. It even ran at some points during production hours with zero impact due to StealthPatrol.
The 3.1TB of space reclaimed on the 3PAR SAN equates to a 7% reduction of overall consumed space, and we currently only have PerfectStorage installed on 14% of the fleet.
The next step going forward will be to upgrade the balance of VMs to hardware version 9, and deploy the PerfectStorage agents out to the balance of VMs and physicals. I will write another blog post once that has been achieved.
Overall I think it’s a great product. Check it out on the Raxco website here: http://www.raxco.com/business/products/perfectstorage
Musings from a past life 
http://blog.raxco.com/2014/01/17/3-1tb-thin-on-thin-space-reclaimed-with-zero-downtime-and-zero-impact/
LikeLike