Today I had an interesting question posed to me by a customer.
“What’s a way we can verify file integrity compliance of our VMware View gold image, at regular intervals, WITHOUT powering it on? Because powering it on would force us to recompose all VMs once it’s powered down again”
In order to verity file integrity of a VM, there will need to be a file integrity check at some point in time. [PCI DSS 10.5.5 & 11.5] This is not a point in doubt. The question arose regarding regular scheduled checks on file integrity, at a pre-set interval. That’s the key. They need to ensure it weekly or monthly.
10.5.5 Use file integrity monitoring or change detection software on logs to ensure that existing log data cannot be changed without generating alerts (although new data being added should not cause an alert)
11.5 Deploy a change-detection mechanism (for example, file-integrity monitoring tools) to alert personnel to unauthorized modification of critical system files, configuration files, or content files; and configure the software to perform critical file comparisons at least weekly
(Note, Netapp does have an API inside call fpolicy, which 3rd party tools and integrate into it, and have instant notification of file changes, and other auditing. In many situations, those may be a better or more comprehensive fit. In this case, we needed a spot solution for a specific type of scenario not already covered by their existing toolsets)
The image would be scanned with a FIM tool such as Tripwire ChangeIQ. The VM would then be shutdown, and a (sha or sha256) checksum of all the VM files would be taken from the (NFS volume) the Datastore. (Possibly by a checked/safe scan box w/ a read-only mount of the NFS volumes)
Being that [CUSTOMER] utilizes Netapp storage, with NFS as the protocol for VMware data stores, there is an easy solution to this problem.
NFS is a file-based protocol. The storage has visibility into the blocks and files. The Netapp can also take snapshots of the volumes which hold the files. These snapshots are read-only views to a point in time mapping of the inodes on the system when the snapshot was created. There is absolutely no way to modify the contents of a snapshot.
Netapp Snapshots of the VMware datastore would be scheduled at regular intervals. Utilizing the view of the filesystem at the point of a snapshot, a checksum can be performed on the files within that snapshot guaranteeing that the VMDKs and config files are not able to be modified by a compromised scanning system or any system which may touch the storage.
The “known good” snapshot checksum would be compared to the file version within the recent snapshot.
This method should be arbitrary to script.
Example. (In this I used the same ESXi server to checksum, but the NFS volume can be presented up to a server to checksum, with the entire NFS mount being readonly as an additional layer of security.)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 |
## Listing which datastore I have available on this vmware server. /vmfs/volumes # esxcfg-nas -l temp_p_removal_cDot1 is /temp_p_removal_cDot1 from 10.40.1.114 mounted available temp_p_removal_cDot2_compressed is /temp_p_removal_cDot2 from 10.40.1.232 mounted available ## Checking the powerstate of a known good VM "bv-esxi01" /vmfs/volumes/ # vim-cmd vmsvc/getallvms | grep bv-esx01 17 bv-esx01 [temp_p_removal_cDot1] bv-esx01/bv-esx01.vmx vmkernel5Guest vmx-10 root password = password /vmfs/volumes/ # vim-cmd vmsvc/power.getstate 17 Retrieved runtime info Powered off # verified powered off. ## Going to the VMs directory /vmfs/volumes # cd temp_p_removal_cDot1 /vmfs/volumes/52315eae-867f89ff # cd bv-esx01 ## Picking a file to checksum /vmfs/volumes/52315eae-867f89ff/bv-esx01 # ls bv-esx01-flat.vmdk bv-esx01.vmdk bv-esx01.vmx bv-esx01_1-flat.vmdk vmware-1.log vmware.log bv-esx01.nvram bv-esx01.vmsd bv-esx01.vmxf bv-esx01_1.vmdk vmware-2.log ## md5sum the file (sha1sum or sha256 recommended, didn't have it on this box) /vmfs/volumes/52315eae-867f89ff/bv-esx01 # md5sum bv-esx01-flat.vmdk 96a8c577010121ed5d5f402c9541154c bv-esx01-flat.vmdk ## Going to the virtual snapshot folder (Netapp volume) for this Datastore /vmfs/volumes/52315eae-867f89ff/bv-esx01 # cd ../../.snapshot/ ## Here you can see some of the snapshots. /vmfs/volumes/52315eae-867f89ff/.snapshot # ls VeeamSourceSnapshot_Backup_Job_5.2014-11-08_1526 hourly.2015-05-28_2205 hourly.2015-05-29_0205 daily.2015-05-28_0010 hourly.2015-05-28_2305 weekly.2015-05-17_0015 daily.2015-05-29_0010 hourly.2015-05-29_0005 weekly.2015-05-24_0015 hourly.2015-05-28_2105 hourly.2015-05-29_0105 ## Picking a randoms snapshot for this. /vmfs/volumes/52315eae-867f89ff/.snapshot # cd hourly.2015-05-29_0205 ## see the contents of the snapshot, basically it's all my VMs. /vmfs/volumes/52315eae-867f89ff/.snapshot/hourly.2015-05-29_0205 # ls bmcc-LoadMaster-demo bv-esx01 gss_Workstation mlo-esxi-02 pr-ha-splunk jk47-smtp pr-ha-labdhcp # Repeat /vmfs/volumes/52315eae-867f89ff/.snapshot/hourly.2015-05-29_0205 # cd bv-esx01 /vmfs/volumes/52315eae-867f89ff/.snapshot/hourly.2015-05-29_0205/bv-esx01 # ls bv-esx01-flat.vmdk bv-esx01.vmdk bv-esx01.vmx bv-esx01_1-flat.vmdk vmware-1.log vmware.log bv-esx01.nvram bv-esx01.vmsd bv-esx01.vmxf bv-esx01_1.vmdk vmware-2.log /vmfs/volumes/52315eae-867f89ff/.snapshot/hourly.2015-05-29_0205/bv-esx01 # md5sum bv-esx01-flat.vmdk 96a8c577010121ed5d5f402c9541154c bv-esx01-flat.vmdk #~~ Notice the checksum withing my snapshot is the same as my main VM, which had it's FIM ran and then shutdown/checksummed. ## Verify that my snapshot is indeed readonly. /vmfs/volumes/52315eae-867f89ff/.snapshot/hourly.2015-05-29_0205/bv-esx01 # touch bv-esx01-flat.vmdk touch: bv-esx01-flat.vmdk: Read-only file system /vmfs/volumes/52315eae-867f89ff/.snapshot/hourly.2015-05-29_0205/bv-esx01 # cat fooo > bv-esx01-flat.vmdk -sh: can't create bv-esx01-flat.vmdk: Permission denied /vmfs/volumes/52315eae-867f89ff/.snapshot/hourly.2015-05-29_0205/bv-esx01 # ls -l bv-esx01-flat.vmdk -rw------- 1 root root 8589934592 Apr 9 2015 bv-esx01-flat.vmdk /vmfs/volumes/52315eae-867f89ff/.snapshot/hourly.2015-05-29_0205/bv-esx01 # dd if=/dev/zero of=bv-esx01-flat.vmdk bs=1024 count=10 dd: can't open 'bv-esx01-flat.vmdk': Permission denied /vmfs/volumes/52315eae-867f89ff/.snapshot/hourly.2015-05-29_0205/bv-esx01 # ls -l bv-esx01-flat.vmdk -rw------- 1 root root 8589934592 Apr 9 2015 bv-esx01-flat.vmdk ## Take it up a notch. Let's scan this file in all snapshots. /vmfs/volumes/52315eae-867f89ff/.snapshot/hourly.2015-05-29_0205/bv-esx01 # cd ../.. /vmfs/volumes/52315eae-867f89ff/.snapshot # for i in * > do > echo "Scanning $i/bv-esx01/bv-esx01-flat.vmdk" > md5sum $i/bv-esx01/bv-esx01-flat.vmdk > done Scanning VeeamSourceSnapshot_Backup_Job_5.2014-11-08_1526/bv-esx01/bv-esx01-flat.vmdk 96a8c577010121ed5d5f402c9541154c VeeamSourceSnapshot_Backup_Job_5.2014-11-08_1526/bv-esx01/bv-esx01-flat.vmdk Scanning daily.2015-05-28_0010/bv-esx01/bv-esx01-flat.vmdk 96a8c577010121ed5d5f402c9541154c daily.2015-05-28_0010/bv-esx01/bv-esx01-flat.vmdk Scanning hourly.2015-05-28_2205/bv-esx01/bv-esx01-flat.vmdk 96a8c577010121ed5d5f402c9541154c hourly.2015-05-28_2205/bv-esx01/bv-esx01-flat.vmdk Scanning hourly.2015-05-28_2305/bv-esx01/bv-esx01-flat.vmdk .. truncated .. |
We can see that as long as the VM is considered good, being shutdown immediately after it’s initial scan, all future checksum matches of the readonly snapshots on the Netapp can be used to illustrate that the source VM is still intact and good.
Additional Reading:
- Netapp TR-4401: PCI DSS 3.0 and Clustered Data ONTAP 8.3 (Not yet public on the TR site, Ask your Netapp SE or VAR for a copy from the Field Portal)
.@BSidesLV Can any #PCI-DSS experts sanity check something I told a client Re: FIM & 11.5? Concerns? http://t.co/VR38sGFD6D