During a recent upgrade of VMWare ESXi from 4.1 to 5.5, I encountered a problem: after the upgrade, systems stored on a shared iSCSI storage platform did not work. In fact, a number of them were damaged. Files could be read and downloaded from the datastore with no issues, but writes to the datastore failed.

A number of symptoms appeared, most tellingly in the VMWare client console about connectivity to the datastore being lost and then restored. However, the vmkernel logs on the hosts themselves showed more interesting failures, specifically reservations and the failure of SCSI ‘WRITE_SAME’ 0x93 errors. This led me to a post about the Linux SCST target software, stating that this doesn’t support this command and to disable VAAI/acceleration features in VMWare.

This did solve the problem and it appears that VMWare has been a bit dumb here. The options are marked as ‘requires compliant hardware’ inside VMWare but 1) they are turned on by default, and 2) are applied to targets with a HW acceleration value of ‘Unknown’, rather than being restricted to validated hardware. Poor design decision in my opinion.

To turn off these features, you need to edit the following on each host:

ESXi Configuration > Advanced Settings

/VMFS3/HardwareAcceleratedLocking > Set to 0

/DataMover/HardwareAcceleratedMove > Set to 0

/DataMover/HardwareAcceleratedInit > Set to 0

 

(Originally posted 2015-06-24)

So, I ran into something interesting and confusing today.

I found that my PDC was stuck on ‘Local CMOS Clock’ as it’s source, despite it being configured to use an external NTP server (in this case, my local Cisco router is configured as an NTP server). Finding this odd, I verified the configuration and then executed a resync command:
w32tm /resync
and received this output:
Sending resync command to local computer
The computer did not resync because no time data was available.
Curious, as the NTP server is running and other devices are syncing correctly. Logging into the Cisco router and executing ‘show ntp associations’ resulted in this:
 address         ref clock       st   when   poll reach  delay  offset   disp
  192.168.1.11    .INIT.          16      –  32768     0  0.000   0.000 15937.
  192.168.1.12    .INIT.          16      –  32768     0  0.000   0.000 15937.
 ~2610:20:6F15:15::27
                  .STEP.          16      –   1024     0  0.000   0.000 15937.
*~128.138.140.44  .NIST.           1      2     64   377 44.098   0.552  3.122
Well, that’s odd. The system is syncing correctly, but why are the domain controllers showing up as stratum 16?
‘show ntp associations detail’
192.168.1.11 dynamic, insane, invalid, unsynced, stratum 16
ref ID .INIT., time 00000000.00000000 (16:00:00.000 PST Wed Dec 31 1899)
our mode passive, peer mode unspec, our poll intvl 32768, peer poll intvl 131072
root delay 0.00 msec, root disp 0.00, reach 0, sync dist 15956.69
delay 0.00 msec, offset 0.0000 msec, dispersion 15937.50
precision 2**24, version 3
Well, that’s not good, it’s trying to register itself as an NTP peer with the Cisco router. Googling some bits finally led me to a bug report between certain versions of Cisco IOS and W32Time.
The fix?
Change the w32time configuration:
w32tm /config /manualpeerlist:IPADDRESS,0x8 /syncfromflags:manual /reliable:yes /update
net stop w32time && net start w32time
w32tm /resync
That ‘0x8’ allows it to work.

(Originally posted 2012-06-16)

Based on ASDM 6.4(7), and ASA OS 8.2(5).

If you want to use your AD accounts for Administrative logins to the ASA, as well as Execute access, this will help you configure it. You’ll need to configure a group in AD and add the relevant accounts that you wish to grant access to beforehand.

ASA AAA for Administration

Expand: ASDM > Configuration > Device Management > Users/AAA

Select AAA Server Groups
Select ADD under ‘AAA Server Groups‘ pane
AAA Server Group: LDAP_DOMAIN
Protocol: LDAP
Reactivation: Depletion
Dead Time: 10 minutes
Max Failed Attempts: 3
Ok

Highlight ‘LDAP_DOMAIN‘ in the list

Select ADD under ‘Servers in the Selected Group‘ pane
Interface Name: Select Interface which the LDAP server is accessible on
Server Name or IP Address: IP of LDAP server
Timeout: 10 seconds
Enable LDAP over SSL: uncheck  (if you have a proper PKI certificate configured, you can enable this)
Server Port: 389   (port 636 by default for SSL, if you have enabled it)
Server Type: Microsoft
Base DN: Base bind point of the directory search in CN=XXX,OU=XXX,DC=XXX,DC=XXX format
Scope: Sub
Naming Attributes: sAMAccountName
Login DN: CN=LDAP Reader,OU=XXXX,DC=XXXX,DC=XXXX  (just a domain user, not a domain admin)
Login Password: password of LDAP Reader account
LDAP Attribute Map: –None–  (note, we’ll come back to this setting)
SASL MD5 Authentication: uncheck
SASL Kerberos Authentication: uncheck
Group Base DN: leave blank
Group Search Timeout: 10
OK

Expand ‘LDAP Attribute Map‘ at the bottom

Select ADD
Name: LDAP_MemberOf_ServiceType
Under Mapping of Attribute Name tab:
LDAP Attribute Name: memberOf
Cisco Attribute Name: IETF-Radius-Service-Type
Select ‘Add >>
Select ‘Mapping of Attribute Value’ tab
LDAP Attribute Value: group name in CN format to valid users
Cisco Attribute Value: 6
Select ‘Add >>
OK

Select Server added under ‘Servers in the Selected Group

Select Edit
Change LDAP Attribute Map to ‘LDAP_MemberOf_ServiceType
OK

Highlight the Server that was just added in the list

Select Test
Select Authorization
Enter user that should be authorized successfully
Ok
Select Test
Select Authorization
Enter user that should NOT be authorized successfully
Ok

You can repeat the above steps, except for the LDAP Attribute Map, which can be reused, for multiple servers for redundancy. I highly recommend this.

Select ‘User Accounts‘ in navigation pane

Select ‘Add
Username: some_admin_user
Password: some_admin_password
Confirm Password: some_admin_password
Select ‘Full Access
Change Privilege Level to ‘15
OK

**This is a LOCAL database user that will be used for emergency access to the ASA if the LDAP servers are unavailable. This account cannot log in if the LDAP servers are available.**

Select AAA Access in navigation pane

Select ‘Authentication‘ tab
Under ‘Require authentication to allow use of privileged mode commands
Check ‘Enable‘, Server Group: LDAP_DOMAIN, check ‘Use LOCAL when server group fails
Under ‘Require authentication for the following types of connections
Check ‘SSH‘, Server Group: LDAP_DOMAIN, check ‘Use LOCAL when server group fails
Leave HTTP/ASDM, Serial and Telnet unchecked, this way if you’ve screwed up somewhere, you haven’t locked yourself out of the ASA.
Select ‘Authorization‘ tab
Under ‘Enable authorization for ASA command access
Uncheck ‘Enable
Under ‘Perform authorization for exec shell access
Check ‘Enable‘, ‘Remote server
OK

Apply configuration

Test connecting to the ASA via SSH via domain credentials; no need to specify domain name with the username. Verify that executing ‘enable’ and specifying your AD password works.

If successful, save configuration to flash. Otherwise, review configuration via ASDM and change/test until it does.

Once satisfied that access is correctly being handled via SSH, enable LDAP AAA for HTTP/ASDM as well.

Select AAA Access in navigation pane
Select ‘Authentication‘ tab
Under ‘Require authentication for the following types of connections
Check ‘HTTP/ASDM‘, Server Group: LDAP_DOMAIN, check ‘Use LOCAL when server group fails
OK

Save configuration to flash/startup-configuration.

(Originally published 2010-08-19)

Configuring ASA 8.2 for Remote VPN Access, with LDAP Authentication and Authorization

The following components need to be configured:
– AAA Server
– LDAP Attribute Map
– Address Pools
– Group Policy
– Connection Profile/Tunnel Group

This guide assumes you are using ASDM 6.2 for configuration of the ASA.

This guide was written and tested against a Cisco ASA 5520, Firmware version 8.2(2), ASDM version 6.2(5)

LDAP Attribute Map Configuration

– Create a normal Domain User account for the ASA to use to bind to the Directory to query. This user needs no special access other than to be able to query the directory. It certainly does not need to be a domain administrator account!
– Create an AD Group to be used for access to the VPN. Name it something sensible.

– Get the DN’s for the user and group created, you’ll need them for the next steps. You can use dsquery or the Object tab in ADUC to get them.

– On the ASA, go to Configuration > Remote Access VPN > AAA/Local Users > LDAP Attribute Map
– Create a new map, call it something like ‘LDAP_memberOf
– For the Mapping of Attribute Name:
LDAP Attribute Name: ‘memberOf
Cisco Attribute Name: ‘Group-Policy
*NOTE: Below ASA 8.2, this is IETF-Radius-Class, it has been renamed for 8.2+
– For the Mapping of Attribute Value:
LDAP Attribute Value: full DN of the Remote Access Group from AD e.g. ‘CN=RemoteGroup,OU=Groups,DC=domain,DC=com
Cisco Attribute Value: The name of the ASA VPN Group Policy you are going to use (we haven’t created it yet) e.g. ‘RemoteAccess_Grp
-Save

AAA Server Configuration

– Go to Configuration > Remote Access VPN > AAA/Local Users > AAA Server Groups
– Add a new AAA Server Group:
Server Group: Descriptive Name i.e. ‘LDAP_DOMAIN
Protocol: LDAP
Reactivation Mode: Depletion
Dead Time: 10 minutes
Max Failed Attempts: 3
– OK
– Now, with the LDAP_DOMAIN Server Group selected in the top frame, click on Add for the ‘Servers in the Selected Group’ frame
Interface Name: INSIDE or whatever interface you have your Domain Controller behind
Server Name or IP Address: IP Address of a Domain Controller
Timeout: 10 seconds
Server Port: 389 (unless you enable the LDAP over SSL option)
Server Type: Microsoft
Base DN: ‘DC=domain,DC=com‘ or whatever your base DN is
Scope: ‘All levels beneath the Base DN
*Note: One level is quicker, but you need to set the base DN to where your AD Access Group is located
Naming Attribute(s): ‘sAMAccountName
Login DN: full DN of the normal AD User created above
Login Password: password of the AD User account
LDAP Attribute Map: ‘LDAP_memberOf‘ or whatever you called the LDAP Attribute Map previously created
SASL options unchecked
Group Base DN: leave this blank
– OK

Address Pools

– You can either use a DHCP server, or assign a static pool to the ASA to hand out to VPN clients.
– For DHCP: Configuration > Remote Access VPN > Network (Client) Access > Address Assignment > Assignment Policy:
– Uncheck ‘Use Authentication Server
– Check ‘Use DHCP
– Uncheck ‘Use Internal Address Pools
– For Static Pool (which I recommend): Configuration > Remote Access VPN > Network (Client) Access > Address Assignment > Assignment Policy:
– Uncheck ‘Use Authentication Server
– Uncheck ‘Use DHCP
– Check ‘Use Internal Address Pools
– Uncheck ‘Allow the reuse of an IP address XX minutes after it is released
Configuration > Remote Access VPN > Network (Client) Access > Address Assignment > Address Pools
– Add New Pool
Name: Descriptive Name e.g. ‘RemoteVPN_Pool’
Starting Address: IP
Ending Address: IP (note these are inclusive!)
Subnet Mask: Mask
– Ok

*Note: The rest of this guide assumes a Static Address pool was created.

Address Pool created

Group Policies (Note, the terminology sucks. This is a policy for a group on the ASA, not an AD GPO)

– Go to Configuration > Remote Access VPN > Network (Client) Access > Group Policies
– Create a new policy to deny access:
– Add Internal Group Policy
– General:
Name: NoAccess
Address Pools: uncheck ‘Inherit‘ and leave blank
Expand ‘More Options’:
Simultaneous Logins: uncheck ‘Inherit‘ and set to 0
– Ok

This policy will be used to deny unauthorized & unauthenticated users from connecting.

– Create a new policy to for authenticated users:
– Add Internal Group Policy
– General:
Name: ‘RemoteAccess_Grp
Address Pools: uncheck ‘Inherit‘ and select the Address Pool previously created
Simultaneous Logins: Uncheck ‘Inherit’ and set the value to the number of Remote users you have licenses for or the number of valid IP addresses in your VPN pool
– Servers:
DNS Servers: I generally explicitly set them. Delimit with comma or space
WINS Servers: If in use, I generally explicitly set them. Delimit with comma or space
Expand More Options:
Default Domain: uncheck ‘Inherit‘ and set explicitly
– Advanced:
Split Tunneling: If you want to enable split tunneling, set as below. The default is tunnel all.
DNS Names: ‘Inherit
Policy: uncheck ‘Inherit‘ and select ‘Tunnel Network List Below
Network List: uncheck ‘Inherit‘, click ‘Manage
Create a new ACL with permit ACEs that include the hosts/netblocks you WANT tunneled.
Other options: Leave rest at default of ‘Inherit
– Ok

Group Policies created

Connection Profiles / Tunnel Groups:

Go to: Configuration > Remote Access VPN > Network (Client) Access > IPsec Connection Profiles
Click on Add
Name: Descriptive name e.g. ‘RemoteAccess_TunnelGroup’
IKE Peer Authentication:
Pre-shared Key: enter IKE Pre-shared key here
Identity Certificate: None
User Authentication:
Server Group: ‘LDAP_DOMAIN
Fallback: ‘Use LOCAL if Server Group fails‘, check this if you want to fall back to local users if AD is unavailable
Client Address Assignment:
DHCP Servers: leave blank
Client Address Pools: Select Address Pool previously created
Default Group Policy:
Group Policy: NoAccess
*Important, the default policy should be to lock them out, successful authentication&authorization provides the access policy
Advanced
Authorization:
Server Group: ‘LDAP_Domain
Users must exist in the authorization database to connect: Check this
-Ok

Enabling access to VPN clients:

To enable IPsec access:
– Go to Configuration > Remote Access VPN > Network (Client) Access > IPsec Connection Profiles
– Under Access Interfaces, check ‘Allow Access‘ for your ‘OUTSIDE‘ interface

To enable AnyConnect SSL / Legacy SSL:
– Go to Configuration > Remote Access VPN > Network (Client) Access > AnyConnect Connection Profiles
– Under Access Interfaces, check ‘Allow Access‘ for your ‘OUTSIDE‘ interface
– Under Connection Profiles, edit the tunnel group you created and give it an alias (The end users will see this in AnyConnect)
Login Page Setting: check ‘Allow user to select connection profile….
– SSL VPN clients also require a SSL certificate to be installed on the interface:
Configuration > Remote Access VPN > Advanced > SSL Settings
Edit your ‘outside’ interface and either create a self-signed certificate (warning will show on the client, but will work) or provide an existing certificate

NAT Rules
Configuration > Firewall > NAT Rules
If ‘Enable traffic through the firewall without address translation‘ is checked, you don’t need to add anything here
Otherwise:
Create a new NAT Exempt Rule:
Original
Interface: INSIDE
Source: Network object containing tunneled networks (generally your internal netblock, but you can use this to restrict devices VPN users can get to)
Destination: Create a new network object containing your VPN Client addresses
NAT Exempt Direction:
NAT Exempt outbound traffic from interface ‘INSIDE’ to lower security interfaces(Default)
Description:
Write a handy reminder for yourself when you are staring at this rule 6 months from now during a security audit.
– Ok

NAT Rules done

That should be everything, you should have IPsec and SSL VPN capability to your ASA now. Install the appropriate client on the end device and test.

Clients:

Both the Anyconnect SSL client and the IPsec VPN client are available for a variety of operating systems, and can be pre-installed and configured for the end-user.

On the IPsec VPN client, the Group name and Password are the Tunnel-Group name and the IKE pre-shared key. It will prompt for the user credentials afterwards.
PCF files can be created to securely distribute the IKE PSK to remote end-users (saved as a one way hash in the PCF file).