Posted on

Trouble Ticket #4: Unable to download NAT policy for ACE

Introduction

This was a fun one. Coming in to work on a Friday morning for what you hope is an uneventful segue in to the weekend and your colleague looks up from his fast scrolling terminal screen and says “we may have a problem”.

Unable to download NAT policy for ACE

The change he had implemented was simple enough. Add a couple of new sub-interfaces to the Cisco ASA firewall, add the required security ACLs and configure the NAT and no-NAT (NAT0) rules. The firewall code was still pre 8.3 on 8.0(4), so used the older NAT syntax.

The problem arose when the no-NAT config was applied, specifically adding ACE entries to the ACL that the no-NAT applied to the new interfaces referenced. The firewall threw up the following message:

[sourcecode language=”plain”]
Unable to download NAT policy for ACE
[/sourcecode]

In the context of the above sequence of events, this message isn’t actually that obscure. Pre version 8.3, the Cisco ASA uses policy based NAT. For the no-NAT, it uses an ACL to decide which traffic should not be NAT’d as it comes in to an interface. As the new ACEs were being put in to the firewall, the above message is effectively telling us that the firewall was unable to apply this to the no-NAT policy. So the ACE shows up in the config, but it isn’t having any effect.

In addition to this, we had also lost management access to certain networks through the firewall as part of this change.

The fix

The config was rolled back as a matter of course but the issue remained. Running packet tracer on the firewall showed that the issue was down to the no-NAT, although comparing the config with a backup showed no differences.

Based on our gut feelings and the message we saw, the NAT0 statement was removed and re-added and the issue vanished. Searching Cisco.com brought up this bug (CSCsl46310). Cisco recommend reloading the firewall as a workaround prior to reapplying the NAT0 statement, but that wasn’t required in our case.

Known fixed releases are supposedly 8.2(0.79), 8.0(3.2) and 8.1(0.130), although on the download site, 8.2.5 is a recommended version so I think that will be my first stop.

Summary

It is actually a pleasant surprise when a bug at least produces behaviour and a system message that can be used to troubleshoot without too much effort.

Till the next time.

Posted on

ASDM inaccessible on Cisco ASA 9.1(1)

Introduction

I’ll keep this short but sweet and hopefully this will save somebody a lot of head scratching out there. I unwrapped a couple of brand new ASA5512X firewalls but found the ASDM inaccessible.

ASDM inaccessible

All the standard stuff was in there, entered in global configuration mode:

I had enabled the http server and told the ASA which host address to accept connections from. I had enabled local authentication and a user name. When I connected, I got a “This webpage is not available” message.

After some sniffing around, I found a line of config that is critical:

This is the default on the two ASA devices I received. A security device. That has DES encryption as the default setting. Not very good Cisco. Not only is it weak encryption, but it stops my Chrome connecting to ASDM. Funnily enough, IE8, which was another browser on the jump box I was using allows the connection but I missed this until after I fixed it due to being a convert to Chrome for quite some time. So one option is to use an older, less secure browser. Or…

Resolution

The right fix would be to change the default ssl encryption as below:

This now allows more recent (and secure) browsers to connect to ASDM. The command above is shown in the default config in version 9.1(1). In older versions, you would need to type:

I also believe the default in older versions would be to enable pretty much all levels of SSL encryption:

Summary

The key point to remember here is, when you use a new version of software that you have become familiar with, try and find out what the differences are! I’ve not checked at which point this change was made and whilst it’s not a show stopper it is annoying.

By the way, you may have noticed I set the encryption at AES128 and some of you may be aware that AES256 is an option. The reason I currently choose AES128 over the 192 or 256 bit versions is I’ve read about vulnerabilities (albeit non-critical) with those key lengths. I’d be interested in anybody else’s take on this.

Till the next time.

Posted on

Trouble Ticket #3: Cisco Secure Desktop, ASDM not reflecting CLI

Introduction

Just in case anybody was thinking ‘wow, this guy must have a cushy job, only three trouble tickets over the last 18 months’, then think again. The purpose of these trouble ticket posts is to write about those issues that don’t occur every day, the crafty quirks that have you scratching your head. In this case, I was witnessing something with the Cisco Secure Desktop feature that didn’t make sense and was frustrating to say the least.

Cisco Secure Desktop

For those not in the know, Cisco Secure Desktop (CSD) is a feature of the Cisco ASA platform that allows remote access VPN clients to go through a number of health checks before being allowed to connect. In the scenario discussed here, I had configured a couple of client certificate checks and wanted to add a third for a specific client type.

Diagram

CSD_ASDM

The diagram above shows the third certificate check added at the bottom right. If this third check fails, the client is denied login.

Problem

When I saved the changes, everything looked as it should. The policy in the diagram above is not saved in the normal configuration file on the ASA but in this file: ‘flash:/sdesktop/data.xml’. As you can see from the snippet below, immediately after entering the third certificate check, the GUI and CLI agree on the configuration (certain details changed\removed):

The problem reared it’s ugly head when I closed ASDM and reloaded it. The XML file always shows correctly what the ASA is doing, but ASDM was now missing that third check in the GUI so now it appeared it was only carrying out the original certificate checks. That meant that to delete the third check, or add yet another check, I would need to hack the XML file.

Troubleshooting steps

I tried reloading the ASA. That made the GUI update to reflect the current XML file, but again, as soon as I made any changes and reloaded ASDM, the GUI would revert back to the two certificate check, regardless of what other checks I added. It was as if there was a cache somewhere.

I upgraded CSD to the latest version and still no joy. At that time, I got Cisco TAC involved. The person who I was dealing with was helpful and took all the details so he could lab it up. At first he reported he couldn’t replicate the issue but a few days later got back to me to report it was occurring but intermittently.

He requested some Java debug logs, which I duly gave and then waited for about a week while he escalated it to the development team.

Workaround

Finally, he came back to me to advise that this was being listed as a new bug:

CSCuh08360

Lucky me, I got my first bug listed. Something tells me it won’t be my last. There is a better workaround than rebooting the ASA each time though:

Summary

Basically, disabling and re-enabling CSD re-syncs the ASDM with the XML file, so until the bug is fixed, if you see this issue, I’d suggest doing this workaround before you make any changes to your CSD policies. FYI, I was using version 1.7 of Java and apparently 1.6 doesn’t have the same problem. So, to manage your firewall security device, feel free to use a less secure Java client. Sigh…

Till the next time.

Posted on

Cisco ASA – OSPF passive interface is…inactive

Introduction

I recently came across a quirk in the ASA’s implementation of OSPF that is widely known but I thought I’d share it here anyway. With the firewall being placed at the security boundary that quite often connects to the Internet on the outside interface, you would most often have a default route redistributed in to OSPF, although of course you could also be talking OSPF to your service provider depending on your situation.

However, in my home lab, the lab itself terminates on an ASA, which has it’s outside interface on my home LAN. I connect in to my lab from that LAN through the ASA and so wish to advertise that network in to the lab for reachability.

All well and good. Except that I have no OSPF speakers on my home LAN and the ASA, by virtue of line 2, is now desperately trying to chat something up on 192.168.1.0/24. In my situation, this is mostly harmless. In another, it poses a security risk as anybody could in theory fire up an OSPF speaker, learn about the internal routes and inject it’s own. Obviously, there are ways to protect this from happening e.g. MD5 authentication but quickly firing up Wireshark shows me the traffic is there and I hate seeing traffic on the wire that doesn’t have to be (Dropbox LAN Sync, I’m looking right at you!).

OSPF passive interface

So it makes perfect sense to use the OSPF passive interface functionality in this scenario. This allows me to turn OSPF chatter off on an interface. In conjunction with the network statement on line 2 above, I can advertise the network in to OSPF, but OSPF will not try to talk on the interface. Job done.

Except that for some reason, the ASA does not support this. The command exists under the RIP and EIGRP configuration modes but not OSPF. One possible way to resolve this would be:

Let’s remove the network statement. This removes the network from being advertised and also stops OSPF talking on that interface. Line 3 redistributes connected subnets (duh, obviously!) in to the OSPF process.

As a side note, line 3 advertises subnets as they are configured on the interface e.g. 192.168.1.128/25 would be advertised as such. If you miss the ‘subnets’ keyword off, it will only advertise classful networks, in our previous example, 192.168.1.128/25 would not be redistributed. Also, if you have the subnets keyword already added, negate the full line before adding it back in without the subnets keyword. It warns that only classful networks will be redistributed, but if you check the config, the subnets keyword remains and, in our example, 192.168.1.128/25 would be redistributed. More quirkiness.

Summary

Perhaps somebody reading this has insight as to why this functionality has been missed off the ASA platform. The workaround discussed above isn’t perfect either. Anything redistributed will show as an external route in your routing table and quite often that isn’t what you want.

Till the next time.

Posted on

Logging options on the Cisco ASA

Introduction

Logging is a critical function of any device in your network, but perhaps even more so on a firewall. Whether you are troubleshooting an issue, following an audit trail or just wanting to know what is going on at any time, being able to view generated logs is highly valuable. This post looks at logging options on the Cisco ASA and discusses some of the things you need to consider.

Tick tock

It’s all very well looking through your logs as individual events but if you want to tie them together, particularly across multiple devices, then you need to ensure that all of your devices have the correct time configured. Nothing says ‘I love you’ more than a time stamp that you can trust. If you use a centralised logging solution, you can filter logs across multiple devices to help determine root cause of issues. You can configure time on the ASA manually by using the following commands:

The above lines configure the time, the timezone relative to UTC and the daylight savings time settings respectively. There is a battery on the motherboard of the ASA that will keep the time should the device lose power or be rebooted. However, locally configured time can drift over…time, so what we really want is to use a trusted external time source that all devices synchronise against and this is where NTP comes in.

Lines 1-2 above dictate that we should be using authentication with NTP for added security and gives a key to use. Line 3 is required to advise the ASA that this key is trusted. Line 4 tells us which server to use, which interface it can be found on and which authentication key to use. It also tells the ASA to prefer this time source over other NTP servers of the same judged accuracy based on what stratum they are in. You should configure at least two NTP servers for redundancy. In the event that all servers are unavailable for an extended period, the ASA can fall back to using the local clock. NTP is a Jekyll and Hyde protocol. It can be as simple to understand as the last section or you can dive deep in to it’s bowels and be lost forever.

Log destination

Logs can be sent to several destinations but before I list them, it should be noted that logs come from two key sources, system events and network events. System events include things like CPU errors, network events include packets being denied on a certain interface. Both types of messages are dealt with by the logging subsystem and are then potentially filtered prior to being sent to one of the following destinations:

  • Console – logs sent here can be viewed in real time when you are connected to the serial port. As this causes CPU interrupts for each message, you need to be careful when enabling this
  • ASDM – logs can be viewed in the ASDM GUI. From here, you can quickly build filters, colour code the logs by severity and save the log as a local text file to be dealt with later or simply archived
  • Monitor – logs to a Telnet or SSH session. But you don’t still use Telnet for management do you?!?
  • Buffered – this is the internal memory buffer
  • Host – a remote syslog server
  • SNMP – rather than sending logs remotely using the syslog syntax, you can use SNMP to send a trap
  • Mail – send generated logs via SMTP
  • Flow-export-syslogs – send event messages via NetFlow v9

Log severity levels

Before I show some examples of how to configure different logging, it’s worth looking at the different severity levels available to us. There are eight in total as per Cisco’s definitions below:

Numeric level Name Definition
0 Emergencies  Extremely critical “system unusable” messages
1 Alerts Messages that require immediate administrator action
2 Critical A critical condition
3 Errors An error message (also the level of many access list deny messages)
4 Warnings A warning message (also the level of many other access list deny messages)
5 Notifications A normal but significant condition (such as an interface coming online)
6 Informational An informational message (such as a session being created or torn down)
7 Debugging A debug message or detailed accounting message

By selecting a lower severity (with a higher number), you are also opting in to everything with a higher severity e.g. level 4 will not only log all warnings but all errors, critical, alert and emergency logs. Be wary of selecting too low a severity level, particularly on the console. You can quickly bring a device to it’s knees if it’s getting hammered.

Examples

Here are some examples to show how to get things up and running.

Line 1 enables logging globally. We then enable timestamps on the log messages, without which it’s difficult to tell when an event occurred. Line 3 configures the size of the local buffer memory. Once this fills up, it is overwritten in a circular fashion. Lines 4 and 5 configure the buffered and monitor destinations previously discussed for the same level, the first using the keyword ‘warnings’ and the second using the equivalent numerical value. Both are interchangeable but will show in various command outputs using the keyword regardless (except in the logs themselves, where the numerical form will display). Lines 6 and 7 are configured together for remote syslog logging. Line 6 enables the logging at the specified level (in this case informational) and line 7 configures the syslog server IP address and the interface it can be found on.

Line 8 allows various other attributes to be included in each log message. In this case, it will include the hostname but can also include the firewall IP address of a particular interface, the context name (where used) or a specific string. The latter could be useful for using regular expressions for refining logs at a more granular level.

Finally, the show logging command will firstly show the different settings for each logging destination and then the current contents of the local log buffer. Below is an example of its output with just the first log entry for brevity (please note the enabled settings below are not by any means ideal for a production environment, you need to consider what is best for yours):

One thing to note about logging to Telnet\SSH sessions using the monitor destination. Whilst you may have this enabled and will be able to see the messages logged count in the above output rising each time, you may find yourself confused as to why, whilst SSH’d on to your ASA, you aren’t seeing the logs on your screen. To view logs for the current session, assuming they are enabled, you need to type this command in whilst connected:

and the logs will start appearing according to the severity level you have set. In a move that I can only attribute to Cisco allowing a drunken intern to come up with the command to negate the above one, somebody settled on:

Thanks drunken intern guy. To finish off this post, I’ll bundle some other commands together with a brief description:

If you have a pair of firewalls configured in a failover configuration, you can enter the first command to enable logging on the standby unit also. Just be aware of the increase in traffic if logging externally to the ASA. The second line will additionally send debug messages to any configured syslog servers, which is disabled by default. Again, this can cause a severe increase in traffic, especially if you enable lots of debugs. The last command changes messages to a proprietary Cisco format and to be honest, I don’t think its used much at all.

Summary

Hopefully you will have learnt a couple of extra things you can do with logging from this post but you can dive even deeper and I suggest you do to get the most out of this critical function. For example, you can archive buffered logs to the local flash or to a remote FTP server, you can disable certain messages completely or just filter them from certain destinations. You can also change the default severity of individual messages to better suit your environment. It would require a certain amount of initial work but would be easily repeatable across your estate.

A great place to learn more is to use the ASDM console which, despite me being a CLI fiend on the ASA, comes in to its own when configuring and reviewing logs. Also pay special attention to what level of logging you have for each destination. I’ve only covered a couple of key points on how best to do this (e.g. disable console logging) as what works best depends on your environment. If possible though, try to use a centralised Syslog server and use the ASDM logging due to it’s immediate nature and filtering capabilities.

Till the next time.

Posted on

Cisco ASA Identity Firewall

Introduction

When the CTO approached me asking how access to a subnet was restricted, I advised him that the people who needed access were given a DHCP reservation and an ACL on a Cisco ASA limited those IP addresses to certain destination hosts on certain ports. It wasn’t scalable and it wasn’t particularly secure. For example, if one of those users wanted to log on to a different machine, they would get a different IP address and access would break. If their computer NIC got sick and needed replacing, same thing. Worse still, anybody could log on to their machine and get access to the same resources or give themselves a static IP when the other user had their computer turned off and nobody would be the wiser.

I had been looking at a number of methods to address this, such as 802.1x authentication\application proxy and during our discussion, we both came to the conclusion that it would be pretty cool if we could restrict access via usernames, especially if this could remain transparent to those users. This would give us the flexibility and transparency we were looking for whilst maintaining the required level of security. Isn’t that the perfect balance that we strive for in IT security?

!!!DRUM ROLL!!!

Enter the Identity Firewall feature on the Cisco ASA platform. This is a new feature available from software version 8.4(2). The Identity Firewall integrates with Active Directory using an external (to the ASA) agent. The Cisco.com website has a host of documentation on the feature which you should follow to get it up and running but below is a summary of how it works, some things to be aware of and my thoughts on the feature.

How it works

First of all, this feature has three main components. The ASA itself, Active Directory and an agent which sits on one of your domain machines and feeds information back to the ASA. There are a number of prerequisites you need to make sure are in place and rules you must follow before rolling this out e.g. AD must be installed on certain OS versions, same for the server the agent is on, you can have multiple agents for redundancy but only one installed on a domain controller, you must configure the correct level of domain wide security auditing, ensure host firewalls are opened accordingly. The configuration guide does a pretty good job of listing all of these and I would heartily recommend you check it out if deploying the feature.

There is also the relatively small matter of creating an account in AD that the ASA will use to communicate with AD. You also need to create some config on the ASA itself made up of a mix of AAA and specific ‘user-identity’ commands. The agent is configured via the command prompt (or Powershell prompt as I tend to always use since I fell in love with it deploying Exchange 2007). You set up which AD servers it should be monitoring (you want to point it to all DCs that will be logging account logon\logoff events) and then which clients (ASAs) the agents will be reporting to. You can also configure Syslog servers for better accounting.

In a nutshell, the agent is checking the security logs of the DCs to see who is logging on and with what IP address. It adds this information to a local cache and sends it to the configured ASA clients. This is done using Radius. The ASA also talks directly to AD using LDAP to get AD group membership. The final piece of the puzzle is enabling the Identity Firewall so that all of these components start talking. At this stage, there are various commands you can run via the agent or on the ASA to confirm the communication is working as expected.

It is then a matter of creating ACLs that utilise users and\or groups and giving them access to resources. You can also combine this with source IP which means you can say Fred can access Server X when he is in Subnet Y but if he moves, he loses access. If Janet logs on to Fred’s original machine, she could have the same IP but won’t be given the same access due to her username not being in the ACL.

I found setting it all up very easy. However, circumstance led to it being some time before all interested parties were happy with the outcome e.g. the firewall being used had recently been upgraded from 8.0 code to 8.4, and a load of migrated NAT rules started playing up, muddying the waters. The positive side to this was that the firewall config has been streamlined considerably. There was also the morning spent troubleshooting a single user’s issue that ended up being down to him not being in the correct AD group as assumed. That said, I think it’s a pretty robust feature for what is effectively a first iteration.

Testing

I’ve tried to break it a number of ways e.g. pull an authorised user’s network cable out of their machine, give myself their static IP, but without being logged on as them, my username\IP pairing doesn’t match what the firewall thinks. There are a number of timers involved with this feature that might trip you up if you don’t understand them e.g. the firewall can only check group membership every hour at best, meaning if you remove a user from a group, they could still retain access. Worse still, if you disable their account in AD but keep them in the group, they will still have access until their Kerberos ticket expires. You can manually update group membership on the ASA if this would be an issue for you. I would guess that if it is, the person has been marched out of the building at that point anyway.

VPN logon vs AD logon

Another thing to be aware of is remote VPN access. When you remote VPN on, you get authenticated to the firewall. This could be via local ASA accounts, Radius, TACACS, ACS, LDAP etc. If you use AAA LDAP authentication (using Active Directory in this case), you are not logging on to the domain as you VPN in, you are simply saying ‘here are my AD credentials, please authenticate me on the firewall’. At that point, one of two things happens with the Identity Firewall. If you are using a domain computer to remote on, that machine will automatically try to make contact with a DC. When it finds one (over the VPN), it will log on to the domain, create a security log and the AD agent will let the ASA know. Any rules assigned to that user, that don’t filter on source IP, will now come in to effect. However, if the machine is not joined to the domain, there will be no logon event (the username\password given at connection was only for VPN authentication), and so any user-identity ACLs will not apply.

In the latter case, you may want to create a specific VPN connection profile with specific DHCP pool for example and go back to restricting by IP, only allowing users in a certain AD group for example to be able to connect to that profile. Not ideal, but it would work.

Summary

In summary, I really like this feature. It answers a number of questions specific to the environment I have implemented it in. Things I particularly like are its ease of setup including the troubleshooting commands to aid with this. Learning the syntax of the ACLs is a simple task, as they extend upon existing IP-based ACLs. I also like the fact that there are several ‘under the hood’ settings you can tweak to improve security further still e.g. remove users from the cache if their machine as been idle for a certain amount of time, force LDAP lookups over SSL.

Something that I wasn’t so keen on was the split communication i.e. the ASA getting group membership directly from AD, IP mapping from the agent. I would like to see the agent providing the group membership too. It should also not be too hard to tell the agent which groups to keep a closer eye on (the ones used in ACLs sounds like a good start). This would allow the ACL to be updated much quicker should a group be amended.

I would love to hear what others’ experience of this feature has been. Overall, I am very pleased with it and look forward to upcoming improvements.

Edit: additional points

A few other points I should highlight, if you move to a different subnet the new IP address will be picked up as long as your device authenticates with the domain. I tested this early on by disconnecting my laptop from the wired network with a persistent ping happening in the background. I then VPN’d back to the office via the WiFi network and within a couple of seconds of the connection be made, the ping reestablished.

The second point which will hopefully save somebody a head ache out there; a user testing this functionality was reporting it working as expected until they would open a spreadsheet that would make a data connection to a SQL database in a different subnet. The user had access to this SQL database as allowed by the Identity Firewall ACLs but as soon as the data was refreshed in the spreadsheet, he would get kicked. I could see it happening using the ASA and AD agent troubleshooting commands too. After a little bit of head scratching, we realised that the spreadsheet data connection was using a different set of domain credentials to connect to the database. This triggered a logon event for the user’s IP address but with the user account for the data connection. In turn, the AD Agent updated the cache with the new user and IP combination and denied the traffic based on the ACL. One answer to this would be to add the data connection account to the ACL but in the end, we just connected to the database in the spreadsheet using the user’s own credentials. This highlights the fact that, whilst a user can be affiliated with multiple IP addresses, only one user can be affiliated with any one IP address, which is the key to making this feature secure.

One last thing which is perhaps the most important, considering this is a discussion about a security feature. If your users connect via a shared IP address, you should be aware. For example, if your users log on to a Terminal Server, they will all be coming from the same IP address by default. That means that if a user with elevated permissions logs on to the TS, all other users will effectively gain the same access as permitted via the Identity Firewall. My advice would be to deny access via a shared IP address to any resources that need restricted access.

Till the next time.