Posted on

Trouble Ticket #4: Unable to download NAT policy for ACE

Introduction

This was a fun one. Coming in to work on a Friday morning for what you hope is an uneventful segue in to the weekend and your colleague looks up from his fast scrolling terminal screen and says “we may have a problem”.

Unable to download NAT policy for ACE

The change he had implemented was simple enough. Add a couple of new sub-interfaces to the Cisco ASA firewall, add the required security ACLs and configure the NAT and no-NAT (NAT0) rules. The firewall code was still pre 8.3 on 8.0(4), so used the older NAT syntax.

The problem arose when the no-NAT config was applied, specifically adding ACE entries to the ACL that the no-NAT applied to the new interfaces referenced. The firewall threw up the following message:

[sourcecode language=”plain”]
Unable to download NAT policy for ACE
[/sourcecode]

In the context of the above sequence of events, this message isn’t actually that obscure. Pre version 8.3, the Cisco ASA uses policy based NAT. For the no-NAT, it uses an ACL to decide which traffic should not be NAT’d as it comes in to an interface. As the new ACEs were being put in to the firewall, the above message is effectively telling us that the firewall was unable to apply this to the no-NAT policy. So the ACE shows up in the config, but it isn’t having any effect.

In addition to this, we had also lost management access to certain networks through the firewall as part of this change.

The fix

The config was rolled back as a matter of course but the issue remained. Running packet tracer on the firewall showed that the issue was down to the no-NAT, although comparing the config with a backup showed no differences.

Based on our gut feelings and the message we saw, the NAT0 statement was removed and re-added and the issue vanished. Searching Cisco.com brought up this bug (CSCsl46310). Cisco recommend reloading the firewall as a workaround prior to reapplying the NAT0 statement, but that wasn’t required in our case.

Known fixed releases are supposedly 8.2(0.79), 8.0(3.2) and 8.1(0.130), although on the download site, 8.2.5 is a recommended version so I think that will be my first stop.

Summary

It is actually a pleasant surprise when a bug at least produces behaviour and a system message that can be used to troubleshoot without too much effort.

Till the next time.

Posted on

Technical rewind

Introduction

I was recently thinking about the future of this blog and had been considering whether to bin it or come back to it with renewed enthusiasm. After all, there are thousands of other blogs out there that cover similar topics, ranging in quality from barely readable to excellent. Whilst I hope that mine falls no further down than the middle of that scale, I asked myself what value do people get from my own posts.

When I logged on to the admin portal for the first time in a while, I  saw two key things that made me realise that I should continue writing, perhaps not as frequently as some other bloggers, but with more posts that are close to my heart and hopefully that will shine through in my writing. The first was that, whilst my viewing figures are not particularly spectacular, they have been constant throughout my recent absence so people are still coming over, both to check out what is on offer and also from search results. The second thing I noticed was that there were almost a dozen updates for WordPress itself, the theme and some plugins and I found myself feeling quite protective and applied the relevant TLC.

Technical rewind

I’ve worked in IT for well over 10 years, achieved my CCNA back in 2009 and my CCNP in about 2012. I got past the half way point towards my CCNP Security and then something dawned on me. Something that made me down my certification tools and take a long look at myself. My appointment to a management role in the last year has only cemented my thinking.

The quest I was on to further my knowledge according to Cisco’s road map in addition to my new, less hands on role had left my foundational routing and switching knowledge less polished than I would have liked. I still function as a good network engineer, but I get a certain satisfaction from having nuts and bolts knowledge at my fingertips and I’ve been aware that this has slipped since the new year.

Regarding the certification path, the blueprints for most of the exams never match the on the job knowledge requirements. So in a busy world, you spend huge amounts of time learning about things that Cisco want you to learn, but your boss isn’t bothered about and quite often, nor should you be. They are just not relevant for the day to day or even tomorrow.

With that in mind and with the time that I am currently able to commit to studying, I am going to aim for the CCIE R&S Written as a way of refreshing my current certs but more importantly, I will deep dive in to all the relevant topics to give that much needed polish. Those studies will hopefully provide me with some good topics on which to blog too.

Summary

As I recently tweeted, I find that knowledge is a foundation to build upon rather than a skip to fill up. Being self aware of when that knowledge needs some maintenance is a key skill for any engineer to prevent it all falling down about them. Do your core skills need brushing up on?

Till the next time.

Posted on

Running ASDM and WebVPN on the same interface

Introduction

So you are thinking of running ASDM and WebVPN on the same interface? This is quite a rare configuration for the simple reason that ASDM is a management tool and WebVPN is usually enabled on the outside interface and best practice would dictate using an internal or even dedicated management interface to allow ASDM\CLI connections to. However, in a lab environment, this isn’t such an issue. In fact, in my labs, the machine I manage the ASA from is also the machine I test VPN connectivity from so this is a requirement for me.

Running ASDM and WebVPN on same interface

You basically have two options. You can change the port that ASDM runs on, or change the port that WebVPN runs on. As stated, this is mostly seen in a non-production environment so it probably doesn’t matter too much which way you do it but if for any reason you had to use this configuration in production, you would probably want to change the ASDM port so your remote users don’t have to worry about changing ports.

Both options are very simple to implement. To change the ASDM port, you enter a modified version of the command you enter to enable ASDM:

This changes the ASDM port to 4343. As stated, missing out ‘4343’ still enables ASDM but on the default port of 443.

To change the WebVPN port only requires an extra line:

Of course, both services can be run on the same port if required, but you need to know the URL to access ASDM. (The WebVPN URL is the default and so will load with just the IP address\hostname). The ASDM URL at time of writing on software version 9.1(2) is:

https://ipaddress/admin/public/index.html

Once you have downloaded and installed the ASDM launcher, you again don’t need to worry about having different ports as the launcher itself connects to the correct URL automatically.

Summary

There may not be many situations in which you would consider running ASDM and WebVPN on the same interface but it’s good to know it can be done from both a port and URL point of view.

Till the next time.

Posted on

ASDM inaccessible on Cisco ASA 9.1(1)

Introduction

I’ll keep this short but sweet and hopefully this will save somebody a lot of head scratching out there. I unwrapped a couple of brand new ASA5512X firewalls but found the ASDM inaccessible.

ASDM inaccessible

All the standard stuff was in there, entered in global configuration mode:

I had enabled the http server and told the ASA which host address to accept connections from. I had enabled local authentication and a user name. When I connected, I got a “This webpage is not available” message.

After some sniffing around, I found a line of config that is critical:

This is the default on the two ASA devices I received. A security device. That has DES encryption as the default setting. Not very good Cisco. Not only is it weak encryption, but it stops my Chrome connecting to ASDM. Funnily enough, IE8, which was another browser on the jump box I was using allows the connection but I missed this until after I fixed it due to being a convert to Chrome for quite some time. So one option is to use an older, less secure browser. Or…

Resolution

The right fix would be to change the default ssl encryption as below:

This now allows more recent (and secure) browsers to connect to ASDM. The command above is shown in the default config in version 9.1(1). In older versions, you would need to type:

I also believe the default in older versions would be to enable pretty much all levels of SSL encryption:

Summary

The key point to remember here is, when you use a new version of software that you have become familiar with, try and find out what the differences are! I’ve not checked at which point this change was made and whilst it’s not a show stopper it is annoying.

By the way, you may have noticed I set the encryption at AES128 and some of you may be aware that AES256 is an option. The reason I currently choose AES128 over the 192 or 256 bit versions is I’ve read about vulnerabilities (albeit non-critical) with those key lengths. I’d be interested in anybody else’s take on this.

Till the next time.

Posted on

Trouble Ticket #3: Cisco Secure Desktop, ASDM not reflecting CLI

Introduction

Just in case anybody was thinking ‘wow, this guy must have a cushy job, only three trouble tickets over the last 18 months’, then think again. The purpose of these trouble ticket posts is to write about those issues that don’t occur every day, the crafty quirks that have you scratching your head. In this case, I was witnessing something with the Cisco Secure Desktop feature that didn’t make sense and was frustrating to say the least.

Cisco Secure Desktop

For those not in the know, Cisco Secure Desktop (CSD) is a feature of the Cisco ASA platform that allows remote access VPN clients to go through a number of health checks before being allowed to connect. In the scenario discussed here, I had configured a couple of client certificate checks and wanted to add a third for a specific client type.

Diagram

CSD_ASDM

The diagram above shows the third certificate check added at the bottom right. If this third check fails, the client is denied login.

Problem

When I saved the changes, everything looked as it should. The policy in the diagram above is not saved in the normal configuration file on the ASA but in this file: ‘flash:/sdesktop/data.xml’. As you can see from the snippet below, immediately after entering the third certificate check, the GUI and CLI agree on the configuration (certain details changed\removed):

The problem reared it’s ugly head when I closed ASDM and reloaded it. The XML file always shows correctly what the ASA is doing, but ASDM was now missing that third check in the GUI so now it appeared it was only carrying out the original certificate checks. That meant that to delete the third check, or add yet another check, I would need to hack the XML file.

Troubleshooting steps

I tried reloading the ASA. That made the GUI update to reflect the current XML file, but again, as soon as I made any changes and reloaded ASDM, the GUI would revert back to the two certificate check, regardless of what other checks I added. It was as if there was a cache somewhere.

I upgraded CSD to the latest version and still no joy. At that time, I got Cisco TAC involved. The person who I was dealing with was helpful and took all the details so he could lab it up. At first he reported he couldn’t replicate the issue but a few days later got back to me to report it was occurring but intermittently.

He requested some Java debug logs, which I duly gave and then waited for about a week while he escalated it to the development team.

Workaround

Finally, he came back to me to advise that this was being listed as a new bug:

CSCuh08360

Lucky me, I got my first bug listed. Something tells me it won’t be my last. There is a better workaround than rebooting the ASA each time though:

Summary

Basically, disabling and re-enabling CSD re-syncs the ASDM with the XML file, so until the bug is fixed, if you see this issue, I’d suggest doing this workaround before you make any changes to your CSD policies. FYI, I was using version 1.7 of Java and apparently 1.6 doesn’t have the same problem. So, to manage your firewall security device, feel free to use a less secure Java client. Sigh…

Till the next time.

Posted on

10 tenets of working in IT – Tenet 4, Cross Pollinate

Introduction

This may be a generalisation, but in my experience the larger the company you work at, the more siloed you become. The smaller the company, the more broad your skill set usually needs to be. This isn’t always the case of course but it has been for every single one of my jobs. This post is aimed mainly at those people who do find themselves in a silo. Don’t limit your skill set. Talk to your colleagues in the next cubicle. Learn storage, Windows, Linux, scripting. Get multi-vendor skills. Do all of this to the depth to make you better at your job and less reliant on others. A good IT engineer should be able to engage with his peers with other skill sets. Get a hobby – doesn’t have to be related to your work but it lets the mind grow.

Generalise versus specialise

This post inevitably brings up the question of generalisation versus specialisation but I want to keep it short. Perhaps I’ll cover this never-ending discussion in more depth in another post but in my opinion, its not a ‘choose one’ answer. In simple terms, you can specialise in fewer topics and generalise in more. The depth you go to also affects the number of skills you acquire. The answer about what balance to strike depends on a number of factors e.g. the job market now, trends, your current employers’ requirements and of course you i.e. the answer changes for each person based on a large number of factors. Enough said for now…

Summary

It’s 2013 and at no earlier point has it been more obvious that most people who work in IT need to have a wider range of skills and knowledge to do their day to day job. Those that don’t are either mining a vein of speciality wealth or will inevitably be left behind in the wake of technology.

Till the next time.

Posted on

Logging options on the Cisco ASA

Introduction

Logging is a critical function of any device in your network, but perhaps even more so on a firewall. Whether you are troubleshooting an issue, following an audit trail or just wanting to know what is going on at any time, being able to view generated logs is highly valuable. This post looks at logging options on the Cisco ASA and discusses some of the things you need to consider.

Tick tock

It’s all very well looking through your logs as individual events but if you want to tie them together, particularly across multiple devices, then you need to ensure that all of your devices have the correct time configured. Nothing says ‘I love you’ more than a time stamp that you can trust. If you use a centralised logging solution, you can filter logs across multiple devices to help determine root cause of issues. You can configure time on the ASA manually by using the following commands:

The above lines configure the time, the timezone relative to UTC and the daylight savings time settings respectively. There is a battery on the motherboard of the ASA that will keep the time should the device lose power or be rebooted. However, locally configured time can drift over…time, so what we really want is to use a trusted external time source that all devices synchronise against and this is where NTP comes in.

Lines 1-2 above dictate that we should be using authentication with NTP for added security and gives a key to use. Line 3 is required to advise the ASA that this key is trusted. Line 4 tells us which server to use, which interface it can be found on and which authentication key to use. It also tells the ASA to prefer this time source over other NTP servers of the same judged accuracy based on what stratum they are in. You should configure at least two NTP servers for redundancy. In the event that all servers are unavailable for an extended period, the ASA can fall back to using the local clock. NTP is a Jekyll and Hyde protocol. It can be as simple to understand as the last section or you can dive deep in to it’s bowels and be lost forever.

Log destination

Logs can be sent to several destinations but before I list them, it should be noted that logs come from two key sources, system events and network events. System events include things like CPU errors, network events include packets being denied on a certain interface. Both types of messages are dealt with by the logging subsystem and are then potentially filtered prior to being sent to one of the following destinations:

  • Console – logs sent here can be viewed in real time when you are connected to the serial port. As this causes CPU interrupts for each message, you need to be careful when enabling this
  • ASDM – logs can be viewed in the ASDM GUI. From here, you can quickly build filters, colour code the logs by severity and save the log as a local text file to be dealt with later or simply archived
  • Monitor – logs to a Telnet or SSH session. But you don’t still use Telnet for management do you?!?
  • Buffered – this is the internal memory buffer
  • Host – a remote syslog server
  • SNMP – rather than sending logs remotely using the syslog syntax, you can use SNMP to send a trap
  • Mail – send generated logs via SMTP
  • Flow-export-syslogs – send event messages via NetFlow v9

Log severity levels

Before I show some examples of how to configure different logging, it’s worth looking at the different severity levels available to us. There are eight in total as per Cisco’s definitions below:

Numeric level Name Definition
0 Emergencies  Extremely critical “system unusable” messages
1 Alerts Messages that require immediate administrator action
2 Critical A critical condition
3 Errors An error message (also the level of many access list deny messages)
4 Warnings A warning message (also the level of many other access list deny messages)
5 Notifications A normal but significant condition (such as an interface coming online)
6 Informational An informational message (such as a session being created or torn down)
7 Debugging A debug message or detailed accounting message

By selecting a lower severity (with a higher number), you are also opting in to everything with a higher severity e.g. level 4 will not only log all warnings but all errors, critical, alert and emergency logs. Be wary of selecting too low a severity level, particularly on the console. You can quickly bring a device to it’s knees if it’s getting hammered.

Examples

Here are some examples to show how to get things up and running.

Line 1 enables logging globally. We then enable timestamps on the log messages, without which it’s difficult to tell when an event occurred. Line 3 configures the size of the local buffer memory. Once this fills up, it is overwritten in a circular fashion. Lines 4 and 5 configure the buffered and monitor destinations previously discussed for the same level, the first using the keyword ‘warnings’ and the second using the equivalent numerical value. Both are interchangeable but will show in various command outputs using the keyword regardless (except in the logs themselves, where the numerical form will display). Lines 6 and 7 are configured together for remote syslog logging. Line 6 enables the logging at the specified level (in this case informational) and line 7 configures the syslog server IP address and the interface it can be found on.

Line 8 allows various other attributes to be included in each log message. In this case, it will include the hostname but can also include the firewall IP address of a particular interface, the context name (where used) or a specific string. The latter could be useful for using regular expressions for refining logs at a more granular level.

Finally, the show logging command will firstly show the different settings for each logging destination and then the current contents of the local log buffer. Below is an example of its output with just the first log entry for brevity (please note the enabled settings below are not by any means ideal for a production environment, you need to consider what is best for yours):

One thing to note about logging to Telnet\SSH sessions using the monitor destination. Whilst you may have this enabled and will be able to see the messages logged count in the above output rising each time, you may find yourself confused as to why, whilst SSH’d on to your ASA, you aren’t seeing the logs on your screen. To view logs for the current session, assuming they are enabled, you need to type this command in whilst connected:

and the logs will start appearing according to the severity level you have set. In a move that I can only attribute to Cisco allowing a drunken intern to come up with the command to negate the above one, somebody settled on:

Thanks drunken intern guy. To finish off this post, I’ll bundle some other commands together with a brief description:

If you have a pair of firewalls configured in a failover configuration, you can enter the first command to enable logging on the standby unit also. Just be aware of the increase in traffic if logging externally to the ASA. The second line will additionally send debug messages to any configured syslog servers, which is disabled by default. Again, this can cause a severe increase in traffic, especially if you enable lots of debugs. The last command changes messages to a proprietary Cisco format and to be honest, I don’t think its used much at all.

Summary

Hopefully you will have learnt a couple of extra things you can do with logging from this post but you can dive even deeper and I suggest you do to get the most out of this critical function. For example, you can archive buffered logs to the local flash or to a remote FTP server, you can disable certain messages completely or just filter them from certain destinations. You can also change the default severity of individual messages to better suit your environment. It would require a certain amount of initial work but would be easily repeatable across your estate.

A great place to learn more is to use the ASDM console which, despite me being a CLI fiend on the ASA, comes in to its own when configuring and reviewing logs. Also pay special attention to what level of logging you have for each destination. I’ve only covered a couple of key points on how best to do this (e.g. disable console logging) as what works best depends on your environment. If possible though, try to use a centralised Syslog server and use the ASDM logging due to it’s immediate nature and filtering capabilities.

Till the next time.

Posted on

Overview of Cisco Catalyst 3850 switch

Introduction

As many of you will be aware, Cisco announced the release of the Catalyst 3850 switch at Cisco Live 2013 in London only last week. As I blogged at that time, this wasn’t the world’s best kept secret. Several people were talking about it online and I’d come across a few pages on different parts of Cisco’s website hinting that it was coming. There was mixed reaction to the news from ‘is this not just a 3750 with an integrated Wireless LAN Controller?’ to more warm and welcoming feedback. I’ll try and leave my own judgement until the end of the post but for now, let me list some of the specs of the 3850 and make the obvious comparison to the 3750X using data from Cisco’s website:

Comparison of Catalyst 3750-X and 3850 Switches

Features Cisco Catalyst 3750-X Cisco Catalyst 3850
Stacking bandwidth 64 Gbps 480 Gbps
Cisco IOS® Software wireless controller No Yes
Queues per port 4 8
Quality-of-service (QoS) model MLS MQC
Uplinks 4 x 1GE2 x 10GE NM4 x 1 GE or 2 x 10GE SM 4 x 1GE2 x 1/10GE4 x 1/10GE(on 48 port model)
StackPower Yes Yes
Flexible NetFlow support Yes (C3KX-SM-10G required) Yes
Multicore CPU for hosted services No Yes
Flash size 64 MB 2 GB
Operating system Cisco IOS Software Cisco IOS-XE Software

The first thing that is immediately obvious is I need to find a better way to format tables on my site!

The second thing is that, putting the integrated wireless functionality of the 3850 to one side for now, it is clear that the 3850 offers improvements in several areas; far greater bandwidth across a switch stack (where more than one of these switches are connected together as a single ‘virtual switch’. The actual stacking cables themselves are much improved too), more queues per port, a preferable QoS model and a move to IOS-XE which in itself has a number of improvements over vanilla IOS. Take a visit to various places on the web and you will find many more spec sheets that show improvements of all sorts e.g. more ACEs for security, QoS and PBR, a bigger TCAM and many more.

Integrated WLC

Whilst we all love having more of everything to play with on our favourite devices, I think that the feature that gives this announcement some punch is the wireless capabilities of the switch and all in a 1U form factor. You could also get this functionality in a 3750X but only on a 2U switch from what I recall. Of course, if you want to stack your switches and want redundancy in the WLC also, then 1U wins over 2U every time, 4U over 8U, etc.

The WLC integrated in to the 3850 has some features that you might want to see in any Cisco controller e.g. Clean Air, EnergyWise, QoS. One switch will support 50 WAPs and 2000 clients. Although I haven’t looked at purchasing these yet, I was told by a number of Cisco people at Cisco Live that the price is going to be comparable to a 3750X, but you will probably need to add on the WLC licencing to that base price.

Summary

If you consider that you are saving yourself the requirement for a standalone WLC on top of all of the increases in capabilities, the move to IOS-XE, the improvement in the stacking technology etc., the 3850 looks like a very capable and tempting upgrade to the 3750X. Cisco are classifying this product under Unified Access, bringing wired and wireless access in at the same point. I just wish I’d had the opportunity to put them in to our office network last year when I opted to use a pair of stacked 3750X switches with a 2504 WLC.

Till the next time.

Posted on

Catalyst 6800 switch? Maybe, maybe not…

Introduction

Just a quick piece of conjecture here before last call at the airport. I was hearing rumours of a Cisco Catalyst 6800 series switch over the last few days. Some quick and dirty ‘facts’:

  • It is being touted as the next generation chassis for the 6500. This is to allow the limits of the 6500 back-plane to be upgraded in order to accept some increasingly sophisticated line cards e.g. 100G, service-a-riffic <long week, that’s all your getting
  • Will accept half width line cards
  • Will be available in 2013

I only got a whiff of this on the last day so wasn’t able to delve any deeper but it definitely gives weight to the statements coming from Cisco that the 6500 platform is being invested in, even though they aren’t calling it the 6500 anymore! We’ll just have to wait and see.

Till the next time.

Posted on

Cisco Live London 2013 – part 2

Introduction

All good things come to an end and so it is with Cisco Live London 2013. Friday’s sessions were a deep dive on the Nexus 1000v and one covering cloud architectures. Both were good and provided lots of useful information that I need to get up to speed on. The fact that my ears were still ringing from last night’s Customer Appreciation Event and I’ve started to lose my voice from singing along hasn’t dampened my spirits.

Thursday

Thursday saw me attending three sessions. The first was on tuning Cisco’s IPS. If you use an IPS, check this session out. It was the best one I’ve attended this week and you could feel the presenter’s passion for his work coming through. The second session was on SDN but it didn’t work for me and I called it a day after an hour and went off to do a LISP walk in lab which gave me a taste of what it looks like in the CLI. The last session on Thursday was an advanced BGP session, explaining some of the new features that are in the pipeline. It was a bit dry but contained some useful nuggets.

Bonus

Before the CAE kicked off, I got an email from Cisco saying I’d won a £100 Amazon voucher. All that swag collecting apparently had the side effect of collecting points in an accumulator. I’ll let you know this time next year if that voucher has helped offset the email spam that I am likely to receive.

Time to party

I skipped last year’s CAE to meet up with some folks at the Fox outside the Excel centre who didn’t have a ticket and others that I had met during the week. This year, there was news that the guest performer was a British rock legend and I felt like rocking out so turned up. The food and drink were both plentiful and there was lots of entertainment put on for us such as beach volleyball, surfing, bucking bronco, some crazy limbo dude and a helter skelter.

Music initially came in the form of a Bob Marley tribute band who were awesome. Then a band I’d not heard of called the RPJ band (Rick Parfitt Junior being the front man). Not sure if they do their own material but they stuck to covers of some classic tunes such as Journey’s ‘Don’t Stop Believing’, The Killers’ ‘Mr Brightside’ and Nirvana’s ‘Smells Like Teen Spirit’. I’d manoeuvred myself to the front row with my beer and enjoyed some head banging and foot stomping. The band were very entertaining.

I should have known from the warm up band who the rock icon was going to be…junior’s dad Rick Parfitt from ‘The Quo’. I’ll be honest. When he came out, I thought ‘oh dear’. A couple of strums in to one of the three chords he had brought with him however and I was rocking out. Staff came along the front with inflatable guitars to hand out and that was all I needed to have a great night. I looked across at senior at one point and he gave me a wry smile and nodded in appreciation of my air guitar. Bugger didn’t invite me on stage though. A soon as the music ended after a couple of great hours, I realised why I don’t tend to go to gigs anymore (at my age!)…persistent ringing in my ears. But it was worth it.

On the train back to the flat, I started to feel a little peckish so headed down Brick Lane for a nice hot curry washed down with a night-cap.

Summary

Cisco Live 2013 was a different beast from last year. I was doing sessions from 09:00-18:00 last year. I scaled back on them this year and spent far more time in the World of Solutions, getting to know products in more depth, both those that I knew something about and others that I didn’t previously have any knowledge on whatsoever.

The WiFi was vastly improved this year. Where I was frequently unable to connect in 2012, I suffered no such issues this time. Lunches and refreshments were spot on again. I’ve got some gym time to rack up methinks. It was great meeting up with a lot of other techies from all different fields and putting faces to Twitter handles etc. and I hope to keep in touch with them.

Thanks to everybody who helped make this such a fun and instructive event including Cisco, the exhibition staff and of course the attendees. I now have the seemingly impossible task of trying to get a spot for next year’s event in Milan. I’m going to go dark for a few days now and spend time with my family but I have some scaffolding in place for technical posts in the near future so keep your eyes peeled. The final good point for me to make about Cisco Live is the legacy effect it has on me. I’m raring to get back to studying to complete my CCNP Security, dive deeper on some data centre\virtualisation stuff and later in the year, take a long hard look at the CCIE.

Till the next time.

Posted on

Cisco ASA Identity Firewall

Introduction

When the CTO approached me asking how access to a subnet was restricted, I advised him that the people who needed access were given a DHCP reservation and an ACL on a Cisco ASA limited those IP addresses to certain destination hosts on certain ports. It wasn’t scalable and it wasn’t particularly secure. For example, if one of those users wanted to log on to a different machine, they would get a different IP address and access would break. If their computer NIC got sick and needed replacing, same thing. Worse still, anybody could log on to their machine and get access to the same resources or give themselves a static IP when the other user had their computer turned off and nobody would be the wiser.

I had been looking at a number of methods to address this, such as 802.1x authentication\application proxy and during our discussion, we both came to the conclusion that it would be pretty cool if we could restrict access via usernames, especially if this could remain transparent to those users. This would give us the flexibility and transparency we were looking for whilst maintaining the required level of security. Isn’t that the perfect balance that we strive for in IT security?

!!!DRUM ROLL!!!

Enter the Identity Firewall feature on the Cisco ASA platform. This is a new feature available from software version 8.4(2). The Identity Firewall integrates with Active Directory using an external (to the ASA) agent. The Cisco.com website has a host of documentation on the feature which you should follow to get it up and running but below is a summary of how it works, some things to be aware of and my thoughts on the feature.

How it works

First of all, this feature has three main components. The ASA itself, Active Directory and an agent which sits on one of your domain machines and feeds information back to the ASA. There are a number of prerequisites you need to make sure are in place and rules you must follow before rolling this out e.g. AD must be installed on certain OS versions, same for the server the agent is on, you can have multiple agents for redundancy but only one installed on a domain controller, you must configure the correct level of domain wide security auditing, ensure host firewalls are opened accordingly. The configuration guide does a pretty good job of listing all of these and I would heartily recommend you check it out if deploying the feature.

There is also the relatively small matter of creating an account in AD that the ASA will use to communicate with AD. You also need to create some config on the ASA itself made up of a mix of AAA and specific ‘user-identity’ commands. The agent is configured via the command prompt (or Powershell prompt as I tend to always use since I fell in love with it deploying Exchange 2007). You set up which AD servers it should be monitoring (you want to point it to all DCs that will be logging account logon\logoff events) and then which clients (ASAs) the agents will be reporting to. You can also configure Syslog servers for better accounting.

In a nutshell, the agent is checking the security logs of the DCs to see who is logging on and with what IP address. It adds this information to a local cache and sends it to the configured ASA clients. This is done using Radius. The ASA also talks directly to AD using LDAP to get AD group membership. The final piece of the puzzle is enabling the Identity Firewall so that all of these components start talking. At this stage, there are various commands you can run via the agent or on the ASA to confirm the communication is working as expected.

It is then a matter of creating ACLs that utilise users and\or groups and giving them access to resources. You can also combine this with source IP which means you can say Fred can access Server X when he is in Subnet Y but if he moves, he loses access. If Janet logs on to Fred’s original machine, she could have the same IP but won’t be given the same access due to her username not being in the ACL.

I found setting it all up very easy. However, circumstance led to it being some time before all interested parties were happy with the outcome e.g. the firewall being used had recently been upgraded from 8.0 code to 8.4, and a load of migrated NAT rules started playing up, muddying the waters. The positive side to this was that the firewall config has been streamlined considerably. There was also the morning spent troubleshooting a single user’s issue that ended up being down to him not being in the correct AD group as assumed. That said, I think it’s a pretty robust feature for what is effectively a first iteration.

Testing

I’ve tried to break it a number of ways e.g. pull an authorised user’s network cable out of their machine, give myself their static IP, but without being logged on as them, my username\IP pairing doesn’t match what the firewall thinks. There are a number of timers involved with this feature that might trip you up if you don’t understand them e.g. the firewall can only check group membership every hour at best, meaning if you remove a user from a group, they could still retain access. Worse still, if you disable their account in AD but keep them in the group, they will still have access until their Kerberos ticket expires. You can manually update group membership on the ASA if this would be an issue for you. I would guess that if it is, the person has been marched out of the building at that point anyway.

VPN logon vs AD logon

Another thing to be aware of is remote VPN access. When you remote VPN on, you get authenticated to the firewall. This could be via local ASA accounts, Radius, TACACS, ACS, LDAP etc. If you use AAA LDAP authentication (using Active Directory in this case), you are not logging on to the domain as you VPN in, you are simply saying ‘here are my AD credentials, please authenticate me on the firewall’. At that point, one of two things happens with the Identity Firewall. If you are using a domain computer to remote on, that machine will automatically try to make contact with a DC. When it finds one (over the VPN), it will log on to the domain, create a security log and the AD agent will let the ASA know. Any rules assigned to that user, that don’t filter on source IP, will now come in to effect. However, if the machine is not joined to the domain, there will be no logon event (the username\password given at connection was only for VPN authentication), and so any user-identity ACLs will not apply.

In the latter case, you may want to create a specific VPN connection profile with specific DHCP pool for example and go back to restricting by IP, only allowing users in a certain AD group for example to be able to connect to that profile. Not ideal, but it would work.

Summary

In summary, I really like this feature. It answers a number of questions specific to the environment I have implemented it in. Things I particularly like are its ease of setup including the troubleshooting commands to aid with this. Learning the syntax of the ACLs is a simple task, as they extend upon existing IP-based ACLs. I also like the fact that there are several ‘under the hood’ settings you can tweak to improve security further still e.g. remove users from the cache if their machine as been idle for a certain amount of time, force LDAP lookups over SSL.

Something that I wasn’t so keen on was the split communication i.e. the ASA getting group membership directly from AD, IP mapping from the agent. I would like to see the agent providing the group membership too. It should also not be too hard to tell the agent which groups to keep a closer eye on (the ones used in ACLs sounds like a good start). This would allow the ACL to be updated much quicker should a group be amended.

I would love to hear what others’ experience of this feature has been. Overall, I am very pleased with it and look forward to upcoming improvements.

Edit: additional points

A few other points I should highlight, if you move to a different subnet the new IP address will be picked up as long as your device authenticates with the domain. I tested this early on by disconnecting my laptop from the wired network with a persistent ping happening in the background. I then VPN’d back to the office via the WiFi network and within a couple of seconds of the connection be made, the ping reestablished.

The second point which will hopefully save somebody a head ache out there; a user testing this functionality was reporting it working as expected until they would open a spreadsheet that would make a data connection to a SQL database in a different subnet. The user had access to this SQL database as allowed by the Identity Firewall ACLs but as soon as the data was refreshed in the spreadsheet, he would get kicked. I could see it happening using the ASA and AD agent troubleshooting commands too. After a little bit of head scratching, we realised that the spreadsheet data connection was using a different set of domain credentials to connect to the database. This triggered a logon event for the user’s IP address but with the user account for the data connection. In turn, the AD Agent updated the cache with the new user and IP combination and denied the traffic based on the ACL. One answer to this would be to add the data connection account to the ACL but in the end, we just connected to the database in the spreadsheet using the user’s own credentials. This highlights the fact that, whilst a user can be affiliated with multiple IP addresses, only one user can be affiliated with any one IP address, which is the key to making this feature secure.

One last thing which is perhaps the most important, considering this is a discussion about a security feature. If your users connect via a shared IP address, you should be aware. For example, if your users log on to a Terminal Server, they will all be coming from the same IP address by default. That means that if a user with elevated permissions logs on to the TS, all other users will effectively gain the same access as permitted via the Identity Firewall. My advice would be to deny access via a shared IP address to any resources that need restricted access.

Till the next time.

Posted on

How to prepare for a Cisco exam

Introduction

Having just passed my 642-902 ROUTE exam, I thought I would write a post to explain how I set out to walk out with a smile on my face and not egg. I’m not going to discuss the details of the exam itself for obvious reasons but thought I would blog about the training path I took and some general points of exam taking. As I often get asked how to prepare for a Cisco exam, this post will hopefully be useful for a wide audience.

For those that haven’t read my first couple of posts (and why is that??), I passed my CCNA via the ICND1 and ICND2 route back in early 2009. At that time I was a Microsoft systems engineer but saw the light and when I had the chance to become a networking engineer last year, I sat the CCNA exam to renew the certification. I moved in to the new role officially in November 2011 but had already begun to study towards the 642-813 Switch exam, which I passed on November 25th. It’s worth noting that I effectively scraped through this exam as far as I was concerned and I put that down to my preparation, which was not as complete as it should have been.

Videos

I used the CBTNuggets video series but, after the CCNA series by Jeremy Cioara which was simply excellent, I found the Switch series to be a disappointment and it included many references to the old BCMSN exam, which told me that the content wasn’t bang up to date. OK, fair enough, the topics might not have changed a whole lot but if you are going to resell something an as upgrade, please don’t just stick a different badge on it! I ended up losing interest and watched the INE video series instead.

Reading

I also used the official certification guide from Cisco Press but here lay another issue, this time with myself. As part of the move to networking, I felt a certain pressure to get up to speed as quickly as possible. This wasn’t a real pressure, it was something that I imagined but it meant instead of reading the book from cover to cover as I should have done, I skimmed some chapters and skipped a couple of topics. This is exactly why my score was not up to my usual self-imposed standards. It was also what made me determind to put time pressures to one side and make sure that I understood all the material before going in to the next exam.

For the 642-902 exam, I basically used the materials\methods below and I’ll briefly go in to a little more detail on how I blended all these together to give myself the best chance of passing the exam:

  1. Cisco Press exam guide book
  2. CBTNuggets video series
  3. Cisco Live
  4. Labs
  5. INE R&S workbooks
  6. INE video series
  7. Work experience
  8. Boson exams

Firstly I broke the book down in to 6 sections; EIGRP, OSPF, BGP, Redistribution, IPv6, WAN\Branch offices. Straight away, it ceased to be a 700 page book and became 6 individual topics that weren’t so daunting anymore. I gave myself deadlines to read each topic and made sure I hit them by increasing the page count per day if I skipped any days, which I made sure was a rare event. I read them pretty much in the order above, except for BGP which I left until last.

As I was covering each topic in the book, I would watch the corresponding CBTNuggets videos. The Route series is a vast improvement over the Switch videos. Jeremy uses GNS3 labs to cover the topics and the topology files he uses are available to subscribers on their website so you can ‘play along’ with Jezzer.

Filling in the gaps

I was lucky enough to get along to Cisco Live in London this year and found it to be very inspirational. The technical sessions were top notch and gave me a head start on a number of ROUTE related topics, such as IPv6 which I had previously not really ‘got’, but a 4 hour hands on lab gave me a massive boost, as did some of the related breakout sessions. The fact that, up until then I had pencilled in a date of June for sitting the exam but brought it back two months speaks volumes about the effect it had on my motivation.

With the book finished and the CBTNuggets videos wrapped up three weeks before the exam date, I knuckled down to some labbing. Again, I broke it down to the six topics and focused on these, even more so on the routing protocols and redistribution and used the INE CCIE Routing and Switching materials to give me a real sense that I was going beyond the requirements for the Route exam. I should point out that I am lucky in regard to the training materials I have access to. My company have a dedicated training budget and were happy to pay for all the books, subscriptions and the Cisco Live ticket, in addition to the exam cost.

As a form of ‘detail revision’, I also decided to go through the 19 hours or so of INE videos in the Route series and was watching a couple of videos each day whilst labbing. I found that this really helped it all sink in and gel. Whilst I could have rewatched the CBTNuggets videos, I think another trainer’s perspective is quite often useful and so it proved.

On the job training

The day to day tasks that I do as a network engineer really helped. For example, I work for an ISP that runs BGP and OSPF in our core and using this live environment to see how the various topics knit together is priceless. It’s also given me a few tasks to keep me busy over the next few weeks and months as I’ve noticed where improvements and tweaks could be made and let’s not forget the IPv6 implementation plan!

Practice exams

Finally, the Boson exams gave me great insight in to which areas I was still weak in. After completing an exam, I would go back to the book and read up on the weak points. The day before the exam, I did 108 questions and got 907 which made me feel more confident.

The methods used between the Switch and Route exams were worlds apart and I know which one I preferred. Putting the effort in really makes the difference and every hour you use for studying now will save you countless hours of head scratching at a later date. With one more exam to go for the CCNP, I am getting a feeling of anticipation but fully intend to apply the same regime to studying, despite the fact I hear from many sources that if you have been working in IT for any number of years, you should be able to pass the TSHOOT exam with minimal study. That doesn’t tempt me in the slightest. I want to make sure my CCNP is as solid as it can be. After all, this is the foundation for my entire networking career from now on. I have the desire to go on to the CCIE at some point, perhaps with some design certs along the way, maybe the CCIP\CCNP SP and some specialisations such as Wireless and Security.

One thing I have realised is that there is no rush for these career making skills and that is why I’ll be going back to the Switch topics and applying the same process again to them that got me here with the Route. In fact, INE have a deep dive series specifically on Layer 2 that sounds like just the ticket. On a final note, this was my 5th Cisco exam and, despite me loving the CCNA exams the first time around, was my favourite so far. Things are really starting to gel now and I have to say I have a strange attraction to BGP that I will be pursuing further…

The real exams

This last section (which I originally missed out due to being giddy about going on holiday the day after my exam!) is about the exam itself. Oh yeah…that bit!! As you progress through your studies, you should start getting a better idea of when you will be ready to sit the exam. My suggestion is to book the exam about 4-6 weeks before the date itself. This will hopefully give you a last burst of energy in the final stage – there is nothing like a target to aim for. I always try to book the exam for about one week (and usually no more than two) after finishing the books, videos and labs, giving me that 1-2 weeks for exams and final reading up.

What are my thoughts on postponing an exam? It all depends on whether you mind about having to sit some exams more than once before you nail it. If you do care (and I’ll admit I have this obsession about NOT failing an IT exam based on a failed university chapter earlier in my life), then feel free to push it back a week or more, but don’t do this more than once. If you are not bothered about a failure here and there, then stick to the original date. Either way, I think you should try to be as ready as possible, although I can see the benefits of sitting an exam when you might not be 100% ready (examples include your 1st exam when you don’t know what to expect, a renewal that has crept up on you and you must take it before a certain date).

For the exam day itself, I can offer some basic tips. Make sure you have your ID with you, book the exam for a time that suits you (e.g. if you usually feel sleepy mid afternoon, book a morning exam), make sure you know where the test centre is, where parking is etc. Leave plenty of time to get there – most centres I’ve been to have let me start early anyway. If yours doesn’t, you will at least have time to settle your nerves and maybe have a cup of tea\water\etc., (or nip to the loo…).

The exam itself should be an exercise in self-control. Make sure you read the pre-exam blurb carefully, especially if you are fairly new to exam taking. Ask for the paper and pen that you are usually allowed to take in so you can make notes. Before the exam starts proper, you should be told how long you have and how many questions are waiting for you. This is important information. Use it to determine roughly how long you have on each question. I say roughly as some questions will take seconds to answer but a simulation could take 20 minutes or more. The point is, if you have two hours to do 50 questions and you find yourself on question 10 with 30 minutes left, you’ve managed your time poorly. Rather than doing the maths on a question by question basis, I would check my time every 30 mins (in the example above) and try to ensure I was 25% further in. With that in mind, don’t be afraid to drop a question if you’ve hit a road block. In my last exam (ROUTE), I got stuck on a simulation at question 40 ish with 30 minutes left. 8 minutes later, I had done about half of the required work but was going around in circles. What did I do? I set myself a target of dumping the question with no less than 15 minutes left. At that time, I had progressed further but still not nailed it but continued to the next question regardless. As I clicked ‘END’ on my last question, I had exactly 28 seconds left on the clock. My hard decision had allowed me a chance to answer all the remaining questions.

And finally

My last bit of exam advice would be to make yourself as comfortable as you can. For me, that usually means being in the room alone as I like to talk to myself out loud, stand up and stretch my legs from time to time and even sing\hum to myself to chill out! Find what works for you, that doesn’t upset other exam takers.

Till the next time.

Posted on

Cisco Live London 2012 Day 1

First of all, WOW. The vibe at Cisco Live London 2012 is quite amazing. A two minute walk from the Princes Regent DLR stop takes you in to the Excel exhibition centre and the registration process was over in another two minutes and the first souvenir of the week, the obligatory CL backpack, was in hand.

Need to look for a new laptop to fit…
Vendor stalls at the back, Meet the Engineer pods in white

The technical seminar I had signed up for was the ‘catchy’ sounding ‘TECVIR-2002 Enabling the Cloud: Data Center Virtualization – Applications, Compute, Networking and Best Practices’.

The three presenters over the day, which stretched to nine hours, were Carlos Pereira, Santiago Freitas and Ray O’Hanlon. Each had their own style but all were very capable speakers\presenters which kept me engaged for the individual parts which ran up to two hours each. Carlos in particular was a natural and the demonstrations given by Santiago were nothing short of breathtaking.

From the left: Santiago Freitas, Carlos Pereira and Ray O’Hanlon

I did think if nine hours was enough to cover the broad range of topics in any real depth but these guys have done this before and the fluff was kept to a minimum, at least for the first half of the day. Any attempt for me to judge the quality in the afternoon would be futile as I was just trying to understand as much as I could, despite the fact I have the slides to refer back to.

Fabricpath, UCS, OTV, LISP, FCoE, VXLAN all got good representation and of course how they relate to ‘the cloud’. I am thoroughly relieved to know that my idea of what cloud is matched fairly well to Cisco’s.  Note that this post is a general overview of the day. If you want to learn about the specifcs of these technologies, there are already plenty of online resources which do a better job than I could at this stage…my head is still, at 22:30 filing whatever it can remember away. Where it was evident that the topics could have been turned up further on the nerd meter to 12, references were made to the specific technical sessions later in the week with a suggestion to attend. Despite having swapped my schedule about several times in the preceding weeks, I think tonight will see yet another juggle!

What I liked today was that nobody’s knowledge level was taken for granted. The presenters were very good at sensing the tone when something being discussed needed more depth…probably the furrowed brows around the room. It was also amusing that some people were using today as a ‘how do I fix this issue in my production network’  session.

Matt’s takeaways

Firstly, I still struggle to see what questions a lot of the new technologies are trying to answer. For example, take OTV, please (OK, old joke). After discussing the innards of this technology, a quick poll around the room to count the number of people who were extending their layer 2 domain across physical sites caused one slightly shaky hand to raise. And it seemed that nobody was going to return to the office next week to implement it.

Secondly, as Bob Dylan said, the times are a changin’. Networking is undergoing a huge metamorphosis, unlike anything I’ve seen in my years in IT. Love it or loath it, cloud is here to stay and it’s going to take a whole new skillset just to understand it, let alone plan, design, implement and operate. The current standard of logging on to 50 TOR switches to configure individually could very well be coming to an end as the control plane is centralised. Add a super smart management platform on top and productivity has the potential to go through the roof. That’s once the questions are properly defined and the right answers agreed upon. That’s not even talking about the questions that are only relevant to you.

Finally, Cisco Intelligent Automation for Cloud (CIAC) looks like it has the potential to put a few people out of work, to say the least. The demonstration of LISP and OTV working together was very impressive, with a VMotion between data centres causing only a single ping packet to drop but what really stood out for me was the self-service portal demonstration which showed a brand new ESX host being deployed as production ready in less than 30 minutes with just a few clicks. In addition, a VM was deployed to another host with correct network settings (both at the VM and network ‘pod’ level) and security settings applied. It looked like a lot of work to set up, but a dream to run.

I’m goosed and have another 3.5 days to get through. Luckily, the rest of the week’s sessions are shorter. Here’s to learning new things.

Till the next time.

Posted on

Welcome to my blog

Welcome

Welcome and thanks for at least coming this far! I’ve considered running a blog since the word was invented. I’ve had numerous sites over the years but they all went through a dozen changes and not one involved interesting content to be perfectly honest. I’ve been holding off on getting the ball rolling but with my first visit to Cisco Live coming up in a few weeks, thought that now is as good a time as any.

Initially I looked at Blogspot, liked the look of a couple of blogs and thought I’d write a small number of hopefully useful posts, outlining my rise in the world of the network engineer, in particular working with Cisco kit. But two posts in, I thought to myself, why not get the domain name I’ve always wanted and host the blog there instead, which is where we are today.

To give a bit of background as to who I am and where I’ve been, I’ve worked in IT full time since 2002 as a Microsoft engineer, attaining an MCSE 2003:Security, MCITP:Server and Enterprise Administrator and specialising in Exchange 2007\2010 in that time. In 2008, I started studying for the CCNA certification to broaden my horizons and six months later, having taken the ICND1\ICND2 path, was the proud owner of a CCENT and CCNA. I carried on specialising in Microsoft technologies, in particular Exchange and put my CCNA skills to use with basic configuration\troubleshooting on our internal network and on some of our customer’s infrastructures.

A few months ago, I was aware that my CCNA was going to expire (Feb 2012) and it was at that point that I was in the fortunate position of suggesting to my line manager a move to being a full time network engineer, which both he and the company supported…result! Within six weeks, I’d resat my CCNA as I wanted to reaffirm my foundational skills before moving on to the next step, the CCNP. I’m originally from Manchester but with family ties in Scotland. For the last four years I’ve worked for an ISP\hosting company in the North East. The initial aim of this blog was to document my journey through the valley of Cisco certification, but I soon realised that I would be restricting my content. So in short, this will be a technology blog with a heavy emphasis on networking.

Although my plans may change in terms of the order of things, I intend on gaining my CCNP in the next 9 months (have already passed my SWITCH exam), spending the following 12-18 months looking to gain some design certs (CCDA\CCDP), perhaps CCNA Security or Wireless or perhaps even a currently job relevant CCIP. No more than three years from now, I hope to be in a ‘comfortable’ position to take on the CCIE R&S written exam and lab.

If somebody ends up finding it useful, then all the better. In fact, if somebody ends up finding it at all, I’ll be happy. As a final note, please feel free to contact me at (vegaskid at vegaskid dot net) if you have any suggestions or questions and do make yourself at home. Till the next time…

😉