Cisco SNMP Monitoring

Cases for use

To store metrics about how your Cisco routers and swtiches are performing, you’ll probably find yourself needing to
enable SNMP. This protocol enables the gathering of statistics through MIBs (Management Information Bases) over the SNMP protocol.
You can then track bandwith utilisation to help plan upgrades, de-commissions, and the like.

Versions

v1 is all but completely unused.
v2c is the most common implementation. We’ll discuss that here and I’ll add a further section regarding v3 and it’s authentication options.

 Configuration

When retrieving statistics from the device, you don’t want interfaces re-numbering and changing otherwise your hard work
will be for nothing when you’re polling interfaces and they change on you after a reload or installation of a network module.

snmp-server ifindex persist

If you’re working in a large enough organisation to be able to personalise the systems so that you have areas of responsibility you can add
basic contact details for the devices.

snmp-server contact ITDept
snmp-server location TheLondonDungeon

Because you likely don’t want any system to be able to contact the device using SNMP, you can determine which systems should be able to poll the SNMP Server by creating an ACL.
Perhaps you have one server that you want to access SNMP on the device and an entire Management subnet.

ip access-list standard 10
 permit host 192.168.1.20
 permit 192.168.4.0 0.0.0.255

Consider the read-only community string you will use for your devices and configure this, along with the access-list and assign it a read-only permission.

snmp-server community <string> ro 10

You can now configure your monitoring device to poll all the routers and swtiches in your network to store statistics.

Configuring Cisco IP SLA and Object Tracking

Cases for use

1. You would like to modify your network routing in response to a change of conditions either in your network or outside your network.
2. You have an interest in determining performance characteristics for latency/bandwidth across your network to provide metrics over time.
3. A combination of both of the above.

Restrictions

Using 3750 Switches as an example, if you’re running IPBase, you’ll only be able to configure IP SLA Responders. Full IP SLA features are available in advanced IOS images.
Check your platform and feature set of IOS for your ability to run IP SLA features.

Use Case 1

You have two egress routes from different ISP’s available from a single Layer 3 Device. you wish to modify the default route from your Layer 3 device depending on
upstream connectivity to an address you define on each link.

Scenario

12.2(55) IPServices images on 3750 MLS Platforms are being used.

Two IP Next hops for routing your data plane traffic are :

Link 1 = 1.1.1.1
Link 2 = 2.2.2.2

The device on the primary ISP’s network you’re tracking to determine routing preference is :

1.1.10.10

Configuration

Create a simple icmp-echo SLA instance number 10 to check availability on your preferred upstream link.

ip sla 10
 icmp-echo 1.1.10.10
 timeout 500
 frequency 3
ip sla schedule 1 life forever start-time now

Create a tracked object number 99 checking your SLA probe 10 for reachability.

track 99 ip sla 10 reachability

Configure a default route to the preferred provider with a tracking object associated with it.

ip route 0.0.0.0 0.0.0.0 1.1.1.1 track 99

Add a new default route with a higher metric than your existing default route (3). This will only be installed into the routing table if the tracked route is removed due to the primary link’s own tracked object being down.

ip route 0.0.0.0 0.0.0.0 2.2.2.2 3

So the result is the static configuration is the secondary route which could be described as a catch all after the dynamics of the primary route fail.
Again to summarise, the primary route and, the SLA probe and tracking is where all the logic and dynamics are configured.

 

Use Case 2

You’re using a First Hop Redundancy Protocol, in this case, HSRP, configured between three Layer 3 devices.
You wish to modify which of the HSRP Devices is forwarding traffic depending on the status of an object you define on each Layer 3 Device.

Scenario

12.2(55) IPServices images on 3750 MLS Platforms are being used.

HSRP Router 1 = VLAN10 10.1.10.2
HSRP Router 2 = VLAN10 10.1.10.3
HSRP Router 3 = VLAN10 10.1.10.4

HSRP Virtual IP = 10.1.10.1

The object’s you’re tracking

HSRP Router 1 – Upstream Device = 8.8.8.8
HSRP Router 2 – Upstream Interface = GigabitEthernet 0/1
HSRP Router 3 – Nothing

Configuration

Create IP SLA and Tracked Objects

Router 1

ip sla 10
 icmp-echo 8.8.8.8
 timeout 500
 frequency 3
ip sla schedule 1 life forever start-time now
track 99 ip sla 10 reachability

Configure HSRP

Router 1

interface vlan 10
standby 10 ip 10.1.10.1
standby 10 preempt
standby 10 priority 110
standby 10 track 99 30

Router 2

interface vlan 10
standby 10 ip 10.1.10.1
standby 10 preempt
standby 10 track GigabitEthernet 0/1 20

Router 3

interface vlan 10
standby 10 ip 10.1.10.1
standby 10 priority 90

This configuration shows that Router 1 is the preferred HSRP gateway with Priority 110, then Router 2, then Router 3 with Priority 90. Default HSRP Priority is 100 (Router 2).

Notice you don’t have to create a tracked object for Router 2 as the HSRP track command can monitor interfaces local to the device as part of the HSRP configuration. 

You also don’t need pre-emption on Router 3 because It’ll never find itself with a higher priority than the other two routers unless they have failed and decremented their priority because of a failure in the objects that they are tracking. 

Fractured 5th Metatarsal experience

Greetings all,

Over Christmas I managed to snap my 5th Metatarsal in a Snowboarding accident. The accident happened just before NYE in Chamonix and resulted in this pretty mess
Image
It matters not how I did it which was far more rubbish than you might imagine, but more to the point how long it’s taken me to recover and I wish to offer up my advice to anyone as unfortunate as myself to experience such a rubbish and debilitating break in their foot.

To give you some perspective about my physical state, I’m a keen gym go-er and keen cyclist. I was training for the Caledonia eTape until the injury and was in Base 2 of my training working up to 12 hours or more on the bike at that point. Being very active and training around 2 hours a day, 6 days a week meant that this injury presented me with some pretty big challenges physically and mentally.

6 days after the fracture I had the foot operated on by Eoin Baldwin who did a fantastic job in re-attaching the splintered bones with two fixing screws. That was the complicated bit out of the way, the rest was down to me and my body’s healing process.

In Jan ’12 it was pretty cold in the UK, no more than normal, but still, cold. After getting up in the morning I had real trouble keeping my foot warm in the bootie cast that I had on. It was far from painful and I stopped taking the codeine based painkillers 1 day after the Op. It simply wasn’t hurting enough for me to want to take them.

Sadly, 2 days after the Op, I came down with the Flu which was kindly given to me by one of my holiday buddies. That was horrid, being club footed and Flu’d up all at the same time was one of the most rotten feelings ever. 4 days later I was starting to feel human (albeit club footed) again.

No-mans land ensued. I was signed off from work but still completely mentally able. 12 days after the Op, I returned to work – a couple of days earlier than recommended, but I was going nuts and had a lot of work on. I struggled after the journey to work to keep the foot warm still. After a 07:30 get up, by 10am my foot would be uncomfortably cold and I’d try my best but would fail in trying to warm the thing up until I got home and had a bath with one foot hanging out the side.

As soon as I was back to work, I was back at the gym too. I’ve read elsewhere that other people have completely avoided all exercise to ‘let the body do the healing process’. I honestly disagree with that. This implies that they think their body can only do one thing at once, or it takes power away from the healing process by your body doing anything else. For example your body has 100% of ‘healing power’ and by doing Strength Training on your remaining functional parts of your body, you are detracting from that 100% ‘healing power’ by diverting some of that healing toward the applied stresses of Strength Training.
Assuming you are eating ‘well’ taking into account you are what you eat and you’re eating ‘enough’ which I think would be easy if you’re doing Strength Training with limitations i.e. your lower body is out of action thanks to an injury like this then for me there is no reason to lay back and sit still for months waiting for a bone to fuse.

I’ve ranted briefly on this as I had ALOT of arguments with people telling me I shouldn’t be going to the gym whilst I had the injury. Notably, none of these people were people who, without being harsh, were people that ate ‘well’ or had a good physique or good relationship with food or exercise so their advice fell on ears which were not tuned in to their concern.

I will stress though that you should not attempt any lower leg exercises. I did.
Leg Extensions using the Leg Extension machine were the only exercise I felt comfortable using both my legs so as not to train only one leg and cause imbalance.
I suggest you avoid these too as although there’s no direct stress on the foot in any shape or form, after my news which I’ll talk about in a moment, I suggest disengaging your temptation to use your lower body at all for the time being.

Some 20+ days after the Op, I visited the Hospital for a follow up x-ray only to have the Surgeon stand in front of the light box and utter ‘Oh’. That short noise was the last thing in the world I wanted to hear.

 

Image

 

That ‘Oh’ was the realisation that somehow I’d managed to pull the repair apart so the two fixating screws were although aligned but holding the bone with quite a degree of separation again.

It was one of the worst things I’ve ever heard and was very very upset and depressed for a couple of days.
Because the up down and side to side alignment was actually okay meaning the functional operation of the foot was looking okay, the decision was to leave it be for a while and see how it was looking in another 2 weeks. Now, just to mention here, there was expected to be some fusing already, which there wasn’t either because I’d split it or simply it hadn’t started.
The problem with the 5th metatarsal is it’s a small bone, it’s also about as far away from your heart as you can get in your body, and is on the outside of your foot with a relatively small amount of blood flow.
These are all facts that worked against me.

This leads to my first piece of advice.
KEEP YOUR FOOT VERY WARM! Never let the foot go cold.
Cold = No blood flow. No blood flow = No healing.
Do whatever you have to do to keep your foot warm at all times. This includes your Thigh and Lower Leg as they’ll be supplying the blood to your foot. Long Johns, super big socks and over sized slippers are all good things. In emergency perhaps some chemically activated hand warmers stuffed into your foot may help. Make this your personal priority.

My second piece of advice
Do nothing that puts your foot under any pressure until your surgeon advises you otherwise
I cannot state how important this is.
Getting up out of bed puts pressure on your trailing foot, be careful! Getting up from being one legged in front of the fridge and unknowingly the supporting (broken) foot that’s out behind you is put under top side pressure to help with the one legged get up.
Falling over, which I did too many times and hated every one.
Not using any lower leg gym equipment, no matter what muscles you think you are aren’t working. It simply isn’t worth the risk or a re-op or a situation like mine with a mis-aligned bone.
I cannot state how difficult it is to exist with only one foot on the ground for months at a time.
I cannot explicitly state which moment cause my repair to separate. There were plenty (too many) moments including the falls which put far too much pressure on my foot and it could have been any of them.

Lastly, Do Exercise!
Do Upper Body strength training. Unless you have one of those upper body cycling things you’re not going to be able to do any cardio, so simply engaging your upper body will stop you going nuts and keep some resemblance of your physique.

After three months I’m out of casts and actively load bearing without crutches now, but I’m awaiting my next x-ray to see the state of the repair and will update this post when I have that aswell as thoughts on muscle loss and diet during that recovery period.
In the meantime. Good luck if you’re going through this too!

Ciao! x

CCIE in 2012?

It’s April 2012. I’ve had my CCIE Lab for almost 2 years now and I’m going to have to do something about it. According to IPExpert there will be a Blueprint change from v4 to v5 around the time the year is out which leaves me 8 months to get my shit together and pass TSHOOT, CCIE Written and Lab before Christmas unless I want to modify the lab which would be a pain in the ass to be honest.

I’ve booked the calendar out, exams have been scheduled and I’m going to give it a shot. TSHOOT booked for late May and CCIE Written booked for late August.

I’ll be sharing my most painful learning experiences with you all as usual. 🙂
Good luck to any of you trying to shoehorn the qualification into your lives too!

Fibre Channel and NAS SMB Storage

I’d like to pen my experiences of storage protocols to hopefully benefit some other soul who’s responsibility is an entire organisations data asset.

 

In the past, I’ve been fortunate enough to be pretty much exclusive to HP Server and Storage Hardware. HP do build fantastic Server Systems. They are very good at the Desktop space too,
There, I said it, I’m an HP fan for the compute environment.

 

I’ve experienced using MSA1000, 2000, EVA 3000, 4000 and 4400 storage systems which are all solid robust units which I would bet any business on. Sadly though they all seem to come with the mystical Fibre Channel baggage. I’m not particularly averse to dealing with a Fibre Channel network as long as it doesn’t have a Storage Virtualisation product which is misconfigured and miscabled in the mix (my recent nightmare) but, that being said, it is still a pain in the ass.
If you were (and you should) to follow the vendor compatibility matrices for the associated HBA firmware, FC Switch OS, and Storage Processor Firmware’s you will likely find yourself in situation more often than you’d like (I’d prefer never) where you have to say “I need to take the whole SAN offline to update this, and because of this and the compatibility required to stay supported, I have to update this, this and this” all these “this” are usually Host HBAs and FC Switch Firmwares.

 

It’s a dark day when it comes, it really is. I don’t care how much process you put in for it, it’s a dread. I’m sure the Storage Consultants and Field Engineers out there are of a different opinion, but when a multi faceted engineer that’s part of a small team like myself is presented with that situation. I stand my ground in saying it’s a bad day at the office.

 

Don’t get me wrong. I didn’t ever experience data loss or extended outage in FC world, but the knowledge that you’re dealing with an entire organisations data and the paranoia of knowing hundreds of people can’t do their job should something go awry is less than appealing.
You can shout ‘you should have it backed up’ all you like. The prospect of restoring all your backups to another storage repository – because of course you have those lying around don’t you – is equally as appealing.

 

Fast forward to, for me, today, and NFS. NFS is an old school protocol. Been around for ages, now I believe in v4 should you choose to use it, but widely in use as v3.

NFS as you can imagine by nature is used across Ethernet. Now up till fairly recently that meant 100Mb/1Gb connectivity options. Today 10Gb CNAs are gaining uptake, though not for me. So the connectivity options now exceed the available Fibre Channel speeds that I know of (2, 4 and 8Gb). That’s of course if you require a large amount of throughput from your Hosts to your storage.

 

There’s the key right there. Do yourself a favour. Get some kind of performance metrics out of your Fibre Channel infrastructure. Do it now, start Googling how to get numbers out of your Brocade, HP or Cisco or whatever flavour of FC switching you own. You owe it to yourself to find this out.

 

If you already know your throughput requirements require 10Gb CNAs or FC 4, 8Gb HBAs you probably know that as you are running Enterprise Class systems, you have responsibility for looking after an infrastructure that likely doesn’t fit into the normal SMB market. Or, you have a special use case that can’t be canned with those two statements and the remainder of my musings won’t have as great an effect on you.

For the rest of us, running business around the world that aren’t supporting data warehouses and need to run a modest budget and keep some head space at the same time, I think you’d struggle to find your performance reports show that your doing anything that couldn’t be satisfied by Gigabit Ethernet.

 

The key to most storage systems performing is, in a large part, determined by how many IOPS you require and how many IOPS the array can deliver. You can right size your array in terms of capacity but hugely undersell yourself in terms of performance. I would never buy an array based primarily on capacity any more. Of course in the past I did before I knew better, luckily I didn’t get stung in any major ways. It would definitely have been better if I had right sized using IOPS, but that was then and this is now.

These days in the world of VMware and some other peoples take on Virtualisation, you need to provide storage that can handle A LOT of different machines all accessing the same shared storage.
That includes machines that are barely active to machines that are Tier1 such as Exchange and SQL environments. They key to that is providing enough spindles to satisfy your I/O requirements. Only then, once you’ve satisfied your I/O req’s should you look back at your new array design and say “Is that enough room for my data?”. I’ll bet you it’s plenty once you’ve right sized for your I/O req’s.

 

Back to the Transport. NFS.
I don’t think these days there’s any trouble with MS Supporting their OS’s and applications on VMware. One exception I know off the top of my head is the lack of support for Exchange Mailbox Servers using their datastores over NFS. From my point of view, I’ve ignored this caveat and should my organisation arise at an issue where we need support, we’ve decided we’ll use sVmotion the VM to either some DAS or iSCSI system for the duration of the support case should we get busted.

Other than this potentially important caveat there’s no issue in terms of support for your entire infrastructure running from NFS served datastores.

 

So how does my Datacenter look now I’ve migrated 10TB of production data to NFS using sVmotion over the course of a week? Beautifully simple.

Host – Storage Switch – Storage Processor. Done.

Oh my gosh. Gone are the days of Zoning. Gone are the days of FC domain compatibility matrices. Gone are the days of WWNs, F-Ports, N-Ports, NPIV and all that shizzle.

Hello Ethernet.

 

I think a major component in getting a successful NFS implementation is ensuring your deploying either on autonomous storage switches, emulating the FC Switch topology model, switches that stack with fabric interconnects, and on switches that have enough Fabric bandwidth to accommodate for all the ports your going to provision running at full rate. So I’m saying no over subscription and no unmanaged rubbish devices here essentially. There’s also a lot of VMware design guides to help you along with NFS which I won’t bother you with as this isn’t a ‘how to’, it’s simply a ‘what it’s like now I have’.

 

So if you’re going to be in the market for a new storage system. I encourage you to engage with your Technology Partners and ask them what NFS based storage solutions are available. Ask them for a demo or a Webex to some Lab systems.

 

You will be very glad you did.

 

Ciao for now

ESXi installable on USB

A very simple subject but whilst setting up my home VCP5 setup I had the requirement of getting the hypervisor installed on a USB key without burning it to a CD to use in the destination server – For those with iLO or equivalent you can continue browsing more interesting parts of the Internet – but to achieve this mean feat my instructions are as follows:-

Fire up VMware Workstation
Create a VM with no HDD and no network adapters, but added a USB controller
Start the VM with a long boot delay, enough for me to attach the USB key to the VM once the VM was powered up and attached the ESXi 5 installer ISO to the CD drive of the vm.
Let the VM boot and viola, you’ll be installing to the USB key of the VM from your installation media.
You can then take said key to your machine of choice (remember this isn’t supported in production by VMware unless you’re using a manufacturer provided and somewhat costly ESXi USB key) and fire up your new ESXi host from the key.

Sorted me out anyways!
Ciao for now
P x

All.I.Can released in one month

A great looking new feature called All.I.Can is released on Oct16th in the UK and I can’t wait to see it.
It’s showing at the Prince Charles – Leicester Square so it’s a cool screening venue. The film itself looks to me a cross between Steep and Home, and I love both those films so I hope I’m in for a treat!

Ciao!

VMware 4.x NICs showing down after upgrade from 3.x

Dang. I spent a couple of days trying to figure out what was going on with an IBM xSeries server after upgrading a working system from 3.x to 4.1U1 , troubleshooting cables, adapters, firmwares only to find that it’s a ‘feature’ in the way 4.x deals with the specific type of Intel Gig NIC that I have installed in my estate which are the 82571EB Dual and Quad port adapters.

This KB article will help set you straight after I was pointed in the right direction by VMware support.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010313

Hope this fixes you up quicker than the couple of days it took me to resign to the support case.