Wednesday, November 2, 2016

QoS troubleshooting commands for 4500E/3650 networks

4500E tips from StackExchange

show platform hardware interface ten1/1/31 statistics
that should show you input bytes by CoS.
show platform hardware qos interface foo X/Y
shows queue lengths and flow counts
show interface foo X/Y counter detail
shows interface egress packets by queue, queue drops and DBL drops.
show policy-map interface foo X/Y
mostly useful if you are doing custom policy maps
show interface foo X/Y capabilities
if you want to know what the number and type of queues are available on a particular interface
show qos maps
to show the DSCP-CoS mappings

3650 tips

TBD

Monday, October 17, 2016

Handy Palo Alto Networks SNMP traps

For those of you running Palo Alto Networks devices in your environment, here are a few snmp traps I've found useful.

panROUTINGRoutedBGPPeerLeftEstablishedTrap - sent when a BGP peer drops off and leaves an Established state

panROUTINGRoutedBGPPeerEnterEstablishedTrap - sent when a BGP peer re-enters an Established state 

Wednesday, October 12, 2016

Handy Regex search strings

This post will be updated occasionally to include strings I've found to be useful in day to day work.  A great way to test these is to simply go to regexr.com, paste in the body of text/strings you want to match against, and verify with your own regex strings whether they match or not.


  • Negative lookahead for syslog string alerting.  Used primarily on Solarwinds syslog viewer, this should be placed in the message type field to discriminate against anything that includes the string "PLATFORM" in a syslog message.  All other messages will apply as normal.

    ^(?!.*PLATFORM).*$

  • IPv4 address matching
           [0-9]+(?:\.[0-9]+){3}

Monday, July 11, 2016

EEM redirecting logs to an ftp server

Today I was asked to help with a request from our voice engineer.  She on occasion has to run debugs for specific call functions on our ISR 4K routers, which show up in the buffered log.  We tried sending those log entries to our syslog server, but since the transport protocol was udp/514 (default), some of the messages were coming in out of order.  We tried a second syslog server and got the same result.

Another option was to use tcp instead, so we attempted to get that working.  Unfortunately the syslog server was not listening on that port, so we had to improvise.  We came up with the idea of exporting the full log file to an ftp server every so often, and using EEM to accomplish this.

Our initial script runs as a privilege 15 user once per day, and uses a while loop to run every 60 minutes of 3600 seconds.  It simply dumps the entire log buffer into a text file with a variable which increments once per hour.

event manager session cli username "neteng" privilege 15
event manager applet syslog_redirect
 event timer cron cron-entry "0 0 * * *" maxrun 86400
 action 0.00 cli command "term len 0"
 action 0.01 set i "0"
 action 0.02 while $i le 23
 action 0.03  cli command "show log | redirect ftp://10.10.10.10/logs/$i.txt"
 action 0.04  increment i 1
 action 0.05  wait 3600
 action 0.06 end

This worked pretty well, and we saw the script running once per hour as expected.  Unfortunately the log buffer is set to 70000000 (due to the large volume of debug entries written for each call), so writing the files takes about 4-5 minutes.  The script actually generates the log files at 12:04am, 1:08am, and so on.  At first I was apprehensive about this and wanted the file to correspond to the variable integer, but found that we were looking at the timestamp of the file rather than the filename itself.

Overall, a very basic script that does what we wanted.  However, it sometimes gets stuck we have to ask the question, why the hell are we letting a process run for 24 hours? That eats up cpu/memory cycles.  Here is a better version:

event manager applet syslog_redirect
 event timer cron cron-entry "0 * * * *"
 action 0.0 cli command "show clock"
 action 0.1 string range "$_cli_result" 2 4
 action 0.2 set hour "$_string_result"
 action 0.3 string range "$_cli_result" 6 7
 action 0.4 set mins "$_string_result"
 action 0.5 string range "$_cli_result" 20 22
 action 0.6 set day "$_string_result"
 action 0.7 cli command "show log | redirect ftp://10.13.5.36/vgw/test/$day$hour.$mins.txt"
 action 0.8 cli command "clear log" pattern "confirm"
 action 0.9 cli command "yes"

It will even clear the log after running, making your next log dumps smaller in size.  Depending on your "show clock" output, you may need to adjust the numerical variables on the right, as they are literal positions within the output itself.