Tufin Firewall Expert Tip #4: Vendor and model-specific tips for optimizing firewall performance

Do your firewalls need a tune-up? In our last column, we looked at general ways that firewall performance can be improved to overcome problems such as high CPU utilization, low throughput, and slow applications.

This time, we are following up with a list of vendor and model-specific tips and best practices that can help you to optimize your firewall infrastructure. Thank you to everybody who responded to the last column and sent in their own tips!

Check Point

1. Use networks instead of address ranges in NAT.
2. Avoid rules with Ident.
3. Replace nested groups by flat groups.
4. Be aware of configurations that SecureXL templates (fastpath) cannot handle, for example, security server, or syndefender.
5. Note that SecureXL templates can be disabled from a certain rule onwards due to certain configurations such as client auth, time objects, etc.
6. Be aware of configurations that SecureXL cannot handle, for example:
   • FloodGate-1 (automatically disables SecureXL)
   • Rules with user authentication
   • Services with a port number range (disables connection-rate acceleration)
   • Time object associated with the rule (disables connection-rate acceleration)
7. Be aware of SmartDefense configurations that may impact performance:
   • Network Security–>Fingerprint scrambling–>ISN spoofing
   • Network Security–>Fingerprint scrambling –>TTL

Cisco all models

1. Debug messages are known to affect performance.

PIX 6.3

1. TCP Intercept is known to impact performance.
2. If you are not using NAT and have no DNAT communications, disable the ILS fixup.

Cisco IOS Firewall

1. Performance may be affected if the value of ‘ip inspect one−minute high’ is far greater than the value in the ’show ip audit stat’ command.

Cisco ASA

1. Verifying TCP checksums may impact firewall performance.
2. Ideal performance is achieved when traffic enters and exits ports on the same adapter or ports on adapters serviced by the same I/O bridge (ASA 5580).

Cisco FWSM

1. Deep packet inspection may cause high CPU (all inspection engines except for SMTP are handled in software).
2. Before release 3.1, non UDP or TCP or ICMP flows are handled on a packet by packet basis. With 3.1 and higher, the FWSM creates flows in NP1 and NP2.
3. Be aware of features that are not offloaded to network processors, they will use the CPU.
4. Built-in ACL optimization algorithm: FWSM Release 4.0 incorporates an algorithm capable of optimizing ACLs by coalescing contiguous subnets referred to in different access-control entries into a single statement and detecting overlaps in port ranges. Note that after the optimization process, the ACL is likely to be different from the original one.

Juniper (ScreenOS)

1. ALG (application layer gateway) is applied globally to all policies by default but may have a major impact on performance. Disabling it on specific policies can make a significant improvement.
2. On high-end firewall platforms, NS-5000, ISG-1000 and ISG-2000, with ScreenOS 6.2 and above, Juniper switched the default rule search algorithm from “hardware” (ASIC) to “software” (CPU). The software search algorithm provides faster policy search time compared to older versions, when the number of “rules” for a pair zone is more than 500 rules, but it could cause high CPU during policy changes.
3. ScreenOS 6.1: using wildcard address/wildcard policy causes a performance penalty.

Fortinet

1. Enable only the required management features you need. If you don’t need SSH or SNMP, don’t enable them.
2. Enable only the required application inspections.
3. Minimize use of alert systems. If you export syslog, you may not need SNMP or email alerts.
4. Establish auto-updates (scheduled update) at a reasonable rate. Every 4 or 5 hours should be ok on most cases.
5. Minimize use of Protection Profiles. If you don’t need a Protection Profile on a firewall rule, don’t put it there.
6. Minimize use of Virtual Domains and avoid them completely on low-end models.
7. Avoid Traffic Shaping if you need maximum performance. By definition, Traffic Shaping slows down traffic.

As usual, I’d love to have your feedback,
Reuven.

Tufin Firewall Expert Tip #3: Best practices for optimizing firewall performance

Is your firewall overloaded? Symptoms include high CPU, low throughput and slow applications.  Before upgrading your hardware, it is worth checking whether the firewall configuration can be optimized.

Optimization techniques can be divided into two groups – general best practices, and vendor-specific, model-specific configurations. This column focuses on best practices. Next time, we will look at vendor-specific tips, so if you have any to share, we would like to hear from you.

Optimizing firewalls for better performance and throughput:

1. Remove bad traffic and clean up the network.  Notify server administrators about servers hitting the firewall directly with outbound denied DNS/NTP/SMTP/HTTP(S) requests as well as dropped/rejected internal devices. The administrators should then reconfigure the servers not to send this type of unauthorized outbound traffic, thereby taking load off the firewall.
2. Filtering unwanted traffic can be spread among firewalls and routers to balance the performance and effectiveness of the security policy.
   1. Identify the top inbound dropped requests that are candidates to move upstream to the router as ACL filters. This can be time consuming, but it is a good method for moving blocks upstream to the router and saving firewall CPU and memory.
   2. If you have an internal choke router inside your firewall, also consider moving common outbound traffic blocks to your choke routers, freeing more processing on your firewall.
3. Remove unused rules and objects from the rule bases.
4. Reduce rule base complexity – rule overlapping should be minimized.
5. Create a rule to handle broadcast traffic (bootp, NBT, etc.) with no logging.
6. Place the heavily used rules near the top of the rule base. Note that some firewalls (such as Cisco Pix, ASA version 7.0 and above, FWSM 4.0 and certain Juniper Networks models) don’t depend on rule order for performance since they use optimized algorithms to match packets.
7. Avoid DNS objects requiring DNS lookup on all traffic.
8. Your firewall interfaces should match your switch and/or router interfaces.  If your router is half duplex your firewall should be half duplex.  If your switch is 100 Mbit your firewall interface should be hard-set to match your switch; both should most likely be hard-set to 100 Mbit full duplex. Your switch and firewall should both report the same speed and duplex mode. If your switch is gigabit, your switch and firewall should both be set to auto-negotiate both speed and duplex.  If your gigabit interfaces do not match between your firewall and switch, you should try replacing the cables and patch panel ports. Gigabit interfaces that are not linking at 1000 Mbit full duplex are almost always a sign of other issues.
9. Separate firewalls from VPNs to offload VPN traffic and processing.
10. Offload UTM features from the firewall: AV, AntiSpam, IPS, URL scanning.
11. Upgrade to the latest software version. As a rule of thumb, newer versions contain performance enhancements but also add new capabilities, so a performance gain is not guaranteed.

If you use Tufin SecureTrack, you can automate a number of these tasks.

Here are a few ways that SecureTrack can help:

1. Identify unused rules and objects with the Rule and Object Usage Report, and consider removing them. The longer the reporting period, the more reliable the rule usage status will be. Remember that certain rules, like the ones allowing disaster recovery services, are only used rarely. You can also identify and cleanup unused group members.
2. Analyze rule shadowing with Policy Analysis. Run Policy Analysis with “Any;Any;Any;Any” to identify completely shadowed rules. These rules are redundant and should be deleted. You can re-validate the redundancy with an unused rules report.
3. Identify the most-used rules with the Rule and Object Usage Report and move them up in the rule base hierarchy. To find the top-most location for placing a rule without affecting connectivity, run an “Any;Any;Any;Any” policy analysis query, then, for each most-used rule:
   1. If it is not shadowed, move it to any higher location.
   2. If it is shadowed, find the lowest-ranked shadowing rule with a contradictory action and place the most-used rule below that one.
4. Other things to keep in mind when re-ordering rules:
   1. You’ll probably want to preserve the rule base structure, for example, rule grouping by service or application, source or destination, projects etc.
   2. Be careful with policies containing rules with special actions such as authentication or encryption – shadowing becomes more tricky in this case.
5. You may also use the Best Practices Rule Order Optimization test to quickly identify candidates for relocation.
6. Use the Automatic Policy Generator (APG) to identify and remove unwanted traffic from the firewall. Read more about APG here.
7. Use the “Software Version Compliance Report” to control your firewall software versions.

Last but not least, remember that optimization can have a price, too – beyond the time you’ve invested. If you are not careful, you can wind up with a rule base which is too hard to maintain.  If you have the budget, there are times when upgrading the hardware is the easiest alternative.

In our next column, we’ll focus on ways to optimize specific models of firewalls from the different vendors. If you have any tips that you would like to share, please contact me at rh@tufin.com.

Tufin Firewall Expert Tip #2: Analyzing Network Connectivity Problems

Network connectivity problems are some of the most common – and aggravating – for business users. With distributed systems, as soon as an application does not behave as expected, the firewall is suspect. There are many other possible points of failure – the client application, the user’s PC, intermediate switches, routers, filters, load balancers and the application itself. But, because of its nature (secretive and designed to keep people out) the firewall is a prime suspect. As a firewall administrator, you are guilty until proven innocent.

So how can you quickly determine if the problem is due to the firewall or not?

One approach is to analyze the firewall traffic logs. Contact the user, obtain his IP address and ask him to access the application again. Ideally, this should trigger the connection in question. Then you can review the firewall traffic logs and locate the dropped or accepted packets. How easy this is depends on the tools – unless you have a smart log browser, you may have to work with syslogs.  Normally there will be a lot of logs so a filter on the source IP and, if possible, on the destination IP or port will make things easier.

Unfortunately, in many cases, you will find nothing. One possible reason is that the rule that allows or blocks the relevant traffic is not configured to generate logs. Another possibility is that you are not looking at the right firewall or are simply missing the relevant logs.

Another method is to analyze the firewall rule base. In many cases this is not feasible due to the size and complexity of the network and firewall policies. But if the network is relatively small and you know the rule base very well, you may be able to narrow the problem down to a specific rule or to a recent change that might have affected the application flow.

If you have Tufin SecureTrack, you can use the Policy Analysis tool to query the rule base. Get the user’s IP address, the IP address of the application and the service or port, if possible. Log into SecureTrack and create a policy analysis query with these inputs. You can run the query on all firewalls or, if you are sure which ones are relevant, on a subset. For the report, you can choose to show all traffic or only dropped or accepted traffic.  SecureTrack does not send any packets over the network. It analyzes its own copy of the rule base, which is always up to date from continuous monitoring. Since SecureTrack does not depend on traffic logs, it doesn’t matter whether log data is missing or unavailable.

Policy Analysis will quickly determine whether the firewalls are allowing the user’s traffic or not. If it turns out that the firewall is, in fact, blocking traffic, Policy Analysis will point you to the rule that’s causing the problem as well as when it was last changed, and by whom.

Tufin Firewall Expert Tip #1: Relocating a Server

An IT group needs to move a server and you need to update the firewall policy. The question is this: Which rules need to be changed? What is the fastest and safest way to get the job done?

You have two choices on where to start: the firewall rule base or the traffic logs. The rule base has one obvious advantage – it’s smaller. But it may be difficult to figure out which rules are relevant. Often the server will not be referenced explicitly, so you need to manually check every rule. This could be a tricky task, particularly if your rule base has been growing for a few years and has had multiple administrators. Rule shadowing complicates things even more.

The other alternative is reviewing firewall traffic logs. By identifying the traffic to and from the server over a reasonable period of time, you can create a new rule base. You do not need to worry about shadowing and potentially, this method will enable you to build a very accurate policy that does not include a lot of overly permissive rules. The downside is the large number of logs that must be analyzed. How far do you need to go back to make sure you’ve covered all legitimate use? One month? Three months? A year? What about your disaster recovery hot site traffic? Remember, you won’t have any logs for that unless you have tested your failover.

Whichever way you go, here are some things to look out for:

• Creating over-permissive rules. Though they provide a quick fix, hackers may use them as attack vectors later on.
• Access that is currently allowed and not really required.
• Implicit connectivity that seems superfluous but could actually be essential for business continuity.

If you have Tufin SecureTrack™, try using the Automatic Policy Generator (APG) to prepare for server relocation. Run APG on firewall logs from or to the server going back a year. It will generate a set of rules that allows the actual server access that took place during that period.

You should review it to make sure there are no rules that stem from malicious traffic, such as a port scan, (even if it runs slowly over several days), a conficker virus or a generic botnet.Change the server’s address to the new one and implement the new rules on relevant firewalls above any potentially blocking rules – you can consult with SecureTrack’s Policy Analysis tool or the Policy Change Advisor in SecureChange™ Workflow. After the relocation is done, you can use the rule and object usage report to clean up the obsolete access rules.

Automatic Policy Generation with Permissive Rule Analysis

One of my favourite activities, as CTO and founder, is meeting our users and talking to them about their needs and wishes in the areas of firewall policy management and beyond. It’s always nice to hear how SecureTrack is helping out and what our users like about it, it’s also useful to hear about things that don’t work and need improvement, but what I’m really after is requirements that lead to innovation.

A couple of years ago I was visiting one of our users, a mobile operator, and we were discussing a requirement they had to remove unused objects from the policy. At the time SecureTrack already provided rule usage analysis and this new requirement gave me the idea of mapping traffic logs onto objects within rules. This solution seemed right; unused objects would appear with zero hits and could then be safely deleted from the rule. The user also mentioned something about analyzing rules that are too wide which I kept in mind but didn’t have the bandwidth to deal with.

Anyway, we started brainstorming the object usage analysis requirement and came upon an interesting scenario:

Source	Destination	Service	Action
A B	X Y	HTTP	Accept

Assuming all objects are used, can this rule be improved?

If you like puzzles stop here and think.

Notes from the Blogosphere: PreachSecurity review of SecureTrack 4.5 review

Last week, Rafal Los posted an in-depth review of SecureTrack 4.5 on his PreachSecurity blog (aka Digital Soapbox).  As a former firewall admin, Raf has felt the pain our products were designed to get rid of.

What’s interesting about his review is that unlike the professional reviewers who wrote the semi-recent 5-star review of SecureTrack in SC Magazine, Raf’s familiarity with the issues that can really ruin a firewall admin’s day,  he is not an easy sell.

Not to say that SC Mag review wasn’t a huge achievement for us – getting a five star review from a leading security magazine in a MAJOR  high.   What’s so  appealing about Raf’s review (outside of his  high marks for usability, utility, and ‘worth your time’) is the street cred he brings to the table.

Raf  states: “I recently worked for a company that had 250+ firewalls in a single business unit, mostly Check Point and Cisco Pix… managed either out of one of a handful of Provider-1 consoles, or remotely via telnet (then SSH, as available). That was one of those nightmares I never thought I’d wake up from… so you can probably see why I’m so interested, at least personally, in tools that aim to simplify the life of the firewall admins out there.”

That sounds like a chunk of our customer base, for sure.

So, when he says things like  “Tufin were kind enough to give me a full demo and a demo image to play with and I have to say I’ve been genuinely impressed…”  and  “…where was this [SecureTrack] when I was jockeying firewalls back in ‘98/’99.?” and  especially ” This is a really cool product set with one of the most visible ROI models I have ever seen…”

Well… it’s hard not to feel good a about that.

But the ultimate compliment he gave us  (articulated in true firewall admin style) “… I think this is one of those products you shouldn’t go too long without… like socks or deodorant…”

Amen!

Click here to see this the full review, and while you’re at it, check out other posts on Raf’s blog, they are definitely worth a read.

John Pescatore on firewall complexity

Gartner’s John Pescatore wrote an interesting blog post today on firewall rule base complexity - “A Storm in Any Port” (firewall buffs will appreciate the pun – “Any Port”…)

We’ve been preaching about the complexity and inherent risks of large firewall rule bases for years, and it’s always great to receive validation from one of the most influential people in IT security.

Here are a couple of key quotes, in my opinion:

“… it is pretty rare to find an enterprise firewall policy that anyone is really sure about exactly what policy the rule set actually implements. Most firewall rule sets have mutated through incremental adds/drops/changes over the years and have turned into gargantuan linear lists that now have a life of their own.”

“Often you’ll find that easily 30% of the exceptions are no longer needed.”

The only thing missing from this great post is the solution – well, guess what:  we have it… Using Tufin SecureTrack, you can manage firewall policies proactively, and clean up unused firewall rules and objects. In some cases, we’ve seen customers with close to 50% of their firewall rule completely unused over a long period of time, out of hundreds of rules.

It’s time for some spring cleaning, folks…

Outsourcing Firewalls to MSSP

To anyone that’s been around the security industry for a while, the outsourcing of firewall management to Managed Security Service Providers (MSSP) is a fact life. The main players in this industry, estimated by Forrester at $3 billion in 2008, are well known names: AT&T, Verizon Business, EDS, Verisign (now SecureWorks), and several others.

Like many other IT outsourcing trends, the underlying reasons are cost savings and operational efficiency, with clear economic advantages for the MSSP vs. internal IT resources. The top MSSP’s focus on security operations management, retain highly-skilled personnel, maintain best practices across their customer base, and manage a 24×7 NOC / SOC. In addition, due to the sheer size of their infrastructure (hundreds to thousands of firewalls), MSSP’s receive major discounts from manufacturers for the underlying firewalls / routers / switches / servers / etc. All of these factors enable MSSP’s to dramatically reduce operational cost, and pass on the cost savings to the customer.

There are various challenges in working with MSSP’s, mostly related to giving up control over a complex and sensitive IT process, with a fine line drawn between the internal IT team and its interface with the MSSP. Organizations that have a corporate security policy and are governed by regulations mandating operational control, need to be able to retain some control over security configuration changes, and to be able to effectively manage the potential risk inherent in the outsourcing of security operations.

Here’s the first article in a series on the issues around outsourcing Firewalls to MSSP’s.

Welcome

Hi everyone,

Welcome to Tufin’s blog!

This is the corporate blog for Tufin Technologies, which leads the Firewall Operations Management market.

Several people are going to write, mostly about security management, Tufin, firewalls and other topics.

I hope you find it interesting, and keep coming back!

Take care,

Ruvi