BadInputFilter revisited

Designed for Tomcat 6, BadInputFilter provides a frontline of defense against common web application security exploits such as SQL injection attacks and cross-site scripting. Unfortunately, BadInputFilter breaks silently in later implementations of Tomcat. In this article, learn how to restore the security benefits of BadInputFilter for all versions of Tomcat, and even for use in other Servlet/JSP containers.

As a software developer and educator, I read a lot of technical books. Some are more useful than others, but one that I have found to be extremely useful for several years is Tomcat: The Definitive Guide (Second Edition) by Jason Brittain and Ian F. Darwin (see Resources). Even though this book was written for Tomcat 6, most of it is still applicable to Tomcat 7, currently the latest stable release. If you use Apache Tomcat as much as I do, this book is an invaluable reference. (Note to Jason and Ian: With Apache Tomcat 8 on the horizon, it’s time to consider writing a third edition!)

Brittain & Darwin provide a thorough introduction to configuring and running Apache Tomcat in production environments. The book also provides source code for a couple of useful classes named BadInputFilter and BadInputValve, which can be used to filter out potentially dangerous requests. The problem is that these two classes rely on knowledge that is specific to Tomcat 6. In the more recent versions of Tomcat, BadInputValve no longer compiles, and a part of the functionality provided by BadInputFilter no longer works. Both the original version of BadInputFilter and an updated version available on SourceForge have this problem (see Resources). Moreover, the failure of BadInputFilter is silent, in the sense that there is no indication that it no longer does part of its job.

In this article I propose an update and correction to BadInputFilter. To limit the scope of the article, I will focus on the original implementation of BadInputFilter from the Brittain & Darwin book. You could use a similar approach for the updated implementation with a few changes that are fairly straightforward. In fact, you’ll find two versions of my updated BadInputFilter in the source code that comes with this article. In the package com.softmoore.filter, BadInputFilter corresponds to the original implementation and BadInputFilter2 corresponds to the updated implementation currently hosted on SourceForge.

If you are reading this article on JavaWorld, I assume that you have programmed in Java and have some familiarity with using Java-based technology for web applications such as servlets, JSP, and so on. I also assume that you know the role that Tomcat plays in these kinds of applications. I will provide a brief introduction to filters, in order to give context for the remainder of the article.

Filters and valves

So what is a filter? As described in the Javadoc for interface Filter (see Resources), a filter is an object that does filtering tasks either on the request to a resource (e.g., a servlet or an .html file), or on the response from a resource, or both. More than one filter can be applied in a chain to a resource or a collection of resources, which is an excellent example of the Chain of Responsibility pattern. Figure 1 illustrates the use of filters in a web application.

Examples of uses for filters include:

1. Authentication
2. Logging and auditing
3. Data compression
4. Encryption

One of the cool things about filters is that they can be added to an application after the fact. For example, suppose that you have an existing web application, and you decide that you want to log some special messages whenever requests are received for certain web resources. You can rewrite the part of the application that serves up these resources, or you can simply write a filter that intercepts requests for the resources, logs the message, and then forwards the request on to be handled in the normal manner. In the second approach, the existing web application remains unmodified except for the web application deployment descriptor, an XML file named web.xml, where you specify the existence of the filter and the resources to which it applies.

Valves are similar to filters except for three very important differences:

1. Valves are specific to Tomcat and, therefore, run only in the Tomcat Servlet/JSP container. Filters, on the other hand, are part of the Java Servlet specification and are designed to be independent of any Servlet/JSP container, so they are portable to other Servlet/JSP containers besides Tomcat. Unfortunately the code for BadInputFilter depends on implementation details in Tomcat 6, which changed with newer versions of Tomcat.
2. If your architecture consists of multiple web applications on the same Tomcat server, valves can be configured in one place to filter requests for all or some of them. Filters need to be configured separately for each web application.
3. Filters are easily configured to run on specific URL patterns. Valves require that you write your own matching code for this purpose.

Using BadInputFilter for web application security

If a web application isn’t designed with security in mind, it is likely vulnerable to external security attacks such a cross-site scripting (XSS), HTML injection, and SQL injection. BadInputFilter was designed to check for potentially dangerous user input and filter out bad requests. Although you probably should take additional security measures, you can think of BadInputFilter as a frontline of defense for some well-known web application security exploits.

Essentially BadInputFilter examines user requests for potential issues. If an issue is found, it performs one of two actions—either (1) it forbids the request or (2) it escapes the bad user input. Performing these actions in a filter allows you to implement the code once and then easily apply it to multiple (or all) resources within the same web application.

Forbidding the user input is accomplished by sending an HTTP response status code of 403 (forbidden) back to the client. For example, if the user input contains certain non-printable control characters, you will likely want to deny the request altogether. The part of BadInputFilter that forbids certain characters within the user input will still work correctly. But the second part, the part that escapes bad user input, will fail.

Where BadInputFilter fails

A number of the web security exploits involve entering characters or phrases that have special meaning within the context of HTML, JavaScript, or SQL. For example, if user input is not properly validated or escaped, it is possible to enter JavaScript that gets executed on the server or that reveals information about the server environment. As an example of what I mean by escaping user input, you might want to look for all occurrences of left- or right-angle brackets (“<” or “>”) and replace them by equivalent HTML/XML entities (“<” or “>” respectively). Thus user input containing <script> (indicating possible JavaScript code) would be rendered harmless in most cases, but would still display properly if echoed back to the user as part of an HTML response page.

In their book, Brittain & Darwin provide a test JSP page (input_test.jsp) that can be used to see how various filtered and unfiltered user input is handled. Configuring BadInputFilter to filter requests to this test page shows that BadInputFilter no longer escapes user input as originally intended. The reason that BadInputFilter fails is that it relies on implementation details from Tomcat 6.

Here is a brief description, from an implementation perspective, of why BadInputFilter fails. Filters have access to user input through an object of type HttpServletRequest. For an HttpServletRequest object, the method getParameterMap() is supposed to return an immutable Map containing parameter names and values. In Tomcat 6, the immutability of this object could be bypassed by using reflection to get access to the setLocked() method and then invoking it with a parameter of false, essentially unlocking the immutable Map. With the unlocked Map, it was possible to change the parameter names and/or values. In later versions of Tomcat, using this approach to unlock and modify request parameters does not work, but the failure is silent. No indication of failure appears anywhere in the system, neither on a webpage nor in a log file. The user’s input is simply passed on unmodified—meaning that it is not escaped.

Modifying BadInputFilter to escape user input

As I’ve described, HttpServletRequest has a method named getParameterMap() that returns an immutable Map, thus mapping parameter names to values. But in order to “escape” user input, you need to be able to modify these parameter names and/or values. The question is how to modify an immutable Map. Obviously you can’t modify the Map directly (or at least you shouldn’t be able to modify the Map), but you can wrap the entire HttpServletRequest in an HttpServletRequestWrapper, which can then be passed along to the destination resource in place of the original request. The “wrapped” request copies and escapes the original parameter names and values, which the destination resource then has access to.

I’ll present the code to correct the existing problems with BadInputFilter in the form of two Java classes. The first class, FilterableRequest, extends HttpServletRequestWrapper in order to provide access and modification (escaping) of the request parameters. The constructor for FilterableRequest makes a local copy of the parameter Map of the HttpServletRequest that it wraps. It then overrides the inherited methods that provide access to the Map so that they use the local copy. Inherited methods still treat the parameter Map as immutable, but FilterableRequest also provides a new method, getModifiableParameterMap(), that allows modification of the parameters. Listing 1 is the source code for FilterableRequest.

Listing 1. FilterableRequest

package com.softmoore.filter;

   import java.util.*;
   import javax.servlet.http.*;

   /**
    * Wraps an <code>HttpServletRequest so that parameters can
    * be modified by a filter.
    *
    * @author John I. Moore, Jr.
    */
   public class FilterableRequest extends HttpServletRequestWrapper
     {
       private Map parameters = null;

       /**
        * Construct a wrapper for the original request.
        *
        * @param request the original HttpServletRequest
        */
       public FilterableRequest(HttpServletRequest request)
         {
           super(request);
           parameters = new TreeMap();
           parameters.putAll(super.getParameterMap());
         }

       @Override
       public String getParameter(final String name)
         {
           String[] values = parameters.get(name);
           return values != null ? values[0] : null;
         }

       @Override
       public Map getParameterMap()
         {
           return Collections.unmodifiableMap(parameters);
         }

       /**
        * Returns a ParameterMap that can be modified.
        */
       protected Map getModifiableParameterMap()
         {
           return parameters;
         }

       @Override
       public Enumeration getParameterNames()
         {
           return Collections.enumeration(parameters.keySet());
         }

       @Override
       public String[] getParameterValues(final String name)
         {
           return parameters.get(name);
         }
     }

Since parts of the implementation for BadInputFilter still work correctly (for example, the initialization code and the part that forbids user input by sending an HTTP response status code of 403), the easiest way to illustrate the correction to Jason Brittain’s version of BadInputFilter is to create a new version that extends the original version and then overrides only those methods that need to change. (This approach is sometimes known as implementation inheritance.)

Source code for the new version of BadInputFilter is shown in Listing 2 below. The essence of the changes can be summarized as follows:

• In method doFilter(), we create a FilterableRequest that wraps the HttpServletRequest parameter and then pass this FilterableRequest to method filterParameters as follows:

  FilterableRequest filterableRequest = new FilterableRequest((HttpServletRequest) request);
  filterParameters(filterableRequest);
  

• In method filterParameters(), we call getModifiableParameterMap() on the filterable request to gain access to the parameters rather than attempting to “unlock” the immutable Map using reflection.

Essentially the rest of the code in these two methods is modified only slightly from the original BadInputFilter implementation. Listing 2 shows the source code for these two methods in my version of BadInputFilter.

Listing 2. Updated BadInputFilter

@Override
   public void doFilter(ServletRequest request, ServletResponse response,
       FilterChain filterChain) throws IOException, ServletException
     {
       // Skip filtering for non-HTTP requests and responses.
       if (!(request instanceof HttpServletRequest)
           || !(response instanceof HttpServletResponse))
         {
           filterChain.doFilter(request, response);
           return;
         }

       // Only let requests through based on the allows and denies.
       if (processAllowsAndDenies(request, response))
         {
           // Filter the input for potentially dangerous JavaScript
           // code so that bad user input is cleaned out of the request
           // by the time Tomcat begins to perform the request.
           FilterableRequest filterableRequest
               = new FilterableRequest((HttpServletRequest) request);
           filterParameters(filterableRequest);

           // Perform the request.
           filterChain.doFilter(filterableRequest, response);
         }
     }

   /**
    * Filters all existing parameters for potentially dangerous content,
    * and escapes any if they are found.
    *
    * @param request The FilterableRequest that contains the parameters.
    */
   public void filterParameters(FilterableRequest request)
     {
       Map<String, String[]> paramMap = request.getModifiableParameterMap();

       // Loop through each of the substitution patterns.
       for (String patternString : parameterEscapes.keySet())
         {
           Pattern pattern = Pattern.compile(patternString);

           // Loop through the list of parameter names.
           for (String name : paramMap.keySet())
             {
               String[] values = request.getParameterValues(name);

               // See if the name contains the pattern.
               boolean nameMatch;
               Matcher matcher = pattern.matcher(name);
               nameMatch = matcher.find();
               if (nameMatch)
                 {
                   // The parameter’s name matched a pattern, so we
                   // fix it by modifying the name, adding the parameter
                   // back as the new name, and removing the old one.
                   String newName = matcher.replaceAll((String) parameterEscapes
                       .get(patternString));
                   paramMap.remove(name);
                   paramMap.put(newName, values);
                   servletContext.log(“Parameter name “ + name
                       + “ matched pattern ”” + patternString
                       + “”.  Remote addr: “
                       + ((HttpServletRequest) request).getRemoteAddr());
                 }
             }

           // Loop through the list of parameter values for each name.
           for (String name : paramMap.keySet())
             {
               String[] values = request.getParameterValues(name);
               // Check the parameter’s values for the pattern.
               if (values != null)
                 {
                   for (int j = 0; j < values.length; j++)
                     {
                       String value = values[j];
                       boolean valueMatch;
                       Matcher matcher = pattern.matcher(value);
                       valueMatch = matcher.find();
                       if (valueMatch)
                         {
                           // The value matched, so we modify the value
                           // and then set it back into the array.
                           String newValue;
                           newValue = matcher.replaceAll((String)
                               parameterEscapes.get(patternString));
                           values[j] = newValue;
                           servletContext.log(“Parameter ”” + name
                               + “”‘s value ”” + value
                               + “” matched pattern ””
                               + patternString + “”.  Remote addr: “
                               + ((HttpServletRequest) request).getRemoteAddr());
                           servletContext.log(“newValue =” + newValue);
                         }
                     }
                 }
             }
         }
     }

Configuring BadInputFilter

The Brittain & Darwin book contains sample configuration settings for BadInputFilter that can be added to the web application deployment descriptor (web.xml). Listing 3 shows the sample configuration code given in the book.

Listing 3. Sample configuration for BadInputFilter

   
   <filter>
     <filter-name>BadInputFilter</filter-name>
     <filter-class>com.oreilly.tomcat.filter.BadInputFilter</filter-class>
     <init-param>
       <param-name>deny</param-name>
       <param-value>x00,x04,x08,x0a,x0d</param-value>
     </init-param>
     <init-param>
       <param-name>escapeQuotes</param-name>
       <param-value>true</param-value>
     </init-param>
     <init-param>
       <param-name>escapeAngleBrackets</param-name>
       <param-value>true</param-value>
     </init-param>
     <init-param>
       <param-name>escapeJavaScript</param-name>
       <param-value>true</param-value>
     </init-param>
   </filter>
   <filter-mapping>
     <filter-name>BadInputFilter</filter-name>
     <url-pattern>/input_test.jsp</url-pattern>
   </filter-mapping>
   

The minimum change needed to use my version of BadInputFilter is to replace com.oreilly.tomcat.filter.BadInputFilter with com.softmoore.filter.BadInputFilter in the <filter-class> element of this configuration. In addition, I think it’s worth considering a few other possible changes to either the configuration code in Listing 3 or to the original source code for BadInputFilter.

First, Eclipse Kepler (4.3.1) and Java 7 will give several warnings related to generics when compiling the original version of BadInputFilter. This is not surprising because the original code was written when generics were relatively new to Java. At one point in Jason Brittain’s original code there is a SuppressWarnings annotation to get rid of one such warning, but other warnings now exist. There are several ways to get rid of these warnings, but following Jason’s code we can simply add SuppressWarnings annotations as follows:

1. Add @SuppressWarnings(“rawtypes”) before method processAllowsAndDenies().
2. Replace @SuppressWarnings(“unchecked”) with @SuppressWarnings({ “unchecked”, “rawtypes” }) before method filterParameters.
3. Remove @SuppressWarnings(“unchecked”) from the middle of method filterParameters() since the change in item 2 above makes it redundant.

Second, when the configuration parameter escapeQuotes is set to true, as shown in Listing 3, BadInputFilter will replace all occurrences of a single-quote (apostrophe) with the entity “'”. This is an essential defense against several web security attacks, especially SQL injection attacks. It can present problems, however, if one of the parameters is used for a search based on last names and you want to allow searching for names like “O’Neil.” The safest thing to do here is to not make any changes to either the configuration file or BadInputFilter but to unescape the value of the parameter as part of additional validation before performing the search. (Of course, you should always prefer a JDBC PreparedStatement over a simple Statement when accessing the database; doing so not only is potentially more efficient, but PreparedStatements are less susceptible to SQL injection attacks. See the Resources section to learn more about using PreparedStatement for web application security.)

Third, the configuration settings in Listing 3 will forbid access (response status code 403) if a parameter contains either “x0a” (aka “newline” or “n”) or “x0d” (aka “carriage return” or “r”). One or both of these control characters will be generated when the user presses the Enter key, depending on whether the user is on a Unix/Linux computer or a Windows computer.

Forbidding these characters as part of request parameters would be an appropriate response for most websites where forms contain simple input fields such as checkboxes, radio-buttons, and textfields. In fact, the common default response when a user presses the Enter key is to submit the form, not to put control characters in the field. If, however, the form contains a textarea element, then the user is allowed to enter multiple lines of text. Pressing Enter does not submit the form but places one or both of the control characters into the parameter value.

So for websites using textarea elements, an alternate approach is to not forbid these control characters but to allow and escape them. In other words, remove them from the configuration settings for the deny parameter in Listing 3, modify BadInputFilter to add a configuration boolean parameter escapeNewlines (which is similar to the other escape parameters), and then replace all occurrences of “r” with a space character (“ “) and “n” with the HTML “ ” (or “ ” for XHTML). This escape replacement should work fine for most Windows and Unix/Linux environments, including Apple OS/X. You would need to make one more small change in order for the modification to work, however.

It is important to be certain when escaping parameters that you escape the angle bracket (<) before you escape newlines, since the escape for newlines contains an angle bracket. But since all escape patterns in BadInputFilter are stored in a HashMap, you have no guarantee of the iteration order when retrieving the entries. The solution is twofold:

1. Use a LinkedHashMap rather than a HashMap; for field parameterEscapes; that is, in the code for BadInputFilter, replace the declaration:

protected HashMap<String, String> parameterEscapes = new HashMap<String, String>();

with

protected Map<String, String> parameterEscapes 
    = new LinkedHashMap<String, String>();

(Note that “LinkedHashMap<String, String>()” can be shortened to “LinkedHashMap<>()” in Java 7 and later.)

2. Ensure that the entries to escape newlines are added to the parameterEscapes Map after the entries to escape angle brackets. It is important to ensure that angle brackets are escaped first.

In conclusion

If you develop and/or maintain a website that is open to the public, you should anticipate that someone will try to hack into it at some point in time. One way to reduce your website’s vulnerability is by validating all user input. The class BadInputFilter was originally designed to provide a frontline of defense for several well-known web application security exploits by filtering requests and either forbidding the request or escaping the potentially malicious input. With more recent versions of Tomcat, part of the functionality provided by BadInputFilter no longer works. In this article I’ve described changes to BadInputFilter that will restore the lost functionality. Moreover, unlike the original version of BadInputFilter, the updated version does not depend on the implementation of Tomcat and should run correctly in any Servlet/JSP container.