When you think ASP, think...
Recent Articles
All Articles
ASP.NET Articles
ASPFAQs.com
Message Board
Related Web Technologies
User Tips!
Coding Tips

Sections:
Sample Chapters
Commonly Asked Message Board Questions
JavaScript Tutorials
MSDN Communities Hub
Official Docs
Security
Stump the SQL Guru!
XML Info
Information:
Feedback
Author an Article
ASP ASP.NET ASP FAQs Message Board Feedback
Print this page.
Published: Wednesday, November 12, 2008

Troubleshooting Website Problems by Examining the HTTP Traffic

By Scott Mitchell


Introduction
I started my career as a web developer with Microsoft's Active Server Pages (ASP), the predecessor to ASP.NET. ASP was a very simple scripting engine and lacked the tools that ASP.NET developers today take for granted, most notably a debugger. Debugging an ASP script typically involved littering the code with Response.Write statements to output the values of variables at different points in time of the script's life-cycle. Debugging an ASP.NET page is so much easier thanks to the Visual Studio debugger, which allows you to set breakpoints, step through executing code, use Watch windows to keep an eye on variable values as they change, and an Intermediate window to evaluate statements during debug time.

While the Visual Studio debugger has greatly improved the debugging story, there are certain scenarios where a server-side debugger is of little or no help. In certain cases the problem is not in the server-side code but instead in what is being sent from the client to the server (or vice-a-versa). These types of scenarios are quite common when creating AJAX-enabled web applications, as the data exchanged between the client and server during a partial page postback affects the code executed on the server-side and how the page is updated on response. This technique is also quite useful when debugging pages that perform different Response.Redirects based on various parameters, or when trying to ascertain why images, videos, or other external content is not properly loading on a web page.

Unlike debugging server-side code, examining the HTTP traffic sent between the client and the server is typically done on the client - namely, from the browser rather than from within Visual Studio. Fiddler is a free, excellent tool for debugging HTTP traffic. This article provides an overview of Fiddler and shows how to use Fiddler to assist with debugging. Read on to learn more!

- continued -

Why Bother With Viewing the HTTP Traffic?
Every time a web browser makes a request to a web server - be it for a web page, an image within a web page, a CSS file, an external JavaScript file, a web service call, or whatever - the browser does so by making an HTTP request to the server. In the HTTP request the browser sends the server the name of the resource it wants (e.g., Default.aspx) along with other, optional information. For instance, when submitting a form the browser sends along the names and values of the form fields. Upon receipt of this message, the web server retrieves the requested resource and returns the resource's content. This content may be binary data (for an image, MP3, ZIP file, etc.), static text content in a text file or HTML page, or dynamically-generated text content, in the case of an ASP.NET web page or ASP.NET web service. The browser then displays the returned content based on the content type.

The low-level information being ferried between the client and server is usually "correct" and works as expected. Typically when there's a problem in an ASP.NET web page the problem lies in the code in the ASP.NET page's code-behind class. In such cases the Visual Studio debugger is usually the best tool for finding the problem and fixing it. But in other scenarios the problem - or an explanation of the problem - does not lie in the server-side code but rather in the information exchanged between the client and the server. For such scenarios, examining the traffic ferried between the client and the server is instrumental in discovering the root of the problem.

Consider the following issue: you have an ASP.NET page with a form that contains a TextBox control, a Button control, and a Label control. The idea is that the user types some value into the TextBox and submits the form and some information is then displayed on screen in the Label. You'd implement this functionality by creating an event handler for the Button's Click event and writing code to display the appropriate information in the Label control based on the user's input in the TextBox. After writing this code you go to test it and, sure enough, if you type in text into the TextBox and then click the Button, the page posts back and the expected output is displayed. After some more testing you notice something a bit fishy - everything works as expected if the user submits the form by clicking the Button, but if you hit the Enter key while focused in the TextBox the page posts back but the Label control does not display anything. What's more, this oddity only exists when testing with Internet Explorer. What gives?

Your first stab at solving this problem might be to use the Visual Studio debugger. You'd set a breakpoint in the Page_Load and Click event handlers. This would show you that the page was posted back to when the Button was clicked or when the user hit the Enter key while focused in the TextBox control, and would reveal that the Button's Click event handler only executes when the Button is explicitly clicked (and not when the user hits the Enter key). But this type of debugging provides no explanation as to why this difference in behavior exists.

This is the type of problem where analyzing the HTTP traffic sent between the client and the server is key. If you compare the HTTP traffic sent when clicking the Button versus the HTTP traffic sent when hitting the Enter key you'd discover that the browser is not sending the name and value of the Button when the user hits the Enter key, but this name/value pair is being sent when the Button is clicked. In order for ASP.NET to know that the Button was clicked - and therefore raise its Click event - the Button name/value pair must be sent from the browser. This difference explains the behavior observed. (For more background on this behavior as well as workarounds, be sure to read Enter and the Button Click Event.)

The above example may sound like a very rare and narrow case where HTTP traffic inspection is paramount to successfully debugging the problem. However, this type of analysis is quite useful in a myriad of scenarios. Inspecting the HTTP traffic is usually one of the first things to do when debugging an AJAX-enabled web application. HTTP traffic analysis is useful for scenarios where there are potentially many redirects from one page to another in order to ascertain what page(s) were visited and what information was sent to each page. Similarly, HTTP traffic analysis is helpful in determining the cause of broken images or other missing external content. For example, if an image file or CSS file is in a directory that is inaccessible because of file permissions or URL authorization rules, the browser will show a broken image icon or won't apply the CSS rule. To determine the root cause you can inspect the HTTP response returned from these external files when requested by the browser, where you'll clearly see that the content could not be retrieved because of a permission issue (or whatever the case may be).

Inspecting the HTTP Traffic with Fiddler
Fiddler is a free HTTP debugging tool created by Microsoft. Fiddler works with a host of browsers, but works most naturally with Internet Explorer. (This article shows using Fiddler with Internet Explorer. For tips on having Fiddler work with Firefox, see Fiddler for Firefox or read the "How can I configure other browsers to use Fiddler" answer in the Fiddler FAQ.

To get started with Fiddler visit the Fiddler home page and download and install it. Once installed, Fiddler is accessible from Internet Explorer from the Tools menu. Launching Fiddler opens a new window that tracks each HTTP request made by the browser in a list in the left pane. Selecting a request from this list loads its details in the right panes. This includes a performance summary along with detailed information about the request (the content sent from the client to the server) and the response (the content returned from the server to the client).

The screen shot below shows Fiddler in action. The left pane lists the various HTTP requests. I started by visiting www.google.com and then searched on the term "ASP.NET". (The various calls to clients1.google.com for /complete/search?hl=en... were AJAX requests to the auto-complete search box on Google's homepage.) After viewing the search results I clicked on the link to www.asp.net. My browser then loaded the homepage for that site plus the external files associated with the site: CSS files (.css), JavaScript files (.js and WebResource.axd and ScriptResource.axd files), and image files. Selecting a request from the left displays its details on the right. The initial request to www.google.com is selected on the left and the statistics are shown on the right.

Fiddler in Action...

You can also view the request and response details in the right pane by selecting the "Session Inspector" tab. From this tab you can inspect the raw data sent in the HTTP request and response. The screen shot below shows this raw data for the request and response to www.google.com.

Fiddler in Action...

Fiddler is an indispensible application that every web developer should have in his toolbox. The only gotcha that virtually every developer who first uses Fiddler runs into is that by default Fiddler cannot inspect HTTP traffic sent to localhost. The good news is that there are a couple of easy workarounds you can apply to get Fiddler to start tracking HTTP requests to localhost. From the Fiddler FAQ "Why don't I see traffic sent to http://localhost or http://127.0.0.1":

IE7 and the .NET Framework are hardcoded not to send requests for Localhost through any proxies, and as a proxy, Fiddler will not receive such traffic.

The workaround is to use your machine name as the hostname instead of Localhost or 127.0.0.1. So, for instance, rather than hitting http://localhost:8081/mytestpage.aspx, instead visit http://machinename:8081/mytestpage.aspx.

There are other workarounds for getting Fiddler to work in localhost; refer to the Further Readings section at the end of this article.

Use Fiddler for Evaluating Your Site's Performance
In addition to inspecting the raw HTTP traffic, Fiddler is useful for analyzing the performance of your website. If you are worried that a particular page is loading too slowly, use Fiddler to see if there is a particular external resource that's the culprit. Each request listed in the left pane includes the size of the downloaded content (in bytes) and the request list can be sorted by this column. In short, you can quickly see if there are any extraneously large requests, such as bloated image files or JavaScript files that are ripe for compression.

The Statistics tab in the right pane shows the total bytes sent and received for the selected requests in the left pane. In addition to the total bytes the Statistics also include the actual time it took to make the selected requests, the bytes broken down by content type, and estimated download times at various geographical locations with various download speeds (DSL vs. modem). and the time it took to make the requests.

The "Session Inspector" tab indicates whether a selected request was compressed on the server before being sent to the client. If no HTTP compression was applied you can choose one of the standard HTTP compression techniques (GZIP or DEFLATE) and see how many bytes would have been saved had compression been implemented.

Using Fiddler to Analyze HTTP Traffic from Client Applications
If you have client applications that make Web Service requests, do screen scrapes, or perform some other sort of HTTP request, you can use Fiddler during debug-time to inspect these requests and responses. In short, you need to start Fiddler and then update your client application's code to Fiddler as a proxy. The following code should do the trick:

' Create the class that calls the Web Service
Dim service As New MyWebServiceProxyClass()

' Use Fiddler as a web proxy
service.Proxy = New WebProxy("127.0.0.1", 8888)

' Make the HTTP request
service.MethodToInvoke(...)

That's all there is to it! With this code in place the Web Service request will be routed through Fiddler and you will see the request and response in Fiddler. Pretty neat, eh?

For more information on this technique, along with some great step-by-step instructions on setting up Fiddler to work in browsers such as Firefox, be sure to read Rich Strahl's blog entry Monitoring HTTP Output with Fiddler in .NET HTTP Clients and WCF Proxies.

Conclusion
Most website problems are best tackled by using the Visual Studio debugger. However, there are a class of bugs that are not due to server-side code errors but instead have to do with the raw information exchanged between the client and the server. For such bugs examining the traffic between a client and a server is key to understanding the problem. The good news is that there is Fiddler, an excellent, free tool for inspecting HTTP traffic.

Happy Programming!

  • By Scott Mitchell


    Further Readings

  • Fiddler Homepage
  • Fiddler Demonstration Videos
  • Fiddler for Firefox
  • Localhost HTTP Debugging with Fiddler
  • Working with Fiddler on Localhost
  • Monitoring HTTP Output with Fiddler in .NET HTTP Clients and WCF Proxies

  • ASP.NET [1.x] [2.0] | ASPMessageboard.com | ASPFAQs.com | Advertise | Feedback | Author an Article