Published: Wednesday, November 12, 2008
Troubleshooting Website Problems by Examining the HTTP Traffic
By Scott Mitchell
Introduction
I started my career as a web developer with Microsoft's Active Server Pages (ASP), the predecessor to ASP.NET. ASP was a very simple scripting engine
and lacked the tools that ASP.NET developers today take for granted, most notably a debugger. Debugging an ASP script typically involved littering
the code with Response.Write statements to output the values of variables at different points in time of the script's life-cycle. Debugging
an ASP.NET page is so much easier thanks to the Visual Studio debugger, which allows you to set breakpoints, step through executing code, use Watch
windows to keep an eye on variable values as they change, and an Intermediate window to evaluate statements during debug time.
While the Visual Studio debugger has greatly improved the debugging story, there are certain scenarios where a server-side debugger is of little or no
help. In certain cases the problem is not in the server-side code but instead in what is being sent from the client to the server (or vice-a-versa).
These types of scenarios are quite common when creating AJAX-enabled web applications, as the data exchanged between the client and server during a
partial page postback affects the code executed on the server-side and how the page is updated on response. This technique is also quite useful when
debugging pages that perform different Response.Redirects based on various parameters, or when trying to ascertain why images, videos,
or other external content is not properly loading on a web page.
Unlike debugging server-side code, examining the HTTP traffic sent between the client and the server is typically done on the client - namely, from
the browser rather than from within Visual Studio. Fiddler is a free, excellent tool for debugging HTTP traffic. This article provides an overview
of Fiddler and shows how to use Fiddler to assist with debugging. Read on to learn more!
Why Bother With Viewing the HTTP Traffic?
Every time a web browser makes a request to a web server - be it for a web page, an image within a web page, a CSS file, an external JavaScript file,
a web service call, or whatever - the browser does so by making an HTTP request
to the server. In the HTTP request the browser sends the server the name of the resource it wants (e.g., Default.aspx) along with other,
optional information. For instance, when submitting a form the browser sends along the names and values of the form fields.
Upon receipt of this message, the web server retrieves the requested resource and returns the resource's content. This content may be binary data
(for an image, MP3, ZIP file, etc.), static text content in a text file or HTML page, or dynamically-generated text content, in the case of an ASP.NET
web page or ASP.NET web service. The browser then displays the returned content based on the content type.
The low-level information being ferried between the client and server is usually "correct" and works as expected. Typically when there's a problem in
an ASP.NET web page the problem lies in the code in the ASP.NET page's code-behind class. In such cases the Visual Studio debugger is usually the best
tool for finding the problem and fixing it. But in other scenarios the problem - or an explanation of the problem - does not lie in the server-side
code but rather in the information exchanged between the client and the server. For such scenarios, examining the traffic ferried between the client
and the server is instrumental in discovering the root of the problem.
Consider the following issue: you have an ASP.NET page with a form that contains a TextBox control, a Button control, and a Label control. The idea is
that the user types some value into the TextBox and submits the form and some information is then displayed on screen in the Label. You'd implement
this functionality by creating an event handler for the Button's Click event and writing code to display the appropriate information
in the Label control based on the user's input in the TextBox. After writing this code you go to test it and, sure enough, if you type in text into
the TextBox and then click the Button, the page posts back and the expected output is displayed. After some more testing you notice something a bit
fishy - everything works as expected if the user submits the form by clicking the Button, but if you hit the Enter key while focused in the
TextBox the page posts back but the Label control does not display anything. What's more, this oddity only exists when testing with Internet Explorer.
What gives?
Your first stab at solving this problem might be to use the Visual Studio debugger. You'd set a breakpoint in the Page_Load and
Click event handlers. This would show you that the page was posted back to when the Button was clicked or when the user hit the Enter key
while focused in the TextBox control, and would reveal that the Button's Click event handler only executes when the Button is explicitly
clicked (and not when the user hits the Enter key). But this type of debugging provides no explanation as to why this difference in
behavior exists.
This is the type of problem where analyzing the HTTP traffic sent between the client and the server is key. If you compare the HTTP traffic sent when
clicking the Button versus the HTTP traffic sent when hitting the Enter key you'd discover that the browser is not sending the name and value of
the Button when the user hits the Enter key, but this name/value pair is being sent when the Button is clicked. In order for ASP.NET to know
that the Button was clicked - and therefore raise its Click event - the Button name/value pair must be sent from the browser.
This difference explains the behavior observed. (For more background on this behavior as well as workarounds, be sure to read
Enter and the Button Click Event.)
The above example may sound like a very rare and narrow case where HTTP traffic inspection is paramount to successfully debugging the problem. However,
this type of analysis is quite useful in a myriad of scenarios. Inspecting the HTTP traffic is usually one of the first things to do when debugging
an AJAX-enabled web application. HTTP traffic analysis is useful for scenarios where there are potentially many redirects from one page to another
in order to ascertain what page(s) were visited and what information was sent to each page. Similarly, HTTP traffic analysis is helpful in determining
the cause of broken images or other missing external content. For example, if an image file or CSS file is in a directory that is inaccessible because
of file permissions or URL authorization rules, the browser will show a broken image icon or won't apply the CSS rule. To determine the root cause you
can inspect the HTTP response returned from these external files when requested by the browser, where you'll clearly see that the content could not
be retrieved because of a permission issue (or whatever the case may be).
Inspecting the HTTP Traffic with Fiddler
Fiddler is a free HTTP debugging tool created by Microsoft. Fiddler works with a host of browsers,
but works most naturally with Internet Explorer. (This article shows using Fiddler with Internet Explorer. For tips on having Fiddler work with Firefox,
see Fiddler for Firefox or read the "How can I configure other browsers to use Fiddler"
answer in the Fiddler FAQ.
To get started with Fiddler visit the Fiddler home page and download and install it. Once installed,
Fiddler is accessible from Internet Explorer from the Tools menu. Launching Fiddler opens a new window that tracks each HTTP request made by the browser
in a list in the left pane. Selecting a request from this list loads its details in the right panes. This includes a performance summary along with
detailed information about the request (the content sent from the client to the server) and the response (the content returned from the server to the
client).
The screen shot below shows Fiddler in action. The left pane lists the various HTTP requests. I started by visiting www.google.com and then searched
on the term "ASP.NET". (The various calls to clients1.google.com for /complete/search?hl=en... were AJAX requests to the
auto-complete search box on Google's homepage.) After viewing the search results I clicked on the link to www.asp.net. My browser then loaded the
homepage for that site plus the external files associated with the site: CSS files (.css), JavaScript files (.js and
WebResource.axd and ScriptResource.axd files), and image files. Selecting a request from the left displays its details
on the right. The initial request to www.google.com is selected on the left and the statistics are shown on the right.
You can also view the request and response details in the right pane by selecting the "Session Inspector" tab. From this tab you can inspect the
raw data sent in the HTTP request and response. The screen shot below shows this raw data for the request and response to www.google.com.
Fiddler is an indispensible application that every web developer should have in his toolbox. The only gotcha that virtually every developer who first
uses Fiddler runs into is that by default Fiddler cannot inspect HTTP traffic sent to localhost. The good news is that there are a couple of easy workarounds
you can apply to get Fiddler to start tracking HTTP requests to localhost. From the Fiddler
FAQ "Why don't I see traffic sent to http://localhost or http://127.0.0.1":
IE7 and the .NET Framework are hardcoded not to send requests for Localhost through any proxies, and as a proxy, Fiddler will not receive such traffic.
The workaround is to use your machine name as the hostname instead of Localhost or 127.0.0.1. So, for instance, rather than hitting
http://localhost:8081/mytestpage.aspx, instead visit http://machinename:8081/mytestpage.aspx.
There are other workarounds for getting Fiddler to work in localhost; refer to the Further Readings section at the end of this article.
| Use Fiddler for Evaluating Your Site's Performance |
|---|
|
In addition to inspecting the raw HTTP traffic, Fiddler is useful for analyzing the performance of your website. If you are worried that a
particular page is loading too slowly, use Fiddler to see if there is a particular external resource that's the culprit. Each request listed in
the left pane includes the size of the downloaded content (in bytes) and the request list can be sorted by this column. In short, you can quickly
see if there are any extraneously large requests, such as bloated image files or JavaScript files that are ripe for compression.
The Statistics tab in the right pane shows the total bytes sent and received for the selected requests in the left pane. In addition to the total
bytes the Statistics also include the actual time it took to make the selected requests, the bytes broken down by content type, and estimated
download times at various geographical locations with various download speeds (DSL vs. modem). and the time it took to make the requests.
The "Session Inspector" tab indicates whether a selected request was compressed on the server before being sent to the client. If no HTTP compression
was applied you can choose one of the standard HTTP compression techniques (GZIP or DEFLATE) and see how many bytes would have been saved had
compression been implemented.
|
Using Fiddler to Analyze HTTP Traffic from Client Applications
If you have client applications that make Web Service requests, do screen scrapes, or perform some other sort of HTTP request, you can use
Fiddler during debug-time to inspect these requests and responses. In short, you need to start Fiddler and then update your client application's
code to Fiddler as a proxy. The following code should do the trick:
' Create the class that calls the Web Service
Dim service As New MyWebServiceProxyClass()
' Use Fiddler as a web proxy
service.Proxy = New WebProxy("127.0.0.1", 8888)
' Make the HTTP request
service.MethodToInvoke(...)
|
That's all there is to it! With this code in place the Web Service request will be routed through Fiddler and you will see the request and
response in Fiddler. Pretty neat, eh?
For more information on this technique, along with some great step-by-step instructions on setting up Fiddler to work in browsers such
as Firefox, be sure to read Rich Strahl's blog entry
Monitoring HTTP Output with Fiddler in .NET HTTP Clients and WCF Proxies.
Conclusion
Most website problems are best tackled by using the Visual Studio debugger. However, there are a class of bugs that are not due to server-side code
errors but instead have to do with the raw information exchanged between the client and the server. For such bugs examining the traffic between a
client and a server is key to understanding the problem. The good news is that there is Fiddler, an excellent, free tool for inspecting HTTP traffic.
Happy Programming!
By Scott Mitchell
Further Readings
Fiddler Homepage
Fiddler Demonstration Videos
Fiddler for Firefox
Localhost HTTP Debugging with Fiddler
Working with Fiddler on Localhost
Monitoring HTTP Output with Fiddler in .NET HTTP Clients and WCF Proxies
