When you think ASP, think...
Recent Articles
All Articles
ASP.NET Articles
ASPFAQs.com
Message Board
Related Web Technologies
User Tips!
Coding Tips
Search

Sections:
Book Reviews
•Sample Chapters
Commonly Asked Message Board Questions
JavaScript Tutorials
MSDN Communities Hub
Official Docs
Security
Stump the SQL Guru!
Web Hosts
XML
Information:
Advertise
Feedback
Author an Article
Jobs




ASP ASP.NET ASP FAQs Message Board Feedback ASP Jobs
 
Print this Page!
Published: Wednesday, September 8, 2004

An Extensive Examination of Web Services: Part 10

By Scott Mitchell

An Extensive Examination of Web Services Index
An Extensive Examination of Web Services is a multi-part article series spanning several months. Listed below are the current "parts," along with a very brief synopsis and the date published.
  • Part 1 - Examines the basics of Web services, what Web services are, and the technologies and standards that serve as the underpinnings of Web services. (October 8th, 2003)
  • Part 2 - Examines creating Web services using Visual Studio .NET. Looks underneath the hood of the code created by VS.NET. (October 15th, 2003)
  • Part 3 - Examines creating a client application that consumes a Web service. Discusses the purpose and structure of a WSDL document, along with creating and using proxy classes to consume a Web service. (November 5th, 2003)
  • Part 4 - Examines the utility of Web services and common scenarios where Web services make sense. A business-oriented look at Web services. (November 19th, 2003)
  • Part 5 - Takes an in-depth look at XML serialization, which is the process of converting a data type, such as an integer, array, or custom class, into its XML representation, and back again. Every time a message is passed to or from a Web service, XML serialization transpires. (December 17th, 2003)
  • Part 6 - Looks at sending metadata to a Web method through the use of SOAP headers. Examines defining and accepting a SOAP header on the Web service end, and looks at sending a populated SOAP header from the client. (December 31st, 2003)
  • Part 7 - Examines how the incoming and outgoing messages to a Web service can be programmatically modified via SOAP Extensions. (January 21st, 2004)
  • Part 8 - Learn about the Web Service Enhancements (WSE) and Microsoft's free class library for implementing the WSE standards. (June 30th, 2004)
  • Part 9 - See how to implement UsernameToken authentication using the WSE 2.0 Toolkit. (July 14, 2004)
  • Part 10 - Learn how to send large amounts of data as attachments using DIME and WS-Attachments. (September 8th, 2004)
  • Part 11 - Create (and consume) client-side script-accessible Web Services using Microsoft's ASP.NET AJAX framework. (September 26th, 2007)

Introduction


As discussed in previous installments of this article series, Web services utilize a number of core standards, such as XML, SOAP, and WSDL. The XML and SOAP standards spell out the structure with which the messages sent between a client and a Web service are formatted; WSDL indicates how a Web service can formally define its functionality in a machine-readable syntax. While these core standards are sufficient for implementing Web services, they don't indicate how to accomplish common business tasks, such as authenticating a requestor, encrypting a message, and so forth.

To provide a standardized means for accomplishing everyday Web service scenarios, a number of industry leaders - Microsoft, IBM, BEA, and others - joined forces to craft a number of emerging Web service standards. These standards, called the Web Service Enhancements (WSE, for short), were the topic of discussion in Part 8 of this article series. WSE includes a vast array of emerging standards to accomplish a variety of common tasks, including standards related to security, process flow, privacy, and messaging, among others. Many of these standards are fairly new, and are still being discussed, while others are more mature and have been officially adopted as standards by open standards group, such as OASIS. In Part 8 we also discussed how Microsoft has released a free toolkit - the WSE 2.0 Toolkit - for using the more mature WSE standards. Armed with this toolkit you can easily utilize the supported WSE standards when creating .NET Web services and .NET Web service clients.

In this article we'll examine how to use the WSE Toolkit to efficiently send binary attachments through a Web service call, both from the client to the Web service, and from the Web service back to the client. As we'll discuss in this article, Microsoft's WSE Toolkit allows large attachments to be sent along with a Web service method using the DIME and WS-Attachments standards. We'll examine these standards and why they are more efficient than sending large amounts of binary data in a Web service call through other common means.

- continued -

Techniques for Sending Large Amounts of Data in a Web Service Call


There may be times where you find yourself creating a Web service that needs to be sent some (potentially) large amount of data from the client consuming the service. For example, a job hunting Web site might offer a Web service for recruiters that provided a means for them to post new jobs. (A Web service interface might be preferred over a Web site interface for recruiters since they may have to post numerous job offers. Through a Web site, a recruiter would have to enter the information by hand; with a Web service, the task could be automated.) In addition to passing details about the job posting, a recruiter might also want to send some large amount of data, such as a picture of the facilities where the job is being hosted, or a PDF that includes a formal description of the job.

There are three general techniques for passing potentially large amounts of data into a Web service:

  1. As an input parameter to the Web service method - to allow a recruiter to send a PDF containing a formal description of the job being posted, the Web service method PostJob() could be augmented to include a byte array input parameter, whose contents would be the binary content of the PDF file.
  2. Using SOAP with Attachments - SOAP with Attachments provides a standardized way to send a MIME-encoded message that includes the SOAP envelope in one MIME part, and an arbitrary number of attachments in the other MIME parts. With this technique, the PDF file could be sent as a single MIME part in a two-part MIME message.
  3. Using DIME and WS-Attachments - DIME - short for Direct Internet Message Encapsulation - is a standard for sending a message in explicitly-sized chunks. WS-Attachments is a standard that defines how to use DIME to send attachments in a Web service call. For our recruiter example, one chunk could contain the SOAP envelope while another chunk could contain the PDF attachment.
The preferred technique today is to use DIME and WS-Attachments for reasons we will examine shortly. As we'll see, the WSE 2.0 Toolkit makes it easy to send and receive DIME attachments - you just need to write a line or two of code, and the WSE Toolkit will take care of the rest for you.

MTOM - the Next Step in the Evolution of Sending Attachments
MTOM - short for Message Transmission Optimization Mechanism - is the next step in the evolution of sending attachments in a Web service. MTOM is very new - a W3C Candidate Recommendation for MTOM was released just a couple of weeks ago (August 26, 2004). While MTOM is not supported in the WSE 2.0 Toolkit, it will be used heavily in Indigo, Microsoft's upcoming messaging system. To learn more about MTOM check out the MTOM specification.

Why DIME and WS-Attachments is the Preferred Technique (Today)


To understand why DIME and WS-Attachments are the preferred technique for passing large attachments in a Web service setting, let's first look at the shortcomings of sending large data as a method parameter and SOAP with Attachments. Of the two techniques, sending a large amount of data as a Web service method parameter is the least efficient. All of the input parameter values passed into a Web service method from a client, are serialized into XML and sent in the SOAP Body. If you are sending large, binary data through a method's input parameter, you'll likely pass in a byte array. That is, you'd define your method in the Web service like:

_
Public Sub PostJob(title as String, salary as Decimal, ..., PDF() as Byte)
   ...
End Sub

While this approach will definitely work, it is horribly efficient when passing large amounts of data. Why? Well, the client must encode the binary data into text using base64 encoding, and then send the contents inside the SOAP Body. That is, the SOAP envelope being sent from the client to the recipient might look something like: 

<soap:Envelope ...>
  <soap:Body>
    <PostJob>
      <title>Product Tester</title>
      <salary>55000.00</salary>
      ...
      <PDF>
        QfvW1lJL4ye7abHPruebye7abHPrueiF14Il
        W1lJL4oW1lJye7abHPrueL4...==
      </PDF>
    </PostJob>
  </soap:Body>
</soap:Envelope>

Of course the contents in the <PDF> element may be hundreds, if not thousands of kilobytes. The reason this is inefficient is because crafting and parsing XML documents is not trivial, and requires substantial memory and processor resources. By making the XML document several hundreds of kilobytes or megabytes in size is an easy way to quickly bring your Web service to its knees.

To improve the situation, we want to move the large data outside of the SOAP Envelope, so it does not hinder the creation and parsing of the XML. By moving this data out of band, it is called an "attachment." That is, ideally we would want to send the message in a format like: 

<soap:Envelope ...>
  <soap:Body>
    <PostJob>
      <title>Product Tester</title>
      <salary>55000.00</salary>
      ...
      <PDF>
        PDF DOCUMENT AT END OF ENVELOPE
      </PDF>
    </PostJob>
  </soap:Body>
</soap:Envelope>

QfvW1lJL4ye7abHPruebye7abHPrueiF14Il
W1lJL4oW1lJye7abHPrueL4...==

Of course the challenge that now faces us is:

  1. How to we add the PDF contents to the end of the message so that it can quickly be found. (This process is complicated further since we need to be able to send, perhaps, multiple attachments, and not just one.)
  2. How do we tie the <PDF> element in the SOAP Body to the appropriate attachment?
There are different ways to solving these two problems. One such way, used by the SOAP with Attachments standard, is to send the message using the MIME standard, and having the attachments appear in separate MIME boundaries. SOAP with Attachments has its downsides, the primary one being that while MIME boundaries make it easy for the sender of the attachment to add an arbitrary number of attachments, it takes considerably more work to pick out the precise attachments from the receiver's standpoint, since each attachment must be scanned in its entirety, in order to find the MIME boundaries.

Understanding MIME
MIME - short for Multipurpose Internet Mail Extensions - is a standardized means for sending attachments in email. MIME works by dividing the email content into distinct boundaries, separating each boundary with a unique boundary string. With email, the first MIME boundary contains the email text, while the remaining boundaries contain any related attachments. The following shows a sample email, with the MIME boundaries made bold: 

Date: Sun, 07 Dec 2003 10:45:53 -0800
From: "John Smith" 
Subject: URGENT - NIGERIA MONEY IS YOURS TO BE CLAIMED!!
To: Scott Mitchell 
Content-type: multipart/related; boundary="Boundary_(ID_FXnn6OVMs/pgPvsHzyUcwA)"

This is a multi-part message in MIME format.

--Boundary_(ID_IZ1Lhy3/0i8IoHkif3PKpA)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7bit

I am Charles Smith, from the Nation of Nigeria, with URGENT news regarding a bank
account with the sum of 14 MILLION USD.  Attached is a PDF file with instructions
on how to claim this vast fortune.  ACT TODAY.

--Boundary_(ID_FXnn6OVMs/pgPvsHzyUcwA)
Content-id: <433473118@07122003-0700>
Content-type: x-application/pdf; name=scam.pdf
Content-transfer-encoding: base64
Content-disposition: attachment; filename=scam.pdf

R0lGODlhmQA3APcAAAAAAIAAAACAAICAAAAAgIAAgACAgMDAwMDcwKbK8EAgAGAgAIAgAKAgAMAg
AOAgAABAACBAAEBAAGBAAIBAAKBAAMBAAOBAAABgACBgAEBgAGBgAIBgAK...

Working with DIME and WS-Attachments


DIME and WS-Attachments are the preferred technique for sending attachments. Unlike SOAP with Attachments, which uses MIME boundaries to separate the attachments from one another, DIME uses explicitly-sized chunks. Each chunk has a record number and a size associated with it, so the receiver of a DIME-formatted message can quickly move from one attachment to another. With SOAP with Attachments, moving from one attachment to another requires examining the first attachment in its entirety, looking for the MIME boundary; with DIME, the receiver only needs to check the size of the current chunk and then skip ahead however many bytes specified to get to the beginning of the next chunk.

Using DIME to send an attachment in a Web service call is easy, thanks to Microsoft's WSE 2.0 Toolkit. For this article, let's create a Web service that accepts a binary attachment and simply saves it to the Web server's file system. To start, create a Web service application, and configure it to use the WSE 2.0 Toolkit. (Recall that this requires right-clicking on the Project Name, choosing the WSE 2.0 Settings option - see the screenshot to the right - and checking both checkboxes in the General tab.)

Next, in the Web service class create a method called UploadPDF(). This method will accept a DIME attachment and save it to the Web server's file system. The code for this method is fairly simple, and shown below. Note that the method does not take in the attachment as an input parameter of any kind - it simply references the attachment through the Attachments collection within the code. (For this code be sure to add a using or Imports statement at the top of the class file for the namespaces Microsoft.Web.Services2 and Microsoft.Web.Services2.Dime, as well as for System.IO (since we'll be saving the attachment to the Web server's file system)). 

 _
Public Sub UploadPDF()
    'Make sure precisely one attachment was sent
    If RequestSoapContext.Current.Attachments.Count <> 1 Then
        Throw New ArgumentException("Please upload one file at a time...")
    End If

    'Save the file to LatestPDF.pdf
    Dim fs As FileStream = File.Create(HttpContext.Current.Server.MapPath("LatestPDF.pdf"))
    Dim pdfLength As Integer = RequestSoapContext.Current.Attachments(0).Stream.Length
    Dim buffer(pdfLength) As Byte
    RequestSoapContext.Current.Attachments(0).Stream.Read(buffer, 0, pdfLength)
    fs.Write(buffer, 0, pdfLength)
    fs.Close()
End Sub

As this example shows, the Web service has access to the set of attachments passed in using DIME via the RequestSoapContext.Current.Attachments collection. This collection contains a set of DimeAttachment instances, each of which contain a Stream property. In UploadPDF(), the attachment size is first checked to ensure that precisely one attachment was passed in. Next, a FileStream object is used to save the contents of the uploaded PDF to the Web service's directory on the file system.

All that remains is to create the client. For this demo I used an ASP.NET Application as the client, although a WinForms application would work just as well. The client presents the user with a file upload box, allowing them to choose a PDF. Upon submitting the Web Form, the ASP.NET page creates a DimeAttachment with the contents of the uploaded file, adds it to the outgoing Attachments collection - RequestSoapContext.Current.Attachments - and invokes the Web service's UploadPDF() method. The complete code, along with a screenshot, is shown below. (For more information on uploading files with ASP.NET be sure to read Uploading in ASP.NET, by Tribikram Rath.) 

Private Sub btnUpload_Click(ByVal sender As System.Object, _
                 ByVal e As System.EventArgs) Handles btnUpload.Click
    'Call the Web service
    Dim proxy As New localhost.DIMEDemoWse

    'Add the attachment
    Dim attachment As _
         New DimeAttachment("PDF", TypeFormat.MediaType, _
                                Me.fileUpload.PostedFile.InputStream)
    proxy.RequestSoapContext.Attachments.Add(attachment)

    'Call the Web service's UploadPDF() method
    proxy.UploadPDF()

    ...
End Sub

In the code we start by creating an instance of the WSE-enabled proxy class, and then add a DimeAttachment instance to its Attachments collection. The DimeAttachment constructor takes in, among other inputs, a stream to the data to upload. Since we are uploading a file using the file upload control provided by ASP.NET, we reference this uploaded file's stream using the PostedFile class's InputStream property. Once the DimeAttachment has been added, we simply invoke the UploadPDF() method.

Screenshot of Client in Action...


Closing Comments on DIME with the WSE 2.0 Toolkit


In this article we saw a demo where a client sends an attachment to a Web service, and the Web service saves the attachment on its file system. There are scenarios where you may want to reverse this direction. That is, the client might request a particular file, and the Web service might return it as an attachment using DIME. If you want to send an attachment from the Web service to the requesting client, simply create a DimeAttachment in the Web service (as we did in the client example above), and add it to the ResponseSoapContext.Current.Attachments collection (as opposed to RequestSoapContext.Current.Attachments). Similarly, to receive the attachment from the client, after the call to the Web service method that returns an attachment, the attachment will be available in the proxyClassInstance.ResponseSoapContext.Attachments collection.

Finally, when sending attachments with DIME it is vitally important that you do not close the stream of the attachment being sent in the DimeAttachment. That is, don't do something like: 

Dim attachment As _
     New DimeAttachment("PDF", TypeFormat.MediaType, _
                          Me.fileUpload.PostedFile.InputStream)
proxy.RequestSoapContext.Attachments.Add(attachment)

'DO NOT DO THIS!
fileUpload.PostedFile.InputStream.Close()

proxy.UploadPDF()

While it may seem logical to "clean up" after yourself, closing the underlying stream will result in an error when attempting to send the attachment in the Web service request or response. This is because the WSE Toolkit needs to have access to the data that is being attached to the SOAP request or response, in order to craft the appropriate message format. If you close the underlying stream, the WSE Toolkit won't be able to properly access the data it needs. Rest assured, the WSE Toolkit will close the stream for you - so don't do it yourself, or you may end up with a cryptic exception.

Conclusion


In this article we looked at various options for sending large amounts of data in a Web service call. Of the three current options, the best one is DIME with WS-Attachments. Sending attachments with DIME moves the contents of the data outside of the SOAP Envelope, and formats the attachments in a means that can be quickly accessed by the recipient of the message. Microsoft's WSE 2.0 Toolkit provides support for sending and receiving DIME attachments with Web services and Web service clients.

Happy Programming!

  • By Scott Mitchell

    Attachments:


  • Download the DIME Web Service and Client Files Used in this Article

    Beginner's .NET XML Web Services

    Beginner's .NET XML Web Services DVD

    Beginner's .NET XML Web Services offers nearly eight hours of training on .NET XML Web services in a video format. This two-disc DVD set, presented by author Scott Mitchell, offers a unique opportunity for learning about the fundamentals of Web services. The 14 lessons begin with an examination of the core Web service standards, and then quickly move into showing you how to create and consume Web services in Microsoft's .NET Framework. There are in-depth lessons for each of the core Web service standards - XML, SOAP, and WSDL - along with examples of the Web Service Enhancements (WSE). Scattered throughout each of these chapters are extensive demos, depicting how to build, deploy, and access Web services using Microsoft's Visual Studio .NET. The DVD also contains a thorough examination of a real-world, end-to-end Web service application.

    Scott Mitchell is the editor and founder of 4GuysFromRolla.com, author of the An Extensive Examination of the Web Services article series, and author of numerous ASP and ASP.NET books.

    [Buy this DVD]

    Why get stuck with a 600-page paperweight when you can learn about .NET Web Services through an interesting and interactive video presentation?

    • Software Developer / Programmer - Distributed Systems (NYC)
    Next Step Systems
    US-NY-New York

    Justtechjobs.com Post A Job | Post A Resume

    ASP.NET [1.x] [2.0] | ASPMessageboard.com | ASPFAQs.com | Advertise | Feedback | Author an Article