User:AnuragT/HTTP

= Hypertext Transfer Protocol =

1. Overview
Have you ever wondered how your computer talks to other computers on your local LAN or to other systems on the internet?

Understanding the processes of how computers interact is an important part of networking based interaction. In this chapter, we will make an attempt to discuss the concept of communication from the very basic fundamental level that needs to be understood by everybody.

Further sections will address a brief Introduction to HTTP, its history, various associated terminologies, purpose of HTTP, its message pattern, architecture and its stateless nature. Also a short introduction to cookies and concept of content negotiation shall be a part of the chapter.

2. Introduction
Before moving on to the details, it can be helpful to get a hold of scenario so far. Previous chapters have covered,

•	Basic networking that enables the communication via different layers.

•	Ethernet as implemented on the base level.

•	TCP/IP protocols enabling the communication over higher layers.

•	Underlying infrastructure The Internet.

•	Other technologies that supplement networking such as DNS.

•	Web Services i.e. WWW

Observe the image below. Computer X is attempting a communication with Computer Y and with our prior knowledge of network layers we can see the flow of information from X to Y. In addition we can also see the different protocols acting on each layer to enable this exchange.



So far we are aware of, •	The Internet the underlying infrastructure which enables the data (i.e. information) exchange.

•	The physical medium (i.e. cables) which enable the exchange on a physical (Hardware) level.

•	The role of Data Link Layer in transmitting the data (i.e. information) via frames.

•	The role of Network Layer to further transfer (on sender’s end)/receive (on receiver’s end) frames via IP packets.

•	The role of Transport Layer to further transfer (on sender’s end)/receive (on receiver’s end) IP packets via TCP segments.

Now moving ahead in the flow the data (i.e. information) reaches the Application Layer. The Application Layer which takes care of various applications or processes (within the computers) that uses underlying layers and their protocols to deliver the data to destination computers.

By now we observe at each layer there are certain protocol options to carry out the task designated to that particular layer. Similarly, application layer also has various protocols that applications use to communicate with the second layer, the transport layer. Some of the popular application layer protocols are:

• HTTP (Hypertext transfer protocol)

• FTP (File transfer protocol)

• SMTP (Simple mail transfer protocol)

• SNMP (Simple network management protocol) etc.

This chapter aims to cover the HTTP protocol implemented over the Application Layer. Understanding the working of HTTP takes us a step closer to how information is exchanged between computers and the role of HTTP in facilitating the exchange.

2.1 History of HTTP
The HTTP protocol first appeared in yearly 90’s as a solution for document management in CERN, a huge European research center. They needed to maintain and manage a large number of scientific documents, each having many links to other documents (similar to modern day Wikipedia).

Using hypertext (a pre-existing concept in practice was an obvious and reasonable decision to build such a vast system of interconnected documents. Then, Sir Tim Berners-Lee came up with a quite improvised solution: to store documents on a central server and manage those using client applications (browsers) installed on all computers. To build the link between the server and the client application, the HTTP was invented. The first version of the protocol had only one method, namely GET, which would request a page from a server and returned only HTML pages.

Nowadays, HTTP/1.1 is the most commonly used version of the HTTP protocol, which is specified in RFC2616 internet standard document.

Version History

• HTTP/0.9 in 1991 was the first documented version, corresponding to the original version.

1. Limitation - Single method (GET) and returned only HTML pages.

• HTTP/1.0 in 1992, a version was published, with some minor changes, was finalized by May 1996 (RFC 1945).

2. Improvements - Ability to transmit files of different types like images, videos, scripts, CSS documents.

• HTTP/1.1 in 1995, with IETF developing a new version of HTTP. It quickly spread into wide usage, and it was officially standardized in 1997 in RFC 2068, with minor fixes in RFC 2616 two years later.

3. Improvements - HTTP/1.1 brought the ability to reuse established connections for subsequent requests, greatly improving the performance of the protocol by lowering the latency between them.

2.2 Terminologies
• Browser - A software application used to locate, retrieve and also display content on the World Wide Web, including Web pages, images, video and other files.

• Web Page - A document on the World Wide Web. Every Web page is identified by a unique URL (Uniform Resource Locator).

• URI - A URI identifies a resource either by location, or a name, or both. Hence it is an identifier or a reference to a source and not a filename or its location itself. With a proper protocol it is possible to retrieve the actual resource, that’s called dereferencing a URI.

• URL - A URL is a specialization of URI that defines the network location of a specific resource. Unlike a URI that is just an identifier for some resource; the URL defines how the resource can be obtained.

A relation between URI and URL, all URLs are URIs but not vice-versa.

• Hypertext – A special type of database system that enables a user to access particular locations in webpages or other electronic documents by clicking on links within specific webpages or documents.

• Media – Text, graphics, voice, animation, video.

• Hypermedia - An extension to hypertext that supports linking graphics, sound, and video elements in addition to text elements.

• Protocol - An agreed-upon format for transmitting data between two devices.

• HTTP - An underlying protocol used by the World Wide Web, defines how messages are formatted and transmitted, and what actions Web servers and browsers should take in response to various commands (such as URLs entered in the browser).

• HTML - The authoring language used to create documents on the World Wide Web. It defines the structure and layout of a Web document on the browser.

2.3 Purpose of HTTP
HTTP functions as a request-response protocol in the client-server computing model. The modern day web for the most part is based on a Client-Server Architecture as illustrated below. A client is any computed that is a part of the network, while a server is a powerful computer designed with one goal that is to service requests from various clients on the network.

The services may include,

• Data retrieval.

• Access to devices such as printers, scanners (again on the network).

• Manage disk drives (in case of file servers).

Hence the purpose of HTTP is to enable these “requests” and “responses” to and fro in between the source (client) and target (server). An illustration to demonstrate the above mentioned process.

• A simple client server works typically with a client machine sending a request to the server. This may be illustrated as, a client    entering a URL on his/her browser to request a web page (which can be a hypertext document or media located on the server).

• The URL as discussed is actually a path to the target document (being requested by the client) on the server.

• The request is sent to the server in form of an HTTP request which is received by the server.

• The requested document or media is then located on the server’s file system with the path (URL) provided by the client’s request.

• Once located the server prepares a response message (HTTP Response format) which contains the requested document/media and sent back to the client’s browser.

• The browser is already equipped to understand the HTTP response and hence the browser displays the response in a way that the user can comprehend.

• The representation of the document/media on the client’s browser is taken care by HTML.



The above scenario indicates the role and purpose of HTTP within the World Wide Web. HTTP is the foundation of data communication for the World Wide Web. It is used for communication between web browsers and web servers, though in principle it can be used for other purposes as well.

3. Nature of HTTP
As displayed in the previous graphic HTTP functions on a request/response basis. This is one of the basic ways via with computers communicate. An analogy of a simple telephone call made and answered demonstrates nature of HTTP functionality. The method is incorporated in the client-server model as well, where multiple clients place requests which are then met with a response from a server.

For simplicity and performance this method of communication is usually implemented in a purely synchronous fashion. The term synchronous implies that when a connection (request) initiated by a web service, HTTP tends to keep the connection open until it is met by a response or a preset timeout period expires on which the connection is terminated to reduce unnecessary load on the server.

However there are also asynchronous implementations in cases where the response may be relayed later in time, after the request has been placed.

4.1 An HTTP Session
Because HTTP is a client-server protocol, an HTTP session consists of three phases:

1. The client establishes a TCP connection (or the appropriate connection if the transport layer is not TCP).

2. The client sends its request and then waits for the answer.

3. The server processes the request and sends back its answer, containing a status code and the appropriate data.

Note : With HTTP/1.1, the connection is no longer closed after the third phase, as the client is allowed to issue another request at this point: the second and third phases can therefore be done several times.

4.2 Establishing A Connection
Because HTTP is a client-server protocol, it always is the client that establishes the connection. Opening a connection in HTTP really is establishing a connection in the underlying transport layer, usually TCP.

With TCP, the default port for an HTTP server on a computer is port 80, though others are often used, like 8000 or 8080. The URL of a page to fetch contains both the domain name and the port number, though the latter can be omitted if it is 80.

4.3 HTTP Request
Once the connection is established, the user-agent can send its request. (A user-agent is typically a web browser, but need not be.) A client request consists of text directives, separated by CRLF (carriage return, followed by line feed), divided in three blocks:

1. The first line contains a request method followed by its parameters:

♦ The path of the document, i.e., an absolute URL without the protocol and the domain name

♦ The HTTP protocol version used

2. The subsequent lines each represent a specific HTTP header, giving the server some information about what kind of data is appropriate (e.g., what language) or some data altering its behavior (e.g., not sending an answer if it is already cached). These HTTP headers form a block that ends with an empty line.

3. The final block is the optional data block, which contains further data and is mainly used by the POST method.

Examples of requests

Fetching the root page of developer.mozilla.org, i.e. http://developer.mozilla.org/, and telling the server that the user-agent would prefer the page in French, if possible:

GET / HTTP/1.1

Host: developer.mozilla.org

Accept-Language: fr

4.4 HTTP Response
After the connected agent has sent its request, the web server handles it, and finally sends a response back. Similarly to a client request, a server response is formed of text directives, separated by CRLF, though divided in three different blocks:

1. The first line, the status line, consists of an acknowledgment of the HTTP version used followed by a status request.

2. The subsequent lines each represent a specific HTTP header giving the client some information about the data sent (e.g., type, data size, compression algorithm used). Similarly to the block of HTTP headers for a client request, these HTTP headers form a block that ends with an empty line.

3. The final block is the data block, which contains the data (if any).

Examples of responses

Successful reception of a web page:

HTTP/1.1 200 OK

Date: Sat, 09 Oct 2010 14:28:02 GMT

Server: Apache

Last-Modified: Tue, 01 Dec 2009 20:18:22 GMT

ETag: "51142bc1-7449-479b075b2891b"

Accept-Ranges: bytes

Content-Length: 29769

Content-Type: text/html

<!DOCTYPE html... (here comes the 29769 bytes of the requested web page)

4.5 Headers
TTP message headers are used to precisely describe the resource being fetched or the behavior of the server or the client. Custom proprietary headers can be added using the 'X-' prefix; others are listed in an IANA registry, whose original content was defined in RFC 4229. IANA also maintain a registry of proposed new HTTP message headers.

4.6 HTTP Response Codes
HTTP Response Codes indicate whether a specific HTTP requests has been successfully completed.

Responses are grouped in five classes:

♦ Informational Responses

♦ Successful Responses

♦ Redirections

♦ Client Errors

♦ Servers Errors

The following link lists them all, with their respective meanings:

5. Stateless Nature of HTTP
HTTP has been stated down as a stateless protocol. A small background on stateful and stateless at this juncture can provide a valuable insight on the arguable nature of the protocol being discussed. Stateful and stateless are terms (adjectives) that imply whether or not a protocol is engineered to remember single or multiple preceding events in a given sequence of interactions with a user, another computer or application, a device, or other outside element.

Stateful implies the computer or an application keeps track of the interaction (connection), usually achieved by storing connection details in fields designated for that purpose. An example for the same can be how a TCP connection works. Stateless on the other hand implies that there is no record of previous interactions (connection) and each connection request has to be handled based entirely on information that comes with it. An example for the same can be how HTTP supposedly functions.

Now a fair argument can arise if HTTP is based on TCP protocol why and how is it stateless and stateful. A few observations can help us shed some light on the argument at hand. HTTP doesn't "inherit" from TCP, but rather uses it for a transport. HTTP uses TCP to establish connections those are stateful in nature, but then disconnects on the request is completed with an apt response from the server. Later if required (say a new HTTP request is initiated) it will re-establish a connection again. So, while you browse through a web site you create many different connections. Each one of those connections is stateful, but the conversation as a whole is not, since you are dropping the connection with every conversation.

Let’s take an analogy for better insight. Consider the phone service to be TCP and consider your relationship with distant family members to be HTTP. You will contact them with the phone service. Each call to them would be a stateful TCP connection. However, you don't constantly stay on the phone with them, as you will disconnect and call them back again at a later time. You would certainly expect them to remember what you talked about on the last call. HTTP in itself does not do that, but it is rather a function of the web server that maintains the state of the overall conversation.

The reason for this design of HTTP is fairly justified and simple. A popular web-server easily takes up to millions of hits per day. Each of those hit is of quick connect/disconnect nature. If this was not the case then the server ends up with an open connection for each user (so a million hits makes a million users which in turn makes a million open connections) logged on to web pages hosted on the server resulting in potential overload and at some point maybe even a server crash.

While the stateless nature of HTTP was suited to the initial tasks which included mostly included fetching documents (static web pages), but since the web went ahead to become more complex (dynamic web applications, online stores, etc.) with multiple media formats hosted, shared and transferred a shortcoming of HTTP was realized. This was because these are inherently stateful activities. This shortcoming of HTTP is being addressed in a number of new technologies, including cookies.

5.1 Origin of Cookies
One of the biggest issues in the early days of the web was how to manage “state”. Managing “state” refers to the scenario where the server had no way of knowing if two requests came from the same web browser. Cookies are instrumental in maintaining state on the Web. "State" in this case refers to an application's ability to work interactively with a user, remembering all data since the application started, and differentiating between multiple users and their individual data sets.

Let’s take an analogy of a laundry cleaner's shop. You drop your bag of clothes off, and get a token in return. When you return with the token, you get your clothes back. If you don't have the token, then the laundry man doesn't know which clothes are yours. In fact, he won't be able to tell whether you are there to pick up clothes, or a brand new customer. As such, the token is critical to maintaining state between you and the laundry man.

As discussed in the preceding section stateless nature of HTTP makes the server unable to determine if a specific visitor is a new user or old. So a mechanism is needed to mark the visitor (that is, hand him/her a "laundry token!") to keep a track. Cookies can help accomplish the same.

We have now come to observe how the original HTTP functionality can pose a challenge when dealing with the dynamic content on the web. Fact being that the web for the most of its part is dynamic in nature today; cookies are one of the mechanisms to make HTTP function in tandem with the new requirements of web applications.

5.1.1 What is a Cookie?
Cookie is a small text-only string file stored in the client’s memory. The pieces of information are stored as name-value pairs. The file is composed of plain text and no part of it is an executable code by definition. A web page or the server prompts the browser to store these files on the user’s machine. Web servers can later use the same file (cookie) to identify the individual user on subsequent visits.

For example, I visit facebook.com, and the site places a cookie on my machine. The cookie file for facebook.com contains the following information:

♦  UserID

♦  BSD77TSX8C8S7S9XZY

♦  www.facebook.com/

Facebook.com has stored on my machine a single name-value pair. The name of the pair is UserID, and the value is BSD77TSX8C8S7S9XZY (Session ID). The first time I visit facebook.com; the site assigns me a unique ID value (Session ID) and stores it on my machine.

A typical application of cookies can be observed on web sites that require credentials to login. The web sites often tend to store the credentials on a cookie (once they are initially verified after the user manually logs in). The cookie then allows the user to freely navigate the site without having to log in to it, at any point of time as long as the cookie is present and validated in the user’s system.

There are multiple reasons as to why web sites tend to use cookies,

• To personalize the user’s browsing session like My Yahoo etc.

• To assist users on online shopping platforms by saving history of prior browsed products. E.g. Amazon Books or eBay

• Collecting demographic information. E.g. DoubleClick

5.1.2 How do Web-Sites use Cookie?
As discussed already cookies help a Web site remember what state your browser is in. A cookie accomplishes this by allowing a site to store state information on your machine.

An USERID is a simple piece of state information. That is the presence of USERID on your system implies the website knows you are not a new visitor.

Let’s see the example in the image below to analyze the way cookies function.

The following scenario displays a simple login client (user) server interaction. Let’s define the scenario elaborately. Assume the user as “david” and the website as, say www.cookiedemo.com. A quick introduction to a few terms will enable a clear understanding of the demo.

• Session ID – A long alphanumeric string uniquely assigned to users and stored in the cookies which in turn are stored on their machine. The Session ID is used as the key or identifier when storing information about the user or session.

• Set-Cookie - The “setcookie” function sends an HTTP cookie to a client.

• Session - The session of activity that a user with a unique IP address spends on a Web site during a specified period of time.

Now we will follow the interaction in steps,

1. A user “david” in this case logs in to a site, www.cookiedemo.com.

2. If the credentials entered by “david” are validated (that is in case the username and password entered by david are authentic), the server sends a cookie to david’s browser via setcookie method.

3. At this point (after successful login, david has now initiated a session with the website). The cookie sent to david contains a SESSION ID.

4. An entry of this SESSION ID is also made on the server database.

5. The database contains a mapping of SESSION ID and corresponding username. In our case the Session Id in the figure is mapped to the username- david.

6. Now the next time the user david visits the website www.cookiedemo.com the cookie stored on his machine is fetched by the server and the entry of the SESSION ID and username pair is crosschecked with the pair entry on the server database.

7. On successful validation the user david is granted access and he can resume his session. The above example is also instrumental in demonstrating how the cookies function in making the ideally stateless HTTP protocol to behave otherwise. As per the functionality of HTTP under ideal circumstances the login attempts of user david in Step 1 and again in Step 6 would have been treated as independent HTTP requests requiring the user to manually login again every time he accessed the site. However with the cookies in the picture that does not seems to be the case and hence we observe how cookies address the shortcomings of original HTTP as pointed out earlier.



5.1.3 Fun Facts
• Navigator 1.0 was the first browser to implement cookies.

• A cookie is actually a well-known computer science term that is used when describing an opaque piece of data held by an intermediary.

6. Content Negotiation
As complex the term sounds, it is rather quite a simple mechanism. In essence it is the way of controlling the representations of information on web by determining the most appropriate way out of the existing representations. On the web, these existing and equivalent representations of information are referred to as “variants” and the mechanism itself is Content Negotiation.

6.1 Need for Content Negotiation
Content negotiation lends the browser the power and capability to share its preferences with regards to the information being requested from the server. The term negotiation can be applied to various aspects of the information like,

• Language as suited to the user

• Media type as supported by the browser

In order to meet the browser’s preferences the browser must first send its preferences to the server. For example if the browser is installed on a German-language based machine then it should indicate the server that it can accept data in German.

Similarly most common scenario of content negotiation arises in choosing the data based on types of media available. For example, the server may have image data in various formats say JPEG and GIF. The browser can indicate the server on what format is acceptable and may even in some cases prefer to JPEG over GIF as they are quicker to download.

URIs play an instrumental role in delivering content negotiation hence a short recap of URI can help to get a better insight on the topic.

6.2 Why URI?
Before proceeding further a clear understanding of the term URI is a pre-requisite. As discussed prior a URI is an identifier or a reference to a resource on the web and the filename or location itself. This is how it differs from a URL. By using a proper protocol, it is possible to retrieve the actual resource, that’s called dereferencing a URI. Both these terms are used interchangeably and hence it is important at this juncture to point out the difference. The abstraction offered by the URI in contrast to URL raises a fair question on the need of URI when in most cases the resource will happen to be stored in a file anyway, and the URI will be directly mapped to the file name. An analogy can help here. Consider a bar code for a product. A bar code is a reference to a product. In the case of the bar code and associated product, it is important to note that the product is not a specific box of cookies on a shelf, the referred product is actually the type of cookies of a certain brand, and all share the same bar code.

This is the same idea behind URIs. A URI refers to a resource on web, but the resource is not just one file on the web server. For example a resource say “Weather in Koblenz” can be a plain HTML document containing weather details of Koblenz or it can be an image representing a map with indicators about the weather data or it can be a video briefing on the same. Both these files can qualify as appropriate representations for the resource “Weather in Koblenz”.



6.3 Best Deal For Everyone
As discussed earlier content negotiation is all about deciding the best variant to offer from existing variants. Another analogy here can help. When discussing on phone while scheduling an appointment with the doctor. The conversation may go as follows,

You: Hello Doctor, Would it be fine to schedule the appointment on Wednesday or Friday?

Doctor: Friday sounds good, as I have a free slot.

Sounds simple right? Now just think of the day as “resource” and yourself as a “client requesting the resource” and the doctor as the “server” accepting the request based on the preferences of the client and its own availabilities. This is how content negotiation is implemented on web.

We can hence summarize that Content Negotiation serves the best variant of a resource, based on:

• Availability of the variant and server preferences.

• Preferences and limitations of client. In HTTP, this is achieved by the client sending, its request with specifications such as,

--> Accept headers (Accept, Accept-Language and Accept-Encoding), to communicate its capabilities and preferences in Format, Language and Encoding, respectively.

The best deal is determined by three factors,

•	Specific HTTP headers provided by the client – Server-Driven Negotiation

•	Using the 300- Multiple Choice response code or 406 Not Acceptable HTTP response codes by the server -- Agent-Driven Negotiation

•	A cache -- Transparent Negotiation

These are nothing but three types of content negotiation.

6.5 Agent-Driven Negotiation
Server-driven negotiation suffers from a few downsides:

• It not salable. There is one header per feature used in the negotiation. If one wants to use screen size, resolution or other dimensions, a new HTTP header must be created.

• Sending of the headers must be done on every request. This is not too problematic with few headers, but with the eventual multiplications of them, the message size would lead to a decrease in performance.

• The more headers are sent, the more entropy is sent, allowing for better HTTP fingerprinting and corresponding privacy concern.

HTTP allowed from the start another negotiation type, agent-driven negotiation. In this negotiation, when facing an ambiguous request, the server sends back a page containing links to the available alternative resources. The user is presented the resources and chooses the one to use.

Unfortunately, the HTTP standard does not specify the format of the page allowing to choose between the available resources, preventing to easily automatize the process. Beside fallback of the server-driven negotiation, this method is almost always used in conjunction with scripting, especially with JavaScript redirection: after having checked for the negotiation criteria, the script performs the redirection. A second problem is that one more request is needed in order to fetch the real resource, slowing the availability of the resource to the user.

6.6 Transparent Negotiation
Both agent and server-driven content negotiations have certain drawbacks. In order to get the best of both worlds, the HTTP/1.1 standard allowed then another type of content negotiation, transparent content negotiation.

A transparent content negotiation is initiated by the agent by adding a Negotiate: header. This header tells the server that the client is supporting this type of negotiation. In its response, the server will gives back the result of its server-driven content negotiation, i.e. the most-likely page according the Accept-*: headers sent by the agent in its request, and several HTTP headers (TCN:, Alternates: and Variant-Vary:) indicating what alternative pages does exists. If the browser find the alternatives more useful than the default page, it performs the second part a client-driven content negotiation and fetch the better page. That way, each time the server-driven content negotiation succeed only one HTTP request is needed, which is optimal. A second request is needed only when the Accept-*: headers do not convey enough information to let the server do an optimal choice.

Section 7: Conclusion
''In the conclusion you wrap up the most important points mentioned before on roughly half a page and provide a paragraph of outlook to later chapters. You may also mention important issues that have not been considered in this chapter (e.g. if UDP is not considered in the IP chapter).''

8: Quiz
TO BE COMPLETED.

9. References
Introduction
 * 1) https://developer.mozilla.org/en-US/docs/HTTP
 * 2) http://www.webopedia.com/

Cookies- Contributed to Cookie Central by David Whalen
 * 1) http://cscie12.dce.harvard.edu/lecture_notes/2011/20110504/handout.html
 * 2) http://www.cookiecentral.com/faq/
 * 3) http://www.nczonline.net/blog/2009/05/05/http-cookies-explained/

Stateless Nature of HTTP
 * 1) http://whatis.techtarget.com/definition/stateless
 * 2) http://stackoverflow.com/questions/4913763/what-does-it-mean-when-they-say-http-is-stateless

Nature of HTTP
 * 1) http://en.wikipedia.org/wiki/Request-response

HTTP Architecture
 * 1) http://www.ask.com/wiki/Hypertext_Transfer_Protocol#Technical_overview
 * 2) https://developer.mozilla.org/en-US/docs/HTTP/Headers
 * 3) http://www.jmarshall.com/easy/http/#resources

Content Negotiation
 * 1) https://developer.mozilla.org/en-US/docs/HTTP/Content_negotiation