Improved search efficiency
Content crawling method improves the search engine efficiency.Google, All search engines like Google, Yahoo updating their
content crawling method, MSN also joins the party. To improve the efficiency of its ability to
crawl and index your websites, live search has made
many updates in the crawler.
Two important updates are made:
1.‘HTTP compression’ and
2.‘Conditional get’.
HTTP compression:
“HTTP compression makes faster transmission time. Faster Transmission time is possible with compressing static files and application responses, reducing network load between your servers and our crawler”-said Fabrice Canel, of live search crawling team.. This feature is now supported by most of the browsers and popular search engines like Google.
Conditional Get:
As stated in official Live Search blog, “Last time we crawled a page and then some changes are made to that page, in that case we will not download the page unless it has changed. Our crawler will include the "If-Modified-Since" header and last download time in the GET request and when available, our crawler will include in the GET request "If-None-Match" header and the ETag value. Web server will respond with a 304 HTTP response, if the content hasn't changed.” Fabrice has given an explanation on how to configure the Conditional get in your website and it is very simple to do. With the help of online tools, you can simply check whether your website already supports the "If-Modified-Since.
You can check it:
For Internet Explorer - using Fiddler
(or)
for Firefox - using Live Headers .
By using with these tools you can create a custom GET request and send it to your server.
Note:Make sure that your request includes the "If-Modified-Since" header.
A sample “If modified-Since” header-Given by Fabrice
“GET /sa/3_12_0_163076/webmaster/webmaster_layout.css HTTP/1.1
Host: webmaster.live.com
If-Modified-Since: Tue, 22 Jan 2008 01:28:49 GMT
You will get a server response to the following simplified sample:
"HTTP/1.x 304 Not Modified”
“If you have not yet configured conditional get on your site, we would strongly encourage you to do so, as it can significantly help reduce server load as most browsers and crawlers already support this feature” says Fabrice Canel.
Live search has also upgraded the user agent to "msnbot/1.1" to optimize the crawling. According to official MSN space, there are many more updates that are on the way.
Two important updates are made:
1.‘HTTP compression’ and
2.‘Conditional get’.
HTTP compression:
“HTTP compression makes faster transmission time. Faster Transmission time is possible with compressing static files and application responses, reducing network load between your servers and our crawler”-said Fabrice Canel, of live search crawling team.. This feature is now supported by most of the browsers and popular search engines like Google.
Conditional Get:
As stated in official Live Search blog, “Last time we crawled a page and then some changes are made to that page, in that case we will not download the page unless it has changed. Our crawler will include the "If-Modified-Since" header and last download time in the GET request and when available, our crawler will include in the GET request "If-None-Match" header and the ETag value. Web server will respond with a 304 HTTP response, if the content hasn't changed.” Fabrice has given an explanation on how to configure the Conditional get in your website and it is very simple to do. With the help of online tools, you can simply check whether your website already supports the "If-Modified-Since.
You can check it:
For Internet Explorer - using Fiddler
(or)
for Firefox - using Live Headers .
By using with these tools you can create a custom GET request and send it to your server.
Note:Make sure that your request includes the "If-Modified-Since" header.
A sample “If modified-Since” header-Given by Fabrice
“GET /sa/3_12_0_163076/webmaster/webmaster_layout.css HTTP/1.1
Host: webmaster.live.com
If-Modified-Since: Tue, 22 Jan 2008 01:28:49 GMT
You will get a server response to the following simplified sample:
"HTTP/1.x 304 Not Modified”
“If you have not yet configured conditional get on your site, we would strongly encourage you to do so, as it can significantly help reduce server load as most browsers and crawlers already support this feature” says Fabrice Canel.
Live search has also upgraded the user agent to "msnbot/1.1" to optimize the crawling. According to official MSN space, there are many more updates that are on the way.
