c# webcrawling basics

To download the metadata and the content of a website you have some opportunities like this for example:

HttpWebRequest httpWebRequest = (HttpWebRequest)HttpWebRequest.Create(url);
WebRequest webRequest = (WebRequest)httpWebRequest;
WebResponse webResponse = webRequest.GetResponse();
StreamReader sr = new StreamReader(webResponse.GetResponseStream());
string htmlStr = sr.ReadToEnd();

While doing this you create a http-web-request and a web-response. With the stream-reader you will get the metadata and the content of the website and are able to write this into string.

Advertisements