A sitemap is a list of pages available on a website. This was used to help visitors to provide an overall navigation of the website, but now sitemaps are mainly used for listing pages and their updates to search engines. Sitemaps are no more in HTML format, but in plain text format or commonly HTML.

This article explains another technique in Web Scraping, which has been an important function of search engine crawling - extraction of all links for any given URL. This is quite a simple procedure, especially when done using Regular Expressions. The HTML content from any given url is downloaded as a string, and all occurences of hyperlinks are extracted from it.