Web scraping is scraping data from online, as long as it is allowed by the websites. The workflow of web scraping not only includes getting data online but also includes the process of turning the data into something readable and usable since in most cases, the data scraped are unstructured. Specifically, the steps of web scraping are:
- locate the URL for which you want to scrape data from;
- Inspect the webpage to identify the “Tags”/“Path”/“Selector”/“Attributes” of the content you want to scrape;
- Write the code and make sure your code works;
- Generalize your codes to scrape more webpages, but make sure to not let yourself be blocked. I will talk about some tips in the last section.
In this article, I will demonstrate the steps by web scraping Yahoo Finance news. Yahoo Finance provides numerous financial data for companies, including their stock prices, balance sheet statements, etc, and they are web-scraping friendly. There are a lot of tutorials talking about scraping numerical data from Yahoo Finance, this article will discuss and follow the web scraping steps above to extract news article and related information from Yahoo Finance.
