1. Python Web Scraping With Beautiful Soup
2. Web Scraping With Beautifulsoup
3. How To Use Beautiful Soup

Web scraping is the technique to extract data from a website.

The module BeautifulSoup is designed for web scraping. The BeautifulSoup module can handle HTML and XML. It provides simple method for searching, navigating and modifying the parse tree.

This technique is called web scraping or web harvesting or web data extraction. This article discusses the steps involved in web scraping using the implementation of a Web Scraping framework of Python called Beautiful Soup. Steps involved in web scraping: Send an HTTP request to the URL of the webpage you want to access. You’ll find that Beautiful Soup will cater to most of your parsing needs, from navigating to advanced searching through the results. In this tutorial, you’ve learned how to scrape data from the Web using Python, requests, and Beautiful Soup. You built a script that fetches job postings from the Internet and went through the full web scraping process from start to finish.

Related course:
Browser Automation with Python Selenium

Get links from website

The example below prints all links on a webpage:

It downloads the raw html code with the line:

A BeautifulSoup object is created and we use this object to find all links:

Extract links from website into array

To store the links in an array you can use:

Function to extract links from webpage

Python Web Scraping With Beautiful Soup

If you repeatingly extract links you can use the function below:

Related course:
Browser Automation with Python Selenium

• Searching the Tree

Introduction

Before reading it, please read the warnings in my blog Learning Python: Web Scraping.

A brief introduction of Beautiful Soup can be found in my blog Learning Python: Web and Databases.It creates a parse tree for parsed pages that can be used to extract data from HTML, which is useful for web scraping.

Create a BeautifulSoup object that represents the document as a nested data structure.

Beautiful Soup supports the HTML parser included in Python’s standard library, but it also supports a number of third-party Python parsers. One is the lxml parser. For instance, BeautifulSoup(markup, 'lxml').It is very fast and lenient.

Web Scraping With Beautifulsoup

Objects in Beautiful Soup

Beautiful Soup transforms a complex HTML document into a complex tree of Python objects.The objects are mainly four kinds: Tag, NavigableString, BeautifulSoup, and Comment.

1. BeautifulSoup: the BeautifulSoup object itself represents the document as a whole.
2. Tag: a Tag object corresponds to an XML or HTML tag in the original document. Every tag has a name (accessible as .name) and any number of attributes (accessible by treating like a dictionary).
3. NavigableString: a string corresponds to a bit of text within a tag. You can’t edit a string in place, but you can replace one string with another, using replace_with().
4. Comment: the Comment object is just a special type of NavigableString.

Beautiful Soup defines classes for anything else that might show up in an XML document: CData, ProcessingInstruction, Declaration, and Doctype.

Navigating the Tree

In a HTML/XML document, tag may contain texts and other tags.Beautiful Soup provides many attributes for navigating and iterating over tree.

1. Directly use the name of the tag. Using a tag name as an attribute will give you only the first tag by that name.
2. .contents give a list of a tag’s direct children.
3. .children generator can be used to iterate over a tag’s direct children.
4. .descendants allows to iterate over all of a tag’s children recursively including its direct children, the children of its direct children and so on.
5. .string gives the text in this tag if it has only one NavigableString child. It gives the text in a tag’s child if this tag has only one tag child and this tag child has a string. If a tag contains more than one thing, it is no clear and is defined to be None.
6. .parent can access an element’s parent.
7. .parents can iterate over all of an element’s parents.
8. .next_sibling and .previous_sibling can navigate between elements that are on the same level.
9. .next_siblings and .previous_siblings can iterate over a tag’s siblings.
10. .next_element and .previous_element can navigate to the next or previous element of a tag. It is different from .next_sibling and .previous_sibling.

Searching the Tree

Beautiful Soup also provides many methods for searching the tree.Two main methods are find() and find_all().

Some basic usage of the methods:

1. use a string;
2. use regular expression;
3. use a list;
4. use True;
5. use a self defined function.

There are some arguments for these two methods to use.

How To Use Beautiful Soup

Method: find_all()

find_all(name, attrs, recursive, string, limit, **kwargs):

1. name: it can be a string, a regular expression, a list a function or True.
2. attrs: Any argument that’s not recognized will be turned into a filter on one of a tag’s attributes. Sometimes, the attributes cannot be used as a keyword argument. Then use attrs to pass attribute name and its value. It would be a little different if you want to search by CSS class. It uses the keyword argument class_.
3. recursive: if you set this value to False like recursive=False, it will only search through the direct children instead of all the descendants.
4. string: with it you can search for strings instead of tags. It also accepts a string, a regular expression, a list, a function or True.
5. limit: it limits the number of results.

Method: find()

find(name, attrs, recursive, string, **kwargs): it only returns the first result.If find() can’t find anything, it returns None instead of an empty list.

Other Methods

There are some other methods: find_parents(), find_parent(), find_next_siblings(), find_next_sibling(), find_previous_siblings(), find_previous_sibling(), find_all_next(), find_next(), find_all_previous() and find_previous(). They are all similar. So I will not describe them in detail here.

As of version 4.7.0, Beautiful Soup supports most CSS4 selectors via the SoupSieve project.

Example

KassiesA: UEFA European Cup Football contains a lot of soccer data for the matches of UEFA Champions League and Europa League.

I will give an example using Beautiful Soup to extract the results of all the matches in UEFA European Cup Matches 2017/2018.

The HTML content in the page looks like:

Based on the structure of the page, I develop a simple program to save all the match results in file with CSV format.

Some parts of the file:

Then we can use them for our own analysis.

Further

Besides parsing tree, Beautiful Soup also allows to modify the tree and write the changes as a new HTML or XML document.

More information in detail can be found in Beautiful Soup Official Documentation.

References

Please enable JavaScript to view the comments powered by Disqus.blog comments powered by Disqus