Getting to Philosophy With Python
There’s a famous Wikipedia phenomena that by clicking the first link in the main text of the article on the English Wikipedia, you’ll eventually end up on the philosophy page. An explanation can be found here. Briefly, it’s because of Wikipedia Manual of Style guidelines that recommend that articles begin by telling “what or who the subject is, and often when and where”.
This was true for roughly 97% of articles, so there’s a big chance that by entering a random Wikipedia page and following the procedure you’ll indeed end up on Philosophy. I could test this by hand, but this wouldn’t be a dev.to article without writing some code. We’ll start with how to download Wikipedia articles.
How to get data
It’s simple - just request contents of and article with urllib3. Wikipedia follows a convenient pattern for naming its articles. After the usual en.wikipedia.org/ there’s a /wiki and then /article_name (or media! we’ll deal with that later) for example, en.wikipedia.org/wiki/Data_mining.
Firstly, I’ll create a pool from which I’ll make requests to Wikipedia.
import urllib3
from bs4 import BeautifulSoup
pool = urllib3.PoolManager()
From now on, I’ll could download the articles one by one. To automate the process of crawling through the site, the crawler will be recursive. Each iteration of it will return (current_url, [crawler for link_on site]), the recursion will stop, at given depth. In the end, I’ll end up with tree structure.
def crawl(
pool: urllib3.PoolManager,
url,
phrase=None,
deep=1,
sleep_time=0.5,
n=5,
prefix="https://en.wikipedia.org",
verbose=False,
):
"""
Crawls given Wikipedia `url` (article) with max depth `deep`. For each page
extracts `n` urls and if `phrase` is given check if `phrase` in urls.
Parameters
----------
pool : urllib3.PoolManager
Request pool
phrase : str
Phrase to search for in urls.
url : str
Link to wikipedia article
deep : int
Depth of crawl
sleep_time : float
Sleep time between requests.
n : int
Number of links to return
prefix : str, default="https://en.wikipedia.org""
Site prefix
Returns
-------
tuple
Tuple of url, list
"""
if verbose:
site = url.split("/")[-1]
print(f"{deep} Entering {site}")
# Sleep to avoid getting banned
time.sleep(sleep_time)
site = pool.request("GET", url)
soup = BeautifulSoup(site.data, parser="lxml")
# Get links from wiki (I'll show it later)
links = get_links_from_wiki(soup=soup, n=n, prefix=prefix)
# If phrase was given check if any of the links have it
is_phrase_present = any([phrase in link for link in links]) and phrase is not None
if deep > 0 and not is_phrase_present:
return (
url,
[
crawl(
pool=pool,
url=url_,
phrase=phrase,
deep=deep - 1,
sleep_time=sleep_time,
n=n,
prefix=prefix,
verbose=verbose,
)
for url_ in links
],
)
return url, links
If you read the code carefully, you’d notice a function get_links_from_wiki. get_links_from_wiki function parses the article. It works by finding a div that contains the whole article, then iterates through all paragraphs (or lists) and finds all links that match pattern /wiki/article_name. Because there’s no domain in that pattern, it is added at the end.
def get_links_from_wiki(soup, n=5, prefix="https://en.wikipedia.org"):
"""
Extracts `n` first links from wikipedia articles and adds `prefix` to
internal links.
Parameters
----------
soup : BeautifulSoup
Wikipedia page
n : int
Number of links to return
prefix : str, default="https://en.wikipedia.org""
Site prefix
Returns
-------
list
List of links
"""
arr = []
# Get div with article contents
div = soup.find("div", class_="mw-parser-output")
for element in div.find_all("p") + div.find_all("ul"):
# In each paragraph find all <a href="/wiki/article_name"></a> and
# extract "/wiki/article_name"
for i, a in enumerate(element.find_all("a", href=True)):
if len(arr) >= n:
break
if (
a["href"].startswith("/wiki/")
and len(a["href"].split("/")) == 3
and ("." not in a["href"] and ("(" not in a["href"]))
):
arr.append(prefix + a["href"])
return arr
Now we have everything to check the phenomena. I’ll set max depth to 50 and set n=1 (to only expand first link in the article).
crawl(pool, "https://en.wikipedia.org/wiki/Doggart", phrase="Philosophy", deep=50, n=1, verbose=True)
Output:
50 Entering Doggart
49 Entering Caroline_Doggart
48 Entering Utrecht
47 Entering Help:Pronunciation_respelling_key
...
28 Entering Mental_state
27 Entering Mind
26 Entering Thought
25 Entering Ideas
('https://en.wikipedia.org/wiki/Doggart',
[('https://en.wikipedia.org/wiki/Caroline_Doggart',
[('https://en.wikipedia.org/wiki/Utrecht',
[('https://en.wikipedia.org/wiki/Help:Pronunciation_respelling_key',
[('https://en.wikipedia.org/wiki/Pronunciation_respelling_for_English',
...
[('https://en.wikipedia.org/wiki/Ideas',
['https://en.wikipedia.org/wiki/Philosophy'])])])])])])])])])])])])])])])])])])])])])])])])])])
As you can see after 25 iterations indeed we found Philosophy page.
Found this post interesting? Check out my Github @finloop