---
title: "Web Crawler with Python Twisted"
description: "Building an asynchronous web crawler using Python Twisted framework. Practical notes and code examples on Web Crawler with."
url: "https://h3manth.com/scribe/web-crawler-with-python-twisted/"
canonical_url: "https://h3manth.com/scribe/web-crawler-with-python-twisted/"
markdown_url: "https://h3manth.com/scribe/blog/web-crawler-with-python-twisted.md"
publishDate: "2013-04-26T00:00:00.000Z"
author: "Hemanth HM"
category: "python"
tags: ["python","twisted","web-crawler"]
word_count: 123
reading_time_minutes: 1
x_markdown_tokens: 397
content_signal: "ai-train=yes, search=yes, ai-input=yes"
---

# Web Crawler with Python Twisted

> Building an asynchronous web crawler using Python Twisted framework. Practical notes and code examples on Web Crawler with.

A simple HTTP crawler built with Python [Twisted](http://twistedmatrix.com/trac/).

```python
#!/usr/bin/env python
from twisted.web.client import getPage
from BeautifulSoup import BeautifulSoup
from twisted.python import log
from twisted.internet import defer, task
import re

def parallel(iterable, count, callable, *args, **named):
    coop = task.Cooperator()
    work = (callable(elem, *args, **named) for elem in iterable)
    return defer.DeferredList([coop.coiterate(work) for i in xrange(count)])

def union(p, q):
    for e in p:
        if e not in q:
            print e
            q.append(e)

def extractLinks(html):
    soup = BeautifulSoup(html)
    soup.prettify()
    return [
        str(anchor['href']) 
        for anchor in soup.findAll('a', attrs={'href': re.compile("^http://")})
        if anchor['href']
    ]

def crawlPage(url, urlList):
    d = getPage(url)
    d.addCallback(extractLinks)
    d.addCallback(union, urlList)
    d.addErrback(log.err)
    return d

def main(reactor, *args):
    urls = list(args)
    return parallel(urls, len(urls), crawlPage, urls)

if __name__ == '__main__':
    task.react(main, ["http://h3manth.com", "http://www.test.com"])
```

Requires: PyOpenSSL and Twisted 12.3+