This is a simple test/benchmark I wrote to test out the new urllib3 library that I found on the internets today. I currently use the excellent httplib2, so I used it for comparison.
A typical run looks something like this:
simple sequential download, first run: urllib3 took 2.396 seconds httplib2 took 1.851 seconds simple sequential download, second run: urllib3 took 1.705 seconds httplib2 took 0.104 seconds now, urllib3 with 4 threads and 4 connections: took 1.124 seconds
Each timed run downloads 15 URLs, all from the same google domain.
In the first sequential benchmark, httplib2 and urllib3 are pretty even. Sometimes one is faster, sometimes the other is. It's a wash. This makes sense because they both have persistent connections.
In the second sequential benchmark, httplib2 is always much faster, due to it pulling everything from its on-disk cache.
In the last benchmark, urllib3 tends to execute in 1/3 to 1/2 the time it would sequentially, by using a thread pool and a connection pool, each with a size of 4. There is no comparison with httplib2 here, because httplib2 currently is not thread-safe.