Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
1.1k views
in Technique[技术] by (71.8m points)

multithreading - Understanding python GIL - I/O bound vs CPU bound

From python threading documentation

In CPython, due to the Global Interpreter Lock, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation). If you want your application to make better use of the computational resources of multi-core machines, you are advised to use multiprocessing. However, threading is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously.

Now I have a thread worker like this

def worker(queue):
    queue_full = True
    while queue_full:
        try:
            url = queue.get(False)
            w = Wappalyzer(url)
            w.analyze()
            queue.task_done()

        except Queue.Empty:
            queue_full = False

Here w.analyze() doing two things

  1. Scrape the url using requests library
  2. Analyzing the scraped html using pyv8 javascript library

As far as I know, 1 is I/O bound and 2 is CPU bound.

Does that mean, GIL applied for 2 and my program won't work properly?

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Answer

0 votes
by (71.8m points)

The GIL description does not say anything about correctness, only about efficiency.

If 2 is CPU bound, you will not be able to get multicore performance out of threading, but your program will still perform correctly.

If you care about CPU Parallelism, you should use Python's multiprocessing library.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...