Python多線程并發模型在處理CPU密集任務時退化為多線程串行執行

原因

由于GIL的存在，同一時刻只有一個線程的字節碼可以被Python解釋器執行，
因此無論CPU有多個核心，程序執行的是CPU密集任務還是I/O密集型任務，多個線程也無法并行執行。

線程被I/O阻塞的時候會釋放GIL，因此多線程并發執行I/O密集型任務的時候，并發度尚可。

多線程執行純CPU任務的時候，不會釋放GIL，因此多線程執行純CPU任務的線程并發執行時，接近退化為串行執行。

線程池版本-代碼運行結果

(base) sgj@sgj-laptop:~/my-ostep/chap26/cpu-busy$ python thread_pool.py
Seconds per round: 1.0589
Total rounds: 55
Total real time:66.0324
Total theory time: 1.0589 * 55 = 58.2401

線程池版本

#!/usr/bin/env python

import time
import sys
from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import as_completed


def spin(duration):
    i = 0
    while i < 1.18 * duration* 10**7:
        i += 1
    return duration

def main():
    # CPU密集任務中, Python 的多線程并發會退化為多線程串行
    t1 = time.time()
    spin(1)
    t2 = time.time()
    one = t2 - t1
    print(f"Seconds per round: {one:.4f}")

    durations = list(range(1, 11))
    with ThreadPoolExecutor() as e:
        start = time.time()
        futures = [e.submit(spin, x) for x in durations]
        gen = as_completed(futures)
        for f in gen:
            pass
        end = time.time()
        total_rounds = sum(durations)
        print(f"Total rounds: {total_rounds}")
        print(f"Total real time:{(end - start):.4f}")
        print(f"Total theory time: {one:.4f} * {total_rounds} = {(one * total_rounds):.4f}")

if __name__ == '__main__':
    main()

普通多線程版本-代碼運行結果

(base) sgj@sgj-laptop:~/my-ostep/chap26$ python threads_threading.py
Seconds per round: 1.0434
Total real time:64.4664
Total rounds: 55
Total theory time: 1.0434 * 55 = 57.3856

普通多線程版本

#!/usr/bin/env python

import time
import sys
import threading


def spin(duration):
    i = 0
    while i < 1.18 * duration* 10**7:
        i += 1
    return duration

def main():
    # CPU密集任務中, Python 的多線程并發會退化為多線程串行
    t1 = time.time()
    spin(1)
    t2 = time.time()
    one = t2 - t1
    print(f"Seconds per round: {one:.4f}")

    durations = list(range(1, 11))
    lst = list()
    for duration in durations:
        t = threading.Thread(target=spin, args=(duration,))
        lst.append(t)

    start = time.time()
    for t in lst:
        t.start()

    for t in lst:
        t.join()
    end = time.time()

    total_rounds = sum(durations)
    print(f"Total real time:{(end - start):.4f}")
    print('Total rounds:', sum(durations))
    print(f"Total theory time: {one:.4f} * {total_rounds} = {(one * total_rounds):.4f}")

if __name__ == '__main__':
    main()

進程池版本-代碼運行結果

(base) sgj@sgj-laptop:~/my-ostep/chap26/cpu-busy$ python process_pool.py
Seconds per round: 1.1104
Total real time:22.4188
Total rounds: 55
Total theory time: 1.1104 * 55 = 61.0702

進程池版本

#!/usr/bin/env python

import time
import sys
from concurrent.futures import ProcessPoolExecutor
from concurrent.futures import as_completed


def spin(duration):
    i = 0
    while i < 1.18 * duration* 10**7:
        i += 1
    return duration

def main():
    # python的多進程模型在多核CPU上可以并行執行
    t1 = time.time()
    spin(1)
    t2 = time.time()
    one = t2 - t1
    print(f"Seconds per round: {one:.4f}")

    durations = list(range(1, 11))
    with ProcessPoolExecutor() as e:
        start = time.time()
        futures = [e.submit(spin, duration) for duration in durations]
        gen = as_completed(futures)
        for f in gen:
            pass
        end = time.time()
        total_rounds = sum(durations)
        print(f"Total real time:{(end - start):.4f}")
        print(f"Total rounds: {total_rounds}")
        print(f"Total theory time: {one:.4f} * {total_rounds} = {(one * total_rounds):.4f}")

if __name__ == '__main__':
    main()

普通多進程版本-代碼運行結果

Python的多進程模型不受GIL的限制，我的代碼中創建了10個進程，但我的電腦只有8個核心，所以總的運行時間為20.6785
如果我的電腦有10個核心，可以預見總運行時間在會在15秒左右

(base) sgj@sgj-laptop:~/my-ostep/chap26$ python process_multiprocessing.py
Seconds per round: 1.0555
Total real time:20.6785
Total rounds: 55
Total theory time: 1.0555 * 55 = 58.0537

普通多進程版本

#!/usr/bin/env python

import time
import sys
from multiprocessing import Process

def spin(duration):
    i = 0
    while i < 1.18 * duration* 10**7:
        i += 1
    return duration

def main():
    # python的多進程模型在多核CPU上可以并行執行
    t1 = time.time()
    spin(1)
    t2 = time.time()
    one = t2 - t1
    print(f"Seconds per round: {one:.4f}")

    durations = list(range(1, 11))
    lst = list()
    for duration in durations:
        p = Process(target=spin, args=(duration,))
        lst.append(p)

    start = time.time()
    for p in lst:
        p.start()

    for p in lst:
        p.join()
    end = time.time()

    total_rounds = sum(durations)
    print(f"Total real time:{(end - start):.4f}")
    print('Total rounds:', sum(durations))
    print(f"Total theory time: {one:.4f} * {total_rounds} = {(one * total_rounds):.4f}")

if __name__ == '__main__':
    main()

posted @ 2025-08-14 23:22 Guanjie255 閱讀(9) 評論(0) 收藏舉報

刷新頁面返回頂部

Guanjie Sun

Python多線程并發模型在處理CPU密集任務時退化為多線程串行執行

原因

線程池版本-代碼運行結果

普通多線程版本-代碼運行結果

進程池版本-代碼運行結果

普通多進程版本-代碼運行結果

公告