cython并行性能-計算滾動求和 rolling function
cython通過編譯為C程序提高性能有很多例子,通過OpenMP并行的性能沒那么多。
今天嘗試了一下似乎gcc對parallelism reduction優化的很厲害,加上OpenMP并行可以提高20倍性能(相對于pandas rolling),這不是簡單的2 core帶來的性能提高。
滾動求和 rolling sum的例子
最簡單的實現pandas.rolling,通過操作numpy array,速度也還算能接受。
# test_para.py
import numpy as np
#import pyximport; pyximport.install(reload_support=True, setup_args={"include_dirs":np.get_include()})
import timeit
import pandas as pd
import para.cpara as cpara
X = -1 + 2*np.random.rand(100000)
ss = pd.Series(X)
ss.rolling(100).apply(np.sum,raw=True)
print('==============')
print('multi thread')
start_time = timeit.default_timer()
sum_cython=pd.Series(cpara.window_sum(X, 100))
print(timeit.default_timer() - start_time)
print('single thread')
start_time = timeit.default_timer()
sum_pandas=ss.rolling(100).apply(np.sum,raw=True)
print(timeit.default_timer() - start_time)
print(np.max(np.abs(sum_cython - sum_pandas)))
cython源文件
# cpara.pyx
cimport cython
import numpy as np
from cython.parallel import prange,parallel
cimport numpy as cnp
from libc.stdlib cimport malloc
@cython.boundscheck(False)
def window_sum(cnp.ndarray[double, ndim=1] arr, int window):
cdef h = np.zeros_like(arr)
cdef int imax = arr.shape[0]
cdef double *buffer = <double *>malloc(imax * sizeof(double))
cdef double result = 0.0
cdef int i, j
with nogil, parallel():
for i in prange(imax, schedule='dynamic'):
buffer[i] = 0.0
if i >= window-1:
for j in range(window):
buffer[i] += arr[i-j]
for i in range(imax):
if i < window -1:
h[i] = np.nan
else:
h[i] = buffer[i]
return h
setup.py中要加入openmp的編譯鏈接參數
EXT = Extension("*",
["para/*.pyx"],
define_macros=[('CYTHON_TRACE', CYTHON_DEBUG),
('CYTHON_TRACE_NOGIL', CYTHON_DEBUG),
('CYTHON_BINDING', CYTHON_DEBUG),
("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION"),
('CYTHON_FAST_PYCCALL', '1')],
extra_compile_args = ["-fopenmp" ],
extra_link_args=['-fopenmp'],
include_dirs=[".", np.get_include()])
性能比較
%timeit pd.Series(cpara.window_sum(X, 100))
23.4 ms ± 325 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)
%timeit ss.rolling(100).apply(np.sum,raw=True)
536 ms ± 3.96 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
536/23.4=22.9
測試環境:i3-7100U 2core 2T CPU, ubuntu 18.04 LTS

浙公網安備 33010602011771號