作用:使用计算机的多核
python中的多线程其实并不是真正的多线程,如果想要充分地使用多核CPU的资源,在python中大部分情况需要使用多进程。
Python 的 multiprocessing
库用于创建多进程程序。它提供了一个易于使用的接口来并行运行多个进程,从而充分利用多核处理器的计算能力。该库允许你创建进程、管理进程池、共享内存和通信。它适用于执行CPU密集型任务,如科学计算、图像处理等,可以显著提高程序的运行效率,是实现并行计算的重要工具。
import multiprocessing
print (multiprocessing.cpu_count())
import multiprocessing as mp
def job(a,b):
print("aaaaa")
if __name__=="__main__":
p1 = mp.Process(target = job,args = (1,2))
p1.start()
p1.join()
import multiprocessing as mp # 引入multiprocessing
def job(a,b):
print("aaaaa")
if __name__=="__main__":
p1 = mp.Process(target = job,args = (1,2)) # 指定方法
p1.start() # 开始工作
p1.join() # 主进程等待进程池中的所有子进程结束
p1 = mp.Process(target = job,args = (1,2)) # 指定方法理解成要for循环开启
import multiprocessing as mp
def job(q):
res = 0
for i in range(1000):
res += i+i**2+i**3
q.put(res)
if __name__=="__main__":
q = mp.Queue()
p1 = mp.Process(target = job,args = (q,))
p2 = mp.Process(target = job,args = (q,))
p1.start()
p2.start()
p1.join()
p2.join()
res1 = q.get()
res2 = q.get()
print(res1)
print(res2)
print(res1 + res2)
import multiprocessing as mp
def job(q):
res = 0
for i in range(1000):
res += i+i**2+i**3
q.put(res) # 3、获得内容
if __name__=="__main__":
q = mp.Queue() # 1、实例化Queue
p1 = mp.Process(target = job,args = (q,)) # 2、传入Queue
p2 = mp.Process(target = job,args = (q,))
p1.start()
p2.start()
p1.join()
p2.join()
res1 = q.get() # 4、获得内容
res2 = q.get()
print(res1)
print(res2)
print(res1 + res2)
import multiprocessing as mp
def job(x):
return x*x
def multicore():
pool = mp.Pool()
res = pool.map(job,range(100))
print(res)
if __name__=="__main__":
multicore()
import multiprocessing as mp
def job(x):
return x*x
def multicore():
pool = mp.Pool() # 1、实例化
res = pool.map(job,range(100)) # 2、指定任务
print(res)
if __name__=="__main__":
multicore()
import multiprocessing as mp
def job(x):
return x*x
def multicore():
pool = mp.Pool(processes = 2) # 1、实例化,核数,不写就全部一起
res = pool.map(job,range(100))
print(res)
if __name__=="__main__":
multicore()
import multiprocessing
import time
def func(name):
print("start: %s" % name)
time.sleep(2)
return "end: %s" % name
if __name__ == "__main__":
name_list = ["winter", "elly", "james", "yule"]
res_list = []
pool = multiprocessing.Pool(3) # 创建一个进程总数为3的进程池
for member in name_list:
res = pool.apply(func, (member,)) # 创建子进程,并执行,不需要start
print(res)
pool.close()
pool.join() # 调用join之前,先调用close函数,否则会出错。执行完close后不会有新的进程加入到pool
print("all done...")
start: winter
end: winter
start: elly
end: elly
start: james
end: james
start: yule
end: yule
all done...
import multiprocessing
import time
def func(name):
print("start: %s" % name)
time.sleep(2)
return "end: %s" % name
def func_exp(msg):
print("callback: %s" % msg)
if __name__ == "__main__":
name_list = ["winter", "elly", "james", "yule"]
res_list = []
pool = multiprocessing.Pool() # 创建一个进程总数为3的进程池
for member in name_list:
res = pool.apply_async(func, (member,), callback=func_exp) # 创建子进程,并执行,不需要start
res_list.append(res) #注意这里是append了res,不是res.get(),不然又要阻塞了
for res_mem in res_list:
print(res_mem.get())
pool.close()
pool.join() # 调用join之前,先调用close函数,否则会出错。执行完close后不会有新的进程加入到pool
print("all done...")
start: winter
start: elly
start: james
start: yule
callback: end: winter
end: winter
callback: end: elly
end: elly
callback: end: james
end: james
callback: end: yule
end: yule
all done...
def func(i):
mysqldb = pymysql.connect(host="127.0.0.1", user="root", password="123456", db="data",
port=3306, charset="utf8")
mysqlcursor2 = mysqldb.cursor() # 获取指针以操作数据库