Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added CPU tests/1 worker time.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added CPU tests/1 worker.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added CPU tests/10 workers time.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added CPU tests/10 workers.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added CPU tests/2 workers time.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added CPU tests/2 workers.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added CPU tests/4 workers time.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added CPU tests/4 workers.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added CPU tests/6 cores.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
22 changes: 22 additions & 0 deletions CPU-bound.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
from hashlib import md5
from random import choice, randint
import concurrent.futures


def is_coin(n):
while True:
s = "".join([choice("0123456789") for i in range(50)])
h = md5(s.encode('utf8')).hexdigest()

if h.endswith("00000"):
return s + ' ' + h


def main():
with concurrent.futures.ProcessPoolExecutor(max_workers=10) as executor:
for coin in zip(executor.map(is_coin, range(9))):
print(coin)


if __name__ == '__main__':
main()
Binary file added IO tests/10 workers time.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added IO tests/10 workers.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added IO tests/100 workers time.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added IO tests/100 workers.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added IO tests/5 workers time.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added IO tests/5 workers.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added IO tests/synchronous ask time.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added IO tests/synchronous ask.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
19 changes: 19 additions & 0 deletions IO-bound.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
from urllib.request import urlopen
from urllib.parse import unquote
from bs4 import BeautifulSoup
from tqdm import tqdm

url = 'https://ru.wikipedia.org/wiki/%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F:%D0%A1%D0%BB%D1%83%D1%87' \
'%D0%B0%D0%B9%D0%BD%D0%B0%D1%8F_%D1%81%D1%82%D1%80%D0%B0%D0%BD%D0%B8%D1%86%D0%B0 '

res = open('res.txt', 'w', encoding='utf8')

for i in tqdm(range(100)):
html = urlopen(url).read().decode('utf8')
soup = BeautifulSoup(html, 'html.parser')
links = soup.find_all('a')

for l in links:
href = l.get('href')
if href and href.startswith('http') and 'wiki' not in href:
print(href, file=res)
51 changes: 51 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,3 +190,54 @@ while True:
* Во время работы посмотрите с использованием стандартных утилит вашей OC загрузку памяти, процессора, сети, время работы. Зависят ли они от количества воркеров и как?
* Убедитесь в том, что так как задача CPU bound, наращивать количество воркеров, большее количества ядер, бесполезно.

## Тесты для IO-bound

![synchronous ask](https://github.com/Miarel/multi/blob/main/IO%20tests/synchronous%20ask.png)

![synchronous ask time](https://github.com/Miarel/multi/blob/main/IO%20tests/synchronous%20ask%20time.png)

* Синхронная проверка ссылок. Так как ссылок очень много(1000+) процесс занимает примерно 23 минуты. Сильной нагрузги на процессор и память нет. Нагрузка на сеть также очень низкая в пределах от 0.1 мбит/c до 0.6 мбит/c.

![5 workers](https://github.com/Miarel/multi/blob/main/IO%20tests/5%20workers.png)

![5 workers time](https://github.com/Miarel/multi/blob/main/IO%20tests/5%20workers%20time.png)

* 5 воркеров. Нагрузка на процессор и память осталась примерно такой-же как и при синхронной проверке, но увеличилась нагрузка на сеть(от 0.5 мбит/c до 6 мбит/c). Время работы ощутимо уменьшилось в сравнении с синхронной проверкой.

![10 workers](https://github.com/Miarel/multi/blob/main/IO%20tests/10%20workers.png)

![10 workers time](https://github.com/Miarel/multi/blob/main/IO%20tests/10%20workers%20time.png)

* 10 воркеров. Относительно 5 воркеров слегка увеличилась нагрузка на память, но незначительно, нагрузка на сеть также осталась в пределах от 0.5 мбит/c до 6 мбит/c. Время работы сократилось.

![100 workers](https://github.com/Miarel/multi/blob/main/IO%20tests/100%20workers.png)

![100 workers time](https://github.com/Miarel/multi/blob/main/IO%20tests/100%20workers%20time.png)

* 100 воркеров. Относительно 10 воркеров слегка увеличилась нагрузка на память, но незначительно, нагрузка на сеть также осталась в пределах от 0.5 мбит/c до 6 мбит/c. Время работы сократилось.

## Тесты для CPU-bound

* 1 воркер

![1 worker](https://github.com/Miarel/multi/blob/main/CPU%20tests/1%20worker.png)
![1 worker time](https://github.com/Miarel/multi/blob/main/CPU%20tests/1%20worker%20time.png)

* 2 воркера

![2 workers](https://github.com/Miarel/multi/blob/main/CPU%20tests/2%20workers.png)
![2 workers time](https://github.com/Miarel/multi/blob/main/CPU%20tests/2%20workers%20time.png)

* 4 воркера

![4 workers](https://github.com/Miarel/multi/blob/main/CPU%20tests/4%20workers.png)
![4 workers time](https://github.com/Miarel/multi/blob/main/CPU%20tests/4%20workers%20time.png)

* 10 воркеров

![10 workers](https://github.com/Miarel/multi/blob/main/CPU%20tests/10%20workers.png)
![10 workers time](https://github.com/Miarel/multi/blob/main/CPU%20tests/10%20workers%20time.png)

Скриншоты идут в последовательности увелечения количества воркеров. 1 воркер создвет примерно 12.4% нагрузки на процессор и 8.4 мб на память. Следовтельно с увеличением количества воркеров нагрузка пропорционально растет. Тесты производились на ПК с шестиядерным процессором, поэтому при попытках указать более 6 воркеров нагрузка идет только на 6 ядер.

![6 cores](https://github.com/Miarel/multi/blob/main/CPU%20tests/6%20cores.png)
Loading