Skip to content

Commit 3858f3a

Browse files
v2.4.0rc8 (#1052)
* [Windows] Remove `--threadless` from `TestCase` (#1025) [Windows] Remove `--threadless` from `TestCase` * Handle `SIGINFO`. Try `kill -s INFO <pid>`. (#1024) * Handle `SIGINFO`. Try `kill -s INFO <pid>`. Also remove dirty hack added in flags to incorporate `--basic-auth` flag. Add `__pycache__` to ignore list. Disable http proxy during acceptor benchmark. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * not on windows * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * no cover * # pylint: disable=E1101 * Curl retry on error and check_output 2 minute timeout * check output timeout None * Update `faulthandler_timeout` to 2 minutes * Disable `test_circular_imports`, `isort` integration now works * Fix curl flags * Revert back to older flags * SIGINFO attribute might not even exist Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add a skeleton app to demonstrate how to use `proxy.py` for standalone projects (#1029) * Add a skeleton app structure * Update `README.md` for skeleton app * Add `skeleton-app` to pre commit * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update readme Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Move `Tls` module within core (#1030) * [ReverseProxy] Move within core lib with ability to write its plugin (#1033) * Move reverse proxy core within core lib * Fix main test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix doc and lint * Fix lint * Top level category for reverse plugin Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [Multiport] Prepare for listening on multiple ports (#1031) * Convert `--port` to list of integers * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Prepare for multiport listener * Multi listeners, but will fail in theory as no port override is currently performed * Separate `listener` module * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update flags in readme * Fix imports in listener test * Fix flag parsing for `port` and `ports` * Fix tests Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [Multiport] `Acceptor` can now accept work from multiple descriptors (#1036) * Update acceptor to accept work from multiple ports * import order * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix acceptor tests * Tweak * fix tests Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [ReverseProxy] Add integration tests (#1037) * Add reverse proxy integration test * Fix reverse proxy url for https integration * Enable reverse proxy for docker build and test * [ReverseProxyPlugin] Remove redundant `ca_file` flag when wrapping upstream (#1046) * Fix #1045 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * no conn log * Do not edit host header by default Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [SOCKS4] Packet parser & builder (#1047) * Initial commit * Parse socks4 packet * Lint check * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix lint issues * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add handler test skeleton * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [CacheByContentTypePlugin] Prepare for content type parsing (#1038) * Move default cache directory within `proxy.py` instance data directory * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add `CacheByContentTypePlugin` skeleton and remove unnecessary mixin * Fix startup issues after removal of mixin * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * . * Fix broken mock * doc fix * doc Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [Work] Refactor for generic work type (#1048) * Add `TcpOrTlsSocket` type * isort * Update to use Fd executor * Define `HostPort` type * Fix fileno * spellfix * [Work] Refactor as FD work (#1049) * Abstract out FD based work implementation * No need of local/remote abstractions * fix type * Add `BaseLocalExecutor` * Fix lint and tests * [BaseRemoteExecutor] Base executor for processing work remotely (#1050) * Abstract out FD based work implementation * No need of local/remote abstractions * fix type * Add `BaseLocalExecutor` * Fix lint and tests * Add a `BaseRemoteExecutor` * [Work] `kwargs` independent `work_klass` creation and work core (#1051) * `kwargs` independent work klass and core * Fix tests * Add a `create` method to base class * Lint fixes * Pre `v2.4.0rc8` cleanups (#1053) * Pre-release cleanups * Add listener pool test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add multi listener test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2 parents 471f776 + d192046 commit 3858f3a

File tree

102 files changed

+1830
-783
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

102 files changed

+1830
-783
lines changed

.github/workflows/test-library.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -597,6 +597,8 @@ jobs:
597597
$CONTAINER_TAG
598598
--hostname 0.0.0.0
599599
--enable-web-server
600+
--enable-reverse-proxy
601+
--plugin proxy.plugin.ReverseProxyPlugin
600602
&&
601603
./tests/integration/test_integration.sh 8899
602604

.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@
1010
.tox
1111
.python-version
1212

13+
__pycache__
14+
1315
coverage.xml
1416
proxy.py.iml
1517

.pre-commit-config.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -169,6 +169,7 @@ repos:
169169
- --strict-optional
170170
- benchmark/
171171
- examples/
172+
- skeleton/
172173
- proxy/
173174
- tests/
174175
pass_filenames: false

README.md

Lines changed: 107 additions & 76 deletions
Large diffs are not rendered by default.

benchmark/_proxy.py

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,9 +22,10 @@
2222
backlog=65536,
2323
open_file_limit=65536,
2424
enable_web_server=True,
25-
disable_proxy_server=False,
26-
num_acceptors=10,
27-
local_executor=1,
25+
disable_proxy_server=True,
26+
num_acceptors=4,
27+
num_workers=10,
28+
local_executor=0,
2829
log_file='/dev/null',
2930
) as _:
3031
while True:

check.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@
3737
list(REPO_ROOT.glob('*.py')) +
3838
list((REPO_ROOT / 'proxy').rglob('*.py')) +
3939
list((REPO_ROOT / 'examples').rglob('*.py')) +
40+
list((REPO_ROOT / 'skeleton').rglob('*.py')) +
4041
list((REPO_ROOT / 'benchmark').rglob('*.py')) +
4142
list((REPO_ROOT / 'tests').rglob('*.py'))
4243
)

docs/conf.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -283,6 +283,7 @@
283283
(_any_role, 'work_klass'),
284284
(_py_class_role, '_asyncio.Task'),
285285
(_py_class_role, 'asyncio.events.AbstractEventLoop'),
286+
(_py_class_role, 'BaseListener'),
286287
(_py_class_role, 'CacheStore'),
287288
(_py_class_role, 'Channel'),
288289
(_py_class_role, 'HttpParser'),
@@ -316,7 +317,10 @@
316317
(_py_class_role, 'connection.Connection'),
317318
(_py_class_role, 'EventQueue'),
318319
(_py_class_role, 'T'),
320+
(_py_class_role, 'HostPort'),
321+
(_py_class_role, 'TcpOrTlsSocket'),
319322
(_py_obj_role, 'proxy.core.work.threadless.T'),
320323
(_py_obj_role, 'proxy.core.work.work.T'),
321324
(_py_obj_role, 'proxy.core.base.tcp_server.T'),
325+
(_py_obj_role, 'proxy.core.work.fd.fd.T'),
322326
]

examples/ssl_echo_server.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,8 +21,8 @@ class EchoSSLServerHandler(BaseTcpServerHandler[TcpClientConnection]):
2121
"""Wraps client socket during initialization."""
2222

2323
@staticmethod
24-
def create(**kwargs: Any) -> TcpClientConnection: # pragma: no cover
25-
return TcpClientConnection(**kwargs)
24+
def create(*args: Any) -> TcpClientConnection: # pragma: no cover
25+
return TcpClientConnection(*args)
2626

2727
def initialize(self) -> None:
2828
# Acceptors don't perform TLS handshake. Perform the same

examples/task.py

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
# -*- coding: utf-8 -*-
2+
"""
3+
proxy.py
4+
~~~~~~~~
5+
⚡⚡⚡ Fast, Lightweight, Pluggable, TLS interception capable proxy server focused on
6+
Network monitoring, controls & Application development, testing, debugging.
7+
8+
:copyright: (c) 2013-present by Abhinav Singh and contributors.
9+
:license: BSD, see LICENSE for more details.
10+
"""
11+
import time
12+
import argparse
13+
import threading
14+
import multiprocessing
15+
from typing import Any
16+
17+
from proxy.core.work import (
18+
Work, ThreadlessPool, BaseLocalExecutor, BaseRemoteExecutor,
19+
)
20+
from proxy.common.flag import FlagParser
21+
from proxy.common.backports import NonBlockingQueue
22+
23+
24+
class Task:
25+
"""This will be our work object."""
26+
27+
def __init__(self, payload: bytes) -> None:
28+
self.payload = payload
29+
print(payload)
30+
31+
32+
class TaskWork(Work[Task]):
33+
"""This will be our handler class, created for each received work."""
34+
35+
@staticmethod
36+
def create(*args: Any) -> Task:
37+
"""Work core doesn't know how to create work objects for us, so
38+
we must provide an implementation of create method here."""
39+
return Task(*args)
40+
41+
42+
class LocalTaskExecutor(BaseLocalExecutor):
43+
"""We'll define a local executor which is capable of receiving
44+
log lines over a non blocking queue."""
45+
46+
def work(self, *args: Any) -> None:
47+
task_id = int(time.time())
48+
uid = '%s-%s' % (self.iid, task_id)
49+
self.works[task_id] = self.create(uid, *args)
50+
51+
52+
class RemoteTaskExecutor(BaseRemoteExecutor):
53+
54+
def work(self, *args: Any) -> None:
55+
task_id = int(time.time())
56+
uid = '%s-%s' % (self.iid, task_id)
57+
self.works[task_id] = self.create(uid, *args)
58+
59+
60+
def start_local(flags: argparse.Namespace) -> None:
61+
work_queue = NonBlockingQueue()
62+
executor = LocalTaskExecutor(iid=1, work_queue=work_queue, flags=flags)
63+
64+
t = threading.Thread(target=executor.run)
65+
t.daemon = True
66+
t.start()
67+
68+
try:
69+
i = 0
70+
while True:
71+
work_queue.put(('%d' % i).encode('utf-8'))
72+
i += 1
73+
except KeyboardInterrupt:
74+
pass
75+
finally:
76+
executor.running.set()
77+
t.join()
78+
79+
80+
def start_remote(flags: argparse.Namespace) -> None:
81+
pipe = multiprocessing.Pipe()
82+
work_queue = pipe[0]
83+
executor = RemoteTaskExecutor(iid=1, work_queue=pipe[1], flags=flags)
84+
85+
p = multiprocessing.Process(target=executor.run)
86+
p.daemon = True
87+
p.start()
88+
89+
try:
90+
i = 0
91+
while True:
92+
work_queue.send(('%d' % i).encode('utf-8'))
93+
i += 1
94+
except KeyboardInterrupt:
95+
pass
96+
finally:
97+
executor.running.set()
98+
p.join()
99+
100+
101+
def start_remote_pool(flags: argparse.Namespace) -> None:
102+
with ThreadlessPool(flags=flags, executor_klass=RemoteTaskExecutor) as pool:
103+
try:
104+
i = 0
105+
while True:
106+
work_queue = pool.work_queues[i % flags.num_workers]
107+
work_queue.send(('%d' % i).encode('utf-8'))
108+
i += 1
109+
except KeyboardInterrupt:
110+
pass
111+
112+
113+
# TODO: TaskWork, LocalTaskExecutor, RemoteTaskExecutor
114+
# should not be needed, abstract those pieces out in the core
115+
# for stateless tasks.
116+
if __name__ == '__main__':
117+
flags = FlagParser.initialize(
118+
['--disable-http-proxy'],
119+
work_klass=TaskWork,
120+
)
121+
start_remote_pool(flags)
122+
# start_remote(flags)
123+
# start_local(flags)

examples/tcp_echo_server.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,8 @@ class EchoServerHandler(BaseTcpServerHandler[TcpClientConnection]):
2020
"""Sets client socket to non-blocking during initialization."""
2121

2222
@staticmethod
23-
def create(**kwargs: Any) -> TcpClientConnection: # pragma: no cover
24-
return TcpClientConnection(**kwargs)
23+
def create(*args: Any) -> TcpClientConnection: # pragma: no cover
24+
return TcpClientConnection(*args)
2525

2626
def initialize(self) -> None:
2727
self.work.connection.setblocking(False)

0 commit comments

Comments
 (0)