Skip to content

Commit d85abc0

Browse files
rubyroobsrubyroobs
committed
Preparing for OSS release
0 parents  commit d85abc0

File tree

21 files changed

+983
-0
lines changed

21 files changed

+983
-0
lines changed

‎.github/workflows/main.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
name: CI Tests
2+
3+
on: [push]
4+
5+
jobs:
6+
build:
7+
runs-on: ubuntu-latest
8+
strategy:
9+
max-parallel: 5
10+
matrix:
11+
python-version: [3.6, 3.7, 3.8, 3.9]
12+
13+
steps:
14+
- uses: actions/checkout@v1
15+
- name: Set up Python ${{ matrix.python-version }}
16+
uses: actions/setup-python@v2
17+
with:
18+
python-version: ${{ matrix.python-version }}
19+
- name: Install dependencies
20+
run: |
21+
python -m pip install --upgrade pip
22+
pip install tox tox-gh-actions
23+
- name: Test with tox
24+
run: tox

‎.gitignore‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# general things to ignore
2+
build/
3+
dist/
4+
*.egg-info/
5+
*.egg
6+
*.py[cod]
7+
__pycache__/
8+
*.so
9+
*~
10+
11+
# due to using tox and pytest
12+
.tox
13+
.cache

‎LICENSE‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
Copyright (c) 2021 Ruby Nealon
2+
3+
Permission is hereby granted, free of charge, to any person obtaining a copy
4+
of this software and associated documentation files (the "Software"), to deal
5+
in the Software without restriction, including without limitation the rights
6+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
7+
copies of the Software, and to permit persons to whom the Software is
8+
furnished to do so, subject to the following conditions:
9+
10+
The above copyright notice and this permission notice shall be included in all
11+
copies or substantial portions of the Software.
12+
13+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
14+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
15+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
16+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
17+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
18+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
19+
SOFTWARE.

‎MANIFEST.in‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
2+
include pyproject.toml
3+
include *.md
4+
include LICENSE
5+
recursive-include src *.py
6+
recursive-include example_outputs *.py
7+
recursive-include example_outputs *.vcl

‎README.md‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
# m2vcl
2+
3+
![CI Tests status badge](https://github.com/rubyroobs/m2vcl/workflows/CI%20Tests/badge.svg)
4+
5+
Experimental extension of [m2cgen](https://github.com/BayesWitnesses/m2cgen) to export statistical models to [Varnish Configuration Language](https://varnish-cache.org/docs/trunk/users-guide/vcl.html), for use in the Varnish cache. Right now only Fastly-flavored VCL is the only target supported, though this could theoretically partially target core Varnish in the future.
6+
7+
## Examples
8+
9+
For code examples and their generated VCL outputs, see the [example_outputs](https://github.com/rubyroobs/m2vcl/tree/master/example_outputs) directory.
10+
11+
## Usage
12+
13+
Use `export_to_fastly_vcl` to export to Fastly-flavored VCL. The `export_to_fasty_vcl` function takes arguemnts `indent` (defaults to 4, indent size in the generated VCL) and `sub_name` (defaults to `score`, the prefix for the generated subroutine and input/output header names). Inputs for the subroutine can be set on the headers `req.http.<prefix>_input_<index>` and outputs will be set on the header `req.http.<prefix>_output_<index>`.
14+
15+
A working demo is available in [this Fastly fiddle](https://fiddle.fastlydemo.net/fiddle/754b1898), with the source provided below:
16+
17+
### Generating Python code
18+
19+
```
20+
from sklearn.model_selection import train_test_split
21+
from sklearn.datasets import load_iris
22+
from sklearn.tree import DecisionTreeClassifier
23+
24+
import m2vcl
25+
26+
iris = load_iris()
27+
X = iris.data
28+
y = iris.target
29+
X_train, X_test, y_train, y_test = train_test_split(
30+
X, y, random_state=0)
31+
32+
clf = DecisionTreeClassifier(max_leaf_nodes=3, random_state=0)
33+
clf.fit(X_train, y_train)
34+
print(m2vcl.export_to_vcl(clf))
35+
```
36+
37+
### Output VCL
38+
39+
```
40+
sub score {
41+
declare local var.input_3 FLOAT;
42+
set var.input_3 = std.atof(req.http.score_input_3);
43+
44+
declare local var.input_2 FLOAT;
45+
set var.input_2 = std.atof(req.http.score_input_2);
46+
47+
declare local var.var0_0 FLOAT;
48+
declare local var.var0_1 FLOAT;
49+
declare local var.var0_2 FLOAT;
50+
if (var.input_3 <= 0.800000011920929) {
51+
set var.var0_0 = 1.0;
52+
set var.var0_1 = 0.0;
53+
set var.var0_2 = 0.0;
54+
} else {
55+
if (var.input_2 <= 4.950000047683716) {
56+
set var.var0_0 = 0.0;
57+
set var.var0_1 = 0.9166666666666666;
58+
set var.var0_2 = 0.08333333333333333;
59+
} else {
60+
set var.var0_0 = 0.0;
61+
set var.var0_1 = 0.02564102564102564;
62+
set var.var0_2 = 0.9743589743589743;
63+
}
64+
}
65+
set req.http.score_output_0 = var.var0_0;
66+
set req.http.score_output_1 = var.var0_1;
67+
set req.http.score_output_2 = var.var0_2;
68+
return;
69+
}
70+
```
71+
72+
### VCL Usage
73+
74+
```
75+
# VCL_DELIVER
76+
set req.http.score_input_2 = "1.23456789";
77+
set req.http.score_input_3 = "9.87654321";
78+
call score;
79+
set resp.http.Score-Result-0 = req.http.score_output_0;
80+
set resp.http.Score-Result-1 = req.http.score_output_1;
81+
set resp.http.Score-Result-2 = req.http.score_output_2;
82+
```
83+
84+
## Known limitations
85+
86+
* Precision is limited due to limitations of Fastly, and will be lost for each subroutine the AST is broken down into due to the required float -> string -> float conversion.
87+
* Only tested with a small subset of models i.e. highly experimental - make sure to sanity check outputs
88+
89+
## Todo
90+
91+
* Improve test coverage by performing end to end testing on Fastly
92+
* Create tests for more models
93+
* Support core Varnish (may require a VMOD to provide equivalent functionality of [Fastly's math trig](https://developer.fastly.com/reference/vcl/functions/math-trig/))

‎example_outputs/decision_tree/output.vcl‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
sub score {
2+
declare local var.input_3 FLOAT;
3+
set var.input_3 = std.atof(req.http.score_input_3);
4+
5+
declare local var.input_2 FLOAT;
6+
set var.input_2 = std.atof(req.http.score_input_2);
7+
8+
declare local var.var0_0 FLOAT;
9+
declare local var.var0_1 FLOAT;
10+
declare local var.var0_2 FLOAT;
11+
if (var.input_3 <= 0.800000011920929) {
12+
set var.var0_0 = 1.0;
13+
set var.var0_1 = 0.0;
14+
set var.var0_2 = 0.0;
15+
} else {
16+
if (var.input_2 <= 4.950000047683716) {
17+
set var.var0_0 = 0.0;
18+
set var.var0_1 = 0.9166666666666666;
19+
set var.var0_2 = 0.08333333333333333;
20+
} else {
21+
set var.var0_0 = 0.0;
22+
set var.var0_1 = 0.02564102564102564;
23+
set var.var0_2 = 0.9743589743589743;
24+
}
25+
}
26+
set req.http.score_output_0 = var.var0_0;
27+
set req.http.score_output_1 = var.var0_1;
28+
set req.http.score_output_2 = var.var0_2;
29+
return;
30+
}

‎example_outputs/decision_tree/source.py‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
from sklearn.model_selection import train_test_split
2+
from sklearn.datasets import load_iris
3+
from sklearn.tree import DecisionTreeClassifier
4+
5+
import m2vcl
6+
7+
iris = load_iris()
8+
X = iris.data
9+
y = iris.target
10+
X_train, X_test, y_train, y_test = train_test_split(
11+
X, y, random_state=0)
12+
13+
clf = DecisionTreeClassifier(max_leaf_nodes=3, random_state=0)
14+
clf.fit(X_train, y_train)
15+
print(m2vcl.export_to_vcl(clf))

‎example_outputs/linear_regression/output.vcl‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
sub score {
2+
declare local var.input_0 FLOAT;
3+
set var.input_0 = std.atof(req.http.score_input_0);
4+
5+
declare local var.input_1 FLOAT;
6+
set var.input_1 = std.atof(req.http.score_input_1);
7+
8+
declare local var.input_2 FLOAT;
9+
set var.input_2 = std.atof(req.http.score_input_2);
10+
11+
declare local var.input_3 FLOAT;
12+
set var.input_3 = std.atof(req.http.score_input_3);
13+
14+
declare local var.input_4 FLOAT;
15+
set var.input_4 = std.atof(req.http.score_input_4);
16+
17+
declare local var.input_5 FLOAT;
18+
set var.input_5 = std.atof(req.http.score_input_5);
19+
20+
declare local var.input_6 FLOAT;
21+
set var.input_6 = std.atof(req.http.score_input_6);
22+
23+
declare local var.input_7 FLOAT;
24+
set var.input_7 = std.atof(req.http.score_input_7);
25+
26+
declare local var.input_8 FLOAT;
27+
set var.input_8 = std.atof(req.http.score_input_8);
28+
29+
declare local var.input_9 FLOAT;
30+
set var.input_9 = std.atof(req.http.score_input_9);
31+
32+
declare local var.input_10 FLOAT;
33+
set var.input_10 = std.atof(req.http.score_input_10);
34+
35+
declare local var.input_11 FLOAT;
36+
set var.input_11 = std.atof(req.http.score_input_11);
37+
38+
declare local var.input_12 FLOAT;
39+
set var.input_12 = std.atof(req.http.score_input_12);
40+
41+
declare local var.var0_0 FLOAT;
42+
set var.var0_0 = var.input_0;
43+
set var.var0_0 *= -0.10801135783679545;
44+
declare local var.var1_0 FLOAT;
45+
set var.var1_0 = var.var0_0;
46+
declare local var.var2_0 FLOAT;
47+
set var.var2_0 = 36.459488385090125;
48+
set var.var2_0 += var.var1_0;
49+
declare local var.var3_0 FLOAT;
50+
set var.var3_0 = var.var2_0;
51+
declare local var.var4_0 FLOAT;
52+
set var.var4_0 = var.input_1;
53+
set var.var4_0 *= 0.04642045836688176;
54+
declare local var.var5_0 FLOAT;
55+
set var.var5_0 = var.var4_0;
56+
declare local var.var6_0 FLOAT;
57+
set var.var6_0 = var.var3_0;
58+
set var.var6_0 += var.var5_0;
59+
declare local var.var7_0 FLOAT;
60+
set var.var7_0 = var.var6_0;
61+
declare local var.var8_0 FLOAT;
62+
set var.var8_0 = var.input_2;
63+
set var.var8_0 *= 0.02055862636707862;
64+
declare local var.var9_0 FLOAT;
65+
set var.var9_0 = var.var8_0;
66+
declare local var.var10_0 FLOAT;
67+
set var.var10_0 = var.var7_0;
68+
set var.var10_0 += var.var9_0;
69+
declare local var.var11_0 FLOAT;
70+
set var.var11_0 = var.var10_0;
71+
declare local var.var12_0 FLOAT;
72+
set var.var12_0 = var.input_3;
73+
set var.var12_0 *= 2.6867338193448966;
74+
declare local var.var13_0 FLOAT;
75+
set var.var13_0 = var.var12_0;
76+
declare local var.var14_0 FLOAT;
77+
set var.var14_0 = var.var11_0;
78+
set var.var14_0 += var.var13_0;
79+
declare local var.var15_0 FLOAT;
80+
set var.var15_0 = var.var14_0;
81+
declare local var.var16_0 FLOAT;
82+
set var.var16_0 = var.input_4;
83+
set var.var16_0 *= -17.766611228300167;
84+
declare local var.var17_0 FLOAT;
85+
set var.var17_0 = var.var16_0;
86+
declare local var.var18_0 FLOAT;
87+
set var.var18_0 = var.var15_0;
88+
set var.var18_0 += var.var17_0;
89+
declare local var.var19_0 FLOAT;
90+
set var.var19_0 = var.var18_0;
91+
declare local var.var20_0 FLOAT;
92+
set var.var20_0 = var.input_5;
93+
set var.var20_0 *= 3.809865206809212;
94+
declare local var.var21_0 FLOAT;
95+
set var.var21_0 = var.var20_0;
96+
declare local var.var22_0 FLOAT;
97+
set var.var22_0 = var.var19_0;
98+
set var.var22_0 += var.var21_0;
99+
declare local var.var23_0 FLOAT;
100+
set var.var23_0 = var.var22_0;
101+
declare local var.var24_0 FLOAT;
102+
set var.var24_0 = var.input_6;
103+
set var.var24_0 *= 0.0006922246403425021;
104+
declare local var.var25_0 FLOAT;
105+
set var.var25_0 = var.var24_0;
106+
declare local var.var26_0 FLOAT;
107+
set var.var26_0 = var.var23_0;
108+
set var.var26_0 += var.var25_0;
109+
declare local var.var27_0 FLOAT;
110+
set var.var27_0 = var.var26_0;
111+
declare local var.var28_0 FLOAT;
112+
set var.var28_0 = var.input_7;
113+
set var.var28_0 *= -1.475566845600255;
114+
declare local var.var29_0 FLOAT;
115+
set var.var29_0 = var.var28_0;
116+
declare local var.var30_0 FLOAT;
117+
set var.var30_0 = var.var27_0;
118+
set var.var30_0 += var.var29_0;
119+
declare local var.var31_0 FLOAT;
120+
set var.var31_0 = var.var30_0;
121+
declare local var.var32_0 FLOAT;
122+
set var.var32_0 = var.input_8;
123+
set var.var32_0 *= 0.30604947898517226;
124+
declare local var.var33_0 FLOAT;
125+
set var.var33_0 = var.var32_0;
126+
declare local var.var34_0 FLOAT;
127+
set var.var34_0 = var.var31_0;
128+
set var.var34_0 += var.var33_0;
129+
declare local var.var35_0 FLOAT;
130+
set var.var35_0 = var.var34_0;
131+
declare local var.var36_0 FLOAT;
132+
set var.var36_0 = var.input_9;
133+
set var.var36_0 *= -0.01233459391657437;
134+
declare local var.var37_0 FLOAT;
135+
set var.var37_0 = var.var36_0;
136+
declare local var.var38_0 FLOAT;
137+
set var.var38_0 = var.var35_0;
138+
set var.var38_0 += var.var37_0;
139+
declare local var.var39_0 FLOAT;
140+
set var.var39_0 = var.var38_0;
141+
declare local var.var40_0 FLOAT;
142+
set var.var40_0 = var.input_10;
143+
set var.var40_0 *= -0.9527472317072923;
144+
declare local var.var41_0 FLOAT;
145+
set var.var41_0 = var.var40_0;
146+
declare local var.var42_0 FLOAT;
147+
set var.var42_0 = var.var39_0;
148+
set var.var42_0 += var.var41_0;
149+
declare local var.var43_0 FLOAT;
150+
set var.var43_0 = var.var42_0;
151+
declare local var.var44_0 FLOAT;
152+
set var.var44_0 = var.input_11;
153+
set var.var44_0 *= 0.009311683273793711;
154+
declare local var.var45_0 FLOAT;
155+
set var.var45_0 = var.var44_0;
156+
declare local var.var46_0 FLOAT;
157+
set var.var46_0 = var.var43_0;
158+
set var.var46_0 += var.var45_0;
159+
declare local var.var47_0 FLOAT;
160+
set var.var47_0 = var.var46_0;
161+
declare local var.var48_0 FLOAT;
162+
set var.var48_0 = var.input_12;
163+
set var.var48_0 *= -0.5247583778554923;
164+
declare local var.var49_0 FLOAT;
165+
set var.var49_0 = var.var48_0;
166+
declare local var.var50_0 FLOAT;
167+
set var.var50_0 = var.var47_0;
168+
set var.var50_0 += var.var49_0;
169+
set req.http.score_output_0 = var.var50_0;
170+
return;
171+
}

‎example_outputs/linear_regression/source.py‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
from sklearn.datasets import load_boston
2+
from sklearn.linear_model import LinearRegression
3+
4+
import m2vcl
5+
6+
boston = load_boston()
7+
X, y = boston.data, boston.target
8+
estimator = LinearRegression()
9+
estimator.fit(X, y)
10+
print(m2vcl.export_to_vcl(estimator))

0 commit comments

Comments
 (0)