Is your feature request related to a problem? Please describe.
The acro versions of queries in Stata are much slower to run than the native ones
Describe the solution you'd like
Stata is fast because (1) it is compiled (not interpreted) and (2) it keeps everything in memory.
We could try and reduce the overhead by not passing as much data through the SDI interface to python.
For example, if there is an exclusion ( if() statement) in the Stata query,
we should see if we can apply the exclusion before the data is bundled up by the sfi interface and passed to python (I.e. in acro.ado) rather than afterwards in acro_stata_parser.py
Describe alternatives you've considered
- Find a big open source dataset.
- Run a profiler on the code to see where time is spent - could be something else inefficient in
acro_stata_parser.py
Additional context
Is your feature request related to a problem? Please describe.
The acro versions of queries in Stata are much slower to run than the native ones
Describe the solution you'd like
Stata is fast because (1) it is compiled (not interpreted) and (2) it keeps everything in memory.
We could try and reduce the overhead by not passing as much data through the SDI interface to python.
For example, if there is an exclusion (
if()statement) in the Stata query,we should see if we can apply the exclusion before the data is bundled up by the sfi interface and passed to python (I.e. in
acro.ado) rather than afterwards inacro_stata_parser.pyDescribe alternatives you've considered
acro_stata_parser.pyAdditional context