Working with profiler

This page describes how we use the profiler for pyhton

python -m cProfile .\tool_script_runner.py .\scripts\scripts_regression.txt log.profile.run swiss4 >profile_log.txt

Description can be found here: https://docs.python.org/3/library/profile.html

         370951672 function calls (367090349 primitive calls) in 725.402 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   198535    0.695    0.000  417.256    0.002 embedding.py:10(embedding_udf)
   711882    4.032    0.000  402.673    0.001 dbaccess.py:891(update_embedded_referencing)
4607663/1096348  378.885    0.000  397.228    0.000 dbaccess.py:1157(recursive_attributes)
   198536    0.552    0.000  241.834    0.001 xmlserializer.py:39(unmarshall)
   198545    0.939    0.000  241.213    0.001 bases.py:35(parse)
   198545    0.800    0.000  240.176    0.001 lxml.py:16(parse)
   198545   25.999    0.000  239.262    0.001 lxml.py:44(process_context)
  4062931    4.366    0.000  138.309    0.000 bases.py:131(end)
  1950403    4.846    0.000   82.534    0.000 element.py:79(bind)
  4062931   13.958    0.000   74.954    0.000 bases.py:68(start)
  4373619    4.987    0.000   67.931    0.000 utils.py:83(parse_var)
  4373619    4.511    0.000   62.944    0.000 utils.py:130(parse_value)
  4373619   16.938    0.000   58.429    0.000 converter.py:84(deserialize)
  1947974    2.803    0.000   48.797    0.000 primitive.py:28(bind)
  3777589   10.189    0.000   46.902    0.000 element.py:440(child)
  1950403    6.021    0.000   39.231    0.000 element.py:120(bind_content)
   524807    3.890    0.000   34.502    0.000 pickleserializer.py:18(unmarshall)
   524807   25.532    0.000   30.612    0.000 {built-in method _pickle.loads}
  1950403    4.798    0.000   28.505    0.000 element.py:151(bind_attrs)

For line based analysis in critical parts we base on https://stackoverflow.com/questions/3927628/how-can-i-profile-python-code-line-by-line We first start with https://github.com/pyutils/line_profiler

To profile a python script:

• Install line_profiler: pip install line_profiler.
• In the relevant file(s), import line profiler and decorate function(s) you want to profile with @line_profiler.profile.
• Set the environment variable LINE_PROFILE=1 and run your script as normal. When the script ends a summary of profile results, files written to disk, and instructions for inspecting details will be written to stdout.