Profile syntax highlighting performance

[filipelbc]

Note: in order to profile syntax highlighting performance, you need to have Vim compiled with +profile,
and use the :syntime command.

See http://vimhelp.appspot.com/syntax.txt.html#%3Asyntime

[jzba]

Vim was getting terribly slow when editing org files. It went back to normal after disabling syntax highlight (with syntax off).

I've use syntax highlight for many file types, and never got this performance issue. The orgmode plugin really needs a patch.

I've profile VIM after this answer.

The summary of the profile output is listed bellow:

FUNCTIONS SORTED ON TOTAL TIME
count  total (s)   self (s)  function
   61   0.230085   0.017993  <SNR>87_OnTextChangedInsertMode()
   61   0.140299   0.125118  <SNR>87_InvokeCompletion()
  299   0.056960             <SNR>87_Pyeval()
  117   0.046580   0.006361  <SNR>87_PollCompletion()
   61   0.044449   0.031410  <SNR>87_IdentifierFinishedOperations()
  125   0.025520   0.007315  <SNR>87_OnInsertChar()
  204   0.023586   0.005001  <SNR>87_AllowedToCompleteInCurrentBuffer()
  204   0.018585   0.016632  <SNR>87_AllowedToCompleteInBuffer()
  125   0.014820             <SNR>64_Highlight_Matching_Pair()
   61   0.013786   0.001850  <SNR>87_InsideCommentOrStringAndShouldStop()
   61   0.011936             <SNR>87_InsideCommentOrString()
    1   0.011077   0.002058  <SNR>87_OnInsertLeave()
    2   0.008983   0.008819  <SNR>87_OnFileReadyToParse()
   63   0.007612   0.002379  <SNR>58_HighlightEOLWhitespaceExceptCurrentLine()
   29   0.007237             <SNR>87_OnCompleteDone()
   38   0.006574   0.003145  41()
   61   0.006571   0.000780  <SNR>87_OnBlankLine()
   64   0.005325   0.002734  <SNR>58_PerformMatchHighlight()
  117   0.005033   0.003981  <SNR>87_Complete()
    1   0.004517             <SNR>19_TrimTrailingWhitespace()

FUNCTIONS SORTED ON SELF TIME
count  total (s)   self (s)  function
   61   0.140299   0.125118  <SNR>87_InvokeCompletion()
  299              0.056960  <SNR>87_Pyeval()
   61   0.044449   0.031410  <SNR>87_IdentifierFinishedOperations()
   61   0.230085   0.017993  <SNR>87_OnTextChangedInsertMode()
  204   0.018585   0.016632  <SNR>87_AllowedToCompleteInBuffer()
  125              0.014820  <SNR>64_Highlight_Matching_Pair()
   61              0.011936  <SNR>87_InsideCommentOrString()
    2   0.008983   0.008819  <SNR>87_OnFileReadyToParse()
  125   0.025520   0.007315  <SNR>87_OnInsertChar()
   29              0.007237  <SNR>87_OnCompleteDone()
  117   0.046580   0.006361  <SNR>87_PollCompletion()
  204   0.023586   0.005001  <SNR>87_AllowedToCompleteInCurrentBuffer()
    1              0.004517  <SNR>19_TrimTrailingWhitespace()
  117   0.005033   0.003981  <SNR>87_Complete()
   38   0.006574   0.003145  41()
   64   0.005325   0.002734  <SNR>58_PerformMatchHighlight()
   63   0.007612   0.002379  <SNR>58_HighlightEOLWhitespaceExceptCurrentLine()
  213   0.002412   0.002219  <SNR>87_CloseCompletionMenu()
    1   0.011077   0.002058  <SNR>87_OnInsertLeave()
   64   0.002591   0.001976  <SNR>58_ShouldHighlight()

[filipelbc]

If it is the syntax highlighting that is slow, you need to profile it using syntime. See http://vimhelp.appspot.com/syntax.txt.html#%3Asyntime

Not surprisingly, the slow pattern is the orgTableHeader.

Feel free to try to improve it.

[filipelbc]

Turns out the biggest performance bottleneck was not that pattern, but the overall way the table syntax highlight was implemented. It has been improved in #75

[jzba]

Thanks for the fix! It indeed improved the usability a lot.

However, I think there is still room for a lot of improvement. After the fix, If I type regularly I get not delay, but I still notice a delay if I type like mad (random key presses, just for the sake of stressing vim).

I've profiled the syntax highlight on a 300 lines org file, as well as a 300 lines python file (GG's feed.py). The top 10 results and the totals are shown below. The total time is 10x smaller on python, and the average time of the slowest operation is 7 times smaller on python.

org

 TOTAL      COUNT  MATCH   SLOWEST     AVERAGE   NAME               PATTERN
  0.002090   44     0       0.000736    0.000048  orgCode            \(^\|\s\|[-({\"']\|\(^\|\s\|[-({\"']\)[*/+_]\)\@<==[^ ]
  0.001486   46     4       0.000310    0.000032  orgItalic          \(^\|\s\|[-({\"']\)\@<=/[^ ]
  0.000973   44     0       0.000222    0.000022  orgStrikethrough   \(^\|\s\|[-({\"']\)\@<=+[^ ]
  0.000929   44     0       0.000189    0.000021  orgUnderline       \(^\|\s\|[-({\"']\)\@<=_[^ ]
  0.000675   46     2       0.000118    0.000015  orgBold            \(^\|\s\|[-({\"']\)\@<=\*[^ ]
  0.000667   44     0       0.000118    0.000015  orgCode            \(^\|\s\|[-({\"']\|\(^\|\s\|[-({\"']\)[*/+_]\)\@<=\~[^ ]
  0.000540   36     5       0.000196    0.000015  orgFold0           \ze\n\*\{1}
  0.000381   44     5       0.000128    0.000009  orgConfig          ^\s*#+\k\+:
  0.000288   48     8       0.000063    0.000006  orgSection0        ^\*\{1} .*
  0.000270   4      4       0.000113    0.000068  orgConfigValue     .*$

  0.118738   29349

python

TOTAL      COUNT  MATCH   SLOWEST     AVERAGE   NAME               PATTERN
  0.000336   50     0       0.000144    0.000007  pythonRawString    [rR]'
  0.000299   50     0       0.000046    0.000006  pythonError        [$?]
  0.000265   50     0       0.000047    0.000005  pythonFloat        \<\d\+\.\d*\%([eE][+-]\=\d\+\)\=[jJ]\=
  0.000264   50     0       0.000058    0.000005  pythonFloat        \<\d\+[eE][+-]\=\d\+[jJ]\=\>
  0.000263   50     0       0.000057    0.000005  pythonNumber       \<[1-9]\d\+\>
  0.000257   50     0       0.000037    0.000005  pythonNumberError  \<\d\+\D\>
  0.000247   50     0       0.000048    0.000005  pythonBytes        [bB]'''
  0.000247   50     0       0.000039    0.000005  pythonNumber       \<\d\+[jJ]\>
  0.000245   50     0       0.000039    0.000005  pythonBytes        [bB]"
  0.000242   50     0       0.000029    0.000005  pythonError        \<\d\+\D\+\>

  0.018242   2679

These reports were generated with:

:syntime on
:redraw!
:syntime report

[filipelbc]

Patches are welcome!