[Discovery][PHP] Python discovery method not supported

[mlessio]

While trying to execute the latest version of the tpframework we are getting an error message while executing some specific discovery rules on a PHP project.

The executed command was:

tpframework discovery -a -t in/REDACTED_PHP_Project -i -l PHP --ignore-measurements

In the following output it is possible to notice that the CPG generation was successfully executed:

Discovery for patterns started...
16:50 - INFO - Discovery for patterns started...
16:50 - INFO - Generation of CPG for in/REDACTED_PHP_Project: started...
2023-06-29 18:13:30.277Z  info [PhpToCpg] Dominator(s) done.  - (PhpToCpg.scala:142)
2023-06-29 18:17:48.161Z  info [PhpToCpg] DDG done.  - (PhpToCpg.scala:148)
2023-06-29 18:17:49.188Z  info [PhpToCpg] Stub creation done.  - (PhpToCpg.scala:204)
2023-06-29 18:17:49.372Z  info [PhpToCpg] Call finishing pass done.  - (PhpToCpg.scala:218)
executing /tp-framework/tp_framework/core/cpgTest.sc with params=Map(name -> /tp-framework/out/discovery_2023-06-29-16-50-52_PHP_REDACTED_PHP_Project/cpg_2023-06-29-16-50-52_PHP_REDACTED_PHP_Project.bin)
Creating project `cpg_2023-06-29-16-50-52_PHP_REDACTED_PHP_Project.bin` for CPG at `/tp-framework/out/discovery_2023-06-29-16-50-52_PHP_REDACTED_PHP_Project/cpg_2023-06-29-16-50-52_PHP_REDACTED_PHP_Project.bin`
Creating working copy of CPG to be safe
Loading base CPG from: /tp-framework/workspace/cpg_2023-06-29-16-50-52_PHP_REDACTED_PHP_Project.bin/cpg.bin.tmp
2023-06-29 18:22:31.556 WARN ReachingDefPass: dlr_main has more than 4000 definitions
2023-06-29 18:22:31.558 WARN ReachingDefPass: Skipping.
The graph has been modified. You may want to use the `save` command to persist changes to disk.  All changes will also be saved collectively on exit
script finished successfully
()
18:23 - INFO - Generation of CPG for in/REDACTED_PHP_Project: done.
[...]

After this step, the discovery phase started and then it failed with the following error message:

18:41 - INFO - pattern 55 instance 1 - prepare discovery rule /tp-framework/testability_patterns/PHP/55_goto/1_instance_55_goto/1_instance_55_goto.py...
18:41 - INFO - pattern 55 instance 1 - running discovery rule...
18:41 - ERROR - Discovery method `python` is not supported.
NoneType: None
18:41 - ERROR - pattern 55 instance 1 - Discovery rule failure for this instance: Discovery method `python` is not supported.
Traceback (most recent call last):
  File "/usr/local/bin/tpframework", line 33, in <module>
    sys.exit(load_entry_point('tp-framework', 'console_scripts', 'tpframework')())
  File "/tp-framework/tp_framework/cli/main.py", line 46, in main
    discovery_pattern_cmd.execute_command(args)
  File "/tp-framework/tp_framework/cli/tpf_commands.py", line 263, in execute_command
    interface.run_discovery_for_pattern_list(target_dir, l_pattern_id, language, tool_parsed, tp_lib_path,
  File "/tp-framework/tp_framework/cli/interface.py", line 74, in run_discovery_for_pattern_list
    d_res = discovery.discovery(Path(src_dir), pattern_id_list, tp_lib_path, itools, language, build_name,
  File "/tp-framework/tp_framework/core/discovery.py", line 259, in discovery
    return discovery_ignore_measurement(cpg, l_tp_id, tp_lib_path, language, build_name, disc_output_dir,
  File "/tp-framework/tp_framework/core/discovery.py", line 394, in discovery_ignore_measurement
    "discovery": discovery_for_tpi(tpi_instance, tpi_json_path, cpg, disc_output_dir,
  File "/tp-framework/tp_framework/core/discovery.py", line 438, in discovery_for_tpi
    already_executed[d_tpi_discovery["rule_hash"]] = findings
UnboundLocalError: local variable 'findings' referenced before assignment

The following command can be used to reproduce the issue, on a vanilla docker installation of the framework, which basically executes the discovery phase on a PHP sample codebase which is self contained in the project:

tpframework discovery -a -t testability_patterns/PHP/1_static_variables/ -i -l PHP --ignore-measurements

The currently installed Joern version is:

Version: 1.2.1

[compaluca]

Thanks @mlessio.

We tried to solve these kind of issues already in branch https://github.com/testable-eu/sast-tp-framework/tree/cpg_only_merged_and_slightly_modified.

Can you give it a try? As soon as @pr0me confirmed that branch is fine, we will merge it into the main one.

@felix-20 : today you were encountering another similar issue? In case please update the branch.

[felix-20]

Yes, I encountered, the same issue, and I pushed my fix to the cpg_only_merged_and_slightly_modified branch.

[mlessio]

Sorry guys, but the mentioned branch seems to have broken the Joern installation:

root@7adf33a6afcc:/tp-framework# joern
Exception in thread "main" java.lang.NoClassDefFoundError: scala/runtime/LazyVals$
	at scopt.OptionParser.<clinit>(OptionParser.scala:74)
	at io.joern.console.BridgeBase.parseConfig(BridgeBase.scala:44)
	at io.joern.console.BridgeBase.parseConfig$(BridgeBase.scala:39)
	at io.joern.joerncli.console.AmmoniteBridge$.parseConfig(AmmoniteBridge.scala:5)
	at io.joern.joerncli.console.AmmoniteBridge$.delayedEndpoint$io$joern$joerncli$console$AmmoniteBridge$1(AmmoniteBridge.scala:7)
	at io.joern.joerncli.console.AmmoniteBridge$delayedInit$body.apply(AmmoniteBridge.scala:5)
	at scala.Function0.apply$mcV$sp(Function0.scala:39)
	at scala.Function0.apply$mcV$sp$(Function0.scala:39)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
	at scala.App.$anonfun$main$1(App.scala:76)
	at scala.App.$anonfun$main$1$adapted(App.scala:76)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:926)
	at scala.App.main(App.scala:76)
	at scala.App.main$(App.scala:74)
	at io.joern.joerncli.console.AmmoniteBridge$.main(AmmoniteBridge.scala:5)
	at io.joern.joerncli.console.AmmoniteBridge.main(AmmoniteBridge.scala)
Caused by: java.lang.ClassNotFoundException: scala.runtime.LazyVals$
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
	... 18 more

The above output comes from a fresh installation of the whole framework.
It seems that the issue is due to Joern, and also a fresh install of the version 1.2.1 on another container is not working anymore. @pr0me can you please support on this?

[pr0me]

Yes, we are working on it (#66)

[pr0me]

We merged some changes that should make things better.
#70