diff --git a/CITATION.cff b/CITATION.cff index 2b77079c..160f2112 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -18,6 +18,6 @@ authors: given-names: "Eric" orcid: "https://orcid.org/0000-0003-3470-3647" title: "SVFA-Scala: an implementation of SVFA for Java" -version: 0.6.0 +version: 0.6.1-SNAPSHOT date-released: 2025-09-06 url: "https://github.com/PAMunb/svfa" diff --git a/README.md b/README.md index 8dd29d5c..555be2f0 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,7 @@ This project follows a **modular architecture** with three focused modules: - **`core`**: Essential SVFA framework + Android analysis support - **`securibench`**: Java security vulnerability analysis benchmarks - **`taintbench`**: Android malware analysis benchmarks +- **`scripts`**: snippets of code use to automatize metric computation ## Quick Start @@ -23,7 +24,7 @@ This project follows a **modular architecture** with three focused modules: Add to your `build.sbt`: ```scala resolvers += Resolver.githubPackages("PAMunb", "svfa") -libraryDependencies += "br.unb.cic" %% "svfa-core" % "0.6.0" +libraryDependencies += "br.unb.cic" %% "svfa-core" % "0.6.1-SNAPSHOT" ``` #### Using svfa-core in Java/Maven Projects @@ -40,7 +41,7 @@ Add to your `pom.xml`: br.unb.cic svfa-core_2.12 - 0.6.0 + 0.6.1-SNAPSHOT ``` @@ -59,7 +60,7 @@ repositories { } dependencies { - implementation 'br.unb.cic:svfa-core_2.12:0.6.0' + implementation 'br.unb.cic:svfa-core_2.12:0.6.1-SNAPSHOT' } ``` @@ -110,12 +111,24 @@ sbt testTaintbench ## API Usage -Implement a class that extends the `JSVFA class` (see examples in the tests). You must provide implementations for: +Implement a class that extends the `JSVFA class` (i.e., scala/br/unb/cic/svfa/JSVFATest.scala) + +You must provide implementations for: * `getEntryPoints()` - Set up the "main" methods (returns List of Soot methods) * `sootClassPath()` - Set up the soot classpath (returns String) * `analyze(unit)` - Identify node types (source, sink, simple node) in the graph +The framework implements a flexible approach (`trait`) to store the set of node types (source, sink) +Then, this trait is available to be manipulated in method `analyze(unit)` + +```scala +trait SecuribenchSpec { + val sinkList: Seq[String] = List() + + val sourceList: Seq[String] = List() +``` + ### Example Usage ```scala @@ -138,15 +151,15 @@ class MyAnalysis extends JSVFATest { ## Available Commands -| Command | Description | -|---------|-------------| -| `sbt testCore` | Run core SVFA tests | +| Command | Description | +|-----------------------|--------------------------------------------------| +| `sbt testCore` | Run core SVFA tests | | `sbt testSecuribench` | Run security vulnerability tests (93 test cases) | -| `sbt testTaintbench` | Run Android malware tests | -| `sbt testRoidsec` | Run specific Roidsec test | -| `sbt compileAll` | Compile all modules | -| `sbt publishCore` | Publish core module to GitHub Packages | -| `sbt publishAllLocal` | Publish all modules to local Maven repository | +| `sbt testTaintbench` | Run Android malware tests | +| `sbt testRoidsec` | Run specific Roidsec test | +| `sbt compileAll` | Compile all modules | +| `sbt publishCore` | Publish core module to GitHub Packages | +| `sbt publishAllLocal` | Publish all modules to local Maven repository | ## Scripts @@ -190,92 +203,197 @@ This project integrates 2 well-known benchmarks. ### Securibench -This benchmark was integrated because it is also used in the [FlowDroid Project](https://github.com/secure-software-engineering/FlowDroid) and tests cases are in `src/test/java/securibench`. - -#### JSVFA metrics (old) - -> failed: 46, passed: 57 of 103 tests - (55.34%) - -| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | -|:--------------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:| -| aliasing | 10 | 12 | 2/6 | 8 | 1 | 3 | 0.89 | 0.73 | 0.80 | -| arrays | 0 | 9 | 1/10 | 0 | 0 | 9 | 0.00 | 0.00 | 0.00 | -| basic | 60 | 60 | 36/42 | 52 | 3 | 3 | 0.95 | 0.95 | 0.95 | -| collections | 3 | 15 | 1/14 | 1 | 1 | 13 | 0.50 | 0.07 | 0.12 | -| datastructures | 7 | 5 | 4/6 | 4 | 2 | 0 | 0.67 | 1.00 | 0.80 | -| factories | 4 | 3 | 2/3 | 2 | 1 | 0 | 0.67 | 1.00 | 0.80 | -| inter | 10 | 18 | 7/14 | 8 | 0 | 8 | 1.00 | 0.50 | 0.67 | -| session | 0 | 3 | 0/3 | 0 | 0 | 3 | 0.00 | 0.00 | 0.00 | -| strong_updates | 0 | 1 | 4/5 | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| TOTAL | 94 | 126 | 57/103 | 75 | 8 | 40 | 0.90 | 0.65 | 0.75 | - -To have detailed information about each group of tests run, [see here.](modules/securibench/src/docs-metrics/jsvfa/jsvfa-metrics-v0.3.0.md) (*computed in March 2023.*) - -#### JSVFA 2.0 metrics (v0.3.4) - -> failed: 38, passed: 65 of 103 tests - (63.11%) - -| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | -|:--------------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:| -| Aliasing | 4 | 12 | 1/6 | 1 | 1 | 9 | 0.50 | 0.10 | 0.17 | -| Arrays | 11 | 9 | 5/10 | 5 | 4 | 2 | 0.56 | 0.71 | 0.63 | -| Basic | 59 | 60 | 37/42 | 53 | 2 | 3 | 0.96 | 0.95 | 0.95 | -| Collections | 8 | 15 | 5/14 | 5 | 1 | 8 | 0.83 | 0.38 | 0.52 | -| Datastructures | 5 | 5 | 4/6 | 4 | 1 | 1 | 0.80 | 0.80 | 0.80 | -| Factories | 4 | 3 | 2/3 | 2 | 1 | 0 | 0.67 | 1.00 | 0.80 | -| Inter | 12 | 18 | 8/14 | 9 | 0 | 6 | 1.00 | 0.60 | 0.75 | -| Session | 0 | 3 | 0/3 | 0 | 0 | 3 | 0.00 | 0.00 | 0.00 | -| StrongUpdates | 3 | 1 | 3/5 | 1 | 2 | 0 | 0.33 | 1.00 | 0.50 | -| TOTAL | 106 | 126 | 65/103 | 80 | 12 | 32 | 0.87 | 0.71 | 0.78 | - -To have detailed information about each group of tests run, [see here.](modules/securibench/src/docs-metrics/jsvfa/jsvfa-metrics-v0.3.4.md) (*computed in September 2025.*) +This benchmark was integrated because it is also used in the [FlowDroid Project](https://github.com/secure-software-engineering/FlowDroid) and this integration +is implemented in `securibench` module. + +The result are presented in a table that contains the following information. + +- **Expected:** The amount of taint flows presented by TAINTBENCH +- **Actual:** The amount of taint flows detected by JSVFA +- **Status:** If the test PASS OR FAIL +- **TP:** True Positive +- **FP:** False Positive +- **Precision:** TP/(TP + FP) +- **Recall:** TP/P +- **F-score:** (2 x Precision x Recall)/(Precision + Recall) + +#### Old metrics (v0.3.0) + +> failed: 59, passed: 63 of 122 tests - (51.63%) + +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | Pass Rate | +|:--------------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:|:---------:| +| aliasing | 10 | 12 | 2/6 | 8 | 1 | 3 | 0.89 | 0.73 | 0.80 | 33.33 | +| arrays | 0 | 9 | 1/10 | 0 | 0 | 9 | 0.00 | 0.00 | 0.00 | 10 | +| basic | 60 | 60 | 36/42 | 52 | 3 | 3 | 0.95 | 0.95 | 0.95 | 85.71 | +| collections | 3 | 15 | 1/14 | 1 | 1 | 13 | 0.50 | 0.07 | 0.12 | 7.14 | +| datastructures | 7 | 5 | 4/6 | 4 | 2 | 0 | 0.67 | 1.00 | 0.80 | 66.67 | +| factories | 4 | 3 | 2/3 | 2 | 1 | 0 | 0.67 | 1.00 | 0.80 | 66.67 | +| inter | 10 | 18 | 7/14 | 8 | 0 | 8 | 1.00 | 0.50 | 0.67 | 50 | +| session | 0 | 3 | 0/3 | 0 | 0 | 3 | 0.00 | 0.00 | 0.00 | 0 | +| strong_updates | 0 | 1 | 4/5 | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | 80 | +| Pred | 8 | 5 | 6/9 | 5 | 3 | 0 | 0.63 | 1.00 | 0.77 | 66.67% | +| Reflection | 0 | 4 | 0/4 | 0 | 0 | 4 | 0.00 | 0.00 | 0.00 | 0% | +| Sanitizers | 0 | 4 | 0/6 | 0 | 0 | 6 | 0.00 | 0.00 | 0.00 | 0% | +| TOTAL | 102 | 139 | 63/122 | 80 | 11 | 50 | 0.88 | 0.62 | 0.72 | 51.64 | + +To have detailed information about each test category run, [see here.](modules/securibench/src/docs-metrics/jsvfa/jsvfa-metrics-v0.3.0.md) (*computed in June 2023.*) + +#### New metrics (v0.6.1) + +> failed: 47, passed: 75 of 122 tests - (61.48%) + +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | Pass Rate | +|:--------------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:|:---------:| +| Aliasing | 10 | 12 | 2/6 | 8 | 1 | 3 | 0.89 | 0.73 | 0.80 | 33.33% | +| Arrays | 11 | 9 | 5/10 | 5 | 4 | 2 | 0.56 | 0.71 | 0.63 | 50% | +| Basic | 57 | 60 | 38/42 | 55 | 1 | 4 | 0.98 | 0.93 | 0.95 | 90.48% | +| Collections | 8 | 15 | 5/14 | 5 | 1 | 8 | 0.83 | 0.38 | 0.52 | 35.71% | +| Datastructures | 5 | 5 | 4/6 | 4 | 1 | 1 | 0.80 | 0.80 | 0.80 | 66.67% | +| Factories | 4 | 3 | 2/3 | 2 | 1 | 0 | 0.67 | 1.00 | 0.80 | 66.67% | +| Inter | 12 | 18 | 8/14 | 9 | 0 | 6 | 1.00 | 0.60 | 0.75 | 57.14% | +| Session | 0 | 3 | 0/3 | 0 | 0 | 3 | 0.00 | 0.00 | 0.00 | 0% | +| StrongUpdates | 3 | 1 | 3/5 | 1 | 2 | 0 | 0.33 | 1.00 | 0.50 | 60% | +| Pred | 8 | 5 | 6/9 | 5 | 3 | 0 | 0.63 | 1.00 | 0.77 | 66.67% | +| Reflection | 0 | 4 | 0/4 | 0 | 0 | 4 | 0.00 | 0.00 | 0.00 | 0% | +| Sanitizers | 2 | 6 | 2/6 | 1 | 0 | 4 | 1.00 | 0.20 | 0.33 | 33.33% | +| TOTAL | 120 | 141 | 75/122 | 95 | 14 | 35 | 0.87 | 0.73 | 0.79 | 61.48% | + +To have detailed information about each test category run, [see here.](modules/securibench/src/docs-metrics/jsvfa/jsvfa-metrics-v0.6.1.md) (*computed in November 2025.*) + +##### Common issues +From the 47 tests, we have categorized nine (9) issues. + +[i] **Wrong counting**: Some tests from the Securibench benchmark are incorrectly labeled, leading to wrong expected values. +We have mapped four cases: `(8.51%)` +- Aliasing2 +- Aliasing4 +- Inter4 +- Inter5 + +[ii] Array Indexes: The actual implementation is unable to recognize tainted in specific indexes from an array. Currently, it marks all the array as tainted. +We have mapped six cases: `(12.77%)` +- Aliasing3 +- Arrays2 +- Arrays5 +- Arrays8 +- Arrays9 +- Arrays10 + +[iii] Support Class Missing: Some tests use methods from securibench that are not mocked. +We have mapped seven cases: `(14.89%)` +- Basic31 +- Basic36 +- Basic38 +- Session1 +- Session2 +- Session3 +- Sanitizers5 + +[iv] Missing Context: The logic for handling context is not entirely flawless, resulting in certain edge cases that lead to bugs such as: + [a] Nested structures as HashMap, LinkedList, and others, + [b] Loop statement as "for" or "while", + [c] Parameters passed in the constructor. +We have mapped 16 cases: `(34.04%)` +- Aliasing5 +- Basic42 +- Collections3 +- Collections5 +- Collections6 +- Collections7 +- Collections8 +- Collections9 +- Collections10 +- Collections12 +- Collections13 +- Datastructures4 +- Datastructures5 +- Factories3 +- Inter9 +- Inter12 + +[v] Reflection: The current implementation does not address the reflection feature, +We have mapped 5 cases: `(10.64%)` +- Inter6 +- Refl1 +- Refl2 +- Refl3 +- Refl4 + +[vi] Global variables references: There are unaddressed edge cases regarding the handling of the definition of global variables., +We have mapped two cases: `(4.26%)` +- StrongUpdates3 +- StrongUpdates5 + + +[vii] Path for conditional: The current logic always evaluates two paths for a conditional, regardless of whether the condition is set to True or False, +We have mapped three cases: `(6.38%)` +- Pred3 +- Pred6 +- Pred7 + +[viii] Sanitizer method: The current implementation fails to deal with the intermediary method utilized by the sanitizer. +We have mapped three cases: `(6.38%)` +- Sanitizers2 +- Sanitizers4 +- Sanitizers6 + +[ix] Flaky +We have mapped one cases: `(2.13%)` +- Inter11 #### FLOWDROID - failed: 36, passed: 67 of 103 tests. `(65.05%)` -| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | -|----------------|-------|----------|--------|----|----|----|-----------|--------|------| -| Aliasing | 11 | 11 | 4/6 | 9 | 1 | 1 | 0.90 | 0.90 | 0.90 | -| Arrays | 14 | 9 | 6/10 | 6 | 5 | 0 | 0.55 | 1.00 | 0.71 | -| Basic | 38 | 61 | 26/42 | 33 | 1 | 24 | 0.97 | 0.58 | 0.73 | -| Collections | 14 | 14 | 12/14 | 12 | 1 | 1 | 0.92 | 0.92 | 0.92 | -| Datastructures | 5 | 5 | 4/6 | 3 | 1 | 1 | 0.75 | 0.75 | 0.75 | -| Factories | 1 | 3 | 1/3 | 1 | 0 | 2 | 1.00 | 0.33 | 0.50 | -| Inter | 15 | 16 | 13/14 | 15 | 0 | 1 | 1.00 | 0.94 | 0.97 | -| Session | 0 | 3 | 0/3 | 0 | 0 | 3 | 0.00 | 0.00 | 0.00 | -| StrongUpdates | 0 | 1 | 4/5 | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| TOTAL | 98 | 126 | 67/103 | 77 | 9 | 37 | 0.90 | 0.68 | 0.77 | +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | Pass Rate | +|----------------|-------|----------|--------|----|----|----|-----------|--------|------|-----------| +| Aliasing | 11 | 11 | 4/6 | 9 | 1 | 1 | 0.90 | 0.90 | 0.90 | 66.67% | +| Arrays | 14 | 9 | 6/10 | 6 | 5 | 0 | 0.55 | 1.00 | 0.71 | 60% | +| Basic | 38 | 61 | 26/42 | 33 | 1 | 24 | 0.97 | 0.58 | 0.73 | 61.90% | +| Collections | 14 | 15 | 11/14 | 12 | 1 | 2 | 0.92 | 0.86 | 0.89 | 78.57% | +| Datastructures | 5 | 5 | 4/6 | 3 | 1 | 1 | 0.75 | 0.75 | 0.75 | 66.67% | +| Factories | 1 | 3 | 1/3 | 1 | 0 | 2 | 1.00 | 0.33 | 0.50 | 33.33% | +| Inter | 15 | 18 | 11/14 | 13 | 0 | 3 | 1.00 | 0.81 | 0.90 | 78.57% | +| Session | 0 | 3 | 0/3 | 0 | 0 | 3 | 0.00 | 0.00 | 0.00 | 0% | +| StrongUpdates | 0 | 1 | 4/5 | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | 80% | +| Pred | - | - | - | - | - | - | - | - | - | - | * NO EXECUTED +| Reflection | - | - | - | - | - | - | - | - | - | - | * NO EXECUTED +| Sanitizers | - | - | - | - | - | - | - | - | - | - | * NO EXECUTED +| **TOTAL** | 98 | 126 | 67/103 | 77 | 9 | 37 | 0.90 | 0.68 | 0.77 | 65.05% | To have detailed information about each group of tests run, [see here.](modules/securibench/src/docs-metrics/flowdroid/flowdroid-metrics.md) #### JOANA -> failed: 32, passed: 71 of 103 tests. `(68.93%)` - -| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | -|----------------|-------|----------|--------|----|----|----|-----------|--------|------| -| Aliasing | 6 | 11 | 2/6 | 2 | 2 | 7 | 0.50 | 0.22 | 0.31 | -| Arrays | 10 | 9 | 9/10 | 9 | 1 | 0 | 0.90 | 1.00 | 0.95 | -| Basic | 45 | 61 | 25/42 | 26 | 6 | 22 | 0.81 | 0.54 | 0.65 | -| Collections | 15 | 15 | 14/14 | 15 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Datastructures | 6 | 5 | 5/6 | 5 | 1 | 0 | 0.83 | 1.00 | 0.91 | -| Factories | 3 | 3 | 3/3 | 3 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Inter | 13 | 18 | 9/14 | 9 | 0 | 5 | 1.00 | 0.64 | 0.78 | -| Session | 3 | 3 | 3/3 | 3 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| StrongUpdates | 5 | 1 | 1/5 | 1 | 4 | 0 | 0.20 | 1.00 | 0.33 | -| TOTAL | 106 | 126 | 71/103 | 73 | 14 | 34 | 0.84 | 0.68 | 0.75 | +> failed: 37, passed: 85 of 122 tests. `(69.67%)` + +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | Pass rate | +|----------------|-------|----------|--------|----|----|----|-----------|--------|-------|-----------| +| Aliasing | 6 | 11 | 2/6 | 2 | 2 | 7 | 0.50 | 0.22 | 0.31 | 33.33% | +| Arrays | 10 | 9 | 9/10 | 9 | 1 | 0 | 0.90 | 1.00 | 0.95 | 90% | +| Basic | 45 | 61 | 25/42 | 26 | 6 | 22 | 0.81 | 0.54 | 0.65 | 59.52% | +| Collections | 15 | 14 | 13/14 | 14 | 1 | 0 | 0.93 | 1.00 | 0.96 | 92.86% | +| Datastructures | 6 | 5 | 5/6 | 5 | 1 | 0 | 0.83 | 1.00 | 0.91 | 83.33% | +| Factories | 3 | 3 | 3/3 | 3 | 0 | 0 | 1.00 | 1.00 | 1.00 | 100% | +| Inter | 13 | 16 | 11/14 | 11 | 0 | 3 | 1.00 | 0.79 | 0.88 | 78.57% | +| Session | 3 | 3 | 3/3 | 3 | 0 | 0 | 1.00 | 1.00 | 1.00 | 100% | +| StrongUpdates | 5 | 1 | 1/5 | 1 | 4 | 0 | 0.20 | 1.00 | 0.33 | 20% | +| Pred | 8 | 5 | 6/9 | 5 | 3 | 0 | 0.63 | 1.00 | 0.77 | 66.67% | +| Reflection | 3 | 4 | 3/4 | 3 | 0 | 1 | 1.00 | 0.75 | 0.86 | 75% | +| Sanitizer | 6 | 6 | 4/6 | 4 | 1 | 1 | 0.80 | 0.80 | 0.80 | 66.67% | +| TOTAL | 123 | 138 | 85/122 | 86 | 19 | 34 | 0.82 | 0.72 | 0.77 | 69.67% | To have detailed information about each group of tests run, [see here.](modules/securibench/src/docs-metrics/joana/joana-metrics.md) -#### METRICS SUMMARY +### METRICS SUMMARY -| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | Pass Rate | -|:---------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:|----------:| -| JSVFA | 94 | 126 | 57/103 | 75 | 8 | 40 | 0.90 | 0.65 | 0.75 | 55.34% | -| JSVFA 2.0 | 106 | 126 | 65/103 | 80 | 12 | 32 | 0.87 | 0.71 | 0.78 | 63.11% | -| Flowdroid | 98 | 126 | 67/103 | 77 | 9 | 37 | 0.90 | 0.68 | 0.77 | 65.05% | -| Joana | 106 | 126 | 71/103 | 73 | 14 | 34 | 0.84 | 0.68 | 0.75 | 68.93% | +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | Pass Rate | +|:------------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:|----------:| +| JSVFA v0.3.0 | 102 | 139 | 63/122 | 80 | 11 | 50 | 0.88 | 0.62 | 0.72 | 51.64% | +| JSVFA v0.6.1 | 120 | 141 | 75/122 | 95 | 14 | 35 | 0.87 | 0.73 | 0.79 | 61.48% | +| Flowdroid | 98 | 126 | 67/103 | 77 | 9 | 37 | 0.90 | 0.68 | 0.77 | 65.05% | +| Joana | 123 | 138 | 85/122 | 86 | 19 | 34 | 0.82 | 0.72 | 0.77 | 69.67% | ### TAINTBENCH: @@ -285,8 +403,11 @@ and it is introduced by [Paper: TaintBench: Automatic real-world malware benchma which in its result section presents six experiments to answer one of its RQ: *How effective are taint analysis tools on TaintBench compared to DroidBench*, where FLOWDROID and AMANDROID as the chosen tools. -In the next sections, we will focus in **Experiment 2** and **Experiment 3** and use our tool (JSVFA) to reproduce them. -After that, we will compare the already computed results for FLOWDROID to our results. +In the next sections, we will focus in **Experiment 2** and **Experiment 3** and use the latest version of JSVFA to reproduce them. +Both experiments compare the matches between the set of leaks reported by each APK execution TaintBench paper, referred to as `expected`, +against the number of leaks identified by JSVFA, which is termed `actual`. A `match` is when the source and sink are the same in both the actual and expected sets. +We have created a file `taintbench.properties` to set the configurations in module `taintbench`. + #### Environment Setup @@ -331,19 +452,6 @@ ANDROID_SDK=/path/to/android/sdk TAINT_BENCH=/path/to/taintbench sbt testRoidsec ANDROID_SDK=/path/to/android/sdk TAINT_BENCH=/path/to/taintbench sbt "testOnly br.unb.cic.android.RoidsecTest" ``` -- The result for each APK tested using JSVFA are presented in a table that contains the following information. - - **Expected:** The amount of taint flows presented by TAINTBENCH - - **Actual:** The amount of taint flows detected by JSVFA - - **Status:** If the test PASS OR FAIL - - **TP:** True Positive - - **FP:** False Positive - - **Precision:** TP/(TP + FP) - - **Recall:** TP/P - - **F-score:** (2 x Precision x Recall)/(Precision + Recall) -- We have created a file `taintbench.properties` in `src/test/resources` to set the configurations - -**Disclaimer**: Although TAINTBENCH contains 203 expected [taint flows](https://taintbench.github.io/taintbenchSuite/), -we have decided to use only 186 expected cases because the mentioned paper, uses as a reference, works with those amounts. #### EXPERIMENT I @@ -351,27 +459,13 @@ This case emulates **Experiment 2 - TB2** that states: >All tools are configured with sources and sinks defined in benchmark suite. -The mentioned sources and sinks can be found in [TB_SourcesAndSinks](https://github.com/TaintBench/TaintBench/blob/main/TB_SourcesAndSinks.txt), -and we have stored them in `src/test/scala/br/unb/cic/android/TaintBenchSpec.scala`. - -As a result, we got `36, passed: 3 of 39 tests` and comparing to FLOWDROID we computed the next metrics: -~~a better `precision(0.82)`, the same `recall(0.22)` value and a slightly better `F-score(0.35)`.~~ - -- JSVFA metrics, to have detailed information about each group of tests run, [see here.](modules/taintbench/src/docs-metrics/taintbench-experiment-I.md) - -| Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | Pass Rate | -|:-----:|:--------:|:------:|:--:|:--:|:---:|:---------:|:------:|:-------:|:---------:| -| 60 | 203 | 3/39 | 4 | 12 | 155 | 0.25 | 0.03 | 0.05 | 7.69% | - -- FLOWDROID metrics from [Paper](https://doi.org/10.1007/s10664-021-10013-5) - -| Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | Pass Rate | -|:-----:|:--------:|:------:|:--:|:--:|:--:|:---------:|:------:|:-------:|:---------:| -| 55 | 186 | ? | 41 | 14 | ? | 0.75 | 0.22 | 0.34 | ? | - -##### Observation -- From the 37 failing tests, 28 of them reported zero flows. +The mentioned sources and sinks can be found in [TB_SourcesAndSinks](https://github.com/TaintBench/TaintBench/blob/main/TB_SourcesAndSinks.txt). +As a result, it finds fewer leaks than the expected, and, it gets only 10 matches, +which means, it has found 76 new leakages. +| Actual Findings | Expected Findings | Matches | +|:---------------:|:-----------------:|:-------:| +| 86 | 216 | 10 | #### EXPERIMENT II @@ -381,29 +475,17 @@ This case emulates **Experiment 3 - TB3** that configures: configure all tools. Each tool analyzes each benchmark app with the associated list of sources and sinks -The mentioned lists can be found in https://taintbench.github.io/taintbenchSuite/, and we have stored them by individual -files in `src/test/scala/br/unb/cic/android/specs`. +The mentioned lists can be found in [TB_SourcesAndSinks](https://taintbench.github.io/taintbenchSuite). +As a result, it finds more leaks than the expected, however, it still gets only 10 matches as in the last experiment. +which means, it has found 645 new leakages. -As a result, we got `failed: 35, passed: 4 of 39 tests` and comparing to FLOWDROID -~~awe detect a several better amount of `TP(135)` but also a several amount of `FP(318)` and about metric, we got a significant less `precision(0.30)` due to the high amount of FP; however, -a good `recall(0.73)` value and a better `F-score(0.42)`.~~ +| Actual Findings | Expected Findings | Matches | +|:---------------:|:-----------------:|:-------:| +| 655 | 216 | 10 | -- JSVFA metrics, to have detailed information about each group of tests run, [see here.](modules/taintbench/src/docs-metrics/taintbench-experiment-II.md) - -| Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | Pass Rate | -|:-----:|:--------:|:------:|:--:|:---:|:--:|:---------:|:------:|:-------:|:---------:| -| 620 | 203 | 4/39 | 29 | 463 | 46 | 0.06 | 0.39 | 0.10 | 10.26% | - -- FLOWDROID metrics from Paper https://doi.org/10.1007/s10664-021-10013-5 - -| Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | Pass Rate | -|:-----:|:--------:|:------:|:--:|:--:|:--:|:---------:|:------:|:-------:|:---------:| -| 57 | 186 | ? | 43 | 14 | ? | 0.75 | 0.23 | 0.35 | ? | - -##### Observation -- We got a big amount of FP. ## Tasks + ### WIP - [ ] Add set up project documentation. - [ ] Fix bugs for Securibench in folders @@ -413,13 +495,6 @@ a good `recall(0.73)` value and a better `F-score(0.42)`.~~ - [ ] Strong Update - [ ] Aliasing -### TO-DO -- - -### DONE -- [X] Create Git Action flow. -- [X] Integrate Taintbench. - ## License This project is licensed under the MIT License. diff --git a/build.sbt b/build.sbt index e73b1dc4..9d622e32 100644 --- a/build.sbt +++ b/build.sbt @@ -3,7 +3,7 @@ ThisBuild / scalaVersion := "2.12.20" ThisBuild / organization := "br.unb.cic" -ThisBuild / version := "0.6.0" +ThisBuild / version := "0.6.1-SNAPSHOT" // Global settings ThisBuild / publishConfiguration := publishConfiguration.value.withOverwrite(true) diff --git a/modules/core/src/main/scala/br/unb/cic/soot/graph/Graph.scala b/modules/core/src/main/scala/br/unb/cic/soot/graph/Graph.scala index 7ab6c913..505ae327 100644 --- a/modules/core/src/main/scala/br/unb/cic/soot/graph/Graph.scala +++ b/modules/core/src/main/scala/br/unb/cic/soot/graph/Graph.scala @@ -523,6 +523,12 @@ class Graph() { val csOpenAndClose = csOpen ++ csClose + /** + TO-DO: Implement a better way to calculate the right csOpen and Close + because the right one can lead to a bug in some edges cases. + */ + + csOpenAndClose.foreach(open => { if (open.value.context.nonEmpty) { cs = cs + open.value.context.head @@ -568,21 +574,7 @@ class Graph() { f(stmt) ) - def reportConflicts( - useUniquePaths: Boolean = false - ): scala.collection.Set[String] = { - val conflicts = findConflictingPaths() - - if (useUniquePaths) { - var conflictsByUniquePaths: Set[String] = Set.empty[String] - conflicts.foreach(path => { - conflictsByUniquePaths += s"source: ${path.head.show()} - sink: ${path.last.show()}" - }) - conflictsByUniquePaths - } else { - conflicts.map(p => p.toString) - } - } + def reportConflicts(): scala.collection.Set[List[GraphNode]] = findConflictingPaths() def findConflictingPaths(): scala.collection.Set[List[GraphNode]] = { if (fullGraph) { diff --git a/modules/core/src/main/scala/br/unb/cic/soot/svfa/SVFA.scala b/modules/core/src/main/scala/br/unb/cic/soot/svfa/SVFA.scala index 84cdbf0e..50e09328 100644 --- a/modules/core/src/main/scala/br/unb/cic/soot/svfa/SVFA.scala +++ b/modules/core/src/main/scala/br/unb/cic/soot/svfa/SVFA.scala @@ -27,10 +27,8 @@ abstract class SVFA extends SootConfiguration { svg.toDotModel() } - def reportConflictsSVG( - useUniquePaths: Boolean = false - ): collection.Set[String] = { - svg.reportConflicts(useUniquePaths) + def reportConflictsSVG(): scala.collection.Set[List[GraphNode]] = { + svg.reportConflicts() } def executionTime(): Double = { diff --git a/modules/core/src/main/scala/br/unb/cic/soot/svfa/jimple/JSVFA.scala b/modules/core/src/main/scala/br/unb/cic/soot/svfa/jimple/JSVFA.scala index a9692466..0bf01409 100644 --- a/modules/core/src/main/scala/br/unb/cic/soot/svfa/jimple/JSVFA.scala +++ b/modules/core/src/main/scala/br/unb/cic/soot/svfa/jimple/JSVFA.scala @@ -317,7 +317,7 @@ abstract class JSVFA case (p: Local, q: ArrayRef) => // p = q[i] loadArrayRule(assignStmt.stmt, q, method, defs) case (p: Local, q: InvokeExpr) => - invokeRule(assignStmt, q, method, defs) // call a method + invokeRule(assignStmt, q, method, defs) // p = myObject.method() : call a method and assign its return value to a local variable case (p: Local, q: Local) => copyRule(assignStmt.stmt, q, method, defs) case (p: Local, _) => copyRuleInvolvingExpressions(assignStmt.stmt, method, defs) @@ -367,10 +367,18 @@ abstract class JSVFA }) } - /** Handles invocation rules for a call statement by traversing the call - * graph. This method avoids infinite recursion and limits the traversal - * depth for performance. - */ + /** + * Handles invocation rules for a call statement by traversing the call + * graph. This method avoids infinite recursion and limits the traversal + * depth for performance. + * + * i.e: + * + * myObject.method() + * myObject.method(q) + * this.method() + * this.method(q) + */ private def invokeRule( callStmt: Statement, exp: InvokeExpr, @@ -1039,14 +1047,28 @@ abstract class JSVFA case v => v } - /** CASE 1: UPDATE EDGE(S) "FROM" each stmt where the variable, passed as an - * argument, is defined. "TO" stmt where the method is called (call-site - * stmt). - * - * CASE 2: ??? - * - * CASE 2: ??? - */ + /** CASE #1: UPDATE EDGE(S) "FROM" each stmt where the variable, passed as an + * argument, is defined. "TO" stmt where the method is called (call-site + * stmt). i.e: [s1 -> s2] + * + * ------------------- + * s1: p = ... + * s2: myObj.method(p) + * ------------------- + * + * CASE #2: + * TO-DO + * + * CASE #3: UPDATE EDGE(S) "FROM" from definition of base object "TO" where it + * calls any of its methods. The expression must be type (invoke). + * i.e: [s1 -> s2] + * + * ------------------- + * s1: myObj = new Object() + * s2: myObj.method() + * ------------------- + * + */ private def defsToCallOfSinkMethod( stmt: Statement, exp: InvokeExpr, @@ -1054,7 +1076,7 @@ abstract class JSVFA defs: SimpleLocalDefs ) = { - // CASE 1 + // CASE #1 exp.getArgs .stream() .filter(a => a.isInstanceOf[Local]) @@ -1069,22 +1091,21 @@ abstract class JSVFA updateGraph( source, target - ) // update 'edge(s)' FROM "declaration stmt(s) for args" TO "call-site stmt" (current stmt) + ) }) - // CASE 2 + // CASE #2 if (local.getType.isInstanceOf[ArrayType]) { val stores = arrayStores.getOrElseUpdate(local, List()) stores.foreach(sourceStmt => { val source = createNode(caller, sourceStmt) val target = createNode(caller, targetStmt) - updateGraph(source, target) // add comment + updateGraph(source, target) }) } }) - // CASE 3 - // edges from definition to base object of an invoke expression + // CASE #3 if (isFieldSensitiveAnalysis() && exp.isInstanceOf[InstanceInvokeExpr]) { if (exp.asInstanceOf[InstanceInvokeExpr].getBase.isInstanceOf[Local]) { val local = @@ -1095,7 +1116,7 @@ abstract class JSVFA .forEach(sourceStmt => { val source = createNode(caller, sourceStmt) val target = createNode(caller, targetStmt) - updateGraph(source, target) // add comment + updateGraph(source, target) }) } } diff --git a/modules/core/src/test/scala/br/unb/cic/metrics/TestResult.scala b/modules/core/src/test/scala/br/unb/cic/metrics/TestResult.scala index 11641a8c..689bcca6 100644 --- a/modules/core/src/test/scala/br/unb/cic/metrics/TestResult.scala +++ b/modules/core/src/test/scala/br/unb/cic/metrics/TestResult.scala @@ -191,9 +191,9 @@ trait TestResult { ) val header = - "| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | Execution Time |" + "| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score |" val sep = - "|:--------------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:|:--------------:|" + "|:--------------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:|" println(header) println(sep) var totalFound = 0 @@ -229,7 +229,7 @@ trait TestResult { val totalF1 = f1Score() val totalStatus = s"${totalPassed}/${totalTests}" println( - f"| TOTAL | ${totalFound}%5d | ${totalExpected}%8d | ${totalStatus}%6s | ${totalTP}%2d | ${totalFP}%2d | ${totalFN}%3d | ${totalPrec}%9.2f | ${totalRec}%6.2f | ${totalF1}%7.2f | ${totalExecutionTime}%9.2f ms |" + f"| TOTAL | ${totalFound}%5d | ${totalExpected}%8d | ${totalStatus}%6s | ${totalTP}%2d | ${totalFP}%2d | ${totalFN}%3d | ${totalPrec}%9.2f | ${totalRec}%6.2f | ${totalF1}%7.2f |" ) } } diff --git a/modules/securibench/src/docs-metrics/flowdroid/flowdroid-metrics.md b/modules/securibench/src/docs-metrics/flowdroid/flowdroid-metrics.md index 258e3280..769cd2e7 100644 --- a/modules/securibench/src/docs-metrics/flowdroid/flowdroid-metrics.md +++ b/modules/securibench/src/docs-metrics/flowdroid/flowdroid-metrics.md @@ -1,21 +1,24 @@ #### FLOWDROID metrics -### SUMMARY +### SUMMARY (*computed in November 2025.*) > failed: 36, passed: 67 of 103 tests. `(65.05%)` -| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | -|----------------|-------|----------|--------|----|----|----|-----------|--------|------| -| Aliasing | 11 | 11 | 4/6 | 9 | 1 | 1 | 0.90 | 0.90 | 0.90 | -| Arrays | 14 | 9 | 6/10 | 6 | 5 | 0 | 0.55 | 1.00 | 0.71 | -| Basic | 38 | 61 | 26/42 | 33 | 1 | 24 | 0.97 | 0.58 | 0.73 | -| Collections | 14 | 15 | 11/14 | 12 | 1 | 2 | 0.92 | 0.86 | 0.89 | -| Datastructures | 5 | 5 | 4/6 | 3 | 1 | 1 | 0.75 | 0.75 | 0.75 | -| Factories | 1 | 3 | 1/3 | 1 | 0 | 2 | 1.00 | 0.33 | 0.50 | -| Inter | 15 | 18 | 11/14 | 13 | 0 | 3 | 1.00 | 0.81 | 0.90 | -| Session | 0 | 3 | 0/3 | 0 | 0 | 3 | 0.00 | 0.00 | 0.00 | -| StrongUpdates | 0 | 1 | 4/5 | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| TOTAL | 98 | 126 | 67/103 | 77 | 9 | 37 | 0.90 | 0.68 | 0.77 | +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | Pass Rate | +|----------------|-------|----------|--------|----|----|----|-----------|--------|------|-----------| +| Aliasing | 11 | 11 | 4/6 | 9 | 1 | 1 | 0.90 | 0.90 | 0.90 | 66.67% | +| Arrays | 14 | 9 | 6/10 | 6 | 5 | 0 | 0.55 | 1.00 | 0.71 | 60% | +| Basic | 38 | 61 | 26/42 | 33 | 1 | 24 | 0.97 | 0.58 | 0.73 | 61.90% | +| Collections | 14 | 15 | 11/14 | 12 | 1 | 2 | 0.92 | 0.86 | 0.89 | 78.57% | +| Datastructures | 5 | 5 | 4/6 | 3 | 1 | 1 | 0.75 | 0.75 | 0.75 | 66.67% | +| Factories | 1 | 3 | 1/3 | 1 | 0 | 2 | 1.00 | 0.33 | 0.50 | 33.33% | +| Inter | 15 | 18 | 11/14 | 13 | 0 | 3 | 1.00 | 0.81 | 0.90 | 78.57% | +| Session | 0 | 3 | 0/3 | 0 | 0 | 3 | 0.00 | 0.00 | 0.00 | 0% | +| StrongUpdates | 0 | 1 | 4/5 | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | 80% | +| Pred | - | - | - | - | - | - | - | - | - | - | +| Reflection | - | - | - | - | - | - | - | - | - | - | +| Sanitizers | - | - | - | - | - | - | - | - | - | - | +| **TOTAL** | 98 | 126 | 67/103 | 77 | 9 | 37 | 0.90 | 0.68 | 0.77 | 65.05% | According to Flowdroid Paper (https://www.bodden.de/pubs/far+14flowdroid.pdf), they computed the next values. @@ -38,7 +41,7 @@ According to Flowdroid Paper (https://www.bodden.de/pubs/far+14flowdroid.pdf), t | StrongUpdate | 0/0 | 0 | | **TOTAL** | 117/121 | 9 | -### Details +### DETAILS - ✅ : PASSED; ❌ : FAIL @@ -55,7 +58,7 @@ According to Flowdroid Paper (https://www.bodden.de/pubs/far+14flowdroid.pdf), t | TOTAL | 11 | 11 | 4/6 | 9 | 1 | 1 | 0.90 | 0.90 | 0.90 | -- **ArraysTest** - failed: 4, passed: 6 of 10 tests. `(60.00%)` +- **ArraysTest** - failed: 4, passed: 6 of 10 tests. `(60%)` | Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | |----------|-------|----------|--------|----|----|----|-----------|--------|------| @@ -197,7 +200,7 @@ According to Flowdroid Paper (https://www.bodden.de/pubs/far+14flowdroid.pdf), t | TOTAL | 0 | 3 | 0/3 | 0 | 0 | 3 | 0.00 | 0.00 | 0.00 | -- **StrongUpdateTest** - failed: 1, passed: 4 of 5 tests. `(80.00%)` +- **StrongUpdateTest** - failed: 1, passed: 4 of 5 tests. `(80%)` | Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | |----------------|-------|----------|--------|----|----|----|-----------|--------|------| diff --git a/modules/securibench/src/docs-metrics/joana/joana-metrics.md b/modules/securibench/src/docs-metrics/joana/joana-metrics.md index b84e7d2e..89056fbc 100644 --- a/modules/securibench/src/docs-metrics/joana/joana-metrics.md +++ b/modules/securibench/src/docs-metrics/joana/joana-metrics.md @@ -1,39 +1,42 @@ -## Joana Metrics +## JOANA Metrics -> SUMMARY +> SUMMARY (*computed in November 2025.*) -- **Securibench** - failed: 32, passed: 71 of 103 tests. `(68.93%)` +- **Securibench** - failed: 37, passed: 85 of 122 tests. `(69.67%)` -| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | -|----------------|-------|----------|--------|----|----|----|-----------|--------|------| -| Aliasing | 6 | 11 | 2/6 | 2 | 2 | 7 | 0.50 | 0.22 | 0.31 | -| Arrays | 10 | 9 | 9/10 | 9 | 1 | 0 | 0.90 | 1.00 | 0.95 | -| Basic | 45 | 61 | 25/42 | 26 | 6 | 22 | 0.81 | 0.54 | 0.65 | -| Collections | 15 | 15 | 14/14 | 15 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Datastructures | 6 | 5 | 5/6 | 5 | 1 | 0 | 0.83 | 1.00 | 0.91 | -| Factories | 3 | 3 | 3/3 | 3 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Inter | 13 | 18 | 9/14 | 9 | 0 | 5 | 1.00 | 0.64 | 0.78 | -| Session | 3 | 3 | 3/3 | 3 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| StrongUpdates | 5 | 1 | 1/5 | 1 | 4 | 0 | 0.20 | 1.00 | 0.33 | -| TOTAL | 106 | 126 | 71/103 | 73 | 14 | 34 | 0.84 | 0.68 | 0.75 | +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | Pass rate | +|----------------|-------|----------|--------|----|----|----|-----------|--------|-------|-----------| +| Aliasing | 6 | 11 | 2/6 | 2 | 2 | 7 | 0.50 | 0.22 | 0.31 | 33.33% | +| Arrays | 10 | 9 | 9/10 | 9 | 1 | 0 | 0.90 | 1.00 | 0.95 | 90% | +| Basic | 45 | 61 | 25/42 | 26 | 6 | 22 | 0.81 | 0.54 | 0.65 | 59.52% | +| Collections | 15 | 14 | 13/14 | 14 | 1 | 0 | 0.93 | 1.00 | 0.96 | 92.86% | +| Datastructures | 6 | 5 | 5/6 | 5 | 1 | 0 | 0.83 | 1.00 | 0.91 | 83.33% | +| Factories | 3 | 3 | 3/3 | 3 | 0 | 0 | 1.00 | 1.00 | 1.00 | 100% | +| Inter | 13 | 16 | 11/14 | 11 | 0 | 3 | 1.00 | 0.79 | 0.88 | 78.57% | +| Session | 3 | 3 | 3/3 | 3 | 0 | 0 | 1.00 | 1.00 | 1.00 | 100% | +| StrongUpdates | 5 | 1 | 1/5 | 1 | 4 | 0 | 0.20 | 1.00 | 0.33 | 20% | +| Pred | 8 | 5 | 6/9 | 5 | 3 | 0 | 0.63 | 1.00 | 0.77 | 66.67% | +| Reflection | 3 | 4 | 3/4 | 3 | 0 | 1 | 1.00 | 0.75 | 0.86 | 75% | +| Sanitizer | 6 | 6 | 4/6 | 4 | 1 | 1 | 0.80 | 0.80 | 0.80 | 66.67% | +| TOTAL | 123 | 138 | 85/122 | 86 | 19 | 34 | 0.82 | 0.72 | 0.77 | 69.67% | > DETAILS - **AliasingTest** - failed: 4, passed: 2 of 6 tests. `(33.33%)` -| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | -|-----------|-------|----------|--------|----|----|----|-----------|--------|------| -| Aliasing1 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Aliasing2 | 1 | 0 | ❌ | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | -| Aliasing3 | 1 | 0 | ❌ | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | -| Aliasing4 | 1 | 2 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| Aliasing5 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Aliasing6 | 1 | 7 | ❌ | 0 | 0 | 6 | 0.00 | 0.00 | 0.00 | -| TOTAL | 6 | 11 | 2/6 | 2 | 2 | 7 | 0.50 | 0.22 | 0.31 | +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | +|------------|-------|----------|--------|----|----|----|-----------|--------|------| +| Aliasing1 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Aliasing2 | 1 | 0 | FAIL | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | +| Aliasing3 | 1 | 0 | FAIL | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | +| Aliasing4 | 1 | 2 | FAIL | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | +| Aliasing5 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Aliasing6 | 1 | 7 | FAIL | 0 | 0 | 6 | 0.00 | 0.00 | 0.00 | +| TOTAL | 6 | 11 | 2/6 | 2 | 2 | 7 | 0.50 | 0.22 | 0.31 | -- **ArraysTest** - failed: 1, passed: 9 of 10 tests. `(90.00%)` +- **ArraysTest** - failed: 1, passed: 9 of 10 tests. `(90%)` | Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | |----------|-------|----------|--------|----|----|----|-----------|--------|------| @@ -52,54 +55,54 @@ - **BasicTest** - failed: 17, passed: 25 of 42 tests. `(59.52%)` -| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | -|----------|-------|----------|--------|----|----|----|-----------|--------|------| -| Basic1 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic2 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic3 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic4 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic5 | 1 | 3 | ❌ | 0 | 0 | 2 | 0.00 | 0.00 | 0.00 | -| Basic6 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic7 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic8 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic9 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic10 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic11 | 1 | 2 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| Basic12 | 1 | 2 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| Basic13 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| Basic14 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| Basic15 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic16 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic17 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic18 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic19 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| Basic20 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| Basic21 | 0 | 4 | ❌ | 0 | 0 | 4 | 0.00 | 0.00 | 0.00 | -| Basic22 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic23 | 2 | 3 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| Basic24 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic25 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic26 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic27 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic28 | 1 | 2 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| Basic29 | 1 | 2 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| Basic30 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic31 | 0 | 3 | ❌ | 0 | 0 | 3 | 0.00 | 0.00 | 0.00 | -| Basic32 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic33 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| Basic34 | 2 | 2 | ✅ | 2 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic35 | 12 | 6 | ❌ | 0 | 6 | 0 | 0.00 | 0.00 | 0.00 | -| Basic36 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic37 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic38 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic39 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Basic40 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| Basic41 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| Basic42 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| TOTAL | 45 | 61 | 25/42 | 26 | 6 | 22 | 0.81 | 0.54 | 0.65 | - - -- **CollectionTest** - failed: 0, passed: 14 of 14 tests. `(100.00%)` +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | +|----------|--------|-----------|---------|----|----|----|-----------|--------|------| +| Basic1 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic2 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic3 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic4 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic5 | 1 | 3 | ❌ | 0 | 0 | 2 | 0.00 | 0.00 | 0.00 | +| Basic6 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic7 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic8 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic9 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic10 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic11 | 1 | 2 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | +| Basic12 | 1 | 2 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | +| Basic13 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | +| Basic14 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | +| Basic15 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic16 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic17 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic18 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic19 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | +| Basic20 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | +| Basic21 | 0 | 4 | ❌ | 0 | 0 | 4 | 0.00 | 0.00 | 0.00 | +| Basic22 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic23 | 2 | 3 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | +| Basic24 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic25 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic26 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic27 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic28 | 1 | 2 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | +| Basic29 | 1 | 2 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | +| Basic30 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic31 | 0 | 3 | ❌ | 0 | 0 | 3 | 0.00 | 0.00 | 0.00 | +| Basic32 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic33 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | +| Basic34 | 2 | 2 | ✅ | 2 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic35 | 12 | 6 | ❌ | 0 | 6 | 0 | 0.00 | 0.00 | 0.00 | +| Basic36 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic37 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic38 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic39 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic40 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | +| Basic41 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | +| Basic42 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | +| TOTAL | 45 | 61 | 25/42 | 26 | 6 | 22 | 0.81 | 0.54 | 0.65 | + + +- **CollectionTest** - failed: 1, passed: 13 of 14 tests. `(92.86%)` | Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | |---------------|-------|----------|--------|----|----|----|-----------|--------|------| @@ -111,13 +114,13 @@ | Collections6 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | | Collections7 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | | Collections8 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Collections9 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Collections9 | 1 | 0 | FAIL | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | *FLAKY | Collections10 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | | Collections11 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | | Collections12 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | | Collections13 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | | Collections14 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| TOTAL | 15 | 15 | 14/14 | 15 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| TOTAL | 15 | 14 | 13/14 | 14 | 1 | 0 | 0.93 | 1.00 | 0.96 | - **DataStructureTest** - failed: 1, passed: 5 of 6 tests. `(83.33%)` @@ -133,7 +136,7 @@ | TOTAL | 6 | 5 | 5/6 | 5 | 1 | 0 | 0.83 | 1.00 | 0.91 | -- **FactoryTest** - failed: 0, passed: 3 of 3 tests. `(100.00%)` +- **FactoryTest** - failed: 0, passed: 3 of 3 tests. `(100%)` | Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | |------------|-------|----------|--------|----|----|----|-----------|--------|------| @@ -143,28 +146,28 @@ | TOTAL | 3 | 3 | 3/3 | 3 | 0 | 0 | 1.00 | 1.00 | 1.00 | -- **InterTest** - failed: 5, passed: 9 of 14 tests. `(64.29%)` +- **InterTest** - failed: 3, passed: 11 of 14 tests. `(78.57%)` -| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | -|---------|-------|----------|--------|----|----|----|-----------|--------|------| -| Inter1 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Inter2 | 1 | 2 | FAIL | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| Inter3 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Inter4 | 1 | 2 | FAIL | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| Inter5 | 1 | 2 | FAIL | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| Inter6 | 0 | 1 | FAIL | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| Inter7 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Inter8 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Inter9 | 1 | 2 | FAIL | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | -| Inter10 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Inter11 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Inter12 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Inter13 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| Inter14 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | -| TOTAL | 13 | 18 | 9/14 | 9 | 0 | 5 | 1.00 | 0.64 | 0.78 | +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | +|---------|--------|-----------|---------|-----|-----|-----|------------|---------|------| +| Inter1 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Inter2 | 1 | 2 | FAIL | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | +| Inter3 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Inter4 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Inter5 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Inter6 | 0 | 1 | FAIL | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | +| Inter7 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Inter8 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Inter9 | 1 | 2 | FAIL | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | +| Inter10 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Inter11 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Inter12 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Inter13 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Inter14 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| TOTAL | 13 | 16 | 11/14 | 11 | 0 | 3 | 1.00 | 0.79 | 0.88 | -- **SessionTest** - failed: 0, passed: 3 of 3 tests. `(100.00%)` +- **SessionTest** - failed: 0, passed: 3 of 3 tests. `(100%)` | Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | |----------|-------|----------|--------|----|----|----|-----------|--------|------| @@ -174,7 +177,7 @@ | TOTAL | 3 | 3 | 3/3 | 3 | 0 | 0 | 1.00 | 1.00 | 1.00 | -- **StrongUpdateTest** - failed: 4, passed: 1 of 5 tests. `(20.00%)` +- **StrongUpdateTest** - failed: 4, passed: 1 of 5 tests. `(20%)` | Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | |----------------|-------|----------|--------|----|----|----|-----------|--------|------| @@ -183,4 +186,46 @@ | StrongUpdates3 | 1 | 0 | ❌ | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | | StrongUpdates4 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | | StrongUpdates5 | 1 | 0 | ❌ | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | -| TOTAL | 5 | 1 | 1/5 | 1 | 4 | 0 | 0.20 | 1.00 | 0.33 | \ No newline at end of file +| TOTAL | 5 | 1 | 1/5 | 1 | 4 | 0 | 0.20 | 1.00 | 0.33 | + + +> Extras + +- **PredTest** - failed: 3, passed: 6 of 9 tests. `(66.67%)` + +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | +|--------|---------|-----------|---------|-----|-----|-----|------------|---------|------| +| Pred1 | 0 | 0 | PASS | 0 | 0 | 0 | 0.00 | 0.00 | 0.00 | +| Pred2 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Pred3 | 1 | 0 | FAIL | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | +| Pred4 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Pred5 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Pred6 | 1 | 0 | FAIL | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | +| Pred7 | 1 | 0 | FAIL | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | +| Pred8 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Pred9 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| TOTAL | 8 | 5 | 6/9 | 5 | 3 | 0 | 0.63 | 1.00 | 0.77 | + + +- **ReflectionTest** - failed: 1, passed: 3 of 4 tests. `(75%)` + +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | +|--------|--------|-----------|---------|-----|-----|-----|------------|---------|------| +| Refl1 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Refl2 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Refl3 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Refl4 | 0 | 1 | FAIL | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | +| TOTAL | 3 | 4 | 3/4 | 3 | 0 | 1 | 1.00 | 0.75 | 0.86 | + + +- **SanitizersTest** - failed: 2, passed: 4 of 6 tests. `(66.67%)` + +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F1 | +|-------------|---------|------------|----------|------|------|------|------------|---------|------| +| Sanitizers1 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Sanitizers2 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Sanitizers3 | 1 | 0 | FAIL | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | +| Sanitizers4 | 1 | 2 | FAIL | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | +| Sanitizers5 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Sanitizers6 | 1 | 1 | PASS | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| TOTAL | 6 | 6 | 4/6 | 4 | 1 | 1 | 0.80 | 0.80 | 0.80 | \ No newline at end of file diff --git a/modules/securibench/src/docs-metrics/jsvfa/jsvfa-metrics-v0.3.0.md b/modules/securibench/src/docs-metrics/jsvfa/jsvfa-metrics-v0.3.0.md index 60eedfe4..207910cf 100644 --- a/modules/securibench/src/docs-metrics/jsvfa/jsvfa-metrics-v0.3.0.md +++ b/modules/securibench/src/docs-metrics/jsvfa/jsvfa-metrics-v0.3.0.md @@ -2,7 +2,7 @@ > SUMMARY (*computed in June 2023.*) -- **securibench.micro** - failed: 46, passed: 57 of 103 tests - (55.34%) +- **securibench.micro** - failed: 59, passed: 63 of 122 tests - (51.63%) | Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | Pass Rate | |:--------------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:|:---------:| @@ -15,8 +15,10 @@ | inter | 10 | 18 | 7/14 | 8 | 0 | 8 | 1.00 | 0.50 | 0.67 | 50 | | session | 0 | 3 | 0/3 | 0 | 0 | 3 | 0.00 | 0.00 | 0.00 | 0 | | strong_updates | 0 | 1 | 4/5 | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | 80 | -| TOTAL | 94 | 126 | 57/103 | 75 | 8 | 40 | 0.90 | 0.65 | 0.75 | 55.34 | - +| Pred | 8 | 5 | 6/9 | 5 | 3 | 0 | 0.63 | 1.00 | 0.77 | 66.67% | +| Reflection | 0 | 4 | 0/4 | 0 | 0 | 4 | 0.00 | 0.00 | 0.00 | 0% | +| Sanitizers | 0 | 4 | 0/6 | 0 | 0 | 6 | 0.00 | 0.00 | 0.00 | 0% | +| TOTAL | 102 | 139 | 63/122 | 80 | 11 | 50 | 0.88 | 0.62 | 0.72 | 51.64 | > Details diff --git a/modules/securibench/src/docs-metrics/jsvfa/jsvfa-metrics-v0.6.1.md b/modules/securibench/src/docs-metrics/jsvfa/jsvfa-metrics-v0.6.1.md new file mode 100644 index 00000000..1c0daa9a --- /dev/null +++ b/modules/securibench/src/docs-metrics/jsvfa/jsvfa-metrics-v0.6.1.md @@ -0,0 +1,236 @@ + +> SUMMARY (*computed in November 2025.*) + +- **securibench.micro** - failed: 47, passed: 75 of 122 tests - (61.48%) + +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | Pass Rate | +|:--------------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:|:---------:| +| Aliasing | 10 | 12 | 2/6 | 8 | 1 | 3 | 0.89 | 0.73 | 0.80 | 33.33% | +| Arrays | 11 | 9 | 5/10 | 5 | 4 | 2 | 0.56 | 0.71 | 0.63 | 50% | +| Basic | 57 | 60 | 38/42 | 55 | 1 | 4 | 0.98 | 0.93 | 0.95 | 90.48% | +| Collections | 8 | 15 | 5/14 | 5 | 1 | 8 | 0.83 | 0.38 | 0.52 | 35.71% | +| Datastructures | 5 | 5 | 4/6 | 4 | 1 | 1 | 0.80 | 0.80 | 0.80 | 66.67% | +| Factories | 4 | 3 | 2/3 | 2 | 1 | 0 | 0.67 | 1.00 | 0.80 | 66.67% | +| Inter | 12 | 18 | 8/14 | 9 | 0 | 6 | 1.00 | 0.60 | 0.75 | 57.14% | +| Session | 0 | 3 | 0/3 | 0 | 0 | 3 | 0.00 | 0.00 | 0.00 | 0% | +| StrongUpdates | 3 | 1 | 3/5 | 1 | 2 | 0 | 0.33 | 1.00 | 0.50 | 60% | +| Pred | 8 | 5 | 6/9 | 5 | 3 | 0 | 0.63 | 1.00 | 0.77 | 66.67% | +| Reflection | 0 | 4 | 0/4 | 0 | 0 | 4 | 0.00 | 0.00 | 0.00 | 0% | +| Sanitizers | 2 | 6 | 2/6 | 1 | 0 | 4 | 1.00 | 0.20 | 0.33 | 33.33% | +| TOTAL | 120 | 141 | 75/122 | 95 | 14 | 35 | 0.87 | 0.73 | 0.79 | 61.48% | + + +> Details + +[//]: # () + +[//]: # ) + +- **securibench.micro.aliasing** - failed: 4, passed: 2 of 6 tests - (33.33%) + +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | +|:---------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:| +| Aliasing1 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Aliasing2 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [i] +| Aliasing3 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [ii] +| Aliasing4 | 2 | 1 | ❌ | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | * issue [i] +| Aliasing5 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [iv] +| Aliasing6 | 7 | 7 | ✅ | 7 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| TOTAL | 10 | 12 | 2/6 | 8 | 1 | 3 | 0.89 | 0.73 | 0.80 | + + +- **securibench.micro.arrays** - failed: 5, passed: 5 of 10 tests - (50.0%) + +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | +|:--------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:| +| Arrays1 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Arrays2 | 3 | 1 | ❌ | 0 | 2 | 0 | 0.00 | 0.00 | 0.00 | * issue [ii] +| Arrays3 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Arrays4 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Arrays5 | 1 | 0 | ❌ | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | * issue [ii] +| Arrays6 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Arrays7 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Arrays8 | 2 | 1 | ❌ | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | * issue [ii] +| Arrays9 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [ii] +| Arrays10 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [ii] +| TOTAL | 11 | 9 | 5/10 | 5 | 4 | 2 | 0.56 | 0.71 | 0.63 | + + +- **securibench.micro.basic** - failed: 4, passed: 38 of 42 tests - (90.48%) + +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | +|:-------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:| +| Basic0 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic1 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic2 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic3 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic4 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic5 | 3 | 3 | ✅ | 3 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic6 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic7 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic8 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic9 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic10 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic11 | 2 | 2 | ✅ | 2 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic12 | 2 | 2 | ✅ | 2 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic13 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic14 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic15 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic16 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic17 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic18 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic19 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic20 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic21 | 4 | 4 | ✅ | 4 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic22 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic23 | 3 | 3 | ✅ | 3 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic24 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic25 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic26 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic27 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic28 | 2 | 2 | ✅ | 0 | 0 | 2 | 1.00 | 1.00 | 1.00 | +| Basic29 | 2 | 2 | ✅ | 2 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic30 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic31 | 0 | 2 | ❌ | 0 | 0 | 2 | 0.00 | 0.00 | 0.00 | * issue [iii] +| Basic32 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic33 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic34 | 2 | 2 | ✅ | 2 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic35 | 6 | 6 | ✅ | 6 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic36 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [iii] +| Basic37 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic38 | 2 | 1 | ❌ | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | * issue [iii] +| Basic39 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic41 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Basic42 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [iv] +| TOTAL | 57 | 60 | 38/42 | 55 | 1 | 4 | 0.98 | 0.93 | 0.95 | + + +- **securibench.micro.collections** - failed: 9, passed: 5 of 14 tests - (35.71%) + +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | +|:-------------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:| +| Collections1 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Collections2 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Collections3 | 1 | 2 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [iv] +| Collections4 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Collections5 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [iv] +| Collections6 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [iv] +| Collections7 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [iv] +| Collections8 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [iv] +| Collections9 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [iv] +| Collections10 | 2 | 1 | ❌ | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | * issue [iv] +| Collections11 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Collections12 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [iv] +| Collections13 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [iv] +| Collections14 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| TOTAL | 8 | 15 | 5/14 | 5 | 1 | 8 | 0.83 | 0.38 | 0.52 | + + +- **securibench.micro.datastructures** - failed: 2, passed: 4 of 6 tests - (66.67%) + +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | +|:---------------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:| +| Datastructures1 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Datastructures2 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Datastructures3 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Datastructures4 | 1 | 0 | ❌ | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | * issue [iv] +| Datastructures5 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [iv] +| Datastructures6 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| TOTAL | 5 | 5 | 4/6 | 4 | 1 | 1 | 0.80 | 0.80 | 0.80 | + + +- **securibench.micro.factories** - failed: 1, passed: 2 of 3 tests - (66.67%) + +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | +|:----------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:| +| Factories1 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Factories2 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Factories3 | 2 | 1 | ❌ | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | * issue [iv] +| TOTAL | 4 | 3 | 2/3 | 2 | 1 | 0 | 0.67 | 1.00 | 0.80 | + + +- **securibench.micro.inter** - failed: 6, passed: 8 of 14 tests - (57.14%) + +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | +|:-------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:| +| Inter1 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Inter2 | 2 | 2 | ✅ | 2 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Inter3 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Inter4 | 1 | 2 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [i] +| Inter5 | 1 | 2 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [i] +| Inter6 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [v] +| Inter7 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Inter8 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Inter9 | 1 | 2 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [iv] +| Inter10 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Inter11 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [ix] +| Inter12 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [iv] +| Inter13 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Inter14 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| TOTAL | 12 | 18 | 8/14 | 9 | 0 | 6 | 1.00 | 0.60 | 0.75 | + + +- **securibench.micro.session** - failed: 3, passed: 0 of 3 tests - (0.0%) + +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | +|:--------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:| +| Session1 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [iii] +| Session2 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [iii] +| Session3 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [iii] +| TOTAL | 0 | 3 | 0/3 | 0 | 0 | 3 | 0.00 | 0.00 | 0.00 | + + +- **securibench.micro.strong_updates** - failed: 2, passed: 3 of 5 tests - (60.0%) + +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | +|:--------------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:| +| StrongUpdates1 | 0 | 0 | ✅ | 0 | 0 | 0 | 0.00 | 0.00 | 0.00 | +| StrongUpdates2 | 0 | 0 | ✅ | 0 | 0 | 0 | 0.00 | 0.00 | 0.00 | +| StrongUpdates3 | 1 | 0 | ❌ | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | * issue [vi] +| StrongUpdates4 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| StrongUpdates5 | 1 | 0 | ❌ | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | * issue [vi] +| TOTAL | 3 | 1 | 3/5 | 1 | 2 | 0 | 0.33 | 1.00 | 0.50 | + + +> Extra Tests + +These tests are not executed by Flowdroid + +- **securibench.micro.pred** - failed: 3, passed: 6 of 9 tests - (66.67%) + +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | +|:-----:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:| +| Pred1 | 0 | 0 | ✅ | 0 | 0 | 0 | 0.00 | 0.00 | 0.00 | +| Pred2 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Pred3 | 1 | 0 | ❌ | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | * issue [vii] +| Pred4 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Pred5 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Pred6 | 1 | 0 | ❌ | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | * issue [vii] +| Pred7 | 1 | 0 | ❌ | 0 | 1 | 0 | 0.00 | 0.00 | 0.00 | * issue [vii] +| Pred8 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Pred9 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| TOTAL | 8 | 5 | 6/9 | 5 | 3 | 0 | 0.63 | 1.00 | 0.77 | + + +- **securibench.micro.reflection** - failed: 4, passed: 0 of 4 tests - (0.0%) + +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | +|:-----:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:| +| Refl1 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [v] +| Refl2 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [v] +| Refl3 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [v] +| Refl4 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [v] +| TOTAL | 0 | 4 | 0/4 | 0 | 0 | 4 | 0.00 | 0.00 | 0.00 | + + +- **securibench.micro.sanitizers** - failed: 4, passed: 2 of 6 tests - (33.33%) + +| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | +|:-----------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:| +| Sanitizers1 | 1 | 1 | ✅ | 1 | 0 | 0 | 1.00 | 1.00 | 1.00 | +| Sanitizers2 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [viii] +| Sanitizers3 | 0 | 0 | ✅ | 0 | 0 | 0 | 0.00 | 0.00 | 0.00 | +| Sanitizers4 | 1 | 2 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [viii] +| Sanitizers5 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [iii] +| Sanitizers6 | 0 | 1 | ❌ | 0 | 0 | 1 | 0.00 | 0.00 | 0.00 | * issue [viii] +| TOTAL | 2 | 6 | 2/6 | 1 | 0 | 4 | 1.00 | 0.20 | 0.33 | \ No newline at end of file diff --git a/modules/securibench/src/test/java/securibench/micro/aliasing/Aliasing7.java b/modules/securibench/src/test/java/securibench/micro/aliasing/Aliasing7.java deleted file mode 100644 index 4e6281a9..00000000 --- a/modules/securibench/src/test/java/securibench/micro/aliasing/Aliasing7.java +++ /dev/null @@ -1,52 +0,0 @@ -/** - @author Benjamin Livshits - - $Id: Aliasing6.java,v 1.1 2006/04/21 17:14:27 livshits Exp $ - */ -package securibench.micro.aliasing; - -import java.io.IOException; -import java.io.PrintWriter; - -import javax.servlet.http.HttpServletRequest; -import javax.servlet.http.HttpServletResponse; - -import securibench.micro.BasicTestCase; -import securibench.micro.MicroTestCase; - -/** - * @servlet description="false positive of aliasing with copy propagation" - * @servlet vuln_count = "0" - * */ -public class Aliasing7 extends BasicTestCase implements MicroTestCase { - private static final String FIELD_NAME = "name"; - - protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException { - String[] names = new String[2]; - names[0] = req.getParameter(FIELD_NAME); - names[1] = "Foo"; - Object - o1, o2, o3, o4, o5, o6, o7, o8, o9, o10, o11, o12, o13, o14, o15, o16, o17, o18, o19, o20, - o21, o22, o23, o24, o25, o26, o27, o28, o29, o30, o31, o32, o33, o34, o35, o36, o37, o38, o39, o40; - o1 = o2 = o3 = o4 = o5 = o6 = o7 = o8 = o9 = o10 = o11 = o12 = o13 = o14 = o15 = o16 = o17 = o18 = o19 = o20 = - o21 = o22 = o23 = o24 = o25 = o26 = o27 = o28 = o29 = o30 = o31 = o32 = o33 = o34 = o35 = o36 = o37 = o38 = o39 = o40 = - names[1]; - - PrintWriter writer = resp.getWriter(); - writer.println(o1); /* OK */ - writer.println(o2); /* OK */ - writer.println(o3); /* OK */ - writer.println(o4); /* OK */ - writer.println(o32); /* OK */ - writer.println(o37); /* OK */ - writer.println(o40); /* OK */ - } - - public String getDescription() { - return "false positive of aliasing with copy propagation"; - } - - public int getVulnerabilityCount() { - return 0; - } -} \ No newline at end of file diff --git a/modules/securibench/src/test/java/securibench/micro/aliasing/Aliasing8.java b/modules/securibench/src/test/java/securibench/micro/aliasing/Aliasing8.java deleted file mode 100644 index eedee156..00000000 --- a/modules/securibench/src/test/java/securibench/micro/aliasing/Aliasing8.java +++ /dev/null @@ -1,58 +0,0 @@ -/** - @author Benjamin Livshits - - $Id: Aliasing6.java,v 1.1 2006/04/21 17:14:27 livshits Exp $ - */ -package securibench.micro.aliasing; - -import java.io.IOException; -import java.io.PrintWriter; - -import javax.servlet.http.HttpServletRequest; -import javax.servlet.http.HttpServletResponse; - -import securibench.micro.BasicTestCase; -import securibench.micro.MicroTestCase; - -/** - * @servlet description="aliasing in an array index" - * @servlet vuln_count = "1" - * */ -public class Aliasing8 extends BasicTestCase implements MicroTestCase { - private static final String FIELD_NAME = "name"; - - protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException { - String[] names = new String[2]; - names[0] = req.getParameter(FIELD_NAME); - names[1] = "Foo"; - Object - o1, o2, o3, o4, o5, o6, o7, o8, o9, o10, o11, o12, o13, o14, o15, o16, o17, o18, o19, o20, - o21, o22, o23, o24, o25, o26, o27, o28, o29, o30, o31, o32, o33, o34, o35, o36, o37, o38, o39, o40; - o1 = o2 = o3 = o4 = o5 = o6 = o7 = o8 = o9 = o10 = o11 = o12 = o13 = o14 = o15 = o16 = o17 = o18 = o19 = o20 = - o21 = o22 = o23 = o24 = o25 = o26 = o27 = o28 = o29 = o30 = o31 = o32 = o33 = o34 = o35 = o36 = o37 = o38 = o39 = o40 = - names[1]; - - o2 = names[0]; - - PrintWriter writer = resp.getWriter(); - writer.println(o1); /* BAD */ - writer.println(o2); /* BAD */ - writer.println(o3); /* BAD */ - writer.println(o4); /* BAD */ - writer.println(o32); /* BAD */ - writer.println(o37); /* BAD */ - writer.println(o40); /* BAD */ - - sink(names); /* This should be consider a leak too, because one array element is tainted */ - } - - public String getDescription() { - return "aliasing in an array index"; - } - - public int getVulnerabilityCount() { - return 7; - } - - public void sink(String[] data) { } -} \ No newline at end of file diff --git a/modules/securibench/src/test/java/securibench/micro/aliasing/Aliasing9.java b/modules/securibench/src/test/java/securibench/micro/aliasing/Aliasing9.java deleted file mode 100644 index 73635096..00000000 --- a/modules/securibench/src/test/java/securibench/micro/aliasing/Aliasing9.java +++ /dev/null @@ -1,51 +0,0 @@ -/** - @author Benjamin Livshits - - $Id: Aliasing6.java,v 1.1 2006/04/21 17:14:27 livshits Exp $ - */ -package securibench.micro.aliasing; - -import java.io.IOException; -import java.io.PrintWriter; - -import javax.servlet.http.HttpServletRequest; -import javax.servlet.http.HttpServletResponse; - -import securibench.micro.BasicTestCase; -import securibench.micro.MicroTestCase; - -/** - * @servlet description="interprocedural aliasing in an array index" - * @servlet vuln_count = "1" - * */ -public class Aliasing9 extends BasicTestCase implements MicroTestCase { - private static final String FIELD_NAME = "name"; - - protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException { - String[] names = source(req); - Object o1, o2; - o1 = o2 = names[1]; - - o2 = names[0]; - - PrintWriter writer = resp.getWriter(); - writer.println(o1); /* OK */ - writer.println(o2); /* BAD */ - } - - public String[] source(HttpServletRequest req) { - String[] names = new String[2]; - names[0] = req.getParameter(FIELD_NAME); - names[1] = "Foo"; - - return names; - } - - public String getDescription() { - return "aliasing with copy propagation"; - } - - public int getVulnerabilityCount() { - return 7; - } -} \ No newline at end of file diff --git a/modules/securibench/src/test/scala/br/unb/cic/metrics/TestResult.scala b/modules/securibench/src/test/scala/br/unb/cic/metrics/TestResult.scala index 6b8d8654..4169d502 100644 --- a/modules/securibench/src/test/scala/br/unb/cic/metrics/TestResult.scala +++ b/modules/securibench/src/test/scala/br/unb/cic/metrics/TestResult.scala @@ -191,9 +191,9 @@ trait TestResult { ) val header = - "| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score | Execution Time |" + "| Test | Found | Expected | Status | TP | FP | FN | Precision | Recall | F-score |" val sep = - "|:--------------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:|:--------------:|" + "|:--------------:|:-----:|:--------:|:------:|:--:|:--:|:---|:---------:|:------:|:-------:|" println(header) println(sep) var totalFound = 0 @@ -212,7 +212,7 @@ trait TestResult { val f1 = f1Score(testName) val shortTestName = testName.split('.').last.padTo(14, ' ') println( - f"| $shortTestName| ${m.found}%5d | ${m.expected}%8d | ${status}%6s | ${m.truePositives}%2d | ${m.falsePositives}%2d | ${m.falseNegatives}%3d | ${prec}%9.2f | ${rec}%6.2f | ${f1}%7.2f | ${m.executionTime}%9.2f ms |" + f"| $shortTestName| ${m.found}%5d | ${m.expected}%8d | ${status}%6s | ${m.truePositives}%2d | ${m.falsePositives}%2d | ${m.falseNegatives}%3d | ${prec}%9.2f | ${rec}%6.2f | ${f1}%7.2f |" ) totalFound += m.found totalExpected += m.expected @@ -229,7 +229,7 @@ trait TestResult { val totalF1 = f1Score() val totalStatus = s"${totalPassed}/${totalTests}" println( - f"| TOTAL | ${totalFound}%5d | ${totalExpected}%8d | ${totalStatus}%6s | ${totalTP}%2d | ${totalFP}%2d | ${totalFN}%3d | ${totalPrec}%9.2f | ${totalRec}%6.2f | ${totalF1}%7.2f | ${totalExecutionTime}%9.2f ms |" + f"| TOTAL | ${totalFound}%5d | ${totalExpected}%8d | ${totalStatus}%6s | ${totalTP}%2d | ${totalFP}%2d | ${totalFN}%3d | ${totalPrec}%9.2f | ${totalRec}%6.2f | ${totalF1}%7.2f |" ) } } diff --git a/modules/securibench/src/test/scala/br/unb/cic/securibench/deprecated/SecuribenchDeprecatedExtraTest.scala b/modules/securibench/src/test/scala/br/unb/cic/securibench/deprecated/SecuribenchDeprecatedExtraTest.scala deleted file mode 100644 index 59ea1389..00000000 --- a/modules/securibench/src/test/scala/br/unb/cic/securibench/deprecated/SecuribenchDeprecatedExtraTest.scala +++ /dev/null @@ -1,176 +0,0 @@ -package br.unb.cic.securibench.deprecated - -import br.unb.cic.securibench.SecuribenchTest -import org.scalatest.FunSuite - -class SecuribenchDeprecatedExtraTest extends FunSuite { - - /** PRED TESTs - */ - - test( - "in the class Pred1 we should detect 0 conflict(s) of a Pred test case" - ) { - val svfa = new SecuribenchTest("securibench.micro.pred.Pred1", "doGet") - svfa.buildSparseValueFlowGraph() - assert(svfa.reportConflictsSVG().size == 0) - } - - test( - "in the class Pred2 we should detect 1 conflict(s) of a Pred test case" - ) { - val svfa = new SecuribenchTest("securibench.micro.pred.Pred2", "doGet") - svfa.buildSparseValueFlowGraph() - assert(svfa.reportConflictsSVG().size == 1) - } - - ignore( - "in the class Pred3 we should detect 0 conflict(s) of a Pred test case" - ) { - val svfa = new SecuribenchTest("securibench.micro.pred.Pred3", "doGet") - svfa.buildSparseValueFlowGraph() - assert(svfa.reportConflictsSVG().size == 0) - } - - test( - "in the class Pred4 we should detect 1 conflict(s) of a Pred test case" - ) { - val svfa = new SecuribenchTest("securibench.micro.pred.Pred4", "doGet") - svfa.buildSparseValueFlowGraph() - assert(svfa.reportConflictsSVG().size == 1) - } - - test( - "in the class Pred5 we should detect 1 conflict(s) of a Pred test case" - ) { - val svfa = new SecuribenchTest("securibench.micro.pred.Pred5", "doGet") - svfa.buildSparseValueFlowGraph() - assert(svfa.reportConflictsSVG().size == 1) - } - - ignore( - "in the class Pred6 we should detect 0 conflict(s) of a Pred test case" - ) { - val svfa = new SecuribenchTest("securibench.micro.pred.Pred6", "doGet") - svfa.buildSparseValueFlowGraph() - assert(svfa.reportConflictsSVG().size == 0) - } - - ignore( - "in the class Pred7 we should detect 0 conflict(s) of a Pred test case" - ) { - val svfa = new SecuribenchTest("securibench.micro.pred.Pred7", "doGet") - svfa.buildSparseValueFlowGraph() - assert(svfa.reportConflictsSVG().size == 0) - } - - test( - "in the class Pred8 we should detect 1 conflict(s) of a Pred test case" - ) { - val svfa = new SecuribenchTest("securibench.micro.pred.Pred8", "doGet") - svfa.buildSparseValueFlowGraph() - assert(svfa.reportConflictsSVG().size == 1) - } - - test( - "in the class Pred9 we should detect 1 conflict(s) of a Pred test case" - ) { - val svfa = new SecuribenchTest("securibench.micro.pred.Pred9", "doGet") - svfa.buildSparseValueFlowGraph() - assert(svfa.reportConflictsSVG().size == 1) - } - - /** REFLECTION TESTs - */ - ignore( - "in the class Refl1 we should detect 1 conflict(s) of a Reflection test case" - ) { - val svfa = - new SecuribenchTest("securibench.micro.reflection.Refl1", "doGet") - svfa.buildSparseValueFlowGraph() - assert(svfa.reportConflictsSVG().size == 1) - } - - ignore( - "in the class Refl2 we should detect 1 conflict(s) of a Reflection test case" - ) { - val svfa = - new SecuribenchTest("securibench.micro.reflection.Refl2", "doGet") - svfa.buildSparseValueFlowGraph() - assert(svfa.reportConflictsSVG().size == 1) - } - - ignore( - "in the class Refl3 we should detect 1 conflict(s) of a Reflection test case" - ) { - val svfa = - new SecuribenchTest("securibench.micro.reflection.Refl3", "doGet") - svfa.buildSparseValueFlowGraph() - assert(svfa.reportConflictsSVG().size == 1) - } - - ignore( - "in the class Refl4 we should detect 1 conflict(s) of a Reflection test case" - ) { - val svfa = - new SecuribenchTest("securibench.micro.reflection.Refl4", "doGet") - svfa.buildSparseValueFlowGraph() - assert(svfa.reportConflictsSVG().size == 1) - } - - /** SANITIZERS TESTs - */ - ignore( - "in the class Sanitizers1 we should detect 1 conflict(s) of a Sanitizers test case" - ) { - val svfa = - new SecuribenchTest("securibench.micro.sanitizers.Sanitizers1", "doGet") - svfa.buildSparseValueFlowGraph() - assert(svfa.reportConflictsSVG().size == 1) - } - - ignore( - "in the class Sanitizers2 we should detect 1 conflict(s) of a Sanitizers test case" - ) { - val svfa = - new SecuribenchTest("securibench.micro.sanitizers.Sanitizers2", "doGet") - svfa.buildSparseValueFlowGraph() - assert(svfa.reportConflictsSVG().size == 1) - } - - test( - "in the class Sanitizers3 we should detect 0 conflict(s) of a Sanitizers test case" - ) { - val svfa = - new SecuribenchTest("securibench.micro.sanitizers.Sanitizers3", "doGet") - svfa.buildSparseValueFlowGraph() - assert(svfa.reportConflictsSVG().size == 0) - } - - ignore( - "in the class Sanitizers4 we should detect 2 conflict(s) of a Sanitizers test case" - ) { - val svfa = - new SecuribenchTest("securibench.micro.sanitizers.Sanitizers4", "doGet") - svfa.buildSparseValueFlowGraph() - assert(svfa.reportConflictsSVG().size == 2) - } - - ignore( - "in the class Sanitizers5 we should detect 1 conflict(s) of a Sanitizers test case" - ) { - val svfa = - new SecuribenchTest("securibench.micro.sanitizers.Sanitizers5", "doGet") - svfa.buildSparseValueFlowGraph() - assert(svfa.reportConflictsSVG().size == 1) - } - - ignore( - "in the class Sanitizers6 we should detect 1 conflict(s) of a Sanitizers test case" - ) { - val svfa = - new SecuribenchTest("securibench.micro.sanitizers.Sanitizers6", "doGet") - svfa.buildSparseValueFlowGraph() - assert(svfa.reportConflictsSVG().size == 1) - } -} diff --git a/modules/securibench/src/test/scala/br/unb/cic/securibench/deprecated/SecuribenchTestSuite.scala b/modules/securibench/src/test/scala/br/unb/cic/securibench/deprecated/SecuribenchTestSuite.scala index 57ec6841..e2f7c806 100644 --- a/modules/securibench/src/test/scala/br/unb/cic/securibench/deprecated/SecuribenchTestSuite.scala +++ b/modules/securibench/src/test/scala/br/unb/cic/securibench/deprecated/SecuribenchTestSuite.scala @@ -5,8 +5,9 @@ import org.scalatest.FunSuite class SecuribenchTestSuite extends FunSuite { - /** ALIASING TESTs - */ + /** + ALIASING TESTs + */ test( "in the class Aliasing1 we should detect 1 conflict of a simple aliasing test case" @@ -65,7 +66,7 @@ class SecuribenchTestSuite extends FunSuite { assert(svfa.reportConflictsSVG().size == expectedConflicts) } - ignore("in the class Aliasing6 we should detect 7 conflicts") { + test("in the class Aliasing6 we should detect 7 conflicts") { val testName = "Aliasing6" val expectedConflicts = 7 @@ -272,7 +273,6 @@ class SecuribenchTestSuite extends FunSuite { assert(svfa.reportConflictsSVG().size == expectedConflicts) } - // FLAKY test( "in the class Basic6 we should detect 1 conflict of a complex derived string test" ) { @@ -1130,7 +1130,7 @@ class SecuribenchTestSuite extends FunSuite { assert(svfa.reportConflictsSVG().size == expectedConflicts) } -// FLAKY + // FLAKY: It only fails in the Github action pipeline ignore( "in the class Inter11 we should detect 1 conflict of a simple inter test case" ) { @@ -1263,7 +1263,7 @@ class SecuribenchTestSuite extends FunSuite { assert(svfa.reportConflictsSVG().size == expectedConflicts) } - // It only fails in the Github action pipeline + // FLAKY: It only fails in the Github action pipeline ignore( "in the class StrongUpdates4 we should detect 1 conflict of a simple strong update test case" ) { @@ -1291,4 +1291,179 @@ class SecuribenchTestSuite extends FunSuite { svfa.buildSparseValueFlowGraph() assert(svfa.reportConflictsSVG().size == expectedConflicts) } + + /** + * + * EXTRA TESTs + * + */ + + /** PRED TESTs + */ + + test( + "in the class Pred1 we should detect 0 conflict(s) of a Pred test case" + ) { + val svfa = new SecuribenchTest("securibench.micro.pred.Pred1", "doGet") + svfa.buildSparseValueFlowGraph() + assert(svfa.reportConflictsSVG().size == 0) + } + + test( + "in the class Pred2 we should detect 1 conflict(s) of a Pred test case" + ) { + val svfa = new SecuribenchTest("securibench.micro.pred.Pred2", "doGet") + svfa.buildSparseValueFlowGraph() + assert(svfa.reportConflictsSVG().size == 1) + } + + ignore( + "in the class Pred3 we should detect 0 conflict(s) of a Pred test case" + ) { + val svfa = new SecuribenchTest("securibench.micro.pred.Pred3", "doGet") + svfa.buildSparseValueFlowGraph() + assert(svfa.reportConflictsSVG().size == 0) + } + + test( + "in the class Pred4 we should detect 1 conflict(s) of a Pred test case" + ) { + val svfa = new SecuribenchTest("securibench.micro.pred.Pred4", "doGet") + svfa.buildSparseValueFlowGraph() + assert(svfa.reportConflictsSVG().size == 1) + } + + test( + "in the class Pred5 we should detect 1 conflict(s) of a Pred test case" + ) { + val svfa = new SecuribenchTest("securibench.micro.pred.Pred5", "doGet") + svfa.buildSparseValueFlowGraph() + assert(svfa.reportConflictsSVG().size == 1) + } + + ignore( + "in the class Pred6 we should detect 0 conflict(s) of a Pred test case" + ) { + val svfa = new SecuribenchTest("securibench.micro.pred.Pred6", "doGet") + svfa.buildSparseValueFlowGraph() + assert(svfa.reportConflictsSVG().size == 0) + } + + ignore( + "in the class Pred7 we should detect 0 conflict(s) of a Pred test case" + ) { + val svfa = new SecuribenchTest("securibench.micro.pred.Pred7", "doGet") + svfa.buildSparseValueFlowGraph() + assert(svfa.reportConflictsSVG().size == 0) + } + + test( + "in the class Pred8 we should detect 1 conflict(s) of a Pred test case" + ) { + val svfa = new SecuribenchTest("securibench.micro.pred.Pred8", "doGet") + svfa.buildSparseValueFlowGraph() + assert(svfa.reportConflictsSVG().size == 1) + } + + test( + "in the class Pred9 we should detect 1 conflict(s) of a Pred test case" + ) { + val svfa = new SecuribenchTest("securibench.micro.pred.Pred9", "doGet") + svfa.buildSparseValueFlowGraph() + assert(svfa.reportConflictsSVG().size == 1) + } + + /** REFLECTION TESTs + */ + ignore( + "in the class Refl1 we should detect 1 conflict(s) of a Reflection test case" + ) { + val svfa = + new SecuribenchTest("securibench.micro.reflection.Refl1", "doGet") + svfa.buildSparseValueFlowGraph() + assert(svfa.reportConflictsSVG().size == 1) + } + + ignore( + "in the class Refl2 we should detect 1 conflict(s) of a Reflection test case" + ) { + val svfa = + new SecuribenchTest("securibench.micro.reflection.Refl2", "doGet") + svfa.buildSparseValueFlowGraph() + assert(svfa.reportConflictsSVG().size == 1) + } + + ignore( + "in the class Refl3 we should detect 1 conflict(s) of a Reflection test case" + ) { + val svfa = + new SecuribenchTest("securibench.micro.reflection.Refl3", "doGet") + svfa.buildSparseValueFlowGraph() + assert(svfa.reportConflictsSVG().size == 1) + } + + ignore( + "in the class Refl4 we should detect 1 conflict(s) of a Reflection test case" + ) { + val svfa = + new SecuribenchTest("securibench.micro.reflection.Refl4", "doGet") + svfa.buildSparseValueFlowGraph() + assert(svfa.reportConflictsSVG().size == 1) + } + + /** SANITIZERS TESTs + */ + test( + "in the class Sanitizers1 we should detect 1 conflict(s) of a Sanitizers test case" + ) { + val svfa = + new SecuribenchTest("securibench.micro.sanitizers.Sanitizers1", "doGet") + svfa.buildSparseValueFlowGraph() + assert(svfa.reportConflictsSVG().size == 1) + } + + ignore( + "in the class Sanitizers2 we should detect 1 conflict(s) of a Sanitizers test case" + ) { + val svfa = + new SecuribenchTest("securibench.micro.sanitizers.Sanitizers2", "doGet") + svfa.buildSparseValueFlowGraph() + assert(svfa.reportConflictsSVG().size == 1) + } + + test( + "in the class Sanitizers3 we should detect 0 conflict(s) of a Sanitizers test case" + ) { + val svfa = + new SecuribenchTest("securibench.micro.sanitizers.Sanitizers3", "doGet") + svfa.buildSparseValueFlowGraph() + assert(svfa.reportConflictsSVG().size == 0) + } + + ignore( + "in the class Sanitizers4 we should detect 2 conflict(s) of a Sanitizers test case" + ) { + val svfa = + new SecuribenchTest("securibench.micro.sanitizers.Sanitizers4", "doGet") + svfa.buildSparseValueFlowGraph() + assert(svfa.reportConflictsSVG().size == 2) + } + + ignore( + "in the class Sanitizers5 we should detect 1 conflict(s) of a Sanitizers test case" + ) { + val svfa = + new SecuribenchTest("securibench.micro.sanitizers.Sanitizers5", "doGet") + svfa.buildSparseValueFlowGraph() + assert(svfa.reportConflictsSVG().size == 1) + } + + ignore( + "in the class Sanitizers6 we should detect 1 conflict(s) of a Sanitizers test case" + ) { + val svfa = + new SecuribenchTest("securibench.micro.sanitizers.Sanitizers6", "doGet") + svfa.buildSparseValueFlowGraph() + assert(svfa.reportConflictsSVG().size == 1) + } } diff --git a/modules/taintbench/src/test/scala/br/unb/cic/android/AndroidTaintBenchSuiteExperiment2Test.scala b/modules/taintbench/src/test/scala/br/unb/cic/android/AndroidTaintBenchSuiteExperiment2Test.scala index 2588f22f..ca2d704c 100644 --- a/modules/taintbench/src/test/scala/br/unb/cic/android/AndroidTaintBenchSuiteExperiment2Test.scala +++ b/modules/taintbench/src/test/scala/br/unb/cic/android/AndroidTaintBenchSuiteExperiment2Test.scala @@ -23,7 +23,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -36,7 +36,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -49,7 +49,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -62,7 +62,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -75,7 +75,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -88,7 +88,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -101,7 +101,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -114,7 +114,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -127,7 +127,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -140,7 +140,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -153,7 +153,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -166,7 +166,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -179,7 +179,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -192,7 +192,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -205,7 +205,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -218,7 +218,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -231,7 +231,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -244,7 +244,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -258,7 +258,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -271,7 +271,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -284,7 +284,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -297,7 +297,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -310,7 +310,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -323,7 +323,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -336,7 +336,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -349,7 +349,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -362,7 +362,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -375,7 +375,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -388,7 +388,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -401,7 +401,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -415,7 +415,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -428,7 +428,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -444,7 +444,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -460,7 +460,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -473,7 +473,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -486,7 +486,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -499,7 +499,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } @@ -512,7 +512,7 @@ class AndroidTaintBenchSuiteExperiment2Test extends FunSuite createJsonReport(svfa.conflictPaths() , nameAPK, "modules/taintbench/src/docs-metrics/experiment-II/findings") - val found = svfa.reportConflictsSVG(true).size + val found = svfa.reportConflictsSVG().size assert(found == expected) } diff --git a/release_notes.txt b/release_notes.txt index f8473165..9f2ff551 100644 --- a/release_notes.txt +++ b/release_notes.txt @@ -20,4 +20,8 @@ v0.6.0 - Organize metrics results, - Create module for scripts, - Set up github workflow to not use tokens, -- Create logic to generate reports. \ No newline at end of file +- Create logic to generate reports. + +v0.6.1 +- Update readme information, +- Compute new test categories: Preds, Reflections, and Sanitizers. \ No newline at end of file