From 0458ca29dfb3acdeb4c1b898c1680cedb6c26e22 Mon Sep 17 00:00:00 2001 From: Raphael Heitjohann <5891816+rheitjoh@users.noreply.github.com> Date: Thu, 24 Jun 2021 14:56:54 +0200 Subject: [PATCH 1/7] Rework benchmarking.md --- docs/benchmarking.md | 151 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 129 insertions(+), 22 deletions(-) diff --git a/docs/benchmarking.md b/docs/benchmarking.md index f739ed1..b03cd3b 100644 --- a/docs/benchmarking.md +++ b/docs/benchmarking.md @@ -4,11 +4,24 @@ toc: true mathjax: true --- -In this document we discuss your options when it comes to benchmarking code written using Cryptimeleon libraries as well as group operation counting provided by Cryptimeleon Math. +In this document we discuss your options when it comes to benchmarking code written using Cryptimeleon libraries. -# Runtime Benchmarking +There are many different kinds of possible performance metrics. +These include runtime in the form of CPU cycles or CPU time, memory usage, and network usage. +Furthermore, one may want to collect hardware-independent information such as the number of group operations or pairings. +Both of these types of metrics have their use-cases. +Counting group operations and pairings has the advantage of being hardware-independent. +To the layperson and potential user, applied metrics such as CPU time or memory usage can be more meaningful as they demonstrate practicality better than the more abstract group operations metrics. -## Lazy Eval +# Collecting Applied Metrics + +Applied metrics, such as runtime or memory usage, can be collected using any existing Java benchmark framework. +An example of such a framework is the Java Microbenchmarking Harness (JMH). +It allows for very accurate measurements and integrates with many existing profilers. +Due to these existing options, we have decided against implementing any such capabilities ourselves. +Therefore, you will need to use one of these existing benchmark frameworks. + +## Problems With Lazy Evaluation While useful for automatic optimization, the [lazy evaluation]({% link docs/lazy-eval.md %}) features of Math can create some benchmarking problems if not handled correctly. @@ -17,15 +30,96 @@ During setup for your verification benchmark, you will probably execute the sign If evaluation is deferred until later, group operations done during signing will only be executed once the result is needed during verification of your signature. This will obviously falsify your results for the verification benchmark. -Therefore, you should make sure to compute all group operations via any of the methods that force a blocking evaluation of the `LazyGroupElement` instance, for example using `computeSync()`. +Furthermore, the signing algorithm may compute precomputations. +These precomputations could then be used during the verification algorithm, resulting in lower runtime than is actually the case. + +We show how to compensate for this in the JMH example that is part of the next section. +Specifically, by calling `getRepresentation()` on the signature you can force all the involved computations to be done in a blocking manner. +To remove the precomputations, you can use a method like `restoreSignature()` to recreate the `Signature` object. +This new signature does not have precomputations. + +## JMH Example + +As an example we look at how to use JMH to measure runtime for our implementation of the verification algorithm of the signature scheme from Section 4.2 of Pointcheval and Sanders [PS18]. + +Given the public key $$\textsf{pk} = (\tilde{g}, \tilde{X}, \tilde{Y}_1, \dots, \tilde{Y}_{r+1})$$, the message vector $$\textbf{m} = (m_1, \dots, m_r)$$, and the signature $$\sigma = (m', \sigma_1, \sigma_2)$$, the verification algorithm works as follows: +Check whether $$\sigma_1$$ is the multiplicative identity of $$\mathbb{G}_1$$ and output $$0$$ if yes. +Lastly, output $$1$$ if $$e(\sigma_1, \tilde{X} \cdot \prod_{j = 1}^r{\tilde{Y}_j^{m_j} \cdot \tilde{Y}_{r+1}^{m'}}) = e(\sigma_2, \tilde{g})$$ holds, and $$0$$ if not. +Here $$e$$ denotes the pairing operation. + +We test the verification operation using messages of length one and of length 10. +```java +@State(Scope.Thread) +public class PS18VerifyBenchmark { + + // Test with one message and ten + @Param({"1", "10"}) + int numMessages; + + PS18SignatureScheme scheme; + PlainText plainText; + Signature signature; + VerificationKey verificationKey; + + // The setup method that creates the signature and verificationKey used by the verify benchmark + @Setup(Level.Iteration) + public void setup() { + PSPublicParameters pp = new PSPublicParameters(new MclBilinearGroup()); + scheme = new PS18SignatureScheme(pp); + SignatureKeyPair keyPair = + scheme.generateKeyPair(numMessages); + RingElementPlainText[] messages = new RingElementPlainText[numMessages]; + for (int i = 0; i < messages.length; i++) { + messages[i] = new RingElementPlainText(pp.getZp().getUniformlyRandomElement()); + } + plainText = new MessageBlock(messages); + signature = scheme.sign(plainText, keyPair.getSigningKey()); + verificationKey = keyPair.getVerificationKey(); + // Computations in sign and/or key gen may be done non-blocking. + // To make sure these are not done as part of the verification benchmark, + // we force the remaining computations to be done blocking via getRepresentation() + signature.getRepresentation(); + verificationKey.getRepresentation(); + } + + // The benchmark method. Includes settings for JMH + @Benchmark + @BenchmarkMode(Mode.SingleShotTime) + @Warmup(iterations = 3, batchSize = 1) + @Measurement(iterations = 10, batchSize = 1) + @OutputTimeUnit(TimeUnit.MILLISECONDS) + public Boolean measureVerify() { + // Running the signing or key gen algorithms may have done precomputations + // for the verification key and/or signature. + // To reset these precomputation such that they do not make the verification + // algorithm faster than it would be without, we recreate the objects + // without precomputations. + signature = scheme.restoreSignature(signature.getRepresentation()); + verificationKey = scheme.restoreVerificationKey(verificationKey.getRepresentation()); + return scheme.verify(plainText, signature, verificationKey); + } +} +``` -## Micro-Benchmarking +JMH outputs the following results for the above benchmark (using an i5-4210m on Ubuntu 20.04): -If you care about accuracy, we recommend using a micro-benchmarking framework such as [JMH](https://openjdk.java.net/projects/code-tools/jmh/). -We also strongly recommend working through the [JMH Samples](https://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/) to make sure you avoid any bad practices that could potentially invalidate your benchmarks. +|Benchmark | numMessages| Mode| Cnt | Score | Error| Units | +|-------------------|---------------------------|--------|----|-------|--------|-------| +|measureVerify | 1 | ss | 50 | 5.263 |$$\pm$$ 0.641 | ms/op | +|measureVerify | 10 | ss | 50| 8.157 |$$\pm$$ 1.506 | ms/op| + +The above example is also part of our [Benchmark](https://github.com/cryptimeleon/benchmark) project. +There you can find more examples like it. + +If you want to use JMH, we strongly recommend working through the [JMH Samples](https://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/) to make sure you avoid any bad practices that could potentially invalidate your benchmarks. + +## Using JMH with Gradle -[Our own benchmarks](https://github.com/cryptimeleon/benchmark) use JMH. Since JMH is made to be used with Maven, you will probably want to add a Gradle task for executing your JMH tests (if you use Gradle). +To run the JMH tests, you need to run JMH's `Main` class with the classpath set to the folder where your tests lie. +The code below gives an example Gradle task that can run JMH tests inside the `jmh` source set. +It also enables support for certain JMH parameters. +For example, using the `include` parameter you can explicitly state the test classes you want to run. ```groovy task jmh(type: JavaExec) { @@ -53,19 +147,27 @@ task jmh(type: JavaExec) { args '-rff', resultFile } ``` -Above is the script we use for our Cryptimeleon Benchmark project. -It allows us to use certain JMH parameters in addition to just running all tests contained in the `jmh` source set. + +For example, to execute the previous JMH example (and only it), you would run +```bash +./gradlew -q jmh -Pinclude="PS18VerifyBenchmark" +``` +inside the folder of your Gradle project. # Group Operation Counting -Cryptimeleon Math includes capabilities for group operation counting. -Specifically, it allows for tracking group inversions, squarings, operations, as well as (multi-)exponentiations for a specific group. +The collection of hardware-independent metrics such as group operations is implemented by the Cryptimeleon Math library. +The main points of interest here are the `DebugBilinearGroup` and `DebugGroup` classes. +The former allows for counting pairings, and the latter allows for counting group operations, group squarings (relevant for elliptic curves), group inversions, exponentiations, and multi-exponentiations. +It is also able to track the number of times group elements have been serialized. ## DebugGroup The functionality of group operation counting is provided by using a special group, the `DebugGroup`. -*Note: Keep in mind that `DebugGroup` uses \\(\mathbb{Z}_n\\) under the hood, and so is only to be used when testing and/or counting group operations, not for other performance benchmarks.* +By simply using the `DebugGroup` to perform the computations, it automatically counts the operations done within it. + +*Note: Keep in mind that `DebugGroup` uses $$\mathbb{Z}_n$$ under the hood, and so is only to be used when testing and/or counting group operations, not for other performance benchmarks.* ```java import org.cryptimeleon.math.structures.groups.debug.DebugGroup; @@ -84,18 +186,23 @@ System.out.println(debugGroup.getNumSquaringsTotal()); 1 ``` -As seen above, `DebugGroup` provides the same interfaces as any other group in Math does, just with some additional features. - -Whenever a group operation is performed, `DebugGroup` tracks it internally. -The user can access the data via a variety of methods. -These methods for data access can be separated in two categories: -Methods whose names end in `Total` and ones whose names end in `NoExpMultiExp`. -The former includes all group operations, even the ones done in (multi-)exponentiation algorithms while `NoExpMultiExp` methods only retrieve operation counts of operations *not* done in (multi-)exponentiations. +The counting is done in two modes: The `NoExpMultiExp` mode and the `Total` mode. +Group operations metrics from the `NoExpMultiExp` mode disregard operations done inside (multi-)exponentiations while the `Total` mode does account for operations inside (multi-)exponentiations. +`NoExpMultiExp` measurements are therefore independent of the actual (multi-)exponentiation algorithm while `Total` measurements are more expressive in regards to the actual runtime (since estimating group operation runtime is easier than that of a (multi-)exponentiation). This is useful if you want to track (multi-)exponentiations only as a single unit and not the underlying group operations. -That data can be accessed via the `getNumExps()` and `getMultiExpTermNumbers()` methods, where the latter returns an array containing the number of bases in each multi-exponentiation done. +Exponentiation and multi-exponentiation data can be accessed via the `getNumExps()` and `getMultiExpTermNumbers()` methods, where the latter returns an array containing the number of bases in each multi-exponentiation done. Additionally, `resetCounters()` can be used to reset all operation counters, and `formatCounterData()` provides a printable string that summarizes all collected data. +As an example we consider the computation of $$g^a \cdot h^b$$. +The `NoExpMultiExp` mode counts this as a single multi-exponentiation with two terms. +No group operations are counted since they are all part of the multi-exponentiation. +The `Total` mode does not consider the multi-exponentiation as its own unit. +Instead, it counts the group operations, inversions, and squarings that are part of evaluating the multi-exponentiation (using a wNAF-type algorithm). +Combining these metrics gives us therefore a more complete picture of the computational costs. + +A more detailed (code) example is given below: + ```java DebugGroup debugGroup = new DebugGroup("DG1", 1000000); GroupElement elem = debugGroup.getUniformlyRandomNonNeutral(); @@ -145,7 +252,7 @@ The count is accessible via `getNumRetrievedRepresentations()`. ## DebugBilinearGroup Cryptimeleon Math also provides a `BilinearGroup` implementation that can be used for counting, the `DebugBilinearGroup` class. -It uses a simple (not secure) \\(\mathbb{Z}_n\\) pairing. +It uses a simple (not secure) $$\mathbb{Z}_n$$ pairing. In addition to the usual group operation counting done by the three `DebugGroup` instances contained in the bilinear group, `DebugBilinearGroup` also allows you to track number of pairings performed. From 97dbf0379ccc2f7b4c6035a8c9dcec63e3670712 Mon Sep 17 00:00:00 2001 From: Raphael Heitjohann <5891816+rheitjoh@users.noreply.github.com> Date: Thu, 24 Jun 2021 15:55:10 +0200 Subject: [PATCH 2/7] Rewrite the problems with lazy evaluation section of benchmarking.md --- docs/benchmarking.md | 74 +++++++++++++++++++++++++++----------------- 1 file changed, 46 insertions(+), 28 deletions(-) diff --git a/docs/benchmarking.md b/docs/benchmarking.md index b03cd3b..27751b6 100644 --- a/docs/benchmarking.md +++ b/docs/benchmarking.md @@ -21,26 +21,40 @@ It allows for very accurate measurements and integrates with many existing profi Due to these existing options, we have decided against implementing any such capabilities ourselves. Therefore, you will need to use one of these existing benchmark frameworks. -## Problems With Lazy Evaluation +## Problems That Can Falsify Your Benchmarks -While useful for automatic optimization, the [lazy evaluation]({% link docs/lazy-eval.md %}) features of Math can create some benchmarking problems if not handled correctly. - -Let's take for example a signature scheme. -During setup for your verification benchmark, you will probably execute the signing algorithm. -If evaluation is deferred until later, group operations done during signing will only be executed once the result is needed during verification of your signature. -This will obviously falsify your results for the verification benchmark. +We now want to look at some potential problems when creating a benchmark. +We will illustrate these problems using the example of benchmarking the verification algorithm of a signature scheme. -Furthermore, the signing algorithm may compute precomputations. -These precomputations could then be used during the verification algorithm, resulting in lower runtime than is actually the case. +### Lazy Evaluation -We show how to compensate for this in the JMH example that is part of the next section. -Specifically, by calling `getRepresentation()` on the signature you can force all the involved computations to be done in a blocking manner. -To remove the precomputations, you can use a method like `restoreSignature()` to recreate the `Signature` object. -This new signature does not have precomputations. +While useful for automatic optimization, the [lazy evaluation]({% link docs/lazy-eval.md %}) features of Math can create some benchmarking problems if not handled correctly. +To benchmark your verification algorithm, you will need valid signatures, and these signatures will be provided by executing the signing algorithm. +Inside the signing algorithm group operations might be executed. +By default, group operations in Cryptimeleon Math are evaluated lazily, i.e. deferred until needed. +Therefore, group operations done during signing will only be executed once the result is needed during verification of your signature. +This will increase the runtime of your verification algorithm and therefore falsify the results. +By serializing the `Signature` object using the `getRepresentation()` method *before* running verification, you can force all remaining computations involving fields of the `Signature` object to be completed in a blocking manner. + +### Mutability + +Another problem is the mutability of the `Signature` object. +When repeatedly verifying the same `Signature` object, it may be possible that executing the verification algorithm leads to certain optimizations that change the runtime of the verification for the succeeding verification runs. +An example of this problem is the storing of *precomputations*. +The `Signature` object will most likely contain group elements. +Those group elements can store precomputations which help to make future exponentiations more efficient. +The first run of your verification may create such precomputations that can then be used in future invocations of the verification algorithm. +These latter runs will then run faster than the first one, leading to skewed results. + +The underlying problem here is the mutability of the `Signature` object and the reuse of it for multiple runs of the verification algorithm. +Therefore, the solution is to recreate the `Signature` object for each invocation of the verification algorithm. +This can be done using our [representation framework]({% link docs/representations.md %}). +By serializing and then deserializing it for each invocation of the verification algorithm, you obtain a fresh `Signature` object each time with the same state as before serialization. ## JMH Example As an example we look at how to use JMH to measure runtime for our implementation of the verification algorithm of the signature scheme from Section 4.2 of Pointcheval and Sanders [PS18]. +Here we will also see how to implement the mitigations for the benchmarking problems discussed in the previous section. Given the public key $$\textsf{pk} = (\tilde{g}, \tilde{X}, \tilde{Y}_1, \dots, \tilde{Y}_{r+1})$$, the message vector $$\textbf{m} = (m_1, \dots, m_r)$$, and the signature $$\sigma = (m', \sigma_1, \sigma_2)$$, the verification algorithm works as follows: Check whether $$\sigma_1$$ is the multiplicative identity of $$\mathbb{G}_1$$ and output $$0$$ if yes. @@ -58,10 +72,11 @@ public class PS18VerifyBenchmark { PS18SignatureScheme scheme; PlainText plainText; - Signature signature; - VerificationKey verificationKey; + Representation signatureRepr; + Representation verifyKeyRepr; - // The setup method that creates the signature and verificationKey used by the verify benchmark + // The setup method that creates the signature and verification key + // used by the verify benchmark @Setup(Level.Iteration) public void setup() { PSPublicParameters pp = new PSPublicParameters(new MclBilinearGroup()); @@ -78,8 +93,8 @@ public class PS18VerifyBenchmark { // Computations in sign and/or key gen may be done non-blocking. // To make sure these are not done as part of the verification benchmark, // we force the remaining computations to be done blocking via getRepresentation() - signature.getRepresentation(); - verificationKey.getRepresentation(); + signatureRepr = signature.getRepresentation(); + verifyKeyRepr = verificationKey.getRepresentation(); } // The benchmark method. Includes settings for JMH @@ -94,8 +109,8 @@ public class PS18VerifyBenchmark { // To reset these precomputation such that they do not make the verification // algorithm faster than it would be without, we recreate the objects // without precomputations. - signature = scheme.restoreSignature(signature.getRepresentation()); - verificationKey = scheme.restoreVerificationKey(verificationKey.getRepresentation()); + signature = scheme.restoreSignature(signatureRepr); + verificationKey = scheme.restoreVerificationKey(verifyKeyRepr); return scheme.verify(plainText, signature, verificationKey); } } @@ -164,15 +179,14 @@ It is also able to track the number of times group elements have been serialized ## DebugGroup The functionality of group operation counting is provided by using a special group, the `DebugGroup`. - By simply using the `DebugGroup` to perform the computations, it automatically counts the operations done within it. -*Note: Keep in mind that `DebugGroup` uses $$\mathbb{Z}_n$$ under the hood, and so is only to be used when testing and/or counting group operations, not for other performance benchmarks.* +*Note: Keep in mind that `DebugGroup` uses $$\mathbb{Z}_n$$ under the hood and is way faster than any secure group, and so is only to be used when testing and/or counting group operations, not for other performance benchmarks.* ```java import org.cryptimeleon.math.structures.groups.debug.DebugGroup; -// instantiate the debug group with a name and its size +// Instantiate the debug group with a name and its size DebugGroup debugGroup = new DebugGroup("DG1", 1000000); // Get a random non-neutral element and square it @@ -240,10 +254,10 @@ As you can see, the "Total group operation data" block has much higher numbers t ### Lazy Evaluation -`DebugGroup` does use lazy evaluation, meaning that `compute()` calls are necessary before retrieving tracked operation data, else the operation might have not been executed yet. -However, `compute()` has been changed to behave like `computeSync()` in that it blocks until the computation is done. -This is because non-blocking computation can lead to race conditions when printing the result of tracking the group operations, i.e. the computation has not been performed yet when the data is printed. -So make sure to always call `compute()` on every `DebugGroupElement` before accessing any counter data. +`DebugGroup` does use lazy evaluation, meaning that you need to ensure all lazy computations have finished before retrieving the tracked results. +One way to do this is to call `computeSync()` on all operations. +However, for your convenience, `DebugGroup` also overrides `compute()` to behave like `computeSync()` in that it blocks until the computation is done. +So make sure to always call `compute()` on every involved `DebugGroupElement` before accessing any counter data, or call `getRepresentation()` to serialize any involved objects as this also leads to a blocking computation. ### Serialization Tracking `DebugGroup` not only allows for tracking group operations, it also counts how many calls of `getRepresentation()` have been called on elements of the group. This has the purpose of allowing you to track serializations. @@ -272,4 +286,8 @@ System.out.println(bilGroup.getNumPairings()); ``` ``` 1 -``` \ No newline at end of file +``` + +# References + +[PS18] David Pointcheval and Olivier Sanders. “Reassessing Security of Randomizable Signatures”. In: Topic in Cryptology - CT-RSA 2018. Ed. by Nigel P. Smart. Springer International Publishing, 2018, pp 319-338. \ No newline at end of file From 9b8b605de841cc7900d41f8b9bb6391d03f9e46c Mon Sep 17 00:00:00 2001 From: Raphael Heitjohann <5891816+rheitjoh@users.noreply.github.com> Date: Wed, 30 Jun 2021 16:33:41 +0200 Subject: [PATCH 3/7] Fix indentation in JMH example --- docs/benchmarking.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/benchmarking.md b/docs/benchmarking.md index 27751b6..e47690e 100644 --- a/docs/benchmarking.md +++ b/docs/benchmarking.md @@ -66,9 +66,9 @@ We test the verification operation using messages of length one and of length 10 @State(Scope.Thread) public class PS18VerifyBenchmark { - // Test with one message and ten - @Param({"1", "10"}) - int numMessages; + // Test with one message and ten + @Param({"1", "10"}) + int numMessages; PS18SignatureScheme scheme; PlainText plainText; @@ -77,8 +77,8 @@ public class PS18VerifyBenchmark { // The setup method that creates the signature and verification key // used by the verify benchmark - @Setup(Level.Iteration) - public void setup() { + @Setup(Level.Iteration) + public void setup() { PSPublicParameters pp = new PSPublicParameters(new MclBilinearGroup()); scheme = new PS18SignatureScheme(pp); SignatureKeyPair keyPair = @@ -95,15 +95,15 @@ public class PS18VerifyBenchmark { // we force the remaining computations to be done blocking via getRepresentation() signatureRepr = signature.getRepresentation(); verifyKeyRepr = verificationKey.getRepresentation(); - } + } - // The benchmark method. Includes settings for JMH - @Benchmark - @BenchmarkMode(Mode.SingleShotTime) - @Warmup(iterations = 3, batchSize = 1) - @Measurement(iterations = 10, batchSize = 1) - @OutputTimeUnit(TimeUnit.MILLISECONDS) - public Boolean measureVerify() { + // The benchmark method. Includes settings for JMH + @Benchmark + @BenchmarkMode(Mode.SingleShotTime) + @Warmup(iterations = 3, batchSize = 1) + @Measurement(iterations = 10, batchSize = 1) + @OutputTimeUnit(TimeUnit.MILLISECONDS) + public Boolean measureVerify() { // Running the signing or key gen algorithms may have done precomputations // for the verification key and/or signature. // To reset these precomputation such that they do not make the verification @@ -112,7 +112,7 @@ public class PS18VerifyBenchmark { signature = scheme.restoreSignature(signatureRepr); verificationKey = scheme.restoreVerificationKey(verifyKeyRepr); return scheme.verify(plainText, signature, verificationKey); - } + } } ``` From 87bd1044077aea01fc297c526a26b2902192d6c0 Mon Sep 17 00:00:00 2001 From: Raphael Heitjohann <5891816+rheitjoh@users.noreply.github.com> Date: Wed, 30 Jun 2021 16:46:50 +0200 Subject: [PATCH 4/7] Add note about selecting exp algs for debug groups --- docs/benchmarking.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/benchmarking.md b/docs/benchmarking.md index e47690e..e2c6a7f 100644 --- a/docs/benchmarking.md +++ b/docs/benchmarking.md @@ -259,6 +259,14 @@ One way to do this is to call `computeSync()` on all operations. However, for your convenience, `DebugGroup` also overrides `compute()` to behave like `computeSync()` in that it blocks until the computation is done. So make sure to always call `compute()` on every involved `DebugGroupElement` before accessing any counter data, or call `getRepresentation()` to serialize any involved objects as this also leads to a blocking computation. +### Configuring Used (Multi-)exponentiation Algorithm + +`DebugGroup` makes use of efficient exponentiation and multi-exponentiation algorithms. +The exact algorithm used changes the resulting group operation counts. +To manually configure these algorithms, `DebugGroup` (and `DebugBilinearGroup`) offers setter and getter methods such as `getSelectedMultiExpAlgorithm` and `setSelectedMultiExpAlgorithm`. +Furthermore, you can configure the precomputation and exponentiation window sizes used for those algorithms. +These are the same methods as offered by `LazyGroup`. + ### Serialization Tracking `DebugGroup` not only allows for tracking group operations, it also counts how many calls of `getRepresentation()` have been called on elements of the group. This has the purpose of allowing you to track serializations. The count is accessible via `getNumRetrievedRepresentations()`. @@ -288,6 +296,7 @@ System.out.println(bilGroup.getNumPairings()); 1 ``` + # References [PS18] David Pointcheval and Olivier Sanders. “Reassessing Security of Randomizable Signatures”. In: Topic in Cryptology - CT-RSA 2018. Ed. by Nigel P. Smart. Springer International Publishing, 2018, pp 319-338. \ No newline at end of file From 07362ac403f2ee5353be5506df66080e4c4494c8 Mon Sep 17 00:00:00 2001 From: Raphael Heitjohann <5891816+rheitjoh@users.noreply.github.com> Date: Thu, 22 Jul 2021 13:23:12 +0200 Subject: [PATCH 5/7] Reorder jmh benchmarking to first present basic example and then fix issues by example --- docs/benchmarking.md | 186 ++++++++++++++++++++++++++++--------------- 1 file changed, 123 insertions(+), 63 deletions(-) diff --git a/docs/benchmarking.md b/docs/benchmarking.md index e2c6a7f..7d24f2b 100644 --- a/docs/benchmarking.md +++ b/docs/benchmarking.md @@ -1,5 +1,5 @@ --- -title: Benchmarking and Group Operation Counting +title: Benchmarking and group operation counting toc: true mathjax: true --- @@ -13,7 +13,7 @@ Both of these types of metrics have their use-cases. Counting group operations and pairings has the advantage of being hardware-independent. To the layperson and potential user, applied metrics such as CPU time or memory usage can be more meaningful as they demonstrate practicality better than the more abstract group operations metrics. -# Collecting Applied Metrics +# Collecting applied metrics Applied metrics, such as runtime or memory usage, can be collected using any existing Java benchmark framework. An example of such a framework is the Java Microbenchmarking Harness (JMH). @@ -21,37 +21,7 @@ It allows for very accurate measurements and integrates with many existing profi Due to these existing options, we have decided against implementing any such capabilities ourselves. Therefore, you will need to use one of these existing benchmark frameworks. -## Problems That Can Falsify Your Benchmarks - -We now want to look at some potential problems when creating a benchmark. -We will illustrate these problems using the example of benchmarking the verification algorithm of a signature scheme. - -### Lazy Evaluation - -While useful for automatic optimization, the [lazy evaluation]({% link docs/lazy-eval.md %}) features of Math can create some benchmarking problems if not handled correctly. -To benchmark your verification algorithm, you will need valid signatures, and these signatures will be provided by executing the signing algorithm. -Inside the signing algorithm group operations might be executed. -By default, group operations in Cryptimeleon Math are evaluated lazily, i.e. deferred until needed. -Therefore, group operations done during signing will only be executed once the result is needed during verification of your signature. -This will increase the runtime of your verification algorithm and therefore falsify the results. -By serializing the `Signature` object using the `getRepresentation()` method *before* running verification, you can force all remaining computations involving fields of the `Signature` object to be completed in a blocking manner. - -### Mutability - -Another problem is the mutability of the `Signature` object. -When repeatedly verifying the same `Signature` object, it may be possible that executing the verification algorithm leads to certain optimizations that change the runtime of the verification for the succeeding verification runs. -An example of this problem is the storing of *precomputations*. -The `Signature` object will most likely contain group elements. -Those group elements can store precomputations which help to make future exponentiations more efficient. -The first run of your verification may create such precomputations that can then be used in future invocations of the verification algorithm. -These latter runs will then run faster than the first one, leading to skewed results. - -The underlying problem here is the mutability of the `Signature` object and the reuse of it for multiple runs of the verification algorithm. -Therefore, the solution is to recreate the `Signature` object for each invocation of the verification algorithm. -This can be done using our [representation framework]({% link docs/representations.md %}). -By serializing and then deserializing it for each invocation of the verification algorithm, you obtain a fresh `Signature` object each time with the same state as before serialization. - -## JMH Example +## JMH example As an example we look at how to use JMH to measure runtime for our implementation of the verification algorithm of the signature scheme from Section 4.2 of Pointcheval and Sanders [PS18]. Here we will also see how to implement the mitigations for the benchmarking problems discussed in the previous section. @@ -62,6 +32,11 @@ Lastly, output $$1$$ if $$e(\sigma_1, \tilde{X} \cdot \prod_{j = 1}^r{\tilde{Y}_ Here $$e$$ denotes the pairing operation. We test the verification operation using messages of length one and of length 10. +The below code measures verification as follows: +For each iteration, a new signature and verification key are generated. +Then the verification of these is measured once. +This process makes up one iteration. +The results of all the iterations are then combined by JMH. ```java @State(Scope.Thread) public class PS18VerifyBenchmark { @@ -72,11 +47,14 @@ public class PS18VerifyBenchmark { PS18SignatureScheme scheme; PlainText plainText; - Representation signatureRepr; - Representation verifyKeyRepr; + Signature signature; + VerificationKey verificationKey; // The setup method that creates the signature and verification key - // used by the verify benchmark + // used by the verify benchmark. + // Level.Iteration tells us that it is run before each new iteration + // (an iteration consists of a sequence of invocations of the benchmark method; see sample 06 of the + // JMH samples we link at the end of this section) @Setup(Level.Iteration) public void setup() { PSPublicParameters pp = new PSPublicParameters(new MclBilinearGroup()); @@ -90,42 +68,20 @@ public class PS18VerifyBenchmark { plainText = new MessageBlock(messages); signature = scheme.sign(plainText, keyPair.getSigningKey()); verificationKey = keyPair.getVerificationKey(); - // Computations in sign and/or key gen may be done non-blocking. - // To make sure these are not done as part of the verification benchmark, - // we force the remaining computations to be done blocking via getRepresentation() - signatureRepr = signature.getRepresentation(); - verifyKeyRepr = verificationKey.getRepresentation(); } // The benchmark method. Includes settings for JMH @Benchmark - @BenchmarkMode(Mode.SingleShotTime) + @BenchmarkMode(Mode.SingleShotTime) // method is called once per iteration @Warmup(iterations = 3, batchSize = 1) @Measurement(iterations = 10, batchSize = 1) @OutputTimeUnit(TimeUnit.MILLISECONDS) public Boolean measureVerify() { - // Running the signing or key gen algorithms may have done precomputations - // for the verification key and/or signature. - // To reset these precomputation such that they do not make the verification - // algorithm faster than it would be without, we recreate the objects - // without precomputations. - signature = scheme.restoreSignature(signatureRepr); - verificationKey = scheme.restoreVerificationKey(verifyKeyRepr); return scheme.verify(plainText, signature, verificationKey); } } ``` -JMH outputs the following results for the above benchmark (using an i5-4210m on Ubuntu 20.04): - -|Benchmark | numMessages| Mode| Cnt | Score | Error| Units | -|-------------------|---------------------------|--------|----|-------|--------|-------| -|measureVerify | 1 | ss | 50 | 5.263 |$$\pm$$ 0.641 | ms/op | -|measureVerify | 10 | ss | 50| 8.157 |$$\pm$$ 1.506 | ms/op| - -The above example is also part of our [Benchmark](https://github.com/cryptimeleon/benchmark) project. -There you can find more examples like it. - If you want to use JMH, we strongly recommend working through the [JMH Samples](https://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/) to make sure you avoid any bad practices that could potentially invalidate your benchmarks. ## Using JMH with Gradle @@ -169,7 +125,112 @@ For example, to execute the previous JMH example (and only it), you would run ``` inside the folder of your Gradle project. -# Group Operation Counting +## Potential problems with benchmarks + +While being the most obvious approach, the verification benchmark we presented previously has some problems. +We will now discuss these problems and show to apply fixes for them to the example. + +### Lazy evaluation + +While useful for automatic optimization, the [lazy evaluation]({% link docs/lazy-eval.md %}) features of Math can create some benchmarking problems if not handled correctly. +To benchmark your verification algorithm, you will need valid signatures, and these signatures will be provided by executing the signing algorithm. +Inside the signing algorithm group operations might be executed. +By default, group operations in Cryptimeleon Math are evaluated lazily, i.e. deferred until needed. +Therefore, group operations done during signing will only be executed once the result is needed during verification of your signature. +Hence, when measuring runtime of the verification algorithm, we are actually also potentially measuring the deferred operations from the signature algorithm. +This will increase the runtime of your verification algorithm and therefore falsify the results. +By serializing the `Signature` object using the `getRepresentation()` method *before* the benchmark method, you can force all remaining computations involving fields of the `Signature` object to be completed in a blocking manner. + +See below for how to apply this to our verification example: +```java + @Setup(Level.Iteration) + public void setup() { + PSPublicParameters pp = new PSPublicParameters(new MclBilinearGroup()); + scheme = new PS18SignatureScheme(pp); + SignatureKeyPair keyPair = + scheme.generateKeyPair(numMessages); + RingElementPlainText[] messages = new RingElementPlainText[numMessages]; + for (int i = 0; i < messages.length; i++) { + messages[i] = new RingElementPlainText(pp.getZp().getUniformlyRandomElement()); + } + plainText = new MessageBlock(messages); + signature = scheme.sign(plainText, keyPair.getSigningKey()); + verificationKey = keyPair.getVerificationKey(); + // We force the group elements making up signature and verificationKey to be fully computed + // such that the computation does not spill over into the benchmark method + signature.getRepresentation(); + verificationKey.getRepresentation(); + } +``` + +### Optimizations stored in objects + +Another problem is the reuse of the `Signature` object. +When repeatedly verifying the same `Signature` object, it may be possible that executing the verification algorithm leads to certain optimizations that change the runtime of the verification for the succeeding verification runs. +An example of this problem is the storing of *precomputations*. +The `Signature` object will most likely contain group elements. +Those group elements can store precomputations which help to make future exponentiations more efficient. +The first run of your verification may create such precomputations that can then be used in future invocations of the verification algorithm. +These latter runs will then run faster than the first one, leading to skewed results. + +We have slightly changed the benchmark method from the JMH example to showcase this problem: +```java + // The benchmark method. Includes settings for JMH + @Benchmark + @BenchmarkMode(Mode.SingleShotTime) // method is called once per iteration + @Warmup(iterations = 3, batchSize = 5) + @Measurement(iterations = 10, batchSize = 5) + @OutputTimeUnit(TimeUnit.MILLISECONDS) + public Boolean measureVerify() { + return scheme.verify(plainText, signature, verificationKey); + } +``` +The `batchSize` inside the `@Warmup` and `@Measurement` annotations has been increased to `5`, meaning that, per iteration, five invocations of `measureVerify` are done. +In the first iteration, precomputations may be stored inside `signature` which can make the subsequent invocations more efficient. + +The underlying problem here is the reuse of the `Signature` object for multiple runs of the verification algorithm. +Therefore, the solution is to recreate the `Signature` object for each invocation of the verification algorithm. +This can be done using our [representation framework]({% link docs/representations.md %}). +By serializing and then deserializing it for each invocation of the verification algorithm, you obtain a fresh `Signature` object each time with the same state as before serialization. + +The improved benchmark method then looks as follows: +```java + // New fields + Representation signatureRepr; + Representation verifyKeyRepr; + + @Setup(Level.Iteration) + public void setup() { + // ... + signatureRepr = signature.getRepresentation(); + verifyKeyRepr = verificationKey.getRepresentation(); + } + + // The benchmark method. Includes settings for JMH + @Benchmark + @BenchmarkMode(Mode.SingleShotTime) + @Warmup(iterations = 3, batchSize = 5) + @Measurement(iterations = 10, batchSize = 5) + @OutputTimeUnit(TimeUnit.MILLISECONDS) + public Boolean measureVerify() { + // Running the signing or key gen algorithms may have done precomputations + // for the verification key and/or signature. + // To reset these precomputation such that they do not make the verification + // algorithm faster than it would be without, we recreate the objects + // without precomputations. + signature = scheme.restoreSignature(signatureRepr); + verificationKey = scheme.restoreVerificationKey(verifyKeyRepr); + return scheme.verify(plainText, signature, verificationKey); + } +``` + +Notice that we are also serializing and deserializing the verification key in the above example. +This represents the scenario where we are using a different verification key for each verification. +In another scenario we may be reusing the verification key, meaning that serializing and deserializing it before every verification would not be representative. + +Therefore, make sure that the way you design the benchmark matches the scenario that you actually want to benchmark. + +# Group operation counting The collection of hardware-independent metrics such as group operations is implemented by the Cryptimeleon Math library. The main points of interest here are the `DebugBilinearGroup` and `DebugGroup` classes. @@ -252,12 +313,11 @@ System.out.println(debugGroup.formatCounterData()); As you can see, the "Total group operation data" block has much higher numbers than the block below it, due to counting operations done during the multi-exponentiation and exponentiation. -### Lazy Evaluation +### Lazy evaluation `DebugGroup` does use lazy evaluation, meaning that you need to ensure all lazy computations have finished before retrieving the tracked results. -One way to do this is to call `computeSync()` on all operations. -However, for your convenience, `DebugGroup` also overrides `compute()` to behave like `computeSync()` in that it blocks until the computation is done. So make sure to always call `compute()` on every involved `DebugGroupElement` before accessing any counter data, or call `getRepresentation()` to serialize any involved objects as this also leads to a blocking computation. +For your convenience, `DebugGroup` also overrides `compute()` to behave like `computeSync()` in that it blocks until the computation is done. ### Configuring Used (Multi-)exponentiation Algorithm From b725852271aa7201a90b6fd8f031368d22dafdfe Mon Sep 17 00:00:00 2001 From: Raphael Heitjohann <5891816+rheitjoh@users.noreply.github.com> Date: Wed, 28 Jul 2021 11:24:21 +0200 Subject: [PATCH 6/7] Add explanation of new bucket system --- docs/benchmarking.md | 188 ++++++++++++++++++++++++++++++------------- 1 file changed, 131 insertions(+), 57 deletions(-) diff --git a/docs/benchmarking.md b/docs/benchmarking.md index 7d24f2b..9af8e5e 100644 --- a/docs/benchmarking.md +++ b/docs/benchmarking.md @@ -234,15 +234,10 @@ Therefore, make sure that the way you design the benchmark matches the scenario The collection of hardware-independent metrics such as group operations is implemented by the Cryptimeleon Math library. The main points of interest here are the `DebugBilinearGroup` and `DebugGroup` classes. -The former allows for counting pairings, and the latter allows for counting group operations, group squarings (relevant for elliptic curves), group inversions, exponentiations, and multi-exponentiations. +The former allows for counting pairings, and the latter allows for counting group operations, group squarings (relevant for elliptic curves), group inversions, exponentiations, multi-exponentiations, as well as serializations of group elements via `getRepresentation()`. It is also able to track the number of times group elements have been serialized. -## DebugGroup - -The functionality of group operation counting is provided by using a special group, the `DebugGroup`. -By simply using the `DebugGroup` to perform the computations, it automatically counts the operations done within it. - -*Note: Keep in mind that `DebugGroup` uses $$\mathbb{Z}_n$$ under the hood and is way faster than any secure group, and so is only to be used when testing and/or counting group operations, not for other performance benchmarks.* +*Note: Keep in mind that `DebugGroup` and `DebugBilinearGroup` use $$\mathbb{Z}_n$$ under the hood and is way faster than any secure group, and so is only to be used when testing and/or counting group operations, not for other performance benchmarks.* ```java import org.cryptimeleon.math.structures.groups.debug.DebugGroup; @@ -255,12 +250,40 @@ GroupElement elem = debugGroup.getUniformlyRandomNonNeutral(); elem.op(elem).compute(); // Print number of squarings in group -System.out.println(debugGroup.getNumSquaringsTotal()); +System.out.println(debugGroup.getNumSquaringsTotalDefault()); ``` ``` 1 ``` +In the above example, we instantiate a `DebugGroup`, compute a squaring within it, and then use the `getNumSquaringsTotalDefault()` method to obtain the number of squarings done from the `DebugGroup` object (in this case one). +In this case we selected an arbitrary size for the `DebugGroup`, but in practice you should use the same size as the group you are replacing with `DebugGroup`. +The reason for that is that the number of operations done in exponentiation algorithms depends on the group's size. +Therefore, if the sizes do not match, the number of counted operations can be incorrect. + +`DebugGroup` has a multitude of such getter methods for retrieving various operation statistics. +Their names all are constructed to the following structure: +The first part denotes the type of count the getter retrieves, in this case `getNumSquarings` and number of squarings done. +The second part denotes the *counting mode* (here `Total`) and the third, if present, relates to the bucket system (here `Default`). +`DebugBilinearGroup` has similar methods for retrieving pairing data. + +Instead of using the getter methods, you can also use the provided `formatCounterData` methods which automatically format the count data for printing. +The `formatCounterData` methods of `DebugBilinearGroup` summarize the pairing data as well as the data of \\(G_1\\), \\(G_2\\) and \\(G_T\\). + +As seen in the previous example, `DebugGroup` does use lazy evaluation, meaning that you need to ensure all lazy computations have finished before retrieving the tracked results. +So make sure to always call `compute()` on every involved `DebugGroupElement` before accessing any counter data, or call `getRepresentation()` to serialize any involved objects as this also leads to a blocking computation. +For your convenience, `DebugGroup` also overrides `compute()` to behave like `computeSync()` in that it blocks until the computation is done. + +## Do I need to keep reading? + +In the following section, we will explain some of the more advanced feature of the counting system. +If your benchmark uses only a single `DebugGroup` and/or `DebugBilinearGroup` instance (specifically no interactive protocols), you may not need to know about those features. +In that case you can just use `DebugGroup` and `DebugBilinearGroup`, and then retrieve the data using the `formatCounterDataDefault()` formatting method. +If you want to use the getter methods with suffix `Default`, you should read at least the following section about counting modes to understand the difference between `Total` and `NoExpMultiExp` getter methods. +If your application uses multiple `DebugGroup` instances and/or multiple parties, reading the sections about static counting and the bucket system is recommended. + +## Counting modes + The counting is done in two modes: The `NoExpMultiExp` mode and the `Total` mode. Group operations metrics from the `NoExpMultiExp` mode disregard operations done inside (multi-)exponentiations while the `Total` mode does account for operations inside (multi-)exponentiations. `NoExpMultiExp` measurements are therefore independent of the actual (multi-)exponentiation algorithm while `Total` measurements are more expressive in regards to the actual runtime (since estimating group operation runtime is easier than that of a (multi-)exponentiation). @@ -273,10 +296,10 @@ As an example we consider the computation of $$g^a \cdot h^b$$. The `NoExpMultiExp` mode counts this as a single multi-exponentiation with two terms. No group operations are counted since they are all part of the multi-exponentiation. The `Total` mode does not consider the multi-exponentiation as its own unit. -Instead, it counts the group operations, inversions, and squarings that are part of evaluating the multi-exponentiation (using a wNAF-type algorithm). +Instead, it counts the group operations, inversions, and squarings that are part of evaluating the multi-exponentiation (using a wNAF-type algorithm by default, but the algorithm used can be configured). Combining these metrics gives us therefore a more complete picture of the computational costs. -A more detailed (code) example is given below: +Let's see the difference in action by performing an exponentiation and examining the number of operations done in both counting modes: ```java DebugGroup debugGroup = new DebugGroup("DG1", 1000000); @@ -285,77 +308,128 @@ GroupElement elem2 = debugGroup.getUniformlyRandomNonNeutral(); GroupElement elem3 = debugGroup.getUniformlyRandomNonNeutral(); GroupElement elem4 = debugGroup.getUniformlyRandomNonNeutral(); -// Perform a multi-exponentiation with 4 bases -elem.pow(10).op(elem2.pow(10)).op(elem3.pow(10)).op(elem4.pow(10)).compute(); // An exponentiation elem.pow(10).compute(); -// Squaring, group op and inversion -elem.op(elem).op(elem2).inv().compute(); -// Print summary of all data -System.out.println(debugGroup.formatCounterData()); +System.out.println("Total ops: " + debugGroup.getNumOpsTotalDefault()); +System.out.println("Ops not done in exp: " + debugGroup.getNumOpsNoExpMultiExpDefault()); ``` ``` -------- Operation data for DebugGroup(Lazy DG1;Lazy DG1) ------- ------ Total group operation data: ----- - Number of Group Operations: 34 - Number of Group Inversions: 1 - Number of Group Squarings: 9 ------ Group operation data without operations done in (multi-)exp algorithms: ----- - Number of Group Operations: 1 - Number of Group Inversions: 1 - Number of Group Squarings: 1 ------ Other data: ----- - Number of exponentiations: 1 - Number of terms in each multi-exponentiation: [4] - Number of retrieved representations (via getRepresentation()): 0 +Total ops: 8 +Ops not done in exp: 0 ``` -As you can see, the "Total group operation data" block has much higher numbers than the block below it, due to counting operations done during the multi-exponentiation and exponentiation. +As you can see, `debugGroup.getNumOpsNoExpMultiExpDefault()` returns `0`, while `debugGroup.getNumOpsTotalDefault()` returns `8`. +This is because the only group operations done are inside the exponentiation. -### Lazy evaluation +## Static counting -`DebugGroup` does use lazy evaluation, meaning that you need to ensure all lazy computations have finished before retrieving the tracked results. -So make sure to always call `compute()` on every involved `DebugGroupElement` before accessing any counter data, or call `getRepresentation()` to serialize any involved objects as this also leads to a blocking computation. -For your convenience, `DebugGroup` also overrides `compute()` to behave like `computeSync()` in that it blocks until the computation is done. +Counting in `DebugGroup` and `DebugBilinearGroup` is done statically, meaning that different instances of `DebugGroup` share their counts. -### Configuring Used (Multi-)exponentiation Algorithm +This is illustrated by the following example: -`DebugGroup` makes use of efficient exponentiation and multi-exponentiation algorithms. -The exact algorithm used changes the resulting group operation counts. -To manually configure these algorithms, `DebugGroup` (and `DebugBilinearGroup`) offers setter and getter methods such as `getSelectedMultiExpAlgorithm` and `setSelectedMultiExpAlgorithm`. -Furthermore, you can configure the precomputation and exponentiation window sizes used for those algorithms. -These are the same methods as offered by `LazyGroup`. +```java +DebugGroup debugGroup1 = new DebugGroup("DG1", 1000000); +GroupElement elem1 = debugGroup1.getUniformlyRandomNonNeutral(); +DebugGroup debugGroup2 = new DebugGroup("DG2", 1000000); + +elem1.op(elem1).compute(); -### Serialization Tracking -`DebugGroup` not only allows for tracking group operations, it also counts how many calls of `getRepresentation()` have been called on elements of the group. This has the purpose of allowing you to track serializations. -The count is accessible via `getNumRetrievedRepresentations()`. +System.out.println("DG1: " + debugGroup1.getNumSquaringsTotalDefault()); +System.out.println("DG2: " + debugGroup2.getNumSquaringsTotalDefault()); +``` +``` +DG1: 1 +DG2: 1 +``` -## DebugBilinearGroup +Technically, the only squaring that is done is the one on `elem1` which is an element of `debugGroup1`. +Due to static counting, however, the squaring is also counted by `debugGroup2`. +This has the advantage that count data of operations done inside internal `DebugGroup` instances (internal in the sense of hidden inside the application and not accessible to the benchmarker) can be retrieved via any other `DebugGroup` instance. -Cryptimeleon Math also provides a `BilinearGroup` implementation that can be used for counting, the `DebugBilinearGroup` class. -It uses a simple (not secure) $$\mathbb{Z}_n$$ pairing. +This sharing of data does *not* apply to the `DebugGroup` instances exposed by `DebugBilinearGroup`'s `getG1()`, `getG2()`, and `getGT()` methods. +These all have their own shared count data pools. +Specifically, all instances of `DebugGroup` obtained via `getG1()` share their count data, and similarly for `getG2()` and `getGT()`. -In addition to the usual group operation counting done by the three `DebugGroup` instances contained in the bilinear group, `DebugBilinearGroup` also allows you to track number of pairings performed. +We illustrate this via the following example: ```java -DebugBilinearGroup bilGroup = new DebugBilinearGroup(100); -// Get G1 and G2 of the bilinear group -DebugGroup groupG1 = (DebugGroup) bilGroup.getG1(); -DebugGroup groupG2 = (DebugGroup) bilGroup.getG2(); +DebugGroup debugGroup = new DebugGroup("DG1", 1000000); +GroupElement elem = debugGroup.getUniformlyRandomNonNeutral(); +DebugBilinearGroup debugBilinearGroup = new DebugBilinearGroup(BigInteger.valueOf(1000000), BilinearGroup.Type.TYPE_3); +DebugGroup bilinearG1 = (DebugGroup) debugBilinearGroup.getG1(); +GroupElement elemG1 = bilinearG1.getUniformlyRandomNonNeutral(); +DebugGroup bilinearG2 = (DebugGroup) debugBilinearGroup.getG2(); + +elem.op(elem).compute(); +elemG1.op(elemG1).compute(); +elemG1.op(elemG1).compute(); + +System.out.println("DG1: " + debugGroup.getNumSquaringsTotalDefault()); +System.out.println("Bilinear G1: " + bilinearG1.getNumSquaringsTotalDefault()); +System.out.println("Bilinear G2: " + bilinearG2.getNumSquaringsTotalDefault()); +``` +``` +DG1: 1 +Bilinear G1: 2 +Bilinear G2: 0 +``` -GroupElement elemG1 = groupG1.getUniformlyRandomNonNeutral(); -GroupElement elemG2 = groupG2.getUniformlyRandomNonNeutral(); +As you can see, the squaring computed inside `debugGroup` is not counted by `bilinearG1` or `bilinearG2`. +The two squarings are only counted by `bilinearG1` and not the others. +The reason for this is that \\(G_1\\), \\(G_2\\) and \\(G_T\\) can have different costs for operations, and so it makes sense to count them separately. -// Compute a paring -bilGroup.getBilinearMap().apply(elemG1, elemG2).compute(); +To summarize: All `DebugGroup` instances obtained by directly instantiating them via the constructors of `DebugGroup` share their count data. +The `DebugGroup` instances obtained via `getG1()`, `getG2()`, and `getGT()` each have their own count data pools. +Keep in mind that `DebugGroup` instances obtained via the `getG1()` (or `getG2()` and `getGT()`) methods of different `DebugBilinearGroup` instances also share their count data. -System.out.println(bilGroup.getNumPairings()); +## The bucket system + +All the count data getter methods we used so far had the suffix `Default`. +This is related to the bucket system of `DebugGroup` and `DebugBilinearGroup`. +Due to the static counting, `DebugGroup` instances share their count data. +The bucket system allows different `DebugGroup` instances to count separately by using what we call "buckets". +Each bucket has its own count data tracking. +Using the `setBucket(String bucketName)` method on a `DebugGroup` instance, one can tell that `DebugGroup` instance to use the bucket with name `bucketName` for counting. +Any operations done on elements of that `DebugGroup` instance are counted inside the currently activated bucket of that instance. +This allows different `DebugGroup` instances to track their count data separately. +The data of a specific bucket can be obtained using the getter methods that take a `String` argument such as `getNumOpsTotal(String bucketName)`. + +After initialization, and before any calls to `setBucket`, a default bucket is used, whose data as we saw already can be obtained using the getter methods ending with the suffix `Default`. +This default bucket has no name, so you don't need to worry about conflicting names. +The getter methods ending with the suffix `AllBuckets` summarize the data across all buckets, including the default bucket. + +```java +DebugGroup debugGroup = new DebugGroup("DG1", 1000000); +GroupElement elem = debugGroup.getUniformlyRandomNonNeutral(); + +elem.op(elem).compute(); +debugGroup.setBucket("bucket1"); +elem.op(elem).compute(); + +System.out.println("Default bucket: " + debugGroup.getNumSquaringsTotalDefault()); +System.out.println("bucket1 bucket: " + debugGroup.getNumSquaringsTotal("bucket1")); ``` ``` -1 +Default bucket: 1 +bucket1 bucket: 1 ``` +In the above example, we do a squaring before calling `setBucket`, meaning that the squaring is counted by the default bucket. +We then switch `debugGroup` to the bucket `bucket1` which then counts the second squaring. +The printed results confirm this view, each bucket has one squaring. +Keep in mind that static counting still holds, so if you use the same bucket name for different `DebugGroup` instances, they will share the count data. +The bucket system also applies to `DebugBilinearPairing` and its pairing counter. + +An example application of the bucket system is benchmarking an interactive protocol. +By using separate buckets for each of the participating parties, you can count operations per party. +## Configuring used (multi-)exponentiation algorithms + +`DebugGroup` makes use of efficient exponentiation and multi-exponentiation algorithms. +The exact algorithm used changes the resulting group operation counts. +To manually configure these algorithms, `DebugGroup` (and `DebugBilinearGroup`) offers setter and getter methods such as `getSelectedMultiExpAlgorithm` and `setSelectedMultiExpAlgorithm`. +Furthermore, you can configure the precomputation and exponentiation window sizes used for those algorithms. +These are the same methods as offered by `LazyGroup`. # References From f4a40574f78458ae91fd6169b94ce18d8028d8d6 Mon Sep 17 00:00:00 2001 From: Raphael Heitjohann <5891816+rheitjoh@users.noreply.github.com> Date: Wed, 28 Jul 2021 13:40:32 +0200 Subject: [PATCH 7/7] Rewrite benchmarking document to remove Default suffix stuff --- docs/benchmarking.md | 37 ++++++++++++++++++------------------- 1 file changed, 18 insertions(+), 19 deletions(-) diff --git a/docs/benchmarking.md b/docs/benchmarking.md index 9af8e5e..b3a7e74 100644 --- a/docs/benchmarking.md +++ b/docs/benchmarking.md @@ -250,7 +250,7 @@ GroupElement elem = debugGroup.getUniformlyRandomNonNeutral(); elem.op(elem).compute(); // Print number of squarings in group -System.out.println(debugGroup.getNumSquaringsTotalDefault()); +System.out.println(debugGroup.getNumSquaringsTotal()); ``` ``` 1 @@ -264,12 +264,14 @@ Therefore, if the sizes do not match, the number of counted operations can be in `DebugGroup` has a multitude of such getter methods for retrieving various operation statistics. Their names all are constructed to the following structure: The first part denotes the type of count the getter retrieves, in this case `getNumSquarings` and number of squarings done. -The second part denotes the *counting mode* (here `Total`) and the third, if present, relates to the bucket system (here `Default`). +The second part denotes the *counting mode* (here `Total`) and the third, in this case not present, relates to the bucket system. `DebugBilinearGroup` has similar methods for retrieving pairing data. - Instead of using the getter methods, you can also use the provided `formatCounterData` methods which automatically format the count data for printing. The `formatCounterData` methods of `DebugBilinearGroup` summarize the pairing data as well as the data of \\(G_1\\), \\(G_2\\) and \\(G_T\\). +Additionally, both classes provide reset methods such as `resetCounters()` that can be used to reset the counters. +If you only want to measure certain procedures, for example you only want to benchmark the verification algorithm and not the signing before, then resetting the counters before the procedure to measure is advised. + As seen in the previous example, `DebugGroup` does use lazy evaluation, meaning that you need to ensure all lazy computations have finished before retrieving the tracked results. So make sure to always call `compute()` on every involved `DebugGroupElement` before accessing any counter data, or call `getRepresentation()` to serialize any involved objects as this also leads to a blocking computation. For your convenience, `DebugGroup` also overrides `compute()` to behave like `computeSync()` in that it blocks until the computation is done. @@ -278,8 +280,8 @@ For your convenience, `DebugGroup` also overrides `compute()` to behave like `co In the following section, we will explain some of the more advanced feature of the counting system. If your benchmark uses only a single `DebugGroup` and/or `DebugBilinearGroup` instance (specifically no interactive protocols), you may not need to know about those features. -In that case you can just use `DebugGroup` and `DebugBilinearGroup`, and then retrieve the data using the `formatCounterDataDefault()` formatting method. -If you want to use the getter methods with suffix `Default`, you should read at least the following section about counting modes to understand the difference between `Total` and `NoExpMultiExp` getter methods. +In that case you can just use `DebugGroup` and `DebugBilinearGroup`, and then format the data using `formatCounterData()`. +If you want to use the getter methods, you should read at least the following section about counting modes to understand the difference between `Total` and `NoExpMultiExp` getter methods. If your application uses multiple `DebugGroup` instances and/or multiple parties, reading the sections about static counting and the bucket system is recommended. ## Counting modes @@ -290,7 +292,6 @@ Group operations metrics from the `NoExpMultiExp` mode disregard operations done This is useful if you want to track (multi-)exponentiations only as a single unit and not the underlying group operations. Exponentiation and multi-exponentiation data can be accessed via the `getNumExps()` and `getMultiExpTermNumbers()` methods, where the latter returns an array containing the number of bases in each multi-exponentiation done. -Additionally, `resetCounters()` can be used to reset all operation counters, and `formatCounterData()` provides a printable string that summarizes all collected data. As an example we consider the computation of $$g^a \cdot h^b$$. The `NoExpMultiExp` mode counts this as a single multi-exponentiation with two terms. @@ -311,15 +312,15 @@ GroupElement elem4 = debugGroup.getUniformlyRandomNonNeutral(); // An exponentiation elem.pow(10).compute(); -System.out.println("Total ops: " + debugGroup.getNumOpsTotalDefault()); -System.out.println("Ops not done in exp: " + debugGroup.getNumOpsNoExpMultiExpDefault()); +System.out.println("Total ops: " + debugGroup.getNumOpsTotal()); +System.out.println("Ops not done in exp: " + debugGroup.getNumOpsNoExpMultiExp()); ``` ``` Total ops: 8 Ops not done in exp: 0 ``` -As you can see, `debugGroup.getNumOpsNoExpMultiExpDefault()` returns `0`, while `debugGroup.getNumOpsTotalDefault()` returns `8`. +As you can see, `debugGroup.getNumOpsNoExpMultiExp()` returns `0`, while `debugGroup.getNumOpsTotal()` returns `8`. This is because the only group operations done are inside the exponentiation. ## Static counting @@ -335,8 +336,8 @@ DebugGroup debugGroup2 = new DebugGroup("DG2", 1000000); elem1.op(elem1).compute(); -System.out.println("DG1: " + debugGroup1.getNumSquaringsTotalDefault()); -System.out.println("DG2: " + debugGroup2.getNumSquaringsTotalDefault()); +System.out.println("DG1: " + debugGroup1.getNumSquaringsTotal()); +System.out.println("DG2: " + debugGroup2.getNumSquaringsTotal()); ``` ``` DG1: 1 @@ -365,9 +366,9 @@ elem.op(elem).compute(); elemG1.op(elemG1).compute(); elemG1.op(elemG1).compute(); -System.out.println("DG1: " + debugGroup.getNumSquaringsTotalDefault()); -System.out.println("Bilinear G1: " + bilinearG1.getNumSquaringsTotalDefault()); -System.out.println("Bilinear G2: " + bilinearG2.getNumSquaringsTotalDefault()); +System.out.println("DG1: " + debugGroup.getNumSquaringsTotal()); +System.out.println("Bilinear G1: " + bilinearG1.getNumSquaringsTotal()); +System.out.println("Bilinear G2: " + bilinearG2.getNumSquaringsTotal()); ``` ``` DG1: 1 @@ -385,17 +386,15 @@ Keep in mind that `DebugGroup` instances obtained via the `getG1()` (or `getG2() ## The bucket system -All the count data getter methods we used so far had the suffix `Default`. -This is related to the bucket system of `DebugGroup` and `DebugBilinearGroup`. Due to the static counting, `DebugGroup` instances share their count data. -The bucket system allows different `DebugGroup` instances to count separately by using what we call "buckets". +The bucket system allows different `DebugGroup` (and `DebugBilinearGroup`) instances to count separately by using what we call "buckets". Each bucket has its own count data tracking. Using the `setBucket(String bucketName)` method on a `DebugGroup` instance, one can tell that `DebugGroup` instance to use the bucket with name `bucketName` for counting. Any operations done on elements of that `DebugGroup` instance are counted inside the currently activated bucket of that instance. This allows different `DebugGroup` instances to track their count data separately. The data of a specific bucket can be obtained using the getter methods that take a `String` argument such as `getNumOpsTotal(String bucketName)`. -After initialization, and before any calls to `setBucket`, a default bucket is used, whose data as we saw already can be obtained using the getter methods ending with the suffix `Default`. +After initialization, and before any calls to `setBucket`, a default bucket is used, whose data as we saw already can be obtained using the getter methods that do not take a `String` argument and also do not end with the suffix `AllBuckets`. This default bucket has no name, so you don't need to worry about conflicting names. The getter methods ending with the suffix `AllBuckets` summarize the data across all buckets, including the default bucket. @@ -407,7 +406,7 @@ elem.op(elem).compute(); debugGroup.setBucket("bucket1"); elem.op(elem).compute(); -System.out.println("Default bucket: " + debugGroup.getNumSquaringsTotalDefault()); +System.out.println("Default bucket: " + debugGroup.getNumSquaringsTotal()); System.out.println("bucket1 bucket: " + debugGroup.getNumSquaringsTotal("bucket1")); ``` ```