Further enhancement for adopt with customer k8s

[SamYuan1990]

@SamYuan1990 I have some comments/questions for you

1. Caliper tries not to be closer to the application, it's connectors are minimal so tries to be closer to the SUT. However using the 1.4 and 2.2 does add overheard to caliper but not to the SUT so the results should not be influenced by the use of the SDK (unless you haven't allocated enough resource to caliper). It does behave in a similar manner to how a client application would interact with the SUT ie deciding on which peers/orderers to target and then sending proposal requests and transaction broadcasts as well as making discovery requests (if discovery is specified). With the advent of the new 2.4 gateway the client does even less work and just makes requests directly to the peer but again it is driving the SUT in a manner the SUT would expect from a client.
2. As part of the sandbox is there any monitoring of the cluster and the nodes where the SUT is running ? It's really useful to know CPU/Memory/Disk IO of pods etc. Another useful aspect would be log capturing as well.
3. Although I guess it's handy to have everything installed in the same machine if all these processes are trying to use resource then this can influence results. The SUT should be separated from the Load generator, metrics collators and UI providers. I don't know if that's what you can do already but I just thought I would mention it.

Originally posted by @davidkel in #60 (comment)

• add resource limitation for pod. (https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) #69
• add node label selection for pod. (https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes-using-node-affinity/) #68
• add nodeAffinity for PV(C). (https://kubernetes.io/docs/concepts/storage/volumes/#local) #67
• update a map for deployment logic for current fabric network. #70
• clean up script. #66

[SamYuan1990]

@davidkel,

for question 1, yes and I am try to integrated with caliper for now as #30 and #17, the current blocking for me just waiting for caliper 0.5 release. For my opinion, well, any kind of traffic generation can be seen as an application from SUT side. No matter caliper or application itself. For new gateway, (please correct me for any mistake), it reduce client application side coding and configuration work, but as there is no changes for fabric transaction work flow.(endorsement, orderer, validation) When we focus on specific transaction, the total network traffic nearly the same.

Client -> peers -> orderer/ orderer -> peers

Client -> peer, peer handles the workflow.

For me, it's glad to see if the new gateway can speed up the total TPS by any kind of improvement. I am not sure if we have reuse any grpc connection between peers and orderers in new gateway SDK/implementation? to reduce total traffic and network workload?

for question 2, I will have a try and give you feedback soon. ideally as prometheus been integrated, it's easy to get pod and node CPU/Memory/IO etc from prometheus and grafana.

for question 3, from the design of performance sandbox, it reuse 'kubectl' and which verified with KIND and minikube. so you are able to run the basic framework with any CNCF certificated k8s platform. (I suppose that)
To avoid running traffic on different physical device, I am agree that performance sandbox user should modify with kube deployment files to make the pod running on different node and apply with the k8s cluster as bring their own env.
but it's a good point maybe, as we adding node labels. so that
making deployment on same machine as CI and local development with KIND. for real world usage, user can give define with the own node label to make deployment adjust with their own k8s cluster in a user friendly way.

[SamYuan1990]

@davidkel, yes, it's able to find the CPU/memory usage etc from k8s, with my test with screen shot above.

[davidkel]

@SamYuan1990 caliper 0.5.0 has been released

[davidkel]

The new Peer Gateway service means that the client side only talks to a single peer. For a submit the client

1. requests an endorse and waits for a response
2. decides whether to commit that response or not (this is still sent to the peer, the peer will then send it to the orderer)
3. waits for the peer to receive the block with that txn in

For an evaluate request the client just requests an evaluate and the peer decides where that will be run (itself or another peer)

[SamYuan1990]

The new Peer Gateway service means that the client side only talks to a single peer. For a submit the client

1. requests an endorse and waits for a response
2. decides whether to commit that response or not (this is still sent to the peer, the peer will then send it to the orderer)
3. waits for the peer to receive the block with that txn in

For an evaluate request the client just requests an evaluate and the peer decides where that will be run (itself or another peer)

yes, I see. In current point of view, I still see tape as an option for traffic generation, as tape make traffic without gateway sdk.
maybe, we can have a compare after this project integrated with caliper and to see relationship before and after gateway?

[davidkel]

I think it's good you support different load generators and Tape generates a different kind of load compared to caliper (and caliper has the flexibility to generate multiple different load profiles to try to simulate real world scenarios). I need to find some time myself to look at Tape :-)
One thing I have found with caliper is that I'm not sure if running caliper in a cluster is worthwhile. From my experiments it's better to run it in a VM separate to the SUT. In fact what I end up doing is having multiple VMs each running a set of remote caliper workers that communicate back to a caliper manager for co-ordination and results this ensures I have enough cpu power allocated to caliper to push the SUT to it's limits. Obviously that overhead will be reduced significantly via the new peer gateway service as the client SDK does much less work now

[SamYuan1990]

I think it's good you support different load generators and Tape generates a different kind of load compared to caliper (and caliper has the flexibility to generate multiple different load profiles to try to simulate real world scenarios). I need to find some time myself to look at Tape :-) One thing I have found with caliper is that I'm not sure if running caliper in a cluster is worthwhile. From my experiments it's better to run it in a VM separate to the SUT. In fact what I end up doing is having multiple VMs each running a set of remote caliper workers that communicate back to a caliper manager for co-ordination and results this ensures I have enough cpu power allocated to caliper to push the SUT to it's limits. Obviously that overhead will be reduced significantly via the new peer gateway service as the client SDK does much less work now

so far this project is like local dev mode, and you'd better apply the project with your own cluster.
My idea for this is to use podAffinity.
for local dev, the node have all labels, well for apply with user's own cluster, user able to label their node with different but specific label, so that they are able to make workload on different node.
What do you think?

[davidkel]

Unfortunately I'm not familiar with K8s, sorry

[SamYuan1990]

Unfortunately I'm not familiar with K8s, sorry

I see, I am going to share you with related document

[SamYuan1990]

Node affinity is conceptually similar to nodeSelector, allowing you to constrain which nodes your Pod can be scheduled on based on node labels. There are two types of node affinity:

according to k8s node label, @davidkel, we have two options.(as far as I know)

1. with a node selector, force pod (as org1-peer1 for sample) deployed on node with label as org1-peer.
2. with a node affinity, make the pod deployed on node with label org1-peer as primary option, if there's no enough resource, let k8s schedule the pod deployment.

[SamYuan1990]

Pre discussed with @davidkel offline, we are going to do enhancement with task in list below:

• add resource limitation for pod. (https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
• add node label selection for pod. (https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes-using-node-affinity/)
• add nodeAffinity for PV(C). (https://kubernetes.io/docs/concepts/storage/volumes/#local)
• update a map for deployment logic for current fabric network.
• clean up script.

The challenge for CI/CD is to guarantee that the environment where the benchmarks are run are clean and completely identical on each run, If they are not the same then you can't compare benchmarks, unless CI/CD is going into pre-prod of course and the point is to benchmark effectively what your production env results will be as your pre-prod is identical and you are explicitly changing the pre-prod resources to try to improve TPS. If you are using CI/CD to benchmark other changes such as application code then you need to be able to do reliable benchmark comparison.

So that we need to completed with enhancement above to ensure we can provide user a way to adopt this project with their own pre prod/prod env for performance related usage. By ensure they are able to deploy SUT with clean and completely identical for their each run.