New Quartermaster fully-distributed architecture

[marcomicera]

This proposal intends to abandon Quartermaster's monolithic architecture in terms of cloud infrastructure and repository management.

Table of contents

• Goals
• Current status
• Proposal
  • More distributed
  • Easier to maintain
    • Snapshots
  • More customizable
  • More robust
• Sequence diagram
• Open points
• Possible next steps

Goals

These changes would make Quartermaster:

• distributed, with multiple modules running independently in the cloud;
• easier to maintain, with multiple repositories aiming to do one thing well;
• more customizable, using a declarative configuration (in contrast with the imperative approach currently being used);
• more robust, as the current implementation lacks proper synchronization between modules.

These points will be individually addressed down below.

Current status

Besides initContainers and a StatefulSet for our DGraph database, Quartermaster runs its entire logic in two containers:

1. The qmstr-client container: it executes commands according to the Quartermaster's workflow.
   It is in charge of starting the so-called phases (e.g., build, analysis, etc.).
   Currently, it builds the project, and it successively delegates the execution of the other phases to the qmstr-master container.
2. The qmstr-master container: despite its name, it does not behave as such.
   It takes orders from the qmstr-client container and executes them.

Proposal

Following up, a more detailed plan on how to achieve the aforementioned goals.

More distributed

Some progress has already been done on the infrastructure side: the Quartermaster's main branch has already a dedicated Kubernetes folder, making Quartermaster capable of running in a cluster.
Its logic, however, is not yet fully-distributed as all Quartermaster's modules are running in only two containers (i.e., everything in qmstr-master except for the build phase).

The biggest design change would consist of changing these containers' roles while maintaining a relatively-similar workflow.
First, the master should behave as such, meaning that it shouldn't do anything rather than:

1. imposing orders;
2. synchronizing modules;
3. inserting data into the database¹.

Secondly, the qmstr-client container should be split into multiple entities so that each of those only takes care of a specific task.

More specifically, it would be split into the following entities:

1. Builder(s): the first module to start.
   It only builds the project using the proper Quartermaster wrapper so that build info will be stored into the database through the master¹.
   There would be multiple builders, at least one for every programming language.
   They would all have to be gRPC servers as they would need to wait for a signal from the master commanding them to start their execution.
   • Dependencies: none.
2. Analyzer(s): it would take care of running a specific analyzer on the project.
   Results would be sent to the master¹⁻⁷.
   They would all have to be gRPC servers as well.
   • Dependencies: all builders.
3. Reporter(s): takes care of agglomerating license and compliance information from the master¹ (or the database itself²) and presenting different ways, e.g., a fully exhaustive web page, a short compliance report to be included in a CI step, etc.
   • Dependencies: all the analyzers.

Easier to maintain

Separating those modules into multiple repositories can simplify maintenance and the overall project structure⁹.
Modules would be stored together with their corresponding Docker images and CI/CD stages, breaking down the current pipeline without having to go through the hassle of writing conditional stages⁴.

In particular, the following could end up in a dedicated repository:

• Modules (with their Docker images)
  • Master
  • Analyzers
  • Builders
  • Reporters
• Libraries used by multiple modules
• Deployment files⁵
• Proto messages
• Unique documentation in the website repository
  • Ideally, it would gather modules' documentation (README files) in an automated fashion.

Snapshots

Enhanced maintainability should also mean that developers are able to build and test their modules without the need of running those that depend on them
(i.e., building a reporter without having to launch at least one builder and at least one analyzer before).
While the current implementation stores snapshots in between phases, to my understanding it doesn't make use of them.
Also in case the previous statement turns out to be false, such functionality should be readjusted to this new fully-distributed architecture anyway.
Snapshots would not be implementable without having a long-running, production-ready, and multi-tenant⁸ DGraph instance always running in the background.

More customizable

Currently, Quartermaster goes through the different phases (e.g., build, analysis, etc.) imperatively: it's the qmstr-client container that dictates the workflow and the phases to be executed.
Yet, the master requires a ConfigMap containing configuration for all phases to be run.
Essentially, Quartermaster follows an imperative approach when it comes to workflow definition while having at the same time a declarative configuration for the qmstr-master container.

As emphasized in the "More distributed" paragraph, Quartermaster should follow a declarative approach exclusively.
The phases to be executed and their order should not be imposed by the qmstr-client container issuing commands, but rather from the configuration file given to the master.
The latter would orchestrate and synchronize modules, effectively behaving like a "master".

More robust

Synchronization between the master, client, and database, has always been addressed with simple sleep commands.
While this trivial solution works fine locally, a more complex synchronization mechanism is required when different entities may be scheduled at different times in the cluster⁶.
These entities should also take disruptions and evictions into account.

Sequence diagram

WIP, just a sketch.

Open points

• ¹ Which entities will have access/will be aware of the database?
  • Master and reporter² only
    • Fewer communication channels, simpler sequence diagram, fewer configuration fields, namely, modules do not have to know about the existence of the DB, only the master has to
  • All other modules as well
• ² Should the reporter interrogate the database directly or shall it do it through the master?
• ³ How to link together different pieces after splitting the main repository?
  • Go libraries: importing the corresponding go modules
  • Java libraries
  • Python libraries
• ⁴ Maybe this is not a problem at all, and having multiple repositories would be too dispersive.
• ⁵ Does Quartermaster need a super project or is it simply the deployment repository that binds everything together?
• ⁶ Microservices synchronization.
  Related: softwareengineering.stackexchange.com/a/374033
• ⁷ How would all the different modules know the master's address? Does Quartermaster need a Message Broker?
  • Environment variables
  • "Callback address" field in the protobuf message
• ⁸ Database multi-tenancy: the DGraph StatefulSet might serve different Quartermaster instances as the former is a long-running service and the latter a batch job.
  A trivial solution may consist of adding a "Quartermaster instance ID" field to all objects that are being inserted in the database and let every Quartermaster instance operate only on their own.
  This would also allow snapshots.
• ⁹ While this could simplify per-module testing, this would make integration testing more complicated.

Possible next steps

This paragraph tries to come up with a possible plan for the next achievable steps, assuming that the previous open points have been already addressed.

1. Java builder module separation
   • Goals
     • Split the smallest (and the earliest in the workflow) module into a dedicated container
     • Choose a reliable and easy inter-pod communication system that is needed to test the success of this step
   • Steps
     1. Containerize the Java builder module into its own Docker image⁴⁻⁶
     2. Containerize a small (or the smallest) Quartermaster module taking care of receiving building information¹
     3. Launch a DGraph instance⁸
     4. Let the Java builder module be able to send building information to the Quartermaster module⁷
     5. Check whether DGraph contains the correct building information, comparing its content to a parallel instance launched using the monolithic architecture
   • What has to be addressed
     • ⁴ Should the Java builder module have its own repository, should it end up in the "builders" repository, or should it just go into the unique Quartermaster central repository? How to glue them all together⁵?
     • ⁶ How would the Java builder module start?
     • ¹ Should the Java builder module send its building information directly to the DGraph instance?
     • ⁷ How do modules communicate?
     • ⁸ Is DGraph going to be a long-running service? We might not need necessarily need snapshots at this point.
2. Per-module CI definition
   • Goals
     • Come up with a detailed plan on how images should be built and/or tested and/or deployed
3. Integration tests definition
   • What has to be addressed
     • ⁴⁻⁹ Integration testing highly depends on how repositories will be managed⁴.
4. Separation of all other builders
   • Goals
     • Address the missing programming languages previously supported with the monolithic architecture
5. Separation of analyzers
   • What has to be addressed
     • ⁸ Database multi-tenancy and snapshots to avoid wasting time during development.
6. Separation of reporters

[marijnmur]

General remark: I did not find a way to comment on a line, only as a complete comment. Maybe, or at least I think, it is better to make normal document and do a pull request so we can comment per line and than have a discussion per line. That way a discussion per point can be made and you don't have to quote all the time what you are referring to?

More Distributed:

• what does imposing orders mean. imposing on what, or whom. What is the connection?
• synchronizing modules between what? the newly developed separate containers?
• inserting data into the database? from where. Does data first have to be delivered to the master then before going to the database? This does not sound like something a master should be doing to me. This is not a management task. Database writes are atomic so why do you need a master to do that?
  If I think about making a monolithical design distributed, I would expect the master to be doing only one thing, not three, but that is maybe to idealistically thought.

The split of the client looks okay for starters.

More Customizable

but rather from the configuration file given to the master what would this configuration file look like. defining the different checks to run for a project? How do you change this file? Do you want to follow a sort Map-Reduce flow?

More Robust

So what do you propose here, this is the difficult part I think. How do you make sure that you know an assigned task is done and reported completely. Do you want to implement a sort of messaging system where pods get a task assigned by the master and report to the master again when they finished?

open points

1. how do the modules get the data that they need, or are they stateless? On the one hand you say only master and reporter will be aware of the db, to say in the next line all the other modules as well? How does that work?
2. I think if you let all data flows go through one master that becomes a bottle neck and does not scale well. The master as an orchestrator of work (if that is the goal) should not have the role of data hub as well. that should be split.
3. I would split on functionality before language. but that is debatable I think. I don't know how well the programms/functions can be split
4. generally it makes sense to have a repository for a module. If you have a distributed setup, than your code should be distributed as well and thus live in a separate repository.(again debatable and personal preference). I think in the long run the maintainability improves when the different modules/functions are split into seperate repositories
7. that is dependant on what the synchronization method will become... I would expect something like pub/sub(kafka)
8. not enough technical understanding of the setup to comment...

Possible next steps

The previous paragraph looks like foot notes, but refer to things coming here, that is confusing.

Here I would need more technical knowledge on how QM works to provide comments.
5

[marcomicera]

General remark: I did not find a way to comment on a line, only as a complete comment. Maybe, or at least I think, it is better to make normal document and do a pull request so we can comment per line and than have a discussion per line. That way a discussion per point can be made and you don't have to quote all the time what you are referring to?

While I agree that discussing things in PRs would be more convenient, I don't think we should be pushing such README files in the repo, that's why I created the discussion. But good point indeed. You can quote the reply and then write comments right underneath the lines that you wanna comment, kinda like I'm doing now.

More Distributed:

• what does imposing orders mean. imposing on what, or whom. What is the connection?

The new Quartermaster (master) container should dictate the order of execution of the modules, i.e., builders, analyzers, and reporters.
Connection is still an open point⁷, I'd personally look for a Kubernetes-ready message broker.

• synchronizing modules between what? the newly developed separate containers?

Yes. Ideally, the module containers shouldn't talk to each other, but I imagine a star-shaped communication diagram with the Quartermaster (master) container in the middle.

• inserting data into the database? from where.

If I'm not wrong, builders and analyzers write in the database; reporters should only retrieve data from it.

Does data first have to be delivered to the master then before going to the database? This does not sound like something a master should be doing to me. This is not a management task. Database writes are atomic so why do you need a master to do that?
If I think about making a monolithical design distributed, I would expect the master to be doing only one thing, not three, but that is maybe to idealistically thought.

Agreed. So open point² involves all kinds of modules, really. The only advantage I can think of with that approach (using DB through the master) would be to reduce the number of communication happening in Quartermaster, but I don't think that's such a big of an advantage.

More Customizable

but rather from the configuration file given to the master what would this configuration file look like. defining the different checks to run for a project? How do you change this file? Do you want to follow a sort Map-Reduce flow?

Something along the lines of:

project:
  builders:
  - name: maven
    other: ...
  - name: cpp
    other: ...
  analyzers:
  - name: scancode
    other: ...
  - name: spdx
    other: ...
  reporters:
  - name: html
    address: http://example.com
  - name: ci
    other: ...

As of now, this file only configures the qmstr-master container, but it does not define which tasks are going to be executed. Currently, the qmstr-client container is in control of that. So for instance, if qmstr-client issues a command relative to a module whose configuration is absent, I expect the qmstr-master container to simply fail.

I'd just follow a declarative approach, i.e., that file determines which modules are going to be executed.
The order would always be:

1. Builders
2. Analyzers
3. Reporters

More Robust

So what do you propose here, this is the difficult part I think. How do you make sure that you know an assigned task is done and reported completely. Do you want to implement a sort of messaging system where pods get a task assigned by the master and report to the master again when they finished?

I completely agree, this is the hard part. Maybe with a message broker. KubeMQ is getting my attention...

open points

1. how do the modules get the data that they need, or are they stateless?

Good point. Kubernetes volumes might be enough: modules need the complete repository in their filesystem so that they can build/analyze it.
So AFAIK they shouldn't need any additional input data.

On the one hand you say only master and reporter will be aware of the db, to say in the next line all the other modules as well? How does that work?

I think we should go for one of the two extremes: either all entities can interact with the database or just the Quartermaster (master) container. It's still an open point¹.

2. I think if you let all data flows go through one master that becomes a bottle neck and does not scale well. The master as an orchestrator of work (if that is the goal) should not have the role of data hub as well. that should be split.

Agreed!

4. generally it makes sense to have a repository for a module. If you have a distributed setup, than your code should be distributed as well and thus live in a separate repository.(again debatable and personal preference). I think in the long run the maintainability improves when the different modules/functions are split into seperate repositories

I like this approach too, but I'm afraid this would create way too many repos.
For instance, we have builders. Do you think it's better to create separate repos for different builders (e.g., maven-builder, cpp-builder, python-builder, etc.), or just one repo for module type (e.g., builders, analyzers, reporters)?
I like the latter the most, but we would then need to write conditional CI steps in order to build/test/push only the Docker images that have been modified in a certain commit. Or maybe that's what happens by default?

7. that is dependant on what the synchronization method will become... I would expect something like pub/sub(kafka)

That's where I need advice: pub/sub, message broker, ...? Do I forget anything?

@marijnmur thanks a lot for all the inputs!

[endoroko]

Great work. I read it several times and looked up a couple of things that I didn't know. You seem to have spent quite some thoughts into this concept. Great work!

The idea of using separate builders for each programming language is very interesting! I also like the separation of builder and analyser, but then I think that this could result in quite a number of analysers. Would this still be easy to handle? I wonder if it might be more practical to bundle let's say Python builder with Python analysers in one unit.

Modules would be stored together with their corresponding Docker images and CI/CD stages

I would imagine Docker images reside in a dedicated (private) container registry. Modules and CI/CD in one repository.

Is security (secret management, service mesh, encryption for intra-cluster communication) part of your concept? Might be useful to include security from the beginning.

I will read again tomorrow with a clear brain and add thoughts if need be.

[marcomicera]

Great work. I read it several times and looked up a couple of things that I didn't know. You seem to have spent quite some thoughts into this concept. Great work!

Thanks!

The idea of using separate builders for each programming language is very interesting! I also like the separation of builder and analyser, but then I think that this could result in quite a number of analysers. Would this still be easy to handle? I wonder if it might be more practical to bundle let's say Python builder with Python analysers in one unit.

This was not my idea at all: Quartermaster currently has several builders, analyzers, and reporters. I'm only suggesting to containerize them individually.

Modules would be stored together with their corresponding Docker images and CI/CD stages
I would imagine Docker images reside in a dedicated (private) container registry. Modules and CI/CD in one repository.

I guess the second line of this quote was originally intended to be part of your response, correct if I'm wrong.
If so, I don't see the reason for splitting Docker images from their code and CI, especially if we're gonna be using something like GitHub Actions. I'd rather have a logically isolated piece of code residing in a separate repository along with its Dockerfiles and CI, I think it's more convenient. Did I get it wrong? Was the second line meant to be part of your input?

Is security (secret management, service mesh, encryption for intra-cluster communication) part of your concept? Might be useful to include security from the beginning.

Good point. In short: no, it wasn't... except for service mesh. I'd love some input on this!

[endoroko]

I wasn't aware that builders, analysers and reporters are already existing, so disregard my very unqualified comment ;-)

Yes, the second line was my answer. But apparently you have to add a second CR/LF to end the quote. Otherwise Github treats subsequent lines as a multi-line quote.

Regarding security: it totally depends on the requirements. By default, all communication inside the cluster is unencrypted. If you trust your network and don't have the requirement to encrypt intra-traffic, then you could ignore encryption inside the cluster. Please keep in mind that a service mesh introduces a lot of complexity. In the end it all comes down to: What do you need and why?
And the other aspect is secret management. It could be a good idea to think about how you want to store secrets that you need inside the cluster. Is the k8s default enough for you or does it make sense to use a KMS.
In general, the k8s defaults are okay, but I saw how much work it is to change these things in a running cluster, that's why I suggest you think about it beforehand.

[marcomicera]

By default, all communication inside the cluster is unencrypted. If you trust your network and don't have the requirement to encrypt intra-traffic, then you could ignore encryption inside the cluster.

Do you think it's reasonable to assume that our GKE cluster is not reachable from the outside world unless of course, we expose something with a Service?
Also, Quartermaster is intended to be launched by users in their own Kubernetes clusters. Shall we delegate security issues to them? Wouldn't it be way too complicated for users to implement their security policies within Quartermaster?

Is the k8s default enough for you or does it make sense to use a KMS.

In general, when do you think it's more convenient to use a KMS instead of classic Kubernetes Secrets?

[endoroko]

Do you think it's reasonable to assume that our GKE cluster is not reachable from the outside world unless of course, we expose something with a Service?

Yes, that's very reasonable to protect your endpoints. GCP offers an Identity-Aware Proxy (IAP) that you can put "in front of" your endpoints: https://cloud.google.com/iap/docs/enabling-kubernetes-howto
Like this, no endpoint is publicly available. But there are similar options in the non-Google world. I think it's a good practise to have no public endpoints as they increase the attack surface. And there are several ways to connect more sercurely: something IAP-like, VPN tunnel or even a bastion/jump host.

Also, Quartermaster is intended to be launched by users in their own Kubernetes clusters. Shall we delegate security issues to them? Wouldn't it be way too complicated for users to implement their security policies within Quartermaster?

Oh, if you really need inter-cluster communication (cluster to cluster), then this would be a reason to use a service mesh. But as I said, it will make things more complicated for everyone involved.
Of course, users would not want to deal with security. No one likes security. It's always in the way of convenience :-)

In general, when do you think it's more convenient to use a KMS instead of classic Kubernetes Secrets?

When you speak about convenience, then classic Kubernetes Secrets. BUT then you would have to store secrets in a (public?) repository. And do you want to do that? We use Sealed Secrets:

Sealed Secrets is composed of two parts:

• A cluster-side controller / operator
• A client-side utility: kubeseal
  The kubeseal utility uses asymmetric crypto to encrypt secrets that only the controller can decrypt.

[somayeh-najafi]

• How would all the different modules know the master's address? Does Quartermaster need a Message Broker?

Lots of questions :)

To come to a conclusion about this point we need to answer these questions:
A - What factors make us use Message Broker? list them.
B - If the answer is yes, then Which Message Broker is the best option to use?

A :
Do we need/have these factors in our project?

   1. Redundancy via Persistence
   2. Traffic Spikes
   3. Improve Web Application Page Load Times
   4. Batching for Efficiency
   5. Asynchronous Messaging
   6. Decoupling by Using Data Contracts
   7. Transaction Ordering and Concurrency Challenges
   8. Improve Scalability
   9. Create Resiliency
   10. Guarantee Transaction Occurs Once
   11. Break Larger Tasks into Many Smaller Ones
   12. Monitoring

B:
We assume that we need a Broker, so what features do we need in the following categories?

1. General |  

   Kubernetes Native
   Run Anywhere
   Zero Configuration
   Instant Deployment
   Native Observability
   Persistency
   High Availability
   Unlimited Payloads
   No Other Dependencies

2. Queue |  

   Exactly once delivery
   Message Expiration
   Delayed Delivery
   Dead-Letter
   Long Polling
   Message Visibility Management
   Push Mode
   Pull Mode
   Message Peeking
   Ack-All Queues
   Message Batching

3. Stream |  

   At least once delivery
   Message replay
   Load Balancing
   Consumer Groups
   Pub/Sub Realtime
   At most once delivery
   Fast - In Memory
   Wildcards Partitions
   Load Balancing
   Consumer Groups

4. RPC |  

   Commands - Request Reply
   Queries - Request Reply
   Response Caching
   Built-in Timeouts handling
   Load Balancing
   Consumer Groups
    

[marcomicera]

A - What factors make us use Message Broker? list them.

This article made me think that we might wanna use a message queue instead of a pub-sub as it

ensures that each message for a given topic or channel is delivered to and processed by exactly one consumer

We don't want multiple builders/analyzers to run for a given project.
In case we want to run different types of reporters (that totally makes sense), the master would issue multiple messages, one for each reporter, as they might need different configurations.

Maybe we can still do all of this with pub-sub solutions and I'm missing other key differences...

B - If the answer is yes, then Which Message Broker is the best option to use?

I think we need to compare KubeMQ, Kafka, and RabbitMQ.

A :
Do we need/have these factors in our project?

   1. Redundancy via Persistence
   2. Traffic Spikes
   3. Improve Web Application Page Load Times
   4. Batching for Efficiency
   5. Asynchronous Messaging
   6. Decoupling by Using Data Contracts
   7. Transaction Ordering and Concurrency Challenges
   8. Improve Scalability
   9. Create Resiliency
   10. Guarantee Transaction Occurs Once
   11. Break Larger Tasks into Many Smaller Ones
   12. Monitoring

Those are 12 reasons to use message queuing. They're all pros, no cons, so I don't think it's worth discussing this.

B:
We assume that we need a Broker, so what features do we need in the following categories?

1. General |  
   [...]
2. Queue |  
   [...]
3. Stream |  
   [...]
4. RPC |  
   [...]

This seems to be taken from this KubeMQ comparison table, divided into four different use cases. Nice table, but of course every solution proclaims itself as the best one on their own website hehehe.
I like the "Application Decoupling" use case from their website:

Connectivity solutions such as Application programing interfaces, databases, and storage devices would act as a router to send messages to the consumers. This means they connect with each other and distributes information’s among them to send a unified data to the end-users

[AdinaFra]

I also agree with most of the ideas presented in the proposal with some exceptions :-)

The biggest design change would consist of changing these containers' roles while maintaining a relatively-similar workflow.
First, the master should behave as such, meaning that it shouldn't do anything rather than:

1. imposing orders;
2. synchronizing modules;
3. inserting data into the database¹.

The master should be just the coordinator, without having also roles as synchronising modules or inserting data into databases.

Open points

• ¹ Which entities will have access/will be aware of the database?

  • Master and reporter² only

    • Fewer communication channels, simpler sequence diagram, fewer configuration fields, namely, modules do not have to know about the existence of the DB, only the master has to

  • All other modules as well

Here I think, that the most suitable approach would be to have a dedicated microservice that takes care of all the communication with the database. If the master or any other component/module needs data from the database, it should
request it from that microservice (Single-responsibility_principle)

• ² Should the reporter interrogate the database directly or shall it do it through the master?

It should go through the dedicated microservice

• ³ How to link together different pieces after splitting the main repository?

  • Go libraries: importing the corresponding go modules
  • Java libraries
  • Python libraries

• ⁴ Maybe this is not a problem at all, and having multiple repositories would be too dispersive.

Every module should have its own repository. This would make sharing the module development/responsibility between teams easier.

• ⁵ Does Quartermaster need a super project or is it simply the deployment repository that binds everything together?
• ⁶ Microservices synchronization.
  Related: softwareengineering.stackexchange.com/a/374033

For the microservices synchronization: Kafka

• ⁷ How would all the different modules know the master's address? Does Quartermaster need a Message Broker?

  • Environment variables
  • "Callback address" field in the protobuf message

For service discovery, check out the spring netflix eureka project from the java world.

• ⁸ Database multi-tenancy: the DGraph StatefulSet might serve different Quartermaster instances as the former is a long-running service and the latter a batch job.
  A trivial solution may consist of adding a "Quartermaster instance ID" field to all objects that are being inserted in the database and let every Quartermaster instance operate only on their own.
  This would also allow snapshots.

I don't know if it's a good idea to store the instance ID in the database. This seems like a workaround. What will happen when you get new instance IDs?

• ⁹ While this could simplify per-module testing, this would make integration testing more complicated.

I think docker composer could be a solution for local testing. I also found this article.

[marcomicera]

Here I think, that the most suitable approach would be to have a dedicated microservice that takes care of all the communication with the database. If the master or any other component/module needs data from the database, it should
request it from that microservice (Single-responsibility_principle)

Agreed, will do so.

For the microservices synchronization: Kafka

Nice, thanks. We're thinking of using a Message Broker as we don't need multiple consumers to consume a single message (as it can happen in pub-sub). We've found KubeMQ for this: do you see any crucial disadvantages in using that instead of Kafka?

For service discovery, check out the spring netflix eureka project from the java world.

I'm afraid this can only be used in Spring applications, am I right?
Anyway, I didn't think of automating service discovery in the cloud, I'll look for other solutions.

I don't know if it's a good idea to store the instance ID in the database. This seems like a workaround. What will happen when you get new instance IDs?

I think it's the only way to "tag" nodes belonging to different Quartermaster instances.
Usually, there's no such "tagging" feature in graph databases as they don't have a predefined schema, by definition. The only other alternative I can imagine would consist of spinning a new DGraph StatefulSet every time we launch Quartermaster, which is what we're doing right now, and I don't think it's a wise use of resources. Another key point that I was missing though is the following: which nodes can be shared across multiple Quartermaster instances (no "instance ID tagging")? Which nodes are exclusively related to one specific instance (taggable with an "instance ID")?

I think docker composer could be a solution for local testing. I also found this article.

Using Docker-compose for integration testing is a great idea.
I didn't think of how could we mock other containers during unit testing, though.

Thanks a lot @AdinaFra for your input!

[AdinaFra]

For the microservices synchronization: Kafka

Nice, thanks. We're thinking of using a Message Broker as we don't need multiple consumers to consume a single message (as it can happen in pub-sub). We've found KubeMQ for this: do you see any crucial disadvantages in using that instead of Kafka?

No, I think KubeMQ looks also fine.

For service discovery, check out the spring netflix eureka project from the java world.

I'm afraid this can only be used in Spring applications, am I right?
Anyway, I didn't think of automating service discovery in the cloud, I'll look for other solutions.

Yes, true. I was thinking more about this kind of solution, not precisely this. Sorry, I wasn't clear enough.

I don't know if it's a good idea to store the instance ID in the database. This seems like a workaround. What will happen when you get new instance IDs?

I think it's the only way to "tag" nodes belonging to different Quartermaster instances.
Usually, there's no such "tagging" feature in graph databases as they don't have a predefined schema, by definition. The only other alternative I can imagine would consist of spinning a new DGraph StatefulSet every time we launch Quartermaster, which is what we're doing right now, and I don't think it's a wise use of resources. Another key point that I was missing though is the following: which nodes can be shared across multiple Quartermaster instances (no "instance ID tagging")? Which nodes are exclusively related to one specific instance (taggable with an "instance ID")?

So there are multiple quartermaster instances and each has a DGraph StatefulSet liked to it?

• ⁹ While this could simplify per-module testing, this would make integration testing more complicated.

I think docker composer could be a solution for local testing. I also found this article.

Using Docker-compose for integration testing is a great idea.
I didn't think of how could we mock other containers during unit testing, though.

Yes, I was thinking of Docker-compose only for integration testing.

[marcomicera]

So there are multiple quartermaster instances and each has a DGraph StatefulSet liked to it?

Right now, every Quartermaster instance has its own DGraph StatefulSet, yes. By "database multi-tenancy" I meant multiple Quartermaster instances using the same DGraph StatefulSet to avoid generating the same information over and over again, e.g., build information of a given project.