Add the support to install event sources

[houshengbo]

Describe the bug
We cannot install any event resource available in https://github.com/knative/eventing-contrib.

Expected behavior
Install the event sources available in the repo eventing-contrib. Make the CR able to configure the list of event sources, to be installed in the knative eventing.

The custom resource will look like this:

apiVersion: operator.knative.dev/v1alpha1
kind: KnativeEventing
metadata:
  name: knative-eventing
  namespace: knative-eventing
spec:
  event-contrib:
    version: 0.12.1
    sources: github,kafka

We can use a comma separated string to specify all the event sources to be installed. Or use the following CR

apiVersion: operator.knative.dev/v1alpha1
kind: KnativeEventing
metadata:
  name: knative-eventing
  namespace: knative-eventing
spec:
  event-contrib:
    version: 0.12.1
    sources:
      - github
      - kafka

to specify the event sources in a list.

A key work "all" can be used to install all event sources.

To Reproduce
Install the operator, knative eventing can be installed via CR, but not the event sources

Knative release version
All.

Additional context
Add any other context about the problem here such as proposed priority

[houshengbo]

I would like to see that we can specify which version of the event sources to be installed, so we can read and load the manifest of event-contrib online. I am not sure whether new version of knative eventing can work with old version of event sources. For example, will 0.12.0 eventing work fine with 0.11 of event sources or prior versions?

For a certain version of knative eventing, how do we make sure the version of the event sources is a valid one, able to work with the eventing?

[lionelvillard]

@matzew

[lionelvillard]

an alternative it to have one operator per event source. eg: https://github.com/openshift-knative/knative-kafka-operator

[matzew]

I think I am not really sure we should add that to the KnativeEventing CR ...

apiVersion: operator.knative.dev/v1alpha1
kind: KnativeEventingSources
...

would be IMO a better separation? Also, not all users want sources from contrib mixed in the "core install" for Eventing

[matzew]

an alternative it to have one operator per event source. eg: https://github.com/openshift-knative/knative-kafka-operator

that does currently install both: Source and Channel, via one CR. We are looking to change that, so that our "knative kafka operator" can install the bits indiviually, you pick what you want, e.g. just the source, or just the channel, or both

[houshengbo]

Will eventing sources be independent of serving and eventing?
The reason I ask this question is that there is discussion going on knative/serving-operator#275 about whether to merge the eventing and serving operators.

Does event source need to be taken into account or will it be separate regardless of the decision?

[matzew]

I think that SERVING/EVENTING are more core, and more stable than "any random" source from contrib. So, not sure we should have all sources in the one and only operator for knative

[houshengbo]

Per the discussion in knative/serving-operator#275, it seems the vast majority of the knative community prefer to have one operator for knative. If this is an inevitable direction, we have to think about putting future knative CRDs, including the event sources we talked about here, into existing repo, instead of creating new repos. If we still move on the operator work into a multi-operator structure, it is against what the majority's opinion, at least for now.

I personally do not have strong opinion on merging or keeping separate. Just hope we can go on the same direction, based on different discussions in the community.

[aliok]

@slinkydeveloper you might want to comment on the idea you shared with us (plugins etc.)

[aliok]

So, not sure we should have all sources in the one and only operator for knative

+1 on this.

I think we can have a single operator for core (serving+eventing) and a single operator for contrib(sources+channels+...).

Regarding versioning: I don't think we should care about version mismatches here. Operator versions should be same and they should install the same version of eventing and contrib.
I think it would be a huge burden to support the scenarios like where we have 0.11 of contrib and 0.12 of eventing or vice versa.

[lberk]

ISTM this kind of change puts a large burden on downstream distributions of the knative eventing operator, which is especially troublesome when considering it results in forcing support of a greater selection of upstream eventing-contrib sources.
Wouldn't it be better to create a separate sources operator resource? Then allowing downstream to layer on top of the sources.operators resource in a way that provides flexibility for the various distributions and the sources they choose to add/support? Is there a particular benefit in forcing distributions of knative to support more sources by default?

[houshengbo]

@aliok It is hard to define the boundary between knative or knative core and non knative core. The vast majority of the community lean on to merge the eventing and serving operators(if this becomes our ultimate goal), the benefit to users is that they can have one single operator as the one-stop entry to use knative, instead of two or even more.

If we create more operators for knative, e.g. event source operator, is it just going backward against our intention? People has got the benefit of one operator to install knative, but they need to turn to other operators for event sources. How do we determine what should be put in the knative operator? What should set up their own operators?

[evankanderson]

The list of operators at https://github.com/knative/eventing-contrib is nowhere near exhaustive; if you look at the list at https://knative.dev/docs/eventing/sources/ (which is our public registry of available sources, but should not be considered exhaustive), the breakdown is as follows:

• 5 sources in eventing-contrib
• 2 sources in eventing
• 5 sources outside eventing-contrib (knative-gcp, nachocano)
• 2 "container sources" in eventing-contrib
• 10 "container sources" outside eventing-contrib (triggermesh, vaikas, harwayne)

If you look at what an eventing source is, it's basically a (small) operator:

• CRD defining how to register for events from a particular system
• controller for configuring the system to send events and provision a data plane component
• data plane component for extracting events and converting to CloudEvents
• ServiceAccounts and Roles / RoleBindings for permissioning all the above

An easy way to package these event sources and then install them (helm charts or operatorhub.io) seems like it would be a general benefit here; if we are successful, I'd expect to see companies such as SAP, VMware, and GitHub taking responsibility for publishing their own Event Sources (like GCP has done). These sources should work as well (and with the same experience) as the ones maintained by the core team in knative/eventing-contrib.