Upgrade issue: Only one reference can have Controller set to true.

[houshengbo]

This is an issue, when you try to upgrade from version 0.10.0 to 0.12.x.

The old version of the CRD is called Eventing, and the new version of the CRD is called KnativeEventing. When I tested if it is able to do the upgrade, by installing the new CR, with the old CR still existing, this was the error message I have got:

message: 'ServiceAccount "eventing-controller" is invalid: metadata.ownerReferences:
      Invalid value: []v1.OwnerReference{v1.OwnerReference{APIVersion:"operator.knative.dev/v1alpha1",
      Kind:"KnativeEventing", Name:"knative-eventing", UID:"957e1232-2033-11ea-9797-025000000001",
      Controller:(*bool)(0xc007f54920), BlockOwnerDeletion:(*bool)(0xc007f54921)},
      v1.OwnerReference{APIVersion:"operator.knative.dev/v1alpha1", Kind:"Eventing",
      Name:"knative-eventing", UID:"29b21071-202d-11ea-9797-025000000001", Controller:(*bool)(0xc007f54922),
      BlockOwnerDeletion:(*bool)(0xc007f54923)}}: Only one reference can have Controller
      set to true. Found "true" in references for knative-eventing and knative-eventing'

The Eventing used to be the owner. If we change the CRD to KnativeEventing, how can I make the ownership switch to the KnativeEventing during the upgrade. This is the issue I am encountering.

[houshengbo]

@jcrossley3 @n3wscott

[bbrowning]

@houshengbo It looks like the eventing operator needs special logic to handle renaming the CRD during upgrade. One of those special places where it needs logic will be in handling anything with owner references to the old CR, like this.

A more general upgrade strategy needs to be defined here. How long will we support the old CRD name in some deprecated form? Will the operator now reconcile both API names equally? Will it rewrite existing CRs using the old name to the new name?

There will have to be explicit knowledge about this change added to the Eventing Operator if we want users to have a smooth upgrade path here.

[houshengbo]

@bbrowning For eventing-operator, we decided to release 0.11.0 with the CRD kind Eventing. After that, we will change the CRD kind into KnativeEventing.

I am currently trying to implement a PoC for the upgrade path here: #53. I create a separate reconciler called version controller dedicated to do the upgrade and even the downgrade.

If we have an old version of operator installed. We upgrade the operator by applying the new release of eventing-operator. version controller reconciler will convert the Old CR into the new CR, and we still only reconcile the new API. The new CR will use the same name as the old CR. If they both belong to the same kind and group, new CR overwrites the old CR. If they do not have the same kind and group, new CR will be a separate CR.

I am not sure how long we will keep the old CRD for compatibility, but as long as we claim a upgrade path to support, we should keep the deprecated CRD in the source code. For example, we decide to support 0.11.0 is able to upgraded to 0.14.0, but not to 0.15.0. For 0.14.0, we keep the code to convert 0.11 to 0.14, and for 0.15 release, we drop the support of 0.11 CRD.