Error Updating from 4.16.0.0 to 4.16.1.0

[AlexanderKgr]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Update issue

CLOUDSTACK VERSION

4.16.0.0 - 4.16.1.0

CONFIGURATION

Ubuntu Server 18.04
Advanced networking

OS / ENVIRONMENT

SUMMARY

Followed instructions for update me test lab and have the following issue.

STEPS TO REPRODUCE

Installed new system vm template
Stopped management server 
Made the database backup
Fixed repository.
And updated cloudstack management package
The above  error came.

EXPECTED RESULTS

Cloudstack to start, but it didn't

ACTUAL RESULTS

See the above screenshot

[nvazquez]

Hi @AlexanderKgr have you tried upgrading twice? Or testing a 4.16.1 RC version on this environment? It is failing due to duplicated columns from the schema file for the upgrade: https://github.com/apache/cloudstack/blob/4.16.1.0/engine%2Fschema%2Fsrc%2Fmain%2Fresources%2FMETA-INF%2Fdb%2Fschema-41600to41610.sql#L22

As a workaround you can consider removing the column secondary_object from the table vm_work_job and retry the upgrade. Please consider also taking a backup of the database prior to the column removal

[AlexanderKgr]

Hi @AlexanderKgr have you tried upgrading twice? Or testing a 4.16.1 RC version on this environment? It is failing due to duplicated columns from the schema file for the upgrade: https://github.com/apache/cloudstack/blob/4.16.1.0/engine%2Fschema%2Fsrc%2Fmain%2Fresources%2FMETA-INF%2Fdb%2Fschema-41600to41610.sql#L22

As a workaround you can consider removing the column secondary_object from the table vm_work_job and retry the upgrade. Please consider also taking a backup of the database prior to the column removal

Hi @nvazquez . I haven't tried to upgrade twice, neither i have tried 4.16.1 RC because of lack of time. I have backed up everything and also it is a vm with snapshot.
I will try to remove the column.

[AlexanderKgr]

error removing column.

reinstalling package didn't fix it

[sureshanaparti]

@AlexanderKgr Have you upgraded with the packages at "https://download.cloudstack.org/ubuntu/dists/bionic/4.16/pool/" ? The packages there are rebuild and uploaded again, can you check with latest packages.

[AlexanderKgr]

@AlexanderKgr Have you upgraded with the packages at "https://download.cloudstack.org/ubuntu/dists/bionic/4.16/pool/" ? The packages there are rebuild and uploaded again, can you check with latest packages.

i will check it again. thanks for the answer

i have upgraded with this package

cloudstack-management_4.16.1.0~bionic_all.deb | 2022-03-08 15:55

and still facing the error

[nvazquez]

Hi @AlexanderKgr it seems like the duplicate record did not allow the column removal, once you can remove it and remove the column please retry - otherwise it will complain since the column still exists

[AlexanderKgr]

Hi @AlexanderKgr it seems like the duplicate record did not allow the column removal, once you can remove it and remove the column please retry - otherwise it will complain since the column still exists

Look my comment above. It is not possible to remove the column

[nvazquez]

Yes @AlexanderKgr I have seen that - have also been investigating why your database contains that column if version 4.16.0 does not contain it and was introduced only on 4.16.1.

Found this PR which introduced the constraint that is blocking you from removing the column: https://github.com/apache/cloudstack/pull/5541/files#diff-c3fcc536a9c25282503a8d8f0c07b5320ab9c6ffea9e199bafdba6e306f0aa8bR796. But this PR was reverted(#5665) and not included on 4.16.0. It seems like you were not using the 4.16.0 release packages but a development version. This is not included on 4.16.0:

ALTER TABLE cloud.vm_work_job ADD CONSTRAINT vm_work_job_step_and_objects UNIQUE KEY (step,vm_instance_id,secondary_object);

Anyways, to revert this constraint I would suggest the following SQL and then retry dropping the column:

ALTER TABLE cloud.vm_work_job DROP CONSTRAINT vm_work_job_step_and_objects;

Let us know if that fixes the problem, after dropping the column you should be able to upgrade

[sureshanaparti]

Hi @AlexanderKgr, I've checked the upgrade path, and as @nvazquez pointed the column secondary_object doesn't exists in 4.16.0

DB before upgrade:

[cloud]> desc vm_work_job;
+----------------+---------------------+------+-----+---------+-------+
| Field          | Type                | Null | Key | Default | Extra |
+----------------+---------------------+------+-----+---------+-------+
| id             | bigint(20) unsigned | NO   | PRI | NULL    |       |
| step           | char(32)            | NO   | MUL | NULL    |       |
| vm_type        | char(32)            | NO   | MUL | NULL    |       |
| vm_instance_id | bigint(20) unsigned | NO   | MUL | NULL    |       |
+----------------+---------------------+------+-----+---------+-------+
4 rows in set (0.00 sec)

Upgrade log =>
2022-03-10 08:51:25,906 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- Schema upgrade from 4.16.0.0 to 4.16.1.0
2022-03-10 08:51:25,906 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) --;
2022-03-10 08:51:25,907 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) ALTER TABLE cloud.vm_work_job ADD COLUMN secondary_object char(100) COMMENT 'any additional item that must be checked during queueing' AFTER vm_instance_id
2022-03-10 08:51:25,930 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- Stored procedures to handle cloud and cloud_schema changes
2022-03-10 08:51:25,931 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) -- Idempotent ADD COLUMN

DB after upgrade:

[cloud]> desc cloud.vm_work_job;
+------------------+---------------------+------+-----+---------+-------+
| Field            | Type                | Null | Key | Default | Extra |
+------------------+---------------------+------+-----+---------+-------+
| id               | bigint(20) unsigned | NO   | PRI | NULL    |       |
| step             | char(32)            | NO   | MUL | NULL    |       |
| vm_type          | char(32)            | NO   | MUL | NULL    |       |
| vm_instance_id   | bigint(20) unsigned | NO   | MUL | NULL    |       |
| secondary_object | char(100)           | YES  |     | NULL    |       |
+------------------+---------------------+------+-----+---------+-------+
5 rows in set (0.00 sec)

Can you try with the steps @nvazquez suggested, and update us. Thanks.

[AlexanderKgr]

@nvazquez you were right.
i was trying to update a snapshot version of 4.16.
Reverted all my lab to 4.15 and upgraded to 4.16.1 without any issue.
Thanks again for the help

[nvazquez]

Many thanks @AlexanderKgr - great it has worked!