Missing columns after OC 10.5 -> NC 21.0.0beta2 upgrade

[PVince81]

Steps to reproduce

1. Upgrade from OC 10.5.0 to NC 21.0.0beta2 by replacing the source code and keeping config and data
2. Run occ upgrade => no errors
3. Run db:convert-filecache-bigint

Expected behaviour

Success

Actual behaviour

Error about missing columns "oc_share_external.parent"

Missing columns

Expected from NC 21.0.0beta2

MariaDB [nextcloud_beta]> describe oc_share_external;
+-----------------+---------------+------+-----+---------+----------------+
| Field           | Type          | Null | Key | Default | Extra          |
+-----------------+---------------+------+-----+---------+----------------+
| id              | bigint(20)    | NO   | PRI | NULL    | auto_increment |
| parent          | bigint(20)    | YES  |     | -1      |                |
| share_type      | int(11)       | YES  |     | NULL    |                |
| remote          | varchar(512)  | NO   |     | NULL    |                |
| remote_id       | varchar(255)  | YES  |     |         |                |
| share_token     | varchar(64)   | NO   |     | NULL    |                |
| password        | varchar(64)   | YES  |     | NULL    |                |
| name            | varchar(64)   | NO   |     | NULL    |                |
| owner           | varchar(64)   | NO   |     | NULL    |                |
| user            | varchar(64)   | NO   | MUL | NULL    |                |
| mountpoint      | varchar(4000) | NO   |     | NULL    |                |
| mountpoint_hash | varchar(32)   | NO   |     | NULL    |                |
| accepted        | int(11)       | NO   |     | 0       |                |
+-----------------+---------------+------+-----+---------+----------------+

Actual

MariaDB [owncloud]> describe oc_share_external;
+-----------------+---------------------+------+-----+---------+----------------+
| Field           | Type                | Null | Key | Default | Extra          |
+-----------------+---------------------+------+-----+---------+----------------+
| id              | bigint(20)          | NO   | PRI | NULL    | auto_increment |
| remote          | varchar(512)        | NO   |     | NULL    |                |
| share_token     | varchar(64)         | NO   |     | NULL    |                |
| password        | varchar(64)         | YES  |     | NULL    |                |
| name            | varchar(64)         | NO   |     | NULL    |                |
| owner           | varchar(64)         | NO   |     | NULL    |                |
| user            | varchar(64)         | NO   | MUL | NULL    |                |
| mountpoint      | varchar(4000)       | NO   |     | NULL    |                |
| mountpoint_hash | varchar(32)         | NO   |     | NULL    |                |
| remote_id       | varchar(255)        | NO   |     | -1      |                |
| accepted        | int(11)             | NO   |     | 0       |                |
| lastscan        | bigint(20) unsigned | YES  |     | NULL    |                |
+-----------------+---------------------+------+-----+---------+----------------+

=> need to:

• add "parent" column
• remove "lastscan" column
• PR for share_external here: Add migration for oc_share_external columns #24829

@juliushaertl not sure how these were missed, maybe some of the migrations did not run

[PVince81]

• oc_systemtag.assignable column not properly deleted: https://github.com/nextcloud/server/pull/23044/files#r547282236

  • also need to delete it from the key

• oc_systemtag_object_mapping has unique key instead of PRIMARY KEY (objecttype,objectid,systemtagid) => solved by occ command

[PVince81]

• oc_comments.reference_id is missing => optional
• oc_comments_read_markers index differences solved by occ command
• oc_dav_properties not deleted?

[PVince81]

• oc_federated_reshares:
  • remote_id default is different
  • index is not primary key solved by occ command

is:

  `remote_id` varchar(255) COLLATE utf8mb4_bin NOT NULL COMMENT 'share ID at the remote server',

should be:

  `remote_id` varchar(255) COLLATE utf8mb4_bin DEFAULT '',

[nickvergessen]

oc_comments.reference_id is missing

That's an optional column which can be added with an occ command

[PVince81]

• oc_filecache missing fs_mtime and fs_size keys => solved by occ command

[PVince81]

• oc_notifications, missing icon column: but this is because I had the notifications app disabled for many many OC versions and now when enabling the NC notifications app, the column did not get added. Not sure if / how we want to support such cases...

[PVince81]

• should we delete oc_privatedata ?

• oc_properties differences:

  • field lengths
  • defaults
  • missing "property_index": should probably be added to the occ command ?

is:

CREATE TABLE `oc_properties` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `userid` varchar(64) COLLATE utf8mb4_bin DEFAULT '',
  `propertypath` varchar(255) COLLATE utf8mb4_bin NOT NULL,
  `propertyname` varchar(255) COLLATE utf8mb4_bin NOT NULL DEFAULT '',
  `propertyvalue` varchar(255) COLLATE utf8mb4_bin NOT NULL,
  PRIMARY KEY (`id`),
  KEY `properties_path_index` (`userid`,`propertypath`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin ROW_FORMAT=COMPRESSED;

should be:

CREATE TABLE `oc_properties` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `userid` varchar(64) COLLATE utf8mb4_bin NOT NULL DEFAULT '',
  `propertypath` varchar(255) COLLATE utf8mb4_bin NOT NULL DEFAULT '',
  `propertyname` varchar(255) COLLATE utf8mb4_bin NOT NULL DEFAULT '',
  `propertyvalue` longtext COLLATE utf8mb4_bin NOT NULL,
  PRIMARY KEY (`id`),
  KEY `property_index` (`userid`),
  KEY `properties_path_index` (`userid`,`propertypath`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin ROW_FORMAT=COMPRESSED;

[PVince81]

I could not run the index occ yet due to the bigint one blocking, so maybe some of the index discrepancies will be solved there.

[PVince81]

I have a local solution for the first bits (share_external and assignable)