[16.0][ADD] module_auto_update: allow updates in a minimal environment

[hbrunn]

without this, updates fail when you add columns to models that are used here, namely res.users and ir.model

[StefanRijnhart]

Thanks! Can you squash the fixup commit?

[github-actions]

There hasn't been any activity on this pull request in the past 4 months, so it has been marked as stale and it will be closed automatically if no further activity occurs in the next 30 days.
If you want this PR to never become stale, please ask a PSC member to apply the "no stale" label.

[fkantelberg]

Usually 3 models make problems: res.users, res.partner and res.company. For res.company I only saw it happen only a few times depending on some field definitions. res.users is fixed with that one. However res.partner still causes issues. Because it happens only occasionally the workaround was enough. The reason behind this is the context_get call for the shell which tries to load a context from an user because of prefetch_fields it loads more than necessary (lang + tz) from the database causing SQL exceptions.

A way to solve it would be to hook into it and somehow signal that we don't want to prefetch the fields. If environment variables like below are the best solution is up for discussion but the following code seems to work for res.partner, res.users and res.company without the fix in this PR. The shell was started with:

ODOO_NO_PREFETCH_FIELDS=1 ./odoo-bin shell -c ../../etc/odoo.cfg

from odoo import api

_original_environment_new = api.Environment.__new__


def environment_new(cls, cr, uid, context, *args, **kwargs):
    if os.environ.get("ODOO_NO_PREFETCH_FIELDS"):
        context = dict(context or {}, prefetch_fields=False)

    return _original_environment_new(cls, cr, uid, context, *args, **kwargs)


api.Environment.__new__ = environment_new

What do you think about it? Do you think it's valid to use an environment variable or do you know a better way? The monkey patch is sadly not great. A different way would be to overwrite context_get of res.users and do it similarly. I look forward to it. The PR is a good solution for res.users without invoking the shell differently and monkey patching Environment.

[hbrunn]

@fkantelberg which issues do field additions to res.partner cause in this context? I've used this with many changes to res.partner and never encountered a problem.

Is your proposal an alternative to this PR? It's unclear to me what your use case is, so I can't really form an opinion about env var versus extra parameter, config key, move the code to a module added to --load, ...

[fkantelberg]

If you add a field in an already installed module on res.partner and initialize a normal shell like below. The python code recognizes the newly created field because it loads the module properly. Because res.users inherits res.partner the prefetch_fields for the user context loads all fields of the partner and the SQL query includes the newly created field which isn't added to the table. This already happens on initialization of the python shell. Before you can actually call the functions of module_auto_update. This case wasn't handled by this PR and I thought if we want to fix these annoyances we can try to fix it fully. It works if I add a field for res.users which is also an improvement.

My current guess is: context_get of res.users only loads lang and tz which are only part of res.partner and therefore the fields of res.users aren't fetch and only res.partner fields are fetched during the initialization.

./odoo-bin shell -c ../../etc/odoo.cfg 

ERROR: column res_partner.abct_test does not exist
LINE 1: ..."."signup_type", "res_partner"."phone_sanitized", "res_partn...

Traceback (most recent call last):
  File "/srv/odoo/odoo/parts/odoo/./odoo-bin", line 6, in <module>
    odoo.cli.main()
    ~~~~~~~~~~~~~^^
  File "/srv/odoo/odoo/parts/odoo/odoo/cli/command.py", line 133, in main
    command().run(args)
    ~~~~~~~~~~~~~^^^^^^
  File "/srv/odoo/odoo/parts/odoo/odoo/cli/shell.py", line 162, in run
    self.shell(dbnames[0])
    ~~~~~~~~~~^^^^^^^^^^^^
  File "/srv/odoo/odoo/parts/odoo/odoo/cli/shell.py", line 140, in shell
    ctx = api.Environment(cr, uid, {})['res.users'].context_get()
  File "/srv/odoo/odoo/parts/odoo/odoo/tools/cache.py", line 98, in lookup
    return self.lookup(*args, **kwargs)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/srv/odoo/odoo/parts/odoo/odoo/tools/cache.py", line 155, in lookup
    value = self.method(*args, **kwargs)
  File "/srv/odoo/odoo/parts/odoo/odoo/addons/base/models/res_users.py", line 700, in context_get
    context = self.env.user.read(['lang', 'tz'], load=False)[0]
              ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/odoo/odoo/parts/odoo/odoo/addons/base/models/res_users.py", line 569, in read
    return super().read(fields=fields, load=load)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/odoo/odoo/parts/odoo/odoo/orm/models.py", line 3487, in read
    return self._read_format(fnames=fields, load=load)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/odoo/odoo/parts/odoo/odoo/orm/models.py", line 3744, in _read_format
    vals[name] = convert(record[name], record, use_display_name)
                         ~~~~~~^^^^^^
  File "/srv/odoo/odoo/parts/odoo/odoo/orm/models.py", line 6680, in __getitem__
    return self._fields[key].__get__(self)
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/srv/odoo/odoo/parts/odoo/odoo/orm/fields.py", line 1746, in __get__
    self.compute_value(recs)
    ~~~~~~~~~~~~~~~~~~^^^^^^
  File "/srv/odoo/odoo/parts/odoo/odoo/orm/fields.py", line 1917, in compute_value
    records._compute_field_value(self)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/srv/odoo/odoo/parts/odoo/odoo/orm/models.py", line 4949, in _compute_field_value
    determine(field.compute, self)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/srv/odoo/odoo/parts/odoo/odoo/orm/fields.py", line 84, in determine
    return needle(records, *args)
  File "/srv/odoo/odoo/parts/odoo/odoo/orm/fields.py", line 720, in _compute_related
    record[self.name] = self._process_related(value[self.related_field.name], record.env)
                                              ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/odoo/odoo/parts/odoo/odoo/orm/models.py", line 6680, in __getitem__
    return self._fields[key].__get__(self)
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/srv/odoo/odoo/parts/odoo/odoo/orm/fields.py", line 1702, in __get__
    recs._fetch_field(self)
    ~~~~~~~~~~~~~~~~~^^^^^^
  File "/srv/odoo/odoo/parts/odoo/odoo/orm/models.py", line 3769, in _fetch_field
    self.fetch(fnames)
    ~~~~~~~~~~^^^^^^^^
  File "/srv/odoo/odoo/parts/odoo/odoo/orm/models.py", line 3809, in fetch
    fetched = self._fetch_query(query, fields_to_fetch)
  File "/srv/odoo/odoo/parts/odoo/odoo/orm/models.py", line 3914, in _fetch_query
    rows = self.env.execute_query(query.select(*sql_terms))
  File "/srv/odoo/odoo/parts/odoo/odoo/orm/environments.py", line 534, in execute_query
    self.cr.execute(query)
    ~~~~~~~~~~~~~~~^^^^^^^
  File "/srv/odoo/odoo/parts/odoo/odoo/sql_db.py", line 426, in execute
    self._obj.execute(query, params)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
psycopg2.errors.UndefinedColumn: column res_partner.abct_test does not exist
LINE 1: ..."."signup_type", "res_partner"."phone_sanitized", "res_partn...

[hbrunn]

hmm, my commandline is

ocb/odoo-bin shell --addons-path ocb/addons,ocb/odoo/addons,custom-addons -d odoo16 --smtp False --http-interface localhost --http-port 42420 --limit-time-real 6000 $* <<ODOODOC
env['ir.module.module'].upgrade_changed_checksum_shell()
ODOODOC

and this works to update both res.user and res.partner changes

[hbrunn]

haha, because I've merged in hbrunn/OCB@eb35a10 in my projects for this reason :-)

Well, indeed we'll need to patch this away somehow. I guess making this a real command as in https://github.com/OCA/OCB/blob/16.0/odoo/cli/command.py would be the cleanest way to do this?

[fkantelberg]

True that would be a nice option and we could initialize the environment with an empty context. I would love this solution even more.

[StefanRijnhart]

Aside from how this is going to be solved here, would you like to update that PR to work against master @hbrunn? And then maybe we can all throw our weight behind it to get it merged.

[hbrunn]

yes, I'm not going to touch this PR.

Meanwhile

adding a class in cli/__init__.py (it matters that there's a folder called cli in the module and that this is imported by the module's root __init__):

import odoo
from odoo import api, SUPERUSER_ID
from odoo.cli.command import Command
from odoo.cli.shell import Shell
from odoo.tools import config


class ModuleAutoUpgrade(Command):
    """Run auto upgrade"""
    def run(self, args):
        Shell.init(self, args)
        registry = odoo.registry(config['db_name'])
        with registry.cursor() as cr:
            minimal_env = api.Environment(cr, SUPERUSER_ID, {'prefetch_fields': False})
            minimal_env["ir.module.module"].upgrade_changed_checksum(
                overwrite_existing_translations=config['overwrite_existing_translations'],
            )

seems to do the trick, now I can say

ocb/odoo-bin --addons-path=ocb/addons,ocb/odoo/addons,custom-addons moduleautoupgrade -d odoo16

to trigger a module update that's run in a bare environment.

(it matters that --addons-path= - note the equals sign - is the first parameter in case your deployment doesn't put all custom modules in Odoo's standard module paths)

With the interesting part done now I'll put this on my backlog for picking up eventually, but will happily review if anybody wants to continue from here :-)

[fkantelberg]

Wasn't quite working like that (maybe because your patch was still deployed in your environment) but #3487 works nicely. The --addons-path= is quite annoying but I can live with that for now but that's something only a patch in Odoo can fix.

[hbrunn]

superseded by #3487