-
-
Notifications
You must be signed in to change notification settings - Fork 497
[16.0][IMP] account_statement_import_online: Pull wizard with dates and one fetch #798
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: 16.0
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -188,20 +188,24 @@ def _compute_update_schedule(self): | |
| )[0][1], | ||
| } | ||
|
|
||
| def _pull(self, date_since, date_until): | ||
| def _pull(self, date_since, date_until, one_fetch=False): | ||
| """Pull data for all providers within requested period.""" | ||
| is_scheduled = self.env.context.get("scheduled") | ||
| debug = self.env.context.get("account_statement_online_import_debug") | ||
| debug_data = [] | ||
| for provider in self: | ||
| statement_date_since = provider._get_statement_date_since(date_since) | ||
| if one_fetch: | ||
| statement_date_since = date_since | ||
| while statement_date_since < date_until: | ||
| # Note that statement_date_until is exclusive, while date_until is | ||
| # inclusive. So if we have daily statements date_until might | ||
| # be 2020-01-31, while statement_date_until is 2020-02-01. | ||
| statement_date_until = ( | ||
| statement_date_since + provider._get_statement_date_step() | ||
| ) | ||
| if one_fetch: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How checking this by default is compatible with providers that limit fetch to X days per fetch?
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you put any example of that limit? All that I know don't have such limit. And anyway, if you put a big interval, you get in return the message error if so, so you can limit your interval in the wizard. Remember that this is only for the interactive pull wizard.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. E.g. PayPal - one year only, Wise.com - one year only. User may as well select 5 years when importing for the first time on new setup. Such providers should set the step to 1 year making the Obviously you're aiming to fix experience for some specific providers - what step do they set? Since if they set one day but can handle much more so that one_fetch works - those providers' step should be fixed and not an override added.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah, and if they do that, an error will be arisen from the Paypal provider, so I don't see any problem. Instead, with the default value that is daily, they will make 365*5 = 1825 requests, taking a lot of time and make them to think that it's hung. We can put a warning if you prefer saying that the provider may not accept the petition in one block, but what is not sane is to not do it by default.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Let's use just one of the two threads :D why adding
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Because you want to do this only once, not for every automatic pull that happens once you do the initial pulling, that is only needed to be updated. Another possible use is any temporary problem that prevented to download an interval, and in such case, you also want only one pull, not one by one. Example: during a week, you have invalid credentials. Once you put the valid ones, having create statements as daily, only last day will be downloaded, so you click on the pull wizard and download the last week. Why would you want to do this by default through 7 requests instead of just one?
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Let's pause for a second to investigate the idea behind If I got your argument for I fully acknowledge the API rate limiting issue that may arise for some providers when you pull yearly data with day granularity. However I strongly disagree with the proposed approach. As a counter-proposal, I'd suggest:
that would allow to fetch data in minimal amount of requests (one for some providers, couple for providers like PayPal) while respecting the statement grouping granularity of day/week/month |
||
| statement_date_until = date_until | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Statement grouping side effect: Setting Fetch optimization and statement grouping are separate concerns that should not be coupled through a single flag. |
||
| try: | ||
| data = provider._obtain_statement_data( | ||
| statement_date_since, statement_date_until | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,6 +4,7 @@ | |
| # License AGPL-3.0 or later (https://www.gnu.org/licenses/agpl). | ||
|
|
||
| import pprint | ||
| from datetime import datetime | ||
|
|
||
| from odoo import fields, models | ||
|
|
||
|
|
@@ -12,15 +13,13 @@ class OnlineBankStatementPullWizard(models.TransientModel): | |
| _name = "online.bank.statement.pull.wizard" | ||
| _description = "Online Bank Statement Pull Wizard" | ||
|
|
||
| date_since = fields.Datetime( | ||
| string="From", | ||
| required=True, | ||
| default=fields.Datetime.now, | ||
| ) | ||
| date_until = fields.Datetime( | ||
| string="To", | ||
| required=True, | ||
| default=fields.Datetime.now, | ||
| date_since = fields.Date(string="From", required=True, default=fields.Date.today) | ||
|
alexey-pelykh marked this conversation as resolved.
|
||
| date_until = fields.Date(string="To", required=True, default=fields.Date.today) | ||
| one_fetch = fields.Boolean( | ||
| string="One fetch", | ||
| help="If checked, retrieve the whole interval in only one request to the " | ||
| "provider. This is useful if the provider sets request limits.", | ||
| default=True, | ||
|
pedrobaeza marked this conversation as resolved.
|
||
| ) | ||
|
|
||
| def _get_provider(self): | ||
|
|
@@ -35,21 +34,31 @@ def _get_provider(self): | |
| provider = active_record | ||
| return provider | ||
|
|
||
| def _get_datetimes(self): | ||
| """Return the datetime values for the entered dates, including whole starting | ||
| and ending days. | ||
| """ | ||
| self.ensure_one() | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Timezone concern: Consider using |
||
| since = datetime.combine(self.date_since, datetime.min.time()) | ||
| until = datetime.combine(self.date_until, datetime.max.time()) | ||
| return since, until | ||
|
|
||
| def action_pull(self): | ||
| """Pull statements from provider and then show list of statements.""" | ||
| provider = self._get_provider() | ||
| provider._pull(self.date_since, self.date_until) | ||
| since, until = self._get_datetimes() | ||
| provider._pull(since, until, one_fetch=self.one_fetch) | ||
| action = self.env.ref("account.action_bank_statement_tree").sudo().read([])[0] | ||
| action["domain"] = [("journal_id", "=", provider.journal_id.id)] | ||
| return action | ||
|
|
||
| def action_debug(self): | ||
| """Pull statements in debug and show result.""" | ||
| provider = self._get_provider().with_context( | ||
| active_test=False, | ||
| account_statement_online_import_debug=True, | ||
| active_test=False, account_statement_online_import_debug=True | ||
| ) | ||
| data = provider._pull(self.date_since, self.date_until) | ||
| since, until = self._get_datetimes() | ||
| data = provider._pull(since, until, one_fetch=self.one_fetch) | ||
| wizard = self.env["online.bank.statement.pull.debug"].create( | ||
| {"data": pprint.pformat(data)} | ||
| ) | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Design concern: This overrides
_get_statement_date_since()which providers may use to align the start date to statement boundaries (e.g., start of week/month for weekly/monthly statement creation modes). Bypassing it means the fetched data may not align with the provider's expected statement boundaries.The
_get_statement_date_sincemethod exists to allow providers to customize this behavior -- silently skipping it via a boolean flag breaks that contract.