Master forpr by Yuang-Deng · Pull Request #6 · Yuang-Deng/tsbench

Yuang-Deng · 2022-07-06T02:31:27Z

Integrate autogluon to tsbench

it not necessary for all users

Autogluon integration

Yuang-Deng

clean the code

…into master_forpr

jaheba

Can you ensure to run black for unified code formatting?

Also, I'm wondering why lines are commented out instead of removing them?

jaheba · 2022-07-07T12:12:44Z

pyproject.toml

 fastparquet = "^0.6.1"
 fbprophet = "^0.7.1"
-gluonts = {git = "https://github.com/awslabs/gluon-ts.git", rev = "7c94c1149875f6ad2e0d7b0a6bcee952f14d3fb1"}
+gluonts = "0.9.5"


Should we use 0.10?

I also would not fix the patch version.

ok, I will change it.

I notice that version 0.10 has removed save_datasets, save_metadata, is there an alternate interface?

jaheba · 2022-07-07T12:13:06Z

src/cli/datasets/__init__.py


 from ._main import datasets
-from .compute_catch22 import compute_catch22  # type: ignore
+# from .compute_catch22 import compute_catch22  # type: ignore


Why not just remove this?

This function rely on a package called catch22, but this package was renamed suddenly, and there was a problem with the installation, and the old version of the package could not be installed, so we comment it and will add it after solving the problem of this package.

jaheba · 2022-07-07T12:13:31Z

src/cli/datasets/upload.py

    for config in tqdm(DATASET_REGISTRY.values()):
        upload_directory(
-            s3, path / config.name(), bucket, f"{prefix}/{config.name()}"
+            s3, path + '/' + config.name(), bucket, f"{prefix}/{config.name()}"


Why is this needed?

Since both path and config.name() are of type str, "path / config.name()" will raise an error.

jaheba · 2022-07-07T12:14:08Z

src/cli/evaluations/download.py

    The evaluations are downloaded to the provided directory.
    """
    target = Path(evaluations_path)
+    target = Path.joinpath(target, experiment)


Suggested change

target = Path.joinpath(target, experiment)

target /= experiment

target = Path(evaluations_path) / experiment
we might be able to do this with one line of code?

jaheba · 2022-07-07T12:16:15Z

src/tsbench/forecasts/prediction.py

+try:
+    # TODO this problem may be caused by the version of gluonts?
+    from gluonts.support.util import maybe_len
+except:
+    from gluonts.itertools import maybe_len


If the gluon-ts version is fixed, only one should be necessary.

ok, I will fix the version of gluon-ts and remove redundancy code

huibinshen · 2022-07-11T09:07:16Z

src/cli/evaluations/schedule.py

    all_configurations = generate_configurations(Path(config_path))

    # Then, we can run the training, passing parameters as required
+    source_bucket_prefix = "source"


In this case, I would just add the "source" to the code_location path since we are not expect anyone to change it.

I will add a parameter to control the code_location path

huibinshen · 2022-07-11T09:08:37Z

src/cli/evaluations/summarize.py

+
+METRICS = ["mase", "smape", "nrmse", "nd", "ncrps"]
+
+DATASETS = ["m3_yearly", "m3_quarterly", "m3_monthly", "m3_other", "m4_quarterly", "m4_monthly", 


Could we add a "datasets" options so we don't have to modify the file every rime?

huibinshen · 2022-07-11T09:30:59Z

src/cli/evaluations/summarize.py

+                        res['model'] = model
+                    res['dataset'] = ds
+                    res.update(performance['performances'][-1]['testing'])
+                    val_loss = performance['performances'][n-1]['evaluation']['val_loss'] if 'evaluation' in performance['performances'][n-1] else -1


Could you add a comment on why this is needed?

Also, here we assume we always take the last model. Actually when there are multiple epochs, we may want to take the iteration with the minimum val_loss.

huibinshen · 2022-07-11T10:09:14Z

src/tsbench/config/dataset/_factory.py

    assert name in DATASET_REGISTRY, f"Dataset name '{name}' is unknown."
    return DATASET_REGISTRY[name](Path(path))
+
+def construct_pandas_frame_from_iterable_dataset(


Is this already supported in autogluon.timeseries? And if this is not used, we could remove it.

ok, this code was created when I integrated auto pytorch, it is for testing, I will remove it

Yuang-Deng added 30 commits June 10, 2022 08:27

autogluon integration development

6305ce0

hyperparameter set and add a runbook draft

04ce206

updata runbook for config in EC2

02d1182

hyper parameter se

e830d8b

result collect script and run-time set

10f4b02

add runbook run locally

1ef2670

run time set fix

93c933a

format results as table

9f6e03d

result collect and visualizaion

8a324a4

add parameter to control build docker with local code or remote

2db949f

complete runbook and fixed common tsbench command

0c50fab

add retry for log download

9db5471

download process_map fix

8526f68

val log fix and install autogluon by poetry

2f61043

benchmark and eval_metric

cbd58fd

save results as csv

25b4462

for pr

480c5cc

remov useless comments

88934c9

remove useless code

b9ccb57

remove install autogoluon locally

270dbf2

it not necessary for all users

remove temp file

dcde269

remove temp file

b6382d7

remove temp file

718c0ce

remove useless comments

c688af4

remove useless comments

109a83e

remove useless code

2a6bcb6

remove useless code

0df7616

remove useless code

07aad4a

remove useless file

c6ce53f

clean the code

7ba4962

Yuang-Deng added 19 commits July 4, 2022 13:53

remove useless code

8876ccd

remove useless code

63786c3

remove useless code

0fc838d

remove unused file

b832c53

remove unused file

bb2c712

remove unused file

97a6f79

set default for autogluon

1d76611

clean code

a199344

Update auto_gluon_predictor.py

85332c7

remove useless code

8ea72ba

remove useless import

d544b42

Update evaluate.py

c6a80e4

Update evaluate.py

a935208

remove useless code

df68fb4

remove unused parameter

b7a1609

Merge pull request #2 from Yuang-Deng/autogluon_dev

3dc5432

Autogluon integration

Update Dockerfile_local

765a4e0

decouple format and download

b76784e

Update download.py

76290ee

Yuang-Deng commented Jul 6, 2022

View reviewed changes

Yuang-Deng marked this pull request as ready for review July 6, 2022 02:36

Yuang-Deng added 2 commits July 7, 2022 07:07

fix leaderboard

a57e705

Merge branch 'master_forpr' of https://github.com/Yuang-Deng/tsbench …

aaa974d

…into master_forpr

jaheba reviewed Jul 7, 2022

View reviewed changes

huibinshen reviewed Jul 11, 2022

View reviewed changes

	target = Path.joinpath(target, experiment)
	target /= experiment


		METRICS = ["mase", "smape", "nrmse", "nd", "ncrps"]

		DATASETS = ["m3_yearly", "m3_quarterly", "m3_monthly", "m3_other", "m4_quarterly", "m4_monthly",

Conversation

Yuang-Deng commented Jul 6, 2022

Uh oh!

Yuang-Deng left a comment

Choose a reason for hiding this comment

Uh oh!

jaheba left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jaheba Jul 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

huibinshen Jul 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jaheba Jul 7, 2022 •

edited

Loading

huibinshen Jul 11, 2022 •

edited

Loading