DF/data4es: Stage 040 (progress data) (refined history). #381
Conversation
If the input message for some reason does not contain required fields like 'taskid' or 'task_timestamp' -- we cannot process it properly, but it should not interrupt the whole ETL process.
Evildoor
left a comment
Evildoor
left a comment
There was a problem hiding this comment.
Data samples seem to be wrong.
In 040's output, the index is production_progress, while in the code it is progress.
In 019's output, index of all records is tasks_production, while new ones are supposed to go into their own index - most likely due to the previous problem, since 019's ES_INDEX contains 'progress' => getenv('ES_INDEX_PROGRESS'), where the latter is set to production_progress by sourcing ES config.
|
@Evildoor, fixed now. |
Evildoor
left a comment
Evildoor
left a comment
There was a problem hiding this comment.
Sorry, did not notice it before: 019's input now correctly points to 040, but 019's README still says that its samples are made from 017's ones. And 040's README has no section for samples at all.
Remove links to origins of the input data samples: * link to Stage 017 is obsolete; * actual information can be obtained from the symlinks in the `input/` directory; * when `data4es` shape is changed it should affect only the new/removed stages and the shape itself, not stages before and/or after the new/removed ones.
Add 'Samples' sectoion.
Fixed now. |
Waits for:
(#368,#365)->#369->#380,#366Add new stage (040) "progress".
This stage generates documents of new type --
task_progress-- to be indexed in a separate index and used to produce aggrerated statistics on task/campaign progress.This is the last PR in a row of PRs that replace #359.
Overrides #374.