-
Notifications
You must be signed in to change notification settings - Fork 2
Open
Description
It would be nice to allow for even simpler, higher-level dataflow concepts.
For example, consider an ML dataflow model along these lines:
{
"choice": "machine_learning_training",
"machine_learning_training": {
"epochs": 10,
"training_data": {
"size": 10000000000,
"batch_size": 10000000
"storage_nodes": [
"S1"
]
},
"gradient_size": 10000000,
"include_parameter_server": false,
"compute_nodes": [
"C1", "C2"
],
"async_choice": "async",
"async": {
"max_stale": 10
},
"all_reduce": "ring",
}
}This high-level model would than get translated into lower level constructs (scatters, computes, broadcasts, ...) before execution.
It would allow users with less knowledge of ML data flow to generate workloads that still mimic ML training.
Metadata
Metadata
Assignees
Labels
No labels