Skip to content

Add a differed topology unbuild in pytest plugin #3

@carlos-jenkins

Description

@carlos-jenkins

Currently, the unbuild of a topology is linear. Before a new suite starts, the previous needs to be unbuild. Depending of the topology being built, the unbuild step can be a large part of the time spent executing the suite.

Given the fact that topology_docker can run multiple suites in parallel, we can differ the unbuild of the topologies until some heuristic or event happen. In this way, the data of the test execution is available sooner.

If this mode is enabled, the unbuild will be differed to a specific "garbage collection" subprocess that will trigger the stop and removal of all the containers upon request or activation by an event or heuristic.

At the implementation details, the plugin can submit to the garbage collector all the topology objects to be discarted. The garbage collector process will query from time to time for resources, and decide to kick in if some parameter of disk space, number of containers, ram or processing is matched.

As part of this optimization, it is important to consider that the current implementation of the stop command is blocking:

https://github.com/HPENetworking/topology_docker/blob/master/lib/topology_docker/node.py#L188
https://github.com/HPENetworking/topology_docker/blob/master/lib/topology_docker/platform.py#L211

An optimization could be that once the topology is submitted to the garbage collector, the garbage collector subprocess request stop of all the containers, but do not wait or block until done. Once the event trigger the removal, the garbage collector will then wait if not stopped and then remove them.
Wait and removal of the container can be done in different threads (consider the GIL), and until all of them finish then the garbage collector will be done.

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions