control-plane: design for improved data-plane monitoring and shard restarts

[jgraettinger]

Today, we use a rather blunt means for restarting failed shards (every five minutes, we un-assign any shard which is currently FAILED).

This is simultaneously too large a delay for a first flaky failure of an otherwise healthy task, and too short a delay for a task which only ever errors. We'd like to instead track failures of a task over time and use a more graduated back-off if it continues to fail after successive restarts, likely ultimately disabling the task automatically after sustained failure.

We also now have multiple data-planes, and we want a consolidated mechanism for managing shard failures across all data-planes.