Crash and restart hosts in a runtime

[dtwitty]

My goal is to crash and restart hosts randomly according to some distribution. This way I can see the long-running behavior of my code under duress.

Currently, this looks like

loop {
  sim.step();
  if sim.elapsed() > next_crash_time {
    sim.crash(host);
    next_crash_time = ...
  }
}

However, what would be really nice is if I could crash hosts in a runtime, like this:

loop {
    let next_crash_in = ...;
    sleep(next_crash_in).await;
    sim.crash(host);
    let repair_in = ...;
    // Eventually restart the host.
    spawn(async move {
        sleep(restart_in).await;
        warn!("Restarting node {}", node_to_crash);
        sim.bounce(node_to_crash);
    });
}

I tried to do this in a client, but it doesn't work because client already holds the &mut sim, and I need a &mut sim to crash a host. This means the future can't be Send, so the program doesn't compile.

Is it possible to do this? If not perhaps we could:

• expose a free function, like how elapsed() and hold() work
• expose a runtime so we can inject tasks into the sim

[dtwitty]

Also, is this a good first issue? If so, I wouldn't mind taking this issue if I could get some guidance on how to start

[mcches]

I can't tell what the difference is between the two examples. The same amount of time would be elapsing in between each event. It's structured like this due to how the state is managed and mutated. I don't see a good way to do this right now, but you are welcome to take a look and propose one.