Feature: leader replicates without waiting for its own log flush (0.9)

[glennswest]

What

In 0.9, RaftCore::run_command(Command::AppendInputEntries) awaits the log store's LogFlushed (append_to_log) before the Replicate commands queued after it run. So on a leader, the local fsync and a follower's fsync happen one after the other, and a lone write on a 3-node cluster costs two fsyncs.

This PR changes an AppendInputEntries that carries this node's own leader vote (engine.leader.vote == vote):

• It submits the append with RaftLogStorage::append and goes on, so Replicate goes out at once. The trait already requires the entries to be readable once append returns.
• One forwarder task waits for the flushes in submission order and sends a new Notify::LocalLogFlushed.
• Only then is the leader's own progress updated (update_local_progress), and only if its vote is unchanged. A commit therefore still needs a quorum of durable copies; the leader's copy just no longer has to be first.
• A flush error stops RaftCore, as append_to_log did.

Followers are unchanged: they still answer AppendEntries only after their flush.

Bounds, so a slow or stuck disk behaves as before:

• At most 16 leader appends can be unflushed; the next one waits.
• If the oldest unflushed append has been pending for a heartbeat interval, RaftCore waits for every submitted flush before it sends anything to a follower, heartbeats included. Before this change every append blocked RaftCore, so a leader with a stalled disk went silent and its followers elected another leader. That still happens.

One consequence for applications: the state machine can be asked to apply an entry, committed on followers, before the leader's own flush of it returns. An application that must never apply ahead of its local log has to wait for that flush itself. openraft's startup already handles last_applied > last_log_id by purging the hole, so this is safe either way.

Why

This came from fastetcd, an etcd v3 implementation on openraft. There, a 3-member cluster paid two fsyncs per lone write where etcd pays one (etcd's leader writes its WAL while the entries are on their way to the followers).

Measured on 3 in-process members over gRPC, with every member's fsync delayed by 200 ms, nine lone writes, three rounds:

	median lone write
0.9.24	433 / 438 / 436 ms
with this change	246 / 231 / 215 ms

fastetcd has run this change, vendored on 0.9.24, through its full test suite, crash (SIGKILL) tests, leader-stall election tests, and rolling upgrade and downgrade against unpatched members.

Tests on this branch

• cargo test -p openraft --lib: 188 passed.
• cargo test -p tests --no-fail-fast: everything passed, twice. The base (release-0.9, d60f28b) gave the same result.
• cargo clippy -p openraft --all-targets -- -D warnings and cargo fmt --check are clean.
• membership::t30_commit_joint_config::commit_joint_config_during_0_to_012, run alone (cargo test -p tests --test membership t30_commit_joint_config), fails 20 of 20 on this branch and 20 of 20 on the base, with the same the cluster is already undergoing a configuration change error from its two concurrent add_learner calls. It passes inside the full run. That is unrelated to this change.

0.10's IO pipeline separates submitting from flushing, so this is meant for 0.9 users until then. If you'd rather have it behind a Config flag, I'm happy to add one.

---

This change is