feat(home): read Home Assistant devices back after a change and never call it done when they didn't change

CHANGELOG.md

@@ -6,6 +6,11 @@ and versions follow [Semantic Versioning](https://semver.org/).
 ## [Unreleased]
 
 ### Added
+- **Home Assistant devices are checked after Sentient changes them.** A moment after it turns on a light, locks a
+  door, opens the blinds, arms the alarm or sets the thermostat, Sentient asks Home Assistant for the device's state
+  again, for a few seconds, and only calls it done when the device really changed. If it didn't, you see it plainly
+  ("Hallway light still reports off."), in chat, in quick commands and in task runs, and never a success. You can
+  change how long it waits in Settings (#386).
 - **Folders and chat apps follow your memory places too.** In Settings, Folders, each folder or Obsidian vault now has
   a **Used in** choice: every chat (as before, and what your folders already have), Personal, Work or a project. A
   folder kept for Work only comes up in work chats, so a personal chat never brings up your work notes unless you ask.

docs/API.md

@@ -954,15 +954,25 @@ or `Tool.verify`, and every MCP tool that is not a look-up (its kind guessed fro
 Today that is `file_write`, `gmail_send`, `gmail_reply`, `email_imap_send`, `gcal_create_event`, `gcal_quick_add`,
 `slack_post_message`, `slack_reply_in_thread`, `discord_send_message`, `whatsapp_send_message`, the Home Assistant
 actions (`home_assistant_turn_on`/`turn_off`/`set_temperature`/`activate_scene`/`run_script`/`open_cover`/`close_cover`/
-`lock`/`unlock`/`set_alarm`, result only) and MCP tools. A call
+`lock`/`unlock`/`set_alarm`) and MCP tools. A call
 the user declined or undid, a try run's noted call and a call handed to the user are never checked.
 1. The result says it worked: no `error`, no `ok`/`success`/`sent`/`created: false`, no failed `status`, and no
    non-empty `failed`/`errors`/`rejected`/`bounced`/`not_sent` list, looked for one level down too (an MCP result's
    `structured` part, or `content` that is JSON). Text `content` starting with "Error" or "Failed" counts as failed.
 2. Read back, where the tool declares `verify(arguments, result, ctx) -> {ok, detail?}` (sync or async, 15 s at most):
    `file_write` reads the file (it exists and holds what was written, line endings aside), Gmail finds the sent
-   message with the `SENT` label, Google Calendar finds the event and it is not cancelled. A read-back that can't run
-   (offline) leaves the result's own answer standing.
+   message with the `SENT` label, Google Calendar finds the event and it is not cancelled. Home Assistant (#386)
+   waits a moment (0.5 s), then reads each device's state again (`GET /api/states/<id>`, about once a second, for
+   `integrations.home_assistant_check_s` seconds in all, default 4, 0 turns it off) until it is as asked: on or off
+   for lights, switches, fans and the like (a media player, thermostat or water heater that was turned on only has to
+   be not off), `locked`/`unlocked`, `open`/`closed`, `armed_<mode>`/`disarmed`, and the asked `hvac_mode` and target
+   `temperature`. A device still `opening`, `closing`, `locking`, `unlocking`, `arming` or `pending` when the time is
+   up counts as fine, with `detail` "Garage door is still opening". One that still shows the old state, or is
+   `unavailable`, fails with `detail` "Hallway light still reports off" and `missed` its entity id (`targets` of a
+   Home Assistant check are the call's entity ids). Scenes, scripts and kinds with no plain on
+   and off are not read back. A read-back that can't run (offline) leaves the result's own answer standing.
+   When a read-back fails, the tool message the model reads ends with "[Sentient checked afterwards and it did not
+   work: <detail>. Tell the user plainly; don't say it worked.]", so the reply never calls it done.
 
 **OutcomeCheck** `{tool, app, kind: "email"|"message"|"file"|"event"|"home"|null, action: "email sent", ok, detail,
 how: "result"|"read_back", targets: [...], missed: [...], settled?, message?}`. `detail` is the plain reason it failed
@@ -980,7 +990,9 @@ that tool, if it worked.
   clean run logs one `info` update: "Checked: file written, email sent." The report writer gets the same line as
   `checked_outcome` and is told to believe it. Try runs and swarm runs are not checked.
 - Chat replies carry the reply's `checks` on `done` and on the saved message; the app shows "Checked: ..." under the
-  reply, or the failures in a Needs your attention box. A failed check also counts in the turn's `tool_errors`
+  reply, or the failures in a Needs your attention box. Quick commands (section 23) are checked the
+  same way: when the read-back fails the reply is "I tried, but Bedroom lamp still reports off." instead of "Turned on
+  the bedroom lamp.". A failed check also counts in the turn's `tool_errors`
   (`chat.turn_completed`), so the self-improvement reviewer sees it.
 
 ### Where results go
@@ -3397,7 +3409,8 @@ sends, buys or deletes anything ("delete the timer" goes to the model). It runs
 asking: with approvals mode `always`, or an "Ask" or "Never" rule on the tool or on `reminders`, the model answers as
 usual. A home command calls `home_assistant_turn_on` or `home_assistant_turn_off` (with `brightness_pct` for a
 level) once for every device it names, and says "Turned off the living room lights." or "Set the bedroom lamp to
-30%.". Its words name exactly that change, so in approvals mode `ask` they count as the yes for it (effective risk
+30%.", or "I tried, but Bedroom lamp still reports off." when reading the device back shows it didn't change
+(section 4, "Checking the outcome", #386). Its words name exactly that change, so in approvals mode `ask` they count as the yes for it (effective risk
 `write` only, ADR 0028); approvals mode `always`, an "Ask", "Hand it to me" or "Never" rule on the tool or on
 `home_assistant`, or a Read only connection send it to the model. Not used for a message with attachments. Turned
 off with `chat.quick_commands`.

sentient/agent/fastpath/base.py

@@ -83,6 +83,11 @@ def listing(self, reminders: list[dict], now: datetime) -> str: ...
 
     def failed(self, error: str) -> str: ...
 
+    def unconfirmed(self, detail: str) -> str:
+        """What to say when the call said it worked but reading it back shows it didn't (#386): ``detail`` is
+        "Hallway light still reports off"."""
+        ...
+
     def home_done(self, command: Command, count: int) -> str:
         """What to say after a home command switched ``count`` devices ("Turned off the living room lights.")."""
         ...

sentient/agent/fastpath/en.py

@@ -874,6 +874,11 @@ def failed(error: str) -> str:
     return f"I couldn't do that: {error}" if error else "I couldn't do that."
 
 
+def unconfirmed(detail: str) -> str:
+    detail = detail.strip().rstrip(".")
+    return f"I tried, but {detail}." if detail else "I tried, but it didn't seem to work."
+
+
 def timer_done(reminder: dict) -> tuple[str, str]:
     label = str(reminder.get("label") or "")
     if label:

sentient/agent/loop.py

@@ -1461,6 +1461,8 @@ async def _record(
         folder = incognito_outputs.outputs_folder(ctx.session_id) if ctx.incognito and ctx.session_id else "outputs"
         content = await self._tool_content(res, tc.id, tool=p.tool, limit=result.tool_result_limit, folder=folder)
         model_content = self.outside_content(content, p.tool, tc.arguments) if p.tool is not None else content
+        if note := outcomes.model_note(check):  # the result said done, reading it back says not (#386)
+            content, model_content = f"{content}\n{note}", f"{model_content}\n{note}"
         messages.append({"role": "tool", "tool_call_id": tc.id, "name": tc.name, "content": model_content})
         if persist:  # stored as it is; marked again when the transcript is read back (mark_outside_messages)
             extra = {"approval": p.approval} if p.approval else {}
@@ -1905,24 +1907,36 @@ async def _quick_turn(
             ctx = self.tool_context(session_id, channel)
             ctx.incognito = bool((session or {}).get("incognito"))
             ctx.extra = {**ctx.extra, "quick": True, "quick_language": plan.language.code}
-            outcomes: list[tuple[Any, bool]] = []
+            results: list[tuple[Any, bool]] = []
+            checks: list[dict] = []
             if plan.calls:
                 await self.store.add_message(session_id, "assistant", None, tool_calls=[c.to_openai() for c in plan.calls])
             for call in plan.calls:
                 yield ToolCallEvent(call_id=call.id, name=call.name, arguments=call.arguments, **ev)
                 res, is_error, ms = await self._run_tool(call, ctx)
-                content = await self._tool_content(res, call.id, tool=self.registry.get(call.name))
+                tool = self.registry.get(call.name)
+                content = await self._tool_content(res, call.id, tool=tool)
+                # what the device really did, read back (#386): a light that stayed off is never "Turned on"
+                check = await self.check_outcome(tool, call.arguments, res, is_error, ctx) if tool is not None else None
+                if check is not None:
+                    checks.append(check)
+                if note := outcomes.model_note(check):
+                    content = f"{content}\n{note}"
                 await self.store.add_message(session_id, "tool", content, tool_call_id=call.id, name=call.name)
-                outcomes.append((res, is_error))
-                yield ToolResultEvent(call_id=call.id, name=call.name, result=res, is_error=is_error, duration_ms=ms, **ev)
-            reply = plan.answer(outcomes)
+                results.append((res, is_error))
+                yield ToolResultEvent(
+                    call_id=call.id, name=call.name, result=res, is_error=is_error, duration_ms=ms, check=check, **ev
+                )
+            missed = [c for c in checks if c.get("ok") is False and c.get("how") == "read_back"]
+            reply = plan.language.unconfirmed(str(missed[0].get("detail") or "")) if missed else plan.answer(results)
+            checks = outcomes.finalize(checks, reply)
             yield TextDelta(text=reply, **ev)
-            message_id = await self.store.add_message(session_id, "assistant", reply)
+            message_id = await self.store.add_message(session_id, "assistant", reply, checks=checks or None)
             log.info(
                 "quick command: %s (%s) done in %d ms with no model call", plan.command.action, plan.language.code,
                 int((time.perf_counter() - started) * 1000),
             )
-            yield Done(content=reply, message_id=message_id, **ev)
+            yield Done(content=reply, message_id=message_id, checks=checks, **ev)
         except Exception as exc:  # every turn still ends with an error and one done (docs/API.md section 1, #334)
             log.exception("quick command %s failed", plan.command.action)
             yield Error(message=plan.language.failed(""), recoverable=False, **ev)

sentient/agent/outcomes.py

@@ -221,6 +221,10 @@ def _targets(kind: str | None, arguments: dict) -> list[str]:
                 text = str(v)
                 found += _ADDRESS.findall(text) or [p.strip() for p in re.split(r"[,;]", text) if p.strip()]
         return list(dict.fromkeys(t.lower() for t in found))
+    if kind == "home" and args.get("entity_id"):  # one id, several separated by commas, or a list
+        value = args["entity_id"]
+        ids = value if isinstance(value, list) else re.split(r"[,\s]+", str(value))
+        return list(dict.fromkeys(str(i).strip().lower() for i in ids if str(i).strip()))
     for key in ("name", "path", "filename", "summary", "title", "text"):
         if isinstance(args.get(key), str) and args[key].strip():
             return [args[key].strip().lower()]
@@ -268,9 +272,22 @@ async def check(tool: Any, arguments: dict, res: Any, is_error: bool, ctx: Any,
         out["how"] = "read_back"
         if not verified.get("ok"):
             out.update(ok=False, detail=_clip(verified.get("detail") or f"{app} doesn't show it"))
+            if isinstance(verified.get("missed"), list):  # which of the things it acted on didn't change
+                out["missed"] = [str(m).lower() for m in verified["missed"]]
+        elif verified.get("detail"):  # fine, with a note ("Garage door is still opening")
+            out["detail"] = _clip(verified["detail"])
     return out
 
 
+def model_note(check: dict | None) -> str:
+    """A line for the model after a call whose result said it worked but whose read-back shows it didn't (#386), so
+    the reply never calls it done; "" otherwise (a failed result is already in the result the model reads)."""
+    if not check or check.get("ok") is not False or check.get("how") != "read_back":
+        return ""
+    detail = str(check.get("detail") or "it did not happen").rstrip(" .")
+    return f"[Sentient checked afterwards and it did not work: {detail}. Tell the user plainly; don't say it worked.]"
+
+
 def claimed(text: str, kind: str | None) -> bool:
     """True when a final answer says the action happened ("I've sent the email")."""
     return bool((_CLAIMS.get(kind or "") or _CLAIM_ANY).search(text or ""))

sentient/config/schema.py

@@ -637,6 +637,12 @@ def _clean_read_only(cls, value: object) -> object:
         description="Listen to Home Assistant for changes (a door opening, a light turning on) so tasks can start "
         "when they happen. Uses no model calls.",
     )
+    home_assistant_check_s: float = Field(
+        4.0, ge=0, le=10,
+        description="After Sentient turns on a light, locks a door or changes another device in Home Assistant, how "
+        "many seconds it waits for the device to report its new state, so it never says done when the device didn't "
+        "change. 0 turns this check off.",
+    )
     fast_sync_seconds: int = Field(
         60, ge=0, le=3600,
         description=(