BQAA Java preview readiness tracker: correctness, lifecycle, and Python contract parity

[caohy1988]

Summary

This is the consolidated readiness tracker for the Java BigQuery Agent Analytics (BQAA) plugin.

Current verdict: not ready to exit Preview or claim Python-compatible Preview behavior. Full Python feature parity is not required, but the P1 items below can produce incorrect analytics, duplicate telemetry, retained per-invocation resources, or incomplete cleanup on supported Java runtime paths.

Latest comparison, pinned on 2026-07-10:

• Java google/adk-java@08f4fdc6c9ea80ec46723b2c6b71a4409c77feed
• Python google/adk-python@9d306f5d322eccbc5022ac72407fa8a994f64cf6

P1: must fix before broad/public Preview

• Make trace state safe for concurrent ParallelAgent branches.

  • Java stores one TraceManager per invocation and one global LIFO stack inside it.
  • ParallelAgent merges concurrently scheduled branches with the same invocation ID. If branch A starts, branch B starts, and A completes first, A pops B's span.
  • Use task-local, branch-keyed, or otherwise ownership-checked trace state.
  • Add a deterministic same-invocation overlap test that completes branches out of start order and validates span IDs, parents, and STARTING/COMPLETED pairing.

• Stop creating plugin-owned OpenTelemetry spans.

  • TraceManager.pushSpan() starts a real OTel span and pop/cleanup ends it.
  • With an SDK exporter configured, BQAA therefore exports a duplicate analytics span tree alongside ADK's framework spans.
  • Match Python's ID-only internal execution tree while inheriting an ambient trace_id when available.
  • Add an in-memory exporter regression test asserting BQAA exports zero plugin-owned spans.

• Apply sensitive-key redaction at the final attributes boundary.

  • Content and session-metadata paths now use the redacting formatter, but the final assembled attributes map is not passed through that boundary.
  • state_delta, custom tags, labels, and other extra attributes can therefore carry sensitive-key values into BigQuery unchanged.
  • Redact the complete attributes tree immediately before serialization, regardless of which producer populated it. Python currently has the same gap, so parity is not a safety defense.
  • Add nested secret-key fixtures for state deltas, custom tags, labels, and generic metadata.

• Fix per-invocation processor/task lifecycle.

  • BatchProcessor.start() discards the ScheduledFuture returned by scheduleWithFixedDelay().
  • Per-invocation close() does not cancel it, so every completed invocation leaves a periodic task retaining its closed processor/writer until plugin-wide shutdown.
  • Store/cancel the task during idempotent close and add a high-cardinality invocation test proving scheduled work/resources are released.

• Make append, final drain, and timeout cleanup genuinely bounded.

  • writer.append(...).get() has no total deadline, and final processor drain runs outside the effective shutdownTimeout bound.
  • Pending parse/offload futures are not cancelled or lifecycle-gated after timeout. A late completion can recreate a BatchProcessor for an invocation that has already been finalized, with no later invocation cleanup.
  • Apply a total append/drain deadline, reject/account appends after invocation close, and test a never-completing append plus a late GCS/parse completion.

• Surface and account writer-construction and continuation failures.

  • StreamWriter creation occurs inside computeIfAbsent; if the future completes exceptionally, the pending task is removed without a durable error signal or drop counter.
  • Rows can therefore disappear before a processor exists, outside the current loss-accounting path.
  • Record a specific drop reason, emit an observable error, and make late/exceptional continuations respect the invocation lifecycle gate.

• Correct HITL and long-running-tool pause/resume semantics.

  • Java currently labels both HITL FunctionCall requests and FunctionResponse results as *_COMPLETED.
  • User-message handling inspects FunctionCall where resumed input arrives as FunctionResponse.
  • Emit pairable TOOL_PAUSED / resumed TOOL_COMPLETED rows with function_call_id and pause_kind, without double-routing HITL completions.
  • Add a Runner-level two-turn test for HITL and non-HITL long-running tools.

• Add terminal agent/run failure analytics and guaranteed cleanup.

  • Java's plugin API exposes model/tool error callbacks but no agent/run error callbacks.
  • afterAgent / afterRun are concatenated after successful completion, so an upstream exception can skip AGENT_ERROR, INVOCATION_ERROR, per-invocation flush, and trace cleanup.
  • Add framework error hooks and finally-style invocation cleanup that preserves the original exception.

Contract gaps: fix or explicitly exclude from a narrow Preview

• Align or version the AGENT_RESPONSE row/view contract.

  • Java emits content.text_summary and exposes text_summary in its view.
  • Python emits content.response and exposes response_text.
  • The same event type should be queryable across SDKs; add a cross-language golden-row test.

• Complete supported workflow taxonomy.

  • Java runtime models transfer, compaction, and long-running tool IDs, but Java BQAA does not emit Python-equivalent AGENT_TRANSFER, EVENT_COMPACTION, and pairable TOOL_PAUSED rows/views.
  • Implement these before a parity claim, or list each omission in Preview scope documentation.

• Add the minimum attributes.adk producer envelope.

  • At minimum stamp schema version, app name, and available source-event identity/branch fields.
  • Python-only node/scope fields can follow when Java exposes equivalent runtime data.

• Publish an executable Java Preview guide and data contract.

  • Cover dependency/setup, ADC and IAM, dataset/table/view defaults, explicit close behavior, drop-stat monitoring, GCS/privacy controls, event taxonomy, row/view contract, and documented Python-parity exclusions.

P2 reliability and operational readiness

• Move first-use table bootstrap off the agent callback thread; coalesce concurrent setup and use bounded retry/backoff after failures.
• Make enabled=false avoid ADC resolution and BigQuery client initialization.
• Add explicit loss accounting for setup-not-ready, writer construction, parse timeout, and rejected late continuations.
• Add a gated real-BigQuery smoke test using public construction, real Storage Write append/drain, and query readback. The current BigQueryAgentAnalyticsPluginE2ETest mocks BigQuery, BigQueryWriteClient, and StreamWriter.
• Decide and document Java/Python default differences (agent_analytics.events, views disabled vs Python agent_events, views enabled).
• Make model, tool, and BigQuery exception-message capture null-safe; Java exceptions may legally return a null message, while the generated event value objects require a non-null optional payload.
• Validate all duration and retry settings at config build time, including positive flush/shutdown durations and valid retry-attempt bounds.

Lower-priority Python gaps that do not independently block a narrow Preview

• Schema-first payload-column projection / denylist.
• Generic custom-metadata allowlisting.
• Full node/scope/route/render/rewind envelope parity.
• Structured LLM tool declaration parity.
• Public flush/view-management helper parity.

These should remain roadmap items unless the Preview explicitly promises full Python query/config parity.

Confirmed fixed on current Java main

• Sensitive-key redaction for content/session-metadata paths and fail-closed custom content formatting.
  • Final assembled attributes remain open under the P1 above and are not included in this completed item.
• Empty state deltas no longer emit STATE_DELTA.
• Timestamp partitioning for newly created tables.
• Failed table ensure is retryable; schema drift is logged.
• Root agent initialization and Event.author fallback.
• Drop-stat folding, regional routing, shutdown hook, headers, and numeric/config validation.

Acceptance tests for Preview readiness

• Same-invocation ParallelAgent overlap preserves branch-local span ownership.
• Recording OTel exporter receives zero BQAA-owned spans.
• Final attributes redaction covers nested secrets in state deltas, custom tags, labels, and generic metadata.
• Many completed invocations leave no periodic flush tasks/processors/writers retained.
• shutdownTimeout bounds a never-completing append and final drain.
• A parse/offload future completing after timeout cannot recreate an invocation processor.
• Writer-construction failures increment a specific drop counter and emit an observable error.
• HITL and ordinary long-running tools produce correctly named and pairable pause/resume rows.
• Agent and invocation exceptions produce terminal rows, preserve the original error, and finalize state exactly once.
• Null-message model/tool/BigQuery exceptions do not break analytics callbacks.
• Invalid flush/shutdown durations and retry settings fail at config construction.
• Java/Python golden fixtures agree on common event names, row fields, content JSON, trace linkage, and view projections.
• Slow/failing first-use table setup does not stall agent progress or trigger one control-plane RPC per event.
• Real BigQuery write/readback smoke test passes in a disposable dataset.

Current verification signal

• Java focused BQAA suite: 135 passed, 0 failed.
• Java full core main profile: 1,591 passed, 0 failed/errors, 24 skipped; additional profiles also passed.
• Python pinned plugin suite: 306 passed, 6 skipped.
• These suites do not currently cover the production shapes listed in the acceptance checklist above.

Pinned references

• Java plugin package: https://github.com/google/adk-java/tree/08f4fdc6c9ea80ec46723b2c6b71a4409c77feed/core/src/main/java/com/google/adk/plugins/agentanalytics
• Java ParallelAgent:

  adk-java/core/src/main/java/com/google/adk/agents/ParallelAgent.java

  Lines 172 to 184 in 08f4fdc

  	protected Flowable<Event> runAsyncImpl(InvocationContext invocationContext) {
  	List<? extends BaseAgent> currentSubAgents = subAgents();
  	if (currentSubAgents == null || currentSubAgents.isEmpty()) {
  	return Flowable.empty();
  	}
  	var updatedInvocationContext = setBranchForCurrentAgent(this, invocationContext);
  	List<Flowable<Event>> agentFlowables = new ArrayList<>();
  	for (BaseAgent subAgent : currentSubAgents) {
  	agentFlowables.add(subAgent.runAsync(updatedInvocationContext).subscribeOn(scheduler));
  	}
  	return Flowable.merge(agentFlowables)
  	.takeUntil((Event event) -> event.actions().escalate().orElse(false));

• Python reference plugin: https://github.com/google/adk-python/blob/9d306f5d322eccbc5022ac72407fa8a994f64cf6/src/google/adk/plugins/bigquery_agent_analytics_plugin.py

Historical investigation and withdrawn/down-scoped findings remain available in the comments below.

[caohy1988]

Additional reviewed findings (write-path reliability, OTel export, scalability)

Updated after direct line-by-line checks against:

• adk-java issue-pinned SHA ce6af87e209e9cc718fb987ef1c01665e0be2432
• adk-python reference SHA 400f512d812a0c92ebee5d13b5e7448516a55eae

Note: Java main has moved since this issue was opened, so Java line references below are against the pinned SHA above.

These are intended to be non-overlapping with the issue body: the body already covers schema/taxonomy/parity items such as redaction, HITL resume, unconditional STATE_DELTA, table partitioning / tableEnsured, defaults, ADK envelope, root_agent_name, and no-current-agent guard.

Correctness / data-loss (P1-class)

• No drop accounting / health signal for lost rows. Python exposes structured counters (queue_full, retry_exhausted, non_retryable, arrow_prep_failed, unexpected_error) through get_drop_stats(). Java only logs queue-full and write failures (BatchProcessor.append, BatchProcessor.flush), so hosts cannot programmatically detect that analytics rows were dropped.

• Plugin-owned real OTel spans can be exported to the user's trace backend. Java TraceManager.pushSpan() starts a real OTel span and popSpan() ends it. Python now records internal span IDs only and explicitly avoids creating OTel spans, because exporter-enabled hosts can otherwise see duplicate analytics spans. This is separate from the already-filed ambient span tree issue: Java should either stop owning OTel spans for BQAA bookkeeping or make export behavior explicit and tested.

• Append/schema failures can lose a drained batch with no counter, salvage, retry, or DLQ. Java drains rows from the queue before append; if AppendRowsResponse.hasError() or AppendSerializationError happens, the batch is logged and discarded. The default batchSize=1 limits blast radius today, but any real batching setting turns one bad row or append failure into multi-row data loss.

Reliability hardening (P2-class)

• Storage Write regional routing is not explicitly set or tested. The control-plane BigQuery client uses setLocation(config.location()), but the Java write path does not call StreamWriter.Builder.setLocation(config.location()) and does not set x-goog-request-params itself. Python explicitly passes x-goog-request-params: write_stream=... for append routing. Add a regional dataset test, and set the StreamWriter location or otherwise prove non-US/EU appends route correctly.

• No JVM shutdown hook / exit-time warning. Java drains only when the host calls close() or an invocation reaches ensureInvocationCompleted(). Python registers atexit cleanup, but note that Python's handler mainly warns/drains queued items rather than performing a full BigQuery flush at interpreter exit. For Java preview, either register a conservative shutdown hook or document/test the required close/flush lifecycle.

• Per-invocation StreamWriter creation is likely too expensive at scale. PluginState.getBatchProcessor(invocationId) creates a new BatchProcessor, StreamWriter, Arrow allocator, and root per invocation. High invocation throughput means many writers and allocators; createWriter() failures surface as VerifyException from inside async append setup.

Scalability / operational polish (P3)

• batchSize=1 default keeps batching effectively disabled. Python currently shares this default, so this is not Java-only, but it is worth deciding before preview whether the default is for correctness/simplicity or whether Java should ship a throughput-oriented preview default.

• Processed-invocation cache edge should be documented or tested. The cache is maximumSize(10000) with expireAfterWrite(10min). This is not a bug for invocations that merely run longer than 10 minutes; the realistic edge is callbacks that arrive more than 10 minutes after completion, or after heavy completed-invocation churn, getting reprocessed.

• JsonFormatter fallback loses observability. IllegalArgumentException in smartTruncate() / convertToJsonNode() falls back to string conversion without logging. This is survivable, but preview users will have little signal that structured metadata was degraded.

• Hot-path INFO logs are chatty. Examples: duplicate processed invocation, pending-task wait/removal, and table ensure. Most of these should likely be FINE / FINER for preview.

• org.threeten.bp and java.time are mixed in the write path. This looks type-contained today, but it adds avoidable conversion risk and confusion in PluginState / retry / shutdown code.

Suggested additional tests

• Drop counters increment on queue-full, retry-exhausted, non-retryable, serialization, and unexpected-error paths.
• Host with an OTel SDK/exporter enabled does not receive duplicate plugin-owned spans, while BigQuery rows still have a consistent span_id / parent_span_id tree.
• Regional dataset append verifies the Storage Write routing path (setLocation or x-goog-request-params) for non-default locations.
• A batch containing a schema-invalid row records data loss and either salvages good rows, requeues retryable rows, or sends failed rows to a documented dead-letter path.
• JVM close/shutdown lifecycle is tested: explicit close() drains, and shutdown-hook / exit-time behavior is documented and covered.
• LLM success and error paths produce consistent span_id / parent_span_id pairing. This is a regression guard, not a confirmed current bug.

Withdrawn or down-scoped after verification

• Withdrawn: onModelErrorCallback wrong parent_span_id. Current Java appears consistent with Python: success captures the parent before pop, while error pops first and then reads the new current span as the popped span's parent.
• Withdrawn: no retry jitter. GAX RetrySettings defaults to jittered retries; the jitter setter is deprecated with "Retries always jitter."
• Withdrawn: latency_ms / Duration serialization bug. Java converts Duration to millis in a Map before JSON conversion, and JsonFormatter uses ObjectMapper.findAndRegisterModules(). A round-trip test is still fine, but this is not a confirmed defect.
• Withdrawn: part_attributes Java STRING vs Python RECORD. Python also defines part_attributes as STRING and stores serialized JSON.
• Down-scoped: aggregate truncation budget. Java truncation is per-field, but Python is also effectively per-field for this path. Track only if preview requires a top-level row-size/depth guard, not as a Java/Python parity defect.
• Down-scoped: dry-run mode. I did not find a Python dry-run mode either; this is an optional onboarding feature, not a parity blocker.

Net-new findings from 2026-07-01 follow-up

• Schema auto-upgrade can early-return on the version label alone. BigQueryUtils.maybeUpgradeSchema() returns immediately when the stored adk_schema_version label equals the constant, before diffing the actual table schema. A table stamped with the current label but missing columns will never be reconciled until the constant changes. Also, schemaFieldsMatch() only handles missing fields and nested STRUCT additions; non-STRUCT type/mode drift is ignored and later surfaces as Storage Write failures. Fix by always diffing actual schema and logging or failing on incompatible drift.

• EventData.java is missing the Apache license header. Every other file in this package carries the Google Apache header; EventData.java starts directly with the package declaration and also has stale Javadoc saying fields are passed to _log_event. Trivial fix, but should be handled before release/preview.

• ensureInvocationCompleted() timeout can close the writer while parse tasks keep running. On timeout, doFinally() flushes/closes the invocation BatchProcessor, but the timed-out CompletableFutures are not cancelled. A late parse future can then finish and call processor.append(row) against a closed StreamWriter path. Either cancel/ignore late tasks after timeout or make appends after close a no-op with accounting.

• View DDL uses unvalidated identifier interpolation. createAnalyticsViews() splices project, dataset, table, and view prefix into CREATE OR REPLACE VIEW using String.format. These are operator-supplied config values and createViews=false by default, so this is low severity, but a backtick or invalid identifier can break or redirect DDL. Validate BigQuery identifiers before building SQL.

• Numeric config values are not validated. batchSize, queueMaxSize, and maxContentLength accept zero/negative values through the builder. Add builder validation for positive queue/batch sizes and documented behavior for maxContentLength.

• Minor polish bucket: sub-millisecond latencies floor to 0 via Duration.toMillis(); cycle sentinel strings differ from Python; table-create conflict/already-exists is noisy; close() can briefly spin if a scheduled flush holds flushLock; deep nesting still has no recursion-depth guard.

• logEvent runs blocking/synchronous work on the agent callback thread, most acutely on the first event. logEvent() executes its body synchronously and returns an already-completed Completable (only content parse + append are async). That synchronous body calls ensureTableExistsOnce() (BigQueryAgentAnalyticsPlugin.java:232), which on the first event of the process performs blocking BigQuery RPCs — getTable(), then create() and/or maybeUpgradeSchema() — on the callback thread, stalling that invocation until table setup finishes. Every event additionally pays synchronous getResolvedTraceIds() and getAttributes(), including smartTruncate(session.state()) (:352) over potentially-large session state, before returning. This is distinct from the tableEnsured race in the issue body (that is retry-after-failure; this is hot-path latency). Suggested fix: run the first-run table ensure off-thread or at construction, and move ID resolution / attribute building / session-state truncation into the async stage. Related: view creation is submitted to the shared 2-thread flush executor (:176), so a slow DDL can occupy a flush thread.

[hemasekhar-p]

Hi @caohy1988, Thank you for creating this issue and for your contribution. To facilitate the review process, could you please complete the pull request and transition it from a draft to open state? This will allow us to begin reviewing the Issue and the associated changes. We appreciate your work on this.