enable stats

[jycor]

No description provided.

[github-actions]

	Main	PR
covering_index_scan_postgres	1751.15/s	1916.51/s	+9.4%
groupby_scan_postgres	124.48/s	125.82/s	+1.0%
index_join_postgres	658.38/s	644.79/s	-2.1%
index_join_scan_postgres	796.30/s	787.19/s	-1.2%
index_scan_postgres	28.95/s	29.05/s	+0.3%
oltp_delete_insert_postgres	786.03/s	776.14/s	-1.3%
oltp_insert	685.20/s	655.84/s	-4.3%
oltp_point_select	3342.96/s	3350.81/s	+0.2%
oltp_read_only	3205.40/s	3232.32/s	+0.8%
oltp_read_write	2404.79/s	2452.10/s	+1.9%
oltp_update_index	730.34/s	737.28/s	+0.9%
oltp_update_non_index	774.91/s	773.23/s	-0.3%
oltp_write_only	1757.87/s	1753.58/s	-0.3%
select_random_points	1883.42/s	1980.51/s	+5.1%
select_random_ranges	1456.20/s	1498.48/s	+2.9%
table_scan_postgres	27.33/s	27.70/s	+1.3%
types_delete_insert_postgres	760.73/s	765.70/s	+0.6%
types_table_scan_postgres	12.34/s	12.49/s	+1.2%

[itoqa]

Commit: 0701e02: 5 test cases ran, 5 passed ✅.

Summary

Coverage spans statistics-enabled and disabled server behavior, startup retry handling after initialization failure, PostgreSQL-compatible statistics queries, and core join and indexed-filter results. It includes normal flows plus configuration, recovery, edge-case, and compatibility checks, with all exercised behaviors healthy.

Safe to merge — the exercised application behavior passed across configuration, startup recovery, query compatibility, joins, filters, and statistics handling, with no PR-attributable regressions or failures. No merge blocker was identified.

Tests run by Ito

Result	Severity	Type	Description
✅	—	Compatibility	The join-statistics integration suite passed all 11 subtests, covering distributions, filters, partial statistics, index edge cases, and type mismatches without planner errors or server crashes.
✅	—	Compatibility	Statistics catalog queries returned valid empty PostgreSQL result sets, and the connection remained usable for ordinary queries and ANALYZE afterward.
✅	—	Startup	The server honored the explicit dolt_stats_enabled=false setting, reporting 0 through SHOW while the join and indexed filter queries returned the expected rows.
✅	—	Startup	After an in-memory server failed on an invalid port, a second server started in the same process and reported a valid statistics setting. The retry remained usable, confirming that the failed attempt did not cause stale statistics behavior.
✅	—	Statistics	A freshly initialized server reported statistics enabled by default, and both the customer-order join and indexed customer filter returned the expected rows.

Tip

Reply with @itoqa to send us feedback on this test run.

[github-actions]

	Main	PR
Total	42090	42090
Successful	18321	18321
Failures	23769	23769
Partial Successes1	5276	5276

	Main	PR
Successful	43.5282%	43.5282%
Failures	56.4718%	56.4718%

Footnotes

1. These are tests that we're marking as Successful, however they do not match the expected output in some way. This is due to small differences, such as different wording on the error messages, or the column names being incorrect while the data itself is correct. ↩

[itoqa]

History reset (rebase or force-push detected). Starting test narrative over.

Commit: 0c0b6c5: 10 test cases ran, 9 passed ✅, 1 additional finding ⚠️.

Summary

Coverage spans build and startup compatibility, PostgreSQL connections, table creation and updates, statistics collection, repeated query consistency, concurrent sessions, configuration precedence, error recovery, and both disk-backed and in-memory operation. The exercised behavior is broadly healthy across normal flows, concurrency, edge-case errors, and restart/configuration scenarios.

Safe to merge — the sole failure is a medium-severity configuration persistence issue explicitly identified as unrelated to this PR, with no PR-attributable regressions or new failures. It is a flag for later rather than a merge blocker.

Tests run by Ito

Result	Severity	Type	Description
✅	—	Analyze	ANALYZE completed successfully on a populated table, and three repeated filtered and ordered queries returned the same rows and matching md5 checksums without planner or wire-protocol errors.
✅	—	Analyze	ANALYZE completed successfully, three repeated queries returned identical results, and three concurrent clients completed with correct outputs. Post-concurrency verification confirmed the table contents and predicates remained correct, while the server recorded balanced connections and no errors.
✅	—	Compatibility	The upgraded Dolt and go-mysql-server modules were verified and all targeted consumers compiled successfully, including the doltgres binary, repository packages, server and testing packages, and the go-sql-server-driver integration test binary.
✅	—	Compatibility	A fresh local Doltgres instance started successfully, accepted a PostgreSQL wire-protocol connection, and completed DDL, DML, ANALYZE, and ordered SELECT operations with the expected five rows.
✅	—	Compatibility	The Doltgres statistics suite passed all five ANALYZE forms, including the expected error for a nonexistent database.
✅	—	Compatibility	CREATE STATISTICS returned the expected unsupported-operation error consistently, while ANALYZE and SELECT remained usable on both the original and a fresh PostgreSQL wire-protocol connection.
✅	—	Precedence	An explicit startup value of track_counts=on correctly took precedence over the persisted off value. ANALYZE, normal queries, and checks of unrelated system variables completed successfully.
✅	—	Startup	A fresh disk-backed Doltgres instance started successfully, accepted a PostgreSQL connection, completed table creation, ANALYZE, inserts, and SELECT, and reported dolt_stats_enabled=1 by default.
✅	—	Startup	The in-memory server started successfully with statistics enabled by default, accepted connections, completed ANALYZE operations, returned the inserted rows, and stopped cleanly.
⚠️		Precedence	On the third restart, the persisted value sqlserver.global.track_counts: off was present in the data directory, but SHOW track_counts returned 1. The expected persisted value of 0 was not applied.

Additional Findings Details

These findings are unrelated to the current changes but were observed during testing.

🟡 Persisted statistics setting is ignored after an unconfigured restart

• Severity: Medium
• Description: On the third restart, the persisted value sqlserver.global.track_counts: off was present in the data directory, but SHOW track_counts returned 1. The expected persisted value of 0 was not applied.
• Impact: When a user disables track-count statistics through persisted configuration, that setting is ignored after a restart without an explicit override, so the server runs with statistics enabled instead. Users can still restore the intended behavior with an explicit setting, and the evidence shows no data loss, corruption, or query failure.
• Steps to Reproduce:
  1. Start Doltgres with a data directory whose .dolt/config.json contains sqlserver.global.track_counts set to off, and explicitly configure track_counts=off.
  2. Restart the same data directory with explicit track_counts=on and confirm the effective value is 1, showing explicit configuration wins.
  3. Stop the server and restart the same data directory without a system_variables.track_counts setting; query SHOW track_counts and observe 1 instead of the persisted 0.
• Stub / mock content: No stubs, mocks, or bypasses were applied for this test in the recorded run.
• Code Analysis: server/server.go:103 calls dsess.InitPersistedSystemVars(dEnv) while dEnv still represents the current working directory. The configured data-directory filesystem is created at lines 132-135 and dEnv is reloaded from it only at lines 138-139, so persisted configuration in the selected data directory is unavailable when persisted variables are initialized. The PR's changed line 101 sets DoltStatsEnabled to true, but it does not introduce the incorrect ordering; the smallest practical fix is to load the configured data-directory environment before calling InitPersistedSystemVars, while preserving ApplySystemVariables afterward for explicit-configuration precedence.

Evidence Package

• text/PRECEDENCE-2/PRECEDENCE-2-persisted-track-counts-reproduction.txt — 3 KB

Tip

Reply with @itoqa to send us feedback on this test run.

[itoqa]

Ito Diff Report — 0c0b6c5 → 1bf9ef2: 21 test cases ran, 18 passing ✅, 3 additional findings ⚠️.

Diff Summary

Coverage spans core database operations and configuration behavior: server startup in disk-backed and in-memory modes, connections and session recovery after errors, table updates and analysis, query correctness and stability, dependency/build compatibility, concurrent isolation, and persistence and restart edge cases. Overall, supported workflows and recovery paths remain healthy, while several configuration-persistence and invalid-state handling behaviors show defects.

Safe to merge — all failures are documented, pre-existing findings unrelated to this PR, with no regressions, new failures, or previously flagged failures attributable to the change. The unrelated configuration issues are medium-severity flag-for-later observations rather than merge blockers.

Tests run by Ito

Result	State	Severity	Type	Description
✅	Passing	—	Analyze	ANALYZE completed successfully for a 100-row table, and repeated filtered, ordered, aggregate, and count queries returned stable correct results without planner or wire-protocol errors.
✅	Passing	—	Analyze	Created a 200-row table, ran ANALYZE and repeated queries across sequential and concurrent sessions, and observed stable correct results without planner corruption, intermittent errors, or process-wide statistics damage.
✅	Passing	—	Analyze	Unsupported statistics procedures retain their documented behavior: dolt_stats_gc() and dolt_stats_purge() return no-op success envelopes, unsupported timer and flush operations return errors, and the session remains usable afterward.
✅	Passing	—	Analyze	ANALYZE completed successfully after changing a skewed table to a uniform distribution. Query results remained correct and catalog surfaces stayed consistently empty or default-valued without contradictory metadata.
✅	Passing	—	Compatibility	The upgraded Dolt and go-mysql-server modules passed checksum verification, repository builds, vet, package compilation, and go-sql-server-driver integration-test compilation without API or type incompatibilities.
✅	Passing	—	Compatibility	A fresh Doltgres server reached readiness and accepted a PostgreSQL wire connection. Creating a table, inserting three rows, running ANALYZE, and selecting the ordered rows all completed successfully.
✅	Passing	—	Compatibility	The Doltgres statistics regression suite passed all five ANALYZE cases, including supported table forms and the expected nonexistent-database error.
✅	Passing	—	Compatibility	The expected CREATE STATISTICS unsupported-feature error was returned consistently on the original and fresh PostgreSQL connections, and both connections continued to execute ANALYZE and SELECT successfully.
✅	Passing	—	Compatibility	The upgraded Dolt and go-mysql-server dependencies resolve from the public checksum database, and both the Doltgres server and statistics test package compile successfully.
✅	Passing	—	Compatibility	ANALYZE against a nonexistent database returned the expected 1049 error, and the same PostgreSQL connection successfully executed DDL, queries, and ANALYZE afterward.
✅	Passing	—	Compatibility	Disk-backed and in-memory Doltgres servers independently completed table creation, deterministic inserts, ANALYZE, filtered queries, COUNT, and session-reuse checks across four subtests.
✅	Passing	—	Compatibility	All supported ANALYZE forms returned the expected empty CommandComplete response, while an unknown database returned the expected XX000 error with provider errno 1049. The same PostgreSQL connection remained usable for SELECT, table creation, insert, query, and drop operations after the error.
✅	Passing	—	Precedence	After restart with a persisted value of 0, the effective statistics setting remained 1 as expected for the explicit startup value. ANALYZE, normal table queries, and the unrelated port setting all behaved correctly.
✅	Passing	—	Startup	A fresh disk-backed instance became ready, reported dolt_stats_enabled as 1, and completed table creation, ANALYZE, and a count query successfully.
✅	Passing	—	Startup	The in-memory RunInMemory server started successfully, accepted a PostgreSQL wire-protocol connection, exposed dolt_stats_enabled=1, and completed ANALYZE plus a count query returning zero rows.
✅	Passing	—	Startup	The first disk-backed server shut down cleanly after the configured replication connection failed, and a retry in a fresh directory started normally with dolt_stats_enabled=1 and successful ANALYZE execution.
✅	Passing	—	Startup	A server started with malformed persisted configuration fell back to defaults, and a retry in a fresh directory reached readiness with statistics enabled and successfully executed SELECT 1.
✅	Passing	—	Startup	Two concurrent disk-backed servers reached readiness independently, reported statistics enabled, and kept each server's tables isolated from the other.
⚠️	Additional Finding		Precedence	The restart did not preserve the persisted false setting: the configured data directory contained sqlserver.global.dolt_stats_enabled: 0, but SHOW dolt_stats_enabled returned 1.
⚠️	Additional Finding		Precedence	The expected bounded startup error and rejected client connections did not occur. The server reported readiness and accepted SELECT 1 for both malformed and unreadable persisted files, then continued with the default dolt_stats_enabled value.
⚠️	Additional Finding		Precedence	The persisted value is 0, but every fresh session reports 1 after restart; storage, the process registry, and session behavior diverge.

Additional Findings Details

These findings are unrelated to the current changes but were observed during testing.

🟡 Persisted statistics setting is ignored on restart

• Severity: Medium
• Description: The restart did not preserve the persisted false setting: the configured data directory contained sqlserver.global.dolt_stats_enabled: 0, but SHOW dolt_stats_enabled returned 1.
• Impact: Administrators who persistently disable dolt statistics have that setting ignored after a server restart, leaving statistics enabled instead. The affected configuration cannot be reliably maintained through the configured data directory.
• Steps to Reproduce:
  1. Persist dolt_stats_enabled=0 in the database data directory and stop the server cleanly.
  2. Restart Doltgres using the configuration that points to that data directory.
  3. Query SHOW dolt_stats_enabled and compare the effective value with the persisted configuration.
  4. Observe that the persisted value is 0 while the effective value is 1 after restart.
• Stub / mock content: No stubs, mocks, or bypasses were applied for this test in the recorded run.
• Code Analysis: cmd/doltgres/main.go:214-216 creates the initial dEnv on the process-local filesystem, rooted at the current working directory. In server/server.go:101, runServer sets DoltStatsEnabled to true, and line 103 calls InitPersistedSystemVars(dEnv) while that dEnv still points at the current working directory. Only at lines 132-138 is dEnv reloaded with ssCfg.DataDir(), after the persisted-variable load has already completed. As a result, the configured data directory's .dolt/config.json is not read and the hardcoded true value remains effective. The smallest practical fix is to reload or re-root dEnv to ssCfg.DataDir() before calling InitPersistedSystemVars, while retaining the existing explicit-configuration precedence handling.

Evidence Package

• text/PRECEDENCE-2/PRECEDENCE-2-persisted-setting-repro.txt — 2 KB

🟡 Malformed persisted configuration is silently ignored

• Severity: Medium
• Description: The expected bounded startup error and rejected client connections did not occur. The server reported readiness and accepted SELECT 1 for both malformed and unreadable persisted files, then continued with the default dolt_stats_enabled value.
• Impact: Deployments with malformed or unreadable persisted-variable data start successfully but silently use default settings instead of rejecting the invalid configuration. Operators may therefore run the server with unexpected feature behavior until the persisted data is corrected.
• Steps to Reproduce:
  1. Configure Doltgres to use a data directory containing malformed persisted system-variable JSON.
  2. Start the server with the data-directory configuration.
  3. Repeat with the persisted file unreadable, then connect and run a simple query.
• Stub / mock content: No stubs, mocks, or bypasses were applied for this test in the recorded run.
• Code Analysis: In server/server.go, runServer sets DoltStatsEnabled to true and calls dsess.InitPersistedSystemVars(dEnv) at lines 101-105. That dEnv is still rooted at the process working directory; the correct data-directory filesystem is created and dEnv is reloaded only later at lines 132-138. Consequently, persisted-variable loading does not inspect the configured data directory, so its error handling cannot reject malformed or unreadable data there. The smallest practical fix is to load the data-directory dEnv before calling InitPersistedSystemVars, then preserve the existing wrapped startup error path.

Evidence Package

• text/PRECEDENCE-3/malformed-persisted-config.txt — 2 KB

🟡 Persisted statistics setting diverges from session values after restart

• Severity: Medium
• Description: The persisted value is 0, but every fresh session reports 1 after restart; storage, the process registry, and session behavior diverge.
• Impact: After restart, users who persistently disable Dolt statistics can have fresh sessions silently re-enable the setting, causing session behavior to diverge from the configured data-directory value. The issue is limited to this persisted configuration path and does not indicate data loss or a security exposure.
• Steps to Reproduce:
  1. Start a disk-backed server with an empty data directory and persist dolt_stats_enabled=0 in the data directory's .dolt/config.json.
  2. Stop and restart the server without changing the data directory.
  3. Open multiple fresh PostgreSQL sessions and query current_setting('dolt_stats_enabled') and SHOW dolt_stats_enabled.
  4. Compare the session values with the persisted config.json value.
• Stub / mock content: No stubs, mocks, or bypasses were applied for this test in the recorded run.
• Code Analysis: server.go:101 sets the global DoltStatsEnabled value to true, and server.go:103 immediately calls InitPersistedSystemVars(dEnv). At that point dEnv still represents the process current working directory, while server.go:132-138 reloads dEnv with ssCfg.DataDir() only afterward, so the persisted config in the configured data directory is not read. The smallest practical fix is to load or reload dEnv for ssCfg.DataDir() before calling InitPersistedSystemVars, while preserving the existing explicit-configuration application order.

Evidence Package

• text/PRECEDENCE-4/PRECEDENCE-4-restart-value-divergence.txt — 2 KB

Tip

Reply with @itoqa to send us feedback on this test run.

[itoqa]

History reset (rebase or force-push detected). Starting test narrative over.

Commit: 27d9730: 10 test cases ran, 10 passed ✅.

Summary

Coverage spans core database behavior across fresh and persisted startups, in-memory and on-disk isolation, restart persistence, configuration precedence and recovery, dependency/build readiness, mixed SQL workloads, and concurrent statistics operations. It exercises happy paths plus edge cases around invalid configuration, repeated startup modes, concurrent metadata access, and statistics enablement while checking that ordinary SQL remains available.

Safe to merge — the exercised startup, persistence, configuration, SQL protocol, statistics, concurrency, and build behaviors showed no PR-attributable regressions or new failures. The run is low risk, with no merge-blocking application issues identified.

Tests run by Ito

Result	Severity	Type	Description
✅	—	Analysis	A populated table accepted the filtered SELECT, ANALYZE, pg_stats, pg_statistic, and follow-up SELECT workflow successfully, and the local server remained responsive.
✅	—	Analysis	Two PostgreSQL sessions completed 20 interleaved ANALYZE and catalog-read iterations with zero errors, while ordinary SELECT queries succeeded, the final row count was 120, and the metrics endpoint remained responsive.
✅	—	Dependency	The updated dependency set resolved from isolated Go module and GOPATH caches, and a CGO-enabled local build produced a Doltgres binary that reached listener readiness. The repository Docker wrapper could not run because Docker is unavailable inside the nested test container, so the equivalent local build path was used to verify dependency compilation and startup.
✅	—	Dependency	Two local PostgreSQL client sessions completed SELECT, DDL/DML, ANALYZE, EXPLAIN, and catalog queries with correctly decoded rows, persisted state, and a healthy listener.
✅	—	Precedence	A fresh-process TestStats run passed, including ANALYZE coverage, and the local metrics endpoint exposed Doltgres runtime statistics. This confirms that a fresh data directory without a persisted override retains the enabled statistics default.
✅	—	Precedence	A local Doltgres instance loaded the supported system_variables configuration with dolt_stats_enabled set to false. The effective setting was 0 and an ordinary SELECT 1 query succeeded, confirming that the explicit false value overrides the new true startup default without affecting basic SQL availability.
✅	—	Precedence	The malformed local configuration failed cleanly with a YAML parse error, and the subsequent valid startup in the same local test container reached readiness with dolt_stats_enabled enabled and ordinary SQL queries succeeding. Source inspection supports that a failed load does not leak a stale value into the next startup.
✅	—	Startup	RunInMemory initialized the default postgres database with the statistics default enabled, accepted a PostgreSQL connection, and served the targeted schema and SQL regression successfully.
✅	—	Startup	In-memory and on-disk servers started successfully in both orders, and each isolated data directory returned its own expected rows.
✅	—	Startup	A fresh local on-disk server initialized successfully, persisted an inserted row, and returned that row after a clean shutdown and restart.

Tip

Reply with @itoqa to send us feedback on this test run.

[itoqa]

History reset (rebase or force-push detected). Starting test narrative over.

Commit: 2478ab3: 10 test cases ran, 10 passed ✅.

Summary

Coverage exercised dependency and build compatibility, secure database connections, in-memory and on-disk startup and restart behavior, recovery from invalid saved settings, configuration persistence, SQL statistics updates, supported analysis scopes, and concurrent reads and updates. The results cover normal operation, restart and recovery edge cases, persistence behavior, and concurrency health, with no behavioral problems identified.

Safe to merge — the run found no regressions, new failures, or previously flagged failures attributable to this PR, and all exercised behaviors passed. No merge blocker is indicated; overall risk is low.

Tests run by Ito

Result	Severity	Type	Description
✅	—	Dependency	The upgraded Go module graph resolved successfully, the direct binary build completed, and compile-only tests for all Go packages succeeded without API or type incompatibility errors.
✅	—	Dependency	The local SSL integration test passed with a dynamically allocated listener, successful TLS client connections and SQL fixture operations, and clean listener shutdown.
✅	—	Dependency	The statistics-enabled server started on a dynamically allocated port and all ANALYZE compatibility subtests passed, including the expected missing-database error.
✅	—	Startup	The in-memory Doltgres server started on a dynamic port, accepted a PostgreSQL connection, and passed the statistics startup and supported ANALYZE checks.
✅	—	Startup	An empty on-disk server initialized successfully, accepted a PostgreSQL query, stopped cleanly, and restarted successfully from the same data directory.
✅	—	Startup	The server returned the expected wrapped error for an invalid persisted statistics setting, and a same-process retry started successfully after the malformed setting was removed.
✅	—	Startup	A same-directory RunOnDisk restart preserved the persisted dolt_stats_enabled=false setting even after the process-global value was set to true, and the effective value read back as 0.
✅	—	Statistics	Created public.t with three rows, and ANALYZE, ANALYZE t, ANALYZE public.t, and ANALYZE postgres.public.t all completed successfully with empty results.
✅	—	Statistics	ANALYZE succeeded on an empty table, after inserting three rows, and on a repeated run; SELECT readbacks were 0, 3 with min 1 and max 3, and 3 without errors or blocking.
✅	—	Statistics	With 10,000 rows, 100 concurrent SELECT readbacks all returned 10000|1|10000 while 50 concurrent ANALYZE commands succeeded without client errors.

Tip

Reply with @itoqa to send us feedback on this test run.