Address Dexcom and Abbott data ingestion issues by darinkrauss · Pull Request #958 · tidepool-org/platform

darinkrauss · 2026-06-13T05:49:55Z

Address Dexcom and Abbott data ingestion issues
Retry Dexcom API calls with exponential fallback during task run
Do not specify last sync time for Dexcom data range API, per Dexcom
Handle oddities in Dexcom API response
Drain and close HTTP response bodies
Allow refresh of provider session, meaning particular to provider
Do not persist expired token to allow retry refresh
Increase summary update running maximum duration to four minutes
Use platform parser for token data
Redact provider session for logging
Refactor UnstickTasks to report tasks unstuck
Allow clearing error from data source and task
Handle error marshaling failures due to invalid Meta field
Add debug logging for unusual data source and provider session conditions
Fix issues with provenance collection
Rename structure NotParsed to ReportNotParsed
Add NotParsed to return all not parsed fields
Add total_insulin tool
Various minor updates
Add and update tests
https://tidepool.atlassian.net/browse/BACK-3927
https://tidepool.atlassian.net/browse/BACK-3613
https://tidepool.atlassian.net/browse/BACK-3973
https://tidepool.atlassian.net/browse/BACK-3964

Copilot

Pull request overview

This PR focuses on improving reliability and correctness of third-party data ingestion (notably Dexcom, plus some Abbott-related dependency updates) and tightening request/response handling across services. It introduces a generic retry mechanism, improves parsing/“not parsed” reporting, adds provider-session refresh support, and addresses a number of operational issues such as draining HTTP response bodies.

Changes:

Add a reusable request retrier with exponential backoff and wire it into the Dexcom client; adjust Dexcom fetch logic to match Dexcom guidance (e.g., omit lastSyncTime) and handle response oddities.
Refactor parsing utilities to return unparsed fields and separate “report not parsed” behavior; improve error serialization robustness (e.g., marshal failures due to invalid Meta).
Add provider-session refresh capability (API + client + provider hook), improve provenance collection utilities, and standardize draining/closing HTTP response bodies.

Reviewed changes

Copilot reviewed 83 out of 87 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
work/store/structured/mongo/mongo.go	Logging/error-handling cleanups around Mongo operations.
work/service/coordinator.go	Adjust error message for invalid process result.
test/rest/rest.go	Add Response interface + go:generate for mocks.
test/rest/rest_mocks.go	New generated gomock for Response.
task/task.go	Use Serializable.HasError() when checking task error presence.
task/store/store.go	Change UnstickTasks to return task list instead of count.
task/store/mongo/mongo.go	Refactor UnstickTasks implementation and type-filter selectors.
task/service/service/service.go	Inject Dexcom retrier into Dexcom client construction.
task/queue/queue.go	Update unstick/dispatch/complete logging and dispatch flow.
summary/task/updaterunner.go	Increase max summary update duration to 4 minutes.
structure/parser/object_parser.go	Switch to `any`, add NotParsed map + ReportNotParsed behavior.
structure/parser/object_parser_test.go	Update tests for NotParsed/ReportNotParsed semantics and `any`.
structure/parser/array_parser.go	Switch to `any`, add NotParsed map + ReportNotParsed behavior.
structure/parser/array_parser_test.go	Update tests for NotParsed/ReportNotParsed semantics and `any`.
structure/parser.go	Update parser interfaces for `any` + NotParsed/ReportNotParsed split.
services/tools/total_insulin/total_insulin.go	Add CLI tool to compute total insulin totals by type/subtype.
service/middleware/auth.go	Update token hashing calls to new crypto hash signatures.
request/values_parser.go	Switch to `any`, add NotParsed map + ReportNotParsed behavior.
request/retrier.go	New generic retrier with exponential backoff + jitter helpers.
request/retrier_test.go	Tests for retrier behavior (success/failure/backoff/jitter/logging).
request/request.go	Add DrainAndClose helper for safely draining/clsoing bodies.
request/parser.go	Use DrainAndClose on responses; switch to ReportNotParsed.
provider/test/provider.go	Add provider test hook for OnRefresh.
provider/provider.go	Extend provider interface with OnRefresh.
provider/debug.go	Add debugger for data sources/provider sessions with redaction.
plugin/abbott/go.sum	Add go.sum entries for new Abbott plugin dependency.
plugin/abbott/go.mod	Add indirect dependency github.com/gowebpki/jcs.
oura/webhook/webhook.go	Use fmt.Appendf to avoid string formatting allocation patterns.
oura/service/api/v1/event.go	Comment tweak for provider sessions pluralization.
oura/provider/config_test.go	Remove AuthStyleInParams assignment in test setup.
oauth/work/mixin.go	Avoid persisting expired token to allow future refresh retries.
oauth/work/mixin_test.go	Update tests to match new ExpireToken behavior (no persistence).
oauth/test/oauth_mocks.go	Extend mock provider with OnRefresh.
oauth/provider/provider.go	Add no-op OnRefresh; use fmt.Appendf in state hashing.
errors/test/errors.go	Use math/rand/v2; switch helper maps/meta to `any`.
errors/errors.go	Add errors.As/LastCause, Serializable.HasError, safer marshal handling, `any` conversions.
errors/errors_test.go	Update tests for NewSerializable(nil) behavior and serialized outputs.
dexcom/provider/provider.go	Add provider OnRefresh implementation to trigger task availability.
dexcom/fetch/runner.go	Adjust Dexcom range fetch and unauth retry cause detection; avoid persisting expired tokens.
dexcom/fetch/runner_test.go	Update expectations for data range behavior and call counts.
dexcom/event.go	Handle Dexcom health/value oddities; default missing blood glucose unit.
dexcom/event_test.go	Update/extend tests for new Dexcom event validation behavior.
dexcom/dexcom_test.go	Naming cleanup for transmitter ID constant.
dexcom/client/client.go	Add retrier support + request duration threshold constant; retry Dexcom API requests.
dexcom/client/client_test.go	Update constructor tests for new retrier argument.
data/source/test/source.go	Use crypto test helper for md5 hashes.
data/source/store/structured/mongo/mongo.go	Use Serializable.HasError for error persistence decisions.
data/source/source.go	Always retain parsed Serializable (even if nil error) when error field exists.
data/source/source_test.go	Update expected parsed Error values under new Serializable semantics.
data/service/api/v1/v1.go	Add provenance extraction helper + improved XFF selection helper.
data/service/api/v1/v1_test.go	Add tests for provenance/XFF selection helpers.
data/service/api/v1/users_datasets_create.go	Switch to `any` and use ReportNotParsed + new provenance helper.
data/service/api/v1/summary.go	Drain/close response bodies on certain error paths; rename clinicID variable.
data/service/api/v1/summary_test.go	New test file for realtime request query parsing.
data/service/api/v1/datasets_data_create.go	Switch to `any`, use new provenance helper, import cleanup.
data/service/api/v1/datasets_data_create_provenance_test.go	Remove old provenance collection tests (replaced by v1_test.go).
data/raw/store/structured/mongo/mongo.go	Logging cleanup; fix some error returns and post-update re-fetch.
data/raw/store/structured/mongo/mongo_test.go	Add missing-context and invalid-document tests for raw store APIs.
data/data_set.go	Use ParseDataSetClient helper instead of manual parsing.
crypto/test/crypto.go	Update hashing helpers to pass bytes to crypto hash APIs.
crypto/crypto.go	Change hex hash APIs to accept bytes to avoid implicit conversions.
crypto/crypto_test.go	Update tests for new byte-based crypto hash APIs.
consent/loader/loader_test.go	Remove unnecessary `.Times(1)` expectations.
clinics/test/clinics.go	Refactor imports/types; add RandomClinicID helper.
client/client.go	Replace local drainAndClose with request.DrainAndClose helper.
blob/service/api/v1/v1.go	DrainAndClose blob content bodies.
blob/client/client.go	DrainAndClose bodies on header parse errors.
auth/token.go	Add OAuthToken.Redacted for logging-safe token representation.
auth/test/provider_session.go	Add ProviderSessionRefresh test helpers + object builders.
auth/test/provider_session_client.go	Add RefreshProviderSession test client support.
auth/test/auth.go	Rename New* helpers to Random* (keep deprecated wrappers).
auth/test/auth_mocks.go	Add RefreshProviderSession method to auth client mock.
auth/service/test/service_mocks.go	New generated gomock for auth service interfaces.
auth/service/service/client.go	Add RefreshProviderSession service method calling provider OnRefresh.
auth/service/service.go	Add go:generate for service mocks.
auth/service/api/v1/provider_session.go	Add POST /refresh endpoint for provider sessions.
auth/service/api/v1/provider_session_test.go	Add tests for provider session refresh endpoint behavior.
auth/providersession/test/provider_session_mocks.go	Add RefreshProviderSession to provider session client mock.
auth/provider_session.go	Add ProviderSessionRefresh model + ProviderSession.Redacted/ProviderSessions.Redacted.
auth/provider_session_test.go	Add parse/validate/serialize tests for ProviderSessionRefresh.
auth/client/external.go	Parse token validation response via TokenData using structure parser; DrainAndClose serverlogin responses.
auth/client/client.go	Add client method for RefreshProviderSession endpoint.
auth/auth_test.go	Fix nil context test to use context.Context(nil).
appvalidate/palm_tree_secrets.go	DrainAndClose partner API response body.
appvalidate/coastal_secrets.go	DrainAndClose partner API response body.

Files not reviewed (3)

auth/providersession/test/provider_session_mocks.go: Generated file
auth/service/test/service_mocks.go: Generated file
auth/test/auth_mocks.go: Generated file

Comments suppressed due to low confidence (1)

data/service/api/v1/summary.go:129

In the clinics.ErrorCodeClinicClientFailure branch, res may be nil (or the meta may not be a *http.Response), but the code unconditionally uses res.StatusCode/res.Body, which can panic. Guard the Reader call and fall back to InternalServerError when the response meta is missing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 99 out of 103 changed files in this pull request and generated 4 comments.

Files not reviewed (3)

auth/providersession/test/provider_session_mocks.go: Generated file
auth/service/test/service_mocks.go: Generated file
auth/test/auth_mocks.go: Generated file

Copilot

Pull request overview

Copilot reviewed 99 out of 103 changed files in this pull request and generated 2 comments.

Files not reviewed (3)

auth/providersession/test/provider_session_mocks.go: Generated file
auth/service/test/service_mocks.go: Generated file
auth/test/auth_mocks.go: Generated file

toddkazakov

LGTM

The base branch was changed.

coderabbitai · 2026-08-04T21:50:35Z

Important

Review skipped

Too many files!

This PR contains 186 files, which is 86 over the limit of 100.

To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch.

Upgrade to a paid plan to raise the limit.

This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: a3672141-7c66-4e9a-b7ba-44b9f82d8e38

📥 Commits

Reviewing files that changed from the base of the PR and between 0880a3a and b119d40.

⛔ Files ignored due to path filters (2)

go.sum is excluded by !**/*.sum
plugin/abbott/go.sum is excluded by !**/*.sum

📒 Files selected for processing (186)

.travis.yml
Makefile
README.md
appvalidate/coastal_secrets.go
appvalidate/palm_tree_secrets.go
auth/client/client.go
auth/client/client_test.go
auth/client/external.go
auth/provider_session.go
auth/provider_session_test.go
auth/providersession/test/provider_session_mocks.go
auth/providersession/work/mixin.go
auth/service/api/v1/metrics.go
auth/service/api/v1/oauth.go
auth/service/api/v1/provider_session.go
auth/service/api/v1/provider_session_test.go
auth/service/api/v1/router.go
auth/service/service.go
auth/service/service/client.go
auth/service/test/service_mocks.go
auth/test/auth.go
auth/test/auth_mocks.go
auth/test/provider_session.go
auth/test/provider_session_client.go
auth/token.go
blob/client/client.go
blob/service/api/v1/v1.go
client/client.go
client/client_test.go
client/config.go
client/prometheus.go
client/prometheus_test.go
client/round_tripper.go
client/round_tripper_test.go
clinics/test/clinics.go
consent/loader/loader_test.go
consent/store/mongo/consent_record_repository.go
consent/store/mongo/consent_repository.go
crypto/crypto.go
crypto/crypto_test.go
crypto/test/crypto.go
data/data_set.go
data/raw/store/structured/mongo/mongo.go
data/raw/store/structured/mongo/mongo_test.go
data/raw/work/mixin.go
data/service/api/v1/datasets_data_create.go
data/service/api/v1/datasets_data_create_provenance_test.go
data/service/api/v1/metrics.go
data/service/api/v1/summary.go
data/service/api/v1/summary_test.go
data/service/api/v1/users_datasets_create.go
data/service/api/v1/v1.go
data/service/api/v1/v1_test.go
data/service/service/standard.go
data/set/work/mixin.go
data/source/source.go
data/source/source_test.go
data/source/store/structured/mongo/mongo.go
data/source/store/structured/mongo/mongo_test.go
data/source/test/source.go
data/source/work/mixin.go
data/store/mongo/mongo_datum.go
data/store/mongo/mongo_test.go
data/work/mixin.go
dexcom/alert.go
dexcom/calibration.go
dexcom/client/client.go
dexcom/client/client_test.go
dexcom/data_range.go
dexcom/data_range_test.go
dexcom/device.go
dexcom/dexcom_test.go
dexcom/egv.go
dexcom/event.go
dexcom/event_test.go
dexcom/fetch/runner.go
dexcom/fetch/runner_test.go
dexcom/moment.go
dexcom/moment_test.go
dexcom/provider/provider.go
dexcom/provider/provider_test.go
ehr/reconcile/runner.go
ehr/reconcile/runner_test.go
ehr/reconcile/task.go
ehr/reconcile/task_test.go
ehr/sync/runner.go
ehr/sync/task.go
ehr/sync/task_test.go
env.sh
env.test.sh
errors/errors.go
errors/errors_test.go
errors/test/errors.go
go.mod
log/test/serializer.go
oauth/client/client.go
oauth/client/client_test.go
oauth/provider/client/client.go
oauth/provider/provider.go
oauth/test/oauth_mocks.go
oauth/test/token_source.go
oauth/test/token_source_source.go
oauth/work/mixin.go
oauth/work/mixin_test.go
oura/client/client.go
oura/client/client_test.go
oura/oura.go
oura/oura_test.go
oura/provider/config_test.go
oura/provider/provider.go
oura/provider/provider_test.go
oura/service/api/v1/event.go
oura/webhook/webhook.go
plugin/abbott/go.mod
pointer/default.go
pointer/default_test.go
prescription/store/mongo/prescription_repository.go
private/plugin/abbott
prometheus/test/prometheus.go
provider/debug.go
provider/factory/factory.go
provider/provider.go
provider/test/factory.go
provider/test/provider.go
request/condition.go
request/inspector.go
request/parser.go
request/request.go
request/retrier.go
request/retrier_test.go
request/values_parser.go
service/middleware/auth.go
services/tools/dexcom_analyze/dexcom_analyze.go
services/tools/total_insulin/total_insulin.go
store/structured/condition.go
store/structured/mongo/config.go
store/structured/mongo/result.go
store/structured/mongo/result_test.go
store/structured/mongo/test/config.go
structure/parser.go
structure/parser/array_parser.go
structure/parser/array_parser_test.go
structure/parser/object_parser.go
structure/parser/object_parser_test.go
summary/fetcher/data.go
summary/fetcher/fetcher.go
summary/reporters/realtime.go
summary/store/bgm_test.go
summary/store/cgm_test.go
summary/store/continuous_test.go
summary/store/summary.go
summary/summary.go
summary/task/migrationrunner.go
summary/task/migrationrunner_test.go
summary/task/updaterunner.go
summary/task/updaterunner_test.go
summary/test/summary_mocks.go
task/client/client.go
task/queue/multi.go
task/queue/multi_test.go
task/queue/queue.go
task/queue/queue_internal_test.go
task/queue/queue_test.go
task/queue/runner.go
task/queue/test/runner.go
task/service/api/v1/v1.go
task/service/service/client.go
task/service/service/service.go
task/store/mongo/mongo.go
task/store/mongo/mongo_test.go
task/store/store.go
task/store/test/task_session.go
task/task.go
task/task_test.go
task/test/client.go
task/test/task.go
task/test/task_accessor.go
task/test/task_mocks.go
test/http/http.go
test/rest/rest.go
test/rest/rest_mocks.go
test/test.go
test/time.go
work/base/processor.go
work/service/coordinator.go
work/store/structured/mongo/mongo.go

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

- Address Dexcom and Abbott data ingestion issues - Retry Dexcom API calls with exponential fallback during task run - Do not specify last sync time for Dexcom data range API, per Dexcom - Handle oddities in Dexcom API response - Drain and close HTTP response bodies - Allow refresh of provider session, meaning particular to provider - Do not persist expired token to allow retry refresh - Increase summary update running maximum duration to four minutes - Use platform parser for token data - Redact provider session for logging - Refactor UnstickTasks to report tasks unstuck - Allow clearing error from data source and task - Update Dexcom task runner to correctly update last import time - Update Dexcom task runner to correctly reschedule - Handle error marshaling failures due to invalid Meta field - Add debug logging for unusual data source and provider session conditions - Fix issues with provenance collection - Rename structure NotParsed to ReportNotParsed - Add NotParsed to return all not parsed fields - Add checks for task claims that have been lost - Add total_insulin tool - Various minor updates - Add and update tests - Add and update comments - https://tidepool.atlassian.net/browse/BACK-3927 - https://tidepool.atlassian.net/browse/BACK-3613 - https://tidepool.atlassian.net/browse/BACK-3973 - https://tidepool.atlassian.net/browse/BACK-3964

darinkrauss requested a review from Copilot June 13, 2026 05:49

Copilot started reviewing on behalf of darinkrauss June 13, 2026 05:50 View session

Copilot AI reviewed Jun 13, 2026

View reviewed changes

Comment thread provider/debug.go Outdated

Comment thread auth/service/service/client.go Outdated

Comment thread task/store/mongo/mongo.go Outdated

Comment thread task/store/mongo/mongo.go Outdated

Comment thread errors/errors.go

darinkrauss force-pushed the darin/cloud-data-fixes branch 2 times, most recently from f7275ad to 46f0e9d Compare June 14, 2026 00:08

darinkrauss requested a review from Copilot June 14, 2026 00:09

Copilot started reviewing on behalf of darinkrauss June 14, 2026 00:09 View session

Copilot AI reviewed Jun 14, 2026

View reviewed changes

Comment thread dexcom/provider/provider.go

Comment thread data/service/api/v1/v1.go Outdated

Comment thread auth/client/external.go Outdated

Comment thread store/structured/mongo/result.go

darinkrauss force-pushed the darin/cloud-data-fixes branch from 46f0e9d to e17e01e Compare June 14, 2026 17:25

darinkrauss requested a review from Copilot June 14, 2026 17:25

Copilot started reviewing on behalf of darinkrauss June 14, 2026 17:25 View session

Copilot AI reviewed Jun 14, 2026

View reviewed changes

Comment thread store/structured/mongo/result.go

Comment thread request/parser.go

darinkrauss requested a review from toddkazakov June 14, 2026 17:37

toddkazakov previously approved these changes Jun 26, 2026

View reviewed changes

Comment thread dexcom/event.go

Comment thread auth/token.go Outdated

Comment thread dexcom/event.go

Base automatically changed from BACK-3960-oura-connection to master July 1, 2026 22:25

darinkrauss force-pushed the darin/cloud-data-fixes branch from e17e01e to a5cbbca Compare July 1, 2026 22:31

toddkazakov previously approved these changes Jul 14, 2026

View reviewed changes

darinkrauss dismissed toddkazakov’s stale review via 9a77b84 August 4, 2026 21:50

darinkrauss force-pushed the darin/cloud-data-fixes branch from a5cbbca to 9a77b84 Compare August 4, 2026 21:50

darinkrauss force-pushed the darin/cloud-data-fixes branch 2 times, most recently from 4c7e742 to 8f2fcf3 Compare August 6, 2026 04:22

darinkrauss force-pushed the darin/cloud-data-fixes branch from 8f2fcf3 to b119d40 Compare August 7, 2026 15:34

darinkrauss changed the base branch from master to darin/dexcom-slowdown August 7, 2026 16:05

Navigation Menu

Uh oh!

Uh oh!

Address Dexcom and Abbott data ingestion issues - #958

Address Dexcom and Abbott data ingestion issues#958
darinkrauss wants to merge 1 commit into
darin/dexcom-slowdownfrom
darin/cloud-data-fixes

darinkrauss commented Jun 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

toddkazakov left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot commented Aug 4, 2026 •

edited

Loading

Review skipped

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

darinkrauss commented Jun 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

toddkazakov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

coderabbitai Bot commented Aug 4, 2026 •

edited

Loading