Repository navigation
DUM bucket map YAML lacks effective configuration management, causing weblog uploads to fail with HTTP 500 #410
Description
Checked for duplicates
Yes - I've already checked
🐛 Describe the bug
A data provider uploading Apache weblog files (7 .gz files, 1 batch) with pds-ingress-client consistently fails at the "Requesting Batch 1" step with:
Ingress request failed with HTTP status 500 (Internal Server Error)
The ingestion service encountered an internal error and could not process this request.
Please contact PDS Engineering for assistance.
Authentication succeeds, file indexing and checksum manifest generation complete (7/7 files, 1/1 batches), and the failure occurs before any file is uploaded. The failure reproduced on client v2.5.8 and again on v2.6.3 after upgrading, and on a later retry, so it does not appear to be a one-off transient failure.
The client error message does not include enough detail to determine the root cause (client-side vs. service-side, network vs. Lambda error).
🕵️ Expected behavior
- The ingress request for the batch succeeds and files are uploaded, or
- If the service fails, the client reports an actionable error (e.g. response body / request ID / Lambda error detail) and logs enough diagnostic information to identify the cause.
📜 To Reproduce
- Install
pds-data-upload-managerv2.6.3 - Run:
pds-ingress-client -c <config path> -n <node ID> -w apache -p <weblog dir> -r <report path>.json <weblog dir>/access_log.26090[3-9]*.gz - Observe authentication succeeds, manifest generation completes, then the batch request fails with HTTP 500
🖥 Environment Info
- Version of this software: v2.6.3 (also reproduced on v2.5.8)
- Operating System: Linux
📚 Version of Software Used
pds-data-upload-manager v2.6.3
🩺 Test Data / Additional context
- Uses the weblog ingress mode (
-w apache) with 7 gzipped access log files, single batch (batch size 100) - Reported by a data provider on 2026-11-06; the provider retried and got the same failure
- Next step is to inspect the Ingress Lambda CloudWatch logs for the failed request time windows
🦄 Related requirements
N/A
For Internal Dev Team To Complete
⚙️ Engineering Details
🎉 Integration & Test
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
None yet
Effort
None yet
Product
Data Upload Manager
Work Area
Core Data Services
Start date
None yet
Target date
None yet
Projects
- Status
- Status
Activity
jordanpadams commentedon Nov 7, 2026
Investigation findings
What the logs show
Searching the ingress Lambda CloudWatch logs for the failed request windows shows that both the staging and archive bucket checks return 404 Not Found (
head_bucket).process_ingress_requestthen raises aRuntimeError,lambda_handlerfails fast, and API Gateway returns the 500 the client reports. The same sequence appears for both of the reporter's attempts.This is not a permissions failure. A permissions problem would log
exists but access is forbidden (403). The buckets named in the bucket map do not exist in the account and region of the Lambda.What it is not
-w apache(short form of--weblogs) together with-p <prefix>, which is the required combination. The client therefore sent destination paths of the formweblogs/<node>-apache/..., and all seven files passed indexing, gzip validation and manifest generation before the batch request failed.Likely root cause: bucket map configuration
The bucket map in the deployed Lambda (
config/bucket-map.yaml) is generated by Terraform fromsrc/pds/ingress/service/config/bucket-map.yaml.tmpl, with thevenuevariable substituted into bucket names.In the template:
pds-<node>-staging-<venue>andpds-<node>-archive-<venue>.paths:override routingweblogs/*to the shared web analytics bucket.paths:entries.So a NAIF weblog path does not match any override and falls through to NAIF's regular staging and archive buckets. In this deployment those two buckets do not exist, hence the 404s.
Still to confirm
config/bucket-map.yaml).venuevalue are wrong. If they succeed, only the weblog routing is missing.weblogs/*routing should be added for the other nodes as well as SBN.Related improvements (#411)
[-]Ingress client fails with HTTP 500 when requesting batch for Apache weblog upload[/-][+]DUM bucket map YAML lacks effective configuration management, causing weblog uploads to fail with HTTP 500[/+]jordanpadams commentedon Nov 7, 2026
Resolution
Root cause: a configuration management gap. The bucket map YAML (
config/bucket-map.yamlin the ingress Lambda) was lost in the most recent deployment. The correct configuration, including weblog routing, was in the deployment documentation, but restoring it was missed in the deployment process. With the entries gone, weblog paths fell through to node buckets that do not exist, so the Lambda failed its bucket check and returned an HTTP 500.Remediation: the bucket map was restored by hand with weblog routing for every node. It passes the repo's Yamale schema, and each node's
weblogs/*path resolves to the web analytics bucket for both staging and archive. This was a manual edit, so it is replaced the next time the service is deployed unless the deployment process applies it.Not yet verified: an end-to-end weblog upload against the restored configuration has not been completed. Both attempts to run the smoke test stopped on client INI config problems before reaching the service.
Follow-ups
Closing for now. Reopen if the end-to-end check fails.
jordanpadams commentedon Nov 7, 2026
Update: verified in production
The weblog smoke test (added in #411) has been run in production and verified. Weblog ingress works end to end with the restored bucket map, which resolves the "Not yet verified" item in the resolution comment above.