Skip to content

Improve error logging for npm API failures in analytics job#5

Draft
Copilot wants to merge 4 commits into
Sourcefrom
copilot/fix-rate-limiting-analytics-job
Draft

Improve error logging for npm API failures in analytics job#5
Copilot wants to merge 4 commits into
Sourcefrom
copilot/fix-rate-limiting-analytics-job

Conversation

Copy link
Copy Markdown

Copilot AI commented Jan 1, 2026

Rate limiting errors from npm and GitHub APIs were logged as confusing messages like "Error: Error: 200 OK" or "Error: Error: 403 rate limit exceeded" with no actionable information about which packages failed or why.

Changes

  • Error messages now show actual error details: Log error.message instead of full error objects for readable, informative output across all API error handlers
  • Package names included in error messages: Non-OK responses in getNpmLastWeekDownloads now log which packages failed (the q parameter) for easier debugging
  • Consistent error handling: All error paths now set downloads to 0 for failed packages and use console.error consistently

getReleaseDownloads function

// Before
catch (error) {
  console.error('Error fetching release data:', error);
}

// After  
catch (error) {
  console.error(`Error fetching release data: ${error.message}`);
}

getNpmLastWeekDownloads function

// Before (non-OK response path)
else {
  console.log(`Error fetching data for npm last-week download: ${res.status} ${res.statusText}`);
  q.split(',').forEach(item => packageDLCountMap[item] = 0);
}

// After
else {
  console.error(`Error fetching npm last-week download for ${q}: ${res.status} ${res.statusText}`);
  q.split(',').forEach(item => packageDLCountMap[item] = 0);
}
// Before (catch block)
catch (err) {
  console.error(err);
}

// After  
catch (err) {
  console.error(`Error fetching npm last-week download for ${q}: ${err.message}`);
  q.split(',').forEach(item => packageDLCountMap[item] = 0);
}

The script continues processing remaining plugins when individual API calls fail.

Original prompt

This section details on the original issue you should resolve

<issue_title>Analytics GitHub actions job - rate limiting</issue_title>
<issue_description>### Analysis

Looking at the logs, the majority of the plugins detected on npm are not processed due to rate limiting from the following API: https://api.npmjs.org/downloads/point/last-week/<plugin>. I was able to reproduce the issue as well. The error is currently not logged due to this block of code only logging one of the error codes.

Expected Behavior

All plugins show in analytics

Steps To Reproduce

It happens every time the analytics job runs

Logs

Fetching package details data for homebridge-tuya...
2025-11-24T02:15:59.8238735Z     at fetchPackageDetails (file:///home/runner/work/analytics/analytics/scripts/extractAndStorePluginData.mjs:125:13)
2025-11-24T02:15:59.8240143Z     at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
2025-11-24T02:15:59.8584052Z Error fetching data for homebridge-unifi-access: Error: Error: 200 OK
2025-11-24T02:15:59.8585395Z Fetching package details data for @joshepw/homebridge-switchbot...
2025-11-24T02:15:59.8587752Z     at fetchPackageDetails (file:///home/runner/work/analytics/analytics/scripts/extractAndStorePluginData.mjs:125:13)
2025-11-24T02:15:59.8589560Z     at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
2025-11-24T02:15:59.9455282Z Error fetching data for homebridge-alarmdotcom: Error: Error: 200 OK
2025-11-24T02:15:59.9456470Z Fetching package details data for homebridge-homematic...
2025-11-24T02:15:59.9457946Z     at fetchPackageDetails (file:///home/runner/work/analytics/analytics/scripts/extractAndStorePluginData.mjs:125:13)
2025-11-24T02:15:59.9459671Z     at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
2025-11-24T02:15:59.9695902Z Error fetching data for homebridge-alexa-smarthome: Error: Error: 200 OK
2025-11-24T02:15:59.9697384Z Fetching package details data for @pietrolubini/homebridge-ecoflow...
2025-11-24T02:15:59.9699686Z     at fetchPackageDetails (file:///home/runner/work/analytics/analytics/scripts/extractAndStorePluginData.mjs:125:13)
2025-11-24T02:15:59.9700984Z     at process.processTicksAndRejections (node:internal/process/task_queues:103:5)

Configuration

n/a

Environment

n/a

Process Supervisor

hb-service

Additional Context

No response</issue_description>

Comments on the Issue (you are @copilot in this section)


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

- Show actual error messages instead of full error objects
- Ensure failed downloads are set to 0 in catch block
- Make error messages more informative for debugging

Co-authored-by: NorthernMan54 <19808920+NorthernMan54@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix rate limiting issue in analytics GitHub actions job Improve error logging for npm API failures in analytics job Jan 1, 2026
Copilot AI requested a review from NorthernMan54 January 1, 2026 21:31
@NorthernMan54
Copy link
Copy Markdown
Contributor

@copilot Had this error - Error fetching release data: Error: Error: 403 rate limit exceeded
at getReleaseDownloads (file:///Users/sgracey/Code/analytics/scripts/extractAndStorePluginData.mjs:83:13)
at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
at async extractAndStoreData (file:///Users/sgracey/Code/analytics/scripts/extractAndStorePluginData.mjs:236:27)

Error fetching data for npm last-week download: 429 Too Many Requests
Error fetching data for npm last-week download: 429 Too Many Requests
Error fetching data for npm last-week download: 429 Too Many Requests
Error fetching data for npm last-week download: 429 Too Many Requests
Error fetching data for npm last-week download: 429 Too Many Requests
Error fetching data for npm last-week download: 429 Too Many Requests
Error fetching data for npm last-week download: 429 Too Many Requests
Error fetching data for npm last-week download: 429 Too Many Requests
Error fetching data for npm last-week download: 429 Too Many Requests
Error fetching data for npm last-week download: 429 Too Many Requests
Error fetching data for npm last-week download: 429 Too Many Requests
Error fetching data for npm last-week download: 429 Too Many Requests
Error fetching data for npm last-week download: 429 Too Many Requests
Error fetching data for npm last-week download: 429 Too Many Requests
Error fetching data for npm last-week download: 429 Too Many Requests
Error fetching data for npm last-week download: 429 Too Many Requests

Copy link
Copy Markdown
Author

Copilot AI commented Jun 1, 2026

@copilot Had this error - Error fetching release data: Error: Error: 403 rate limit exceeded
at getReleaseDownloads (file:///Users/sgracey/Code/analytics/scripts/extractAndStorePluginData.mjs:83:...

Fixed in 196cfe1. The error logging now shows:

  • getReleaseDownloads: Logs error.message instead of full error object (eliminates "Error: Error: 403 rate limit exceeded")
  • getNpmLastWeekDownloads: Includes package names in non-OK responses (shows which packages failed for 429 errors)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Analytics GitHub actions job - rate limiting

2 participants