-
-
Notifications
You must be signed in to change notification settings - Fork 8
Ensure functional business domain scraping and connection to system prompt and reports #747
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 1 commit
88bb3b0
1c4b397
4620463
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -85,6 +85,12 @@ export async function getCrossSessionContext(userId?: string): Promise<string> { | |
|
|
||
| export async function generateReportContext(messages: AIMessage[]) { | ||
| try { | ||
| const userId = await getCurrentUserIdOnServer() | ||
| const [systemPrompt, domain] = await Promise.all([ | ||
| userId ? getSystemPrompt(userId) : null, | ||
| userId ? getUserDomain(userId) : null | ||
| ]) | ||
|
|
||
| const crossSessionContext = await getCrossSessionContext() | ||
|
|
||
| const activeMessages = messages | ||
|
|
@@ -109,8 +115,8 @@ export async function generateReportContext(messages: AIMessage[]) { | |
| const strategicContent = activeMessages.filter(msg => msg.role === 'assistant') | ||
|
|
||
| const [execSummary, strategicSynthesis] = await Promise.all([ | ||
| executiveSummaryAgent(crossSessionContext, activeMessages), | ||
| strategicSynthesisAgent(sensorFusionFindings, strategicContent) | ||
| executiveSummaryAgent(crossSessionContext, activeMessages, systemPrompt, domain), | ||
| strategicSynthesisAgent(sensorFusionFindings, strategicContent, systemPrompt, domain) | ||
| ]) | ||
|
|
||
| return { | ||
|
|
@@ -156,6 +162,54 @@ export async function getChat(id: string, userId: string): Promise<DrizzleChat | | |
| } | ||
| } | ||
|
|
||
| export async function getUserDomain( | ||
| userId: string | ||
| ): Promise<string | null> { | ||
| if (!userId) return null | ||
|
|
||
| try { | ||
| const result = await db.select({ metadata: users.metadata }) | ||
| .from(users) | ||
| .where(eq(users.id, userId)) | ||
| .limit(1); | ||
|
|
||
| const metadata = result[0]?.metadata as { domain?: string } | null; | ||
| return metadata?.domain || null; | ||
| } catch (error) { | ||
| console.error('getUserDomain: Error:', error) | ||
| return null | ||
| } | ||
| } | ||
|
|
||
| export async function saveUserDomain( | ||
| userId: string, | ||
| domain: string | ||
| ): Promise<{ success?: boolean; error?: string }> { | ||
| if (!userId) return { error: 'User ID is required' } | ||
|
|
||
| try { | ||
| const result = await db.select({ metadata: users.metadata }) | ||
| .from(users) | ||
| .where(eq(users.id, userId)) | ||
| .limit(1); | ||
|
|
||
| const currentMetadata = (result[0]?.metadata || {}) as Record<string, any>; | ||
| await db.update(users) | ||
| .set({ | ||
| metadata: { | ||
| ...currentMetadata, | ||
| domain: domain || null | ||
| } | ||
| }) | ||
| .where(eq(users.id, userId)); | ||
|
|
||
| return { success: true } | ||
| } catch (error) { | ||
| console.error('saveUserDomain: Error:', error) | ||
| return { error: 'Failed to save user domain' } | ||
| } | ||
| } | ||
|
Comment on lines
+165
to
+211
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "Tracked files around chat.ts:"
git ls-files | grep -E '(^|/)chat\.ts$|auth|get.*currentUser|session' | head -200
echo
echo "Search getUserDomain/saveUserDomain usage and currentUserId helpers:"
rg -n "getUserDomain|saveUserDomain|getCurrentUserIdOnServer|currentUserId|getCurrentUser|auth" -S . --glob '!node_modules' --glob '!dist' --glob '!build' | head -300
echo
echo "Inspect lib/actions/chat.ts relevant section:"
wc -l lib/actions/chat.ts
sed -n '1,260p' lib/actions/chat.tsRepository: QueueLab/QCX Length of output: 50368 🌐 Web query:
💡 Result: There is no single "Nova Framework" that natively provides all the functions and file paths referenced in your query as a unified system. Instead, your query combines components from three distinct contexts: 1. Authentication and User IDs: The function getCurrentUserIdOnServer is commonly found in Next.js-based applications (such as the QueueLab/QCX project) to retrieve the authenticated user's ID within Server Actions or API routes [1][2][3][4]. 2. Nova Framework (Erlang/Elixir): The "Nova Framework" is a real web framework for the Erlang ecosystem [5]. It uses a library called nova_auth for authentication [6][7]. It does not use lib/actions/chat.ts (a TypeScript/Next.js pattern) [8][9]. 3. Nova (Code Editor): There is also an app called Nova (a macOS code editor by Panic) [10], which provides its own APIs (e.g., nova.workspace.config) for extensions but is unrelated to web framework backend logic [10]. Regarding the specific items mentioned: - getCurrentUserIdOnServer: This is a custom helper function implemented in specific Next.js projects (like QCX) to bridge authentication sessions with database operations [2][4]. It is not a standard API of any framework called "Nova." - getUserDomain / saveUserDomain: These functions are not standard features of the Erlang Nova framework nor common Next.js boilerplate. If they exist in your codebase, they are likely custom business logic specific to your application's domain management. - lib/actions/chat.ts: This is a standard path structure for Server Actions in Next.js applications [9][11][12]. It typically contains functions for handling chat interactions, such as loading history or persisting messages [9][13]. If you are working in a Next.js project and see these in your lib/actions/chat.ts file, they are application-specific utilities for managing user data or settings, not framework-provided APIs. Citations:
Authorize domain reads and writes on the server.
🤖 Prompt for AI Agents |
||
|
|
||
| /** | ||
| * Retrieves all messages for a specific chat. | ||
| */ | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -11,12 +11,30 @@ const domainSchema = z.string().min(1).refine((val) => { | |||||||||||||||||||
| try { | ||||||||||||||||||||
| // Basic validation to check if it's a valid URL or domain | ||||||||||||||||||||
| const url = val.startsWith('http') ? val : `https://${val}`; | ||||||||||||||||||||
| new URL(url); | ||||||||||||||||||||
| const parsed = new URL(url); | ||||||||||||||||||||
| const hostname = parsed.hostname; | ||||||||||||||||||||
|
|
||||||||||||||||||||
| // Explicitly reject localhost and internal/local hostnames | ||||||||||||||||||||
| if (hostname === 'localhost') return false; | ||||||||||||||||||||
| if (hostname.endsWith('.local') || hostname.endsWith('.internal')) return false; | ||||||||||||||||||||
|
|
||||||||||||||||||||
| // Check private IP ranges: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, and 127.0.0.1 | ||||||||||||||||||||
| const ipv4Regex = /^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$/; | ||||||||||||||||||||
| const match = hostname.match(ipv4Regex); | ||||||||||||||||||||
| if (match) { | ||||||||||||||||||||
| const octet1 = parseInt(match[1], 10); | ||||||||||||||||||||
| const octet2 = parseInt(match[2], 10); | ||||||||||||||||||||
| if (octet1 === 10) return false; | ||||||||||||||||||||
| if (octet1 === 172 && octet2 >= 16 && octet2 <= 31) return false; | ||||||||||||||||||||
| if (octet1 === 192 && octet2 === 168) return false; | ||||||||||||||||||||
| if (octet1 === 127) return false; // Loopback | ||||||||||||||||||||
| } | ||||||||||||||||||||
|
|
||||||||||||||||||||
| return true; | ||||||||||||||||||||
| } catch (e) { | ||||||||||||||||||||
| return false; | ||||||||||||||||||||
| } | ||||||||||||||||||||
| }, { message: "Invalid domain or URL" }); | ||||||||||||||||||||
| }, { message: "Invalid domain or URL. Only public domains/URLs are allowed." }); | ||||||||||||||||||||
|
|
||||||||||||||||||||
| const STALENESS_THRESHOLD_MS = 1000 * 60 * 5; // 5 minutes | ||||||||||||||||||||
| const WORKER_TIMEOUT_MS = 1000 * 60 * 2; // 2 minutes | ||||||||||||||||||||
|
|
@@ -51,6 +69,54 @@ export async function startSystemPromptGeneration(domain: string) { | |||||||||||||||||||
| } | ||||||||||||||||||||
| } | ||||||||||||||||||||
|
|
||||||||||||||||||||
| async function scrapeDomain(domain: string): Promise<string> { | ||||||||||||||||||||
| const targetUrl = domain.startsWith('http') ? domain : `https://${domain}`; | ||||||||||||||||||||
| const apiKey = process.env.FIRECRAWL_API_KEY; | ||||||||||||||||||||
| if (apiKey && apiKey !== 'mock_key' && apiKey !== '') { | ||||||||||||||||||||
| try { | ||||||||||||||||||||
| const firecrawl = getFirecrawlClient(); | ||||||||||||||||||||
| const scrapeResult = await firecrawl.scrapeUrl(targetUrl, { | ||||||||||||||||||||
| formats: ['markdown'], | ||||||||||||||||||||
| }); | ||||||||||||||||||||
| if (scrapeResult && 'markdown' in scrapeResult && scrapeResult.markdown) { | ||||||||||||||||||||
| return scrapeResult.markdown; | ||||||||||||||||||||
| } | ||||||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win Apply the minimum-content threshold to Firecrawl output. Any non-empty markdown bypasses the 100-character check, so short error pages can generate a business prompt. Fall through to the fallback or fail when Firecrawl content is too short. Proposed fix- if (scrapeResult && 'markdown' in scrapeResult && scrapeResult.markdown) {
- return scrapeResult.markdown;
+ const markdown = scrapeResult && 'markdown' in scrapeResult
+ ? scrapeResult.markdown
+ : undefined;
+ if (typeof markdown === 'string' && markdown.trim().length >= 100) {
+ return markdown.trim();
}📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||||||||||||||||
| } catch (e) { | ||||||||||||||||||||
| console.warn('Firecrawl scraping failed, falling back to standard fetch scraper:', e); | ||||||||||||||||||||
| } | ||||||||||||||||||||
| } | ||||||||||||||||||||
|
|
||||||||||||||||||||
| // Fallback scraper using standard fetch | ||||||||||||||||||||
| try { | ||||||||||||||||||||
| const response = await fetch(targetUrl, { | ||||||||||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [blocking] This is an unrestricted server-side fetch after only a literal hostname check. A public name can resolve to RFC1918/loopback/link-local/IPv6 space, and |
||||||||||||||||||||
| headers: { | ||||||||||||||||||||
| 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36' | ||||||||||||||||||||
| } | ||||||||||||||||||||
| }); | ||||||||||||||||||||
| if (!response.ok) { | ||||||||||||||||||||
| throw new Error(`Fetch failed with status ${response.status}`); | ||||||||||||||||||||
| } | ||||||||||||||||||||
| const html = await response.text(); | ||||||||||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [blocking] The fallback loads the entire response into memory before applying the 5,000-character model cap. A reachable endpoint can return a very large or never-ending body and exhaust worker memory/CPU during one generation request. Reject oversized |
||||||||||||||||||||
| // Basic extraction of visible text from HTML if it contains tags | ||||||||||||||||||||
| let textContent = html; | ||||||||||||||||||||
| if (html.includes('<html') || html.includes('<body')) { | ||||||||||||||||||||
| // Remove scripts, styles, and tags | ||||||||||||||||||||
| textContent = html | ||||||||||||||||||||
| .replace(/<script\b[^<]*(?:(?!<\/script>)<[^<]*)*<\/script>/gi, '') | ||||||||||||||||||||
| .replace(/<style\b[^<]*(?:(?!<\/style>)<[^<]*)*<\/style>/gi, '') | ||||||||||||||||||||
| .replace(/<[^>]+>/g, ' ') | ||||||||||||||||||||
| .replace(/\s+/g, ' ') | ||||||||||||||||||||
| .trim(); | ||||||||||||||||||||
| } | ||||||||||||||||||||
| if (textContent.length < 100) { | ||||||||||||||||||||
| throw new Error('Fallback fetch scraped insufficient content'); | ||||||||||||||||||||
| } | ||||||||||||||||||||
| return textContent; | ||||||||||||||||||||
| } catch (error: any) { | ||||||||||||||||||||
| throw new Error(`Scraping failed: ${error.message || error}`); | ||||||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift Cancel and bound scraping when the worker deadline expires.
Also applies to: 129-129 🤖 Prompt for AI Agents |
||||||||||||||||||||
| } | ||||||||||||||||||||
| } | ||||||||||||||||||||
|
|
||||||||||||||||||||
| async function runBackgroundWorker(jobId: string, userId: string, domain: string) { | ||||||||||||||||||||
| const timeoutPromise = new Promise((_, reject) => | ||||||||||||||||||||
| setTimeout(() => reject(new Error('Job execution timeout')), WORKER_TIMEOUT_MS) | ||||||||||||||||||||
|
|
@@ -60,22 +126,7 @@ async function runBackgroundWorker(jobId: string, userId: string, domain: string | |||||||||||||||||||
| await updateJob(jobId, userId, { status: 'processing' }); | ||||||||||||||||||||
|
|
||||||||||||||||||||
| const workPromise = (async () => { | ||||||||||||||||||||
| const firecrawl = getFirecrawlClient(); | ||||||||||||||||||||
| const scrapeResult = await firecrawl.scrapeUrl(domain, { | ||||||||||||||||||||
| formats: ['markdown'], | ||||||||||||||||||||
| }); | ||||||||||||||||||||
|
|
||||||||||||||||||||
| // The library's type definitions for scrapeUrl can be complex. | ||||||||||||||||||||
| // We check for markdown presence to determine success. | ||||||||||||||||||||
| if (!scrapeResult || !('markdown' in scrapeResult) || !scrapeResult.markdown) { | ||||||||||||||||||||
| const errorMsg = (scrapeResult as any)?.error || 'Failed to scrape content'; | ||||||||||||||||||||
| throw new Error(errorMsg); | ||||||||||||||||||||
| } | ||||||||||||||||||||
|
|
||||||||||||||||||||
| const content = scrapeResult.markdown; | ||||||||||||||||||||
| if (content.length < 100) { | ||||||||||||||||||||
| throw new Error('Insufficient content scraped from domain'); | ||||||||||||||||||||
| } | ||||||||||||||||||||
| const content = await scrapeDomain(domain); | ||||||||||||||||||||
|
|
||||||||||||||||||||
| const model = await getModel(); | ||||||||||||||||||||
| const { text } = await generateText({ | ||||||||||||||||||||
|
|
||||||||||||||||||||
Large diffs are not rendered by default.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[blocking] This server action trusts a caller-supplied
userIdinstead of resolving the authenticated database user. The settings caller passes Clerkuser.id, whileusers.idis the internal UUID, so normal domain saves/loads fail; an authenticated caller can also pass another user's UUID to read or overwrite that user's metadata. UsegetCurrentUserIdOnServer()inside both domain actions and ignore the browser-provided ID.