feat: Implement Dynamic FAQ Handler with Web Search for Organizationa… #119
Conversation
|
Important Review skippedReview was skipped due to path filters ⛔ Files ignored due to path filters (1)
CodeRabbit blocks several paths by default. You can override this behavior by explicitly including those paths in the path filters. For example, including You can disable this status message by setting the WalkthroughThis change refactors the FAQ handler pipeline to use a dynamic, multi-step process that leverages web search and an LLM for synthesizing organizational answers. It updates function signatures and tool node wiring to inject a search tool and LLM, introduces robust error handling and response formatting, and enhances supervisor node validation and logging. Discord cog loading is disabled, and dependency requirements are adjusted. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant DevRelAgent
participant FAQHandler
participant SearchTool
participant LLM
User->>DevRelAgent: Asks organizational FAQ
DevRelAgent->>FAQHandler: Passes query, search_tool, llm
FAQHandler->>LLM: Refine query for org-specific search
LLM-->>FAQHandler: Refined query
FAQHandler->>SearchTool: Search with refined query
SearchTool-->>FAQHandler: Top search results
FAQHandler->>LLM: Synthesize answer from results
LLM-->>FAQHandler: Synthesized answer
FAQHandler->>DevRelAgent: Formatted answer with citations
DevRelAgent->>User: Responds with answer
Estimated code review effort🎯 4 (Complex) | ⏱️ ~40 minutes Assessment against linked issues
Assessment against linked issues: Out-of-scope changes
Possibly related PRs
Suggested reviewers
Poem
✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (5)
backend/app/agents/devrel/nodes/handlers/faq.py (1)
68-94: Well-designed query refinement with effective prompt engineering.The prompt provides clear instructions and helpful examples for transforming user questions into effective search queries. The focus on official organizational information is appropriate.
Consider adding a max token limit to the LLM call to ensure concise query generation:
- response = await llm.ainvoke([HumanMessage(content=refinement_prompt)]) + response = await llm.ainvoke([HumanMessage(content=refinement_prompt)], max_tokens=50)backend/app/agents/devrel/nodes/react_supervisor.py (3)
10-14: Consider making configuration constants environment-configurableThese hardcoded values might need adjustment based on deployment environment or use case. Consider loading them from environment variables or a configuration file for better flexibility.
+import os + # Configuration constants -MAX_ITERATIONS = 10 -MAX_CONVERSATION_HISTORY = 5 +MAX_ITERATIONS = int(os.getenv("REACT_MAX_ITERATIONS", "10")) +MAX_CONVERSATION_HISTORY = int(os.getenv("REACT_MAX_CONVERSATION_HISTORY", "5")) VALID_ACTIONS = ["web_search", "faq_handler", "onboarding", "github_toolkit", "complete"]
136-139: Redundant iteration checkThis iteration count check duplicates the logic already present in
react_supervisor_node(line 33). Since the supervisor node already handles this check and creates a completion response, this check will never trigger.Consider removing this redundant check or documenting why double-checking is necessary:
- # Safety check for infinite loops - iteration_count = state.context.get("iteration_count", 0) - if iteration_count > MAX_ITERATIONS: - logger.warning(f"Max iterations reached for session {state.session_id}") - return "complete"
231-249: Consider validating against AgentState schemaWhile the current validation checks essential attributes, consider validating against the full AgentState schema for more comprehensive validation.
You could leverage Pydantic's validation:
def _validate_state(state: AgentState) -> bool: """Validate state before processing""" try: if not state: return False # Leverage Pydantic validation if hasattr(state, 'model_validate'): state.model_validate(state.model_dump()) # Check required fields if not hasattr(state, 'session_id') or not state.session_id: logger.error("Invalid state: missing session_id") return False if not hasattr(state, 'context'): logger.error("Invalid state: missing context") return False return True except Exception as e: logger.error(f"Error validating state: {e}") return Falsebackend/integrations/discord/cogs.py (1)
48-77: Consider consolidating the cleanup logic.Both cleanup approaches execute the same core functionality but with different scheduling mechanisms. Consider extracting the common cleanup logic.
+ async def _execute_cleanup(self): + """Common cleanup logic used by both scheduling approaches""" + try: + print("--> Running token cleanup task...") + await cleanup_expired_tokens() + print("--> Token cleanup task finished.") + except Exception as e: + logger.error(f"Error during token cleanup: {e}") if TASKS_AVAILABLE: @tasks.loop(minutes=5) async def cleanup_expired_tokens(self): """Periodic cleanup of expired verification tokens""" - try: - print("--> Running token cleanup task...") - await cleanup_expired_tokens() - print("--> Token cleanup task finished.") - except Exception as e: - logger.error(f"Error during token cleanup: {e}") + await self._execute_cleanup() async def _manual_cleanup_loop(self): """Manual cleanup loop if tasks extension is not available""" await self.bot.wait_until_ready() while True: try: await asyncio.sleep(300) # 5 minutes - print("--> Running manual token cleanup task...") - await cleanup_expired_tokens() - print("--> Manual token cleanup task finished.") + await self._execute_cleanup() except asyncio.CancelledError: break - except Exception as e: - logger.error(f"Error during manual token cleanup: {e}")
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
poetry.lockis excluded by!**/*.lock
📒 Files selected for processing (10)
backend/app/agents/devrel/agent.py(1 hunks)backend/app/agents/devrel/nodes/handlers/faq.py(1 hunks)backend/app/agents/devrel/nodes/handlers/web_search.py(2 hunks)backend/app/agents/devrel/nodes/react_supervisor.py(1 hunks)backend/app/agents/devrel/tool_wrappers.py(1 hunks)backend/app/core/handler/faq_handler.py(1 hunks)backend/integrations/discord/bot.py(5 hunks)backend/integrations/discord/cogs.py(5 hunks)backend/main.py(3 hunks)pyproject.toml(1 hunks)
🧰 Additional context used
🧠 Learnings (7)
📚 Learning: in the devr.ai backend, the devrelagent follows a singleton pattern where only one instance exists f...
Learnt from: smokeyScraper
PR: AOSSIE-Org/Devr.AI#75
File: backend/app/agents/devrel/agent.py:34-35
Timestamp: 2025-06-13T21:56:19.183Z
Learning: In the Devr.AI backend, the DevRelAgent follows a singleton pattern where only one instance exists for the entire application lifetime, using InMemorySaver with thread-based conversation management to persist user conversations across sessions.
Applied to files:
backend/integrations/discord/bot.py
📚 Learning: in the devr.ai codebase, user_id directly maps as thread_id for memory persistence in the agent syst...
Learnt from: smokeyScraper
PR: AOSSIE-Org/Devr.AI#76
File: backend/app/agents/shared/base_agent.py:22-44
Timestamp: 2025-06-14T05:57:43.872Z
Learning: In the Devr.AI codebase, user_id directly maps as thread_id for memory persistence in the agent system, ensuring thread_id is always non-empty and unique per user.
Applied to files:
backend/integrations/discord/bot.py
📚 Learning: the handle_technical_support_node function in backend/app/agents/devrel/nodes/handle_technical_suppo...
Learnt from: smokeyScraper
PR: AOSSIE-Org/Devr.AI#72
File: backend/app/agents/devrel/nodes/handle_technical_support_node.py:6-17
Timestamp: 2025-06-08T13:15:40.536Z
Learning: The handle_technical_support_node function in backend/app/agents/devrel/nodes/handle_technical_support_node.py is intentionally minimal and will be extended after database configuration is completed.
Applied to files:
backend/app/agents/devrel/tool_wrappers.pybackend/app/agents/devrel/agent.pybackend/app/agents/devrel/nodes/handlers/faq.pybackend/app/agents/devrel/nodes/react_supervisor.py
📚 Learning: in backend/app/agents/devrel/tools/search_tool.py, the tavilysearchtool.search() method has partial ...
Learnt from: smokeyScraper
PR: AOSSIE-Org/Devr.AI#72
File: backend/app/agents/devrel/nodes/handle_web_search_node.py:31-42
Timestamp: 2025-06-08T13:31:11.572Z
Learning: In backend/app/agents/devrel/tools/search_tool.py, the TavilySearchTool.search() method has partial error handling for missing API key, AttributeError, ConnectionError, and TimeoutError, but lacks a comprehensive Exception catch-all block, so calling functions may still need additional error handling for other potential exceptions.
Applied to files:
backend/app/agents/devrel/tool_wrappers.pybackend/app/agents/devrel/nodes/handlers/web_search.pybackend/app/agents/devrel/agent.pybackend/app/agents/devrel/nodes/handlers/faq.py
📚 Learning: in backend/app/agents/devrel/tools/search_tool.py, the tavilysearchtool.search() method already incl...
Learnt from: smokeyScraper
PR: AOSSIE-Org/Devr.AI#72
File: backend/app/agents/devrel/nodes/handle_web_search_node.py:31-42
Timestamp: 2025-06-08T13:31:11.572Z
Learning: In backend/app/agents/devrel/tools/search_tool.py, the TavilySearchTool.search() method already includes comprehensive error handling that catches all exceptions and returns an empty list instead of raising them, so calling functions don't need additional try-catch blocks.
Applied to files:
backend/app/agents/devrel/nodes/handlers/web_search.pybackend/app/agents/devrel/nodes/handlers/faq.py
📚 Learning: in the devr.ai project, the application is designed to run from the backend directory, making import...
Learnt from: smokeyScraper
PR: AOSSIE-Org/Devr.AI#87
File: backend/app/api/v1/health.py:1-12
Timestamp: 2025-06-28T23:14:21.089Z
Learning: In the Devr.AI project, the application is designed to run from the backend directory, making import paths like `from main import DevRAIApplication` correct for the current setup. Import path adjustments for deployment will be configured later during the deployment process.
Applied to files:
backend/main.py
📚 Learning: in the devr.ai project, smokeyscraper prefers to defer test updates and fixes (like missing imports ...
Learnt from: smokeyScraper
PR: AOSSIE-Org/Devr.AI#87
File: tests/test_supabase.py:1-3
Timestamp: 2025-06-28T23:15:13.374Z
Learning: In the Devr.AI project, smokeyScraper prefers to defer test updates and fixes (like missing imports after module reorganization) to separate PRs rather than expanding the scope of module update/chore PRs to include comprehensive test refactoring.
Applied to files:
backend/integrations/discord/cogs.py
🧬 Code Graph Analysis (2)
backend/app/core/handler/faq_handler.py (1)
backend/app/core/events/enums.py (1)
EventType(10-32)
backend/app/agents/devrel/nodes/react_supervisor.py (2)
backend/app/agents/state.py (1)
AgentState(18-73)backend/app/agents/devrel/nodes/generate_response.py (1)
_get_latest_message(97-101)
🔇 Additional comments (17)
backend/app/agents/devrel/nodes/handlers/web_search.py (1)
54-54: LGTM! Minor formatting cleanup.The removal of the trailing whitespace and extra space improves code consistency without affecting functionality.
Also applies to: 64-64
backend/app/core/handler/faq_handler.py (1)
4-4: LGTM! Import cleanup.Removing the unused
PlatformTypeimport improves code cleanliness.backend/app/agents/devrel/agent.py (1)
46-47: LGTM! Correctly updated for dynamic FAQ handling.The faq_handler_tool_node instantiation now properly passes search_tool and llm parameters, aligning with the refactored dynamic FAQ implementation.
backend/app/agents/devrel/tool_wrappers.py (1)
19-24: LGTM! Function signature correctly updated.The faq_handler_tool_node function signature and implementation are properly updated to support the dynamic FAQ handler, passing search_tool and llm instead of the static faq_tool.
backend/app/agents/devrel/nodes/handlers/faq.py (5)
8-28: LGTM! Well-structured handler implementation.The refactored handle_faq_node properly extracts the user message, delegates to the dynamic FAQ process, and returns a well-structured response with appropriate metadata.
30-67: LGTM! Robust implementation with excellent error handling.The 5-step dynamic FAQ process is well-implemented with:
- Clear step-by-step logging for debugging
- Comprehensive error handling with fallback response
- Proper handling of empty search results
95-129: LGTM! Excellent synthesis implementation.The synthesis function effectively:
- Limits context to top 5 results for optimal LLM performance
- Provides clear, detailed instructions for response generation
- Maintains appropriate response length constraints
- Ensures conversational tone without exposing internal mechanics
130-145: LGTM! Clean and user-friendly response formatting.The formatting function appropriately:
- Presents the synthesized answer first
- Adds a clear sources section with emoji for visual appeal
- Limits to top 3 sources for readability
- Uses proper markdown formatting for clickable links
146-156: LGTM! Thoughtful fallback response design.The fallback response excellently:
- Acknowledges the user's question
- Provides transparent reasons for failure
- Offers actionable suggestions for the user
- Maintains a helpful and professional tone
backend/app/agents/devrel/nodes/react_supervisor.py (2)
72-128: Well-structured parsing implementationThe enhanced parsing logic effectively handles multi-line LLM responses with proper section detection and content accumulation. The validation against VALID_ACTIONS and fallback behavior ensures robust operation.
250-271: Well-structured response helpersThese helper functions provide consistent response structures for both error and normal completion scenarios, which aids in debugging and state management throughout the agent workflow.
backend/integrations/discord/bot.py (1)
68-68: LGTM - Code formatting improvements.These whitespace and formatting adjustments improve code hygiene without affecting functionality.
Also applies to: 74-74, 94-94, 104-104, 117-117, 142-142
backend/main.py (2)
49-57: Clarify the temporary disabling of Discord cog extensions.The Discord cog loading has been completely commented out, which disables slash commands and periodic cleanup tasks. While the comment mentions "troubleshooting import issues," this creates an inconsistency with the PR objectives that focus on implementing FAQ functionality.
This change effectively breaks Discord slash commands like
/help,/reset, and/verify_github. Please clarify:
- Is this temporary disabling intentional for this PR?
- How does this align with implementing the Dynamic FAQ Handler?
- Should the FAQ functionality work without these extensions?
Consider whether the FAQ handler should be implemented as part of the existing cog structure or as a separate component.
135-135: LGTM - Minor formatting improvement.backend/integrations/discord/cogs.py (3)
14-21: LGTM - Robust fallback for optional tasks extension.The conditional import with a feature flag is a good approach to handle environments where the
tasksextension may not be available.
42-47: LGTM - Proper cleanup task cancellation.The cog_unload method properly handles cancellation for both cleanup approaches.
141-141: LGTM - Code formatting improvements.Also applies to: 156-156, 217-217
| return add_tool_result(state, "web_search", tool_result) | ||
|
|
||
| async def faq_handler_tool_node(state: AgentState, faq_tool) -> Dict[str, Any]: | ||
| async def faq_handler_tool_node(state: AgentState, search_tool, llm) -> Dict[str, Any]: |
There was a problem hiding this comment.
Let the FAQ tool be independent of the search_tool. One of the SOLID principle while designing systems.
There was a problem hiding this comment.
I don't think any changes done in this file are relevant? What do you think? Could you please detail out why you wanted these changes? I don't think there is any relevance on changing the react_supervisor code for each handler? FAQ tool can work with the initial system too?
2 participants
🌟 Feature Implementation: Dynamic FAQ Handler
Problem Solved
Key Features Implemented
✅ Smart Query Routing: ReAct Supervisor intelligently identifies organizational queries
✅ Dynamic Web Search: Real-time information retrieval from organization's public presence
✅ Automated Synthesis: Converts search results into comprehensive, user-friendly responses
✅ Source Attribution: Provides links for users to explore topics further
Testing Results
Successfully tested with query: "How does this organization work?"
Technical Implementation
Ready for review and production deployment.
Closes #97
bandicam.2025-08-04.11-01-48-554.mp4
bandicam.2025-08-04.13-33-21-813.mp4
Summary by CodeRabbit
New Features
Improvements
tasksextension.Bug Fixes
Chores
Style