backend: Handle error when web scraping fails in Hybrid search

[tianjing-li]

Thank you for contributing to the Cohere Toolkit!

• PR title: "area: description"

  • Where "area" is whichever of interface, frontend, model, tools, backend, etc. is being modified. Use "docs: ..." for purely docs changes, "infra: ..." for CI changes.
  • Example: "deployment: add Azure model option"

• PR message: Delete this entire checklist and replace with

  • Description: a description of the change
  • Issue: the issue # it fixes, if applicable
  • Dependencies: any dependencies required for this change

• Add tests and docs: Please include testing and documentation for your changes

• Lint and test: Run make lint and make run-tests

AI Description

This pull request introduces changes to two Python scripts, src/backend/tools/hybrid_search.py and src/backend/tools/web_scrape.py, which are part of a backend system for web scraping and search functionality.

src/backend/tools/hybrid_search.py

The rerank_results function has been updated to include a more robust check for the presence of the 'text' key in the result dictionary. This change ensures that the function only processes results that are dictionaries and contain the 'text' key.

src/backend/tools/web_scrape.py

The call function now returns a list of dictionaries instead of a single dictionary in the case of errors. This modification standardises the error handling and ensures that the returned data structure is consistent.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.

Project coverage is 77.97%. Comparing base (320f6ef) to head (2939367).

Files with missing lines	Patch %	Lines
src/backend/tools/web_scrape.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #879   +/-   ##
=======================================
  Coverage   77.97%   77.97%           
=======================================
  Files         259      259           
  Lines       11123    11123           
=======================================
  Hits         8673     8673           
  Misses       2450     2450           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.