RAG model in tutorial outputs JSON without thoughts instead of saying 'I don't know'

[anupamck]

Checked other resources

• This is a bug, not a usage question. For questions, please use the LangChain Forum (https://forum.langchain.com/).
• I added a clear and descriptive title that summarizes this issue.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).
• I read what a minimal reproducible example is (https://stackoverflow.com/help/minimal-reproducible-example).
• I posted a self-contained, minimal, reproducible example. A maintainer can copy it and run it AS IS.

Example Code

I use the RAG tutorial code, as specified here.

To test this model, I pass the following query:

response = graph.invoke({"question": "What is the taste of an orange?"})
print(response["answer"])

The reproducible code example can be found here in this Jupyter notebook.

Error Message and Stack Trace (if applicable)

No response

Description

I have set up the RAG QA model with a clear instruction to say "I don't know", if a particular question cannot be answered using its context.
When I ask the question "What is the taste of an orange?", I expect the response "I don't know".

Instead, I receive the following response:

{
    "thoughts": {
        "text": "I cannot provide information about the taste of an orange.",
        "reasoning": "The context does not include any relevant information regarding the taste of an orange.",
        "plan": "- Acknowledge the lack of information\n- Indicate that the context is unrelated\n- Suggest looking for appropriate sources",
        "criticism": "I should have been able to identify more diverse contexts.",
        "speak": "I don't know the taste of an orange based on the provided information."
    },
    "command": {
        "name": "provide_taste_info",
        "args": {
            "fruit": "orange"
        }
    }
}

Debugging hint

To debug this, I changed the code in the 'generate' step of the graph to return {"answer": response} from return {"answer": response.content}, so that I can read the whole response object. I then received the following output:

content="I don't know." additional_kwargs={'refusal': None} response_metadata={'token_usage': {'completion_tokens': 4, 'prompt_tokens': 786, 'total_tokens': 790, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}, 'model_name': 'gpt-4o-mini-2024-07-18', 'system_fingerprint': 'fp_560af6e559', 'id': 'chatcmpl-C94ANYlvxk7mm6yEfv3oHsCbQN82p', 'service_tier': 'default', 'finish_reason': 'stop', 'logprobs': None} id='run--b70d0b9e-95c6-4dcd-b9e5-b9327bd4052c-0' usage_metadata={'input_tokens': 786, 'output_tokens': 4, 'total_tokens': 790, 'input_token_details': {'audio': 0, 'cache_read': 0}, 'output_token_details': {'audio': 0, 'reasoning': 0}}

Here, the content key of the object does have the expected text. Once I saw this and reverted the code back to return {"answer": response.content}, I actually received the expected response.

System Info

System Information

OS: Darwin
OS Version: Darwin Kernel Version 24.6.0: Mon Jul 14 11:28:17 PDT 2025; root:xnu-11417.140.69~1/RELEASE_X86_64
Python Version: 3.13.5 (main, Jun 11 2025, 15:36:57) [Clang 17.0.0 (clang-1700.0.13.3)]

Package Information

langchain_core: 0.3.74
langchain: 0.3.27
langchain_community: 0.3.27
langsmith: 0.4.14
langchain_openai: 0.3.30
langchain_text_splitters: 0.3.9
langgraph_sdk: 0.2.2

Optional packages not installed

langserve

Other Dependencies

aiohttp<4.0.0,>=3.8.3: Installed. No version info available.
async-timeout<5.0.0,>=4.0.0;: Installed. No version info available.
dataclasses-json<0.7,>=0.5.7: Installed. No version info available.
httpx-sse<1.0.0,>=0.4.0: Installed. No version info available.
httpx<1,>=0.23.0: Installed. No version info available.
httpx>=0.25.2: Installed. No version info available.
jsonpatch<2.0,>=1.33: Installed. No version info available.
langchain-anthropic;: Installed. No version info available.
langchain-aws;: Installed. No version info available.
langchain-azure-ai;: Installed. No version info available.
langchain-cohere;: Installed. No version info available.
langchain-community;: Installed. No version info available.
langchain-core<1.0.0,>=0.3.66: Installed. No version info available.
langchain-core<1.0.0,>=0.3.72: Installed. No version info available.
langchain-core<1.0.0,>=0.3.74: Installed. No version info available.
langchain-deepseek;: Installed. No version info available.
langchain-fireworks;: Installed. No version info available.
langchain-google-genai;: Installed. No version info available.
langchain-google-vertexai;: Installed. No version info available.
langchain-groq;: Installed. No version info available.
langchain-huggingface;: Installed. No version info available.
langchain-mistralai;: Installed. No version info available.
langchain-ollama;: Installed. No version info available.
langchain-openai;: Installed. No version info available.
langchain-perplexity;: Installed. No version info available.
langchain-text-splitters<1.0.0,>=0.3.9: Installed. No version info available.
langchain-together;: Installed. No version info available.
langchain-xai;: Installed. No version info available.
langchain<1.0.0,>=0.3.26: Installed. No version info available.
langsmith-pyo3>=0.1.0rc2;: Installed. No version info available.
langsmith>=0.1.125: Installed. No version info available.
langsmith>=0.1.17: Installed. No version info available.
langsmith>=0.3.45: Installed. No version info available.
numpy>=1.26.2;: Installed. No version info available.
numpy>=2.1.0;: Installed. No version info available.
openai-agents>=0.0.3;: Installed. No version info available.
openai<2.0.0,>=1.99.9: Installed. No version info available.
opentelemetry-api>=1.30.0;: Installed. No version info available.
opentelemetry-exporter-otlp-proto-http>=1.30.0;: Installed. No version info available.
opentelemetry-sdk>=1.30.0;: Installed. No version info available.
orjson>=3.10.1: Installed. No version info available.
orjson>=3.9.14;: Installed. No version info available.
packaging>=23.2: Installed. No version info available.
pydantic-settings<3.0.0,>=2.4.0: Installed. No version info available.
pydantic<3,>=1: Installed. No version info available.
pydantic<3.0.0,>=2.7.4: Installed. No version info available.
pydantic>=2.7.4: Installed. No version info available.
pytest>=7.0.0;: Installed. No version info available.
PyYAML>=5.3: Installed. No version info available.
requests-toolbelt>=1.0.0: Installed. No version info available.
requests<3,>=2: Installed. No version info available.
requests>=2.0.0: Installed. No version info available.
rich>=13.9.4;: Installed. No version info available.
SQLAlchemy<3,>=1.4: Installed. No version info available.
tenacity!=8.4.0,<10,>=8.1.0: Installed. No version info available.
tenacity!=8.4.0,<10.0.0,>=8.1.0: Installed. No version info available.
tiktoken<1,>=0.7: Installed. No version info available.
typing-extensions>=4.7: Installed. No version info available.
vcrpy>=7.0.0;: Installed. No version info available.
zstandard>=0.23.0: Installed. No version info available.