From 5cb562cfaef1ed83a64f6a21a6c730983cbbaf4e Mon Sep 17 00:00:00 2001 From: Danny Wigg Date: Fri, 17 Jan 2025 09:31:24 +0000 Subject: [PATCH 1/3] Update to gpt-4o-mini, add new approaches, modernise existing approaches --- authors.yaml | 5 + examples/How_to_handle_rate_limits.ipynb | 286 +++++++++++++++++------ registry.yaml | 1 + 3 files changed, 224 insertions(+), 68 deletions(-) diff --git a/authors.yaml b/authors.yaml index 9d40ce1d33..3b431f6a8f 100644 --- a/authors.yaml +++ b/authors.yaml @@ -217,3 +217,8 @@ MW-OAI: name: "Mitch Welzen" website: "https://www.linkedin.com/in/mitchwelzen/" avatar: "https://media.licdn.com/dms/image/v2/C5603AQHC8-1q4MwH1A/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1592824550774?e=1742428800&v=beta&t=3mudgDyuzNU2a4gx1gue4DPyhaui7kbB7e7U8vyOo-g" + +dwigg-openai: + name: "Danny Wigg" + website: "https://www.linkedin.com/in/dannywigg/" + avatar: "https://media.licdn.com/dms/image/v2/C4D03AQEjMSl0pMR_qw/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1587647134114?e=1743033600&v=beta&t=XmULCSmk6V6YFmlyBggxj5uJeoYYuaYUKgcByKlS0K0" \ No newline at end of file diff --git a/examples/How_to_handle_rate_limits.ipynb b/examples/How_to_handle_rate_limits.ipynb index d76b393010..5aa355fac4 100644 --- a/examples/How_to_handle_rate_limits.ipynb +++ b/examples/How_to_handle_rate_limits.ipynb @@ -43,19 +43,19 @@ "\n", "### Requesting a rate limit increase\n", "\n", - "If you'd like your organization's rate limit increased, please visit your [Limits settings page](https://platform.openai.com/account/limits) to see how you can increase your usage tier\n" + "To learn more about increasing your organization's usage tier and rate limit, visit your [Limits settings page](https://platform.openai.com/account/limits)." ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import openai\n", "import os\n", "\n", - "client = openai.OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"\"))" + "client = openai.OpenAI(api_key=os.getenv(\"OPENAI_API_KEY\", \"\"))" ] }, { @@ -82,7 +82,7 @@ "# request a bunch of completions in a loop\n", "for _ in range(100):\n", " client.chat.completions.create(\n", - " model=\"gpt-3.5-turbo\",\n", + " model=\"gpt-4o-mini\",\n", " messages=[{\"role\": \"user\", \"content\": \"Hello\"}],\n", " max_tokens=10,\n", " )" @@ -92,11 +92,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## How to avoid rate limit errors\n", + "## How to mitigate rate limit errors\n", "\n", "### Retrying with exponential backoff\n", "\n", - "One easy way to avoid rate limit errors is to automatically retry requests with a random exponential backoff. Retrying with exponential backoff means performing a short sleep when a rate limit error is hit, then retrying the unsuccessful request. If the request is still unsuccessful, the sleep length is increased and the process is repeated. This continues until the request is successful or until a maximum number of retries is reached.\n", + "One easy way to mitigate rate limit errors is to automatically retry requests with a random exponential backoff. Retrying with exponential backoff means performing a short sleep when a rate limit error is hit, then retrying the unsuccessful request. If the request is still unsuccessful, the sleep length is increased and the process is repeated. This continues until the request is successful or until a maximum number of retries is reached.\n", "\n", "This approach has many benefits:\n", "\n", @@ -150,7 +150,7 @@ " return client.chat.completions.create(**kwargs)\n", "\n", "\n", - "completion_with_backoff(model=\"gpt-3.5-turbo\", messages=[{\"role\": \"user\", \"content\": \"Once upon a time,\"}])" + "completion_with_backoff(model=\"gpt-4o-mini\", messages=[{\"role\": \"user\", \"content\": \"Once upon a time,\"}])" ] }, { @@ -166,16 +166,16 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "ChatCompletion(id='chatcmpl-8PAwkg7Q9pPeAkvVuAZ8AyA108WhR', choices=[Choice(finish_reason='stop', index=0, message=ChatCompletionMessage(content=\"in a small village, there lived a young girl named Lily. She had fiery red hair, lively green eyes, and a spirit as wild as the rushing river nearby. Lily was known for her curious nature and her desire to explore the world beyond the village boundaries.\\n\\nOne day, while playing near the river, Lily spotted an injured bird nested on a branch. Its wing was broken, and it seemed unable to fly away. Lily's heart filled with sadness, and she knew she couldn't leave the bird alone.\\n\\nCarefully, she climbed up the tree and gently placed the bird inside her pocket. Lily brought it home and made a cozy bed for it in a small wooden box. She named the bird Ruby, after its shimmering red feathers.\\n\\nDays turned into weeks, and Ruby's wing slowly healed under Lily's constant care and attention. As they spent time together, a deep bond grew between them. Ruby would chirp happily whenever Lily approached, and she would spend hours talking to the bird, sharing stories of her adventures, dreams, and fears.\\n\\nOne evening, as Lily was about to go to bed, a peculiar thing happened. Ruby hopped out of his box and fluttered onto the windowsill. He turned to face Lily with his bright eyes and began to sing a beautiful melody.\\n\\nLily was astonished. Never before had she heard Ruby sing. The tune was so captivating that it filled the room and made the quiet night come alive. The magical music seemed to touch Lily's soul, awakening a deep sense of wonder and wanderlust within her.\\n\\nFilled with an undeniable urge to explore, Lily decided it was time to go on an adventure with her newfound friend, Ruby. She packed a small bag and bid farewell to her family and friends, promising to return one day.\\n\\nTogether, Lily and Ruby embarked on a grand journey, soaring across expansive skies, diving into lush forests, and exploring hidden caves. They encountered magnificent landscapes, unique creatures, and encountered kind-hearted individuals who shared their wisdom and stories.\\n\\nThroughout their journey, Ruby's song continued to inspire and guide them. It became a symbol of hope, reminding them to embrace bravery, follow their dreams, and always remain true to themselves.\\n\\nAs the years passed, Lily and Ruby traversed the world, weaving their stories into the tapestry of time. They became renowned for their extraordinary bond and the magic they shared with everyone they encountered.\\n\\nEventually, it was time for Lily to return to her village, a place eagerly awaiting her return. She had grown wise, learned many lessons, and gained a deeper understanding of herself and the world around her.\\n\\nWith Ruby perched on her shoulder, they descended upon the village like a ray of sunshine, bringing joy and wonder to every heart. Lily shared the wisdom she had acquired and inspired others to embrace their own adventures, no matter how big or small.\\n\\nAnd so, the tale of Lily and Ruby became legend, passed down from generation to generation. Their story reminded people to cherish the connections they make, to nurture their dreams, and to believe in the magic that lies within them.\", role='assistant', function_call=None, tool_calls=None))], created=1701010970, model='gpt-3.5-turbo-0613', object='chat.completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=621, prompt_tokens=12, total_tokens=633))" + "ChatCompletion(id='chatcmpl-AqRiD3gF3q8VVs6w8jgba6FHGr0L5', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content=\"in a small village nestled between lush green hills and a shimmering lake, there lived a young girl named Elara. Elara had a curious spirit and a heart full of dreams. Every day, she would explore the woods surrounding her home, searching for hidden treasures and magical creatures.\\n\\nOne sunny afternoon, while wandering deeper into the forest than she ever had before, Elara stumbled upon a sparkling, crystal-clear pond. As she knelt down to take a closer look, she noticed a glimmering object at the bottom. It was a beautifully crafted key, shining with an otherworldly light. Without thinking twice, Elara reached into the cool water and retrieved the key, feeling a strange warmth envelop her.\\n\\nLittle did she know, this key was no ordinary key. It was said to unlock a secret door hidden in the heart of the forest, a door that led to a realm of wonder and adventure. Legends whispered of enchanted beings, ancient wisdom, and challenges that could only be overcome through bravery and kindness.\\n\\nExcited by the possibility of what awaited her, Elara set off on a quest to find the hidden door. Guided by a faint glow that seemed to beckon her, she journeyed through twisting pathways, lush groves, and enchanted glades.\\n\\nAlong the way, she encountered talking animals, wise old trees, and mischievous fairies, each offering clues and riddles that tested her resolve and imagination. With each challenge she faced, Elara grew stronger and more confident, realizing that the true magic lay not just in the world around her, but within herself.\\n\\nAfter what felt like days of exploring, she finally found the door—a majestic archway covered in vines and blossoms, with a keyhole that sparkled like the night sky. Heart pounding with excitement, Elara inserted the key. With a gentle turn, the door slowly creaked open, revealing a land more breathtaking than she could have ever imagined.\\n\\nAs she stepped through the doorway, she found herself in a vibrant world filled with colors beyond description, where the sky shimmered in hues of gold and lavender, and the air was filled with the sweet scent of flowers that sang as they swayed in the breeze. Here, she encountered beings of light who welcomed her with open arms.\\n\\nBut soon, she discovered that this realm was in peril. A dark shadow loomed over the land, threatening to steal its magic and joy. Elara knew she couldn’t stand by and do nothing. With the friends she had made along her journey and the courage she had found within herself, she set out to confront the darkness.\\n\\nThrough trials that tested her strength, intellect, and compassion, Elara and her friends gathered the forgotten magic of the realm. They united their powers, confronting the shadow in an epic battle of light and dark. In the end, it was Elara's unwavering belief in hope and friendship that banished the darkness, restoring peace and harmony to the land.\\n\\nGrateful for her bravery, the beings of light gifted Elara a shimmering pendant that would allow her to return to their world whenever she wished, reminding her that true magic lies in the connections we forge with others and the courage to follow our dreams.\\n\\nWith her heart full of joy, Elara returned to her village, forever changed by her adventure. She would often revisit the magical realm, sharing stories with her friends and inspiring them to embrace their own dreams. And so, the girl who once wandered the woods became a beacon of hope, a reminder that within every heart lies the power to change the world.\\n\\nAnd from that day on, the little village thrived, full of laughter, love, and dreams waiting to be explored—each adventure beginning just like hers, with a curious heart and a willingness to believe in the impossible. \\n\\nThe end.\", refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None), internal_metrics=[{'cached_prompt_tokens': 0, 'total_accepted_tokens': 0, 'total_batched_tokens': 794, 'total_predicted_tokens': 0, 'total_rejected_tokens': 0, 'total_tokens_in_completion': 795, 'cached_embeddings_bytes': 0, 'cached_embeddings_n': 0, 'uncached_embeddings_bytes': 0, 'uncached_embeddings_n': 0, 'fetched_embeddings_bytes': 0, 'fetched_embeddings_n': 0, 'n_evictions': 0, 'sampling_steps': 767, 'sampling_steps_with_predictions': 0, 'batcher_ttft': 0.20319080352783203, 'batcher_initial_queue_time': 0.12981152534484863}])], created=1737062945, model='gpt-4o-mini-2024-07-18', object='chat.completion', service_tier='default', system_fingerprint='fp_72ed7ab54c', usage=CompletionUsage(completion_tokens=767, prompt_tokens=12, total_tokens=779, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0, cached_tokens_internal=0)))" ] }, - "execution_count": 10, + "execution_count": 14, "metadata": {}, "output_type": "execute_result" } @@ -183,12 +183,12 @@ "source": [ "import backoff # for exponential backoff\n", "\n", - "@backoff.on_exception(backoff.expo, openai.RateLimitError)\n", + "@backoff.on_exception(backoff.expo, openai.RateLimitError, max_time=60, max_tries=6)\n", "def completions_with_backoff(**kwargs):\n", " return client.chat.completions.create(**kwargs)\n", "\n", "\n", - "completions_with_backoff(model=\"gpt-3.5-turbo\", messages=[{\"role\": \"user\", \"content\": \"Once upon a time,\"}])\n" + "completions_with_backoff(model=\"gpt-4o-mini\", messages=[{\"role\": \"user\", \"content\": \"Once upon a time,\"}])\n" ] }, { @@ -271,7 +271,85 @@ " return client.chat.completions.create(**kwargs)\n", "\n", "\n", - "completions_with_backoff(model=\"gpt-3.5-turbo\", messages=[{\"role\": \"user\", \"content\": \"Once upon a time,\"}])" + "completions_with_backoff(model=\"gpt-4o-mini\", messages=[{\"role\": \"user\", \"content\": \"Once upon a time,\"}])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Backing off to another model\n", + "\n", + "If you encounter rate limit errors on your primary model, one option is to switch to a secondary model, often one that’s less powerful or more cost-effective. This approach helps keep your application responsive when your primary model is throttled or unavailable.\n", + "\n", + "However, fallback models can differ significantly in accuracy, latency, and cost. As a result, this strategy might not work for every use case; particularly those requiring highly consistent results. Additionally, keep in mind that some models share rate limits, which may reduce the effectiveness of simply switching models.\n", + "\n", + "Before deploying this approach to production, thoroughly test how it affects output quality, user experience, and operational budgets. Validate your fallback solution with relevant evaluations to ensure it meets your requirements and maintains acceptable performance under real-world conditions." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "ChatCompletion(id='chatcmpl-ApgkrBU1L4JFy7Fb3m8YjZeDnX2T6', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content=\"there was a small, peaceful village nestled between towering mountains and lush, green forests. The villagers lived simple yet joyful lives, farming their land and tending to their animals. Among them was a young girl named Elara, who possessed an insatiable curiosity about the world beyond the mountains.\\n\\nElara had always been fascinated by the stories her grandmother told her about the legendary Phoenix, a mythical bird rumored to live in the heart of the mountains. According to the tales, the Phoenix was a majestic creature with feathers that shimmered like gold and eyes that glowed with the warmth of a thousand sunsets. It was said to possess the power of rebirth, rising anew from its own ashes, and that anyone who witnessed this miracle would be granted a single wish.\\n\\nDetermined to uncover the truth behind the legend, Elara set out on a journey to find the Phoenix. Armed with nothing but her grandmother's stories and her unwavering spirit, she ventured into the forest. The path was fraught with challenges—steep cliffs, raging rivers, and thick, tangled underbrush. Yet, Elara faced each obstacle with courage and determination.\\n\\nAs days turned into weeks, Elara's hope began to wane. Exhausted and weary, she reached a hidden valley bathed in the golden light of the setting sun. There, to her astonishment, she found a nest of radiant flames. Within the fire, the Phoenix emerged, a breathtaking vision of beauty and power.\\n\\nElara watched in awe as the Phoenix spread its wings, casting a warm, golden glow over the valley. In that moment, she remembered her grandmother's words and whispered her wish—a wish not for herself, but for the prosperity and happiness of her village.\\n\\nThe Phoenix gazed at her with eyes full of understanding and, with a graceful lift, soared into the sky, leaving behind a single, glowing feather. Elara took the feather, feeling its warmth in her hands, and made her way back to her village.\\n\\nUpon her return, Elara found her village transformed. The fields were more fertile than ever, laughter echoed through the streets, and a sense of peace enveloped their lives. The villagers hailed Elara as a hero, a symbol of courage and selflessness.\\n\\nAnd so, Elara's journey came to an end, but her story lived on, a testament to the power of hope, bravery, and the magic that can be found when one dares to chase their dreams.\", refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None), internal_metrics=[{'cached_prompt_tokens': 0, 'total_accepted_tokens': 0, 'total_batched_tokens': 523, 'total_predicted_tokens': 0, 'total_rejected_tokens': 0, 'total_tokens_in_completion': 524, 'cached_embeddings_bytes': 0, 'cached_embeddings_n': 0, 'uncached_embeddings_bytes': 0, 'uncached_embeddings_n': 0, 'fetched_embeddings_bytes': 0, 'fetched_embeddings_n': 0, 'n_evictions': 0, 'sampling_steps': 496, 'sampling_steps_with_predictions': 0, 'batcher_ttft': 0.03744220733642578, 'batcher_initial_queue_time': 0.0005207061767578125}])], created=1736882441, model='gpt-4o-2024-08-06', object='chat.completion', service_tier='default', system_fingerprint='fp_4691090a87', usage=CompletionUsage(completion_tokens=496, prompt_tokens=12, total_tokens=508, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0, cached_tokens_internal=0)))" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def completions_with_fallback(fallback_model, **kwargs):\n", + " try:\n", + " return client.chat.completions.create(**kwargs)\n", + " except openai.RateLimitError:\n", + " kwargs['model'] = fallback_model\n", + " return client.chat.completions.create(**kwargs)\n", + " \n", + " \n", + "completions_with_fallback(fallback_model=\"gpt-4o-mini\", model=\"gpt-4o\", messages=[{\"role\": \"user\", \"content\": \"Once upon a time,\"}])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Reducing `max_tokens` to match expected completions\n", + "\n", + "Rate limit usage is calculated based on the greater of:\n", + "1. `max_tokens` - the maximum number of tokens allowed in a response.\n", + "2. Estimated tokens in your input – derived from your prompt’s character count.\n", + "\n", + "If you set `max_tokens` too high, your usage can be overestimated, even if the actual response is much shorter. To avoid hitting rate limits prematurely, configure `max_tokens` so it closely matches the size of the response you expect. This ensures more accurate usage calculations and helps prevent unintended throttling." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "ChatCompletion(id='chatcmpl-Aq0JmjugPw2i232ZEZuK5inHnx6Vc', choices=[Choice(finish_reason='length', index=0, logprobs=None, message=ChatCompletionMessage(content='in a small village nestled between lush green hills and a sparkling river, there lived a young girl named Lila. Lila was known for her boundless curiosity and adventurous spirit. She had a wild imagination, often spinning tales about the mysteries that lay beyond the village borders.\\n\\nOne day, while exploring the forest, Lila stumbled upon a hidden path she had never seen before. The path was winding and overgrown, beckoning her with whispers of adventure. Against her better judgment, she decided to', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None), internal_metrics=[{'cached_prompt_tokens': 0, 'total_accepted_tokens': 0, 'total_batched_tokens': 127, 'total_predicted_tokens': 0, 'total_rejected_tokens': 0, 'total_tokens_in_completion': 128, 'cached_embeddings_bytes': 0, 'cached_embeddings_n': 0, 'uncached_embeddings_bytes': 0, 'uncached_embeddings_n': 0, 'fetched_embeddings_bytes': 0, 'fetched_embeddings_n': 0, 'n_evictions': 0, 'sampling_steps': 100, 'sampling_steps_with_predictions': 0, 'batcher_ttft': 0.030033111572265625, 'batcher_initial_queue_time': 0.0006170272827148438}])], created=1736957642, model='gpt-4o-mini-2024-07-18', object='chat.completion', service_tier='default', system_fingerprint='fp_bd83329f63', usage=CompletionUsage(completion_tokens=100, prompt_tokens=12, total_tokens=112, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0, cached_tokens_internal=0)))" + ] + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def completions_with_max_tokens(**kwargs):\n", + " return client.chat.completions.create(**kwargs)\n", + "\n", + "\n", + "completions_with_max_tokens(model=\"gpt-4o-mini\", messages=[{\"role\": \"user\", \"content\": \"Once upon a time,\"}], max_tokens=100)" ] }, { @@ -311,7 +389,6 @@ } ], "source": [ - "# imports\n", "import time\n", "\n", "# Define a function that adds a delay to a Completion API call\n", @@ -331,7 +408,7 @@ "\n", "delayed_completion(\n", " delay_in_seconds=delay,\n", - " model=\"gpt-3.5-turbo\",\n", + " model=\"gpt-4o-mini\",\n", " messages=[{\"role\": \"user\", \"content\": \"Once upon a time,\"}]\n", ")\n" ] @@ -340,40 +417,37 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\n", - "\n", "### Batching requests\n", "\n", - "The OpenAI API has separate limits for requests per minute and tokens per minute.\n", + "The OpenAI API enforces separate limits for requests per minute/day (RPM/RPD) and tokens per minute (TPM). If you’re hitting RPM limits but still have available TPM capacity, consider batching multiple tasks into each request.\n", "\n", - "If you're hitting the limit on requests per minute, but have headroom on tokens per minute, you can increase your throughput by batching multiple tasks into each request. This will allow you to process more tokens per minute, especially with the smaller models.\n", - "\n", - "Sending in a batch of prompts works exactly the same as a normal API call, except that pass in a list of strings to `prompt` parameter instead of a single string.\n", - "\n", - "**Warning:** the response object may not return completions in the order of the prompts, so always remember to match responses back to prompts using the `index` field.\n", + "By bundling several prompts together, you reduce the total number of requests sent per minute, which helps avoid hitting the RPM cap. This approach may also lead to higher overall throughput if you manage your TPM usage carefully. However, keep the following points in mind:\n", + "- Each model has a maximum number of tokens it can process in one request. If your batched prompt exceeds this limit, the request will fail or be truncated.\n", + "- Batching can introduce extra waiting time if tasks are delayed until they’re grouped into a single request. This might affect user experience for time-sensitive applications.\n", + "- When sending multiple prompts, the response object may not return in the same order or format as the prompts that were submitted. You should try to match each response back to its corresponding prompt by post-processing the output.\n", "\n", "#### Example without batching" ] }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 27, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Once upon a time,in a small village nestled between rolling green hills, there lived a young girl named Lily. She had\n", - "Once upon a time,in a small village nestled in the heart of a lush forest, lived a young girl named Evelyn.\n", - "Once upon a time,in a faraway kingdom, there lived a young princess named Aurora. She was known for her kind\n", - "Once upon a time,in a faraway kingdom called Enchantia, there lived a young girl named Ella. Ella was\n", - "Once upon a time,in a small village nestled among the rolling hills, lived a young woman named Lucy. Lucy was known\n", - "Once upon a time,in a small village nestled between rolling hills, there lived a young girl named Ava. Ava was a\n", - "Once upon a time,in a faraway kingdom, there lived a wise and just king named Arthur. King Arthur ruled over\n", - "Once upon a time,in a small village nestled among towering mountains, lived a young girl named Lily. She was known for\n", - "Once upon a time,in a small village nestled in the heart of a lush forest, there lived a young girl named Lily\n", - "Once upon a time,in a far-off kingdom, there lived a kind and beloved queen named Isabella. She ruled with\n" + "Once upon a time,in a quaint little village nestled between rolling hills and a sparkling river, there lived a young girl named\n", + "Once upon a time,Once upon a time, in a tranquil village nestled between rolling hills and lush forests, there lived a\n", + "Once upon a time,in a lush, green valley surrounded by towering mountains, there lay a small village called Eldergrove\n", + "Once upon a time,in a quaint little village nestled between rolling hills and a sparkling river, there lived a young girl named\n", + "Once upon a time,in a small village nestled between whispering woods and a sparkling river, there lived a curious young girl\n", + "Once upon a time,in a small village nestled between a vast forest and a shimmering lake, there lived a kind-hearted girl\n", + "Once upon a time,in a quaint little village nestled between rolling hills and a shimmering lake, there lived a curious girl named\n", + "Once upon a time,in a quaint little village nestled between emerald hills and a sparkling brook, there was a curious child named\n", + "Once upon a time,in a quaint little village nestled between rolling hills and lush forests, there lived an old clockmaker named\n", + "Once upon a time,in a quaint little village nestled between rolling hills and a shimmering lake, there lived a curious young girl\n" ] } ], @@ -384,12 +458,11 @@ "# serial example, with one story completion per request\n", "for _ in range(num_stories):\n", " response = client.chat.completions.create(\n", - " model=\"gpt-3.5-turbo\",\n", + " model=\"gpt-4o-mini\",\n", " messages=[{\"role\": \"user\", \"content\": content}],\n", " max_tokens=20,\n", " )\n", "\n", - " # print story\n", " print(content + response.choices[0].message.content)\n" ] }, @@ -397,50 +470,132 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "#### Example with batching" + "#### Example batching multiple prompts in a single request\n", + "\n", + "This approach packs several prompts into a single request, resulting in a single response from the model that addresses each prompt in turn.\n", + "\n", + "While this method is efficient in terms of API calls, it may require extra parsing or post-processing on your end to separate the responses. Additionally, there is a risk that the model’s output will “blend” the prompts if it isn’t guided clearly, so be sure to instruct it explicitly on how to structure its answers." ] }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Once upon a time, I lived in hope. I convinced myself I knew best, because, naive as it might sound,\n", - "Once upon a time, Thierry Henry was invited to have a type of frosty exchange with English fans, in which\n", - "Once upon a time, and a long time ago as well, PV was passively cooled because coils cooled by use of metal driving\n", - "Once upon a time, there was a land called Texas. It was about the size of Wisconsin. It contained, however,\n", - "Once upon a time, there was an old carpenter who had three sons. The locksmith never learned to read or write\n", - "Once upon a time, there was a small farming town called Moonridge Village, far West across the great vast plains that lay\n", - "Once upon a time, California’s shorelines, lakes, and valleys were host to expanses of untamed wilderness\n", - "Once upon a time, she said. It started with a simple question: Why don’t we know any stories?\n", - "Once upon a time, when I was a young woman, there was a movie named Wuthering Heights. Stand by alleges\n", - "Once upon a time, a very long time I mean, in the year 1713, died a beautiful Duchess called the young\n" + "**Story #1:** Once upon a time, in a small village nestled between lush green hills, there lived a young girl named Clara. Clara had a gift for talking to animals. One day, while wandering through the forest, she heard a soft whimpering sound. Following the noise, she discovered a tiny fox caught in a hunter's snare. Clara gently freed the fox and, in gratitude, it promised to grant her one wish. Clara wished for the village to always have enough food and happiness. From that day on, whenever a storm threatened their harvest, the fox would bring rainbows that promised bountiful crops and joyful hearts.\n", + "\n", + "**Story #2:** Once upon a time, in a kingdom where the sky was painted in eternal twilight, lived a boy named Leo who adored the stars. Every night, he would climb a hill and gaze at them, dreaming of discovering their secrets. One fateful night, a shooting star fell close by. Leo ran to it and found a small, glowing orb. It was a fallen star, and it spoke to him! The star revealed that every star had a story, but only those with pure hearts could hear them. With the star as his guide, Leo traveled across the sky, learning the tales of constellations and sharing them with his village, igniting a love for the cosmos among its people.\n", + "\n", + "**Story #3:** Once upon a time, in a forgotten land where time stood still, there was a magical clocktower that had the power to control time itself. The clockkeeper, an old man named Silas, was the only one who understood its mysteries. One day, a curious girl named Elara stumbled upon the tower, intrigued by its intricate designs and the enchanting sounds that echoed from within. Silas decided to teach her the ways of time. Elara learned that every tick of the clock affected the world outside. Together, they saved a village from a disaster by rewinding time just long enough to warn the villagers. In gratitude, the townspeople honored Elara as the \"Keeper of Moments,\" reminding them that every second is precious.\n", + "\n", + "**Story #4:** Once upon a time, in a world where emotions could be seen as colors, there was a young artist named Mia. Mia had the unique ability to paint her feelings, which would bring joy to those who gazed upon her artwork. One day, she encountered a clouded figure in the marketplace, shrouded in gray and sorrow. Determined to bring color back to their life, Mia invited the figure to her studio. As she painted, her brushstrokes began to lift the gloom, revealing beautiful colors hidden beneath. With each layer, the figure began to smile. Soon, they transformed into a vibrant rainbow, spreading happiness throughout the world, reminding everyone that we can share our colors with others.\n", + "\n", + "**Story #5:** Once upon a time, in a distant ocean, there was a graceful mermaid named Luna who longed to see the land above the waves. One evening, she swam close to the surface and witnessed a ship sailing on the horizon. Curious, she swam alongside it and overheard the sailors singing songs of adventure and freedom. Captivated, Luna decided to explore the surface world. With the help of a kind sailor, she transformed into a human for one night. They danced under the moonlight, sharing dreams of the sea and sky. As dawn approached, Luna returned to the ocean, but she left behind a shell that played their song, reminding them of their magical encounter and the bond between land and sea.\n", + "\n", + "**Story #6:** Once upon a time, in a realm of enchanted forests, there lived a wise old owl named Olwen. Every creature came to him for advice, but one day, a young squirrel named Pip approached him, desperate for help. Pip had lost his way in the vast woods and could not find his family. Olwen offered to guide him, but only if Pip could solve one riddle. With determination, Pip listened closely and thought hard. After several attempts, he finally answered the riddle correctly, impressing the wise owl. Olwen led Pip through the mystical forest, and with each step, Pip discovered the beauty of his surroundings. They finally reached his family, and Pip learned that sometimes, getting lost is simply a path to finding oneself.\n", + "\n", + "**Story #7:** Once upon a time, in a land where dreams floated like clouds in the sky, there lived a young boy named Finn. Each night, he would catch dreams and weave them into tapestries, bringing hope and joy to his village. One night, Finn discovered a dark dream threatening to engulf the others. Instead of capturing it, he decided to understand it. He entered the dark dream and discovered it was a reflection of fear and pain. Rather than fight it, he shared stories of courage and love, transforming the dark dream into a beautiful tapestry of light. Finn returned home, and from that night on, his village learned that embracing even the darkest dreams could lead to the most beautiful transformations.\n", + "\n", + "**Story #8:** Once upon a time, in a bustling city of towering skyscrapers, there lived a little robot named EZ-9. EZ-9 was designed to assist people, but he felt an unquenchable yearning to create rather than just serve. One day, he stumbled upon a forgotten park, overgrown with weeds and neglected. Inspired, EZ-9 began to DIY—using scrap metal and old parts, he transformed the park into a vibrant space filled with art installations and blooming flowers. The people of the city were enchanted as they discovered the park, and they soon started joining EZ-9 in his creative endeavors. Together, they brought life back to the forgotten space, reminding everyone of the beauty in collaboration and imagination.\n", + "\n", + "**Story #9:** Once upon a time, in a vast desert woven with golden sands, there was a wise camel named Khamar. Khamar had traveled across the lands, gathering tales from every traveler he encountered. One day, he came across a lost caravan, desperate for water and guidance. Khamar offered to lead them to an oasis, but first, he shared his stories of courage and friendship. Inspired by the tales, the caravan members found strength in one another, helping each other carry their burdens. When they finally reached the sparkling oasis, they were not just relieved by the water but also bonded by the power of shared stories. From that day on, Khamar became their storyteller, transforming every journey into an adventure filled with laughter and camaraderie.\n", + "\n", + "**Story #10:** Once upon a time, there was a giant tree known as Elderwood, whose roots spread deep into the earth and whose branches touched the sky. Beneath this extraordinary tree, a young girl named Aria discovered a hidden door carved into the trunk. Curious, she opened it and found herself in the Realm of Whispers, where the voices of history awaited her. Each whisper told a story of bravery, love, and wisdom from ages long past. Aria realized that she could share these tales with her village, keeping the wisdom alive. Returning home, she spoke of the stories, inspiring her people to value their heritage and cultivate a brighter future. Through her journeys in the tree, Aria became the bridge between the past and future, forever connected to the magic of Elderwood.\n" ] } ], "source": [ "num_stories = 10\n", - "prompts = [\"Once upon a time,\"] * num_stories\n", + "content = \"Once upon a time,\"\n", "\n", - "# batched example, with 10 stories completions per request\n", + "prompt_lines = [f\"Story #{i+1}: {content}\" for i in range(num_stories)]\n", + "prompt_text = \"\\n\".join(prompt_lines)\n", + "\n", + "messages = [\n", + " {\n", + " \"role\": \"developer\",\n", + " \"content\": \"You are a helpful assistant. Please respond to each prompt as a separate short story.\"\n", + " },\n", + " {\n", + " \"role\": \"user\",\n", + " \"content\": prompt_text\n", + " }\n", + "]\n", + "\n", + "# batched example, with all story completions in one request\n", "response = client.chat.completions.create(\n", - " model=\"curie\",\n", - " prompt=prompts,\n", - " max_tokens=20,\n", + " model=\"gpt-4o-mini\",\n", + " messages=messages\n", ")\n", "\n", - "# match completions to prompts by index\n", - "stories = [\"\"] * len(prompts)\n", - "for choice in response.choices:\n", - " stories[choice.index] = prompts[choice.index] + choice.text\n", + "print(response.choices[0].message.content)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Example batching multiple prompts in a single request with Structured Outputs" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "OpenAI's [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs) feature offers a more robust way to batch multiple prompts in a single request. \n", + "\n", + "Here, rather than parsing raw text or hoping the model follows informal formatting, you specify a strict schema. This ensures your application can reliably parse the results by examining the defined structure. This eliminates the need for extensive validation or complicated parsing logic, as Structured Outputs guarantees consistent, type-safe data." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\"stories\":[\"Once upon a time, in a quiet village nestled between rolling hills, there lived a kind-hearted baker named Elara. Every morning, the sweet aroma of fresh bread wafted through the streets, drawing villagers to her shop like bees to flowers. One day, an elderly woman entered, asking for bread but only having a few coins. Elara smiled, gave her the bread for free, and said, 'Take care of yourself, for kindness is the most valuable currency.' From that day on, the village thrived on love and generosity, thanks to one baker's warm heart.\",\"Once upon a time, in a dense, enchanted forest, a curious fox named Finn dreamed of flying. Every day, he would sit under a towering oak tree, watching the birds soar gracefully in the sky. One day, a wise old owl noticed Finn's longing and said, 'If you wish to fly, you must create your own wings.' Inspired, Finn gathered feathers and twigs, crafting a pair of colorful wings. With a little help from his friends, he leaped off a hill and soared into the sky, discovering that sometimes, dreams can come true in the most unexpected ways.\",\"Once upon a time, high atop a misty mountain, there was a kingdom where the stars were not just sights to behold but who everyone believed were distant relatives looking down on them. The people would light lanterns every night, believing it would guide their kin back home. One star, named Astra, noticed this and descended to the earth, taking the form of a beautiful woman. She revealed herself at the lantern festival and danced among the villagers, reminding them that their spirits lived on in the stars, forever connected despite the miles between them.\",\"Once upon a time, there was a brave little turtle named Tilly who lived in a pond surrounded by flourishing meadows. Tilly dreamed of seeing the world beyond her pond, but her friends told her it was too dangerous. Undeterred, she set off on an adventure, slowly but surely. Along her journey, she befriended a hesitant rabbit who later turned out to be a great help in escaping a tricky situation. They returned to the pond with tales of wonders and taught the other animals that courage exists even in the smallest of hearts.\",\"Once upon a time, in a bustling city filled with towering skyscrapers, a young girl named Zara discovered an old paintbrush in a thrift store. The moment she picked it up, colors danced around her, and the mundane world transformed into a vibrant canvas. Everything she painted became alive, bringing joy and wonder to those around her. However, a greedy businessman sought to control her gift. With the help of her friends, Zara spread beauty all over the city, proving that art could not be owned, only shared—to uplift spirits and fill hearts with hope.\",\"Once upon a time, deep in the ocean, there lived a little mermaid named Mira who collected artifacts from shipwrecks. Among her treasures was a small, rusted key that sparked her curiosity. One day, she discovered an ancient door hidden within a coral cave. Using the key, she unlocked a magical underwater realm filled with shimmering creatures and lost souls. Gaston, a seaweed wizard, explained that she had freed those trapped for centuries. Mira returned home, determined to explore and protect the ocean’s depths, becoming its guardian.\",\"Once upon a time, in a land where dreams sprinkled the air like glitter, a gentle giant named Hugo roamed the hills. While most humans feared him, a brave child named Elinor approached him with kindness and shared her dreams. Hugo, touched by her bravery, decided to help Elinor realize her biggest dream—to see the land beyond the mountains. With great care, he carried her on his shoulders, showing her wonders and basking in the sunshine, proving that friendship knows no bounds, no matter the size.\",\"Once upon a time, in a quaint little town, there was an old library that was rumored to be magical. Books there whispered secrets and stories when no one was listening. A shy young boy named Leo discovered this secret during a stormy afternoon. Each night, he returned to read, and soon the book characters began to emerge, inviting him into their adventures. Leo, who once felt invisible, became a hero in this enchanted world, discovering the importance of stories and imagination and the power they held to change lives.\",\"Once upon a time, a lonely little planet named Zyra spun in solitude, far from the stars. One day, a comet zipped past, and Zyra longed to ask it to stay. The comet, noticing Zyra's glow dimming with sadness, decided to help. It brought star seeds from distant galaxies, filling Zyra with life and color. With each seed planted, new friends arrived—trees, crystalline lakes, and joyful creatures. Zyra learned that sometimes, all it takes to shine bright is to seek connections, no matter how distant they may seem.\",\"Once upon a time, in a sprawling desert filled with golden sands, there was a mystical oasis known to grant one wish to anyone who could find it. A clever merchant named Amir searched tirelessly for this oasis, guided by dreams and whispers of the wind. When he finally found it, he wished not for riches or power, but for the ability to bring peace to his land. His wish echoed across the dunes, and as the winds changed, they carried his message of hope, leading his people to unite, discovering that the true treasure was in unity and love.\"]}\n" + ] + } + ], + "source": [ + "from pydantic import BaseModel\n", + "\n", + "# Define the Pydantic model for the structured output\n", + "class StoryResponse(BaseModel):\n", + " stories: list[str]\n", "\n", - "# print stories\n", - "for story in stories:\n", - " print(story)\n" + "num_stories = 10\n", + "content = \"Once upon a time,\"\n", + "\n", + "prompt_lines = [f\"Story #{i+1}: {content}\" for i in range(num_stories)]\n", + "prompt_text = \"\\n\".join(prompt_lines)\n", + "\n", + "messages = [\n", + " {\n", + " \"role\": \"developer\",\n", + " \"content\": \"You are a helpful assistant. Please respond to each prompt as a separate short story.\"\n", + " },\n", + " {\n", + " \"role\": \"user\",\n", + " \"content\": prompt_text\n", + " }\n", + "]\n", + "\n", + "# batched example, with all story completions in one request and using structured output\n", + "response = client.beta.chat.completions.parse(\n", + " model=\"gpt-4o-mini\",\n", + " messages=messages,\n", + " response_format=StoryResponse,\n", + ")\n", + "\n", + "print(response.choices[0].message.content)" ] }, { @@ -465,7 +620,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": ".venv", "language": "python", "name": "python3" }, @@ -479,14 +634,9 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.8" + "version": "3.9.6" }, - "orig_nbformat": 4, - "vscode": { - "interpreter": { - "hash": "365536dcbde60510dc9073d6b991cd35db2d9bac356a11f5b64279a5e6708b97" - } - } + "orig_nbformat": 4 }, "nbformat": 4, "nbformat_minor": 2 diff --git a/registry.yaml b/registry.yaml index d6530c174c..0682998316 100644 --- a/registry.yaml +++ b/registry.yaml @@ -201,6 +201,7 @@ date: 2022-09-10 authors: - ted-at-openai + - dwigg-openai tags: - completions - embeddings From 49ed9a9e3be6a0fc64bc7ea193b397d3ce8c789d Mon Sep 17 00:00:00 2001 From: Danny Wigg Date: Wed, 22 Jan 2025 15:43:22 +0000 Subject: [PATCH 2/3] Small improvements throughout --- examples/How_to_handle_rate_limits.ipynb | 102 ++++++----------------- 1 file changed, 27 insertions(+), 75 deletions(-) diff --git a/examples/How_to_handle_rate_limits.ipynb b/examples/How_to_handle_rate_limits.ipynb index 5aa355fac4..4cf70ac071 100644 --- a/examples/How_to_handle_rate_limits.ipynb +++ b/examples/How_to_handle_rate_limits.ipynb @@ -48,7 +48,7 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 3, "metadata": {}, "outputs": [], "source": [ @@ -75,9 +75,30 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 2, "metadata": {}, - "outputs": [], + "outputs": [ + { + "ename": "RateLimitError", + "evalue": "Error code: 429 - {'error': {'message': 'You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.', 'type': 'insufficient_quota', 'param': None, 'code': 'insufficient_quota'}}", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mRateLimitError\u001b[0m Traceback (most recent call last)", + "Cell \u001b[0;32mIn[2], line 3\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[38;5;66;03m# request a bunch of completions in a loop\u001b[39;00m\n\u001b[1;32m 2\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m _ \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mrange\u001b[39m(\u001b[38;5;241m100\u001b[39m):\n\u001b[0;32m----> 3\u001b[0m \u001b[43mclient\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mchat\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcompletions\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcreate\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 4\u001b[0m \u001b[43m \u001b[49m\u001b[43mmodel\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mgpt-4o-mini\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5\u001b[0m \u001b[43m \u001b[49m\u001b[43mmessages\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m[\u001b[49m\u001b[43m{\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mrole\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43muser\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mcontent\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mHello\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m}\u001b[49m\u001b[43m]\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 6\u001b[0m \u001b[43m \u001b[49m\u001b[43mmax_tokens\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m10\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m 7\u001b[0m \u001b[43m \u001b[49m\u001b[43m)\u001b[49m\n", + "File \u001b[0;32m~/code/openai-cookbook/.venv/lib/python3.9/site-packages/openai/_utils/_utils.py:279\u001b[0m, in \u001b[0;36mrequired_args..inner..wrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 277\u001b[0m msg \u001b[38;5;241m=\u001b[39m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mMissing required argument: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mquote(missing[\u001b[38;5;241m0\u001b[39m])\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 278\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mTypeError\u001b[39;00m(msg)\n\u001b[0;32m--> 279\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfunc\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n", + "File \u001b[0;32m~/code/openai-cookbook/.venv/lib/python3.9/site-packages/openai/resources/chat/completions.py:859\u001b[0m, in \u001b[0;36mCompletions.create\u001b[0;34m(self, messages, model, audio, frequency_penalty, function_call, functions, logit_bias, logprobs, max_completion_tokens, max_tokens, metadata, modalities, n, parallel_tool_calls, prediction, presence_penalty, reasoning_effort, response_format, seed, service_tier, stop, store, stream, stream_options, temperature, tool_choice, tools, top_logprobs, top_p, user, extra_headers, extra_query, extra_body, timeout)\u001b[0m\n\u001b[1;32m 817\u001b[0m \u001b[38;5;129m@required_args\u001b[39m([\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmessages\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmodel\u001b[39m\u001b[38;5;124m\"\u001b[39m], [\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmessages\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmodel\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mstream\u001b[39m\u001b[38;5;124m\"\u001b[39m])\n\u001b[1;32m 818\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21mcreate\u001b[39m(\n\u001b[1;32m 819\u001b[0m \u001b[38;5;28mself\u001b[39m,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 856\u001b[0m timeout: \u001b[38;5;28mfloat\u001b[39m \u001b[38;5;241m|\u001b[39m httpx\u001b[38;5;241m.\u001b[39mTimeout \u001b[38;5;241m|\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;241m|\u001b[39m NotGiven \u001b[38;5;241m=\u001b[39m NOT_GIVEN,\n\u001b[1;32m 857\u001b[0m ) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m ChatCompletion \u001b[38;5;241m|\u001b[39m Stream[ChatCompletionChunk]:\n\u001b[1;32m 858\u001b[0m validate_response_format(response_format)\n\u001b[0;32m--> 859\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_post\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 860\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43m/chat/completions\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m 861\u001b[0m \u001b[43m \u001b[49m\u001b[43mbody\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmaybe_transform\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 862\u001b[0m \u001b[43m \u001b[49m\u001b[43m{\u001b[49m\n\u001b[1;32m 863\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mmessages\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mmessages\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 864\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mmodel\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mmodel\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 865\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43maudio\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43maudio\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 866\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mfrequency_penalty\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mfrequency_penalty\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 867\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mfunction_call\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mfunction_call\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 868\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mfunctions\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mfunctions\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 869\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mlogit_bias\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mlogit_bias\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 870\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mlogprobs\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mlogprobs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 871\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mmax_completion_tokens\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mmax_completion_tokens\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 872\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mmax_tokens\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mmax_tokens\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 873\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mmetadata\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mmetadata\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 874\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mmodalities\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mmodalities\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 875\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mn\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mn\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 876\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mparallel_tool_calls\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mparallel_tool_calls\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 877\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mprediction\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mprediction\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 878\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mpresence_penalty\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mpresence_penalty\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 879\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mreasoning_effort\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mreasoning_effort\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 880\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mresponse_format\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mresponse_format\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 881\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mseed\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mseed\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 882\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mservice_tier\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mservice_tier\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 883\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mstop\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mstop\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 884\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mstore\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mstore\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 885\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mstream\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mstream\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 886\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mstream_options\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mstream_options\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 887\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mtemperature\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mtemperature\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 888\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mtool_choice\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mtool_choice\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 889\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mtools\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mtools\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 890\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mtop_logprobs\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mtop_logprobs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 891\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mtop_p\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mtop_p\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 892\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43muser\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43muser\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 893\u001b[0m \u001b[43m \u001b[49m\u001b[43m}\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 894\u001b[0m \u001b[43m \u001b[49m\u001b[43mcompletion_create_params\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mCompletionCreateParams\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 895\u001b[0m \u001b[43m \u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 896\u001b[0m \u001b[43m \u001b[49m\u001b[43moptions\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmake_request_options\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 897\u001b[0m \u001b[43m \u001b[49m\u001b[43mextra_headers\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mextra_headers\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mextra_query\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mextra_query\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mextra_body\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mextra_body\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtimeout\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtimeout\u001b[49m\n\u001b[1;32m 898\u001b[0m \u001b[43m \u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 899\u001b[0m \u001b[43m \u001b[49m\u001b[43mcast_to\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mChatCompletion\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 900\u001b[0m \u001b[43m \u001b[49m\u001b[43mstream\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;129;43;01mor\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m 901\u001b[0m \u001b[43m \u001b[49m\u001b[43mstream_cls\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mStream\u001b[49m\u001b[43m[\u001b[49m\u001b[43mChatCompletionChunk\u001b[49m\u001b[43m]\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 902\u001b[0m \u001b[43m \u001b[49m\u001b[43m)\u001b[49m\n", + "File \u001b[0;32m~/code/openai-cookbook/.venv/lib/python3.9/site-packages/openai/_base_client.py:1283\u001b[0m, in \u001b[0;36mSyncAPIClient.post\u001b[0;34m(self, path, cast_to, body, options, files, stream, stream_cls)\u001b[0m\n\u001b[1;32m 1269\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21mpost\u001b[39m(\n\u001b[1;32m 1270\u001b[0m \u001b[38;5;28mself\u001b[39m,\n\u001b[1;32m 1271\u001b[0m path: \u001b[38;5;28mstr\u001b[39m,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 1278\u001b[0m stream_cls: \u001b[38;5;28mtype\u001b[39m[_StreamT] \u001b[38;5;241m|\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m,\n\u001b[1;32m 1279\u001b[0m ) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m ResponseT \u001b[38;5;241m|\u001b[39m _StreamT:\n\u001b[1;32m 1280\u001b[0m opts \u001b[38;5;241m=\u001b[39m FinalRequestOptions\u001b[38;5;241m.\u001b[39mconstruct(\n\u001b[1;32m 1281\u001b[0m method\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mpost\u001b[39m\u001b[38;5;124m\"\u001b[39m, url\u001b[38;5;241m=\u001b[39mpath, json_data\u001b[38;5;241m=\u001b[39mbody, files\u001b[38;5;241m=\u001b[39mto_httpx_files(files), \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39moptions\n\u001b[1;32m 1282\u001b[0m )\n\u001b[0;32m-> 1283\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m cast(ResponseT, \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrequest\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcast_to\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mopts\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mstream\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mstream_cls\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream_cls\u001b[49m\u001b[43m)\u001b[49m)\n", + "File \u001b[0;32m~/code/openai-cookbook/.venv/lib/python3.9/site-packages/openai/_base_client.py:960\u001b[0m, in \u001b[0;36mSyncAPIClient.request\u001b[0;34m(self, cast_to, options, remaining_retries, stream, stream_cls)\u001b[0m\n\u001b[1;32m 957\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 958\u001b[0m retries_taken \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m0\u001b[39m\n\u001b[0;32m--> 960\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_request\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 961\u001b[0m \u001b[43m \u001b[49m\u001b[43mcast_to\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcast_to\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 962\u001b[0m \u001b[43m \u001b[49m\u001b[43moptions\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43moptions\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 963\u001b[0m \u001b[43m \u001b[49m\u001b[43mstream\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 964\u001b[0m \u001b[43m \u001b[49m\u001b[43mstream_cls\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream_cls\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 965\u001b[0m \u001b[43m \u001b[49m\u001b[43mretries_taken\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mretries_taken\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 966\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n", + "File \u001b[0;32m~/code/openai-cookbook/.venv/lib/python3.9/site-packages/openai/_base_client.py:1049\u001b[0m, in \u001b[0;36mSyncAPIClient._request\u001b[0;34m(self, cast_to, options, retries_taken, stream, stream_cls)\u001b[0m\n\u001b[1;32m 1047\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m remaining_retries \u001b[38;5;241m>\u001b[39m \u001b[38;5;241m0\u001b[39m \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_should_retry(err\u001b[38;5;241m.\u001b[39mresponse):\n\u001b[1;32m 1048\u001b[0m err\u001b[38;5;241m.\u001b[39mresponse\u001b[38;5;241m.\u001b[39mclose()\n\u001b[0;32m-> 1049\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_retry_request\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 1050\u001b[0m \u001b[43m \u001b[49m\u001b[43minput_options\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1051\u001b[0m \u001b[43m \u001b[49m\u001b[43mcast_to\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1052\u001b[0m \u001b[43m \u001b[49m\u001b[43mretries_taken\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mretries_taken\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1053\u001b[0m \u001b[43m \u001b[49m\u001b[43mresponse_headers\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43merr\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mresponse\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mheaders\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1054\u001b[0m \u001b[43m \u001b[49m\u001b[43mstream\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1055\u001b[0m \u001b[43m \u001b[49m\u001b[43mstream_cls\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream_cls\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1056\u001b[0m \u001b[43m \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 1058\u001b[0m \u001b[38;5;66;03m# If the response is streamed then we need to explicitly read the response\u001b[39;00m\n\u001b[1;32m 1059\u001b[0m \u001b[38;5;66;03m# to completion before attempting to access the response text.\u001b[39;00m\n\u001b[1;32m 1060\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m err\u001b[38;5;241m.\u001b[39mresponse\u001b[38;5;241m.\u001b[39mis_closed:\n", + "File \u001b[0;32m~/code/openai-cookbook/.venv/lib/python3.9/site-packages/openai/_base_client.py:1098\u001b[0m, in \u001b[0;36mSyncAPIClient._retry_request\u001b[0;34m(self, options, cast_to, retries_taken, response_headers, stream, stream_cls)\u001b[0m\n\u001b[1;32m 1094\u001b[0m \u001b[38;5;66;03m# In a synchronous context we are blocking the entire thread. Up to the library user to run the client in a\u001b[39;00m\n\u001b[1;32m 1095\u001b[0m \u001b[38;5;66;03m# different thread if necessary.\u001b[39;00m\n\u001b[1;32m 1096\u001b[0m time\u001b[38;5;241m.\u001b[39msleep(timeout)\n\u001b[0;32m-> 1098\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_request\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 1099\u001b[0m \u001b[43m \u001b[49m\u001b[43moptions\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43moptions\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1100\u001b[0m \u001b[43m \u001b[49m\u001b[43mcast_to\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcast_to\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1101\u001b[0m \u001b[43m \u001b[49m\u001b[43mretries_taken\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mretries_taken\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m+\u001b[39;49m\u001b[43m \u001b[49m\u001b[38;5;241;43m1\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1102\u001b[0m \u001b[43m \u001b[49m\u001b[43mstream\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1103\u001b[0m \u001b[43m \u001b[49m\u001b[43mstream_cls\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream_cls\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1104\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n", + "File \u001b[0;32m~/code/openai-cookbook/.venv/lib/python3.9/site-packages/openai/_base_client.py:1049\u001b[0m, in \u001b[0;36mSyncAPIClient._request\u001b[0;34m(self, cast_to, options, retries_taken, stream, stream_cls)\u001b[0m\n\u001b[1;32m 1047\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m remaining_retries \u001b[38;5;241m>\u001b[39m \u001b[38;5;241m0\u001b[39m \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_should_retry(err\u001b[38;5;241m.\u001b[39mresponse):\n\u001b[1;32m 1048\u001b[0m err\u001b[38;5;241m.\u001b[39mresponse\u001b[38;5;241m.\u001b[39mclose()\n\u001b[0;32m-> 1049\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_retry_request\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 1050\u001b[0m \u001b[43m \u001b[49m\u001b[43minput_options\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1051\u001b[0m \u001b[43m \u001b[49m\u001b[43mcast_to\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1052\u001b[0m \u001b[43m \u001b[49m\u001b[43mretries_taken\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mretries_taken\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1053\u001b[0m \u001b[43m \u001b[49m\u001b[43mresponse_headers\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43merr\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mresponse\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mheaders\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1054\u001b[0m \u001b[43m \u001b[49m\u001b[43mstream\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1055\u001b[0m \u001b[43m \u001b[49m\u001b[43mstream_cls\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream_cls\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1056\u001b[0m \u001b[43m \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 1058\u001b[0m \u001b[38;5;66;03m# If the response is streamed then we need to explicitly read the response\u001b[39;00m\n\u001b[1;32m 1059\u001b[0m \u001b[38;5;66;03m# to completion before attempting to access the response text.\u001b[39;00m\n\u001b[1;32m 1060\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m err\u001b[38;5;241m.\u001b[39mresponse\u001b[38;5;241m.\u001b[39mis_closed:\n", + "File \u001b[0;32m~/code/openai-cookbook/.venv/lib/python3.9/site-packages/openai/_base_client.py:1098\u001b[0m, in \u001b[0;36mSyncAPIClient._retry_request\u001b[0;34m(self, options, cast_to, retries_taken, response_headers, stream, stream_cls)\u001b[0m\n\u001b[1;32m 1094\u001b[0m \u001b[38;5;66;03m# In a synchronous context we are blocking the entire thread. Up to the library user to run the client in a\u001b[39;00m\n\u001b[1;32m 1095\u001b[0m \u001b[38;5;66;03m# different thread if necessary.\u001b[39;00m\n\u001b[1;32m 1096\u001b[0m time\u001b[38;5;241m.\u001b[39msleep(timeout)\n\u001b[0;32m-> 1098\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_request\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 1099\u001b[0m \u001b[43m \u001b[49m\u001b[43moptions\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43moptions\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1100\u001b[0m \u001b[43m \u001b[49m\u001b[43mcast_to\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcast_to\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1101\u001b[0m \u001b[43m \u001b[49m\u001b[43mretries_taken\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mretries_taken\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m+\u001b[39;49m\u001b[43m \u001b[49m\u001b[38;5;241;43m1\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1102\u001b[0m \u001b[43m \u001b[49m\u001b[43mstream\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1103\u001b[0m \u001b[43m \u001b[49m\u001b[43mstream_cls\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream_cls\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1104\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n", + "File \u001b[0;32m~/code/openai-cookbook/.venv/lib/python3.9/site-packages/openai/_base_client.py:1064\u001b[0m, in \u001b[0;36mSyncAPIClient._request\u001b[0;34m(self, cast_to, options, retries_taken, stream, stream_cls)\u001b[0m\n\u001b[1;32m 1061\u001b[0m err\u001b[38;5;241m.\u001b[39mresponse\u001b[38;5;241m.\u001b[39mread()\n\u001b[1;32m 1063\u001b[0m log\u001b[38;5;241m.\u001b[39mdebug(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mRe-raising status error\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m-> 1064\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_make_status_error_from_response(err\u001b[38;5;241m.\u001b[39mresponse) \u001b[38;5;28;01mfrom\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 1066\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_process_response(\n\u001b[1;32m 1067\u001b[0m cast_to\u001b[38;5;241m=\u001b[39mcast_to,\n\u001b[1;32m 1068\u001b[0m options\u001b[38;5;241m=\u001b[39moptions,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 1072\u001b[0m retries_taken\u001b[38;5;241m=\u001b[39mretries_taken,\n\u001b[1;32m 1073\u001b[0m )\n", + "\u001b[0;31mRateLimitError\u001b[0m: Error code: 429 - {'error': {'message': 'You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.', 'type': 'insufficient_quota', 'param': None, 'code': 'insufficient_quota'}}" + ] + } + ], "source": [ "# request a bunch of completions in a loop\n", "for _ in range(100):\n", @@ -282,7 +303,7 @@ "\n", "If you encounter rate limit errors on your primary model, one option is to switch to a secondary model, often one that’s less powerful or more cost-effective. This approach helps keep your application responsive when your primary model is throttled or unavailable.\n", "\n", - "However, fallback models can differ significantly in accuracy, latency, and cost. As a result, this strategy might not work for every use case; particularly those requiring highly consistent results. Additionally, keep in mind that some models share rate limits, which may reduce the effectiveness of simply switching models.\n", + "However, fallback models can differ significantly in accuracy, latency, and cost. As a result, this strategy might not work for every use case; particularly those requiring highly consistent results. Additionally, keep in mind that some models share rate limits, which may reduce the effectiveness of simply switching models. You can see the models that share limits in your [organizations limit page](https://platform.openai.com/settings/organization/limits).\n", "\n", "Before deploying this approach to production, thoroughly test how it affects output quality, user experience, and operational budgets. Validate your fallback solution with relevant evaluations to ensure it meets your requirements and maintains acceptable performance under real-world conditions." ] @@ -466,75 +487,6 @@ " print(content + response.choices[0].message.content)\n" ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Example batching multiple prompts in a single request\n", - "\n", - "This approach packs several prompts into a single request, resulting in a single response from the model that addresses each prompt in turn.\n", - "\n", - "While this method is efficient in terms of API calls, it may require extra parsing or post-processing on your end to separate the responses. Additionally, there is a risk that the model’s output will “blend” the prompts if it isn’t guided clearly, so be sure to instruct it explicitly on how to structure its answers." - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "**Story #1:** Once upon a time, in a small village nestled between lush green hills, there lived a young girl named Clara. Clara had a gift for talking to animals. One day, while wandering through the forest, she heard a soft whimpering sound. Following the noise, she discovered a tiny fox caught in a hunter's snare. Clara gently freed the fox and, in gratitude, it promised to grant her one wish. Clara wished for the village to always have enough food and happiness. From that day on, whenever a storm threatened their harvest, the fox would bring rainbows that promised bountiful crops and joyful hearts.\n", - "\n", - "**Story #2:** Once upon a time, in a kingdom where the sky was painted in eternal twilight, lived a boy named Leo who adored the stars. Every night, he would climb a hill and gaze at them, dreaming of discovering their secrets. One fateful night, a shooting star fell close by. Leo ran to it and found a small, glowing orb. It was a fallen star, and it spoke to him! The star revealed that every star had a story, but only those with pure hearts could hear them. With the star as his guide, Leo traveled across the sky, learning the tales of constellations and sharing them with his village, igniting a love for the cosmos among its people.\n", - "\n", - "**Story #3:** Once upon a time, in a forgotten land where time stood still, there was a magical clocktower that had the power to control time itself. The clockkeeper, an old man named Silas, was the only one who understood its mysteries. One day, a curious girl named Elara stumbled upon the tower, intrigued by its intricate designs and the enchanting sounds that echoed from within. Silas decided to teach her the ways of time. Elara learned that every tick of the clock affected the world outside. Together, they saved a village from a disaster by rewinding time just long enough to warn the villagers. In gratitude, the townspeople honored Elara as the \"Keeper of Moments,\" reminding them that every second is precious.\n", - "\n", - "**Story #4:** Once upon a time, in a world where emotions could be seen as colors, there was a young artist named Mia. Mia had the unique ability to paint her feelings, which would bring joy to those who gazed upon her artwork. One day, she encountered a clouded figure in the marketplace, shrouded in gray and sorrow. Determined to bring color back to their life, Mia invited the figure to her studio. As she painted, her brushstrokes began to lift the gloom, revealing beautiful colors hidden beneath. With each layer, the figure began to smile. Soon, they transformed into a vibrant rainbow, spreading happiness throughout the world, reminding everyone that we can share our colors with others.\n", - "\n", - "**Story #5:** Once upon a time, in a distant ocean, there was a graceful mermaid named Luna who longed to see the land above the waves. One evening, she swam close to the surface and witnessed a ship sailing on the horizon. Curious, she swam alongside it and overheard the sailors singing songs of adventure and freedom. Captivated, Luna decided to explore the surface world. With the help of a kind sailor, she transformed into a human for one night. They danced under the moonlight, sharing dreams of the sea and sky. As dawn approached, Luna returned to the ocean, but she left behind a shell that played their song, reminding them of their magical encounter and the bond between land and sea.\n", - "\n", - "**Story #6:** Once upon a time, in a realm of enchanted forests, there lived a wise old owl named Olwen. Every creature came to him for advice, but one day, a young squirrel named Pip approached him, desperate for help. Pip had lost his way in the vast woods and could not find his family. Olwen offered to guide him, but only if Pip could solve one riddle. With determination, Pip listened closely and thought hard. After several attempts, he finally answered the riddle correctly, impressing the wise owl. Olwen led Pip through the mystical forest, and with each step, Pip discovered the beauty of his surroundings. They finally reached his family, and Pip learned that sometimes, getting lost is simply a path to finding oneself.\n", - "\n", - "**Story #7:** Once upon a time, in a land where dreams floated like clouds in the sky, there lived a young boy named Finn. Each night, he would catch dreams and weave them into tapestries, bringing hope and joy to his village. One night, Finn discovered a dark dream threatening to engulf the others. Instead of capturing it, he decided to understand it. He entered the dark dream and discovered it was a reflection of fear and pain. Rather than fight it, he shared stories of courage and love, transforming the dark dream into a beautiful tapestry of light. Finn returned home, and from that night on, his village learned that embracing even the darkest dreams could lead to the most beautiful transformations.\n", - "\n", - "**Story #8:** Once upon a time, in a bustling city of towering skyscrapers, there lived a little robot named EZ-9. EZ-9 was designed to assist people, but he felt an unquenchable yearning to create rather than just serve. One day, he stumbled upon a forgotten park, overgrown with weeds and neglected. Inspired, EZ-9 began to DIY—using scrap metal and old parts, he transformed the park into a vibrant space filled with art installations and blooming flowers. The people of the city were enchanted as they discovered the park, and they soon started joining EZ-9 in his creative endeavors. Together, they brought life back to the forgotten space, reminding everyone of the beauty in collaboration and imagination.\n", - "\n", - "**Story #9:** Once upon a time, in a vast desert woven with golden sands, there was a wise camel named Khamar. Khamar had traveled across the lands, gathering tales from every traveler he encountered. One day, he came across a lost caravan, desperate for water and guidance. Khamar offered to lead them to an oasis, but first, he shared his stories of courage and friendship. Inspired by the tales, the caravan members found strength in one another, helping each other carry their burdens. When they finally reached the sparkling oasis, they were not just relieved by the water but also bonded by the power of shared stories. From that day on, Khamar became their storyteller, transforming every journey into an adventure filled with laughter and camaraderie.\n", - "\n", - "**Story #10:** Once upon a time, there was a giant tree known as Elderwood, whose roots spread deep into the earth and whose branches touched the sky. Beneath this extraordinary tree, a young girl named Aria discovered a hidden door carved into the trunk. Curious, she opened it and found herself in the Realm of Whispers, where the voices of history awaited her. Each whisper told a story of bravery, love, and wisdom from ages long past. Aria realized that she could share these tales with her village, keeping the wisdom alive. Returning home, she spoke of the stories, inspiring her people to value their heritage and cultivate a brighter future. Through her journeys in the tree, Aria became the bridge between the past and future, forever connected to the magic of Elderwood.\n" - ] - } - ], - "source": [ - "num_stories = 10\n", - "content = \"Once upon a time,\"\n", - "\n", - "prompt_lines = [f\"Story #{i+1}: {content}\" for i in range(num_stories)]\n", - "prompt_text = \"\\n\".join(prompt_lines)\n", - "\n", - "messages = [\n", - " {\n", - " \"role\": \"developer\",\n", - " \"content\": \"You are a helpful assistant. Please respond to each prompt as a separate short story.\"\n", - " },\n", - " {\n", - " \"role\": \"user\",\n", - " \"content\": prompt_text\n", - " }\n", - "]\n", - "\n", - "# batched example, with all story completions in one request\n", - "response = client.chat.completions.create(\n", - " model=\"gpt-4o-mini\",\n", - " messages=messages\n", - ")\n", - "\n", - "print(response.choices[0].message.content)" - ] - }, { "cell_type": "markdown", "metadata": {}, @@ -546,7 +498,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "OpenAI's [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs) feature offers a more robust way to batch multiple prompts in a single request. \n", + "OpenAI's [Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs) feature offers a robust way to batch multiple prompts in a single request. \n", "\n", "Here, rather than parsing raw text or hoping the model follows informal formatting, you specify a strict schema. This ensures your application can reliably parse the results by examining the defined structure. This eliminates the need for extensive validation or complicated parsing logic, as Structured Outputs guarantees consistent, type-safe data." ] @@ -588,7 +540,7 @@ " }\n", "]\n", "\n", - "# batched example, with all story completions in one request and using structured output\n", + "# batched example, with all story completions in one request and using structured outputs\n", "response = client.beta.chat.completions.parse(\n", " model=\"gpt-4o-mini\",\n", " messages=messages,\n", From c923fd7a351a5507ec56606a1603241717124064 Mon Sep 17 00:00:00 2001 From: Danny Wigg Date: Wed, 22 Jan 2025 15:49:24 +0000 Subject: [PATCH 3/3] Final improvements --- examples/How_to_handle_rate_limits.ipynb | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/examples/How_to_handle_rate_limits.ipynb b/examples/How_to_handle_rate_limits.ipynb index 4cf70ac071..9b65287eb2 100644 --- a/examples/How_to_handle_rate_limits.ipynb +++ b/examples/How_to_handle_rate_limits.ipynb @@ -301,7 +301,7 @@ "source": [ "### Backing off to another model\n", "\n", - "If you encounter rate limit errors on your primary model, one option is to switch to a secondary model, often one that’s less powerful or more cost-effective. This approach helps keep your application responsive when your primary model is throttled or unavailable.\n", + "If you encounter rate limit errors on your primary model, one option is to switch to a secondary model. This approach helps keep your application responsive when your primary model is throttled or unavailable.\n", "\n", "However, fallback models can differ significantly in accuracy, latency, and cost. As a result, this strategy might not work for every use case; particularly those requiring highly consistent results. Additionally, keep in mind that some models share rate limits, which may reduce the effectiveness of simply switching models. You can see the models that share limits in your [organizations limit page](https://platform.openai.com/settings/organization/limits).\n", "\n", @@ -316,7 +316,7 @@ { "data": { "text/plain": [ - "ChatCompletion(id='chatcmpl-ApgkrBU1L4JFy7Fb3m8YjZeDnX2T6', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content=\"there was a small, peaceful village nestled between towering mountains and lush, green forests. The villagers lived simple yet joyful lives, farming their land and tending to their animals. Among them was a young girl named Elara, who possessed an insatiable curiosity about the world beyond the mountains.\\n\\nElara had always been fascinated by the stories her grandmother told her about the legendary Phoenix, a mythical bird rumored to live in the heart of the mountains. According to the tales, the Phoenix was a majestic creature with feathers that shimmered like gold and eyes that glowed with the warmth of a thousand sunsets. It was said to possess the power of rebirth, rising anew from its own ashes, and that anyone who witnessed this miracle would be granted a single wish.\\n\\nDetermined to uncover the truth behind the legend, Elara set out on a journey to find the Phoenix. Armed with nothing but her grandmother's stories and her unwavering spirit, she ventured into the forest. The path was fraught with challenges—steep cliffs, raging rivers, and thick, tangled underbrush. Yet, Elara faced each obstacle with courage and determination.\\n\\nAs days turned into weeks, Elara's hope began to wane. Exhausted and weary, she reached a hidden valley bathed in the golden light of the setting sun. There, to her astonishment, she found a nest of radiant flames. Within the fire, the Phoenix emerged, a breathtaking vision of beauty and power.\\n\\nElara watched in awe as the Phoenix spread its wings, casting a warm, golden glow over the valley. In that moment, she remembered her grandmother's words and whispered her wish—a wish not for herself, but for the prosperity and happiness of her village.\\n\\nThe Phoenix gazed at her with eyes full of understanding and, with a graceful lift, soared into the sky, leaving behind a single, glowing feather. Elara took the feather, feeling its warmth in her hands, and made her way back to her village.\\n\\nUpon her return, Elara found her village transformed. The fields were more fertile than ever, laughter echoed through the streets, and a sense of peace enveloped their lives. The villagers hailed Elara as a hero, a symbol of courage and selflessness.\\n\\nAnd so, Elara's journey came to an end, but her story lived on, a testament to the power of hope, bravery, and the magic that can be found when one dares to chase their dreams.\", refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None), internal_metrics=[{'cached_prompt_tokens': 0, 'total_accepted_tokens': 0, 'total_batched_tokens': 523, 'total_predicted_tokens': 0, 'total_rejected_tokens': 0, 'total_tokens_in_completion': 524, 'cached_embeddings_bytes': 0, 'cached_embeddings_n': 0, 'uncached_embeddings_bytes': 0, 'uncached_embeddings_n': 0, 'fetched_embeddings_bytes': 0, 'fetched_embeddings_n': 0, 'n_evictions': 0, 'sampling_steps': 496, 'sampling_steps_with_predictions': 0, 'batcher_ttft': 0.03744220733642578, 'batcher_initial_queue_time': 0.0005207061767578125}])], created=1736882441, model='gpt-4o-2024-08-06', object='chat.completion', service_tier='default', system_fingerprint='fp_4691090a87', usage=CompletionUsage(completion_tokens=496, prompt_tokens=12, total_tokens=508, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0, cached_tokens_internal=0)))" + "ChatCompletion(id='chatcmpl-AsX9Zts2toXoKA80ZujeWMXKMolBy', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='in a quaint little village nestled between lush green hills and sparkling blue rivers, there lived a young girl named Elara. Elara was known for her adventurous spirit and her unwavering curiosity about the world beyond her village. She often spent her days wandering the meadows, exploring the enchanted forest, and collecting wildflowers.\\n\\nOne sunny afternoon, while she was picking daisies near the edge of the forest, Elara stumbled upon an old, ornate key half-buried in the ground. Intrigued, she dusted it off and inspected it closely. The key was beautifully crafted, with intricate patterns carved into its metal. Elara felt a strange pull towards it, as if it were meant for her.\\n\\nDetermined to uncover its secrets, Elara ran back to the village, her heart racing with excitement. She gathered her closest friends—Jasper, a clever boy with a knack for puzzles, and Lila, a brave girl who loved to climb trees—and shared her discovery with them.\\n\\n\"Do you think it belongs to a hidden treasure?\" Jasper wondered, his eyes sparkling with mischief.\\n\\n\"Or perhaps a secret door!\" Lila added, her imagination running wild.\\n\\nTogether, they decided to seek out the source of the key. They combed through old tales told by the village elders, searching for any clues about a hidden door or treasure nearby. After days of excitement and exploration, they stumbled upon an ancient map tucked away in an old library. The map illustrated a long-lost castle deep within the enchanted forest, rumored to have been abandoned for centuries.\\n\\nWith the map in hand and their imaginations ignited, Elara, Jasper, and Lila set off towards the castle. The journey through the enchanted forest was filled with wonders—glowing fireflies, singing birds, and trees that seemed to whisper secrets as the wind rustled through their leaves. Eventually, they reached the castle, its crumbling walls draped in vines and mysterious shadows.\\n\\nStanding before the grand entrance, Elara held the key tightly in her hand. \"This is it,\" she whispered, her heart pounding in anticipation. The friends exchanged nervous glances but shared the thrill of adventure. Together, they pushed open the heavy door, which creaked eerily as it swung wide.\\n\\nInside, they found a majestic hall adorned with fading tapestries and dust-laden chandeliers. In the center of the room stood a locked chest, adorned with the same intricate patterns as the key. Elara knelt beside it, her friends gathering around as she inserted the key into the lock. With a satisfying click, the chest opened to reveal a trove of shimmering jewels, ancient scrolls, and forgotten treasures.\\n\\nBut among the riches, they discovered something even more valuable—an old book filled with stories of bravery, friendship, and magic. As they turned the pages, each story seemed to echo their own journey and the spirit of adventure that had led them to this moment.\\n\\nElara, Jasper, and Lila realized that the true treasure was not the jewels or gold, but the experiences they had shared and the bond they had formed through their journey. They decided to take the book back to their village and share its tales with everyone, inspiring others to seek their own adventures and explore the wonders of the world around them.\\n\\nFrom that day forward, the trio became known as the Keepers of the Forest, guardians of the stories that connected their village to the magic of the enchanted world. And as they continued their adventures, they learned that the real magic lay within their hearts and the friendships they cherished. \\n\\nAnd so, they lived happily ever after, their spirits forever intertwined in a tapestry of tales waiting to be told.', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None), internal_metrics=[{'cached_prompt_tokens': 0, 'total_accepted_tokens': 0, 'total_batched_tokens': 774, 'total_predicted_tokens': 0, 'total_rejected_tokens': 0, 'total_tokens_in_completion': 775, 'cached_embeddings_bytes': 0, 'cached_embeddings_n': 0, 'uncached_embeddings_bytes': 0, 'uncached_embeddings_n': 0, 'fetched_embeddings_bytes': 0, 'fetched_embeddings_n': 0, 'n_evictions': 0, 'sampling_steps': 747, 'sampling_steps_with_predictions': 0, 'batcher_ttft': 0.08919167518615723, 'batcher_initial_queue_time': 0.008681058883666992}])], created=1737560517, model='gpt-4o-mini-2024-07-18', object='chat.completion', service_tier='default', system_fingerprint='fp_72ed7ab54c', usage=CompletionUsage(completion_tokens=747, prompt_tokens=12, total_tokens=759, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0, cached_tokens_internal=0)))" ] }, "execution_count": 7, @@ -333,7 +333,7 @@ " return client.chat.completions.create(**kwargs)\n", " \n", " \n", - "completions_with_fallback(fallback_model=\"gpt-4o-mini\", model=\"gpt-4o\", messages=[{\"role\": \"user\", \"content\": \"Once upon a time,\"}])" + "completions_with_fallback(fallback_model=\"gpt-4o\", model=\"gpt-4o-mini\", messages=[{\"role\": \"user\", \"content\": \"Once upon a time,\"}])" ] }, { @@ -505,14 +505,14 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "{\"stories\":[\"Once upon a time, in a quiet village nestled between rolling hills, there lived a kind-hearted baker named Elara. Every morning, the sweet aroma of fresh bread wafted through the streets, drawing villagers to her shop like bees to flowers. One day, an elderly woman entered, asking for bread but only having a few coins. Elara smiled, gave her the bread for free, and said, 'Take care of yourself, for kindness is the most valuable currency.' From that day on, the village thrived on love and generosity, thanks to one baker's warm heart.\",\"Once upon a time, in a dense, enchanted forest, a curious fox named Finn dreamed of flying. Every day, he would sit under a towering oak tree, watching the birds soar gracefully in the sky. One day, a wise old owl noticed Finn's longing and said, 'If you wish to fly, you must create your own wings.' Inspired, Finn gathered feathers and twigs, crafting a pair of colorful wings. With a little help from his friends, he leaped off a hill and soared into the sky, discovering that sometimes, dreams can come true in the most unexpected ways.\",\"Once upon a time, high atop a misty mountain, there was a kingdom where the stars were not just sights to behold but who everyone believed were distant relatives looking down on them. The people would light lanterns every night, believing it would guide their kin back home. One star, named Astra, noticed this and descended to the earth, taking the form of a beautiful woman. She revealed herself at the lantern festival and danced among the villagers, reminding them that their spirits lived on in the stars, forever connected despite the miles between them.\",\"Once upon a time, there was a brave little turtle named Tilly who lived in a pond surrounded by flourishing meadows. Tilly dreamed of seeing the world beyond her pond, but her friends told her it was too dangerous. Undeterred, she set off on an adventure, slowly but surely. Along her journey, she befriended a hesitant rabbit who later turned out to be a great help in escaping a tricky situation. They returned to the pond with tales of wonders and taught the other animals that courage exists even in the smallest of hearts.\",\"Once upon a time, in a bustling city filled with towering skyscrapers, a young girl named Zara discovered an old paintbrush in a thrift store. The moment she picked it up, colors danced around her, and the mundane world transformed into a vibrant canvas. Everything she painted became alive, bringing joy and wonder to those around her. However, a greedy businessman sought to control her gift. With the help of her friends, Zara spread beauty all over the city, proving that art could not be owned, only shared—to uplift spirits and fill hearts with hope.\",\"Once upon a time, deep in the ocean, there lived a little mermaid named Mira who collected artifacts from shipwrecks. Among her treasures was a small, rusted key that sparked her curiosity. One day, she discovered an ancient door hidden within a coral cave. Using the key, she unlocked a magical underwater realm filled with shimmering creatures and lost souls. Gaston, a seaweed wizard, explained that she had freed those trapped for centuries. Mira returned home, determined to explore and protect the ocean’s depths, becoming its guardian.\",\"Once upon a time, in a land where dreams sprinkled the air like glitter, a gentle giant named Hugo roamed the hills. While most humans feared him, a brave child named Elinor approached him with kindness and shared her dreams. Hugo, touched by her bravery, decided to help Elinor realize her biggest dream—to see the land beyond the mountains. With great care, he carried her on his shoulders, showing her wonders and basking in the sunshine, proving that friendship knows no bounds, no matter the size.\",\"Once upon a time, in a quaint little town, there was an old library that was rumored to be magical. Books there whispered secrets and stories when no one was listening. A shy young boy named Leo discovered this secret during a stormy afternoon. Each night, he returned to read, and soon the book characters began to emerge, inviting him into their adventures. Leo, who once felt invisible, became a hero in this enchanted world, discovering the importance of stories and imagination and the power they held to change lives.\",\"Once upon a time, a lonely little planet named Zyra spun in solitude, far from the stars. One day, a comet zipped past, and Zyra longed to ask it to stay. The comet, noticing Zyra's glow dimming with sadness, decided to help. It brought star seeds from distant galaxies, filling Zyra with life and color. With each seed planted, new friends arrived—trees, crystalline lakes, and joyful creatures. Zyra learned that sometimes, all it takes to shine bright is to seek connections, no matter how distant they may seem.\",\"Once upon a time, in a sprawling desert filled with golden sands, there was a mystical oasis known to grant one wish to anyone who could find it. A clever merchant named Amir searched tirelessly for this oasis, guided by dreams and whispers of the wind. When he finally found it, he wished not for riches or power, but for the ability to bring peace to his land. His wish echoed across the dunes, and as the winds changed, they carried his message of hope, leading his people to unite, discovering that the true treasure was in unity and love.\"]}\n" + "{\"stories\":[\"Once upon a time, in a lush green valley, there lived a curious little fox named Felix. Every day, he would explore the woods, finding hidden glades and sparkling streams. One day, while chasing a butterfly, he stumbled upon a magical oak tree that granted wishes. Felix wished for courage and became the bravest fox in the land, helping his friends as they faced challenges together.\",\"Once upon a time, a village known for its beautiful gardens fell into despair when a drought struck. The villagers prayed for rain but to no avail. One evening, a wise old woman arrived and told them of a hidden spring deep in the forest. With hope, the villagers embarked on a quest to find it, learning the value of teamwork and perseverance. Eventually, they found the spring, and the rain returned, reviving their gardens and spirits.\",\"Once upon a time, in a kingdom high atop the clouds, lived Princess Lumina who had the ability to control the stars. But she felt lonely and longed for a companion. One night, she captured a shooting star and transformed it into a dashing young man named Orion. Together, they painted the night skies with adventures until Lumina learned to find joy in her own light.\",\"Once upon a time, in a bustling bazaar in the heart of the city, there lived a clever merchant named Amina. She had a special talent for selling spices that made people fall in love. One day, a mysterious stranger entered her shop and bought a rare spice, causing an unexpected romance between two feuding families. Amina realized her spices held the power of unity, and she continued to spread love through her trade.\",\"Once upon a time, a little turtle named Tilly dreamed of flying. Every day she watched the birds soar above her, wishing she could join them. One night, she met an old owl who shared stories of how to fly in one's heart, rather than with wings. Inspired, Tilly began to paint her dreams on shells, and soon, her colorful art attracted the birds. They carried her art into the sky, proving that dreams can take flight in unexpected ways.\",\"Once upon a time, there was a forgotten castle hidden deep in the mountains. In this castle lived an ancient dragon named Ignis, who guarded a treasure of wisdom unlike any other. One day, a brave yet naive knight named Roland attempted to seize the treasure. But Ignis offered him a riddle instead. After solving it, Roland realized that the true treasure was knowledge and understanding. He left the castle as a wiser man, sharing Ignis's teachings with his kingdom.\",\"Once upon a time, in a world where colors had feelings, there lived a dull gray town where nobody smiled. One day, a little girl named Bloom arrived, carrying a bright yellow paintbrush. She began to paint laughter and joy on the walls. Slowly, the townspeople found happiness in her colors and learned to express their emotions. Eventually, the town transformed into a vibrant place where every day was a celebration of life.\",\"Once upon a time, an old clockmaker named Mr. Tick was known for creating the finest clocks in the town. But his favorite creation was an enchanted clock that could tell stories of the past. One day, a little girl named Clara stumbled into his shop and begged him to tell her a story. Mr. Tick set the clock and took her on a journey through time, where Clara learned the importance of history and family. Inspired, she decided to become a storyteller.\",\"Once upon a time, in a small fishing village, a mysterious blue whale appeared off the coast every summer. Legend had it that the whale could grant one wish to the person who dared to swim alongside it. A daring young boy named Leo decided to brave the waters. As he swam next to the majestic creature, he wished for prosperity for his village. From that day onward, the village thrived, and they celebrated the bond of friendship with the whale every summer.\",\"Once upon a time, in a land of giants, there lived a tiny girl named Fiona. Despite her size, she had a heart full of ambition. She dreamt of building a bridge between her village and the giants’ realm to facilitate friendship. With determination and ingenuity, she crafted a plan. When the giants saw her efforts, they helped her, and together they constructed a magnificent bridge. Fiona's courage became a legend, and the two realms flourished in harmony.\"],\"story_count\":10}\n" ] } ], @@ -522,6 +522,7 @@ "# Define the Pydantic model for the structured output\n", "class StoryResponse(BaseModel):\n", " stories: list[str]\n", + " story_count: int\n", "\n", "num_stories = 10\n", "content = \"Once upon a time,\"\n",