Total token count of openai callback does not count embedding usage

[tmbo]

When using embeddings, the total_tokens count of a callback is wrong, e.g. the following example currently returns 0 even though it shouldn't:

from langchain.callbacks import get_openai_callback

with get_openai_callback() as cb:
    embeddings = OpenAIEmbeddings()
    embeddings.embed_query("helo")
    print(cb.total_tokens)

IMO this is confusing (and there is no way to get the cost from the embeddings class at the moment).

[hwchase17]

currently counting tokens via embedding is not supported - this is a good feature request though

[benheckmann]

I will start to work on this as a first issue. Will try to submit a draft in about two weeks. I am following the guidelines and hoping for a successful PR. :)

[thaiminhpv]

This is for those who are looking for a quick workaround solution to count tokens when waiting for the PR to be merged:

import tiktoken

def num_tokens_from_string(string: str) -> int:
    """Returns the number of tokens in a text string."""
    encoding = tiktoken.get_encoding("cl100k_base")
    num_tokens = len(encoding.encode(string))
    return num_tokens

num_tokens_from_string("tiktoken is great!")

Note that tiktoken is a library from OpenAI.

[anignx]

You have no idea how expensive a large file transfer can be

[zacharypodbela]

@thaiminhpv appreciate the workaround! Noob question:

If I'm doing something like...

db = Chroma.from_documents(texts, embeddings)

What would I pass into your function num_tokens_from_string? Iterate through texts and pass each one in? And those would just be the prompt tokens, right? How could I estimate the completion tokens used in the above call?

Thanks!

[thaiminhpv]

@zacharypodbela As far as I understand that

total token used = prompt tokens + completion tokens

In Embedding API models, completion tokens is always 0 so that completion tokens = total token used

So I think your approach is sufficient, and the completion tokens used is 0