Customizable Logit Warping Strategies for Generation #40010 #40403

PamelaBha · 2025-08-23T23:54:15Z

What does this PR do?

Feature request
Improve the generate() API by supporting custom, declarative logit warping strategies. Make it easier for users to plug in standard and custom LogitProcessors via configuration or arguments without needing to subclass or dive into internals.

Motivation
The generation module already supports rich logit manipulation through LogitProcessorList, but:

It is undocumented and hard to use for casual users
Requires advanced subclassing to customize behaviors (e.g., word bans, domain constraints)
Doesn’t support JSON- or dict-style configuration like many other parts of Transformers
Making logit warping more accessible enables:

Prompt engineers and power users to fine-tune generation behavior
Safer generation via blacklists or probability shifting
Dynamic controls like repetition penalties or temperature annealing

Fixes # (issue): 40010

Before submitting

[No ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ Yes] Did you read the contributor guideline,
Pull Request section?
[ Yes] Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case. Customizable Logit Warping Strategies for Generation #40010
[ Yes] Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
[ Yes] Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Rocketknight1 · 2025-08-25T14:11:21Z

cc @gante

gante

Very cool PR! I like where this feature is going 🔥

I've added a few comments, but I think they are simple to solve 🤗 Let me know if you have questions/counter-proposals regarding my comments.

Other global comments:

Missing: documentation of logits_processor in GenerationConfig. I would add a link to the example file inside the docs, your examples are quite clear!
Suggestion: in the documentation of logits_processor in generate(), I would add a mention about the new format (and a link to the examples)
Some classes should be directly imported from the top level, i.e. we should be able to do from transformers import LogitProcessorRegistry. This is done by adding the classes to the outermost __init__.py. I think the only class we need to add there is LogitProcessorRegistry (we can already to from transformers import LogitsProcessor)

.gitignore

gante · 2025-08-26T10:16:27Z

src/transformers/generation/configuration_utils.py

@@ -58,7 +58,16 @@


 if is_torch_available():
-    from .logits_process import SynthIDTextWatermarkLogitsProcessor, WatermarkLogitsProcessor
+    from ..cache_utils import (


(this diff seems related to a merge conflict, we can probably remove these added lines)

gante · 2025-08-26T10:18:00Z

src/transformers/generation/configuration_utils.py

+
+
+


Suggested change

gante · 2025-08-26T10:21:21Z

src/transformers/generation/configuration_utils.py

+        # Special handling for logit_processors - serialize to JSON string
+        if "logit_processors" in output and output["logit_processors"] is not None:
+            if not isinstance(output["logit_processors"], str):
+                # Convert to JSON string
+                output["logit_processors"] = json.dumps(output["logit_processors"])


Is this needed? if self.logit_processors is a list of dicts, then returning it directly should be fine :)

gante · 2025-08-26T10:25:27Z

src/transformers/generation/configuration_utils.py

@@ -1039,6 +1054,21 @@ def from_pretrained(
            config._original_object_hash = hash(config)  # Hash to detect whether the instance was modified
            return config

+    def get_logit_processors(self):


Can we move this function to utils.py?

Goal: let's keep classes as independent as possible -- the generation config doesn't need to know about the expected format that logit_processors takes in generate. It's mostly a data storage class.

gante · 2025-08-26T10:38:14Z

src/transformers/generation/logits_process.py

+
+# Enhanced LogitsProcessorList
+class ConfigurableLogitsProcessorList(LogitsProcessorList):
+    """Extended LogitsProcessorList that supports configuration-based construction."""


missing: an example in the docstring

gante · 2025-08-26T10:39:36Z

src/transformers/generation/utils.py

+
+        # Add any directly passed processors last
+        if logits_processor is not None:
+            processors.extend(logits_processor)


merging with the custom logits_processor already done in self._merge_criteria_processor_list call a few lines above

gante · 2025-08-26T10:40:22Z

src/transformers/generation/utils.py

+        if configured_processors is not None:
+            # Check for duplicates and warn
+            existing_types = {type(p).__name__ for p in processors}
+            config_types = {type(p).__name__ for p in configured_processors}
+            duplicates = existing_types & config_types
+
+            if duplicates:
+                warnings.warn(
+                    f"Duplicate LogitProcessors detected: {duplicates}. "
+                    f"Configured processors will be added after standard ones."
+                )
+
+            processors.extend(configured_processors)


Perhaps we can reuse the logic in _merge_criteria_processor_list ?

gante · 2025-08-26T10:41:45Z

tests/generation/test_logits_process.py

@@ -1391,3 +1403,148 @@ def check_eos_logits(out, logits, batch, channel, eos):
        check_eos_logits(out=out, logits=logits, batch=1, channel=0, eos=eos)
        self.assertTrue(delay_pattern_processor.active_batches.all())
        self.assertTrue((delay_pattern_processor.delay_pattern == torch.tensor(delay_pattern) - 1).all())
+
+class TestLogitProcessorRegistry(unittest.TestCase):


Let's have a single class for everything related to the configurable logits processors 🤗

gante · 2025-08-26T10:45:45Z

examples/pytorch/text-generation/logit_processors_example.py

+    from transformers.generation.logits_process import LogitsProcessor, LogitProcessorRegistry
+    import torch


all the imports in the example should be:
1 - at the top of the file
2 - transformers imports should be top-level imports, i.e. from transformers import LogitsProcessor, LogitProcessorRegistry. See the global PR comment I added about this 🤗

PamelaBha · 2025-08-26T16:42:09Z

Very cool PR! I like where this feature is going 🔥

I've added a few comments, but I think they are simple to solve 🤗 Let me know if you have questions/counter-proposals regarding my comments.

Other global comments:

Missing: documentation of logits_processor in GenerationConfig. I would add a link to the example file inside the docs, your examples are quite clear!

Suggestion: in the documentation of logits_processor in generate(), I would add a mention about the new format (and a link to the examples)

Some classes should be directly imported from the top level, i.e. we should be able to do from transformers import LogitProcessorRegistry. This is done by adding the classes to the outermost __init__.py. I think the only class we need to add there is LogitProcessorRegistry (we can already to from transformers import LogitsProcessor)

thanks a lot for reviewing. I will address these soon and send a new PR

…into Issue40010

PamelaBha · 2025-08-29T00:18:51Z

@gante this test is failing in the latest build but nothing to do with my change I believe: FAILED tests/models/gpt_oss/test_modeling_gpt_oss.py::GptOssModelTest::test_assisted_decoding_matches_greedy_search_1_same - AssertionError: False is not true

any pointers for me?

PamelaBha added 2 commits August 21, 2025 17:19

Declarative logit processor configuration

445fa34

Gitignore conda changes

1f59508

gante reviewed Aug 26, 2025

View reviewed changes

PamelaBha added 17 commits August 27, 2025 16:18

Declarative logit processor configuration

fbd1840

Gitignore conda changes

f90f939

PR comment changes

f3b72a1

More fixes

3a69724

Merge branch 'Issue40010' of https://github.com/PamelaBha/transformers …

c90f0cd

…into Issue40010

Fix merge conflict

bed91c1

Resolve PR comments

5c93e6c

PR comment

e7c342b

syntactic changes

329b1bf

Merge branch 'main' into Issue40010

ffe96a2

syntactic changes

e304ae9

Merge branch 'Issue40010' of https://github.com/PamelaBha/transformers …

957117e

…into Issue40010

Merge branch 'main' into Issue40010

3526f99

wrong space

bd13612

Merge branch 'Issue40010' of https://github.com/PamelaBha/transformers …

4125588

…into Issue40010

lint issues

c881dc2

Lint issues

7bea595

Merge branch 'main' into Issue40010

b3d158f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Customizable Logit Warping Strategies for Generation #40010 #40403

Customizable Logit Warping Strategies for Generation #40010 #40403

Uh oh!

PamelaBha commented Aug 23, 2025

Uh oh!

Rocketknight1 commented Aug 25, 2025

Uh oh!

gante left a comment

Uh oh!

Uh oh!

gante Aug 26, 2025

Uh oh!

gante Aug 26, 2025

Uh oh!

PamelaBha Aug 28, 2025

Uh oh!

gante Aug 26, 2025

Uh oh!

gante Aug 26, 2025

Uh oh!

gante Aug 26, 2025

Uh oh!

gante Aug 26, 2025

Uh oh!

gante Aug 26, 2025

Uh oh!

gante Aug 26, 2025

Uh oh!

gante Aug 26, 2025

Uh oh!

PamelaBha commented Aug 26, 2025

Uh oh!

PamelaBha commented Aug 29, 2025

Uh oh!

Uh oh!

		from transformers.generation.logits_process import LogitsProcessor, LogitProcessorRegistry
		import torch

Customizable Logit Warping Strategies for Generation #40010 #40403

Are you sure you want to change the base?

Customizable Logit Warping Strategies for Generation #40010 #40403

Uh oh!

Conversation

PamelaBha commented Aug 23, 2025

What does this PR do?

Before submitting

Who can review?

Uh oh!

Rocketknight1 commented Aug 25, 2025

Uh oh!

gante left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PamelaBha commented Aug 26, 2025

Uh oh!

PamelaBha commented Aug 29, 2025

Uh oh!

Uh oh!