SmoothQuant Modifier for OBCQ #1758

Satrat · 2023-10-11T13:40:10Z

This PR implements a productionized SmoothQuantModifier for use in OBCQ, with example recipes for OPT and Llama.

Instructions and Testing

To run for OPT, and print out perplexity results

python src/sparseml/transformers/sparsification/obcq/obcq.py facebook/opt-1.3b c4
    --recipe src/sparseml/transformers/sparsification/obcq/example.yaml --eval wikitext

To run for Llama, you'll need a local copy of the model

python src/sparseml/transformers/sparsification/obcq/obcq.py /local/path/to/llama open_platypus
    --recipe src/sparseml/transformers/sparsification/obcq/example_llama.yaml --eval open_platypus

…arsegpt/llama2 # Conflicts: # src/sparseml/experimental/sparsegpt/dispatch.py # src/sparseml/experimental/sparsegpt/layer_compressor.py # src/sparseml/experimental/sparsegpt/main.py # src/sparseml/experimental/sparsegpt/model_preprocessor.py # src/sparseml/experimental/sparsegpt/opt.py # src/sparseml/experimental/sparsegpt/sequential.py # src/sparseml/experimental/sparsegpt/sparsegpt.py

src/sparseml/modifiers/smoothquant/base.py

src/sparseml/modifiers/smoothquant/pytorch.py

src/sparseml/core/model/pytorch.py

src/sparseml/modifiers/smoothquant/base.py

src/sparseml/modifiers/smoothquant/pytorch.py

Co-authored-by: Rahul Tuli <[email protected]>

…nto prod_smooth_quant

natuan and others added 30 commits August 17, 2023 23:28

First abstract structure with OPT, MPT

bd729f3

Merge branch 'main' into tuan/sparsegpt

0476a14

scaffolding

e9c4e95

First abstract structure with OPT, MPT

07a7273

Updated

73ccd2e

Updated

f742f4a

Initial implementation of Llama2 integration

7c2d3d4

Initial implementation of Llama2 integration

3f4c5d8

Fix llama-2 key

c181380

Add modelutils

dd0f5f8

Make smoothquant optional

99f8bbe

probe sequence length from model

db901e4

probe sequence length from model

48ebe81

Fix typo

2be7def

add in recipe and modifier logic

fb346ad

Catch additional arguments to load_model and load_data

c4fc152

Redifinition of seqlen

2b50607

Runable OPT main code path

15b2c10

mpt modifier

34c3d84

Merge branch 'tuan/sparsegpt' into sa/sgpt_modifier

4cac803

Initial start implementation

a3934c2

State before rebasing

6a35b7f

State before rebasing

e02abb2

Initial support for MPT

466213c

add in further completion state for session and events

d953be4

Initial commit for Llama2

7fc49ac

Return model after override Llama attn

ff5b9a1

working refactor

090af1c

More memory-efficient layer compression

594339f

bfineran previously approved these changes Oct 16, 2023

View reviewed changes

Satrat requested a review from anmarques October 16, 2023 20:56

Merge branch 'main' into prod_smooth_quant

8244bc5

Satrat dismissed bfineran’s stale review via 8244bc5 October 19, 2023 15:25

remove pytorch import

45e3630

Satrat requested a review from bfineran October 19, 2023 15:45

anmarques suggested changes Oct 20, 2023

View reviewed changes

src/sparseml/modifiers/smoothquant/base.py Outdated Show resolved Hide resolved

src/sparseml/modifiers/smoothquant/pytorch.py Outdated Show resolved Hide resolved

rahul-tuli previously approved these changes Oct 20, 2023

View reviewed changes

PR comments on logarithmic equalization

6b0393d

Satrat dismissed rahul-tuli’s stale review via 6b0393d October 20, 2023 18:48

Sara Adkins and others added 8 commits October 20, 2023 15:05

Update src/sparseml/modifiers/smoothquant/base.py

3012189

Co-authored-by: Rahul Tuli <[email protected]>

Update src/sparseml/modifiers/smoothquant/base.py

12ab5e6

Co-authored-by: Rahul Tuli <[email protected]>

Update src/sparseml/modifiers/smoothquant/base.py

06b7f7d

Co-authored-by: Rahul Tuli <[email protected]>

PR comments

042cb3f

Merge branch 'prod_smooth_quant' of github.com:neuralmagic/sparseml i…

68b1841

…nto prod_smooth_quant

clean up comments

11d1be3

Merge branch 'main' into prod_smooth_quant

7a83668

Merge branch 'main' into prod_smooth_quant

31b8bab

Satrat requested review from rahul-tuli and anmarques October 20, 2023 19:35

fix calibration model reference, adding tiny integration test

788c16b

anmarques previously approved these changes Oct 31, 2023

View reviewed changes

Merge branch 'main' into prod_smooth_quant

800d08b

Satrat dismissed anmarques’s stale review via 800d08b October 31, 2023 12:07

style

de2b0cd

bfineran approved these changes Oct 31, 2023

View reviewed changes

bfineran merged commit 90250f2 into main Oct 31, 2023

bfineran deleted the prod_smooth_quant branch October 31, 2023 12:21

bfineran pushed a commit that referenced this pull request Nov 16, 2023

SmoothQuant Modifier for OBCQ (#1758)

312f2f0

bfineran pushed a commit that referenced this pull request Nov 16, 2023

SmoothQuant Modifier for OBCQ (#1758)

63aee59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

SmoothQuant Modifier for OBCQ #1758

SmoothQuant Modifier for OBCQ #1758

Uh oh!

Satrat commented Oct 11, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SmoothQuant Modifier for OBCQ #1758

SmoothQuant Modifier for OBCQ #1758

Uh oh!

Conversation

Satrat commented Oct 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Instructions and Testing

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Satrat commented Oct 11, 2023 •

edited

Loading