Skip to content

Commit f347e51

Browse files
mudlerAisuko
mudler
and
Aisuko
authored
feat(conda): conda environments (#1144)
* feat(autogptq): add a separate conda environment for autogptq (#1137) **Description** This PR related to #1117 **Notes for Reviewers** Here we lock down the version of the dependencies. Make sure it can be used all the time without failed if the version of dependencies were upgraded. I change the order of importing packages according to the pylint, and no change the logic of code. It should be ok. I will do more investigate on writing some test cases for every backend. I can run the service in my environment, but there is not exist a way to test it. So, I am not confident on it. Add a README.md in the `grpc` root. This is the common commands for creating `conda` environment. And it can be used to the reference file for creating extral gRPC backend document. Signed-off-by: GitHub <[email protected]> Signed-off-by: Ettore Di Giacinto <[email protected]> * [Extra backend] Add seperate environment for ttsbark (#1141) **Description** This PR relates to #1117 **Notes for Reviewers** Same to the latest PR: * The code is also changed, but only the order of the import package parts. And some code comments are also added. * Add a configuration of the `conda` environment * Add a simple test case for testing if the service can be startup in current `conda` environment. It is succeed in VSCode, but the it is not out of box on terminal. So, it is hard to say the test case really useful. **[Signed commits](../CONTRIBUTING.md#signing-off-on-commits-developer-certificate-of-origin)** - [x] Yes, I signed my commits. <!-- Thank you for contributing to LocalAI! Contributing Conventions ------------------------- The draft above helps to give a quick overview of your PR. Remember to remove this comment and to at least: 1. Include descriptive PR titles with [<component-name>] prepended. We use [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/). 2. Build and test your changes before submitting a PR (`make build`). 3. Sign your commits 4. **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below). 5. **X/Twitter handle:** we announce bigger features on X/Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out! By following the community's contribution conventions upfront, the review process will be accelerated and your PR merged more quickly. If no one reviews your PR within a few days, please @-mention @mudler. --> Signed-off-by: GitHub <[email protected]> Signed-off-by: Ettore Di Giacinto <[email protected]> * feat(conda): add make target and entrypoints for the dockerfile Signed-off-by: Ettore Di Giacinto <[email protected]> * feat(conda): Add seperate conda env for diffusers (#1145) **Description** This PR relates to #1117 **Notes for Reviewers** * Add `conda` env `diffusers.yml` * Add Makefile to create it automatically * Add `run.sh` to support running as a extra backend * Also adding it to the main Dockerfile * Add make command in the root Makefile * Testing the server, it can start up under the env Signed-off-by: GitHub <[email protected]> Signed-off-by: Ettore Di Giacinto <[email protected]> * feat(conda):Add seperate env for vllm (#1148) **Description** This PR is related to #1117 **Notes for Reviewers** * The gRPC server can be started as normal * The test case can be triggered in VSCode * Same to other this kind of PRs, add `vllm.yml` Makefile and add `run.sh` to the main Dockerfile, and command to the main Makefile **[Signed commits](../CONTRIBUTING.md#signing-off-on-commits-developer-certificate-of-origin)** - [x] Yes, I signed my commits. <!-- Thank you for contributing to LocalAI! Contributing Conventions ------------------------- The draft above helps to give a quick overview of your PR. Remember to remove this comment and to at least: 1. Include descriptive PR titles with [<component-name>] prepended. We use [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/). 2. Build and test your changes before submitting a PR (`make build`). 3. Sign your commits 4. **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below). 5. **X/Twitter handle:** we announce bigger features on X/Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out! By following the community's contribution conventions upfront, the review process will be accelerated and your PR merged more quickly. If no one reviews your PR within a few days, please @-mention @mudler. --> Signed-off-by: GitHub <[email protected]> Signed-off-by: Ettore Di Giacinto <[email protected]> * feat(conda):Add seperate env for huggingface (#1146) **Description** This PR is related to #1117 **Notes for Reviewers** * Add conda env `huggingface.yml` * Change the import order, and also remove the no-used packages * Add `run.sh` and `make command` to the main Dockerfile and Makefile * Add test cases for it. It can be triggered and succeed under VSCode Python extension but it is hang by using `python -m unites test_huggingface.py` in the terminal ``` Running tests (unittest): /workspaces/LocalAI/extra/grpc/huggingface Running tests: /workspaces/LocalAI/extra/grpc/huggingface/test_huggingface.py::TestBackendServicer::test_embedding /workspaces/LocalAI/extra/grpc/huggingface/test_huggingface.py::TestBackendServicer::test_load_model /workspaces/LocalAI/extra/grpc/huggingface/test_huggingface.py::TestBackendServicer::test_server_startup ./test_huggingface.py::TestBackendServicer::test_embedding Passed ./test_huggingface.py::TestBackendServicer::test_load_model Passed ./test_huggingface.py::TestBackendServicer::test_server_startup Passed Total number of tests expected to run: 3 Total number of tests run: 3 Total number of tests passed: 3 Total number of tests failed: 0 Total number of tests failed with errors: 0 Total number of tests skipped: 0 Finished running tests! ``` **[Signed commits](../CONTRIBUTING.md#signing-off-on-commits-developer-certificate-of-origin)** - [x] Yes, I signed my commits. <!-- Thank you for contributing to LocalAI! Contributing Conventions ------------------------- The draft above helps to give a quick overview of your PR. Remember to remove this comment and to at least: 1. Include descriptive PR titles with [<component-name>] prepended. We use [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/). 2. Build and test your changes before submitting a PR (`make build`). 3. Sign your commits 4. **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below). 5. **X/Twitter handle:** we announce bigger features on X/Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out! By following the community's contribution conventions upfront, the review process will be accelerated and your PR merged more quickly. If no one reviews your PR within a few days, please @-mention @mudler. --> Signed-off-by: GitHub <[email protected]> Signed-off-by: Ettore Di Giacinto <[email protected]> * feat(conda): Add the seperate conda env for VALL-E X (#1147) **Description** This PR is related to #1117 **Notes for Reviewers** * The gRPC server cannot start up ``` (ttsvalle) @Aisuko ➜ /workspaces/LocalAI (feat/vall-e-x) $ /opt/conda/envs/ttsvalle/bin/python /workspaces/LocalAI/extra/grpc/vall-e-x/ttsvalle.py Traceback (most recent call last): File "/workspaces/LocalAI/extra/grpc/vall-e-x/ttsvalle.py", line 14, in <module> from utils.generation import SAMPLE_RATE, generate_audio, preload_models ModuleNotFoundError: No module named 'utils' ``` The installation steps follow https://github.com/Plachtaa/VALL-E-X#-installation below: * Under the `ttsvalle` conda env ``` git clone https://github.com/Plachtaa/VALL-E-X.git cd VALL-E-X pip install -r requirements.txt ``` **[Signed commits](../CONTRIBUTING.md#signing-off-on-commits-developer-certificate-of-origin)** - [x] Yes, I signed my commits. <!-- Thank you for contributing to LocalAI! Contributing Conventions ------------------------- The draft above helps to give a quick overview of your PR. Remember to remove this comment and to at least: 1. Include descriptive PR titles with [<component-name>] prepended. We use [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/). 2. Build and test your changes before submitting a PR (`make build`). 3. Sign your commits 4. **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below). 5. **X/Twitter handle:** we announce bigger features on X/Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out! By following the community's contribution conventions upfront, the review process will be accelerated and your PR merged more quickly. If no one reviews your PR within a few days, please @-mention @mudler. --> Signed-off-by: GitHub <[email protected]> Signed-off-by: Ettore Di Giacinto <[email protected]> * fix: set image type Signed-off-by: Ettore Di Giacinto <[email protected]> * feat(conda):Add seperate conda env for exllama (#1149) Add seperate env for exllama Signed-off-by: Aisuko <[email protected]> Signed-off-by: Ettore Di Giacinto <[email protected]> * Setup conda Signed-off-by: Ettore Di Giacinto <[email protected]> * Set image_type arg Signed-off-by: Ettore Di Giacinto <[email protected]> * ci: prepare only conda env in tests Signed-off-by: Ettore Di Giacinto <[email protected]> * Dockerfile: comment manual pip calls Signed-off-by: Ettore Di Giacinto <[email protected]> * conda: add conda to PATH Signed-off-by: Ettore Di Giacinto <[email protected]> * fixes * add shebang * Fixups Signed-off-by: Ettore Di Giacinto <[email protected]> * file perms Signed-off-by: Ettore Di Giacinto <[email protected]> * debug * Install new conda in the worker * Disable GPU tests for now until the worker is back * Rename workflows * debug * Fixup conda install * fixup(wrapper): pass args Signed-off-by: Ettore Di Giacinto <[email protected]> --------- Signed-off-by: GitHub <[email protected]> Signed-off-by: Ettore Di Giacinto <[email protected]> Signed-off-by: Aisuko <[email protected]> Signed-off-by: Ettore Di Giacinto <[email protected]> Co-authored-by: Aisuko <[email protected]>
1 parent 9b17af1 commit f347e51

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+1286
-161
lines changed

‎.github/workflows/test-gpu.yml‎ renamed to ‎.github/workflows/disabled/test-gpu.yml‎

File renamed without changes.

‎.github/workflows/image.yml‎

Lines changed: 38 additions & 98 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ concurrency:
1414
cancel-in-progress: true
1515

1616
jobs:
17-
docker:
17+
image-build:
1818
strategy:
1919
matrix:
2020
include:
@@ -29,98 +29,6 @@ jobs:
2929
tag-latest: 'false'
3030
tag-suffix: '-ffmpeg'
3131
ffmpeg: 'true'
32-
33-
runs-on: ubuntu-latest
34-
steps:
35-
- name: Checkout
36-
uses: actions/checkout@v4
37-
- name: Release space from worker
38-
run: |
39-
echo "Listing top largest packages"
40-
pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
41-
head -n 30 <<< "${pkgs}"
42-
echo
43-
df -h
44-
echo
45-
sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
46-
sudo apt-get remove --auto-remove android-sdk-platform-tools || true
47-
sudo apt-get purge --auto-remove android-sdk-platform-tools || true
48-
sudo rm -rf /usr/local/lib/android
49-
sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
50-
sudo rm -rf /usr/share/dotnet
51-
sudo apt-get remove -y '^mono-.*' || true
52-
sudo apt-get remove -y '^ghc-.*' || true
53-
sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
54-
sudo apt-get remove -y 'php.*' || true
55-
sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
56-
sudo apt-get remove -y '^google-.*' || true
57-
sudo apt-get remove -y azure-cli || true
58-
sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
59-
sudo apt-get remove -y '^gfortran-.*' || true
60-
sudo apt-get remove -y microsoft-edge-stable || true
61-
sudo apt-get remove -y firefox || true
62-
sudo apt-get remove -y powershell || true
63-
sudo apt-get remove -y r-base-core || true
64-
sudo apt-get autoremove -y
65-
sudo apt-get clean
66-
echo
67-
echo "Listing top largest packages"
68-
pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
69-
head -n 30 <<< "${pkgs}"
70-
echo
71-
sudo rm -rfv build || true
72-
df -h
73-
- name: Docker meta
74-
id: meta
75-
uses: docker/metadata-action@v5
76-
with:
77-
images: quay.io/go-skynet/local-ai
78-
tags: |
79-
type=ref,event=branch
80-
type=semver,pattern={{raw}}
81-
type=sha
82-
flavor: |
83-
latest=${{ matrix.tag-latest }}
84-
suffix=${{ matrix.tag-suffix }}
85-
86-
- name: Set up QEMU
87-
uses: docker/setup-qemu-action@master
88-
with:
89-
platforms: all
90-
91-
- name: Set up Docker Buildx
92-
id: buildx
93-
uses: docker/setup-buildx-action@master
94-
95-
- name: Login to DockerHub
96-
if: github.event_name != 'pull_request'
97-
uses: docker/login-action@v3
98-
with:
99-
registry: quay.io
100-
username: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
101-
password: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
102-
103-
- name: Build and push
104-
uses: docker/build-push-action@v5
105-
with:
106-
builder: ${{ steps.buildx.outputs.name }}
107-
build-args: |
108-
BUILD_TYPE=${{ matrix.build-type }}
109-
CUDA_MAJOR_VERSION=${{ matrix.cuda-major-version }}
110-
CUDA_MINOR_VERSION=${{ matrix.cuda-minor-version }}
111-
FFMPEG=${{ matrix.ffmpeg }}
112-
context: .
113-
file: ./Dockerfile
114-
platforms: ${{ matrix.platforms }}
115-
push: ${{ github.event_name != 'pull_request' }}
116-
tags: ${{ steps.meta.outputs.tags }}
117-
labels: ${{ steps.meta.outputs.labels }}
118-
119-
120-
docker-gpu:
121-
strategy:
122-
matrix:
123-
include:
12432
- build-type: 'cublas'
12533
cuda-major-version: 11
12634
cuda-minor-version: 7
@@ -162,7 +70,42 @@ jobs:
16270
&& sudo apt-get install -y git
16371
- name: Checkout
16472
uses: actions/checkout@v4
165-
73+
# - name: Release space from worker
74+
# run: |
75+
# echo "Listing top largest packages"
76+
# pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
77+
# head -n 30 <<< "${pkgs}"
78+
# echo
79+
# df -h
80+
# echo
81+
# sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
82+
# sudo apt-get remove --auto-remove android-sdk-platform-tools || true
83+
# sudo apt-get purge --auto-remove android-sdk-platform-tools || true
84+
# sudo rm -rf /usr/local/lib/android
85+
# sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
86+
# sudo rm -rf /usr/share/dotnet
87+
# sudo apt-get remove -y '^mono-.*' || true
88+
# sudo apt-get remove -y '^ghc-.*' || true
89+
# sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
90+
# sudo apt-get remove -y 'php.*' || true
91+
# sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
92+
# sudo apt-get remove -y '^google-.*' || true
93+
# sudo apt-get remove -y azure-cli || true
94+
# sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
95+
# sudo apt-get remove -y '^gfortran-.*' || true
96+
# sudo apt-get remove -y microsoft-edge-stable || true
97+
# sudo apt-get remove -y firefox || true
98+
# sudo apt-get remove -y powershell || true
99+
# sudo apt-get remove -y r-base-core || true
100+
# sudo apt-get autoremove -y
101+
# sudo apt-get clean
102+
# echo
103+
# echo "Listing top largest packages"
104+
# pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
105+
# head -n 30 <<< "${pkgs}"
106+
# echo
107+
# sudo rm -rfv build || true
108+
# df -h
166109
- name: Docker meta
167110
id: meta
168111
uses: docker/metadata-action@v5
@@ -192,6 +135,7 @@ jobs:
192135
registry: quay.io
193136
username: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
194137
password: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
138+
195139
- name: Build and push
196140
uses: docker/build-push-action@v5
197141
with:
@@ -207,7 +151,3 @@ jobs:
207151
push: ${{ github.event_name != 'pull_request' }}
208152
tags: ${{ steps.meta.outputs.tags }}
209153
labels: ${{ steps.meta.outputs.labels }}
210-
- name: Release space from worker ♻
211-
if: always()
212-
run: |
213-
docker system prune -f -a --volumes || true

‎.github/workflows/test.yml‎

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ concurrency:
1414
cancel-in-progress: true
1515

1616
jobs:
17-
ubuntu-latest:
17+
tests-linux:
1818
runs-on: ubuntu-latest
1919
strategy:
2020
matrix:
@@ -67,11 +67,18 @@ jobs:
6767
run: |
6868
sudo apt-get update
6969
sudo apt-get install build-essential ffmpeg
70-
70+
curl https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc | gpg --dearmor > conda.gpg && \
71+
sudo install -o root -g root -m 644 conda.gpg /usr/share/keyrings/conda-archive-keyring.gpg && \
72+
gpg --keyring /usr/share/keyrings/conda-archive-keyring.gpg --no-default-keyring --fingerprint 34161F5BF5EB1D4BFBBB8F0A8AEB4F8B29D82806 && \
73+
sudo /bin/bash -c 'echo "deb [arch=amd64 signed-by=/usr/share/keyrings/conda-archive-keyring.gpg] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" > /etc/apt/sources.list.d/conda.list' && \
74+
sudo /bin/bash -c 'echo "deb [arch=amd64 signed-by=/usr/share/keyrings/conda-archive-keyring.gpg] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" | tee -a /etc/apt/sources.list.d/conda.list' && \
75+
sudo apt-get update && \
76+
sudo apt-get install -y conda
7177
sudo apt-get install -y ca-certificates cmake curl patch
7278
sudo apt-get install -y libopencv-dev && sudo ln -s /usr/include/opencv4/opencv2 /usr/include/opencv2
73-
sudo pip install -r extra/requirements.txt
74-
79+
80+
sudo rm -rfv /usr/bin/conda || true
81+
PATH=$PATH:/opt/conda/bin make -C extra/grpc/huggingface
7582
7683
# Pre-build stable diffusion before we install a newever version of abseil (not compatible with stablediffusion-ncn)
7784
GO_TAGS="tts stablediffusion" GRPC_BACKENDS=backend-assets/grpc/stablediffusion make build
@@ -96,12 +103,11 @@ jobs:
96103
cd grpc && mkdir -p cmake/build && cd cmake/build && cmake -DgRPC_INSTALL=ON \
97104
-DgRPC_BUILD_TESTS=OFF \
98105
../.. && sudo make -j12 install
99-
100106
- name: Test
101107
run: |
102108
ESPEAK_DATA="/build/lib/Linux-$(uname -m)/piper_phonemize/lib/espeak-ng-data" GO_TAGS="tts stablediffusion" make test
103109
104-
macOS-latest:
110+
tests-apple:
105111
runs-on: macOS-latest
106112
strategy:
107113
matrix:

‎Dockerfile‎

Lines changed: 21 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ ARG TARGETARCH
1414
ARG TARGETVARIANT
1515

1616
ENV BUILD_TYPE=${BUILD_TYPE}
17-
ENV EXTERNAL_GRPC_BACKENDS="huggingface-embeddings:/build/extra/grpc/huggingface/huggingface.py,autogptq:/build/extra/grpc/autogptq/autogptq.py,bark:/build/extra/grpc/bark/ttsbark.py,diffusers:/build/extra/grpc/diffusers/backend_diffusers.py,exllama:/build/extra/grpc/exllama/exllama.py,vall-e-x:/build/extra/grpc/vall-e-x/ttsvalle.py,vllm:/build/extra/grpc/vllm/backend_vllm.py"
17+
ENV EXTERNAL_GRPC_BACKENDS="huggingface-embeddings:/build/extra/grpc/huggingface/run.sh,autogptq:/build/extra/grpc/autogptq/run.sh,bark:/build/extra/grpc/bark/run.sh,diffusers:/build/extra/grpc/diffusers/run.sh,exllama:/build/extra/grpc/exllama/run.sh,vall-e-x:/build/extra/grpc/vall-e-x/run.sh,vllm:/build/extra/grpc/vllm/run.sh"
1818
ENV GALLERIES='[{"name":"model-gallery", "url":"github:go-skynet/model-gallery/index.yaml"}, {"url": "github:go-skynet/model-gallery/huggingface.yaml","name":"huggingface"}]'
1919
ARG GO_TAGS="stablediffusion tts"
2020

@@ -77,17 +77,25 @@ RUN curl -L "https://github.com/gabime/spdlog/archive/refs/tags/v${SPDLOG_VERSIO
7777
# Extras requirements
7878
FROM requirements-core as requirements-extras
7979

80+
RUN curl https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc | gpg --dearmor > conda.gpg && \
81+
install -o root -g root -m 644 conda.gpg /usr/share/keyrings/conda-archive-keyring.gpg && \
82+
gpg --keyring /usr/share/keyrings/conda-archive-keyring.gpg --no-default-keyring --fingerprint 34161F5BF5EB1D4BFBBB8F0A8AEB4F8B29D82806 && \
83+
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/conda-archive-keyring.gpg] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" > /etc/apt/sources.list.d/conda.list && \
84+
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/conda-archive-keyring.gpg] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" | tee -a /etc/apt/sources.list.d/conda.list && \
85+
apt-get update && \
86+
apt-get install -y conda
87+
8088
COPY extra/requirements.txt /build/extra/requirements.txt
8189
ENV PATH="/root/.cargo/bin:${PATH}"
8290
RUN pip install --upgrade pip
8391
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
84-
RUN if [ "${TARGETARCH}" = "amd64" ]; then \
85-
pip install git+https://github.com/suno-ai/bark.git diffusers invisible_watermark transformers accelerate safetensors;\
86-
fi
87-
RUN if [ "${BUILD_TYPE}" = "cublas" ] && [ "${TARGETARCH}" = "amd64" ]; then \
88-
pip install torch vllm && pip install auto-gptq https://github.com/jllllll/exllama/releases/download/0.0.10/exllama-0.0.10+cu${CUDA_MAJOR_VERSION}${CUDA_MINOR_VERSION}-cp39-cp39-linux_x86_64.whl;\
89-
fi
90-
RUN pip install -r /build/extra/requirements.txt && rm -rf /build/extra/requirements.txt
92+
#RUN if [ "${TARGETARCH}" = "amd64" ]; then \
93+
# pip install git+https://github.com/suno-ai/bark.git diffusers invisible_watermark transformers accelerate safetensors;\
94+
# fi
95+
#RUN if [ "${BUILD_TYPE}" = "cublas" ] && [ "${TARGETARCH}" = "amd64" ]; then \
96+
# pip install torch vllm && pip install auto-gptq https://github.com/jllllll/exllama/releases/download/0.0.10/exllama-0.0.10+cu${CUDA_MAJOR_VERSION}${CUDA_MINOR_VERSION}-cp39-cp39-linux_x86_64.whl;\
97+
# fi
98+
#RUN pip install -r /build/extra/requirements.txt && rm -rf /build/extra/requirements.txt
9199

92100
# Vall-e-X
93101
RUN git clone https://github.com/Plachtaa/VALL-E-X.git /usr/lib/vall-e-x && cd /usr/lib/vall-e-x && pip install -r requirements.txt
@@ -139,6 +147,7 @@ FROM requirements-${IMAGE_TYPE}
139147
ARG FFMPEG
140148
ARG BUILD_TYPE
141149
ARG TARGETARCH
150+
ARG IMAGE_TYPE=extras
142151

143152
ENV BUILD_TYPE=${BUILD_TYPE}
144153
ENV REBUILD=false
@@ -169,6 +178,10 @@ COPY --from=builder /build/local-ai ./
169178
# do not let stablediffusion rebuild (requires an older version of absl)
170179
COPY --from=builder /build/backend-assets/grpc/stablediffusion ./backend-assets/grpc/stablediffusion
171180

181+
RUN if [ "${IMAGE_TYPE}" = "extras" ]; then \
182+
PATH=$PATH:/opt/conda/bin make prepare-extra-conda-environments \
183+
; fi
184+
172185
# Copy VALLE-X as it's not a real "lib"
173186
RUN if [ -d /usr/lib/vall-e-x ]; then \
174187
cp -rfv /usr/lib/vall-e-x/* ./ ; \

‎Makefile‎

Lines changed: 18 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -290,12 +290,12 @@ run: prepare ## run local-ai
290290
test-models/testmodel:
291291
mkdir test-models
292292
mkdir test-dir
293-
wget https://huggingface.co/nnakasato/ggml-model-test/resolve/main/ggml-model-q4.bin -O test-models/testmodel
294-
wget https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin -O test-models/whisper-en
295-
wget https://huggingface.co/mudler/all-MiniLM-L6-v2/resolve/main/ggml-model-q4_0.bin -O test-models/bert
296-
wget https://cdn.openai.com/whisper/draft-20220913a/micro-machines.wav -O test-dir/audio.wav
297-
wget https://huggingface.co/mudler/rwkv-4-raven-1.5B-ggml/resolve/main/RWKV-4-Raven-1B5-v11-Eng99%2525-Other1%2525-20230425-ctx4096_Q4_0.bin -O test-models/rwkv
298-
wget https://raw.githubusercontent.com/saharNooby/rwkv.cpp/5eb8f09c146ea8124633ab041d9ea0b1f1db4459/rwkv/20B_tokenizer.json -O test-models/rwkv.tokenizer.json
293+
wget -q https://huggingface.co/nnakasato/ggml-model-test/resolve/main/ggml-model-q4.bin -O test-models/testmodel
294+
wget -q https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin -O test-models/whisper-en
295+
wget -q https://huggingface.co/mudler/all-MiniLM-L6-v2/resolve/main/ggml-model-q4_0.bin -O test-models/bert
296+
wget -q https://cdn.openai.com/whisper/draft-20220913a/micro-machines.wav -O test-dir/audio.wav
297+
wget -q https://huggingface.co/mudler/rwkv-4-raven-1.5B-ggml/resolve/main/RWKV-4-Raven-1B5-v11-Eng99%2525-Other1%2525-20230425-ctx4096_Q4_0.bin -O test-models/rwkv
298+
wget -q https://raw.githubusercontent.com/saharNooby/rwkv.cpp/5eb8f09c146ea8124633ab041d9ea0b1f1db4459/rwkv/20B_tokenizer.json -O test-models/rwkv.tokenizer.json
299299
cp tests/models_fixtures/* test-models
300300

301301
prepare-test: grpcs
@@ -306,8 +306,8 @@ test: prepare test-models/testmodel grpcs
306306
@echo 'Running tests'
307307
export GO_TAGS="tts stablediffusion"
308308
$(MAKE) prepare-test
309-
HUGGINGFACE_GRPC=$(abspath ./)/extra/grpc/huggingface/huggingface.py TEST_DIR=$(abspath ./)/test-dir/ FIXTURES=$(abspath ./)/tests/fixtures CONFIG_FILE=$(abspath ./)/test-models/config.yaml MODELS_PATH=$(abspath ./)/test-models \
310-
$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="!gpt4all && !llama && !llama-gguf" --flake-attempts 5 -v -r ./api ./pkg
309+
HUGGINGFACE_GRPC=$(abspath ./)/extra/grpc/huggingface/run.sh TEST_DIR=$(abspath ./)/test-dir/ FIXTURES=$(abspath ./)/tests/fixtures CONFIG_FILE=$(abspath ./)/test-models/config.yaml MODELS_PATH=$(abspath ./)/test-models \
310+
$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="!gpt4all && !llama && !llama-gguf" --fail-fast -v -r ./api ./pkg
311311
$(MAKE) test-gpt4all
312312
$(MAKE) test-llama
313313
$(MAKE) test-llama-gguf
@@ -387,6 +387,16 @@ protogen-python:
387387

388388
## GRPC
389389

390+
prepare-extra-conda-environments:
391+
$(MAKE) -C extra/grpc/autogptq
392+
$(MAKE) -C extra/grpc/bark
393+
$(MAKE) -C extra/grpc/diffusers
394+
$(MAKE) -C extra/grpc/vllm
395+
$(MAKE) -C extra/grpc/huggingface
396+
$(MAKE) -C extra/grpc/vall-e-x
397+
$(MAKE) -C extra/grpc/exllama
398+
399+
390400
backend-assets/grpc:
391401
mkdir -p backend-assets/grpc
392402

‎extra/grpc/README.md‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Common commands about conda environment
2+
3+
## Create a new empty conda environment
4+
5+
```
6+
conda create --name <env-name> python=<your version> -y
7+
8+
conda create --name autogptq python=3.11 -y
9+
```
10+
11+
## To activate the environment
12+
13+
As of conda 4.4
14+
```
15+
conda activate autogptq
16+
```
17+
18+
The conda version older than 4.4
19+
20+
```
21+
source activate autogptq
22+
```
23+
24+
## Install the packages to your environment
25+
26+
Sometimes you need to install the packages from the conda-forge channel
27+
28+
By using `conda`
29+
```
30+
conda install <your-package-name>
31+
32+
conda install -c conda-forge <your package-name>
33+
```
34+
35+
Or by using `pip`
36+
```
37+
pip install <your-package-name>
38+
```

‎extra/grpc/autogptq/Makefile‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
.PONY: autogptq
2+
autogptq:
3+
@echo "Creating virtual environment..."
4+
@conda env create --name autogptq --file autogptq.yml
5+
@echo "Virtual environment created."

‎extra/grpc/autogptq/README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Creating a separate environment for the autogptq project
2+
3+
```
4+
make autogptq
5+
```

‎extra/grpc/autogptq/autogptq.py‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,15 @@
11
#!/usr/bin/env python3
2-
import grpc
32
from concurrent import futures
4-
import time
5-
import backend_pb2
6-
import backend_pb2_grpc
73
import argparse
84
import signal
95
import sys
106
import os
11-
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
12-
from pathlib import Path
7+
import time
8+
9+
import grpc
10+
import backend_pb2
11+
import backend_pb2_grpc
12+
from auto_gptq import AutoGPTQForCausalLM
1313
from transformers import AutoTokenizer
1414
from transformers import TextGenerationPipeline
1515

0 commit comments

Comments
 (0)