Install from source for a specific branch with docker

[agunapal]

Description

This PR allows the user to build TorchServe from source from

• a specific branch and build TS from source
• Build TS from source with Local changes

Fixes #(issue)

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

Feature/Issue validation/testing

• Production image from source and local changes

 ./build_image.sh -s -l -t pytorch/ts:latest
[+] Building 317.5s (28/28) FINISHED                                                                                                                                       docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                 0.0s
 => => transferring dockerfile: 9.62kB                                                                                                                                               0.0s
 => resolve image config for docker.io/docker/dockerfile:experimental                                                                                                                0.5s
 => [auth] docker/dockerfile:pull token for registry-1.docker.io                                                                                                                     0.0s
 => CACHED docker-image://docker.io/docker/dockerfile:experimental@sha256:600e5c62eedff338b3f7a0850beb7c05866e0ef27b2d2e8c02aa468e78496ff5                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                      0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                 0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                                      0.5s
 => [auth] library/ubuntu:pull token for registry-1.docker.io                                                                                                                        0.0s
 => [internal] load build context                                                                                                                                                   63.1s
 => => transferring context: 7.52GB                                                                                                                                                 62.9s
 => CACHED [production-image 1/9] FROM docker.io/library/ubuntu:20.04@sha256:80ef4a44043dec4490506e6cc4289eeda2d106a70148b74b5ae91ee670e9c35d                                        0.0s
 => [compile-image  2/10] RUN --mount=type=cache,id=apt-dev,target=/var/cache/apt     apt-get update &&     apt-get upgrade -y &&     apt-get install software-properties-common   145.0s
 => [production-image 2/9] RUN --mount=type=cache,target=/var/cache/apt     apt-get update &&     apt-get upgrade -y &&     apt-get install software-properties-common -y &&       145.0s
 => [production-image 3/9] RUN useradd -m model-server     && mkdir -p /home/model-server/tmp                                                                                        0.4s
 => [compile-image  3/10] RUN python3.9 -m venv /home/venv                                                                                                                           2.9s
 => [compile-image  4/10] RUN python -m pip install -U pip setuptools                                                                                                                3.1s 
 => [compile-image  5/10] RUN export USE_CUDA=1                                                                                                                                      0.3s 
 => [compile-image  6/10] COPY ./ serve                                                                                                                                             31.0s 
 => [compile-image  7/10] RUN     if echo "true" | grep -q "false"; then         rm -rf serve;        git clone --recursive https://github.com/pytorch/serve.git -b master;     fi   0.3s 
 => [compile-image  8/10] WORKDIR serve                                                                                                                                              0.0s 
 => [compile-image  9/10] RUN     if echo "ubuntu:20.04" | grep -q "cuda:"; then         if [ "" ]; then             python ./ts_scripts/install_dependencies.py --cuda ;        e  40.2s 
 => [compile-image 10/10] RUN     if echo "true" | grep -q "true"; then         python -m pip install -r requirements/developer.txt;        python ts_scripts/install_from_src.py;  70.5s 
 => [production-image 4/9] COPY --chown=model-server --from=compile-image /home/venv /home/venv                                                                                      7.6s 
 => [production-image 5/9] COPY docker/dockerd-entrypoint.sh /usr/local/bin/dockerd-entrypoint.sh                                                                                    0.0s 
 => [production-image 6/9] RUN chmod +x /usr/local/bin/dockerd-entrypoint.sh     && chown -R model-server /home/model-server                                                         0.3s 
 => [production-image 7/9] COPY docker/config.properties /home/model-server/config.properties                                                                                        0.1s 
 => [production-image 8/9] RUN mkdir /home/model-server/model-store && chown -R model-server /home/model-server/model-store                                                          0.2s 
 => [production-image 9/9] WORKDIR /home/model-server                                                                                                                                0.1s 
 => exporting to image                                                                                                                                                              11.4s
 => => exporting layers                                                                                                                                                             11.4s
 => => writing image sha256:b6f4d83f7917ef2ddfae721fa4d690e2fdb2c7c82b2d91c27d54362aefc04f86                                                                                         0.0s
 => => naming to docker.io/pytorch/ts:latest                                                                                       

• Production Image default

./build_image.sh -t pytorch/ts:latest
[+] Building 284.9s (28/28) FINISHED                                                                                                                                       docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                 0.0s
 => => transferring dockerfile: 9.49kB                                                                                                                                               0.0s
 => resolve image config for docker.io/docker/dockerfile:experimental                                                                                                                0.5s
 => [auth] docker/dockerfile:pull token for registry-1.docker.io                                                                                                                     0.0s
 => CACHED docker-image://docker.io/docker/dockerfile:experimental@sha256:600e5c62eedff338b3f7a0850beb7c05866e0ef27b2d2e8c02aa468e78496ff5                                           0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                      0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                                      0.5s
 => [auth] library/ubuntu:pull token for registry-1.docker.io                                                                                                                        0.0s
 => [internal] load build context                                                                                                                                                   23.2s
 => => transferring context: 7.52GB                                                                                                                                                 23.0s
 => CACHED [production-image 1/9] FROM docker.io/library/ubuntu:20.04@sha256:80ef4a44043dec4490506e6cc4289eeda2d106a70148b74b5ae91ee670e9c35d                                        0.0s
 => [compile-image  2/10] RUN --mount=type=cache,id=apt-dev,target=/var/cache/apt     apt-get update &&     apt-get upgrade -y &&     apt-get install software-properties-common   113.2s
 => [compile-image  3/10] RUN python3.9 -m venv /home/venv                                                                                                                           2.8s 
 => [compile-image  4/10] RUN python -m pip install -U pip setuptools                                                                                                                3.2s 
 => [compile-image  5/10] RUN export USE_CUDA=1                                                                                                                                      0.3s 
 => [compile-image  6/10] COPY ./ serve                                                                                                                                             30.9s 
 => [compile-image  7/10] RUN     if echo "false" | grep -q "false"; then         rm -rf serve;        git clone --recursive https://github.com/pytorch/serve.git -b master;     f  79.3s 
 => [compile-image  8/10] WORKDIR serve                                                                                                                                              0.0s 
 => [compile-image  9/10] RUN     if echo "ubuntu:20.04" | grep -q "cuda:"; then         if [ "" ]; then             python ./ts_scripts/install_dependencies.py --cuda ;        e  41.2s 
 => [compile-image 10/10] RUN     if echo "false" | grep -q "true"; then         python -m pip install -r requirements/developer.txt;        python ts_scripts/install_from_src.py;  1.9s 
 => CACHED [production-image 2/9] RUN --mount=type=cache,target=/var/cache/apt     apt-get update &&     apt-get upgrade -y &&     apt-get install software-properties-common -y &&  0.0s 
 => CACHED [production-image 3/9] RUN useradd -m model-server     && mkdir -p /home/model-server/tmp                                                                                 0.0s 
 => [production-image 4/9] COPY --chown=model-server --from=compile-image /home/venv /home/venv                                                                                      5.5s 
 => [production-image 5/9] COPY docker/dockerd-entrypoint.sh /usr/local/bin/dockerd-entrypoint.sh                                                                                    0.0s 
 => [production-image 6/9] RUN chmod +x /usr/local/bin/dockerd-entrypoint.sh     && chown -R model-server /home/model-server                                                         0.2s 
 => [production-image 7/9] COPY docker/config.properties /home/model-server/config.properties                                                                                        0.1s 
 => [production-image 8/9] RUN mkdir /home/model-server/model-store && chown -R model-server /home/model-server/model-store                                                          0.2s
 => [production-image 9/9] WORKDIR /home/model-server                                                                                                                                0.1s
 => exporting to image                                                                                                                                                               3.2s
 => => exporting layers                                                                                                                                                              3.2s
 => => writing image sha256:84f4856f5cce4c996283432eabf9f661c9f789193b229e5590a700c3a69e9405                                                                                         0.0s
 => => naming to docker.io/pytorch/ts:latest                                                       

• Dev Image default

./build_image.sh -bt dev -t pytorch/ts:latest
[+] Building 398.1s (27/27) FINISHED                                                                                                                                       docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                 0.0s
 => => transferring dockerfile: 9.62kB                                                                                                                                               0.0s
 => resolve image config for docker.io/docker/dockerfile:experimental                                                                                                                0.5s
 => [auth] docker/dockerfile:pull token for registry-1.docker.io                                                                                                                     0.0s
 => CACHED docker-image://docker.io/docker/dockerfile:experimental@sha256:600e5c62eedff338b3f7a0850beb7c05866e0ef27b2d2e8c02aa468e78496ff5                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                      0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                 0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                                      0.5s
 => [auth] library/ubuntu:pull token for registry-1.docker.io                                                                                                                        0.0s
 => CACHED [compile-image  1/10] FROM docker.io/library/ubuntu:20.04@sha256:80ef4a44043dec4490506e6cc4289eeda2d106a70148b74b5ae91ee670e9c35d                                         0.0s
 => [internal] load build context                                                                                                                                                    0.7s
 => => transferring context: 2.38MB                                                                                                                                                  0.7s
 => [dev-image 2/8] RUN --mount=type=cache,target=/var/cache/apt     apt-get update &&     apt-get upgrade -y &&     apt-get install software-properties-common -y &&     add-apt  102.3s
 => CACHED [compile-image  2/10] RUN --mount=type=cache,id=apt-dev,target=/var/cache/apt     apt-get update &&     apt-get upgrade -y &&     apt-get install software-properties-co  0.0s
 => CACHED [compile-image  3/10] RUN python3.9 -m venv /home/venv                                                                                                                    0.0s
 => CACHED [compile-image  4/10] RUN python -m pip install -U pip setuptools                                                                                                         0.0s
 => CACHED [compile-image  5/10] RUN export USE_CUDA=1                                                                                                                               0.0s
 => CACHED [compile-image  6/10] COPY ./ serve                                                                                                                                       0.0s
 => CACHED [compile-image  7/10] RUN     if echo "false" | grep -q "false"; then         rm -rf serve;        git clone --recursive https://github.com/pytorch/serve.git -b master;  0.0s
 => CACHED [compile-image  8/10] WORKDIR serve                                                                                                                                       0.0s
 => CACHED [compile-image  9/10] RUN     if echo "ubuntu:20.04" | grep -q "cuda:"; then         if [ "" ]; then             python ./ts_scripts/install_dependencies.py --cuda ;     0.0s
 => CACHED [compile-image 10/10] RUN     if echo "false" | grep -q "true"; then         python -m pip install -r requirements/developer.txt;        python ts_scripts/install_from_  0.0s
 => [dev-image 3/8] COPY ./ serve                                                                                                                                                   46.6s 
 => [dev-image 4/8] RUN     if echo "false" | grep -q "false"; then         rm -rf serve;        git clone --recursive https://github.com/pytorch/serve.git -b master;     fi       76.4s 
 => [dev-image 5/8] COPY --from=compile-image /home/venv /home/venv                                                                                                                  8.5s 
 => [dev-image 6/8] WORKDIR serve                                                                                                                                                    0.0s 
 => [dev-image 7/8] RUN python -m pip install -U pip setuptools     && python -m pip install --no-cache-dir -r requirements/developer.txt     && python ts_scripts/install_from_s  147.2s 
 => [dev-image 8/8] WORKDIR /home/model-server                                                                                                                                       0.1s 
 => exporting to image                                                                                                                                                              15.7s 
 => => exporting layers                                                                                                                                                             15.7s 
 => => writing image sha256:a3d095ad7e7c77b117b1e0d9c92ef768ce061ad227ad89c086cf7527d289d434                                                                                         0.0s 
 => => naming to docker.io/pytorch/ts:latest                                               

• Dev image from source

 ./build_image.sh -bt dev -l -s -t pytorch/ts:latest                                                                                        
[+] Building 404.0s (27/27) FINISHED                                                                                                                                       docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                 0.0s
 => => transferring dockerfile: 9.62kB                                                                                                                                               0.0s
 => resolve image config for docker.io/docker/dockerfile:experimental                                                                                                                0.5s
 => [auth] docker/dockerfile:pull token for registry-1.docker.io                                                                                                                     0.0s
 => CACHED docker-image://docker.io/docker/dockerfile:experimental@sha256:600e5c62eedff338b3f7a0850beb7c05866e0ef27b2d2e8c02aa468e78496ff5                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                      0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                 0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                                      0.5s
 => [auth] library/ubuntu:pull token for registry-1.docker.io                                                                                                                        0.0s
 => [internal] load build context                                                                                                                                                    0.7s
 => => transferring context: 2.38MB                                                                                                                                                  0.7s
 => CACHED [compile-image  1/10] FROM docker.io/library/ubuntu:20.04@sha256:80ef4a44043dec4490506e6cc4289eeda2d106a70148b74b5ae91ee670e9c35d                                         0.0s
 => [dev-image 2/8] RUN --mount=type=cache,target=/var/cache/apt     apt-get update &&     apt-get upgrade -y &&     apt-get install software-properties-common -y &&     add-apt  112.4s
 => CACHED [compile-image  2/10] RUN --mount=type=cache,id=apt-dev,target=/var/cache/apt     apt-get update &&     apt-get upgrade -y &&     apt-get install software-properties-co  0.0s
 => CACHED [compile-image  3/10] RUN python3.9 -m venv /home/venv                                                                                                                    0.0s
 => CACHED [compile-image  4/10] RUN python -m pip install -U pip setuptools                                                                                                         0.0s
 => CACHED [compile-image  5/10] RUN export USE_CUDA=1                                                                                                                               0.0s
 => CACHED [compile-image  6/10] COPY ./ serve                                                                                                                                       0.0s
 => CACHED [compile-image  7/10] RUN     if echo "true" | grep -q "false"; then         rm -rf serve;        git clone --recursive https://github.com/pytorch/serve.git -b master;   0.0s
 => CACHED [compile-image  8/10] WORKDIR serve                                                                                                                                       0.0s
 => CACHED [compile-image  9/10] RUN     if echo "ubuntu:20.04" | grep -q "cuda:"; then         if [ "" ]; then             python ./ts_scripts/install_dependencies.py --cuda ;     0.0s
 => CACHED [compile-image 10/10] RUN     if echo "true" | grep -q "true"; then         python -m pip install -r requirements/developer.txt;        python ts_scripts/install_from_s  0.0s
 => [dev-image 3/8] COPY ./ serve                                                                                                                                                   55.4s 
 => [dev-image 4/8] RUN     if echo "true" | grep -q "false"; then         rm -rf serve;        git clone --recursive https://github.com/pytorch/serve.git -b master;     fi         0.2s 
 => [dev-image 5/8] COPY --from=compile-image /home/venv /home/venv                                                                                                                 32.9s 
 => [dev-image 6/8] WORKDIR serve                                                                                                                                                    0.0s 
 => [dev-image 7/8] RUN python -m pip install -U pip setuptools     && python -m pip install --no-cache-dir -r requirements/developer.txt     && python ts_scripts/install_from_s  185.5s 
 => [dev-image 8/8] WORKDIR /home/model-server                                                                                                                                       0.0s 
 => exporting to image                                                                                                                                                              16.3s 
 => => exporting layers                                                                                                                                                             16.3s 
 => => writing image sha256:03c1048e6d30c8466064e0d2a792e1b14df0dfd45faaf5013666cb8b46b76de7                                                                                         0.0s 
 => => naming to docker.io/pytorch/ts:latest                                 

Checklist:

• Did you have fun?
• Have you added tests that prove your fix is effective or that this feature works?
• Has code been commented, particularly in hard-to-understand areas?
• Have you made corresponding changes to the documentation?

[agunapal]

@msaroufim Do you think we should had a fork arg so that this can be used by anyone

[msaroufim]

@msaroufim Do you think we should had a fork arg so that this can be used by anyone

Yeah I was gonna say it's quite strange to me that this only works with remote branches, I'd expect it to work given a local branch

[agunapal]

@msaroufim Do you think we should had a fork arg so that this can be used by anyone

Yeah I was gonna say it's quite strange to me that this only works with remote branches, I'd expect it to work given a local branch

@msaroufim We can update the dockerfile to copy serve from host machine instead of using git clone. I guess this will hold good for official builds or anyone building docker from their fork since the workflow clones from repo anyway. Does this sound reasonable?