"On-demand" e2e test execution on PRs

[JorTurFer]

During the last Community Standup (October 26th), we were talking about that it would be nice if there are any option to run e2e as part of the PR checks. Executing e2e test is slow and requires external resources, so we were talking about the possibility of using commands on the PR itself (from people who have at least written permissions) to trigger those e2e test only if he/she wants.
Also we suggested running a small subset of e2e test instead of the whole suite as a smoke test but following the approach of something manual which anyone who has written permissions should trigger.

I'm trying to figure out if there is any already done solution for this use case and I haven't been able to find any, so I propose using 2 different actions executed in a row to achieve the desired behavior.

• pull-request-comment-trigger
• has-permission

Do you know about any built-in solution or any GitHub action which already does it? Does this approach make sense for you?

[zroubalik]

The very best thing would be to specify which e2e test to run in the trigger, maybe somehow specify the regex that's been set here

keda/tests/run-all.sh

Line 21 in 7f619bd

for test_case in $(find scalers -name "*.test.ts" | shuf)

? Maybe I am just halucinating, but that would be awesome. This way we can run just e2e for a scaler that is being added and what not 🔥

[JorTurFer]

I think that this could be possible writing our own Github Action. Maybe we can create our own GitHub Action from scratch and match all our requirements, I mean, maybe we can create it and put all things together:

• Check the command
• Check permissions
• Trigger e2e tests

I'm doing the PoC of the first approach, but maybe if we are open to create our own Action, I can create it

[JorTurFer]

As a first approach, this could be enough to trigger another action:

name: Manual run e2e test

on:
  issue_comment:
    types: [created]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2

      - uses: khan/pull-request-comment-trigger@master
        id: check-comment
        with:
          trigger: 'e2e'
          reaction: rocket
          prefix_only: true
        env:
          GITHUB_TOKEN: '${{ secrets.GITHUB_TOKEN }}'

      - name: Check user permission
        id: check-permission
        uses: scherermichael-oss/action-has-permission@master
        with:
          required-permission: write
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Run only if user has sufficient permissions
        if: steps.check-comment.outputs.triggered == 'true' && steps.check-permission.outputs.has-permission        
        run: echo "DO THE STUFFS TO RUN E2E TESTS"
        
      - name: Run only if user has NOT sufficient permissions
        if: "! steps.check-permission.outputs.has-permission"
        run: echo "Sorry!"

Basically, if someone with writing permissions post a comment that starts with e2e, the step Run only if user has sufficient permissions will be executed. We can place there all the things that we want

[JorTurFer]

One important thing that I found is that we would change a bit the process to use different image tags during e2e tests.
Until now we could use main tag but for e2e tests from PR we should use other different tag. Probably if I create the PR we can discuss over the code and all will be easier to understand and have an opinion.

[zroubalik]

Yeah we should create a tag for each PR, something like :prXXXX, where XXXX is the number of the PR. And publish it only on ghcr.io (we should drop DockerHub in a near future) and if it it possible, delete the tag after the job is finished.

[JorTurFer]

I have to test it well, but I think that this could work (I have to prepare the environment)
#2241

I would like to support the option to pass the regex, I will take a look at it tomorrow

[JorTurFer]

This is finally done! 🎉

[zroubalik]

@JorTurFer can we see the test log somewhere, in case the e2e test failed?

[JorTurFer]

Sure, you can find them in the action executions.
Maybe we could answer to the comment with the execution url to do it easier. I mean, now we (github-bot) are reacting to the comment but we could try to improve it adding a comment with the url
Wdty?

[zroubalik]

Yeah, that would be excellent!

[JorTurFer]

Okey, I will try to add a comment in case of failing tests with the execution url or if it's possible, (🤞 ) with the step url