Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[chore] Add a smoke test that is run as part of the scorecard tests V2 #1335

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

andreasgerstmayr
Copy link
Contributor

This is a continuation of PR #1252, rebased on latest main branch with the following updates:

  • a new service account is created to run the scorecard tests (the default service account doesn't have permissions to perform the tasks of the kuttl tests, for example creating a new CR)
  • the e2e cluster environment is set up before running the tests (the kuttl tests require the operator to be installed)

@andreasgerstmayr andreasgerstmayr requested a review from a team as a code owner December 20, 2022 13:44
@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Dec 20, 2022

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: andreasgerstmayr / name: Andreas Gerstmayr (a74867c)

@andreasgerstmayr andreasgerstmayr force-pushed the add-scorecard-smoke-test-v2 branch 3 times, most recently from a74867c to 5d0d2c6 Compare December 20, 2022 15:23
@andreasgerstmayr andreasgerstmayr changed the title Add a smoke test that is run as part of the scorecard tests V2 [chore] Add a smoke test that is run as part of the scorecard tests V2 Dec 20, 2022
- image: quay.io/operator-framework/scorecard-test-kuttl:v2.0.0
labels:
suite: kuttlsuite
test: kuttltest1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is the label used?

Should we use a more descriptive label?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Afaics it's just printed in the scorecard output, i.e.

    {
      "kind": "Test",
      "apiVersion": "scorecard.operatorframework.io/v1alpha3",
      "spec": {
        "image": "quay.io/operator-framework/scorecard-test-kuttl:v2.0.0",
        "labels": {
          "suite": "kuttlsuite",
          "test": "kuttltest1"
        },
        "storage": {
          "spec": {
            "mountPath": {}
          }
        }
      },
      "status": {
        "results": [
          {
            "name": "smoketest",
            "state": "pass",
            "creationTimestamp": null
          }
        ]
      }
    }

Currently all tests of this kuttl suite (bundle/tests/scorecard/kuttl) are run in parallel as a single scorecard test. If we decide later that we want to split kuttl tests in multiple scorecard tests, we should add a descriptive label to each. For now I think it's fine, as in other projects we also run all kuttl tests in the same kuttl suite (i.e. tests/e2e is a single kuttle suite).

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Apr 21, 2023

CLA Signed

The committers listed above are authorized under a signed CLA.

Co-authored-by: Kevin Earls <kearls@redhat.com>

Setup service account and e2e env for kuttl scorecard tests

Signed-off-by: Andreas Gerstmayr <agerstmayr@redhat.com>
…Makefile

Signed-off-by: Andreas Gerstmayr <agerstmayr@redhat.com>
Signed-off-by: Andreas Gerstmayr <agerstmayr@redhat.com>
@pavolloffay
Copy link
Member

/easycla

Signed-off-by: Andreas Gerstmayr <agerstmayr@redhat.com>
- op: add
path: /stages/0/tests/-
value:
image: docker.io/andreasgerstmayr/scorecard-test-kuttl:1681909083
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it using a custom image?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It needs kuttl 0.15.0 because this kuttl version contains fixes regarding test cleanup, which is especially important if we run the tests in a single namespace.
Once operator-framework/operator-sdk#6401 is merged and released, we can switch to the official one again.

@@ -45,6 +46,12 @@ stages:
storage:
spec:
mountPath: {}
- image: docker.io/andreasgerstmayr/scorecard-test-kuttl:1681909083
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it using a custom image? Could we switch to the upstream one?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above (this file is auto-generated from the file above).

Signed-off-by: Andreas Gerstmayr <agerstmayr@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants