Improved smoke tests #934
Improved smoke tests #934
Conversation
iNikem
requested review from
anuraaga,
jkwatson,
trask and
tylerbenson
as code owners
iNikem
force-pushed
the
smoke-tests
branch
4 times, most recently
from
b86a30b
to
f5d937e
Compare
|
Build fails with "Previous attempts to find a Docker environment failed. Will not retry. Please see logs and check configuration". Hm... can we run Testcontainers in CircleCI at all?? |
We are still using |
|
But we do want to use testcontainers? Not only for smoke tests, right? Should we hurry up our migration to Github Actions? |
👍 from me I'm feeling more confident about the move to Github Actions given open-telemetry/community#429 (comment) |
|
But I don't think we can do it until we have working Gradle cache and that requires open-telemetry/community#392 |
|
why not? just too slow without the cache? |
|
Yes. Current nightly takes ~35 minutes. I think this is too slow for PR build |
|
Can we use local build cache for CI (at least for now) and save/restore that cache as part of the build? |
|
We can try. I will get back to this in couple of days. |
|
This PR also adds workflows for PR checks and for building applications for smoke tests (for now only for java 11). I also disabled smoke tests in CircleCI as the latter cannot run testcontainers. Github Actions take a lot of time for now, but Gradle cache is in place, so I will monitor how well will it work. All in all, please review, let merge this and see what will happen. |
|
GH Actions fail because there is no such images in this repo's packages yet. Will be solved shortly after merged. I totally failed to split this into working independent pieces :( |
| - name: Test with Gradle | ||
| uses: nick-invision/retry@v1 | ||
| with: | ||
| command: ./gradlew testJava${{ matrix.java }} --stacktrace | ||
| timeout_minutes: 60 | ||
| max_attempts: 3 |
There was a problem hiding this comment.
just checking that we want to do this retry on PRs (not just nightly)?
There was a problem hiding this comment.
I think we still want to protect PRs from flaky test.
There was a problem hiding this comment.
just realized, this will slow down feedback (3x?) on real failing builds, which are common in PRs
There was a problem hiding this comment.
Not exactly 3x, because already passed tests will not be re-run. But you are right. Will see how this goes...
smoke-tests/src/test/groovy/io/opentelemetry/smoketest/SmokeTest.groovy
Outdated
Show resolved
Hide resolved
| //We don't have support for Wildfly Undertow server yet. | ||
| //So this test just verifies that Wildfly has come up. |
There was a problem hiding this comment.
i think this should work now with real span verification, the reason it didn't before is because we were verifying from logging exporter and wildfly messed with stdout/stderr, i can try it out after the merge
There was a problem hiding this comment.
I haven't seen any spans generated during this test
|
Hm, indeed probably there is no need for several Gradle wrappers in one repo... |
|
@anuraaga I still prefer to add Gradle wrappers into those projects. They are meant to be developed independently. You most probably will end up with a separate Idea project for them and it will be nice if they are self-contained. Gradle wrappers are small, less than 80K |
* Preping smoke tests * Improved smoke tests * Make it work in Linux * Add workflow to run tests on PR * Build Smoke tests * Fix PR workflow trigger * Fix smoke test apps trigger * Disable smoke tests in CircleCI and let them run in Github Actions only * Fix smoke test apps docker image names * Polish
New smoke tests based on Testcontainers and OpenTelemetry Collector.
Target applications (Spring Boot, Wildfly and Play) are run inside Testcontainers with attached java agent. Agent is configured to send data to the Collector which uses
fileexporterto write received spans to a json file. Tests then read that file and can assert anything they like about spans.Partly addresses #298. What's left to be done there: write GitHub Action to build and publish target application distributions on their changes; support different java versions for target applications.