Add Netty instrumentation handlers after matched handler rather than at end of pipeline.

[anuraaga]

While I think it's generally a bug, in practice we can't be sure that the pipeline will propagate events we need through to the end because who knows what the user's handlers will be doing. We can be sure that Netty's default handlers do though, so addAfter is safer than addLast.

Fixes #2098

This is close, tests pass but I get this mysterious error. @trask do you know the magic?

Transformed classes match global libraries ignore matcher: [org.springframework.core.ReactiveAdapterRegistry$EmptyCompletableFuture]
java.lang.AssertionError: Transformed classes match global libraries ignore matcher: [org.springframework.core.ReactiveAdapterRegistry$EmptyCompletableFuture]
	at io.opentelemetry.instrumentation.test.AgentTestRunner.agentCleanup(AgentTestRunner.java:111)
	at org.spockframework.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:200)
	at org.spockframework.runtime.model.MethodInfo.invoke(MethodInfo.java:113)
	at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:98)
	at org.spockframework.spring.SpringInterceptor.interceptCleanupSpecMethod(SpringInterceptor.java:82)
	at org.spockframework.runtime.extension.AbstractMethodInterceptor.intercept(AbstractMethodInterceptor.java:39)
	at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:97)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
	at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
	at org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:82)
	at org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:73)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:87)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:53)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:66)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:51)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:87)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:66)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.stop(TestWorker.java:133)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
	at java.base/java.lang.Thread.run(Thread.java:834)

[anuraaga]

Warning this PR will pass, but testLatestDeps is still broken with the error message in the description so should only be merged after manual verification.

Looked here but couldn't find anything that matches that class weirdly

https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/master/javaagent-tooling/src/main/java/io/opentelemetry/javaagent/tooling/matcher/AdditionalLibraryIgnoresMatcher.java#L133

[trask]

Warning this PR will pass, but testLatestDeps is still broken with the error message in the description so should only be merged after manual verification.

Looked here but couldn't find anything that matches that class weirdly

https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/master/javaagent-tooling/src/main/java/io/opentelemetry/javaagent/tooling/matcher/AdditionalLibraryIgnoresMatcher.java#L133

the "org.springframework.core" part is matching, and we need to add exception for this class, e.g.

      if (name.startsWith("org.springframework.core.")) {
        if (name.startsWith("org.springframework.core.task.")
            || name.equals("org.springframework.core.DecoratingClassLoader")
            || name.equals("org.springframework.core.OverridingClassLoader")
            || name.equals("org.springframework.core.ReactiveAdapterRegistry$EmptyCompletableFuture")) {
          return false;
        }
        return true;
      }

[anuraaga]

Thanks @trask - fixed and confirmed locally testLatestDeps works

[anuraaga]

Had to raise timeout since seems to fail on CI at 15 min too nowadays