BrowserContext can't be created for reused VUs

[SunstriderEx]

Brief summary

If browser-level test starts after no-browser (protocol-level) tests, an error in browser.newContext() may occur.
It happens when browser-level test reuses VU, that was already used for protocol-level test (or simply test without BrowserContext creation).

Use case: hybrid load testing, many protocol-level tests and some browser-level ones (for testing ASPX pages with multiple internal requests for example), https://k6.io/docs/using-k6-browser/running-browser-tests/#run-both-browser-level-and-protocol-level-tests-in-a-single-script

k6 version

k6 v0.47.0 (commit/5ceb210056, go1.21.2, windows/amd64)

OS

Windows 10

Docker version and image (if applicable)

No response

Steps to reproduce the problem

Just run following script:

import { sleep } from 'k6';
import { browser } from 'k6/experimental/browser';
import exec from 'k6/execution';

export const options = {
  scenarios: {
    protocolLevel: {
      executor: 'constant-vus',
      exec: 'protocolLevelTest',
      vus: 2,
      duration: '1s',
    },
    browserLevel: {
      options: {
        browser: {
          type: 'chromium',
        },
      },
      executor: 'ramping-vus',
      exec: 'browserLevelTest',
      startVUs: 1,
      stages: [
        { duration: '2s', target: 2 },
        { duration: '2s', target: 2 },
      ],
      startTime: '30s',
    },
  },
};

export function protocolLevelTest() {
  console.log(`${getVuInfo()}: protocol-level test dummy`);
  sleep(0.4);
}

export async function browserLevelTest() {
  try {
    browser.newContext();
    console.log(`${getVuInfo()}: BrowserContext created`);
  } catch (error) {
    console.error(`${getVuInfo()}: ${error}`);
  }

  sleep(1);
}

function getVuInfo() {
  return JSON.stringify(exec.vu);
}

Expected behaviour

All iterations of all tests have to pass without errors.

Actual behaviour

It seems it takes 30 seconds to free VUs used by scenario with constant-vus executor.
Before that moment browser-level test ramping completely new VUs with no problems for browser.newContext(). But at 31 second browser-level test takes VU that was used for protocol-level test. In each iteration of this reused VU occurs an error when trying to create BrowserContext (see logs below):

browser not found in registry. make sure to set browser type option in scenario definition in order to use the browser module

Output:

  execution: local
     script: browserVuErrorTest.js
     output: -

  scenarios: (100.00%) 2 scenarios, 3 max VUs, 1m4s max duration (incl. graceful stop):
           * protocolLevel: 2 looping VUs for 1s (exec: protocolLevelTest, gracefulStop: 30s)
           * browserLevel: Up to 2 looping VUs for 4s over 2 stages (gracefulRampDown: 30s, exec: browserLevelTest, startTime: 30s, gracefulStop: 30s)

INFO[0000] {"idInInstance":2,"idInTest":2,"iterationInInstance":0,"iterationInScenario":0,"tags":{"group":"","scenario":"protocolLevel"},"metrics":{"tags":{"group":"","scenario":"protocolLevel"},"metadata":{}}}: protocol-level test dummy  source=console
INFO[0000] {"idInInstance":1,"idInTest":1,"iterationInInstance":0,"iterationInScenario":0,"tags":{"group":"","scenario":"protocolLevel"},"metrics":{"tags":{"group":"","scenario":"protocolLevel"},"metadata":{}}}: protocol-level test dummy  source=console
INFO[0000] {"idInInstance":1,"idInTest":1,"iterationInInstance":1,"iterationInScenario":1,"tags":{"group":"","scenario":"protocolLevel"},"metrics":{"tags":{"group":"","scenario":"protocolLevel"},"metadata":{}}}: protocol-level test dummy  source=console
INFO[0000] {"idInInstance":2,"idInTest":2,"iterationInInstance":1,"iterationInScenario":1,"tags":{"group":"","scenario":"protocolLevel"},"metrics":{"tags":{"group":"","scenario":"protocolLevel"},"metadata":{}}}: protocol-level test dummy  source=console
INFO[0001] {"idInInstance":2,"idInTest":2,"iterationInInstance":2,"iterationInScenario":2,"tags":{"group":"","scenario":"protocolLevel"},"metrics":{"tags":{"group":"","scenario":"protocolLevel"},"metadata":{}}}: protocol-level test dummy  source=console
INFO[0001] {"idInInstance":1,"idInTest":1,"iterationInInstance":2,"iterationInScenario":2,"tags":{"group":"","scenario":"protocolLevel"},"metrics":{"tags":{"group":"","scenario":"protocolLevel"},"metadata":{}}}: protocol-level test dummy  source=console
INFO[0030] {"idInInstance":3,"idInTest":3,"iterationInInstance":0,"iterationInScenario":0,"tags":{"group":"","scenario":"browserLevel"},"metrics":{"tags":{"group":"","scenario":"browserLevel"},"metadata":{}}}: BrowserContext created  source=console
INFO[0031] {"idInInstance":3,"idInTest":3,"iterationInInstance":1,"iterationInScenario":1,"tags":{"group":"","scenario":"browserLevel"},"metrics":{"tags":{"group":"","scenario":"browserLevel"},"metadata":{}}}: BrowserContext created  source=console
ERRO[0032] {"idInInstance":1,"idInTest":1,"iterationInInstance":3,"iterationInScenario":0,"tags":{"group":"","scenario":"browserLevel"},"metrics":{"tags":{"group":"","scenario":"browserLevel"},"metadata":{}}}: GoError: browser not found in registry. make sure to set browser type option in scenario definition in order to use the browser module  source=console
INFO[0032] {"idInInstance":3,"idInTest":3,"iterationInInstance":2,"iterationInScenario":2,"tags":{"group":"","scenario":"browserLevel"},"metrics":{"tags":{"group":"","scenario":"browserLevel"},"metadata":{}}}: BrowserContext created  source=console
ERRO[0033] {"idInInstance":1,"idInTest":1,"iterationInInstance":4,"iterationInScenario":1,"tags":{"group":"","scenario":"browserLevel"},"metrics":{"tags":{"group":"","scenario":"browserLevel"},"metadata":{}}}: GoError: browser not found in registry. make sure to set browser type option in scenario definition in order to use the browser module  source=console
INFO[0034] {"idInInstance":3,"idInTest":3,"iterationInInstance":3,"iterationInScenario":3,"tags":{"group":"","scenario":"browserLevel"},"metrics":{"tags":{"group":"","scenario":"browserLevel"},"metadata":{}}}: BrowserContext created  source=console

     data_received........: 0 B 0 B/s
     data_sent............: 0 B 0 B/s
     iteration_duration...: avg=705.92ms min=400.46ms med=709.23ms max=1.01s p(90)=1s p(95)=1s
     iterations...........: 12  0.344638/s
     vus..................: 2   min=0      max=2
     vus_max..............: 3   min=3      max=3


running (0m34.8s), 0/3 VUs, 12 complete and 0 interrupted iterations
protocolLevel ✓ [======================================] 2 VUs    1s
browserLevel  ✓ [======================================] 0/2 VUs  4s

As you can see in output there is vus_max = 3, while both protocol-level and browser-level tests configured to use 2 VUs.

Therefore, at the moment, browser-level tests cannot be combined with protocol-level tests due to this bug in most cases.

[SunstriderEx]

The newBrowserRegistry function's called once for each VU and subscribes to their IterStart, IterEnd, Exit events.
If VU runs in a protocol-level scenario at first, then following code executes in the handleIterEvents func (IterStart event):

xk6-browser/browser/registry.go

Lines 241 to 249 in e8b0269

	// If browser module is imported in the test, NewModuleInstance will be called for
	// every VU. Because on VU init stage we can not distinguish to which scenario it
	// belongs or access its options (because state is nil), we have to always subscribe
	// to each VU iter events, including VUs that do not make use of the browser in their
	// iterations.
	// Therefore, if we get an event that does not correspond to a browser iteration, then
	// unsubscribe for the VU events and exit the loop in order to reduce unuseful overhead.
	if !isBrowserIter(r.vu) {
	unsubscribeFn()

The handleExitEvent function also triggers unsubscribe at the same time:

xk6-browser/browser/registry.go

Lines 292 to 293 in e8b0269

	func (r *browserRegistry) handleExitEvent(exitCh <-chan *k6event.Event, unsubscribeFn func()) {
	defer unsubscribeFn()

30 seconds after a protocol-level scenario finishes VU frees. A browser-level scenario launches, takes this VU, but there is no subscription to IterStart event anymore, so the browser initialization code is not executed, the errBrowserNotFoundInRegistry error occures in the getBrowser function.

I tried to fix this, but I don't see a simple solution to this error. It seems to be by design (there is the unsubscribe_on_non_browser_vu test), and to fix this behaviour working with VU's events and browser initialization needs to be reviewed.
Or am I wrong, and the problem is simpler?