metrics select error on clickhouse cluster

[deathalt]

if I run qryn with clickhouse cluster and config like

clickhouseServer: qryn-clickhouse.click-qryn.svc
clusterName: "default"

after I push some logs / tracers / metrics through metrico/otel-collector I cannot access metrics on API or grafana

Qryn logs

Error: Request failed with status code 404
    at createError (/app/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/app/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/app/node_modules/axios/lib/adapters/http.js:269:11)
    at IncomingMessage.emit (node:events:526:35)
    at endReadableNT (node:internal/streams/readable:1589:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  config: {
    url: 'http://admin:123asdzxc@qryn-clickhouse.click-qryn.svc:8123/?database=qryn',
    method: 'post',
    data: "WITH raw AS (select argMaxMerge(last) as `value`,`fingerprint`,intDiv(timestamp_ns, 15000000000) * 15000 as `timestamp_ms` from `metrics_15s_dist` where ((`fingerprint` in ((select `fingerprint` from `time_series_gin` where ((((`key` = '__name__') and (`val` = 'demo_client_request_counts'))) and (`date` >= toDate(fromUnixTimestamp(1701073470))) and (`date` <= toDate(fromUnixTimestamp(1701084270)))) group by `fingerprint` having (groupBitOr(bitShiftLeft(((`key` = '__name__') and (`val` = 'demo_client_request_counts'))::UInt64, 0)) = 1)))) and (`timestamp_ns` >= 1701073470000000000) and (`timestamp_ns` <= 1701084270000000000)) group by `fingerprint`,`timestamp_ms` order by `fingerprint`,`timestamp_ms`), timeSeries AS (select `fingerprint`,arraySort(JSONExtractKeysAndValues(labels, 'String')) as `labels` from `time_series` where (`fingerprint` in ((select `fingerprint` from `time_series_gin` where ((((`key` = '__name__') and (`val` = 'demo_client_request_counts'))) and (`date` >= toDate(fromUnixTimestamp(1701073470))) and (`date` <= toDate(fromUnixTimestamp(1701084270)))) group by `fingerprint` having (groupBitOr(bitShiftLeft(((`key` = '__name__') and (`val` = 'demo_client_request_counts'))::UInt64, 0)) = 1))))) select any(labels) as `stream`,arraySort(groupArray((raw.timestamp_ms, raw.value))) as `values` from raw as `raw` any left join timeSeries as time_series on `time_series`.`fingerprint` = raw.fingerprint group by `raw`.`fingerprint` order by `raw`.`fingerprint` FORMAT RowBinary",
    headers: {
      Accept: 'application/json, text/plain, */*',
      'Content-Type': 'application/x-www-form-urlencoded',
      'User-Agent': 'axios/0.21.4',
      'Content-Length': 1506
    },
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    adapter: [Function: httpAdapter],
    responseType: 'arraybuffer',
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    validateStatus: [Function: validateStatus],
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      clarifyTimeoutError: false
    }
  },
  request: <ref *1> ClientRequest {
    _events: [Object: null prototype] {
      abort: [Function (anonymous)],
      aborted: [Function (anonymous)],
      connect: [Function (anonymous)],
      error: [Function (anonymous)],
      socket: [Function (anonymous)],
      timeout: [Function (anonymous)],
      finish: [Function: requestOnFinish]
    },
    _eventsCount: 7,
    _maxListeners: undefined,
    outputData: [],
    outputSize: 0,
    writable: true,
    destroyed: true,
    _last: false,
    chunkedEncoding: false,
    shouldKeepAlive: true,
    maxRequestsOnConnectionReached: false,
    _defaultKeepAlive: true,
    useChunkedEncodingByDefault: true,
    sendDate: false,
    _removedConnection: false,
    _removedContLen: false,
    _removedTE: false,
    strictContentLength: false,
    _contentLength: 1506,
    _hasBody: true,
    _trailer: '',
    finished: true,
    _headerSent: true,
    _closed: true,
    socket: Socket {
      connecting: false,
      _hadError: false,
      _parent: null,
      _host: 'qryn-clickhouse.click-qryn.svc',
      _closeAfterHandlingError: false,
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 6,
      _maxListeners: undefined,
      _writableState: [WritableState],
      allowHalfOpen: false,
      _sockname: null,
      _pendingData: null,
      _pendingEncoding: '',
      server: null,
      _server: null,
      timeout: 3000,
      parser: null,
      _httpMessage: null,
      [Symbol(async_id_symbol)]: -1,
      [Symbol(kHandle)]: [TCP],
      [Symbol(lastWriteQueueSize)]: 0,
      [Symbol(timeout)]: Timeout {
        _idleTimeout: 3000,
        _idlePrev: [TimersList],
        _idleNext: [TimersList],
        _idleStart: 764232,
        _onTimeout: [Function: bound ],
        _timerArgs: undefined,
        _repeat: null,
        _destroyed: false,
        [Symbol(refed)]: false,
        [Symbol(kHasPrimitive)]: false,
        [Symbol(asyncId)]: 3019,
        [Symbol(triggerId)]: 3017
      },
      [Symbol(kBuffer)]: null,
      [Symbol(kBufferCb)]: null,
      [Symbol(kBufferGen)]: null,
      [Symbol(kCapture)]: false,
      [Symbol(kSetNoDelay)]: true,
      [Symbol(kSetKeepAlive)]: true,
      [Symbol(kSetKeepAliveInitialDelay)]: 1,
      [Symbol(kBytesRead)]: 0,
      [Symbol(kBytesWritten)]: 0
    },
    _header: 'POST /?database=qryn HTTP/1.1\r\n' +
      'Accept: application/json, text/plain, */*\r\n' +
      'Content-Type: application/x-www-form-urlencoded\r\n' +
      'User-Agent: axios/0.21.4\r\n' +
      'Content-Length: 1506\r\n' +
      'Host: qryn-clickhouse.click-qryn.svc:8123\r\n' +
      'Authorization: Basic YWRtaW46MTIzYXNkenhj\r\n' +
      'Connection: keep-alive\r\n' +
      '\r\n',
    _keepAliveTimeout: 0,
    _onPendingData: [Function: nop],
    agent: Agent {
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      defaultPort: 80,
      protocol: 'http:',
      options: [Object: null prototype],
      requests: [Object: null prototype] {},
      sockets: [Object: null prototype] {},
      freeSockets: [Object: null prototype],
      keepAliveMsecs: 1000,
      keepAlive: true,
      maxSockets: Infinity,
      maxFreeSockets: 256,
      scheduling: 'lifo',
      maxTotalSockets: Infinity,
      totalSocketCount: 1,
      [Symbol(kCapture)]: false
    },
    socketPath: undefined,
    method: 'POST',
    maxHeaderSize: undefined,
    insecureHTTPParser: undefined,
    joinDuplicateHeaders: undefined,
    path: '/?database=qryn',
    _ended: true,
    res: IncomingMessage {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      socket: null,
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: true,
      rawHeaders: [Array],
      rawTrailers: [],
      joinDuplicateHeaders: undefined,
      aborted: false,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 404,
      statusMessage: 'Not Found',
      client: [Socket],
      _consuming: true,
      _dumped: false,
      req: [Circular *1],
      responseUrl: 'http://admin:123asdzxc@qryn-clickhouse.click-qryn.svc:8123/?database=qryn',
      redirects: [],
      [Symbol(kCapture)]: false,
      [Symbol(kHeaders)]: [Object],
      [Symbol(kHeadersCount)]: 22,
      [Symbol(kTrailers)]: null,
      [Symbol(kTrailersCount)]: 0
    },
    aborted: false,
    timeoutCb: null,
    upgradeOrConnect: false,
    parser: null,
    maxHeadersCount: null,
    reusedSocket: true,
    host: 'qryn-clickhouse.click-qryn.svc',
    protocol: 'http:',
    _redirectable: Writable {
      _writableState: [WritableState],
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      _options: [Object],
      _ended: true,
      _ending: true,
      _redirectCount: 0,
      _redirects: [],
      _requestBodyLength: 1506,
      _requestBodyBuffers: [],
      _onNativeResponse: [Function (anonymous)],
      _currentRequest: [Circular *1],
      _currentUrl: 'http://admin:123asdzxc@qryn-clickhouse.click-qryn.svc:8123/?database=qryn',
      [Symbol(kCapture)]: false
    },
    [Symbol(kCapture)]: false,
    [Symbol(kBytesWritten)]: 0,
    [Symbol(kNeedDrain)]: false,
    [Symbol(corked)]: 0,
    [Symbol(kOutHeaders)]: [Object: null prototype] {
      accept: [Array],
      'content-type': [Array],
      'user-agent': [Array],
      'content-length': [Array],
      host: [Array],
      authorization: [Array]
    },
    [Symbol(errored)]: null,
    [Symbol(kHighWaterMark)]: 16384,
    [Symbol(kRejectNonStandardBodyWrites)]: false,
    [Symbol(kUniqueHeaders)]: null
  },
  response: {
    status: 404,
    statusText: 'Not Found',
    headers: {
      date: 'Mon, 27 Nov 2023 11:24:37 GMT',
      connection: 'Keep-Alive',
      'content-type': 'application/octet-stream',
      'x-clickhouse-server-display-name': 'qryn-clickhouse-shard2-1.qryn-clickhouse-headless.click-qryn.svc.lux4.kube',
      'transfer-encoding': 'chunked',
      'x-clickhouse-query-id': 'e48f4c04-d311-443b-bfbd-de94be9eca6f',
      'x-clickhouse-format': 'RowBinary',
      'x-clickhouse-timezone': 'Etc/UTC',
      'x-clickhouse-exception-code': '60',
      'keep-alive': 'timeout=3',
      'x-clickhouse-summary': '{"read_rows":"11087","read_bytes":"894581","written_rows":"0","written_bytes":"0","total_rows_to_read":"11087","result_rows":"0","result_bytes":"0"}'
    },
    config: {
      url: 'http://admin:123asdzxc@qryn-clickhouse.click-qryn.svc:8123/?database=qryn',
      method: 'post',
      data: "WITH raw AS (select argMaxMerge(last) as `value`,`fingerprint`,intDiv(timestamp_ns, 15000000000) * 15000 as `timestamp_ms` from `metrics_15s_dist` where ((`fingerprint` in ((select `fingerprint` from `time_series_gin` where ((((`key` = '__name__') and (`val` = 'demo_client_request_counts'))) and (`date` >= toDate(fromUnixTimestamp(1701073470))) and (`date` <= toDate(fromUnixTimestamp(1701084270)))) group by `fingerprint` having (groupBitOr(bitShiftLeft(((`key` = '__name__') and (`val` = 'demo_client_request_counts'))::UInt64, 0)) = 1)))) and (`timestamp_ns` >= 1701073470000000000) and (`timestamp_ns` <= 1701084270000000000)) group by `fingerprint`,`timestamp_ms` order by `fingerprint`,`timestamp_ms`), timeSeries AS (select `fingerprint`,arraySort(JSONExtractKeysAndValues(labels, 'String')) as `labels` from `time_series` where (`fingerprint` in ((select `fingerprint` from `time_series_gin` where ((((`key` = '__name__') and (`val` = 'demo_client_request_counts'))) and (`date` >= toDate(fromUnixTimestamp(1701073470))) and (`date` <= toDate(fromUnixTimestamp(1701084270)))) group by `fingerprint` having (groupBitOr(bitShiftLeft(((`key` = '__name__') and (`val` = 'demo_client_request_counts'))::UInt64, 0)) = 1))))) select any(labels) as `stream`,arraySort(groupArray((raw.timestamp_ms, raw.value))) as `values` from raw as `raw` any left join timeSeries as time_series on `time_series`.`fingerprint` = raw.fingerprint group by `raw`.`fingerprint` order by `raw`.`fingerprint` FORMAT RowBinary",
      headers: [Object],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      adapter: [Function: httpAdapter],
      responseType: 'arraybuffer',
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: -1,
      maxBodyLength: -1,
      validateStatus: [Function: validateStatus],
      transitional: [Object]
    },
    request: <ref *1> ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 7,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: false,
      chunkedEncoding: false,
      shouldKeepAlive: true,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 1506,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [Socket],
      _header: 'POST /?database=qryn HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/x-www-form-urlencoded\r\n' +
        'User-Agent: axios/0.21.4\r\n' +
        'Content-Length: 1506\r\n' +
        'Host: qryn-clickhouse.click-qryn.svc:8123\r\n' +
        'Authorization: Basic YWRtaW46MTIzYXNkenhj\r\n' +
        'Connection: keep-alive\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/?database=qryn',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: true,
      host: 'qryn-clickhouse.click-qryn.svc',
      protocol: 'http:',
      _redirectable: [Writable],
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    data: <Buffer 43 6f 64 65 3a 20 36 30 2e 20 44 42 3a 3a 45 78 63 65 70 74 69 6f 6e 3a 20 52 65 63 65 69 76 65 64 20 66 72 6f 6d 20 71 72 79 6e 2d 63 6c 69 63 6b 68 ... 183 more bytes>
  },
  isAxiosError: true,
  toJSON: [Function: toJSON]
}

if I comment in my qryn deployment

  clusterName: "default"

I can access the metrics, but some of the log and traceroute queries are dropped because the tables are created in a cluster and competitive queries are killed by clickhouse.

Clickhouse logs with access to node directly without "clustername" env

``` 2023.11.27 11:35:13.042149 [ 67 ] {} qryn.samples_v3 (59fcf967-1b10-480e-82d1-92bdde363b19): auto DB::StorageReplicatedMergeTree::processQueueEntry │ │ (ReplicatedMergeTreeQueue::SelectedEntryPtr)::(anonymous class)::operator()(DB::StorageReplicatedMergeTree::LogEntryPtr &) const: Code: 235. DB::Exception: │ │ Part 1701043200_3200_3200_0 (state Active) already exists. (DUPLICATE_DATA_PART), Stack trace (when copying this message, always include the lines below): │ │ │ │ 0. DB::Exception::Exception(std::__1::basic_string, std::__1::allocator > const&, int, bool) @ 0x78d0bba in /opt/bi │ │ tnami/clickhouse/bin/clickhouse │ │ 1. DB::MergeTreeData::checkPartCanBeAddedToTable(std::__1::shared_ptr&, std::__1::unique_lock&) const @ 0x12b00e7f │ │ in /opt/bitnami/clickhouse/bin/clickhouse │ │ 2. DB::MergeTreeData::renameTempPartAndReplaceImpl(std::__1::shared_ptr&, DB::MergeTreeData::Transaction&, std::__1::unique_lock&, std::__1::shared_ptr, std::__1::vector, std::__1::allocato │ │ r > >*) @ 0x12b00133 in /opt/bitnami/clickhouse/bin/clickhouse │ │ 3. DB::MergeTreeData::renameTempPartAndReplace(std::__1::shared_ptr&, DB::MergeTreeData::Transaction&, std::__1::shared_ptr) @ 0x12b01c7f in /opt/bitnami/clickhouse/bin/clickhouse ```

[deathalt]

fixed for me

thanks!