-
-
Notifications
You must be signed in to change notification settings - Fork 63
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can I provide replication and distributed SQL statements for Clickhouse clusters #188
Comments
When I use the following statement to create a database and data table, it has the following error:
qryn logs:
|
Hello @ktpktr0 - did you also set the following?
|
@lmangani Yes, I have set
|
@ktpktr0 the wiki has been updated, could you recheck against it? https://github.com/metrico/qryn/wiki/qryn-tables-replication-support |
@lmangani |
@ktpktr0 please attach the errors and make sure any relevant part of troubleshooting section is also completed |
|
The issue is still in ClickHouse and those are the relevant logs most likely? |
I have tried to copy tables and distributed tables, and it works normally |
The database is not, you need to look at what happened during table creation. Perhaps @akvlad has more suggestions. |
I tried a cluster of two nodes (qryn-clickhouse-cluster/docker/docker-compose-clickhouse-cluster.yml), but it also failed to work |
@ktpktr0 I pushed a new tag with more logs on ping. |
@akvlad
qryn log:
|
@ktpktr0 |
How does qryn connect when my Clickhouse is a cluster? Using chproxy? |
|
For example, I have a Clickhouse cluster with 3 partitions About 2: Ningx -- > qryn1 + slice 1; Qryn2 + fragment 2; Qryn2 + slice 3 About 3: Based on the previous architecture, which tables need to be written using distributed tables? Regarding the first two architectures, when querying logs, if a fragment does not exist, will it query other nodes? |
@ktpktr0 https://clickhouse.com/docs/en/engines/table-engines/special/distributed#distributed-reading-data In fact you don't need to connect qryn-js to each node of the cluster. Distributed table will do all the work. It's just to spread queries load among the nodes. |
Multiple qryn are connected to the Clickhouse cluster for high availability of qryn. When using distributed write and read, perhaps multiple qryn can be used to connect the same partition? For distributed tabular tables, all tables need to be created and used for querying and writing. Then the example I provided should work. |
@ktpktr0 if you're scaling qryn, the same configuration logic applies to all instances in the same "cluster" - since they are basically stateless, as long as they all talk to the same CH cluster (distributed table or otherwise) things should work and scale fine. You can also have instances dedicated to querying w/o any ingestion if you want to separate the two loads |
Load separation refers to read / write separation? One or more qryn is used to read logs, and it connects the first copy of the Clickhouse fragment; One or more qryns are used to query logs, and it connects the second copy of the Clickhouse partition? |
When I change to a Clickhouse node, it still reports an error
|
Whether a configuration (cluster) can be added. When Clickhouse is a single machine, it is set to no; When Clickhouse is a cluster, it is set to yes. There is no need to manually execute SQL statements. |
@ktpktr0 good point - we'll discuss this and come up with some possibilities such as allowing provisioning templates. note: from the logs you seem to be using an outdated version of qryn |
I look forward to these improvements. In addition, when multiple service logs are stored in one table, will the performance of a single table decrease if the data volume is large or the storage time is long? |
{"level":50,"time":1663951939638,"pid":19,"hostname":"qryn-947657ddb-phphr","name":"qryn","err":"Error: Request failed with status code 404\nResponse: [404] Code: 60. DB::Exception: Table cloki.tempo_traces doesn't exist. (UNKNOWN_TABLE) (version 22.9.2.7 (official build))\n\nError: Request failed with status code 404\n at createError (/app/node_modules/axios/lib/core/createError.js:16:15)\n at settle (/app/node_modules/axios/lib/core/settle.js:17:12)\n at IncomingMessage.handleStreamEnd (/app/node_modules/axios/lib/adapters/http.js:269:11)\n at IncomingMessage.emit (node:events:525:35)\n at endReadableNT (node:internal/streams/readable:1358:12)\n at processTicksAndRejections (node:internal/process/task_queues:83:21)","msg":"Error starting qryn"} |
@panhaitao does a second restart create the tables? |
I set up a Clickhouse cluster locally, which has three shard 2 replicas. I want to know how distributed and cluster replication work in cloki.
I try the following operation. Is it correct?
When I try to start cloki, it has the following error:
The text was updated successfully, but these errors were encountered: