HBASE-28608 More sensible client meta operation timeout default

[droudnitsky]

https://issues.apache.org/jira/browse/HBASE-28608

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 25s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 44s	master passed
+1 💚	compile	0m 17s	master passed
+1 💚	shadedjars	5m 41s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 17s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 24s	the patch passed
+1 💚	compile	0m 16s	the patch passed
+1 💚	javac	0m 16s	the patch passed
-1 ❌	shadedjars	2m 23s	patch has 10 errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 14s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 25s	hbase-client in the patch passed.
		17m 7s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#6000
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 4885aa4a1a7f 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / cd4c5c3
Default Java	Temurin-1.8.0_352-b08
shadedjars	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/1/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/1/testReport/
Max. process+thread count	306 (vs. ulimit of 30000)
modules	C: hbase-client U: hbase-client
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 13s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 1s	master passed
+1 💚	compile	0m 21s	master passed
+1 💚	shadedjars	5m 13s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 20s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 47s	the patch passed
+1 💚	compile	0m 22s	the patch passed
+1 💚	javac	0m 22s	the patch passed
-1 ❌	shadedjars	2m 15s	patch has 10 errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 18s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 44s	hbase-client in the patch passed.
		17m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6000
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 02f0f7f27169 5.4.0-182-generic #202-Ubuntu SMP Fri Apr 26 12:29:36 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / cd4c5c3
Default Java	Eclipse Adoptium-17.0.10+7
shadedjars	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/1/artifact/yetus-jdk17-hadoop3-check/output/patch-shadedjars.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/1/testReport/
Max. process+thread count	316 (vs. ulimit of 30000)
modules	C: hbase-client U: hbase-client
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 39s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 55s	master passed
+1 💚	compile	0m 21s	master passed
+1 💚	shadedjars	5m 17s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 18s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 48s	the patch passed
+1 💚	compile	0m 21s	the patch passed
+1 💚	javac	0m 21s	the patch passed
-1 ❌	shadedjars	2m 43s	patch has 10 errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 18s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 55s	hbase-client in the patch passed.
		19m 35s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#6000
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux a3ec4dffb6db 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / cd4c5c3
Default Java	Eclipse Adoptium-11.0.17+8
shadedjars	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/1/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/1/testReport/
Max. process+thread count	294 (vs. ulimit of 30000)
modules	C: hbase-client U: hbase-client
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 26s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 19s	master passed
+1 💚	compile	0m 42s	master passed
+1 💚	checkstyle	0m 15s	master passed
+1 💚	spotless	0m 46s	branch has no errors when running spotless:check.
+1 💚	spotbugs	0m 47s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 0s	the patch passed
+1 💚	compile	0m 42s	the patch passed
+1 💚	javac	0m 42s	the patch passed
+1 💚	checkstyle	0m 15s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	5m 44s	Patch does not cause any errors with Hadoop 3.3.6.
-1 ❌	spotless	0m 14s	patch has 34 errors when running spotless:check, run spotless:apply to fix.
+1 💚	spotbugs	0m 51s	the patch passed
		_ Other Tests _
-1 ❌	asflicense	0m 10s	The patch generated 1 ASF License warnings.
		24m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6000
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 4a5f496ede6e 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / cd4c5c3
Default Java	Eclipse Adoptium-11.0.17+8
spotless	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/1/artifact/yetus-general-check/output/patch-spotless.txt
asflicense	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/1/artifact/yetus-general-check/output/patch-asflicense-problems.txt
Max. process+thread count	77 (vs. ulimit of 30000)
modules	C: hbase-client U: hbase-client
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/1/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[droudnitsky]

fixed spotless check

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 26s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 22s	master passed
+1 💚	compile	0m 16s	master passed
+1 💚	shadedjars	5m 40s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 14s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 27s	the patch passed
+1 💚	compile	0m 17s	the patch passed
+1 💚	javac	0m 17s	the patch passed
+1 💚	shadedjars	5m 42s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 14s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 26s	hbase-client in the patch passed.
		20m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#6000
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 3363826a38ae 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / cd4c5c3
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/2/testReport/
Max. process+thread count	301 (vs. ulimit of 30000)
modules	C: hbase-client U: hbase-client
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 13s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 45s	master passed
+1 💚	compile	0m 23s	master passed
+1 💚	shadedjars	5m 11s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 18s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 48s	the patch passed
+1 💚	compile	0m 22s	the patch passed
+1 💚	javac	0m 22s	the patch passed
+1 💚	shadedjars	5m 11s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 18s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 44s	hbase-client in the patch passed.
		20m 5s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6000
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux faf05cec2766 5.4.0-182-generic #202-Ubuntu SMP Fri Apr 26 12:29:36 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / cd4c5c3
Default Java	Eclipse Adoptium-17.0.10+7
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/2/testReport/
Max. process+thread count	311 (vs. ulimit of 30000)
modules	C: hbase-client U: hbase-client
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 39s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 45s	master passed
+1 💚	compile	0m 20s	master passed
+1 💚	shadedjars	5m 18s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 17s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 46s	the patch passed
+1 💚	compile	0m 20s	the patch passed
+1 💚	javac	0m 20s	the patch passed
+1 💚	shadedjars	5m 19s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 17s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 38s	hbase-client in the patch passed.
		20m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#6000
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 7a73ef98eb37 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / cd4c5c3
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/2/testReport/
Max. process+thread count	287 (vs. ulimit of 30000)
modules	C: hbase-client U: hbase-client
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 23s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 15s	master passed
+1 💚	compile	0m 40s	master passed
+1 💚	checkstyle	0m 15s	master passed
+1 💚	spotless	0m 45s	branch has no errors when running spotless:check.
+1 💚	spotbugs	0m 45s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 58s	the patch passed
+1 💚	compile	0m 39s	the patch passed
+1 💚	javac	0m 39s	the patch passed
+1 💚	checkstyle	0m 14s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	5m 38s	Patch does not cause any errors with Hadoop 3.3.6.
+1 💚	spotless	0m 42s	patch has no errors when running spotless:check.
+1 💚	spotbugs	0m 48s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 10s	The patch does not generate ASF License warnings.
		24m 3s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6000
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux f0e229b4f262 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / cd4c5c3
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	78 (vs. ulimit of 30000)
modules	C: hbase-client U: hbase-client
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6000/2/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache9]

For me this is reasonable. But anyway this is a behavior change, better start a discuss thread on the dev list to see if others have other opinions.

Thanks.

[droudnitsky]

Thank you for reviewing @Apache9 . I was originally under the impression that the default behavior was unintentional given the documentation in the hbase reference and the discussion in the work done in HBASE-24956 which added a new critical dependence on meta operation timeout in the locate region/meta lookup codepath in 2.x, but this may not have been unintentional as I originally thought. There seem to be two very different dependencies on meta operation timeout on 2.x:

1. End to end operation timeout for system table operations (2.x and 3)
2. Timeout to acquire userRegionLock to initiate meta scan in locateRegionInMeta (2.x blocking client only), this is the dependence on the meta operation timeout property that brought the 20 min default to my attention.

I am not familiar with how the locate region codepath on branch 3 works. But for branch 2 and the userRegionLock timeout, I don't think it makes sense to have default behavior be such that someone configuring client operation timeout can have operations taking many times longer than their client operation timeout due to contention around userRegionLock if they have not set the meta operation timeout.

I think it may be worth decoupling the userRegionLock timeout from the meta operation timeout, since we have one property that will dictate the behavior of two very different things on 2.x . I will put some more research and thought into this and perhaps raise a seperate issue to propose decoupling userRegionLock timeout from meta table operation timeout, which should make it simpler to reason about a default for meta operation timeout here. I'll start a dev list thread once I have gathered my thoughts on this, would be great to get more opinions on this, thank you for the suggestion and review.