=clt #16948 Use min retries for singleton leaving scenario"

* In 2.4 we derive the number of hand-over/take-over retries from
  the removal margin, but we decided to set that to 0 by default, since
  it is intended for network partition scenarios. maxTakeOverRetries
  became 1. So there must be also be a  min number of retries property.
* The test failed for the leaving scenario because the singleton
  instance was stopped hard without sending the terminationMessage when
  the maxTakeOverRetries was exceeded.

akka-cluster-tools/src/main/scala/akka/cluster/singleton/ClusterSingletonManager.scala

@@ -387,7 +387,13 @@ class ClusterSingletonManager(
 
   val (maxHandOverRetries, maxTakeOverRetries) = {
     val n = (removalMargin.toMillis / handOverRetryInterval.toMillis).toInt
-    (n + 3, math.max(1, n - 3))
+    val minRetries = context.system.settings.config.getInt(
+      "akka.cluster.singleton.min-number-of-hand-over-retries")
+    require(minRetries >= 1, "min-number-of-hand-over-retries must be >= 1")
+    val handOverRetries = math.max(minRetries, n + 3)
+    val takeOverRetries = math.max(1, handOverRetries - 3)
+
+    (handOverRetries, takeOverRetries)
   }
 
   // started when when self member is Up