Multi Node Testing Docs and Sample. See #2349

akka-docs/dev/multi-jvm-testing.rst

@@ -2,7 +2,7 @@
 .. _multi-jvm-testing:
 
 ###################
- Multi-JVM Testing
+ Multi JVM Testing
 ###################
 
 Support for running applications (objects with main methods) and
@@ -12,13 +12,13 @@ ScalaTest tests in multiple JVMs.
 Setup
 =====
 
-The multi-JVM testing is an sbt plugin that you can find here:
+The multi JVM testing is an sbt plugin that you can find here:
 
 http://github.com/typesafehub/sbt-multi-jvm
 
-You can add it as a plugin by adding the following to your project/plugins.sbt::
+You can add it as a plugin by adding the following to your project/plugins.sbt:
 
-   addSbtPlugin("com.typesafe.sbt" % "sbt-multi-jvm" % "0.3.0")
+.. includecode:: ../../project/plugins.sbt#sbt-multi-jvm
 
 You can then add multi-JVM testing to ``project/Build.scala`` by including the ``MultiJvm``
 settings and config. For example, here is an example of how the akka-remote-tests project adds
@@ -223,3 +223,10 @@ classpath. Here is a similar example to the one above but using ScalaTest::
 
 To run just these tests you would call ``multi-jvm:test-only sample.Spec`` at
 the sbt prompt.
+
+Multi Node Additions
+====================
+
+There has also been some additions made to the ``SbtMultiJvm`` plugin to accomodate the
+:ref:`experimental <experimental>` module :ref:`multi node testing <multi_node_testing>`,
+described in that section.

akka-docs/dev/multi-node-testing.rst

@@ -0,0 +1,183 @@
+.. _multi_node_testing:
+
+###################
+ Multi Node Testing
+###################
+
+.. note:: This module is :ref:`experimental <experimental>`. This document describes how to use the features
+ implemented so far. More features are coming in Akka Coltrane. Track progress of the Coltrane milestone in
+ `Assembla <http://www.assembla.com/spaces/akka/tickets>`_.
+
+Preparing Your Project for Multi Node Testing
+=============================================
+
+The multi node testing is a separate jar file. Make sure that you have the following dependency in your project:
+
+.. parsed-literal::
+
+  "com.typesafe.akka" %% "akka-remote-tests-experimental" % "2.1-SNAPSHOT"
+
+If you are using the latest nightly build you should pick a timestamped Akka version from
+`<http://repo.typesafe.com/typesafe/snapshots/com/typesafe/akka/>`_. Don't use ``SNAPSHOT``. Note that the
+Scala version |scalaVersion| is part of the artifactId.
+
+Multi Node Testing Concepts
+===========================
+
+Multi node testing in Akka consist of three main parts.
+
+* `The Test Conductor`_. that coordinates and controls the nodes under test.
+* `The Multi Node Spec`_. that is a convenience wrapper for starting the ``TestConductor`` and letting all
+  nodes connect to it.
+* `The SbtMultiJvm Plugin`_. that starts tests in multiple JVMs possibly on multiple machines.
+
+The Test Conductor
+==================
+
+The basis for the multi node testing is the ``TestConductor``. It is an Akka Extension that plugs in to the
+network stack and it is used to coordinate the nodes participating in the test and provides several features
+including:
+
+* Node Address Lookup: Finding out the full path to another test node (No need to share configuration between
+  test nodes)
+* Node Barrier Coordination: Waiting for other nodes at named barriers.
+* Network Failure Injection: Throttling traffic, dropping packets, unplugging and plugging nodes back in.
+
+This is a schematic overview of the test conductor.
+
+.. image:: ../images/akka-remote-testconductor.png
+
+The test conductor server is responsible for coordinating barriers and sending commands to the test conductor
+clients that act upon them, e.g. throttling network traffic to/from another client.
+
+The Multi Node Spec
+===================
+
+The Multi Node Spec consists of two parts. The ``MultiNodeConfig`` that is responsible for common
+configuration and enumerating and naming the nodes under test. The ``MultiNodeSpec`` that contains all the
+convenience functions for making the test nodes interact with each other.
+
+The setup of the ``MultiNodeSpec`` is configured through properties that you set on all JVMs that's going to run a
+node under test.
+
+These are the available properties:
+  * ``multinode.max-nodes``
+
+    The maximum number of nodes that a test can have.
+
+  * ``multinode.host``
+
+    The host name or IP for this node. Must be resolvable using InetAddress.getByName.
+
+  * ``multinode.port``
+
+    The port number for this node. Defaults to 0 which will use a random port.
+
+  * ``multinode.server-host``
+
+    The host name or IP for the server node. Must be resolvable using InetAddress.getByName.
+
+  * ``multinode.server-port``
+
+    The port number for the server node. Defaults to 4711.
+
+  * ``multinode.index``
+
+    The index of this node in the sequence of roles defined for the test. The index 0 is special and that machine
+    will be the server. All failure injection and throttling must be done from this node.
+
+The SbtMultiJvm Plugin
+======================
+
+The :ref:`SbtMultiJvm Plugin <multi_jvm_testing>` has been updated to be able to run multi node tests, by
+automatically generating the relevant ``multinode.*`` properties. This means that you can easily run multinode tests
+on a single machine by just running them as normal multi jvm tests.
+
+Multi Node Specific Additions
++++++++++++++++++++++++++++++
+
+The plugin also has a number of new ``multi-node-*`` sbt tasks and settings to support running tests on multiple
+machines. The necessary test classes and dependencies are packaged for distribution to other machines with
+`SbtAssembly <https://github.com/sbt/sbt-assembly>`_ into a jar file with a name on the format
+``<projectName>_<scalaVersion>-<projectVersion>-multi-jvm-assembly.jar``
+
+.. note::
+
+   To be able to distribute and kick off the tests on multiple machines, it is assumed that both host and target
+   systems are POSIX like systems with `ssh` and `rsync` available.
+
+These are the available sbt multi-node settings:
+  * ``multiNodeHosts``
+
+    A sequence of hosts to use for running the test, on the form ``user@host:java`` where host is the only required
+    part. Will override settings from file.
+
+  * ``multiNodeHostsFileName``
+
+    A file to use for reading in the hosts to use for running the test. One per line on the same format as above.
+    Defaults to ``multi-node-test.hosts`` in the base project directory.
+
+  * ``multiNodeTargetDirName``
+
+    A name for the directory on the target machine, where to copy the jar file. Defaults to ``multi-node-test`` in
+    the base directory of the ssh user used to rsync the jar file.
+
+  * ``multiNodeJavaName``
+
+    The name of the default Java executable on the target machines. Defaults to ``java``.
+
+Here are some examples of how you define hosts:
+  * ``localhost``
+
+    The current user on localhost using the default java.
+
+  * ``user1@host1``
+
+    User ``user1`` on host ``host1`` with the default java.
+
+  * ``user2@host2:/usr/lib/jvm/java-7-openjdk-amd64/bin/java``
+
+    User ``user2`` on host ``host2`` using java 7.
+
+  * ``host3:/usr/lib/jvm/java-6-openjdk-amd64/bin/java``
+
+    The current user on host ``host3`` using java 6.
+
+Running the Test in Multi Node Mode
++++++++++++++++++++++++++++++++++++
+
+To run all the multi node test in multi-node mode (i.e. distributing the jar files and rkicking off the tests
+remotely) from inside sbt, use the ``multi-node-test`` task:
+
+.. code-block:: none
+
+  multi-node-test
+
+To run individual tests use the ``multi-node-test-only`` task:
+
+.. code-block:: none
+
+  multi-node-test-only akka.remote.RandomRoutedRemoteActor
+
+More than one test name can be listed to run multiple specific tests. Tab completion in sbt makes it easy to
+complete the test names.
+
+A Multi Node Testing Example
+============================
+
+First we need some scaffolding to hook up the `MultiNodeSpec` with your favorite test framework. Lets define a trait
+``STMultiNodeSpec`` that uses ScalaTest to start and stop ``MultiNodeSpec``.
+
+.. includecode:: ../../akka-samples/akka-sample-multi-node/src/test/scala/sample/multinode/STMultiNodeSpec.scala#example
+
+Then we need to define a configuration. Lets use two nodes ``"node1`` and ``"node2"`` and call it
+``MultiNodeSampleConfig``.
+
+.. includecode:: ../../akka-samples/akka-sample-multi-node/src/multi-jvm/scala/sample/multinode/MultiNodeSample.scala
+  :include: package,config
+
+And then finally to the node test code. That starts the two nodes, and demostrates a barrier, and a remote actor
+message send/receive.
+
+.. includecode:: ../../akka-samples/akka-sample-multi-node/src/multi-jvm/scala/sample/multinode/MultiNodeSample.scala
+  :include: package,spec

akka-docs/experimental/index.rst

@@ -24,4 +24,4 @@ of the module over time.
    :maxdepth: 2
 
    ../cluster/index
-
+   ../dev/multi-node-testing

akka-samples/akka-sample-multi-node/src/multi-jvm/scala/sample/multinode/MultiNodeSample.scala

@@ -0,0 +1,59 @@
+/**
+ *  Copyright (C) 2009-2012 Typesafe Inc. <http://www.typesafe.com>
+ */
+//#package
+package sample.multinode
+//#package
+
+//#config
+import akka.remote.testkit.MultiNodeConfig
+
+object MultiNodeSampleConfig extends MultiNodeConfig {
+  val node1 = role("node1")
+  val node2 = role("node2")
+}
+//#config
+
+//#spec
+import akka.remote.testkit.MultiNodeSpec
+import akka.testkit.ImplicitSender
+import akka.actor.{Props, Actor}
+
+class MultiNodeSampleSpecMultiJvmNode1 extends MultiNodeSample
+class MultiNodeSampleSpecMultiJvmNode2 extends MultiNodeSample
+
+class MultiNodeSample extends MultiNodeSpec(MultiNodeSampleConfig)
+  with STMultiNodeSpec with ImplicitSender {
+
+  import MultiNodeSampleConfig._
+
+  def initialParticipants = 2
+
+  "A MultiNodeSample" must {
+
+    "wait for all nodes to enter a barrier" in {
+      enterBarrier("startup")
+    }
+
+    "send to and receive from a remote node" in {
+      runOn(node1) {
+        enterBarrier("deployed")
+        val ponger = system.actorFor(node(node2).toString + "user/ponger")
+        ponger ! "ping"
+        expectMsg("pong")
+      }
+
+      runOn(node2) {
+        system.actorOf(Props(new Actor {
+          def receive = {
+            case "ping" => sender ! "pong"
+          }
+        }), "ponger")
+        enterBarrier("deployed")
+      }
+
+      enterBarrier("finished")
+    }
+  }
+}
+//#spec

akka-samples/akka-sample-multi-node/src/test/scala/sample/multinode/STMultiNodeSpec.scala

@@ -0,0 +1,25 @@
+/**
+ *  Copyright (C) 2009-2012 Typesafe Inc. <http://www.typesafe.com>
+ */
+//#example
+package sample.multinode
+
+//#imports
+import org.scalatest.{ BeforeAndAfterAll, WordSpec }
+import org.scalatest.matchers.MustMatchers
+import akka.remote.testkit.MultiNodeSpecCallbacks
+//#imports
+
+//#trait
+/**
+ * Hooks up MultiNodeSpec with ScalaTest
+ */
+trait STMultiNodeSpec extends MultiNodeSpecCallbacks
+  with WordSpec with MustMatchers with BeforeAndAfterAll {
+
+  override def beforeAll() = multiNodeSpecBeforeAll()
+
+  override def afterAll() = multiNodeSpecAfterAll()
+}
+//#trait
+//#example

project/AkkaBuild.scala

@@ -286,60 +286,71 @@ object AkkaBuild extends Build {
     id = "akka-samples",
     base = file("akka-samples"),
     settings = parentSettings,
-    aggregate = Seq(camelSample, fsmSample, helloSample, helloKernelSample, remoteSample, clusterSample)
+    aggregate = Seq(camelSample, fsmSample, helloSample, helloKernelSample, remoteSample, clusterSample, multiNodeSample)
   )
 
   lazy val camelSample = Project(
     id = "akka-sample-camel",
     base = file("akka-samples/akka-sample-camel"),
     dependencies = Seq(actor, camel),
-    settings = defaultSettings ++ Seq(
-      libraryDependencies ++= Dependencies.camelSample,
-      publishArtifact in Compile := false
-    )
+    settings = sampleSettings ++ Seq(libraryDependencies ++= Dependencies.camelSample)
   )
 
   lazy val fsmSample = Project(
     id = "akka-sample-fsm",
     base = file("akka-samples/akka-sample-fsm"),
     dependencies = Seq(actor),
-    settings = defaultSettings ++ Seq( publishArtifact in Compile := false )
+    settings = sampleSettings
   )
 
   lazy val helloSample = Project(
     id = "akka-sample-hello",
     base = file("akka-samples/akka-sample-hello"),
     dependencies = Seq(actor),
-    settings = defaultSettings ++ Seq( publishArtifact in Compile := false )
+    settings = sampleSettings
   )
 
   lazy val helloKernelSample = Project(
     id = "akka-sample-hello-kernel",
     base = file("akka-samples/akka-sample-hello-kernel"),
     dependencies = Seq(kernel),
-    settings = defaultSettings ++ Seq( publishArtifact in Compile := false )
+    settings = sampleSettings
   )
 
   lazy val remoteSample = Project(
     id = "akka-sample-remote",
     base = file("akka-samples/akka-sample-remote"),
     dependencies = Seq(actor, remote, kernel),
-    settings = defaultSettings ++ Seq( publishArtifact in Compile := false )
+    settings = sampleSettings
   )
 
   lazy val clusterSample = Project(
     id = "akka-sample-cluster-experimental",
     base = file("akka-samples/akka-sample-cluster"),
     dependencies = Seq(cluster, remoteTests % "compile;test->test;multi-jvm->multi-jvm", testkit % "test->test"),
-    settings = defaultSettings ++ multiJvmSettings ++ Seq(
+    settings = sampleSettings ++ multiJvmSettings ++ Seq(
       // disable parallel tests
       parallelExecution in Test := false,
       extraOptions in MultiJvm <<= (sourceDirectory in MultiJvm) { src =>
         (name: String) => (src ** (name + ".conf")).get.headOption.map("-Dakka.config=" + _.absolutePath).toSeq
       },
       scalatestOptions in MultiJvm := defaultMultiJvmScalatestOptions,
-      jvmOptions in MultiJvm := defaultMultiJvmOptions,
-      publishArtifact in Compile := false
+      jvmOptions in MultiJvm := defaultMultiJvmOptions
+    )
+  ) configs (MultiJvm)
+
+  lazy val multiNodeSample = Project(
+    id = "akka-sample-multi-node-experimental",
+    base = file("akka-samples/akka-sample-multi-node"),
+    dependencies = Seq(remoteTests % "test", testkit % "test"),
+    settings = sampleSettings ++ multiJvmSettings ++ Seq(
+      libraryDependencies ++= Dependencies.multiNodeSample,
+      // disable parallel tests
+      parallelExecution in Test := false,
+      extraOptions in MultiJvm <<= (sourceDirectory in MultiJvm) { src =>
+        (name: String) => (src ** (name + ".conf")).get.headOption.map("-Dakka.config=" + _.absolutePath).toSeq
+      },
+      jvmOptions in MultiJvm := defaultMultiJvmOptions
     )
   ) configs (MultiJvm)
 
@@ -377,6 +388,10 @@ object AkkaBuild extends Build {
     publishArtifact in Compile := false
   )
 
+  lazy val sampleSettings = defaultSettings ++ Seq(
+    publishArtifact in Compile := false
+  )
+
   val excludeTestNames = SettingKey[Seq[String]]("exclude-test-names")
   val excludeTestTags = SettingKey[Set[String]]("exclude-test-tags")
   val includeTestTags = SettingKey[Set[String]]("include-test-tags")
@@ -593,6 +608,8 @@ object Dependencies {
   val docs = Seq(Test.scalatest, Test.junit, Test.junitIntf)
 
   val zeroMQ = Seq(protobuf, zeroMQClient, Test.scalatest, Test.junit)
+
+  val multiNodeSample = Seq(Test.scalatest)
 }
 
 object Dependency {

project/plugins.sbt

@@ -1,7 +1,10 @@
 
 resolvers += Classpaths.typesafeResolver
 
+// these comment markers are for including code into the docs
+//#sbt-multi-jvm
 addSbtPlugin("com.typesafe.sbt" % "sbt-multi-jvm" % "0.3.0")
+//#sbt-multi-jvm
 
 addSbtPlugin("com.typesafe.sbtscalariform" % "sbtscalariform" % "0.4.0")
 
@@ -9,10 +12,4 @@ addSbtPlugin("com.typesafe.sbtosgi" % "sbtosgi" % "0.3.0")
 
 addSbtPlugin("com.typesafe" % "sbt-mima-plugin" % "0.1.3")
 
-resolvers ++= Seq(
-  // needed for sbt-assembly, which comes with sbt-multi-jvm
-  Resolver.url("sbtonline", url("http://scalasbt.artifactoryonline.com/scalasbt/sbt-plugin-releases"))(Resolver.ivyStylePatterns),
-  "less is" at "http://repo.lessis.me",
-  "coda" at "http://repo.codahale.com")
-
 // addSbtPlugin("me.lessis" % "ls-sbt" % "0.1.1")