add LoadSnapshotFailed in snapshot protocol, #21842

* treat snapshot load failure in same way as other recovery failures
* if load of snapshot fails the persistent actor will be stopped, since
  we can't assume that a consistent state would be recovered just by
  replaying all events, since events may have been  deleted
* additional recovery docs
* improve log message

akka-docs/rst/java/code/docs/persistence/LambdaPersistenceDocTest.java

@@ -108,6 +108,15 @@ public class LambdaPersistenceDocTest {
     }
     //#recovery-completed
 
+    abstract class MyPersistentActor6 extends AbstractPersistentActor {
+      //#recovery-no-snap
+      @Override
+      public Recovery recovery() {
+        return Recovery.create(SnapshotSelectionCriteria.none());
+      }
+      //#recovery-no-snap
+    }
+
     abstract class MyActor extends AbstractPersistentActor {
       //#backoff
       @Override

akka-docs/rst/java/code/docs/persistence/PersistenceDocTest.java

@@ -85,6 +85,15 @@ public class PersistenceDocTest {
           }
           //#recovery-completed
         }
+
+        abstract class MyPersistentActor6 extends UntypedPersistentActor {
+            //#recovery-no-snap
+            @Override
+            public Recovery recovery() {
+                return Recovery.create(SnapshotSelectionCriteria.none());
+            }
+            //#recovery-no-snap
+        }
         
         abstract class MyActor extends UntypedPersistentActor {
           //#backoff

akka-docs/rst/java/lambda-persistence.rst

@@ -160,12 +160,25 @@ only be received by a persistent actor after recovery completes.
   as the original sender is presumed to be long gone. If you indeed have to notify an actor during
   recovery in the future, store its ``ActorPath`` explicitly in your persisted events.
 
+.. _recovery-custom-java-lambda:
+
 Recovery customization
 ^^^^^^^^^^^^^^^^^^^^^^
 
 Applications may also customise how recovery is performed by returning a customised ``Recovery`` object
-in the ``recovery`` method of a ``AbstractPersistentActor``, for example setting an upper bound to the replay
-which allows the actor to be replayed to a certain point "in the past" instead to its most up to date state:
+in the ``recovery`` method of a ``AbstractPersistentActor``.
+
+To skip loading snapshots and replay all events you can use ``SnapshotSelectionCriteria.none()``.
+This can be useful if snapshot serialization format has changed in an incompatible way.
+It should typically not be used when events have been deleted.
+
+.. includecode:: code/docs/persistence/LambdaPersistenceDocTest.java#recovery-no-snap
+
+Another example, which can be fun for experiments but probably not in a real application, is setting an 
+upper bound to the replay which allows the actor to be replayed to a certain point "in the past" 
+instead to its most up to date state. Note that after that it is a bad idea to persist new 
+events because a later recovery will probably be confused by the new events that follow the 
+events that were previously skipped.
 
 .. includecode:: code/docs/persistence/LambdaPersistenceDocTest.java#recovery-custom
 
@@ -339,6 +352,8 @@ next message.
 
 If there is a problem with recovering the state of the actor from the journal when the actor is
 started, ``onRecoveryFailure`` is called (logging the error by default), and the actor will be stopped.
+Note that failure to load snapshot is also treated like this, but you can disable loading of snapshots
+if you for example know that serialization format has changed in an incompatible way, see :ref:`recovery-custom-java-lambda`.
 
 Atomic writes
 -------------

akka-docs/rst/java/persistence.rst

@@ -168,12 +168,25 @@ They are cached and received by a persistent actor after recovery phase complete
   as the original sender is presumed to be long gone. If you indeed have to notify an actor during
   recovery in the future, store its ``ActorPath`` explicitly in your persisted events.
 
+.. _recovery-custom-java:
+
 Recovery customization
 ^^^^^^^^^^^^^^^^^^^^^^
 
 Applications may also customise how recovery is performed by returning a customised ``Recovery`` object
-in the ``recovery`` method of a ``UntypedPersistentActor``, for example setting an upper bound to the replay
-which allows the actor to be replayed to a certain point "in the past" instead to its most up to date state:
+in the ``recovery`` method of a ``UntypedPersistentActor``.
+
+To skip loading snapshots and replay all events you can use ``SnapshotSelectionCriteria.none()``.
+This can be useful if snapshot serialization format has changed in an incompatible way.
+It should typically not be used when events have been deleted.
+
+.. includecode:: code/docs/persistence/PersistenceDocTest.java#recovery-no-snap
+
+Another example, which can be fun for experiments but probably not in a real application, is setting an 
+upper bound to the replay which allows the actor to be replayed to a certain point "in the past" 
+instead to its most up to date state. Note that after that it is a bad idea to persist new 
+events because a later recovery will probably be confused by the new events that follow the 
+events that were previously skipped.
 
 .. includecode:: code/docs/persistence/PersistenceDocTest.java#recovery-custom
 
@@ -359,6 +372,8 @@ next message.
 
 If there is a problem with recovering the state of the actor from the journal when the actor is
 started, ``onRecoveryFailure`` is called (logging the error by default), and the actor will be stopped.
+Note that failure to load snapshot is also treated like this, but you can disable loading of snapshots
+if you for example know that serialization format has changed in an incompatible way, see :ref:`recovery-custom-java`.
 
 Atomic writes
 -------------