Commit graph

148 commits

Author SHA1 Message Date
Patrik Nordwall
6ed3295acd Merge branch 'master' into wip-multi-dc-merge-master-patriknw 2017-08-31 10:51:12 +02:00
Lukas Phaf
06e00fdf62 DeadLetterSuppression for Replicator.UpdateSuccess in distributed data #23553 2017-08-22 14:49:40 +02:00
Jimin Hsieh
f623d10522 Rename addr to address in non-public API #21874 2017-08-08 13:18:56 +02:00
Martynas Mickevičius
bc0f2ee26d Load MiMa filters from file (#23083) 2017-07-27 12:33:14 +02:00
Patrik Nordwall
fe2bf91659 increase timeout due to sometimes slow initialization of lmdb, #23284 (#23302) 2017-07-07 04:12:02 +02:00
Patrik Nordwall
0b1ce7223d fix usage of the the leader sorted set in Replicator
* since the ordering can change based on the member's status
  it's not possible to use ordinary - for removal
* similar issue at a few places where ageOrdering was used
2017-07-05 14:56:12 +02:00
Patrik Nordwall
bb9549263e Rename team to data center, #23275 2017-07-04 17:11:21 +02:00
Patrik Nordwall
e0fe0bc49e Make cluster sharding DC aware, #23231
* Sharding only within own team (coordinator is singleton)
* the ddata Replicator used by Sharding must also be only within own team
* added support for Set of roles in ddata Replicator so that can be used
  by sharding to specify role + team
* Sharding proxy can route to sharding in another team
2017-07-04 15:04:43 +02:00
Patrik Nordwall
1a732fba71 slow init of lmdb in DurableDataSpe, #22082 2017-06-22 15:08:37 +02:00
Patrik Nordwall
0d890e6364 enable debug logging in DurableDataSpec, #22082 2017-06-20 20:50:28 +02:00
Patrik Nordwall
2970287f95 discard large deltas, #23025
* to avoid OversizedPayloadException
* some complex deltas grow for each update operation, e.g.
  when updating different keys in ORMap (PNCounterMap)
* such large deltas can safely be discarded and disseminated as full
  state instead
* added ReplicatedDeltaSize interface to be able to define the "size"
  and when that size exceeds configured threshold the delta is discarded
2017-06-16 14:20:26 +02:00
Patrik Nordwall
edef9e34c7 serialize-creators=off in tests, #23003 2017-05-22 20:11:03 +02:00
Philippus Baalman
ef9c7313b6 Extend copyright into 2017 (#22833) 2017-05-04 15:14:33 +02:00
Patrik Nordwall
3ab101039f Lazy init of LmdbDurableStore, #22759 (#22779)
* Lazy init of LmdbDurableStore, #22759

* to avoid creating files (and initializing db) when not needed,
  e.g. cluster sharding that is not using remember entities
* enable MiMa against 2.5.0

* use OptionVal instead
2017-04-28 15:12:14 +02:00
gosubpl
0bbb6900d7 fix race condition in ReplicatorDeltaSpec (#22737) 2017-04-21 16:18:49 +02:00
gosubpl
3a8eef4506 fixes to ORSet mergeRemoveDelta and ORMap deltaMerge (#22648) 2017-04-03 18:15:53 +02:00
Patrik Nordwall
5293666a30 Use full state for the ORSet and ORMap removals until #22648 is fixed
* some tests disabled, search for
  "FIXME use full state for removals, until issue #22648 is fixed"
  to find them
2017-03-30 13:26:35 +02:00
Patrik Nordwall
dc070e4e1c fix wrong log in receiveDeltaPropagation 2017-03-30 13:02:10 +02:00
Patrik Nordwall
9855e2896f Handle delta None correctly, #22655
When a DeltaReplicatedData returns None from delta it must still be
treated as a delta that increase the version counter in DeltaPropagationSelector.
Otherwise a later delta might be applied before the full state gossip is received
and thereby violating RequiresCausalDeliveryOfDeltas.
2017-03-30 08:12:03 +02:00
Patrik Nordwall
24ddc8f381 Merge pull request #22634 from gosubpl/wip/22350-coalescing-deltas
delta-aggregation in the ORMap deltas (#22633)
2017-03-28 20:28:21 +02:00
gosubpl
7c42627ea9 delta-aggregation in the ORMap deltas (#22633) 2017-03-27 23:51:09 +02:00
Björn Antonsson
f8b4fb55ca Remove use of deprecated Scala features #22581 2017-03-27 19:05:54 +03:00
gosubpl
2caef783dc fix ORMap duplicated delta propagation (#22606) 2017-03-23 10:49:42 +01:00
gosubpl
dfbf50538e fixed serialization of RemoveKey 2017-03-21 09:43:33 +02:00
Patrik Nordwall
cc7065601a fix serialization of delta ops, #22604 2017-03-21 09:43:27 +02:00
Patrik Nordwall
191bd0fe41 =str #22584 make subfusing work if diff materialize() is called
Also: removed unecessary method with wrong name, #22350
2017-03-17 12:42:13 +01:00
Patrik Nordwall
edee6ae544 Merge pull request #22508 from gosubpl/wip/22350-delta-crdt-orset-friends
ORMap and friends are delta-CRDTs (#22350)
2017-03-17 10:27:52 +01:00
gosubpl
1f2ef60174 ORMap and friends have deltas (#22350) 2017-03-17 02:47:50 +01:00
Johan Andrén
d51f7abd63 Give a good error message on unassigned ports #22552 2017-03-15 18:03:52 +01:00
Patrik Nordwall
6b5b819c73 Merge pull request #22378 from akka/wip-delta-in-direct-write-patriknw
Use delta in direct write also, #22188
2017-03-15 16:19:29 +01:00
Patrik Nordwall
233e784154 Use delta in direct write also, #22188
* Follow up on the causal delivery of deltas.
* The first implementation used full state for the direct
  Write messages, i.e. updates with WriteConsistency != LocalWrite
* This is an optimization so that delatas are tried first and if
  they can't be applied it falls back to full state.
* For simultanious updates the messages may be reordered because we
  create separate WriteAggregator actor and such, but normally they
  will be sent in order so the deltas will typically be received in
  order, otherwise we fall back to retrying with full state in the
  second round in the WriteAggregator.
2017-03-15 11:38:44 +01:00
Johan Andrén
1e6b8ec86b Make sure the hashcode is not negative when selecting key chunk #22487 2017-03-13 17:57:03 +01:00
Johan Andrén
7a0e5b31f8 Avoid Array.ofDim where possible #22516 2017-03-13 17:49:45 +01:00
ortigali
adf7e8271e less verbose debug logging in ddata (#22384)
* less verbose debug logging in ddata #22170

* less verbose debug logging in ddata 2 #22170
2017-02-24 10:38:14 +01:00
Patrik Nordwall
f2fe8582fa fix serialization of pruning in DurableDataEnvelope 2017-02-22 15:22:44 +01:00
gosubpl
3d51111b3c proper implementation of mergeRemoveDelta, #22188 2017-02-22 14:27:36 +01:00
Patrik Nordwall
b700b840d1 causal delivery of deltas, #22188
* keep track of delta interval versions and skip deltas
  that are not consequtive, i.e. when some delta message was lost
* send the delta versions in the full state gossip to sync up the
  expected versions after dropped deltas
* implementation of deltas for ORSet
* refactoring of the delta types to allow for different type for the
  delta and the full state
* extensive tests
* mima filter
* performance optimizations
* simple pruning of deltas
* Java API
* update documentation
* KeyId type alias
* Use InternalApi annotation
2017-02-22 14:27:36 +01:00
gosubpl
d470321051 GSet ported to delta-CRDT (#22187) 2017-02-16 13:06:57 +01:00
Patrik Nordwall
9df49be5d0 Enable additional-serialization-bindings by default, #22169
* and thereby it was also possibe to simplify the documentation
  in remoting
2017-01-24 18:52:49 +01:00
Patrik Nordwall
3e7ffd6b96 Add support for delta-CRDT, #21875
* delta GCounter and PNCounter
* first stab at delta propagation protocol
* send delta in the direct write
* possibility to turn off delta propagation
* tests
* protobuf serializer for DeltaPropagation
* documentation
2017-01-24 13:24:45 +01:00
Patrik Nordwall
952be31a7d make pruning of CRDT garbage work, #21647
* fix merge issues of DataEnvelope and its pruning
* simplify by removing the tombstones, which didn't work in all cases anyway
* keep the PruningPerformed markers in the DataEnvelope until configured
  TTL has elapsed (wall clock)
* simplify PruningState structure
* also store the pruning markers in durable data
* collect removed nodes from the data, listing on MemberRemoved is not enough
* possibility to disable pruning altogether
* documented caveat for durable data
2017-01-23 16:54:02 +01:00
Patrik Nordwall
37679d307e rememberingEntities with ddata mode, #22154
* one Replicator per configured role
* log LMDB directory at startup
* clarify the imporantce of the LMDB directory
* use more than one key to support many entities
2017-01-23 11:57:52 +01:00
Patrik Nordwall
572d0c8040 stash messages while Replicator is loading durable data, #22150 2017-01-19 12:55:28 +01:00
inakov
e043a9fffe Add minimum cap for ReadMajority/WriteMajority #21618 2017-01-18 14:13:40 +02:00
Johan "Party Cannon" Andrén
91d6a3f125 #22147 show the cause of the error stopping durable store 2017-01-18 09:19:49 +01:00
Patrik Nordwall
14e0a70b46 Merge pull request #22142 from inakov/wip-distributed-data-delete-api-update-new-inakov
Update DistributedData Delete API to include optional request context. Clear Pull Request
2017-01-17 17:15:15 +01:00
Patrik Nordwall
fcddd5fa72 Merge pull request #22144 from jgordijn/22140-test-failure-ReplicatedDataSerializer-jgordijn
#22140 fixed ReplicatedDataSerializerSpec error with artery serializers
2017-01-17 10:51:29 +01:00
Patrik Nordwall
84ade6fdc3 add CoordinatedShutdown, #21537
* CoordinatedShutdown that can run tasks for configured phases in order (DAG)
* coordinate handover/shutdown of singleton with cluster exiting/shutdown
* phase config obj with depends-on list
* integrate graceful leaving of sharding in coordinated shutdown
* add timeout and recover
* add some missing artery ports to tests
* leave via CoordinatedShutdown.run
* optionally exit-jvm in last phase
* run via jvm shutdown hook
* send ExitingConfirmed to leader before shutdown of Exiting
  to not have to wait for failure detector to mark it as
  unreachable before removing
* the unreachable signal is still kept as a safe guard if
  message is lost or leader dies
* PhaseClusterExiting vs MemberExited in ClusterSingletonManager
* terminate ActorSystem when cluster shutdown (via Down)
* add more predefined and custom phases
* reference documentation
* migration guide
* problem when the leader order was sys2, sys1, sys3,
  then sys3 could not perform it's duties and move Leving sys1 to
  Exiting because it was observing sys1 as unreachable
* exclude Leaving with exitingConfirmed from convergence condidtion
2017-01-16 09:01:57 +01:00
Jeroen Gordijn
55a1532e56 #22140 fixed ReplicatedDataSerializerSpec error with artery serializers 2017-01-16 08:51:40 +01:00
inakov
65cba329d0 Updated DistributedData Delete API to include optional request context.
#20140
2017-01-14 14:30:58 +02:00