Commit graph

43 commits

Author SHA1 Message Date
Konrad `ktoso` Malawski
563c7fbcf0 Issue 24594: Integration with sbt-headers and initial header population 2018-03-13 15:45:55 +01:00
Christopher Batey
009214ae07
Update copyright to 2018 (#24241) 2018-01-04 17:26:29 +00:00
Patrik Nordwall
6ed3295acd Merge branch 'master' into wip-multi-dc-merge-master-patriknw 2017-08-31 10:51:12 +02:00
Lukas Phaf
06e00fdf62 DeadLetterSuppression for Replicator.UpdateSuccess in distributed data #23553 2017-08-22 14:49:40 +02:00
Patrik Nordwall
0b1ce7223d fix usage of the the leader sorted set in Replicator
* since the ordering can change based on the member's status
  it's not possible to use ordinary - for removal
* similar issue at a few places where ageOrdering was used
2017-07-05 14:56:12 +02:00
Patrik Nordwall
bb9549263e Rename team to data center, #23275 2017-07-04 17:11:21 +02:00
Patrik Nordwall
e0fe0bc49e Make cluster sharding DC aware, #23231
* Sharding only within own team (coordinator is singleton)
* the ddata Replicator used by Sharding must also be only within own team
* added support for Set of roles in ddata Replicator so that can be used
  by sharding to specify role + team
* Sharding proxy can route to sharding in another team
2017-07-04 15:04:43 +02:00
Patrik Nordwall
2970287f95 discard large deltas, #23025
* to avoid OversizedPayloadException
* some complex deltas grow for each update operation, e.g.
  when updating different keys in ORMap (PNCounterMap)
* such large deltas can safely be discarded and disseminated as full
  state instead
* added ReplicatedDeltaSize interface to be able to define the "size"
  and when that size exceeds configured threshold the delta is discarded
2017-06-16 14:20:26 +02:00
Patrik Nordwall
5293666a30 Use full state for the ORSet and ORMap removals until #22648 is fixed
* some tests disabled, search for
  "FIXME use full state for removals, until issue #22648 is fixed"
  to find them
2017-03-30 13:26:35 +02:00
Patrik Nordwall
dc070e4e1c fix wrong log in receiveDeltaPropagation 2017-03-30 13:02:10 +02:00
Patrik Nordwall
9855e2896f Handle delta None correctly, #22655
When a DeltaReplicatedData returns None from delta it must still be
treated as a delta that increase the version counter in DeltaPropagationSelector.
Otherwise a later delta might be applied before the full state gossip is received
and thereby violating RequiresCausalDeliveryOfDeltas.
2017-03-30 08:12:03 +02:00
Johan Andrén
d51f7abd63 Give a good error message on unassigned ports #22552 2017-03-15 18:03:52 +01:00
Patrik Nordwall
6b5b819c73 Merge pull request #22378 from akka/wip-delta-in-direct-write-patriknw
Use delta in direct write also, #22188
2017-03-15 16:19:29 +01:00
Patrik Nordwall
233e784154 Use delta in direct write also, #22188
* Follow up on the causal delivery of deltas.
* The first implementation used full state for the direct
  Write messages, i.e. updates with WriteConsistency != LocalWrite
* This is an optimization so that delatas are tried first and if
  they can't be applied it falls back to full state.
* For simultanious updates the messages may be reordered because we
  create separate WriteAggregator actor and such, but normally they
  will be sent in order so the deltas will typically be received in
  order, otherwise we fall back to retrying with full state in the
  second round in the WriteAggregator.
2017-03-15 11:38:44 +01:00
Johan Andrén
1e6b8ec86b Make sure the hashcode is not negative when selecting key chunk #22487 2017-03-13 17:57:03 +01:00
ortigali
adf7e8271e less verbose debug logging in ddata (#22384)
* less verbose debug logging in ddata #22170

* less verbose debug logging in ddata 2 #22170
2017-02-24 10:38:14 +01:00
Patrik Nordwall
b700b840d1 causal delivery of deltas, #22188
* keep track of delta interval versions and skip deltas
  that are not consequtive, i.e. when some delta message was lost
* send the delta versions in the full state gossip to sync up the
  expected versions after dropped deltas
* implementation of deltas for ORSet
* refactoring of the delta types to allow for different type for the
  delta and the full state
* extensive tests
* mima filter
* performance optimizations
* simple pruning of deltas
* Java API
* update documentation
* KeyId type alias
* Use InternalApi annotation
2017-02-22 14:27:36 +01:00
Patrik Nordwall
3e7ffd6b96 Add support for delta-CRDT, #21875
* delta GCounter and PNCounter
* first stab at delta propagation protocol
* send delta in the direct write
* possibility to turn off delta propagation
* tests
* protobuf serializer for DeltaPropagation
* documentation
2017-01-24 13:24:45 +01:00
Patrik Nordwall
952be31a7d make pruning of CRDT garbage work, #21647
* fix merge issues of DataEnvelope and its pruning
* simplify by removing the tombstones, which didn't work in all cases anyway
* keep the PruningPerformed markers in the DataEnvelope until configured
  TTL has elapsed (wall clock)
* simplify PruningState structure
* also store the pruning markers in durable data
* collect removed nodes from the data, listing on MemberRemoved is not enough
* possibility to disable pruning altogether
* documented caveat for durable data
2017-01-23 16:54:02 +01:00
Patrik Nordwall
572d0c8040 stash messages while Replicator is loading durable data, #22150 2017-01-19 12:55:28 +01:00
inakov
e043a9fffe Add minimum cap for ReadMajority/WriteMajority #21618 2017-01-18 14:13:40 +02:00
Johan "Party Cannon" Andrén
91d6a3f125 #22147 show the cause of the error stopping durable store 2017-01-18 09:19:49 +01:00
inakov
65cba329d0 Updated DistributedData Delete API to include optional request context.
#20140
2017-01-14 14:30:58 +02:00
Jeroen Gordijn
81f081b33c #21648 Prefer reachable nodes in consistency writes/reads 2017-01-09 20:36:54 +01:00
Philippus Baalman
6c7085252a extended copyright into 2017 2017-01-04 17:37:15 +01:00
Patrik Nordwall
38a133ece0 harden DurableDataSpec, #22037 2016-12-21 16:45:50 +01:00
Patrik Nordwall
d6d50a08d0 Add support for durable storage of distributed data, #21645
* using lmdbjava libarary
2016-12-05 12:27:02 +01:00
Nafer Sanabria
32810e1f1d =htc, doc replace usages of deprecated methods of FileIO (#20928) 2016-07-10 17:41:57 +02:00
netcomm
33283da77a =ddata add WeaklyUp cluster member node (#20634)
* add WeaklyUp cluster member node

* fix some small point
2016-07-07 23:44:14 +02:00
Björn Antonsson
c66ce62d63 Update to a working version of Scalariform 2016-06-02 22:12:36 +02:00
Johannes Rudolph
b6cbc7f13a =all remove unused imports 2016-02-23 20:29:22 +01:00
Johan Andrén
62e30b3c08 Update copyrights and links to the new company name #19851 2016-02-23 12:58:39 +01:00
Prayag Verma
b7783968a0 =pro #19068 All copyrights ranges and single years updated to a range ending in 2016 2016-01-25 10:20:30 +01:00
Patrik Nordwall
3c5936aaa1 =cdd #18328 use set operators union diff instead of ++ --
AFTER:

[info] Benchmark                                  (set1Size)   Mode  Cnt      Score     Error   Units
[info] ORSetMergeBenchmark.mergeAddFromBothNodes           1  thrpt   10   1892.687 ± 182.830  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromBothNodes          10  thrpt   10    346.653 ±  22.048  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromBothNodes          20  thrpt   10    215.354 ±  15.486  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromBothNodes         100  thrpt   10     46.544 ±   2.930  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromOtherNode           1  thrpt   10   2483.531 ± 192.286  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromOtherNode          10  thrpt   10    355.292 ±  27.886  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromOtherNode          20  thrpt   10    211.221 ±  14.300  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromOtherNode         100  thrpt   10     47.516 ±   2.750  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromSameNode            1  thrpt   10  10323.210 ± 732.747  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromSameNode           10  thrpt   10   3476.660 ± 512.333  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromSameNode           20  thrpt   10   3490.045 ± 180.549  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromSameNode          100  thrpt   10   2915.627 ± 202.994  ops/ms
[info] ORSetMergeBenchmark.mergeComplex                    1  thrpt   10    556.034 ±  44.991  ops/ms
[info] ORSetMergeBenchmark.mergeComplex                   10  thrpt   10    246.468 ±  14.414  ops/ms
[info] ORSetMergeBenchmark.mergeComplex                   20  thrpt   10    175.547 ±  12.543  ops/ms
[info] ORSetMergeBenchmark.mergeComplex                  100  thrpt   10     47.452 ±   1.467  ops/ms

BEFORE:

[info] Benchmark                                  (set1Size)   Mode  Cnt     Score     Error   Units
[info] ORSetMergeBenchmark.mergeAddFromBothNodes           1  thrpt   10  2007.939 ±  74.673  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromBothNodes          10  thrpt   10   337.110 ±  15.055  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromBothNodes          20  thrpt   10   223.600 ±   8.403  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromBothNodes         100  thrpt   10    46.697 ±   2.136  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromOtherNode           1  thrpt   10  2542.537 ± 120.697  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromOtherNode          10  thrpt   10   365.694 ±  17.571  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromOtherNode          20  thrpt   10   216.323 ±   9.446  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromOtherNode         100  thrpt   10    49.563 ±   2.725  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromSameNode            1  thrpt   10  9883.186 ± 725.672  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromSameNode           10  thrpt   10  3266.528 ± 189.993  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromSameNode           20  thrpt   10  3206.017 ± 124.623  ops/ms
[info] ORSetMergeBenchmark.mergeAddFromSameNode          100  thrpt   10  2709.031 ± 162.182  ops/ms
[info] ORSetMergeBenchmark.mergeComplex                    1  thrpt   10   572.704 ±  21.504  ops/ms
[info] ORSetMergeBenchmark.mergeComplex                   10  thrpt   10   249.226 ±  12.324  ops/ms
[info] ORSetMergeBenchmark.mergeComplex                   20  thrpt   10   170.560 ±  10.320  ops/ms
[info] ORSetMergeBenchmark.mergeComplex                  100  thrpt   10    46.373 ±   1.800  ops/ms
2015-11-25 09:17:12 +01:00
Roland Kuhn
f1abaa1c5e Merge pull request #18875 from ktoso/wip-akka.js-cherries-ktoso
Akka.js cherries to master
2015-11-07 18:01:24 +01:00
Andrea
cd3d68a77c =act switch to java std lib ThreadLocalRandom 2015-11-06 14:04:33 +01:00
Patrik Nordwall
6b935e0c0b =cdd #18768 Cache serialization of read and write msg
The WriteAggregator and ReadAggregator typically send
the same message to several replicas and by caching the serialized bytes
we avoid doing the same thing for each message

and add a test for WriteAggregator
2015-10-30 17:03:32 +01:00
Patrik Nordwall
153b683a0a =cdd #18106 Merge update value with existing 2015-08-14 10:08:42 +02:00
Patrik Nordwall
94a61c7eb2 =cdd #17778 Convert samples to java 2015-07-03 16:19:57 +02:00
Konrad Malawski
7eeaf6cd9f =doc remove duplicated word typos 2015-07-01 23:48:17 +02:00
Patrik Nordwall
83a1f8eab3 =cdd #17796 Avoid duplicate change events 2015-06-29 14:50:36 +02:00
Christopher Hunt
7041c76ba9 +cdd #16799 Add ORMultiMap data type 2015-06-18 16:17:53 +02:00
Patrik Nordwall
cbe5dd2cf5 +cdd #16799 Add Distributed Data module
Previously know as [patriknw/akka-data-replication](https://github.com/patriknw/akka-data-replication),
which was originally inspired by [jboner/akka-crdt](https://github.com/jboner/akka-crdt).

The functionality is very similar to akka-data-replication 0.11.

Here is a list of the most important changes:

* The package name changed to `akka.cluster.ddata`
* The extension was renamed to `DistributedData`
* The keys changed from strings to classes with unique identifiers and type information of the data values,
  e.g. `ORSetKey[Int]("set2")`
* The optional read consistency parameter was removed from the `Update` message. If you need to read from
  other replicas before performing the update you have to first send a `Get` message and then continue with
  the ``Update`` when the ``GetSuccess`` is received.
* `BigInt` is used in `GCounter` and `PNCounter` instead of `Long`
* Improvements of java api
* Better documentation
2015-06-18 15:58:22 +02:00