LATEST VERSION: 8.1.0 - CHANGELOG
Pivotal GemFire® v8.1

Consistency Checking by Region Type

Consistency Checking by Region Type

GemFire performs different consistency checks depending on the type of region you have configured.

Partitioned Region Consistency

For a partitioned region, GemFire maintains consistency by routing all updates on a given key to the GemFire member that holds the primary copy of that key. That member holds a lock on the key while distributing updates to other members that host a copy of the key. Because all updates to a partitioned region are serialized on the primary GemFire member, all members apply the updates in the same order and consistency is maintained at all times. See Understanding Partitioning.

Replicated Region Consistency

For a replicated region, any member that hosts the region can update a key and distribute that update to other members without locking the key. It is possible that two members can update the same key at the same time (a concurrent update). It is also possible that, due to network latency, an update in one member is distributed to other members at a later time, after those members have already applied more recent updates to the key (an out-of-order update). By default, GemFire members perform conflict checking before applying region updates in order to detect and consistently resolve concurrent and out-of-order updates. Conflict checking ensures that region data eventually becomes consistent on all members that host the region. The conflict checking behavior for replicated regions is summarized as follows:
  • If two members update the same key at the same time, conflict checking ensures that all members eventually apply the same value, which is the value of one of the two concurrent updates.
  • If a member receives an out-of-order update (an update that is received after one or more recent updates were applied), conflict checking ensures that the out-of-order update is discarded and not applied to the cache.

How Consistency Checking Works for Replicated Regions and How Destroy and Clear Operations Are Resolved provide more details about how GemFire performs conflict checking when applying an update.

Consistency for WAN Deployments

Regions can also be configured to distribute updates to other GemFire clusters over a WAN. With a distributed WAN configuration, multiple gateway senders asynchronously queue and send region updates to another GemFire cluster. It is possible for multiple sites to send updates to the same region entry at the same time. It is also possible that, due to a slow WAN connection, a cluster might receive region updates after a considerable delay, and after it has applied more recent updates to a region. To ensure that WAN-replicated regions eventually reach a consistent state, GemFire first ensures that each cluster performs consistency checking to regions before queuing updates to a gateway sender for WAN distribution. In order words, region conflicts are first detected and resolved in the local cluster, using the techniques described in the previous sections.

When a GemFire cluster in a WAN configuration receives a distributed update, conflict checking is performed to ensure that all sites apply updates in the same way. This ensures that regions eventually reach a consistent state across all GemFire clusters. The default conflict checking behavior for WAN-replicated regions is summarized as follows:
  • If an update is received from the same GemFire cluster that last updated the region entry, then there is no conflict and the update is applied.
  • If an update is received from a different GemFire cluster than the one that last updated the region entry, then a potential conflict exists. A cluster applies the update only when the update has a timestamp that is later than the timestamp currently recorded in the cache.
Note: If you use the default conflict checking feature for WAN deployments, you must ensure that all GemFire members in all clusters synchronize their system clocks. For example, use a common NTP server for all GemFire members that participate in a WAN deployment.

As an alternative to the default conflict checking behavior for WAN deployments, you can develop and deploy a custom conflict resolver for handling region events that are distributed over a WAN. Using a custom resolver enables you to handle conflicts using criteria other than, or in addition to, timestamp information. For example, you might always prioritize updates that originate from a particular site, given that the timestamp value is within a certain range.

See How Consistency Is Achieved in WAN Deployments and, to configure a custom resolver, see "Implementing a GatewayConflictResolver" under Resolving Conflicting Events.

Non-Replicated Region and Client Cache Consistency

When a member receives an update for an entry in a non-replicated region and applies an update, it performs conflict checking in the same way as for a replicated region. However, if the member initiates an operation on an entry that is not present in the region, it first passes that operation to a member that hosts a replicate. The member that hosts the replica generates and provides the version information necessary for subsequent conflict checking. See How Consistency Checking Works for Replicated Regions.

Client caches also perform consistency checking in the same way when they receive an update for a region entry. However, all region operations that originate in the client cache are first passed onto an available GemFire server, which generates the version information necessary for subsequent conflict checking.

Configuring Consistency Checking

GemFire enables consistency checking by default. You cannot disable consistency checking for persistent regions. For all other regions, you can explicitly enable or disable consistency checking by setting the concurrency-checks-enabled region attribute in cache.xml to "true" or "false."

All GemFire members that host a region must use the same concurrency-checks-enabled setting for that region.

A client cache can disable consistency checking for a region even if server caches enable consistency checking for the same region. This configuration ensures that the client sees all events for the region, but it does not prevent the client cache region from becoming out-of-sync with the server cache.

See <region-attributes>.

Note: Regions that do not enable consistency checking remain subject to race conditions. Concurrent updates may result in one or more members having different values for the same key. Network latency can result in older updates being applied to a key after more recent updates have occurred.

Overhead for Consistency Checks

Consistency checking requires additional overhead for storing and distributing version and timestamp information, as well as for maintaining destroyed entries for a period of time to meet consistency requirements.

To provide consistency checking, each region entry uses an additional 16 bytes. When an entry is deleted, a tombstone entry of approximately 13 bytes is created and maintained until the tombstone expires or is garbage-collected in the member. (When an entry is destroyed, the member temporarily retains the entry with its current version stamp to detect possible conflicts with operations that have occurred. The retained entry is referred to as a tombstone.) See How Destroy and Clear Operations Are Resolved.

If you cannot support the additional overhead in your deployment, you can disable consistency checks by setting concurrency-checks-enabled to "false" for each region. See Consistency for Region Updates.

How Consistency Checking Works for Replicated Regions

Each region stores version and timestamp information for use in conflict detection. GemFire members use the recorded information to detect and resolve conflicts consistently before applying a distributed update.

By default, each entry in a region stores the ID of the GemFire member that last updated the entry, as well as a version stamp for the entry that is incremented each time an update occurs. The version information is stored in each local entry, and the version stamp is distributed to other GemFire members when the local entry is updated.

A GemFire member or client that receives an update message first compares the update version stamp with the version stamp recorded in its local cache. If the update version stamp is larger, it represents a newer version of the entry, so the receiving member applies the update locally and updates the version information. A smaller update version stamp indicates an out-of-order update, which is discarded.

An identical version stamp indicates that multiple GemFire members updated the same entry at the same time. To resolve a concurrent update, a GemFire member always applies (or keeps) the region entry that has the highest membership ID; the region entry having the lower membership ID is discarded.

Note: When a GemFire member discards an update message (either for an out-of-order update or when resolving a concurrent update), it does not pass the discarded event to an event listener for the region, or to any configured gateway senders for WAN distribution. You can track the number of discarded updates for each member using the conflatedEvents statistic. See GemFire Statistics List. Some members may discard an update while other members apply the update, depending on the order in which each member receives the update. For this reason, the conflatedEvents statistic differs for each GemFire member. The example below describes this behavior in more detail.
The following example shows how a concurrent update is handled in a distributed system of three GemFire members. Assume that Members A, B, and C have membership IDs of 1, 2, and 3, respectively. Each member currently stores an entry, X, in their caches at version C2 (the entry was last updated by member C):
  • Step 1: An application updates entry X on GemFire member A at the same time another application updates entry X on member C. Each member increments the version stamp for the entry and records the version stamp with their member ID in their local caches. In this case the entry was originally at version C2, so each member updates the version to 3 (A3 and C3, respectively) in their local caches.

  • Step 2: Member A distributes its update message to members B and C.

    Member B compares the update version stamp (3) to its recorded version stamp (2) and applies the update to its local cache as version A3. In this member, the update is applied for the time being, and passed on to configured event listeners.

    Member C compares the update version stamp (3) to its recorded version stamp (3) and identifies a concurrent update. To resolve the conflict, member C next compares the membership ID of the update to the membership ID stored in its local cache. Because the distributed system ID the update (A3) is lower than the ID stored in the cache (C3), member C discards the update (and increments the conflatedEvents statistic).



  • Step 3: Member C distributes the update message to members A and B.

    Members A and B compare the update version stamp (3) to their recorded version stamps (3) and identify the concurrent update. To resolve the conflict, both members compare the membership ID of the update with the membership ID stored in their local caches. Because the distributed system ID of A in the cache value is less than the ID of C in the update, both members record the update C3 in their local caches, overwriting the previous value.

    At this point, all members that host the region have achieved a consistent state for the concurrent updates on members A and C.



How Destroy and Clear Operations Are Resolved

When consistency checking is enabled for a region, a GemFire member does not immediately remove an entry from the region when an application destroys the entry. Instead, the member retains the entry with its current version stamp for a period of time in order to detect possible conflicts with operations that have occurred. The retained entry is referred to as a tombstone. GemFire retains tombstones for partitioned regions and non-replicated regions as well as for replicated regions, in order to provide consistency.

A tombstone in a client cache or a non-replicated region expires after 8 minutes, at which point the tombstone is immediately removed from the cache.

A tombstone for a replicated or partitioned region expires after 10 minutes. Expired tombstones are eligible for garbage collection by the GemFire member. Garbage collection is automatically triggered after 100,000 tombstones of any type have timed out in the local GemFire member. You can optionally set the gemfire.tombstone-gc-threshold property to a value smaller than 100000 to perform garbage collection more frequently.

Note: To avoid out-of-memory errors, a GemFire member also initiates garbage collection for tombstones when the amount of free memory drops below 30 percent of total memory.

You can monitor the total number of tombstones in a cache using the tombstoneCount statistic in CachePerfStats. The tombstoneGCCount statistic records the total number of tombstone garbage collection cycles that a member has performed. replicatedTombstonesSize and nonReplicatedTombstonesSize show the approximate number of bytes that are currently consumed by tombstones in replicated or partitioned regions, and in non-replicated regions, respectively. See GemFire Statistics List.

About Region.clear() Operations

Region entry version stamps and tombstones ensure consistency only when individual entries are destroyed. A Region.clear() operation, however, operates on all entries in a region at once. To provide consistency for Region.clear() operations, GemFire obtains a distributed read/write lock for the region, which blocks all concurrent updates to the region. Any updates that were initiated before the clear operation are allowed to complete before the region is cleared.

Transactions with Consistent Regions

A transaction that modifies a region having consistency checking enabled generates all necessary version information for region updates when the transaction commits.

If a transaction modifies a normal, preloaded or empty region, the transaction is first delegated to a GemFire member that holds a replicate for the region. This behavior is similar to the transactional behavior for partitioned regions, where the partitioned region transaction is forwarded to a member that hosts the primary for the partitioned region update.

The limitation for transactions on normal, preloaded or or empty regions is that, when consistency checking is enabled, a transaction cannot perform a localDestroy or localInvalidate operation against the region. GemFire throws an UnsupportedOperationInTransactionException exception in such cases. An application should use a Destroy or Invalidate operation in place of a localDestroy or localInvalidate when consistency checks are enabled.

How Consistency Is Achieved in WAN Deployments

When two or more GemFire systems are configured to distribute events over a WAN, each system performs local consistency checking before it distributes an event to a configured gateway sender. Discarded events are not distributed across the WAN.

When a gateway sender distributes an event to another GemFire site, it adds the distributed system ID of the local cluster, as well as a timestamp for the event. In a default configuration, the cluster that receives the event examines the timestamp to determine whether or not the event should be applied. If the timestamp of the update is earlier than the local timestamp, the cluster discards the event. If the timestamp is the same as the local timestamp, then the entry having the highest distributed system ID is applied (or kept).

You can override the default consistency checking for WAN events by installing a conflict resolver plug-in for the region. If a conflict resolver is installed, then any event that can potentially cause a conflict (any event that originated from a different distributed system ID than the ID that last modified the entry) is delivered to the conflict resolver. The resolver plug-in then makes the sole determination for which update to apply or keep.

See Resolving Conflicting Events.