TraversalSideEffects can now logically operate within a distributed OLAP environment. In order to make this possible, it is necessary that each side-effect be registered with a reducing BinaryOperator. This binary operator will combine distributed updates into a single global side-effect at the master traversal. Many of the methods in TraversalSideEffect have been Deprecated, but they are backwards compatible save that TraversalSideEffects.get() no longer returns an Optional, but instead throws an IllegalArgumentException. While the Optional semantics could have remained, it was deemed best to directly return the side-effect value to reduce object creation costs and because all side-effects must be registered apriori, there is never a reason why an unknown side-effect key would be used. In short:

All TraversalSideEffects must be registered upfront. This is because, in OLAP, side-effects map to Memory compute keys and as such, must be declared prior to the execution of the TraversalVertexProgram. If a user’s traversal creates a side-effect mid-traversal, it will fail. The traversal must use GraphTraversalSource.withSideEffect() to declare the side-effects it will use during its execution lifetime. If the user’s traversals use standard side-effect Gremlin steps (e.g. group("m")), then no changes are required.

See: TINKERPOP-1192

In a distributed environment, a side-effect can not be mutated and be expected to exist in the mutated form at the final, aggregated, master traversal. For instance, if the side-effect "myCount" references a Long, the Long can not be updated directly via sideEffects.set("myCount", sideEffects.get("myCount") + 1). Instead, it must rely on the registered reducer to do the merging and thus, the Step must do sideEffect.add("mySet",1), where the registered reducer is Operator.sum. Thus, the below will increment "a". If no operator was provided, then the operator is assumed Operator.assign and the final result of "a" would be 1. Note that Traverser.sideEffects(key,value) uses TraversalSideEffect.add().

See: TINKERPOP-1192, TINKERPOP-1166

OLTP graph providers that have a custom GraphStep implementation should ensure that g.V().hasId(x) and g.V(x) compile to the same representation. This ensures a consistent user experience around random access of elements based on ids (as opposed to potentially the former doing a linear scan). A static helper method called GraphStep.processHasContainerIds() has been added. TinkerGraphStepStrategy was updated as such:

is now

See: TINKERPOP-1219

The Step interface is fundamental to Gremlin. Step.processNextStart() and Step.next() both returned Traverser<E>. We had so many Traverser.asAdmin() and direct typecast calls throughout (especially in TraversalVertexProgram) that it was deemed prudent to have Step.processNextStart() and Step.next() return Traverser.Admin<E>. Moreover it makes sense as this is internal logic where Admins are always needed. Providers with their own step definitions will simply need to change the method signatures of Step.processNextStart() and Step.next(). No logic update is required — save that asAdmin() can be safely removed if used. Also, Step.addStart() and Step.addStarts() take Traverser.Admin<S> and Iterator<Traverser.Admin<S>>, respectively.

The way in which TraverserRequirements are calculated has been changed (for the better). The ramification is that post compilation requirement additions no longer make sense and should not be allowed. To enforce this, Traversal.addTraverserRequirement() method has been removed from the interface. Moreover, providers/users should never be able to add requirements manually (this should all be inferred from the end compilation). However, if need be, there is always RequirementStrategy which will allow the provider to add a requirement at strategy application time (though again, there should not be a reason to do so).

Providers that either have their own ComparatorHolder implementation or reason on OrderXXXStep will need to update their code. ComparatorHolder now returns List<Pair<Traversal,Comparator>>. This has greatly reduced the complexity of comparison-based steps like OrderXXXStep. However, its a breaking API change that is trivial to update to, just some awareness is required.

See: TINKERPOP-1209

Providers that have a custom GraphComputer implementation will have a lot to handle. Note that if the graph system simply uses SparkGraphComputer or GiraphGraphComputer provided by TinkerPop, then no updates are required. This only effects providers that have their own custom GraphComputer implementations.

Memory updates:

VertexProgram updates:

See: TINKERPOP-1166, TINKERPOP-1164, TINKERPOP-951

Operational semantic test cases have been added to GraphComputerTest to ensure that all the above are implemented correctly.

The Barrier interface use to simply be a marker interface. Now it has methods and it is the primary means by which distributed steps across an OLAP job are aggregated and distributed. It is unlikely that Barrier was ever used directly by a provider’s custom step. Instead, a provider most likely extended SupplyingBarrierStep, CollectingBarrierStep, and/or ReducingBarrierStep.

Providers that have custom extensions to these steps or that use Barrier directly will need to adjust their implementation slightly to accommodate a new API that reflects the Memory updates above. This should be a simple change. Note that FinalGet no longer exists and such post-reduction processing is handled by the reducing step (via the new Generating interface).

See: TINKERPOP-1164

The ProcessPerformanceSuite and TraversalPerformanceTest have been deprecated. They are still available, but going forward, providers should implement their own performance tests and not rely on the built-in JUnit benchmark-based performance test suit.

The GraphComputer API has changed with the addition of GraphComputer.vertices(Traversal) and GraphComputer.edges(Traversal). These methods construct a GraphFilter object which is also new to TinkerPop 3.2.0. GraphFilter is a "push-down predicate" used to selectively retrieve subgraphs of the underlying graph to be OLAP processed.

See: TINKERPOP-962

TinkerPop 3.2.0 has integrated VertexPrograms into GraphTraversal. This means, that a single traversal can compile to multiple GraphComputer OLAP jobs. This requires that ComputeResults be chainable. There was never any explicit tests to verify if a provider’s GraphComputer could be chained, but now there are. Given a reasonable implementation, it is likely that no changes are required of the provider. However, to ensure the implementation is "reasonable" GraphComputerTests have been added.

See: TINKERPOP-570

Providers that have custom Gremlin language implementations (e.g. Gremlin-Scala), there is a new class called ScriptTraversal which will handle script-based processing of traversals. The entire GroovyXXXTest-suite was updated to use this new class. The previous TraversalScriptHelper class has been deprecated so immediate upgrading is not required, but do look into ScriptTraversal as TinkerPop will be using it as a way to serialize "String-based traversals" over the network moving forward.

See: TINKERPOP-1154

If the provider has custom steps that leverage by()-modulation, those will now need to implement ByModulating. Most of the methods in ByModulating are default and, for most situations, only ByModulating.modulateBy(Traversal) needs to be implemented. Note that this method’s body will most like be identical the custom step’s already existing TraversalParent.addLocalChild(). It is recommended that the custom step not use TraversalParent.addLocalChild() as this method may be deprecated in a future release. Instead, barring any complex usages, simply rename the CustomStep.addLocalChild(Traversal) to CustomStep.modulateBy(Traversal).

See: TINKERPOP-1153

The TraversalSource infrastructure has been completely rewritten. Fortunately for users, their code is backwards compatible. Unfortunately for graph system providers, a few tweaks to their implementation are in order.

See: TINKERPOP-971

Documentation related to the lower-level APIs used by a provider, that was formerly in the reference documentation, has been moved to its own documentation set that is now referred to as the Provider Documentation.

See: TINKERPOP-937

The semantics of the various clear() methods on GraphProvider didn’t really change, but it would be worth reviewing their implementations to ensure that implementations can be called successfully in an idempotent fashion. Multiple calls to clear() may occur for a single test on the same Graph instance, as 3.1.1-incubating introduced an automated method for clearing graphs at the end of a test and some tests call clear() manually.

See: TINKERPOP-1146

Up until now transaction management has been a feature of sessionless requests only, but the new manageTransaction request argument for the Session OpProcessor changes that. Session-based requests can now pass this boolean value on each request to signal to Gremlin Server that it should attempt to commit (or rollback) the transaction at the end of the request. By default, this value as false, so there is no change to the protocol for this feature.

See: TINKERPOP-1039, Reference Documentation - Considering Sessions

The Gremlin Server protocol now allows the passing of scriptEvaluationTimeout as an argument to the SessionOpProcessor and the StandardOpProcessor. This value will override the setting of the same name provided in the Gremlin Server configuration file on a per request basis.

The RemoteAcceptor now has a new method called allowRemoteConsole(). It has a default implementation that returns false and should thus be a non-breaking change for current implementations. This value should only be set to true if the implementation expects the user to always use :> to interact with it. For example, the tinkerpop.server plugin expects all user interaction through :>, where the line is sent to Gremlin Server. In that case, that RemoteAcceptor implementation can return true. On the other hand, the tinkerpop.gephi plugin, expects that the user sometimes call :> and sometimes work with local evaluation as well. It interacts with the local variable bindings in the console itself. For tinkerpop.gephi, this method returns false.

See: Reference Documentation - Remote Console

There were a number of fixes related to usage of appropriate types in the test suite. There were cases where tests were mixing types, such that a single property key might have two different values. This mixed typing caused problems for some graphs and wasn’t really something TinkerPop was looking to explicitly enforce as a rule of implementing the interfaces.

While the changes should not have been breaking, providers should be aware that improved consistencies in the tests may present opportunities for test failures.

See: TINKERPOP-984, TINKERPOP-990, TINKERPOP-1000

For providers who have built custom serializers in Gryo, there is a new feature open that can be considered. A GryoMapper can now take a custom Kryo ClassResolver, which means that custom types can be coerced to other types during serialization (e.g. a custom identifier could be serialized as a HashMap). The advantage to taking this approach is that users will not need to have the provider’s serializers on the client side. They will only need to exist on the server (presuming that the a type is coerced to a type available on the client, of course). The downside is that serialization is then no longer a two way street. For example, a custom ClassResolver that coerced a custom identifier to HashMap would let the client work with the identifier as a HashMap, but the client would then have to send that identifier back to the server as a HashMap where it would be recognized as a HashMap (not an identifier).

See: TINKERPOP-1064

There were a number of corrections made around the consistency of Features and how they were applied in tests. Corrections fell into two groups of changes:

These changes related to "Feature Consistency" open up a number of previously non-executing tests for graphs that did not support meta-properties, so providers should be wary of potential test failure on previously non-executing tests.

See: TINKERPOP-985, TINKERPOP-997, TINKERPOP-998

There are two new methods on the Spark-Gremlin RDD interfaces.

Note that both these methods have default implementations which simply work with empty RDDs. Most providers will never need to implement these methods as they are specific to file/RDD management for GraphComputer. The four classes that implement these methods are PersistedOutputRDD, PersistedInputRDD, InputFormatRDD, and OutputFormatRDD. For the interested provider, study the implementations therein to see the purpose of these two new methods.

See link:https://issues.apache.org/jira/browse/TINKERPOP-762

The AbstractTransaction class has been abstracted into two different classes supporting two different modes of operation: AbstractThreadLocalTransaction and AbstractThreadedTransaction, where the former should be used when supporting ThreadLocal transactions and the latter for threaded transactions. Of course, providers may still choose to build their own implementation on AbstractTransaction itself or simply implement the Transaction interface.

The AbstractTransaction gains the following methods to potentially implement (though default implementations are supplied in AbstractThreadLocalTransaction and AbstractThreadedTransaction):

See: TINKERPOP-765, TINKERPOP-885

The default behavior for Transaction.close() is to rollback the transaction and is enforced by tests, which previously asserted the opposite (i.e. commit on close). These tests have been renamed to suite the new semantics:

If these tests were referenced in an OptOut, then their names should be updated.

See: TINKERPOP-805

There were numerous changes to the GraphTraversal API. Nearly all changes are backwards compatible with respective "deprecated" annotations. Please review the respective updates specified in the "Graph System Users" section.

Element.Exceptions.elementAlreadyRemoved has been deprecated and test enforcement for consistency have been removed. Providers are free to deal with deleted elements as they see fit.

See: TINKERPOP-297

The VendorOptimizationStrategy has been renamed to ProviderOptimizationStrategy. This renaming is consistent with revised terminology for what were formerly referred to as "vendors".

See: TINKERPOP-876

GraphComputer.configure(String key, Object value) is now a method (with default implementation). This allows the user to specify engine-specific parameters to the underlying OLAP system. These parameters are not intended to be cross engine supported. Moreover, if there are not parameters that can be altered (beyond the standard GraphComputer methods), then the provider’s GraphComputer implementation should simply return and do nothing.

The "rebindings" argument to the "standard" OpProcessor has been renamed to "aliases". While "rebindings" is still supported it is recommended that the upgrade to "aliases" be made as soon as possible as support will be removed in the future. Gremlin Server will not accept both parameters at the same time - a request must contain either one parameter or the other if either is supplied.

See: TINKERPOP-913

If a driver configures the Transaction.onReadWrite() or Transaction.onClose() settings, note that these settings no longer apply to all future requests. If the settings are changed for a sessionless request they will only apply to that one request. If the settings are changed from an in-session request they will only apply to all future requests made in the scope of that session.

See: TINKERPOP-885

It is important to note that this feature of the sub-protocol applies to the SessionOpProcessor (i.e. for session-based requests). Prior to this change, there was no way to explicitly close a session. Sessions would get closed by the server after timeout of activity. This new "op" gives drivers the ability to close the session explicitly and as needed.

See: TINKERPOP-849, Reference Documentation - OpProcessor

Providers can consider the use of the new GraphFactoryClass annotation to specify the factory class that GraphFactory will use to open a new Graph instance. This is an optional feature and will generally help implementations that have an interface extending Graph. If that is the case, then this annotation can be used in the following fashion:

MyGraphFactory must contain the static open method that is normally expected by GraphFactory.

See: TINKERPOP-778

There was a change that affected providers who implemented GraphComputer related tests such as the ProcessComputerSuite. If the provider runs those tests, then edit the GraphProvider implementation for those suites to include the GraphProvider.Descriptor annotation as follows:

See: TINKERPOP-690 for more information.

There were some adjustments to the test suite with respect to how Transaction.close() was being validated. For most providers, this will generally mean checking OptOut annotations for test renaming problems. The error that occurs when running the test suite should make it apparent that a test name is incorrect in an OptOut if there are issues there.

See: TINKERPOP-764 for more information.

Gremlin Server now supports SASL-based authentication. By default, Gremlin Server is not configured with authentication turned on and authentication is not required, so existing drivers should still work without any additional change. Drivers should however consider implementing this feature as it is likely that many users will want the security capabilities that it provides.

See: Reference Documentation - Gremlin Server Authentication