A text/plain MIME type has been added to the HTTP endpoint to return Gremlin Console formatted results in plain text. This format can be helpful for a variety of reasons. Reading JSON formatted results can be difficult sometimes and text/plain is a more simple, readable representation for when that is helpful.

See: TINKERPOP-2947

Prior to GraphSON 3.0, there were options to have GraphSON returned with and without types. The latter was helpful for getting a more simple to process JSON format that relied strictly on the JSON type system. This format was of course lossy in nature and not conducive for use with the language variants that were developing as the primary mechanism for working with Gremlin. Of course, simple HTTP never really went away and forcing the type system on consumers makes that route of connecting harder than it should be.

With HTTP users in mind, this version brings back both versions of untyped GraphSON by adding the following mime types:

Neither is configured by default in Gremlin Server, but can be added if desired by editing the Gremlin Server configuration file.

See: TINKERPOP-2965

Release 3.6.3 will be one of the last versions of 3.6.x to use Go 1.17 for gremlin-go as this runtime is no longer supported. Upcoming releases of gremlin-go will attempt to use the latest available version of Go.

The mergeV() and mergeE() step were added at version 3.6.0. Given some feedback on implementation and usage, some additional changes were needed in order to improve the usability of these steps. These changes could not be made without a breaking change to existing behavior introduced at 3.6.0. The main changes to consider are:

See: TINKERPOP-2850

Python and .NET have support for GraphBinary at least since 3.5.0, but kept GraphSON 3 by default. It now seems safe to make GraphBinary the default in 3.6.x. With this change, all language variants now have GraphBinary as their default serialization format.

To continue using the GraphSON, explicitly specify it as the serializer to use in the configuration.

See: TINKERPOP-2723

The new element() step provides a way to traverse from a Property to the Element that owns it:

See: TINKERPOP-2713, Reference Documentation, Provider Documentation

One of the most commonly used patterns in Gremlin is the use of fold().coalesce(unfold(), …​) to perform upsert-like functionality. While this pattern is quite flexible, it can also be confusing to new users and for certain use cases challenging to get the pattern correctly implemented. For providers, the pattern is difficult to properly optimize because it can branch into complexity quite quickly making it hard to identify a section of Gremlin for an upsert and therefore is not executed as efficiently as it might have been otherwise.

The new mergeV() and mergeE() steps greatly simplify this pattern and as the pattern is condensed into a single step it should be straightforward for providers to optimize as part of their implementations. The following example demonstrates just how much easier implementing a basic upsert of a vertex has gotten:

In a more complex example below, if the vertex is found, then it is updated with an "age" of "30" otherwise it is created with an "age" of 27:

The pattern was even more complicated for upserting edges, but the following example demonstrates how much easier mergeE() is to follow:

For those currently using the fold().coalesce(unfold(), …​) pattern, there is no need to be concerned with incompatibility as a result of these new steps. That pattern is still perfectly usable and valid Gremlin, but whenever possible it would be best to migrate away from it as graph providers ramp up on 3.6.0 support and introduce important write optimizations that will make a big difference in performance.

See: TINKERPOP-2681, mergeE()-step, mergeV()-step

Aliases have been added to Direction to allow for OUT to be referred to as from and IN can be referred to as to, which is a bit more friendly and matches more closely with existing Gremlin syntax.

Pick was formerly a nested class of TraversalOptionParent, but has now been promoted to being a class on its own in org.apache.tinkerpop.gremlin.process.traversal.Pick.

The by() modulator is critical to the usage of Gremlin. When used in conjunction with a step that supports it, the arguments to the by() modulator shift the behavior of the internals of the step. The behavior that by() introduces has not always been consistent with some overloads establishing null traversers, others throwing exceptions that are hard to digest, some filtering, etc.

In 3.6.0, the rules for the by() modulator are made straightforward. If the by() produces a result then it is said to be "productive" and its value is propagated to the step for use. If the by() does not produce a result then the traverser to which it was to be applied is filtered.

The following sections demonstrate the behavior in 3.5.x alongside the new 3.6.0 behavior:

aggregate()

cyclicPath()

dedup()

When using dedup() over labels all labels must produce or the path will be filtered:

group()

There are two by() modulators that can be assigned to group()`. The first modulator is meant to identify the key to group on and will filter values without that key out of the resulting Map.

The second by()` is applied to the result as a reducing operation and will filter away entries in the List value of each key.

groupCount()

math()

The math() step requires that the result of the by() be a Number, so a result of null will still result in a runtime exception. Filtering will eliminate such errors, though a runtime error may still be present should the modulator produce a non-numeric value.

order()

path()

All by() modulators must be productive for the filter to be satisfied.

project()

The project() step will produce an incomplete Map by filtering away keys of unproductive by() modulators.

propertyMap()

sack()

sample()

select()

All by() modulators must be productive for the filter to be satisfied.

simplePath()

tree()

All by() modulators must be productive for the filter to be satisfied.

valueMap()

where()

For the most part, this change largely removes runtime exceptions and since most uses cases are not likely to rely on those for query execution, existing code should not be broken by this upgrade. However, users who relied on 3.5.x behavior where by() might propagate a null would however see a behavioral change. To temporarily restore the old behavior, simply include g.withStrategies(ProductiveByStrategy) in the traversal configuration, which will force the null to be produced. Ultimately, it would be best not to rely on this strategy in the long term however and convert Gremlin that requires it to behave properly without it.

For example, if in 3.5.x there was a traversal like g.V().group().by('age') and "age" is known to not always be a valid key, the appropriate change would be to propagate null explicitly as with: g.V().group().by(coalesce(values('age'), constant(null))).

See: TINKERPOP-2635

A number of graph databases have included support for regular expressions text predicates and now TinkerPop includes a regex() option to TextP:

See: TINKERPOP-2652

There is a new module called gremlin-annotations and it holds the annotations used to construct Java-based Gremlin DSLs. These annotations were formerly in gremlin-core and therefore it will be necessary to modify dependencies accordingly when upgrading to 3.6.0. Package and class names have remained the same and general usage is unchanged.

It is worth noting that gremlin-groovy utilized the DSL annotations to construct the Credentials DSL so the gremlin-annotations package is now explicitly associated to gremlin-groovy but as an <optional> dependency.

See: TINKERPOP-2411

The new fail() step provides a way to immediately terminate a traversal with a runtime exception. In the Gremlin Console, the exception will be rendered as follows which helps provide some context to the failure:

See: TINKERPOP-2635, Reference Documentation

In 3.5.x, calling property() with a key that is of type T and a value that is null or calling addV() with a null label is processed as a valid traversal and default values are used. That approach allows opportunities for unexpected problems should a variable being passed as a parameter end up accidentally shifting to null without the caller’s knowledge. Starting in 3.6.0, such traversals will generate an exception during construction of the traversal.

It is still possible to call addV() with no arguments to assume a default label and id generation remains implementation specific with some graphs accepting id and others ignoring it to generate their own. Both value of T remain immutable.

See: TINKERPOP-2611

In Gremlin Server and Gremlin Console distributions, the default logging implementation of log4j 1.2.x has been replaced by logback 1.2.x given CVE-2019-17571. While it was easy to replace log4j for users of the zip distributions, it was a little harder for users to change our packaged Docker containers which should work more cleanly out of the box.

See: TINKERPOP-2534

Numeric operations around short and byte have not behaved quite like int and long. Here is an example of a sum operation with sack():

Note that the type returned for the the sum should be the largest type encountered in the operation, thus if a long + int would return long or a byte + int would return int. The last example above shows inconsistency in this rule when dealing with types short and byte which simply promote them to int.

For 3.6.0, that inconsistency is resolved and may be a breaking change should code be relying on the integer promotion.

See: TINKERPOP-2610

The gremlin-driver module no longer depends on groovy or groovy-json. It became an <optional> dependency in 3.5.0 and general deprecation of the serializers for the JsonBuilder class from Groovy (which was the only reason the dependency existed in the first place) occurred in 3.5.2. As they were made <optional> it is likely that users who required those packages have already adjusted their dependencies to explicitly include them. As for those who still make use of JsonBuilder serialization for some reason, the only recourse is to find the code in TinkerPop and maintain it independently. The following classes were removed as of this change (links go to their 3.5.1 versions):

See: TINKERPOP-2593

Gryo MessageSerializer implementations were deprecated at 3.4.3 (GryoLite at 3.2.6) in favor of GraphBinary. As GraphBinary has been the default implementation for some time now and given that Gryo is a JVM-only format which reduces its usability within Gremlin Language Variants, it seemed like the right time to remove the Gryo MessageSerializer implementations from the code base. Gryo may still be used for file based applications.

See: TINKERPOP-2639

GroovyTranslator has been removed from the gremlin-groovy module. Please update any code using that class to instead use org.apache.tinkerpop.gremlin.process.traversal.translator.GroovyTranslator which is found in gremlin-core.

See: TINKERPOP-2657

When gremlin-python was first built, it followed the Gremlin step names perfectly and didn’t account well for Python keywords that those steps conflicted with. As this conflict led to problems in usage, steps that matched keywords were renamed to have an underscore suffix (e.g. sum() to sum_()) and the old step names were deprecated.

In 3.6.0, those original conflicting steps names have simply been removed. Please change any of the following steps that may still be in use to instead prefer the underscore suffixed versions:

The full list of steps with the suffix naming can be found in the Reference Documentation.

In addition to removing the conflicting names, camel cased naming has been deprecated for all Gremlin steps and replaced with more Pythonic snake cased names. As this change was merely deprecation, this change is non-breaking at this time, but the camel cased steps will be removed in some future major version.

See: TINKERPOP-2650

The property() step has been extended to take a Map of property key/value pairs as an argument with two new signatures:

When called, each individual key/value pair in the Map is saved as a property to the element. When the Cardinality is specified, that cardinality will be applied to all elements in the map as they are saved to the element.

If users need different cardinalities per property, then please use the existing pattern of stringing multiple property()` calls together.

See: TINKERPOP-2665

gremlin-javascript and gremlint have upgraded from Node 10 to Node 16 as Node 10 has passed end of life. gremlin-go has upgraded from Go 1.17 to Go 1.20 as Go 1.20 has passed end of life.

Introduced max number length (10000 chars), string length (20 000 000 chars), and nesting depth (1000) constraints for GraphSON deserialization due to security vulnerabilities with earlier versions of Jackson Databind. New constraints are not expected to impact most users but can be overriden via GraphSONMapper.Builder or through serializer configuration. Example:

Neo4j-Gremlin has been deprecated due to incompatibility with current versions of Neo4j. See reference docs for more information.

Release 3.5.6 will be the last version of 3.5.x to use Node 10 for gremlin-javascript and Go 1.17 for gremlin-go as both of these runtimes are no longer supported. The following release (3.5.7) will likely contain Node 16 to align with 3.6.x and gremlin-go will use the latest available version of Go.

Gremlin Server Docker image now supports both AMD64 and ARM64. Multi-architecture image can now be found on Docker Hub.

see: TINKERPOP-2852

There are various performance improvements included in this release:

See: TINKERPOP-2526, TINKERPOP-2899, TINKERPOP-2855, TINKERPOP-2919

The Gremlin.NET driver can now be configured for logging to get more insights into its internal state, like when a connection was closed and will therefore be replaced. It uses Microsoft.Extensions.Logging for this so all kinds of different .NET logging implementations can be used.

The following example shows how to provide a LoggerFactory that is configured to log to the console to GremlinClient:

See: TINKERPOP-2471

The Java drivers approach to determine the health of the host to which it is connected is important to how it behaves. The driver has generally taken a pessimistic approach to make this determination, where a failure to borrow a connection from the pool would constitute enough evidence to mark the host as dead. Unfortunately a failure to borrow from the pool is not the best indicator for a dead host. Intermittent network failure, an overly busy client, or other temporary issues could have been at play while the server was perfectly available.

The consequences for marking a host dead are fairly severe as the connection pool is shutdown. While it is shutdown in an orderly fashion so as to allow for existing requests to complete if at all possible, it immediately blocks new requests producing an immediate NoHostAvailableException (assuming there are actually no other hosts to route requests to). There is then some delay to reconnect to the server and re-establish each of the connections in the pool, which might take some time especially if there was a large pool. If an application were sending hundreds of requests a second, it would quickly equate to hundreds of exceptions a second, filling logs quickly and making it hard to decipher the original root cause from that initial inability to simply borrow a connection.

For 3.5.5, the driver has dropped some of its pessimistic ways and has taken a more optimistic posture when it comes to considering host availability. When the driver now has a problem borrowing a connection from the pool, it does not assume the worst of a failed host. It instead checks if the other connections in the pool are still alive. If at least one is alive it assumes the host is still available and the user simply gets a TimeoutException for waiting beyond the maxWaitForConnection setting. In the event that there are no connections deemed alive, the driver will try to issue an immediate reconnect to determine host health. If the reconnect succeeds then the host remains in an available state for future requests and if it fails it is finally marked as dead and thus unavailable.

If a host is marked unavailable, the driver, like before, will try to route requests to other configured hosts that are healthy. If there are no such other hosts and all are unhealthy, the driver will now make an immediate attempt to route the request to a random unavailable host on the chance it will come back online and the connection pool thus re-established. So, rather than a NoHostAvailableException flood, the user would instead see a TimeoutException per request made.

Since the driver was already throwing NoHostAvailableException and TimeoutException, there shouldn’t be too many code changes required on upgrade to this version. The main difference to consider is that NoHostAvailableException should appear less often, so any code that mitigated against a flood of these exceptions might not be necessary anymore. One should also consider that if code was relying on the asynchronous properties that Client.submitAsync() implies in its name, that code may now see more blocking behavior than before. In doing this refactoring, it was noted that the connection borrowing behavior has never behaved in a truly asynchronous manner and only the fast NoHostAvailableException aspect of that method behavior really contributed to an asynchronous looking call for failed connection borrowing. With the fast NoHostAvailableException gone, it may be more likely now to note blocking behavior when a connection cannot be obtained.

See: TINKERPOP-2813

Previously, a server does not distinguish amongst the different types of clients connecting to it. We have now added user agent to web socket handshake in all the drivers, each with their own configuration to enable or disable user agent. User agent is enabled by default for all drivers.

See: TINKERPOP-2480

Previously, the java driver relies on the default 10 second SSL handshake timeout defined by Netty. We have removed the default SSL handshake timeout. The SSL handshake timeout will instead be capped by setting connectionSetupTimeoutMillis.

See: TINKERPOP-2814

A utility class SparkIOUtil is introduces, which allows users to load graph into Spark RDD via loadVertices(org.apache.commons.configuration2.Configuration, JavaSparkContext) method.

After introducing a number of release candidates over the past several months, the official release of Gremlin-Go is now available starting at 3.5.4. There are a few simple prerequisites required to get started:

In the root directory of your project, containing the go.mod file, run the following command:

go get github.com/apache/tinkerpop/gremlin-go/v3@v3.5.4

Afterwards, we can quickly get started with writing Gremlin in Go:

For full documentation on the available API to give Gremlin Go a try, click here.

See: TINKERPOP-2692

Gremlin JavaScript now also supports GraphBinary. GraphSON3 still remains the default serialization format.

To enable GraphBinary support, set the mime type in the connection options.

Deprecated and removed functionality of the connectOnStartup option for Gremlin Javascript to resolve potential unhandledRejection and race conditions. Setting connectOnStartup to true will only trigger a console warning.

New DriverRemoteConnection objects no longer initiate connection by default at startup. Call open() explicitly if one wishes to connect on startup. For example:

See: TINKERPOP-2708

It should be noted however that compression might make an application susceptible to attacks like CRIME/BREACH. Compression should therefore be turned off if the application sends sensitive data to the server as well as data that could potentially be controlled by an untrusted user. Compression can be disabled via the disableCompression parameter on the GremlinClient constructor.

See: TINKERPOP-2682

A Translator can translate a Gremlin traversal from one programming language to another. Gremlin.NET now comes with a GroovyTranslator which translates a Gremlin.NET traversal into a string that contains a Gremlin-Groovy traversal:

See: TINKERPOP-2367

After Javascript, .NET and Python are now the second and third non-JVM variants of Gremlin to get support for remote transactions.

An example of the .NET Tx() syntax can be found in the .NET Transaction Section.

An example of the Python tx() syntax can be found in the Python Transaction Section

See: TINKERPOP-2556 for .NET and TINKERPOP-2555 for Python

Gremlin in native programming languages can all construct a native date and time object. In Java, that would probably be a java.util.Date object while in Javascript it would likely be the Node.js Date. In any of these cases, these native objects would be serialized to millisecond-precision offset from the unix epoch to be sent over the wire to the server (in embedded mode for Java, it would be up to the graph database to determine how the date is handled).

The gap is in Gremlin scripts which do not have a way to natively construct dates and times other than by using Groovy variants. As TinkerPop moves toward a more secure method of processing Gremlin scripts by way of the gremlin-language model, it was clear that this gap needed to be filled. The new datetime() function can take a ISO-8601 formatted datetime and internally produce a Date with a default time zone offset of UTC (+00:00).

This functionality, while syntax of gremlin-language, is also exposed as a component of gremlin-groovy so that it can be used in the Gremlin Console and through the GremlinScriptEngine in Gremlin Server.

The above examples use the Java Date method toGMTString() to properly format the date for demonstration purposes. From a Gremlin language perspective there are no functions that can be called on the return value of datetime().

See: TINKERPOP-2596

Some default value settings were changed to be consistent across different languages.

See: TINKERPOP-2379

The fold() step has long been used for reducing the traversal stream to a List as in:

This step has always had a lesser known second signature though which has two arguments of a "seed" and "foldFunction":

There was a bit of incomplete (or perhaps incorrect) logic that did not allow the folding of Map and as of this version, that issue has been resolved which enables Map entry merges:

See: TINKERPOP-2667

Release 3.5.0 introduce the ability for there to be traversers that contained a null value. Since that time it has been noted that supplying a null value as an argument to certain Gremlin steps might cause exception behaviors that were either not clear or perhaps even unexpected. The following changes were made to help make these behaviors more consistent and to further solidify the semantics of null usage in the Gremlin language:

See: TINKERPOP-2605, TINKERPOP-2620

Gremlin steps that take a by() modulator have had varying behaviors for some time now and depending on the step and context of the argument given to the by(), the traversal might return a result, throw an exception, produce a null if no value is present, or filter results. While all of these behaviors cannot be reconciled in 3.5.x without breaking changes it is possible to start to deal with the exception behavior in a more consistent way.

When a traversal given to by() does not contain a result, the traversal is deemed "unproductive". As mentioned above, unproductive arguments to by() lead to different results depending on the case:

With ProductiveByStrategy now installed by default in 3.5.2, the exception behavior is changed to follow the null behavior forcing the unproductive traversal to return something:

Note that group() step has some special behaviors that made it virtually impossible for ProductiveByStrategy to fully retain them, so ultimately the strategy ignores the modulators on that step except for the simple cases where the key modulator is a basic ValueTraversal (e.g. by('name')'). As a result, for group() the behavior should be unchanged with exceptions retained:

See: TINKERPOP-2635

Javascript is now the first non-JVM variant of Gremlin to get support for remote transactions. An example of the tx() syntax can be found in the Javascript Transaction Section.

See: TINKERPOP-2557

The Gremlint website became an official part of the TinkerPop project for 3.5.0. In 3.5.1, the javascript library that contains the logic for the site has been made available as a code library that can be used independently and has been published to npm.

See: TINKERPOP-2551

When connecting to an SSL enabled server using Java, errors with that connectivity would result in a RuntimeException that wrapped an underlying NoHostAvailableException which did not surface information about the SSL problem. For this sort of failure in 3.5.2, the RuntimeException has been replaced with the NoHostAvailableException which in turn wraps a more specific SSLException. While this change represents a small break in exception handling, it seemed a worthwhile change to take, so as to help make this error more clear when it arises.

See: link: TINKERPOP-2616

TinkerPop implements Gremlin in a variety of different programming languages. For 3.5.0, there are a number of major upgrades to those programming language environments:

See: TINKERPOP-2076, TINKERPOP-2317, TINKERPOP-2335

Gremlint, the JavaScript library powering gremlint.com, is now part of the TinkerPop project. It provides utilities for formatting Gremlin queries, taking parameters such as indentation, maximum line length and dot placement into account.

Until recently Gremlint has been an independent project maintained by Øyvind Sæbø, a developer at Ardoq. Ardoq has since donated Gremlint, as well as gremlint.com, to TinkerPop so that it can serve as TinkerPop’s canonical Gremlin code formatter.

See: TINKERPOP-2530

Translator implementations were moved from a mostly quiet and internal feature of TinkerPop to a documented and more readily accessible form in 3.4.9. For 3.5.0, the functionality has been expanded significantly in a number of ways. First, for Java, gremlin-core now has a JavascriptTranslator and a DotNetTranslator which completes the set of Gremlin translation functions for the programming languages that TinkerPop supports. It is therefore now possible to convert Gremlin bytecode to string representations that can compile in C#, Groovy, Javascript and Python.

The following example demonstrates what this functionality looks like in Javascript but it is quite similar for each Translator implementation:

In addition, a native Python Translator implementation has been added that will generate a string of Gremlin that will compile on the JVM.

See: TINKERPOP-2451, TINKERPOP-2452, TINKERPOP-2527

Apache Commons

There is a major breaking change in the use of Configuration objects. Prior to 3.5.0, Configuration objects were from the Apache Commons commons-configuration library, but in this version, they are of commons-configuration2. While this is a breaking change, the fix for most implementations will be quite simple and amounts to changing the import statements from:

to

It is also worth noting that default list handling in configurations is treated differently. TinkerPop disabled the default list handling approach in Configuration 1.x, but if that functionality is still needed, it can be reclaimed by setting the LegacyListDelimiterHandler - details for doing taking this step and other relevant upgrade information can be found in the 2.x Upgrade Documentation.

Neo4j

There were two key changes to the neo4j-gremlin module:

Groovy Dependency in Java Driver

The gremlin-driver module made its dependency on Groovy optional as its only reason for inclusion was to support JsonBuilder serialization and this feature is little known and perhaps even less used. Read more about this change here in the Upgrade Documentation in the Serialization Section under the subsection title "GraphSON and JsonBuilder".

See: TINKERPOP-2185

The gremlin-driver has an additional packaging which may make it easier to upgrade for some users who may have extensive dependency chains.

The above dependency with the shaded classifier shades all the non-optional dependencies of gremlin-driver and includes gremlin-core and tinkergraph-gremlin in an unshaded form. The slf4j dependency was not included because shading it can cause problems with its operations.

See: TINKERPOP-2476

Java and Gryo

Since the first release of TinkerPop 3.x, Gryo has been the default serialization format for Gremlin Server and Java Driver. It was also used as the default serialization format for Gremlin Console remote connectivity to Gremlin Server. As of this release, Gryo has been replaced as the default by GraphBinary. All packaged configuration files and programmatic defaults have been modified as such.

It is still possible to utilize Gryo as a message serialization format by modifying Gremlin Server configuration files to include the appropriate Gryo configurations. If using Gryo, do not use earlier versions of the driver and server with 3.5.0. Use a 3.5.0 client to connect to a 3.5.0 server. Generally speaking, mixed version combinations will appear to work properly, but problems will likely occur during general course of usage and it is therefore not advisable to take this approach.

For best compatibility between 3.4.x and 3.5.x, please use GraphBinary.

GraphSON and JsonBuilder

GraphSON serialization support for Groovy’s JsonBuilder has been present since the first version of GraphSON. That approach to returning results has never materialized as a standardized way to use Gremlin as originally envisioned. While support for this serialization form is still present, the dependency on Groovy in gremlin-driver has been changed to "optional", which means that users who wish to continue to return JsonBuilder results for some reason must explicitly include groovy and groovy-json dependencies in their applications. For Maven this would mean adding the following dependencies:

The ${groovy.version} should match the version specified in TinkerPop’s root pom.xml.

.NET: GraphBinary

Gremlin.NET now also supports GraphBinary. GraphSON 3 however still remains the default serialization format as GraphBinary should be considered experimental for this version in .NET:

.NET: New JSON Library

Gremlin.NET now uses System.Text.Json instead of Newtonsoft.Json as System.Text.Json is already included in .NET Core 3.0 and higher which removes a dependency and offers better performance. Most users should not notice this change, however users who have implemented their own GraphSON serializers or deserializers will need to modify them accordingly. The same applies to users that let Gremlin.NET return data without deserializing it first as the returned data types will change in this case, for example from Newtonsoft.Json’s JObject or JToken to JsonElement with System.Text.Json.

Python dict Deserialization

In Gremlin it is common to return a dict or list as a key value in another dict. The problem for Python is that these values are not hashable and will result in an error. By introducing a HashableDict and Tuple for those keys (respectively), it is now possible to return these types of results and not have to work around them:

See: TINKERPOP-2259, TINKERPOP-2349, TINKERPOP-2395, TINKERPOP-2407, TINKERPOP-2460, TINKERPOP-2472, Custom JSON serialization with Gremlin.NET

The TinkerPop Transaction API and its related features have not changed much since TinkerPop 3.x was initially released. Transactions that extend beyond the scope of a single traversal (or request) have remained a feature for embedded use cases and script execution (where supported) even in the face of the rise of remote graph use cases. With the varying contexts that exist for how and when transactions can be used, it has led to a fair bit of confusion.

For 3.5.0, TinkerPop introduces a change in approach to transactions that has the goal of unifying the API and features for all use cases, while addressing the more current state of the graph ecosystem which has shifted heavily toward remote communication and more diverse programming language ecosystems beyond the JVM.

The new model for using a transaction looks like this:

The first important point to take away here is that the same transaction code will work for either embedded or remote use cases. The second important point is that for remote cases, transaction support with bytecode is a wholly new feature, which was implemented by providing support for bytecode in sessions. Up until this time, sessions could only support script-based requests, so that makes another added feature related to this one.

See: TINKERPOP-2537, Reference Documentation - Transactions

TinkerPop conventions for child traversals is to spawn them anonymously from __, therefore:

or more succinctly via static import as:

Some users have chosen to instead write the above as:

which spawns a child traversal from a GraphTraversalSource. When spawned this way, a traversal is bound to a "source" and therefore is not anonymous. While the above code worked, it is important that there be less ways to do things with Gremlin so as to avoid confusion in examples, documentations and mailing list answers.

As of 3.5.0, attempting to use a traversal spawned from a "source" will result in an exception. Users will need to modify their code if they use the unconventional syntax.

See: TINKERPOP-2361

Gremlin has traditionally disallowed null as a value in traversals and not always in consistent ways:

Note how null can produce exception behavior or act as a filter. For 3.5.0, TinkerPop has not only made null usage consistent, but has also made it an allowable value within a Traversal:

TinkerGraph can be configured to support null as a property value and all graphs may not support this feature (for example, Neo4j does not). Please be sure to check the new supportsNullPropertyValues() feature (or the documentation of the graph provider) to determine if the Graph implementation allows null as a property value.

With respect to null in relation to properties, there was a bit of inconsistency in the handling of null in calls to property() depending on the type of mutation being executed demonstrated as follows in earlier versions:

This behavior has been altered to become consistent. First, assuming null is not supported as a property value, the setting of a property to null should have the behavior of removing the property in the same way in which you might do g.V().properties().drop():

Then, assuming null is supported as a property value, it would simply store the null for the key:

The above described changes also have an effect on steps like group() and groupCount() which formerly produced exceptions when keys could not be found:

For situations where the key did not exist, the approach was to filter away vertices that did not have the available key so that such steps would work properly or to write a more complex by() modulator to better handle the possibility of a missing key. With the latest changes however none of that is necessary unless desired:

In conclusion, this improved support of null may affect the behavior of existing traversals written in past versions of TinkerPop as it is no longer possible to rely on null to expect a filtering action for traversers. Please review existing Gremlin carefully to ensure that there are no unintended consequences of this change and that there are no opportunities to improve existing logic to take greater advantage of this expansion of null semantics.

See: TINKERPOP-2235, TINKERPOP-2099

The new ByModulatorOptimizationStrategy attempts to re-write by() modulator traversals to use their more optimized forms which can provide a major performance improvement. As a simple an example, a traversal like by(id()) would be replaced by by(id), thus replacing a step-based traversal with a token-based traversal.

See: TINKERPOP-1682

The new SeedStrategy allows the user to set a seed value for steps that make use of Random so that the traversal has the ability to return deterministic results. While this feature is useful for testing and debugging purposes, there are also some practical applications as well.

See: TINKERPOP-2014

The Property interface is not included in the hierarchy of Element. This means that an edge property or a meta-property are not considered elements the way that a VertexProperty is. As a result, some usages of T in relation to properties do not work consistently. One such example is by(T), a token-based traversal, where the following works for a VertexProperty but will not for edge properties or meta-properties:

For a Property you would need to use key()-step:

Aside from the inconsistency, this issue also presents a situation where performance is impacted as token-based traversals are inherently faster than step-based ones. In 3.5.0, this issue has been resolved in conjunction with the introduction of ByModulatorOptimizationStrategy which will optimize by(key()) and by(value()) to their appropriate token versions automatically.

See: TINKERPOP-1682

The match() step behavior might have seemed inconsistent those first using it. While there are a number of examples that might demonstrate this issue, the easiest one to consume would be:

The output is unexpected if there isn’t awareness of some underlying optimizations at play, where match() as the final step in the traversal implies that the user wants all of the labels as part of the output. With the addition of the extra steps, unfold() and identity() in the above case, the implication is that the traversal must be explicit in the labels to preserve from match, thus:

Being explicit, as is the preference in writing Gremlin to good form, helps restrict the path history required to execute the traversal and therefore preserves memory. Of course, making match() a special form of end step is a confusing approach as the behavior of the step changes simply because another step is in play. Furthermore, correct execution of the traversal, relies on the execution of traversal strategies when the same traversal should produce the same results irrespective of the strategies applied to it.

In 3.5.0, we look to better adhere to that guiding design principle and ensure a more consistent output for these types of traversals. While the preferred method is to specify the labels to preserve from match() with a following select() step as shown above, match() will now consistently return all labels when they are not specified explicitly.

See: TINKERPOP-2481, TINKERPOP-2499

Remote SideEffects

Remote traversals no longer support the retrieval of remote side-effects. Users must therefore directly return side-effects as part of their query if they need that data. Note that server settings for TraversalOpProcessor, which formerly held the cache for these side-effects, no longer have any effect and can be removed.

Audit Logging

The authentication.enableAuditlog configuration property is deprecated and replaced by the enableAuditLog property to also make it available to Authorizer implementations. With the new setting enabled, there are slight changes in the formatting of audit log messages. In particular, the name of the authenticated user is included in every message.

Authorization

While Gremlin Server has long had authentication options to determine if a user can connect to the server, it now also contains the ability to apply a level of authorization to better control what a particular authenticated user will have access to. Authorization is controlled by the new Authorizer interface, which can be implemented by users and graph providers to provide this custom functionality.

UnifiedChannelizer

Gremlin Server uses a Channelizer abstraction to configure different Netty pipelines which can then offer different server behaviors. Most commonly, users configure the WebSocketChannelizer to enable the websocket protocol to which the various language drivers can connect.

TinkerPop 3.5.0 introduces a new Channelizer implementation called the UnifiedChannelizer. This channelizer is somewhat similar to the WsAndHttpChannelizer in that combines websocket and standard HTTP protocols in the server, but it provides a new and improved thread management approach as well as a more streamlined execution model. The UnifiedChannelizer technically replaces all existing implementations, but is not yet configured by default in Gremlin Server. To use it, modify the channelizer setting in the server yaml file as follows:

As the UnifiedChannelizer is tested further, it will eventually become the default implementation. It may however be the preferred channelizer when using large numbers of short-lived sessions as the the threading model of the UnifiedChannelizer is better suited for such situations. If using this new channelizer, there are a few considerations to keep in mind:

See: TINKERPOP-2245, TINKERPOP-2269, TINKERPOP-2389, Reference Documentation

Some error conditions are temporary in nature and therefore an operation that ends in such a situation may be tried again as-is to potential success. In embedded use cases, an exception that implements the TemporaryException interface implies that the failing operation can be retried. For remote use cases, a ResponseStatusCode of 596 which equates to SERVER_ERROR_TEMPORARY is an indicator that a request may be retried.

With this more concrete and generalized approach to determining when retries should happen, the need to trap provider specific exceptions or to examine the text of error messages are removed. Before replacing existing code that might do these things currently, it may be best to include this sort of retry checking in addition to current methods as it may take time for providers to support these new options. Alternatively, if you can confirm that a provider does support this functionality then feel free to proceed wholly with this generalized TinkerPop approach.

Finally, it is important to note that TinkerPop drivers do not automatically retry when these conditions are met. It is up to the application to determine if retry is desired and how best to do so.

See: TINKERPOP-2517

With the removal of Python 2.x support the transport layer of gremlin-python has been rewritten to use a library that utilizes the asyncio event loop of Python 3. AIOHTTP utilizes Python 3’s event loop with a minimal HTTP abstraction and is now used for the transport layer. From a user’s perspective there is not much of a change except there is now new configuration options available through named parameters, see AIOHTTP ws_connect for more details. This change fixed a number of issues that were related to the IOLoop of the old Tornado transport layer, which has been completely removed from the library. An additional config which enables the driver to be used from within an event loop has been added and can be used by setting call_from_event_loop=True when connecting.

See: TINKERPOP-1886, TINKERPOP-2388, TINKERPOP-2484, TINKERPOP-2546,

The Python Driver now supports Kerberos based authentication:

See: TINKERPOP-1641, Reference Documentation

The following deprecated classes, methods or fields have been removed in this version:

Certain elements of the API were not or could not be deprecated in prior versions and were simply renamed for this release:

See: TINKERPOP-2080, TINKERPOP-2231, TINKERPOP-2233, TINKERPOP-2239, TINKERPOP-2269, TINKERPOP-2273, TINKERPOP-2455, 3.2.10 Upgrade Documentation for SSL

Various TraversalStrategy implementations inject barrier() instances to improve traversal performance. It is often helpful in debugging situations or in tweaking traversal performance activities that it may be necessary to prevent automated barrier() injection. Formerly, that involved removing LazyBarrierStrategy, RepeatUnrollStrategy, and PathRetractionStrategy as all three of those strategies had the potential to add a barrier(). While this approach would work, it may be hard to recall those three strategies easily and you may be removing optimizations that might have been useful in addition to the barrier().

In 3.4.11, it is only necessary to remove LazyBarrierStrategy. Doing so will disable barrier() additions in other strategies and as this is the new model of execution, it can be expected to work as the approach in the future even if other strategies attempt to add barrier() steps for some reason. Note of course that this rule only applies to TinkerPop-enabled strategies. It is possible that provider-specific strategies could add barrier()-steps and thus still require direct removal.

See: TINKERPOP-1994

In version 3.4.9, it became possible to set client-side read/write timeouts. Those timeouts were defaulted to thirty seconds which could put the client at odds with other timeout settings which could then cause unexpected failures. The client-side timeouts have now been defaulted to None leaving it to users to set the timeout themselves if they so desire. In this way, 3.4.10 has the same timeout behavior as 3.4.8 without any additional workarounds.

The timeouts can be configured as shown in the below example:

See: TINKERPOP-2505

One of the silent features of Gremlin is the ScriptTranslator. More specifically, the implementation of this interface which will convert a Traversal object (or Gremlin Bytecode) into a proper String representation that is syntactically correct for the implementation language.

Some Gremlin users may already be aware of the implementations for Groovy and Python from previous versions. These classes have largely been used for testing purposes, but users have found helpful use cases for them and they have now been promoted to gremlin-core from their original locations. The old versions in gremlin-groovy and gremlin-python have been deprecated. There have been some improvements to the GroovyTranslator such that the Gremlin generated will not match character for character with the deprecated version. There may also be some potential for the newer version in 3.4.9 to generate scripts that will not work in earlier versions. It is therefore best to use 3.4.9 translators within environments where 3.4.9 is uniformly supported. If older versions are in place, it may be better to continue use of the deprecated versions.

See: TINKERPOP-2461

The :bytecode command in the Gremlin console includes two new options: reset and config. Both options provide ways to better control the GraphSONMapper used internally by the command. The reset option will replace the current GraphSONMapper with a new one with some basic defaults: GraphSON 3.0 with extension and TinkerIoRegistry if present. The config option provides a way to specify a custom GraphSONMapper or additional configurations to the default one previously described.

See: TINKERPOP-2479

The withStrategies() configuration step accepts a variable number of TraversalStrategy instances. In Java, those instances are typically constructed with instance() if it is a singleton or by way of a builder pattern which provides a fluent, type safe method to create the object. For Groovy, which is highly applicable to those who use Gremlin scripts in their applications or work a lot within tools similar to the Gremlin Console, the builder syntax can work but doesn’t really match the nature of the Groovy language. Using a strategy in this script context would look something like:

While this method will still work, it is now possible to use a more succinct syntax for Groovy scripts:

The rules are straightforward. If a strategy can be instantiated with instance() as a singleton then use just the class name as a shortcut. Interestingly, many users try this syntax when they first get started and obviously fail. With the syntax present, they will have one less error to contend with in their early days of Gremlin. For strategies that take configurations, these strategies will use named arguments in the constructor where the names match the expected builder methods. This named argument syntax is common to Groovy and not something special to Gremlin - it is just now exposed for this purpose.

See: TINKERPOP-2466

The AnonymousTraversalSource was introduced in 3.3.5 and is most typically used for constructing remote TraversalSource instances, but it also provides a way to construct a TraversalSource from an embedded Graph instance:

The withGraph(Graph) method is now deprecated in favor the new withEmbedded(Graph) method that is more explicit about its intent:

This change is mostly applicable to JVM languages where embedded Graph instances are available. For Gremlin Language Variants not on the JVM, the withGraph(Graph) method has simply been deprecated and not replaced (with the preference to use withRemote() variants).

See: TINKERPOP-2413

Using SubgraphStrategy, PartitionStrategy and other TraversalStrategy implementations is now possible in Javascript.

See: TINKERPOP-2054

Gremlin Server now supports standard WebSocket compression (per RFC 7692). Both the Java and Python drivers support this functionality from the client’s perspective. Compression is enabled by default and should be backward compatible, thus allowing older versions of the driver to connect to newer versions of the server and vice versa. Using the compression-enabled drivers with a server that also supports that functionality will greatly reduce network IO requirements.

See: TINKERPOP-2441, TINKERPOP-2453

The Java Driver was designed with the idea that a Cluster instance would be created once and then used for the life of the application. As a result, the cost of setup and termination of that instance was typically sunk into the general startup and shutdown of the application itself. In some environments, where applications were short-lived, this cost was quite apparent and undesirable given that it might take several seconds to initialize and then a similar amount of time for proper shutdown.

In 3.4.9, the initialization and shutdown of the Cluster object has been improved dramatically, which should be especially helpful to those aforementioned ephemeral environments. The following micro-benchmark results demonstrate the difference in performance between 3.4.8 and 3.4.9:

See: TINKERPOP-2445

With Java it has been possible to pass per-request settings for both scripts and bytecode. While Javascript, Python, and .NET allowed this in various ways, it wasn’t quite as convenient as Java, nor was it well documented. In this release, the approach for making this sort of per-request configurations is now much more consistent across languages. We see this most evident in bytecode based requests:

Java

C#

Javascript

Python

Please see the new "Per Request Settings" sections for each language in the Gremlin Drivers and Variants section for information on how to send scripts with specific request configurations.

See: TINKERPOP-2296, TINKERPOP-2420, TINKERPOP-2421

The org.apache.tinkerpop.gremlin.server.util.CheckedGraphManager can be used instead of org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager (in gremlin-server.yml to ensures that Gremlin Server fails to start if all graphs fails. This configuration option can be useful for a number of different situations (e.g. use CheckedGraphManager on a Kubernetes cluster to ensure that a pod will be restarted when it cannot properly handle requests.) As a final note, DefaultGraphManager is no longer final and thus can be extended.

See: TINKERPOP-2436

Lambda scripts can now be utilized in gremlin-javascript and follows roughly the same pattern as Python does:

See: TINKERPOP-2001

The Gremlin.NET driver now automatically tries to reconnect to a server when no open connection is available to submit a request. This should enable the driver to handle cases where the server is only temporarily unavailable or where the server has closed connections which some graph providers do when no requests were sent for some time.

See: TINKERPOP-2288

Gremlin Console now has the :cls command to clear the screen. This feature acts as an alternative to platform specific clear operations and provides a common way to perform that function.

TINKERPOP-2357

In 3.4.5 the equality of the Property object changed to allow language features like dedup() to work more consistently. An unnoticed side-effect of that change was that calling drop() on properties that had the same values would not properly remove all their instances. This problem affected edge and meta property instances but not the properties of vertices as their equality definitions had not changed.

This issue is now resolved with the side-effect being that the inclusion of drop() will disable LazyBarrierStrategy which prevents automatic bulking. In most common cases, the impact of that optimization loss should be minimal and could be added back manually with barrier() steps in the appropriate places.

See: TINKERPOP-2338

It is quite common to use the by(String) modulator when doing some for of selection operation where the String is the key to the value of the current Traverser, demonstrated as follows:

Of course, this approach usually only works when the current Traverser is an Element. If it is not an element, the error is swift and severe:

and while it is perhaps straightforward to see the problem in the above example, it is not always clear exactly where the mistake is. The above example is the typical misuse of by(String) and comes when one tries to treat a Map the same way as an Element (which is quite reasonable).

In 3.4.5, the issue of using by(String) on a Map and the error messaging have been resolved as follows:

See: TINKERPOP-2314

Previously, hasKey()-step and hasValue()-step only applied to vertex properties. In order to support more generalized scenarios, the behavior of these steps were modified to allow them to be applied to both edge properties and meta-properties.

The original behavior is demonstrated as follows:

The new behavior of hasKey() with edge property can be seen as:

Similar behavior of for hasKey() can be seen with meta-properties:

The new behavior for hasValue() with edge property is as follows:

and similarly with hasValue() for meta-properties:

TINKERPOP-1733

There was some inconsistency in terms of Property equality in earlier versions of Gremlin. Equality is now defined as follows: two properties are equal only if their key and value are equal, even if their parent elements are not equal. It makes sense when comparing properties regardless of parent elements to just focus on the property itself as it yields more uniform and concise results to reason about. The benefit of this change is that the behavior of property comparison and dedup()-step are predictable, and it will not affect the result if the property is detached from the parent element.

The old behavior can be shown using "The Crew" toy graph as follows:

The new more consistent behavior is demonstrated below:

See: TINKERPOP-2318

There is now support for GraphBinary in Python. As with Java, it remains a working but experimental format that is still under evaluation. This new serializer can be used by first ensuring that it is available on the server and then configuring the connection as follows:

TINKERPOP-2279

Since graph elements (i.e. Vertex, Edge, and VertexProperty) are returned from remote sources as references (i.e. without properties), one of the more common needs for Gremlin users is the ability to easily return a Map representation of the elements that they are querying. Typically, such transformations are handled by valueMap():

or by way of project():

While valueMap() works reasonably well for Vertex and VertexProperty transformations it does less well for Edge as it fails to include incident vertices:

This limitation forces a fairly verbose use of project() for what is a reasonably common requirement:

By introducing elementMap()-step, there is now a single step that covers the most common transformation requirements for all three graph elements:

See: TINKERPOP-2284, Reference Documentation

The store()-step and aggregate()-step do the same thing in different ways, where the former is lazy and the latter is eager in the side-effect collection of objects from the traversal. The different behaviors can be thought of as differing applications of Scope where global is eager and local is lazy. As a result, there is no need for both steps when one will do.

As of 3.4.3, store(String) is now deprecated in favor of aggregate(Scope, String) where the Scope should be set to local to ensure the same functionality as store(). Note that aggregate('x') is the same as aggregate(global,'x').

See: TINKERPOP-1553

Gryo is now deprecated as a serialization format for Gremlin Server, however, it is still configured as a default option in the sample configuration files packaged with the server. The preferred serialization option should now be GraphBinary. Note that Gremlin Console is now configured to use GraphBinary by default.

See: TINKERPOP-2250

In 3.4.0, the notion of RequestOptions were added so that users could have an easier way to configure settings on individual requests made through the Java driver. While that change provided a way to set those configurations for script based requests, it didn’t include options to make those settings in a Traversal submitted via Bytecode. In this release those settings become available via with() step on the TraversalSource as follows:

See: TINKERPOP-2211

The Gremlin Console timeout that is set by :remote config timeout x was client-side only in prior versions, which meant that if the console timeout was less than the server timeout the client would timeout but the server might still be processing the request. Similarly, a longer timeout on the console would not change the server and the timeout would occur sooner than expected. These discrepancies often led to confusion.

As of 3.4.0, the Java Driver API allowed for timeout settings to be more easily passed per request, so the console was modified for this current version to pass the console timeout for each remote submission thus yielding more consistent and intuitive behavior.

See: TINKERPOP-2203

In the initial release of sparql-gremlin it was only possible to execute a SPARQL query and have it translate to Gremlin. Therefore, it was only possible to write a query like this:

In this release, however, it is now possible to further process that result with Gremlin steps:

See: TINKERPOP-2171, Reference Documentation

The sparql-gremlin module is a SPARQL to Gremlin compiler, which allows SPARQL to be executed over any TinkerPop-enabled graph system.

See: TINKERPOP-1878, Reference Documentation

The Gremlin.NET driver now uses request pipelining. This allows connections to be reused for different requests in parallel which should lead to better utilization of connections. The ConnectionPool now also has a fixed size whereas it could previously create an unlimited number of connections. Each Connection can handle up to MaxInProcessPerConnection requests in parallel. If this limit is reached for all connections, then a NoConnectionAvailableException is thrown which makes this a breaking change.

These settings can be set as properties on the ConnectionPoolSettings instance that can be passed to the GremlinClient.

See: TINKERPOP-1774, TINKERPOP-1775, Reference Documentation

TinkerPop 3.4.0 adds a new index()-step, which allows users to transform simple collections into index collections or maps.

The valueMap() step now supports by and with modulation, which also led to the deprecation of valueMap(true) overloads.

In previous versions within() and without() performed strict number comparisons; that means these predicates did not only compare number values, but also the type. This was inconsistent with how other predicates (like eq, gt, etc.) work. All predicates will now ignore the number type and instead compare numbers only based on their value.

Old behavior:

New behavior:

See: TINKERPOP-2058

Gremlin Server has had some inconsistent behavior in the serialization of the results it returns. Remote traversals based on Gremlin bytecode always detach returned graph elements to "reference" (i.e. removes properties and only include the id and label), but scripts would detach graph elements and include the properties. For 3.4.0, TinkerPop introduces the ReferenceElementStrategy which can be configured on a GraphTraversalSource to always detach to "reference".

The packaged initialization scripts that come with Gremlin Server now pre-configure the sample graphs with this strategy to ensure that both scripts and bytecode based requests over any protocol (HTTP, websocket, etc) and serialization format all return a "reference". To revert to the old form, simply remove the strategy in the initialization script.

It is recommended that users choose to configure their GraphTraversalSource instances with ReferenceElementStrategy as working with "references" only is the recommended method for developing applications with TinkerPop. In the future, it is possible that ReferenceElementStrategy will be configured by default for all graphs on or off Gremlin Server, so it would be best to start utilizing it now and grooming existing Gremlin and related application code to account for it.

See: TINKERPOP-2075

Gremlin now supports simple text predicates on top of the existing P predicates. Both, the new TextP text predicates and the old P predicates, can be chained using and() and or().

See: TINKERPOP-2041

The infix notation of and() and or() now supports an arbitrary number of traversals and ConnectiveStrategy produces a traversal with proper AND and OR semantics.

Furthermore, previous versions failed to apply 3 or more and() steps using the infix notation, this is now fixed.

See: TINKERPOP-2029

GraphBinary is a new language agnostic, network serialization format designed to replace Gryo and GraphSON. At this time it is only available on the JVM, but support will be added for other languages in upcoming releases. The serializer has been configured in Gremlin Server’s packaged configuration files. The serializer can be configured using the Java driver as follows:

See: TINKERPOP-1942, IO Documentation

The Gremlin Server protocol allows for status attributes to be returned in responses. These attributes were typically for internal use, but were designed with extensibility in mind so that providers could place return their own attributes to calling clients. Unfortunately, unless the client was being used with protocol level requests (which wasn’t convenient) those attributes were essentially hidden from view. As of this version however, status attributes are fully retrievable for both successful requests and exceptions.

See: TINKERPOP-1913

This version of TinkerPop introduces the with()-step to Gremlin. It isn’t really a step but is instead a step modulator. This modulator allows the step it is modifying to accept configurations that can be used to alter the behavior of the step itself. A good example of its usage is shown with the revised syntax of the pageRank()-step which now uses with() to replace the old by() options:

A similar change was made for peerPressure()-step:

Note that the by() modulators still work, but should be considered deprecated and open for removal in a future release where breaking changes are allowed.

See: TINKERPOP-1975, Reference Documentation

Calculating the shortest path between vertices is a common graph use case. While the traversal to determine a shortest path can be expressed in Gremlin, this particular problem is common enough that the feature has been encapsulated into its own step, demonstrated as follows:

See: TINKERPOP-1990, Reference Documentation

In prior version of TinkerPop, it was recommended that the identification of Connected Component instances in a graph be computed by way of a reasonably complex bit of Gremlin that looked something like this:

The above approach had a number of drawbacks that included a large execution cost as well as incompatibilities in OLAP. To simplify usage of this commonly use graph algorithm, TinkerPop 3.4.0 introduces the connectedComponent() step which reduces the above operation to:

It is important to note that this step does require the use of a GraphComputer to work, as it utilizes a VertexProgram behind the scenes.

See: TINKERPOP-1967, Reference Documentation

There have been some important changes to IO operations for reading and writing graph data. The use of Graph.io() has been deprecated to further remove dependence on the Graph (Structure) API for users and to extend these basic operations to GLV users by making these features available as part of the Gremlin language.

It is now possible to simply use Gremlin:

While io()-step is still single-threaded for OLTP style loading, it can be utilized in conjunction with OLAP which internally uses CloneVertexProgram and therefore any graph InputFormat or OutputFormat can be configured in conjunction with this step for parallel loads of large datasets.

It is also worth noting that the io()-step may be overridden by graph providers to utilize their native bulk-loading features, so consult the documentation of the implementation being used to determine if there are any improved efficiencies there.

See: TINKERPOP-1996, Reference Documentation

The Java driver now allows for various options to be set on a per-request basis via new overloads to submit() that accept RequestOption instances. A good use-case for this feature is to set a per-request override to the scriptEvaluationTimeout so that it only applies to the current request.

See: TINKERPOP-1342

Previously min() and max() were only working for numeric values. This has been changed and these steps can now operate over any Comparable value. The common workaround was the combination of order().by() and limit() as shown here:

Any attempt to use min() or max() on non-numeric values lead to an exception:

With the changes in this release these kind of queries became a lot easier:

Traversals now support nesting of repeat() loops.

These can now be used to repeat another traversal while in a looped context, either inside the body of a repeat() or in its step modifiers (until() or emit()).

See: TINKERPOP-967

There were some minor modifications to how EventStrategy is constructed and what can be expected from events raised from the addition of new properties.

With respect to the change in terms of EventStrategy construction, the detach() builder method formerly took a Class as an argument and that Class was meant to be one of the various "detachment factories" or null. That approach was a bit confusing, so that signature has changed to detach(EventStrategy.Detachment) where the argument is a more handy enum of detachment options.

As for the changes related to events themselves, it is first worth noting that the previously deprecated vertexPropertyChanged(Vertex, Property, Object, Object…​) on MutationListener has been removed for what should have originally been the correct signature of vertexPropertyChanged(Vertex, VertexProperty, Object, Object…​). In prior versions when this method and its related edgePropertyChanged() and vertexPropertyPropertyChanged() were triggered by way of the addition of a new property a "fake" property was included with a null value for the "oldValue" argument to these methods (as it did not exist prior to this event). That was a bit awkward to reason about when dealing with that event. To make this easier, the event now raises with a KeyedVertexProperty or KeyedProperty instance, which only contains a property key and no value in them.

TINKERPOP-1831

The behavior of min(), max(), mean() and sum() has been modified to return no result if there’s no input. Previously these steps yielded the internal seed value:

These traversals will no longer emit a result. Note, that this also affects more complex scenarios, e.g. if these steps are used in by() modulators:

Since software vertices have no outgoing edges and thus no weight values to sum, software will no longer show up in the result. In order to get the same result as before, one would have to add a coalesce()-step:

See: TINKERPOP-1777

The order of select scopes has been changed to: maps, side-effects, paths. Previously the order was: side-effects, maps, paths - which made it almost impossible to select a specific map entry if a side-effect with the same name existed.

The following snippets illustrate the changed behavior:

Above is the old behavior; the second select("a") has no effect, it selects the side-effect a again, although one would expect to get the map entry a. What follows is the new behavior:

See: TINKERPOP-1522

In earlier versions of TinkerPop, BulkSet was coerced to a List for GraphSON which was convenient in that it didn’t add a new data type to support, but inconvenient in that it meant that certain process tests were not consistent in terms of how they ran and the benefits of the BulkSet were "lost" in that the "bulk" was being resolved server side. With the addition of BulkSet as a GraphSON type the "bulk" is now resolved on the client side by the language variant. How that resolution occurs depends upon the language variant. For Java, there is a BulkSet object which maintains that structure sent from the server. For the other variants, the BulkSet is deserialized to a List form which results in a much larger memory footprint than what is contained the BulkSet.

See: TINKERPOP-2111

Bindings were formerly created using a Python 2-tuple as a bit of syntactic sugar, but all other language variants used an explicit Bindings object which gremlin-python already had in place. To make all work variants behave consistently, the 2-tuple syntax has been removed in favor of the explicit Bindings.of() option.

See: TINKERPOP-2116

This section describes newly deprecated classes, methods, components and patterns of usage as well as which previously deprecated features have been officially removed or repurposed.

While TinkerPop doesn’t recommend the use of sessions for most use cases, it does remain a feature that is available and exposed on the server. As such, providing support in all Gremlin Language Variants for this feature is useful in ensuring a consistent implementation for all programming languages.

The maxWaitForSessionClose setting for the Java driver has been deprecated and in some sense replaced by the maxWaitForClose setting. The two settings perform different functions, but expect maxWaitForSessionClose to be removed in future versions. The maxWaitForClose performs a more useful function than maxWaitForSessionClose in the sense that it tells the driver how long it should wait for pending messages from the server before closing the connection. The maxWaitForSessionClose on the other hand is how long the driver should wait for the server to respond to a session close message (i.e. an actual response from the server). Waiting for that specific response to the session close message could result in the driver hanging on calls to Client.close() if there is a long run query running on the server and close message is stacked behind that in queue.

Future versions will remove support for that particular message and simply close the session when the connection is closed. As a result that setting will no longer be useful. The old setting is really only useful for connecting to older versions of the server prior to 3.3.11 that do not have the session shutdown hook bound to the close of the connection.

See: TINKERPOP-2336

Once a traversal has been executed (i.e. iterated) it’s internal state is such that it cannot be re-used:

To re-use a traversal it must be copied with clone() as follows:

The ability to clone() has been exclusive to Java and was a missing component of other supported languages like Python, Javascript and .NET. This feature has now been added for all the language variants making the ability to re-use traversals consistent in all ecosystems.

See: TINKERPOP-2315

Jython support in gremlin-python has been deprecated to focus on native Python 3.x support for 3.5.0 where Jython support will be wholly removed. This change does mean that TinkerPop will no longer support the ability to submit lambdas as native Python scripts. They will need to be submitted as Groovy and the library will be defaulted to do as such.

See: TINKERPOP-2322

ReservedKeysVerificationStrategy is a new VerificationStrategy that can be used to help prevent traversals from adding property keys that are protected. They may be protected as a result of the key being a reserved keyword of the underlying graph system or they key may simply violate some standard conventions.

TINKERPOP-2280

Gremlin Javascript now enables more robust error handling by way of a ResponseError which provides access to more information from the server. Specifically, it includes the statusMessage and statusCode which were formerly packed into the Error.message property, which meant that the error message string had to be parsed if there was a need to take a specific action based on that information. The ResponseError also includes the statusAttributes which is a Map object that will typically incorporate server exceptions and stackTrace keys, but could also include provider specific error information.

The original error messaging has remained unchanged and therefore users who were message parsing should not expect changes in behavior, however, future versions will eliminate the "overuse" of the Error.message property, so it is advised that users update their code to take advantage of the ResponseError features.

See: TINKERPOP-2285

Expect a NoHostAvailableException rather than a more generic TimeoutException if the Java driver is unable to connect to a Host. This sort of failure can occur in a number of different scenarios, but can often occur when there are configuration problems with authentication and SSL, preventing the connection to the Host to be established.

TINKERPOP-2132

The scriptEvaluationTimeout controls the length of time a request is processed by Gremlin Server (or at its core, the GremlinExecutor). Of course, with the introduction of bytecode-based requests many versions ago, this naming has failed to make complete sense in more recent times. Therefore, scriptEvaluationTimeout has been deprecated and replaced by evaluationTimeout. Both configurations are still respected, but it is advised that users switch to evaluationTimeout as scriptEvaluationTimeout will be removed in a later version.

Note that when configuring Gremlin Server’s evaluationTimeout that the scriptEvaluationTimeout should be set to -1 (the default) or else it will use that value in its initialization of the server and ignore the evaluationTimeout.

See: TINKERPOP-2213

Prior to this version, branch steps (in particular BranchStep and ChooseStep) could only handle constant values and Pick tokens. Starting in this version, these steps will also accept predicates and traversals as show in the example below.

See: TINKERPOP-1084

With GraphSON, the g:Date is meant to represent a millisecond-precision offset from the unix epoch, but earlier version of Gremlin Python were using the timezone of the local system to handle serialization and deserialization, thus resulting in incorrect conversion. This issue is now resolved. It may be necessary to remove workarounds that have been introduced to combat this problem.

See: TINKERPOP-2264

Gremlin-Javascript now supports withComputer() syntax, which means that it is now possible in Javascript to utilize Gremlin steps that require a GraphComputer to execute (e.g. pageRank() and peerPressure()).

See: TINKERPOP-2251

There is now greater consistency across Gremlin Language Variants with hasNext() support added to both JavaScript and .NET. Note that in .NET, the proper method name follows C# capitalization semantics and is referred to as HasNext().

See: TINKERPOP-1921

Traversal.getSideEffects() has been deprecated for purposes of external calls by end-users. While the method is still present TinkerPop no longer guarantees its existence in future versions or consistency of its behavior, especially for Gremlin Language Variants and remote traversal execution. If this method is currently in use to gather side-effect results after traversal execution, please change such code to use cap()-step. For example, code like this:

should be converted to something like:

TINKERPOP-2265

Prior TinkerPop 3.3.6 EventStrategy did not work with multi-properties. The EventStrategy behavior for single-valued properties has not changed; if a property is added to a multi-valued VertexProperty, then a VertexPropertyChangedEvent will be now be fired. The arguments passed to the event depend on the cardinality type.

See: TINKERPOP-2159

Gremlin DSL developers had to rely on monkey patching to build a Domain Specific Language in JavaScript. With this latest version, the process is a bit more elegant and largely handled by extending existing core classes of Gremlin-JavaScript which is a bit more in-line with other language variant patterns for DSL development.

See: TINKERPOP-2089, Reference Documentation

There are occasions where a traversal (or other evaluation) takes longer than expected to complete. Common examples of these situations typically include:

In earlier versions, the only option was to kill the Gremlin Console (typically with ctrl+c), but as of this version ctrl+c will be intercepted and attempt to interrupt whatever process is currently executing.

See TINKERPOP-2181

Gremlin Console included references to:

to provide some benchmarking and profiling tools for the Utilities Plugin. Neither project is well maintained at this point and given that TinkerPop has moved past the versions of Groovy that are supported by these tools it doesn’t make sense to retain the dependencies. Users who wish to continue to use these tools can obviously do so still by manually adding the dependencies to the Gremlin Console or with the following command:

and then import the appropriate classes and methods to use.

See: TINKERPOP-2182

The Gremlin Recipes documentation now has a new section which discusses anti-patterns in Gremlin. Its subsections detail commonly seen approaches to the Gremlin language which can lead to poorly written and/or under-performing traversals. The new Anti-Patterns Section can be found here.

See: TINKERPOP-2114

The AnonymousTraversalSource provides for a more unified syntax for TraversalSource construction by placing greater user emphasis on the creation of the source rather than the Graph it is connected to. It has a number of different static traversal() methods available and when imported as:

allows TraversalSource construction syntax such as:

Typically, this syntax is used for "remote" traversal construction for bytecode based requests, but has the option to bind a local Graph instance to it as well. It doesn’t save much typing to do so obviously, so it may not be best used in that situation. Python, Javascript and .NET have similar syntax.

See: TINKERPOP-2078

The Gremlin Console now has a new :bytecode command to help users work more directly with Gremlin bytecode. The command is more of a debugging tool than something that would be used for every day purposes. It is sometimes helpful to look at Gremlin bytecode directly and the process for viewing it in human readable format is not a single step process. It is also not immediately clear how to convert bytecode to a Gremlin string. The :bytecode command aims to help with both of these issues:

See: TINKERPOP-2050, Reference Documentation - Console Commands

The "class map" cache in Gremlin Server (specifically the GremlinGroovyScriptEngine) that holds compiled scripts is now fully configurable via the GroovyCompilerGremlinPlugin.classMapCacheSpecification.

See: TINKERPOP-2038, Reference Documentation - Cache Management

A new strategy named EarlyLimitStrategy was added. The strategy will try to find a better spot for any RangeStep, which is as early as possible in the traversal. If possible it will also merge multiple RangeStep`s into a single one by recalculating the range for the first step and removing the second. If it turns out that the merge of two steps won’t produce a valid range (an empty result), then the `EarlyLimitStrategy will remove the RangeStep instances and insert a NoneStep instead.

This strategy is particularly useful when a provider implementation generates the queries to the underlying database. By making sure that the ranges are applied as early as possible, we can ensure that the underlying database is only asked for the least amount of data necessary to continue the traversal evaluation.

The Order enum originally introduced incr for ascending order and decr for descending order. It’s not clear why they were named this way when common querying parlance would call for asc and desc for those respective cases. Note that incr and decr have not been removed - just deprecated and thus marked for future removal. Prefer asc and desc going forward when writing Gremlin and look to update existing code using the deprecated values.

See: TINKERPOP-1956

In Gremlin Server, it is best not to use the timedInterrupt option on GroovyCompilerGremlinPlugin because it can compete with the scriptEvaluationTimeout setting and produce a different error path. Simply rely on scriptEvaluationTimeout as it covers both script evaluation time and result iteration time.

See: TINKERPOP-1778

The Credential DSL has been modified to work as a standard Java-based Gremlin DSL. The now deprecated old approach used a "graph wrapping" style that was developed long before the recommended method for building DSLs was published. Under this new model, the DSL is initialized via traversal as follows:

See: TINKERPOP-1903, Reference Documentation - Security

Graph traversals that return a Set from Java are now coerced to a List in Python. This change ensures that Python results match Java results for the same traversal. It is possible to see this problem in prior versions of gremlin-python where a Set of numbers of different types are returned. In Java, a set of:

would be deserialized to the following in Python:

Now that the Java Set is coerced to a List in Gremin Python, the Java Set can be fully represented. Users who require a Set will need to manually convert their List to a Set.

See: TINKERPOP-1844

There was a bug in GraphSON 3.0 serialization that prevented proper handling of results contain Path object. As a result, traversals that used and returned results from the path()-step in Python would return unusable results, but did not actually cause an exception condition. This problem is now resolved.

See: TINKERPOP-1799

GraphTraversal.math(String) was added. This step provides scientific calculator capabilities to a Gremlin traversal.

See: TINKERPOP-1632

The from() and to()-steps of GraphTraversal have a Traversal<E,Vertex> overload. The E has been changed to ? in order to reduce < >-based coersion in strongly type Gremlin language variants.

See: TINKERPOP-1793

The GraphTraversal and GraphTraversalSource methods of addV() and addE() have been extended to support dynamic label determination upon element creation. Both these methods can take a Traversal<?, String> where the first String returned by the traversal is used as the label of the respective element.

See: TINKERPOP-1793

There were two major bugs in the way in which PageRank was being calculated in PageRankVertexProgram. First, teleportation energy was not being distributed correctly amongst the vertices at each round. Second, terminal vertices (i.e. vertices with no outgoing edges) did not have their full gathered energy distributed via teleportation.

For users upgrading, note that while the relative rank orders will remain "the same," the actual PageRank values will differ from prior TinkerPop versions.

Normalization preserved through computation:

Two other additions to PageRankVertexProgram were provided as well.

See: TINKERPOP-1783

While 3.3.0 released Gryo 3.0 and GraphSON 3.0 and these versions were defaulted in a number of places, it seems that some key defaults were missed. Specifically, calls to Graph.io(graphson()) and Graph.io(gryo()) were still using the old versions. The defaults have now been changed to ensure 3.0 is properly referenced in those cases.

See Neo4j’s 3.2 Upgrade FAQ for a complete guide on how to upgrade from the previous 2.3.3 version. Also note that many of the configuration settings have changed from neo4j 2x to 3x

In particular, these properties referenced in TinkerPop documentation and configuration were renamed:

TinkerPop has always packaged sample graphs with its zip distributions. As of 3.3.0, the distributions will only include Gryo 3.0, GraphSON 3.0 and GraphML (which is unversioned) files. Other versions are not included, but could obviously be generated using the IO API directly.

GraphTraversal.hasXXX(), where XXX is Id, Label, Key, Value, was faulty in that they relied on calling an intermediate method for flattening Object[] arguments and thus, yielding a non 1-to-1 correspondence between GraphTraversal and Bytecode. This has been remedied. Most users will not notice this change. Perhaps only some users that may use Java reflection over GraphTraversal might have a simple problem.

See: TINKERPOP-1520

Graphite and Ganglia are no longer packaged with the Gremlin Server distribution. They are considered optional dependencies and therefore must be installed manually by the user.

See: TINKERPOP-1550, Reference Documentation - Metrics

SelectStep and SelectOneStep (select()) are the only Scoping steps that default to Pop.mixed as their labeled path selection criteria. All other steps, like match(), where() and dedup(), use Pop.last. In order to better enable optimizations around total Pop.last traversals, the select()-steps now default to Pop.last. Most users will not notice a difference as it is rare for repeated labels to be used in practice. However, formal backwards compatibility is possible as outlined below.

Assuming that x is not a Pop argument:

If an explicit Pop argument is provided, then no changes are required.

See: TINKERPOP-1541

The optional()-step was previously implemented using ChooseStep. However, if the optional branch contained side-effects, then unexpected behaviors can emerge. Thus, a potential backwards compatibility issue arises if side-effects were being used in optional(). However, the behavior would be unpredictable so this backwards incompatibility is desirable.

See TINKERPOP-1506

It is no longer possible to intialize the Gremlin Console with a script without use of -e. In other words, prior versions allowed:

Such a command must now be written as:

See: TINKERPOP-1283, TINKERPOP-1651

GraphTraversal.valueMap(includeTokens,propertyKeys…​) now returns a Map<Object,E> to account for the presence of T.id or T.label if you pass true to it.

See: TINKERPOP-1483

The TinkerPop reference documentation has always mentioned that the gremlin-spark /lib directory needed to be added to HADOOP_GREMLIN_LIBS environment variable. In reality, that was not truly necessary. With Spark 1.x having gremlin-spark in HADOOP_GREMLIN_LIBS hasn’t been a problem, but Spark 2.0 introduces a check for duplicate jars on the path which will cause job initialization to fail. As a result, going forward with TinkerPop 3.3.0, the gremlin-spark lib directory should not be included in HADOOP_GREMLIN_LIBS.

The following deprecated classes, methods or fields have been removed in this version:

Please see the javadoc deprecation notes or upgrade documentation specific to when the deprecation took place to understand how to resolve this breaking change.

See: TINKERPOP-832, TINKERPOP-833, TINKERPOP-834, TINKERPOP-999, TINKERPOP-1010, TINKERPOP-1028, TINKERPOP-1040, TINKERPOP-1046, TINKERPOP-1049, TINKERPOP-1142, TINKERPOP-1169, TINKERPOP-1171, TINKERPOP-1275, TINKERPOP-1283, TINKERPOP-1289, TINKERPOP-1291, TINKERPOP-1420, TINKERPOP-1421, TINKERPOP-1465, TINKERPOP-1481, TINKERPOP-1526, TINKERPOP-1603, TINKERPOP-1612, TINKERPOP-1622, TINKERPOP-1651, TINKERPOP-1694, TINKERPOP-1700, TINKERPOP-1706, TINKERPOP-1721, TINKERPOP-1719, TINKERPOP-1720, TINKERPOP-880, TINKERPOP-1170, TINKERPOP-1729

gremlin-server.sh is now also an init script and can no longer be started without parameters. To start it in the foreground with defaults like previous usage, please use the console parameter. Also, gremlin-server.sh will continue to start in the foreground when provided a yaml configuration file.

How to install as a service has been added to the Reference Documentation - As A Service.

The switch name has changed for installing dependencies. -i has been deprecated and replaced by install.

See: TINKERPOP-980, Reference Documentation - Server Configuration.

The userMapperFromGraph serialization configuration option was used to allow the IO configurations of a specific graph to be assigned to a specific serializer. This feature has been removed completely now. Please use the ioRegistries configuration option to add one or more specific Graph serialization capabilities to a serializer.

see: TINKERPOP-1699

The switch name has changed for installing dependencies. -i has been deprecated and replaced by install.

Historically, SparkGraphComputer has used GryoSerializer to handle the serialization of objects in Spark. The reason this exists is because TinkerPop uses a shaded version of Kryo and thus, couldn’t use the standard KryoSerializer-model provided by Spark. However, a "shim model" was created which allows for the shaded and unshaded versions of Kryo to interact with one another. To this end, KryoSerializer can now be used with a GryoRegistrator. The properties file for a SparkGraphComputer now looks as follows:

If the old GryoSerializer model is desired, then the properties file should simply look as before:

See: TINKERPOP-1389

The API for the script provided to a ScriptInputFormat has changed slightly. The signature for parse(line, factory) is now simply parse(line). The inclusion of factory was deprecated in 3.1.2. Instead of using the {{factory}} to get the {{StarGraph}} there is a {{graph}} variable in the glocal context of the script. Simply use that directly in the script.

See: TINKERPOP-1137, Reference Documentation - Script I/O Format

The Gremlin Javascript Driver now supports SASL Plain Text authentication against a Gremlin Server.

See: TINKERPOP-1977, Reference Documentation - Gremlin Javascript

TinkerPop improves its security posture by removing insecure defaults and adding forward-looking standards support.

Gremlin Server no longer supports automatically creating self-signed certificates. Self-signed certificates can still be created manually outside of Gremlin Server. If ssl is enabled, a key store must be configured.

Cluster client no longer trusts all certs by default as this is an insecure configuration. Instead, if no trust store is configured, Cluster will use the default CA certs. To revert to the previous behavior and accept all certs, it must be explicitly configured.

This release introduces JKS and PKCS12 support. JKS is the legacy Java Key Store. PKCS12 has better cross-platform support and is gaining in adoption. Be aware that JKS is the default on Java 8. Java 9 and higher use PKCS12 as the default. Both Java keytool and OpenSSL tools can create, read, update PKCS12 files.

Other new features include specifying SSL protocols and cipher suites. The packaged *-secure.yaml files now restrict the protocol to TLSv1.2 by default.

PEM-based configurations are deprecated and may be removed in a future release.

See also Connecting via Java Configuration, Gremlin Server Configuration.

TINKERPOP-2022 TINKERPOP-2023

TinkerPop has provided some general methods for importing and exporting data, but more and more graph providers are producing their own bulk import/export facilities and they are more efficient and easier to use than TinkerPop’s methods. As a result, TinkerPop will now refer users to the bulk import/export features of individual graph providers and as such, has deprecated BulkLoaderVertexProgram as of this release.

As part of this change, the BulkDumperVertexProgram has been renamed to CloneVertexProgram with the former being deprecated. CloneVertexProgram is more aptly named, as it essentially copies a graph from a graph InputFormat to a graph OutputFormat.

TINKERPOP-1985

Docker images are now available on Docker Hub for Gremlin Console and Gremlin Server.

See: TINKERPOP-1897, Reference Documentation - Gremlin Console, Reference Documentation - Gremlin Server,

In Gremlin Server, it is best not to use 'TimedInterruptCustomizerProvider' because it can compete with the 'scriptEvaluationTimeout' setting and produce a different error path. Simply rely on 'scriptEvaluationTimeout' as it covers both script evaluation time and result iteration time.

See: TINKERPOP-1778

The Grateful Dead dataset has been with TinkerPop since the early days of 1.x. It has always been available as a packaged dataset that needed to be loaded through the various IO options available, while other toy graphs had the benefit of TinkerFactory to help get them bootstrapped. For 3.2.10, Grateful Dead is now more conveniently loaded via that same method as the other toy graphs with TinkerFactory.createGratefulDead().

Gremlin Javascript can now submit script, with optional bindings, using the Client class:

and also allows translation of bytecode steps into script: