While the concept of an open source contribution can refer to doing development work on the code base, there are many other ways outside of coding to contribute to Apache TinkerPop. Participating on the various mailing lists, offering ideas, reporting bugs, writing documentation are all welcome contributions to the project that help improve the TinkerPop. This section of the document is designed to help provide some structure for potential contributors and to give them ideas for how they could get started becoming more involved in the TinkerPop community.

Code changes can take a fair bit of effort from both the contributor doing the work, as well as the people who will be reviewing and testing that work. It is beneficial to all involved that some basic considerations be made by the contributor to ensure the best use is made of everyone’s time.

Before proceeding, contributors should evaluate if the proposed change is likely to be relevant, new and actionable:

Before considering how to contribute code, it is useful to understand how code is reviewed, and why changes may be rejected. Simply put, changes that have many or large positives, and few negative effects or risks, are much more likely to be merged, and merged quickly. Risky and less valuable changes are very unlikely to be merged, and may be rejected outright rather than receive iterations of review.

Positives

Negatives, Risks

Generally, TinkerPop uses JIRA to track logical issues, including bugs and improvements, and uses GitHub pull requests to manage the review and merge of specific code changes. That is, JIRA issues are used to describe what should be fixed or changed, and high-level approaches, and pull requests describe how to implement that change in the project’s source code.

As a first step to making a contribution, consider JIRA:

Next, make changes and prepare a pull request:

Once the pull request has been placed it will go into review:

On successful review, the pull request will be merged to the main repository and the JIRA issue will be closed.

TinkerPop version numbers follow a format of MAJOR.MINOR.PATCH, where:

The number for MAJOR rarely changes and when it does, it typically represents a complete re-write of the software and thus massive changes can be expected. As a result of the definition for MAJOR, we tend to focus on the other two numbers when considering where changes will land.

The driving question for determining if a change is suitable for MINOR or PATCH versions is whether or not the change introduces some form of incompatibility. As the focus for making the determination is "breaking change", we therefore allow the option for new features to land in PATCH versions. Taking this approach, lengthens the life span of MINOR versions considerably which is helpful to graph providers looking for stable long-lived support for a particular release line.

With all of the above rules in mind, there are some subtle considerations that have come into play in the past when trying to determine where a change should land:

In short, the dev mailing list is open to discussion on where a change will land. The discussion just needs to have good reasoning and use the general principles above as a framework for coming to a consensus.

At a minimum, development of TinkerPop requires Java 1.8.0_40+ and Maven 3.2.5+. Maven is used as the common build system, which even controls the builds of non-JVM GLVs such as gremlin-python. Java and Maven are described as a "minimum" for a development environment, because they will only build JVM portions of TinkerPop and many integration tests will not fire with this simple setup. It is possible to get a clean and successful build with this minimum, but it will not be possible to build non-JVM aspects of the project and those will go untested.

To gain the ability to execute all aspects of the TinkerPop build system, other environmental configurations must be established. Those prerequisites are defined in the following subsections.

The following commands are a mix of Maven flags and shell scripts that handle different build operations

TinkerPop provides a shell script, that can start several build tasks within a Docker container. The required Docker images will be built automatically if they don’t exist yet. Thus the first invocation of the Docker script is expected to take some time.

The script can be found under PROJECT_HOME/docker/build.sh. The following tasks are currently supported:

A list of command line options is provided by docker/build.sh --help. The container will install, configure and start all required dependencies, such as Hadoop.

Options can be passed to Docker by setting the TINKERPOP_DOCKER_OPTS environment variable. A speed boost can be gained at the expense of memory by using tmpfs and the special directory /usr/src/tinkermem.

A custom maven settings.xml can be supplied, for example, to point to a local proxy. Copy the settings.xml to the PROJECT_HOME/ directory. The Docker script will detect and copy it to the running container.

If the container is used to generate the user docs, it will start a web server and show the URL that is used to host the HTML docs.

After finishing all tasks, the script will immediately destroy the container.

Docker can also be helpful to developers who do not want to run tests from a Maven environment, which may be a bit opaque when dealing with test failures and largely unhelpful for debugging. This situation is typically case for developers doing work on Gremlin Language Variants (e.g. Python). To help alleviate this problem, developers can start a standalone Gremlin Server with its standard test configuration that is used in the standard Maven build.

Generally speaking, most developers will want to test their code against the latest build of Gremlin Server in the TinkerPop repository. To do that, first be sure to build a Docker image of the current code:

Next, generate the a Docker image for Gremlin Server with:

Finally, start the server with:

Starting Gremlin Server this way makes it possible to run Gremlin Language Variant tests without Maven (for example, directly from a debugger) which should greatly reduce development friction for these environments.

It is also possible to specify the exact version of Gremlin Server to run with the test configuration. This version should be an existing Docker image version and must be an explicit version that maps to an actual TinkerPop artifact:

To be a bit more clear, the version can not be a Docker tag like "latest" because there is no such TinkerPop artifact that has been published with that version number.

This section refers specifically to setup within Intellij. TinkerPop has a module called gremlin-shaded which contains shaded dependencies for some libraries that are widely used and tend to introduce conflicts. To ensure that Intellij properly interprets this module after importing the Maven pom.xml perform the following steps:

Note that it may be necessary to re-execute these steps if the gremlin-shaded pom.xml is ever updated.

Developers working on the neo4j-gremlin module should enabled the include-neo4j Maven profile in Intellij. This will ensure that tests will properly execute within the IDE.

If Intellij complains about "duplicate sources" for the Groovy files when attempting to compile/run tests, then install the GMavenPlus Intellij plugin.

The gremlin-core module uses a Java annotation processor to help support DSLs. To support this capability be sure that:

Once the Apache TinkerPop PMC has sent an invitation to a contributor to become a new committer and that contributor has accepted that invitation and provided their iCLA to Apache, then there are some administrative steps that the new committer can expect to go through to complete the process so that they have access to Apache resources like the Git repository. While the information for completing the process can be found in a multitude of places the following listing provides a summary of what to do next:

TinkerPop has a user mailing list and a dev mailing list. As a committer, it is a good idea to join both.

TinkerPop also has a Slack channel for more real-time communication about the project among contributors, though it must be kept in mind that project discussion and decisions must occur on the dev mailing list mentioned above.

Occasionally, online meetings via video conference are held. These meetings are schedule via the dev mailing list about a week before they are to occur to find a day and time that is available for those interested in attending. On the day of the meeting, the meeting organizer will create a Google Hangout (or similar video conferencing link). At that point, all who are interested can attend. Meeting minutes should be taken and added to the Meetings section of this document using the pattern already established.

There is a two-pronged approach to maintaining the change log and preparing the release notes.

Given the dependence on the JIRA report for generating additions to the CHANGELOG.asciidoc, which uses the title of the issue as the line presented in the release note report, titles should be edited prior to release to be useful in that context. In other words, an issue title should be understandable as a change in the fewest words possible while still conveying the gist of the change.

Changes that break the public APIs should be marked with a "breaking" label and should be distinguished from other changes in the release notes.

TinkerPop has several release branches:

Changes to earlier branches should merge forward toward master (e.g. 3.4-dev should merge to master). Please read more about this process in the Pull Requests section. Note that 4.0-dev is rebased on master and currently behaves as a fresh repository as all 3.x content was removed.

Other branches may be created for collaborating on features or for RFC’s that other developers may want to inspect. It is suggested that the JIRA issue ID be used as the prefix, since that triggers certain automation, and it provides a way to account for the branch lifecycle, i.e. "Who’s branch is this, and can I delete it?"

For branches that are NOT associated with JIRA issues, developers should utilize their Apache ID as a branch name prefix. This provides a unique namespace, and also a way to account for the branch lifecycle.

Developers should remove their own branches when they are no longer needed.

Tags are used for milestones, release candidates, and approved releases. Please refrain from creating arbitrary tags, as they produce permanent clutter.

TinkerPop uses Apache JIRA as its issue tracker. JIRA is a very robust piece of software with many options and configurations. To simplify usage and ensure consistency across issues, the following conventions should be adhered to:

Contributors should examine the current code base to determine what the code style patterns are and should match their style to what is already present. Of specific note however, TinkerPop does not use "import wildcards" - IDEs should be adjusted accordingly to not auto-wildcard the imports.

TinkerPop uses Travis for CI services. The build status can be found here. Note that the CI process does not run all possible tests (e.g. Neo4j-related tests) as a full execution would likely exceed the allowable times for builds on these servers. Instead Travis runs a basic cross-section of tests selected to provide a reasonably high degree of confidence in the branch built.

When possible, committers should avoid direct "breaking" change (e.g. removing a method from a class) and favor deprecation. Deprecation should come with sufficient documentation and notice especially when the change involves public APIs that might be utilized by users or implemented by providers:

The JIRA issues that track removal of deprecated methods should be periodically evaluated to determine if it is prudent to schedule them into a release.

TinkerPop has a wide variety of test types that help validate its internal code as well as external provider code. There are "unit tests" and "integration tests". Unit tests execute on standard runs of mvn clean install. These tests tend to run quickly and provide a reasonable level of coverage and confidence in the code base. Integration tests are disabled by default and must be explicitly turned on with a special build property by adding -DskipIntegrationTests=false to the mvn execution. Integration tests run slower and may require external components to be running when they are executed. They are "marked" as separate from unit tests by inclusion of the suffix "IntegrateTest".

Here are some other points to consider when developing tests:

Benchmarks are a useful tool to track performance between TinkerPop versions and also as tools to aid development decision making. TinkerPop uses OpenJDK JMH for benchmark development. The JMH framework provides tools for writing robust benchmarking code that avoid many of the pitfalls inherent in benchmarking JIT compiled code on the JVM. Example JMH benchmarks can be found here.

TinkerPop benchmarks live in the gremlin-benchmark module and can either be run from within your IDE or as a standalone uber-jar. The uber-jar is the JMH recommended approach and also makes it easy to distribute artifacts to various environments to gather benchmarking numbers. Having said that, in most cases it should be sufficient to run it from within the IDE.

Benchmarks will not run by default because they are time consuming. To enable benchmarks during the test phase do -DskipBenchmarks=false. To change the number of warmup iterations, measurement iterations, and forks you can do mvn clean test -DskipBenchmarks=false -DdefaultForks=5 -DmeasureIterations=20 -DwarmupIterations=20. Benchmark results will be output by default to the benchmarks directory in JSON format.

Benchmarks may also be run from the command line using the JMH runner. Build the uber-jar and simply run java -jar gremlin-benchmark-TP-VERSION.jar. To see a list of JMH runner options, add the -h flag.

The JUnit/JMH integration was inspired by the Netty projects microbenchmarking suite. Please refer to the Netty docs for more details. Presently there are 3 abstract benchmark classes that may be used as building blocks for your benchmarks; AbstractBenchmarkBase, AbstractGraphBenchmark, and AbstractGraphMutateBenchmark.

Code modifications must go through a review-then-commit (RTC) process before being merged into a release branch. All committers should follow the pattern below, where "you" refers to the committer wanting to put code into a release branch.

For those performing reviews as part of this process it is worth noting that the notion of "review" is fairly wide for our purposes. TinkerPop has grown into a large and complex code base and very few people (if anyone) is knowledgeable on all of its modules. Detailed code reviews might often be difficult or impossible as a result.

To be clear, a "review" need not be specifically about the exact nature of the code. It is perfectly reasonable to review (and VOTE) in the following fashion:

Non-committers are welcome to review and VOTE as well and while their VOTEs are not binding, they will be taken as seriously as non-binding VOTEs on releases. Reviewing and VOTEing on pull requests as a non-committer is a great way to contribute to the TinkerPop community and get a good pulse on the changes that are upcoming to the framework.

The following exceptions to the RTC (review-then-commit) model presented above are itemized below. It is up to the committer to self-regulate as the itemization below is not complete and only hints at the types of commits that do not require a review.

When the committer chooses CTR, it is considered good form to include something in the commit message that explains that CTR was invoked and the reason for doing so. For example, "Invoking CTR as this change encompasses minor adjustments to text formatting." CTR based commits will still require manual merging through all release branches. Merges should occur in reverse order, starting with the latest release version first (e.g. if the fix is going to 3.3.x then the change should be merged in the following order master, 3.4-dev, 3.3-dev).

There are many dependencies on other open source libraries in TinkerPop modules. When adding dependencies or altering the version of a dependency, developers must consider the implications that may apply to the TinkerPop LICENSE and NOTICE files. There are two implications to consider:

Understanding these implications is important for insuring that TinkerPop stays compliant with the Apache 2 license that it releases under.

Regarding the first item, refer to the Apache Legal for a list of approved licenses that are compatible with the Apache 2 license.

The second item requires a bit more effort to follow. The Apache website offers a how-to guide on the approach to maintaining appropriate LICENSE and NOTICE files, but this guide is designed to offer some more specific guidance as it pertains to TinkerPop and its distribution.

To get started, TinkerPop has both "source" and "binary" LICENSE/NOTICE files:

The documentation for TinkerPop is stored in the git repository in docs/src/ and are then split into several subdirectories, each representing a "book" (or its own publishable body of work). If a new AsciiDoc file is added to a book, then it should also be included in the index.asciidoc file for that book, otherwise the preprocessor will ignore it. Likewise, if a whole new book (subdirectory) is added, it must include an index.asciidoc file to be recognized by the AsciiDoc preprocessor.

Adding a book also requires a change to the root pom.xml file. Find the "asciidoc" Maven profile and add a new <execution> to the asciidoctor-maven-plugin configuration. For each book in docs/src/, there should be a related <execution> that generates the HTML from the AsciiDoc. Follows the patterns already established by the existing <execution> entries, paying special attention to the pathing of the '<sourceDirectory>', <outputDirectory> and <imagesdir>. Note that the <outputDirectory> represents where the book will exist when uploaded to the server and should preserve the directory structure in git as referenced in <sourceDirectory>.

Please see the Building and Testing section for more information on how to generate the documentation.

The content for the TinkerPop home page and related pages that make up the web site at tinkerpop.apache.org is stored in the git repository under /docs/site. In this way, it becomes easier for the community to provide content presented there, because the content can be accepted via the standard workflow of a pull request. To generate the site for local viewing, run bin/generate-home.sh, which will build the site in target/site/. PMC members can officially publish the site with bin/publish-home.sh <username>.

"Publishing" does not publish documentation (e.g. reference docs, javadocs, etc) and only publishes what is generated from the content in /docs/site. Publishing the site can be performed out of band with the release cycle and is no way tied to a version. The master branch should always be considered the "current" web site and publishing should only happen from that branch.

TinkerPop uses SLF4j for logging and typically leans back on Log4j as the implementation. Configuring log outputs for debugging purposes within tests can be altered by editing the log4j-test.properties file in each module’s test resources. That file gets copied to the target/test-classes on build and surefire and failsafe plugins in maven are then configured to point at that area of the file system for those configuration files. The properties files can be edited to fine tune control of the log output, but generally speaking the current configuration is likely best for everyone’s general purposes, so if changes are made please revert them prior to commit.

The IO Documentation provides more details into GraphML, GraphSON and Gryo with a special focus on the needs of developers who are working directly with these formats. GraphSON gets the greatest focus here as it is used as the primary IO format for GLVs. This documentation is largely generated from the gremlin-io-test module found under gremlin-tools. The gremlin-io-test module also includes a testing framework which validates that formats don’t break between TinkerPop versions. Unfortunately, this module requires some maintenance to ensure that the documentation and tests both stay updated.

The gremlin-io-test module contains a set of files in the test resources that are statically bound to the version in which they were generated. Older versions should never be modified. The only time changes to these resources should be accepted should be for the current SNAPSHOT version. The test resources are generated from the Model class which contains the objects that will undergo serialization for purpose of testing. Note that these same objects in the Model are also used to generate documentation.

To generate these test resources and documentation snippets based on the Model, use this Maven command:

This command will generate two directories in the /target output directory of gremlin-io-test: test-case-data and dev-docs. The contents of test-case-data represents the serialized Model objects that can be copied to the test resources and the contents of the dev-docs contains asciidoc snippets that can be copied to the IO documentation.

Generating data files in the fashion mentioned above with Maven is only good for versions of TinkerPop on the 3.3.x line because the gremlin-io-test module did not exist in 3.2.x. Of course, compatibility is still tested back to those older versions. To generate test data from 3.2.x, there are Groovy scripts in the comments of the gryo.asciidoc and graphson.asciidoc files that can copy/pasted to the Gremlin Console. They will generate the batch of test files needed for gremlin-io-test.

When does this command need to be executed?

The second case, does require some additional discussion. When a new version is added the following classes will need to be updated in the following ways:

GraphBinaryCompatibility - Include new GraphBinary 1.0 enums for the current SNAPSHOT.

GryoCompatibility - Include new Gryo 1.0 and 3.0 enums for the current SNAPSHOT.

GraphSONCompatibility - Include new GraphSON enums for each of the various GraphSON configurations and versions.

GraphBinaryCompatibilityTest - Add the newly included GraphBinaryCompatibility enums to the test parameters being careful to match the appropriate "mapper" to the right version.

GryoCompatibilityTest - Add the newly included GryoCompatibility enums to the test parameters being careful to match the appropriate "mapper" to the right version.

GraphSONUntypedCompatibilityTest - Add the newly included GraphSON 1.0 and 2.0 "untyped" enums to the test parameters being careful to match the appropriate "mapper" to the right version.

GraphSONTypedCompatibilityTest - Add the newly included GraphSON 3.0 and 2.0 "typed" enums to the test parameters being careful to match the appropriate "mapper" to the right version.

At this point, all of the IO tests are rigged up properly and assuming the test resources are available a standard mvn clean install should execute the compatibility tests and validate that everything is working as expected and that there are no breaks in serialization processes.

A "development version" or "snapshot" (in Java parlance) is not an "official" release. Artifacts produced for a snapshot are solely for the convenience of providers and other developers who want to use the latest releases of TinkerPop. These releases do not require a VOTE and do not require a "release manager". Any PMC member can deploy them. It is important to note that these releases cannot be promoted outside of the developer mailing list and should not be recommended for use beyond the purpose of evaluation and testing.

For JVM-based artifacts, simply use the following command:

and artifacts will be pushed to the Apache Snapshot Repository. GLV development artifacts must be generated and deployed separately with additional commands:

Python, .NET and Javascript do not use the snapshot model the JVM does, however, the build is smart in that it will dynamically generate a development version number for the GLV artifacts when "-SNAPSHOT" is in the pom.xml.

If you wish to verify that mvn deploy works before doing it officially then:

If this is your first time as release manager, you will need to setup keys for signing purposes per the Apache release process. Generally speaking, this will mean that you will need to generate a key-pair and then upload your public key to a public keyserver.

For a general overview of key basics, refer to this. For detailed step-by-step instructions, please follow the instructions here.

After completing the key-pair setup instructions, be sure to add yourself to the PGP signature section of bin/validate-distribution.sh.

The "pre-flight check" is a list of things performed by the release manager during the weeks leading up to a scheduled day to release. These checks will help to ensure that that release day goes smoothly by identifying problems up early and communicating with other members of the community.

A release candidate is an unofficial release that is represented by a tagged version in the Git repository. It is offered in cases where there is significant change in a particular version and the potential for upgrades and problems might be high. Release candidates do not require a vote thread. Lazy consensus is acceptable for confirming their deployment.

Note that release candidates need not require the release of all artifacts. For example, if the risk is with one particular GLV, then a release candidate can be prepared of just that particular artifact. In those cases, a tag on the commit that represents the release candidate is sufficient and does not necessarily require that the versions be bumped to reflect the actual "-rc" version. In other words, if preparing a release candidate for .NET, then there is no need to go through the processing of bumping versions in Java artifacts and all GLVs. Nor is it necessary to alter the version of .NET to include the release candidate versioning. It can simply be altered locally by the release manager and deployed.

This section describes the process that process that prepares a release for VOTE by the community. If there are multiple releases (as there usually are) being prepared, it may be best for downstream releases to wait for upstream releases to complete their process to assure that no last minute commits are required to get the upstream release completed. It is up to the discretion of the release managers to decide how they wish to proceed with respect to preparing releases in parallel or in a more serial fashion.

A number of administration tasks should be taken care of after release is public. Some of these items can be performed during the VOTE period at the release manager’s discretion, though it may be wise to wait until a successful VOTE is eminent before reopening development. When there are multiple release managers, it’s best to coordinate these tasks as one individual may simply just handle them all.

When a candidate is identified by a PMC member as someone who might be a good official committer to TinkerPop, the PMC member should open a DISCUSS thread on the private TinkerPop mailing list. The thread should provide some background and reasoning for why that member believes the candidate would be a good committer. Given enough time for feedback on the candidate and presuming there is still positive interest in doing so, a VOTE thread on the private TinkerPop mailing list is started to get the official stance. As per usual, the VOTE will be made open for no less than 72 hours.

If the VOTE closes with a successful positive vote to make the candidate a committer, then send the following email to the candidate and copy the private TinkerPop mailing list:

Assuming the iCLA included items from step 3, the secretary will handle account creation, otherwise it will be up to the PMC chair to handle such things. Once the account is established the PMC chair will then need to:

Finally, new committers should be sent an email that covers some of the administrative elements of their new role. Generally speaking the email can largely just point them to the Initial Setup section of this documentation.

The process for bringing on new PMC members is not so different from the one for new committers. The process begins with a DISCUSS thread to the private mailing list for building consensus followed by a VOTE thread to confirm. Presuming the new PMC member is a committer already (which is mostly likely for TinkerPop), there should be no need for any additional paperwork. On successful vote however, a NOTICE should be sent to the board@apache.org (copying private@tinkerpop.apache.org):

If the candidate does not (yet) have an Apache account, then please note that fact in the notification email. Provide the board 72 hours to respond with objection before extending the invite to the new PMC member. Once that time has elapsed, the invite may be sent to the new PMC member. The following template may be used for the invitation (be sure to copy private@ when sending the email):

If they accept, then the PMC chair should refer to the How to Add a New PMC Member section in the Apache docs for how to complete the process. Use Whimsy to update PMC membership.

The PMC Chair is responsible for submitting a report to the board on a quarterly basis. TinkerPop reports on the following months: January, April, July, October. The report has the following format:

A draft of the report should be sent to the TinkerPop developer mailing list for review at least three days prior to submitting to the board. The final report should be sent in plain-text format to board@apache.org and should be committed to the appropriate meeting agenda in SVN at:

Contributions can come in many forms and extend well beyond code contributions. TinkerPop strives to ensure that contributions are honestly and accurately recognized for the benefit of the individuals who help make TinkerPop possible.

One of the ways in which this recognition is supported is by way of the Contributor Listing on the TinkerPop home page, which list those who have contributed to the project in some way. It lists both committers and PMC members and provides a short "bio" (i.e. a description of their contributions) as well as an indication of whether they are currently participating in the project or not, i.e. active or inactive. The bios have a slightly different form depending on that status:

Committers and PMC members can keep this information up to date themselves as they have direct access to the Git repository. Reminders to update this information will be sent to the dev list on each release as part of code freeze week. Those who have not shown an update to their bio in git in the past year or have otherwise not replied to the dev list email to indicate their status in that time will be assumed "inactive" for the purpose of the listing.

The following is a template for the "Bio Update" email:

The meeting was scheduled for 12:00pm EST, started at 12:05pm when sufficient attendance to constitute a quorum was reached and was held via Google Hangout hosted by Marko Rodriguez. Meeting adjourned at 2:35pm EST.

Committers/PPMC Members

Guests

None

Minutes

The following tickets were assigned to 3.1.1-incubating:

The following items were identified for 3.2.0-incubating:

The following issues were simply closed during review - the reasons for closing can be found in the comments of the issues themselves:

The meeting was scheduled for 1:00pm EST, started on time and was held via Google Hangout hosted by Stephen Mallette. Meeting adjourned at 1:45pm EST.

Committers/PPMC Members

Guests

None

Minutes

Upon review of the tickets in JIRA assigned to 3.1.0-incubating, the following were removed from the 3.1.0-incubating roadmap: