Commit 85449b0
Deflake tests in GraphSONTypedCompatibilityTest, GraphSONUntypedCompatibilityTest and GraphBinaryCompatibilityTest (#3237)
We observed several tests in GraphSONTypedCompatibilityTest that exhibited flaky behavior when executed with NonDex. Specifically speaking, we can reproduce them by using the following commands.
```
mvn clean install -DskipTests -Drat.skip=true
```
* Test shouldReadWriteEdge[expect(v2)]
```bash
mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteEdge[expect(v2)]" -Drat.skip=true
```
* Test shouldReadWriteEdge[expect(v3)]
```bash
mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteEdge[expect(v3)]" -Drat.skip=true
```
* Test shouldReadWritePath[expect(v2)]
```bash
mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWritePath[expect(v2)]" -Drat.skip=true
```
* Test shouldReadWritePath[expect(v3)]
```bash
mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWritePath[expect(v3)]" -Drat.skip=true
```
* Test shouldReadWriteProperty[expect(v2)]
```bash
mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteProperty[expect(v2)]" -Drat.skip=true
```
* Test shouldReadWriteProperty[expect(v3)]
```bash
mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteProperty[expect(v3)]" -Drat.skip=true
```
* Test shouldReadWriteTraverser[expect(v2)]
```bash
mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteTraverser[expect(v2)]" -Drat.skip=true
```
* Test shouldReadWriteTraverser[expect(v3)]
```bash
mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteTraverser[expect(v3)]" -Drat.skip=true
```
* Test shouldReadWriteVertexProperty[expect(v2)]
```bash
mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteVertexProperty[expect(v2)]" -Drat.skip=true
```
* Test shouldReadWriteVertexProperty[expect(v3)]
```bash
mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteVertexProperty[expect(v3)]" -Drat.skip=true
```
* Test shouldReadWriteVertex[expect(v2)]
```bash
mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteVertex[expect(v2)]" -Drat.skip=true
```
* Test shouldReadWriteVertex[expect(v3)]
```bash
mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONTypedCompatibilityTest#shouldReadWriteVertex[expect(v3)]" -Drat.skip=true
```
And the error should be something like this:
```
[ERROR] Failures:
[ERROR] GraphSONTypedCompatibilityTest>AbstractTypedCompatibilityTest.shouldReadWriteEdge:322 expected:<e[17][7-develops->10]> but was:<e[13][1-develops->10]>
[ERROR] GraphSONTypedCompatibilityTest>AbstractTypedCompatibilityTest.shouldReadWriteEdge:322 expected:<e[17][7-develops->10]> but was:<e[13][1-develops->10]>
```
Upon our investigation, the root cause is the use of:
```
protected Map<Object, Vertex> vertices = new ConcurrentHashMap<>();
protected Map<Object, Edge> edges = new ConcurrentHashMap<>();
```
in TinkerGraph.java, which does not guarantee a deterministic order.
The simplest fix would be to replace ConcurrentHashMap with LinkedHashMap, as we did in a previous PR. We've confirmed that this change could remove the flakiness of these tests. However, we are concerned that such a change might introduce unintended side effects in the code under test. Another possible fix would be to deterministically select a fixed id, but that approach would make the test become sensitive to future implementation changes. Thus, we decided to use this sorting-based approach to deflake this test.
This fix also deflakes the entire GraphSONUntypedCompatibilityTest and GraphBinaryCompatibilityTest, which used to fail under NonDex in the 3.7-dev branch, this could be verified by:
```
mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONUntypedCompatibilityTest" -Drat.skip=true
```
```
mvn clean -pl gremlin-util edu.illinois:nondex-maven-plugin:2.2.1:nondex -Dtest="org.apache.tinkerpop.gremlin.structure.io.graphbinary.GraphBinaryCompatibilityTest" -Drat.skip=true
```
Co-authored-by: Siddhi Jhunjhunwala <siddhi2000jhun@gmail.com>1 parent f004085 commit 85449b0
1 file changed
Lines changed: 22 additions & 7 deletions
Lines changed: 22 additions & 7 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| 43 | + | |
43 | 44 | | |
44 | 45 | | |
45 | 46 | | |
| |||
115 | 116 | | |
116 | 117 | | |
117 | 118 | | |
118 | | - | |
119 | | - | |
120 | | - | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
121 | 128 | | |
122 | | - | |
123 | | - | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
124 | 137 | | |
125 | 138 | | |
126 | 139 | | |
| |||
153 | 166 | | |
154 | 167 | | |
155 | 168 | | |
156 | | - | |
| 169 | + | |
157 | 170 | | |
158 | 171 | | |
159 | 172 | | |
| |||
192 | 205 | | |
193 | 206 | | |
194 | 207 | | |
195 | | - | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
196 | 211 | | |
197 | 212 | | |
198 | 213 | | |
| |||
0 commit comments