Do better showing differences between x,y in test() failure

[MichaelChirico]

Closes #7834. NB: merge target is not master currently

Output from test(1, c(1:10, 11:20, 30:21, 31:1000), c(1:10, 20:11, 30:21, 1000:31))

Test 1 ran without errors but failed check that x equals y:
> x = c(1:10, 11:20, 30:21, 31:1000) 
First 6 different of 980 (1000 total, type 'integer'): 
x[11] x[12] x[13] x[14] x[15] x[16] 
   11    12    13    14    15    16 
> y = c(1:10, 20:11, 30:21, 1000:31) 
First 6 different of 980 (1000 total, type 'integer'): 
y[11] y[12] y[13] y[14] y[15] y[16] 
   20    19    18    17    16    15 
Mean relative difference: 0.9406426

It's a bit visually dense; we might prefer this alternative using a matrix representation:

Test 1 ran without errors but failed check that x equals y:
> x = c(1:10, 11:20, 30:21, 31:1000) 
First 6 different of 980 (1000 total, type 'integer'): 
                        
index: 11 12 13 14 15 16
value: 11 12 13 14 15 16
> y = c(1:10, 20:11, 30:21, 1000:31) 
First 6 different of 980 (1000 total, type 'integer'): 
                        
index: 11 12 13 14 15 16
value: 20 19 18 17 16 15
Mean relative difference: 0.9406426

See also other test cases:

test(1, NA_character_, 'abc')
# Test 1 ran without errors but failed check that x equals y:
# > x = NA_character_ 
# First 1 different of 1 (1 total, type 'character'): 
# x[1] 
#   NA 
# > y = "abc" 
# First 1 different of 1 (1 total, type 'character'): 
#  y[1] 
# "abc" 
# 'is.NA' value mismatch: 0 in current 1 in target

x = c(1, 2, 3, NA, 4, NA)
y = c(NA, 2, 4, NA, 4, 5)
test(1, x, y)
# Test 1 ran without errors but failed check that x equals y:
# > x = x 
# First 3 different of 3 (6 total, type 'double'): 
# x[1] x[3] x[6] 
#    1    3   NA 
# > y = y 
# First 3 different of 3 (6 total, type 'double'): 
# y[1] y[3] y[6] 
#   NA    4    5 
# 'is.NA' value mismatch: 2 in current 2 in target

A case that's still not completely satisfactory -- it gives the impression of showing x[11] == y[11] -- but I'm willing to ignore for now:

test(1, c(1:10, 1.0000001), c(1:10, 1))
# Test 1 ran without errors but failed check that x equals y:
# > x = c(1:10, 1.0000001) 
# First 1 different of 1 (11 total, type 'double'): 
# x[11] 
#     1 
# > y = c(1:10, 1) 
# First 1 different of 1 (11 total, type 'double'): 
# y[11] 
#     1 
# Mean relative difference: 9.999999e-08

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.02%. Comparing base (895067d) to head (746918d).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #7836   +/-   ##
=======================================
  Coverage   99.02%   99.02%           
=======================================
  Files          88       88           
  Lines       17266    17266           
=======================================
  Hits        17097    17097           
  Misses        169      169           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

No obvious timing issues in HEAD=test-err-show-diff

Generated via commit 746918d

Download link for the artifact containing the test results: ↓ atime-results.zip

Task	Duration
R setup and installing dependencies	2 minutes and 54 seconds
Installing different package versions	1 minutes and 12 seconds
Running and plotting the test cases	7 minutes and 42 seconds

[ben-schwen]

Also not 100% satisfied with the index representation. Maybe something like

First 6 of 980 differing elements (1000 total):
 index  x  y
    11 11 20
    12 12 19
    13 13 18
    14 14 17
    15 15 16
    16 16 15
Types: x integer, y integer

[MichaelChirico]

Maybe something like

The problem with this one is it requires a bit more of a refactor since the current approach is "show x difference then show y difference" as opposed to "show a holistic comparison of x,y".

[ben-schwen]

The problem with this one is it requires a bit more of a refactor since the current approach is "show x difference then show y difference" as opposed to "show a holistic comparison of x,y".

True. Maybe we put that as follow up. The index notation is already much better TY!