"forEachChild" method provides different results for Cache and Gtm providers

[killmenot]

Hi,

The following test falls when 'gtm' provider is used for running integration tests
https://github.com/robtweed/ewd-document-store/blob/master/spec/integration/ewd-document-store.spec.js#L295

It seems that works fine for InterSystem Caché (have no chance to run it without cache.node) but fails when Gtm used.

The problem is here:
(1) https://github.com/robtweed/ewd-document-store/blob/master/lib/proto/forEach.js#L83
(2) https://github.com/robtweed/ewd-document-store/blob/master/lib/proto/forEach.js#L87

For Gtm (2) returns the following result

node before =  '{"global":"temp","subscripts":["foo","Davisþ"]}'
end / result = undefined

However, if I comment out (1), (2) returns the following:

node before =  '{"global":"temp","subscripts":["foo","Davis"]}'
end/result =  'Douglas'

and seems works.

Also, please note that FIELD_MARK also used here
https://github.com/robtweed/ewd-document-store/blob/master/lib/proto/forEach.js#L75

@robtweed Do you have any ideas?

[robtweed]

What data are you using for this test? Where is it defined so I can try it out manually by myself?

[killmenot]

@robtweed data object for forEachChild tests is defined here:
https://github.com/robtweed/ewd-document-store/blob/master/spec/integration/ewd-document-store.spec.js#L179

[robtweed]

OK it appears to be a problem introduced into YottaDB release 122. I'll figure out a solution

[killmenot]

Also, another issue with "gtm" (maybe related) for this test case:
https://github.com/robtweed/ewd-document-store/blob/master/spec/integration/ewd-document-store.spec.js#L273

Some log output:
gtm (without FIELD_MARK)

%YDB-I-MUFILRNDWNSUC, File /root/.yottadb/r1.22_x86_64/g/yottadb.dat successfully rundown
line 87 (lib/proto/forEach.js) node: {"global":"temp","subscripts":["foo","D"]}
line 87 (lib/proto/forEach.js) res: {"ok":true,"global":"temp","subscripts":["foo","Davies"],"result":"Davies"}
*** end = Davies; direction = forwards
start node: {"global":"temp","subscripts":["foo",""]}
result = Barton
result = Briggs
result = Davies

gtm (with FIELD_MARK)

%YDB-I-MUFILRNDWNSUC, File /root/.yottadb/r1.22_x86_64/g/yottadb.dat successfully rundown
line 87 (lib/proto/forEach.js) node: {"global":"temp","subscripts":["foo","Dþ"]}
line 87 (lib/proto/forEach.js) res: {"ok":false,"errorCode":150373642,"errorMessage":"order+13^v4wNode,%YDB-E-RPARENMISSING, Right parenthesis expected"}
*** end = undefined; direction = forwards
start node: {"global":"temp","subscripts":["foo",""]}
result = Barton

For run GTM, I use the killmenot/qewd-server:tests docker (https://github.com/killmenot/docker-qewd-server) that I created by clone rtweed/qewd-server in the past

I guess the problem is not in FIELD_MARK because this functionality works for me correctly with https://github.com/killmenot/ewd-memory-globals (however, one patch is needed to be pull but it's not important for this case) but in the error message:

{"ok":false,"errorCode":150373642,"errorMessage":"order+13^v4wNode,%YDB-E-RPARENMISSING, Right parenthesis expected"}

Do you have any ideas how to fix it?

[killmenot]

ydb_env here: https://pastebin.com/aVY31kc8

[killmenot]

Regarding the patch I noted above, once I have gtm (or cache but I need cache.node) working correctly to run integration tests I will be able to pull and discuss it

[robtweed]

The field mark is required in order to push the so-called collating sequence forward so I get the correct seed value. The problem turns out to be that either YottaDB or NodeM is no longer able to handle a string containing a character greater than ASCII 127. I need to find out why this is.

In the meantime, change line 37 of forEach.js to this:

    const FIELD_MARK = String.fromCharCode(127);

and it will work

Rob

[killmenot]

@robtweed yeah, it fixes the issue but requires me to update 2 unit tests.

In the same time 2 integration tests are still faling

(1) https://github.com/robtweed/ewd-document-store/blob/master/spec/integration/ewd-document-store.spec.js#L160
(2) https://github.com/robtweed/ewd-document-store/blob/master/spec/integration/ewd-document-store.spec.js#L380

The problem is that expected data expect number as a string but it returns as number

[robtweed]

Yes you'll need to make a change to the way you open GT.M to fix that, due to a change in NodeM

In your test script, when you open YottaDB / GT.M, you need to do it this way:

  var ok = db.open({
    mode: 'strict'
  });

This tells NodeM to behave exactly like cache.node, and not perform any automatic type conversions

[robtweed]

You'll have to wait while I find out the solution to the ASCII 254 issue. Don't change your tests for this just yet

[killmenot]

@robtweed sure, np.

Actually, there are only 2 changes:

1. support different db providers for integration tests
2. fix for memory db provider in forEach

but both them can wait ofc

[robtweed]

OK, once again it's a NodeM enhancement which, in your tests, you need to fix by changing the open() parameters:

var ok = db.open({
mode: 'strict',
charset: 'm'
});

Revert to the original version of forEach.js and now your tests should work.

I'll be making this the default way of opening YottaDB/GT.M in the next build of QEWD

[robtweed]

I've updated the ewd-qoper8-gtm module (on Github and NPM) to address this issue

[killmenot]

@robtweed updated tests for ewd-qoper8-gtm module robtweed/ewd-qoper8-gtm#3

[killmenot]

@robtweed Also, please review this one: #20