error run NormaliseNVPX.py

[atanasj]

Hi,

I seem to be getting an error when trying to to convert form .nvpx to .norm. See below:

| => python2 NormaliseNVPX.py /Users/atanas/OneDrive\ -\ University\ of\ Wollongong/PhD/PR_qual/unzip/2019_05_29-qual-conv-error-test\ copy.nvpx /Users/atanas/OneDrive\ -\ University\ of\ Wollongong/PhD/PR_qual/unzip/sqlite:///filename.db
Started database server on port 62267
Traceback (most recent call last):
  File "NormaliseNVPX.py", line 149, in <module>
    NVivo.Normalise(args)
  File "/Users/atanas/OneDrive - University of Wollongong/PhD/PR_qual/unzip/nvivotools/NVivo.py", line 206, in Normalise
    nvivoAnnotation    = Table('Annotation',    nvivomd, autoload=True)
  File "<string>", line 2, in __new__
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/deprecations.py", line 130, in warned
    return fn(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 496, in __new__
    metadata._remove_table(name, schema)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 491, in __new__
    table._init(name, metadata, *args, **kw)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 585, in _init
    resolve_fks=resolve_fks,
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 626, in _autoload
    _extend_on=_extend_on,
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2146, in run_callable
    with self._contextual_connect() as conn:
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2226, in _contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2266, in _wrap_pool_connect
    e, dialect, self
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1536, in _handle_dbapi_exception_noconnection
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 363, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 760, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
    self._dec_overflow()
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
    self.__connect(first_connect_check=True)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 639, in __connect
    connection = pool._invoke_creator(self)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 451, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 522, in connect
    return Connection(args, kwargs)
  File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 538, in __init__
    parent = Connection.cls_parent = Root("PYTHON")
  File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 464, in __init__
    'libdbcapi_r.dylib')
  File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 456, in load_library
    raise InterfaceError("Could not load dbcapi.  Tried: " + ','.join(map(str, names)))
sqlalchemy.exc.InterfaceError: (sqlanydb.InterfaceError) (u'Could not load dbcapi.  Tried: None,dbcapi.dll,libdbcapi_r.so,libdbcapi_r.dylib', 0)
(Background on this error at: http://sqlalche.me/e/rvf5)

Is there something I'm doing wrong? My system info is:

ProductName:	Mac OS X
ProductVersion:	10.14.5
BuildVersion:	18F132

My python2 version is Python 2.7.16

List of python2 libraries

| => pip2 list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Package                            Version
---------------------------------- --------
appnope                            0.1.0
argh                               0.26.2
autopep8                           1.4.4
backports-abc                      0.5
backports.functools-lru-cache      1.5
backports.shutil-get-terminal-size 1.0.0
certifi                            2019.3.9
chardet                            3.0.4
configparser                       3.7.4
contextlib2                        0.5.5
dateutils                          0.6.6
decorator                          4.4.0
docopt                             0.6.2
enum34                             1.1.6
fpdf                               1.7.2
future                             0.17.1
futures                            3.2.0
idna                               2.8
importlib-metadata                 0.15
ipykernel                          4.10.0
ipython                            5.8.0
ipython-genutils                   0.2.0
jedi                               0.13.3
jupyter-client                     5.2.4
jupyter-core                       4.4.0
List                               1.3.0
lxml                               4.3.3
mccabe                             0.6.1
numpy                              1.16.4
opencv-python                      4.1.0.25
pandoc-fignos                      1.4.2
pandoc-tablenos                    1.4.2
pandoc-xnos                        1.2.0
pandocfilters                      1.4.2
parso                              0.4.0
pathlib2                           2.3.3
pathtools                          0.1.2
pdfminer                           20140328
pexpect                            4.7.0
pickleshare                        0.7.5
Pillow                             6.0.0
pip                                19.1.1
pipupgrade                         1.5.0
pluggy                             0.12.0
prompt-toolkit                     2.0.9
psutil                             5.6.2
ptyprocess                         0.6.0
pycodestyle                        2.5.0
pydocstyle                         3.0.0
pyflakes                           2.1.1
Pygments                           2.4.2
PyMuPDF                            1.14.16
python-dateutil                    2.8.0
python-jsonrpc-server              0.1.2
python-language-server             0.26.1
pytz                               2019.1
PyYAML                             5.1
pyzmq                              18.0.1
requests                           2.22.0
rope                               0.14.0
scandir                            1.10.0
scipy                              1.2.1
select-backport                    0.2
setuptools                         41.0.1
simplegeneric                      0.8.1
singledispatch                     3.4.0.3
six                                1.12.0
snowballstemmer                    1.2.1
SQLAlchemy                         1.3.4
sqlalchemy-sqlany                  1.0.3
sqlanydb                           1.0.10
tornado                            5.1.1
traitlets                          4.3.2
urllib3                            1.25.3
watchdog                           0.9.0
wcwidth                            0.1.7
wheel                              0.33.4
yapf                               0.27.0
zipp                               0.5.1

I'm not sure what of the above is relevant, but just though I'd include everything. Let me know if you need more info. Cheers.

[jschultz]

You need to start the scripts via the shell scripts NormaliseNVPX.sh etc. These do various things including setting up the SQL Anywhere database server, which is the reason why it's falling over when it tries to connect to the server.

[atanasj]

I cannot seem to find these scripts... where are they? When clicking on the hyperlinks I get this error:

[jschultz]

Oh sorry, I incorporated that code into the Python scripts. But I'm not clear which hyperlinks you are referring to in your last message.

Do you get any output when you run the script, or is that error the first thing you see?

Have you installed SQL Anywhere yourself or are you relying on the version packaged with NVivo?

[atanasj]

I only get the error. I just installed SQL Anywhere and ran the the sqlanysrv.sh script:

| => /Users/atanas/OneDrive\ -\ University\ of\ Wollongong/PhD/PR_qual/unzip/nvivotools/helpers/sqlanysrv.sh
/Users/atanas/OneDrive - University of Wollongong/PhD/PR_qual/unzip/nvivotools/helpers/sqlanysrv.sh: line 44: /Applications/SQLAnywhere17/System/bin64/dbeng??: No such file or directory

Trying to convert and I still get the error as above, copied below just in case it is different:

| => python2 NormaliseNVPX.py /Users/atanas/OneDrive\ -\ University\ of\ Wollongong/PhD/PR_qual/unzip/nvivotools-test.nvpx /Users/atanas/OneDrive\ -\ University\ of\ Wollongong/PhD/PR_qual/unzip/nvivotools-test.norm
/Users/atanas/OneDrive - University of Wollongong/PhD/PR_qual/unzip/nvivotools/helpers/sqlanysrv.sh: line 44: /Applications/SQLAnywhere17/System/bin64/dbeng??: No such file or directory
Started database server on port 63722
Traceback (most recent call last):
  File "NormaliseNVPX.py", line 149, in <module>
    NVivo.Normalise(args)
  File "/Users/atanas/OneDrive - University of Wollongong/PhD/PR_qual/unzip/nvivotools/NVivo.py", line 206, in Normalise
    nvivoAnnotation    = Table('Annotation',    nvivomd, autoload=True)
  File "<string>", line 2, in __new__
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/deprecations.py", line 130, in warned
    return fn(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 496, in __new__
    metadata._remove_table(name, schema)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 491, in __new__
    table._init(name, metadata, *args, **kw)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 585, in _init
    resolve_fks=resolve_fks,
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 626, in _autoload
    _extend_on=_extend_on,
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2146, in run_callable
    with self._contextual_connect() as conn:
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2226, in _contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2266, in _wrap_pool_connect
    e, dialect, self
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1536, in _handle_dbapi_exception_noconnection
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 363, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 760, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
    self._dec_overflow()
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
    self.__connect(first_connect_check=True)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 639, in __connect
    connection = pool._invoke_creator(self)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 451, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 522, in connect
    return Connection(args, kwargs)
  File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 538, in __init__
    parent = Connection.cls_parent = Root("PYTHON")
  File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 464, in __init__
    'libdbcapi_r.dylib')
  File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 456, in load_library
    raise InterfaceError("Could not load dbcapi.  Tried: " + ','.join(map(str, names)))
sqlalchemy.exc.InterfaceError: (sqlanydb.InterfaceError) (u'Could not load dbcapi.  Tried: None,dbcapi.dll,libdbcapi_r.so,libdbcapi_r.dylib', 0)
(Background on this error at: http://sqlalche.me/e/rvf5)

[jschultz]

OK you shouldn't run sqlanysrv.sh yourself - you need to let the python scripts run it because they supply additional arguments. But still... it should give a clearer error message than that.

[atanasj]

Any suggestions what I can do?

[jschultz]

What happens when you run the script without calling sqlanysrv.sh? I can try to duplicate the issue when I have time.

…

On Wednesday, 29 May 2019, atanasj wrote: Any suggestions what I can do? -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #7 (comment)

[atanasj]

You mean just running NormaliseNVPX.py? Error info form the SQLAlchemy page here outlined what it is, but I'm not too sure how to address it... Is it not an issues with nvivotools then? Sorry, I'm very much a novice in things git and opensource.

Here's the output, but not sure if I've run it as you asked:

| => /Users/atanas/OneDrive\ -\ University\ of\ Wollongong/PhD/PR_qual/unzip/nvivotools/NormaliseNVPX.py nvivotools-test.nvpx
/Users/atanas/OneDrive - University of Wollongong/PhD/PR_qual/unzip/nvivotools/helpers/sqlanysrv.sh: line 44: /Applications/SQLAnywhere17/System/bin64/dbeng??: No such file or directory
Started database server on port 49302
Traceback (most recent call last):
  File "/Users/atanas/OneDrive - University of Wollongong/PhD/PR_qual/unzip/nvivotools/NormaliseNVPX.py", line 149, in <module>
    NVivo.Normalise(args)
  File "/Users/atanas/OneDrive - University of Wollongong/PhD/PR_qual/unzip/nvivotools/NVivo.py", line 206, in Normalise
    nvivoAnnotation    = Table('Annotation',    nvivomd, autoload=True)
  File "<string>", line 2, in __new__
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/deprecations.py", line 130, in warned
    return fn(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 496, in __new__
    metadata._remove_table(name, schema)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 491, in __new__
    table._init(name, metadata, *args, **kw)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 585, in _init
    resolve_fks=resolve_fks,
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 626, in _autoload
    _extend_on=_extend_on,
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2146, in run_callable
    with self._contextual_connect() as conn:
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2226, in _contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2266, in _wrap_pool_connect
    e, dialect, self
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1536, in _handle_dbapi_exception_noconnection
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 363, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 760, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
    self._dec_overflow()
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
    self.__connect(first_connect_check=True)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 639, in __connect
    connection = pool._invoke_creator(self)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 451, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 522, in connect
    return Connection(args, kwargs)
  File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 538, in __init__
    parent = Connection.cls_parent = Root("PYTHON")
  File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 464, in __init__
    'libdbcapi_r.dylib')
  File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 456, in load_library
    raise InterfaceError("Could not load dbcapi.  Tried: " + ','.join(map(str, names)))
sqlalchemy.exc.InterfaceError: (sqlanydb.InterfaceError) (u'Could not load dbcapi.  Tried: None,dbcapi.dll,libdbcapi_r.so,libdbcapi_r.dylib', 0)
(Background on this error at: http://sqlalche.me/e/rvf5)

[jschultz]

OK thanks that's useful. I should be able to find some time to look into this tomorrow and get back to you.

[atanasj]

Great, thanks so much for your help.

[jschultz]

OK I managed to replicate and solve your problem.

In short, and for the record, the problem is related to OSX's System Integrity Protection, which forbids user applications from using shared libraries that are not in the standard location by modifying DYLIB_LIBRARY_PATH For the moment, it is still happy to read dynamic libraries from the current working directory, so we make this work by temporarily changing the working directory to the one containing the SQL Anywhere libraries. Really this is an SQL Anywhere problem and they should come up with a solution.

I also stumbled on another problem related to a bug in the sqlalchemy-sqlany library. You need to install my fork of that library, which fixes the bug. See this pull request.

To pick up these changes you'll need to pull the latest commits, then reinstall sqlalchemy-sqlany either using requirements.txt or by following the (changed) instructions.

[atanasj]

Great, thanks for this. Now, being a novice I will ask some questions that may seem to have obvious answers to the initiated:

1. How do I temporarily changing the working directory to the one containing the SQL Anywhere libraries?
2. How would I use the requirements.txt to do this? Do I install your fork via pip install https://github.com/BarraQDA/sqlalchemy-sqlany.git or via git?

Thanks in advance for your help.

Cheers,
Atanas.

[jschultz]

1. How do I temporarily changing the working directory to the one containing the SQL Anywhere libraries?

The updated scripts do all this. All you have to do is refresh your copy of them ('git pull' from the nvivotools directory.

2. How would I use the `requirements.txt` to do this? Do I install your fork via `pip install https://github.com/BarraQDA/sqlalchemy-sqlany.git` or via `git`?

The former of these should do the trick. Or you could do 'pip -r requirements.txt' although this might fail on pymssql. Let me know how you get on.

[atanasj]

Thanks for that. I'm in the middle of some uni work that I need to get out, whilst simultaneously refreshing my Mac after some type of corruption requiring a clean install of macos and reinstalling from a backup image... So, it may take me a several days to get to this but I will definitely let you know how I go.

Cheers,
Atanas.

[atanasj]

Okay, so I tried step 2 above, and I got this error:

| => pip2 install https://github.com/BarraQDA/sqlalchemy-sqlany.git
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting https://github.com/BarraQDA/sqlalchemy-sqlany.git
  Downloading https://github.com/BarraQDA/sqlalchemy-sqlany.git
     | 112kB 1.8MB/s
  ERROR: Cannot unpack file /private/var/folders/77/7ht3td2j1vb00csyp1l4wlv40000gn/T/pip-unpack-HFLkZr/sqlalchemy-sqlany.git (downloaded from /private/var/folders/77/7ht3td2j1vb00csyp1l4wlv40000gn/T/pip-req-build-gSpYgK, content-type: text/html; charset=utf-8); cannot detect archive format
ERROR: Cannot determine archive format of /private/var/folders/77/7ht3td2j1vb00csyp1l4wlv40000gn/T/pip-req-build-gSpYgK

Any thoughts?