env: python: No such file or directory

[psychtek]

Nvivo version 12
Getting the error below when trying to convert a nvpx to a norm file. Have read through all the issues and #13 seemed to have more luck but I am unable to get anything to convert.

Python Config

Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Monterey 12.4
python:         /usr/local/bin/python3.10
libpython:      /usr/local/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/config-3.10-darwin/libpython3.10.dylib
pythonhome:     /usr/local/Cellar/python@3.10/3.10.7/Frameworks/Python.framework/Versions/3.10:/usr/local/Cellar/python@3.10/3.10.7/Frameworks/Python.framework/Versions/3.10
version:        3.10.7 (main, Sep 15 2022, 01:51:29) [Clang 14.0.0 (clang-1400.0.29.102)]
numpy:          /usr/local/lib/python3.10/site-packages/numpy
numpy_version:  1.23.2 

Packages:

             package     version              requirement
1          argrecord       0.1.3         argrecord==0.1.3
2     beautifulsoup4      4.11.1   beautifulsoup4==4.11.1
3                bs4       0.0.1               bs4==0.0.1
4            chardet       5.0.0           chardet==5.0.0
5          dateutils      0.6.12        dateutils==0.6.12
6         et-xmlfile       1.1.0        et-xmlfile==1.1.0
7             future      0.18.2           future==0.18.2
8           greenlet       1.1.3          greenlet==1.1.3
9           networkx       2.8.7          networkx==2.8.7
10             numpy      1.23.2            numpy==1.23.2
11          openpyxl      3.0.10         openpyxl==3.0.10
12            pandas       1.4.3            pandas==1.4.3
13          pdfminer    20191125       pdfminer==20191125
14            Pillow       9.2.0            Pillow==9.2.0
15          psycopg2       2.9.3          psycopg2==2.9.3
16      pycryptodome      3.15.0     pycryptodome==3.15.0
17           pymssql       2.2.5           pymssql==2.2.5
18            pyodbc      4.0.34           pyodbc==4.0.34
19   python-dateutil       2.8.2   python-dateutil==2.8.2
20              pytz    2022.2.1           pytz==2022.2.1
21               six      1.16.0              six==1.16.0
22         soupsieve 2.3.2.post1   soupsieve==2.3.2.post1
23        SQLAlchemy      1.4.41       SQLAlchemy==1.4.41
24 sqlalchemy-sqlany       1.0.4 sqlalchemy-sqlany==1.0.4
25          sqlanydb      1.0.11         sqlanydb==1.0.11

Converting

python3 NormaliseNVPX.py -nv 12 ../Projects/Round3_Merged.nvpx

Error

Found SQLAnywhere installation at /Applications/NVivo 12.app/Contents/SQLAnywhere/bin64
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Applications/NVivo 12.app/Contents/SQLAnywhere/lib64/libdbcapi_r.rpath.dylib
env: python: No such file or directory

And the sqlanywhere config script output

Found SQLAnywhere installation at /Applications/NVivo.app/Contents/SQLAnywhere/bin64
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Applications/NVivo.app/Contents/SQLAnywhere/lib64/libdbcapi_r.rpath.dylib
BASH_REMATCH=([0]="NVivo")
CHDIR=/Applications/NVivo.app/Contents/SQLAnywhere/lib64
PIPESTATUS=([0]="1")
SQLANY_API_DLL=libdbcapi_r.rpath.dylib
_=/Applications/NVivo.app/Contents/SQLAnywhere/bin64/sa_config.sh
arch=bin64
binpath=/Applications/NVivo.app/Contents/SQLAnywhere/bin64
bits=64
dbeng=/Applications/NVivo.app/Contents/SQLAnywhere/bin64/dbeng16
libpath=/Applications/NVivo.app/Contents/SQLAnywhere/lib64
nvivo=1
nvivoversion=
source=NVivo.app
sqlanywhere=/Applications
static=0
version=16

Hopefully this is an easy fix and I am simply missing a step somewhere!? Thanks

[jschultz]

The call to install_name_tool is in sqlanyenv.sh which is called from NormaliseNVPX.py. After that something must happen that makes env python fail. If you are able to add debug files into NormaliseNVPX.py you ought to be able to track down the problem. Likely it's a Mac thing - there's a bit of hackery in there to make it work which hasn't been tested on recent MacOS releases.

[psychtek]

I started doing a deep dive and I think I found where it was going funky. I'll recreate and post what I come up with.

Has this been tested with Python3? I read somewhere that there's been some changes to how Python reads binary. 🐍

[jschultz]

OK good luck.
I updated all the tools for Python 3 after Python 2 was discontinued. There were certainly changes around Unicode strings if that's what you mean. It's conceivable that this has created problems, but wouldn't rate it very highly. One thing in particular that isn't really tested is the NVivo 12 stuff. I've never done a comprehensive upgrade for NVivo 12, just made a few ad hoc changes to solve problems that users reported.

[psychtek]

My python 🐍 is pretty average as I work full time with R, so apologies for the approach. I pulled the guts of the NormaliseNVPX script and worked through each section and recorded it here to see if any of it helps. The previous error must have been a PATH setting as this seems to be a non issue now.

A super phenomenal, thank you on this tool set! It has been a lifesaver for archiving our workflow.

Aside from the warnings there are two ForeignKeyConstraint errors. If I comment these out and 'skip' then I can successfully convert a npx to a norm file.

These two lines produce an error:

>>> nvivoAnnotation    = Table('Annotation',    nvivomd, autoload=True)
ArgumentError: ForeignKeyConstraint with duplicate source column references are not supported.
>>> nvivoNodeReference = Table('NodeReference', nvivomd, autoload=True)
ArgumentError: ForeignKeyConstraint with duplicate source column references are not supported.

When I comment out the two lines from Nvivo.py that are creating the previous issues:

            #nvivoAnnotation    = Table('Annotation',    nvivomd, autoload=True)
            nvivoCategory      = Table('Category',      nvivomd, autoload=True)
            nvivoExtendedItem  = Table('ExtendedItem',  nvivomd, autoload=True)
            nvivoItem          = Table('Item',          nvivomd, autoload=True)
            #nvivoNodeReference = Table('NodeReference', nvivomd, autoload=True)
            nvivoProject       = Table('Project',       nvivomd, autoload=True)
            nvivoRole          = Table('Role',          nvivomd, autoload=True)
            nvivoSource        = Table('Source',        nvivomd, autoload=True)
            nvivoUserProfile   = Table('UserProfile',   nvivomd, autoload=True)

And then skip these in the arguments, I get a .norm file:

❯ python3 NormaliseNVPX.py -nv 12 -a 'skip' -t 'skip'  ../../../Nvivo\ Projects/repliCATS_KnownOutcomeClaims_Round3_Merged.nvpx
Found SQLAnywhere installation at /Applications/NVivo 12.app/Contents/SQLAnywhere/bin64
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Applications/NVivo 12.app/Contents/SQLAnywhere/lib64/libdbcapi_r.rpath.dylib
Started database server on port 65332
/usr/local/lib/python3.10/site-packages/sqlalchemy_sqlany/base.py:464: SAWarning: Dialect sqlany:<module 'sqlanydb' from '/usr/local/lib/python3.10/site-packages/sqlanydb.py'> will not make use of SQL compilation caching as it does not set the 'supports_statement_cache' attribute to ``True``.  This can have significant performance implications including some performance degradations in comparison to prior SQLAlchemy versions.  Dialect maintainers should seek to set this attribute to True after appropriate development and testing for SQLAlchemy 1.4 caching support.   Alternatively, this attribute may be set to False which will disable this warning. (Background on this error at: https://sqlalche.me/e/14/cprf)
  return connection.scalar(
Normalising users
/Users/awwillc/Repos/RAT/Scripts/NVivo_Extraction/nvivotools/NVivo.py:175: SAWarning: TypeDecorator UUID() will not produce a cache key because the ``cache_ok`` attribute is not set to True.  This can have significant performance implications including some performance degradations in comparison to prior SQLAlchemy versions.  Set this attribute to True if this type object's state is safe to use in a cache key, or False to disable this warning. (Background on this error at: https://sqlalche.me/e/14/cprf)
  for row in conn.execute(select([table.c[column] for column in columns]))]
Normalising project
Normalising node categories
Normalising nodes
Normalising node attributes
Normalising source categories
Normalising sources
Normalising source attributes

Where would I find the Annotation and NodeReference tables? Is this on the project setup side of things in NVivo?

[psychtek]

Reading through #7 to check a few things and came across this post about the argument error so I ran the code as you recommended and returned the following error in the console:

Collecting git+git://github.com/BarraQDA/sqlalchemy-sqlany
  Cloning git://github.com/BarraQDA/sqlalchemy-sqlany to /private/var/folders/1m/r7b19ccj54sfvx6ymrlkgd186l0xc5/T/pip-req-build-5z6r51ne
  Running command git clone --filter=blob:none --quiet git://github.com/BarraQDA/sqlalchemy-sqlany /private/var/folders/1m/r7b19ccj54sfvx6ymrlkgd186l0xc5/T/pip-req-build-5z6r51ne
  fatal: unable to connect to github.com:
  github.com[0: 20.248.137.48]: errno=Operation timed out

  error: subprocess-exited-with-error

  × git clone --filter=blob:none --quiet git://github.com/BarraQDA/sqlalchemy-sqlany /private/var/folders/1m/r7b19ccj54sfvx6ymrlkgd186l0xc5/T/pip-req-build-5z6r51ne did not run successfully.
  │ exit code: 128
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× git clone --filter=blob:none --quiet git://github.com/BarraQDA/sqlalchemy-sqlany /private/var/folders/1m/r7b19ccj54sfvx6ymrlkgd186l0xc5/T/pip-req-build-5z6r51ne did not run successfully.
│ exit code: 128
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[psychtek]

oops ok so I needed to change:

pip3 install git+git://github.com/BarraQDA/sqlalchemy-sqlany

git to https to the following:

pip3 install git+https://github.com/BarraQDA/sqlalchemy-sqlany

Installing collected packages: sqlalchemy-sqlany
  Attempting uninstall: sqlalchemy-sqlany
    Found existing installation: sqlalchemy-sqlany 1.0.4
    Uninstalling sqlalchemy-sqlany-1.0.4:
      Successfully uninstalled sqlalchemy-sqlany-1.0.4
Successfully installed sqlalchemy-sqlany-1.0.3

I than reran as previous without the -skip args set and getting a bunch of other errors now. I am a little lost 😅

Found SQLAnywhere installation at /Applications/NVivo 12.app/Contents/SQLAnywhere/bin64
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Applications/NVivo 12.app/Contents/SQLAnywhere/lib64/libdbcapi_r.rpath.dylib
Started database server on port 50672
Traceback (most recent call last):
  File "/Users/awwillc/Repos/RAT/Scripts/NVivo_Extraction/nvivotools/NormaliseNVPX.py", line 165, in <module>
    NVivo.Normalise(args)
  File "/Users/awwillc/Repos/RAT/Scripts/NVivo_Extraction/nvivotools/NVivo.py", line 212, in Normalise
    nvivoAnnotation    = Table('Annotation',    nvivomd, autoload=True)
  File "<string>", line 2, in __new__
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/util/deprecations.py", line 309, in warned
    return fn(*args, **kwargs)
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/sql/schema.py", line 616, in __new__
    with util.safe_reraise():
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/sql/schema.py", line 612, in __new__
    table._init(name, metadata, *args, **kw)
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/sql/schema.py", line 687, in _init
    self._autoload(
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/sql/schema.py", line 720, in _autoload
    insp = inspection.inspect(autoload_with)
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/inspection.py", line 64, in inspect
    ret = reg(subject)
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/engine/reflection.py", line 182, in _engine_insp
    return Inspector._construct(Inspector._init_engine, bind)
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/engine/reflection.py", line 117, in _construct
    init(self, bind)
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/engine/reflection.py", line 128, in _init_engine
    engine.connect().close()
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/engine/base.py", line 3315, in connect
    return self._connection_cls(self, close_with_result=close_with_result)
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
    else engine.raw_connection()
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/engine/base.py", line 3394, in raw_connection
    return self._wrap_pool_connect(self.pool.connect, _connection)
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/engine/base.py", line 3361, in _wrap_pool_connect
    return fn()
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/pool/base.py", line 320, in connect
    return _ConnectionFairy._checkout(self)
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/pool/base.py", line 884, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/pool/base.py", line 486, in checkout
    rec = pool._do_get()
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/pool/impl.py", line 145, in _do_get
    with util.safe_reraise():
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
    return self._create_connection()
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/pool/base.py", line 266, in _create_connection
    return _ConnectionRecord(self)
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/pool/base.py", line 381, in __init__
    self.__connect()
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/pool/base.py", line 691, in __connect
    )._exec_w_sync_on_first_run(self.dbapi_connection, self)
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/event/attr.py", line 320, in _exec_w_sync_on_first_run
    self(*args, **kw)
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/event/attr.py", line 334, in __call__
    fn(*args, **kw)
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/util/langhelpers.py", line 1695, in go
    return once_fn(*arg, **kw)
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/engine/create.py", line 674, in first_connect
    dialect.initialize(c)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy_sqlany/base.py", line 470, in initialize
    super(SQLAnyDialect, self).initialize(connection)
  File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/engine/default.py", line 414, in initialize
    self.default_schema_name = self._get_default_schema_name(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy_sqlany/base.py", line 465, in _get_default_schema_name
    text("SELECT current user",
TypeError: text() got an unexpected keyword argument 'typemap'

[jschultz]

OK you're doing well.

The last problem was simply that I hadn't merged upstream fixes to sqlalchemy into my fork. If you do the pip3 install ... command now you should get past this latest error.

[psychtek]

Ok that all seems to have fixed that, thank you!

I started fresh and then that error returned to haunt me.

env: python: Not a directory

When I comment out line 89 that's spawning a new process it runs successfully.

# On non-Windows OS, need to set up environment for SQL Anywhere server and restart process.
if os.name != 'nt':
    # Check if already done
    if not os.environ.get('_restart'):
        if args.sqlanywhere:
            os.environ['sqlanywhere'] = args.sqlanywhere
        envlines = subprocess.check_output(helperpath + 'sqlanyenv.sh', text=True).splitlines()
        for envline in envlines:
            env = re.match(r"(?P<name>\w+)=(?P<quote>['\"]?)(?P<value>.*)(?P=quote)", envline, re.MULTILINE | re.DOTALL).groupdict()
            os.environ[env['name']] = env['value']

        os.environ['_restart'] = 'TRUE'
        #os.execve(sys.argv[0], sys.argv, os.environ)

Like you suggested, likely a mac thing. But this seems to be a work around for now for things on my end.