pyarrow.lib.ArrowInvalid: Field named <column_name> is not found

[bencoldham]

The following code produced an error on the transform function. The fit function works correctly. This error is reproduced for every feature in the original dataset.

X = df.drop(["Target"], axis=1)
y = df["Target"]

ofe = OpenFE()
ofe.fit(data=X, label=y, categorical_features=cat_cols, n_jobs=11)

train_x, test_x = transform(X, test, ofe.new_features_list[:50], n_jobs=11 )

Traceback (most recent call last):
  File "/home/ben/projects/kaggle/.venv/lib/python3.10/site-packages/openfe/utils.py", line 102, in _cal
    _data = pd.read_feather('./openfe_tmp_data.feather', columns=base_features).set_index('openfe_index')
  File "/home/ben/projects/kaggle/.venv/lib/python3.10/site-packages/pandas/io/feather_format.py", line 124, in read_feather
    return feather.read_feather(
  File "/home/ben/projects/kaggle/.venv/lib/python3.10/site-packages/pyarrow/feather.py", line 226, in read_feather
    return (read_table(
  File "/home/ben/projects/kaggle/.venv/lib/python3.10/site-packages/pyarrow/feather.py", line 262, in read_table
    table = reader.read_names(columns)
  File "pyarrow/_feather.pyx", line 114, in pyarrow._feather.FeatherReader.read_names
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Field named evaluations is not found

[ZhangTP1996]

Could you please provide a minimal reproducible example with code and data so that I can fix it?

[scar-fx]

Could you please provide a minimal reproducible example with code and data so that I can fix it?

i have same issue
here is how my data looks .

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 76368 entries, 0 to 76367
Data columns (total 36 columns):

Column Non-Null Count Dtype

---

0 Marital status 76368 non-null int64
1 Application mode 76368 non-null int64
2 Application order 76368 non-null int64
3 Course 76368 non-null int64
4 Daytime/evening attendance 76368 non-null int64
5 Previous qualification 76368 non-null int64
6 Previous qualification (grade) 76368 non-null float64
7 Nacionality 76368 non-null int64
8 Mother's qualification 76368 non-null int64
9 Father's qualification 76368 non-null int64
10 Mother's occupation 76368 non-null int64
11 Father's occupation 76368 non-null int64
12 Admission grade 76368 non-null float64
13 Displaced 76368 non-null int64
14 Educational special needs 76368 non-null int64
15 Debtor 76368 non-null int64
16 Tuition fees up to date 76368 non-null int64
17 Gender 76368 non-null int64
18 Scholarship holder 76368 non-null int64
19 Age at enrollment 76368 non-null int64
20 International 76368 non-null int64
21 Curricular units 1st sem (credited) 76368 non-null int64
22 Curricular units 1st sem (enrolled) 76368 non-null int64
23 Curricular units 1st sem (evaluations) 76368 non-null int64
24 Curricular units 1st sem (approved) 76368 non-null int64
25 Curricular units 1st sem (grade) 76368 non-null float64
26 Curricular units 1st sem (without evaluations) 76368 non-null int64
27 Curricular units 2nd sem (credited) 76368 non-null int64
28 Curricular units 2nd sem (enrolled) 76368 non-null int64
29 Curricular units 2nd sem (evaluations) 76368 non-null int64
30 Curricular units 2nd sem (approved) 76368 non-null int64
31 Curricular units 2nd sem (grade) 76368 non-null float64
32 Curricular units 2nd sem (without evaluations) 76368 non-null int64
33 Unemployment rate 76368 non-null float64
34 Inflation rate 76368 non-null float64
35 GDP 76368 non-null float64
dtypes: float64(7), int64(29)
memory usage: 21.0 MB

the is also a target column that is on it own .

if you need more ifno let know and i will provide it thanks .

[scar-fx]

here is the full error message :
raceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/openfe/utils.py", line 102, in _cal
_data = pd.read_feather('./openfe_tmp_data.feather', columns=base_features).set_index('openfe_index')
File "/opt/conda/lib/python3.10/site-packages/pandas/io/feather_format.py", line 124, in read_feather
return feather.read_feather(
File "/opt/conda/lib/python3.10/site-packages/pyarrow/feather.py", line 226, in read_feather
return (read_table(
File "/opt/conda/lib/python3.10/site-packages/pyarrow/feather.py", line 262, in read_table
table = reader.read_names(columns)
File "pyarrow/_feather.pyx", line 114, in pyarrow._feather.FeatherReader.read_names
File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Field named credited is not found

---

_RemoteTraceback Traceback (most recent call last)
_RemoteTraceback:
"""
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/openfe/utils.py", line 102, in _cal
_data = pd.read_feather('./openfe_tmp_data.feather', columns=base_features).set_index('openfe_index')
File "/opt/conda/lib/python3.10/site-packages/pandas/io/feather_format.py", line 124, in read_feather
return feather.read_feather(
File "/opt/conda/lib/python3.10/site-packages/pyarrow/feather.py", line 226, in read_feather
return (read_table(
File "/opt/conda/lib/python3.10/site-packages/pyarrow/feather.py", line 262, in read_table
table = reader.read_names(columns)
File "pyarrow/_feather.pyx", line 114, in pyarrow._feather.FeatherReader.read_names
File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Field named credited is not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/lib/python3.10/concurrent/futures/process.py", line 246, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/opt/conda/lib/python3.10/site-packages/openfe/utils.py", line 111, in _cal
exit()
NameError: name 'exit' is not defined
"""

The above exception was the direct cause of the following exception:

NameError Traceback (most recent call last)
Cell In[62], line 2
1 #transform the train and test data according to generated features.
----> 2 f_x_train, f_x_val = transform(x_train, x_val, features, n_jobs=1)

File /opt/conda/lib/python3.10/site-packages/openfe/utils.py:147, in transform(X_train, X_test, new_features_list, n_jobs, name)
145 cat_feats = []
146 for i, res in enumerate(results):
--> 147 is_cat, d1, d2, f = res.result()
148 names.append('autoFE_f_%d' % i + name)
149 names_map['autoFE_f_%d' % i + name] = f

File /opt/conda/lib/python3.10/concurrent/futures/_base.py:451, in Future.result(self, timeout)
449 raise CancelledError()
450 elif self._state == FINISHED:
--> 451 return self.__get_result()
453 self._condition.wait(timeout)
455 if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:

File /opt/conda/lib/python3.10/concurrent/futures/_base.py:403, in Future.__get_result(self)
401 if self._exception:
402 try:
--> 403 raise self._exception
404 finally:
405 # Break a reference cycle with the exception in self._exception
406 self = None

NameError: name 'exit' is not defined

[ZhangTP1996]

It seems that the error comes from _data = pd.read_feather('./openfe_tmp_data.feather', columns=base_features).set_index('openfe_index'). But credited is not in the columns. Are you trying to run multiple openfe processes in the same machine?

[scar-fx]

only one openfe process am runing it on kaggle notebook .