Knitro attribute not available on Knitro 13.2

[flucoe]

Hi Jeff,

I was trying to use Knitro but I'm getting an odd message for version 13.2.0

Specifically, I can import it (and actually use it for non-pyblp things, such as the examples*.py provided with Knitro) but when defining the optimization (kn = pyblp.Optimization('knitro')) I get

  File [relevant_path/] pyblp/configurations/optimization.py", line 470, in knitro_context_manager
    knitro_context = knitro.KTR_new()
AttributeError: module 'knitro' has no attribute 'KTR_new'

I tried this on my local machine and on an HPRC system (to make sure it was not my installation that was generating the problem), and in both cases I get the same message (with different path to optimization.py, of course). Any ideas of what could be happening? I was not getting this with previous versions of Knitro such as 13.0.1.

Also, I can I use Knitro without trouble outside pyblp.

Thanks.

[jeffgortmaker]

Ugh, I was hoping Knitro would keep their software backwards-compatible. My best guess is they dropped compatibility with their old API in one of the newest versions, but I don't have the newest version installed on my machine, so I can't check right now.

When you use Knitro outside of PyBLP, are you calling functions that look different than the ones I call in the following code?

pyblp/pyblp/configurations/optimization.py

Lines 345 to 486 in 8ff613d

	def knitro_optimizer(
	initial_values: Array, bounds: Optional[Iterable[Tuple[float, float]]], objective_function: ObjectiveFunction,
	iteration_callback: Callable[[], None], compute_gradient: bool, **knitro_options: Any) -> Tuple[Array, bool]:
	"""Optimize with Knitro."""
	with knitro_context_manager() as (knitro, knitro_context):
	iterations = 0
	cache: Optional[Tuple[Array, ObjectiveResults]] = None
	def combined_callback(
	request_code: int, _: Any, __: Any, ___: Any, ____: Any, values: Array, _____: Any,
	objective_store: Array, ______: Any, gradient_store: Array, *_______: Any) -> int:
	"""Handle requests to compute either the objective or its gradient (which are cached for when the next
	request is for the same values) and call the iteration callback when there's a new major iteration.
	"""
	nonlocal iterations, cache
	# call the iteration callback if this is a new iteration
	current_iterations = knitro.KTR_get_number_iters(knitro_context)
	while iterations < current_iterations:
	iteration_callback()
	iterations += 1
	# compute the objective or used cached values
	if cache is None or not np.array_equal(values, cache[0]):
	cache = (values.copy(), objective_function(values))
	objective, gradient = cache[1]
	# define a function that normalizes values so they can be digested by Knitro
	normalize = lambda x: min(max(float(x), -sys.maxsize), sys.maxsize)
	# handle request codes
	if request_code == knitro.KTR_RC_EVALFC:
	objective_store[0] = normalize(objective)
	return knitro.KTR_RC_BEGINEND
	if request_code == knitro.KTR_RC_EVALGA:
	assert compute_gradient and gradient is not None
	for index, gradient_value in enumerate(gradient.flatten()):
	gradient_store[index] = normalize(gradient_value)
	return knitro.KTR_RC_BEGINEND
	return knitro.KTR_RC_CALLBACK_ERR
	# configure Knitro callbacks
	callback_mapping = {
	knitro.KTR_set_func_callback: combined_callback,
	knitro.KTR_set_grad_callback: combined_callback
	}
	for set_callback, callback in callback_mapping.items():
	code = set_callback(knitro_context, callback)
	if code != 0:
	raise RuntimeError(f"Encountered error code {code} when registering {set_callback.__name__}.")
	# configure Knitro parameters
	for key, value in knitro_options.items():
	set_parameter = knitro.KTR_set_param_by_name
	if isinstance(value, str):
	set_parameter = knitro.KTR_set_char_param_by_name
	code = set_parameter(knitro_context, key, value)
	if code != 0:
	raise RuntimeError(f"Encountered error code {code} when configuring '{key}'.")
	# initialize the problem
	bounds = bounds or [(-np.inf, +np.inf)] * initial_values.size
	with warnings.catch_warnings():
	warnings.simplefilter('ignore')
	code = knitro.KTR_init_problem(
	kc=knitro_context,
	n=initial_values.size,
	xInitial=initial_values,
	lambdaInitial=None,
	objGoal=knitro.KTR_OBJGOAL_MINIMIZE,
	objType=knitro.KTR_OBJTYPE_GENERAL,
	xLoBnds=np.array([b[0] if np.isfinite(b[0]) else -knitro.KTR_INFBOUND for b in bounds]),
	xUpBnds=np.array([b[1] if np.isfinite(b[1]) else +knitro.KTR_INFBOUND for b in bounds]),
	cType=None,
	cLoBnds=None,
	cUpBnds=None,
	jacIndexVars=None,
	jacIndexCons=None,
	hessIndexRows=None,
	hessIndexCols=None
	)
	if code != 0:
	raise RuntimeError(f"Encountered error code {code} when initializing the Knitro problem solver.")
	# solve the problem
	values_store = np.zeros_like(initial_values)
	with warnings.catch_warnings():
	warnings.simplefilter('ignore')
	return_code = knitro.KTR_solve(
	kc=knitro_context, x=values_store, lambda_=np.zeros_like(initial_values), evalStatus=0,
	obj=np.array([0], np.float64), c=None, objGrad=None, jac=None, hess=None, hessVector=None,
	userParams=None
	)
	# Knitro was only successful if its return code was 0 (final solution satisfies the termination conditions for
	# verifying optimality) or between -100 and -199 (a feasible approximate solution was found)
	return values_store, return_code > -200
	@contextlib.contextmanager
	def knitro_context_manager() -> Iterator[Tuple[Any, Any]]:
	"""Import Knitro and initialize its context."""
	try:
	import knitro
	except OSError as exception:
	if 'Win32' in repr(exception):
	raise EnvironmentError("Make sure both Knitro and Python are 32- or 64-bit.") from exception
	raise
	# modify older version of Knitro to work with NumPy
	try:
	# noinspection PyUnresolvedReferences
	import knitroNumPy
	knitro.KTR_array_handler._cIntArray = knitroNumPy._cIntArray
	knitro.KTR_array_handler._cDoubleArray = knitroNumPy._cDoubleArray
	knitro.KTR_array_handler._userArray = knitroNumPy._userArray
	knitro.KTR_array_handler._userToCArray = knitroNumPy._userToCArray
	knitro.KTR_array_handler._cToUserArray = knitroNumPy._cToUserArray
	except ImportError:
	pass
	# create the Knitro context and attempt to free it if anything goes wrong
	knitro_context = None
	try:
	knitro_context = None
	try:
	knitro_context = knitro.KTR_new()
	except RuntimeError as exception:
	if 'Error while initializing parameter' not in str(exception):
	raise
	if not knitro_context:
	raise OSError(
	"Failed to find a Knitro license. Make sure that Knitro is properly installed. You may have to create "
	"the environment variable ARTELYS_LICENSE and set it to the location of the directory with the license "
	"file."
	)
	yield knitro, knitro_context
	finally:
	try:
	knitro.KTR_free(knitro_context)
	except Exception:
	pass

If so, I'll leave this issue open as a reminder that PyBLP's Knitro wrapper has to be updated for newer versions. Until then, you have a few options:

1. Downgrade your Knitro version so it's PyBLP-compatible.
2. Write a custom Optimization method that uses the new Knitro API. It might help to refer to the above code I use to interface with the old API.
3. Make a pull request that adds support for the new API to the above code. I'm happy to look over it, etc.

[flucoe]

Hi Jeff. I'm sorry for the slow reply.

I'll check whether it is possible to downgrade for now. If not, I'll try to figure out how to use the custom optimization method. Regarding my use of Knitro: I have been using the approach in exampleSciPy.py within examples/Python/examples with scipy minimize() function but with the kn_minimize method.

[jeffgortmaker]

Sounds good -- if you do end up making a custom optimization method and don't mind posting it here, it would be a great starting point for making the package compatible with the newer interface.

[allen107]

Hi fluce, @flucoe

I have encounter same probelm here. Can you share your code that uses kn_minimize method? Thx.