Skip to content

Add inference_function() function #78

@DzmitryPihulski

Description

@DzmitryPihulski

The user will write it's own async inference function, something like:

async get_answer(input):
    return output

And will pass it to the inference_function(), something like:

from llmsql import inference_function

result = inference_function(inference_function=get_answer, requests_per_minute=60)

Metadata

Metadata

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions