spinf-serve
A GPU server that reads content once to calculate answer-choice probabilities for multiple classification questions.
What does it do?
When sent content and multiple questions with answer choices, spinf-serve reads the content once and returns a probability for each choice in each question. Responses also include baseline probabilities without the content and probability mass outside the choices. Prompt packs containing questions and scoring rules can be loaded in advance to return category scores and decisions. It can run on the user’s own hardware; the documented default setup uses one 24GB NVIDIA GPU and Linux.
Key features
- Reads a content item once to calculate answer-choice probabilities for multiple questions.
- Returns baseline probabilities without content and residual probability outside the choices.
- Loads prompt packs to include category scores and decisions in responses.
- Accepts classification requests through an OpenAI-compatible API.
- Runs on a self-hosted GPU server under the Apache 2.0 license.