Concept
Inference
The process in which a trained AI model responds to a concrete request — the actual ‘use’ of a model.
Inference denotes applying an already trained model: a request goes in, the model computes an answer. Unlike training, which is a one-off and very compute-intensive, inference happens millions of times in ongoing operation.
Because every use is inference, it determines the running cost and the latency of an AI service. Cloud providers bill inference per token; when you run it yourself, hardware utilization determines the price per request.
An inference server is the machine that performs these computations — usually GPU-backed. How many requests it handles stably at once depends less on the model than on the operating layer.