17 articles
Browse guides
An API is how software talks to software. For AI it means you post a request containing your prompt and settings and receive generated output, paying per token. It is what makes a model a component you can build on rather than a website you visit. In practice: One HTTPS request in, one JSON response out, billed by tokens used.
An API key identifies your account to the provider. Anyone holding it can spend your budget and read your usage, so keys belong on a server or in a secrets manager, never in front-end code or a public repository. Rotate them and scope them wherever the provider allows. In practice: A key committed to a public repo gets found and used within minutes.
Batch inference processes a queue of inputs where latency does not matter, using hardware far more efficiently. Providers often discount it substantially. If a human is not waiting for the answer, it is the cheaper path. In practice: Classifying a year of support tickets overnight at half price.
Chunking is the least glamorous and most consequential step in a RAG pipeline. Chunks that are too large dilute the embedding and waste context; too small and they lose the meaning that made them relevant. Splitting on structure — headings, sections — beats splitting on character count. In practice: Cutting a contract mid-clause, so retrieval returns half a sentence that means nothing.
GPUs perform the massive matrix multiplications behind training and inference thousands of times faster than a CPU. Their memory capacity sets the ceiling on model size, which is why quantization matters. Supply and price of GPUs shape the whole market. In practice: Whether a model fits in 24GB of VRAM decides if it runs on your desk.
Hybrid search runs lexical and semantic retrieval together and merges the rankings. Keywords nail exact matches — product codes, names, error strings — while vectors handle paraphrase. In production RAG this combination reliably beats either alone. In practice: ‘Error E-4471 refund’ needs the exact code and the concept of refunds.
Input and output tokens are usually priced differently, with output the expensive side. Costs scale with conversation length because history is resent each turn, which is why naive chat apps get expensive fast. Caching, shorter context, and routing to smaller models are the standard levers. In practice: Resending a 50-page document with every follow-up question, and paying for it every time.
For LLMs, latency splits into time-to-first-token, which drives perceived speed, and total generation time. Streaming attacks the first; smaller models and shorter outputs attack the second. Reasoning models trade latency for accuracy on purpose. In practice: 400ms to first word feels instant; four seconds of blank screen feels dead.
LLMOps shifts the concerns because you usually are not training the model. What you version, test, and monitor is prompts, retrieval, tool definitions, evals, latency, spend, and output safety. The core discipline is the same: nothing ships without a test that would catch its failure. In practice: Every prompt change runs the eval suite in CI before it reaches production.
MLOps applies DevOps thinking to ML, with extra problems: data versioning, training reproducibility, drift monitoring, and retraining pipelines. The insight it encodes is that a model in a notebook is roughly 10% of the work. In practice: Automatic retraining triggered when monitored accuracy drops below a threshold.