17 articles
Browse guides
On-device inference keeps data local, works offline, and has no per-token cost — genuinely attractive for privacy and regulated contexts. The trade is capability: local models are smaller and slower than frontier APIs. Quantization and distillation are what make it viable at all. In practice: Transcribing a confidential meeting without the audio leaving the laptop.
When many requests share a long, stable prefix — a system prompt, a document, a set of examples — caching lets the provider skip recomputing it. The saving is large and mostly free: put the stable content first and the variable content last. In practice: A 20,000-token manual cached once, then queried a thousand times cheaply.
Rate limits protect shared capacity and shape your architecture: any production integration needs backoff, retries, and a queue. Limits usually rise with account tier and usage history, so plan for the limit you have today, not the one you hope for. In practice: HTTP 429 at 3pm because everyone’s traffic peaks together.
An SDK saves you from hand-rolling HTTP calls, retries, and streaming. It handles auth, typing, and error handling in idiomatic code. Using one is almost always faster and safer than talking to the raw endpoint. In practice: Three lines of Python instead of thirty lines of request plumbing.
Throughput is the capacity question, distinct from latency’s speed question. Batching raises throughput while raising individual latency, which is exactly the trade you want for offline jobs and exactly wrong for chat. In practice: Batch a million classifications overnight; nobody is watching the clock.
TPUs are ASICs designed specifically for the tensor operations in neural networks, available through Google Cloud rather than as hardware you buy. They are one example of a broader move to purpose-built AI silicon as general-purpose GPUs stop being the only option. In practice: Large training runs on Google infrastructure typically use TPU pods.
Vector search embeds the query and returns the nearest stored vectors, so ‘how do I get my money back’ finds a refund policy that never uses those words. It is the retrieval engine behind RAG. It is also weaker than keyword search on exact identifiers, which is why hybrid search exists. In practice: A query with no shared vocabulary still surfaces the right document.