Introduction
Software development includes writing, testing, deploying and maintaining applications. AI affects it in two ways: it helps developers work faster, and it lets them add new capabilities, such as chat, search and agents, to the products they build. This guide focuses on the second: building reliable AI features.
How AI applies
A language model is a service you call with text and receive text from. Around it you build the parts that make it dependable: your own data through retrieval, tools it may use, validation of its outputs, and monitoring. Good engineering matters more than any single model choice.
Applications and use cases
- Coding assistance: completion, explanation, refactoring and test generation.
- Code review: first-pass checks for bugs and style.
- Documentation: drafts from code and commit history.
- Product features: semantic search, summarisation, chat and recommendations.
- Agents: assistants that call tools such as search, databases or ticketing APIs.
Technologies involved
Model APIs (hosted) or open-source models you run yourself; embedding models and vector databases for retrieval; agent frameworks or plain code for tool use; Python and JavaScript SDKs; and observability tools to trace prompts, cost and latency. Choose models on quality for your task, latency, cost, data-handling terms and licence, using your own evaluation set.
Implementation guide
- Prototype: call a model API with a clear prompt and examples.
- Ground it: add retrieval so answers use your documents.
- Constrain output: request structured formats and validate them.
- Evaluate: run a fixed test set after every change.
- Harden: add retries, timeouts, caching, rate limits and logging.
- Deploy and monitor: track quality, cost and failures.
Technical resources
1. Calling a model reliably
The provider call is generic here. Use your provider's current SDK and model names.
import json, time
def call_json(llm, prompt, retries=3):
for attempt in range(retries):
try:
raw = llm(prompt + "\nReturn valid JSON only.", timeout=30)
return json.loads(raw) # validate before using
except (ValueError, TimeoutError):
time.sleep(2 ** attempt) # exponential backoff
raise RuntimeError("Model did not return valid JSON")2. Retrieval: finding relevant passages
import numpy as np
def top_chunks(query_vec, chunk_vecs, chunks, k=3):
q = query_vec / np.linalg.norm(query_vec)
m = chunk_vecs / np.linalg.norm(chunk_vecs, axis=1, keepdims=True)
scores = m @ q # cosine similarity
best = np.argsort(scores)[::-1][:k]
return [(chunks[i], float(scores[i])) for i in best]Use a vector database once you have more than a few thousand chunks. Embed your documents once and store the vectors.
3. A minimal agent loop
TOOLS = {"search_docs": search_docs, "get_order": get_order} # your own functions
def run_agent(llm, task, max_steps=5):
history = [task]
for _ in range(max_steps): # hard limit prevents runaway loops
step = llm(history, tools=list(TOOLS)) # model picks a tool or answers
if step.final_answer:
return step.final_answer
result = TOOLS[step.tool](**step.args) # allow-list: only your functions
history.append(result)
return "Stopped: step limit reached"Business benefits
Teams aim for faster delivery of routine code, richer product features and less time on documentation. Productivity effects vary by task and team, so measure them on your own work.
Challenges and risks
- Incorrect or insecure code: review and test everything generated.
- Prompt injection: treat retrieved and user content as untrusted, and limit what tools can do.
- Data exposure: check what leaves your environment and the provider's data terms.
- Non-determinism: outputs vary, so test with evaluation sets, not single examples.
- Cost and lock-in: track token usage and keep model calls behind an interface you control.
Examples
Illustrative scenarios: a SaaS product adds natural-language search over its help centre; an engineering team generates test cases for legacy code and reviews each one; an internal agent looks up order status through an approved API.