The one-line rule
Prompting changes the LLM's behavior for one call. Fine-tuning changes it for all calls. RAG changes what the LLM knows. Choose by what's varying in your problem.
Default order
Always try prompting first. Add RAG if you need facts the model doesn't have. Fine-tune last, when prompting + RAG won't get behavior consistent enough.
When prompting is enough
Prompting handles the majority of production LLM use cases.
- →Task is well-described in plain English
- →A few in-context examples shape it to your liking
- →Output format is enforced via JSON schema / function calling
- →Behavior is consistent enough for your QA bar
When you actually need RAG
RAG is for external knowledge — yours, not the model's.
- →Answers depend on private/internal data the LLM was never trained on
- →Knowledge changes faster than you can retrain
- →You need citations for compliance or trust
- →Volume of knowledge exceeds practical context window
When fine-tuning is the right call
Fine-tuning is for behavior. It's the most expensive option — pick it deliberately.
- →You need a specific output style (e.g., terse, JSON-only, in a domain dialect) every time
- →Few-shot examples in the prompt are eating too many tokens
- →Domain reasoning is consistent but unusual (legal phrasing, medical formatting)
- →Cost per request matters — small fine-tuned models often beat large generic ones
The decision tree
- →Step 1: Try prompting with few-shot + CoT + JSON mode. If quality bar met → ship.
- →Step 2: Quality issues from missing knowledge? → Add RAG.
- →Step 3: Quality issues from inconsistent behavior, formatting, or domain phrasing? → Consider fine-tuning (LoRA).
- →Step 4: Cost / latency too high? → Distill or fine-tune a smaller model to match.
Most real systems combine all three
In practice, a polished production LLM app uses prompting (the surface), RAG (the facts), and fine-tuning (the style + domain). Each handles one axis cleanly. Treat them as orthogonal levers, not competitors.