What We’ve Learned From A Year of Building with LLMs


Hey friends,

Recently a couple of friends and I got together to write about some challenges and hard-won lessons from a year of building with LLMs. One thing led to another and this is now published on O'Reilly in three sections:

  • Tactics: Prompting, RAG, workflows, caching, when to finetune, evals, guardrails
  • Ops: Looking at data, working with models, product and risk, building a team
  • Strategy: "No GPUs before PMF", "the system not the model", how to iterate, cost

We have a dedicated site with all 42 lessons https://applied-llms.org/ and a job board with vetted jobs: https://applied-llms.org/jobs (get 20% off with discount code "2B212D9F").

While I'd typically paste the content here so you can read it from the convenience of your inbox, the current piece is long and probably best read in your browser. Nonetheless, here's the behind the scenes of how this started. I hope you find our writeup useful!

Eugene Yan

I build ML, RecSys, and LLM systems that serve customers at scale, and write about what I learn along the way. Join 7,500+ subscribers!

Read more from Eugene Yan

Hey friends, I've been thinking and experimenting a lot with how to apply, evaluate, and operate LLM-evaluators and have gone down the rabbit hole on papers and results. Here's a writeup on what I've learned, as well as my intuition on it. It's a very long piece (49 min read) and so I'm only sending you the intro section. It'll be easier to read the full thing on my site. I appreciate you receiving this, but if you want to stop, simply unsubscribe. 👉 Read in browser for best experience (web...

Hey friends, Just got back from the AI Engineer World's Fair and it was a blast! I had the opportunity to give the closing keynote, as well as host GitHub CEO Thomas Dohmke for a fireside chat. Along the same lines, I've been thinking about how to interview for ML/AI engineers and scientists, and got together with Jason to write about the technical and non-technical skills to look for, how to phone screen, run interview loops, and debrief, and some tips for interviewers and hiring managers....

Hey friends, I've been helping teams with their prompts lately and was sad to see how they didn't have a good understanding of the basics, even as they reached for advanced techniques and complicated prompting tools. This spurred me to write this piece on the fundamentals of prompting. By mastering these, we should get 80 - 90% of they way to the optimal prompt. Aside: My friend Hamel Husain is organizing an LLM Conference + Finetuning Workshop: 11 talks by world-class practitioners like...