Hey friends,
A short post of my 2023 in review. I'll go through the goals I set in 2022, some highlights for the year (e.g., diving into language modeling), goals for 2024, and some stats from 2023. If you have a review of your own, please reply to this with it—I'd love to read it.
I appreciate you receiving this, but if you want to stop, simply unsubscribe.
• • •
👉 Read in browser for best experience (web version has extras & images) 👈
2023 was a peaceful year of small, steady steps. There were no major lifestyle changes and I had the time and energy to explore new interests and focus on learning. Here’s my 2023 year in review, including goals, highlights, and statistics.
First, checking in on the goals I had set for myself last year:
I wished I could have found more time to meditate and read fiction but it’s been tough this year with all the learning I’ve had to do. Nonetheless, I’m happy with the progress on the other goals. Beyond the goals, I also had a few other highlights.
With the step-change improvement from gpt-3.5-turbo, I started paying more attention to language modeling (LM) at the start of 2023. Thankfully, I had some familiarity with the subject, having previously applied LM models to recommender systems and experimented with gpt-3 for simple summarizations. To catch up, I did some experimentation and prototyping to learn the strengths and limits of autoregressive LMs. These prototypes included Discord bots, UIs for interacting with LLMs, Obsidian copilot based on lexical and embedding-based RAG, and finetuning a hallucination classifier on out-of-domain data.
This self-learning set me up for success at internal hackathons. Our first prototype, codenamed Dewey, did well (shout out to my awesome partner-in-crime Kelly N!) This gave me the courage to ask if I could start working on GenAI at least one day per week, and perhaps eventually transition to half-time. This request went better than expected…
Since then, my Charter has expanded to include working on GenAI initiatives across the org. This includes educating the working level and senior leadership, as well as figuring out how to deploy GenAI reliably and cost-effectively at scale. We’ve organized two fruitful hackathons: Two prototypes have been launched while two more are in progress.
Although I didn’t hit my goal of 26 posts, I’m proud of the writing I’ve done this year. Some pieces that have been impactful include:
Much of my writing is a byproduct of my work building ML systems, and in return it contributes to me being more effective at the work. Writing helps identify and fill my knowledge gaps, refine my thinking, and scale my sharing. (I learned that my leadership reads my writing and tweets 😱)
As usual, I found writing easier when I write with a specific person in mind (instead of a vague general audience). Some example pieces and their audience-of-one include:
I also had the opportunity to speak at the inaugural AI Engineer Summit. The energy was inspiring (my recap here) and I got the chance to connect and stay in touch with several practitioners as we figure out how to use this new technology in production.
In addition, I gave an invited keynote at the Amazon Machine Learning Conference. I took the chance to share the awesome work our team had done on session-based retrieval, contextual ranking, and cost-effective, just-in-time infrastructure. (I previously presented a public version at RecSys 2022 as a keynote too.)
Together with a few friends, we started a paper club to read and discuss fundamental papers in the LM space. I believe we’ve learned more as a group than we could have individually, by pooling together our shared knowledge, experience, and questions. Here are one-sentence summaries for the earlier papers.
I made three angel investments in ML and tooling startups. This will likely be the volume of investments going forward. I’ve also become more selective, focusing on startups where I can provide the most value, mostly in the field of data and ML.
I had a minor health scare (that involved bruising easily) that prompted me to reexamine my diet and exercise. My wife and I paid more scrutiny to our nutrition, such as reducing saturated fat and alcohol. We continued to keep sugar and processed food to a minimum.
Also, while I’ve been consistent with weight training, I’ve been neglecting cardio. Thus, I started forcing myself to jog at least twice a week. On challenging weeks, I make do with 45-minute brisk walks. This has improved my resting heart rate and VO2 max.
I wrote my first mission statement in 2013 and have been revising it every year or so. Here’s the latest iteration, written while on the flight back from SF after a particularly inspiring and energizing week. Fun fact: It started as a tweet and fits in 280 chars.
Here’s a word cloud of my writing in 2023. The top themes (user, data, model) have been consistent though the focus on LLMs is new. (Word cloud from a previous year here.)
![]() |
Word cloud of my writing in 2023
This site saw 259k unique visitors in 2023, an increase of 21% from 2022.
![]() |
Number of unique visitors in 2023
The incoming channels were mostly direct, organic search, and organic social. The US is the largest source by far (and it looks like my audience in Singapore is tapering 😢).
![]() |
Channel and geography of readers in 2023
The top pages in 2023 were mostly broader pieces on patterns and system design. Old but gold pieces continue to be in the top 10, including: (i) recsys system design, (ii) writing why what how, and (iii) real-time retrieval.
![]() |
Top 10 most visited pages in 2023
Clicks via Google search was mostly flat at 77.6k, though impressions increased from 2.14M to 2.71M (+27%). As a result, average CTR dropped from 3.6% to 2.9% (-19%).
![]() |
Google search traffic in 2023
Miscellaneous social metrics
![]() |
Email subscriber growth in 2023
I build ML, RecSys, and LLM systems that serve customers at scale, and write about what I learn along the way. Join 7,500+ subscribers!
Hey friends, I've been thinking and experimenting a lot with how to apply, evaluate, and operate LLM-evaluators and have gone down the rabbit hole on papers and results. Here's a writeup on what I've learned, as well as my intuition on it. It's a very long piece (49 min read) and so I'm only sending you the intro section. It'll be easier to read the full thing on my site. I appreciate you receiving this, but if you want to stop, simply unsubscribe. 👉 Read in browser for best experience (web...
Hey friends, Just got back from the AI Engineer World's Fair and it was a blast! I had the opportunity to give the closing keynote, as well as host GitHub CEO Thomas Dohmke for a fireside chat. Along the same lines, I've been thinking about how to interview for ML/AI engineers and scientists, and got together with Jason to write about the technical and non-technical skills to look for, how to phone screen, run interview loops, and debrief, and some tips for interviewers and hiring managers....
Hey friends, Recently a couple of friends and I got together to write about some challenges and hard-won lessons from a year of building with LLMs. One thing led to another and this is now published on O'Reilly in three sections: Tactics: Prompting, RAG, workflows, caching, when to finetune, evals, guardrails Ops: Looking at data, working with models, product and risk, building a team Strategy: "No GPUs before PMF", "the system not the model", how to iterate, cost We have a dedicated site...