Summary
Even with the basic GPT-4-o1 it worked great, but needed a 2-phase prompt to get good result
Reading takes time; there’s too much to read
There is a vast world of high-quality writing on the internet. No-one has time to read it all. So you filter, you read other people’s lists, etc.
But even that’s too much: If you meticulously save every ‘great’ document you find … you collect so many docs that you can’t remember which is which. And how do you give colleagues the info they need to self-serve in looking through your Awesome List Of Awesome Resources – many people say “read them all till you find the one you want” which is more of a curse than a gift.
Commercial options: Google NotebookLM
NotebookLM was created for this use-case. NotebookLM is sadly bad at this use-case and I find it generally not worth the effort: it too often returns factually false statements, and usually misses the key information in a document set. It is also hobbled by Google’s absurd policies (e.g. limiting documents to extremely short size, forcing you to manually split them before uploading).
Free options: Ask an LLM
If you prompt an LLM something like:
“Summarise this document”
… you’ll get a very random response in quality, accuracy, precision, usefulness. It’s what we call a FailPrompt: it’s so badly written (by you) that it’s playing to an LLM’s weaknesses instead of its strengths.
Approach: Use ChatGPT to ‘meta summarise’
I’m building up some reading resources for clients and colleagues to help them find their way through the best stuff about LLMs. One of the resources I like is https://github.com/humanlayer/12-factor-agents – it’s a single-page document, but 90% of the content is split-out into 12 separate webpages, stored in github (not famous as a document publishing platform!).
Knowing how good LLMs today (2025) are if you take away the guardrails … instead of downloading the files, I picked a page which direct-links to all 12 of the webpages, and which is easy for LLMs to parse (its semantically distinct, with clear separation of content from navigation, etc). I then ran a 2-phase process to get the summaries I wanted.
- Phase 1: push it to find the “interesting points” spread across the 12 articles.
- Phase 2: enrich that by asking it for the underlying ‘why’ — but prevent it from choosing a ‘why’ per ‘point’: instead look for ‘whys’ that span ALL the points
Phase 2 is key: if we let the LLM do a 1-for-1 (its default approach) it’ll have a weakest-link-in-the-chain problem: the output will get worse for each enrichment. Instead we re-mingle the results after each phase, causing the LLM to start to see the bigger picture, and use that as its basis.
Phase 1: find ‘interesting points’ in any doc
Prompt used
Outcome
Several interesting things happened – and none of them were positive
Analysis:
- It only gave one quote, for one doc. Useless (but easy to fix: followon prompt is “continue for all 12 docs”)
- It did a websearch of twitter looking for what people ACTUALLY SAID about this URL. Intriguing, and cool – but mostly useless: this is deliberately dumbing-down the AI to the worst of humanity, by averaging social media commentary (most of which is mindless ‘like’ button clicking by people who haven’t read the docs).
- The quote was pretty meaningless overall
But we’re not giving up … we planned a 2-phase process, so lets continue. First we’ll push it to finish the task it bailed on (thanks, OpenAI, for dumbing-down ChatGPT to be lazy and stupid: we know the LLM is much better than this out-of-the-box, but that OpenAI do this on purpose, avoiding answering the question (to save money etc))
Prompt used (addendum)
… with this it completed phase 1 outputting 12 x individual pullquotes.
Analysis:
- this time it did NOT use Twitter and websearch
- … instead it actually did what we asked (note: the previous response literally didn’t do what was asked, instead it just used glorified Google search to avoid doing any work)
- all the pullquotes listed were accurate to the documents I’d read
- … BUT they were unexciting and failed to surface what was valuable in this doc-set
When LLMs fail to pick-up on the ‘what is actually INTERESTING here?’ there are a lot of things you can try – this is one of my personal pet research topics for 2024/2025. A quick glance at one of the docs I liked reminded me that most of what the author is saying is interesting because he keeps providing justifications for his claims.
This overlaps with one of the most famous techniques used by humans: the Five Whys – something we’ve used for decades in every large company I’ve worked at.
Phase 2: find underlying ‘why’ across the doc-set
Prompt used
Items to note:
- that second paragraph is critical: it’s where we comingle the outputs of each doc into a shared pool, and we actively forbid the AI from doing a 1:1 matching (which experience tells us is nearly always poor quality with every LLM)
Outcome
Analysis:
- Genuinely useful quotes
- Genuinely interesting
- Give a good overview of why the source document is worth reading!
Net
I’ve used the 8 bullet points output above as my ‘summary’ for this resource without editing/changing them – post is here: 12 x Principles for building AI-powered software that delivers real-world value to customers – and over time I’ll build up more of my hand-picked resources, hopefully I’ll be able to increasingly automate the ‘describing each resource’ part (even though I still have to do all the hard work of finding + reading + curating each one!)