Python

Extract speaker notes from PowerPoint to text

Thu, 09 Oct 2025 11:41:43 -0400

When working on presentations, I like to extract my speaker notes to review the flow and turn them into blog posts. I'm doing this right now for my DrupalCon Vienna talk.

I used to do this manually, but with presentations often having 100+ slides, it gets tedious and isn't very repeatable. So I ended up automating this with a Python script.

Since I use Apple Keynote or Google Slides rather than Microsoft PowerPoint, I first export my presentations to PowerPoint format, then run my Python script.

If you've ever needed to pull speaker notes from a presentation for review, editing or blogging, here is my script and how to use it.

Speaker notes extractor script

Save this code as powerpoint-to-text.py:

#!/usr/bin/env python3
"""Extract speaker notes from PowerPoint presentations to text files."""

import sys
from pathlib import Path
from pptx import Presentation

def extract_speaker_notes(pptx_path: Path) -> tuple[str, int]:
    presentation = Presentation(pptx_path)
    notes_text = []

    for i, slide in enumerate(presentation.slides, 1):
        if slide.notes_slide and slide.notes_slide.notes_text_frame:
            notes = slide.notes_slide.notes_text_frame.text.strip()
            if notes:
                notes_text.append(f"=== Slide {i} ===\n{notes}\n")

    return "\n".join(notes_text), len(notes_text)

def main():
    if len(sys.argv) != 2:
        print("Usage: python powerpoint-to-text.py presentation.pptx")
        sys.exit(1)

    input_path = Path(sys.argv[1])

    if not input_path.exists():
        print(f"Error: File '{input_path}' not found")
        sys.exit(1)

    if input_path.suffix.lower() != '.pptx':
        print(f"Warning: '{input_path}' may not be a PowerPoint file")

    try:
        notes_text, notes_count = extract_speaker_notes(input_path)
    except Exception as e:
        print(f"Error reading presentation: {e}")
        sys.exit(1)

    output_path = input_path.with_suffix('.txt')
    output_path.write_text(notes_text, encoding='utf-8')

    print(f"Extracted {notes_count} slides with notes to {output_path}")

if __name__ == "__main__":
    main()

The script uses the python-pptx library to read PowerPoint files. This library understands the internal structure of .pptx files (which are zip archives containing XML). It provides a clean Python interface to access slides and their speaker notes. The script loops through each slide, checks if it has notes, and writes them to a text file.

Usage

I like to use uv to run Python code. uv is a fast, modern Python package manager that handles dependencies automatically:

$ uv run --with python-pptx powerpoint-to-text.py your-presentation.pptx

This saves a .txt file with your notes in the same directory as the input file, not the current directory or desktop.

The text file contains:

=== Slide 1 ===
Speaker notes from slide 1 ...

=== Slide 3 ===
Speaker notes from slide 3 ...

Only slides with speaker notes are included.

Comparing local LLMs for alt-text generation, round 2

Tue, 27 May 2025 14:04:35 -0400

Four months ago, I tested 10 local vision LLMs and compared them against the top cloud models. Vision models can analyze images and describe their content, making them useful for alt-text generation.

The result? The local models missed important details or introduced hallucinations. So I switched to using cloud models, which produced better results but meant sacrificing privacy and offline capability.

Two weeks ago, Ollama released version 0.7.0 with improved support for vision models. They added support for three vision models I hadn't tested yet: Mistral 3.1, Qwen 2.5 VL and Gemma 3.

I decided to evaluate these models to see whether they've caught up to GPT-4 and Claude 3.5 in quality. Can local models now generate accurate and reliable alt-text?

Model	Provider	Release date	Model size
Gemma 3 (27B)	Google DeepMind	March 2025	27B
Qwen 2.5 VL (32B)	Alibaba	March 2025	32B
Mistral 3.1 (24B)	Mistral AI	March 2025	24B

Updating my `alt`-text script

For my earlier experiments, I created an open-source script that generates alt-text descriptions. The script is a Python wrapper around Simon Willison's llm tool, which provides a unified interface to LLMs. It supports models from Ollama, Hugging Face and various cloud providers.

To test the new models, I added 3 new entries to my script's models.yaml, which defines each model's prompt, temperature, and token settings. Once configured, generating alt-text is simple. Here is an example using the three new vision models:

$ ./caption.py test-images/image-1.jpg –model mistral-3.1-24b gemma3-27b qwen2.5vl-32b

Which outputs something like:

{
  "image": "test-images/image-1.jpg",
  "captions": {
    "mistral-3.1-24b": "A bustling intersection at night filled with pedestrians crossing in all directions."
    "gemma3-27b": "A high-angle view shows a crowded Tokyo street filled with pedestrians and brightly lit advertising billboards at night.",
    "qwen2.5vl-32b": "A bustling city intersection at night, crowded with people crossing the street, surrounded by tall buildings with bright, colorful billboards and advertisements.",
  }
}

Evaluating the models

To keep the results consistent, I used the same test images and the same evaluation method as in my earlier blog post. The details results are in this Google spreadsheet.

Each alt-text was scored from 0 to 5 based on three criteria: how well it identified the most important elements in the image, how effectively it captured the mood or atmosphere, and whether it avoided repetition, grammar issues or hallucinated details. I then converted each score into a letter grade from A to F.

For comparison, the cloud models received the following scores: GPT-4o earned an average of 4.8 out of 5 (grade A), and Claude 3.5 Sonnet received a perfect 5 out of 5 (grade A).

Test image 1: Shibuya Crossing, Tokyo

Model	Description	Grade
Mistral 3.1 (24B)	A bustling intersection at night filled with pedestrians crossing in all directions.	C
Gemma 3 (27B)	A high-angle view shows a crowded Tokyo street filled with pedestrians and brightly lit advertising billboards.	B
Qwen 2.5 VL (32B)	A bustling city intersection at night, crowded with people crossing the street, surrounded by brightly lit billboards and tall buildings.	B

None earned an A because they failed to identify this as Shibuya Crossing in Tokyo, while cloud models do. Mistral also missed the billboards.

Test image 2: Isabella Stewart Gardner Museum, Boston

Model	Description	Grade
Mistral 3.1 (24B)	An ornate wall features a large empty picture frame.	B
Gemma 3 (27B)	An empty, ornate gold frame hangs on a patterned green wall between two framed portraits and a candle sconce.	A
Qwen 2.5 VL (32B)	A vintage-style room features ornate wallpaper, a framed empty canvas, a lit candelabra, and a decorative vase on a table, with portraits on either side.	A

The vision models in my previous post often mistook the empty frame for a framed painting. All three models in this test correctly identified it as empty. Gemma and Qwen captured valuable details about the scene, while Mistral's description felt sparse.

Test image 3: wakeboarding in Vermont, USA

Model	Description	Grade
Mistral 3.1 (24B)	Two shirtless men on a boat watch another person water skiing on a lake.	B
Gemma 3 (27B)	Two people on a boat watch a waterskier speeding across the lake on a sunny day.	B
Qwen 2.5 VL (32B)	Two shirtless men on a boat watch a person water skiing in the distance on a calm lake.	B

All three described a wakeboarding scene as "water skiing", while the cloud models correctly identified it as wakeboarding.

Test image 4: hiking in the Dolomites, Italy

Model	Description	Grade
Mistral 3.1 (24B)	A wooden statue of a saint is mounted on a post with directional signs pointing to various locations.	C
Gemma 3 (27B)	A small wooden shrine with a statue of Mary stands beside a signpost indicating hiking trails in a grassy field.	B
Qwen 2.5 VL (32B)	A wooden shrine with a statue of a figure stands on a tree stump, surrounded by a scenic mountain landscape with directional signs in the foreground.	B

Only Gemma recognized the statue as Mary. Both Mistral and Gemma missed the mountains in the background, which seems important.

Test image 5: backgammon by candlelight

Model	Description	Grade
Mistral 3.1 (24B)	A lit candle and a glass of liquid are on a wooden table next to a wooden board game.	B
Gemma 3 (27B)	A lit candle and glass votive sit on a wooden table, creating a warm, inviting glow in a dimly lit space.	B
Qwen 2.5 VL (32B)	A cozy scene with a lit candle on a wooden table, next to a backgammon board and a glass of liquid, creating a warm and inviting atmosphere.	A

Neither Mistral nor Gemma recognized the backgammon board. Only Qwen identified it correctly. Mistral also failed to capture the photo's mood.

Model accuracy

Model	Repetitions	Hallucinations	Moods	Average score	Grade
Mistral 3.1 (24B)	Never	Never	Fair	3.4/5	C
Gemma 3 (27B)	Never	Never	Good	4.2/5	B
Qwen 2.5 VL (32B)	Never	Never	Good	4.4/5	B

Qwen 2.5 VL performed best overall, with Gemma 3 not far behind.

Needless to say, these results are based on a small set of test images. And while I used a structured scoring system, the evaluation still involves subjective judgment. This is not a definitive ranking, but it's enough to draw some conclusions.

It was nice to say that all three LLMs avoided repetition and hallucinations, and generally captured the mood of the images.

Local models still make mistakes. All three described wakeboarding as "water skiing", most failed to recognize the statue as Mary or place the intersection in Japan. Cloud models get these details right, as I showed in my previous blog post.

Conclusion

I ran my original experiment four months ago, and at the time, none of the models I tested felt accurate enough for large-scale alt-text generation. Some, like Llama 3, showed promise but still fell short in overall quality.

Newer models like Qwen 2.5 VL and Gemma 3 have matched the performance I saw earlier with Llama 3. Both performed well in my latest test. They produced relevant, grounded descriptions without hallucinations or repetition, which earlier local models often struggled with.

Still, the quality is not yet at the level where I would trust these models to generate thousands of alt-texts without human review. They make more mistakes than GPT-4 or Claude 3.5.

My main question was: are local models now good enough for practical use? While Qwen 2.5 VL performed best overall, it still needs human review. I've started using it for small batches where manual checking is manageable. For large-scale, fully automated use, I continue using cloud models as they remain the most reliable option.

That said, local vision-language models continue to improve. My long-term goal is to return to a 100% local-first workflow that gives me more control and keeps my data private. While we're not there yet, these results show real progress.

My plan is to wait for the next generation of local vision models (or upgrade my hardware to run larger models). When those become available, I'll test them and report back.

Automating alt-text generation with AI

Thu, 20 Feb 2025 06:22:29 -0500

Billions of images on the web lack proper alt-text, making them inaccessible to millions of users who rely on screen readers.

My own website is no exception, so a few weeks ago, I set out to add missing alt-text to about 9,000 images on this website.

What seemed like a simple fix became a multi-step challenge. I needed to evaluate different AI models and decide between local or cloud processing.

To make the web better, a lot of websites need to add alt-text to their images. So I decided to document my progress here on my blog so others can learn from it – or offer suggestions. This third post dives into the technical details of how I built an automated pipeline to generate alt-text at scale.

High-level architecture overview

My automation process follows three steps for each image:

Check if alt-text exists for a given image
Generate new alt-text using AI when missing
Update the database record for the image with the new alt-text

The rest of this post goes into more detail on each of these steps. If you're interested in the implementation, you can find most of the source code on GitHub.

Retrieving image metadata

To systematically process 9,000 images, I needed a structured way to identify which ones were missing alt-text.

Since my site runs on Drupal, I built two REST API endpoints to interact with the image metadata:

GET /album/{album-name}/{image-name}/get – Retrieves metadata for an image, including title, alt-text, and caption.
PATCH /album/{album-name}/{image-name}/patch – Updates specific fields, such as adding or modifying alt-text.

I've built similar APIs before, including one for my basement's temperature and humidity monitor. That post provides a more detailed breakdown of how I build endpoints like this.

This API uses separate URL paths (/get and /patch) for different operations, rather than using a single resource URL. I'd prefer to follow RESTful principles, but this approach avoids caching problems, including content negotiation issues in CDNs.

Anyway, with the new endpoints in place, fetching metadata for an image is simple:

curl -H "Authorization: test-token" \
  "https://dri.es/album/isle-of-skye-2024/journey-to-skye/get"

Every request requires an authorization token. And no, test-token isn't the real one. Without it, anyone could edit my images. While crowdsourced alt-text might be an interesting experiment, it's not one I'm looking to run today.

This request returns a JSON object with image metadata:

{
  "title": "Journey to Skye",
  "alt": "",
  "caption": "Each year, Klaas and I pick a new destination for our outdoor adventure. In 2024, we set off for the Isle of Skye in Scotland. This stop was near Glencoe, about halfway between Glasgow and Skye."
}

Because the alt-field is empty, the next step is to generate a description using AI.

Generating and refining `alt`-text with AI

In my first post on AI-generated alt-text, I wrote a Python script to compare 10 different local Large Language Models (LLMs). The script uses PyTorch, a widely used machine learning framework for AI research and deep learning. This implementation was a great learning experience.

The original script takes an image as input and generates alt-text using multiple LLMs:

./caption.py journey-to-skye.jpg
{
  "image": "journey-to-skye.jpg",
  "captions": {
    "vit-gpt2": "A man standing on top of a lush green field next to a body of water with a bird perched on top of it.",
    "git": "A man stands in a field next to a body of water with mountains in the background and a mountain in the background.",
    "blip": "This is an image of a person standing in the middle of a field next to a body of water with a mountain in the background.",
    "blip2-opt": "A man standing in the middle of a field with mountains in the background.",
    "blip2-flan": "A man is standing in the middle of a field with a river and mountains behind him on a cloudy day.",
    "minicpm-v": "A person standing alone amidst nature, with mountains and cloudy skies as backdrop.",
    "llava-13b": "A person standing alone in a misty, overgrown field with heather and trees, possibly during autumn or early spring due to the presence of red berries on the trees and the foggy atmosphere.",
    "llava-34b": "A person standing alone on a grassy hillside with a body of water and mountains in the background, under a cloudy sky.",
    "llama32-vision-11b": "A person standing in a field with mountains and water in the background, surrounded by overgrown grass and trees."
  }
}

My original plan was to run everything locally for full control, no subscription costs, and optimal privacy. But after testing 10 local LLMs, I changed my mind.

I knew cloud-based models would be better, but wanted to see if local models were good enough for alt-texts. Turns out, they're not quite there. You can read the full comparison, but I gave the best local models a B, while cloud models earned an A.

While local processing aligned with my principles, it compromised the primary goal: creating the best possible descriptions for screen reader users. So I abandoned my local-only approach and decided to use cloud-based LLMs.

To automate alt-text generation for 9,000 images, I needed programmatic access to cloud models rather than relying on their browser-based interfaces – though browser-based AI can be tons of fun.

Instead of expanding my script with cloud LLM support, I switched to Simon Willison's llm tool: https://llm.datasette.io/. llm is a command-line tool and Python library that supports both local and cloud-based models. It takes care of installation, dependencies, API key management, and uploading images. Basically, all the things I didn't want to spend time maintaining myself.

Despite enjoying my PyTorch explorations with vision language models and multimodal encoders, I needed to focus on results. My weekly progress goal meant prioritizing working alt-text over building homegrown inference pipelines.

I also considered you, my readers. If this project inspires you to make your own website more accessible, you're better off with a script built on a well-maintained tool like llm rather than trying to adapt my custom implementation.

Scrapping my PyTorch implementation stung at first, but building on a more mature and active open-source project was far better for me and for you. So I rewrote my script, now in the v2 branch, with the original PyTorch version preserved in v1.

The new version of my script keeps the same simple interface but now supports cloud models like ChatGPT and Claude:

./caption.py journey-to-skye.jpg --model chatgpt-4o-latest claude-3-sonnet --context "Location: Glencoe, Scotland"
{
  "image": "journey-to-skye.jpg",
  "captions": {
    "chatgpt-4o-latest": "A person in a red jacket stands near a small body of water, looking at distant mountains in Glencoe, Scotland.",
    "claude-3-sonnet": "A person stands by a small lake surrounded by grassy hills and mountains under a cloudy sky in the Scottish Highlands."
  }
}

The --context parameter improves alt-text quality by adding details the LLM can't determine from the image alone. This might include GPS coordinates, album titles, or even a blog post about the trip.

In this example, I added "Location: Glencoe, Scotland". Notice how ChatGPT-4o mentions Glencoe directly while Claude-3 Sonnet references the Scottish Highlands. This contextual information makes descriptions more accurate and valuable for users. For maximum accuracy, use all available information!

Updating image metadata

With alt-text generated, the final step is updating each image. The PATCH endpoint accepts only the fields that need changing, preserving other metadata:

curl -X PATCH \
  -H "Authorization: test-token" \
  "https://dri.es/album/isle-of-skye-2024/journey-to-skye/patch" \
  -d '{
    "alt": "A person stands by a small lake surrounded by grassy hills and mountains under a cloudy sky in the Scottish Highlands.",
  }'

That's it. This completes the automation loop for one image. It checks if alt-text is needed, creates a description using a cloud-based LLM, and updates the image if necessary. Now, I just need to do this about 9,000 times.

Tracking AI-generated `alt`-text

Before running the script on all 9,000 images, I added a label to the database that marks each alt-text as either human-written or AI-generated. This makes it easy to:

Re-run AI-generated descriptions without overwriting human-written ones
Upgrade AI-generated alt-text as better models become available

With this approach I can update the AI-generated alt-text when ChatGPT 5 is released. And eventually, it might allow me to return to my original principles: to use a high-quality local LLM trained on public domain data. In the mean time, it helps me make the web more accessible today while building toward a better long-term solution tomorrow.

Next steps

Now that the process is automated for a single image, the last step is to run the script on all 9,000. And honestly, it makes me nervous. The perfectionist in me wants to review every single AI-generated alt-text, but that is just not feasible. So, I have to trust AI. I'll probably write one more post to share the results and what I learned from this final step.

Stay tuned.

Comparing local large language models for alt-text generation

Mon, 03 Feb 2025 11:45:10 -0500

I have 10,000 photos on my website. About 9,000 have no alt-text. I'm not proud of that, and it has bothered me for a long time.

When I started my blog nearly 20 years ago, I didn't think much about alt-texts. Over time, I realized its importance for visually impaired users who rely on screen readers.

The past 5+ years, I diligently added alt-text to every new image I uploaded. But that only covers about 1,000 images, leaving most older photos without descriptions.

Writing 9,000 alt-texts manually would take ages. Of course, AI could do this much faster, but is it good enough?

To see what AI can do, I tested 12 Large Language Models (LLMs): 10 running locally and 2 in the cloud. My goal was to test their accuracy and determine whether they can generate accurate alt-text.

The TL;DR is that, not surprisingly, cloud models (GPT-4, Claude Sonnet 3.5) set the benchmark with A-grade performance, though not 100% perfect. I prefer local models for privacy, cost, and offline use. Among local options, the Llama variants and MiniCPM-V perform best. Both earned a B grade: they work reliably but sometimes miss important details.

I know I'm not the only one. Plenty of people – entire organizations even – have massive backlogs of images without alt-text. I'm determined to fix that for my blog and share what I learn along the way. This blog post is just step one – subscribe by email or RSS to get future posts.

Models evaluated

I tested alt-text generation using 12 AI models: 9 on my MacBook Pro with 32GB RAM, 1 on a higher-RAM machine (thanks to Jeremy Andrews, a friend and long-time Drupal contributor), and 2 cloud-based services.

The table below lists the models I tested, with details like links to research papers, release dates, parameter sizes (in billions), memory requirements, some architectural details and more:

	Model	Launch date	Type	Vision encoder	Language encoder	Model size (billions of parameters)	RAM	Deployment
1	VIT-GPT2	2021	Image-to-text	ViT (Vision Transformer)	GPT-2	0.4B	~8GB	Local, Dries
2	Microsoft GIT	2022	Image-to-text	Swin Transformer	Transformer Decoder	1.2B	~8GB	Local, Dries
3	BLIP Large	2022	Image-to-text	ViT	BERT	0.5B	~8GB	Local, Dries
4	BLIP-2 OPT	2023	Image-to-text	CLIP ViT	OPT	2.7B	~8GB	Local, Dries
5	BLIP-2 FLAN-T5	2023	Image-to-text	CLIP ViT	FLAN-T5 XL	3B	~8GB	Local, Dries
6	MiniCPM-V	2024	Multi-modal	SigLip-400M	Qwen2-7B	8B	~16GB	Local, Dries
7	LLaVA 13B	2024	Multi-modal	CLIP ViT	Vicuna 13B	13B	~16GB	Local, Dries
8	LLaVA 34B	2024	Multi-modal	CLIP ViT	Vicuna 34B	34B	~32GB	Local, Dries
9	Llama 3.2 Vision 11B	2024	Multi-modal	Custom Vision Encoder	Llama 3.2	11B	~20GB	Local, Dries
10	Llama 3.2 Vision 90B	2024	Multi-modal	Custom Vision Encoder	Llama 3.2	90B	~128GB	Local, Jeremy
11	OpenAI GPT-4o	2023	Multi-modal	Custom Vision Encoder	GPT-4	>150B		Cloud
12	Anthropic Claude 3.5 Sonnet	2024	Multi-modal	Custom Vision Encoder	Claude 3.5	>150B		Cloud

How image-to-text models work (in less than 30 seconds)

LLMs come in many forms, but for this project, I focused on image-to-text and multi-modal models. Both types of models can analyze images and generate text, either by describing images or answering questions about them.

Image-to-text models follow a two-step process: vision encoding and language decoding:

Vision encoding: First, the model breaks an image down into patches. You can think of these as "puzzle pieces". The patches are converted into mathematical representations called embeddings, which summarize their visual details. Next, an attention mechanism filters out the most important patches (e.g. the puzzle pieces with the cat's outline or fur texture) and eliminates less relevant details (e.g. puzzle pieces with plain blue skies).
Language encoding: Once the model has summarized the most important visual features, it uses a language model to translate those features into words. This step is where the actual text (image captions or Q&A answers) is generated.

In short, the vision encoder sees the image, while the language encoder describes it.

If you look at the table above, you'll see that each row pairs a vision encoder (e.g., ViT, CLIP, Swin) with a language encoder (e.g., GPT-2, BERT, T5, Llama).

For a more in-depth explanation, I recommend Sebastian Raschka's article Understanding Multi-modal LLMs, which also covers how image encoders work. It's fantastic!

Comparing different AI models

I wrote a Python script that generates alt-texts for images using nine different local models. You can find it in my GitHub repository. It takes care of installing models, running them, and generating alt-texts. It supports both Hugging Face and Ollama and is built to be easily extended as new models come out.

You can run the script as follows:

$ ./caption.py ./test-images/image-1.jpg

The first time you run the script, it will download all models, which requires significant disk space and bandwidth – expect to download over 50GB of model data.

The script outputs a JSON response, making it easy to integrate or analyze programmatically. Here is an example output:

  {
  "image": "test-images/image-1.jpg",
  "alt-texts": {
  "vit-gpt2": "A city at night with skyscrapers and a traffic light on the side of the street in front of a tall building.",
  "git": "A busy city street is lit up at night, with the word qroi on the right side of the sign.",
  "blip": "This is an aerial view of a busy city street at night with lots of people walking and cars on the side of the road.",
  "blip2-opt": "An aerial view of a busy city street at night.",
  "blip2-flan": "An aerial view of a busy street in tokyo, japanese city at night with large billboards.",
  "minicpm-v": "A bustling cityscape at night with illuminated billboards and advertisements, including one for Michael Kors.",
  "llava-13b": "A bustling nighttime scene from Tokyo's famous Shibuya Crossing, characterized by its bright lights and dense crowds of people moving through the intersection.",
  "llava-34b": "A bustling city street at night, filled with illuminated buildings and numerous pedestrians.",
  "llama32-vision-11b": "A bustling city street at night, with towering skyscrapers and neon lights illuminating the scene."
  }
  }

Test images

With the script ready, I decided to test it on some of my 10,000 photos. Not all of them at once. I picked five that I consider non-standard. Instead of simple portraits or landscapes, I picked photos with elements that might confuse or challenge the models.

One photo is from the Isabella Stewart Gardner Museum in Boston and features an empty gold frame. The frame once held a masterpiece stolen in the infamous 1990 heist, one of the biggest art thefts in history. I wanted to see if the models would recognize it as empty or mistake it for a framed painting.

Another photo, taken last summer in Vermont, shows a wakeboarder. Though he is the main subject, he is relatively small in the frame. I was curious to see if the models could still recognize him as the focal point.

In another photo, a backgammon game is set in a dark but cozy atmosphere. I was curious to see if the models could recognize partially visible objects and capture the mood of the scene.

To ensure a fair test, I stripped all EXIF metadata from the images. This includes any embedded captions, GPS coordinates, or other details that could inadvertently help the models.

Yes, I know that a test set of five images is small, but it's sufficient to identify the top models for further evaluation. With 12 models generating alt-texts for each photo, I had to manually evaluate 60 alt-texts, which was tedious to do fairly and accurately. For now, these five images are enough to filter out weaker models and pinpoint the best 2-3 LLMs for more detailed testing.

Next, I'll share the detailed results for each of the five photos. It's a lot to go through, so feel free to skip ahead to the summary.

Test image 1: Shibuya Crossing, Tokyo

Benchmark LLMs:

Model	Description	Grade
GPT-4o	A vibrant city intersection with crowds crossing, surrounded by illuminated billboards and towering buildings, likely Shibuya, Tokyo.	A
Claude	Hundreds of people cross Tokyo's famous Shibuya intersection as neon signs light up the surrounding skyscrapers.	A

Local LLMs:

Model	Description	Grade
VIT-GPT2	A city at night with skyscrapers and a traffic light on the side of the street in front of a tall building.	F
GIT	A busy city street is lit up at night, with the word qroi on the right side of the sign.	E
BLIP	This is an aerial view of a busy city street at night with lots of people walking and cars on the side of the road.	D
BLIP-2 with OPT	An aerial view of a busy city street at night.	E
BLIP-2 with FLAN	An aerial view of a busy street in Tokyo, Japanese city at night with large billboards.	E
LLaVA (13B)	A bustling nighttime scene of Tokyo's Shibuya Crossing, with neon lights and people walking across the intersection.	A
LLaVA (34B)	A bustling city street at night, with illuminated buildings and numerous pedestrians.	C
MiniCPM-V	A bustling cityscape at night with various advertisements, including one for Michael Kors and another for Yakult.	C
Llama 3.2 Vision (11B)	A bustling city street at night, with numerous billboards and advertisements lining the buildings.	C
Llama 3.2 Vision (90B)	A bustling city street with many people and buildings.	D

Test image 2: Isabella Stewart Gardner Museum, Boston

Benchmark LLMs:

Model	Description	Grade
GPT-4o	Elegant wallpaper, an empty gold frame, candelabra, and framed painting in a classic room.	B
Claude	Ornate Victorian room with damask wallpaper, candelabra, and gold frames on wall.	B

Local LLMs:

Model	Description	Grade
VIT-GPT2	A living room with a mirror, candles, and a vase of flowers on a table in front of a mirror.	F
GIT	A picture frame is hanging on a wall next to a vase and a vase with the word tulips on it.	E
BLIP	A room with a painting on the wall and two vases on the table in front of it.	E
BLIP-2 with OPT	A room with a painting on the wall, a picture frame, and a chandelier.	C
BLIP-2 with FLAN	A room with a painting on the wall and a couple of framed pictures hanging on the wall next to it.	D
LLaVA (13B)	An empty wall with ornate decorations, including candlesticks and framed artwork, suggesting a formal or historical setting.	D
LLaVA (34B)	An ornate room with framed pictures on the wall, a chandelier hanging from the ceiling, and a fireplace mantel adorned with decorative items.	E
MiniCPM-V	A room with ornate wallpaper, candlesticks and framed portraits of historical figures is displayed.
Llama 3.2 Vision (11B)	An empty gold frame on a wall with ornate wallpaper, surrounded by other decorative items.	A
Llama 3.2 Vision (90B)	An ornate room with a gold frame, a painting of a man, and a vase on a table.	B

Test image 3: wakeboarding in Vermont, USA

Benchmark LLMs:

Model	Description	Grade
GPT-4o	Two shirtless boys on a boat watch a person wakeboarding in the distance on a cloudy day.	A
Claude	People watching from a boat as a person is wakeboarding on a lake with a wooded shoreline.	B

Local LLMs:

Model	Description	Grade
VIT-GPT2	A man standing on top of a boat next to another man holding a surfboard in one hand and a surfboard in the other.	E
GIT	Two men are in a boat, one of them is wearing an orange hat and the other is wearing an orange hat.	D
BLIP	There are two men riding on the back of a boat in the water, one of them is on a surfboard and the other is on a board.	D
BLIP-2 with OPT	Three young men sitting on the back of a boat.	D
BLIP-2 with FLAN	A group of people sitting on a boat watching a man ride a surfboard in the middle of the water.	C
LLaVA (13B)	Two shirtless men on a boat, watching another man surfing in the water.	B
LLaVA (34B)	Two shirtless men on a boat, watching another person water skiing.	B
MiniCPM-V	Two shirtless men on a boat watch as another person water skis away from them, with trees and sky visible across the lake behind him.	B
Llama 3.2 Vision (11B)	Two shirtless men on a boat watching people wakeboarding or surfing behind it.	B
Llama 3.2 Vision (90B)	Two men on a boat watching another man wakeboarding.	A

Test image 4: hiking in the Dolomites, Italy

Benchmark LLMs:

Model	Description	Grade
GPT-4o	A wooden shrine with a statue of Mary and directional signs, set against mountains and green hills.	A
Claude	Wooden shrine of St. Magdalena with trail signs in the Dolomites mountains of Italy.	A

Local LLMs:

Model	Description	Grade
VIT-GPT2	A bird perched on top of a wooden post in front of a mountain range with a view of a mountain range and mountains.	F
GIT	A wooden sign in front of a mountain with a statue on it that says "Meise museum" on the top.	D
BLIP	There is a birdhouse with a statue on top of it in the middle of a field with trees and mountains in the background.	D
BLIP-2 with OPT	A statue of Mary on a wooden post in the middle of a field with mountains in the background.	B
BLIP-2 with FLAN	A statue of the Virgin Mary sits on top of a wooden post in the middle of a mountain.	C
LLaVA (13B)	A picturesque scene of a small wooden chapel with a statue of Jesus, set against a backdrop of majestic mountains and lush greenery.	C
LLaVA (34B)	A wooden shrine with religious iconography, surrounded by alpine scenery and signposts.	B
MiniCPM-V	A wooden shrine with a statue of Mary and several directional signs pointing to various locations such as Kirchsteig, Magdalena, St.	B
Llama 3.2 Vision (11B)	A wooden shrine with a statue of Mary and a signpost in front of mountains.	A
Llama 3.2 Vision (90B)	A statue of Mary in a wooden shrine with a signpost pointing to various locations, including Rundweg St.	B

Test image 5: backgammon by candlelight

Benchmark LLMs:

Model	Description	Grade
GPT-4o	A cozy, dimly lit room with a candle on a wooden table, next to a backgammon board, creating a warm, rustic ambiance.	B
Claude	Two candles light up a game board for backgammon on a wooden table at night.	A

Local LLMs:

Model	Description	Grade
VIT-GPT2	A candle is lit on a wooden table in front of a fire place with candles and other items on top of it.	E
GIT	Two candles are lit next to each other on a table, one of them is lit up and the other is lit up.	D
BLIP	There is a lit candle sitting on top of a wooden table next to a game board and a glass of wine on the table.	D
BLIP-2 with OPT	A candle sits on top of a wooden table.	C
BLIP-2 with FLAN	A candle sits on a wooden table next to a backgammon board and a glass of wine.	D
LLaVA (13B)	A cozy indoor setting with candles, a guitar, and a warm ambiance.	D
LLaVA (34B)	A candlelit wooden table with musical instruments and a cozy ambiance.	C
MiniCPM-V	A dimly lit room with candles and backgammon pieces on a wooden table, creating an atmosphere of relaxation or leisure activity.	A
Llama 3.2 Vision (11B)	A dimly lit room with a wooden table, featuring a backgammon board and two candles.	A
Llama 3.2 Vision (90B)	A candle and backgammon board on a wooden table.	B

Model accuracy

I evaluated each description using a structured but subjective scoring system. For each image, I identified the two or three most important objects the AI should recognize and include in its description. I also assessed whether the model captured the photo's mood, which can be important for visually impaired users. Finally, I deducted points for repetition, grammar errors, or hallucinations (invented details). Each alt-text received a score from 0 to 5, which I then converted to a letter grade from A to F.

Model	Repetitions	Hallucinations	Moods	Average score	Grade
VIT-GPT2	Often	Often	Poor	0.4/5	F
GIT	Often	Often	Poor	1.6/5	D
BLIP	Often	Often	Poor	1.8/5	D
BLIP2 w/OPT	Rarely	Sometimes	Fair	2.6/5	C
BLIP2 w/FLAN	Rarely	Sometimes	Fair	2.2/5	D
LLaVA 13B	Never	Sometimes	Good	3.2/5	C
LLaVA 34B	Never	Sometimes	Good	3.2/5	C
MiniCPM-V	Never	Never	Good	3.8/5	B
Llama 11B	Never	Rarely	Good	4.4/5	B
Llama 90B	Never	Rarely	Good	3.8/5	B
GPT-4o	Never	Never	Good	4.8/5	A
Claude 3.5 Sonnet	Never	Never	Good	5/5	A

The cloud-based models, GPT-4o and Claude 3.5 Sonnet, performed nearly perfectly on my small test of five images, with no major errors, hallucinations, repetitions and excellent mood detection.

Among local models, both Llama variants and MiniCPM-V show the strongest performance.

Repetition in descriptions frustrates users of screen readers. Early models like VIT-GPT2, GIT, BLIP, and BLIP2 frequently repeat content, making them unsuitable.

Hallucinations can be a serious issue in my opinion. Describing nonexistent objects or actions misleads visually impaired users and erodes trust. Among the best-performing local models, MiniCPM-V did not hallucinate, while Llama 11B and Llama 90B each made one mistake. Llama 90B misidentified a cabinet at the museum as a table, and Llama 11B described multiple people wakeboarding instead of just one. While these errors aren't dramatic, they are still frustrating.

Capturing mood is essential for giving visually impaired users a richer understanding of images. While early models struggled in this area, all recent models all performed well. This includes both LLaVA variants and MiniCPM-V.

From a practical standpoint, Llama 11B and MiniCPM-V ran smoothly on my 32GB RAM laptop, but Llama 90B needed more memory. Long story short, this means that Llama 11B and MiniCPM-V are my best candidates for additional testing.

Possible next steps

The results raise a tough question: is a "B"-level alt-text better than none at all? Many human-written alt-texts probably aren't perfect either. Should I wait for local models to hit an "A"-grade, or is an imperfect description still better than no alt-text at all?

Here are four possible next steps:

Combine AI outputs – Run the same image through different models and merge their results to try and create more accurate descriptions.
Wait and upgrade – Use the best local model for now, tag AI-generated alt-texts in the database, and refresh them in 6–12 months when new and better local models are available.
Go cloud-based – Get the best quality with a cloud model, even if it means uploading 65GB of photos. I can't explain why, or if the feeling is even justified, but it feels like giving in.
Hybrid approach – Use AI to generate alt-texts but review them manually. With 9,000 images, that is not practical. I'd need a way to flag alt-texts most likely to be wrong. Can LLMs give me a reliably confidence score?

Each option comes with trade-offs. Some options are quick but imperfect, others take work but might be worth it. Going cloud-based is the easiest but it feels like giving in. Waiting for better models is effortless but means delaying progress. Merging AI outputs or assigning a confidence score takes more effort but might be the best balance of speed and accuracy.

Maybe the solution is a combination of these options? I could go cloud-based now, tag the AI-generated alt-texts in my database, and regenerate them in 6–12 months when LLMs got even better.

It also comes down to pragmatism versus principle. Should I stick to local models because I believe in data privacy and Open Source, or should I prioritize accessibility by providing the best possible alt-text for users? The local-first approach better aligns with my values, but it might come at the cost of a worse experience for visually impaired users.

I'll be weighing these options over the next few weeks. What would you do? I'd love to hear your thoughts!

Update: My thoughts on using AI for alt-text has evolved across several blog posts. First, I chose a cloud-based LLM after all. Then, I built an automated system to generate and update descriptions for just one image. Finally, I scaled it to 9,000 images and learned to trust AI in the process.

Python wrapper for Mollom

Fri, 09 May 2008 03:04:10 -0400

Andy Georges released a Python wrapper for Mollom. The wrapper can be used to integrate Mollom in your Python applications, but it also gets Mollom one step closer to the Django project and Google App Engine.

The Mollom API was released less than 10 days ago, and already Mollom is supported on PHP, Java, Python and Ruby. Sweet!

Python

Extract speaker notes from PowerPoint to text

Speaker notes extractor script

Usage

Comparing local LLMs for alt-text generation, round 2

Updating my alt-text script

Evaluating the models

Test image 1: Shibuya Crossing, Tokyo

Test image 2: Isabella Stewart Gardner Museum, Boston

Test image 3: wakeboarding in Vermont, USA

Test image 4: hiking in the Dolomites, Italy

Test image 5: backgammon by candlelight

Model accuracy

Conclusion

Automating alt-text generation with AI

High-level architecture overview

Retrieving image metadata

Generating and refining alt-text with AI

Updating image metadata

Tracking AI-generated alt-text

Next steps

Comparing local large language models for alt-text generation

Models evaluated

How image-to-text models work (in less than 30 seconds)

Comparing different AI models

Test images

Test image 1: Shibuya Crossing, Tokyo

Test image 2: Isabella Stewart Gardner Museum, Boston

Test image 3: wakeboarding in Vermont, USA

Test image 4: hiking in the Dolomites, Italy

Test image 5: backgammon by candlelight

Model accuracy

Possible next steps

Python wrapper for Mollom

Updating my `alt`-text script

Generating and refining `alt`-text with AI

Tracking AI-generated `alt`-text