umer9538
An observation tool that records and compares changes in on device AI models across OS updates
SITE CAPTUREBased on information published on the service's website. Selection and review policy
WHAT IT DOES
What does it do?
It captures responses to the same set of prompts for each OS build and publishes what changed. Each prompt is run multiple times to measure determinism as well. Results are committed as JSON files so the differences can be reproduced and build numbers checked.
KEY FEATURES
Key features
- Capture prompt responses by OS build
- Compare differences between captures
- Measure determinism through repeated runs of each prompt
- Publish results as JSON files
- Classify results as ok, refusal, unsupported-language, context-exceeded, or error
EDITOR'S NOTE
Why did it stand out?
Its approach of fixing the prompt set and run conditions, then preserving the results, stands out as a way to see how the same model changes after an OS update. It compares guardrails and format compliance as well as response changes.