neatlogs recorded all 3 failures I planted. It didn't warn me about any of them.
I connected the AI agent in a product I work on to neatlogs, asked it 100 questions, and planted 3 failures I knew the answers to: a tool that crashed, a tool that froze for 96 seconds, and an AI model name that doesn't exist. Then I checked 12 things a new user would expect.
5 of 12checks passed
- Worked · 5
- Didn't work · 7
Worked
-
Saves every run in detail
Each step, with its input and output. The crash and the wrong model name were marked as errors; the freeze showed up as the 3 timeouts it caused.
-
Lists errors in one place
The Analytics page grouped all 20 errors by message, including the 3 planted failures.
-
neat AI explains a failed run
It found the tool that froze for 96 seconds. I made the change it suggested, and the same run worked.
-
Keeps a conversation together
Two messages from one conversation showed up together, in order.
-
Suggests checks from my own data
It proposed 4 detections based on my runs, such as one for a background job that processed nothing.
Didn't work
-
Warns me when something breaks
20 errors, and no detection fired. The daily digest said "0 of 1 projects need a look". The built-in Tool Failure check flagged none of the 15 failed tool steps, and the Agent Execution Error check only watches an "agent" step, which the setup guide never asks you to add.
-
Shows which runs failed
Failed runs get a green tick in the Traces list (it means "Ready") but a red "Error" in Sessions. Analytics reports a 15.9% error rate; only 2.8% of runs actually failed.
-
Catches slow runs
The suggested over-15-seconds check flagged 5 runs. At least 12 took longer, including the slowest: 103 seconds, and it failed.
-
neat AI answers reliably
It said 4 detections had fired (none had) and that 10 runs held my API key (at least 47 did). 2 of 3 tries to explain one run ended in an error.
-
Keeps secrets safe
My own code leaked an API key into at least 47 runs. Nothing warned me, redaction is off by default, and neat AI printed the full key when I asked about it.
-
Opens pages quickly
Pages took up to 11 seconds to open, while their data arrived in under 2. Details below.
-
Setup guide fits my app
It has no steps for apps that call AI models over plain web requests, which is how mine calls OpenRouter. It also puts my live neatlogs key in the text I paste into a coding agent, while telling that agent never to ask me to paste keys into chat.
Why the pages felt slow
The data arrived fast. Building the page did not.
Seconds to open, measured in Chrome
- Under 2 s
- Over 8 s
Analytics
Traces, first visit
Traces, later visit
- Clicks register in about 0.07 seconds, so the browser isn't the slow part.
- The sign-in page's response headers show it is built on a US server for every request, with caching turned off. From India it takes 0.6 seconds to start loading. I couldn't check the logged-in pages the same way.
- Twice, a tab switch timed out and the app reloaded the whole page instead. The browser console showed 65 to 75 errors after a few clicks.
- A one-line question to neat AI took about a minute to answer.
What I'd fix first
- A built-in check for any failed step, with an alert. I tried one on my data and it caught all 20 errors.
- One clear outcome per run: worked, recovered or failed. Use it in the Traces list and in the error rate.
- API keys redacted by default. And never repeated in neat AI answers.
- Faster page builds. The data is ready in under 2 seconds; the pages take up to 11.