It is common knowledge nowadays that assistants like Claude or GPT are useful but sometimes produce very convincing but wrong answers. I am aware of this behavior, and I even tried to exploit this for good while still working on educational AI in the early days of GPT-2 (paper). Yet sometimes they still get you…

We were experimenting with a retrieval system for large-scale semantic search utilizing LanceDB. As part of the experiments, a colleague was measuring different open- and closed-source embedding models on an open retrieval benchmark. The models worked well at first, hitting similar numbers to the published ones. Yet, the moment he turned on IVF_RQ quantization, we noticed a severe collapse in retrieval quality, rendering the index completely unusable. Naturally, we first suspected a bug in our own retrieval pipeline but, on review, everything was fine. So, like every modern engineer, my natural next move was to discuss this problem with AI assistants to see if they could figure out what went wrong. Given my problem description, Claude and Gemini answered confidently with mathematical formulas and references to scientific papers, arguing that there is a mathematical property inherent to the embeddings at hand that makes them hard to quantize. Having two “independent” answers and little time to cross-check the math and references, they convinced me that we most likely just had to switch embeddings going forward, using less optimal embeddings for the task.

Thankfully, in the meantime, my excellent colleague stayed skeptical and spent more time on the problem. Finally, he identified the root cause not in any inherent embedding property, but in the LanceDB Rust implementation of the IVF_RQ quantization. For embeddings of larger dimensions, some Rust variable overflowed during quantized distance calculation, destroying distance ordering. Fixing this on his local branch and rebuilding the index using the same settings that had previously been erroneous brought the performance of the quantized index much closer to the performance of the unquantized index.

So, the lesson from this might be unsurprising but worth reiterating. For any decision that has actual consequences, relying on AI judgment trades time saved now for the risk of technical debt or incorrectness. You can often get the best of both worlds by reviewing AI answers for correctness. Yet time pressure sometimes makes trusting plausible answers a very tempting road. So, this is now my personal cautionary tale about the risks this carries. Sometimes slowing down and diving deep into the details is crucial, and one must take a stand to make time for that.