A while back i wrote a small data aggregator that would do some background processing and populate a database with it’s calculated results. The main reason to implement this was that running the queries in real-time on the production server was causing memory exhaust issues. Once the new feature was done and deployed. We quickly noticed we were running into the same kind of issues again. And the script would halt almost each time it was executed.
Fatal error: Allowed memory size of xxxxxxxxx bytes exhausted (tried toallocate xxxxx bytes) in
After some hacking and optimization tricks. The script’s memory consumption seemed stable enough to deploy it to production. However. Each time we deployed it. We were hit by memory exhaust issues almost instantly. Time to do some debugging and code tracing.
And we’re able to narrow it down to a single mysql* (don’t get me started on the mysql* functions, so let’s ignore that for now) call. Namely mysql_fetch_field. Why on earth would this generate a memory exhaust issue on the server you might think. And that’s exactly what we were thinking.
The problem was. That the offset passed in as the field offset was incorrect. And resulted in a notice. These notices however were suppressed with the @ symbol. And therefor never noticeable during development. But that’s no reason for memory exhaust issues… right?
Right! Further inspection showed that in production PHP’s APM extension was loaded and active. And this was the real problem behind the memory exhaust issue. Thousands of queries were executed. And almost all of them were throwing an error notice for the fetch field function. Which was ignored by the @ symbol. But APM was still logging them. And apparently had a very hard time keeping up.
It looks like a bug in APM. But that’s another story. The issue was fixed by passing in the correct offset of course. Another nice example of logging bringing down a production server.