#1424 - Server Max out and Critical Bail out
-
By Lhasadreams
- Added
- 0 views
| Identifier | #1424 |
|---|---|
| Issue type | Major issue (breaks an entire feature) |
| Title | Server Max out and Critical Bail out |
| Status | Closed (cannot reproduce) |
| Handling member | Chris Graham |
| Addon | General / Uncategorised |
| Description | Hi Chris,
At about 3pm today and yesterday our server started consumming high CPU and stopped servicing the the CRON Daemon. End users saw a Critial Error bail out message. This lasted for about 30 minutes and then everything returned to normal. I was wondering if it has anything to do with the extra code that is in the cron job to deal with the unpinning of topics ? During the high CPU I got this: mysql> SHOW FULL PROCESSLIST; +---------+-------------+-----------+------------------+----------------+------+--------------------+-----------------------+ | Id | User | Host | db | Command | Time | State | Info | +---------+-------------+-----------+------------------+----------------+------+--------------------+-----------------------+ | 4785544 | DELAYED | localhost | vwgolfm_cms_live | Delayed insert | 0 | Waiting for INSERT | | | 5231932 | DELAYED | localhost | vwgolfm_cms_live | Delayed insert | 0 | Waiting for INSERT | | | 6062863 | DELAYED | localhost | eximstats | Delayed insert | 1 | Waiting for INSERT | | | 6100531 | root | localhost | vwgolfm_cms_live | Query | 0 | NULL | SHOW FULL PROCESSLIST | | 6101675 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 1 | | NULL | | 6101704 | eximstats | localhost | eximstats | Sleep | 1 | | NULL | | 6101715 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101731 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101737 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101754 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101755 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101763 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101769 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101774 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101778 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101779 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101792 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101793 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101798 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101807 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101808 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101811 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101816 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101817 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101822 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101829 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101830 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101831 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101836 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | | 6101843 | vwgolfm_cms | localhost | vwgolfm_cms_live | Sleep | 0 | | NULL | +---------+-------------+-----------+------------------+----------------+------+--------------------+-----------------------+ 30 rows in set (0.00 sec) Which gradually reduced to this: mysql> SHOW FULL PROCESSLIST; +---------+---------+-----------+------------------+----------------+------+--------------------+-----------------------+ | Id | User | Host | db | Command | Time | State | Info | +---------+---------+-----------+------------------+----------------+------+--------------------+-----------------------+ | 4785544 | DELAYED | localhost | vwgolfm_cms_live | Delayed insert | 6 | Waiting for INSERT | | | 5231932 | DELAYED | localhost | vwgolfm_cms_live | Delayed insert | 14 | Waiting for INSERT | | | 6062863 | DELAYED | localhost | eximstats | Delayed insert | 170 | Waiting for INSERT | | | 6105186 | DELAYED | localhost | eximstats | Delayed insert | 175 | Waiting for INSERT | | | 6105228 | root | localhost | NULL | Query | 0 | NULL | SHOW FULL PROCESSLIST | +---------+---------+-----------+------------------+----------------+------+--------------------+-----------------------+ 5 rows in set (0.00 sec) Those waiting for insert are still there now. Any idea ? |
| Steps to reproduce | |
| Funded? | No |
The system will post a comment when this issue is modified (e.g., status changes). To be notified of this, click "Enable comment notifications".
Comments
Delayed inserts are intentional and normal. We actually do this for performance reasons now, it prevents locks, and hence deadlock situations. Typically these will be stat logging or other similar logging that doesn't need to be kept up-to-date.
Curiously the same Connection identifier seems to be being recycled for different delayed inserts. The time is resetting so I'm pretty sure they are different ones. I'll look into that, it's at least interesting.
I will see if I can find any logged errors.
http://dev.mysql.com/doc/refman/5.5/en/thread-pool-operation.html
i.e. threads are re-used. It's not the same query getting stuck.
I will keep an eye on it. I did capture the critical error and entered it in to a tracker, but i then think I must of hit the wrong button because it never appeared :-( and I didn't keep a copy !!