Thread: 503 error

We checked the server's logs and found that the server was overloaded which caused the reported errors. Here are some of the latest log records:
Code:

Sep 08 07:29:16 c21148: High load (25.53) reached!
Sep 08 07:29:16 c21148: Critical load (25.53) reached! Killing all php,ftp,smtp,mailnull,archivers procs!
Sep 08 07:29:18 c21148: High load (25.53) reached!
Sep 08 07:29:18 c21148: Critical load (25.53) reached! Killing all php,ftp,smtp,mailnull,archivers procs!
Sep 08 07:29:21 c21148: High load (22.87) reached!
Sep 08 07:29:21 c21148: Critical load (22.87) reached! Killing all php,ftp,smtp,mailnull,archivers procs!
Sep 08 07:29:23 c21148: High load (22.87) reached!
Sep 08 07:29:23 c21148: Critical load (22.87) reached! Killing all php,ftp,smtp,mailnull,archivers procs!
Sep 08 07:29:25 c21148: High load (22.87) reached!
Sep 08 07:29:25 c21148: Critical load (22.87) reached! Killing all php,ftp,smtp,mailnull,archivers procs!
Sep 08 07:29:27 c21148: High load (15.06) reached!
Sep 08 07:29:29 c21148: High load (15.06) reached!
Sep 08 07:29:31 c21148: High load (15.06) reached!
Sep 08 07:30:34 c21148: Killed long proc(3553) owner(asdasd91) :: /usr/local/php53/bin/php index.php
Sep 08 07:34:21 c21148: Killed long proc(28417) owner(asdasd91) :: /usr/local/php53/bin/php index.php
Sep 08 07:44:54 c21148: Killed long proc(30562) owner(asdasd91) :: /usr/local/php53/bin/php index.php
Sep 08 07:44:54 c21148: Killed long proc(2124) owner(asdasd91) :: /usr/local/php53/bin/php index.php
Sep 08 07:44:54 c21148: Killed long proc(4284) owner(asdasd91) :: /usr/local/php53/bin/php index.php
Sep 08 07:44:54 c21148: Killed long proc(4315) owner(asdasd91) :: /usr/local/php53/bin/php index.php
Sep 08 07:45:07 c21148: Killed long proc(15746) owner(asdasd91) :: /usr/local/php53/bin/php index.php
Sep 08 07:45:11 c21148: Killed long proc(16576) owner(asdasd91) :: /usr/local/php53/bin/php index.php


I generated a full report for asdasd91 account and I see that the CPU usage if this account is very high. Here is the report:
Code:

=== Top 10 Executed Scripts ================================================== ===================================
Count Script Local Path
----- --------------------------------------------- -----------------------------------------------------------
3335 http://matlob.co.il/index.php /home/asdasd91/public_html/matlob/index.php
1905 http://matlob.co.il/wp-cron.php /home/asdasd91/public_html/matlob/wp-cron.php
100 http://matlob.co.il/wp-admin/admin-ajax.php /home/asdasd91/public_html/matlob/wp-admin/admin-ajax.php
18 http://matlob.co.il/wp-admin/edit.php /home/asdasd91/public_html/matlob/wp-admin/edit.php
10 http://matlob.co.il/wp-admin/load-scripts.php /home/asdasd91/public_html/matlob/wp-admin/load-scripts.php
10 http://matlob.co.il/wp-admin/post.php /home/asdasd91/public_html/matlob/wp-admin/post.php
-----------------------------------------------------------------------------------------------------------------

=== Top 10 IP Addresses ================================================== ================================================== =======
Hits IP Address UserAgent
---- --------------- ------------------------------------------------------------------------------------------------------------
423 37.60.238.81 WordPress/4.6.1; https://matlob.co.il
296 41.142.101.52 Mozilla/5.0 (Windows NT 6.1; rv:48.0) Gecko/20100101 Firefox/48.0
251 87.70.49.108 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWeb ...... ML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
212 185.97.127.81 Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_4 like Mac ...... ko) GSA/18.0.130791545 Mobile/13G35 Safari/600.1.4
171 176.13.11.65 Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_2 like Mac ...... like Gecko) Version/9.0 Mobile/13F69 Safari/601.1
169 185.120.125.128 Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_3 like Mac ...... like Gecko) Version/9.0 Mobile/13G34 Safari/601.1
141 86.108.124.198 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (K ...... ML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
140 109.66.2.78 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (K ...... ML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
136 89.139.149.68 Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_5 like Mac ...... leWebKit/601.1.46 (KHTML, like Gecko) Mobile/13G36
127 109.65.11.216 Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_5 like Mac ...... like Gecko) Version/9.0 Mobile/13G36 Safari/601.1
-----------------------------------------------------------------------------------------------------------------------------------

=== Last 24 hours - Hive Statistics ================================================== ===========================================
| Script Statistics | Cron Statistics | MySQL Statistics
Date From | Exec CPU MEM RIO WIO AVG | Exec CPU MEM RIO WIO AVG | Exec CPU AVG
---------------- - ------ -------- --------- -------- ------- ----- - ------ ----- ----- ----- ----- ----- - ------ ----- -----
2016-09-07 10:00 | 1168 32110.6 262.3 GB 348.0 KB 96.0 KB 27.5 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-07 11:00 | 1147 30892.1 249.8 GB 260.0 KB 96.0 KB 26.9 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-07 12:00 | 1069 22756.1 230.9 GB 2.7 MB 1.2 MB 21.3 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-07 13:00 | 1110 37700.7 255.0 GB 856.0 KB 8.0 KB 34.0 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-07 14:00 | 1218 42618.4 274.2 GB 724.0 KB 8.0 KB 35.0 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-07 15:00 | 1206 66480.2 288.9 GB 32.0 KB 0 B 55.1 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-07 16:00 | 1616 41315.4 373.3 GB 4.0 KB 0 B 25.6 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-07 17:00 | 1122 27614.2 249.3 GB 0 B 0 B 24.6 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-07 18:00 | 898 25876.5 198.1 GB 0 B 0 B 28.8 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-07 19:00 | 676 19830.9 148.7 GB 16.0 KB 0 B 29.3 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-07 20:00 | 837 17932.9 177.8 GB 72.0 KB 72.0 KB 21.4 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-07 21:00 | 735 9824.8 159.3 GB 0 B 0 B 13.4 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-07 22:00 | 1242 14564.6 279.5 GB 4.0 KB 0 B 11.7 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-07 23:00 | 703 12438.9 151.5 GB 0 B 0 B 17.7 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-08 00:00 | 871 13428.2 193.4 GB 8.0 KB 0 B 15.4 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-08 01:00 | 1234 28112.6 278.6 GB 0 B 0 B 22.8 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-08 02:00 | 1721 21902.4 388.0 GB 16.0 KB 0 B 12.7 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-08 03:00 | 1243 23354.6 279.6 GB 40.0 KB 0 B 18.8 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-08 04:00 | 1518 33104.1 345.5 GB 84.0 KB 16.0 KB 21.8 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-08 05:00 | 1211 40995.2 279.4 GB 92.0 KB 12.0 KB 33.9 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-08 06:00 | 1156 43221.1 264.3 GB 8.0 KB 0 B 37.4 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-08 07:00 | 1193 35406.7 266.3 GB 4.0 KB 0 B 29.7 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-08 08:00 | 1188 32490.5 270.5 GB 664.0 KB 0 B 27.3 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
2016-09-08 09:00 | 923 37095.1 202.2 GB 28.0 KB 0 B 40.2 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
Total | 27005 711066.8 6066.3 GB 5.9 MB 1.5 MB 26.3 | 0 0.0 0 B 0 B 0 B 0.0 | 0 0.0 0.0
---------------------------------------------------------------------------------------------------------------------------------


I have then checked the running processes and noticed that the high CPU usage is caused by the MySQL service. Currently there are 3 slow MySQL processes in the memory generated by the asdasd91_d1 database waiting for for more than 1 minute to complete their request.

I have checked the MySQL slow queries log and found that the website installed within /home/asdasd91/public_html/matlob folder that use asdasd91_d1 database generates these slow queries.

Below is an excerpt of some of the slow queries generated by this website:
Code:

Executed 18h 30m 14s ago for 70.098332 sec on Database --> asdasd91_d1
Date: 2016-09-07 14:44:41 Query_time: 70.098332 Rows_examined: 26659: Rows_sent 79 Lock_time: 0.001667
# Schema: asdasd91_d1 Last_errno: 1160 Killed: 0 SELECT post_id FROM ( SELECT post_id, ( 6371 * acos( cos( radians(32.699635) ) * cos( radians(lat) ) * cos( radians(lng) - radians(35.303546) ) + sin( radians(32.699635) ) * sin( radians(lat) ) ) ) AS distance, post_date FROM ( SELECT latitude.post_id, lat, lng, latitude.post_date FROM ( SELECT post_id, meta_value lat, post_date FROM wp_posts LEFT JOIN wp_postmeta ON wp_posts.ID = wp_postmeta.post_id WHERE meta_key = '_jr_geo_latitude' AND wp_posts.post_status = 'publish' ) latitude, ( SELECT post_id, meta_value lng, post_date FROM wp_posts LEFT JOIN wp_postmeta ON wp_posts.ID = wp_postmeta.post_id WHERE meta_key = '_jr_geo_longitude' AND wp_posts.post_status = 'publish' ) longitude WHERE latitude.post_id = longitude.post_id AND latitude.post_id IN ( SELECT wp_postmeta.post_id FROM wp_postmeta WHERE meta_key = 'geo_short_address' AND meta_value <> '') ) as geo_data HAVING distance < 5 UNION ( SELECT ID, 999999999 as distance, post_date FROM wp_posts WHERE ( ID NOT IN( SELECT wp_postmeta.post_id FROM wp_postmeta WHERE meta_key = 'geo_short_address' ) OR ID IN ( SELECT wp_postmeta.post_id FROM wp_postmeta WHERE meta_key = 'geo_short_address' AND meta_value = '' ) ) AND wp_posts.post_status = 'publish' AND post_type = 'job_listing' ORDER BY post_date DESC ) ORDER BY COALESCE(distance, 999999999) ASC, post_date DESC ) as jobs;
------------------------------------------------------------------------------------------------------
Executed 3h 30m 32s ago for 67.077373 sec on Database --> asdasd91_d1
Date: 2016-09-08 05:44:23 Query_time: 67.077373 Rows_examined: 26643: Rows_sent 74 Lock_time: 0.001246
# Schema: asdasd91_d1 Last_errno: 1160 Killed: 0 SELECT post_id FROM ( SELECT post_id, ( 6371 * acos( cos( radians(32.517127) ) * cos( radians(lat) ) * cos( radians(lng) - radians(35.148529) ) + sin( radians(32.517127) ) * sin( radians(lat) ) ) ) AS distance, post_date FROM ( SELECT latitude.post_id, lat, lng, latitude.post_date FROM ( SELECT post_id, meta_value lat, post_date FROM wp_posts LEFT JOIN wp_postmeta ON wp_posts.ID = wp_postmeta.post_id WHERE meta_key = '_jr_geo_latitude' AND wp_posts.post_status = 'publish' ) latitude, ( SELECT post_id, meta_value lng, post_date FROM wp_posts LEFT JOIN wp_postmeta ON wp_posts.ID = wp_postmeta.post_id WHERE meta_key = '_jr_geo_longitude' AND wp_posts.post_status = 'publish' ) longitude WHERE latitude.post_id = longitude.post_id AND latitude.post_id IN ( SELECT wp_postmeta.post_id FROM wp_postmeta WHERE meta_key = 'geo_short_address' AND meta_value <> '') ) as geo_data HAVING distance < 5 UNION ( SELECT ID, 999999999 as distance, post_date FROM wp_posts WHERE ( ID NOT IN( SELECT wp_postmeta.post_id FROM wp_postmeta WHERE meta_key = 'geo_short_address' ) OR ID IN ( SELECT wp_postmeta.post_id FROM wp_postmeta WHERE meta_key = 'geo_short_address' AND meta_value = '' ) ) AND wp_posts.post_status = 'publish' AND post_type = 'job_listing' ORDER BY post_date DESC ) ORDER BY COALESCE(distance, 999999999) ASC, post_date DESC ) as jobs;


The above logs show that these queries were executed for more than 60 seconds, normally the execution of SQL queries takes no more than 2-3 seconds. Please, note that slow queries are not part of the normal website operation and they are usually due to large or not properly designed MySQL databases or query logic.

You should contact your web developers and have them review the websites and perform the necessary optimizations to avoid the slow queries on the server. This will reduce the CPU usage generated by the MySQL service.

If such optimization is not possible and you wish these queries to be executed on the server you should consider upgrading your hardware. We would recommend you to add at least 2 more CPU cores to the resources of your Cloud server. This way the server will be able to work better and handle your websites correctly.