m5

Hello,

I have a command-line PHP script--called Listener--that is designed
to run indefinitely with a predictable CPU usage and memory
footprint. In a nutshell, it's a multi-client socket server that
waits for incoming connections, processes incoming data, stores
results in a MySQL database, and basically gets on with its life. And
it works. No errors or serious problems to speak of. And I've been
running it for a couple years on an Xserve dual-G5 2GHz w/ OS X
Server 10.4.11). Six months ago, the program would run for days, even
a couple weeks, without a hitch. The only reason I would stop the
script is for some other purpose, like a software update.

However, the number of socket clients connecting in the past 3-4
months has steadily increased, and this seems to have exposed (if not
created) a strange performance "issue" with PHP 5.2.4, MySQL 5.0.45
and/or Mac OS X Server 10.4.11. (I say "and/or" because I am unsure
where the problem's cause really lies.) Basically, after the script
has been running for a day or so (processing essentially the amount
data that used to take two weeks), the CPU usage of the machine goes
from 30% (normal) to 80-90%. This appears to be irrespective of the
number of clients connected to the server at the time, but rather the
amount of time the script has been running (and therefore cumulative
cycles it's operated, data processed, MySQL queries executed, etc.).
And the CPU usage stays high, even when the actual load (number of
clients) decreases. At this time, if I run top, I get the following
info:

22512 mysqld 91.6% 8:22:12 31 106 125 305M+ 3.20M
260M 475M
17104 php 7.2% 81:14.01 1 15 145 5.08M 7.80M
10.9M- 87.5M
22537 Terminal 6.6% 2:59:59 22 176 240 12.3M 21.2M
18.3M- 236M
[...]

At first I thought, okay, it's MySQL's fault. Maybe a lot of slow-
running queries. But the slow query log is pretty clean. So maybe
it's a combination of Mac OS X and MySQL and PHP? I Googled for a
similar problem, and finally ran across this article:

http://www.shawnhogan.com/2005/10/my...-x-server.html

....where the author describes a very similar CPU usage pattern. I
tried his suggested fixes and they seemed to have helped a little (or
maybe it's my wishful thinking--hard to tell), since the high CPU
load issue doesn't appear to happen as soon... But it still happens.

Anyway, I'm really stumped as to what to do next, where to look, etc.
If I stop the script, and restart it (but not the MySQL itself), CPU
usage goes back to normal--for about a day or two.

The only thing I thought might be connected is how many short-lived
PHP child processes Listener creates--around 20-30,000 per day.
Sounds higher, but on average it's just one every 2-3 seconds.
Anyway, although the number of child processes isn't concurrent,
would there be a problem with the number of historical child
processes in view of ulimits or kern.maxfilesperproc?

Anyway suggestions, tips, or links are much appreciated. Thanks.

....Rene

lists

I have a server that "listens" like yours does. I get 80k - 85k connections a day to it.

When I first started it, I was only getting about 3k of connections aday. Then I upped the
listening pattern and it tanked. I noticed that all my mail/web/db connections just sat there.

When I investigated, I found that the number of connections to the server was being overloaded. So
I increased the kern.maxfilesperproc setting to 32000. All the problems went away.

I have about have the horse power you do, running OpenBSD 4.1, and it runs great now as my
"listener" / web / ftp / mail / named / database / spam filter / etc...

One question about the "listener" program, does it maintain a connection to the DB or does it
open/close a connection upon each socket connection? If it does the latter, you might look into
using a constant connection rather then opening/closing on a per connection basis.

--
Jim Lucas

"Some men are born to greatness, some achieve greatness,
and some have greatness thrust upon them."

Twelfth Night, Act II, Scene V
by William Shakespeare

m5

Hi Jim,


Thanks for the reply. (Not many people seem to be doing what I'm
doing in the way I'm doing it... so I really appreciate the feedback.)

I don't think the kern.maxfilesperproc setting is a problem. I'm
currently set to 102400 per Shawn Hogan's advice. And my Listener
program only receives ~1000 connections per day. However, each
connection involves multiple MySQL queries, often as many as 50 or so
per connection each day. So perhaps your second observation applies.
MySQL is showing ~ 1,500 connections per hour. And I'm using
mysql_connect() in Listener. I will see if using mysql_pconnect(),
and reducing the number of connections helps.

My.cnf's max connections, presently at 100, may have been too low in
view using mysql_connect(). If mysql_pconnect() doesn't improve
things, maybe I should bump up max_connections to 500?

...Rene

trogers

Hi,

RF> I have a command-line PHP script--called Listener--that is designed
RF> to run indefinitely with a predictable CPU usage and memory
RF> footprint. In a nutshell, it's a multi-client socket server that
RF> waits for incoming connections, processes incoming data, stores
RF> results in a MySQL database, and basically gets on with its life. And
RF> it works. No errors or serious problems to speak of. And I've been
RF> running it for a couple years on an Xserve dual-G5 2GHz w/ OS X
RF> Server 10.4.11). Six months ago, the program would run for days, even
RF> a couple weeks, without a hitch. The only reason I would stop the
RF> script is for some other purpose, like a software update.
RF> However, the number of socket clients connecting in the past 3-4
RF> months has steadily increased, and this seems to have exposed (if not
RF> created) a strange performance "issue" with PHP 5.2.4, MySQL 5.0.45
RF> and/or Mac OS X Server 10.4.11. (I say "and/or" because I am unsure
RF> where the problem's cause really lies.) Basically, after the script
RF> has been running for a day or so (processing essentially the amount
RF> data that used to take two weeks), the CPU usage of the machine goes
RF> from 30% (normal) to 80-90%. This appears to be irrespective of the
RF> number of clients connected to the server at the time, but rather the
RF> amount of time the script has been running (and therefore cumulative
RF> cycles it's operated, data processed, MySQL queries executed, etc.).
RF> And the CPU usage stays high, even when the actual load (number of
RF> clients) decreases. At this time, if I run top, I get the following
RF> info:
RF> 22512 mysqld 91.6% 8:22:12 31 106 125 305M+ 3.20M
RF> 260M 475M
RF> 17104 php 7.2% 81:14.01 1 15 145 5.08M 7.80M
RF> 10.9M- 87.5M
RF> 22537 Terminal 6.6% 2:59:59 22 176 240 12.3M 21.2M
RF> 18.3M- 236M
RF> [...]
RF> At first I thought, okay, it's MySQL's fault. Maybe a lot of slow-
RF> running queries. But the slow query log is pretty clean. So maybe
RF> it's a combination of Mac OS X and MySQL and PHP? I Googled for a
RF> similar problem, and finally ran across this article:
RF> http://www.shawnhogan.com/2005/10/my...-x-server.html
RF> ...where the author describes a very similar CPU usage pattern. I
RF> tried his suggested fixes and they seemed to have helped a little (or
RF> maybe it's my wishful thinking--hard to tell), since the high CPU
RF> load issue doesn't appear to happen as soon... But it still happens.
RF> Anyway, I'm really stumped as to what to do next, where to look, etc.
RF> If I stop the script, and restart it (but not the MySQL itself), CPU
RF> usage goes back to normal--for about a day or two.
RF> The only thing I thought might be connected is how many short-lived
RF> PHP child processes Listener creates--around 20-30,000 per day.
RF> Sounds higher, but on average it's just one every 2-3 seconds.
RF> Anyway, although the number of child processes isn't concurrent,
RF> would there be a problem with the number of historical child
RF> processes in view of ulimits or kern.maxfilesperproc?
RF> Anyway suggestions, tips, or links are much appreciated. Thanks.
RF> ...Rene


Put a usleep(1000) in the listen while() loop and give the cpu a
break.

--
regards,
Tom

m5

Good advice, but I've already been doing that. The thing is, when the
script first starts up, the CPU rarely exceeds 30%, even when many
clients (200+) are simultaneously connected and sending data. When a
few clients are connected, CPU is typically below 10%. Again, it's
only after 24-48 hours that, all of a sudden, CPU usage increases by
40-50%. And it stays high until I stop the script and restart it.

One question I have though is, is there actually any benefit to using
mysql_pconnect(), since the script simply loops? My understanding is
that pconnect only benefits if a script would otherwise be using
mysql_connect repeatedly--and this script doesn't, since it calls
mysql_[p]connect() just once, in the start tof execution.

....Rene

lists

This makes me think about asking if you have to short of a timeout on your receiving connection?

What are you using to setup your connection? fsockopen() or stream_socket_server() ?

here is a snippet of what I have

<?php

$conn = mysql_connect(...);

if ( $socket = @stream_socket_server('udp://'.LISTEN_IP.':'.LISTEN_PORT,
$errno, $errstr, STREAM_SERVER_BIND) ) {

while ( true ) {

/* Get the exact same packet again, but remove it from the buffer this time. */
$buff = stream_socket_recvfrom($socket, 1024, 0, $remote_ip);

# do stuff with your incoming data and mysql connection


}
fclose($socket); } mysql_close($conn);
?>

I don't have a timeout set on the *_recvfrom() call. I just wait until the next connection comes in.

You don't need to use mysql_pconnect(), especially if you are using, what in essence is, a daemon.

Just don't open and close the connection constantly, leave it open.

Also, make sure you are not using an array that you are not re-initializing through each iteration
of the loop. If the array keeps getting bigger, PHP might $*%& on itself. Always re-initialize
arrays to clean them up.

Hope some of this helps!

--
Jim Lucas

"Some men are born to greatness, some achieve greatness,
and some have greatness thrust upon them."

Twelfth Night, Act II, Scene V
by William Shakespeare

jochem

even then he may still have creeping memory ... in which it might be possible to have a mother process that
spawns and watchs a child process .. the child process is the actual deamon, the child could then keep a track
of it's own memory usage and then kill itself when it gets too big ... the mother in turn would automatically
spawn a new child deamon process upon seeing it's child has committed suicide.

does that make sense?

m5

One second on stream_socket_server(), with a 900 second timeout stream_socket_accept().


[...]

Thanks for sharing your code. Seems pretty similar to mine at first glance.


Yeah, after thinking about it, that's what I figured. Thanks for confirming though.


Yes, I just open it once at the top of the script. And that's it.


All good advice. I will check my arrays, although I don't think this
is a problem since I monitor the scripts memory usage with
memory_get_usage() and memory_get_peak_usage(), and it never tops 3MB
(max allocated is 16MB). There are a few little parts to the daemon.
One thing I'm doing that could be problematic is running an include
(); on a couple files each time a socket has new data (this allows me
to adjust the processing logic on the fly without having to start the
script and wait for clients to reconnect)--but I can see this being
expensive in terms of performance and resources. Actually, I wonder
if THAT is not what's starving the script of resources over time--
each fread() involves several includes(); I'll have to look into that...

FWIW, here's the stripped-down skeleton of the server:
As always, constructive criticism is very welcome.

<?php

$socket = stream_socket_server("tcp://127.0.0.1:9876", $errno, $errstr);

if ($socket) {

$master[] = $socket;
$read = $master;
$write = $master;

while (1) {

$read = $master;
$write = $master;
$mod_fd = stream_select($read, $_w = NULL, $_e = NULL, 1);

if ($mod_fd === FALSE) {
break;
}

for ($i = 0; $i < $mod_fd; ++$i) {
if ($read[$i] === $socket) { // NEW SOCKET

$conn = stream_socket_accept($socket, 900);
$master[] = $conn;
$key_num = array_search($conn, $master, TRUE);

} else {

$sock_data = fread($read[$i], 32768);

if (strlen($sock_data) === 0) { // CONNECTION GONE

$key_to_del = array_search($read[$i], $master, TRUE);
fclose($read[$i]);
unset($master[$key_to_del]);

} elseif ($sock_data === FALSE) { // CONNECTION BROKEN

$key_to_del = array_search($read[$i], $master, TRUE);
fclose($read[$i]);
unset($master[$key_to_del]);

} else { // READ INCOMING DATA

// include (somefiles);
// include (somefiles);
// include (somefiles);
// [ ... ]

}
}
}
}
}

?>

jochem

it may ... but it's besides the point ... apparently I'm having difficulty even reading the subject of question!?@#$? :-P

lists

The follow part, I think, is where your problem is.


This line tells the system to wait 1 second and then continue, whether
you have an inbound connection or not.

Then here you are testing for success or failure of the last call.


Problem, if you don't have a connection, then the will fail constantly.
once every second X (times) the number of connections you have...


Here, you are not removing the successful connections from $master, so
it keeps growing.... on and on...


As for the includes, well, I would turn them into function calls and
then have a different function for the different ways you want the program to react.


I didn't see anything about your DB connections. Are those located in
the includes?

How long, on average, does your processing of the incoming data take?
Because, to me, it looks like you might have a blocking problem with the
in-coming connections. If the initial connection takes too long, then
the following connections might be getting blocked. You might want to
look into pcntl_* functions to do forking if you need it.