michael lockhart

I am having an ongoing problem with my exchange cluster. We are running a 3
node (active/passive) exchange 2003 ENT cluster on windows server 2003. All
works ok for a while, and then the HTTP Virtual Server Instance on node 1
fails. Event ID's are below. I fail the cluster over to node 3 and the EVS
Instance runs fine without a problem. i have noticed that the EVS will fail
on node 1 3-4 days after a reboot. I have now been running on the 3rd node
for 2 weeks without a hitch. I have installed Exchange SP2 and this has not
made any difference.

Event 1069: "Cluster resource 'Exchange HTTP Virtual Server Instance 100
(FPCEVS1)' in Resource Group 'Exchange Server 1 (FPCEX1)' failed."
Event 1005: "Exchange HTTP Virtual Servce Instance 100 (FPCEVS1): The
IsAlive check for this resource failed."

Anyone come across this before? Any help would be greatly appreciated!

evan dodds msft

Michael -

What is the error code in the Event 1005? The 1005 event tells that the HTTP
clustered instance failed, but the error code associated with the failure
may help explain why. It's unusual that it would run for several days before
failure. That sounds to me like possibly a resource exhaustion issue on the
failing node.

--
Evan Dodds
Program Manager
Microsoft Exchange Server

****** Disclaimer ******
This posting is provided "AS IS" with no warranties, and confers no rights.
Note: Please do not reply to this e-mail address. It is used for newsgroup
purposes only.

michael lockhart

Evan,

the full event log entry is below:
Exchange HTTP Virtual Server Instance 100 (FPCEVS1): The IsAlive check for
this resource failed.

For more information, click http://www.microsoft.com/contentredirect.asp.

For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp.

Data: Bytes - 0000: 46 27 00 00 F'..

Data: Words - 0000: 00002746


The following is from the cluster.log file on node 1, which is having the
issues.

00000dd8.00001540::2005/12/25-20:49:11.267 ERR Microsoft Exchange DAV
Server Instance <Exchange HTTP Virtual Server Instance 100 (FPCEVS1)>:
[EXRES] DwCheckProtocolBanner: failed in send. Error 10054.
00000dd8.00001540::2005/12/25-20:49:11.267 ERR Microsoft Exchange DAV
Server Instance <Exchange HTTP Virtual Server Instance 100 (FPCEVS1)>:
[EXRES] ExchangeCheckIsAlive: IsAlive failed, will retry in 50 msec.
00000dd8.00001540::2005/12/25-20:49:11.329 ERR Microsoft Exchange DAV
Server Instance <Exchange HTTP Virtual Server Instance 100 (FPCEVS1)>:
[EXRES] DwCheckProtocolBanner: failed in send. Error 10054.
00000dd8.00001540::2005/12/25-20:49:11.329 ERR Microsoft Exchange DAV
Server Instance <Exchange HTTP Virtual Server Instance 100 (FPCEVS1)>:
[EXRES] ExchangeCheckIsAlive: IsAlive failed, will retry in 100 msec.
00000dd8.00001540::2005/12/25-20:49:11.439 ERR Microsoft Exchange DAV
Server Instance <Exchange HTTP Virtual Server Instance 100 (FPCEVS1)>:
[EXRES] DwCheckProtocolBanner: failed in send. Error 10054.
00000dd8.00001540::2005/12/25-20:49:11.439 ERR Microsoft Exchange DAV
Server Instance <Exchange HTTP Virtual Server Instance 100 (FPCEVS1)>:
[EXRES] ExchangeCheckIsAlive: IsAlive failed, will retry in 200 msec.
00000dd8.00001540::2005/12/25-20:49:11.642 ERR Microsoft Exchange DAV
Server Instance <Exchange HTTP Virtual Server Instance 100 (FPCEVS1)>:
[EXRES] DwCheckProtocolBanner: failed in send. Error 10054.
00000dd8.00001540::2005/12/25-20:49:11.642 ERR Microsoft Exchange DAV
Server Instance <Exchange HTTP Virtual Server Instance 100 (FPCEVS1)>:
[EXRES] ExchangeCheckIsAlive: IsAlive failed, will retry in 400 msec.
00000dd8.00001540::2005/12/25-20:49:12.048 ERR Microsoft Exchange DAV
Server Instance <Exchange HTTP Virtual Server Instance 100 (FPCEVS1)>:
[EXRES] DwCheckProtocolBanner: failed in send. Error 10054.

Michael

evan dodds msft

Michael,

Error 10054 is:
WSAECONNRESET winerror.h
# An existing connection was forcibly closed by the remote host.

This sounds to me like the IIS server is not accepting the connection on the
interface that the cluster service is checking against. The IsAlive check
will actually be doing an HTTP connection (or HTTPS, depending on how the
HTTP virtual server is configured) from the host IP address against the EVS
IP address. If it cannot successfully make this connection -- as in this
case, where it's being forcibly disconnected with a "connection reset" --
then it will log that as an IsAlive failure for the resource.

Since it's happening only after several days, this sounds even more like a
resource exhaustion failure -- perhaps your system is running out of some
kernel resources (paged-pool, non-paged-pool memory, PTEs, etc) and IIS is
beginning to reject connections. Might be useful to collect perfmon data to
watch if these are running out. Might want to turn on IIS logging and have a
look at whether these connections are making it into the server. Also, might
be useful to have a look at the three "flash" newsletters the Exchange
product group has posted to the EHLO blog (the 3rd one is here:
http://blogs.technet.com/exchange/archive/2005/12/28/416551.aspx -- and has
links to the earlier two also). There are a number of topics covered in
these postings that would be relevant to tips and tuning for resource
exhaustion issues, if that turns out to be the problem.

--
Evan Dodds
Program Manager
Microsoft Exchange Server

****** Disclaimer ******
This posting is provided "AS IS" with no warranties, and confers no rights.
Note: Please do not reply to this e-mail address. It is used for newsgroup
purposes only.

michael lockhart

Evan,

Thanks for your help on this.


Michael

simon walsh

Hi Michael,

This is not a solution to the problem but I would suggest unchecking the
"Affect the group" checkbox on the http resource.
That way the EVS will not fail over. It will merely allow the HTTP resource
to fail on its own.
You can then troubleshoot the http problems without afftecting any othere
resources.

Regards
Simon

michael lockhart

Simon,

the "Affect the group" checkbox is not selected. The EVS does not currently
fail-over, i fail it over manually, as when the resource fails, none of the
clients on that VS can use OWA, and those on MAC's cannot connect.

Michael

simon walsh

Did you try to just bring the resource online again without failing the
entire EVS over? That would do.

Of course it doesn't solve the problem but if you follow Evan's advice you
should find a more permanent solution.

Simon

michael lockhart

i can bring the failed resource back online, but it fails again within a few
minutes.