Hi,
I am using async IO and completion ports to generate I/O to storage
devices. Problems begin when IO error is encountered. According to my
experiments, when IO error is received, there are three possibilities:
1.Error is returned synchronously and IOCP completion routine isn't
called.
2.ERROR_IO_PENDING is retuned synchronously and when the IOCP
completion routine is called GetOverlappedResult is used to get the
error.
3.Error is returned synchronously (Not ERROR_IO_PENDING) and still the
IOCP callback is called later on.

To determine when my callback routine will be called, I thought to use
GetOverlappedResult after sending the IO to determine whether the IO is
still pending in the kernel (An IRP for it still exists and the
completion routine will eventually be called).
However, there were cases in which I received a synchronous error
(ERROR_ACCESS_DENIED because this was a clustered volume and it was
offline) and calling GetOverlappedResult I saw the the IO is still
pending (IO_INCOMPLETE) but the completion routine was never called
with the corresponding overlapped structure.

I have the following questions:
1.Does anybody have any insights as to why GetOverlappedResult return
IO_INCOMPLETE when the I/O never completes and is it legitimate to use
GetOverLappedResult even if ReadFile/WriteFile didn't return with
ERROR_IO_PENDING.
2.Is there some deterministic way to determine when my callback
function will be called or not as ERROR_IO_PENDING in the synchronous
call doesn't seem as a good indication.

The environment I am working on is windows 2003 server machines. The
storage is located on the SAN and is accessed via ATTO or QLOGIC HBAs.

Regards,
ERAN.

sarge

Eran,

After hooking a file handle into an existing completion port, all overlapped
ReadFile and WriteFile completions (whether successful or not) will trigger
GetQueuedCompletionStatus, with one exception:

ReadFile or WriteFile returns FALSE and GetLastError() != ERROR_IO_PENDING.

This is the *only* time that you won't get an overlapped completion thru the
completion port.

Even if ReadFile or WriteFile returns TRUE, indicating a synchronous
completion, GetQueuedCompletionStatus will trigger with the overlapped
structure.

Therefore, minus the one exception above, you should always handle your
completions (and failures) from your completion port worker thread.

GetOverlappedResult is generally used with non-completion port overlapped
I/O. Note that non-completion port overlapped I/O needs a valid
manual-reset event handle to be present in the overlapped structure.
GetOverlappedResult uses this event handle to wait on if the bWait parameter
is TRUE. On the other hand, completion port I/O does *not* use events (the
SDK even says you can use that member of the overlapped structure for
private data if you like). That is why you must use
HasOverlappedIoCompleted if you need to check the status.

Are you re-using overlapped structures? One peculiarity I've noticed with
named pipes and sockets on a completion port is that sometimes
HasOverlappedIoCompleted returns FALSE even after the I/O completion or
failure has triggered through the completion port. This is because the
kernel does not always clear the STATUS_PENDING flag from the
OVERLAPPED::Internal member. So if you reuse overlapped structures, always
memset(0) the structure before using it. And never modify any members of an
overlapped structure while the kernel still has it. The only time it's safe
to reuse or free an overlapped structure is *after*
GetQueuedCompletionStatus returns it to your application.

Now for your top 3 statements:

1. Completion will always be called unless ReadFile or WriteFile returns
FALSE and GetLastError() != ERROR_IO_PENDING.

2. You should not use GetOverlappedResult to retrieve an I/O error on a
completion port. When GetQueuedCompletionStatus returns FALSE, you must
immediately call GetLastError() (from the same thread) to obtain the reason
for failure.

3. The completion port does *not* trigger when ReadFile or WriteFile returns
FALSE and GetLastError() != ERROR_IO_PENDING. That is why you must handle
those errors immediately. The call that you made to GetLastError is where
it ends.

Hope this helps.

--Sarge