beth

Thought as much...

Ah, yes, that's Hutch's automatic utility I was mentioning, I do believe...

Basically, yes...

TASM is slightly less fussy because it doesn't even bother with the
parameters it takes...for example, inside "import32.lib", you'd find
just "ExitProcess", whereas MASM calls it something like
"_ExitProcess@4"...although, despite the ugly syntax, MASM offers an
improvement on TASM here because it'll also allow you to type "call
__imp__ExitProcess@4", which links in a more direct and faster way
than TASM (or MASM using "call _ExitProcess@4") does...this more
"direct" version actually makes an indirect call - assembled to
something like "call [420000h]" - whereas the less direct way
assembles to "call 42000h" and, at the address it calls (420000h),
there's a "JMP" statement which makes the actual final jump to the DLL
function...

The need for this weirdness is because, at compile-time and link-time,
it cannot know the final run-time addresses of the DLLs (the addresses
will be different for different versions of the DLL...plus, DLLs can
be relocated to different places in memory that it's not possible to
"hard-wire" exact addresses into your final code ...so the code is
assembled to make calls to an "import address table"...the actual
Windows loader then fills out this table with the DLL run-time
addresses when the module (DLL or EXE) is loaded...the headers for
your executable will be filled in by the linker with the names of the
DLLs and DLL functions you need, which the loader reads so as to know
exactly what addresses you want put into the "import address table"
(IAT for short)...

With MASM's "direct" method, it makes an indirect call via the
addresses stored in the IAT ("call [ 420000h ]" grabbing the address
stored at 420000h - where the IAT starts in my example here - and then
calls the DLL function directly)...with TASM and MASM when you're not
using the more "direct" method of calling functions, the assembled
instruction looks more like "call 420000h"...and every DLL function is
actually NOT called directly but is called _via_ the "Import Address
Table"...inside the IAT, in fact, you'd find a series of "JMP"
instructions and this is what is actually called by your
program...though the "JMP" instructions, of course, doesn't effect the
return address on the stack which the "CALL" instruction made so it'll
"JuMP" to the DLL function and when that function issues "RET", it
correctly returns back to your program...

This is somewhat complicated, granted, but, really, there's little
choice about doing it this way...because, as DLLs can have different
versions where the same functions could be at different addresses
(e.g. Windows XP's KERNEL32.DLL has been modified and improved on
Windows 95's KERNEL32.DLL...the XP version still has all the same
basic functions in it but they've been changed, improved or extra
functions added ...and, also, DLLs can be "relocated" in memory too
(e.g. one DLL is loaded at 800000h and then another DLL asks to be
loaded into the same location...well, this is not possible because the
other DLL is there...so, the loader simply uses the relocation
information in the DLL and moves it elsewhere in memory...which, by
the way, is why you _should_ have relocation information attached for
DLLs but you _DON'T_ strictly need relocation information for an EXE
because every EXE gets its own address space and, thus, by definition,
every EXE _will_ get its chosen base address...the only exception to
this was "WIN32S" which was an extension that MS added to Windows 3.x
to make it able to run some Win32 programs...I'm presuming that,
basically, you won't be using Windows 3.x or that you'd want WIN32S
compatibility, as no-one uses Windows 3.x anymore ...

Because there's no way of knowing at assemble-time / link-time which
version of the DLLs you're going to be linking to (nor whether that
DLL has been "relocated"...for efficiency reasons of not wasting
memory, Windows does try to load the system DLLs into the same place
for all processes so that it only needs to load a single copy of the
DLL into memory and all processes can share this same copy, so that
only _one_ KERNEL32.DLL needs to be loaded for all the programs
running...Windows will, in situations where this is somehow not
possible, load multiple copies of a DLL...but its default behaviour is
to try its best to load them all to the same locations...note, also,
that the main Windows system DLLs have been linked so that they all
have different base addresses near the end of memory to ensure that
none of them "collide"...it's good practice, also, for any DLLs you
might also create to try to find a memory location where it won't be
"clashing" with any other DLL or system DLL, to avoid the need of
relocating it...it will still work if you don't - it'll just relocate
the DLL - but things are much more efficient when the base addresses
are pre-calculated to best avoid any need for relocation ...so,
though complicated and slightly slower than a direct call straight to
the DLL would be, your EXE will actually be calling all its DLL
functions _via_ the "Import Address Table"...the loader itself will
fill out this table with the actual _exact_ addresses for the
functions when it loads your program (this is the reason, by the way,
that DLLs are always loaded first before any EXE that references
them...it has to know for certain where the DLLs are going to finally
reside in memory to know what addresses to fill out in the "Import
Address Table" ...

The "Import Address Table" is basically just a table of "JMP address"
instructions for each DLL function you need...as this is slightly
confusing, let's try to explain it more clearly just to make it
perfectly clear what's happening (note, in the following, I'll be
putting a space in the MASM names after the "@" sign...this space
shouldn't actually be in a real program at all but it has to be added
to stop newsreaders like Outlook Express thinking that it's an Email
address - because of the "@" sign - and turning it into a hyperlink
...

For MASM's "direct" way:

CALL __imp__ExitProcess@ 4

Assembles to something like:

CALL [ 420001h ] ; note, actual address varies and will almost
; certainly be different...but I'm using this
; address as an example value

At address 420001h, we find the address "76543210h", for
example...again, this address is just an example...but the address
will basically be the _actual_ address of the real "ExitProcess" API
in the KERNEL32 DLL copy loaded into memory...this address is actually
calculated at _load-time_ by the loader...

For TASM (and MASM's "indirect" way), it's an almost similar situation
except that the CALL instruction is slightly different...it now looks
something like:

TASM: CALL ExitProcess
MASM: CALL _ExitProcess@ 4

Assembles to:

CALL 420000h

And, this time, rather than the CALL instruction just using the
address value stored in the "Import Address Table", it actually makes
a call into it...so, we literally call address 420000h...at this
address, we find something like:

JMP 76543210h

And this JMP jumps directly to the start of the actual DLL
function...note that the "JMP" instruction doesn't effect the return
address that was pushed to the stack by "CALL" at all, so the DLL
function's "RET" instruction will still work as expected and return
back your program (the instruction after the CALL) rather than back to
the IAT...

[ Amazingly, this process is actually be done for every call to a DLL
function (including all the OS API you make in your program...plus,
if you think that's bad, then, on NT-based systems (NT, XP , the
DLLs like KERNEL32.DLL actually often don't do the work themselves but
they themselves use another DLL - NTDLL.DLL - to actually do the real
work...well, I say "real work" but, even then, this DLL often has to
make slow "user -> kernel" priviledge level transitions too...before
it goes on to defer it through multiple layers of the OS, potentially
all the way down a perhaps long chain of "layers" until it actually
reaches a device driver which does any actual interfacing with the
real hardware...

And people wonder why I'm not exactly the greatest fan of "layered" OS
architectures and am not always "nice" about Window's speed and
design...if only they knew what was going on then they'd realise I'm
not be at all unreasonable in finding all this often extraneous stuff
not exactly the best thing since sliced bread ]


The Import Library contains information about the functions inside a
particular DLL...unlike an ordinary static library, this information
actually _isn't_ the exact address of the function inside the
DLL...this information _can't_ be known at link-time (as we don't know
which version of a DLL we've going to be linking to...nor whether it's
been "relocated" from its usual base address) but is only known at
load-time...so, actually, there are no "addresses" in the import
library...but there is information in them about each DLL function
that the linker uses to fill out the EXE's headers...when the loader
is loading in your EXE, it looks at this information to know which
DLLs to load into memory...once it does this, it looks through more
information in the EXE headers which tell it which "imports" you
actually want from the DLLs and, with this information, it fills out
the _exact_ DLL function addresses into the "Import Address
Table"...your EXE was assembled and linked to work with the "Import
Address Table"...

The rough idea of what you heard is basically right...but, strictly,
the import library does not have information about the exact
addresses, it has information about the DLLs and the DLL functions
which the linker puts into the EXE headers so that the _loader_ knows
how to fill out the "import address table" properly with the right
addresses...

[ In fact, though the import library is a binary file, it's still
instructional for you to simply open up something like "import32.lib"
or "kernel32.lib" with Notepad...as it's not a text file but a binary
file, there'll be lots of junk characters...but every so often, you'll
see ASCII text sequences like "KERNEL32.DLL" and "ExitProcess"...well,
this is the basic information that an import library contains...not
addresses but names of DLLs, names of API, how many parameters they
take and that sort of thing...this information is used by the
_loader_...so, when it sees "KERNEL32.DLL" listed in the EXE headers,
it knows to load in KERNEL32.DLL into memory...the information about
the API functions in the EXE headers tell the linker exactly which API
function it is that you want to use and it then gets the address of
the named API and puts them in the right "Import Address Table"
positions...once it's done this, the EXE is ready to run because it'll
make all of it's CALLs via this import table...oh, if you do look at
the contents of these files in Notepad, remember NOT to "save changes"
when you close it...you don't want to alter it or it might stop
working properly...think of it as "read only" ]

But, yes, the whole process is rather confusing and
convoluted...unfortunately, it basically has to be this way due to the
design of Windows (flat memory model) and in order to allow for
different version of the same DLLs and to allow the DLLs to be
relocated at load-time / run-time...

[ Interestingly, COM (as used by DirectX and OLE uses an
object-orientated way that automatically "links" at run-time...so,
other than the initial "CreateDirectDraw" API or whatever needed to
get the first "object", it actually doesn't suffer as much pointless
overhead as the "static" linking way...so much for all those people
who claim that object-orientation makes things less efficient, eh? COM
actually uses the more "direct" indirect CALL instructions all the
time and doesn't require the EXE headers to be filled up with
information for the loader...this is why Microsoft are using COM more
and more for all their new DLLs, like DirectX...unfortunately, the
main system DLLs - KERNEL32.DLL, USER32.DLL, GDI32.DLL - were
originally using the other method and basically have to continue to do
so for "backwards compatibility" reasons...or, otherwise, looking at
how MS are making all their new DLLs COM-based, it appears they've
decided that this new method is infinitely better...

Plus, of note, you can also do all of the linking _manually_ at
run-time using the "LoadLibrary", "GetProcAddress" and "FreeLibrary"
APIs in KERNEL32.DLL (these three will have to be statically linked in
the usual way but, thereafter, you can do all the loading and linking
at run-time using these API rather than linking with import
libraries)...

Unfortunately, though, both of these methods - COM and run-time
linking - are a _LOT_ more hard work to actually code...so, it
actually sort of "cancels out" the benefits because you have to write
a lot more complicated code to do things this way...so, it's NOT
really worth doing these things UNLESS you have a good reason to do so
(for example, if you are using "plug-ins" where you'd load a DLL at
run-time which may or may not be there...for instance, looking inside
a "PlugIns" folder on the hard drive for DLLs and then loading
them...which is stuff in the "very advanced" category, really...it's
very nice how things like Internet Explorer and Netscape Navigator can
do all this "plug-in" business...but, well, it's very complicated and
a lot of work to actually code this sort of thing ...use the import
library way if you've not got any reason to do otherwise because the
other ways - though they have their uses - are very headache-causing ]


Yes, this probably also works...because all the needed information is
available...

Note, actually, that "lib.exe" (which you'd use to create your import
libraries in MASM _is_ "link.exe"...Microsoft actually combined the
librarian and the linker and so forth into the same "link.exe" file
and the "/lib" switch makes it behave as "LIB" rather than "LINK" (if
you type "link /lib", you'll notice that it calls itself "LIB" in the
help output, exactly as if you'd typed "lib" alone..."lib.exe" is not
actually a separate program - as link.exe now includes both "link" and
"lib" in the same utility - but is a "shortcut" program which simply
calls "link" with the "/lib" switch always enabled ...

Thus, because of the fact that, really, "lib" and "link" are actually
the same program then it does make sense that you could just use
"ml.exe", "link.exe" and the appropriate ".def" file...because
"link.exe" can just temporarily change to its "lib.exe" alter ego,
process the ".def" file (as you'd use "lib.exe" to create any import
library, anyway and then switch back to being "link.exe" again to
finish off the linking...

This, no doubt, is exactly the reason _why_ Microsoft combined
"lib.exe" and "link.exe" into the same utility...an attempt to make
linking easier...because, yes, that whole "import library" thing can
often be problematic as well as confusing...

I've personally never tried the "/DLL" way you're mentioning
here...but it makes perfect sense that "link.exe" would be able to do
this, as long as you supply the correct ".def" file...well, you learn
something new every day...when I get time, I'll have a go at doing
this, I think, just to see if it really does work

sure.

Well, I don't know for sure either...but it's certainly possible and
would actually make plenty of sense

It's simple, really...the MASM32 package doesn't actually come with
"kernel32.lib", "user32.lib", etc. installed as files...instead, Hutch
has taken a rather clever route to getting these files instead...when
you install MASM32, it grabs the _actual system DLLs_ on your hard
drive and runs those programs you mention to generate all the
necessary ".lib" files for each system DLL it finds on your system...

"inc32.exe", if I've got it right, is actually responsible for
creating all the ".inc" files...these aren't installed in the package
either...instead, just as it does for creating a ".lib" file out of
each of the system DLL, it uses "inc21.exe" on each DLL file also to
create the corresponding ".inc" file...

[ Interestinly, if you actually look at the "kernel32.inc" file that
"inc21.exe" creates, then _all_ parameters are "DWORDs"...there's no
"HINSTANCE" or "HWND" like there is in the Win32 do***entation...the
reason for this is actually rather ingenious...under WIN32, _ALL_
parameters to API just so happen to be "DWORDs" (with only one or two
rare exceptions)...thus, "inc21.exe" is able to create the include
files just by looking at the system DLLs without having to know
anything special about them...the DLL already carries information
about how many bytes of parameters an API function takes (you'll note
that, in MASM, you actually type this information in yourself as part
of the function's name: "_ExitProcess@ 4" (again, the space isn't in
the real name, I have to put it in to stop Outlook thinking that it's
an Email address and turning it into a hyperlink )...so, Hutch's
tool actually does something very simple but, ingeniously, it works
perfectly every time thanks to the fact that _all_ WIN32 API use DWORD
parameters...it just divides the amount of bytes an API function takes
by 4 and then spits out that many "DWORDs" in the API function's
prototype for the include file...as "dumb" and "blind" as this might
sound, it actually works brilliantly due to the fact that Microsoft
are totally consistent in making every parameter a DWORD in size ]

The reason for these programs running many times whilst you install
MASM32 is that they are being called for each system DLL one at a
time, in order to create the corresponding ".lib" and ".inc" file for
each system DLL you have installed on your hard drive...after all, it
would actually be a waste of space for the MASM32 package to carry
around these files when all the information in them _already exists_
on your system _actually inside_ the system DLLs themselves...plus,
theoretically, MASM32 can handle any new system DLLs that Microsoft
come up with, without needing to be updated, just by having these
tools run on the new system DLL file...hence, it also makes Hutch's
life a hundred times easier to keep MASM32 up-to-date...he just needs
to add any new system DLLs to the installation procedure and the tools
will automatically spit out the necessary ".lib" and ".inc"
files...this is thanks to Microsoft being very consistent with their
system DLLs...it's actually quite a sensible approach because Windows
DLLs are numerous and massive that manually writing your own ".inc"
files by hand would simply take forever to do (and it would be very
tedious...after all, it's a very "automatic" translation because,
well, MASM32 installation _does_ actually do it all automatically
...

I Hope that explains it all to you

Beth

charles d. quarles

Hi, Beth


OK, but how does Unix work then, if Unix doesn't use something like this?
In fact, how does any multi-user, multi-tasking OS work if it doesn't do something like this?


Charles

beth

this?

Where did I say I was a fan of UNIX either?

1. Relocations; The loader applies relocations directly...requires
relocation information...DOS MZ executables undergo this...

2. Interrupts; Routines are called via software interrupt...the IVT /
IDT (real and protected mode names respectively interrupt tables
are loaded with the correct addresses...BIOS and DOS calls work this
way...as does Linux (multi-user, multi-tasking enough for you? , via
INT 80h...

3. Method tables; Following an object-orientated paradigm, pointers to
structures containing the addresses of methods / functions /
(sub)routines / procedures* are used...Microsoft's COM technology
defines one such implementation...preferred by all of MS's newer
libraries, such as DirectX (one has the feeling MS would now prefer to
re-implement the whole lot in COM, if it could (as most recent work by
them prefers to be COM-based)...but can't because of
"compatibility")...Windows is multi-tasking and newer versions are
properly multi-user...

4. Run-time dynamic linking; Applications manually load in libraries
and get the procedure addresses at run-time...how the application
wishes to deal with the addresses is up to the application (it could
physically relocate itself, perform indirection via a table of
addresses, etc. ...possible in Windows via "LoadLibrary" and
"GetProcAddress"...

[ 5. Segmentation; The x86 architecture permits multiple virtual
address spaces via segmentation...each function can be given its own
segment which starts always at virtual address zero (regardless of
what it's actually physical address is); Thus, the GDT / LDT* can be
used in a similar fashion to how the IVT / IDT* was used for point 2
(interrupts)...not used by any system I know of, as it's limited,
horrible, slow and complicated...but it's _possible_ so it makes it
into the list ...

6. Run-time static linking plus relative addressing; An even more
insane idea that no-one would actually implement...the loader
"stitches" together code contiguously, as if the multiple modules
files were actually one file...this code is carefully crafted to use
only relative addressing throughout...utterly impractical and next to
impossible with the x86 architecture, as its instruction set is not
designed well for "relative address only" relocatable code... ]

Give me some time and I can think up some more, if you like...up to
point 4, these are _actual_ devices _used_ in various contexts by
various OSes...

Anyway, you missed my point...it's the layered OS architecture that I
was mostly complaining about...also, the comment was only suppposed to
apply to the second "CALL to JMP to API" method...and, clearly, there
is a better alternative to this, as MASM demonstrates with the first
"indirect CALL" method...

I do apologise; I don't kneel before any false gods...sorry if my
frank observations and personal opinions on the matter disturbed any
particular "OS worship" that you engage in...I do try to respect all
Faiths and didn't mean to cause offence to anyone...I was merely
speaking my own mind and failed to account for those who have sense of
humour failure...I will endevour to do better next time...

Perhaps...if I can be bothered...

Best of Luck,
Beth

* Delete as applicable

tim roberts

Yes, but being a Unix clone does not imply very much about the kernel
implementation. Linux happens to use INT 80h to invoke system services on
x86 machines. It could also have used a call gate or an invalid
instruction.
--
- Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.

beth

linux.

I know the name (or, at least, I do know the name for it but had
temporarily forgotten when writing that response ...it's only
"insane" if you attempt it with the x86 instruction set "as is"...what
you propose below (adding an offset onto everything) isn't "insane",
granted...it's quite clever, actually...I just wasn't thinking about
doing it that way (what I was thinking about at the time was
restricting a program to only using instructions with relative
addressing..."jmp short" (-127 to +128 but NOT "jmp near" as that's an
absolute address instruction that replaces (E)IP and that sort of
thing...with the x86 instruction set's design, it's a bit silly
("insane") to try to avoid any instructions that have absolute
addresses because, unlike some CPUs (which generate position
independent code quite naturally), the x86 isn't well-designed for
this...

Now, that said, the thing I didn't consider at the time - which you've
now highlighted - is that you can use the "[ base address + offset ]"
addressing mode throughout and then only need to calculate the base
address at the start of the program...now, true enough, that isn't at
all "insane"...I wasn't thinking about this when I wrote the above

Actually, now that you mention this, I'm beginning to see Windows'
"hInstance" in a new light...because, in fact, under Windows, the
"hInstance" just so happens to also be the base address that the
program was loaded with...therefore, you could actually just use this
under Windows and you wouldn't have to calculate it yourself...were
Microsoft thinking of this at the time? Hmmm, I doubt it - at least,
not for Win32 - because the "hInstance" parameter comes from pre-win32
days and, to be honest, doesn't serve much purpose in Win32 (each
process gets its own address space, anyway) so is only kept around for
"backwards compatibility" reasons...another reason I'm not greatly
happy with "backwards compatibility" as all these API insist on
"hInstance" because they insisted upon it - needed it - under old
Win16 and still insist upon it under Win32, even though the different
memory model makes it quite useless (every program, unless otherwise
instructed, is loading to 400000h and has the exact same
"hInstance"...Win32 is actually telling them apart by other means and
gives each process a different address space so they don't share
memory anymore ...so, in a typical Windows program, you have to pass
the "hInstance" around to all these API functions, even though,
really, that's just "backwards compatibility" and the API aren't using
this value to tell processes apart, anyway (they couldn't...the value
ends up being the _same_ for every process, unless the linker defaults
are altered, which usually doesn't happen and isn't really necessary to do)...

Yes, there's two ways here, really...you can either use the "[
BaseAddress + offset ]" addressing mode or an application could
"self-relocate" and alter its own addresses using self-modification
(to do what the loader would do in relocating a program but the
program does it to itself instead ...plus, you could also do the
"self-relocation" in two ways too...either literally changing every
reference in the program manually or, instead, you could follow the
"import table" style of doing things and merely create a table of
addresses that all address references go via...that is, when the
program starts, you calculate the base address and then fill out a
table of addresses where everything is and then any reference picks up
this address in the table and uses that...

Although, as the relative address (the base address) is going to be
fixed then the table method seems a particularly wasteful of doing
it...it'll work, though...

Ah, wait, now I remember why I wasn't including this method as part of
"point 6" and called point 6 "insane"...this self-relocation was
mentioned in an earlier point so, for point 6, it's NOT included and
it's strictly just "don't use instructions with absolute
addresses"...which is a bit "insane"...

But, true, if you "mix and match" point 6 with other points (using
point 1's "relocation" with "self-relocation"), it's not "insane" any
longer...but I wasn't originally including this when I made point
6...it was strictly a case of "avoid absolute address instructions
throughout a program", which is a little "insane" because that
restriction is a major restriction due to the way that the x86
instruction set uses absolute address instructions everywhere

Yup, the stack and local variables don't ever need to be considered in
these sorts of schemes because they automatically work off where the
ESP register is and, thus, are "relocated" automatically...the stack
always is a relatively addressed data structure, anyway, by definition
that it's always worked out from the "top of stack" which ESP keeps track of


Ah...but if you're following good modular design practices, then you'd
never really want to have one module dip its hand inside another
module and play with the global variables...you would, quite
naturally, only allow access by passing a pointer or by(OOP-style)
doing the manipulations via the module's functions (the module doesn't
reach inside other modules but calls their functions to change things
"on behalf of" that module ...thus, if you were using good modular
design practice and / or OOP principles then this wouldn't be a
restriction at all, as you according to those principles you shouldn't
actually be directly tampering with another module's global variables,
anyway (after all, that's sort of the point of splitting things up
into modules so that they are made into independent chunks of code
that can work together or apart...if the module is directly reaching
into other modules, then it has a dependency on that other module and
it's spoiling its own "re-use" in doing that...that is why OOP
principles insist on the less-than-efficient "only methods can change
variables" thing to preserve "re-use" potential ...


Yup, that's what I was talking about in saying that the x86
architecture isn't designed for this sort of thing...you could
potentially "hack" out some "position independent code" by, as you
say, grabbing (E)IP and then using the "[ base + offset]" addressing
throughout and modifying any absolute address instructions (not
totally efficient but it could be made to work ...but it's just a
very tricky (okay, maybe "insane" is overstating it thing with the
x86...on other architectures, though, you're right...it's a breeze to
do because they automatically support this way of doing things that
it's not at all awkward to code...

Beth