Re: [Ctypes-devel] cinvoke

[Eyal L. <eya...@gm...>]

On 10/21/05, Thomas Heller <th...@py...> wrote:
> I've only looked briefly at the code, so I don't have detailed comments.
> One disadvantage I see so far is that cinvoke would only work when
> compiled with gcc on an x86 system - correct?

Well, it has a bit of inline asm to call arbitrary function pointers
with arbitrary stacks. But that is a trivial piece of inline assembly,
and only that needs to be ported to various architectures.

> Then: ctypes is used to call functions in dlls/shared libraries, without
> the need to write a Python extension.  It seems (but I can only guess
> here, you have to provide use cases, rationales, and code examples),
> that cinvoke is compiled into a C application, and allows to call
> functions in that program from another process.  So, is this to allow to
> script applications?
Nope. It is mainly used to test C libraries and programs.
Though scripting C programs with it is also much easier than embedding
a Python interpreter, and more powerful.

> And, let me ask:  What's your real use case, and why can't you use
> ctypes for that?
Because I cannot use ctypes to control a C program that has its own
mainloop and cannot be embedded as loadable library.
Because if I use ctypes, my C program is must run in the context of
the Python interpreter process, and testing it becomes less
representative, in terms of its memory management, signal handlers it
installs, etc.  With cinvoke, the C program can live in its "natural"
environment.
Because I cannot use ctypes to control multiple C processes that
communicate with each other, especially when those C processes need to
sit on separate computers.  I have already needed this to test complex
C programs.

> There are several Python modules to allow access and manipulate C
> compatible data structures, the standard struct module, higher level
> wrappers around the struct module, and ctypes-lookalike type module (Bob
> Ippolito has 'ptypes', iirc).  Personally, I don't have a large need to
> manipulate these structures without beeing able to pass them to
> C-functions, so for me it makes no sense to separate them.

But the idea is to use the same data type library to represent C data,
AND to use those data variables to pass them to functions. This is
what cinvoke does.
I did this in cinvoke because I wanted to be able to directly access
the data symbols of remote processes and manipulate them without
having to create duplicate data type declarations for function calls
and for data manipulation and representation.

Note that the data library within cinvoke is generically more powerful
than existing struct modules and such, for at least some purposes.  It
allows "attaching" of a data-type to an arbitrary "space" object, and
defining pointer dereference semantics by assigning an "address_space"
attribute into pointers. This "space" object can be the actual address
space of a remote cinvoke server, or just a portion of a file. This
allows to "map" data types into files, dereference pointers in files
or in the address space via the same parsing code, and so on.

This data manipulation, by the way, is also very useful for testing.
Especially as with cinvoke, you can have multiple cinvoke servers
running on multiple computers, and then you can use:

# Get a symbol from aprocess's address space:
a_symbol =3D a.symbol(MyBigStruct, 'some_symbolname')
# Call a function in bprocess's address space with a pointer to a's symbol:
b.call.modify_given_ptr(node_ptr(a_symbol))

And this will transparently copy a_symbol into the context of
bprocess, run the function, and then transparently copy the data back
into a_symbol in the context of aprocess.

Also, if I want to test a C function's data corruption handling, I can use:
a_symbol.nodes.some_data_member.value =3D Corrupted_value
assert TYPE_OF_FAILURE =3D=3D a.call.some_api(node_ptr(a_symbol))

> Finally, function pointers are also C data types, so the separation
> between the type library and the 'call' module is not so clear.  And,
> nowadays ctypes is portable to quite a lot of systems (with the help of
> libffi, of course).
Well, cinvoke is portable to a lot of architectures in the same way
that Linux is, except there is only one trivial asm function to port.
Function Pointers are the only C data type that is only useful when
calling, so it makes sense for this data type to be tightly coupled to
ctypes/cinvoke. However, it does not make sense to define the same
structs, unions and enums multiple times in order to call functions
with it and in order to parse/build/manipulate data with it.