Here is the situation. I have a Visual C++ 6.0 Dll that contains a dialog
box. I have a function declared in the dll that is available for export
(declared using DllExport) that creates an instance of the dialog, calls
DoModal(), and then does some data manipulation. The function needs to return
a BYTE*, whether it be as a param or a return value I don't care, but it
seems as though C# only allows return parameters. I've created a class in
my C# App and have imported the above function using DllImport(). Here is
my problem. When I call the dll function from C#, nothing happens. It appears
as though the function is getting called (I've done some debugging to determine
that it actually gets into the function), but the dialog is never displayed
(it seems like it always defaults to IDCANCEL). Therefore, my return parameter
does not get set. I've also tried creating a simpler function that just
changes the value of an integer that gets passed in. This also fails, giving
me an error message like Object reference not set, or something like that.
What am I doing wrong? Can anyone give me an in-depth example or point
me in the right direction? The docs are not helpful at all on this topic.
If you need more info, or even source code to better understand the situation,
I would be glad to provide it.