Delphi open modal form from DLL

I need to add some plugin functionality to application, and ability to dynamically load and open plugins.

In my application (in main form), I have following code:

procedure TfrmMain.PluginClick(Sender: TObject);
Var
  DllFileName : String;
  DllHandle   : THandle;
  VitoRunPlugin : procedure (AppHandle, FormHandle : HWND);
begin
  DllFileName := (Sender AS TComponent).Name + '.dll';
  DllHandle := LoadLibrary(PWideChar (DllFileName));

  if DllHandle <> 0 then
  Begin
    @VitoRunPlugin := GetProcAddress (DllHandle, 'VitoRunPlugin');
    VitoRunPlugin (Application.Handle, Self.Handle);
  End Else Begin
    ShowMessage ('Plugin load error');
  End;

  FreeLibrary (DllHandle);
end;

And my plugin library is (just for testing now):

library plugintest;

uses
  System.SysUtils, WinApi.Windows,
  Vcl.Forms,
  System.Classes,
  Vcl.StdCtrls;

{$R *.res}

Procedure VitoRunPlugin (AppHandle, FormHandle : HWND);
  Var F : TForm;  B: TButton;
Begin
  F := TForm.CreateParented(FormHandle);
  F.FormStyle := fsNormal;

  B := TButton.Create(F);
  B.Left := 5; B.Top := 5; B.Height := 50; B.Width := 50;
  B.Caption := 'Touch me!';
  B.Parent := F;

  F.ShowModal;
  F.Free;
End;

exports VitoRunPlugin;

begin
end.

The form opening OK, but nothing working: I can't neither press the button, nor close the form. I can only close it by Alt+F4.

What's wrong?

CreateParented makes the form be a child window. And you cannot show a child window modally. So, who knows what happens when your form is shown? I'm sure I cannot predict what happens when you pass a VCL form window handle to the CreateParented constructor of another VCL form.

Change the form creation to be like so:

F := TForm.Create(nil);

In order to make the form have the right owner (and here I mean owner in the Win32 sense), you may need to override CreateParams as follows:

procedure TMyForm.CreateParams(var Params: TCreateParams);
begin
  inherited;
  Params.WndParent := FormHandle;
end;

Obviously you'll need to declare a derived TMyForm class add some plumbing to allow its overridden CreateParams method to gain access to the owner form handle.

If you want the button to do something, you need to code that. Either an an OnClick event handler, or set the button's ModalResult property.