I am creating a C# Windows application using the Leap Motion Controller. I am developing on Windows 8 and Visual Studio 2010. I use SetCursorPos and mouse_event from user32.dll to move the cursor and simulate clicks.
I want the cursor to be moved when in any application. When I run/debug it from Visual Studio it only works when in the application itself or in Visual Studio. When in other applications the mouse does not move and the clicks don't work, but if I try to move the cursor with the real mouse it goes back to the place it was at. When run standalone it does not move in Visual Studio and the mouse can be moved with the real mouse in other applications.
I have this code to use SetCursorPos and mouse_event:
[DllImport("user32.dll")]
public static extern long SetCursorPos(int x, int y);
[DllImport("User32.Dll")]
public static extern void mouse_event(uint dwFlags, uint dx, uint dy, uint cButtons, uint dwExtraInfo);
public static void MouseClick(uint x, uint y) {
mouse_event(MOUSEEVENTF_LEFTDOWN | MOUSEEVENTF_LEFTUP, x, y, 0, 0);
}
In my form class I have the mouse location stored in two fields. In a Timer's tick I set the cursor position and do necessary clicks like this:
if (!frame.Hands.IsEmpty) {
Hand hand = frame.Hands.Leftmost;
if (!hand.Fingers.IsEmpty) {
// Get coordinates...
SetCursorPos(mousex, mousey);
}
}
foreach (Gesture gesture : gestures) {
if (gesture.Type == Gesture.GestureType.TYPESCREENTAP) {
MouseClick(mousex, mousey);
}
}
The if statements are for the Leap device; I want to move the mouse and do other stuff only when there is a hand to get the coordinates.
Is it even possible to move the cursor or simulate mouse clicks for other applications? And if it is, how?
You've inspired me to refactor some of my automation code:
NativeMethods.cs - got most of this from online:
using System;
using System.Runtime.InteropServices;
public class NativeMethods
{
[DllImport( "user32.dll", SetLastError = true )]
internal static extern Int32 SendInput( Int32 cInputs, ref INPUT pInputs, Int32 cbSize );
[StructLayout( LayoutKind.Explicit, Pack = 1, Size = 28 )]
internal struct INPUT
{
[FieldOffset( 0 )] public InputType dwType;
[FieldOffset( 4 )] public MOUSEINPUT mi;
[FieldOffset( 4 )] public KEYBDINPUT ki;
[FieldOffset( 4 )] public HARDWAREINPUT hi;
}
[StructLayout( LayoutKind.Sequential, Pack = 1 )]
internal struct MOUSEINPUT
{
public Int32 dx;
public Int32 dy;
public Int32 mouseData;
public MOUSEEVENTF dwFlags;
public Int32 time;
public IntPtr dwExtraInfo;
}
[StructLayout( LayoutKind.Sequential, Pack = 1 )]
internal struct KEYBDINPUT
{
public Int16 wVk;
public Int16 wScan;
public KEYEVENTF dwFlags;
public Int32 time;
public IntPtr dwExtraInfo;
}
[StructLayout( LayoutKind.Sequential, Pack = 1 )]
internal struct HARDWAREINPUT
{
public Int32 uMsg;
public Int16 wParamL;
public Int16 wParamH;
}
internal enum InputType : int
{
Mouse = 0,
Keyboard = 1,
Hardware = 2
}
[Flags()]
internal enum MOUSEEVENTF : int
{
MOVE = 0x1,
LEFTDOWN = 0x2,
LEFTUP = 0x4,
RIGHTDOWN = 0x8,
RIGHTUP = 0x10,
MIDDLEDOWN = 0x20,
MIDDLEUP = 0x40,
XDOWN = 0x80,
XUP = 0x100,
VIRTUALDESK = 0x400,
WHEEL = 0x800,
ABSOLUTE = 0x8000
}
[Flags()]
public enum KEYEVENTF : int
{
EXTENDEDKEY = 1,
KEYUP = 2,
UNICODE = 4,
SCANCODE = 8
}
/// <summary>The MapVirtualKey function translates (maps) a virtual-key code into a scan
/// code or character value, or translates a scan code into a virtual-key code
/// </summary>
/// <param name="uCode">[in] Specifies the virtual-key code or scan code for a key.
/// How this value is interpreted depends on the value of the uMapType parameter</param>
/// <param name="uMapType">[in] Specifies the translation to perform. The value of this
/// parameter depends on the value of the uCode parameter.</param>
/// <returns>Either a scan code, a virtual-key code, or a character value, depending on
/// the value of uCode and uMapType. If there is no translation, the return value is zero</returns>
/// <remarks></remarks>
[DllImport( "User32.dll", SetLastError = false, CallingConvention = CallingConvention.StdCall, CharSet = CharSet.Auto )]
public static extern UInt32 MapVirtualKey( UInt32 uCode, MapVirtualKeyMapTypes uMapType );
/// <summary>The set of valid MapTypes used in MapVirtualKey
/// </summary>
/// <remarks></remarks>
public enum MapVirtualKeyMapTypes : uint
{
/// <summary>uCode is a virtual-key code and is translated into a scan code.
/// If it is a virtual-key code that does not distinguish between left- and
/// right-hand keys, the left-hand scan code is returned.
/// If there is no translation, the function returns 0.
/// </summary>
/// <remarks></remarks>
MAPVK_VK_TO_VSC = 0x0,
/// <summary>uCode is a scan code and is translated into a virtual-key code that
/// does not distinguish between left- and right-hand keys. If there is no
/// translation, the function returns 0.
/// </summary>
/// <remarks></remarks>
MAPVK_VSC_TO_VK = 0x1,
/// <summary>uCode is a virtual-key code and is translated into an unshifted
/// character value in the low-order word of the return value. Dead keys (diacritics)
/// are indicated by setting the top bit of the return value. If there is no
/// translation, the function returns 0.
/// </summary>
/// <remarks></remarks>
MAPVK_VK_TO_CHAR = 0x2,
/// <summary>Windows NT/2000/XP: uCode is a scan code and is translated into a
/// virtual-key code that distinguishes between left- and right-hand keys. If
/// there is no translation, the function returns 0.
/// </summary>
/// <remarks></remarks>
MAPVK_VSC_TO_VK_EX = 0x3,
/// <summary>Not currently documented
/// </summary>
/// <remarks></remarks>
MAPVK_VK_TO_VSC_EX = 0x4
}
}
MouseInput.cs
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Drawing;
using System.Linq;
using System.Runtime.InteropServices;
using System.Text;
using System.Windows.Forms;
namespace SendInput
{
public class MouseInput
{
public static void LeftClick()
{
DoMouse( NativeMethods.MOUSEEVENTF.LEFTDOWN, new System.Drawing.Point( 0, 0 ) );
DoMouse( NativeMethods.MOUSEEVENTF.LEFTUP, new System.Drawing.Point( 0, 0 ) );
}
public static void LeftClick( int x, int y )
{
DoMouse( NativeMethods.MOUSEEVENTF.MOVE | NativeMethods.MOUSEEVENTF.ABSOLUTE, new System.Drawing.Point( x, y ) );
DoMouse( NativeMethods.MOUSEEVENTF.LEFTDOWN, new System.Drawing.Point( x, y ) );
DoMouse( NativeMethods.MOUSEEVENTF.LEFTUP, new System.Drawing.Point( x, y ) );
}
public static void ClickBoundingRectangleByPercentage( int xPercentage, int yPercentage, System.Drawing.Rectangle bounds )
{
double additional = 0.0;
if ( xPercentage == 99 )
additional = 0.5;
int xPixel = Convert.ToInt32( bounds.Left + bounds.Width * ( xPercentage + additional ) / 100 );
int yPixel = Convert.ToInt32( bounds.Top + bounds.Height * ( yPercentage ) / 100 );
LeftClick( xPixel, yPixel );
}
public static void RightClick()
{
DoMouse( NativeMethods.MOUSEEVENTF.RIGHTDOWN, new System.Drawing.Point( 0, 0 ) );
DoMouse( NativeMethods.MOUSEEVENTF.RIGHTUP, new System.Drawing.Point( 0, 0 ) );
}
public static void RightClick( int x, int y )
{
DoMouse( NativeMethods.MOUSEEVENTF.MOVE | NativeMethods.MOUSEEVENTF.ABSOLUTE, new System.Drawing.Point( x, y ) );
DoMouse( NativeMethods.MOUSEEVENTF.RIGHTDOWN, new System.Drawing.Point( x, y ) );
DoMouse( NativeMethods.MOUSEEVENTF.RIGHTUP, new System.Drawing.Point( x, y ) );
}
public static void MoveMouse( Point p )
{
MoveMouse( p.X, p.Y );
}
public static void MoveMouse( System.Windows.Point p )
{
MoveMouse( Convert.ToInt32( p.X ), Convert.ToInt32( p.Y ) );
}
public static void MoveMouse( int x, int y )
{
DoMouse( NativeMethods.MOUSEEVENTF.MOVE | NativeMethods.MOUSEEVENTF.ABSOLUTE, new System.Drawing.Point( x, y ) );
}
public static System.Drawing.Point GetMousePosition()
{
return Cursor.Position;
}
public static void ScrollWheel( int scrollSize )
{
DoMouse( NativeMethods.MOUSEEVENTF.WHEEL, new System.Drawing.Point( 0, 0 ), scrollSize );
}
private static void DoMouse( NativeMethods.MOUSEEVENTF flags, Point newPoint, int scrollSize = 0 )
{
NativeMethods.INPUT input = new NativeMethods.INPUT();
NativeMethods.MOUSEINPUT mi = new NativeMethods.MOUSEINPUT();
input.dwType = NativeMethods.InputType.Mouse;
input.mi = mi;
input.mi.dwExtraInfo = IntPtr.Zero;
// mouse co-ords: top left is (0,0), bottom right is (65535, 65535)
// convert screen co-ord to mouse co-ords...
input.mi.dx = newPoint.X * 65535 / Screen.PrimaryScreen.Bounds.Width;
input.mi.dy = newPoint.Y * 65535 / Screen.PrimaryScreen.Bounds.Height;
input.mi.time = 0;
input.mi.mouseData = scrollSize * 120;
// can be used for WHEEL event see msdn
input.mi.dwFlags = flags;
int cbSize = Marshal.SizeOf( typeof ( NativeMethods.INPUT ) );
int result = NativeMethods.SendInput( 1, ref input, cbSize );
if ( result == 0 )
Debug.WriteLine( Marshal.GetLastWin32Error() );
}
}
}
Yes, you want to use the "SendInput" function.
see: SendInput doesn't perform click mouse button unless I move cursor
Also,
http://www.pinvoke.net/default.aspx/user32.sendinput
and
http://www.pinvoke.net/default.aspx/Structures/INPUT.html
I've fixed the first hyper link- see the comment about:
There are a few things you should consider when using the SendInput function.
If you do not specify the MOUSEEVENTF_ABSOLUTE flag then dx and dy (MouseInputData structure) are relative coordinates to the current mouse position. If you do specify MOUSEEVENTF_ABSOLUTE then dx and dy are absolute coordinates between 0 and 65535. So if your x and y coordinates are screen coordinates you should use the following function to calculate dx and dy:
and
http://msdn.microsoft.com/en-us/library/ms646310%28VS.85%29.aspx
I managed to solve the problem myself.
There were two problems. The first (and the main) one was that the Leap device was not sending frames to my application when it was in the background. That was fixed by adding this code to the listener's onConnect() method, as suggested here:
controller.SetPolicyFlags(Controller.PolicyFlag.POLICYBACKGROUNDFRAMES);
The second problem was that when run standalone the application didn't have access to send input to other applications. I followed instructions found here (scroll down, there's info about uiAccess) and here and signed my application and added this to the manifest:
<requestedExecutionLevel level="asInvoker" uiAccess="true" />
