Proposal: Improved Cursor APIs

[adrientetar]

Proposal: Improved Cursor APIs

Summary

UWP's story around custom cursors currently is suboptimal – one has to create a .cur file and build it into a .rc file packaged at compile time in order to use custom cursors in the code. We ought to allow runtime-creation of cursors (thereby bringing feature parity with win32) and more broadly consider more ergonomic APIs in order to facilitate building and iterating on cursors for developers so they can bring a more efficient and tailored experience to their users.

Rationale

Custom cursors in UWP are fabricated from a .cur file, a collection of BMP images of differing sizes, and packaged into the AppX bundle which means:

1. Custom cursors are necessarily static, i.e. they can't be created dynamically at runtime. That's a significant limitation for complex apps, or apps that wish to let their users extend the core functionality through plugins (Adobe XD, for instance could be interested in this proposed API).
   This is a limitation of UWP specifically, as win32, Cocoa, or other toolkits such as GTK all have APIs to create a cursor from a bitmap object (as opposed to packaged assets).
2. Developers currently have to go through a tedious process of generating bitmaps, packaging them and fetching the resulting resource ID from resource.h in order to even see their cursor on screen. This makes custom cursors hard to use.
   For instance, Paint3D uses built-in OS cursors for almost all of its tools, where mspaint did use fully custom cursors all-around. Custom cursors signal the currently active tool to the user, or with special markers the action that will result from clicking at a particular spot (hit-testing, see pictures below), and as such are critical for high-quality experience in design applications (e.g. Adobe apps, Office etc.).
   When talking about ergonomics, I do think we should consider how to facilitate handling of different DPI sizes, i.e. have an API to specify cursor as vector image of some sort, as was done for splashscreen icons (though that might not be desirable if antialiasing is a concern).
3. A common scenario of making custom cursors is what I refer to as "markers". A marker is a small icon that adds to the base system icon at a certain point (e.g. the spinner pointer) – this is often used in canvas-based apps to signal that clicking on this particular spot will result in a specific action taken.
   It would be great to have some way to reuse system cursor and add an extra symbol to them (kinda like the spinner pointer, see below), I'm thinking an API that would "decal" the system cursor into a bitmap, alternatively specify to use a system cursor and then the custom color is overlaid with it. [Maybe even allow specifying a scaling factor (e.g. 3/4), if the source of system assets are vectorized that shouldn't technically be a problem.]
   
   
   This is Fontlab 5 on Windows (font editor), showing special highlight when hovering over a point. This lets the user know that they're hit-testing the point, since the threshold of hit testing for such small items isn't necessarily intuitive esp. while one is moving quickly over the canvas
   
   Glyphs on macOS
4. A common use-case of custom cursors is to set a cursor on a given widget; we could add a Cursor property to controls. For reference the current API goes like this:

void OnPointerEntered(object sender, PointerRoutedEventArgs e)
{
    Windows.UI.Xaml.Window.Current.CoreWindow.PointerCursor = new Windows.UI.Core.CoreCursor(
            Windows.UI.Core.CoreCursorType.Hand, 1);
}
void OnPointerExited(object sender, PointerRoutedEventArgs e)
{
    Windows.UI.Xaml.Window.Current.CoreWindow.PointerCursor = new Windows.UI.Core.CoreCursor(
            Windows.UI.Core.CoreCursorType.Arrow, 1);
}

TL;DR: great apps, primarily document-centered or design apps do need custom cursors, and we ought to make it easy for developers to prototype and iterate on them. Would be great if a XAML PM could spearhead this effort.

Functional Requirements

#	Feature	Priority
1	A developer can create and set a custom cursor at runtime (or set of diff. size bitmaps)	Must
1b	Cursor customization easily supports different DPIs (for example, could accept SVG or other vector sources)	Should
2	Custom cursors are specified in a version control-friendly way (no bin files)	Should
3	Developer can specify a control's custom cursor by setting the Cursor property	Could
4	A developer can create a cursor from a system cursor + their own extra drawing	Could
5	API to create an animated cursor?	Could

Important Notes

1. In win32 a cursor can be created from a bitmap at runtime using CreateIconIndirect()
2. In regards to item #‌4, the community toolkit does have an extension for setting a system cursor over a specific UIElement in XAML.

Open Questions

We might want to think about animated cursors also (though that's not a priority for me). If there's an API for making a cursor from bitmap should there be one for animated cursors (e.g., a series of bitmaps?)

[mdtauk]

Microsoft is overhauling the Cursor settings in 19H1 to include scaling and colourisation. Whatever APIs get added, there needs to be consideration with how custom cursors could work with that,

Perhaps the cursors can become SVG based, and colourised by the OS. Font Glyph based, or masked so they adopt the colours chosen.

Then what about inverted states, and cursor shadows/trails?

[michael-hawker]

As called out for the reason for custom cursors being needed, another auxiliary topic that can be discussed here is that it's a chore to change them dynamically, especially to track the cursor last used if moving between things that both have custom cursors.

It's why we have the Mouse.Cursor XAML extension property in the toolkit. It may be helpful to think about if our approach from the toolkit could be improved being part of this work or otherwise? I know there were some issues with FrameworkPropertyMetadata.Inherits not being supported in UWP.

[adrientetar]

So I think 19H1 is feature complete by now but is this something that could get added to 19H2, @ryandemopoulos ?

[ryandemopoulos]

@adrientetar My apologies for the long delay on this. I should introduce myself--I'm a PM Lead for WinUI, and @SavoySchuler's manager. Savoy has mentioned to me that you've been helpful on the NumberBox issue--so let me say "thanks" for your help!

To answer your 19H2 question, as you may have already gathered from our WinUI Roadmap, there isn't really a "19H2" release for WinUI--we're decoupled from the Windows ship cycle so our releases will be numbered 2.2, 2.3, 3.0, 3.1, etc.. from this point forward, with these releases happening every 4 months.

To determine if/when we could get this functionality added, a few questions first (and I'd like to add that you're write-up was hugely helpful--I'm not a cursor expert so thanks for all the info!):

1. Am I correct in understanding that cursors can be swapped at runtime, but the main problem is that it's tedious to do this (having to create them ahead of time as resources rather than just a simple bitmap load)?
2. For augmenting a system cursor with some additional graphic, do you have a sense of whether developers would actually use this API vs. just providing their own pre-built system+custom+graphic cursor? I'm wondering how this was done in Win32, and whether software like FontLab augmented the system cursors (via API), or whether they just pre-built those cursors themselves ahead of time to have the system cursor look.
3. I'm assuming you'd like this proposal to have both options of just changing the cursor on a whim, but also the WCT option of a Cursor property on specific UIElements. (correct?)
4. For animating the cursor, is there a standard asset format that was used in Win32 for doing this?

[mdtauk]

I can help with 4.
In Win32 there is the .cur format for cursors, as well as .ani for animated cursors.
Wikipedia ANI file format

I am not sure if these formats work with the new Cursor settings in Windows 10.

[ryandemopoulos]

@mdtauk Thanks for pointing out the new Cursor settings and the formats. If we decided to do this work we'd have to look into whether these new settings use that. I definitely like the idea of just re-using the existing formats if they're already familiar to Windows developers and there is already a corpus of these out in the wild that would be easy to bring forward when adopting WinUI.

[adrientetar]

@ryandemopoulos Thanks for getting back to me! I understood you being a notch higher in the hierarchy means you aren't as availabel.

To answer your 19H2 question, as you may have already gathered from our WinUI Roadmap, there isn't really a "19H2" release for WinUI

Well, I thought cursor APIs might be in the platform rather than in WinUI, but with the 3.0 coming I'm not sure if that's still the case (I wrote the proposal before it was announced).

Am I correct in understanding that cursors can be swapped at runtime, but the main problem is that it's tedious to do this (having to create them ahead of time as resources rather than just a simple bitmap load)?

Right, but the main point is it's impossible to spawn cursors dynamically at runtime. If I want to add plugins support to my app, and someone writes a plugin that adds a new drawing tool, currently they couldn't add a custom cursor as part of it. This API gap is really the most important point in this issue.

For augmenting a system cursor with some additional graphic, do you have a sense of whether developers would actually use this API vs. just providing their own pre-built system+custom+graphic cursor? I'm wondering how this was done in Win32, and whether software like FontLab augmented the system cursors (via API), or whether they just pre-built those cursors themselves ahead of time to have the system cursor look.

No, apps like Fontlab or Adobe XD or Acrobat Reader just roll their own set of cursors entirely including the arrow pointer, with different design than the OS. I did the same in the previous iterations of my app, I could try to mimic the stock arrow pointer but Windows has a different way of drawing them at different DPI and if the user changes it somehow (customization) I wouldn't be inline with it.
But having a canvas control that shows a different arrow cursor style when you hover that control is always kinda weird, so being able to add to the system cursor would be something I'd use personally. It would help apps blend it more with the OS's look while also extending it. I don't think any GUI toolkit has such an API, so I think this would be a novel thing.

I'm assuming you'd like this proposal to have both options of just changing the cursor on a whim, but also the WCT option of a Cursor property on specific UIElements. (correct?)

That would be nice to have. Though as I said in response to your first question, some items in this proposal are of higher priority than others (like this one).

For animating the cursor, is there a standard asset format that was used in Win32 for doing this?

.cur and .ani as mdtauk said, though these are quite old formats by now so I'm not sure if they're still valid for modern APIs, but I could look into it.. Mainly I'm thinking in regards to support for multiple DPI displays that we might want to have SVG-style drawing commands that re-rasterize when DPI changes, kinda like Windows redraws things like the taskbar icons or mouse cursor to adapt to the current DPI (since DPI scaling is a slider, you would need hundreds of assets otherwise). Do you know if that ability could be exposed as part of the custom cursor API?)

[mdtauk]

You could work in the modern resource system, so add "TargetSize" or "Scale200" to filenames for the cursor images - but if the new cursor system in the OS is now vector, and can be recoloured - maybe all you need to provide in the app is an SVG or PATH element?

[ryandemopoulos]

@adrientetar, @mdtauk OK, I think I have my mind wrapped around this--thanks for helping me understand the space! (@adrientetar FWIW, I really like the novel idea here of having an additive graphic to the existing system cursor--clever idea. :)

I've added the needs-winui-3 label to this since I think this will require that work to be done first, so I think we'll leave this in the freezer for a while until we get closer to having the release come out.

@adrientetar I also took the liberty of re-wording the second scoping line a little bit--I focused on the ability to have the solution adapt to the modern reality of different DPIs (and one potential way to address that would be SVG, but I wanted the driving goal of DPI awareness/adaptibility to be very apparent in the wording). Let me know if that change looks good to you.

Assuming this is all good, we can revisit later when WinUI 3 gets closer. I think once the first pre-release is out, then that might be a good time to revisit.

I should also note that I don't have a sense of where this will fall in terms of priority; we're starting to accumulate a lot of "we'd like to do this thing, but it needs WinUI 3 first" items, and I can foresee a resourcing logjam coming where even once WinUI 3 becomes a reality we'll still have a lot of competing priorities to get to stuff. I'm hoping that the community can help out with some of these features (providing PRs for them), so this could end up in that camp if we can't prioritize it down the road.

[ryandemopoulos]

(and I'll unassign myself from it now--our pattern of items that need to wait for WinUI 3 is to unassign them for the time being, and revisit later)