window_shape cheat sheet

This is a "cheat sheet" for "window_shape" extension by YellowAfterlife.
The extension can be found on itch.io.
Click on sections to expand/collapse them.
Quick display controls: Categories · Sections · Everything · Toggle night mode

Concepts

This extension uses `SetWindowRgn`, which uses GDI regions.

GDI regions are vector-based (consisting of rectangles, polygons, and ellipses) rather than pixel-based, which means that you can't have semi-transparent pixels on the window edge, but you can do binary operations on shapes (like combining them together or applying a mask).

If you've ever worked with vector graphics editors, this should sound familiar.

There is a newer API (`UpdateLayeredWindow`) that's pixel-based, but it requires broader changes to how application renders graphics, which I would expect to backfire in a variety of unpredictable ways.
Update! I found a workaround with a relatively small number of caveats.

Some functions consume the inputs, meaning that the underlying native object is destroyed and the reference is invalidated - you don't have to call window_shape_destroy on them and you cannot use them in further functions.

Changing a window's shape forces a redraw

Setting up

- The underlying API is primarly intended for use with borderless windows.
  While you can use it with regular windows, doing so degrades the window to some kind of Win2K-era appearance... you have to see it yourself, really.

  You can enable borderless window mode in

  - GMS1: Global Game Settings - Windows - Graphics
  - GMS2: Game Options - Windows - Graphics

- Import the extension to your project:

  - GMS1: Right-click "Extensions" in resource tree, pick "Import", pick the GMEZ file.
  - GMS2.2: Drag-and-drop the YYMP file onto the workspace area of the IDE.
  - GMS2.3: menu:Tools - Import Local Asset Package
- Call `window_shape_init()` somewhere on game start / before using other functions.
- Create a shape
- Apply it to the window

window_shape_init()

Initializes the extension.

Should be ran before doing anything else.

Creating shapes

Coordinates are in pixels relative to window's top-left corner.

window_shape_create_empty()​➜

Creates an empty shape (formally a zero-size rectangle).

Basic shapes
window_shape_create_rectangle(x1, y1, x2, y2)​➜

Creates a rectangle shape.

window_shape_create_round_rectangle(x1, y1, x2, y2, w, h)​➜

Creates a rounded rectangle shape (w, h being corner radii).

window_shape_create_ellipse(x1, y1, x2, y2)​➜

Creates an elliptical shape.

window_shape_create_circle(x, y, radius)​➜

Creates a circular shape, by which I mean: this is just a shorthand for the ellipse shape.

Polygons
window_shape_create_polygon_from_array(array, mode, ?count)​➜

Takes an array of x,y coordinate pairs and creates a polygon shape out of it.

`mode` can be:

- `window_shape_polygon_mode_winding`
  The polygon is filled normally.
- `window_shape_polygon_mode_alternate`
  XOR-like logic is applied to polygon when it intersects itself.

`count` is the number of points in the array.
If not specified, it is set to (length of the array)/2.

For a practical example, if you do

    var arr = [];
    var angle = 30;
    var mode = window_shape_polygon_mode_alternate;
    repeat (5) {
        array_push(arr,
            200 + lengthdir_x(200, angle),
            200 + lengthdir_y(200, angle),
        );
        angle += 360/5*2;
    }
    window_shape_set(window_shape_create_polygon_from_array(arr, mode));

with "alternate" mode, the star would have a hole in the middle, while with "winding" mode it won't.

window_shape_create_polygon_from_buffer(buffer, mode, count = -1)​➜

Same as above, but accepts a buffer with series of `buffer_s32` x,y coordinate pairs.

If `count` is -1, it is set to `buffer_tell(buf) div 8`.

Buffer can be safely destroyed afterwards.

window_shape_create_polygon_from_path(path, mode)​➜

Creates a shape from a GameMaker path.

Supports smooth paths somewhat accurately.

""Pixel""" shapes

By which I mean, a pile of rectangles (one per row of connected pixels).

These are kind of slow and I strongly encourage you to use polygons instead.

window_shape_create_rectangles_from_rgba(buffer, tolerance, width, height)​➜

Creates a shape from pixel data in a buffer - any pixel with opacity above `tolerance` will be filled.

`width` and `height` specify dimensions, in pixels.

window_shape_create_rectangles_from_surface(surface, tolerance)​➜

A convenience wrapper for above, creates a shape from pixels in a surface.

window_shape_create_rectangles_from_sprite(surface, tolerance)​➜

A convenience wrapper for above, creates a shape from pixels in a sprite.

Shape manipulations
window_shape_duplicate(shape)​➜

Creates an independent copy of the given shape.

window_shape_destroy(shape)​➜

Destroys the given shape.

Transformations:

window_shape_shift(shape, xoffset, yoffset)​➜

Moves the shape around by a specified offset.

window_shape_transform(shape, m11, m12, m21, m22, dx, dy)​➜

Transforms a shape through a 2d matrix, but... the documentation for underlying function says that this converts the shape to rectangles first (??), so the results vary.

The following would give the window an appearance of a slightly skewed circle, for example

    var _circle = window_shape_create_circle(100, 100, 100);
    var _transf = window_shape_transform(_circle, 1, -0.2, -0.2, 1, 50, 50);
    window_shape_set(_transf);
    window_shape_destroy(_circle);

Binary operations:

window_shape_combine(shape1, shape2, op)​➜

Combines the two shapes into a new one and returns it.

The inputs are consumed in the process.

`op` can be one of the following:

- `window_shape_operation_and`
  The result is the overlap between the shapes.
- `window_shape_operation_copy`
  The result is the first shape.
- `window_shape_operation_diff`
  Subtracts shape2 from shape1.
- `window_shape_operation_or`
  The result is a union between two shapes.
- `window_shape_operation_xor`
  The result is an exclusion between two shapes (overlaps are empty).

See MSDN for an illustrated example of what each mode does.

The following would give the window an appearance of a donut (a circle with a hole in the middle):

    window_shape_set(window_shape_combine(
        window_shape_create_circle(100, 100, 100),
        window_shape_create_circle(100, 100, 40),
        window_shape_operation_diff
    ));

window_shape_combine_nc(shape1, shape2, op)​➜

Like `window_shape_combine`, but does not consume the shapes.

window_shape_concat(shape1, shape2, op)​➜

Like `window_shape_combine`, but the result is kept in `shape1` instead of creating a new one.

The second shape is consumed in the process.

Returns whether successful.

window_shape_concat_nc(shape1, shape2, op)​➜

Like `window_shape_concat`, but does not consume anything.

Utilities:

window_shape_contains_point(shape, x, y)​➜

Contains whether the point is within a shape.

For purposes of checking whether the mouse is over a custom-shaped window, you will probably want to use `display_mouse_get_x() - window_get_x()` rather than `window_mouse_get_x()` (which would only update while the mouse is over the window).

window_shape_contains_rectangle(shape, x1, y1, x2, y2)​➜

Contains whether a rectangle is within a shape.

Setting and resetting
window_shape_set(shape)

Applies the given shape to the game window.

This function consumes the shape.

That isn't my idea - the WinAPI function has it that after the call the region is confiscated from you and the system will manage it (including destroying it once the window closes or a different region is applied).

window_shape_set_nc(shape)

Like above, but does not consume the shape (by giving WinAPI a copy rather than the original).

window_shape_reset()

Returns your window to its original, rectangular form.

Alpha

This is a simple multiplier that applies over any other window shape manipulations.

window_set_alpha(alpha)

Changes the game window's opacity.

`alpha` should be in 0 .. 1 range.

window_get_alpha()​➜

Returns the game window's current opacity.

Chromakey

This can similarly be used with other shape modifiers.

window_set_chromakey(color)

Changes the game window's chromakey color.

Pixels that fully match this color will be see-through and click-through.

Pass `-1` instead of the color to disable chromakey.

window_get_chromakey()​➜

Returns the game window's chromakey color (`-1` if disabled).

Per-pixel alpha
window_enable_per_pixel_alpha()

This function calls DwmEnableBlurBehindWindow for a zero-sized region.

Doing so also happens to enable partial transparency support for the contents of the window ("The alpha values in the window are honored").

For example, if you did

    draw_clear_alpha(c_black, 0);

in a Draw event of the only instance in a room, your window would look like an empty frame.

For a proper setup, make sure that "Clear Display Buffer", "Enable Viewports" (with at least one view set up), and "Clear Viewport Background" are enabled for the room; see `desktop_friend` project for a slightly more sophisticated example.

Now let's talk about the conditions and caveats:

Premultiplied alpha

Windows expects the window pixels to have premultiplied alpha.

In short, instead of just `color`, the pixels should be colored with

    merge_color(c_black, color, alpha)

For sprites, all recent GameMaker versions have a "Premultiply alpha" checkbox in the Texture Settings;

For text and primitives, you'll need to use the above formula;

For mixing multiple semi-transparent images together, see the classic GameMaker surface problems and solutions.

If you don't use pre-multiplied alpha, your window will still work, but the semi-transparent bright portions will appear lighter than they should be.

Hit testing

According to Windows, just because a pixel is now almost or completely transparent, that doesn't mean that you can't click it!

You can use the extension's other functions (be it shapes or chromakey) to narrow down the window's shape, or use the window_set_clickthrough function from my free Window Commands extension to make the window clickable/un-clickable based on where the cursor is.

One-way ticket

The function has been around since Windows 7, yet it doesn't seem like you can go back to an opaque window once you have called it - even if you pass `dwFlags` of `0`.

You can "top-up" the opacity of the window contents by drawing a black rectangle with `bm_add` blend mode or any rectangle with `gpu_set_colourwriteenable` (or `draw_set_colour_write_enable` for GMS1) of `(0, 0, 0, 1)`.