Develop for Shortcuts to Process Images

The Shortcuts app is the official automation solution supported throughout the Apple ecosystem. It allows the user to fit your app into their workflow, and you don’t even have to design UI for the action.

The idea of the Shortcuts action in BORD, my app that un-crops photo and adds adaptive colour backgrounds.

In this tutorial, I will demonstrate how you can develop and expose a Shortcuts action in the Shortcuts app. In my app, BORD, this action adds coloured borders to your photos, so that a 2:3 portrait DSLR photo can fit Instagram’s 4:5 aspect ratio requirements.

For sake of the tutorial, we will develop an action that pass through input as output. However, you are free to do any image operations on the input as you wish.

Table of contents—

• You Don’t Need an App Extension
• Project Setup
• Write Some Code
  • Step 1: Create a handler class
  • Step 2: Tell Application Delegate who’s the handler
• Test It Out
• Next Steps

You Don’t Need an App Extension

Let’s get this out of the way—when developing for iOS 14 or 15, you do not need an App Extension to develop for Shortcuts.

Conventionally, when you search for a tutorial on SiriKit and Intents, you are instructed to go through a step that creates an App Extension. But it’s not necessary (and frankly won’t be a viable route) for our purpose: to develop and expose a Shortcut action that processes photos.

Why are App Extensions not necessary? Starting with iOS 14, SiriKit Intents can be handled by the main app, without an SiriKit Intent App Extension. Without adding an App Extension target to the app bundle, things are easier for the developer (that’s you). Your app can still expose a Shortcuts action, and this action can process input without the app having to open in the foreground.

Why are App Extensions not an option? Also starting with iOS 14, App Extensions are assigned very limited resources. This was aligned with the introduction of iOS Widgets, which are also powered by SiriKit Intents. Once the App Extension reach 30 MB of RAM use, the extension is terminated. This is far below what’s typically required for image processing.

Why do most tutorials instruct us to create an App Extension target? The main reason is that historically, App Extension used to be the only way for Intents to work in Shortcuts.¹ Therefore, most tutorials you can find—including the SiriKit introduction on Apple’s documentation website—will instruct developers to use an App Extension. Having a separate App Extension target also helps you organize your code, and is beneficial when you develop a complex app that run on both iOS and watchOS, as an example.

Project Setup

Before we go to coding, there are a few project setups to do.

1. Enable Siri in Capabilities. Go to your app project, under Targets, select the app bundle. Go to “Signing & Capabilities” and add “Siri”.

   

2. Add Intents Definition file. Click File -> New -> File… and select SiriKit Intent Definition File.

   

3. Define your Intent. For this tutorial, we name it ProcessImage. Apple advises developers to adopt a verb-noun idiom.
   1. In the Intent Definition file, add a new custom intent and name it ProcessImage.
   2. Under Parameters, select File in the Type dropdown menu.
   3. Check Supports multiple values.
   4. Check User can edit value in Shortcuts, widgets, and Add to Siri.
   5. Uncheck other boxes.
   6. Under Shortcuts app, select inputFiles for Input Parameter.
   7. Select inputFiles for Key Parameter.
   8. Enter readable text in Summary in each of the Supported Combinations items.

   

4. Define your Intent response.
   1. Click Response of the ProcessImage intent.
   2. Under Properties, add a new property outputFiles. Make it of File type, and check Supports multiple values.
   3. Also under Properties, select outputFiles in the dropdown menu.

   

5. Configure your app to support multiple windows. This option doesn’t appear if it’s an iPhone-only app. In that case, temporarily check iPad under Deployment Info to expose Support multiple windows. Check Support multiple windows and uncheck iPad.

6. Expose custom Intent for app to handle. Go to the app bundle’s settings, click General, and under Supported Intents add the ProcessImage class name. Leave authentication to None as this intent doesn’t access any sensitive information, or has any real-world impact such as monetary transactions.

7. Do not link Intents framework in the main app bundle.

Write Some Code

Now that we have Siri Intents and project set up, we are ready to write some code. There are simply two steps.

Step 1: Create a handler class

The Intent Definition file generated a few classes and protocols:

• The class ProcessImageIntent, which has property inputFiles
• The protocol ProcessImageIntentHandling
• The class ProcessImageIntentResponse and its related enum ProcessImageIntentReponseCode. The response class has outputFiles property

Now we need to create a handler class to handle the generated intent. To do that, create a new Swift file and use the following code:

import Intents

class ProcessImageIntentHandler: NSObject, ProcessImageIntentHandling {

	func handle(intent: ProcessImageIntent, completion: @escaping (ProcessImageIntentResponse) -> Void) {
			guard let inputFiles = intent.inputFiles else {
				print("Failed to get file from input")
				completion(.init(code: .failure, userActivity: nil))
				return
			}
			let resp = ProcessImageIntentResponse(code: .success, userActivity: nil)
			resp.outputFiles = inputFiles.compactMap({processInputFile($0)})
			completion(resp)
	}
	
	private func processInputFile(_ inputFile: INFile) -> INFile? {
		// The code below is a passthrough. Note you cannot pass the inputFile 
		// directly; you have to generate a new one.
		let outputFile = INFile(data: inputFile.data, filename: inputFile.filename, typeIdentifier: inputFile.typeIdentifier)
		return outputFile
	}

}

Note that in the above code, processInputFile(_:) uses optionals in case certain image files cannot be processed. In the handle(intent:,completion:) method, we use compactMap to directly omit any files that cannot be processed. This passthrough code cannot fail, but it is practical once you operate on image files. You may want to implement your own logic for handling stuff.

Step 2: Tell Application Delegate who’s the handler

Now that you have a class that handles ProcessImageIntent, you need to tell the app which class can handle the processing.

Under AppDelegate, add the following piece:

class AppDelegate: UIResponder, UIApplicationDelegate {

	// ...

	func application(_ application: UIApplication, handlerFor intent: INIntent) -> Any? {
		switch intent {
		case is ProcessImageIntent:
			return ProcessImageIntentHandler()
		default:
			return nil
		}
	}

}

Now your app knows who to look for when they receive a ProcessImageIntent instance. It will launch without a UIWindowScene instance attached.

Build your app and run it once, so that the app register with Shortcuts in the system.

Test It Out

Open the Shortcuts app on your testing device. Add a new shortcut that looks like below:

This Shortcuts action works like any other action—it doesn’t open the app in the foreground, and returns your input photos directly as output. You are free to use this action with a Share Sheet–invoked shortcut, too.

When you want to debug the action, you debug it as you’d normally would debugging the main app—run the main app, go to Home Screen, and launch Shortcuts. If you step through code, you can see breakpoints as you normally would.

There you have it!

Next Steps

There are a few steps to polish up your Shortcuts action:

1. Implement image processing, so that your Shortcuts action actually does something. Currently it passes through input as output.
2. Provide more options with this Shortcuts action, by adding enums and other values. My app BORD, for example, asks the user to choose aspect ratio, background colour, and border width.
3. Localize your SiriKit Intents definition. There are tutorials that you can easily find with Google, so I’m not repeating them here.