A Summary of the WWDC25 Group Lab - Apple Intelligence

Question

DTS Engineer OP

Apple

Created 1w

Replies 2

Boosts 0

Participants 1

At WWDC25 we launched a new type of Lab event for the developer community - Group Labs. A Group Lab is a panel Q&A designed for a large audience of developers. Group Labs are a unique opportunity for the community to submit questions directly to a panel of Apple engineers and designers. Here are the highlights from the WWDC25 Group Lab for Apple Intelligence.

Can I integrate writing tools in my own text editor?

UITextView, NSTextView, and SwiftUI TextEditor automatically get Writing Tools on devices that support Apple Intelligence. For custom text editors, check out Enhancing your custom text engine with Writing Tools.

Given that Foundation Models are on-device, how will Apple update the models over time? And how should we test our app against the model updates?

Model updates are in sync with OS updates. As for testing with updated models, watch our WWDC session about prompt engineering and safety, and read the Human Interface Guidelines to understand best practices in prompting the on-device model.

What is the context size of a session in Foundation Models Framework? How to handle the error if a session runs out of the context size?

Currently the context size is about 4,000 tokens. If it’s exceeded, developers can catch the .exceededContextWindowSize error at runtime. As discussed in one of our WWDC25 sessions, when the context window is exceeded, one approach is to trim and summarize a transcript, and then start a new session.

Can I do image generation using the Foundation Models Framework or is only text generation supported?

Foundation Models do not generate images, but you can use the Foundation Models framework to generate prompts for ImageCreator in the Image Playground framework. Developers can also take advantage of Tools in Foundation Models framework, if appropriate for their app.

My app currently uses a third party server-based LLM. Can I use the Foundation Models Framework as well in the same app? Any guidance here?

The Foundation Models framework is optimized for a subset of tasks like summarization, extraction, classification, and tagging. It’s also on-device, private, and free. But at 3 billion parameters it isn’t designed for advanced reasoning or world knowledge, so for some tasks you may still want to use a larger server-based model.

Should I use the AFM for my language translation features given it does text translation, or is the Translation API still the preferred approach?

The Translation API is still preferred. Foundation Models is great for tasks like text summarization and data generation. It’s not recommended for general world knowledge or translation tasks.

Will the TranslationSession class introduced in ios18 get any new improvments in performance or reliability with the new live translation abilities in ios/macos/ipados 26? Essentially, will we get access to live translation in a similar way and if so, how?

There's new API in LiveCommunicationKit to take advantage of live translation in your communication apps. The Translate framework is using the same models as used by Live Communication and can be combined with the new SpeechAnalyzer API to translate your own audio.

How do I set a default value for an App Intent parameter that is otherwise required?

You can implement a default value as part of your parameter declaration by using the @Parameter(defaultValue:) form of the property wrapper.

How long can an App Intent run?

On macOS there is no limit to how long app intents can run. On iOS, there is a limit of 30 seconds. This time limit is paused when waiting for user interaction.

How do I vary the options for a specific parameter of an App Intent, not just based on the type?

Implement a DynamicOptionsProvider on that parameter. You can add suggestedEntities() to suggest options.

What if there is not a schema available for what my app is doing?

If an app intent schema matching your app’s functionality isn’t available, take a look to see if there’s a SiriKit domain that meets your needs, such as for media playback and messaging apps. If your app’s functionality doesn’t match any of the available schemas, you can define a custom app intent, and integrate it with Siri by making it an App Shortcut. Please file enhancement requests via Feedback Assistant for new App intent schemas that would benefit your app.

Are you adding any new app intent domains this year?

In addition to all the app intent domains we announced last year, this year at WWDC25 we announced that Visual Intelligence will be added to iOS 26 and macOS Tahoe.

When my App Intent doesn't show up as an action in Shortcuts, where do I start in figuring out what went wrong?

App Intents are statically extracted. You can check the ExtractMetadata info in Xcode's build log.

What do I need to do to make sure my App Intents work well with Spotlight+?

Check out our WWDC25 sessions on App Intents, including Explore new advances in App Intents and Develop for Shortcuts and Spotlight with App Intents. Mostly, make sure that your intent can run from the parameter summary alone, no required parameters without default values that are not already in the parameter summary.

Does Spotlight+ on macOS support App Shortcuts?

Not directly, but it shows the App Intents your App Shortcuts are sitting on top of.

I’m wondering if the on-device Foundation Models framework API can be integrated into an app to act strictly as an app in-universe AI assistant, responding only within the boundaries of the app’s fictional context. Is such controlled, context-limited interaction supported?

FM API runs inside the process of your app only and does not automatically integrate with any remaining part of the system (unless you choose to implement your own tool and utilize tool calling). You can provide any instructions and prompts you want to the model.

If a country does not support Apple Intelligence yet, can the Foundation Models framework work?

FM API works on Apple Intelligence-enabled devices in supported regions and won’t work in regions where Apple Intelligence is not yet supported

Boost

Answer 1

DTS Engineer OP

Apple

1w

(Continued)

Can Vision Intelligence in Apple Intelligence be used to analyze a live camera feed and detect the dimensions of a box in real time? Is this possible using the new FoundationModels API, or would ARKit / RoomPlan be required?

FoundationModels API is not multimodal and provides access to Apple’s on device large language model which is unrelated to machine learning models you’d use for bounding box detection. The Visual Intelligence framework does not support this use case as it is more focused on Visual Search and classification.

How powerful is the apple intelligence on device model in terms of model size, architecture, and capabilities like reasoning, generalization, and zero-shot performance.

It’s a large language model with 3 billion parameters, each quantized to 2 bits so good for summarization, extraction, classification, and many more use cases. It’s not designed for world knowledge or advanced reasoning.

Knowing App Intents is the core element to integrate our apps with the system via Apple Intelligence; on which type of intents you suggest we put our focus on to offer a great experience to the users?

IndexedEntity leveraging semantic index in CoreSpotlight is a good place to start. Consider compose-able intents: input and output. Provide a good parameter summary for Shortcuts and Spotlight.

Is there a certain Apple Intelligence call limit, and if there is, can you pay for more calls?

There is no rate limit on foreground requests currently but there is a rate limit on background requests. On-device requests are free i.e. no cost.

Is it possible to train an Adaptor for the Foundation Models to produce Generable output? If so what would the response part of the training data need to look like?

Yes, you can train an adapter and use Generable output with it. The response part of training data should look like a JSON of your generable object.

FoundationModels question: can we control the generated depth when generating @Generable structures with recursion?

There is no direct control and the model takes that responsibility. Try prompt engineering or using an alternative generable structure to influence generated depth.

Where is it documented which AI API's are on device or not?

Many Apple Intelligence features leverage on-device models: things like Writing Tools, Genmoji, Image Playground as well as new features like the Foundation Models framework. Users can also choose to use the on-device model with Use Model in Shortcuts. Don't commit to a specific list, instead use generic language for on-device and Private Cloud Compute for secure on-line AI.

If an App core functionality requires Apple Intelligence, and foundation models, is the a way on the App Store to prevent the App to be downloaded to unsupported devices?

There’s no way to block the app from being downloaded from the App Store. Handle unavailable cases with your UI with the Availability API. See API for checking availability i.e. SystemLanguageModel.Availability

Can Foundation Model accept images as input?

Foundation Models API is text-to-text only right now. Use image inputs with the Vision and Image Playgrounds APIs to achieve great results.

When we expose our app functions via App intent, how’s that information being used in Shortcuts and Siri

AppIntent Usage: used by Shortcuts to generate action UI using parameter types, titles, and descriptive strings.

Shortcut Editing: EnumerableEntityQuery and DynamicOptionsProvider can be used to provide selection UI when editing a shortcut.

AppShortcuts: available in Shortcuts, Spotlight, and can be run by Siri using custom phrases.

Does Foundation Models have the concept of "Memory"? Like ChatGPT.

A given transcript exists only in RAM memory for your app. You can serialize a transcript, store it and then retrieve it later for use at runtime again. There is no automatic mechanism for this.

From a technical developer perspective, what is the difference between "Apple Intelligence" and "Apple Machine learning and AI frameworks"?

Apple Intelligence is a personal intelligence system deeply integrated into apps and experiences across our platforms — from helpful Writing Tools to Genmoji, Image Playground, Image Wand, Clean Up, visual intelligence and a seamless connection to ChatGPT.

Foundation Models framework leverages the on-device model at the core of Apple Intelligence and joins a suite of other Machine Learning APIs and tools you can use to tap into intelligence in your app.

Some additional AIML frameworks for developers include CoreML, MLX, Speech etc.

See Apple Intelligence and machine learning for detailed overview.

I have been experimenting with the Foundation Models framework to summarize news in my RSS app but I'm hitting content guardrails often - any story about politics, obituary, etc. Is this something that I should expect to work or is news summarization not a supported use case?

Try more prompt engineering to effectively within the constraints of guardrails. File feedback for cases where guardrails weren’t accurate or weren’t helpful.

0

Answer 2

DTS Engineer OP

Apple

1w

(Continued)

Other than App Intents, is there anything else we should focus on to offer a great integration of our apps with Apple Intelligence and the rest of the system?

Entities: Donate to CoreSpotlight semantic index. Use Model action: AttributedString support, including tables.

What are the best practices for testing Apple Intelligence features in iOS apps, especially for QA Engineers?

Collect prompts to cover all major use cases of your app. Create a dedicated command line tool or a UI tester app to collect data. Observe data changes over time by manual inspection or asking another LLM to evaluate quality.

Other than the broad categories in the videos e.g. Tagging, Summarization etc. are there any specific examples you can give of types of tasks the model might be good at?

Text Extraction, Personalization, Classification, Structured Answers, Short form generated content, revision etc. A helpful key is providing data to the model through tool calling. You can experiment with Guides to tailor responses to more specific use cases, by providing detailed descriptions of the intended output.

Can we use Foundation Models from App Intents?

Yes, they work great from app intents. We encourage you to leverage all capabilities across different frameworks.

Can I give Foundation Models additional data? For example, if I have an array of products, could I pass that as context, so the user could ask a question about the products?

Yes, you can pass in the extra information as part of instructions or as a output from a tool. Decide which method to use based on if the information is always needed. See session Explore prompt design & safety for on-device foundation models for more details.

If I want to provide some app intents, but my app requires a login before the data is downloaded, what's the best way to handle that? Are there any examples that show how to provide conditional intents?

App Intents cannot be made conditionally available.

Use IntentModes and continueInForeground to show login screen.

0