With the help of researchers at the University of California at Santa Barbara (UC Santa Barbara), Apple has developed an AI-powered image editor that responds to text-based commands. A demo is now live on Hugging Face, though it’s unclear whether the project will ever make it to any Apple products.
Apple’s model is called MLLM-guided image editing, or MGIE, with MLLM being short for “multimodal large language models.” This jargon means MGIE leverages a large language model capable of mixing natural language processing with image inputs, which is vital for any AI-based image editor. Rather than using sliders, filters, and other manual image-editing tools, MGIE users can simply tap out a quick text prompt that tells the model what they want their result to look like.
In preparation for this year’s International Conference on Learning Representations (ICLR), Apple and UC Santa Barbara published a paper to the arXiv detailing MGIE’s process. The user starts with an input image—say, a picture of an A-frame cabin in a lush, wooded area. Then they enter a text prompt: “The cabin in the desert is surrounded by sand dunes and some vegetation, such as cacti or small shrubs.” With this, MGIE seeks matching assets from its IPr2Pr dataset, then uses diffusion to jointly train the model and edit the user’s image. In one of the developers’ experiments, at least, the result is a fairly realistic image of the original cabin in what appears to be the Sonoran desert.
MGIE appears capable of everything from basic edits, like contrast, brightness, and saturation increases, to erasing backgrounds or removing blemishes. MGIE can also be used to crop, resize, and rotate photos. Ideally, MGIE will edit images without sacrificing their legibility or integrity. The developers note in their paper that MGIE isn’t the first AI image editor to use text prompts. Still, previous models tend to be “limited to unrealistic synthesis,” making them useless to many users.
Apple hasn’t yet said whether MGIE is part of a longer-term project or if it will make its way to any future OS updates. However, the company did release the open-source model on GitHub, and you can test it out in your web browser by visiting Hugging Face.
