For about a year, Google has been previewing a number of Gemini-branded and other AI features in consumer apps. Here’s everything that’s been announced and when they might be available.
Pixel
At the end of Made by Google 2023, Zoom Enhance was offered for the Pixel 8 Pro, which “intelligently fills in the gaps between pixels and predicts fine details.” Using a “custom generative AI image model” built into the device, Google has made this useful when you forget to zoom.
It’s an incredible application of generative AI that opens up a range of possibilities for framing and editing your photos. So the kind of zoom enhancement you see in science fiction – it’s in the phone in your hand.
In October, Google said it was “coming later.” It has yet to arrive after the Three Pixel Feature Drop. It is unclear if the model Google is referring to is the Gemini Nano with Multimodality. At this point, it could debut with the Pixel 9 Pro as the phone’s flagship photography feature.
Google Home
The Google Home app will use generative artificial intelligence to summarize events into a “simplified view of what happened recently.” This “quick and easy summary” will use bullet points, but you’ll also be able to “Ask about your home” by speaking to find video history clips and get automations. “Experimental features” will come to Nest Aware subscribers in 2024.
Fitbit
Fitbit Labs will allow Fitbit Premium users to test and provide feedback on experimental AI capabilities.
One such feature is a chatbot that lets you ask natural and conversational questions about your Fitbit data. This “personalized training” that takes fitness goals into account aims to create “actionable messages and instructions” with responses that can include personalized schedules.
- “For example, you can learn more about how many active zone minutes (AZM) you get and the relationship with how restorative your sleep is.”
- “…this model can analyze variations in your sleep patterns and sleep quality and then make recommendations about how you can change the intensity of your exercise based on those insights.”
Behind the scenes, it’s powered by Fitbit and the new Personal Health LLM from Google Research built on Gemini. As of March, it’s coming “later this year” to “a limited number of Android users enrolled in the Fitbit Labs app on the Fitbit mobile app.”
Google Photos
Ask Photos will allow you to ask questions about photos and videos in your library. Apart from finding images, it can extract information and give you a text response. Sample prompts powered by the twins include “Show me the best photo from each national park I’ve seen” and “What themes did we have for Lena’s birthday parties?” includes. It can be used to “suggest the best photos” and create captions for them. Query Images is an “experimental feature” that Google is already rolling out soon, teasing more possibilities in the future.
Gmail + Google Workspace
In Gmail for Android and iOS, you’ll find a Twins button in the top-right corner that lets you display the mobile equivalent of a sidebar for entering full prompts. Gmail also gets Contextual Smart Replies, which offer more personalized, detailed, and nuanced suggestions. This will be rolled out to Workspace Labs in July.
At Cloud Next 2024 in April, Google also previewed voice prompts to help with typing in mobile Gmail. Meanwhile, an “instant polish” feature will “turn rough notes into a complete email with one click.”
On the desktop web, the sidebar is available in Gmail, Google Drive, and Docs/Sheets/Slides. Gemini comes to Google Chat to wrap up further conversations and answer questions.
Google Maps
Back in February, Google Maps announced it would use LLMs to power its Ask About chatbot. You can use it to find places that match your query with support for follow-up questions. It contains details about 250 million places and user-submitted photos, videos and reviews.
Chrome
Gemini Nano is coming to desktop Chrome to power browser features like Help Me Type. It should be available on most modern laptops and desktops.
Google search
In addition to launching AI Previews, Google previewed a number of upcoming features coming to Search Labs for the first time:
- You’ll be able to take the Original AI Review and make it “Simpler” (just a few sentences) or “Break it down” (a longer answer).
- Multi-step reasoning capabilities will allow you to ask a complex question all at once instead of splitting it into multiple queries.
- Food and travel planning
- AI-organized search results page
- Video searches: Record a video and ask a question about it
Android
Gemini Nano with multimodality will be available on the Pixel “later this year” and power features such as on-device/offline TalkBack images and real-time fraud alerts that listen to the call for alert patterns. Google will share more details later this year.
At I/O 2024, Google also previewed how Gemini on Android will soon be an overlay panel instead of opening a full-screen UI to display results. In addition to preserving the context, this will allow you to drag and drop the created image into the conversation. For Gemini Advanced subscribers, the “Ask this video” and “Ask this PDF” buttons will digest Gemini videos and documents, respectively. It’s rolling out “over the next few months.” In addition, Dynamic Offers will use Gemini Nano with Multimodality to understand what’s on your screen:
For example, if you activate Gemini in a conversation about pickleball, suggestions might include “Find pickleball clubs near me” and “Pickball rules for beginners.”
Another addition that will be especially useful on mobile devices is the Gemini Extension for Google Calendar, Tasks, and Keep. This will allow you to capture a page with many upcoming dates that can be converted into Gemini Calendar events. In the coming months, Utilities will give mobile Gemini access to Android’s Clock app.
We also expect mobile Gemini to come to the Pixel Tablet this summer.
Twins
Live will allow you to have a two-way conversation with a Gemini. To make the experience more natural, Gemini will give short answers that you can interrupt to add new information or ask for clarification. You can choose from 10 different voices, and Google envisions Gemini Live as useful for interview preparation or speech practice. It will be available to Gemini Advanced members “in the coming months.”
“Later this year,” Gemini Live will let you launch a live cam mode. Just point to something in the real world and ask a question about it. It is powered by Project Astra.
Gems are personalized versions of Gemini, allowing you to get a “gym buddy, chef, coding partner, or creative writing guide.” Gemini Advanced members will be able to create custom ones, while all users will have access to pre-made Gems like the Learning Coach.
Simply describe what you want your Gemini to do and how to respond – like “you’re my running coach, give me a daily running plan and be positive, upbeat and encouraging”. Gemini will take these instructions and, with one click, refine them to create a Gem that meets your specific needs.
Gemini Advanced users will also get an “immersive planner” that goes beyond just suggesting activities, but actually takes into account travel times and stops, as well as people’s interests, to create a detailed itinerary. Gemini will use Gmail for flight/travel details, Google Maps recommendations for dining and museums near your hotel, and Search for other activities.
FTC: We use automatic affiliate links that generate income. More.