Background

Background

This project has gone through several phases and iterations:

Phase 1: Experience Design in Ubiquitous Computing (Graduate Course) - I worked with a group of computer science graduate students to conduct surveys, ethnographic research, ideate, prototype, and write the original paper that was accepted into the 2019 UbiComp Conference in London.

Phase 2: Thesis Project - I spent the final semester of my graduate program conducting market research, competitive analysis, and user testing to redesign the product from the ground up with information architecture, wireframes and user testing, and finally a robust prototype in InVision.

Phase 3: Post-Grad Iteration - A year or two later in 2021 I decided to redesign the product to continue to improve my craft.

This project has gone through several phases and iterations:

Phase 1: Experience Design in Ubiquitous Computing (Graduate Course) - I worked with a group of computer science graduate students to conduct surveys, ethnographic research, ideate, prototype, and write the original paper that was accepted into the 2019 UbiComp Conference in London.

Phase 2: Thesis Project - I spent the final semester of my graduate program conducting market research, competitive analysis, and user testing to redesign the product from the ground up with information architecture, wireframes and user testing, and finally a robust prototype in InVision.

Phase 3: Post-Grad Iteration - A year or two later in 2021 I decided to redesign the product to continue to improve my craft.

Goals

Goals

This project was started with two goals in mind and in expanded into something much bigger:

  1. Make cooking at home easier and more efficient.

  2. Help people learn how to cook.

  3. Reduce pollution and waste from both meal delivery services and grocery shopping.

This project was started with two goals in mind and in expanded into something much bigger:

  1. Make cooking at home easier and more efficient.

  2. Help people learn how to cook.

  3. Reduce pollution and waste from both meal delivery services and grocery shopping.

Tools Used

Tools Used

Sketch, InVision

Sketch, InVision

My Role

My Role

UX/UI Designer & Researcher

UX/UI Designer & Researcher

Length

Length

3 Months

3 Months

Results

Results

UbiComp Conference

Project paper accepted into and delivered at international conference (20% acceptance rate)

Project paper accepted into and delivered at international conference (20% acceptance rate)

500+

Project Paper Downloads

Project Paper Downloads

7

Paper Cited 7 times

Paper Cited 7 times

“Cooking is a craft, I like to think, and a good cook is a craftsman—not an artist. There's nothing wrong with that: The great cathedrals of Europe were built by craftsmen—though not designed by them. Practicing your craft in expert fashion is noble, honorable, and satisfying” - Anthony Bourdain

“The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.” - Mark Weiser

The Problem

The Problem

Core Problem:

  1. Finding recipes to cook based on the ingredients a person already has in their kitchen can often be difficult, especially for novices.

  2. It can be difficult to focus on cooking tasks while following recipe instructions at the same time.

Bigger Picture Problem:

  1. Meal delivery services like HelloFresh and BlueApron create a lot pollution due to the use of large amounts of plastic packaging.

  2. Many people don’t end up even cooking these meals because they can take a long time to cook, which is even more wasteful.

Core Problem:

  1. Finding recipes to cook based on the ingredients a person already has in their kitchen can often be difficult, especially for novices.

  2. It can be difficult to focus on cooking tasks while following recipe instructions at the same time.

Bigger Picture Problem:

  1. Meal delivery services like HelloFresh and BlueApron create a lot pollution due to the use of large amounts of plastic packaging.

  2. Many people don’t end up even cooking these meals because they can take a long time to cook, which is even more wasteful.

The Solution

The Solution

  1. Streamline the process of entering and tracking ingredients on hand.

  1. Provide recipe recommendations based on the ingredients a person already has (don’t need to go to the grocery store for).

  1. Provide easy step-by-step visual recipe instructions.

Gathering Initial Insights of the Problems People Have with Cooking at Home

Gathering Initial Insights of the Problems People Have with Cooking at Home

Conducting Ethnographic Research by Observing and Interviewing People While They Cook at Home

Conducting Ethnographic Research by Observing and Interviewing People While They Cook at Home

Passive Observation: We observed people in their kitchens as they went through their normal process of deciding what to cook / finding a recipe and then the cooking process itself.

Contextual Interviews: We also asked some of the participants questions as they went through this process in their kitchen.

Passive Observation: We observed people in their kitchens as they went through their normal process of deciding what to cook / finding a recipe and then the cooking process itself.

Contextual Interviews: We also asked some of the participants questions as they went through this process in their kitchen.

“Recipe recommendations with respect to daily nutrition and difficulty of cooking.”

“Recipe recommendations with respect to daily nutrition and difficulty of cooking.”

“A machine that can help make decisions .”

“A machine that can help make decisions .”

“App that creates possible menus with the ingredients I have in my fridge.”

“App that creates possible menus with the ingredients I have in my fridge.”

“I want a simulation that allows me to cook with what few ingredients my refrigerator has.”

“I want a simulation that allows me to cook with what few ingredients my refrigerator has.”

Reviewing Existing Literature on the Topic to Gain Insights and Inspiration

Reviewing Existing Literature on the Topic to Gain Insights and Inspiration

Our team wanted to understand what research had already been done in the area of smart kitchens and cooking. This helped our team find inspiration for designing possible solutions to the challenges we discovered from surveys and ethnographic research. Below are some summaries from published research papers we found.

Our team wanted to understand what research had already been done in the area of smart kitchens and cooking. This helped our team find inspiration for designing possible solutions to the challenges we discovered from surveys and ethnographic research. Below are some summaries from published research papers we found.

Technology for the Fridge

Technology for the Fridge

iFridge is an intelligent system that leverages technology to collect food information, understand the user’s activity, locate specific foods, and then uses a “cooking recipe recommendation” application to provide recipes based off of the user’s daily eating habits.

iFridge is an intelligent system that leverages technology to collect food information, understand the user’s activity, locate specific foods, and then uses a “cooking recipe recommendation” application to provide recipes based off of the user’s daily eating habits.

Multimodal Machine Learning

Multimodal Machine Learning

Modeling a cooking process can be viewed as a multimodal machine-learning problem, which is the issue we focus on in this study. Modal refers to the basic human sensory channels-- such as visual and language signals, whereas multimodal machine learning is a way to build machine-learning models on multiple heterogeneous inputs.

Modeling a cooking process can be viewed as a multimodal machine-learning problem, which is the issue we focus on in this study. Modal refers to the basic human sensory channels-- such as visual and language signals, whereas multimodal machine learning is a way to build machine-learning models on multiple heterogeneous inputs.

Applying Human Emotion to a Decision Model

Applying Human Emotion to a Decision Model

A proposed system in which recipes can be searched for according to the user’s mood. It shows that the user’s mood is determined in six aspects and the result is helpful in making decisions that determine the menu.

A proposed system in which recipes can be searched for according to the user’s mood. It shows that the user’s mood is determined in six aspects and the result is helpful in making decisions that determine the menu.

Using Big Data for Cooking Recipes and Images

Using Big Data for Cooking Recipes and Images

Javier proposed the large-scale Recipe1M dataset, which contains one million structured cooking recipes and their images. Recipe1M has collected, organized, and provided analysis of it contents. This data can be used to evaluate the accuracy and effectiveness of our system.

Javier proposed the large-scale Recipe1M dataset, which contains one million structured cooking recipes and their images. Recipe1M has collected, organized, and provided analysis of it contents. This data can be used to evaluate the accuracy and effectiveness of our system.

Visualizing the Users Journey with Storyboarding

Visualizing the Users Journey with Storyboarding

Visualizing the Users Journey with Storyboarding

This exercise allowed our team to organize our thoughts based on the information we gathered from user research. It allowed us to begin thinking through the journey or process that a person could go through and help us brainstorm solutions.

This exercise allowed our team to organize our thoughts based on the information we gathered from user research. It allowed us to begin thinking through the journey or process that a person could go through and help us brainstorm solutions.

Brainstorming Solutions with Paper Sketches

Brainstorming Solutions with Paper Sketches

We sketched a wide variety of solutions that included a kitchen user-interface along with various sensors (not all of the sketches are included here).

We sketched a wide variety of solutions that included a kitchen user-interface along with various sensors (not all of the sketches are included here).

Design Critique: Gathering Feedback from Classmates

  • People like the idea of having a single interface in front of them while they cook.

  • People like the idea of using a mobile app to scan food, instead of needing a bunch of different sensors in their kitchen.

  • How do you track food items after they’ve been used and disposed of?

    • Answer: The ingredients are removed automatically based on the serving size of the recipes that are used. 

  • People like the idea of having a single interface in front of them while they cook.

  • People like the idea of using a mobile app to scan food, instead of needing a bunch of different sensors in their kitchen.

  • How do you track food items after they’ve been used and disposed of?

    • Answer: The ingredients are removed automatically based on the serving size of the recipes that are used. 

Collaborating with Computer Science Teammates to Create the System Design

Collaborating with Computer Science Teammates to Create the System Design

Our system is based around four main modules: a series of user-facing interfaces, an image recognition model, a search and recommendation engine, and a database. Our architecture supports multiple interfaces, including a web-based portal accessible via a fixed, kitchen-based terminal (the main display) or via users' mobile phones.

Our system is based around four main modules: a series of user-facing interfaces, an image recognition model, a search and recommendation engine, and a database. Our architecture supports multiple interfaces, including a web-based portal accessible via a fixed, kitchen-based terminal (the main display) or via users' mobile phones.

Module 1: Fixed & Mobile User Interfaces

Module 1: Fixed & Mobile User Interfaces

The main user interface is designed to be displayed on a mounted/stationary device located in the kitchen where a person can easily view it while completing cooking tasks.

Users can use the system's mobile interface to send refrigerator and pantry images to the image recognition system.

Both interfaces can be used to review recipe recommendations from our recice search and recommendation engine.

The main user interface is designed to be displayed on a mounted/stationary device located in the kitchen where a person can easily view it while completing cooking tasks.

Users can use the system's mobile interface to send refrigerator and pantry images to the image recognition system.

Both interfaces can be used to review recipe recommendations from our recice search and recommendation engine.

Module 2: Image Recognition Module

Module 2: Image Recognition Module

Our image recognition module identifies potential recipe ingredients. We designed and developed our image recognition model using deep-learning techniques— in this case, a computational neural network (CNN).

We tested our model on the Food 101 data set, holding out 20% of the training set as a development dataset. We tested the classifier that performed best on the development set, instead of heavily tuning the model's parameters. When running the algorithm on the test set, we obtained an accuracy score of 52.63%, outperforming the original by 2%.

Our image recognition module identifies potential recipe ingredients. We designed and developed our image recognition model using deep-learning techniques— in this case, a computational neural network (CNN).

We tested our model on the Food 101 data set, holding out 20% of the training set as a development dataset. We tested the classifier that performed best on the development set, instead of heavily tuning the model's parameters. When running the algorithm on the test set, we obtained an accuracy score of 52.63%, outperforming the original by 2%.

Module 3: Search & Recommendation System

Module 3: Search & Recommendation System

After receiving a list of recognized ingredients from the neural model, the search and recommendation system retrieves recipe information and generates a suite of cooking recommendations.

Our search and recommendation engine is based on Elesticsearch, with the recipe data our database, adapted from the OpenRecipes corpus, indexed by recipe name and ingredient.

After receiving a list of recognized ingredients from the neural model, the search and recommendation system retrieves recipe information and generates a suite of cooking recommendations.

Our search and recommendation engine is based on Elesticsearch, with the recipe data our database, adapted from the OpenRecipes corpus, indexed by recipe name and ingredient.

Validating Solution at UbiComp Conference 2019 - London, UK

Validating Solution at UbiComp Conference 2019 - London, UK

On behalf of my team, I presented our project at the ACM International Joint Conference on Ubiquitous and Pervasive Computing (UbiComp) in London, UK.

On top of being a great experience, it helped validate the concept and encouraged me to take the project further for my thesis design project, which is laid out in the next steps.

Gaining Deeper Understanding of the Competition, Bigger Societal Problems, and Product Market Fit

Gaining Deeper Understanding of the Competition, Bigger Societal Problems, and Product Market Fit

Uchef is healthier and better for the environment

Hello Fresh, Blue Apron, and Freshly make it easier to cook delicious recipes by delivering you the specific ingredients and instructions. However, these services use a ton of plastic to package their ingredients. Not only does this create a negative impact on the environment, but there’s a lot of research showing the negative health impacts of the absorption of micro-plastics into the human body. In particular, Freshly sends you whole meals in plastic containers that you’re supposed to microwave!! That’s a proven method to have micro-plastics leach into your food. Restaurant delivery services are great, but expensive and meals are often packaged hot and ready in a plastic package!

Uchef is healthier and better for the environment

The NYT Cooking app offers some really amazing recipes as well as some cool features like creating a grocery shopping list based on the ingredients you’ll need to cook it. You still have to go through your kitchen to see which ingredients you already have though. Ucook bases recipe recommendations on the ingredients you already have at home. Not only does this work great for busy people, but it opens up more fun and easy grocery shopping opportunities by allowing people to go to the store without a grocery list and to just buy whatever ingredients look good to them. Ucook will do the rest by finding delicious recipe options for you. The possibilities are endless!

Competitive Landscape

There is no other product that is focused on both an ingredient-based solution and providing recipe recommendations for cooking at home. By ingredients-based solution I mean that the product either provides you with the necessary ingredients or in the case of UCHEF (formally UCOOK) the product makes sure to only provide recipes based on the ingredients you already have.

Mapping Out the Structure for a Better Smart Kitchen Interface

Mapping Out the Structure for a Better Smart Kitchen Interface

Having already completed a large amount of user research, analysis, and initial concept creation— I had a strong framework for how to design the IA, especially given the additional context of competitive analysis.

Although conducting card sorting, tree testing, or other methods could have been valuable beforehand in order to help inform and validate the IA even further, at the time I felt it was a bit of an overkill and that these methods could be implemented later on to refine the archtecture and design.

Having already completed a large amount of user research, analysis, and initial concept creation— I had a strong framework for how to design the IA, especially given the additional context of competitive analysis.

Although conducting card sorting, tree testing, or other methods could have been valuable beforehand in order to help inform and validate the IA even further, at the time I felt it was a bit of an overkill and that these methods could be implemented later on to refine the archtecture and design.

Optimizing the Core User Experience of Finding and Cooking Recipes

Optimizing the Core User Experience of Finding and Cooking Recipes

After I had organized the high-level structure for the main kitchen interface (tablet), I was able to then think through the user flows of core user experience for both the main kitchen interface (tablet) and the mobile app (companion) interface.

After I had organized the high-level structure for the main kitchen interface (tablet), I was able to then think through the user flows of core user experience for both the main kitchen interface (tablet) and the mobile app (companion) interface.

Exploring Solutions Visually for Main Kitchen Interface with Low-Fidelity Wireframes

Exploring Solutions Visually for Main Kitchen Interface with Low-Fidelity Wireframes

Validating Function, Not Fashion with Usability Testing

Validating Function, Not Fashion with Usability Testing

Due to the Covid-19 Pandemic lockdowns, I used an online usability testing platform to validate the core process of finding recipes, browsing inventory, and cooking recipes.

Due to the Covid-19 Pandemic lockdowns, I used an online usability testing platform to validate the core process of finding recipes, browsing inventory, and cooking recipes.

Evolving the Design through User Data

Evolving the Design through User Data

Evolving Concepts through User Data

Evolving Concepts through User Data

  • Simplified the home screen (Recipes) and the navigation.

  • Moved the "Begin” button further up on the screen and increased the size of the button.

  • Changed the top navigation to a hamburger menu to reduce cognitive overload.

  • Moved some features to preferences/settings, such as “Connect Mobile Device”, which would allow users to connect and sync their smart phone application to the main Ucook tablet application.

  • Simplified the home screen (Recipes) and the navigation.

  • Moved the "Begin” button further up on the screen and increased the size of the button.

  • Changed the top navigation to a hamburger menu to reduce cognitive overload.

  • Moved some features to preferences/settings, such as “Connect Mobile Device”, which would allow users to connect and sync their smart phone application to the main Ucook tablet application.

Easily Narrow Down Recipe Options

Easily Narrow Down Recipe Options

  • Select which type of meal you want

  • Choose from a wide variety of cuisine types

  • Choose a specific ingredient that you need or are craving.

  • Ucook will give you two (2) recipe options based on your choices and the ingredients you currently have in your kitchen. If you’d like to see more options then simply click “more options” and Ucook will give you two more options.

  • Go at your own pace with with easy step-by-step visual cooking instructions that you don’t need to pause, rewind, or read tiny words from a single page in a cook book. And a tablet can always be cleaned, while a page from a recipe book can easily become contaminated or nasty from touching it with messy fingers.

  • Select which type of meal you want

  • Choose from a wide variety of cuisine types

  • Choose a specific ingredient that you need or are craving.

  • Ucook will give you two (2) recipe options based on your choices and the ingredients you currently have in your kitchen. If you’d like to see more options then simply click “more options” and Ucook will give you two more options.

  • Go at your own pace with with easy step-by-step visual cooking instructions that you don’t need to pause, rewind, or read tiny words from a single page in a cook book. And a tablet can always be cleaned, while a page from a recipe book can easily become contaminated or nasty from touching it with messy fingers.

Your ingredients organized into a Visual and Intuitive User-Interface

Your ingredients organized into a Visual and Intuitive User-Interface

  • Dry goods will automatically be placed into the Cabinets menu, while anything that requires refrigeration will automatically be placed into the Refrigerator menu.

  • Ingredients are organized intuitively into categories such as meat, seafood, vegetables, fruit, drinks, and grains.

  • Select any ingredient to view, edit, or remove the item, as well as add a new item manually if you’d like to.

Meal History

Meal History

  • View a full history of the recipes you’ve previously cooked.

  • View a visual summary of your daily, weekly, and monthly nutrition based on the meals you’ve cooked using Ucook.

  • View all of the recipes you’ve saved so that you can browse through them and cook them later at any time.

Preferences

Preferences

  • Let Ucook know if you have any allergies/food restrictions and it will automatically remove those options.

  • Set your preferred calorie intake as well as your preferred cook time.

Visualizing the Layout for the Mobile (Companion) App

Visualizing the Layout for the Mobile (Companion) App

Creating the Visual Identity

Creating the Visual Identity

InVision Prototype…. Lost to the Sands of Time

InVision Prototype…. Lost to the Sands of Time

Unfortunately, the complex interactive prototype I put together for this project originally no longer exists because it was built with InVision, which is no longer around.

Unfortunately, the complex interactive prototype I put together for this project originally no longer exists because it was built with InVision, which is no longer around.

Also, do to the sheer amount of content and different screens needed to create a robust and believable prototype for the main kitchen interface (tablet), I was never able to create a complete visual design and prototype for the mobile companion app.

Also, do to the sheer amount of content and different screens needed to create a robust and believable prototype for the main kitchen interface (tablet), I was never able to create a complete visual design and prototype for the mobile companion app.

AI Coming to Save the Day, Kind of…

AI Coming to Save the Day, Kind of…

Fortunately, the coming of AI design tools has allowed me to take all of the tedious work I've done previously, and rapidly iterate on the visual design as well as create an impressively robust interactive prototype. However, it did have a lot of issues with spacing and random artifacts that needed to be cleaned up. Mainly, even with a Pro Claude subscription, a single prompt to fix/edit it caused me to reach my session limit BEFORE IT EVEN FINISHED, and this was immediately after my session amount reset.

Fortunately, the coming of AI design tools has allowed me to take all of the tedious work I've done previously, and rapidly iterate on the visual design as well as create an impressively robust interactive prototype. However, it did have a lot of issues with spacing and random artifacts that needed to be cleaned up. Mainly, even with a Pro Claude subscription, a single prompt to fix/edit it caused me to reach my session limit BEFORE IT EVEN FINISHED, and this was immediately after my session amount reset.

Outcome

Outcome

With over 700 downloads and 9 citations, it's highly likely that many of the recipe and cooking applications that have been released on the Apple App Store and other places have directly utilized the research and concepts from the original "Smart Cook: Making Cooking Easier with Multimodal Learning".

With over 700 downloads and 9 citations, it's highly likely that many of the recipe and cooking applications that have been released on the Apple App Store and other places have directly utilized the research and concepts from the original "Smart Cook: Making Cooking Easier with Multimodal Learning".

This on-going passion project has also given me invaluable experience and knowledge with 0-1 product design, UX research, systems thinking, content heavy platforms, and much more.

This on-going passion project has also given me invaluable experience and knowledge with 0-1 product design, UX research, systems thinking, content heavy platforms, and much more.