As students passionate about using technology for social good, we wanted to create something meaningful for our aging population. Elderly people often live alone and face challenges with medication adherence, health monitoring, and social isolation. Inspired by the idea of a robot caregiver, we set out to build a supportive companion using Misty the robot, capable of interacting with seniors through voice, detecting falls, and ensuring medication plans are followed correctly.
We developed a system that integrates an iOS caregiver app, a Django backend, and Misty the robot. Our solution includes:
- Caregivers can schedule checkups, create custom questions, and assign medication plans with dosage, frequency, and timing.
- Conduct checkups using voice-based questions and temperature readings.
- Remind patients about medication intake.
- Take photos when the patient shows their medication, send the image to Roboflow, and validate if the correct drug is being taken.
- Conduct meditation sessions upon request.
- A centralized Django server that connects Misty, the app, and stores patient data.
python3 -m venv envsource env/bin/activate # Linux/macOS
.\env\Scripts\activate # Windowspip install -r requirements.txtcd misty_backend
python manage.py migrate
python manage.py runserver- How to program Misty's SDK for speech, emotion display, and interactive control.
- How to use Gemini / LLM APIs for handling natural conversations and responses.
- How to integrate Roboflow for real-time drug recognition using computer vision and COCO annotations.
- Building a full-stack system connecting iOS, Django, and hardware in real-time.
- Designing systems that prioritize accessibility and empathy.
- Frontend: Swift iOS app for caregivers to manage plans and checkups and patients to view their info and see their checkups.
- Backend: Django with RESTful APIs for patient/caregiver management and Misty integration.
- Misty SDK: Custom functions for talking, displaying emotions, responding to commands, and capturing images.
- Roboflow: Trained a model to detect 5 medication types using COCO-format data and integrated it with Misty’s photo-check feature.
- Voice Interaction: Used Gemini to help Misty hold simple and kind conversations.
- Misty SDK had limited documentation and needed creative workarounds for some features.
- Real-time image prediction with Roboflow required syncing image capture, upload, and response analysis.
- Coordinating communication between the iOS app, Django backend, and Misty robot reliably.
- Designing a conversation flow that felt natural and reassuring for elderly users.
- Time constraints at the hackathon – so many ideas, so little time!
- Add facial recognition so Misty can greet users by name.
- Expand medication detection with more drug types and pills in different lighting conditions.
- Improve Misty’s conversational memory to support more dynamic and personalized interactions.
- Computer Vision
- Django
- Misty SDK
- Swift