Think together · Start with a story
A girl gives a robot a video with sound, then asks what is happening. Different inputs first need to become representations the model can process.
Break down the process
- Visual representationFrom the frames
- Audio representationFrom the sound
- Language questionDefines the task
- Combined systemOrganizes a response
AI systems that see, hear, and speak usually combine different input representations and output abilities.
Some models process inputs together in a more unified structure; others use multiple components that work together. Seeing one chat box does not mean there is only one model or one kind of computation inside.
Each part has limits. A speech-recognition mistake can affect the answer that follows.
Your turn to try
If the image is clear but the audio is misheard, could the final answer still be wrong?
Ready? Open to see the explanation
Yes. A combined system depends on several kinds of information, and one error can carry through to the final result.
One idea to remember
AI systems that see, hear, and speak usually combine different input representations and output abilities.
Try covering the picture above and explaining each step to your family. If one step is hard to explain, revisit that card.