I’m assuming this has to be happening somewhere. Basically, there should be a video search engine that uses speech-to-text technology (which isn’t perfect, but even if it’s 90% right, that’s better than nothing) and starts indexing videos based on what is being said in the audio. So the software would have to listen/watch every video it’s indexing — which could be a very time consuming task, given the amount of video content out there these days.
Pluggd is doing this for podcasts, so it seems it’d be fairly easy to do this for videos. It’s really neat technology — shows a heatmap in the podcast based on the keyword you’re searching for (so that you can see where in the podcast that topic is being spoken about). My only reservations are what the terms of service are on the video websites.
You could also have this technology see if it recognizes any text or images in the video — thus it might notice a picture of an apple, or a logo, or text that says something.
One problem with this search is that the video might be a comedy, but there’s no way of this technology to know it’s a comedy — it would just know the speech in the audio.