Can the openCCG be used in multimodal language parsing?
Or is there anyone use it in multimodal language parsing
Can the openCCG be used in multimodal language parsing that parse user's input
from multiple modalities, and finds it semantic interpretation.
For example, a user may interact with a computer via speech and gesture.
In front of map displayed on screen, he issues a multimodal
command "move it there" and makes two pointing gesture, one for the item
to move, one for the location to move to. Multimodal language parsing
finds the coherent semantic interpretation of this multimodal command.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
OpenCCG has been used for multimodal realization, but not for parsing. Note that OpenCCG uses a "multimodal" version of CCG, where multimodal is used in the sense of multimodal categorial grammar, not multimodal interaction.
OpenCCG has some facilities for managing words as bundles of factors, which for generation has included pitch accents and gestures associated with words. In principle, this could be used for parsing too, but OpenCCG will not help with the key task of associating gestures with words.
-Mike
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Mike is absolutely correct to stress the difference between multimodal as in modes of grammatical control versus multimodal as in different streams of input.
For a project which uses OpenCCG in the multimodal context (in the sense of speech, vision, etc as modalities), check out the CoSy project:
In particular, the page for CoSy at DFKI has nice stuff, including videos of their mobile robot, which is running OpenCCG for natural language understanding and production:
Can the openCCG be used in multimodal language parsing?
Or is there anyone use it in multimodal language parsing
Can the openCCG be used in multimodal language parsing that parse user's input
from multiple modalities, and finds it semantic interpretation.
For example, a user may interact with a computer via speech and gesture.
In front of map displayed on screen, he issues a multimodal
command "move it there" and makes two pointing gesture, one for the item
to move, one for the location to move to. Multimodal language parsing
finds the coherent semantic interpretation of this multimodal command.
OpenCCG has been used for multimodal realization, but not for parsing. Note that OpenCCG uses a "multimodal" version of CCG, where multimodal is used in the sense of multimodal categorial grammar, not multimodal interaction.
OpenCCG has some facilities for managing words as bundles of factors, which for generation has included pitch accents and gestures associated with words. In principle, this could be used for parsing too, but OpenCCG will not help with the key task of associating gestures with words.
-Mike
Mike is absolutely correct to stress the difference between multimodal as in modes of grammatical control versus multimodal as in different streams of input.
For a project which uses OpenCCG in the multimodal context (in the sense of speech, vision, etc as modalities), check out the CoSy project:
http://www.cognitivesystems.org/
In particular, the page for CoSy at DFKI has nice stuff, including videos of their mobile robot, which is running OpenCCG for natural language understanding and production:
http://www.dfki.de/cosy/www/index.html
Geert-Jan Kruijff has some lectures and papers which could also be of interest:
http://www.dfki.de/~gj/
Jason