Microsoft’s Research has unveiled an innovative conversational question answering model, surpassing traditional methods in terms of efficiency, speed, and accuracy while requiring substantially less computational resources.
Presented as a breakthrough in ranking passages from content, this method has been dubbed ‘Generative Retrieval For Conversational Question Answering’, or GCoQA, in the academic vernacular.
Anticipating future developments, the research team proposes the exploration of this technology’s application in the realm of general web search, signifying a promising horizon for this novel approach.
Conversational Question Answering Through Generative Retrieval
The autoregressive language model primarily attempts to predict subsequent words or phrases based on preceding ones.
Such models utilize “identifier strings,” tantamount to representations of text passages within a document in everyday language.
The strategy here involves the deployment of page and section titles to deduce the overarching relevance of the page and the specific subject matter dealt with in its segments.
When applying this methodology to Wikipedia data, it can be reliably assumed that page and section titles adequately describe their respective contents.
These titles effectively act as compasses, guiding the identification of the document’s main topic and the sub-themes contained within the document’s sections.
Practically speaking, it operates much like using a website’s title to discern its essence and inspecting the headings to comprehend the subject matter of each section.
“Identifiers” serve to encapsulate this knowledge, creating a representation which is then associated with the web page’s passages and titles.
Lastly, the earmarked passages are plugged into another autoregressive model, designed to generate responses to posed queries.
Generative Information Retrieval
As outlined in the research paper, the model employs a method known as “beam search“. This advanced technique is responsible for generating identifiers. These identifiers, essentially precise representations of passages extracted from webpages, are then systematically arranged. The ranking order is determined by the likelihood of each identifier providing the most accurate answer to the query.
The researchers write:
“…we utilize beam search… a commonly-used technique, to generate multiple identifiers instead of just one.
Each generated identifier is assigned a language model score, enabling us to obtain a ranking list of generated identifiers based on these scores.
The ranking identifiers could naturally correspond to a ranking list of passages.”
The research document further elaborates the process, characterizing it as a ‘hierarchical search’.
In the given context, hierarchical implies the initial categorization of results based on the topic of the page, followed by the organization of passages within the page using section headings.
Upon retrieval of these passages, another autoregressive model takes the lead, crafting an answer tailored to the data gathered from these retrieved sections.
Microsoft’s Research Evaluating Against Other Techniques
Employing comparative analyses, researchers observed the GCoQA model, outperforming a multitude of prevailing methods.
The prowess of GCoQA is exhibited in its outstanding ability to bypass limitations or bottlenecks that plague other techniques.
Anticipating a significant shift in the paradigm of conversational question answering, the introduction of this model beckons a new era in the field.
A remarkable demonstration of efficiency is the model’s ability to operate on a mere 1/10th of the memory resources hitherto needed. This not only signifies a monumental progression in conservation resources but also appreciable speed in operation.
The researchers write:
“…it becomes more convenient and efficient to apply our method in practice.”
Microsoft researchers conclude:
“Benefiting from fine-grained cross-interactions in the decoder module, GCoQA could attend to the conversation context more effectively.
Additionally, GCoQA has lower memory consumption and higher inference efficiency in practice.”
Drawbacks of GCoQA
Despite the promising applications, several challenges require attention before the full integration of this model.
Researchers encountered constraints within the Generative Contextualized Question-Answering model, or GCoQA, which hinged on the “beam search” technique. Notably, this affected its ability to encompass “large-scale passages.”
Furthermore, efforts to surmount this constraint by amplifying the beam size only led to decreased operational speed of the model.
On another note, the model functioned optimally with Wikipedia due to its consistent and meaningful use of headings.
However, when expanded to other webpages beyond Wikipedia’s confines, challenges arose which could hinder the model’s effectiveness.
This is largely due to the fact that numerous webpages across the Internet neglect to accurately denote the content of their sections through their headings, a departure from established SEO and publishing norms.
Noteworthy observations from the research paper include:
“The generalizability of GCoQA is a legitimate concern.
GCoQA heavily relies on the semantic relationship between the question and the passage identifiers for retrieving relevant passages.
While GCoQA has been evaluated using three academic datasets, its effectiveness in real-world scenarios, where questions are often ambiguous and challenging to match with the identifiers, remains uncertain and requires further investigation.”
GCoQA: A Rising Star in New Technology
Researchers have indicated that the observed performance enhancements hold a significant merit. Inevitably, addressing the present limitations is a necessary task on their course of progress.
Upon concluding the research paper, it was emphasised that there exists two potential areas that warrant further exploration:
“(1) investigating the use of generative retrieval in more general Web search scenarios where identifiers are not directly available from titles; and (2) examining the integration of passage retrieval and answer prediction within a single, generative model in order to better understand their internal relationships.”
The Importance of GCoQA
One of the research scientists recently publicized their insightful research paper, intriguingly entitled “Generative Retrieval for Conversational Question Answering”.
The said paper can be easily accessed, in PDF format, directly from the aforementioned GitHub page.
However, it’s worthy to note that there is always the potential for such invaluable research materials to eventually fall behind paywalls, which could restrict future accessibility.
While it’s uncertain if or when GCoQA will find its way into mainstream search engines, it nevertheless remains a crucial focus of extensive academic scrutiny.
The true essence of GCoQA lies in demonstrating the progressive efforts of researchers, who are tirelessly probing the potential of generative models to revolutionize the current web search paradigm.
This innovative exploration could very well be a harbinger of what we might expect in the foreseeable evolution of search engines.
Explore the announcement and delve into the abstract of the research paper: