Why Bing Chat is bad

In this article we will check why Bing Chat is bad and how it could be improved.

I’m a big fan of ChatGPT. I have already written multiple articles about using it and I still think that while it has its limitations, it is still the most innovative product we have seen in the last couple years and it has a great potential. Microsoft utilises its power in Bing Chat, so if it uses something good, it should be a great service, right? RIGHT?!

Well, if you are using it for simple tasks, it can be somewhat better than a normal search. If you ask “What places should I visit in Billund?”, it goes through Tripadvisor and suggests you the most popular targets in a well structured format. You can also ask “Where can I buy a raincoat in Billund?” and you will get your answers — again, from Tripadvisor. So, you can ask simple questions and get good answers without needing to dig into endless number of webpages from search results.

Answer retrieved from Tripadvisor

I think that the concept is great and this type of information retrieval can be superior to normal search results for most web searchers. However, if you start digging deeper, you will find some serious issues.

Limited sources

As you have seen from the Billund examples above, in both cases Bing Chat just used Tripadvisor as a source and pushed the top results. It did not try to check more sources and give a summary of multiple websites, just used what the laziest tourist does as well. If you just use Tripadvisor, you get the same answers. However, in Tripadvisor you get the locations with nice images and you can also click on the results to get quickly into detailed description of the suggested results.

By using Bing Chat, you will just get the most common knowledge and miss things like the excellent flee market selling LEGO rarities…

Always relying on the top search results

Bing Chat uses retrieval-augmented generation (RAG), which is a technique for enhancing the accuracy and reliability of generative AI models with facts fetched from external sources. This is no brainer, you are doing a search, you should get the most up-to-date information.

However, Bing Chat’s RAG has gone too far, it does a search and uses only the top search results as a source of information. E.g. if you ask for “scientific articles about icing knee injuries”, it searches for scientific articles about using icing for knee injuries and gives you something from the top results. Somewhat better than using only a single source as in the Billund example, but it still limits your scope, which might not be an issue in case of a perfect search, but let’s face it: Bing is not a perfect engine.

Repeating the same thing

The RAG issue above leads to the next issue: Bing Chat repeating itself over and over. Even if you ask for additional sources, in many cases it will just redo the same search and cite the same results from it — despite what you asked for.

There must be some serious issue with history handling as well. E.g. I have tried to get a link to an article cited by the Chat. I saw the link while Bing generated the answer, but at the end I always got something misformatted. Despite asking for the proper link multiple times (even by inserting the wrong answer in my question), I got the same answer every time.

Context issues

Bing Chat does not always have a proper context. Even Bing says:

Avoid using relative terms, like yesterday or tomorrow, and pronouns, like it and they. Instead, use specifics, such as an exact date or a person’s name.

When I worked on a chatbot, one of the first things we added to it was the user’s location and the date. Quite a simple thing, but it helps a lot in giving proper answers (e.g. what are the seasonal fruits now?). So, Bing Chat fails at the basics already. If you go deeper, it has even more issues. Common things I faced with:

• Not adjusting the search to the location: If you ask something about another country, you will still use your local language search results, therefore not getting the best output.
• Misunderstanding data: When I asked for scientific evidence on the effect of icing on knee injuries, I got non-scientific results (simple webpages) as evidence. Moreover, when I finally got some scientific link, that was about “Cartilage icing and chondrocalcinosis on knee radiographs”, which has nothing to do with icing knee injuries. It is still better than ChatGPT’s early days, when it gave fully made up or irrelevant articles, but very far from something useful.
• Random misinterpretations: Usually Bing Chat still gets what you want, but in the example below, when I asked for ways to cure flu, it offered me some mental help. It gets the answer right in vast majority of the cases, but due to some randomness, it failed when I asked.

Security

Just out of curiosity, I have tested whether I can get unsafe data from Bing Chat. If you phrase your questions carefully, you can get a list of poisonous plants, which you can grow in your garden or even the exact recipe of making explosives from materials available publicly. I admit, you can find this information by doing web searches, but I would expect Microsoft to add some safeguard for its service.

For safety reasons, I won’t add any screenshots here.

Improvement suggestions

As I see, adding ChatGPT to Bing has a great potential. However, the current setup needs to be improved to get to its full potential. Key areas to improve are:

Diversify Information Sources

• Bing Chat should broaden its information sources beyond relying predominantly on platforms like Tripadvisor.
• Incorporating insights from diverse reputable sources ensures users receive more accurate and comprehensive answers, especially for complex topics.

Enhance Retrieval-Augmented Generation (RAG)

• Refine the RAG approach to consider a broader range of relevant information.
• Overcoming the limitation of relying solely on top search results would result in more nuanced and comprehensive responses.

Address Repetition Issues

• Implement refined handling of search history to present varied results, even upon explicit requests for additional sources.
• Enhancing the system’s ability to avoid repetitive information retrieval contributes to an improved user experience.

Contextual Understanding

• Improve contextual understanding by incorporating details such as the user’s location and the date.
• Dynamic adjustment of search parameters based on context ensures users receive more relevant and timely information.

Enhance Information Accuracy

• Improve the precision of information retrieval, particularly in scientific or technical domains.
• A more accurate understanding of user queries prevents the system from providing irrelevant or misleading results.

Mitigate Security Concerns

• Implement safeguards to prevent the dissemination of potentially harmful information.
• Stricter moderation and filtering mechanisms contribute to a safer user experience, addressing concerns related to security and inappropriate content.

Conclusion

By prioritizing these structured improvements, Bing Chat has the potential to evolve into a more reliable, user-friendly, and secure service. These enhancements not only optimize the user experience but also contribute to the platform’s credibility and safety, unlocking the full potential of ChatGPT within the Bing Chat environment.