google-site-verification: googlebc47d07320294fb4.html

TW-BERT: End-to-end query term weighting and the future of Google Search

Search is hard, as Seth Godin wrote in 2005.

I mean, if we think SEO is hard (and it is) imagine if you were trying to build a search engine in a world where:

  • The users vary dramatically and change their preferences over time.
  • The technology they access search advances every day.
  • Competitors nipping at your heels constantly.

On top of that, you’re also dealing with pesky SEOs trying to game your algorithm gain insights into how best to optimize for your visitors.

That’s going to make it a lot harder.

Now imagine if the main technologies you need to lean on to advance came with their own limitations – and, perhaps worse, massive costs.

Well, if you’re one of the writers of the recently published paper, “End-to-End Query Term Weighting” you see this as an opportunity to shine.

What is end-to-end query term weighting?

End-to-end query term weighting refers to a method where the weight of each term in a query is determined as part of the overall model, without relying on manually programmed or traditional term weighting schemes or other independent models.

What does that look like?

All Marketers Are Liars

Here we see an illustration of one of the key differentiators of the model outlined in the paper (Figure 1, specifically).

On the right side of the standard model (2) we see the same as we do with the proposed model (4), which is the corpus (full set of documents in the index), leading to the documents, leading to the terms. 

This illustrates the actual hierarchy into the system, but you can casually think of it in reverse, from the top down. We have terms. We look for documents with those terms. Those documents are in the corpus of all the documents we know about.

To the lower left (1) in the standard Information Retrieval (IR) architecture, you’ll notice that there is no BERT layer. The query used in their illustration (nike running shoes) enters the system, and the weights are computed independently of the model and passed to it.

In the illustration here, the weights are passing equally among the three words in the query. However, it does not have to be that way. It’s simply a default and good illustration. 

What is important to understand is that the weights are assigned from outside the model and entered it with the query. We’ll cover why this is important momentarily.

If we look at the term-weight version on the right side, you’ll see that the query “nike running shoes” enters BERT (Term Weighting BERT, or TW-BERT, to be specific) which is used to assign the weights that would be best applied to that query.

From there things follow a similar path for both, a scoring function is applied and documents are ranked. But there’s a key final step with the new model, that is really the point of it all, the ranking loss calculation.

This calculation, which I was referring to above, makes the weights being determined within the model so important. To understand this best, let’s take a quick aside to discuss loss functions, which is important to really understand what’s going on here.

What is a loss function?

In machine learning, a loss function is basically a calculation of how wrong a system is with said system trying to learn to get as close to a zero loss as possible.

Let’s take for example a model designed to determine house prices. If you entered in all the stats of your house and it came up with a value of $250,000, but your house sold for $260,000 the difference would be considered the loss (which is an absolute value).

Across a large number of examples, the model is taught to minimize the loss by assigning different weights to the parameters it is given until it gets the best result. A parameter, in this case, may include things like square feet, bedrooms, yard size, proximity to a school, etc.

Now, back to query term weighting

Looking back at the two examples above, what we need to focus on is the presence of a BERT model to provide the weighting to the terms down-funnel of the ranking loss calculation. 

To put it differently, in the traditional models, the weighting of the terms was done independent of the model itself and thus, could not respond to how the overall model performed. It could not learn how to improve in the weightings.

In the proposed system, this changes. The weighting is done from within the model itself and thus, as the model seeks to improve it’s performance and reduce the loss function, it has these extra dials to turn bringing term weighting into the equation. Literally.

ngrams

TW-BERT isn’t designed to operate in terms of words, but rather ngrams.

The authors of the paper illustrate well why they use ngrams instead of words when they point out that in the query “nike running shoes” if you simply weight the words then a page with mentions of the words nike, running and shoes could rank well even if it’s discussing “nike running socks” and “skate shoes”.

Traditional IR methods use query statistics and document statistics, and may surface pages with this or similar issues. Past attempts to address this focused on co-occurrence and ordering.

In this model, the ngrams are weighted as words were in our previous example, so we end up with something like:

Core

On the left we see how the query would be weighted as uni-grams (1-word ngrams) and on the right, bi-grams (2-word ngrams).

The system, because the weighting is built into it, can train on all the permutations to determine the best ngrams and also the appropriate weight for each, as opposed to relying only on statistics like frequency.

Zero shot

An important feature of this model is its performance in zero-short tasks. The authors tested in on:

  • MS MARCO dataset – Microsoft dataset for document and passage ranking
  • TREC-COVID dataset – COVID articles and studies
  • Robust04 – News articles
  • Common Core – Educational articles and blog posts

They only had a small number of evaluation queries and used none for fine-tuning, making this a zero-shot test in that the model was not trained to rank documents on these domains specifically. The results were:

Godin

It outperformed in most tasks and performed best on shorter queries (1 to 10 words).

And it’s plug-and-play!

OK, that might be over-simplifying, but the authors write:

“Aligning TW-BERT with search engine scorers minimizes the changes needed to integrate it into existing production applications, whereas existing deep learning based search methods would require further infrastructure optimization and hardware requirements. The learned weights can be easily utilized by standard lexical retrievers and by other retrieval techniques such as query expansion.”

Because TW-BERT is designed to integrate into the current system, integration is far simpler and cheaper than other options.

What this all means for you

With machine learning models, it’s difficult to predict example what you as an SEO can do about it (apart from visible deployments like Bard or ChatGPT).

A permutation of this model will undoubtedly be deployed due to its improvements and ease of deployment (assuming the statements are accurate).

That said, this is a quality-of-life improvement at Google, that will improve rankings and zero-shot results with a low cost.

All we can really rely on is that if implemented, better results will more reliably surface. And that’s good news for SEO professionals.

The post TW-BERT: End-to-end query term weighting and the future of Google Search appeared first on Search Engine Land.

Original source: https://searchengineland.com/tw-bert-end-to-end-query-term-weighting-google-search-431907

93 thoughts on “TW-BERT: End-to-end query term weighting and the future of Google Search”

  1. With havin so much written content do you ever run into any problems of plagorism or copyright infringement?

    My blog has a lot of completely unique content I’ve either created myself or outsourced but it appears a lot
    of it is popping it up all over the web without my authorization. Do you
    know any methods to help protect against content from being stolen? I’d certainly appreciate it.

  2. Hi, I think your site might be having browser compatibility issues.
    When I look at your blog in Chrome, it looks fine but
    when opening in Internet Explorer, it has some overlapping.
    I just wanted to give you a quick heads up! Other then that,
    awesome blog!

  3. A person necessarily assist to make seriously posts I’d state.
    That is the very first time I frequented your web page and up to now?
    I amazed with the research you made to make this actual publish amazing.
    Magnificent activity!

  4. I love what you guys are up too. This kind of clever work and reporting!
    Keep up the great works guys I’ve included you guys to our blogroll.

  5. I like what you guys are up too. This sort of clever work and exposure!
    Keep up the very good works guys I’ve included you guys to blogroll.

  6. Every weekend i used to go to see this web page, for the reason that i
    wish for enjoyment, since this this web page conations genuinely good
    funny information too.

  7. First of all I want to say wonderful blog! I had a
    quick question in which I’d like to ask if you don’t mind.
    I was interested to find out how you center yourself and clear your head before writing.
    I have had a tough time clearing my thoughts in getting my
    ideas out there. I truly do take pleasure in writing but it just seems like the first 10 to 15 minutes are lost simply just trying to figure out
    how to begin. Any recommendations or tips? Many thanks!

  8. Wow that was odd. I just wrote an extremely long comment but after I clicked submit my comment didn’t
    appear. Grrrr… well I’m not writing all that over again. Anyways,
    just wanted to say wonderful blog!

  9. Can I just say what a relief to find someone who genuinely knows what they’re discussing online.

    You definitely understand how to bring a problem to light and
    make it important. More and more people should look
    at this and understand this side of your story.
    I was surprised you aren’t more popular since you most
    certainly possess the gift.

  10. Hey I know this is off topic but I was wondering if you knew of any widgets
    I could add to my blog that automatically tweet my newest twitter updates.
    I’ve been looking for a plug-in like this for quite some time and was hoping
    maybe you would have some experience with something like this.
    Please let me know if you run into anything. I
    truly enjoy reading your blog and I look forward to your new
    updates.

  11. Howdy this is kinda of off topic but I was wondering if blogs use WYSIWYG editors or if you have to manually code with
    HTML. I’m starting a blog soon but have no coding skills so I wanted to get
    guidance from someone with experience. Any help would be greatly appreciated!

  12. Wonderful blog! I found it while surfing around on Yahoo
    News. Do you have any tips on how to get listed in Yahoo News?
    I’ve been trying for a while but I never seem to get there!
    Appreciate it

  13. Hmm is anyone else experiencing problems with the pictures on this blog loading?
    I’m trying to figure out if its a problem on my end or if it’s the blog.
    Any feed-back would be greatly appreciated.

  14. Wow, awesome blog layout! How long have you been blogging for?
    you make blogging look easy. The overall look of your website is wonderful, let alone the content!

  15. Hi there would you mind sharing which blog platform you’re working
    with? I’m looking to start my own blog in the near future but
    I’m having a tough time selecting between BlogEngine/Wordpress/B2evolution and Drupal.

    The reason I ask is because your design seems different then most
    blogs and I’m looking for something completely unique.
    P.S Sorry for getting off-topic but I had to ask!

  16. My relatives all the time say that I am wasting my time here at web, however I know I
    am getting familiarity every day by reading such nice content.

  17. I got this website from my buddy who told me on the topic of this site and at the moment this time I am browsing this
    web page and reading very informative content at this time.

  18. Good day! I know this is kind of off topic but I was wondering which blog platform are you using for this website?
    I’m getting sick and tired of WordPress because I’ve had issues with
    hackers and I’m looking at options for another platform.
    I would be great if you could point me in the direction of
    a good platform.

  19. hey there and thank you for your information – I have definitely picked up
    anything new from right here. I did however expertise several technical issues using
    this site, as I experienced to reload the
    web site many times previous to I could get it to load properly.
    I had been wondering if your web hosting is OK?

    Not that I am complaining, but slow loading instances times will very frequently affect your placement in google
    and could damage your high-quality score if advertising and
    marketing with Adwords. Anyway I am adding this RSS to my email and
    could look out for a lot more of your respective
    fascinating content. Make sure you update this again soon.

  20. I’m really enjoying the design and layout of your blog.
    It’s a very easy on the eyes which makes it much more pleasant for me to
    come here and visit more often. Did you hire out a developer to create your theme?
    Fantastic work!

  21. This is really fascinating, You are a very professional
    blogger. I’ve joined your rss feed and stay
    up for seeking more of your fantastic post. Additionally, I have shared your website in my social networks

  22. Thank you for the auspicious writeup. It in fact
    was a amusement account it. Look advanced to more added agreeable
    from you! By the way, how can we communicate?

  23. A motivating discussion is worth comment. There’s no doubt that that you should write more on this
    subject, it may not be a taboo subject but usually folks don’t discuss such topics.

    To the next! All the best!!

  24. Thanks for every other magnificent article. The place else may just anybody get that type of info in such an ideal approach of
    writing? I have a presentation next week, and I’m on the look for such information.

  25. Wonderful goods from you, man. I’ve understand your stuff previous
    to and you’re just extremely excellent. I really like what you’ve acquired here, really like what you’re stating
    and the way in which you say it. You make it entertaining and you still care for to
    keep it smart. I can not wait to read far more from you.
    This is really a great web site.

  26. Greate article. Keep posting such kind of information on your
    site. Im really impressed by it.
    Hello there, You have performed an incredible job.

    I will certainly digg it and personally suggest to
    my friends. I am sure they will be benefited from this website.

  27. Heya i’m for the first time here. I found this board
    and I in finding It really useful & it helped
    me out much. I hope to give one thing again and help others such as you
    helped me.

  28. Thanks , I have just been looking for info approximately this
    subject for a while and yours is the best I have discovered till now.
    However, what in regards to the conclusion? Are you sure concerning the source?

  29. Hi there to every one, as I am in fact keen of reading this weblog’s post to be updated on a regular basis.
    It consists of nice stuff.

  30. I do believe all of the ideas you’ve introduced in your post.

    They’re very convincing and will certainly work. Still, the posts are very
    short for novices. May just you please extend them a bit from next time?

    Thank you for the post.

  31. Good day! Do you use Twitter? I’d like to follow you if that would be ok.
    I’m undoubtedly enjoying your blog and look forward to new posts.

  32. Magnificent beat ! I wish to apprentice while you amend your website, how
    could i subscribe for a blog web site? The account helped me a acceptable deal.
    I had been tiny bit acquainted of this your broadcast provided bright clear idea

  33. I got this web page from my friend who shared with me
    concerning this site and now this time I am visiting this website and
    reading very informative articles or reviews at this time.

  34. Currently it seems like BlogEngine is the top blogging platform out there right now.

    (from what I’ve read) Is that what you’re using on your blog?

  35. Wow, marvelous blog layout! How long have you been blogging for?
    you make blogging look easy. The overall look of your site is wonderful,
    as well as the content!

  36. Superb site you have here but I was wondering if you knew of any forums that cover the
    same topics discussed here? I’d really love to be a part of community where I can get opinions from other
    experienced people that share the same interest. If you have any recommendations, please let me know.
    Many thanks!

  37. Thank you, I have just been looking for info about this topic for ages and yours is the greatest I have came
    upon so far. However, what concerning the bottom line? Are you certain about the source?

  38. A motivating discussion is worth comment. I do believe that you should publish more on this subject,
    it might not be a taboo subject but generally people don’t discuss these issues.
    To the next! Kind regards!!

  39. I loved as much as you’ll receive carried out right here.
    The sketch is tasteful, your authored subject matter stylish.
    nonetheless, you command get bought an impatience over that you wish be delivering the following.
    unwell unquestionably come more formerly again as exactly the same nearly very often inside case you
    shield this increase.

  40. [url=https://frespinn.fun] Играть в казино[/url]
    Способ получить бездепозитные бонусы казино

    Индустрия онлайн-гемблинга frespinn.fun стремительно развивается. Еще несколько лет назад игроки довольствовались большим выбором игровых автоматов и других приложений. Сегодня ситуация коренным образом изменилась. В Интернете можно найти не только казино с бонусами за регистрацию. Благодаря отделу маркетинга игрового института игроков ждет множество подарков. Как насчет бесплатных вращений или дополнительных кредитов? Простая регистрация разделяет бесплатные спины и деньги.

    Для получения различных бонусов достаточно зарегистрироваться на сайте. Для этого заполните форму, указав реальные данные, логин, пароль, e-mail или воспользуйтесь кнопками быстрого входа. Все, что вам нужно, это страница в одной из популярных социальных сетей – Вконтакте, Твиттер, Фейсбук. Теперь вы можете зарегистрироваться на сайте и получить бездепозитный бонус за регистрацию в казино.

  41. [url=https://bezdepozitniebonusicasino.website] Бездепозитные бонусы в казино[/url]

    [url=https://bezdepozitniebonusicasino.website] Фриспины[/url]

    [url=https://bezdepozitniebonusicasino.website] Бонусы в казино[/url]

    [url=https://bezdepozitniebonusicasino.website] Промокоды в казино[/url]

    [url=https://bezdepozitniebonusicasino.website] Играть в казино[/url]

  42. [url=https://top10casino.website/] Бездепозитные бонусы в казино: лучшие бездепы 2023-2024[/url]
    Бездепы по праву считаются самой желанной разновидностью промо-акций в онлайн-казино. Причина проста – игроку не нужно вносить реальные деньги на баланс и тем самым подвергать свой капитал риску. В плюсе оказывается и игровая площадка, ведь ей удается заинтересовать потенциального клиента, который впоследствии может проявлять много активности.
    Например, пользователь может получить 500 рублей или 100 бесплатных вращений за обычное прохождение регистрации, после чего сможет воспользоваться подарочными средствами в любимых развлечениях. Более того, после отыгрыша любое вознаграждение можно благополучно вывести на банковскую карту, электронный кошелек или криптой.
    На текущей странице собраны актуальные предложения по бездепозитным бонусам на 2023-2024 годы. Игрок может выбрать наиболее предпочтительное предложение, перейти на официальный сайт онлайн-казино и выполнить действие. Как правило, для регистрации достаточно указать email, придумать пароль и выбрать валюту. Зачастую операторы казино в качестве бездепов начисляют фриспины в популярные слоты: Book of Dead, Razor Shark, Space Wars.

  43. [url=https://bezdepozitniebonusicasino.website] Бездепозитные бонусы в казино: лучшие бездепы 2023-2024[/url]
    Бездепы bezdepozitniebonusicasino.website по праву считаются самой желанной разновидностью промо-акций в онлайн-казино. Причина проста – игроку не нужно вносить реальные деньги на баланс и тем самым подвергать свой капитал риску. В плюсе оказывается и игровая площадка, ведь ей удается заинтересовать потенциального клиента, который впоследствии может проявлять много активности.
    Например, пользователь может получить 500 рублей или 100 бесплатных вращений за обычное прохождение регистрации, после чего сможет воспользоваться подарочными средствами в любимых развлечениях. Более того, после отыгрыша любое вознаграждение можно благополучно вывести на банковскую карту, электронный кошелек или криптой.
    На текущей странице собраны актуальные предложения по бездепозитным бонусам на 2023-2024 годы. Игрок может выбрать наиболее предпочтительное предложение, перейти на официальный сайт онлайн-казино и выполнить действие. Как правило, для регистрации достаточно указать email, придумать пароль и выбрать валюту. Зачастую операторы казино в качестве бездепов начисляют фриспины в популярные слоты: Book of Dead, Razor Shark, Space Wars.

  44. Рассматривая понятие инвестиционной деятельности, следует знать, что законом допускается совмещение функций нескольких субъектов. К примеру, инвестор имеет возможность самостоятельно реализовывать проект, а также получать от него прибыль.
    Период первоначальных инвестиционных затрат, в котором еще нет притоков от операционной деятельности, принимается за нулевой.
    9. Министерство Российской Федерации по развитию Дальнего Востока не позднее 10 календарных дней со дня поступления доработанных проектов инвестиционных программ согласовывает инвестиционные программы или направляет мотивированный отказ в их согласовании.
    Keywords: innovative project; the formation scheme; balanced set of investment projects; a set of stages of the investment and innovation project.
    Бонд – облигации (от англ. «Bond»). Облигации выпускают государство, корпорации, в том числе и банки. Инвестор может купить бонды банка и получать с них купонный доход.

    Интересная статья посвящена торговле на форекс вместе с советниками https://telegra.ph/Kak-skalping-stal-populyarnoj-strategiej-na-rynke-Foreks-istoriya-i-sushchnost-09-04
    180. Моисеева Н.К. Управление маркетингом: теория, практика, информационные технологии. Учебное пособие. М: Изд-во «Финансы и статистика», 2004 г.
    Практические рекомендации: Применение системы сбалансированного развития инновационной и инвестиционной деятельности промышленной корпорации позволяет эффективно использовать имеющиеся в наличии ресурсы и рационально реструктуризировать или свернуть инновационный проект в случае резкого ухудшения внешних условий, таких как современный системный финансовый кризис. Своевременная реструктуризация инновационного проекта дает возможность сохранить структуру и инвестиционные возможности промышленной корпорации и довести до успешного завершения инновационный проект от нематериальной идеи до создания производственных мощностей нового поколения.
    Липецкая область обеспечивает равные условия, как для российских, так и для иностранных инвесторов.
    В основе расчета этих коэффициентов лежат дисконтные способы, основанные на принципах временной денежной стоимости. Как правило, ставкой дисконтирования становится величина средневзвешенной стоимости капитала WACC, которая может корректироваться на показатели возможного риска (если возникает такая необходимость).
    Портал ведется на двух языках (русский и английский).

Leave a Reply

Your email address will not be published. Required fields are marked *

+ +