BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate one or more embodiments of the invention and, together with the description, explain the invention. In the drawings,
[0013] FIGS. 1 and 2 are exemplary diagrams of an overview of an implementation in which observed user behavior and known quality ratings associated with a set of advertisements are used to construct a statistical model that can be used for estimating advertisement quality, and advertisements can then be filtered, ranked or promoted based on the estimated advertisement quality;
[0014] FIG. 3 is an exemplary diagram of a network in which systems and methods consistent with the principles of the invention may be implemented;
[0015] FIG. 4 is an exemplary diagram of a client or server of FIG. 3 according to an implementation consistent with the principles of the invention;
[0016] FIG. 5 is a flowchart of an exemplary process for constructing a model of user behavior associated with the selections of multiple on-line advertisements according to an implementation consistent with the principles of the invention;
[0017] FIGS. 6-13 illustrate various exemplary session features, corresponding to observed or logged user actions, that may be used for constructing a statistical model for predicting advertisement quality;
[0018] FIG. 14 is a flowchart of an exemplary process for determining predictive values relating to the quality of an advertisement according to an implementation consistent with the principles of the invention;
[0019] FIG. 15 is a diagram that graphically illustrates the exemplary process of FIG. 14 consistent with an aspect of the invention;
[0020] FIG. 16 is a diagram of an exemplary data structure for storing the predictive values determined in FIG. 14;
[0021] FIGS. 17 and 18 are flow charts of an exemplary process for estimating odds of good or bad qualities associated with advertisements using the predictive values determined in the exemplary process of FIG. 14 consistent with principles of the invention;
[0022] FIG. 19 is a flowchart of an exemplary process for predicting the quality of advertisements according to an implementation consistent with the principles of the invention;
[0023] FIG. 20 is a diagram that graphically illustrates the exemplary process of FIG. 19 consistent with an aspect of the invention;
[0024] FIG. 21 is a flowchart of an exemplary process for filtering, ranking and/or promoting advertisements according to an implementation consistent with principles of the invention;
[0025] FIGS. 22-24 illustrate various examples of advertisement filtering consistent with aspects of the invention;
[0026] FIGS. 25 and 26 illustrate examples of advertisement ranking consistent with aspects of the invention;
[0027] FIGS. 27-29 illustrate examples of advertisement promotion consistent with aspects of the invention; and
[0028] FIG. 30 illustrates an exemplary search result document that includes filtered, ranked and/or promoted advertisements consistent with an aspect of the invention.
DETAILED DESCRIPTION
[0029] The following detailed description of the invention refers to the accompanying drawings. The same reference numbers in different drawings may identify the same or similar elements. Also, the following detailed description does not limit the invention.
[0030] Systems and methods consistent with aspects of the invention may use multiple observations of user behavior (e.g., real-time observations or observations from recorded user logs) associated with user selection of on-line advertisements to more accurately estimate advertisement quality as compared to conventional determinations based solely on CTR. Quality ratings associated with known rated advertisements, and corresponding measured observed user behavior associated with selections (e.g., “clicks”) of those known rated advertisements, may be used to construct a statistical model. The statistical model may subsequently be used to estimate qualities associated with advertisements based on observed user behavior, and/or features of the selected ad or a query used to retrieve the ad, associated with selections of the advertisements. The estimated qualities associated with advertisements may be used for determining which advertisements to provide to users, for ranking the advertisements, and/or for promoting selected ones of the advertisements to a prominent position on a document provided to users.
[0031] A “document,” as the term is used herein, is to be broadly interpreted to include any machine-readable and machine-storable work product. A document may include, for example, an e-mail, a web page or site, a business listing, a file, a combination of files, one or more files with embedded links to other files, a news group posting, a blog, an on-line advertisement, etc. Documents often include textual information and may include embedded information (such as meta information, images, hyperlinks, etc.) and/or embedded instructions (such as Javascript, etc.). A “link,” as the term is used herein, is to be broadly interpreted to include any reference to/from a document from/to another document or another part of the same document.
OVERVIEW
[0032] FIGS. 1 and 2 illustrate an exemplary overview of an implementation of the invention in which observed user behavior may be used to estimate qualities of on-line advertisements and then the estimated qualities may be used in filtering, ranking and/or promoting selected advertisements.
[0033] As shown in FIG. 1, each one of multiple rated advertisements 100-1 through 100-N (collectively referred to herein as ad 100) may be associated with a corresponding document 105-1 through 105-N (collectively referred to herein as document 105). Each document 105 may include a set of search results resulting from a search executed by a search engine based on a search query provided by a user and may further include one or more advertisements in addition to a rated ad 100. Each advertisement 100 may be associated with ratings data 120 provided by human raters who have rated a quality of each rated advertisement 100. Each advertisement 100 may advertise various products or services.
[0034] In response to receipt of an advertisement 100, the receiving user may, based on the “creative” displayed on the advertisement, select 110 the advertisement (e.g., “click” on the displayed advertisement using, for example, a mouse). After ad selection 110, an ad landing document 115 may be provided to the selecting user by a server hosting the advertisement using a link embedded in ad 100. The ad landing document 115 may provide details of the product(s) and/or service(s) advertised in the corresponding advertisement 100.
[0035] Before, during and/or after each ad selection 110 by a user, session features 125 associated with each ad selection 110 during a “session” may be measured in real-time or logged in memory or on disk. A session may include a grouping of user actions that occur without a break of longer than a specified period of time (e.g., a group of user actions that occur without a break of longer than three hours).
[0036] The measured session features 125 can include any type of observed user behavior or actions. For example, session features 125 may include a duration of the ad selection 110 (e.g., a duration of the “click” upon the ad 100), the number of selections of other advertisements before and/or after a given ad selection, the number of selections of search results before and/or after a given ad selection, the number of selections on other types of results (e.g., images, news, products, etc.) before and/or after a given ad selection, a number of document views (e.g., page views) before and/or after a given ad selection (e.g., page views of search results before and/or after the ad selection), the number of search queries before and/or after a given ad selection, the number of queries associated with a user session that show advertisements, the number of repeat selections on a same given advertisement, or an indication of whether a given ad selection was the last selection in a session, the last ad selection in a session, the last selection for a given search query, or the last ad selection for a given search query. Other types of observed user behavior associated with ad selection, not described above, may be used consistent with aspects of the invention.
[0037] Using the measured session features 125 and ad ratings data 120, associated with each ad selection 110 of a corresponding rated advertisement 100, a statistical model 130 may be constructed (as further described below). The statistical model may include a probability model derived using statistical techniques. Such techniques may include, for example, logistic regression, regression trees, boosted stumps, or any other statistical modeling technique. Statistical model 130 may provide a predictive value that estimates the likelihood that a given advertisement is good given measured session features associated with a user selection of the advertisement (e.g., P(good ad|ad selection)=f.sub.g(session features)).
[0038] Subsequent to construction of statistical model 130, ad quality values of advertisements selected by one or more users may be predicted. An ad 135, associated with a document 140 and hosted by a server in a network, may be provided to an accessing user. Session features 155 associated with user selection 145 of ad 135 may be measured or logged in memory or on disk, and the measurements may be provided as inputs into statistical model 130. Statistical model 130 may determine a likelihood that ad 135 is a good ad, given the measured session features, and may predict an ad quality value(s) 160 for ad 135. Though FIG. 1 depicts the prediction of a quality value associated with a single ad 135, ad quality values 160 may estimated for each ad 135 selected by multiple users to produce multiple predicted ad quality values 160.