[0148] In another implementation, the quality parameter QP may include the value P.sub.BAD AD determined in block 1950 above and P.sub.BAD AD may be multiplied by CTR. For example, if P.sub.BAD AD*CTR is high (e.g., greater than a threshold value), then the ad will be disabled and, thus, not provided to the user. FIG. 23 illustrates an example of the filtering of multiple ads 2300-1 through 2300-N. For each ad 2300, a value 2310 for P.sub.BAD AD*CTR may be determined and compared to a threshold value (T) 2320. Ads having values 2310 greater than or equal to the threshold T 2020 may be disabled 2130 and ads having values 2310 less than the threshold T 2320 may be provided 2340 (e.g., shown) to the user.
[0149] In yet another implementation, the ratio P.sub.GOOD AD/P.sub.BAD AD may be used as a disabling rule. For example, if P.sub.GOOD AD/P.sub.BAD AD is less than a threshold value, indicating that the probability that an ad is good is lower than the probability that the ad is bad, then the ad will be disabled and, thus, not provided to the user. FIG. 24 illustrates an example of the filtering of multiple ads 2400-1 through 2400-N. For each ad 2400, a value 2410 for P.sub.GOOD AD/P.sub.BAD AD may be determined and compared to a threshold value (T) 2420. Ads having values 2410 less than the threshold T 2420 may be disabled 2430 and ads having values 2410 greater than or equal to the threshold T 2420 may be provided 2440 (e.g., shown) to the user.
[0150] The ads determined to be relevant may be ranked based on the obtained quality parameter(s) (optional block 2130). The relevant ads may be ranked based on the one or more quality parameters obtained in block 2110 above, or based on any other type of ad quality parameter, other than, or in addition to a CTR. The relevant ads may be ranked based on a functional combination of the obtained quality parameter(s) and other parameters (e.g., CTR). In one implementation, the quality parameter QP may include the value P.sub.GOOD AD, determined in block 1950 above. In other implementations, the quality parameter QP may include the value P.sub.BAD AD.
[0151] The functional combination of the obtained quality parameter QP and other parameters may attempt to maximize value to the ad publishing entity, the advertisers, and the users. In one implementation, ads may be ranked using the following function: RANK.sub.ADx=P.sub.GOOD AD.sub.–.sub.ADx*CTR.sub.ADxCPC.sub.ADx Eqn. (9) where CTR is the click-through-rate and CPC is the “cost per click” for that ad. CPC represents the value of an ad click to a given advertiser. FIG. 25 illustrates an example of the ranking of multiple ads 2500-1 through 2500-N. For each ad 2500, a value 2510 for P.sub.GOOD AD*CTR*CPC may be determined. A value 2510 for each ad in the set of relevant ads may be compared so that the ads may be re-ordered in a ranked order 2520. The ranked order 2520 may, for example, as shown in FIG. 25, rank the ads 2500 in ascending order, with the ad 2500-1 having the highest value 2510-1 being ranked first, and the ads 2500-2 through 2500-N having values 2510 less than value 2510-1 being ranked in descending order.
[0152] In another implementation, ads may be ranked using the following function: RANK.sub.ADx=CTR.sub.ADx*CPC.sub.ADx+ValueOfGoodAdToUser*P.sub.- GOOD AD.sub.–.sub.ADx*CTR.sub.ADx -CostofBadAdToUser*P.sub.BAD AD.sub.–.sub.ADx*CTR.sub.ADx Eqn. (10) where CTR is the click-through-rate, CPC is the cost per click for that ad, ValueOfGoodAdToUser is the incremental gain in revenue that an ad publisher may receive from showing a good ad, and CostOfBadAdToUser is the incremental loss in long-term revenue that the ad publisher may sustain from providing a bad ad to the user. The value CTR*CPC represents the short-term revenue that an ad may receive.
[0153] The values ValueOfGoodAdToUser and CostOfBadAdToUser may be estimated in a number of different ways. In one technique, human factors experiments can be run, where users are shown a series of documents having only good ads, and then the users can be provided with a behavioral task to see how likely they are to use the ads. A different set of users can be shown a series of documents having only bad ads, and then this set of users can be provided with a behavioral task to see how likely they are to not use the ads. This can then be refined to see how many documents it takes to change the likelihood of clicking on ads in the behavioral task, and how varying the mix (e.g., a mix of good and bad ads) will change the likelihood. In another technique, session data may be used to observe the sequences of clicks that a user performs within a session, and to determine (by empirical measurement) the probability of further ad clicks after seeing a bad ad (and the same for a good ad).
[0154] In either of the techniques set forth above, the increased likelihood of a user clicking on an ad (if the user is shown good ads) or the decreased likelihood of a user clicking on an ad (if the user is shown bad ads) can be estimated. To derive the value ValueOfGoodAdToUser, the incremental increase can be multiplied by the average value of a click, while the value CostOfBadAdToUser can be derived by multiplying the incremental decrease by an average value of a click. In some implementations, the values ValueOfGoodAdToUser and CostOfBadAdToUser may be adjusted to customize the cost of a click per country or per-business (e.g., travel, finance, consumer goods, etc.) such that the values ValueOfGoodAdToUser and CostOfBadAdToUser have a different cost per click depending on the country, the language, and/or the business.
[0155] FIG. 26 illustrates an example of the ranking of multiple ads 2600-1 through 2600-N. For each ad 2600, a value 2610 may be determined using Eqn. (9) above. The values 2610 for each ad in the set of relevant ads may be compared so that the ads may be re-ordered in a ranked order 2620. The ranked order 2620 may, for example, as shown in FIG. 26, rank the ads 2600 in ascending order, with the ad 2600-1 having the highest value 2610-1 being ranked first, and the ads 2600-2 through 2600-N having values 2610 less than value 2610-1 being ranked in descending order.
[0156] Selected ones of the ads determined to be relevant may be promoted (optional block 2140). Selection of which ads to be promoted may be based on the one or more quality parameters obtained in block 2110 above, or based on any other type of ad quality parameter, in addition to a CTR. Ads may be promoted based on a functional combination of the obtained quality parameter(s) and other parameters (e.g., CTR). In one implementation, the quality parameter QP may include the value P.sub.GOOD AD determined in block 1950 above. In other implementations, the quality parameter QP may include the value P.sub.BAD AD determined in block 1950 above. In one implementation, for example, if P.sub.GOOD AD*CTR is high (e.g., greater than a threshold), or if P.sub.GOOD AD/P.sub.BAD AD is high (e.g., greater than a threshold), then the ad may be promoted.
[0157] FIG. 27 illustrates an example of the promoting of an ad of multiple ads 2700-1 through 2700-N. For each ad 2700, a value 2710 for P.sub.GOOD AD*CTR may be determined and compared to a threshold value (T) 2720. Ads having values 2710 greater than or equal to the threshold T 2720 may be promoted 2730 and ads having values 2710 less than the threshold T 2720 may not be promoted 2740.
[0158] FIG. 28 illustrates another example of the promoting of an ad of multiple ads 2800-1 through 2800-N. For each ad 2800, a value 2810 for P.sub.GOOD AD/P.sub.BAD AD may be determined and compared to a threshold value (T) 2820. Ads having values 2810 greater than or equal to the threshold T 2820 may be promoted 2840 and ads having values 2810 less than the threshold T 2820 may not be promoted 2840.
[0159] In another implementation, the function set forth in Eqn. (9) above may alternatively be used, with the value CostOfBadAdToUser being set higher than the value used in Eqn. (9) above for ranking ads. Setting the value of CostOfBadAdToUser higher than the value used in Eqn. (9) above indicates that it is more costly to promote a bad ad than to just show a bad ad. FIG. 29 illustrates a further example of the promoting of an ad of multiple ads 2900-1 through 2900-N. For each ad 2900, a value 2910 for Eqn. (9) above may be determined and compared to a threshold value (T) 2920. Ads having values 2910 greater than or equal to the threshold T 2920 may be promoted 2930 and ads having values 2910 less than the threshold T 2920 may not be promoted 2940.
[0160] Certain ones of the ads determined to be relevant may be selectively provided to a user based on the filtering, ranking and/or promoting performed in blocks 2120, 2130 and/or 2140 (block 2150). Relevant ads, which were not disabled in block 2120, may be provided to the user. Relevant ads, which do not include the disabled ads, may further be provided to the user in an order determined by the ranking function in block 2130. One or more of the relevant ads, which may not include the disabled ads, may be promoted as determined in block 2140. FIG. 30 illustrates a search result document 3000 in which search results 3010 are provided to a user that issued a search query. Ranked ads 3020, listed in ranked order, may further be included in the search result document 3000. The ranked ads 3020 may include, for example, links to ad landing documents which provide further details about the product or service being advertised. Promoted ads 3030, placed at a prominent or highlighted position, may additionally be included in the search result document 3000.
CONCLUSION
[0161] The foregoing description of preferred embodiments of the present invention provides illustration and description, but is not intended to be exhaustive or to limit the invention to the precise form disclosed. Modifications and variations are possible in light of the above teachings, or may be acquired from practice of the invention. For example, while series of acts have been described with regard to FIGS. 5, 14, 17, 18, 19 and 21, the order of the acts may be modified in other implementations consistent with the principles of the invention. Further, non-dependent acts may be performed in parallel.
[0162] In addition to the session features described above with respect to FIG. 5, conversion tracking may optionally be used in some implementations to derive a direct calibration between predictive values and user satisfaction. A conversion occurs when a selection of an advertisement leads directly to user behavior (e.g., a user purchase) that the advertiser deems valuable. An advertiser, or a service that hosts the advertisement for the advertiser, may track whether a conversion occurs for each ad selection. For example, if a user selects an advertiser’s ad, and then makes an on-line purchase of a product shown on the ad landing document that is provided to the user in response to selection of the ad, then the advertiser, or service that hosts the ad, may note the conversion for that ad selection. The conversion tracking data may be associated with the identified ad selections. A statistical technique, such as, for example, logistic regression, regression trees, boosted stumps, etc., may be used to derive a direct calibration between predictive values and user happiness as measured by conversion.
[0163] It will be apparent to one of ordinary skill in the art that aspects of the invention, as described above, may be implemented in many different forms of software, firmware, and hardware in the implementations illustrated in the figures. The actual software code or specialized control hardware used to implement aspects consistent with the principles of the invention is not limiting of the invention. Thus, the operation and behavior of the aspects have been described without reference to the specific software code, it being understood that one of ordinary skill in the art would be able to design software and control hardware to implement the aspects based on the description herein.
[0164] No element, act, or instruction used in the present application should be construed as critical or essential to the invention unless explicitly described as such. Also, as used herein, the article “a” is intended to include one or more items. Where only one item is intended, the term “one” or similar language is used. Further, the phrase “based on” is intended to mean “based, at least in part, on” unless explicitly stated otherwise.