[0055] The exemplary process may begin with obtaining ratings data associated with rated advertisements (block 500). The ratings data may include human generated data that rates the quality of each of the rated ads (e.g., one way of rating an ad is to rate how relevant is the ad relative to the query issued). Session features associated with each selection of a rated advertisement may then be obtained (block 510). The session features may be obtained in real-time by observing actual user behavior during a given user session, that occurred before, during and after the presentation of each ad impression to a user, or may be obtained from recorded logs of session features (i.e., user behavior and actions) that were stored in a data structure before, during and/or after the presentation of each ad impression to a user. The obtained session features 125 can include any type of observed user behavior. Each of the session features 125 may correspond to an indirect measurement of user satisfaction with a given advertisement. Certain ones of the session features 125 may be factors in determining how different users have different values for other ones of the session features 125 (e.g., users with dial-up connections may have longer ad selection durations than users who have high speed Internet connections).
[0056] Session features 125 may include, but are not limited to, a duration of an ad selection (e.g., a duration of the “click” upon the advertisement), a number of selections of other advertisements before and/or after a given ad selection, a number of selections of search results before and/or after a given ad selection, a number of selections of other results before and/or after a given ad selection, a number of document views (e.g., page views) before and/or after a given ad selection, a number of search queries before and/or after a given ad selection, a number of search queries associated with a user session that show advertisements, a number of repeat selections on a same given advertisement, or an indication of whether a given ad selection was the last selection in a session, the last ad selection in a session, a last selection for a given search query, or the last ad selection for a given search query. FIGS. 6-13 below depict various exemplary types of user behavior, consistent with aspects of the invention, that may be measured as session features.
[0057] FIG. 6 illustrates the measurement of a duration of an ad selection as a session feature 600. As shown in FIG. 6, an ad 605, that is associated with a document 610, may be provided to a user. In response to receipt of ad 605, the user may select 615 ad 605, and an ad landing document 620 may be provided to the user. A duration 625 of the ad selection (e.g., the period of time from selection of the advertisement until the user’s next action, such as clicking on another ad, entering a new query, etc.) may be measured as a session feature 600.
[0058] FIG. 7 illustrates the measurement of a number of other ad selections before and/or after a particular ad selection as a session feature 700. Given a particular selection 705 of an ad N 710, and provision of an ad landing document 715 in response to the ad selection 705, a number of one or more previous ad selections 720 of ads N-x 725, corresponding to provisions of previous ad landing documents 730, may be measured. Additionally, or alternatively, given a particular selection 705 of an ad N 710, a number of one or more subsequent ad selections 735 of ads N+x 740, corresponding to provisions of subsequent ad landing documents 745, may be measured. The number of other ad selections before and/or after a particular ad selection may be measured as a session feature 700.
[0059] FIG. 8 illustrates the measurement of a number of search result selections before and/or after a particular ad selection as a session feature 800. Given a particular selection 805 of an ad N 810, and provision of an ad landing document 815 in response to the ad selection 805, a number of search result documents 820 viewed by the user before the ad selection 805 may be measured as a session feature 800. The search result documents may be provided to the user based on the execution of a search using a search query issued by the user. Additionally, or alternatively, a number of search result documents 825 viewed by the user after the ad selection 805 may be measured as a session feature 800.
[0060] FIG. 9 illustrates the measurement of a number of documents viewed by a user before and/or after a particular ad selection as a session feature 900. Given a particular selection 905 of an ad 910, and provision of an ad landing document 915 in response to the ad selection 905, a number of documents 920 viewed by a user (e.g., page views) before the ad selection 905 may be measured as a session feature 900. Additionally, or alternatively, a number of documents 925 viewed by a user (e.g., page views) after the ad selection 905 may be measured as a session feature 900.
[0061] FIG. 10 illustrates the measurement of a number of search queries issued by a user before and/or after a particular ad selection as a session feature 1000. Given a particular selection 1005 of an ad 1010, and provision of an ad landing document 1015 in response to the ad selection 1005, a number of search queries 1020 issued by a user before the ad selection 1005 may be measured as a session feature 1000. Additionally, or alternatively, a number of search queries 1025 issued by a user after the ad selection 1005 may be measured as a session feature 1000.
[0062] FIG. 11 illustrates the measurement of a number of search queries, in a session that includes a particular ad selection, that results in the display of an advertisement as a session feature 1100. Given a session that includes a particular ad selection, a number of search queries 1105 may be measured that result in the display of a corresponding ad 1110-1 through 1110-N. The number of search queries may be measured as a session feature 1100. The number of search queries 1105 resulting in the display of an advertisement may indicate the commercial nature of a given user session.
[0063] FIG. 12 illustrates the measurement of a number of repeat selections of the same advertisement by a user as a session feature 1200. As shown in FIG. 12, an ad 1205, that may be associated with multiple documents 1210-1 through 1210-N, may be provided to a user one or more times. In response to each receipt of the ad 1205, the user may select 1215 ad 1205, and an ad landing document 1220 may be provided to the user for each of the repeated user selections. The number of repeat selections of the same advertisement by the user may be measured as a session feature 1200.
[0064] FIG. 13 illustrates the determination of whether an ad selection is the last ad selection for a given search query, or whether the ad selection is the last ad selection for a user session as a session feature 1300. As shown in FIG. 13, a user may issue a search query 1305 during a given session 1310, and one or more ads 1315 may be provided to the user subsequent to issuance of search query 1305. In response to each receipt of the ad(s) 1315, the user may select 1320 ad 1315, and an ad landing document 1325 may be provided to the user. A determination may be made whether the ad selection 1320 is the last ad selection for search query 1305. Thus, if multiple ads were selected by the user that issued search query 1305, then only the last ad selection for search query 1305 may be identified. A determination may also be made whether the ad selection 1320 was the last ad selection for session 1310. Therefore, if multiple ad selections have been made by the user during a given session, then only the last ad selection for the session may be identified.
[0065] Other types of user behavior, not shown in FIGS. 6-13, may be used as session features consistent with principles of the invention. The following lists numerous examples of other exemplary session features: [0066] 1) instead of an ad selection duration, a ratio of a given ad selection duration relative to an average ad selection duration for a given user may be used as a session feature. [0067] 2) a ratio of a given ad selection duration relative to all selections (e.g., search result selections or ad selections); [0068] 3) how many times a user selects a given ad in a given session. [0069] 4) a duration of time, from an ad result selection, until the user issues another search query. This may include time spent on other pages (reached via a search result click or ad click) subsequent to a given ad click. [0070] 5) a ratio of the time, from a given ad result selection until the user issues another search query, as compared to all other times from ad result selections until the user issued another search query. [0071] 6) time spent, given an ad result selection, on viewing other results for the search query, but not on the given ad result. [0072] 7) a ratio of the time spent in 6) above (i.e., the time spent on other results rather than the click duration) to an average of the time spent in 6) across all queries. [0073] 
how many searches (i.e., a unique issued search query) that occur in a given session prior to a given search result or ad selection; [0074] 9) how many searches that occur in a given session after a given search result or ad selection. [0075] 10) rather than searches, how many result page views that occur for a given search query before a given selection. This can be computed within the query (i.e., just for a unique query), or for the entire session; [0076] 11) rather than searches, how many search result page views that occur for a given search query after this selection. This can be computed within the query (i.e., just for the unique query), or for the entire session; [0077] 12) the total number of page views that occur in the session; [0078] 13) the number of page views in the session that show ads; [0079] 14) the ratio of the number of page views in the session that show ads to the total number of page views that occur in the session; [0080] 15) total number of ad impressions shown in the session; [0081] 16) average number of ads shown per query that shows ads, another measure of the commerciality of the session; [0082] 17) query scan time–how long from when the user sees the results of a query to when the user does something else (click on an ad, search result, next page, new query, etc.); [0083] 18) ratio between a given query scan time and all other query scan times; [0084] 19) total number of selections (e.g., clicks) that occured on a given search. These selections include all types of selections (e.g., search, onebox, ads) rather than just ad selections; [0085] 20) total number of selections that occurred on a search before a given ad selection; [0086] 21) total number of selections that occurred on a search after a given ad selection; [0087] 22) total number of ad selections that occurred on a search. May need to be normalized by the number of ads on the page; [0088] 23) total number of ad selections that occurred on a search before a given ad selection; [0089] 24) total number of ad selections that occurred on a search after a given ad selection; [0090] 25) total number of ad selections, that occurred on a search, whose ad positions on a document were located above a position of a given ad on the document; [0091] 26) total number of ad selections, that occurred on a search, whose ad positions on a document were located below a position of a given ad on the document; [0092] 27) total number of ad selections that occurred on a search that are not on a given ad; [0093] 28) total number of search result selections that occurred on a search; [0094] 29) total number of search selections that occurred on a search before a given ad selection; [0095] 30) total number of search result selections that occurred on a search after a given ad selection; [0096] 31) total number of search result selections of a long duration that occurred in the session; [0097] 32) total number of search result selections of a short duration that occurred in the session; [0098] 33) total number of search result selections that are last that occurred in the session. A given user may end a session by clicking on a search result, with no subsequent actions, or the user may end a session in some other fashion (e.g., ad result click, issuing a query and not clicking, etc.); [0099] 34) total number of non-search result and non-ad selections that occurred on a search; [0100] 35) an indication of whether there was a conversion from this ad selection; [0101] 36) an indication of the connection speed of the user (e.g., dialup, cable, DSL); [0102] 37) an indication of what country the user is located in. Different cultures might lead to users reacting differently to the same ad or having different cultural reactions or staying on sites differently; [0103] 38) an indication of what region of the world (e.g., APAC=asia pacific) that the user is located in; [0104] 39) was the keyword for a given ad an exact match to the search query (i.e., has all of the same terms as the query) or is missing one word, more than one word, or has rewrite terms. Often, the quality of an ad can vary (the more exact the match, the higher the quality) and keyword matching can be a reasonable way to segment ads, and predict whether an ad is good or bad separately for different match types. [0105] 40) an indication of an estimated click through rate (CTR) for a given ad; [0106] 41) what cost per click (CPC) did the advertiser pay for a given ad selection? The likelihood that an ad is good may depend on how much the advertiser paid (more is higher quality); [0107] 42) what CPC was the advertiser willing to pay? In ad auctioning, advertiser bidding may be used to set ad ranking and the ad/advertiser ranked lower than a given ad/advertiser sets the price that is actually paid by the next higher ranked ad/advertiser; [0108] 43) effective CPC*predicted CTR; or [0109] 44) bid CPC*predicted CTR. The above describes numerous examples of session features that may be used for the statistical model. However, one skilled in the art will recognize that other session features may be used, alternatively, or in conjunction with any of the above-described session features.