|
Field
Experimental Design (Public)
|
Before
The intervention will involve 79 sites directly managed by Service Ontario (it excludes private ServiceOntario centers).
We will randomly assign each site to one of three conditions. Each site will be in the same condition in the two times in which it will be treated: June 19 2017, and one day in November (or early December) 2017 to be defined. These two days in June and November/December 2017 are the therefore intervention periods. The trial, including the collection of pre-intervention performance data and post intervention performance data, goes from November 2016 to June 2018.
Here is a description of the three conditions.
Group 1 – CSRs in Group 1 offices will receive general information about organ donation, as well as some tips for asking customers to sign up.
Group 2 – CSRs in Group 2 offices will receive general information about organ donation, some tips for asking customers to sign up, as well as quantitative information about their individual registration rates, in numeric as well as graphical form (bar chart). The registration rates that we will report is the average over six months (November 2016-April 2017 for the first round, and May 2017-October 2017 for the second round). We had to limit the historical performance data to the end of April because these were the most updated data that were available when defining the intervention.
Group 3 – CSRs in Group 3 offices will receive general information about organ donation, some tips for asking customers to sign up, as well as quantitative information about their individual registration rates, and the average and 80th percentile registration rates in the region to which the center belongs (there are four regions: Central, Western, Eastern and Northern Ontario). This information, again, will be both in numeric and graphical form (bar chart -- see below). The regional distributions (from which the mean and 80th are derived) are therefore distributions of individual average registration rates over the period of interest.
The individual registration rate that we will report is the average over six months (November 2016-April 2017 for the first round, and May 2017-October 2017 for the second round). We had to limit the historical performance data to the end of April because these were the most updated data that were available when defining the intervention.
In group 2 and 3, the communication will include a note specifying that the registration rate information is exclusively for informational purposes and will not affect any formal evaluation.
CSRs will be assigned to the condition of the office where they are working as of the date of the first intervention. Some CSRs worked at more than one location between November 2016 and the time of the intervention; their personal registration rate that they will see if assigned to Group 2 or 3 is the average over the full six month period from November 1 2016 to April 30 2017 (included) -- we sum all of their registrations and divide them by the total of the customers they served.
To calculate the average and 80th registration rate for a region, however, we will calculate, for CSRs who worked in multiple regions (a minority of the total), separate average registration rate limited to the interactions in a given region. So for example, if in the period Nov 1 2016-Apr 30 2017 a CSR called Y served 200 customers in region A and had 10 registrations, and served 100 customer in region B with 8 registrations (and she is currently working in site X of region B), she will:
1) Be assigned to the treatment condition randomly determined for site X in region B
2) Be informed, if in group 2 or 3, that her registration rate between Nov 1 2016 and Apr 30 2017 was (10+8)/(100+200)=18/300=6%.
In determining the average and 80th signup rate for region A and B to show to CSRs in sites assigned to Group 3 in those region, however, we include a registration rate of 10/200=5% by CSR Y in region A, and a registration rate of 8/100=8% by CSR Y in region B.
Because CSR Y currently worked in an office of region B, if her current office is in Group 3 she will be shown her overall registration rate of 6%, and will be shown the average and 80th registration rate for region B. The reason for these calculation choices is that we want to give a CSR a sense of her own overall performance, but we want to give, as reference in group 3, statistics for the current region, because of potential regional differences. This is also the reason for separating, in calculating overall distributions, the performances of one given CSRs across regions.
Moreover, 3 of the 79 offices were administratively relocated to a different region in early April 2017. Because for the most part of the period for which we collected CSR performance data they were assigned to a region different from the current one, we kept them in the "old" region for the purposes of assigning reference stats (for Group 3) and for the stratified randomization (see below).
The communications just described will occur via email, sent at the dates specified above. An email from the Assistant Deputy Minister-Customer Care at ServiceOntario will inform about further communication to follow about organ donation registrations.
The emails for group 2 and 3 will include a brief explanation that the individual registration rates include the full activity of a CSR between Nov 1 2016 and Apr 30 2017, and will clarify (in Group 3) from what region the distributional info on registration rates comes.
Information on signup rate performance will be in numerical and graphical form.
In Group 2, the graph will be a horizontal bar with the signup rate reported next to it. The horizontal bars will be of the same length for all CSRs, regardless of the actual signup rate of a given CSR. The reason for this choice is to focus on the "absolute" value of the rate rather than the relative one.
In group 3, there will be three bars: the actual performance, and the average and 80th signup rate (or, better, average individual signup rates) for the relevant region. The scale of the bars will be such that the longest of the three bars (i.e., either the 80th percentile bar or the bar representing the individual performance) will be of a defined length, the same for all CSRs, regardless of their actual absolute value. The other two bars will be scaled accordingly. Doing so maximizes the relative difference between bars, especially for low performers.
In order to protect the privacy of the CSRs and to not exert undue influence, we will not inform the CSRs of the intervention under way. The researchers will not have access to the identity of the CSRs or other identifying information. We will elaborate a procedure such that it will be possible to assign CSRs to different treatments using numerical IDs, and to use these IDs to follow CSRs over time.
|
After
The intervention will involve 79 sites directly managed by Service Ontario (it excludes private ServiceOntario centers).
We will randomly assign each site to one of three conditions. Each site will be in the same condition in the two times in which it will be treated: June 19 2017, and one day in November (or early December) 2017 to be defined. These two days in June and November/December 2017 are the therefore intervention periods. The trial, including the collection of pre-intervention performance data and post intervention performance data, goes from November 2016 to June 2018.
Here is a description of the three conditions.
Group 1 – CSRs in Group 1 offices will receive general information about organ donation, as well as some tips for asking customers to sign up.
Group 2 – CSRs in Group 2 offices will receive general information about organ donation, some tips for asking customers to sign up, as well as quantitative information about their individual registration rates, in numeric as well as graphical form (bar chart). The registration rates that we will report is the average over six months (November 2016-April 2017 for the first round, and May 2017-October 2017 for the second round). We had to limit the historical performance data to the end of April because these were the most updated data that were available when defining the intervention.
Group 3 – CSRs in Group 3 offices will receive general information about organ donation, some tips for asking customers to sign up, as well as quantitative information about their individual registration rates, and the average and 80th percentile registration rates in the region to which the center belongs (there are four regions: Central, Western, Eastern and Northern Ontario). This information, again, will be both in numeric and graphical form (bar chart -- see below). The regional distributions (from which the mean and 80th are derived) are therefore distributions of individual average registration rates over the period of interest.
The individual registration rate that we will report is the average over six months (November 2016-April 2017 for the first round, and May 2017-October 2017 for the second round). We had to limit the historical performance data to the end of April because these were the most updated data that were available when defining the intervention.
In group 2 and 3, the communication will include a note specifying that the registration rate information is exclusively for informational purposes and will not affect any formal evaluation.
CSRs will be assigned to the condition of the office where they are working as of the date of the first intervention. Some CSRs worked at more than one location between November 2016 and the time of the intervention; their personal registration rate that they will see if assigned to Group 2 or 3 is the average over the full six month period from November 1 2016 to April 30 2017 (included) -- we sum all of their registrations and divide them by the total of the customers they served.
To calculate the average and 80th registration rate for a region, however, we will calculate, for CSRs who worked in multiple regions (a minority of the total), separate average registration rate limited to the interactions in a given region. So for example, if in the period Nov 1 2016-Apr 30 2017 a CSR called Y served 200 customers in region A and had 10 registrations, and served 100 customer in region B with 8 registrations (and she is currently working in site X of region B), she will:
1) Be assigned to the treatment condition randomly determined for site X in region B
2) Be informed, if in group 2 or 3, that her registration rate between Nov 1 2016 and Apr 30 2017 was (10+8)/(100+200)=18/300=6%.
In determining the average and 80th signup rate for region A and B to show to CSRs in sites assigned to Group 3 in those region, however, we include a registration rate of 10/200=5% by CSR Y in region A, and a registration rate of 8/100=8% by CSR Y in region B.
Because CSR Y currently worked in an office of region B, if her current office is in Group 3 she will be shown her overall registration rate of 6%, and will be shown the average and 80th registration rate for region B. The reason for these calculation choices is that we want to give a CSR a sense of her own overall performance, but we want to give, as reference in group 3, statistics for the current region, because of potential regional differences. This is also the reason for separating, in calculating overall distributions, the performances of one given CSRs across regions.
Moreover, 3 of the 79 offices were administratively relocated to a different region in early April 2017. Because for the most part of the period for which we collected CSR performance data they were assigned to a region different from the current one, we kept them in the "old" region for the purposes of assigning reference stats (for Group 3) and for the stratified randomization (see below).
The communications just described will occur via email, sent at the dates specified above. An email from the Assistant Deputy Minister-Customer Care at ServiceOntario will inform about further communication to follow about organ donation registrations.
The emails for group 2 and 3 will include a brief explanation that the individual registration rates include the full activity of a CSR between Nov 1 2016 and Apr 30 2017, and will clarify (in Group 3) from what region the distributional info on registration rates comes.
Information on signup rate performance will be in numerical and graphical form.
In Group 2, the graph will be a horizontal bar with the signup rate reported next to it. The horizontal bars will be of the same length for all CSRs, regardless of the actual signup rate of a given CSR. The reason for this choice is to focus on the "absolute" value of the rate rather than the relative one.
In group 3, there will be three bars: the actual performance, and the average and 80th signup rate (or, better, average individual signup rates) for the relevant region. The scale of the bars will be such that the longest of the three bars (i.e., either the 80th percentile bar or the bar representing the individual performance) will be of a defined length, the same for all CSRs, regardless of their actual absolute value. The other two bars will be scaled accordingly. Doing so maximizes the relative difference between bars, especially for low performers.
In order to protect the privacy of the CSRs and to not exert undue influence, we will not inform the CSRs of the intervention under way. The researchers will not have access to the identity of the CSRs or other identifying information. We will elaborate a procedure such that it will be possible to assign CSRs to different treatments using numerical IDs, and to use these IDs to follow CSRs over time.
UPDATES AS OF JAN 22 2018: a few administrative delays led to running the second round of the trial in late January. Moreover, Service Ontario approved running a third round, organized as the other two, in june 2018. We will then extend data collection until the end of 2018.
|
|
Field
Randomization Unit
|
Before
Randomization will be at the office or centre level -- all CSRs in one office will receive the same treatment condition. As specified above, for the first intervention wave on June 19 the office for each CSR will be the one at which they are working at that same.
An agreement with ServiceOntario is that we would send an email also to those CSRs who joined after April 30th 2017, for whom we do not have any performance information. These few CSRs, regardless of the office where they are on June 19 2017, will be assigned to Group 1, in June 2017 and again in November/December 2017.
For CSRs who will join after the first intervention date, we will be able to collect performance data but they would not have been treated on June 19th. They will only be part (if still working at ServiceOntario) of the second intervention wave, and will be assigned to the condition of the office where they will be working at the date of the second intervention. At the office level and with the exception of CSRs who joined between April 30 and June 19th, the experimental condition will be the same in both intervention waves.
|
After
Randomization will be at the office or centre level -- all CSRs in one office will receive the same treatment condition. As specified above, for the first intervention wave on June 19 the office for each CSR will be the one at which they are working at that same.
An agreement with ServiceOntario is that we would send an email also to those CSRs who joined after April 30th 2017, for whom we do not have any performance information. These few CSRs, regardless of the office where they are on June 19 2017, will be assigned to Group 1, in June 2017 and again in November/December 2017.
For CSRs who will join after the first intervention date, we will be able to collect performance data but they would not have been treated on June 19th. They will only be part (if still working at ServiceOntario) of the second intervention wave, and will be assigned to the condition of the office where they will be working at the date of the second intervention. At the office level and with the exception of CSRs who joined between April 30 and June 19th, the experimental condition will be the same in both intervention waves.
UPDATES AS OF JAN 22 2018: a few administrative delays led to running the second round of the trial in late January. Moreover, Service Ontario approved running a third round, organized as the other two, in june 2018. We will then extend data collection until the end of 2018.
|