Senior Seminar Midterm Exam

Poly Science 190
Senior Seminar


Senior Seminar Midterm Exam


1. Read this on-line publication from the Office of the Legislative Analyst (California) and answer the following questions: Welfare-to-Work Participation in the CalWORKs Program www.lao.ca.gov/analysis_2002/2002_pandi/pi_part_5c_calworks_anl02.html

A. Who is the client and the analyst in this example of policy research?

The client in the case is the State of California Legislator. The Analyst in the case is the Legislative Analyst Office.

B. How would you describe the information needs of the client in this situation?

The California Legislator has four specific types of information needed to determine if the CalWorks program is meeting the requirements for welfare reform. The first type of information needed is that the data from the study will give the Legislator a sense of effectiveness in the California counties implementation of the CalWorks program and in enforcing the program. Next, the Legislator must determine if welfare recipients are receiving the employment services and work experience needed through CalWorks before their five-year limit on welfare aid runs out. The Legislator also wants to make sure if the State of California is meeting the Federal participation requirements defined by the Personal Responsibility and Work Opportunity Reconciliation Act of 1996 and that California will receive Federal Funding and avoid any penalties for not meeting any requirements. Finally, the Legislator would need information on how the CalWorks program can be improved upon and what changes are needed to make these improvements.

C. Where in the policy sequence did this research originate? How does this affect the type of research it provides? Discuss.

There are four parts to the policy sequence. First, there must be a planning stage that identifies the issues regarding the policy—what is the problem. Second, there is a decision-making stage that will identify who makes the policy. After the decision is made on a policy, the policy is then implemented. The real work begins in attempts to tackle the problem. The final stage is the evaluation of the policy—did the policy solve the problem and if the policy did not solve the problem, then why and how can the policy be adapted to solve this problem?

The policy sequence for the CalWorks program is situated between the implementation stage and the evaluation stage. The CalWorks program was designed to replace the Aid To Families with Dependent Children (AFDC) entitlement program with a new welfare to work program as a response to the 1996 federal welfare reform legislation. California had a government program set up for welfare. But with new changes being made on the federal level for welfare reform, California had to create a new program to reflect those changes. The CalWorks program was designed as a limited welfare-to-work program. It was an experimental program to adapt and reflect the new changes to the welfare reform. And since this program was experimental, the Legislature would need an evaluation of how the program was performing its intended task of moving people off welfare and into jobs, and what changes can be made to the program in order to improve upon it before the program would be enacted on a statewide level as a replacement to AFDC. The policy sequence for CalWorks is situated between the implementation stage and the evaluation stage.

D. What kind(s) of information needs does it seek to address? Discuss the appropriateness of this information; give the needs of the client.

There are two kinds of information needs for the legislature regarding CalWorks. First, is CalWorks reducing the number of individual caseloads on welfare and placing them in employment? Second, is California meeting the federal guidelines for reducing welfare as prescribed by federal welfare reform using the CalWorks program? This information is especially appropriate since it shows the California Legislature whether or not California is meeting the new federal guidelines for welfare using the CalWorks program. Federal law holds states accountable for moving individuals and families off welfare and into work by forcing states to meet statewide participation rates. If the federal guidelines of these statewide participation rates were not met, the federal government could impose penalties on California by reducing up to 5% of the states block grant on welfare funding.

E. What kinds of research methods does it use to provide this information?

The CalWorks policy report uses mainly descriptive research methods. The program was designed and implemented as a result of welfare reform. The report describes the new federal guidelines, what counts as participation in welfare-to-work, identifies the activities involved for welfare-to-work such as vocational training or community service, and what are the penalties imposed on states for non-compliance. There is also some causal research involved. The causal research shows statistical analysis of showing how CalWorks is reducing welfare rolls for single parent and two parent families and comparing California’s welfare rolls with other states welfare rolls. The causal research shows whether CalWorks is meeting California’s participation rates for the federal welfare reform. Finally, there are some formative evaluation methods, which identify how CalWorks can be improved upon. Formative evaluation methods identify that some participants are disengaged in CalWorks program—neither are they participating in the program, nor are they facing penalties for non-compliance. There are also problems with the county staff not strictly enforcing CalWorks program or the staff are focusing their time on recipients who are not participating in CalWorks.

F. Are the findings helpful to the client, in light of you answer to (b.)?

The findings in this policy report are mixed for the client. While the findings do show that California will meet the federal participation requirements for welfare-to-work using the CalWorks program, there are still major problems of the CalWorks program that must be addressed. The CalWorks program stressed a goal of universal participation among able-bodied adults. However, participation status for both single parent and two-parent families combined together show that 33% of families are meeting work participation requirements, while 24 % are participating but for insufficient hours and 43% are not participating at all in CalWorks. The study also found that a significant number of recipients are disengaged from the CalWorks program—they are neither participating in the CalWorks program, nor are they facing penalties for non-compliance. This problem could be explained where CalWorks recipients have very little or no contact with officials in the county welfare department. Finally, the study also identified problems with the various counties implementation of CalWorks program. Many counties did not strictly enforce the CalWorks hourly participation rates for recipients. Interviews with the county staff indicated that the staff concentrated their time and resources on individuals who were not participating in CalWorks program, or they were concentrating on getting individuals to do some participation in CalWorks, but not the strict hourly participation rates that CalWorks required. While the CalWorks program was meeting the federal guidelines for welfare-to-work policy, the program still has need for improvement in order to achieve the policy goal of taking people off the welfare rolls and placing them into work.


2. Contemporary policy research may be said to emphasize a summative or a formative approach. Using the case studies in Part II of the Haas/Springer text, select two case studies, which best exemplify each approach and explain how AND why they do so.

The summative approach to policy research can be specified as an outcome oriented approach. Summative evaluations require specific outcomes in ascertaining if a particular policy can address an issue. The evaluation uses quantitative methodology with a rigorous scientific analysis of statistical data.

The policy evaluation of the California Ignition Interlock Experiment is an example of a summative evaluation. This policy evaluation was a quasi-experiment designed to test a high tech device that, when installed in a vehicle of a convicted drunk driver, would prevent the driver from operating the vehicle after consuming alcohol. This experimental pilot program was authorized by law enforcement after California passed the Farr-Davis Act of 1986.

The experiment was fairly simple. Four California counties were selected to participate in this program, which was set up by the California Office of Traffic Safety (OTS). Judges in these county courts would select convicted offenders who would participate in the interlock program and those offenders who would be in the control group. The crux of the experiment would be to compare the re-conviction rates of the interlock group with those of the matched control group. If there were a reduction of DUI re-conviction rates by more than 10%, then the program would be considered a success.

The interlock device produced some “positive” results, however, the data did not support any definite conclusions because of the limitations of the research model of the quasi-experiment. With a summative analysis, the requirements are outcome oriented and judgmental with solid statistical analysis as a basis of evidence. However, the interlock program used a quasi-experiment, which would attempt to emulate the conditions of a classical scientific experiment, but would not be able to provide definite information or exacting statistical data as required in a classical experiment. The results of the experiment indicated that the interlock program measured a “success” rate of more than 10 % reduction in repeated arrests of individuals for DUI offenses who were involved in the interlock program. However, the program had major problems in its implementation that would affect the measurement of its success rate. While there was a 10% measured reduction in repeated DUI arrests of individuals involved in the interlock program, the samples of the experimental offenders and the samples of the control group were not statistically significant since they did not represent a random sample of the California population. A second problem was with the sentencing of individuals to the interlock program. The county judges who were involved in the interlock program had total discretion in selecting DUI cases for the interlock program. This discretion causes a bias within the experiment. Three possible reasons for judicial discretion were identified. First, only a small number of judges used the program, which could mean there was a judicial resistance to the program. The study also found that some judges were reluctant to use the interlock program since they perceived the program as an option for the wealthy DUI offenders who were able to pay for the installation of the devices. Finally, judges who did consistently used the interlock program, did so on a variety of offenders for different reasons. There was no strict sentencing requirement for the interlock program, thus the experimental group and control group did not have common characteristics of which to achieve an accurate measurement of success for the interlocking program.

The study also found various technical problems with the interlocking devices itself. The effectiveness of the device was based on a number of individuals who had the device installed. However, one fourth of the individuals sentenced to the interlock program never had the device installed. The primary reason for this discrepancy was the lack of compliance with court orders. In addition, the report found that the interlocking devices could easily be bypassed or circumvented.

The California Ignition Interlock Pilot Program is a clear example of using a summative evaluation approach to examine a unique policy experiment in order to reduce the number of drunk drivers in the state of California. While the program may not have been a major success, the program clearly shows that policy makers could try new and unique ideas in order to solve policy issues—in a sense, policy makers could think outside of a box. The ignition interlock technology was on the market. Could such technology help reduce the number of drunk drivers on the road? The ignition interlock program shows that there could be some success, however there are still major improvements needed in the technology and in the way this technology could be implemented.

A second case study, which exemplified a summative evaluation, was the Housing Sales in Urban Neighborhoods in St. Louis. The city of St. Louis lacked residents in the downtown areas and because of this residential decline in the downtown area; there was a decline in commercial revitalization. St. Louis decided to embark on a program of gentrification. Gentrification was a program of physical renovation and social class upgrading of inner-city neighborhoods. St. Louis set up a commercial development-led policy of gentrification with community programs to clean up streets, beautify areas and provide neighborhood security programs. In the end, gentrification did not work out as well as the city had planned. Therefore, a policy study was designed by the Community Development Agency (CDA). The CDA designed a summative evaluation to determine the socio-economic and demographic characteristics of individuals buying homes in the city neighborhoods. This evaluation model used telephone surveys of recent home buyers, and interviews of focus groups such as real estate agents, developers, appraisers, and brokers.

What the CDA found was that the St. Louis gentrification strategy was focused on the goals of attracting higher-income buyers who may have previously owned homes and have families of children into these downtown neighborhoods. However, the conclusions generated by the research data indicated that a majority of the home buyers in the downtown areas were previously renting. This showed that the influx of individual residents moving to downtown were younger, more educated, and possibly single individuals who were purchasing their own home. These individuals had not yet started a family. The main reason for this influx of younger first-time home buyers was the affordability of housing in the area. What is interesting about this summative evaluation is that the data and its conclusions indicated the policy results were completely opposite of the policy decisions the city of St. Louis made in its gentrification program.


The second type of case study is the formative evaluation. Formative evaluation focuses on providing information to policy analysts on how to improve an existing program. Formative evaluation does not judge the outcome of a program or determine if expectations were met as seen in summative analysis. Instead, decision makers use the formative evaluation to ask how a program can be improved upon.

The Extended National Youth Sports Program was designed as a risk protection program to protect disadvantaged youths from illegal drug use. A policy study on ENYSP was developed to determine whether ENYSP was achieving its goals of drug protection. However, there are two problems with drug prevention programs. The first problem is that drug prevention programs may not have any clear objective policy goal, which the program can be measured to determine if it is successful. ENYSP did not have any clear goals as to what it was trying to achieve. The second problem is that drug prevention program resources can only be used in present here and now, but the idealized goals of drug prevention is to keep participants away from illegal drug use over a long term basis. This is a serious conflict of interest between the lack of clear goals of drug prevention and the resources these programs can use for prevention. Formative evaluation can focus on a program’s effectiveness and determine how to improve the program. With ENYSP, formative evaluation showed that the program’s impact was holistic—just about any type of ENYSP activity could be viewed as a drug prevention activity. The important crux of ENYSP was the interaction of the youths and the ENYSP adults. While formative evaluation cannot provide answers to long-term goals, the model can show how the program’s resources are used in the present time and suggest improvements to better utilize those resources. ENYSP had several problems. Research showed that ENYSP was plagued by a lack of consistent attendance of youths in the program and an inexperienced ENYSP staff—many of which were not qualified as drug education specialists. Finally formative evaluation provided method of data collecting, which were not heavily dependent on statistical analysis, but provided a wealth of information. Important data was collected by interviewing key ENYSP staff, selected studies of ENYSP sites and field observations of ENYSP activities.

Another policy study using the formative approach in policy research is the De-Institutionalization and Community Services in Virginia. De-institutionalization is a policy where mental health patients could be treated more effectively in an environment of a normal home life and community ties. De-institutionalization started in Virginia in 1968. By the 1980s, the program was criticized for dumping mentally ill patients on the street. The Virginia state legislature authorized the Joint Legislative Audit and Review Commission (JLARC) to create a policy study on Virginia’s de-institutionalization. This policy study was unique in that JLARC used a combination of summative and formative evaluation whenever each type of evaluation model was needed. For the formative evaluation model of the study, JLARC conducted a wide range of interviews with mental health officials involved in providing services to patients. JLARC also conducted on-site visits to state and local institutions and adult homes, which provided care facilities for patients. These interviews and field visits provided JLARC staff with crucial information as to the workings of de-institutionalization and its structured problems with this policy. What is more important, the interviews and field observations provided information as to why these problems existed with de-institutionalization. The information of why these problems existed could not have been collected by using statistical analysis or a summative approach. Formative evaluation was only one part of a combined research strategy using formative and summative evaluation to provide a complete policy evaluation of the de-institutionalization process in Virginia.


3. Policy research is a meaningless exercise if it doesn't impact actually policy decisions. Using the case studies in Part II of the Haas/Springer text, select the case study that apparently had the *most* impact and the one which apparently had the *least* impact. Defend your answer, and offer an explanation as to why this research was effective (and ineffective).

The case study that had the most impact on affecting policy decisions would be the De Institutionalization and Community Service in Virginia report. De institutionalization became popular in the United States when mental health officials concluded that mental health care could occur more effectively and efficiently within the patient’s own home and community ties. A statewide policy of de institutionalization occurred in Virginia in 1968. However, by the 1980s, de-institutionalization became controversial when critics complained that the policy consisted of dumping mentally ill patients on the city streets.

Because of this criticism, the Virginia State Legislature authorized a policy study of de-institutionalization within the state. This study would be conducted by the Joint Legislative Audit and Review Commission (JLAC), a tough but objective watchdog agency. JLAC had an incredible task—to provide the Virginia state legislature with information that can be used to improve the state’s mental health care system in a short period of time. The problem was vast, complex and especially vague. The information needs of the client—the Virginia state legislature—were murky and lacked clear goals. However, this incredibly murky realm of de-institutionalization provided JLAC with a clean slate to conduct an exploratory research model of the Virginia’s mental health care system. JLAC had the discretion of what to do for researching de-institutionalization and how to do it. JLAC used both summative and formative research models whenever each model was more appropriate. Statistical analysis of various state and county mental health agencies regarding agency records and fiscal data was used showing summative evaluation. At the same time, JLAC staff visited mental health care agencies for field observations and to conduct interviews with mental health care officials. This visits and interviews showed a more formative approach to collecting data.

The results of the policy study showed that Virginia’s de-institutionalization policy contained many problems. Most of the clients who were discharged were young, single, unemployed and required medical attention and supervision. The descriptive profile of these discharged clients were completely opposite of what de-institutionalization was suppose to achieve. In addition, community services for discharged patients were inadequate. There was also a lack of housing for discharged patients with the responsibility for providing such housing spread among different state and local agencies. Finally, many of the de-institutionalized clients did not have access to daily medication needed for treating their illness.

The JLAC study and its findings were released at a public hearing covered by the news media. This was especially important factor since the programs findings were given a high public profile. Because of this public profile, the Virginia state legislature could not simply bury this policy research on the back shelf. Instead, the state had to approve many of the improvements mentioned in the study for de-institutionalization.

A case study that had the least impact on affecting policy decisions would be the California Ignition Interlock Program. This program was an experiment to test the effectiveness of a high tech device, which would prevent the vehicles operation by an individual who had consumed alcohol. The device was installed in vehicles of convicted drunk drivers. The main problem with this program was that the research model used for the program was a quasi-experiment, which attempted to emulate a classical experiment without the exacting scientific rigors of data collecting. At the same time, the program used a summative evaluation model, which required clear, definite expectations to be met in the policy goals, to be backed by quantified statistical data. The results of this conflict between the quasi-experiment and summative evaluation was that experiment of the interlock ignition device provided some measured success in preventing DUI convictions of those individuals in the program, however, the data did not support any definite conclusions to this success. There were also other problems with the implementation of this program. Judicial resistance to the interlock program was one problem. There were also technical problems with the interlock ignition devices where these devices could be easily bypassed or circumvented. Finally, one quarter of the individuals who were sentenced to using the interlock devices never had the devices installed on their vehicles. Wile this program did not have a major impact on policy decisions, the program did allow decision makers the ability to test new ideas on how to change the policy problems and to see if these new ideas could be used to impact a policy decision. The California Ignition Interlock Program was just one unique idea to examine.