The State of Practice in Canada (2005) in Measuring the Impact of Career Services > ICCDPP

Current Articles | Categories | Search | Syndication

Thursday, December 28, 2006

The State of Practice in Canada (2005) in Measuring the Impact of Career Services
By jmcadmin @ 2:19 PM :: 4530 Views :: 0 Comments :: Training and Qualifications, Funding Career Guidance, Assessing Effectiveness

CANADIAN RESEARCH WORKING GROUP FOR EVIDENCE-BASED PRACTICE IN CAREER DEVELOPMENT (CRWG)

The State of Practice in Canada in Measuring Career Service Impact: A CRWG Report:

Principal Researchers and Authors

Kris Magnusson University of Lethbridge, Alberta, Vivian Lalande, University of Calgary, Alberta, December 2005 Published by The Canadian Career Development Foundation © Canadian Career Development Foundation, 2005

Canadian Career Development Foundation

202 – 119 Ross Avenue

Ottawa, Ontario K1Y 0N6 Canada

Tel: (613) 729-6164

Fax: (613) 729-3515

information@ccdf.ca

www.ccdf.ca

MEMBERS OF THE CANADIAN RESEARCH WORKING GROUP FOR EVIDENCE-BASED PRACTICE IN CAREER DEVELOPMENT (CRWG)

􀁸 Robert Baudouin, Moncton University, New Brunswick

􀁸 Lynne Bezanson, Canadian Career Development Foundation

􀁸 William A. Borgen, University of British Columbia

􀁸 Liette Goyer, Laval University, Québec

􀁸 Bryan Hiebert, University of Calgary, Alberta

􀁸 Vivian Lalande, University of Calgary, Alberta

􀁸 Kris Magnusson, University of Lethbridge, Alberta

􀁸 Guylaine Michaud, Sherbrooke University, Québec

􀁸 Céline Renald, Canadian Career Development Foundation

􀁸 Michel Turcotte, Human Resources and Skills Development Canada

Student Research Assistants

􀁸 Allison Roest, University of Lethbridge

􀁸 Mark Slomp, University of Lethbridge

􀁸 Mike Yudcovitch, University of Calgary

iii

ACKNOWLEDGEMENTS

The Canadian Research Working Group for Evidence-Based Practice in Career Development (CRWG) gratefully acknowledges the support of Human Resources and Skill Development Canada.

TABLE OF CONTENTS

MEMBERS OF THE CANADIAN RESEARCH WORKING GROUP FOR EVIDENCE-BASED PRACTICE IN CAREER DEVELOPMENT (CRWG).........................................III

ACKNOWLEDGEMENTS......................................................................................................IV

TABLE OF CONTENTS.........................................................................................................V

LIST OF TABLES..................................................................................................................IX

CHAPTER 1............................................................................................................................1

INTRODUCTION: MAKING THE CASE FOR CAREER DEVELOPMENT IN CANADA......1

Background.....................................................................................................................1

Canadian Research Working Group for Evidence-Based Practice in Career Development (CRWG)......................................................................................................2

References......................................................................................................................4

CHAPTER 2............................................................................................................................5

THE EFFICACY OF CAREER DEVELOPMENT INTERVENTIONS: A SYNTHESIS OF RESEARCH......................................................................................................................5

Background.....................................................................................................................5

Target Audience................................................................................................................6

Populations and Samples.................................................................................................6

Research Methods............................................................................................................7

General Findings...............................................................................................................8

Diverging Theoretical Assumptions...................................................................................9

Conclusions....................................................................................................................10

References.....................................................................................................................12

CHAPTER 3..........................................................................................................................13

STATE OF PRACTICE IMPACT SURVEY: METHODS......................................................13

Introduction...................................................................................................................13

Data Collection................................................................................................................13

Surveys...............................................................................................................13

Telephone Interviews..........................................................................................15

Data Analysis..................................................................................................................17

Survey Data.........................................................................................................17

Telephone Interview Data...................................................................................17

Focus Group........................................................................................................18

CHAPTER 4..........................................................................................................................19

RESULTS...........................................................................................................................19

Demographic Data..........................................................................................................19

Respondents.......................................................................................................19

Comparison of Anglophone and Francophone Respondents..............................19

Kinds of Career Services Provided......................................................................20

Demographics of Canadian Career Services Agency Respondents...................24

Demographics of Clients Served by Practitioner Respondents...........................24

Importance and Practice of Outcome Measurement.......................................................28

Measures of Association.................................................................................................29

Type of Agency and Perceived Importance of Outcome Measurement..............29

Type of Agency and Actual Measurement of Outcomes/Impact.........................31

Size of Service Provider, Perceived Importance and Actual Measurement of Impact..................................................................................................................34

Qualitative Analysis.........................................................................................................35

Most Important Outcomes that Agencies and Practitioners Report.....................36

Outcomes that Go Unreported............................................................................37

Focus Group Data...............................................................................................40

Policy Makers’ and Employers’ Data...............................................................................42

Policy Makers......................................................................................................43

Employers...........................................................................................................44

Summary........................................................................................................................44

CHAPTER 5..........................................................................................................................45

IMPLICATIONS AND DISCUSSION....................................................................................45

Generalizing the Findings...............................................................................................45

The Nature of Impact Assessment..................................................................................46

Improving Outcome Measurement..................................................................................47

Increased Level of Sophistication...................................................................................47

Increased System Support..............................................................................................48

Comments on Statistical Trends.....................................................................................49

Recommendations and Next Steps.................................................................................50

Summary........................................................................................................................52

APPENDIX A. LETTER OF INVITATION TO PARTICIPANTS...........................................53

APPENDIX B. ENGLISH AGENCY QUESTIONNAIRE.......................................................57

APPENDIX C – STATE OF PRACTICE...............................................................................65

APPENDIX D. LETTER OF INVITATION TO POLICY MAKERS........................................75

APPENDIX E. LETTER OF INVITATION FOR EMPLOYERS.............................................77

v i

APPENDIX F. PHONE SURVEY FOR POLICY MAKERS..................................................79

APPENDIX G. PHONE SURVEY FOR EMPLOYERS.........................................................81

vii

LIST OF TABLES

Table 1. Relationship Between Language of Practitioner Respondents and Perceived Value of Measuring Outcomes/Impact of Career Services............................................................20

Table 2. Relationship Between Language of Practitioner Respondents and Actual Measurement of Outcomes/Impact of Career Services.........................................................................20

Table 3. Kind and Frequency of Services Provided by Agencies...............................................21

Table 4. Kind and Frequency of Services Provided by Anglophone Practitioners......................21

Table 5. Kind and Frequency of Services Provided by Francophone Practitioners....................23

Table 6. Comparison of Ranking of Service Provision: Agencies, Anglophone Practitioners, and Francophone Practitioners..............................................................................................23

Table 7. Age Groups Served by Agencies..................................................................................24

Table 8. Age Groups Served by Practitioners.............................................................................25

Table 9. Gender Focus of Practitioner Services.........................................................................25

Table 10. Cultural Focus of Practitioner Services.......................................................................26

Table 11. Employment Status Focus of Practitioner Services....................................................26

Table 12. Employment Setting of Practitioners: Kind of Agency.................................................27

Table 13. Employment Setting of Practitioners: Size of Agency.................................................27

Table 14. Perceived Importance of Outcomes/Impact Measurement.........................................28

Table 15. Actual Practice of Outcomes/Impact Measurement....................................................28

Table 16. Relationship Between Type of Agency/Service Provider and Perceived Value of Measuring Outcomes/Impact of Career Services: Agency Results.................................29

Table 17. Relationship Between Type of Agency/Service Provider and Perceived Value of Measuring Outcomes/Impact of Career Services: Anglophone Practitioner Results......30

Table 18. Relationship Between Type of Agency/Organization/Service Provider and Degree to Which They Evaluate Outcomes/Impact of Services: Agency Results...........................32

Table 19. Relationship Between Type of Agency/Organization/Service Provider and Degree to Which They Evaluate Outcomes/Impact of Services: Anglophone Practitioner Results.33

Table 20. Relationship Between Size of Agency/Service Provider and Perceived Value of Measuring Outcomes/Impact of Services: Agency Results.............................................34

Table 21. Relationship Between Size of Agency/Service Provider and Degree to Which They Evaluate Outcomes/Impact of Services: Agency Results...............................................35

CHAPTER 1

INTRODUCTION: MAKING THE CASE FOR CAREER DEVELOPMENT IN CANADA

BACKGROUND

The field of career development has a rich tradition of theory and research that dates back for almost a century. Today, career development interventions and services span educational, community and workplace settings across the globe. The primary goal of these services is to help individuals connect to, choose and manage education, training and work options which are congruent with their interests, passions, talents and sense of purpose. There is also recognition that, when these individual goals are realized, career development services can have an impact on broader social and economic goals, such as raising community quality of life, increasing opportunities, lowering social barriers to inclusion and economic independence, and generally maintaining a vibrant social and economic order.

Many specific claims can be made about the contributions that quality career development services can make. Perhaps the strongest is that individual quality of life improves, as can be reflected in a number of ways. When individual talents, interests and passions are harnessed, people tend to become more motivated learners. They are more likely to be productive employees when their sense of purpose aligns with that of the employer. There is a reduction in social exclusion, as people become empowered to explore and act upon the things that are meaningful to them. A cumulative impact of these outcomes is a more vibrant, equitable workplace where workers enjoy improved levels of mental health. And when career development services are integrated across social and economic strata, a strong foundation is built for cost-effective skills development and learning systems.

There are many indirect claims pertaining to the impact of effective career development services, particularly related to the direct cost benefits of career development systems of interventions. Articulating purpose with passion results in fewer “trial and error” learners. Students in educational settings have increased program completion rates and fewer within institution program transfers. People stay involved in the workforce longer, resulting in lowered social assistance and unemployment benefit payments. Improved mental health in the workplace reduces levels of stress, thus reducing health care costs. People with identified career paths are less likely to become marginalized in society and more likely to make positive community contributions. In the workplace, employers report that employees perform to a higher standard, there is less staff turnover, fewer people take sick leave, and more participate in professional development and/or learning activities.

Virtually anyone involved with the field of career development can attest, through direct experience and observation, to the veracity of these claims. There is only one small problem: we cannot really prove many, if any, of the above claims! In fact, we still possess very little concrete evidence that career interventions actually “work,” and the evidence that we do have tends to reflect very specific interventions for specific populations (see Chapter 2).

Why does the lack of direct evidence matter? Perhaps the greatest reason is that funding for service delivery is increasingly linked to demonstration of results. Simply put, if services cannot clearly demonstrate the impact they are having, they will be less likely to receive funding. Furthermore, one criterion of a profession is the evidence base for its practice; if we want to

increase our credibility, we will need to be able to document the impact we have. Increased professionalism is also related to public accountability; the people who use our services deserve some level of quality assurance. It will also be important to have a strong evidence base if we hope to influence public or organizational policy in desired directions. Finally, practitioners need evidence of the impact of their interventions for their own professional integrity and accountability.

Saying that evidence is needed is one thing; gathering that evidence is quite another. The difficulties associated with documenting the impact of career interventions may be attributable, in part, to the growing recognition of the complexity of career planning and the related challenges of efficacy research: (a) the range of factors influencing individual choice, (b) the wide variance in client groups, issues and concerns that makes comparison of evidence difficult, and (c) the lack of common outcome measures in the field of career development (Hughes, 2004). It is clear that a systematic framework for creating, collecting and evaluating career services efficacy is needed, one that captures the complexity of career practices and outcomes.

In response to this need, and directly following from the recommendations of Working Connections: A Pan-Canadian Symposium on Career Development, Lifelong Learning and Workforce Development (held in Toronto in November of 2003), a forum was convened to discuss the possibility of creating a comprehensive framework for the effective evaluation of career development services that could serve as a guide for career development service providers, practitioners and policy developers in Canada. At the initial forum, which was held in Ottawa in March, 2004, researchers from across Canada were invited to share their perspectives on the state of career efficacy research, and to discuss what steps can be taken to provide evidence for the “what, how, why, and for whom” of career development services in Canada. The Canadian Research Working Group for Evidence-Based Practice in Career Development (CRWG) was created from this forum.

CANADIAN RESEARCH WORKING GROUP FOR EVIDENCE-BASED PRACTICE IN CAREER DEVELOPMENT (CRWG)

In response to a recommendation from the pan-Canadian Symposium, the CRWG launched, in the fall of 2004, a preliminary study to learn more about how Canadian providers of career services evaluate the impact of their services. The main goal of the research was to explore the current state of practice in Canada in order to create shared understandings of evaluation procedures currently being used as well as the desired outcomes for career development practices. Specifically, the CRWG set out to learn about:

􀁸 the service outcomes actually gathered and reported by front-line career practitioners;

􀁸 the service outcomes gathered and reported at the office, agency or school board levels;

􀁸 the service outcomes which are being achieved by front-line and office, agency, and/or school board levels but which are not reported;

􀁸 how policy makers (who fund services) evaluate the services, what they want back from the services, and the kinds of evaluation information and data they prefer to have about the services; and

􀁸 how employers evaluate career development services in the workplace, what outcomes they want from these services, whether the evaluation information is useful to them, and the kinds of evaluation information they would prefer to receive.

The products of this research are presented in this report, which contains five major sections. Chapter 2 provides a brief synthesis of recent research on the efficacy of career development interventions. In Chapter 3, the methods employed for collecting and analyzing the data are described. Chapter 4 provides a summary of the key findings for each of the research target groups. Conclusions, recommendations and an overview of the proposed research agenda to follow from this work are included in Chapter 5.

The research described in this report is the first concrete step in a much more ambitious research and development agenda in Canada. With the preliminary data in hand, the CRWG will be submitting a large-scale application for extended funding to continue the important work of creating an effective and accurate system for measuring the impact of career services. A complete listing of CRWG members, with contact details, is included in Appendix A with the letter of invitation to participants.

REFERENCES

Herr, E. L. (2003). The future of career counseling as an instrument of public policy. The Career Development Quarterly, 52(1), 8-17.

Hiebert, B. (1994). A framework for quality control, accountability and evaluation: Being clear about the legitimate outcomes of career counselling. Canadian Journal of Counselling, 28(4), 334-345.

Hughes, D. (2004. Creating evidence: Building the case for career development. The Career Counsellor, 16, 2,7.

Roest, A., & Magnusson, K. (2004, February). Annotated bibliography of current research on the efficacy of career development interventions and programs. Paper presented at the meeting of the Canadian Research Working Group for Evidence-Based Practice in Career Development, Ottawa, ON.

Watts, A. G. (2000). Career development and public policy. Journal of Employment Counseling, 37, 62-75.

Watts, A. G. (2003). Career development and public policy. Canadian Career Development Foundation. Retrieved March 2, 2004, from http:www.ccdf.ca/pdf/chapter21/pdf

CHAPTER 2

THE EFFICACY OF CAREER DEVELOPMENT INTERVENTIONS: A SYNTHESIS OF RESEARCH

BACKGROUND

Two recent symposia have highlighted the need for public policy to be guided by evidence pertaining to the efficacy of career development practice. The first was an international symposium entitled Career Guidance and Public Policy: Bridging the Gap. It was held in Toronto in October 2003, with 28 countries represented. The second, held in Toronto in November 2003, was Working Connections: A Pan-Canadian Symposium on Career Development, Lifelong Learning and Workforce Development with all provinces and territories represented. A consistent theme that emerged from both Symposia was the need to develop effective systems for gathering data concerning the impact of career development/career guidance services on a number of levels, such as individual well-being, social inclusion, and economic development. Furthermore, participants in both symposia discussed the need for data to inform and influence public policy related to the provision of career services.

The participants at these Symposia echoed a growing call among researchers for more comprehensive efficacy assessment of career practices. Herr (2003) suggests the development of cost-benefit analyses to document the results of career services, and the creation of national research databases to collect and distribute such information. Watts (2002, 2004) urges efficacy research to link career practices to economic efficiency, social equity and sustainability. In Canada, Hiebert (1994) has called for increased and more precise efficacy assessment in career counselling. Currently, several Canadian researchers, including Bryan Hiebert and Vivian Lalande at the University of Calgary and Bill Borgen and Norm Amundson at the University of British Columbia, have been working on the problem of accountability and efficacy measurement in career services.

Despite increased awareness of the need for better understanding of how and why career services are effective, the number of outcome research studies has actually decreased in the last 20 years (Whiston, Brecheisen & Stephens, 2003). The decline may be attributable, in part, to the growing recognition of the complexity of career planning. Hughes (2004), for example, commenting on the difficulties associated with assessing the impact of career interventions, notes three major challenges to efficacy research: the range of factors influencing individual choice; the wide variance in client groups, issues and concerns that makes comparison of evidence difficult; and the lack of common outcome measures in the field of career development.

It is clear that a framework is needed for creating, collecting and evaluating the efficacy of career development services. An initial step in that process was taken with the compilation of the Annotated Bibliography of Current Research on the Efficacy of Career Development Interventions and Programs (Roest & Magnusson, 2004). The annotated bibliography focused primarily on articles that examined the efficacy of career development services and interventions and had been published in English-language career journals within the past 10 years. A parallel initiative, led by Michel Turcotte, examined articles published in French-language journals. Time constraints did not permit a comprehensive review; however, the articles included provide a representative sampling of research in the field. The central themes and observations derived from the review of 53 English-language articles are presented here, in

the following categories: target audience, populations and samples, research methods, general findings about efficacy, and diverging theoretical assumptions.

TARGET AUDIENCE

Most of the articles reviewed addressed an academic or research audience and, to a lesser extent, practitioners. Since the review focused on academic journals, this is hardly surprising. However, given the need for evidence to inform practice, it does pose a few problems. For the most part, descriptions and results are not presented in a manner that would be accessible to many practitioners. Thus, even when specific positive results are found, they may not find their way into general practice. As a result, efficacy research efforts may be replicated without any systematic building upon known data. The problem with this is that, each time an evaluation needs to be conducted, agencies must create their own processes, and do not benefit from the experience, knowledge and recommendations of earlier work. It is difficult to build systematically upon, and thereby advance, our knowledge of the impact of career services when such a piece meal approach is taken.

The academic nature of the articles reviewed poses a secondary problem for practitioners. Even when positive treatment effects are found, very little description of the nature of the program, service or intervention is provided. Practitioners are thus left on their own to locate more detailed descriptions of exactly what it was that proved to be effective. Furthermore, most of the reports focus on holistic program or intervention effects; there is very little analysis of the impact or efficacy of specific treatment or program components.

It is not only practitioners who do not fully benefit from existing efficacy research. Published academic research studies rarely refer to the implications of the research for public policy. This is somewhat surprising because, as Herr (2003) noted, “Career counseling, in its many manifestations, is largely a creature of public policy” (p. 8). Taking Herr’s comments a step further, it is also the case that the public institutions that develop policy (e.g., governments) are the largest source of funding for career services. It would be prudent for researchers to be sensitive to the implications of their work for the creation and shaping of public policy, because ultimately, that is the source of their funding. However, this does not seem to be the case. Although Herr cautions against too close a linkage between career services and public policy, little focused research is available that would support or better inform policy.

POPULATIONS AND SAMPLES

The primary participants in career efficacy research have been students of educational institutions. To some extent, the articles reviewed in the Annotated Bibliography illustrate the principle of “convenience sampling.” Of the 41 specific research studies reviewed, 34 described intervention effects on students, mostly Caucasian, in educational settings. Of these, 20 studies were conducted with university or college students, 9 with high school students, and 5 with middle school students. This pattern is common in psychological research in general; most studies are done where access to participants is convenient. Although in one sense this a reasonable and understandable approach, it still leaves large gaps in our knowledge about the differential effects that career services may have on other groups, such as women, members of varying ethnic or cultural groups, or people from differing educational or socio-economic backgrounds. Clearly the focus of research needs to be expanded to include a much broader spectrum.

RESEARCH METHODS

The majority of the studies reviewed used quantitative methodology, and some used mixed method designs (i.e., quantitative analysis supplemented by qualitative analysis). The most commonly employed research designs were variations on pretest/post-test, treatment group to control group experimental designs. In some cases, treatment group/control group post-test only designs were employed. Depending on the sophistication of the study, one or more predictor variables were related to one or two criteria variables. In general, the studies attempted to isolate specific treatment effects (e.g., computer assisted guidance systems) on specific outcome measures (e.g., occupational decision-making).

A major concern with interpretation of the efficacy data is the imprecision of the outcome measures. Often instruments with questionable standards of reliability and validity serve as the specific outcome measure. For example, studies of youth often employ measures of career maturity, despite the difficulties associated with measuring the career maturity construct. Possibly even stronger efficacy results would be obtained with more accurate outcome measures.

A second concern pertains to assumptions about the outcome measures. Often specific outcomes are used on the assumption that they are linked to positive career planning. For example, increases in occupational exploration behaviours are commonly used as outcome measures, even though there is little evidence to support the assumption that such increases are related to making sound occupational decisions. An equally plausible hypothesis could be that increasing engagement in meaningful activities, regardless of occupational context, will lead to the discovery of satisfying opportunities. Most of the efficacy research published seems to be rooted in what Weinrach (1979) calls the “structural approach” to career development. However, the underlying assumptions governing the selection and subsequent measurement of appropriate outcomes are rarely made explicit in these studies.

Methods of establishing experimental conditions and of measuring aggregate outcomes are problematic for career efficacy research. Very little attention has been paid to the differential effects that interventions may have on sub-groups within the sample or on diverse samples. Furthermore, few studies compare interventions and their treatment effects. One of the most commonly reported types of study assesses a specific intervention or treatment (“the effects of treatment program A on outcome measure X”). Such studies usually reveal positive, but modest, support for the intervention; however, few studies compare the efficacy of interventions with similar goals (“Is treatment program A any more effective than treatment program B?”). Some studies attempt to assess the effects of computerized systems of guidance, frequently comparing the impact of these types of programs to individual counselling and/or to combine counselling and computerized interventions. More comparisons of this kind are needed. Furthermore, as Brown and Ryan Krane (2000) note, more attention needs to be paid to the combined effect of interventions.

Methods of data aggregation are problematic for career efficacy research, particularly in analysis of the efficacy of programs of intervention. While many program evaluation studies provide multiple outcome measures, very few analyze the differential impact of specific program components. The focus on global outcome measures does not help us understand what components, and in what combination, contributed to the outcome. Furthermore, unless process variables are specifically addressed, there is no way of knowing if poor results are related to actual program content or simply to the lack of adherence to program design. Although Hiebert (1994) called for both process and outcome assessment components in program evaluations

over a decade ago, few such comprehensive evaluations are making their way into academic publications.

There have been a few attempts to conduct meta-analyses of career efficacy research (e.g., Sexton, 1996; Whiston, Brecheisen & Stephens, 2003; Whiston, Sexton & Lasoff, 1998). Most of these attempts were hampered by questionable research methodology, insufficient information, or lack of consistency in the reporting of the data in the original studies. Furthermore, there is very little consistency in the choice of outcome measures, even when measuring identical constructs. Therefore, it is very difficult to draw conclusions pertaining to career intervention efficacy across studies. Despite these problems, most of the authors of the meta-analyses and literature reviews agreed that career development interventions are indeed effective. The problem is that little is known about why, how, or for whom they work. Overall, research in career intervention efficacy is piecemeal, fragmented, and unsystematic.

GENERAL FINDINGS

Given the limitations of audience, population samples and research methodology discussed above, one might wonder what, if anything, we can conclude about career efficacy research. Despite these limitations, a few trends did emerge among the studies reported. The most common finding in the efficacy research was that career interventions or programs had a positive effect on participant satisfaction. For example, even in studies that demonstrated no specific treatment effects, the authors would report that clients were satisfied with the processes or interventions, or that they “reacted positively” to the different treatments. It can be concluded that participants generally express satisfaction with career interventions.

Much of the evidence for the efficacy of career interventions pertains to changes in client competence (37 of 41 studies) or client behaviour (8 of 41 studies). Even though a broad spectrum of interventions is represented in the studies reviewed, career interventions in general have been shown to have significant effect in two main areas. First, career interventions increase client exploratory behaviours. Participants are more likely to engage in activities that broaden their range of information and knowledge of career options after engaging in some form of career intervention. Second, participants in the studies reviewed are more likely to make career decisions after engaging in a career intervention. Unfortunately, there is little evidence to suggest whether the various interventions have differential effects; we do not know if one form of intervention is more effective than another in producing these effects.

Very little attention has been paid to aspects of career planning or career development processes other than exploration and decision-making behaviours. Examples of gaps include the role that engagement plays in career planning (e.g., the use of personal meaning in career planning, or the identification of sources of personal meaning as a motivator/guide for career exploration), the development of prerequisite and planning skills needed to actualize a decision, and the development of systems of social support and/or feedback when implementing career decisions. Overall, the research may be characterized by a central assumption that career planning is largely a cognitive process, and that once a decision is made, it can and will be implemented.

The review of the literature reveals another problem with the assessment of career intervention efficacy; scant attention has been paid to broader outcomes of career interventions. There are little follow-up data to indicate whether clients who use career services attain greater levels of later job satisfaction, work performance or life satisfaction, compared with those who do not

access the services; more longitudinal studies are needed. Given that most agencies and services find themselves in an era of fiscal restraint, research into global outcomes is essential for sustaining existing programs and for providing evidence of the need to develop new ones.

Finally, the global impact of career interventions remains virtually unknown. For example, it is very difficult to determine the economic benefits of career interventions. As Hughes (2004) reports, “Research findings that measure the economic benefits of guidance are problematic, mainly because guidance effectiveness research in the United Kingdom is usually short-term and focused on immediate results” (p. 2). The same observation could be applied to studies conducted in North America, where even less is known about the social impact of career interventions. While it may be reasonable to speculate that good occupational decisions would lead to stronger, more stable families, increased connection with community and decreased isolation or alienation, no studies have been found that address such possibilities. Longitudinal research that is able to build upon multiple sources of research evidence and address multiple factors is clearly needed.

DIVERGING THEORETICAL ASSUMPTIONS

In the studies reviewed for this annotated bibliography, results are often presented as if there is agreement regarding the true goals of career planning. Theoretical assumptions are rarely made explicit, even though there may be a variety of perspectives about what constitutes effective career planning. As noted earlier, most efficacy research seems to have been conducted from a structural perspective (e.g., linking specific individual attributes and occupational choice). Typically, this approach results in the selection of outcome measures such as increased knowledge of self (e.g., through standardized or informal career assessment measures), increased knowledge of the world of work (e.g., increases in occupational information or occupational information seeking behaviours), or the selection of a specific occupational goal (occupational decision-making). The relationship between these variables and broader outcomes such as career satisfaction or career stability is not known. Furthermore, there is little evidence even suggesting that these are the most relevant factors for consideration in career planning.

The lack of description of process variables associated with career planning is also problematic. In most studies, little differentiation is made regarding the process of career planning. Although process approaches to career planning have been described (e.g., Magnusson, 1992; Miller-Tiedeman & Tiedeman, 1990; Super, 1990), research studies rarely identify the process of career planning that interventions intend to address. A dominant although covert assumption seems to be that attending to one component of career planning improves overall career planning. Examples of such covert assumptions include the belief that increasing exploratory behaviours is a desired result, or that the general goal of interventions is decision-making.

In addition to the problems with outcome and process measures, there is also a need to identify client or career problem characteristics that may moderate the effects of treatment. Three specific areas are of concern. First, few if any attempts have been made to link interventions to clients’ presenting problems. In most analyses of psychotherapeutic interventions and their efficacy, the client’s presenting issue or problem is clearly identified and the subsequent intervention for that presenting problem is described and tested. However, in career research, the nature of the client’s presenting concern is usually not described, or if it is, it is given a generic label such as “undecided.” Yet clients who are “undecided” about their career paths may run the gamut from those who are trying to choose between two or more preferred futures, to

those who perceive that no options are available to them. For the former, information strategies may be more relevant, whereas for the latter, issues of self-efficacy may need attention. It would be very difficult to measure the true impact of a general intervention if the intervention is not even appropriate for the presenting problem.

The second concern related to the impact of treatment pertains to the role of intrapersonal processes in career planning. Perhaps because of the dominance of the assumption of the cognitive nature of career decision-making, very little attention has been paid to affective factors in career planning. The role that emotional states such as anxiety, depression, and anger, or even of the more positive emotions such as anticipation, hope or confidence, have on career planning process and outcome has not received sufficient attention in efficacy research.

The third concern related to the differential impact of interventions that needs further attention in efficacy research is the role that interpersonal processes play in career planning. Decision conflict may arise when a client’s occupational aspiration does not fit with family values, cultural mores, parental aspirations, or spousal demands. In a purely structural sense – the matching of individual potential with occupational demand – a decision may be a very good one, but on an interpersonal level the decision may be problematic. It will be important to devise research programs that identify and attend to the multiple variables that are related to career planning. A multivariate approach is more likely to enable us to determine the differential effects of interventions.

CONCLUSIONS

The conclusions drawn from this review echo those of recent researchers (Heppner & Heppner, 2003; Hughes, 2004; Sexton, 1996; Whiston, Brecheisen, & Stephens, 2003; Whiston, Sexton, & Lasoff, 1998). As Hughes (2004) noted, “Much of the research that is conducted has been one-off and fragmented, rather than strategic, and not disseminated widely or effectively” (p. 2). Heppner and Heppner (2003) call for increased research into the career counselling process, so that we can better understand what happens in career counselling and how those processes account for positive outcomes.

Obviously a comprehensive research strategy is needed for assessing the efficacy of career interventions. Given the complexity of career processes and factors, this may seem like a daunting proposition. However, the multiple facets, targets, processes, and outcomes of career development clearly need to be identified. We must deepen our understanding of the presenting issues that clients face, of the differential treatment modalities that may be brought to bear on those issues, the combined effect of those treatments on specific client outcomes, and the general and cumulative impact of client change on individual, social and economic well-being.

Compounding the problems associated with creating a comprehensive research framework are the problems of relevance and practicality. The field of career development is different from its psychotherapeutic cousin, in that its practitioners are often not specifically trained in the theory and practice of career counselling. The presentation of findings, and the means of data collection, must speak to the practical realities facing practitioners, policy makers, employers and researchers.

Finally, the general methodologies that would be included within a comprehensive framework must allow for consistency of data interpretation. Increased attention needs to be paid to the means by which data may be aggregated across context and client concern. The broad range of issues and factors associated with career planning demand a robust means of data aggregation

across impact studies. Furthermore, the long-term effects of career interventions can only be determined by conducting longitudinal, cross-sectional research.

Whether or not it is possible – or even desirable – to create such a framework remains to be seen. An initial task of the CRWG was to discuss what steps could be taken to improve our understanding of the “what, how, why and for whom” of career planning. To help with that process, it would be useful to have access to program evaluation research that has been conducted at agency, municipal, provincial, and federal levels. It will also be important to link the work of Canadian researchers with similar work being conducted in international contexts, particularly Great Britain and the United States. Perhaps the newly created International Centre for Career Development and Public Policy (ICCDPP) would be a useful mechanism for coordinating such international cooperation.

Participants at the Ottawa forum agreed that Canadian career services providers are under increasing scrutiny and pressure to prove that the services they offer are both cost-effective and beneficial. They also agreed that an important first step towards assembling an evidence base for career practice would be to determine the current state of outcome evaluation practice in Canada. To that end, a national on-line survey was created. The survey focused on three major stakeholder groups: representatives from agencies providing career services, front-line practitioners, and those involved either in policy development or as large-scale employers.

Four questions guided the development of the survey:

􀁸 How important is it to assess the impact of career services?

􀁸 How is the impact of career services determined?

􀁸 What results are achieved through the provision of career services?

􀁸 How are those results measured?

The methods used to collect and analyze these data are described in the next chapter.

REFERENCES

Brown, S. D., & Ryan Krane, N. (2000). Four (or five) sessions and a cloud of dust: Old assumptions and new observations about career counseling. In S.D. Brown & R.W. Lent (Eds.), Handbook of counseling psychology (3rd ed.) (pp. 740-766). Toronto, ON: John Wiley & Sons.

Heppner, M. J. & Heppner, P. P. (2003). Identifying process variables in career counseling: A research agenda. Journal of Vocational Behaviour, 62(3), 429-452.

Herr, E. L. (2003). The future of career counseling as an instrument of public policy. Career Development Quarterly, 52(1), 8-17.

Hiebert, B. (1994). A framework for quality control, accountability and evaluation: Being clear about the legitimate outcomes of career counselling. Canadian Journal of Counselling, 28(4), 334-345.

Hughes, D. (2004). Creating evidence: Building the case for career development. The Career Counsellor, 16, 2,7.

Magnusson, K. C. (1992). Career counseling techniques. Edmonton, AB: Life-Role Development Group.

Miller-Tiedeman, A., & Tiedeman, D. (1990). Career decision-making: An individualistic perspective. In D. Brown, L. Brooks and Associates (Eds.), Career choice and development (2nd ed.) (pp. 308-337). San Francisco: Jossey-Bass.

Sexton, T. L. (1996). The relevance of counseling outcome research: Current trends and practical implications. Journal of Counseling and Development, 74, 590-600.

Super, D. E. (1990). A life-span, life-space approach to career development. In D. Brown, L. Brooks and Associates (Eds.), Career choice and development (2nd ed.) (pp. 197-261). San Francisco: Jossey-Bass

Watts, A. G. (2000). Career development and public policy. Journal of Employment Counseling, 37, 62-75.

Watts, A. G. (2003). Career development and public policy. Canadian Career Development Foundation. Retrieved March 2, 2004, from http:www.ccdf.ca/pdf/chapter21/pdf

Weinrach, S. (1979). Career counseling: Theoretical and practical perspectives. New York: McGraw-Hill.

Whiston, S. C., Brecheisen, B. K., & Stephens, J. (2003). Does treatment modality affect career counseling effectiveness? Journal of Vocational Behaviour, 62(3), 390-410.

Whiston, S. C., Sexton, T. L., & Lasoff, D. L. (1998). Career intervention outcome: A replication and extension of Oliver and Spokane (1988). Journal of Counseling Psychology, 45(2), 150-165.

CHAPTER 3

STATE OF PRACTICE IMPACT SURVEY: METHODS

INTRODUCTION

The general purpose of this research was to determine the state of Canadian practice regarding the evaluation of career services efficacy. The research focused on the perceptions of three primary groups: agencies providing career services; practitioners of career services; and those in government and industry responsible for career development policy and employment.

DATA COLLECTION

The research was conducted by collecting data through surveys and telephone interviews. The first two groups (agencies and practitioners who provide career development services) were deemed likely to have a common understanding of the concepts and practices in this field from their training and experience. Consequently, the survey method was selected to collect a large amount of data from this sample.

However, little research has been conducted regarding the efficacy of career development services with individuals or organizations that do not actually deliver the services. Because the research with employers and policy makers was exploratory in nature, it was decided to utilize telephone interviews to collect data from these samples. This allowed the interviewer to provide information when clarification was necessary, and the interviewee to volunteer information that survey questions may not have elicited. Each method of data collection is described below.

SURVEYS

In autumn 2004, two on-line surveys were developed (one for agencies and one for practitioners) to answer the following questions:

􀁸 How important is it to assess the impact of career services?

􀁸 How is the impact of career services determined?

􀁸 What results are achieved through the provision of career services?

􀁸 How are those results measured?

For the purposes of this research, an intervention was defined as any intentional activity that is implemented in the hopes of fostering client change. Another way to think of an intervention is that it is anything that is done to produce an outcome for the client or target group. An outcome was defined as a specific result or product of an intervention that might include any combination of the following factors:

􀁸 A difference in client competence that is reflected by changes in client knowledge, skills and/or attributes.

􀁸 A change in the client’s situation, for example, in employment status, educational status, etc.

􀁸 Broad or long term changes for the client and/or community, such as in the client’s financial situation, social inclusion, family changes, community economic development, etc.

􀁸 These definitions were included in the surveys to provide a common understanding of the focus of the questions.

Parallel forms of the survey were developed: one form was completed by representatives from agencies and organizations providing direct career services to clients, and the second form was completed by practitioners. The structure and items on each form were identical, with only wording changes to reflect the specific target audience. In addition, each form was available in both French and English; potential participants were directed to the appropriate form and language version when they first accessed the survey website.

Invitations to participate were distributed to a master list of agencies and practitioners who provide career development services in Canada, maintained by the Canadian Career Development Foundation (CCDF). This list included participants from Working Connections: A Pan-Canadian Symposium on Career Development, Lifelong Learning and Workforce Development (held in Toronto in November of 2003; n = 150); the Career Circuit Network (n = approximately 3000); The Stakeholder Liaison and Advisory Council (SLAC) of the Canadian Standards and Guidelines for Career Development Practitioners (n = 65); and the Canadian Career Development Foundation Board of Governors (n = 11). The invitations (see Appendix B) included a description of the nature of the research and a link to the online survey Furthermore, invitation recipients were encouraged to forward the invitation to other agencies or practitioners who may not have been on the original list, creating a “fan-out” effect. Thus, while the original distribution list contained over 3200 names, the “fan-out method” made it impossible to tell how many people ultimately received an invitation to participate.

Potential participants, upon accessing the survey website and selecting the appropriate form to complete, were provided a brief introduction that explained the goals of the research and clarified the terms that would be used. Those individuals who agreed to complete a survey were asked to review and agree to a research consent form that outlined the ethical terms of the research involvement; consent was indicated by selecting an “I Accept” button at the end of consent form. Selecting the “I Accept” button was the only way to gain access to the actual surveys.

The surveys contained three main sections. Part I, Context of Services and Agency Profile, included demographic items pertaining to the nature of the agency, the services offered, and the populations served. These items required closed-ended or forced-choice responses. One format of question asked participants to estimate how frequently (“never,” “sometimes,” or “often”) they offered each of a list of career services. A second format of questions asked participants to select appropriate responses from a list provided (i.e., identification of the target of services, including age, gender, culture, employment status, and special needs; and the kind and size of agency). Participants could also indicate “other” if the list provided was not reflective of their services. The data from Part I of the survey were used to develop a context for career service provision in Canada.

Part II, Outcomes of your Services, contained both closed and open questions. To answer the primary research questions, participants were asked to indicate how important they believed it was to collect impact data (“not at all,” “somewhat,” or “very”) and whether or not they actually engaged in the practice of impact evaluation (“yes” or “no”). These data were compared to the

demographic data from Part I to determine whether there were any relationships between agency/service demographics and the importance and practice of impact assessment.

A series of open-ended questions were also posed. Participants who responded “yes” to whether or not they actually engaged in impact assessment were then asked to identify the three main outcomes they report, and the evidence they collect for each outcome. All participants were then asked to indicate the outcomes they believe they were achieving but that do not get reported, and the evidence they have for those outcomes. To provide further context, participants were asked to describe the difficulties they face when trying to collect evidence or document the impact of their services. Three final questions, designed to explore what participants would like to see, concluded Part II of the survey. Participants were asked to indicate the kinds of outcomes they would like to report on if they could, and to suggest ways in which the impact of their services could be better measured. Finally, participants were invited to comment on any other aspects of impact evaluation that they had not yet had an opportunity to mention.

Part III, Follow Up, provided an opportunity for further involvement in impact assessment. Respondents were invited to provide examples of any exemplary evaluation tools that they have used, and were also encouraged to indicate if they were interested in participating in follow-up interviews.

All data collected from the on-line surveys were stored in a custom-designed database. When participants completed an item, the response was automatically stored in the appropriate cell of the database. The composite data set was transferred to a spreadsheet for subsequent data analysis; all quantitative analysis was conducted using the Statistical Package for the Social Sciences (SPSS).

TELEPHONE INTERVIEWS

Telephone interviews were conducted with the third target group of the research, who were divided into two distinct samples: (a) individuals who were employed by governments in policy portfolios connected to career and workforce development and (b) Canadian employers. The sample was identified from a group of individuals who participated in the Working Connections Pan-Canadian Symposium in 2003. A total of 41 policy makers and 23 employers were contacted to participate in the study. Of the policy makers, 37 were obtained from the symposium list and 4 were identified by personal referrals. Twenty employers were obtained from the symposium list, and 3 employers were identified by referrals. Nine policy makers and 7 employers participated in phone interviews that ranged from 10 to 35 minutes in length.

The individuals who participated in the phone interviews were fewer than originally planned and included only 2 Francophone participants (the goal was to interview 50 individuals of which half would be Anglophone and half Francophone). It was difficult to acquire research participants for the following reasons:

􀁸 work roles and responsibilities had changed for many policy makers, and they no longer were responsible for career and workforce development programs and services;

􀁸 many individuals indicated that they did not have time to participate in the interviews;

􀁸 the employers did not think they had enough information, as career development was too peripheral to their work;

􀁸 individuals did not respond to e-mails; and

􀁸 scheduling a suitable time for an interview was problematic with some of the individuals who were contacted.

The first step in collecting data was to send letters of invitation to participate in a research interview (see Appendix C for letter sent to policy makers and Appendix D for letter sent to employers). Follow-up e-mails were then sent to those who did not respond to the initial invitation. Individuals who agreed to participate in the research were sent a copy of the appropriate interview protocol (see Appendix E for the policy maker protocol and Appendix F for the employer protocol).

During the telephone interviews, the policy makers were asked the following questions:

􀁸 What do you want back from career development programs and services as a result of your funding?

􀁸 How do you evaluate if you are getting back what you want from these programs and services?

􀁸 What information are you getting to help you make informed decisions about the programs and services you are funding or will fund?

􀁸 How useful is the information you are getting? If you had a wish list, what information would you like to have to help you evaluate better?

􀁸 Do you have any exemplary examples of what is working well?

􀁸 If yes, can you provide a copy to the research team?

􀁸 What other comments would you like to make about measuring the impact of services?

The employers were asked the following interview questions:

􀁸 What do you want back from these programs and services as a result of your investment?

􀁸 How do you evaluate if you are getting back what you want from these programs and services?

􀁸 Are you getting the information you need to make informed decisions about the programs and services you offer?

􀁸 If you had a wish list, what information would you like to have to help you evaluate better?

􀁸 Do you have any exemplary examples of what is working well?

􀁸 If yes, can you provide a copy to the research team?

􀁸 What other comments would you like to make about measuring the impact of services?

These phone interviews were conducted by a member of the research team and a research assistant between January 2005 and March 2005. Notes were taken of the participants’ responses during the interview.

DATA ANALYSIS

SURVEY DATA

Comparison of English and French data on global importance ratings. Parallel forms of the surveys were prepared for each official language. Prior to analysis of specific data, a comparison of Anglophone and Francophone respondents was conducted. Chi-square tests were used to determine if there was a significant relationship between language of respondent and the two global criteria variables of “Importance of Outcome Measurement” and “Practice of Outcome Measurement.” If no significant differences were found, the data were merged for subsequent analysis. If significant differences were found, the data for Anglophone and Francophone respondents were reported separately.

􀁸 Context of services (frequency analysis). Simple frequency analysis methods were applied to all demographic items on the survey, as well as to the items pertaining to perceived importance and practice of outcome assessment. Raw counts and percentages were tabulated for all of these items.

􀁸 Measures of association (chi-square analysis). To test for relationships between demographic variables and the criteria variables of perceived importance of impact assessment and actual practice of impact assessment, chi-square measures were applied, with alpha set to .05. Chi-square analysis was also used to determine whether or not there were significant differences between respondents who completed the surveys in English and those who completed them in French.

􀁸 Open-ended questions (content analysis). Data from open-ended questions were analyzed using content analysis techniques. Content analysis is a process of identifying general themes from the specific responses. Twenty participant “cases” were randomly drawn from the database, and verbatim responses were first examined for the presence of themes or patterns. As each new case was examined, the responses were coded into themes and categories; when a new theme or category emerged, each prior case was reviewed again to search for the existence of the new theme in that data. This process was repeated until no new themes emerged. As a check on the reliability of the themes, a second set of 20 cases was randomly drawn. No new themes were found, indicating that data saturation had been reached.

TELEPHONE INTERVIEW DATA

The written notes from the telephone interviews were also analyzed using content analysis. The first step in the telephone interview data analysis involved reviewing the participants’ responses to each question and grouping similar responses into themes or categories. The number of responses in each theme was noted. These initial groupings of responses to each question were then reviewed, and higher order themes were identified that represented categories of responses. Because some of the responses to each question were related to some of the responses to other questions, the higher order themes were reviewed across the questions. This resulted in two lists of general themes that represented the responses made by the group of policy makers and those made by the group of employers.

One individual initially conducted the analysis of data; then the original responses and themes were sent to another person who confirmed the interpretation of the results. This enhanced the validity of the content analysis.

FOCUS GROUP

Focus groups were held, in French and English, in order to present the initial themes from the data, and to seek feedback on the accuracy of the themes, as well as clarification and expansion on the themes that were identified through the survey data. In January 2005, separate English and French open sessions were held at the National Consultation on Career Development in Ottawa. The preliminary results were presented to attendees, who were then invited to comment on the accuracy of the themes, to suggest additional themes that may not have been captured through the survey data, and to expand or elaborate on any of the themes. The focus groups served as useful reliability checks on the data from the surveys.

CHAPTER 4

RESULTS

From November 2004 through January 2005, the Canadian Research Working Group for Evidence-Based Practice in Career Development (CRWG) conducted a survey of agencies and practitioners providing career services in Canada. The general purpose of the survey was to determine the state of practice in Canada with respect to the measurement and reporting of the impact of career services delivery. Four questions guided the survey:

􀁸 How important is it to assess the impact of career services?

􀁸 How is the impact of career services determined?

􀁸 What results are achieved through the provision of career services?

􀁸 How are those results measured?

Surveys were sent to representatives of agencies providing career services, and to career development practitioners across Canada. Policy makers and employers were also interviewed to determine their views on the importance of career services impact assessment. The results of the surveys are presented first, followed by the interview results. Only those results that were statistically significant are reported.

DEMOGRAPHIC DATA

RESPONDENTS

The survey was conducted on-line; participants were able to access the site from November 2004 to January 2005. A total of 173 agency representatives completed the survey, 147 English-language respondents and 26 French-language respondents. An additional 214 practitioners completed the survey; of these, 168 were Anglophone practitioners and 46 were Francophone practitioners.

COMPARISON OF ANGLOPHONE AND FRANCOPHONE RESPONDENTS

Respondents had the option of completing the survey in either French or English. To determine if there were any significant differences in the perceived importance and actual practice of outcome measurement between the two language groups, Chi-square analyses were conducted. No significant differences were found between English and French agencies on the importance of outcome measurement, nor on whether the agencies reported on their outcomes. Therefore, these data were merged for subsequent analyses of the agency data.

There was a significant difference between Anglophone and Francophone practitioners on the importance of outcome measurement (Χ2 = 55.14; p = .000). Francophone practitioners were more likely than Anglophone respondents to rate the measurement of outcomes as “Not at all important” (see Table 1). There was also a significant difference between Anglophone and Francophone practitioners on the practice of outcome measurement (Χ2 = 9.39; p = .002). Anglophone practitioners were more likely to report on the outcome of their services (see Table 2). Because of the significant differences on the key criteria variables of the study, the Anglophone and Francophone practitioner data will be reported separately.

Table 1. Relationship Between Language of Practitioner Respondents and Perceived Value of Measuring Outcomes/Impact of Career Services

Language of Respondent	Count		Not at All Important		Somewhat Important		Very Important		Total
Anglophone Practitioners	Actual Expected		1 10.2		34 40.9		131 114.9		166
Francophone Practitioners	Actual Expected		12 2.8		18 11.1		15 31.1		45
Total		13		52		146		211

(Χ2 = 55.14; p = .000)

Table 2. Relationship Between Language of Practitioner Respondents and Actual Measurement of Outcomes/Impact of Career Services

Language of Respondent	Count		Yes		No		Total
Anglophone Practitioners	Actual Expected		118 109.4		48 56.6		166 166
Francophone Practitioners	Actual Expected		21 29.6		24 15.4		45 45
Total		139		72		211

(Χ2 = 9.39; p = .002)

KINDS OF CAREER SERVICES PROVIDED

A number of survey items were designed to provide a general picture of the kinds of agencies and practitioners that responded to the survey. As noted above, the Anglophone and Francophone agency data are combined, and the practitioner data are reported separately.

Respondents were asked to indicate the kind of services that they provided and to estimate in general terms how frequently these services were offered. Respondents were encouraged to respond to as many services as were appropriate. Table 3 details the kind and frequency of services provided by agencies; parallel data for Anglophone practitioners are presented in Table 4, and for Francophone practitioners in Table 5. When the categories of “sometimes” and “often” are combined, the most common services provided by Canadian agencies were career education (100%), job search information (94%), career and labour market information (89%), individual career counselling (86.7%), employment counselling (83.3%), and group counselling (78.6%). Thus, the agency respondents to this survey would seem to provide a representative sample of commonly offered career services and interventions.

Table 3. Kind and Frequency of Services Provided by Agencies

Service Provided	Never n (%)	Sometimes n (%)	Often n (%)	Total Respondents
Individual career counselling	17 (9.8)	35 (20.2)	115 (66.5)	167
Group career counselling	29 (16.8)	86 (49.7)	50 (28.9)	165
Career education programs	0 (0%)	65 (37.6)	71 (41%)	136
Assessment services	34 (19.7)	52 (30.1)	80 (46.2)	166
Career and labour market information services	13 (7.5)	45 (26)	109 (63)	167
Employment counselling	21 (12.1)	38 (22)	106 (61.3)	165
Job search information and/or resources	2 (1.2)	38 (22)	126 (72.8)	166
Work development	36 (20.8)	54 (31.2)	73 (42.2)	163
Other	8 (4.6)	10 (5.8)	45 (26)	63

There were some differences in the pattern of service provision by Anglophone practitioners compared to the reported agency patterns (see Table 4). While all Anglophone practitioners (100%) reported providing career education programs, they also all reported providing assessment services (compared to only 76.2% of the agencies). Higher frequencies were also found for individual counselling (97.6%), career and labour market information (97.6%), employment counselling (94.6%) and “other” (91.8%). However, with the exception of the role of assessment for Anglophone practitioners, the rank order of service provision was much the same (see Table 6).

Table 4. Kind and Frequency of Services Provided by Anglophone Practitioners

Service Provided	Never n (%)	Sometimes n (%)	Often n (%)	Total Respondents
Individual career counselling	4 (2.4%)	24 (14.4%)	139 (83.2%)	167
Group career counselling	34 (20.4%)	79 (47.3%)	54 (32.3%)	167
Career education programs	0 (0%)	66 (46.5%)	76 (53.5%)	142

21 Service Provided Never n (%) Sometimes n (%) Often n (%) Total Respondents

Assessment services	16 (9.7%)	54 (32.7%)	95 (57.6%)	165
Career and labour market information services	4 (2.4%)	49 (29.3%)	114 (68.3%)	167
Employment counselling	9 (5.4%)	47 (28.3%)	110 (66.3%)	166
Job search information and/or resources	2 (1.2%)	28 (16.9%)	136 (81.9%)	166
Work development	28 (16.7%)	80 (47.6%)	60 (35.7%)	168
Other	5 (8.2%)	15 (24.6%)	41 (67.2%)	61

There were some differences in the reporting of service provision between Anglophone and Francophone practitioners. Perhaps the most obvious difference was in the provision of career education programs: all Anglophone respondents reported providing career education programs, but only 45.7% of the Francophone respondents reported doing so. However, it should be noted that only 21 Francophone practitioners responded to this item; all of the respondents answering the question indicated that they provided career education programs “sometimes” or “often.” It may be that the phrase “career education” has a different and very specific connotation for Francophone practitioners; those outside of the school system may not view the term the same way as their Anglophone counterparts. Francophone practitioners were also less likely to report providing assessment services (76.1%) than their Anglophone counterparts (100%). When these items are excluded, the ranking of service provision follows the same pattern as for the Anglophone practitioners: job search information (93.5%), labour market information (93.5%), individual counselling (93.5%), and employment counselling (80.5%). Furthermore, while the provision of group counselling varied in rank from one group to the other, the relative frequencies were very common across the groups (78.6% for agencies, 79.6% for Anglophone practitioners, and 76.1% for Francophone practitioners).

It would seem from the analysis of the frequency of reported service provision that a broad and representative sampling of Canadian career service providers responded to the survey. It also appears that there is general agreement between agencies and practitioners about the relative ranking of the frequency of the provision of the different kinds of career services.

Table 5. Kind and Frequency of Services Provided by Francophone Practitioners

Service Provided	Never n (%)	Sometimes n (%)	Often n (%)	Total Respondents
Individual career counselling	3 (6.5)	7 (15.2)	34 (73.9)	44
Group career counselling	11 (23.9)	18 (39.1)	16 (34.8)	45
Career education programs	0	18 (39.1)	3 (6.5)	21
Assessment services	9 (19.6)	22 (47.6)	13 (28.3)	44
Career and labour market information services	2 (4.3)	15 (32.6)	28 (60.9)	45
Employment counselling	8 (17.4)	16 (34.8)	21 (45.7)	45
Job search information and/or resources	2 (4.3)	21 (45.7)	22 (47.8)	45
Work development	23 (50)	15 (32.6)	7 (15.2)	45
Other	2 (4.3)	3 (6.5)	6 (13)	11

Table 6. Comparison of Ranking of Service Provision: Agencies, Anglophone Practitioners, and Francophone Practitioners

Rank	Agencies	Anglophone Practitioners	Francophone Practitioners
1	Career education	Career education	Job search information
2	Job search information	Assessment	Labour market information
3	Labour market information	Job search information	Individual counselling
4	Individual counselling	Labour market information	Employment counselling
5	Employment counselling	Individual counselling	Group
6	Group	Employment counselling	Assessment

DEMOGRAPHICS OF CANADIAN CAREER SERVICES AGENCY RESPONDENTS

Agency respondents were asked to describe the clients that they served, in terms of a range of characteristics including age, gender, culture, and employment status. Respondents were encouraged to select all options that applied to them.

Table 7. Age Groups Served by Agencies

Age Group Served	Percentage of Agencies Providing Career Services to This Group (n = 173)
Students in Grade 9 or less	19.7
Students in Grade 10 to 12	37
Post-secondary students	41
Young People (< 25) not in school	49.7
Adults, 25 to 45	55
Adults, 45+	32.4

The vast majority of agencies (93%) provided services to males and females. Of the remainder, some (3%) were intended for females only, and 4% did not respond. Most agencies (76%) also did not have a specific cultural focus for their services; the two exceptions were agencies providing services to First Nations members (17.3%) or providing specialized immigrant services (2%). Many agencies did target clients by employment status: 56.6% of the agencies provide career services for unemployed clients, 42.8% for social allowance recipients, 34.7% for clients in educational settings, and 38% said that their services were open to anyone. Interestingly, only 11.6% of the agencies specifically targeted clients who were already working.

Agency respondents were asked to indicate the kind of agency or service that they represented. Most respondents represented not-for-profit agencies (50.3%) or provincial government settings (23.3%). Other kinds of agencies represented included school (K to 12; 8.6%), post-secondary (8.6%), private (for profit) agencies or private practice (6.1%), and federal government agencies (3.1%). Although the representation from school settings was less than hoped for, the remaining kinds of service providers would seem to be fairly representative of the range and relative frequency of career service providers in Canada. The frequencies of the kinds of service agencies were also reflected in the size of agencies: 41.1% of the respondents represented agencies with more than ten career service providers, 19.6% were in agencies with six to ten service providers, 28.8% worked in small (two to five service providers) settings, and 10.4% worked on their own (i.e., one person providing career services).

DEMOGRAPHICS OF CLIENTS SERVED BY PRACTITIONER RESPONDENTS

The demographic data for groups served by practitioners are provided for both Anglophone (n = 168) and Francophone (n = 46) respondents. Because of the large volume of data represented, the data are presented in a series of tables. The age group of client services is presented in Table 8; the gender focus of services in Table 9; the cultural focus of services in Table 10; the

employment status of clients served in Table 11; the kind of agency that the practitioners work for in Table 12; and the size of the practitioner’s agency in Table 13. A brief discussion of salient points from this data follows. Due to the small number of Francophone respondents, caution should be used in making comparisons with the data from the Anglophone respondents.

All client age groups are represented in the services provided by both Anglophone and Francophone practitioners. A greater percentage of Anglophone practitioners provide services to adults, whereas a higher number of Francophone respondents provide services to younger children. Otherwise, services are similarly distributed across age groups (see Table 8).

Table 8. Age Groups Served by Practitioners

Age Group Served	Anglophone Practitioners N = 168 n/%	Francophone Practitioners N = 46 n/%
Students in grade 9 or less	19 (11.3)	16 (34.8)
Students in grades 10 to 12	62 (36.9)	20 (43.5)
Students in post-secondary settings	81 (48.2)	17 (37.0)
Young people (i.e., 19 years old or less) who are not in school settings	82 (48.8)	17 (37.0)
Adults in early to middle age (25 to 45 years old)	118 (70.2)	25 (54.3)
Older adults (i.e., more than 45 years old)	102 (60.7)	18 (39.1)

The vast majority of services for both practitioner groups are open to both men and women, although there seem to be more Francophone practitioners providing services intended for women clients (see Table 9).

Table 9. Gender Focus of Practitioner Services

Gender Focus of Services	Anglophone Practitioners N = 167 n/%	Francophone Practitioners N = 45 n/%
Programs/services specifically designed for women	1 (0.60)	6 (13)
Programs/services specifically designed for men	0 (0)	0 (0)
Programs/services open to women and men	166 (99.4)	39 (86.7)

Similarly, the programs and services of the vast majority of practitioners have no particular cultural focus; the one exception seems to be a larger percentage of programs for First Nations clients provided by Anglophone practitioners (see Table 10).

Table 10. Cultural Focus of Practitioner Services

Cultural Focus of Services	Anglophone Practitioners N = 165 n/%	Francophone Practitioners N = 44 n/%
Programs/services designed with no specific cultural or ethnic group in mind	145 (87.9)	40 (90.9)
Programs/services designed for recent immigrants	3 (1.8)	2 (4.5)
Programs/services designed for First Nations people	17 (10.3)	2 (4.5)
Programs/services designed for other specific cultural groups	0 (0)	0

Small differences are noted in the employment status focus of practitioner services (see Table 11). Anglophone practitioners seem to provide more services to unemployed people and those on social allowance, while their Francophone counterparts provide more services to people in educational settings. However, this trend may be easily explained in terms of the employment settings where the practitioners work: A higher percentage of the Francophone respondents are employed in school settings, whereas more of the Anglophone practitioners work for provincial government agencies (see Table 12).

Table 11. Employment Status Focus of Practitioner Services

Employment Status	Anglophone Practitioners N = 168 n/%	Francophone Practitioners N = 44 n/%
Programs/services designed for unemployed individuals	95 (56.5)	18 (39.1)
Programs/services designed for social allowance recipients	71 (42.3)	14 (30.4)
Programs/services designed for people in educational settings	73 (43.5)	24 (52.2)
Programs/services designed for people employed in specific work settings	29 (17.3)	5 (10.9)
Programs/services designed for any member of the community, regardless of employment status	71 (42.3)	15 (32.6)
Other	16 (9.5)	6 (13.0)

Table 12. Employment Setting of Practitioners: Kind of Agency

Kind of Agency	Anglophone Practitioners N = 168 n/%	Francophone Practitioners N = 45 n/%
Federal government agency	9 (5.4)	0 (0)
Provincial government agency (non-school)	39 (23.2)	3 (6.5)
School within K-12 system	31 (18.5)	15 (32.6)
Post-secondary institute (college, technical institute, or university)	23 (13.7)	4 (8.7)
Career services or human resource unit within a larger company/organization providing services for the benefit of employees	3 (1.8)	2 (4.3)
Not-for-profit agency	50 (29.8)	15 (32.6)
Private (for-profit) career services provider	5 (3.0)	1 (2.2)
Private practice/consultancy (self-employed)	8 (4.8)	5 (10.9)

The practitioners who responded to this survey typically worked in small agencies or employment settings (see Table 13). Among Anglophone practitioners, 57% of the respondents worked in settings that employed 5 or fewer career development service providers, and almost 70% of the Francophone respondents worked in similar settings. Once again, most of the Francophone respondents were employed in either K-12 school settings or in not-for-profit agencies. Conversely, very few Francophone respondents (6.5%) worked in large (10 or more career service providers) settings, whereas more than one-quarter (26.3%) of the Anglophone respondents worked in large settings. These data may be more reflective of the methods used to invite Francophone respondents to participate than of any real differences in employment settings. It seems that the initial distribution list and subsequent “fan-out” methods may have been more inclusive of school settings for Francophone participants.

Table 13. Employment Setting of Practitioners: Size of Agency

Agency Size	Anglophone Practitioners N = 167 n/%	Francophone Practitioners N = 45 n/%
10 or more career development service providers	44 (26.3)	3 (6.5)
More than 5 but less than 10 career development service providers	28 (16.8)	10 (21.7)
2 to 5 career development service providers	71 (42.5)	15 (32.6)
One-person	24 (14.4)	17 (37)

IMPORTANCE AND PRACTICE OF OUTCOME MEASUREMENT

Prior to exploring agency and practitioner practice with respect to outcome measurement and reporting, it was necessary first to determine the importance that they placed on outcome assessment. To that end, respondents were asked, “How important is it to measure the outcomes/impact of career services?” Responses included “Not at all important,” “Somewhat important,” and “Very important.” These data are presented in Table 14. Following the question about importance, respondents were asked, “Does your agency/organization report on the outcomes/impact of its services?” Responses included either “Yes” or “No”; these data are presented in Table 15.

Table 14. Perceived Importance of Outcomes/Impact Measurement

Importance	Agencies	Anglophone Practitioners	Francophone Practitioners
Not at all important	4 (2.3%)	1 (0.60%)	12 (26.7%)
Somewhat important	40 (24.4%)	34 (20.5%)	18 (40%)
Very important	120 (73.2%)	131 (78.9%)	15 (33.3%)
Total	164	166	45

Table 15. Actual Practice of Outcomes/Impact Measurement

Report on Outcomes?	Agencies	Anglophone Practitioners	Francophone Practitioners
Yes	136 (84%)	118 (71.1%)	21 (46.7%)
No	26 (16%)	48 (28.9%)	24 (53.3%)
Total	162	166	45

There was almost universal agreement among agency respondents on the importance of measuring the outcomes or impact of career services; 97.6% of the respondents indicated that it was either “somewhat” (24.4%) or “very” (73.2%) important. When asked if they actually reported on the outcomes or impact of their services, 84% of the agencies replied “Yes.” Thus, there would seem to be a small drop-off between perceived importance of impact measurement and the actual practice of impact measurement among agency respondents.

There was also near universal agreement among Anglophone practitioners on the importance of measuring the outcomes of their services; 99.4% of the respondents rated it as either “somewhat” (20.5%) or “very” (78.9%) important. Francophone practitioners were less in agreement; 26.7% responded that impact measurement was “not at all important,” 40% rated it as “somewhat important,” and 33.3% rated it as “very important.” Once again, there was also a drop-off in the actual practice of impact measurement for both groups: 71.1% of the Anglophone practitioners and 46.7% of the Francophone practitioners actually reported on the outcomes or impact of their services.

MEASURES OF ASSOCIATION

The next stage of data analysis involved an exploration of possible relationships between the demographic variables and the perceived importance and actual measurement of outcome or impact measurement. The combined agency data and Anglophone practitioner data were used to conduct these explorations; tests of association were not conducted for the Francophone respondent pool because of its smaller size.

TYPE OF AGENCY AND PERCEIVED IMPORTANCE OF OUTCOME MEASUREMENT

The first question posed was, “Is there a significant relationship between the type of agency/service provider and the perceived value of measuring the impact of career services?” To answer this question, a chi-square analysis was conducted comparing agency type (Survey Item 3.1) and the perceived importance of measuring the impacts/outcomes of services (Survey Item 4.1). The agency data are provided in Table 16, and the Anglophone practitioner data in Table 17.

Table 16. Relationship Between Type of Agency/Service Provider and Perceived Value of Measuring Outcomes/Impact of Career Services: Agency Results

How Important is it to measure the outcomes/impact of your services?					Total
Not at All Important		Somewhat Important	Very Important
Please indicate which of the following best describes your agency, organization or service.	Federal government agency		Count	1		1	3	5
			Expected Count	.1		1.2	3.7	5.0
	Provincial government agency (non-school)		Count	1		7	30	38
			Expected Count	.9		9.3	27.7	38.0
	School within K-12 system		Count	0		9	5	14
			Expected Count	.3		3.4	10.2	14.0
	Post-secondary institute		Count	0		6	8	14
			Expected Count	.3		3.4	10.2	14.0
	Career services within larger company		Count	0		0	0	0
			Expected Count	.0		.0	.0	0.0
	Not-for-profit agency		Count	2		15	65	82
			Expected Count	2.0		20.1	59.9	82.0

How Important is it to measure the outcomes/impact of your services?				Total
Not at All Important		Somewhat Important	Very Important
Private-for-profit agency	Count		0	2	5	7
	Expected Count		.2	1.7	5.1	7.0
Private practice	Count		0	0	3	3
	Expected Count		.1	.7	2.2	8.0
Total	Count		4	40	119	163
	Expected Count		4.0	40.0	119.0	163.0

(Χ2 = 25.036; df = 12; p = .015)

Table 17. Relationship Between Type of Agency/Service Provider and Perceived Value of Measuring Outcomes/Impact of Career Services: Anglophone Practitioner Results

Question 4.1: How Important is it to measure the outcomes/impact of your services?					Total
Not at All Important		Somewhat Important	Very Important
Question 3.1: Please indicate which of the following best describes your agency, organization or service.	Federal government agency		Count	1		2	6	9
			Expected Count	.1		1.8	7.1	9.0
	Provincial government agency (non-school)		Count	0		3	35	38
			Expected Count	.2		7.8	30.0	38.0
	School within K-12 system		Count	0		14	16	30
			Expected Count	.2		6.1	23.7	30.0
	Post-secondary institute		Count	0		6	17	23
			Expected Count	.1		4.7	18.2	23.0
	Career services within larger company		Count	0		1	2	3
			Expected Count	.0		.6	2.4	3.0

Question 4.1: How Important is it to measure the outcomes/impact of your services?				Total
Not at All Important		Somewhat Important	Very Important
Not-for-profit agency	Count		0	4	46	50
	Expected Count		.3	10.2	39.5	50.0
Private-for-profit agency	Count		0	2	3	5
	Expected Count		.0	1.0	3.9	5.0
Private practice	Count		0	2	6	8
	Expected Count		.0	1.6	6.3	8.0
Total	Count		1	34	131	166
	Expected Count		1.0	34.0	131.0	166.0

(Χ2 = 40.8; df = 14; p = .000)

The results demonstrate that there is a significant relationship between the type of service provider and the perceived value of measuring the impact of career services for agencies (Χ2 = 25.036; df = 12; p = .015). Respondents from K-12 school settings were less likely to rate the impact measurement as being “very important,” and those in post-secondary settings were more likely to say it was “somewhat important.” Overall, respondents who worked in educational settings placed less importance on impact measurement than their counterparts in not-for-profit or government settings. There was also a significant relationship between the type of service provider and the perceived value of measuring the impact of career services for Anglophone practitioners (Χ2 = 40.8; df = 14; p = .000). Practitioners in schools were also less likely to view impact measurement as “very important,” and not-for-profit agencies were more likely to rate it as “very important.”

TYPE OF AGENCY AND ACTUAL MEASUREMENT OF OUTCOMES/IMPACT

The second measure of association asked, “Is there a significant relationship between the type of agency/service provider and the degree to which they measure the impact of career services?” To answer Question B2, a chi-square analysis was conducted comparing agency type (Survey Item 3.1) and the degree to which they measure the impacts/outcomes of services (Survey Item 4.2). The agency results are presented in Table 18, and the Anglophone practitioner results in Table 19.

The results indicate that there is a significant relationship between type of service provider and the actual measurement of service impact for both agencies (Χ2 = 32.388; df = 6; p = .000) and practitioners (Χ2 = 47.812; df = 7; p = .000). Agency respondents representing K-12 settings were less likely to report on the impact of their services; not-for-profit agencies were more likely to

report on the impact of their services. This finding essentially parallels the results noted above for the importance of measuring career services impact. Practitioners in federal government agencies, K-12 schools, and private practice were less likely to engage in impact reporting.

Table 18. Relationship Between Type of Agency/Organization/Service Provider and Degree to Which They Evaluate Outcomes/Impact of Services: Agency Results

Does your agency/organization report on the outcomes/impact of its services?					Total
Yes			No
Please indicate which of the following best describes your agency, organization or service.	Federal government agency		Count		4		1		5
			Expected Count		4.2		.8		5.0
	Provincial government agency		Count		32		6		38
			Expected Count		31.9		6.1		38.0
	School within K-12 system		Count		4		9		13
			Expected Count		10.9		2.1		13.0
	Post-secondary institution		Count		12		2		14
			Expected Count		11.7		2.3		14.0
	Career services within larger organization		Count		0		0		0
			Expected Count		2.1		.9		3.0
	Not-for-profit agency		Count		75		6		81
			Expected Count		67.9		13.1		81.0
	Private-for-profit agency		Count		6		1		7
			Expected Count		5.9		1.1		7.0
	Private practice		Count		2		1		3
			Expected Count		2.5		.5		3.0
Total		Count		135		26		161
		Expected Count		135.0		26.0		161.0

(Χ2 = 32.388; df = 6; p = .000)

Table 19. Relationship Between Type of Agency/Organization/Service Provider and Degree to Which They Evaluate Outcomes/Impact of Services: Anglophone Practitioner Results

Question 4.2: Does your agency/organization report on the outcomes/impacts of its services?					Total
Yes			No
Question 3.1: Please indicate which of the following best describes your agency, organization or service.	Federal government agency		Count		2		7		9
			Expected Count		6.4		2.6		9.0
	Provincial government agency		Count		37		2		39
			Expected Count		27.7		11.3		39.0
	School within K-12 system		Count		15		16		31
			Expected Count		22.0		9.0		31.0
	Post-secondary institution		Count		13		10		23
			Expected Count		16.3		6.7		23.0
	Career services within larger organization		Count		2		1		3
			Expected Count		2.1		.9		3.0
	Not-for-profit agency		Count		43		5		48
			Expected Count		34.1		13.9		48.0
	Private-for-profit agency		Count		4		1		5
			Expected Count		3.6		1.4		5.0
	Private practice		Count		2		6		8
			Expected Count		5.7		2.3		8.0
Total		Count		118		48		166
		Expected Count		118.0		48.0		166.0

(Χ2 = 47.812; df = 7; p = .000)

SIZE OF SERVICE PROVIDER, PERCEIVED IMPORTANCE AND ACTUAL MEASUREMENT OF IMPACT

The third measure of association addressed the question, “Is there a significant relationship between the size of an agency/service provider and the perceived value of measuring the impact of career services?” To