ProfPort Webfolio System:

Implementation, Curriculum and Assessment

 

A paper presented at The 2003 Educause Annual Conference: Balancing Opportunities, Expectations, and Resources, in Anaheim, California, USA.

 

Copyright Paul Gathercoal, Douglas Love & Gerry McKean, 2003. This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

 

Paul Gathercoal, Ph.D.

California Lutheran University

gatherco@clunet.edu

Doug Love, Ph.D.

Illinois State University

dolove@ilstu.edu

Gerry McKean, Ph.D.

Illinois State University

gwmckea@ilstu.edu

 

In recent years, webfolios and ePortfolios have been highly vaunted as the next great innovation in education (Kilbane & Milman, 2003; Educause: NLII, 2002; Gathercoal, Love, Bryde & McKean, 2002; Gathercoal, Love & McKean, in press).  Prominent in the literature is optimistic rhetoric praising the benefits of the heuristic and metacognitive processes native to generating growth and showcase portfolios; most of the literature indicates that the process is so important and so valuable to the student, that the process alone is reason enough to “dump” traditional assessment practices in favor of portfolios, ePortfolios and/or webfolios.

 

Gaining widespread popularity in education, portfolio assessment has tremendous advantages over traditional one-time, objective-based test assessment.  Objective-based test assessment only focuses on the product and limits the learner's ability to demonstrate the learning process. It does not allow learners to focus on specific developmental issues that are important to them, instead forcing them to focus on what the teacher deems important.  Traditional assessment is a "moment in time glimpse" of a learner's ability to perform a task or set of tasks. It does not account for any external forces that may be affecting learners' ability to demonstrate their skills. … In addition, portfolio assessment allows learners to demonstrate the knowledge they felt was crucial to their learning experience. Through properly constructed and thoroughly documented portfolios, learners can chronicle the moments of discovery that they underwent during their learning journey.  (Herman & Morrell, 1999, p.86-87)

 

The process of developing a showcase or growth portfolio is of great value to both the student and the educator.  There are few arguments in the literature that deny portfolio assessment has many advantages over traditional assessments especially in their role as teaching and learning tools and as ways of authentically validating student academic achievements.  Criticisms of portfolio assessment tend to revolve around the more substantive issues of “reliability and validity,” including whose work is it, and “time and effort.”

 

Koretz (1998) alludes to problems with validity inherent in portfolio assessment,

 

Portfolio assessment has attributes that make it particularly appealing to those who wish to use assessment to encourage richer instruction - for example, the 'authentic' nature of some tasks, the reliance on large tasks, the lack of standardization, and the close integration of assessment with instruction. But some of these attributes may undermine the ability of the assessments to provide performance data of comparable meaning across large numbers of schools. One size may not fit all .portfolio scores are used to gauge students' proficiency as writers. The question of validity in this case is simple in principle: to what degree does a given score on a writing portfolio justify the intended inference about the student's proficiency? An examination that offers good support for one inference may provide weak support, or none at all, for a related inference. (p.310, 312)

 

Until significant inconsistencies between students’ portfolio-based responses to classroom learning activities and the more objective measures used to test all students’ knowledge of core concepts are resolved, e.g., the student who receives an “A” on his written essay but scores in the lowest quartile on the Iowa Basic Skills Test, critics will always have legitimate arguments that question the substantive nature of portfolio assessment.

 

The literature also signals that portfolio assessment is time and labor intensive to produce, monitor, support and assess.

 

…educators need to take a critical look at both the demands of portfolio assessment and the nature of their own programs before plunging into this new method of assessment.  Portfolio assessment is far from a panacea for the limitations of standardized testing, and may in fact create problems of its own. An appropriate context for the development and use of portfolios is essential, and this context may not be provided … (Hayes, 1997, p.173)

 

These two criticisms of “reliability and validity” and “time and effort” are based on the old paradigm of portfolio assessment.  However, the use of webfolio systems in education is a paradigm shift that renders these criticisms invalid.  When the paradigm shifts to student-centered webfolio use the “time and effort” criticism is conceded as part of the process of teaching and learning.  And the webfolio technology dramatically increases the educator’s productivity when engaged in portfolio assessment activities. The swing to webfolio system use is a paradigm shift to student-centered teaching and learning.  For example, Gathercoal, Love, Bryde and McKean (2002) point out that the criticism regarding “time and effort” is a substantive problem only when instructor-centered approaches to teaching and learning are practiced. 

The next several years will see it routine for students to place their completed course assignments on the WWW for faculty to access.   Most will not deny this; yet, implicit in the acceptance of this new situation may be an incorrect vision that the webfolio will be a traditional portfolio, simply digitized. With this incorrect vision in place, faculty will dismiss the proposition of a webfolio as yet another failed attempt at integrating technology into the “true” culture of education.  Unless there is a shift in the educational unit’s culture, the unit’s educators run the risk of becoming a leading force in the “neo-Luddite” movement Kurzweil predicts will grow in the year 2009. (p.31)

 

When institutions and educational units make the paradigm shift and the educational community values and promotes authentic assessment, webfolios tend to proliferate as they are used to assess and support teaching and learning simultaneously; and the “time and effort” needed to produce, monitor, support and assess student webfolios simply becomes a natural extension of the teaching and learning process.  It is merely another part of the educational culture of the university or college and it is not even considered a problem.

 

Criticisms surrounding the “validity and reliability” of portfolio assessment are best addressed by investigating the paradigm shift from traditional portfolio assessment to webfolio assessment, evaluation and reporting at once.  Today, standardized tests are driving modern-day educational practices (Koretz, 1998).  Webfolio systems provide a viable alternative to these standards-based tests.  When webfolio systems are fully and properly implemented professional educators can do away with standardized tests in favor of webfolio systems that enable standards-based, authentic assessment, program and instructor evaluation and reporting as the driving force behind educational practices. 

 

Webfolio systems facilitate authentic assessment practices complementary with portfolio assessment; program and instructor evaluation complementary with evaluative observations used to inform instruction in standards-based teaching and learning settings; and authentic reporting of student academic achievement complementary with the practice of sharing student showcase and growth portfolios. The innate ability of webfolio systems to unite authentic assessment linked to educational standards, evaluation of educational programs and instructors, and the ability to report in “authentic ways” academic achievement linked to educational standards to those who have a need to know, irrevocably alters the traditional paradigm of portfolio assessment and denies the old criticisms of “validity and reliability.”  It is this substantive improvement, recognizing and valuing the intrinsic links between portfolio assessment, program and instructor evaluation and the reporting of academic achievement that fortify the promise webfolio systems hold for being the next great innovation in education.

 

Intrinsic Links between Assessment, Evaluation and Reporting

 

Traditionally, educators have differentiated between assessments, evaluation, and reporting, rarely making any formal interconnection between the three.  However, these three educational practices are inseparable (Gathercoal, 1995).

 

Assessment is generally viewed as an educator's professional judgment of a student’s academic achievement in relation to the form and content of a course and its intended outcomes.  It is generally accepted that educators are obliged to convey individual student assessments that are accurate and truthful statements about academic achievement.  In fact, educators have a professional, ethical, and legal responsibility to convey accurately and truthfully their knowledge about their students' academic achievement to those who have a need to know (Gathercoal, 2001). However, accuracy and truth are often confounded by intrinsic links between assessment, evaluation, and reporting.

 

Evaluation is the process of determining the effectiveness of an educator, a course, unit of work, or particular teaching strategy and it includes making value judgments that are based on data derived from student assessments. Every good educator makes decisions about future learning experiences based on students' academic achievements and the significance of those achievements.  For this reason we argue that evaluation and assessment are partners in determining course content, structure, and strategies for instruction as assessments are continually used to develop and inform future curriculum implementation and teaching/learning strategies.  Together, assessment and evaluation act as a compass guiding the course toward its ultimate aim and the lesson toward its objective. This intrinsic link can skew truth and accuracy in student assessments since instructors--on some level of awareness--know that their student assessments are influenced by common perceptions of the curriculum and its implementation, i.e., assessments are a reflection of the instructor and his or her instruction.  So, when students flunk courses, ever-present in the minds of educators is the question, "Is it the student who failed the course or the course that failed the student?"  This chicken and egg question tacitly affects student assessments.

 

The reporting of student academic achievement is also intrinsically linked with assessment and evaluation.  Reporting provides information about a student's academic achievement and it affects the student's future educational and employment opportunities.  Secondarily, reporting indirectly provides feedback that is used to make decisions about (evaluate) the instructor and the course of study.  Students’ academic achievement reports can affect future staffing, levels of student participation, resource allocation, and perceived need for improving curriculum and instruction.  It is this secondary function of reporting that can confound the truth and accuracy about students' academic achievement.

 

In order to divine the truth and report accurately about student academic achievement, teachers necessarily divorce the report (the grade or the comment) from curriculum and implementation (the teaching strategies used in class, the course content, and the methodologies used for assessment and reporting). The need for such a separation between the curriculum and its implementation and the reporting of student academic achievement is tacitly understood. Our society silently concurs that teachers operate on a number of assumptions regarding assessment, evaluation, and reporting; teachers must assume that (a) the course they are teaching is politically correct, (b) their teaching strategies are educationally sound, (c) their intended outcomes are achievable by all students in the class, and (d) the reported student assessments accurately depict the student's academic achievement at the time. These assumptions are generally shared throughout our society. (Gathercoal, 1995, p.59)

 

Rather than considering assessment, evaluation and reporting in isolation from one another, education may be better served by thinking of each practice as part of a holistic process that provides meaningful information to those who have a need to know about students’ academic achievements, the instruction and the instructors, the course of study, and the educational environment in which the learning occurred.

 

Professional educators have a legal, ethical, and professional responsibility to communicate accurate student assessments to a wider concerned audience.  Given this responsibility, it is probably wise to base assessment, evaluation, and reporting practices on sound educational principles that reflect and dignify the student's academic achievement.  Gathercoal (1995) articulated seven principles of assessment that challenge many traditional assessment practices and invite professional educators to think holistically about assessment, evaluation, and reporting.  He suggests educators should:

 

1.      Focus on learning and academic achievement. Use assessment practices that contribute to students' learning…

2.      Provide for equal opportunity. Ensure that student assessment practices are inclusive of class, race, age, gender, sexual orientation, and disability…

3.      Make sure that assessment practices and the values of the discipline are congruent. Scholarship is important, and assessment, evaluation, and reporting should reflect the value that schools, colleges, departments, and faculty place on learning in every discipline…

4.      Recognize limitations. All assessment, evaluation, and reporting practices will have limitations; acknowledge them as they arise and, when appropriate, spell them out in your course syllabus…

5.      Be supportive of relationships. Highly competitive assessment practices tend to adversely affect relationships…

6.      View the student as an active participant in the assessment process.  Invite and value self-assessments and act on them as bases for assessment, evaluation, and reporting…

7.      Report student assessments in a consistent and meaningful way. The report should communicate to a wider audience the student's academic achievement, be consistent with the assessment practices employed, and be meaningful to all who need to know. (p.60-61)

 

Webfolio systems are ripe to help professional educators embrace and put into service a holistic approach to assessment, evaluation and reporting that is sensitive to these seven principles.  With the innate ability to generate for analysis and synthesis, student assessments, information about the course of instruction and educational standards, webfolio systems allow students and professional educators to report to an interested and concerned community about academic achievement and teaching and learning, all within the context of authentic assessment and standards-based evaluation practices.  Such a system for assessment, evaluation and reporting may well revolutionize the way we do education.

 

ProfPort Webfolio System: Assessment, Evaluation and Reporting Example

 

The Faculty-Side of Assessment, Evaluation and Reporting

 

The ProfPort webfolio process begins with the faculty preparing course and unit content for their students before they arrive for the semester.  Faculty generally only need to prepare a complete course or unit once as getting ready for an upcoming semester is usually a simple matter of copying their course management material from the last semester to the new semester.  Faculty simply take a few minutes to select materials to copy and then have the system copy those materials and learning activities from a previous semester to the new academic semester.  Once copied, faculty may add new and revise existing materials as appropriate.

 

Figure 1. displays links (highlighted) in the ProfPort Webfolio System that allow faculty to import syllabi from previous semesters, or from other faculty’s course management materials, and copy selected learning activities (competencies, assignments, standards, etc) from previous semesters, or from other faculty’s course management materials.

 

 

Once materials are copied, it is an uncomplicated transition to update the syllabus and learning activities to meet the needs of the new students.  In this way, the time faculty spend preparing the ProfPort Webfolio System for students is never wasted, especially if they will be teaching the same course each semester or if there are several faculty teaching different sections of the same course.  If there are several faculty teaching different sections of the same course, they can meet and prepare a model course that contains a common syllabus and common learning activities. Once complete, the model course, including syllabus and learning activities, may be copied to other faculties’ courses in the webfolio system.  This is a great tool for ensuring equal opportunity (assessment principle 2) between sections of the same course.

When developing a new course or revising syllabi and learning activities in an existing course, faculty type or copy and paste their syllabi information, along with learning activities into appropriate sections of the ProfPort Webfolio System.  The ProfPort Webfolio System allows faculty to use a built-in web-based “What-You-See-is-What-You-Get” (WYSIWYG) editor that is as simple to use as a word processor. 

Figure 2. shows the “What-You-See-is-What-You-Get” (WYSIWYG) edit box that faculty use to create original content, revise existing content, or copy and paste from a word processing program or other multimedia sources right into the ProfPort Webfolio System.

As illustrated in Figure 3, faculty also tie each learning activity to learning categories, assignment types, curriculum standards, and program goals. In addition, each assignment faculty place in the system includes a brief description of the actual task along with sections providing additional assignment detail, pointers to helpful Internet resources, and criterion referenced measures for assessment (a rubric). 

Figure 3. displays the caption and metadata that faculty tie to the learning activity students will respond to as they work towards a demonstration of mastery of the course content.

 

The Profport Webfolio system supports viewing assessment, evaluation, and reporting as part of the holistic process previously described and works within the framework of the seven assessment principles set forth by Gathercoal (1995).  For example, students place work samples in response to learning activities in their webfolios.  Context for the student’s work sample, the learning activity and syllabus, is maintained by the webfolio system and linked to the work sample.  One or more individuals or groups assess the work sample in conjunction with the learning activity.  Assessment takes the form of feedback and comments stored in an on-line log, scores on individual rubrics, and standardized scores on departmental rubrics.

 

Students view the assessment responses on-line and in conjunction with their work sample and the leaning activity.  A learning dialog is created when the student responds to the assessment by improving their work sample.  The improved work sample is again assessed and new comments are added to the assessment log.  This dialog may continue until the student feels they have completed their work and makes no further changes to their work sample or the faculty completes their assessment and “locks” the work sample from further modifications.  Assessment principles 1, “Focus on learning and academic achievement…”, 3, “Make sure that assessment practices and the values of the discipline are congruent…”, 6, “View the student as an active participant in the assessment process…” , and 7, “Report student assessments in a consistent and meaningful way...“ are supported and exemplified by the webfolio assessment process described.

 

The linkage inherent in assessment, evaluation, and reporting are further supported by the Profport Webfolio System when learning activities and work sample assessment results are reported in conjunction with the learning taxonomy, educational standards, and program goals that are tied to the learning activity and the student work samples as illustrated in Figure 3.

 

There is a great deal of flexibility in defining the assessment rubric as educators are simply provided with a blank WYSIWYG edit box for developing and communicating their rubric for each learning activity and a blank likert scale type assessment tool for summative scoring of student work at the program or departmental level. 

 

Figure 4. exhibits two rubrics generated in the blank WYSIWYG edit box by faculty for two different assignments.

 

              

 

Figure 5. exhibits the likert scale type assessment tool for summative scoring of student work at the departmental level.

 

 

The flexibility allowed for educators to design, develop and communicate their own assessment rubrics or criteria does much to complement assessment principle 1, “Focus on learning and academic achievement. Use assessment practices that contribute to students' learning.”  It also helps educators to be mindful of and sensitive to assessment principle 2, “Provide for equal opportunity. Ensure that student assessment practices are inclusive of class, race, age, gender, sexual orientation, and disability.”  With “template” rubrics and assessment criteria, faculty are sometimes limited to a few choices that may not be appropriate for all students.  The lack of “embedded structure” actually provides faculty with more professional autonomy that will work to the benefit of all students.

 

The ProfPort Webfolio System maintains faculty and student content both as it existed for previous semesters and as it exists for the new semester. This assures that someone looking at a student’s work sample (artifact) several years later also will be able to see the actual assignment as it existed when the student created the artifact.  As an intended by-product of the assessment and evaluation process, faculty’s syllabi and course work continuously improve with the updates and curricular modifications over time. 

 

The Student-Side of Assessment, Evaluation and Reporting

 

The ProfPort Webfolio System allows students to house and display both growth and showcase portfolios.  A student's webfolio contains a myriad of multimedia artifacts (work samples) produced over time.  Consisting of formal course assignments, student life activities, and career planning initiatives, a student’s webfolio demonstrates mastery of program and course standards or competencies as he or she builds an organized omnibus of multimedia artifacts on the World-Wide Web.  Each student’s webfolio illustrates how he or she is interweaving formal course work, career planning, and student life activities and developing his or her own unique educational experience.  They use a built-in web-based What-You-See-is-What-You-Get (WYSIWYG) editor that is as simple to use as a word processor.  Students determine what work samples are displayed in their own webfolio and what groups may view which work samples in their webfolio.  They can also add their own captions and corresponding work samples for learning and out-side of class activities that are not formally provided as part of a course to their webfolio.  The new captions are added within the context of a course or program they have registered for in the webfolio system.  Program standards, course competencies, and guidelines enhance the student’s work samples and all are automatically organized as contextual reference points and viewable as links attached to the student’s artifact or work sample. 

 

Figure 6. exhibits a typical student webfolio as viewed in the “display mode” on the student-side of the ProfPort Webfolio System (please note the student’s name and picture have been altered).  Also displayed are the links that allow students to generate “New Caption,” “Modify Caption,” and “Delete Caption.”  The student’s work sample is given context for assessment, evaluation, and reporting by the links to the learning activity that generated the work sample (Activity/Assignment) and the course syllabus. 

 

As indicated above, students have a great deal of autonomy in determining who will view what work samples in their webfolio.  This autonomy in reporting speaks directly to principles of assessment 3, 4, 5, 6 and 7.  The linked contextual information helps to inform others about the limitations of assessment (principle 4), convey the value of the assigned work (principle 3) and also helps to report in a meaningful way (principle 7) by providing information about the course, via the syllabus, and the assignment description, help, resources and assessment criteria specified by the faculty.  The student’s ability to determine what items will be viewed in the webfolio speaks loudly to principle 6, that students will have some say in the assessment process.  Also, allowing students autonomy in determining who will view their webfolio artifacts works well to address principle 5, as it is supportive of relationships and does much to reduce competition among students.  While student autonomy is one great advantage of the ProfPort Webfolio System, it is nothing without the assistance, guidance and mentoring of faculty and administrators who set up the system.

At the beginning of the semester, students add the new courses and units of work prepared for them to their ProfPort webfolios by selecting from a list of faculty-generated courses and units of work.  When registered for the course, the student can then see every assignment, activity, and project (learning activity) listed in his or her webfolios table of contents.  That is, unless the faculty person has decided to use the automatic scheduling feature to hide the assignment from students until some later date (See Figure 3., “Competency will appear in Webfolio”).  In that case, the assignments will appear in the students webfolios throughout the semester on the faculty’s predetermined dates that indicate when specific assignments should appear.  Either way, when an assignment appears in the table of contents the description, models, resources and rubric for assessment for each assignment are just a click away for every student in the class. 

Invariably at least one student asks if they can see examples from past students’ work.  Some students appear surprised at the glee with which faculty grant these requests, as they simply call up past students’ work from the ProfPort Webfolio System and orally comment on the qualities of the work done by previous students.  Although more subtle in approach, the teacher’s goal is the same as that of the early twentieth century industrialist who took a piece of chalk and scrawled the night shift’s production number on the factory floor for the morning shift to see how productive they had been that night.  By sharing past students’ work with current students, faculty members convey and raise expectations as students will be motivated to meet or beat the quality of the previous piece of work.

 

What happens to the webfolios of graduates?  After just a few assignments, students see the potential value of the webfolios they are producing and want to know how long they will be kept after they graduate.  Those graduating understand that their webfolios are powerful marketing tools when they look for their first job.  They particularly relish the capability to individually set access to each section of their webfolio so that their instructors can view the complete portfolio, but a recruiter, mentor, or parent may only see a subset of work samples that the student wants them to view.  So how long will they be kept?  The webfolio vision is that they should be kept forever.  Cyberspace is cheap and it’s getting cheaper.  Have you filled up your hard drive?

 

Assessment, Evaluation and Reporting

All assessment comments and scores are maintained for each student on each artifact over time.  Both quantitative and qualitative assessments are available.  Since standards are associated with learning activities and with student work samples, the assessments attached to the student work samples can be used to provide a wealth of information regarding program and instructor evaluation.  This simple act involving a few mouse clicks combined with the assessment scores faculty assign to each student’s response to the learning activity can be used to address critical assessment and evaluation questions, like: 

ü      Overall, have program goals and standards been met or improved? 

ü      Are there holes in the fabric of the curriculum?

ü      Have specific program goals and standards been met? 

ü      What is the inter-rater reliability for scoring assignments by different faculty?

ü      Are individual students meeting goals and standards? 

ü      Is the curriculum designed for success? 

 

The ProfPort Webfolio System allows a system administrator to export selected information that is needed to answer these and other critical assessment and evaluation questions.  This information can then be imported into SPSS, SAS, EXCEL, and other analysis and graphical presentation packages. 

 

Figure 7. presents some of the over 5,000 records of metadata that were exported from the ProfPort Webfolio System and imported into EXCEL.

 

 

Graphs can be generated to indicate the percent of student work assessed below, meeting, and exceeding faculty expectations for multiple years.  Charts can be produced that show how mastery of a standard is being developed throughout the curriculum.  The visual impact immediately conveys whether there is proper scope and sequence within the curriculum to meet state and institutional standards and whether the curriculum is helping students to achieve those standards.

 

Figure 8. shows a graphic representation of metadata collected in and exported from the ProfPort Webfolio System and imported into EXCEL and presented in PowerPoint slides.

 

 

Conclusion

 

This article has argued how it is possible that the ProfPort Webfolio System and other webfolio systems like it can challenge the mystique and authority of standardized tests as the guiding force behind education today.   As Gathercoal, Love, Bryde and McKean (2002) mooted in their article, On implementing web-base electronic portfolios, “A well-designed curriculum embedded in a webfolio system, conveying academic standards, appropriate resources and providing vehicles for faculty mentoring, enables student’s development and upkeep of developmental, growth and showcase portfolios at once.  A web-based electronic portfolio system acknowledges and appreciates the intrinsic links between student assessment, faculty and program evaluation and the meaningful reporting of assessments and evaluations to interested third parties.” In the capable hands of professional educators who have the best interests of their students at heart, webfolio systems may permanently transform assessment, evaluation and reporting to comprise authentic assessment, evaluation and reporting.

 

References

 

Educause: National Learning Infrastructure Initiative. (2002). Electronic Portfolios (NLII 2002-2003 Key Theme). http://www.educause.edu/nlii/keythemes/eportfolios.asp

 

Gathercoal, F. (2001) Judicious Discipline (5th Ed.). San Francisco: Caddo Gap Press.

 

Gathercoal, P. (1995).  Principles of assessment.  The Clearing House, 69(1), 59-61.

 

Gathercoal, P., Love, D., Bryde, B. & McKean, G. (2002). On implementing web-base electronic portfolios. Educause Quarterly. 25(2) 29-37.

 

Gathercoal, P., Love, D., & McKean, G. (in press). Portfolios to webfolios and beyond: Levels of maturation. Educause Quarterly

 

Hayes, E. (1997). Portfolio assessment in adult basic education:  A pilot study. Adult Basic Education. 7(3) 165-173.

 

Herman, L. P. & Morrell M. (1999). Educational progressions:  Electronic portfolios in a virtual classroom. T.H.E. Jounal. 26(11) 86-89.

 

Kilbane, C. R. & Milman N. B. (2003). The digital teaching portfolio handbook: A how-to guide for educators. Allyn & Bacon.

 

Koretz, D. (1998). Large-scale portfolio assessments in the US:  Evidence pertaining to the quality of measurement. Assessment in Education. 5(3) 309-334.

 

Simon, M. & Forgette-Giroux, R. (2000). Impact of a content selection framework on portfolio assessment at the classroom level. Assessment in Education. 7(1) 83-101.