Hi, everyone!

Jackie Hoermann-Elliot shared a link to the Composition I/II OER text used at TWU, and I thought I'd go ahead and include that link here as well: https://pressbooks.pub/firstyearcomposition/

Hope you enjoy it!

Jessica

Yes, thank you, Jessica!  I enjoyed Jackie's talk very much and appreciated her taking her time to explain her experiences with OER.  It was very helpful!  The link to the press book is icing on the cake, so to speak, a delicious example of her work.  I need examples and case studies to catch up with the rest of our class!

I like the VIVA Open OER Evaluation Checklist.  It covers the following criteria with specific items to look for when evaluating the materials:

Clarity, Comprehensibility, and Readability

Content and Technical Accuracy

Adaptability and Modularity

Appropriateness and Fit

Representation and Diversity

          Under Representation and Diversity, there’s a comment about names that appear to have an ethnic origin.  As someone who has a last name that appears to be Hispanic, I can say that stereotypes exist from appearances.  My husband’s family doesn’t really have an explanation for the name of Baldon, though they have speculated.  As a result of the name, we get mail, emails, telephone calls, and commercial and political ads aimed at Hispanic people.  We don’t mind, it’s just interesting that somewhere someone is making an assumption based on our name and acting on that assumption.  

Under Rubrics for Evaluating Open Education Resource (OER) Objects, I appreciate the comment: “In general, the rubrics should be applied to the smallest meaningful unit” and that the rubrics are rating “the potential, not actual, effectiveness of a particular object in a learning environment.”  I like the idea of rating the potential of use because my plan would be to take that potential and work with it, not just copy it.

I really like the ISKME Accessibility Checklist because it gives the nuts and bolts of accessibility. 

Equity Rubric for OER Evaluation from Branch Alliance for Educator Diversity by Grotewold, Grunzke, Ianniello, Jiles, & Mayo (2021) is a comprehensive, well-cited manual with an interactive table of contents.  I bookmarked this link for later convenience.

WASHINGTON MODEL RESOURCE:  Screening for Biased Content in Instructional Materials is another gem with instructions for presenting cultural diversity and accounting for the phenomenological circumstances of the students.

The CRAAP Test (CRAAP is an acronym for Currency, Relevance, Authority, Accuracy, and Purpose.) is a way of evaluating sources.  The video is brief and to the point with understandable graphics:  https://www.youtube.com/watch?v=EyMT08mD7Ds&t=33s and directs the viewer to be aware of currency, relevance, authority, accuracy, and purpose.  This is the same advice that I’d give students to watch out for being influenced by propaganda.

I the quote that you cited regarding the potential, not actual, effectiveness of a particular object. I thinks that it is fairly common to hear others criticize the available OER materials without really realizing they have the ability to modify most OER according to the needs of their class. I suppose we are only limited by our creativity, imagination -- and sometimes available software. 

I also appreciate your descriptions of each of the checklists. Your descriptions would also be a valuable resource for someone trying to decide which link to start with based upon the needs of their particular OER item. 

Hi, Suz!

Thanks so much for your post! I agree with your point about the ISKME Accessibility Checklist--it's really helpful for faculty. When I'm on the hunt for OER, sometimes I get so caught up in finding something that aligns with learning outcomes (or that's really 'open') that I forget to check for accessibility. It's a big deal, though, so I like evaluation rubrics with those elements folded into the process. It reminds me that it's better to pick something that checks all the most important boxes the first time, rather than integrating it in my course and then having to go back and fix it later.

Awesome work! I appreciate it.

Jessica

I hope that I'm doing this right. I used the VIVA OER Evaluation Checklist. It was quite honestly the simplest to quickly read through. However, it did not specifically address ADA compliance but I assume that this should fall under representation and diversity. I did change the order a bit in terms of the importance of the evaluation criteria. I thnk that Adaptability and Modularity are the most important because it enables me to adapt and revise any issues with any of the other criteria. 

Here is my list in order of importance. 

• Adaptability and Modularity: This provides the flexibility needed to adapt and revise any materials according to the needs of my class. For example, I am able to modify, rearrange, and select only the sections that I find appropriate for my course objectives and teaching style. 
• Content and Technical Accuracy: While adaptability is important, I would place having accurate content as the second most important criterion. Having accurate content reduces the time and effort required to make corrections and reduces inadvertently missing errors within the content. 
• Representation and Diversity: I feel that this area falls under content accuracy and is equally important. It is vital to ensure that diverse views, experiences, and contributions are accurately represented. However, this area might possibly be more difficult and time-consuming to accurately revise. 
• Clarity, comprehensibility, and Readability: must be appropriate for the target population in order for effective learning occur. However, accurate materials can be adjusted for clarity, comprehensibility, and readability. 
• Appropriateness of fit: I have placed this criterion last because it is a bit subjective. I have located quite a few materials that didn't quite fit my needs or perhaps possessed only a few qualities or components that I wished to utilize. Materials that are adaptable can be modified fit each person's needs. 

I think the order of importance is important to think about as you've pointed out here.  I assume most people would apply a rubric in the order given.  If you alter the order would people pick or judge things differently?  I think about this "problem" with respect to exams and assignments too.  We often assume the order questions are asked doesn't matter much, but I wonder if that is a good or bad assumption.

I really like the VIVA OER Evaluation Checklist as a summary of the evaluation points. It is pretty short and covers all of what I would consider the most important. I'm not sure if I can think of any ways to improve it at this point. All of the others provide a lot more detail on particular aspects of evaluating a source, but realistically speaking, I won't have the time to go through that much evaluation before deciding whether or not to use a source. As I get further along in my OER journey and begin to perfect my course and materials, it may be helpful to review more of these to see how the resources could be improved or which ones may need replacing with better alternatives.

Hi, Peter!

Thanks for your post. You made some good points--I agree that some of the rubrics are really, really thorough, so much that they seem a little daunting. For those, maybe a 'divide and conquer' approach would work best? I'm thinking there about something where a group of faculty (and instructional designers, librarians, etc.) might use longer rubrics to vet potential materials. The person who's the best at accessibility checks, for instance, might take that part of the rubric and evaluate materials, while the person who's more comfortable with assessment and SLO's might take that part. Especially for folks at MCC, that seems like a way to make sure all sources are vetted really well without getting too bogged down in work.

I appreciate your post! Thanks!

Jessica

I am rewriting because I misunderstood the instructions. 

I do find the VIVA OER evaluation checklist to be the simplest to use and probably best for someone who is beginning the process of using and assessing OER content while Open Textbook Library Review Criteria (adapted from and very similar to the BC Open Textbook Accessibility Toolkit) is a very helpful tool for quickly checking the basics for format and ADA compliance.

The CRAAP test is also a very quick and easy tool to use with the obviously helpful mnemonic. I think that it does an excellent giving a catchy mnemonic and checkpoint list to criteria that any would hopefully know to check anyway.

To ensure representation and diversity are adequately met, the Washington Model Resource provides a very helpful screening tool that begins on page 6 that can be used to determines. 

• Representation in terms of roles and character traits (.e., genders and gender identities, race, disabilities), 
• Multiple perspectives and contribution in terms of biographical, contemporary, and historical
• Multicultural representation: appropriate representation of culture, languages, traditions, beliefs, values, and customs.
• Appropriate and accurate imagery and language.
• Family representation that depicts a variety of family situations
• Teacher guidance in terms of teaching strategies and responses and providing a partnership with students and their families to enhance lessons.

If someone required a quick way to get assess OER, the VIVA OER evaluation checklist and either the Open Textbook Library Review or BC Open Accessibility Toolkits would be the tools that I would recommend. They are easy to use and not too overwhelming, especially for those who are just beginning to use OER.  

Hi, Shannon!

Thanks for your post. I totally agree with you about the VIVA OER rubric. It's really helpful, and it's not so long that you feel overwhelmed going through a whole pile of potential materials. The BC Open Accessibility Toolkits are also fabulously helpful. Accessibility is something I feel saves loads of times if you find resources that 'check' those boxes. There's so much OER out there, and sometimes I'll find something that seems like a great fit, but it doesn't have true lists, or the images need alt-text, or something like that.

Great work! I really appreciate it.

Jessica

I think I would just use the VIVA OER Evaluation Checklist as it is clear and concise.  It does need "follows all ADA guidelines" or something similar added to the Representation and Diversity section.  It seems this has been overlooked in many of these rubrics.  I also like the CRAAP Test because again it is concise.

I hope I'm doing this right. It seems like the responses from past OER Essential classes focused on just the rubric, not how a specific OER text fits that rubric. Here's the rubric that I use and the resources that helped shape my thinking:

1. The VIVA OER Evaluation Checklist has "Clarity, Comprehension, and Readability" as its first line item. I would also have this first. The text has to be on the appropriate reading level, the writing style has to be clear, and everything is clearly organized. I would also include graphic design in here which would cover text and font size, and other stuff like that. I remember Bruce Ballenger's The Curious Writer having a bit of a busy page back in the day.

2. I don't value currency as much in a Composition book. New editions of textbooks often update essays to maintain relevant issues which is important, but those can easily be offset with FREE (not OPEN) resources. The same is true for citation methods (MLA, etc.) What I do value, though, is relevancy. I want a textbook to touch on (and care for) similar things that I care for (i.e. genre writing). This category would also cover standards as outlined by Achieve OER Rubric, at least for me. 

3. Content Accuracy (via VIVA). While a book's supplemental essays may be a little dated, the other aspects of the book needs to be spot on. Good, effective pre-writing techniques. Accurate components of types of writing. Good descriptions of elements of rhetoric. 

4. Adaptability and Moblity (via VIVA). Having a source be CC-BY, or some variation that allows for remixing is really important. While I care about relevancy, as described above, the ability for a textbook to hit everything that I was is highly unlikely. I also care about its ability to be imported and integrated with D2L, which is one of the major reasons I went with Openstax.

5. Diversity. The books needs to have diversity of authors and diversity of ideas. The text needs to have the trust of its audience, and since many of my students are more conservative - - - there needs to be more than just liberal authors. The book I used from McGraw-Hill for 1302 last semester was definitely responding to the Trump presidency. I was surprised by many of the essays in the book because they used incendiary language that would easily turn off some of my students. 

Do we need to apply this rubric to a specific text? 

Hi, Chris!

Thanks for your post! I agree with your points about Composition books not needing to be super current, provided they've got good prompts and overviews of the writing process. I also thought you point about diversity of perspectives was good as well. Some books privilege a type of student audience that's more typical at four-year institutions, and I think that bias is reflected in the language. There's a tendency to assume most students are middle-class and have similar values, for instance. When I've taught dual-credit classes in different areas or worked with international students, these approaches really fall flat. Having more diverse perspectives and stories is important, and that'd part of why the remixing/adaptability piece seems like a big deal to me.

Great job!

Jessica

Hi, everyone!

Just like the majority of this forum participants, I really like VIVA OER Evaluation Checklist because it looks pretty accurate and comprehensive, but there are other tools mentioned above that grabbed my attention. Looking through all of them, I will probably single out the following criteria (in the order of importance):

1. Clarity, Comprehensibility, and Readability. This criterion is shown as the first in VIVA OER Evaluation Checklist, and I would give it the first place as well. I know that my students would like to use something simple and clear. In fact, I am the same way: it is really easy to lose motivation when resources seem too complicated or confusing.

2. Organizing Content. This criterion is shown in ISKME Accessibility Checklist, and I cannot underestimate its importance. When content is properly organized, with headings and subheadings, and a discernible structure, it is easy to perceive information and retain it. I noticed that students may get intimidated by an excessive amount of textual information, especially if the text is heavily loaded with specific terms and the font is hard to read, but they are more open to visual presentations, images, and emojis. So a clear organization and an attractive layout may make content more appealing and engage students' interest.

3. Content and Technical Accuracy. This is the second criterion shown in VIVA OER Evaluation Checklist, and I think this is an essential point to consider because after looking at the "form," we should always go deeper and evaluate content, its accuracy and appropriateness, and make sure it aligns with our learning objectives. Also, I would like to provide my students with good-quality resources and would not feel comfortable if there are obvious typos, grammar, and spelling errors.

4. Adaptability and Modularity from VIVA OER Evaluation Checklist is another essential point to consider while evaluating OER. Each group of students is unique and different: something that works for one student may not be effective for another one. So I would like to be able to modify my resource and adapt it to the individual needs of my students.

5. Cultural Sustainability. This criterion is included in Equity Rubric for OER Evaluation developed by the BranchED team. Using the principles of Equity and Inclusive Pedagogy, the authors identify four essential dimensions of equity: learner-centered, critical, culturally sustaining, and universal. So a culturally sustainig resource "draws from a variety of cultures, contexts, and groups" and provides inclusivity for all students no matter the origin or race. This criterion ensures the resource does not exclude anyone and fosters cultural pluralism. I think our population of students is becoming more diverse. I do have more international students every semester, and providing all students with resources tailored to their unique cultural identity will help create an environment conducive to learning.

Thanks for the rubric, Anna. I find the idea of "cultural sustainability" interesting. It's a more expansive criteron than my 5 which I just described as diversity. I wonder what makes a composition book "universal." I keep thinking about the book I used for Comp II last semester, Read, Reason, Write, where many of the selected essays were almost belligerent and would probably be a relic in ten years. 

Hi Christopher,

Thank you for your comments. Your question got me interested, so I decided to go back to Equity Rubric for OER Evaluation and research it a little bit more. In the appendix the authors do specify what "universal" means: it is actually "universally designed for learning," so it looks like I overlooked it, and I apologize for this inaccuracy.

Basically, this criterion emphasizes flexibility, options for customization and clarification, and "internalization with options for comprehension through the supplying of background knowldege, highlighting of patterns, and relationships." I think the last point is really interesting and relates this criterion to the previous one - "cultural sustainability."

Yes, you have mentioned "Organizing Content" in your post, I know it is very important, well organized contents will lead us to teach well and students learn better. Even now in my MATH1314, college algebra, if they miss one step or unknow one tiny thing, it affects them a lot, they could not move on or they get lost easily, I shall pay more sttention on this part in the future when I make my OER course.

Hi Yumei,

Thank you for your commnet. I like it when everything is organized clearly and there is a noticeable structure or pattern that will lead to the next step and make it easier; however, I will admit, too much focus on organization is one of my weaknesses, too. Sometimes students will say, "that's too much structure, too much organization, no room for creativity," so I am trying to get a little bit more laid-back, allow more flexibility, and encourage creativity.

What I still do is trying to be consistent in the way I organize things, then as soon as students do the first essay, they grasp how everything is set up, and I think they feel relieved that things are getting easier and more understandable when the course progresses. They do comment that they really appreciate clarity and consistency, and even if they miss something, it is easy to navigate the course layout and find what they need.

Hi, Anna!

Great job on this post! I agree so much with your points about what each rubric brings to the discussion. For me, the point about content being effectively organized is also important for concerns about accessibility. If the document has headings and subheadings, there's a good chance those might also be formatted with styles, which is a big deal for screenreaders for students who are visually impaired. If those elements aren't included, the students can't really use the source. Your point about inclusive pedagogy is also spot on. When we work with students at MCC, there's always such a diverse mix of learners in the room. We have to pick content that will appeal to all of them, and that's a big challenge. It's possible, but it takes some conscious awareness of who our students are and attention to detail.

Awesome job!

Jessica

That is the beauty of OER in terms of the gaps you locate in an evaluation, is that even if a resource is lacking in some small areas, depending on the license, you can adapt and modify the resource to fill in some of those gaps. And in the spirit of OER, you might be able to collaborate with the creator to fill those gaps!

Hi All,

I think if I follow the rubrics in VIVA OER Evaluation Checklist， I can arrange the materials well for all my mathematics classes. At lease for now, since I have not create any course with all in OER yet.

I would keep the same order of VIVA Checklist.

For sure,  Clarity, comprehensibility, and readability is on the top of the list. Otherwise it is meaningless to make or create the contents and couses more troubles to students beside thye already face the challenge to learn the content. For mathematics, we have to use the same notations from the first page to the last page, consistent all the way.

Next, Content and Technical accuracy is so necessary to my math courses. I think I shall firmly depend on the syllabus from our department to decide the content and to make sure cover all and correct. We always start from definitions to reach to the theorems and the principles, mathematical skills are precise and accurate, and always follow the logical to deal with. This is the base for me to search for the OER resources.

For adaptability and modularity, actually, I have borrowed or used some great resource to my courses partially. It is great to have the adaptability to modify the contents to be used in the classes and also I can improve them on top of the materials.

Then I have Appropriateness and Fit just as in the checklist. The content must be in the level to fit our students to be able to read and understand. This requires how well we know our students and how deep we know the contents. Definitely we need to follow our syllabus to embed the content for the courses. I believe the freedom of education, and I strongly believe we need to follow our objectives for the courses.

Last I have Representation and diversity as in VIVA list. It is interesting to have some photographes from different ethnic, our students are from different places and there are more international students now.  It is a way to make students more included and engaged in the participation of learning activities. More sources from different locations in the world are a great idea when we make the courses.

Hi, Yumei!

Thanks so much for your post this week. You made some excellent points about the rubrics and how faculty can use them to make good choices for our learners.

I agree with your point about content and technical accuracy. When I was an undergrad, I remember using this one faculty member's book for second semester chemistry. He was a nice enough person, but he was also an alcoholic, and his textbook had a lot of errors (I mean, A LOT!). He wasn't always able to address those effectively when we did lecture, so it made studying for exams very difficult. Fortunately, I don't think we have the same problems at MCC with our faculty. At the same time, though, I feel strongly that having accurate content really matters for courses where there is a clear "right" answer to the problems included in the textbook and workbooks for students. If that content isn't accurate, it really frustrates learners and wastes their time (as well as that of instructors).

Excellent job!

Jessica