OER Core Elements Academy 2021

I like the Open Textbook Library format and I will add:

High quality Powerpoints: Powerpoints are covering the most important concepts and are easy for students to follow.

Awesome! Thanks for sharing! 

Donnie Kirk

Thanks for sharing!

I haven't formally used any of these rubrics before, although I did have a checklist I created a year ago at an OER conference. I don't know where that list is anymore. But when looking over OERs, these are the things I focus on:

1. Reliability/Credibility/Quality of Content

2. Scope of Content 

3. Organization & Design

4. Current or up-to-date information

5. Credentials of author(s) 

Some questions I consider as I evaluate for the above are the following: What does it look like? Is it formatted in a user-friendly way? Does it have headings and sub-headings that suggest the author(s) know what they are talking about and that make it easy to use and navigate? Does the organization of the material make sense to me for my course? Does the content show awareness of cultural and internal biases and avoids these? Is the content current or clearly informed regarding innovations or new ideas in the field/subject in question? Who wrote it or put it together (remixed it) and in what courses is it already being used? 

Hi, Melinda! 

I totally agree with all of your points here.  I want to just copypasta your posting because what you've described is essentially what I do, as well, when I'm looking at OERs.  

One thing that is often at the front of my mind when I'm sifting through OER resources is whether or not a resource is fairly comprehensive.  I have found it distracting, disjointed, and "hindering" to send students all over the internet hinterlands with various links to multiple resources, so I try to find one or two major resources that can be an ancor in the class and then add on ancillary or supplemental resources more sparingly.  I aim to keep the resource access for students as streamlined as possible.

(As an aside, "hello out there!" to my office mate!  I hope you are having a relaxing summer!)

Cheers!

-Mandy Palmer

Melinda and Mandy,

Agree with your comments. Similar to what I would look at but appreciate your list of questions to help formalize it.

Thanks,
Pam

Evaluation criteria to Assess a Resource

1. CRAAP Test - Since most OER resources are written by education professionals, the resource usually meets the criteria for Relevancy, Accuracy, Authority, and Purpose. What I need is Currency. If an item hasn't been updated within the last five years, it is not current and not relevant to our time. 

In addtion to passing the CRAAP Test, the resource must be easy to navigate. If I have to click more than three times to find my objective, the resource will frustrate users.

2. Open Textbook Library Review Criteria - As someone in the planning stages of writing a textbook, I find the criteria to be a template in how to successfully format a manuscript.

I teach psychology, sociology, and education courses. When vetting material, I personally use the CRAAP test. 

Currency: within past 5 years sooner preferred, but some foundational teachings need to come from older theoritical works to help understand modern topics
Relevance: Depending on the material I am examining and how it will be used  (Spark interest/teaching concept/ application of material) will vary.
Authority: Who created? Where is it located/published? Has it been vetted by others? 
Accuracy: I want to ensure that I am presenting both sides so students can make their own decisions.
Purpose: Most often, I look for multimedia to help present a point a different way. Many of our students at our community college choose not or cannot read textbooks. Many are first generation or ESOL students. I have found that students will watch a short video or work the material through various activites/avenues. This helps facilitate learning. 

I love using CrashCourse and Khan Academy in my classes to help reinforce concepts or ideas. This also allows "extra" help for students who struggle with "reading" or are auditory/kinesthetic learners. 

I have added currency - within the last 5 years to my criteria as well. That's a smart consideration!

Greetings All,

I have to say, I am a bit overwhelmed with all the information we have seen in the last session. I am new to the OER world, and did not expect to see all content and rubrics available—not to mention the experience I have heard people say they have had with OER! 15 years! Wow!

I currently teach speech communication courses at Vernon College in Wichita Falls, TX. I am more of a “guide on the side” instructor, providing content to students via the LMS for quizzing, then we assemble in a live class session for low stakes application activities that subsequently lead to high stakes performances. In a typical course, there are 45 opportunities for assignment evaluation (quizzing, participation activities, application assignments, and/or high stakes performances).

COVID offered an opportunity to learn how to present my content in virtual environment. In my virtual classroom space, students read/view assigned content (textbook or videos), quiz over assigned materials, create of develop a short project (variety of e-tools), post on the LMS discussion board, and reflect on and relate to the posts of others (must comment on no less than 3). Such low stakes activities then lead up to the high stakes live performances delivered via Microsoft Teams. Students sign up for presentation days/times and must be camera ready, front, center, staged, lit, and dressed appropriate for each session. Students present, comment on, and evaluate performances within each session.

Whether in a face-to-face format, or a hybrid/blended/virtual environment, I am satisfied with the textbook and ancillaries I use at our college. We have selected low cost yet comprehensive text from Fountainhead Press. I find that adequate ancillaries accompany the textbooks we choose. I have cobbled together a variety of sources that complement my content at the moment—many include trade articles on the importance of soft skills in a professional setting (communication, interpersonal skills, teamwork, leadership, problem solving, critical thinking, empathy, and/or conflict management) as well as TedTalks and other relevant YouTube videos that demonstrate such skills.

With that, 20+ years of face-to-face course development and teaching, and only several years of virtual/hybrid experience, I find am always looking for relevant activities and purposeful application exercises to complement course content in both realms. I am always in a state of development…and always welcome new ideas for development, implementation and evaluation.

For OER resources, I am not at the moment interested in finding free textbooks or full courses. I am mainly interested in relevant activities and purposeful application exercises to complement course content—specifically in my virtual environment. While I am creatively using a combination of assignments through our LMS and meeting for specific activities in small groups via Microsoft Teams, I am sure there are digitally native experts who have refined teaching and learning in such a manner. Please help J

When I do search for content, exercises and activities for my communication courses, I find the order I usually follow to evaluate follows the following:

Purpose and Relevance: In many areas of my course, whether it be face-to-face or virtual, I love incorporating multimedia within my content. Multimedia is a great tool for making the difficult, ambiguous and/or theoretical concepts come to life for students. Multimedia can present interpersonal dynamics, provide demonstrations, and offer tutorials. In essence, I am always hoping for multimedia that sparks relevance and engagement. Therefore—my first order of business is usually purpose and relevance.

Culturally Responsive. Whether I am selecting learning tools to complement my course or simple graphics for PPT slides or LMS illustrations, I am very aware of WHO is depicted in the illustrations. Have I found content and illustrations that contain a variety of genders, ethnicities, abilities, etc.? I want all to “see themselves” within the course content.

Current: Whether presenting communication dynamics, providing demonstrations, or tutorials—trade articles, multimedia, and other assorted OER content must be up-to-date. I find students relate to content examples when it is within 3 years old.  

Credible:  While purpose, relevance, and current, up-to-date trade and multimedia are essential, I am also conscious of the credibility of the source. Depending on its use, I stick with current and credible trade articles, current and credible TedTalk demonstrations,  and educational industry experts for LMS/Microsoft Teams embedded tutorials.

Accessibility. When choosing any OER related materials, I am becoming more aware of accessibility. Is it reader/audio/visually friendly?

I found the variety of QER rubrics to be a comprehensive reference to consider when choosing OER supplements. I instantly recognized the “Open Textbook Library Review Criteria” as two of the resources I saved in my “Communication Studies” folder are identified as reviewed by the rubric. Additionally, I shared with two of my superiors how comprehensive the rubrics are as well as the QER Commons resource collection.

Cheers to all on the journey!

Donnie Kirk

The Open Textbook Library Criteria are pretty good and comprehensive. The guidelines are easily to follow and adequate. Things the resource I like: Provide PDF, Hardcopy, and ONLINE formats for readers; the reviews have detail explanation; the affiliations of authors are provided. My suggested additional criteria are: state the numbers of problems and year of revised version if provided; targeted readers and pre-requisite for reading these resources in the website; accessibility formats of the resources: PDF, Online, Hardcopy, any Videos provided.

W.Y., 

Awesome commentary and tips! 

Thanks, 

Donnie

I agree with Donnie.  I like The Open Textbook Library Criteria as well because it is easy to use.  Also,W.Y's suggestions are very important additions in the evaluation.  

Hi, Donnie,

Your inclusion of the "culturally responsive" evaluation criteria is hugely important - I think that's a crucial element to keep in mind when reviewing available resources.  

As to your note about looking not for textbooks but for more specific applications, I appreciate that both the OER Commons and the OERTX Repository sites allow for searching for by "material type" in the advanced search.  I also often filter by the most recent resources, to see what's new.

Mandy!

Thanks so much for the responsea and the search tips!

Cheers,

Donnie 

This is excellent, if anything remote learning has taught us to think out side of the box. it is nice to see an online class that provids chances for activities and not just exams.  I did enjoy your evaluation criteria, thank you.

As a Quality Matters Peer Reviewer, I'm drawn to the Achieve OER Rubric because it is so complete. That being said, I'm not sure it's usable when reviewing many OER documents, so I would probably start with the Open Textbook Library Review and when I've narrowed down the content, use the Achieve OER Rubric.

I am specifically looking for OER content for a Project Management course and at this point, even exercises for my students would be welcome. I am also looking for content for some technical courses I am teaching and will use at least one of these rubrics for considering content.

Thanks for your insight.  I am not a QM reviewer, but I took the course recently and thought the Achieve OER Rubric was best to use since that is the push for our institutions.  But, since you are a reviewer, your opinion about using it is very helpful.  

We decided to use a text from the Open Textbook Library and I found their review criteria to be helpful. Our first concern when picking a text was that it covered the areas and objectives we had set for our course. We also made sure the organization, structure, and flow were easy to follow, that the content was up to date, and that it was culturally relevant. We wanted content that could be easily edited so that we could make it our own and also keep it up to date as needed. 

The Achieve Rubric is my favorite if I HAD to choose one. I am working with two colleagues on creating our own rubric to use when evaluating resources. Above all, credibility has and will be one of our main requirements. In order to teach proper use of the rhetorical tools, we need to use resources that properly utilize those tools.

I am currently looking at resources for Composition, Research, Intro to Lit, and Technical Writing. I am especially interested i the Technical Writing portion as my current textbook is now out of print and I do not want my students to have to pay some outrageous price.

Greetings!

Sorry missed the live session last week while traveling. I was able to review the session recording – it looks we had another great session with lots of information and live discussions. Thanks for sharing the examples of evaluation tools. Each tool provides good references on things to consider in evaluating a resource, I personally like the Open Textbook Library Review Criteria and The CRAAP Test. Here are the evaluation criteria I came up with:

• Validity: have open license, from credible source and author.
• Relevance: content is relevant and fit for the needs, suitable for the target audience and at appropriate level.
• Quality: content is well developed to address the subject and is current, accurate and in high quality.
• Design: object is well designed for ease of use and engaging, and follows accessibility and inclusion standards.
• Impact: will help to meet the learning goals and benefit learners.

Wondering if any Texas universities have developed the OER evaluation tools for use at their own institution?

Although I am a library liaison to several departments and have some knowledge of their fields, I do not consider myself a subject expert in their disciplines. However, I do search for potential OER materials for my professors to look at, consider, and evaluate.

So for my own evaluation criteria:

I look at how “open” it is (check the CC license). The particular textbook I am evaluating at the moment (Fundamentals of Business) is shared under a Creative Commons NonCommercial ShareAlike 4.0 license.  I take into consideration how the professor would use the materials and how that use would align with what the individual licenses will allow.

I look at who created/adapted it and their affiliation (academic or otherwise). Are they professors, are they professionals, are they researchers, etc.. Some position that would indicate subject expertise and credibility. This book provides an extensive list of contributors and reviewers. The main contributors are Business professors at Virginia Tech. 

How current is it? I look for the newest materials I can find. The five year rule is standard; however, professors have told me that with certain books it is not as relevant, stating things like “the basic math is still the same and we can provide additional newer examples for the students.” In this case Fundamentals of Business is the 3^rd edition and was published in 2020.

I would also set my eye as to how accessible the materials are.  This business textbook includes a wonderful “Accessibility Chart” (Adapted from Ashok, A. & Wake Hyde, Z. (2019). Accessibility Assessment. Rebus guide to publishing open textbooks (so far). CC BY 4.0. https://press.rebus.community/the-rebus-guide-to-publishing-open-textbooks/back-matter/accessibility-assessment ), which I think will be useful to me in the future when considering OER materials to offer to the instructors.

I also consider its layout, chapter divisions, and flow before suggesting it to the professors as an option.

This is the first time I have written down an evaluation criteria, so I am sure there are aspects I've missed. The other cohort member's comments are giving me alot to think of, so thanks to all.

I liked best the  evaluation criteria on the CRAAP test-currency, relevance, authority, accuracy, purpose.

I also use Youtube, Kahn Academy, and Crash Course videos for demonstration and clarification purposes.

I'm with you, Marilyn - I've found some great things to use as ancillary resources on YouTube and Khan Academy.  I'll check out Crash Course based on your recommendation here - thanks!

I wanted to create a simple, quick evaluation method that could be used to initially screen OER, so I came up with a Triple A Framework: 

Accessibility: follows generally accepted accessibility guidelines (UDL)

Accuracy: information is correct, up-to-date, and provides references

Adaptability: can be adapted for use in any LMS and to meet needs of the course

I used this framework to asess the Foundations of Educational Technology open textbook:

Accessibility

Good: Fully available online, text is screen-reader friendly and uses appropriate headings and html tags

Bad: Some embedded YouTube videos lack closed captions

Accuracy

Good: Information is thorough and accurate, references are scholarly and strong

Bad: References could be more recent

Adaptability

Good: Online textbook can be linked to from any LMS, full textbook can be downloaded in 11 formats including PDFs, Kindle-friendly formats, and various types of html

Bad: Cannot link to specific chapters or pages

I've co-written OER before and used that book to teach that class, so there might have been biased evaluation there :); but since then I've never taught an OER class and so haven't had to evaluate OER materials.  

A bit like I was saying in the chat, in general, when I'm looking at a textbook for a math class, I look for several things.  One, a narrative style.  It makes it much easier for students to follow along if they're gonna use a textbook, and the best theoretical math textbooks I've ever used had a narrative style, so it's completely doable for credit-level math textbooks.  When I teach math I tell students that math in some sense is a story to me, and I'm just taking them along chapter-by-chapter through that story.  Two, I look for well explained examples.  "Well-explained" means explained simply, efficiently, and in a way that easily connects to the reader; which I think often means that the author employs more of visual style in their layout and in making connections and showing the flow between lines of math work, especially in examples -- boxes, arrows, colors, etc (because this generation is so visual, just think about the input they recieve from their phone; if they were 90's kids, sure, they could read Heading-Subtitle-Paragraph-Paragraph-Paragraph-Subtitle-etc fine and get a lot of out of it).  Three, I look at the exercise sets, since that's what I'm likely gonna be stuck giving them for HW since they'll be using the online HW site.  Often, math exercise sets in textbooks consist of a couple intro/easy problems, several medium level problems, and then they jump abruptly to quite a few hard problems, problems that will still give students in later STEM classes trouble.  There's no sense in penalizing students with needlessly hard questions that don't advance student's abilities or tell the teacher much about their learning on a formative assessment, worse to do that with really inadequate scaffolding or use of any of the concepts of Vygotsky's Zone of Proximial Development in the design of the exercies set itself.  So, in exercise sets I look for a couple intro/easy problems, and then many medium level problems that push students to the next level of what they understand on different key concepts, critical thinking skills, and skill sets from the section's content; and only after that just a couple of hard problems to really push them, and also to separate out those who fully understand everything from those that are not quite there yet.

Looking at these resources though, I would definitely love to use the BranchED's Equity-Oriented Criteria and Washington's Model for Screening for Bias in evaluating OER material in the future, cause it always takes intentional work to filter out our, and our institution's, biases and push the materials we, and our instutions, use towards equity.  The rest of the criteria, from a content perspective, I feel I can evaluate just fine from my content expertise and my experiences as a teacher trying to make things the best they can be for my students.  But it always takes intentional and hard work to see outside of our cultural and economic perspectives and upbringing.....

Hey All: 

As noted in our last session, some of the evaluation rubrics are much more detailed than others. However, as academics we have the chance to apply the very skills that we impart to our students. In English courses, as in many other courses, source credibility is a major focus, especially as the internet has consumed most forms of research and information gathering. I cannot quote the statistic off hand, but very few individuals go beyond the first five pages of an Internet search result, and one search engine, Google, absolutely dominates web search and advertising. Many students have never been asked to consider the implications of search objectivity and website affiliation. However, in conjunction with a conversation about source credibility, slant, and bias, usually comes a discussion of the CRAPP Test, which many mentioned.

Qin Fang added “Validity” and “Impact” to the evaluation criteria, which was fitting. Good idea, Qin.

Evaluative criteria are often universal but the more specific criteria seem to stem from one’s discipline. In English courses, students benefit tremendously from sample essays and models, so one of my additional criteria would be “Samples” or “Student Models.” Students love to grapple with other students’ writing; however, I find they are much more vocal about potential revisions when they are not in an in-class peer reviewed session, unfortunately. This is normal, but does anybody have an idea to get students discussing each other’s work at a more critical level in class?  

I would also add critical thinking as an evaluative measure. Does the OER provide those moments where students can synthesize and analyze? Rather than being a means of only transmitting information the best texts act as a tool for getting students engaged with the material at a deeper level.

One of my OER colleagues in this course, Justin Hill, mentioned narrative style. What a great insight, Justin. Interestingly, this was a concern for him in a math course, rather than an anthropology or history course. His comment had me thinking about the authorial voice within textbooks.

Accessibility will become more and more of a necessity as our institutions catch up with student needs. Quality Matters is the organization that pushed me to go beyond content to structure and accessibility. I’ve learned a ton over the last two years, but I have mountains more to learn about design and code.

Benjamin Gracy highlighted something many are asking: What does equity mean exactly? This is an important question, and we see today that the term is being politicized, a trend that is much too common over the last fifteen years and only seems to be getting worse. Many say that “equity” does not mean “equal opportunity” at all but rather some sort of forced state program that centers on one’s skin color. 

I look forward to joining everyone in our third session.

I would most likely begin with the Open Textbook Libarary criteria as they are fairly broad and basic.  At this point I am one of the few instructors in my institution using OER for my particular course.  Because OER textbook/resource adoption has to go through the same adoption process as purchased (big publisher) content--with subcommittee review and election process, I would be forced to use their rubrics in determining the evaluation and possible utilization of OER.  This would include deep integration with our current classroom mangement system (unfortuately that has not been detailed in any way shape or form).  

Left to my own devices, I certianly would be more detailed and include the rubrics for evaluating OER components from Achieve including the evaluation of the utility of materials designed to support teaching, the quality of assessments (including the level of assessment and the inclusion of critial thinking), the quality of technological interactivity (specifically the utilizaiton of 3D models for instruction), and the opportunties for deeper learning, including clinical cases for students to complete as individual and groups, and lastly, the assurance of accessibility.  

As someone who is brand new to OER, I have not had much of an opportunity to evaluate materials, but I would use a simple set of criteria such as the Open Textbook Library Review Criteria.  Also, I teach math and use online homework, so that informs my Criteria as well:

Completeness

The text encompasses all of the necessary material to cover the learning outcomes for the course.

Accuracy

The content does not contain technical errors and is generally free from grammatical errors.

Reusability

           Material is set up in a way that makes it easy to reuse and keep updated. In general, there should not be excessive reliance on cultural references, etc. 

Clarity

The text is written clearly and is easy to read. Explanations should be concise and accessible.

Modularity

The text should be easy to reorganize with various subunits of a course without presenting much disruption to the reader.

Organization/Structure/Flow

The topics in the text are presented in a logical, clear fashion.

Interface

The text is free of significant interface issues, including navigation problems, distortion of images/charts, and any other display irregularities. 

Online assessment materials

The text is accompanied by some utility which facilitates online assessment,  especially homework assignments. 

This is a modified version of the Open Textbook Library Review Criteria. 

I liked Achieve OER rubric and Open Textbook Library Review criteria. I also liked many of the suggestions made. The following is a list of criteria I plan to use to evaluate OERs:

• Accuracy:  Are there major content errors and/or omissions?
• Relevance/Coverage: Does the information directly address the course learning objectives?
• Accessibility/Integration: Is the resource easily accessible in different formats? 
• Licensing: Does the license allow for reuse, modifications or adaptations of the resources?
• Currency:  Is the resource up-to-date?
• Authority:  What are the author's qualifications for creating the materials?

I mostly follow CRAAP. I did find the Achieve rubrics interesting and would like to incorporate it (via maybe a reduced format). The Open Textbook Library is also a good option for evaluation and it includes the cultural relevance criteria, which meets diversity and inclusion requirements. 

Currency- timeliness
Relevance- to topic
Authority- credible sources
Accuracy- correct/good information
Purpose- what does this information do, add-to, etc.?
Cultural relevance-  is information presented in an unbiased and culturally sensitive way?

I reviewed - Introduction to Women, Gender, Sexuality Studies.

I currently teach a student skills type class and i chose the following resource:

https://oertx.highered.texas.gov/courses/college-success-2

The criteria I used were:

1. The content of thh chapters - the topics covered

2. The language used, was it at an apprporiate level for the audience

3. The structure and organization of the book, did it have all of the elements needed to teach the course

After looking over the resource links and some links embedded within them, I want to say that I am still AMAZED at the amount of information and the resouorces which are so freely shared by educators of all sorts. This is the really big difference between what we are trying to do with OER and the traditional business of book publishing companies. It is definitely about saving students money and providing materials quickly.....But this is really just a very small aspect (though huge benefit to students) of this movement. I am humbled by what my colleagues have developed and shared freely for the sake of learning and education. 

My specific thoughts on the evalution resources do not add much from what has already been stated. The Achieve OER resource is quite thorough and well thought out. The Open Textbook Review is clearly not detailed however it is a very good checklist for reviewing and would be valuable to screen the product. However neither one seems to address the quality of images and tables very well. These items are invaluable within life and some physical science courses. Neither seems to specfically address the student perspective. The idea of student input to evaluation is important albeit challenging especially in lower level courses. 

The BC Campus Accessibility seems well thought out and the link to the Web Content Accessibility has very thorough guidelines. Accessibility is a huge concern for educators. The big publishers have been in court over this (and lost). 

BranchED Equity was a streamlined approach that incorporated a few categories from the Washington Model and the BC Accessibility. This resource (like the OpenTextbook) was a quick reference list. The one truly "stand-out" was the Learner-Centered category. I really like the emphasis on the end-user of the product and the result of it.

The Washington Model Screening Bias was absolutely well-done and quite good. This is one area I would love to see all textbooks screened for compliance. It is so easy to be "turned off" as a reader when you sense that the author has completely ignored your perspective. Even if there is value to the content/information, this oversight is so damaging it can render a product useless.

Thank you again for the inspiration and the generous nature of the pioneers and developers.

I recommend the Open Textbook Library Review as a preliminary guide to those faculty who want to use OER.  To the following 10 items I also append a 11th for faculty to be aware of:

Criteria list (OTLR)
1.    Comprehensiveness
2.    Content Accuracy
3.    Relevance/Longevity
4.    Clarity
5.    Consistency
6.    Modularity
7.    Organization/Structure/Flow
8.    Interface
9.    Grammatical Errors
10.    Cultural Relevance
11.    Meets Target Audience Learning Goals, Objectives, and Outcomes

My evaluation criteria are currently listed in order of what I would consider, but with time would want to create a rating scale for each (maybe 0-10), so if deciding between a few resources that would help with the final choice. Personally, I don't want to provide too many items on one topic as students have said they get overwhelmed and confused. I encourage them instead to go do more research or contact me if they need more on a topic. With a rating system, I would then have a list of vetted extra resources I could send them.

If 1 or 2 below is met, then move down the list. 

1. Covers required content area(s) for the course. We have an "approved" list of required topics for each course as they feed into others. This is nice because you know what students need then have flexibility outside of that to add other topics. 
2. Covers content that is current. Some of my content is static (math equations) but then new areas within the industry emerge and are not in a textbook published last year. 
3. Accurate content.
4. Periodic updates. When was the content last updated?
5. Can it be modified, if needed? Can it be downloaded and posted to an LMS? (Not sure this is viable if it is a course embedded in a website. My hesitancy to use OER has been "what if the content owner takes it down and the whole class is built around that resource?")
6. Easy to navigate - easy for users to find what they need quickly
7. Extra resources available - syllabus, discussion ideas, cases, PPT/outlines for notetakers, links to relevant sites, assignments, test questions, etc.
8. Alignment with "accepted" textbook content (could I tell them go find any edition of these 5 books after 2015 for the basic information; they can get one of those for $5 online). Some students just want a physical book and sometimes it helps to hear the same topic in a different voice.  
9. Reputable source. This is nice but for me not a deal breaker if the content is accurate. 

I really enjoyed this practice.  It made me really evaluate what I am looking for as an instructor for my courses.  Some of the critirea that I personally felt to be important are:

• aligns to course content standards
• modular
• quality of explanations of content
  • target audience
• quantity and quality of example problems and practice problems
• Accessible
• Cultural relevance
• CC license
• Stability of resource
• Deeper learning exercises **this is not as needed but it could be useful. 

All the resources provided in the presentation were great, there was something about each that could be used to create one that suits specific needs.  I found one on the Affordable Learning Georgia Website, that had many aspects of several of the other resources.  

https://www.affordablelearninggeorgia.org/help/finding-7

The tool has the following criteria:

-Clarity, Comprehensibility, and Readability

-Content Accuracy and Technical Accuracy

-Adaptability and Modularity

-Appropriateness

-Accessibility

-Supplementary Resources

I am spending quite a bit of time now analyzing the book that I suggested to make sure that it fits my needs.  As I develop more OER I will use this as part of the  framework in developing each segment of the book or content.

Hello,

I use the Quality Matters Rubric to review courses. Next week, I am presenting at the Accelerating Completion and Employability in Information Technology ACE-IT Symposium. I am planning to share the CRAAP Test - Currency, Relevance, Authority, Accuracy, Purpose https://researchguides.ben.edu/source-evaluation with the audience. I think it is a great resource for the Information Technology faculty. Thank you for your time and consideration. Have a great day.

Andres

I feel like most of all these are the most important issues in evaluating OER:

Relevance

Quality

Peer Review

Accessibility

Interactivity

Copyright

For evaluating OER resources. I ue the same rubric model for evaluating websites. I use as a librarian for websites:

How to Evaluate Websites: How to evaluate websites

1. CURRENCY: the timeliness of the information.
2. RELEVANCE: the importance of the information for your needs.
3. AUTHORITY: the source of the information.
4. ACCURACY: the reliability, truthfulness, and correctness of the content.
5. PURPOSE: the reason the information exists.

The formula is easy to share to faculty and students because I spare the reasons for textbooks in the same context of ensuring the website is credible if you plan to use it in your research 

There are 1,218,423,991 websites in the World as of May 2021. >> https://siteefy.com/how-many-websites-are-there/

Applying the same approach to OER with textbooks for vetting the best OER materials using CRAAP in the same way for websites is fruitful.

Kimberly M. Gay 

Since I teach Women's Studies, I need to evaluate resources both for relevancy and credibility but also for bias, diversity/representation, and appropriateness. 

I start with the CRAAP test and then I move on the more discipline-specific focus. 

Bias: I check that the information is free from bias by evaluating whether the author situates themselves in the text. In my experience, bias comes from trying to maintain objectivity and universaility, rather than acknowledging their own context. 

Diversity: When evaluating resources, I need to make sure I have as many voices as possible represented. So I check if all of my texts are by white authors or male authors, and if so, I search for other sources so students feel presented in the texts. 

Appropriateness: Women's Studies materials can vary in their level of appropriateness. By this, I mean some texts discuss intense subjects that can be triggering for students. I balance this based on the level of the students. 

As I search for specific course content, I consider if the resource:

• has content that aligns with the course objectives and competencies. 
• is presenting the information at the correct level of instruction needed such as undergraduate or graduate level 
• can be used as a link from my course LMS without violating copyright laws 
• can be remixed or revised to align with existing lectures I may already have
• can be integrated with course content I already have

If the content meets these considerations, I keep a document with a link to the content, the course number and abbreviation in which I want to use it, the use licnesing information, and a summary of the content and where I want to use it.

My criteria looks something like this: it took some time to identify and write down what I do subconsciously!

Currency: When was it created? The desired year range varies depending on the topic. For news literacy (“fake news”), I would look only at materials created within the past five years.

            -When was it last updated?

            -How often is it updated?

Credibility: Who is the author/ authors?

            -Where do they get their standing? (Institutional backing, popularity, study?)

Completeness: Does the material include the relevant information about the topic that I want to present to learners? How much of the relevant information does it include? Can I supplement it with another resource?

Purpose: What is the goal of the material’s authors?

Quality: Is the material free of errors?

            Are complex ideas presented in a clear way, or even better, in several different ways for better understanding

Readability: Can the material be understood by users with a variety of reading levels and first languages?

Bias: Is there a natural representation of diversity of apparent race, ethnicity, gender, abilities, age?

            Does the material avoid stereotypes?

            Is the material accessible to users of assistive devices like screen readers?

            Are there multiple ways to access the content?

Culturally responsive: Does the material take into account its use by people from cultures with different standards and expectations?

Licensing: Am I free to make edits and re-share?

Stability: Can I count on being able to retrieve the material throughout the semester at least? (As Pamela Rogers asked, can I download it and embed it in my LMS?)

I submitted a resource (Web Literacy for Student Fact-Checkers) and attached my evaluation based on this criteria as a comment!

I really like this "Evaluating OER" rubric, it is basically evaluting on who, what, how basis but it ask specific questions in regards to the resources including thing like bias, affiliation, quality control, etc. One thing I think I would add after reviewing the resources presented in the module was how much I like the Washington Screening for Biased Content in Instructional Materials.  I think I would combine criteria from the two rubrics to also screen for Content Bias and not just possible creator/sponsor bias. 

Particularly the 'Variety of Roles and Character Traits," and "Multiple Perspectives...," and "Multicultural Representation." 

One of the things about the Washington rubric was I am not sure how you actually asses what they criteria was. For examplel, in the section on "Family Representation," how could you dtermine if they were adopted, foster, or stepparents.  Unless it was specifically spelled out? I like that the rubric asked the evaluator to consider this, but I am not sure how it could actually be measured. 

I reviewed both the Open Textbooks Review and the Library CRAAP Test evaluation criteria to evaluate my resource. However, when I started applying the different evaluation criteria, it seems I was more in line with the CRAAP criteria: Currency,Relevance, Authority, Currency, Accuracy and Purpose.  

I realized that I did not actually apply the rubrics to an OER resource to see how that resource rated.  Our department has been using the Biology from Openstax for 8 years and the Anatomy & Physiology from Openstax for 3 years. I am very, very appreciative of the consortium and how much work has been done by Rice University. The work and the cost savings to students has been substantial.

I will look at the Biology textbook which we have used for 8 years. Please note that Openstax is now on the second edition of this textbook. If I apply the Open Textbook Library’s rubrics, I feel that the textbook would not significantly violate any of the criteria. There are errors in labeling of images that we find occasionally, but it is easy to contact the publisher and that does not detract from its value. There are issues with the interface. If you wish to use the images from the pdf version for presentation, you must get the powerpoint files from Openstax. Adobe DC distorts the resolution/text of the image if it is copied and pasted. Some of the images have color distortion when you import them directly from pdf into word or presentation software.

An area that is not covered by this rubric is accessibility. My colleagues have encountered problems here.  It seems that there are sometimes lost or missing alternate text, problems with formatting of diagrams, figures and tables if downloaded from the site. There is a lack of significant color contrast with some titles, text and linked texts (in the original, not due to distortion of download). Some of the links to outside sources are missing as well.

The resource provides many interactive links and has assessments as well as vocabulary at the end of every chapter, but it does not have embedded section questions or recall which are necessary for “checking for understanding” before advancing. It does not provide good linkage to concepts covered previously, or what I think of as reminders for what you should have learned in other chapters or sections. It does not provide encouragement or reflective opportunities for instructors to engage with students either. While the content is accurate and mostly appropriate to the level of student, it is not really active, it does not encourage a deeper learning within the chapter content or moment of discovery very often.

Another area that is lacking with Openstax are the diagrams and figures. Science needs visual demonstrations along with the verbal explanations. It is really an area where “a picture is worth a thousand words”. This is an expensive part of publishing and so it is completely understandable as to why it is missing, but that does not change the necessity of this for utility by students. Images in general must be aesthetically pleasing not just because we are “shallow” but because the quality will activate memory and invite exploration. Their absence is rather glaring.

Please understand that I am very grateful to Openstax for their work. The textbooks seem to be authored by many individuals who are content experts and avoids the pitfalls of the publishing industry’s limited authorship which provides a limited perspective to the book. This is detrimental to a student especially since they do not even realize it is happening. Openstax Biology is highly collaborative and is a very good resource. Thank you, Patrice

I think that the CRAAP test is the most similar to our protocol for creating training materials,

Currency: the timeliness of the information

• Video tutorils and handouts need to be checked against a spreadsheet every semester to determine what needs to be updated or added.
• Tutorials are created according to importance in the semester.
• Vdeo tutorials that are posted need to be linked to the tutorials page. 

Relevance: the importance of the information for your needs

• The tutorials need to help students or faculty with using basic and advanced eLearning tools.
• Information is compared against official documents from our Learning Management System and what the training team discover during testing and training. 

Authority: the source of the information

• The first resource is the official information from our Learning Mangement System.
• Tutorials are created based on best practices for creating video tutorials.
• The people responsible for the tutorials are instructional designers with several years of experience with the LMS or students creating tutorials with guidance from the trainers.
• eLearning contact information is shared for people with additional questions.
• Our URL shows that the tutorials are part of the campus eLearning team.

Accuracy: the reliability, truthfulness, and correctness of the content

• Information comes from reviewing official documents and testing.
• Videos and handouts are proofed by at least one other person before being published. 

Purpose: the reason the information exists

• Tutorials are meant to support the individuals who need to create or manage an online course.
• The tone of the tutorials is neutral as only the abilty of the tools are shown..
• Tutorials cannot have a bias for the student or for the instructor. 

I like the Achieve OER Rubric due to its comprehensiveness, however, I use a simpler tool, first, to decide if the resources are valuable enough to commit the time necessary to use the more comprehensive rubric. 

I first look for content to match the course outcomes and objectives. For those that get a high score, then I use a modification of CRAAP to evaluate if the resource gets curated. Once there are several resources curated for the course, then I use the Achieve OER Rubric to identify which are the most valuable for the course. 

The evaluation criteria that I use to assess a resource will be the rubric that we as Quality Matters Peer Reviewers derived from the QM course design rubric that we added additional criteria at the bottom of the rubric that we previously designed for online and face to face courses. I will compare it to the Open Textbook Library Review to see if there is anything we missed that is not on the one we developed from QM.

I would use a rubric to evaluate an OER with the following performance indicators for a mechanical engineering course

1. Coverage of the course syllabus.
2. Illustrations for clarifying and explaining the concepts.
3. Solved examples in the main text to elaborate the concepts.
4. Exercise problems to practice learned concepts

Each performance indicator would be ranked on a scale from 1 to 4 (lowest to highest)

Allplication:

I used the above criteria to evaluate one of the following OER.

https://oertx.highered.texas.gov/courses/advanced-structural-analysis

The evaluation is present under the comment section of the resource.

Note:

The above is based on the background in developing and using rubrics for evaluating  ABET Criterion 3: Students Outcome (available on the ABET website with the following link):

https://www.abet.org/accreditation/accreditation-criteria/criteria-for-accrediting-engineering-programs-2019-2020/#GC3 

To evaluate OER on topics like online pedagogy and teaching practices in higher ed, I found ACHIEVE OER rubric to be relevant and comprehensive as it covers both content and assessments. A limitation however is that it does not touch on EDI. To address that I would tag BranchED’s Equity-Oriented Criteria to it as Section IX - Equity, Diversity, Inclusion. The other limitation of the Achieve rubric is that it doesn’t provide a way to directly and easily evaluate smaller, individual objects or artifacts. I think that could be addressed with the CRAAP test. I reviewed an adaptation by Meriam Library, CSU, and would include these items from their list:   

• When was the information originally published? When was it revised /updated? Should it have been recently updated (for example, if it is related to technology)
• Content is at an appropriate level for the intended audience (in my case, higher ed educators)
• Autor’s intent is immediately clear – whether it is to inform, teach, sell, entertain or persuade.
• Author’s credentials and specifically organizational affiliations are clear, to help assess if material may be sponsored or meant for propaganda.
• Transparency of sources – how many are there? Are all clearly identified? Are they non-commercial (.edu .gov .org .net)?
• Tables and images are accompanied by data sources  
• Language is unbiased and free of emotion
• No typos or broken links

I have evaluated the below resource and my comments are in the Teaching Online Folder

Zhou, Molly and Brown, David, "Educational Learning Theories: 2nd Edition" (2015). Education Open Textbooks. 1.
https://oer.galileo.usg.edu/education-textbooks/1. 

The criteria used to evaluate this material was the Equity Oriented Resource Criteria. This rubric was chosen because the evaluation of Bloom's (along with the origins) needed to be responsive to the diverse learners of community college.

Learner-Centered: I found this resource to be very learner centered since it is well designed, has appropriate yet informative language, and includes links to other helpful resources. It also has a link for a downloadable PDF so that instructors/designers can save for future use.

Critical: The language used in this article seems to be very factual based and to the point. The various learning theories (and behavior scholars) seem to be inclusive and non-discriminatory.

Cuturally Responsive: The article also focuses on various domains of learning that evolve as the studies evolve.

Overall, I believe that using both the Equity Oriented Module and CRAAP model can be used as a well rounded assessment and evaluation of the material. By using these two models, I do realize that the resource I chose is very restrictive when it comes to remixing, which is understandable since it is very fact based. In the future, I know that I need to be more aware of this if I want to adapt the content for a course.

Going forward, I will direct my attention to the licensing of materials. I hope to continue my OER journey by becoming more familiar with it.

For reference, here is the resource I posted and evaluated: https://teacherofsci.com/blooms-taxonomy/

I am catching up with all the homework, been an overscheduled summer.  Loved the course. I adapted a version of a rubric and added some scoring for me and I really like it.  I reviewed the case study OER I found with the rubric.  Thanks for showing us all these great resources.  I will add to resources. 

I used the CRAAP criteria (Currensy, Relevance, Authority, Accuracy, and Purpose) to evaluate Python for Everybody: Exploring Data In Python 3. I think this was the best criteria for computer science because whether or not the resource is current is very important. The relevance was good because it deals with informatics, which is something that Python is often used for. The other criteria, authority, accuracy, and purpose, are also highly relevant due to the exact nature of computer science and needing something to be done in a certain way.

OER Evaluation Criteria –

I am hoping to GENERATE OER materials, so thinking about these evaluation criteria helps me to “begin with the end in mind” in addition to giving me an opportunity to review other materials to curate.

After brainstorming a bit I came up with the following - which actually seem pretty aligned with the CRAAP tool.

Relevant- Information that is current, practice-based, and reflects diversity

Critical – Presentation develops rationale and unbiased critical thinking, and includes well-cited references

Organized – Logical progression and easy to navigate sections

Professional – Writing style and accuracy reflect competence and quality.

Accessible – Product reflects Universal Design with readability for most students

Engaging- Material is digestible with embedded links & videos to extend learning, and application activities that build practitioner skills and dispositions

This is totally new to me and I plan to use the VIVA OER Evaluation https://docs.google.com/document/d/1Pz_sGy5KoAP7_lRXy6vqZFwscornwTbr_STSD6Kde0U/edithttps://docs.google.com/document/d/1Pz_sGy5KoAP7_lRXy6vqZFwscornwTbr_STSD6Kde0U/edithttps://docs.google.com/document/d/1Pz_sGy5KoAP7_lRXy6vqZFwscornwTbr_STSD6Kde0U/edithttps://docs.google.com/document/d/1Pz_sGy5KoAP7_lRXy6vqZFwscornwTbr_STSD6Kde0U/edit

The CRAAP test is very informative.  I can follow the currency, relevance, authority, accuracy, and purpose checking.  However, I want my OER materials peer-reviewed.  Unlike journals or books from publishers, it is not easy to find someone in my field to review the contents of the OER books. I hope the use of OER materials to become common, and people will be willing to review those materials. 

Hello all,

I teach biology courses. These are the evaluation criteria that I find the most important:

• Current Materials: I want materials that are relevant and hopefully less than five years old.
• Technical Accuracy: I verify the materials are accurate to the best of my knowledge.
• Authority: Who created it? Where is it located/published? Has it been vetted by others? 
• Readability: I want the resource to have good grammar. I want it to have consistent language and formatting.
• Accessibility: I want the resource to be accessible to students with disabilities and refer to the ISKME checklist. 
• Appropriateness and Fit: I need the materials to fit smoothly into the flow of the topic's discussion.

Usually when I help instructors search for OER materials in their disciplines, I look for:

Who is creating or providing it – faculty, institutions or organization that would produce credible resources.

Appropriate academic level.

If it is up-to-date and accurate. Do the links work?

Verify the open license permissions.

I have not is the past assessed for accessibility, but this would be good going forward (one of many takeaways from this 2^nd session).

Hi, everyone!

I adapted the sample VIVA rubric to come up with these areas:

1. Accuracy and Author Expertise/Affiliation --> Here, I'm looking for accurate content that's free of errors in grammar, mechanics, etc.; I also want something written by an expert (or group of experts) in the field. [The resource I found is written by a junior faculty member, so it's okay but perhaps not great]
2. 'Open' Licensing and Adaptability --> I didn't know all that much about open licensing when this course started, but I've realized that it's important to understand the nuances. I'd want a resource for classes that can be adapted and modified as the instructor needs. [Resource was licensed with Creative Commons and allows modification]
3. Clarity, Fit, and Appropriateness --> Since students have to use the resource, it needs to be clearly written (on their level) and logically organized. If students can't understand or navigate the source, it doesn't do the instructor much good. [The organization is okay, but as an instructor I would want to rearrange some parts; the license allows for that]
4. Accessibility & Technical 'Fit' --> Since I hope to offer the course I'm evaluating resource for online or in a hybrid format, it's essential that the content meet accessibility standards in regard to its organization, use of images, multimedia, etc. The resource should also be formatted (file type) so most students would be able to use it on the equipment available to them.
5. Diversity & Inclusivity --> Content should reflect diverse perspectives (both in the examples and case studies and in the authors and scholarly voices included in the conversation) and avoid stereotypes. [Resource could represent more diverse perspectives]

Right now, the resource I found to review satisfies several criteria here, but I was surprised to see there weren't more diverse examples in it upon closer review. Sigh...

I hope everyone's reviewing goes well! :)

Jessica Zbeida

I developed my Evaluation Criteria checklist based on several of the resources provided above.  Here is OER Evaluation Criteria and the filled in version for my evaluation of the Khan Academy's Carrie Mae Weems' photograph Untitled (Woman Feeding Bird), The Kitchen Table Series, 1989-90.  Thanks!

OER Evaluation Example

I combined and rearranged the CRAAP and VIVA evaluation criteria.  I wanted to make sure I evaluate the source/license first and then dig into the content, accessibility, etc.  If the source is questionable, I feel I can stop and not go further into a full content evaluation.  By combining these two evaluations, I felt it covered both source and content. 

I then used it to evaluate an OER Texas Government book I found in the Digitex library.  Using the checklist I felt the book met some of the criteria but not all.  If I use the book I'd have to supplement it.  That seems to be pretty typical though for OER for Texas Government.  There isn't a whole lot out there that is complete for state government.  The federal government OER's look a lot better.

Hi, I decided to develop an evaluation checklist for OER video resources, compiling and modifying from the discussions above and the resources provided for this activity.

OER Video Review

Relevance: Content aligns with the learning objectives related to the topic. The information is current and has been revised or updated if needed. The content is at the appropriate level of rigor.

Accuracy: Content is accurate and free from errors and major omissions.

Authority: Information is from a reliable source that likely will not go obsolete. The author is qualified, and their credentials or organizational affiliations are given

Presentation: Material is presented in a clear, logical fashion. Clarity, tone and pace of speech by Narrator is sound. Graphics are clear and easy to understand

Contextualization: Connections are made to prior and/or future learning objectives.

Engaging: Presentation is engaging, with ties to real-world connections

Length: Video length between 2-15 minutes

Diversity and Inclusivity. Stereotypes are avoided and diverse people are represented.

Accessibility: Compliance with the ISKME accessibility checklist.

Hi all for my evaluative rubric I decided to add to the CRAAP Test by adding Openness.  Hence, the OCRAAP Test. I placed Openness first, because in searching for OER’s I think that is the first starting point. I used basic CRAAP definitions, that are slightly modified from https://researchguides.ben.edu/source-evaluation.

• Openness – How freely available and
  • Is the resource freely available for use in higher ed
  • What if any limitations are placed on usage via the Creative Commons or other license
  • How easily does it integrate into our Learning Platform(s)
• Currency – How timely is the information
  • When was the information originally published or otherwise made available?
  • When was the content last updated?
  • Is the information current for topic and within any applicable accreditation standards?
• Relevance – How Important is the information in regards to the specific need?
  • Is it related to the topic at hand?
  • Is the information intended for the target audience and appropriate for them?
  • Has it been compared to other sources before selection?
• Authority – How strong is the source of the information
  • Who is the author(s), publisher, source(s), sponsor(s) and what are their credentials?
  • Is the author qualified for the given topic and can you verify the contact and biographical information of them and the publisher?
  • Do any referenced URLs about the information provide additional evaluative information?
• Accuracy – How reliable, truthful, correct, researched, and correct is the content
  • Is information verifiable and supported by evidence?
  • Has the information been subjected to peer review or been refereed?
  • Is the information presented professionally/up to standards both in terms of tone and in terms of spelling and grammar?
• Purpose – For what reason(s) does the information exist
  • What is the purpose of the information and is it clearly stated?
  • Is the information fact, opinion, or propaganda?
  • Is the perspective the information is presented objective and impartial and are there any clear political, ideological, cultural, religious, institutional, or personal biases?

I actually used this live this week to review a resource a faculty member brought to me, a Criminal Investigations course form Barnes & Noble’s OER Plus.  

While the resource fit the criteria in terms of the CRAAP method, it did not meet the criteria for the Openness criteria.  We have seen a lot of similar models of late, things offered at below the current pricing we seem from Cengage, etc. but not truly an OER or an OER that has been package and they are selling the package.

So then the question we are having is do we still move to in in the short terms because the cost savings per student would be over $100 per student compared to their textbooks now.  I would be interested to learn more about what folks are doing in these type of situations in their work?  

Mark, this is a great additon to the CRAAP test, right off the bat one will know what platform is being used.

I decided to make a checklist like the Viva checklist with a few tweaks that were more applicable to myself and my courses. I thought it was a great checklist and very comprehensive. 

I evaluated a lesson from Excelsior College on plagiarism. The lesson was interactive, included  before, during and after learning checkpoints, videos, as well as printables. I thought this would be an excellent lesson to use as I already have a plagiarim lesson and quiz in my own course but I really liked the interactive portions of the lesson. 

As I evaluated the content, I made sure to look at how appropriate and correct the content was, and if it was user friendly. I think the only tweak I would make is to keep my current quiz and just adapt the questions to the lesson as there was no end assessment for the instructor.

Holly

I reviewed the Achieve OER Rubric, Open Textbook Library criteria, and the CRAAP test to put together my own evaluation criteria:

Quality - content is comprehensive, concepts related to the subject matter are easy to understand

Accuracy - material is reliable and can be verified, use of correct terminology, peer-reviewed, free of bias and erros

Currency - material must be recently written or recently updated (3 years or less for Information Technology)

Modularity/Flow - content is organized into chapters/modules/sections, logical flow

Accessibility - is accessible to all learners (ISKME Accessibility Checklist)

I liked the VIVA OER Evaluation Checklist as it was clear and concise. I would add a section on the following:

• Accessibility: Ensure all content, images, videos, etc. are within 508 compliance.
• Ease of integration: Is the resource something that can be easily integrated into the LMS.
• Ancillary materials: Sometimes faculty shy away from OER resources because there is a lack of ancillary materials they receive from publishers. While this item may not be a make or break, it would be helpful to understand what resources did or did not have additional ancillary materials.

I really liked the VIVA Open OER Evaluation Checklist, but I, like other colleagues, added something directly stating accessibility. I also liked CRAAP. However, for formal evaluation, I would probably use the VIVA. I would keep CRAAP in mind when doing initial reviews. Since I am a librarian, I may also use this list to ask the faculty what they valued most and see if I can fit the resource to match their specific needs. For example, one faculty may want to use an entire book, as is. Another faculty may want to mix and match multiple books and chapters and create something new. I would like to know how they want to teach and use the resources in their class before the resource could be properly assessed.

1. Clarity and Readability - Format and organization makes sense for the content

2. Content and Accuracy - Content comes from credible sources and is current (updates or recently published)

3. Adaptability and Modularity - Flexibility of the material for classroom use and adaptability for Course Reading Lists

4. Appropriateness and Fit -  Alignment with student learning objectives and is written reasonably for the course

5. Representation and Diversity - Depictions and examples feature diverse characters and authors represent diverse backgrounds

6. Accessibility - Meet standards as laid out by the ISKME guidelines. 

OER Evaluation Criteria –

I would agree that the VIVA checklist is sufficient for an evaluation process depending on the OER course or program you are researching. The challenge is finding information relating to Homeland Security& Emergency Management. The VIVA checklist would work in my field of study. Still, there are challenges with the security aspect of some information that can be obtained. A majority of information needed to develop an OER course in this field requires someone to access federal, state, and local databases and websites. There are numerous journal articles throughout the web but require payment or subscriptions to use their material. The websites or articles that do not require payment or subscriptions are often outdated. Suppose an OER item is found in the Homeland Security & Emergency Management field. In that case, I will use the VIVAS checklist as-is.

My criteria for resource evaluation are as follows:

Content Accuracy and Credibility: 35 pts. 

Readability: 15 pts.

Accessibility: 15 pts.

DEI: 15 pts.

Adaptability: 10 pts.

Technical Accuracy: 10 pts.

For the course overall, I like the Achieve OER rubric. It contains elements of a course vital to student success, and incorporates alignment to SLOs. I appreciate the explanations for scoring each of the criteria. However, I would change the document by including a summary table that lists the criteria and scores to make evaluating a new resource a little more user-friendly. Users that need clarity on a particular criterion can reference the explanations as needed.  For individual resources that will be added here and there such as articles or homework assignments, I prefer the simplicity of the CRAAP rubric. 

After reviewing the Achieve OER Rubric, the CRAAP Test, and the Viva OER Evaluation Checklist, my criteria would include.

• Relevancy
• Currency
• Authority
• Accuracy
• Purpose
• Openness / Adaptability
• Diverse Representation
• Accessibility

I appreciated all three rubrics, especially the Achieve OER rubric for its comprehensiveness, but I understand the VIVA and CRAAP are perhaps more concise and straightforward. 

I really like this checklist from Texas State University Libraries. https://digital.library.txstate.edu/bitstream/handle/10877/12236/OER%20Evaluation%20Checklist.pdf?sequence=1&isAllowed=y

I feel like this one hits everything I would want to hit, but depending on the resources, I wouldn't need to check every box. When evaluating a textbook that will cost students money, I want as much of the resource to be relevant/check the boxes as possible. However, in my own use of OER materials, I pull from a lot of different sources. I may link students to an OER textbook chapter, and that may be the only part of the resource we use. That can make it easier to evaluate a source as well. The biggest things I look for when evaluating are: relevance to the unit/lesson objectives, accuracy of content, accessibility of content, and correctness (minimal typos). I often give my students a lot of context surrounding an OER source to make sure they know what it is and why we are using it (e.g. "This video was orignally used for ZYX. We are using it to think about ABC" or "This is a link to a free textbook. Chapter 4 addresses the concept we are talking about this week." I do this to make sure students understand why we might only use part of a source or use a source that had a different original purpose from what we are using it for. 

While I like the focus and thoroughness of the Equity and Bias tools, they would not be easy to use when evaluating multiple resources in a limited amount of time. I'd probably begin with Achieve or CRAAP for a first pass.

When searching out resources in the past for my courses, I looked at credibility, relevance, and accuracy above all else. Usability is important, especially in lower level courses. Currency is important in rapidly changing fields like tech, but not as much in other fields.

I would utilize the CRAAP tool.

CRAAP is an acronym for Currency, Relevance, Authority, Accuracy, and Purpose. Use the CRAAP Test to evaluate your sources.

Currency: the timeliness of the information

• When was the information published or posted?
• Has the information been revised or updated?
• Is the information current or out-of date for your topic?
• Are the links functional?   

Relevance: the importance of the information for your needs

• Does the information relate to your topic or answer your question?
• Who is the intended audience?
• Is the information at an appropriate level (i.e. not too elementary or advanced for your needs)?
• Have you looked at a variety of sources before determining this is one you will use?
• Would you be comfortable using this source for a research paper?

Authority: the source of the information

• Who is the author/publisher/source/sponsor?
• Are the author's credentials or organizational affiliations given?
• What are the author's credentials or organizational affiliations given?
• What are the author's qualifications to write on the topic?
• Is there contact information, such as a publisher or e-mail address?
• Does the URL reveal anything about the author or source?
  •  examples:
    • .com (commercial), .edu (educational), .gov (U.S. government)
    • .org (nonprofit organization), or
    • .net (network)

Accuracy: the reliability, truthfulness, and correctness of the content

• Where does the information come from?
• Is the information supported by evidence?
• Has the information been reviewed or refereed?
• Can you verify any of the information in another source or from personal knowledge?
• Does the language or tone seem biased and free of emotion?
• Are there spelling, grammar, or other typographical errors?

Purpose: the reason the information exists

• What is the purpose of the information? to inform? teach? sell? entertain? persuade?
• Do the authors/sponsors make their intentions or purpose clear?
• Is the information fact? opinion? propaganda?
• Does the point of view appear objective and impartial?
• Are there political, ideological, cultural, religious, institutional, or personal biases?

There are many lists of criteria to use; I'm not sure what to say. I was recently introduced to the CRAAP tool or test, which I will use.

1. Currency - iI it current? I need to know it is new and up to date.

2. Relevance - Does it relate to my topic or subject matter; additionally, does it reflect what my students need to know to accomplish the objectives.

3. Authority - Who authored the material? Is this source credible?

4. Accuracy - Is the material presented accurate or aligned with other material in the discipline?

5. Purpose - What is the purpose of the material content? How can I use the material?

I use the CRAAP test, consider if the source is user friendly, and the level of accecibility. 

I accessed and viewed the rubrics that were discussed in this training. However, I have not used one at this time for evaluation purposes.  

I am working on evaluation checklist.

Criteria Topics

• Purpose
• Accuracy
• Accessibility & Navigation
• Relevancy
• Design

Possible Questions

• Is the purpose for the site stated?
• Is the information accurate and factual?
• Is the information accurate?
• Is the information consistent?
• Is the information supported by other sources or references?
• Are the links within the site accessible?
• Are the links easy to navigate?
• Are titles and brief descriptions provided?
• Is the content clearly stated?
• Is the content organized?
• Are there grammar errors?

Sorry about this very tardy post, but sickness got in the way. Anyhow, I completed the evaluation of the Mythology Unbound textbook using the Achieve OER rubric. To sum up my own evaluation of the text: it is a good start or can serve as a good base to build on.

UHD also has a library site with sets of evaluation questions and criteria: https://library.uhd.edu/c.php?g=857187&p=6153699.

Thanks,

Ed

While we use an overall criteria for evaluating OER materials. The criteria that resonated with me the most is Representation/ Diversity and the need for sources to highlight and positively reflect marginalized communities. I looked at the Art Appreciation material and while it is overall a good source searching for diversity in the arts, particularly in art appreciation can be challenging. It did fufill comprehesion, accuracy (in content listed), but lacked in diversity and appropriateness (based on the lack of diversity).

Like many others, I use the CRAAP test to evaluate OERs and I like to consider design, readability, and accessibilty. An exciting visual organization is also a plus.

There are some great evaluation tools.  Honestly, I do not have a lot of experience utilizing these tools.  I like the CRAAP test and the representation and diversity section of the VIVA Open OER Evaluation Checklist. 

Representation and Diversity

• The illustrations and photographs reflect diverse peoples and the context of the depiction does not perpetuate stereotypes.

• Example names with particular ethnic or origin associations are portrayed properly and avoid negative comparisons or stereotypes associated with particular national origins or ethnicities.

• The resource presents diverse contexts and all examples are comprehensible by everyone.

• The correct replacement or re-framing of outmoded or incorrect terminology is presented.

• Contributors or researchers referenced in the resource come from diverse backgrounds.

Evaluation criteria:

Readability and clarity

Clear alignment between learning obejctives and contents

Accuracy

Adaptability

I like the OER evaluation rubrics provided by Achieve. Though it is focused more on high school, I think their rubrics are simple to use and easy to adapt. Specifically, I like Rubric II: Quality of Explanation of the Subject Matter (see below - taken from https://www.achieve.org/files/AchieveOERRubrics.pdf) 

---

This rubric is applied to objects designed to explain subject matter. It is used to rate how thoroughly the subject matter is explained or otherwise revealed in the object. Teachers might use this object with a whole class, a small group, or an individual student. Students might use the object to self-tutor. For objects that are primarily intended for teacher use, the rubric is applied to the explanation of the subject matter not to the planning instructions for the teacher.

Rubric II Scoring Guide:

3: An object is rated superior for explanation of subject matter only if all of the following are true:

 The object provides comprehensive information so effectively that the target audience should be able to understand the subject matter.

 The object connects important associated concepts within the subject matter. 

 The object does not need to be augmented with additional explanation or materials.

 The main ideas of the subject matter addressed in the object are clearly identified for the learner.

2: An object is rated strong for explanation of subject matter if it explains the subject matter in a way that makes skills, procedures, concepts, and/or information understandable. It falls short of superior in that it does not make connections among important associated concepts within the subject matter. For example, a lesson on multi-digit addition may focus on the procedure and fail to connect it with place value.

1: An object is rated limited for explanation of subject matter if it explains the subject matter correctly but in a limited way. This cursory treatment of the content is not sufficiently developed for a first-time learner of the content. The explanations are not thorough and would likely serve as a review for most learners.

0: An object is rated very weak or no value for explanation of subject matter if its explanations are confusing or contain errors. There is little likelihood that this object will contribute to understanding.

N/A: This rubric is not applicable (N/A) for an object that is not designed to explain subject matter, for example, a sheet of mathematical formulae or a map. It may be possible to apply the object in some way that aids a learner’s understanding, but that is beyond any obvious or described purpose of the object

---

As a political scientist, I deal with a lot of theoretical concepts. Hence, when evaluating OER resources I really focus on the quality and applicability of the theoretical content (how accurately is it explained, is it easy to follow, are there real world examples, etc.,). 

Here is my evaluation criteria:

1. Comprehensibility 

• clear, well-written content
• level appropriate
• well-organized; logical sequence
• consistent language and formatting

2.  Accuracy

• accurate, current content
• grammatically correct, no typographical errors

3.  Usability

• table of contents, index, chapter headings
• navigable interface
• working links, video, audio, etc.

4. Adaptability 

• open license
• easily adapted, modified, updated, or rearranged as necessary

5. Accessibility

• simple, readable font in accessible colors; no blinking or moving text
• alternative text for images, charts, and diagrams
• transcripts for audio and video