ABSTRACT

Objective:

This study aimed to evaluate the quality, reliability, and content usefulness of videos created by orthodontists on clear orthodontic aligners.

Methods:

Videos were screened using YouTube^TM by conducting a search for “Invisalign”. After a preliminary evaluation of the first 250 results, 61 videos that met the selection criteria were scored and their length, days since upload, and numbers of views, likes, dislikes, and comments were recorded. These data were used to calculate the interaction index and viewing rate. Video reliability was assessed using a five-item modified DISCERN index, and video quality was assessed using the Video Information and Quality Index. A 10-item content usefulness index was created to determine the usefulness of the video content. Descriptive statistics of the parameters were calculated, and correlation coefficients were calculated to evaluate the relationships between the parameters.

Results:

The mean reliability score was 2.75±1.02 (out of 5), and the total quality score was 11.80±3.38 (out of 20). The total content usefulness index was quite low, with a mean score of 2.52±2.14 (out of 10). Interaction index and viewing rate were positively correlated with reliability score (r=0.463, p<0.01; r=0.295, p<0.05) and total quality score (r=0.365, p<0.01; r=0.295, p<0.01, respectively). The reliability score was positively correlated with the total quality score (r=0.842, p<0.01) and total content usefulness index (r=0.346, p<0.01).

Conclusion:

Videos about orthodontic aligner treatment have average reliability and quality but largely insufficient content.

Main Points

• Videos about orthodontic aligner treatment have average reliability and quality but insufficient content.

• The reliability, quality, and content usefulness of the videos are interrelated.

• Video interaction and viewing rates were associated with video quality and reliability, suggesting that viewers should consider these factors

• Orthodontists should pay attention to issues such as information flow, consistency, image use, and enrichment of the content while creating video content.

INTRODUCTION

Humanity’s striving through the ages to reach a higher esthetic has led to advances in technology that have also impacted the field of orthodontics and brought along with it the search for more esthetic methods to achieve the ideal result. One of the most recent examples of these methods is the use of clear orthodontic aligners. The evolution and adoption of clear aligners, which started with the introduction of thermoplastic tooth positioning appliances by Kesling,¹ accelerated after the US Food and Drug Administration approved clear aligners produced by Align Technology© in 1998. The widespread clinical application and research about clear aligner treatments led to their current popularity.^2,3 Patients welcomed these practices with a wave of curiosity and a desire to learn more about them.

YouTube^TM and similar video sharing sites allow users to share their experiences and knowledge and provide access to audiovisual information about various areas. Studies have shown that most active internet users access health information online.^4,5 Today, parameters such as the smile and physical appearance have an important place in many people’s lives and lead younger people to continually produce more content, especially on YouTube^TM, on subjects such as oral hygiene and dental treatments.⁶ The growing number of dentistry-related videos produced both by “YouTubers” and healthcare professionals has recently attracted researchers’ interest in terms of examining their content, quality, and reliability.^{7,8,9,10,11,12,13}

With their increasing popularity, more videos about orthodontic aligners are appearing daily. However, previous studies have raised questions about whether this increasing amount of content is a source of information pollution and misdirection and about the quality and reliability of the videos.^10,14 While content created by patients is primarily based on sharing personal opinions and experience, orthodontists’ main goals in producing content are to provide accurate information to patients or the people to whom a treatment is targeted and to produce high-quality, reliable content based on scientific data. To the best of our knowledge, no studies have investigated YouTube^TM content related to clear aligners created only by orthodontists.

Considering the ethical responsibility of this information, this study aimed to evaluate the quality, reliability, and usefulness of content in videos created by orthodontists about orthodontic aligners. The null hypothesis of this study was “quality, reliability, and content usefulness of videos created by orthodontists about orthodontic aligners are high”.

METHODS

The Google Trends application (https://trends.google.com) provides users with statistical information about the geographical regions, languages, and frequencies for which words or sentences are searched. This site was utilized to determine the most commonly used search term for “orthodontic aligner” worldwide based on searches using various keywords. Search parameters were limited to the last 5 years and worldwide. The search was conducted using the keywords “aligners,” “clear aligners,” “teeth aligners,” “Invisalign,” “SmileDirectClub,” “ClearCorrect,” “Byte,” and “Candid.” According to comparative search results, the most commonly used search term related to “orthodontic aligners” was “Invisalign” (Google Trends, April 23, 2021).

The YouTube^TM website (https://www.youtube.com) was used to screen videos on “orthodontic aligners.” A search for the word “Invisalign” was conducted, and the results were sorted using the “relevance” filter (April 25, 2021). All cookies and past searches were cleared before searching to prevent bias. The first 250 videos in the search results were evaluated. As the order of the videos shown can change in searches performed on different days, a new playlist was created from the evaluated videos in the same order, and uniform resource locators were saved. Multi-part videos were evaluated as a single video.

In the initial evaluation of the videos, videos created by companies/manufacturers, blogs and promotional videos made by aligner users, videos in languages other than English, videos with no audio and/or subtitles, videos irrelevant to the topic of clear aligners, clinic promotional videos not including orthodontists, and videos longer than 15 min were excluded (Table 1). A total of 61 videos that met these criteria were included in the analysis.

Table 1

RESULTS

The interobserver correlation coefficients were in the range of 0.754-0.981, indicating high agreement between the two raters. Both raters were consistent in repeated assessments, with intraobserver correlation coefficients of 0.941-0.985 for the first rater and 0.885-0.982 for the second rater. Therefore, the statistical analyses were based on the evaluations of the senior orthodontist (E.C.).

The descriptive statistics of the 61 videos evaluated are presented in Table 3. The mean DISCERN reliability score was 2.75±1.02, while the mean VIQI total quality score was 11.80±3.38, approximately half of the maximum possible score of 20. Of the VIQI quality criteria, the mean scores were above average for video flow (3.11±1.29), information accuracy (3.67±0.97), and precision (3.44±1.50), while quality (use of images, use of animations, including interviews with community members, video subtitles, and using a summary report) had the lowest score (1.57±0.88). The total content usefulness index was quite low at 2.52±2.14 (Table 3).

Table 3

When the relationships between video characteristics, reliability score, quality scores, and content usefulness index were evaluated, significant positive correlations were detected between video duration and reliability score (r=0.542, p<0.01), flow (r=0.564, p<0.01), information accuracy (r=0.541, p<0.01), and total quality score (r=0.497, p<0.01). Days since upload negatively correlated with reliability score (r=-0.332, p<0.01) and total quality score (r=-0.263, p<0.05). The number of views was positively correlated with flow (r=0.275, p<0.05), while the number of likes was positively correlated with flow (r=0.375, p<0.01) and information accuracy (r=0.357, p<0.01). In addition, the number of comments was positively correlated with flow (r=0.359, p<0.01), information accuracy (r=0.302, p<0.05), and total quality score (r=0.257, p<0.05). Similarly, both interaction rate and viewing rate were positively correlated with reliability score (r=0.463, p<0.01; r=0.295, p<0.05), flow (r=0.460, p<0.01; r=0.420, p<0.01), information accuracy (r=0.448, p<0.01; r=0.325, p<0.05), and total quality score (r=0.365, p<0.01; r=0.295, p<0.01, respectively) (Table 4).

Table 4

Analysis of the relationships between reliability score, quality scores, and total content usefulness index revealed that reliability score was positively correlated with flow (r=0.842, p<0.01), information accuracy (r=0.786, p<0.01), precision (r=0.533, p<0.01), total quality score (r=0.842, p<0.01), and total content usefulness index (r=0.346, p<0.01). Flow was positively correlated with information accuracy (r=0.773, p<0.01), precision (r=0.371, p<0.01), total quality score (r=0.803, p<0.01), and total content usefulness index (r=.389, p<0.05). There were similar relationships between information accuracy and precision (r=0.543, p<0.01) and total quality score (r=0.847, p<0.01). The quality parameter was positively correlated with the total quality score (r=0.381, p<0.01) and total content usefulness index (r=0.365, p<0.01). There were also significant positive correlations between precision and total quality score (r=0.766, p<0.01) and between total quality score and total content usefulness index (r=0.347, p<0.01) (Table 5).

Table 5

DISCUSSION

Increased sharing of knowledge and experience related to aligners through social media has led to research evaluating this content.^10,14,19,20 To the best of our knowledge, three previous studies in the literature have evaluated videos about clear aligners.^10,14,21 In a study evaluating YouTube^TM content related to orthodontic aligners, Ustdal and Guney¹⁰ reported that the content produced was insufficient and unreliable, with only 12 of the 100 videos selected created by dentists or orthodontists. Sadry and Buyukbasaran²¹ also found YouTube videos lacking as a source of information on orthodontic treatment with clear aligners. Livas et al.¹⁴ conducted another study evaluating patient testimonials. In planning this study, the starting point was to evaluate whether content produced by orthodontists is appropriate for patients, and if video reliability, quality, and content contribute to video interaction and viewing. Therefore, unlike other studies, we comprehensively evaluated video content created only by orthodontists to inform patients, rather than videos made by aligner users. The Google Trends app identified “Invisalign” as the most searched term related to “orthodontic aligners”. “Invisalign” term was used in this study However, the use of other terms such as “clear aligners” during the study would have allowed the video alternatives to be diversified. Therefore, using a single term in searches may be a limitation for this study.

Studies of search engine user behavior have reported that users tend to focus on the first few results encountered without scrolling further down the page.^22,23 In previous studies, it was emphasized that 90% of YouTube^TM users clicked on results within the first 3 pages, and only a small proportion of users continued beyond the first page.^{7,8,9,10,11,12} Considering this, we expanded our search to include the first 250 results. Although evaluating many videos is a strength, it may also be a limitation considering the evidence that lower-ranking studies are less likely to attract attention. Additionally, videos longer than 15 minutes were excluded based on user behavior data, as most sessions are less than 15 minutes.²² This exclusion aimed to ensure user interest and facilitate simultaneous evaluation.

According to the results of this study, the reliability and total quality scores of the videos were near the middle of the possible score range. Similarly, Ustdal and Guney¹⁰ found these parameters to be close to average, whereas Lena and Dindaroğlu⁷ evaluated videos related to lingual orthodontic treatment and reported a slightly higher total quality score. Within the VIQI total quality score, video quality scored the lowest in this study. This was because most of the videos lacked images or animations and did not include the opinions/experiences of treated individuals. Therefore, incorporating more visuals and patient experiences is recommended to enhance video quality. Additionally, the total content usefulness index score of the videos in this study was well below average, compared to similar studies in dentistry.^{7,10,24,25,26} A major limitation was that most videos focused on specific topics. It may be reasonable to focus on specific points related to aligner treatment, and it would be unreasonable to expect all videos to cover all the details relevant to the subject. However, omitting background and key informations may lead to the misconceptions. Therefore, to increase the usefulness of the content, it would be beneficial to provide brief, evidence-based information highlighting the definition and main points of treatment in these videos.

When evaluating the relationships between video characteristics and their reliability, quality, and content usefulness, it was observed that longer video length correlated with higher reliability and total quality scores. This finding aligns with studies by Yavan and Gökçe²⁶, where videos on adult orthodontics with richer content and scored higher in quality.²⁶ Lena and Dindaroğlu⁷ reported that viewers lost interest in longer videos, with the average length of rich-content videos was 7.47 minutes. Although previous studies have shown that long videos are not preferred by viewers, longer durations were associated with better video quality and reliability. While this may seem like a dilemma for content creators, the positive correlation between interaction and viewing rates with video reliability, information flow, accuracy, and total quality score indicates that viewers value and are influenced by these factors. Therefore, orthodontists who create content should consider these aspects and develop videos based on scientific data while keeping them at an acceptable length.

Another interesting finding was that video reliability and total quality score decreased with longer time since upload. This suggests that more recently posted videos are perceived as more reliable and higher in quality. This could be attributed to the continuous improvement in knowledge, experience, and technology related to aligner treatment, as well as advancements in video technology/quality over time. Therefore, regular updates of content can be beneficial for maintaining video quality and reliability.

When the correlations between reliability score, quality score, and total content usefulness index were evaluated, significant positive relationships were observed among them. Previous studies have shown that video quality, reliability, and content are interrelated parameters.^7,10 Therefore, using reliable sources, results in more useful content that provides balanced and consistent information. Similarly, as information flow, accuracy, and precision improve, videos become more reliable and useful. Consequently, it’s essential to consider these parameters collectively rather than separately.

CONCLUSION

The null hypothesis was rejected. The results reveal that videos on aligner treatment have average reliability and quality but largely insufficient content. Video interaction and viewing rates were associated with video quality and reliability, suggesting that viewers should consider these factors. In addition, the reliability, quality, and content usefulness of videos are interrelated. Therefore, when orthodontists create content, providing balanced and current scientific information, paying attention to issues such as information flow, consistency, and image use, and enriching the content accordingly will be beneficial both to ensure that patients are appropriately informed and to generate more interaction.

Video Assessment

All videos were watched in their entirety, and data on the number of views, likes, dislikes, comments, time since upload (in days), and video length (in seconds) were recorded. Using these data, interaction index and viewing rate formulas that have been used in previous studies to determine viewer interaction and viewing rates were employed.^7,10,12 In this study, similar rates for each video were determined using the following formulas:

$\mathrm{Interaction} \mathrm{index} (\%) =\frac{\mathrm{number} \mathrm{of} \mathrm{likes} - \mathrm{number} \mathrm{of} \mathrm{dislikes}}{\mathrm{number} \mathrm{of} \mathrm{views}} \times \hspace{0.25em}100$

$\mathrm{Viewing} \mathrm{rate} (\%) =$

DISCERN (Quality Criteria for Consumer Health Information), a 16-item tool published in 1999 for assessing written information, is valuable for determining the reliability and quality of written text. However, its questions may not be suitable for web and video formats.¹⁵ Therefore, in previous studies, investigators preferred to use a modified version this tool consisting of 5 questions to evaluate the information reliability of videos.^10,11,16 The five-item modified index was also used to assess video reliability in this study (Table 2). While assessing the videos, each question was scored as 0 (no) or 1 (yes), resulting in a reliability score between 0 and 5.

Table 2

Video quality was assessed using the Video Information and Quality Index (VIQI), which corresponds all components of the Global Quality Scale used to assess the quality of websites.^17,18 Although the Global Quality Scale was used t in some similar studies to determine video quality, VIQI was preferred because it is more appropriate for video assessment. In VIQI, video quality is rated on a 5-point Likert-type scale (0=poor quality, 5=high quality) in four different areas: flow, accuracy of the information, quality (1 point each for using images, using animations, including interviews with community members, including subtitles, and using a summary report), and precision (level of agreement between video title and content). These scores are totaled to obtain a total quality score ranging from 0 to 20.⁷

To assess the usefulness of the video content, a 10-part content usefulness index was created: 1. Definition and purpose of aligner treatment, 2. Indications and contraindications of treatment, 3. Advantages and disadvantages, 4. Instructions for using the aligner (daily use time, how it is inserted and removed, cleaning and maintenance instructions), 5. Aligner treatment application procedures, 6. Treatment biomechanics, 7. Comparison with other treatment methods, 8. Effect on quality of life (pain, soft tissue damage, effect on speech, psychosocial effect), 9. Cost of treatment and 10. Duration of treatment. The video was given 1 point for each section it provided information about, yielding a total content usefulness score ranging from 0 to 10.

Statistical Analysis

All evaluations were performed simultaneously and independently by two orthodontists (E.C., 10 years of experience and K.T., 4 years of experience), and interclass correlation coefficients were calculated to evaluate interclass reliability. Two weeks after the first evaluation, 15 of the 61 scored videos were randomly selected using an online randomization website (https://www.randomizer.org) and reevaluated by both researchers. The intraclass correlation coefficient was calculated to determine intrarater reliability. The study data were analyzed using SPSS version 21.0 (IBM Corp, Armonk, NY, USA). The Shapiro- Wilk test was performed to determine whether the data were distributed normally. Descriptive statistics of the parameters were calculated. The Spearman correlation test was used for correlations, and correlation coefficients were calculated to evaluate the relationships between the parameters. A p-value of 0.05 was considered statistically significant.