Validity and Reliability of a Site-Level Assessment Questionnaire to Assess Nutrition and Physical Activity Practices in Schools

Abstract

Background:

Schools are an ideal setting for policy, systems, and environmental approaches to obesity prevention. Although school health environment assessments exist for planning purposes, we developed and tested a comprehensive questionnaire that is suitable for both evaluation and planning.

Methods:

Reliability was measured by comparing data collected by school personnel from low-income elementary schools across California at two time points, an average of 2 months apart (n = 23). To assess convergent validity, school responses were compared with the responses completed by the research team (n = 28). A weighted kappa test statistic and percent agreement were calculated for each question and specific groups of questions (questionnaire section, item topic, and response type).

Results:

Test/retest reliability of the questionnaire yielded kappa statistics that ranged from −0.14 to 1.00 (interquartile range [IQR] 0.36). Percent agreement for reliability ranged from 34.78 to 100.00 (IQR 21.7). Kappa statistics for validity ranged from −0.14 to 1.00 (IQR 0.44). Percent agreement for validity ranged from 14.29 to 100.00 (IQR 39.2). Based on these findings, the tool was revised.

Conclusions:

Study findings indicate that the Site-Level Assessment Questionnaire as tested is a reliable and accurate instrument for use in low-income elementary schools. Revisions may have improved the validity and reliability. We therefore recommend either version for use to support low-income schools in their efforts to assess needs, evaluate progress, and create action plans; and to supply high-quality, aggregable data for large-scale analysis. Additional testing is recommended to validate the revised version, increase generalizability, and determine sensitivity to detect change over time.

Introduction

Schools are widely regarded by the public health field to be a primary target for childhood obesity interventions and there has been an increasing focus on improving the health environment of schools through policy, systems, and environment (PSE) change efforts.^1–4 Mounting evidence indicates that PSE interventions positively impact nutrition, fitness status, and weight of students.^5–9

Initial assessment of a school's health environment, policies, and practices is critical for planning and identifying priority PSE interventions, while reassessment can be used to identify measurable change for ongoing planning and evaluation purposes.^10,11 While instruments are currently available to help programs assess needs and plan health interventions, most are designed exclusively for program planning and lack enough specificity for evaluating comprehensive obesity prevention programs at the school level, focus on a particular strategy (e.g., cafeteria/meal programs, physical activity), or assess written policies rather than implementation.^12–16

Consequently, there remains a need for an instrument that comprehensively assesses school nutrition and physical activity practices and that also includes the necessary measurement properties (e.g., reliability, validity, sensitivity) to allow comparison between two time points and to detect meaningful differences between sites.^17,18 There is a particular need for tools that can be used to measure wellness policy implementation and compliance at the school level, as required by the Local School Wellness Policy Final Rule of the Healthy Hunger-Free Kids Act of 2010.^19–21 Recent research showed that the majority of local school authorities have not met this requirement and identified a need for assessment tools to overcome the challenges school administrators cited in monitoring and evaluating the local wellness policy.²²

We developed the Elementary School Site-Level Assessment Questionnaire (SLAQ) as a comprehensive yet concise instrument that supports planning and evaluation of school-based nutrition and physical activity programs. In California, all local health departments using the Supplemental Nutrition Assistance Program Education (SNAP-Ed) funding to implement PSE interventions in low-income elementary schools are required to work with their partner schools to complete a SLAQ before implementation begins, and annually thereafter.

The questionnaire was designed to be straightforward for school personnel to complete, requiring minimal technical nutrition or physical activity knowledge, and to provide immediate feedback to school personnel and their partners to support an informed program planning process. Initial testing of these instruments demonstrated content validity and feasibility.²³ Revisions identified during each stage of development and testing were incorporated before the present study, which aimed to test the convergent validity and reliability of the SLAQ for elementary schools. All aspects of instrument development and testing were funded by SNAP-Ed.

Materials and Methods

Study Sample

Low-income elementary schools (≥50% of students enrolled in free or reduced price meals) serving students in grades K-6 were recruited from across California between May 2019 and January 2020 through partnerships with local SNAP-Ed implementing agencies (such as health departments and cooperative extension agencies) that offered SNAP-Ed programming to the schools. The implementing agencies referred interested schools to the research team to schedule school site visits and interviews, collect relevant documents, and follow up on SLAQ completion. The research team contacted 41 schools, of which 13 either did not respond or declined to participate after receiving additional study details.

Therefore, a total of 28 schools were included in the validity analysis. Five schools did not complete a second questionnaire, reducing the reliability sample to 23 schools. Each school received a payment of $500 for its participation in this study. School staff participating in key informant interviews received a $5 gift card for each interview section completed. Interviewees included school food service managers, principals, wellness leads, physical education leads, and health education leads; interviewees varied from school to school depending on which staff were the most knowledgeable about a given aspect of nutrition and physical activity policies and practices at that school.

Because this study did not include human subjects and was not considered systematic research designed to contribute to generalizable knowledge, it did not require a human subjects research review by the university Institutional Review Board.

Materials

The Elementary School SLAQ was developed by a team of researchers (a subset of the authors) who conducted a scan of the existing environmental policy and practice assessment instruments from a variety of sources, including the SNAP-Ed Evaluation Framework's recommended evaluation tools and the National Collaborative on Childhood Obesity Research (NCCOR) Measures Registry.^24–26 Items from existing instruments^{12,14,15,27–34} were adapted to a standardized statement-and-response format, new items were constructed to fill gaps, and all items were aligned with regulations and recommendations from state and national government and nongovernment organizations, including the Local School Wellness Policy Final Rule of the Healthy Hunger-Free Kids Act of 2010 and USDA's SNAP-Ed Evaluation Framework.^11,19–21

The version of the questionnaire tested in this study resulted from an expert panel review for content validity, pilot testing at a small sample of schools, and subsequent revisions.²³ The questionnaire contained 129 items (grouped into 100 questions) in nine sections, each consisting of a practice statement followed by response options. To optimize item sensitivity, wherever possible, response options indicated degree of implementation on a 5-point Likert-type scale; some items used a trichotomous scale or yes/no responses. Most items were scored on a scale of 0–4 (5-point: 0, 1, 2, 3, 4; trichotomous: 0, 2, 4; yes/no: 0, 4); a few items were scaled 0–1 or 0–2 because these individual items were judged to merit less weight than the items coded 0–4.

In all cases, a high score represented the best practice for that item. Most items were phrased as best practices; for purposes of reading clarity, some items were phrased in the negative and reverse-coded. The nine questionnaire sections were organized by topic and respondent type, for example, grouping together items to be completed by school food service, to facilitate questionnaire completion. This allows for school personnel to skip to relevant sections of the questionnaire, reducing the completion time for each person. The types of personnel most likely to have the content expertise to complete each question were identified during pilot testing. Questionnaires were completed as fillable Adobe PDFs, programmed with skips to reduce error, with auto-calculated section and overall scores that allow respondents to see results immediately upon completion.

Procedures

School personnel completed the Elementary School SLAQ at two time points, separated by ∼2 months. To determine convergent validity of the questionnaire, we compared school personnel's responses on the questionnaire at time one (original) with questionnaire responses completed by the research team (validation).

The research team conducted on-site observations, reviewed school wellness policies and procedures, reviewed district or school nutrition services menus and web pages, and interviewed district and school staff to collect the information needed to complete the questionnaires. The research team collected validation data within 112 days [mean 38 (28)] of each school's time one responses. To test reliability, we compared school personnel's time one (original) responses with their time two (reliability) responses, collected within 180 days [mean 66 (47)] of each school's time one responses.

Data Analysis

To assess the degree of agreement between the original and validation observations and between the original and reliability observations, we calculated the following statistics for each item. Percent exact agreement was calculated between original and reliability as well as between original and validity records for all items, categorical or continuous. The average percent agreement among items belonging to the same questionnaire section, topic, and response type was also calculated to identify patterns of high or low validity and reliability.

We also calculated weighted kappa statistics to examine the degree of agreement between original and reliability records as well as original and validation records.³⁵ To further understand how well the questionnaire's continuous section scores ranked sites, score tertile cut points were calculated for the original observations, reliability observations, and validation observations. These tertile cut points were then used to classify records into one of three categories: records with section score values in the first tertile, records with section score values in the second tertile, and records with section score values in the third tertile.

Weighted kappa statistics were then calculated to assess the degree of agreement in the ranking of records into these section score tertiles between original and reliability as well as between original and validation records. Following the method developed and tested by Muñoz and Bangdiwala,³⁶ weighted kappa values for each item were assigned to the following strength categories: poor (<0), fair (0–0.2), moderate (0.2–0.45), substantial (0.45–0.75), and almost perfect (0.75–1).

Post hoc power analysis assessed the likelihood of observing differences in kappa based on study parameters, including sample size, number of observers, and observed values of a primary study outcome (question 2.7, more than one vegetable choice is offered at lunch). Using a significance level of 0.05 and the prevalence of responding “Never,” “Not usually,” “Sometimes,” “Usually,” and “Always” to question 2.7 to be 0.1071, 0.0357, 0.0357, 0.0714, and 0.7500, respectively, with 26 schools and 2 raters, we would have 80% power to detect a difference between validity kappa values of 0 and 0.275. Using a significance level of 0.05 and the prevalence of responding “Never,” “Not usually,” “Sometimes,” “Usually,” and “Always” to question 2.7 to be 0.0526, 0.0526, 0.000001, 0.1579, and 0.7368, respectively, with 19 schools and 2 raters, we would have 80% power to detect a difference between reliability kappa values of 0 and 0.344.

Results

Validity analyses included a group of 28 diverse elementary schools representing a range of California locales (Table 1); reliability analyses included 23 of the schools from the validity sample. In both samples, the majority of students attending the study schools were low income and Hispanic.

Table 1.

Characteristics of Schools in Elementary School Site-Level Assessment Questionnaire Study Sample

	Validity sample (N = 28), n (%)	Reliability sample (N = 23), n (%)
School grade levels³⁷
K-5	16 (57.1)	15 (65.2)
K-6	7 (25.0)	5 (21.7)
Other	5 (17.9)	3 (13.0)
Urbanicity³⁸
Urban	10 (35.7)	9 (39.1)
Suburban	15 (53.6)	11 (47.8)
Rural	3 (10.7)	3 (13.0)
Location
Northern California	9 (32.1)	8 (34.8)
Central California	9 (32.1)	6 (26.1)
Southern California	10 (35.7)	9 (39.1)
Student demographics, mean (SD)
Number of students enrolled³⁷	512 (173)	518 (162)
Percent of students receiving FRPM³⁷	82.0 (14.0)	82.3 (14.1)
Percent white students³⁹	9.2 (13.6)	7.4 (6.4)
Percent Hispanic students³⁹	78.1 (18.3)	78.7 (14.0)
Percent English language learner students⁴⁰	40.4 (19.8)	40.9 (19.0)

FRPM, free or reduced price meals; N, number; SD, standard deviation.

Validity

Results of the validity analyses are reported for all items in Table 2. Weighted kappa statistics ranged from −0.14 to 1.00 across all items (interquartile range [IQR] 0.44). Weighted kappa was unable to be computed for seven items, for which original and validity observations were identical. The least valid items asked about using food for rewards and punishments (weighted kappa −0.14, 95% confidence interval [CI] −0.37 to 0.09) and student promotion of the meal program and healthy meals (weighted kappa −0.14, 95% CI −0.31 to 0.03). Two items with weighted kappa values of 1.00 asked about the presence of breakfast programs and noncompliant beverages. Moderate or better validity was obtained for 55.0% of items (Table 2). Percent agreement for validity ranged from 14.29 to 100.00 (IQR 39.2).

Table 2.

Elementary School Site-Level Assessment Questionnaire Item Validity and Reliability Statistics with Strength Ratings and Decision About Items

Item	Validity				Reliability				Decision
Item	n	Weighted kappa (95% CI)	Percent agreement	Strength^a	n	Weighted kappa (95% CI)	Percent agreement	Strength^a	Decision
1.1. Wellness policy compliance and progress^a,b	25	0.60 (0.29 to 0.91)	71.43	Substantial	14	0.29 (−0.21 to 0.78)	43.48	Moderate	Revised
1.2. Active school-level wellness committee^a,b	23	0.33 (0.06 to 0.61)	42.86	Moderate	15	0.77 (0.54 to 0.10)	56.52	Almost perfect	Revised
1.3. Family participation on wellness committee^a,b	14	0.44 (−0.01 to 0.89)	60.71	Moderate	12	1.00 (1.00 to1.00)	82.61	Almost perfect	Revised
1.4. School breakfast program participation^a,b	26	−0.07 (−0.21 to 0.06)	60.71	Poor	19	0.00 (0.00 to 0.00)	82.61	Fair	No change
1.5. Fresh fruit and vegetable program participation^a,b	26	0.21 (−0.15 to 0.57)	57.14	Moderate	19	0.87 (0.63 to 1.00)	82.61	Almost perfect	No change
1.6. Special milk program participation^a,b	26	0.00 (−0.00 to −0.00)	39.29	Fair	19	0.68 (0.35 to 1.00)	73.91	Substantial	Removed
1.7. California State meal program^a,b	25	0.03 (−0.02 to 0.09)	21.43	Fair	17	0.30 (−0.29 to 0.90)	65.22	Moderate	Removed
1.8. Seamless summer option^a,b	26	0.19 (−0.19 to 0.57)	57.14	Fair	18	0.75 (0.44 to 1.00)	73.91	Almost perfect	Removed
1.9. Maximized free/reduced price meal enrollment^a,b	27	0.18 (−0.30 to 0.66)	78.57	Fair	18	0.64 (0.00 to 1.00)	78.26	Substantial	No change
1.10. Protection of student program participation^b,c	27	0.00 (0.00 to 0.00)	92.86	Fair	18	0.00 (0.00 to 0.00)	78.26	Fair	No change
1.11. Written policy for section 1^b	24	−0.08 (−0.22 to 0.06)	57.14	Poor	15	0.21 (−0.31 to 0.73)	52.17	Moderate	Removed
2.1. School breakfast programs^a,b	27	0.73 (0.33 to 1.00)	89.29	Substantial	20	0.77 (0.35 to 1.00)	86.96	Almost perfect	No change
2.1.a. Universal free breakfast^a,c	28	0.55 (0.21 to 0.89)	82.14	Substantial	23	0.51 (0.16 to 0.88)	78.26	Substantial	No change
2.1.b. Breakfast in the classroom^a,c	28	0.89 (0.67 to 1.00)	96.43	Almost perfect	23	0.62 (0.16 to 1.00)	91.30	Substantial	No change
2.1.c. “Grab and go” breakfast^a,c	28	−0.04 (−0.09 to 0.01)	92.86	Poor	23	0.65 (0.01 to 1.00)	95.65	Substantial	No change
2.1.d. “Second chance” breakfast^a,c	28	0.58 (0.20 to 0.95)	85.71	Substantial	23	1.00 (1.00 to 1.00)	100.00	Almost perfect	No change
2.1.e. Breakfast offered, none of above programs^a,c	28	0.47 (−0.13 to 1.00)	92.86	Substantial	23	0.78 (0.36 to 1.00)	95.65	Almost perfect	No change
2.1.f. No breakfast offered^a,c	28	1.00 (1.00 to 1.00)	100.00	Almost perfect	23	NA	100.00	NA	No change
2.2. Meals meet National School Lunch Act standards^b	27	NA	96.43	NA	19	0.00 (0.00 to 0.00)	82.61	Fair	Revised
2.3. Meal entrees prepared from scratch^a	26	0.03 (−0.18 to 0.23)	14.29	Fair	20	0.50 (0.16 to 0.84)	60.87	Substantial	Removed
2.4. Sliced or cut fruit offered with meals^b,c	27	0.26 (−0.00 to 0.52)	39.29	Moderate	20	0.53 (0.28 to 0.77)	56.52	Substantial	No change
2.5. Prominent and attractive display of fruits/vegetables^c	26	0.23 (0.02 to 0.43)	42.86	Moderate	19	0.38 (0.08 to 0.68)	60.87	Moderate	No change
2.6. More than one fruit choice at lunch^b,c	26	0.27 (−0.02 to 0.57)	39.29	Moderate	19	0.54 (0.16 to 0.93)	65.22	Substantial	No change
2.7. More than one vegetable choice at lunch^b,c	26	0.48 (0.17 to 0.80)	60.71	Substantial	19	0.73 (0.44 to 1.00)	69.57	Substantial	No change
2.8. Packaged salads or salad bar^c	26	0.63 (0.42 to 0.84)	60.71	Substantial	18	0.74 (0.52 to 0.96)	60.87	Substantial	Revised
2.9. Seasonings and healthy dips for vegetables^c	25	0.24 (−0.04 to 0.52)	39.29	Moderate	17	0.33 (−0.03 to 0.70)	43.48	Moderate	Removed
2.10. Healthy meal beverages^b,c	26	0.35 (−0.01 to 0.71)	50.00	Moderate	19	0.36 (−0.03 to 0.75)	52.17	Moderate	Revised
2.11. Healthy milk types^b,c	26	0.40 (0.12 to 0.67)	50.00	Moderate	19	0.59 (0.30 to 0.88)	56.52	Substantial	Revised
2.12. Placement of milk types in coolers^c	11	0.27 (−0.03 to 0.57)	32.14	Moderate	10	0.56 (0.17 to 0.95)	39.13	Substantial	Revised
2.13. Drinking water in food service areas^c	21	0.64 (0.01 to 1.00)	71.43	Substantial	12	0.75 (0.30 to 1.00)	60.87	Almost perfect	No change
2.14. Pricing of less nutritious foods and beverages^c	24	−0.06 (−0.14 to 0.02)	75.00	Poor	17	0.6 (0.13 to 1.00)	65.22	Substantial	Removed
2.15. Fruit and vegetable servings with meals^b,c	26	−0.02 (−0.05 to 0.01)	85.71	Poor	19	0.00 (0.00 to 0.00)	78.26	Fair	No change
2.16. Cafeteria reinforcement of nutrition education^a,c	14	−0.09 (−0.27 to 0.08)	32.14	Poor	13	0.15 (−0.17 to 0.47)	34.78	Fair	Revised
2.17. Foods served reinforce nutrition education^a	17	0.03 (−0.39 to 0.46)	32.14	Fair	12	0.43 (−0.02 to 0.87)	34.78	Moderate	Removed
2.18. Appropriate length of lunch period^b	25	0.09 (−0.15 to 0.32)	17.86	Fair	18	0.68 (0.47 to 0.88)	47.83	Substantial	Revised
2.19. Appropriate timing of lunch period^b	24	0.28 (0.03 to 0.52)	57.14%	Moderate	19	0.47 (0.12 to 0.81)	60.87%	Substantial	No change
2.20. Space for students to eat meals^c	26	0.01 (−0.34 to 0.35)	64.29	Fair	19	0.64 (0.00 to 1.00)	82.61	Substantial	No change
2.21. Acceptability of dining facilities^c	26	0.24 (−0.08 to 0.57)	57.14	Moderate	19	0.66 (0.30 to 1.00)	69.57	Substantial	No change
2.22. Invitingness of dining facilities^c	26	0.12 (−0.24 to 0.47)	67.86	Fair	19	−0.14 (−0.28 to −0.01)	60.87	Poor	Removed
2.23. Professional development for food service staff^a,b	24	0.59 (0.18 to 1.00)	75.00	Substantial	16	0.53 (0.08 to 0.99)	56.52	Substantial	Removed
2.24. Written policy for section 2^b	22	0.00 (0.00 to 0.00)	64.29	Fair	14	0.05 (−0.48 to 0.58)	43.48	Fair	Removed
3.1. Food used as reward or punishment^a	26	−0.14 (−0.37 to 0.09)	14.29	Poor	19	0.31 (0.05 to 0.58)	47.83	Moderate	Removed
3.2. Staff modeling of dietary behavior^a,c	27	0.05 (−0.11 to 0.21)	28.57	Fair	21	0.27 (−0.09 to 0.62)	52.17	Moderate	Removed
3.3. Food and beverage advertising^a–c	22	−0.02 (−0.05 to 0.01)	75.00	Poor	17	0.35 (−0.21 to 0.92)	65.22	Moderate	No change
3.4. Fundraising^a	26	−0.05 (−0.26 to 0.16)	17.86	Poor	20	0.34 (0.06 to 0.62)	43.48	Moderate	Revised
3.5. Food at events and celebrations^a	26	0.01 (−0.18 to 0.20)	17.86	Fair	21	0.53 (0.23 to 0.82)	56.52	Substantial	Revised
3.6. Nonmeal foods sold (school day)^a	20	−0.02 (−0.05 to 0.01)	64.29	Poor	16	0.35 (−0.21 to 0.91)	60.87	Moderate	Revised
3.7. Nonmeal foods sold (afterschool)^a	23	0.14 (−0.11 to 0.38)	50.00	Fair	19	0.28 (−0.15 to 0.70)	52.17	Moderate	Revised
3.8. Placement of nonmeal foods^c	22	0.00 (0.00 to 0.00)	57.14	Fair	16	−0.12 (−0.23 to −0.01)	52.17	Poor	Removed
3.9. Venues selling nonmeal foods^a	27	0.28 (−0.10 to 0.65)	71.43	Moderate	21	0.24 (−0.28 to 0.75)	73.91	Moderate	Removed
3.10. Beverages sold on campus^a,c	28	0.29 (−0.10 to 0.67)	75.00	Moderate	23	0.35 (−0.01 to 0.70)	69.57	Moderate	No change
3.10.a. No beverages sold on campus^a,c	28	0.33 (−0.03 to 0.69)	75.00	Moderate	23	0.51 (0.13 to 0.88)	78.26	Substantial	No change
3.10.b. Unflavored water^a,c	28	0.36 (−0.16 to 0.89)	89.29	Moderate	23	0.50 (0.01 to 0.99)	86.96	Substantial	No change
3.10.c. Unflavored low-fat or nonfat milk^a,c	28	0.36 (−0.16 to 0.89)	89.29	Moderate	23	0.70 (0.31 to 1.00)	91.30	Substantial	No change
3.10.d. Unflavored whole or reduced fat milk^a,c	28	0.00 (0.00 to 0.00)	92.86	Fair	23	0.65 (0.01 to 1.00)	95.65	Substantial	No change
3.10.e. Flavored nonfat milk^a,c	28	0.47 (−0.13 to 1.00)	92.86	Substantial	23	0.62 (0.13 to 1.00)	91.30	Substantial	No change
3.10.f. Flavored whole, low-fat, or reduced fat milk^a,c	28	NA	100.00	NA	23	0.00 (0.00 to 0.00)	95.65	Fair	No change
3.10.g. 100% fruit or vegetable juice^a,c	28	−0.09 (−0.18 to −0.00)	82.14	Poor	23	0.00 (0.00 to 0.00)	86.96	Fair	No change
3.10.h. Calorie-free beverages^a,c	28	NA	100.00	NA	23	NA	100.00	NA	No change
3.10.i. Lower calorie beverages^a,c	28	NA	100.00	NA	23	NA	100.00	NA	No change
3.10.j. Regular, nondiet soda^a,c	28	NA	100.00	NA	23	NA	100.00	NA	No change
3.10.k. Diet drinks^a,c	28	NA	100.00	NA	23	NA	100.00	NA	No change
3.10.l. Other presweetened drinks^a,c	28	1.00 (1.00 to 1.00)	100.00	Almost perfect	23	0.00 (0.00 to 0.00)	95.65	Fair	No change
3.10.m. Other beverages sold (write-in)^a,c	28	NA	100.00	NA	23	NA	100.00	NA	No change
3.11. Beverages at celebrations and events^a	25	−0.09 (−0.40 to 0.23)	28.57	Poor	18	0.62 (0.29 to 0.94)	60.87	Substantial	Removed
3.12. Drinking water availability^a,c	25	0.00 (−0.00 to −0.00)	82.14	Fair	20	0.64 (0.01 to 1.00)	82.61	Substantial	Revised
3.13. Student refillable water bottles^a	26	0.00 (−0.00 to −0.00)	82.14	Fair	21	0.77 (0.35 to 1.00)	86.96	Almost perfect	No change
3.14. Written policy for section 3^b	24	0.00 (0.00, 0.00)	60.71	Fair	18	0.64 (0.27 to 1.00)	69.57	Substantial	Removed
4.1.a. Edible garden on site^a,c	28	0.71 (0.46 to 0.97)	85.71	Substantial	23	0.91 (0.75 to 1.00)	95.65	Almost perfect	Revised
4.1.b. Edible community garden access^a,c	28	−0.04 (−0.09 to 0.01)	92.86	Poor	23	0.00 (0.00 to 0.00)	95.65	Fair	Revised
4.1.c. No garden access^a,c	28	0.71 (0.45 to 0.97)	85.71	Substantial	23	1.00 (1.00 to 1.00)	100.00	Almost perfect	Revised
4.2. Garden incorporated into nutrition education^a	12	0.23 (−0.19 to 0.65)	53.57	Moderate	6	0.65 (0.35 to 0.96)	60.87	Substantial	No change
4.3. How many students tend the garden^a,c	12	0.60 (0.31 to 0.90)	60.71	Substantial	8	0.90 (0.71 to 1.00)	78.26	Almost perfect	No change
4.4. How often students tend the garden^a	13	0.59 (0.22 to 0.95)	64.29	Substantial	9	0.71 (0.35 to 1.00)	73.91	Substantial	No change
4.5. Garden produce distributed to families^a	13	0.46 (0.02 to 0.89)	64.29	Substantial	10	0.29 (−0.17 to 0.74)	73.91	Moderate	No change
4.6. Garden produce used in meals or snacks^a	13	0.22 (−0.10 to 0.54)	53.57	Moderate	9	0.41 (0.03 to 0.79)	65.22	Moderate	Revised
4.7. Written policy for section 4^b	10	−0.36 (−0.74 to 0.01)	53.57	Poor	9	0.77 (0.35 to 1.00)	86.96	Almost perfect	Removed
5.1. Nutrition education^a	28	0.38 (0.12 to 0.63)	42.86	Moderate	20	0.65 (0.38 to 0.91)	60.87	Substantial	Revised
5.2. Standards-based nutrition education curricula^a,b	22	0.24 (−0.04 to 0.52)	42.86	Moderate	17	0.85 (0.68 to 1.00)	65.22	Almost perfect	Revised
5.3. Participatory/hands-on nutrition education^a,b	23	0.23 (−0.07 to 0.53)	35.71	Moderate	17	0.74 (0.49 to 0.99)	56.52	Substantial	Revised
5.4. Professional development for nutrition teachers^a,b	23	0.05 (−0.30 to 0.39)	35.71	Fair	17	0.90 (0.80 to 1.00)	60.87	Almost perfect	Revised
5.5. Student feedback on school meal menus^a	27	0.09 (−0.10 to 0.28)	32.14	Fair	20	0.56 (0.30 to 0.83)	52.17	Substantial	Removed
5.6. Student promotion of school meals^a	27	−0.14 (−0.31 to 0.03)	50.00	Poor	20	0.65 (0.34 to 0.96)	65.22	Substantial	Revised
5.7. Student preferences used for menu planning^a	26	0.46 (0.19 to 0.73)	53.57	Substantial	20	0.91 (0.79 to 1.00)	78.26	Almost perfect	Revised
5.8. Written policy for section 5^b	23	−0.08 (−0.23 to 0.07)	53.57	Poor	19	0.55 (0.16 to 0.94)	65.22	Substantial	Removed
6.1. Minutes of PE^a	27	0.00 (−0.00 to −0.00)	82.14	Fair	22	0.05 (−0.35 to 0.45)	65.22	Fair	Revised
6.2. PE time spent in moderate to vigorous activity^a	27	0.04 (−0.22 to 0.30)	57.14	Fair	22	0.49 (0.14 to 0.84)	69.57	Substantial	No change
6.3. Standards-based PE curriculum^a	27	0.22 (−0.24 to 0.68)	82.14	Moderate	22	0.46 (−0.13 to 1.00)	86.96	Substantial	Revised
6.4. Standards-based PE assessment^a	28	0.55 (0.30 to 0.80)	71.43	Substantial	21	0.66 (0.42 to 0.91)	73.91	Substantial	Removed
6.5. Student/teacher ratio in PE classes^a	27	0.17 (−0.18 to 0.52)	75.00	Fair	22	0.69 (0.30 to 1.00)	86.96	Substantial	No change
6.6. PE teacher certification/endorsement^a	28	0.58 (0.32 to 0.84)	67.86	Substantial	20	0.85 (0.65 to 1.00)	78.26	Almost perfect	No change
6.7. PE teacher professional development^a,b	27	0.54 (0.19 to 0.89)	78.57	Substantial	20	0.66 (0.31 to 1.00)	73.91	Substantial	Revised
6.8. Written policy for section 6^b	25	0.00 (0.00 to 0.00)	60.71	Fair	21	0.67 (0.32 to 1.00)	78.26	Substantial	Removed
7.1. Minutes of recess offered^a,b	26	−0.06 (−0.15 to 0.03)	71.43	Poor	21	0.13 (−0.27 to 0.52)	65.22	Fair	No change
7.2. Staff facilitation of recess physical activity^a,c	28	0.15 (−0.03 to 0.33)	17.86	Fair	23	0.70 (0.52 to 0.80)	69.57	Substantial	No change
7.3. Classroom activity breaks^a	28	−0.06 (−0.25 to 0.13)	14.29	Poor	22	0.28 (0.01 to 0.55)	47.83	Moderate	Revised
7.4. Physical activity offered indoors when needed^a	28	0.14 (−0.10 to 0.37)	53.57	Fair	23	0.47 (0.16 to 0.77)	60.87	Substantial	No change
7.5. Physical activity facilities^a,c	28	0.04 (−0.15 to 0.22)	32.14	Fair	22	0.69 (0.55 to 0.84)	47.83	Substantial	Revised
7.6. Physical activity used for punishment^a	28	0.19 (−0.08 to 0.46)	60.71	Fair	23	0.33 (0.03 to 0.64)	56.52	Moderate	Revised
7.7.a. Organized physical activities before school^a,c	28	0.21 (−0.21 to 0.63)	75.00	Moderate	23	0.23 (−0.22 to 0.69)	73.91	Moderate	No change
7.7.b. Organized physical activities afterschool^a,c	28	0.51 (0.03 to 0.99)	89.29	Substantial	23	0.36 (−0.02 to 0.74)	78.26	Moderate	No change
7.7.c. No organized physical activities before/afterschool^a,c	28	0.63 (0.15 to 1.00)	92.86	Substantial	23	0.25 (−0.27 to 0.76)	82.61	Moderate	No change
7.8. Intramural sports programs^a	27	0.53 (0.27 to 0.78)	64.29	Substantial	22	0.61 (0.32 to 0.90)	69.57	Substantial	No change
7.9. Portable play equipment^c	27	−0.20 (−0.45 to 0.04)	25.00	Poor	23	0.62 (0.38 to 0.86)	73.91	Substantial	Revised
7.10. Fixed play equipment^c	28	0.17 (−0.14 to 0.48)	67.86	Fair	23	0.42 (0.09 to 0.75)	69.57	Moderate	Revised
7.11. Written policy for section 7^b	24	0.00 (0.00 to 0.00)	46.43	Fair	20	0.59 (0.23 to 0.94)	69.57	Substantial	Removed
8.1. Parent volunteers in wellness programs^a	27	−0.06 (−0.34 to 0.21)	28.57	Poor	21	0.25 (−0.01 to 0.52)	56.52	Moderate	Removed
8.2. Referrals to community wellness resources^a,b	28	0.10 (0.00 to 0.20)	35.71	Fair	22	0.78 (0.58 to 0.98)	78.26	Almost perfect	Revised
8.3. Nutrition education for parents^a,b	28	0.16 (−0.19 to 0.50)	60.71	Fair	22	0.74 (0.55 to 0.92)	73.91	Substantial	No change
8.4. Physical activity opportunities for parents^a,b	27	0.23 (−0.10 to 0.57)	53.57	Moderate	21	0.35 (−0.00 to 0.69)	52.17	Moderate	No change
8.5. Nutrition information available to families^a,b	27	0.00 (−0.22 to 0.23)	35.71	Fair	19	0.41 (0.08 to 0.75)	47.83	Moderate	Revised
8.6. Wellness policy information distributed to parents^a,b	27	0.31 (0.02 to 0.60)	67.86	Moderate	20	0.39 (−0.03 to 0.81)	65.22	Moderate	No change
8.7. Communication of PE/physical activity opportunities^a,b	27	0.26 (−0.03 to 0.54)	42.86	Moderate	20	0.24 (−0.09 to 0.58)	47.83	Moderate	Revised
8.8. Written policy for section 8^b	25	0.23 (−0.14 to 0.60)	64.29	Moderate	18	0.51 (0.10 to 0.91)	60.87	Substantial	Removed
9.1. Mobile vendors near school^a,c	25	0.84 (0.67 to 1.00)	78.57	Almost perfect	20	0.76 (0.42 to 1.00)	86.96	Almost perfect	Removed
9.2. Vendor purchasing agreements (how many foods)^a	26	0.04 (−0.15 to 0.23)	25.00	Fair	20	0.52 (0.10 to 0.95)	69.57	Substantial	Removed
9.3. Vendor purchasing agreements (how many vendors)^a	25	0.10 (−0.09 to 0.29)	35.71	Fair	21	0.67 (0.31 to 1.00)	78.26	Substantial	Removed
9.4. Programs promoting local/regional foods for meals^a	27	0.27 (−0.07 to 0.62)	60.71	Moderate	20	0.8 (0.54 to 1.00)	78.26	Almost perfect	No change
9.5. School grounds open for joint use^a–c	27	0.38 (0.14 to 0.62)	42.86	Moderate	22	0.83 (0.67 to 0.99)	78.26	Almost perfect	Removed
9.6. School grounds used for joint use^a,c	27	0.45 (0.19 to 0.72)	53.57	Substantial	22	0.63 (0.37 to 0.89)	65.22	Substantial	Revised
9.7. Guidelines for food brought to school^a,b	28	0.22 (−0.18 to 0.63)	75.00	Moderate	19	0.20 (−0.26 to 0.66)	56.52	Fair	Revised
9.8.a. Sufficient, well-marked crosswalks^c	28	0.21 (−0.14 to 0.57)	67.86	Moderate	23	0.35 (−0.04 to 0.74)	73.91	Moderate	No change
9.8.b. Speed bumps/traffic calming^c	28	0.23 (−0.11 to 0.57)	60.71	Moderate	23	0.57 (0.27 to 0.87)	78.26	Substantial	Revised
9.8.c. Organized walk-to-school groups^a,c	28	0.52 (0.07 to 0.97)	89.29	Substantial	23	0.74 (0.41 to 1.00)	91.30	Substantial	No change
9.8.d. Sufficient cross guards^a,c	28	0.55 (0.24 to 0.87)	78.57	Substantial	23	0.46 (0.09 to 0.82)	73.91	Substantial	No change
9.8.e. Adequate bike racks on campus^c	28	0.53 (0.24 to 0.83)	78.57	Substantial	23	0.65 (0.30 to 1.00)	86.96	Substantial	No change
9.8.f. Other active transport practice (write-in)^a,c	28	0.13 (−0.30 to 0.56)	78.57	Fair	23	0.83 (0.51 to 1.00)	95.65	Almost perfect	No change
9.8.g. No active transport practices^a,c	28	0.29 (−0.16 to 0.74)	85.71	Moderate	23	−0.05 (−0.11 to 0.02)	91.30	Poor	No change
9.9. Written policy for section 9^b	26	0.16 (−0.15 to 0.47)	60.71	Fair	20	0.12 (−0.32 to 0.56)	52.17	Fair	Removed

Key: Poor, weighted kappa <0; Fair, weighted kappa 0–0.2; Moderate, weighted kappa 0.2–0.45; Substantial, weighted kappa 0.45–0.75, Almost perfect, weighted kappa 0.75–1.

Validity of the item assessed by comparing school personnel rating with research team member rating based on key informant interview.

Validity of the item assessed by comparing school personnel rating with research team member rating based on document review.

Validity of the item assessed by comparing school personnel rating with research team member rating based on observation.

CI, confidence interval; NA, not applicable; PE, physical education.

Validity statistics by questionnaire section, item topic (environment, policy, practice, or program participation), and response type are reported in Tables 3 and 4. Weighted kappa statistics for questionnaire section score tertiles ranged from −0.06 (95% CI −0.34 to 0.21) for section 3 (Food and Drink around the School) to 0.69 (95% CI 0.51 to 0.88) for section 4 (Gardens). Average percent agreement ranged from 43.30% (8.42) for section 5 (Nutrition Education and Student Involvement) to 72.09% (28.52) for section 4 (Gardens).

Table 3.

Elementary School Site-Level Assessment Questionnaire Section Score Tertile Ranking Validity and Reliability Using Weighted Kappa

Section score tertiles	Validity (N = 28)	Reliability (N = 23)
Section score tertiles	Weighted kappa (95% CI)	Weighted kappa (95% CI)
Section 1	0.48 (0.23 to 0.73)	0.51 (0.18 to 0.83)
Section 2	0.07 (−0.22 to 0.36)	0.14 (−0.16 to 0.45)
Section 3	−0.06 (−0.34 to 0.21)	−0.26 (−0.52 to 0.01)
Section 4	0.69 (0.51 to 0.88)	0.82 (0.65 to 0.98)
Section 5	0.25 (−0.01 to 0.52)	0.49 (0.20 to 0.78)
Section 6	0.39 (0.15 to 0.64)	0.66 (0.43 to 0.89)
Section 7	0.16 (−0.12 to 0.44)	0.48 (0.18 to 0.78)
Section 8	0.36 (0.08 to 0.65)	0.56 (0.29 to 0.83)
Section 9	0.38 (0.11 to 0.65)	0.47 (0.19 to 0.74)

N, number; CI, confidence interval.

Table 4.

Validity and Reliability of Items by Section, Item Topic, and Response Type Using Percent Agreement

	n	Validity			Reliability
	n	Mean percent agreement (SD)	Minimum percent agreement	Maximum percent agreement	Mean percent agreement (SD)	Minimum percent agreement	Maximum percent agreement
Section
Section 1	11	58.11 (19.37)	21.43	92.86	69.96 (13.67)	43.48	82.61
Section 2	30	62.14 (24.65)	14.29	100.00	66.38 (19.17)	34.78	100.00
Section 3	27	72.09 (28.52)	14.29	100.00	77.62 (19.06)	43.48	100.00
Section 4	9	68.25 (15.61)	53.57	92.86	81.16 (14.09)	60.87	100.00
Section 5	8	43.30 (8.42)	32.14	53.57	63.04 (7.71)	52.17	78.266
Section 6	8	71.88 (9.44)	57.14	82.14	76.63 (7.69)	65.22	86.96
Section 7	13	54.67 (25.98)	14.29	92.86	66.56 (10.71)	47.83	82.61
Section 8	8	48.66 (14.90)	28.57	67.86	60.33 (11.49)	47.83	78.26
Section 9	15	64.76 (18.89)	25.00	89.29	77.10 (12.57)	52.17	95.65
Topic
Environment	16	70.98 (19.03)	25.00	92.86	80.71 (14.37)	47.83	100.00
Policy	11	52.92 (12.56)	25.00	64.29	66.01 (13.02)	43.48	86.96
Practice	95	62.78 (24.75)	14.29	100.00	70.07 (16.97)	34.78	100.00
Program Participation	7	53.57 (18.21)	21.43	78.57	76.40 (6.08)	65.22	82.61
Response type
Dichotomous	56	77.04 (17.87)	21.43	100.00	80.98 (15.35)	43.48	100.00
Trichotomous	19	60.53 (19.79)	25.00	89.29	73.23 (12.15)	47.83	86.96
5-Point Likert scale	54	48.02 (20.10)	14.29	96.43	60.79 (11.69)	34.78	82.61

SD, standard deviation.

Average percent agreement between validity and original records was lowest for policy items [52.92% (12.56)] and highest for environment questions [70.98% (19.03)]. Average percent agreement for 5-point scale items was lowest [48.02% (20.10)] and was highest for dichotomous items [77.04% (17.87)], a pattern of results that would be expected based on the probability of agreement for response type.

Reliability

Results of reliability analyses are reported for all items in Table 2. Weighted kappa statistics ranged from −0.14 to 1.00 across all items (IQR 0.36). Weighted kappa statistics were unable to be computed for six items, for which original and reliability observations were identical. The least reliable item asked how inviting the student eating area is (weighted kappa −0.14, 95% CI −0.28 to −0.01). Several items had the highest possible weighted kappa of 1.00, including one that asked about participation of students' family members on a school wellness committee. Moderate or better reliability was obtained for 86.8% of items. Percent agreement for reliability ranged from 34.78 to 100.00 (IQR 21.7).

Reliability statistics by questionnaire section, item topic (environment, policy, practice, or program participation), and response type are reported in Tables 3 and 4. Weighted kappa statistics for questionnaire section score tertiles ranged from −0.26 (95% CI −0.52 to 0.01) for section 3 (Food and Drink around the School) to 0.82 (95% CI 0.65 to 0.98) for section 4 (Gardens). Average percent agreement among items in the same section ranged from 60.33% (11.49) for section 8 (Parent and Family Involvement) to 81.16% (14.09) for section 4 (Gardens).

Average percent agreement for reliability was lowest for policy items [66.01% (13.02)] and highest for environment questions [80.71% (14.37)]. Average percent agreement for 5-point scale items was lowest [60.79% (11.69)] and was highest for dichotomous items [80.98% (15.35)], a pattern of results that would be expected based on the probability of agreement for response type.

Based on these findings, the questionnaire was revised by removing items with low validity or reliability and, where appropriate, revising questions and response options based on feedback from respondents or observations the research team made when collecting validity data. Out of 129 items, 60 were kept with no changes, 39 were revised, and 30 were removed from the instrument (Table 2). Of those that were revised, changes were to the stem only for 14 questions, the response options only for 6, and to both for 19 questions. Revisions were generally minor, reducing wordiness or providing clarification such as changing “milk served on campus” to “milk served with meals.” Revisions resulted in improved item-level readability of revised questions as measured by the Flesch/Kincaid Index on the Online-Utility.org Readability Calculator [mean = 0.76 (0.37) grade-level reduction].⁴¹

Discussion

This article reports the reliability and convergent validity of a new instrument, the Elementary School SLAQ, developed for elementary schools to self-assess healthy eating and physical activity practices, for use in program planning and evaluation. The majority of items were found to have moderate or better reliability (86.8%) and validity (55.0%). These findings are comparable with, and at times better than, those from other measures of school and child care environments. For example, a similar proportion of items on the Nutrition and Physical Activity Self-Assessment for Child Care (NAP SACC) were reported to have moderate or better test/retest reliability (89%) and validity (52%).²⁷ On the School Health Policies and Practices Study (SHPPS) questionnaires, reliability of categorical and ordinal questions was similar to that found for all items in our study (88% moderate or better), while a lower proportion of continuous and ordinal items from SHPPS (56%) were found to have acceptable reliability.²⁹

A strength of the Elementary School SLAQ, when compared with many currently existing assessments of institutional nutrition and physical activity practices, is that it was designed to have the sensitivity to detect change between sites and across time points. Using aggregated scores collected by multiple sites, organizations such as public health agencies and school districts can associate the strength of measured institutional practices with available nutrition and physical activity outcome data. When used to assess the same sites over time, changes in the practice scores measured on the Elementary School SLAQ can likely be correlated with changes in measured outcomes. However, because this initial study represents a single year of data collection, it is currently unknown how sensitive to change items are. Future research is planned to measure sensitivity of questionnaire items for detecting change following interventions, as well as differences between sites where interventions differ.¹⁸

Importantly, schools that self-assess using the Elementary School SLAQ benefit from collecting data closely aligned with the Final Rule of the Healthy, Hunger-Free Kids Act of 2010, which outlines school wellness requirements for all local educational agencies and schools participating in national school meal programs.¹⁹ This alignment means that schools can measure and report the extent to which they meet each school wellness requirement of the Final Rule, and to identify areas of Final Rule implementation that may be lacking. While school-level monitoring and reporting of local wellness policy implementation are required by the Final Rule, only a third of local education agencies have conducted monitoring or reporting activities, and administrators report that a lack of vetted tools that are usable by school/district personnel is a significant challenge to accomplishing this.²²

The Elementary School SLAQ is well suited to fill this need, having been designed for use by school personnel and successfully feasibility tested with this audience.²³ School and district personnel may also appreciate the usefulness of an action planning tool that is available to accompany the Elementary School SLAQ to guide them in turning SLAQ results into actionable steps to improve the school health environment (Supplementary Files 1 and 2).

Although the sample size was sufficient to reveal variability in item reliability and validity, a larger sample could offer stronger generalizability. In addition, while the sample included high-priority schools serving the low-income students most often in need of health promotion efforts, findings obtained from this sample may not be representative of all schools in the United States or beyond, and future testing of the instrument in a sample with greater diversity in income and ethnicity would increase generalizability. However, because this sample represents students of color and low-income populations in greatest need of high-quality obesity prevention programs, findings from this study can confidently be applied to planning and evaluating interventions for these priority populations.

The primary statistic used to assess both validity and reliability was weighted kappa, which adjusts for agreement due to chance.³⁵ However, the kappa statistic is affected by the prevalence of the outcome. In this study, there was very little variation in responses to specific items, reflecting the widespread implementation of some practices, especially those enacted by state or federal law. These include items, such as protecting identities of students receiving free or reduced price meals and not selling sugary drinks, where over 90% of respondents selected a single response category. In cases such as these, an alternate statistic is more appropriate. For this reason, this article presents percent agreement in addition to weighted kappa.

The findings presented here can be used to selectively remove or modify items with lower levels of reliability and/or validity in accordance with the user's needs and objectives. For application by local health departments implementing SNAP-Ed in California, the authors removed or revised items with low validity or reliability. This likely increased the validity and reliability of the questionnaire; ideally retesting should be conducted. While a follow-up study to test the psychometric properties of revised items is not currently feasible, improved readability of revised items suggests that validity and reliability are unlikely to have declined, and key informant interviews are being conducted with school personnel using the revised instrument.

As seen in Table 5, a small number of items with fair or poor reliability or validity kappa strength ratings were retained. These items had good percent agreement and likely received low strength ratings as a result of the limitations of the kappa statistic outlined above. In addition to item-specific changes, the questionnaire has been moved from a fillable Adobe PDF form to an online survey platform as a result of feedback from schools and technical issues. We now recommend use of the revised questionnaire, available from the authors, for assessment in elementary schools.

Table 5.

Validity and Reliability Measures of Items by Decision to Maintain with No Change, Revise, or Remove

	Items maintained with no change (N = 60)	Items revised (N = 39)	Items removed (N = 30)	All items (N = 129)
Validity, mean (SD)
Percent agreement	75.48% (18.45)	53.11% (22.46)	48.57% (19.19)	62.46% (23.27)
Kappa	0.33 (0.27)	0.21 (0.24)	0.10 (0.24)	0.24 (0.27)
Reliability, mean (SD)
Percent agreement	80.36% (13.12)	63.88% (16.21)	63.19% (13.06)	71.38% (13.20)
Kappa	0.51 (0.27)	0.53 (0.27)	0.46 (0.26)	0.50 (0.27)
Validity strength, n (%)
Almost perfect	3 (5.00)	0 (0.00)	1 (3.33)	4 (3.10)
Substantial	17 (28.33)	7 (17.95)	2 (6.67)	26 (20.16)
Moderate	17 (10.00)	13 (33.33)	4 (13.33)	34 (26.36)
Fair	11 (18.33)	11 (28.21)	16 (53.33)	38 (29.46)
Poor	6 (10.00)	7 (17.95)	7 (23.33)	20 (15.50)
NA	6 (10.00)	1 (2.56)	0 (0.00)	7 (5.43)
Reliability strength, n (%)
Almost perfect	10 (16.67)	8 (26.67)	4 (13.33)	22 (17.05)
Substantial	26 (43.33)	15 (38.46)	14 (46.67)	55 (42.64)
Moderate	10 (16.67)	11 (28.21)	8 (26.67)	29 (22.48)
Fair	7 (11.67)	5 (12.82)	2 (6.67)	14 (10.85)
Poor	1 (1.67)	0 (0.00)	2 (6.67)	3 (2.33)
NA	6 (10.00)	0 (0.00)	0 (0.00)	6 (4.65)

N, number; SD, standard deviation.

The questionnaire presented here relies on self-report from school personnel, which has both advantages and disadvantages. Because self-report is subject to social desirability bias, it can be less accurate than other data collection methods such as direct observation or review of school menus. However, overall questionnaire scores ranged from 15% to 85% (mean: 58.6% [standard deviation: 13.4]) of points possible, demonstrating that responses were not uniformly high and respondents were willing to report their practices as less than optimal. The advantages of this self-assessment approach are compelling.

In the context of needs assessment and action planning, the process of self-assessment that actively involves school personnel, particularly school administrators, is critical for ensuring that assessment results are used to inform sustainable change.⁴² In addition, while the information collected by direct observation methods is necessarily limited to what can be observed during brief observation periods, a self-assessment questionnaire is able to capture a broader range of information that more comprehensively captures the nutrition and physical activity practices at a school.

Conclusions

This article describes the psychometric testing of a new SLAQ for elementary schools that can support schools to assess needs, create action plans, and evaluate progress, while also offering valid and reliable data for larger scale analysis. Moreover, this new questionnaire fills an identified gap of providing a usable tool for school-level monitoring of local wellness policy implementation in elementary schools. Measured reliability and validity were reasonably good. Future research to measure the instrument's item sensitivity will further inform its usefulness as an evaluation tool.

Footnotes

Authors' Contributions

C.D.R.: conceptualization, methodology, investigation, writing—original draft, and project administration; J.K.: conceptualization, methodology, validation, investigation, writing—review and editing, and project administration; S.C.H.: formal analysis, data curation, and writing—review and editing; C.M.B.: conceptualization, methodology, software, investigation, data curation, and writing—review and editing; A.L.: conceptualization, methodology, and writing—review and editing; G.W.L.: conceptualization, writing—review and editing, supervision, and funding acquisition.

Acknowledgments

This study would not have been possible without the participation of personnel at participating schools, as well as the public health departments and cooperative extension agencies who partnered with us to recruit them. We also thank Evan Talmage for technical assistance with questionnaire deployment, Kaela Plank for assistance with study development, and Azraa Ayesha and Sana Khader for data collection. The California Department of Public Health Nutrition Education and Obesity Prevention Branch supported the effort. Our institution is an equal opportunity employer.

Funding Information

The project was funded by USDA's Supplemental Nutrition Assistance Program Education.

Author Disclosure Statement

No competing financial interests exist for any of the authors.

Supplementary Material

References

Centers for Disease Control and Prevention. Recommended community strategies and measurements to prevent obesity in the United States: Implementation and measurement guide. Morb Mortal Wkly Rep, 2009; 58:1–30.

Institute of Medicine. Accelerating Progress in Obesity Prevention: Solving the Weight of the Nation. Washington, DC: The National Academies Press, 2012.

Story

, Nanney

, Schwartz

. Schools and obesity prevention: Creating school environments and policies to promote healthy eating and physical activity. Milbank Q, 2009; 87:71–100.

Fagen

, Asada

, Welch

, et al. Policy, systems, and environmentally oriented school-based obesity prevention: Opportunities and challenges. J Prev Interv Community, 2014; 42:95–111.

Heelan

, Bartee

, Nihiser

, Sherry

. Healthier school environment leads to decreases in childhood obesity: The Kearney Nebraska Story. Child Obes, 2015; 11:600–607.

Ickovics

, Duffany

, Shebl

, et al. Implementing school-based policies to prevent obesity: Cluster randomized trial. Am J Prev Med, 2019; 56:e1–e11.

Mansfield

, Savaiano

. Effect of school wellness policies and the Healthy, Hunger-Free Kids Act on food-consumption behaviors of students, 2006–2016: A systematic review. Nutr Rev, 2017; 75:533–552.

Cheung

, Franks

, Kramer

, et al. Elementary school physical activity opportunities and physical fitness of students: A statewide cross-sectional study of schools. PLoS One, 2019; 14:e0210444.

Thompson

, Hewawitharana

, Kao

, et al. SNAP-Ed physical activity interventions in low-income schools are associated with greater cardiovascular fitness among 5th and 7th grade students in California. Prev Med Rep, 2020; 20:101222.

10.

Centers for Disease Control and Prevention. School health guidelines to promote healthy eating and physical activity. MMWR Recomm Rep, 2011; 60:1–76.

11.

Naja-Riese

, Keller

KJM

, Bruno

, et al. The SNAP-Ed Evaluation Framework: Demonstrating the impact of a national framework for obesity prevention in low-income populations. Transl Behav Med, 2019; 9:970–979.

12.

Centers for Disease Control and Prevention. School Health Index: A Self-Assessment and Planning Guide. Elementary School Version: Atlanta, GA, 2017.

13.

Schwartz

, Lund

, Grow

, et al. A comprehensive coding system to measure the quality of school wellness policies. J Am Dietet Assoc, 2009; 109:1256–1262.

14.

Cornell University Institute for Behavioral Science and Consumer Choice. Smarter Lunchrooms Scorecard. Available at https://www.smarterlunchrooms.org/scorecard Last accessed August 2017 .

15.

Rollings

, Wells

. Cafeteria assessment for elementary schools (CAFES): Development, reliability testing, and predictive validity analysis. BMC Public Health, 2018; 18:1154.

16.

Stoepker

, Dauenhauer

, Carson

, Moore

. Comprehensive School Physical Activity Program Policies and Practices Questionnaire (CSPAP-Q). Res Q Exerc Sport, 2021; 92:100–110.

17.

Cates

, Blitstein

, Hersey

, et al. Addressing the Challenges of Conducting Effective Supplemental Nutrition Assistance Program Education (SNAP-Ed) evaluations: A step-by-step guide. Prepared by Altarum Institute and RT, I International for the U.S. Department of Agriculture, Food and Nutrition Service, March 2014. Available at https://snaped.fns.usda.gov/library/materials/addressing-challenges-conducting-effective-supplemental-nutrition-assistance Last accessed April 7, 2021.

18.

Vermeersch

, Lambert

, Burlingame

. Outcome Questionnaire: Item sensitivity to change. J Pers Assess, 2000; 74:242–261.

19.

Food and Nutrition Service, United States Department of Agriculture. Local School Wellness Policy Implementation Under the Healthy, Hunger-Free Kids Act of 2010. Federal Register: Washington, DC, 2016. Available at https://www.fns.usda.gov/school-meals/fr-072916c Last accessed April 7, 2021.

20.

Healthy, Hunger-Free Kids Act of 2010. 42 U.S.C. 1779; 2010. Available at https://www.gpo.gov/fdsys/pkg/PLAW-111publ296/pdf/PLAW-111publ296.pdf Last accessed April l7, 2021.

21.

United States Department of

Agriculture

, Food and Nutrition

Service

. Local School Wellness Policy Implementation under the Healthy, Hunger-Free Kids Act of 2010. Final rule. Fed Regist, 2016; 81:50151–50170.

22.

Turner

, Asada

, Leider

, et al. Can monitoring make it happen? An assessment of how reporting, monitoring, and evaluation can support local wellness policy implementation in US Schools. Nutrients, 2021; 13:193.

23.

Rider

, Linares

, Kao

, et al. Assessing healthful eating and physical activity practices in places children learn. J Extens, 2020; 58:t55.

24.

U.S. Department of Agriculture, Food and Nutrition Service. The Supplemental Nutrition Assistance Program Education (SNAP-Ed) Evaluation Framework: Nutrition, Physical Activity, and Obesity Prevention Indicators: Interpretive Guide to the SNAP-Ed Evaluation Framework. USDA: Arlington, VA, 2016.

25.

Measures Registry. National Collaborative on Childhood Obesity Research. Available at https://www.nccor.org/nccor-tools/measures/ Last accessed April l7, 2021.

26.

McKinnon

, Reedy

, Berrigan

, Krebs-Smith SM

, COR Catalogue and Registry Working

Groups

. The National Collaborative on Childhood Obesity Research catalogue of surveillance systems and measures registry: New tools to spur innovation and increase productivity in childhood obesity research. Am J Prev Med, 2012; 42:433–435.

27.

Rudd Center for Food Policy and Obesity. WellSAT-i: Wellness School Assessment Tool for Implementation. Available at www.wellsat.org/ Last accessed June 2018 .

28.

Benjamin

, Neelon

, Ball

, et al. Reliability and validity of a nutrition and physical activity environmental self-assessment for child care. Int J Behav Nutr Phys Act, 2007; 4:29.

29.

Ward

, Morris

, McWilliams

, et al. Go NAPSACC: Nutrition and Physical Activity Self-Assessment for Child Care, 2nd Edition. Center for Health Promotion and Disease Prevention and Department of Nutrition, University of North Carolina at Chapel Hill, 2014.

30.

Brener

, Kann

, Smith

. Reliability and validity of the School Health Policies and Programs Study 2000 questionnaires. J Sch Health, 2003; 73:29–37.

31.

Kyle

, Brener

, Kann

, et al. Methods: School Health Policies and Programs Study 2006. J Sch Health, 2007; 77:398–407.

32.

Wolfe

, Crosiar

. Cornell Self-Assessment Tool for After-School Food and Physical Activity Environments. Division of Nutritional Sciences, Cornell University, 2010.

33.

Rocky Mountain Prevention Research Center, University of Colorado. Denver Public Schools School Wellness Policy Assessment. Available at www.ucdenver.edu/academics/colleges/PublicHealth/research/centers/RMPRC/resources/Documents/Elementary%20DPS%20Assessment%20Tool%20101310%20ar%20esb.pdf Last accessed August 2017 .

34.

State of Alaska Obesity Prevention and Control

Program

, Alaska Division of Public

Health

. Alaska School Wellness Policy Annual Report Checklist. Available at http://dhss.alaska.gov/dph/Chronic/Pages/SchoolHealth/aswpi.aspx Last accessed August 2017 .

35.

Cohen

Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psycholog Bull, 1968; 70:213–220.

36.

Muñoz

, Bangdiwala

. Interpretation of Kappa and B statistics measures of agreement. J Appl Stat, 1997; 24:105–112.

37.

California Department of Education. Unduplicated Student Poverty FRPM Data 2019–2020. Available at https://www.cde.ca.gov/ds/sd/sd/documents/frpm1920.xlsx Last accessed September 20, 2020.

38.

National Center for Education Statistics

U.S

. Department of

Education

. Institute of Education

Sciences

, National Center for Education

Statistics

. Education Demographic and Geographic

Estimates

. 2019. Available at https://nces.ed.gov/programs/edge/Geographic/LocaleBoundaries Last accessed February 23, 2021.

39.

California Department of Education. Enrollment by School. Available at http://dq.cde.ca.gov/dataquest/dlfile/dlfile.aspx?cLevel=School&cYear=2019-20&cCat=Enrollment&cPage=filesenr.asp Last accessed September 20, 2020.

40.

California Department of Education. English Learners by Grade and Language. Available at https://dq.cde.ca.gov/dataquest/dlfile/dlfile.aspx?cLevel=School&cYear=2019-20&cCat=EL&cPage=fileselsch Last accessed February 23, 2021.

41.

Flesch

A new readability yardstick. J Appl Psychol, 1948; 32:221–233.

42.

Pearlman

, Dowling

, Bayuk

, et al. From concept to practice: Using the School Health Index to create healthy school environments in Rhode Island elementary schools. Prev Chronic Dis, 2005; 2:A09.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.47 MB

1.26 MB