June 27, 2008

print icon

Biased Questions? Faulty Methodology?

Filed under: Education by Leo Casey @ 4:39 pm

This morning, the New York newspapers reported on the publication of the UFT’s survey on the leadership of Chancellor Joel Klein and the Department of Education. Read the New York Times, the Daily News and the Post. While the survey was offered in a constructive spirit of a full 360 degree system of accountability, the DoE spokespersons responded by calling the survey political and questioning its methodology.

But the overwhelming preponderance of questions on the UFT survey were taken verbatim from the DoE’s own Learning Environment Survey. The few exceptions were those questions which addressed subjects not covered on the DoE survey, such as the education of the whole child and the educational efficacy of the DoE’s regimen of testing. To show just how faithful this process was, consider the following side-by-side comparison of the wording from the DoE’s survey and the wording from the UFT’s survey. [For each statement, both the DoE survey and the UFT survey asked respondents to choose among the following answers: strongly agree, agree, disagree and strongly disagree.]

DoE Learning Environment Survey:
School leaders communicate a clear vision for this school.
UFT Survey:
Chancellor Klein communicates a clear vision for the NYC school system.

DoE Learning Environment Survey:
School leaders encourage open and honest communication on important school issues.
UFT Survey:
Chancellor Klein encourages open communication on important school decisions.

DoE Learning Environment Survey:
The principal places the learning needs of children ahead of other interests.
UFT Survey:
Chancellor Klein places the learning needs of children ahead of other interests.

DoE Learning Environment Survey:
The principal is an effective manager who makes the school run smoothly.
UFT Survey:
Chancellor Klein is an effective manager who makes the school system run smoothly.

DoE Learning Environment Survey:
The principal has confidence in the expertise of the teachers.
UFT Survey:
Chancellor Klein has confidence in the expertise of his educators.

DoE Learning Environment Survey:
The principal invites teachers to play a meaningful role in setting goals and making important decisions for this school.
UFT Survey:
Chancellor Klein invites educators to play a meaningful role in setting goals and making important decisions in their schools.
and
Chancellor Klein invites parents and community members to play a meaningful role in setting goals and making important decisions about the education of their children.

DoE Learning Environment Survey:
School leaders encourage collaboration among teachers.
UFT Survey:
Chancellor Klein promotes collaboration between principals and teachers and among all educators.

DoE Learning Environment Survey:
To what extent do you feel supported by your principal?
UFT Survey:
Chancellor Klein works to provide the support I need as an educator.

DoE Learning Environment Survey:
Order and discipline are maintained at my school.
UFT Survey:
Chancellor Klein works to promote order and discipline at my school.

4 Comments »

Comments are open for registered users and do not reflect the views of the UFT. Please read our general rules for commenters.

  • I strongly agreed Chancellor Klein communicated a clear vision for the NYC school system. He envisions no tenure, and no seniority. He’s already realized visions ot “accountability” for unionized employees only, the largest class sizes in the state, kids learning in filthy trailers and unventilated closets, outrageous overcrowding with no end in sight, kids freezing on the streets on the coldest days of the year, chief accountability officers who literally run from involved parents…

    How could anyone who follows the news suggest Chancellor Klein has no vision? In fact, he’s now managed to extend his vision beyond his tenure with a five-year taxpayer-funded contract for the Leadership Academy. Personally, I wouldn’t be surprised if he found other ways to extend his vision.

    I’m afraid I had to disagree with the other statements, though.

    Comment by NYC Educator — June 28, 2008 @ 9:57 am

  • Leo, this is an interesting exchange between DoE and UFT. I like the topics you discuss on your blog.

    Based on my experience, for better or worse, methodology is the first thing education reporters will evaluate when considering surveys. But once you get past the “that’s reasonable” threshold, I think reporters give pretty fair treatment of findings.

    Do you think the specific mention of “Chancellor Klein” affects the questions? That immediately struck me as I read through the survey questions the first time through.

    Usually proper nouns will increase the likelihood of injecting bias in survey questions… not a sure thing depending on your sample population, but increases the possibility.

    If a respondent has a particular feeling with respect to Klein (good or bad), this could affect a response if the intensity of that feeling is greater than what is mentioned in the predicate parts of the questions.

    So if someone is peeved at Klein for something regarding school closings or firings, but nothing to do with the predicate/object in a given question, it is more likely that that feeling will negatively affect the response.

    Polling and surveys are of great interest to me, so I just thought I would throw that out there to see what you and others think.

    Best,
    Paul

    Comment by pdiperna — June 28, 2008 @ 12:15 pm

  • Paul:

    I understand your concern, but if the point of the survey was to evaluate the Chancellor, just as principals and teachers are evaluated, how could one eliminate unrelated feelings of antagonism toward him? Certainly, this concern would be no less valid for the principals evaluated by the survey.

    Leo

    Comment by Leo Casey — June 30, 2008 @ 1:36 am

  • [...] the dueling survey issue in New York, it seems the UFT could have just saved themselves a lot of money on their survey and asked just one question about approval/disapproval of Joel Klein. That’s all these [...]

    Pingback by Department of Efficiency at More About Education — July 1, 2008 @ 12:01 pm

Leave a comment

You must be logged in to post a comment.