Friday, October 17, 2014

Edwin Workshops in November!

The ESE is hosting a one-day workshop to showcase EDWIN (Analytics & Teaching and Learning).  This is not the same workshop that was offered last fall.  There are six available dates in November, and sessions run from 9:00 AM - 2:45 PM.  Two of the workshops are taking place at the Taunton Holiday Inn, on Monday, November 3, 2014 and Tuesday, November 18, 2014.

Participants will leave with a solid understanding of the entire Edwin system, a view of enhancements to the system, and tips on next steps for how to implement Edwin in your district.

Sessions will include:

  • Edwin 101 - An introductory look at the Edwin system (both Edwin Analytics & Teaching and Learning).
  • A District Perspective - A look at how Edwin Analytics and Edwin Teaching and Learning are being used in the field.
  • EWIS - Early Warning Indicator System implementation guidance.
  • Data Quality - An overview of the Department's Data Quality Curriculum.
  • Edwin Teaching and Learning for curriculum and instruction as well as for assessment and reporting.
  • Edwin Analytics Scavenger Hunt (*requires a wi-fi enabled mobile device).
  • Hands on Q&A Session for Edwin Analytics and Edwin Teaching and Learning.

See the online registration page for more agenda details: https://www.surveymonkey.com/s/3MKQFCD 
Teams including a district leader, school leader, data specialist, and teacher are encouraged to register.

Thursday, October 16, 2014

Technology Grant Opportunity-Webinars 10/16 and 10/17!

The ESE's Digital Connections Partnership Schools Grant is a new opportunity for local school districts to upgrade their technology infrastructure through a matching grant program administered by ESE and the Executive Office of Education (EOE) in consultation with MassIT.

The reason I have posted this information is because if you have any interest in learning more, there are two technical assistance webinars that are happening right away.  The first is today, October 16 from 3-4:30 PM, and the second option is October 17 from 9:30-11 AM.  Click here to register:  http://www.doe.mass.edu/conference/?ConferenceID=5968

Five million dollars is available in state bond funds: $2 million for suburban communities, $1 million for rural communities, and $2 million for urban communities.  Cities and towns will be required to contribute between 30 and 70 percent on a sliding scale based on their local contributions to the Chapter 70 formula.  Click here for the link to the grant information:  http://www.doe.mass.edu/grants/2015/DCPSG/

State bonds may be used to fund capital projects that provide high-speed broadband and/or wireless Internet connectivity for schools.  Up to 100 % of the local contribution may fund devices for students and teachers.  Applications must be received by ESE by 5:00 PM on November 17, 2014.


Thursday, September 18, 2014

District-Determined Measures (DDM) Resources


MA ESE Resources that provide targeted guidance for schools and districts at each stage of DDM development, were created with CVTE educators and WestEd last year.  While these resources describe work underway in MA CVTE schools and districts, the recommendations, ideas, and processes will benefit ALL districts as they determine where they are in the process of implementation, and where they want to go.



Some highlights:

In Section II:  Links to informative and practical resource guides for each of the four stages of Implementation.  For those currently in "Stage 3: Evaluating", you will find great information in Guises F and G to support collaborative DDM review and improvement discussions this year
.
Appendix D-1: 1-page Summary of the Stages and Steps in DDM Development and Implementation 

Appendix A:  Includes samples of currently used CVTE Assessments which could be used as DDM's.

Appendix D-2:  District-Determined Measures Review and Discussion Guide

Tuesday, August 12, 2014

2014 ESE Accountability Changes  and Webinar


Click on 2014 Preliminary Accountability Review Webinars to sign up for an upcoming ESE webinar on interpreting the preliminary 2014 district and school accountability data (reports now available in the Accountability 2014 Data application in the Security Portal).  

Webinars will be held on the following dates and times: Wednesday, August 27 at 3:00 PM; Wednesday, September 3 at 12:00 PM; and Wednesday, September 10 at 10:00 AM.

On last week's conference calls, some of this information was discussed.  An example of two points of information from last week:  for 2014, the state median for SGP is 50 on all assessments; and high schools that reengaged 2 or more dropouts in the 2012-2013 school year will earn 25 extra credit PPI points (all students & high needs groups only).  The powerpoint for last week's calls, including information on reporting discrepancies (All Discrepancies to be reported by THURSDAY, AUGUST 14, 2014) was posted on the MCAS Service Center.  

More information regarding 2014 accountability determinations, including the 2014 Accountability Data Overview and the School Leader's Guide to the 2014 Accountability Determinations can be found at http://www.doe.mass.edu/apa/accountability.



Monday, August 11, 2014

EWIS Use Workshop Sept. 23, 2014

Early Warning Indicator System (EWIS) Data Update:

The MA Department of Elementary and Secondary Education anticipates releasing the 2014-15 school year Early Warning Indicator System (EWIS) data via Edwin Analytics by early September. The EWIS identifies students in grades 1-12 as being at low, moderate, or high risk of not meeting an upcoming education milestone. The data may be used to help to systematically identify students who may need additional supports to meet key academic goals. 

The  Department is offering a daylong workshop on how to use EWIS data thoughtfully and cohesively in conjunction with other initiatives and data sources throughout the school year. All participants will receive a copy of a new EWIS guide at the workshop. More information (below) and online registration is available at https://www.surveymonkey.com/s/EWIS9-23-14.  The workshop is scheduled to take place at the DoubleTree in Leominster, MA on Tuesday, September 23, 2014 from 8:45 AM - 2:30 PM.  Registration closes on Friday, September 12, 2014.

PARTICIPATION

This workshop is intended for participation by school-based teams of 3-6 staff (including but not limited to school leadership, student support, teachers). District-level staff are also encouraged to be included as members on school-based teams. There is no limit to the number of school-based teams participating from a district. For example, a district may send 2 elementary school teams, 1 middle school team, and 1 high school team. 
No prior EWIS use experience is necessary; the workshop is designed for users of various experience levels. 


TENTATIVE AGENDA
8:14-8:45 -- Registration
8:45-9:15 -- Welcome and overview of the day
9:15-10:00 -- What is EWIS? An Overview to ground everyone in the basics
10:00-11:30 -- Digging in on your data - a guided team exercise
11:30-12:15 -- Introduction of the EWIS use guide and the 6-step cycle
12:15-1:00 -- Lunch (on your own, available for purchase)
1:00-2:00 -- Applying the 6-step EWIS implementation cycle to your school context
2:00-2:30 -- Next steps, final Q&A, and closing

To ensure that your district receives its 2014-15 EWIS data on time, please: 1) certify all student data collections (i.e., SIMS, SCS, and SSDR) according to the schedule posted at http://www.doe.mass.edu/infoservices/data/schedule.html, and 2) complete the student claiming process by Aug. 22 for all students who are new to your district or who are in a new school within your district. If you have any questions, please contact the Edwin Analytics team at EdwinAnalytics@doe.mass.edu.

Tuesday, July 15, 2014

Passing the Torch

Please welcome Jane Daly as the Regional Data Specialist for the Southeast DSAC. Jane came on board this spring and will now be taking over as the official keeper of the SEED blog (as Laura Tilton is now the District Data Specialist in Marshfield). While the DSAC Data Specialist role is focused primarily on working with Level 3 districts in the region, Jane will continue to use this blog to share data-related information to districts in Southeastern Massachusetts (and beyond!) interested in staying informed.

Jane Daly brings PK-12 educational experience from both urban and suburban schools and districts. She has served in district leadership positions as the Assistant Superintendent for Accountability and School Improvement in New Bedford and Director of Curriculum and Instruction for Seekonk Public Schools. For over ten years, she held school leadership positions in middle schools in Seekonk, Dartmouth, and Bridgewater. Prior to that, she was a Math teacher at New Bedford High School for over ten years. She is experienced in leading, implementing, and monitoring the success of improvement efforts and change in small and large schools and districts. She has served on various MA DESE school panel reviews and committees. Jane holds a B.S. in Math from the University of Massachusetts, Dartmouth, a Master’s Degree in Education Administration from Bridgewater State University, and a C.A.G.S. in Education Leadership Management from Northeastern University.

Welcome, Jane!

Thursday, July 10, 2014

ESE Model Student and Staff Feedback Surveys Posted:


The Department of Elementary & Secondary Education has recently posted student and staff 
model feedback surveys at http://www.doe.mass.edu/edeval/feedback/

Districts may adopt or adapt these surveys, and/or choose to use other feedback instruments.

ESE has also published guidance which includes: information about how to identify appropriate feedback instruments, recommendations on the use of student and staff feedback in the five-step evaluation cycle, and a review of alternative methods for collecting feedback from students and educators.

Posted Model Surveys and Administration Protocols include:
  • Student Surveys - Gr. 3-5
  • Student Surveys - Gr. 6-12
  • Staff Surveys
  • Discussion Prompts for Students in Gr. K-2

Tuesday, June 17, 2014

2014 MCAS and Accountability Release Schedule

The 2014 MCAS and Accountability Release Schedule was recently posted to the DESE Web site. Here is a summary of some key dates:
  • June 23rd - partial* preliminary MCAS Student Rosters posted electronically at DropBox Central in the Security Portal
  • June 26th - partial* preliminary reports available in Edwin Analytics 
  • August 7th - full preliminary MCAS results available in both DropBox Central and Edwin Analytics
  • August 12th - Preliminary Accountability data provided through the Security Portal
  • Aug 7-14th - MCAS discrepancy reporting window
  • Mid-September - Official data released publicly (note, all data are embargoed prior to this date!)
*Partial results include: 
  • ELA, full results 
  • Mathematics, multiple-choice results only 
  • STE, grades 5 and 8, multiple-choice results only 
  • MCAS-Alt, full results in all subjects

Wednesday, March 19, 2014

Using Student Growth Percentiles in Educator Evaluation

The folks at ESE recently released several Implementation Briefs focused on timely questions regarding District Determined Measures and Student Impact Ratings. One of these briefs focuses on the question, "How do districts use median Student Growth Percentile in the evaluation of teachers and administrators?" Many of the questions we have been asking about this use of SGP are addressed in this brief, including clarification about determining which teachers (and administrators) should use SGP as part of their Student Impact Rating determination.

While the median growth score for a group of students is the measure used, it is also very valuable to for an educator to examine the distribution of their students' MCAS growth and performance scores. In Edwin Analytics, we can look at a scatter plot of growth and performance by grade level, but no such report exists at this time for a classroom set of students. Creating this type of plot by hand can be very valuable in helping an educator explore classroom-level growth data in a reflective way.

Here's a link back to the activity (originally posted in August 2012, so the references are to EDW rather than Edwin Analytics) which gives some suggestions for facilitation along with a blank scatter plot and other resources: http://southeastdata.blogspot.com/2012/08/exploring-classroom-growth-data.html. I believe this is a valuable exercise as we talk about SGP in the context of educator evaluation - grounding the overall median score in reflection and explanation!


Monday, February 24, 2014

School and District Profiles Survey

The Department of Elementary and Secondary Education is seeking feedback about the School and District Profiles website to inform its examination of current functionality and future potential use. The Department provides this site as the central online repository for sharing public education data. Please share your feedback through the anonymous survey posted at http://surveys.pcgus.com/s3/Profiles before February 28.

Friday, February 7, 2014

Anticipated MCAS Reporting Schedule 2014

Here are the dates that were shared in last week's MCAS test administration training (FYI, the conference call version of the training is being offered again next Wednesday 2/12). Of course, these dates may be subject to change but are still very useful for planning!
 

Anticipated MCAS Reporting Schedule 2014
March 21
Access February Biology data files and rosters at DropBox Central
April 16
Access March Retest data files and rosters at DropBox Central
May 1
Districts receive February Biology and March Retest Parent/Guardian Reports
Mid-May
Schools and districts receive printed ACCESS for ELLs reports and data posted at DropBox Central
Mid-June
Districts access prelim MCAS and MCAS-Alt student results:
 
ELA: preliminary results, including performance levels, scaled scores, and SGPs
 
Math: multiple choice only
 
STE (5&8): multiple choice only
 
MCAS-Alt
 
MCAS ELA long comps available at MCASservicecenter.com
July 1-July 31
MCAS discrepancy reporting by phone
August 6
Preliminary Math and STE MCAS results (combined with June ELA)
August 6-12
MCAS discrepancy reporting online
mid-September
Official results in security portal for districts
(week of Sept 19)
One day later: Public release of results; Parent/Guardian Reports shipments
Note: data will be in Edwin Analytics within 3 days of DropBox release

Thursday, February 6, 2014

Sharing Some News....and Looking for Your Help!

As some of you may know, I started a new position as the Data Management and Accountability Specialist for the Marshfield Public Schools last week. I'm very excited about this new challenge but will miss working with the dedicated educators in the DSAC districts...and those on the DSAC team as well!

I'm looking for YOUR help in finding a fabulous new Data Specialist for the Southeast DSAC. The position is posted on SchoolSpring at: http://www.schoolspring.com/job.cfm?jid=599682. Please take a look and share with any colleagues you know who may be interested in this unique opportunity! Thank you!

P.S. Yes, I plan to keep up this blog in my new role, although I may tweak it a bit moving forward...stay tuned!

Wednesday, January 8, 2014

MCAS Long Compositions Downloads

Recently, I had inquries from several principals asking how to access their school's long compositions. I don't have access to these documents, but wanted to share the information that was given to me by the Measured Progress and the student assessment office at ESE.

The 2013 student compositions were made available for download at the MCAS Service Center web site last August (and then updated with the official results in September). Student compositions are accessible using the usernames and passwords that are mailed to school and district administrators at the start of each calendar year (schools should have received their letters about a week ago.) The username stays the same from year to year (an 8 digit number which represents a combination of the district and school codes) but the password will have changed since 2013.

To access the student compositions:
  • Visit www.MCASservicecenter.com and choose MCAS and then MCAS Reporting
  • Enter your 8-digit username and 2014 password.
  • Click on Reporting. The compositions are located on a tab at the top of the page (MCAS Student Compositions). 
  • Complete the Basic Filtering options and the student compositions should appear. They can be viewed, printed, or downloaded.

Be sure to keep your up-to-date password handy so that you can access the 2014 composition files in the fall of 2014!

If you have questions, you can contact the MCAS Service Center at 800-737-5103 between the hours of 7 a.m. and 5 p.m., Monday through Friday.

Thursday, January 2, 2014

New Year Updates

I wanted to share some updates that came in at the end of 2013 so that they aren't forgotten as we start a new year.

The 12/20 Commissioner's Weekly Update included a variety of timely items, including an opportunity for non-RTTT districts to access Edwin Teaching and Learning at no cost during the 2014-2015 school year. Interested districts should contact EdwinT&L@doe.mass.edu prior to January 10, 2014.


The materials from the 2014 Curriculum Summit are now posted on the ESE Web site, including information about the Model Curriculum Units (MCUs) and how the Curriculum Embedded Performance Assessments (CEPAs) can be used in District-Determined Measures (DDMs), as part of the Educator Evaluation system.

Finally, ESE has posted a survey about the ESEA waiver, seeking feedback from stakeholders about what changes might be made to further improve the state's accountability system. Parents, teachers, district and school administrators, school committee members, students, and other community members are encouraged to respond.

Happy New Year to all!

Thursday, December 19, 2013

Massachusetts Data Quality Program

The Massachusetts Data Quality Program is designed to help both data collectors and data users understand and address the issues that may affect the production of high-quality data. It provides lesson guides, instructional handouts, and material tailored to the specific needs of the K-12 schools and districts. A video describing the importance of data quality is available online for your reference.

As stated on the ESE Web site, "The Data Quality Program is now required for all districts and the program will be held on a regular basis throughout the state." A new round of dates for this one-day training has been posted online at http://www.doe.mass.edu/conference/?ConferenceID=1749 (as of this posting, the most convenient session for readers in the Southeast region of Massachusetts may be January 15 in Milton.)

Friday, December 6, 2013

Edwin is the Man

"Edwin is the Man." This quote comes from a blog post from The Data Quality Campaign (DQC). DQC is a nonprofit, nonpartisan, national advocacy organization based in Washington, DC, and they sing the praises of Edwin.  In the post, there is also a link to a video piece about EWIS (Massachusetts' Early Warning Indicator System) featuring commentary from several Massachusetts districts. The Data is Power video is definitely worth a look as well!
 
I also learned this week about this jam-packed graphic from the DQC folks about data use, inviting you to “Follow one teacher, Ms. Bullen, and one of her students, Joey, through the school year to see how data help teachers, parents, and others make sure students are meeting education goals.”
 
Time is a precious commodity but if you have a few minutes as the week winds down, definitely check these out!

Thursday, December 5, 2013

Edwin Analytics Update

Here is a summary of yesterday's ESE Edwin Analytics User Group WebEx (thanks to my Data Specialist colleague in Greater Boston, Paula O'Sullivan, for this update as I was unable to attend the WebEx.)

Teacher Access - As presented at the Edwin Workshops in November, ESE is doing a "soft" rollout of teacher access to Edwin Analytics.
  • At this point, access is available only to those participants of the workshops, but districts can email the Edwin Analytics team to request access if they did not go to a workshop. The security role is "Global Access Teacher" and should NOT be assigned in combination with any other Edwin Analytics role.
  • There are three reports available to teachers. These reports are also available to administrators in the Classroom folder.
  • Teachers can view the data for last year's students, and for two years prior. At this point, teachers can not run the reports for their current students, but eventually that will be available for districts that have implemented SIF. Manual student claiming does not include updates to teacher reports for current students.
WIDA/ACCESS reports - You can now run these reports for your "claimed" students, i.e. those currently enrolled, IF your district submits claim files.

Local Assessments Data Uploads - ESE is revisiting the idea of allowing districts to upload local assessment data, particularly data from "off-the-shelf" assessments.

New User Guide - This will be posted any day now, along with a new Security & Access Guide.

Housekeeping - ESE is retiring the remaining references to Education Data Warehouse, EDW and DW. The new email address is edwinanalytics@doe.mass.edu, although the old address still works as well). Security role names have changed (e.g., DW 211 School User is now Edwin Analytics School Administrator).

Cubes - A basic demonstration of the cube functionality was provided.

The Edwin Analytics User Group WebEx is held the first Wednesday of the month at 11a.m. and the third Wednesday of the month at 2p.m. ESE sends email reminders to several lists including the Edwin Analytics Contacts as identified in Directory Administration and to subscribers of the EDWusers email list.

To subscribe to the EDWusers list:
Please send an email to imailsrv@list1.doe.mass.edu
with the following text—and only this text—in the body of the message:
subscribe listname your_full_name (e.g., subscribe edwusers john smith).

Monday, November 25, 2013

District and School Report Cards

The annual District and School Report Cards are now available in the Profiles section of the ESE web site, on the Accountability tab for any given school or district. There is also a Report Card search box that has been added to the main Profiles page. This year there is a Report Card Overview and also a Complete Report Card. The "Overview" page connects a variety of questions to the data, including:
  • How is our school doing overall?
  • How does our school's achievement over time compare to the district and the state?
  • How does our school's growth compare to the district and the state?
  • How does our school's enrollment compare to the district and the state?
  • How do our school's teachers and classrooms compare to the district and the state?
  • How is our school doing on other important measures?
For additional information and guidance around the report cards (distribution requirements, etc), please visit http://www.doe.mass.edu/apa/accountability/reportcard/

Wednesday, November 13, 2013

PARCC Field Tests and the MCAS

As you may know, many questions have surfaced recently regarding the PARCC Field Test and the Spring 2014 MCAS. ESE has heard these questions and has put together resources related to field-test logistics, including a timeline and information related to future accountability determinations.

Additional details are contained within a recent memo to superintendents at http://www.doe.mass.edu/news/news.aspx?id=7849. Please note that the conference call registration and exemption surveys links in the memo are intended for superintendent use only. Other leaders who are not superintendents should coordinate with their superintendent as ESE has requested one phone line per district for the conference calls.

Postsecondary / College and Career Readiness Resources

ESE has released a new DART (District Analysis and Review Tool) that contains a range of high school, postsecondary, and career readiness data. Included in the tool are demographics, high school indicators, high school performance, program of study, postsecondary education outcomes, and career development education over a number of school years. DART helps users to find "comparable" schools and allows them to contrast their school's progress on student outcomes to similar schools and the state. To access DART–Success after High School, go to www.doe.mass.edu/apa/dart.

In addition, there are ongoing resources available from ESE on a variety of topics related to College and Career Readiness, including a February webinar on Graduation and Dropout rate data.
http://www.doe.mass.edu/news/news.aspx?id=7847

Tuesday, October 8, 2013

2012-13 Dropout Data Available for Review

ESE announced last week that the 2012-13 dropout data is available for districts to review. Dropout rates are factored into ESE's accountability determinations for high schools and districts, so now is your chance to ensure the data are accurate! The announcement from ESE is as follows:

"ESE has posted 2012-13 Annual Dropout data to the Security Portal for districts to review and make corrections to individual student data. Aggregate rates have not yet been provided. ESE will not accept corrections through email or over the phone. The only mechanism to make corrections to these data is through the October SIMS submission. ESE will make the final rates publicly available in January 2014 once the October 2013 SIMS data have been certified and the calculations have been completed. 

Data are located under "Dropout Data Summary Report" on the Security Portal at https://gateway.edu.state.ma.us/. The application can be accessed with a username, password, and the SIMS Data User role. Should you or your staff have any questions on the correction process, please contact your District Support Specialist. A list of those individuals is located at http://www.doe.mass.edu/infoservices/data/fts.html. The deadline to correct these dropout data coincides with the October SIMS submission deadline, which is November 7."


As a reminder, dropout rate is one of the core indicators in the Progress and Performance Index (for districts and high schools). All groups with dropout data (districts, schools, and subgroups) are expected to halve the gap between their annual dropout rate, if one exists, and a rate of zero percent by the 2016-17 school year. For accountability determinations in any given year, the annual; dropout rate for the prior year is used. Next year's (2014) PPI will utilize the dropout rate data from 2012-2013 (this allows ESE to use a data set that has been thoroughly reviewed by district and ESE staff).

Friday, September 20, 2013

State Releases 2013 MCAS School and District Results

Today the state released the official 2013 MCAS School and District Results. Information is available in a multitude of places:
  • Today's Press Release
  • The ESE Accountability Web Page now includes an Excel file with commendation school information as well as level/PPI information for all districts and schools by region. This page also contains a variety of support documents around school and district accountability, including some short informative videos about the accountability system, PPI, and percentiles.
  • School and District Profiles have also been updated to include 2013 MCAS information under the "Assessment" and "Accountability" tabs for a school, district, or the state.
  • Edwin Analytics reports in the Security Portal have also been updated with the 2013 official data.

Wednesday, September 11, 2013

Exploring Classroom Growth Data

The "Exploring Classroom Growth Data" activity I posted last fall invites educators to engage with student growth and performance data by creating a scatter plot for a given group of students. I've used this activity extensively with teachers throughout the year and I know others out there have done the same. I wanted to post a link back to the activity and accompanying instructions for those who would like to use it this fall to help educators explore and engage with student growth data at the classroom level.

http://southeastdata.blogspot.com/2012/08/exploring-classroom-growth-data.html 


Thursday, August 22, 2013

Looking at Student Roster Reports

In Edwin Analytics, the "Students, Staff & Classroom" tab contains a number of student-level MCAS reports. Some of these reports are "roster" reports which means that they display data for a given list of students in a tabular format, usually one line of data per student. Student reports that are not in "roster" format generally display a page or two of data for each student.

The students included in a report can be selected in one of two ways - by using filters (such as "all ELL students" or "all students who scored Advanced") OR by selecting students from a list (such as a teacher's class list from last year, from their class coming in this year, etc.) There is a document in the Edwin Analytics handouts folder that provides further instructions.

Some of these roster reports include:
  • IT616 Test Item Analysis Roster - provides test item data as well as achievement level, scaled score, raw score, SGP, % correct, and CD status for each student in a list. One challenge with this report is that the font is VERY small if it is printed on a single sheet of paper. I have found that if you save your report as a PDF file, Adobe Reader has a "poster" option that will allow you to print the file larger across multiple pages. This report is single-subject (ELA, math OR science.)
  • PE617 Student MCAS Roster - provides a list of students and their scaled scores, achivement levels, and SGPs for all tested subjects (ELA, math, AND science) on a single report. This report is very useful if you would like to look at performance on all 3 tests for a given group of students.
  • GR602 Student Growth Roster - provides MCAS scaled score information for 3 years of ELA and math performance, plus the most recent year's SGP for each student. This is very useful for examining individual student performance over time in a compact way.

And here is one more "roster" report that I had not seen until a few days ago when a colleague shared it with me. If it is new to you as well you may want to check it out, particularly if you're interested in a roster report of student performance by question type or standard.
  • PE606 (Student Assessment Detail) report has traditionally been one of those reports that was a page or two per student. However, there is now a "roster" option on the initial prompt page. If you choose this, you'll get a report similar to IT616, except it will give information by question type and standard rather than by item. 
Folks that have access to the Classroom-level reports (available only to those with the DW-Evaluator role) should note that the IT516 and PE506 reports are similar to the 600-level counterparts listed above, but that they can be run for a selected classroom.


  
 

Wednesday, August 21, 2013

Edwin Analytics Workshop Handouts and Tips

As you know, the Education Data Warehouse has now been merged into Edwin Analytics, and by now you may have logged in to run reports with your preliminary 2013 MCAS data. The interface hasn't changed significantly, and in recent workshops the participants have commented that the speed and ease of use in the new system are excellent.

One difference you may have noticed is that the "My Folders" area is not easy to find. "My Folders" is an excellent place to save copies of reports, especially reports that may contain student-level data, as it is extremely secure (unlike a memory stick!) and accessible from any computer.

If you do not see a "My Folders" tab at the very top level of tabs (to the right of "Public Folders"), you need to set the My Folders tab to appear. To do this, click on the "Tab Menu" button to the left of the EOE tab at the very top of the page and select "Add Tabs". Check the box next to "My Folders", then the green arrow in the middle to move the selection over to the right. Click OK. You should now see the My Folders tab at the top, next to Public Folders.

For those that were not able to make the Edwin Analytics workshops this month (and even for those that were!) I'm providing you a link to a Google Drive folder which contains the Edwin Analytics handouts and information sheets used in the workshops. Please let me know what you find useful or if there are questions I can answer for you!

Link to Edwin Analytics Handouts

Monday, August 19, 2013

Educator Evaluation Update, District-Determined Measures

The Commissioner recently released a memo with an update for the timelines and expectations for implementing District-Determined Measures (DDMs) and the Student Impact Rating of the educator evaluation system. The DDM update can be found here http://www.doe.mass.edu/news/news.aspx?id=7640

Tuesday, August 13, 2013

Where can I find....?

I've heard from a number of principals and other school/district leaders over the last week who aren't quite sure where to go to get different pieces of information related to the preliminary MCAS data, so I thought I'd post a quick clarification.

There are actually 3 different applications in the Security Portal where preliminary MCAS-related information is posted. Not everyone will have access to all 3 applications...it depends on what applications you have been assigned access by your Directory Administrator. If you have access, you'll see links to each of these applications when you log into the Security Portal (http://gateway.edu.state.ma.us/)

2013 Accountability Data
This application within the portal connects to 3 layers of reports for your school/district, similar to the Accountability tab in Profiles. Preliminary information about accountability level, PPI indicator values, points, and targets can be found here.

DropBox Central
- Accountability 2013 Data - does not contain 2013 accountability data at this time, as the data is not final
- MCAS 2013 Data - contains the August 2013 preliminary data files in .CSV and .XLS format for all tested students for whom your organization is accountable (one line per student - these are large and somewhat unwieldy files!)

Edwin Analytics 
A variety of preliminary MCAS reports (item, strand, performance/achievement, and growth) can be run for various grades, subjects, and/or subgroups within a school or district.

When the data is finalized (mid-September), there will also be updated 2013 data posted publicly to Profiles. However, the preliminary data is only available through the Security Portal with appropriate access. Please remember that the preliminary data is embargoed until the public release in September!

Saturday, August 10, 2013

Full Preliminary MCAS and Accountability Results

By now, you probably have taken a look at the full preliminary MCAS results and preliminary accountability data that DESE released last week (see the SAS update for more info). DESE held two webinars last week to provide more explanation and guidance. Thanks to my colleague Paula O'Sullivan (Greater Boston DSAC) for passing along some of the key points from the webinar, as I was unable to attend.
  • All data is embargoed until the public release in September! DESE states that you can use it for planning and research with appropriate staff and to let students know they earned a diploma. You should not discuss aggregate results in public or show preliminary aggregate reports at school committee meetings.
  • The MCAS discrepancy window closes on Tues August 13th at 5 p.m.
  • The state has changed slightly how they calculate the growth data, in that they now compare each student to a historical academic peer group. This means that the state SGP for each grade level will no longer necessarily be 50, it may be a bit higher or lower. The state results will be released in September.
  • The student file now includes, for each student, how long that student has been enrolled in your district, and in your school. In the future, Edwin Analytics will provide reports with MCAS results disaggregated by these variables.
  • Classroom reports with EOY SIMS data may be available in early September.
  • Preliminary accountability reports are available in a separate application within the security portal. More detailed webinars will be held in a few weeks. You can register on DESE's site. Updated guidance is also available. 
  • Later this calendar year, DESE anticipates rolling out Edwin Analytics to teachers.
Please let me know if you have any questions as you explore your preliminary data. I hope to see many of you at the upcoming regional Edwin Analytics workshops.

Tuesday, July 16, 2013

Summer Edwin Analytics Workshops Update

Just a reminder about the regional Edwin Analytics workshops I will be offering in August - all 3 sponsoring collaboratives have now posted the registration information, so I wanted to post an update. Please click the links below to register.

-----------------------------------------------------------

I will be offering the "Getting to Know Edwin Analytics" workshop several times in mid-late August (with the expectation that the preliminary MCAS data will be available at that time). There is no cost for these workshops; although priority registration will be given to individuals from Level 3 districts we anticipate to be able to accommodate all who would like to attend.

The workshop will provide a structured environment in which to explore data using the pre-defined reports in Edwin Analytics (formerly the Education Data Warehouse), including the Growth Model reports and classroom-level reports. This workshop is appropriate to new users but can serve as an update for previous users of the Education Data Warehouse. Participants will access their own district or school's data, and will have time to run and save reports for use in planning for the 2013-2014 school year.

All attendees must be assigned appropriate Security Portal access to Edwin Analytics by their district's Director Administrator prior to the workshop. Please register directly with the collaborative through whom the workshop is offered (it is not necessary that your district be a member of the collaborative to attend.)

8/15/13 - Bourne High School
8:30-11:30 am OR 12:15-3:15 pm
Please register through the Cape Cod Collaborative

8/20/13 - Whitman Hanson Regional High School
8:30-11:30 am
Please register through the North River Collaborative

8/22/13 - READS Collaborative, Middleborough
8:30-11:30 am
Please register through the READS Collaborative


Thursday, June 20, 2013

June 2013 Release of Preliminary Results

As many of you already know, preliminary MCAS results were made available to authorized users in DropBox Central (in the security portal) on Tuesday. This release includes full preliminary ELA results and preliminary multiple-choice results for grades 3–8 and 10 Math and grades 5 and 8 Science.

There is an interpretive guide to accompany this release (currently available in DropBox Central, to be posted to the 2013 MCAS Web page shortly) which provides information about various ways that the preliminary data (along with the 2013 released items) can be reviewed and utilized. The raw-to-scaled score conversion tables for ELA have also been posted.

In addition, the latest Student Assessment Update notes that preliminary MCAS-Alt results have been made available through DropBox Central, and ELA long compositions are posted online at www.mcasservicecenter.com.

On Friday, June 21, the same preliminary results will be made available through selected reports in Edwin Analytics.

Per the Interpretive Guide: "All data released prior to the official release of school and district results in mid-September are embargoed, which means the data cannot be released to the public or discussed in public meetings.... Preliminary MCAS data, including the Mathematics and STE multiple-choice results, can and should be used for educational planning purposes. Whenever preliminary results are printed for planning purposes, they should be clearly dated and labeled “preliminary,” with the understanding that some results may change with the official release in September."

As a reminder, I will be offering three regional Edwin Analytics workshops in August after the remainder of the preliminary data has been released, but in the meantime, please do not hesitate to contact me with questions.




Tuesday, June 18, 2013

Release of Spring 2013 MCAS Items

The Massachusetts Department of Elementary and Secondary Education has posted the released items for the Spring 2013 tests by grade and content area at: http://www.doe.mass.edu/mcas/2013/release/default.html.

Monday, June 17, 2013

Reminder: New ESE Security Portal URL

With the preliminary MCAS results being released on Tuesday (see earlier post for more details, along with the Student Assessment Services update sent out via email today), there’s a good chance you’ll want to log into the ESE Security Portal in the very near future to access Dropboxes or Edwin Analytics.

If you haven’t logged into the Security Portal in a while, you will notice some changes. Earlier in June, ESE installed a new “Identity Management System (IMS)” in order to align the Security Portal with the Commonwealth’s new secure access standards. Existing username and passwords will continue to work with the new Identity Management System and there is no need to change either credential, BUT the URL has changed.

New Security Portal URL: https://gateway.edu.state.ma.us/

If you're trying to access Edwin Analytics, you should click the first "Edwin Analytics" link on your portal home page after logging in to the URL above to get the latest version of Edwin Analytics. The second link that says "Edwin Analytics (EDW)" is the old version.

You should change any bookmarks you may have to the new URL. (Note that the "Security Portal" link in the drop-down menu on the ESE home page has been updated to this new URL, so if you normally access the portal this way you'll go to the right spot.) If you notice that you're being asked to log in twice, you may be accessing the portal through an incorrect URL. If you have any questions regarding your access, please contact your district’s Directory Administrator.

Wednesday, June 12, 2013

Summer Edwin Analytics Workshops

I will be offering the "Getting to Know Edwin Analytics" workshop several times in mid-late August (with the expectation that the preliminary MCAS data will be available at that time). There is no cost for these workshops; although priority registration will be given to individuals from Level 3 districts we anticipate to be able to accommodate all who would like to attend.

The workshop will provide a structured environment in which to explore data using the pre-defined reports in Edwin Analytics (formerly the Education Data Warehouse), including the Growth Model reports and classroom-level reports. This workshop is appropriate to new users but can serve as an update for previous users of the Education Data Warehouse. Participants will access their own district or school's data, and will have time to run and save reports for use in planning for the 2013-2014 school year.

All attendees must be assigned appropriate Security Portal access to Edwin Analytics by their district's Director Administrator prior to the workshop. Please register directly with the collaborative through whom the workshop is offered (it is not necessary that your district be a member of the collaborative to attend.)

8/15/13 - Bourne High School
8:30-11:30 am OR 12:15-3:15 pm
Please register through the Cape Cod Collaborative

8/20/13 - Whitman Hanson Regional High School
8:30-11:30 am OR 12:15-3:15 pm
Please register through the North River Collaborative
(info is not yet posted but will be in the next week or so)

8/22/13 - READS Collaborative, Middleborough
8:30-11:30 am OR 12:15-3:15 pm
Please register through the READS Collaborative


Tuesday, June 11, 2013

Edwin Teaching and Learning Pilot Update

The Edwin Teaching and Learning Group at ESE have published an update on the ET&L pilot project, including information about current and future pilot opportunities (for RTTT districts), content updates, and online resources.

The update document can be found under "News and Updates" on the Edwin page on the ESE site.

http://www.doe.mass.edu/edwin/


Friday, May 31, 2013

MCAS Release Schedule 2013

The Massachusetts Department of Elementary and Secondary Education has published the release schedule for 2013 MCAS data. Today's memorandum plus the downloadable schedule can be found at http://www.doe.mass.edu/mcas/.

In summary: Starting on June 18, school leaders will have access to student roster files (ELA all question types, Science multiple choice for grades 5 and 8, Math multiple choice only, MCAS-Alt full results) through Dropbox Central. Selected reports will be available in Edwin Analytics several days later. A second round of data will be released on August 7, with the discrepancy reporting window occuring between August 7-13. Please see the schedule for all the details.

As mentioned in yesterday's post, the Security Portal (home of Dropbox Central and Edwin Analytics) will be undergoing some changes this weekend. The new URL for the portal will be https://gateway.edu.state.ma.us.

I will be holding regional Edwin Analytics workshops on August 15 at Bourne HS and August 22 at READS in Middleboro and will post registration information to the blog next week. If anyone is interested in a late June workshop given that there will be some new data to explore by June 21, please let me know (laura.tilton@dsacma.org) as I could certainly schedule one if the interest exists.

Happy Friday!

Thursday, May 30, 2013

Changes to Security Portal and Edwin Analytics / Friday Webinars

ESE is currently working to install a new Identity Management System (IMS) so that the Security Portal will align with the Commonwealth's new secure access standards. The first phase of this upgrade will take place over the weekend of June 1-2, as the first set of portal applications will be migrated over to the new system. The Security Portal will be unavailable during this time, as well as on two additional dates (June 6 and June 8) when additional migrations are scheduled to take place. During the June 3-9 period you may experience the need to log in twice to the portal as the upgrade will be in progress.

The Edwin Analytics/EDW team will host two WebEx sessions  tomorrow at 11:00 am and 1:00 pm to review the impact of this IMS on Edwin Analytics/EDW. I will also share any new information here on the blog.

The WebEx agenda includes:
1.       Discussion of  IDM final memo, Application Release Schedule
2.       Discussion of new security portal
3.       Review of the new User Interface

To access the webinar:

Telephone number if needed: 1-877-820-7831
Password: 683154# 
To Un-mute/Mute your phone dial *6
For operator help dial *0
http://commonwealthofmass.centurylinkccc.com/CenturylinkWeb/datawarehouse

If you need additional information, please see the Data Warehouse contact or Directory Administrator for your district.

Tuesday, May 28, 2013

Student Assessment Update - ACCESS For ELLs; Retirement of Score Request Tool

The latest Student Assessment Update is now posted on the ESE Web site at http://www.doe.mass.edu/news/news.aspx?id=7487, including updates on ACCESS for ELLs score report interpretation resources and retirement of the MCAS Score Request Tool.

Please note: These updates come out via email before they are posted online. Please see the information listed at the bottom of the update page for more information about receiving the updates via email.

Tuesday, May 21, 2013

District Determined Measures: Technical Guide A

From the May 17 Commissioner's Weekly Update:

Technical Guide A, a supplement to Part VII of the Mass. Model Educator Evaluation System, is now available at http://www.doe.mass.edu/edeval/ddm/TechnicalGuide.pdf. The guide describes key assessment concepts that district teams should consider when selecting district-determined measures. It also provides guidance on how to select or modify existing assessments and how to build a new assessment.

Thursday, May 9, 2013

Educator Evaluation Spring Convening 5/29/2013

Educator Evaluation Spring Convening:
Connecting Policy, Practice, and Practitioners
Wednesday, May 29th, 7:45am-4:30pm


Sturbridge Host Hotel & Conference Center
366 Main Street, Sturbridge, Massachusetts 01566

The Department of Elementary and Secondary Education is excited to bring together public school districts, associations and educator preparation programs for a day of celebration, learning and reflection about promising and emerging practices in the new Educator Evaluation framework. This new framework will be implemented statewide in the 2013-2014 school year; however, many districts have already begun to engage in this work. This is an opportunity for districts currently implementing the framework to share their successes and challenges and indentify new and actionable strategies for improving implementation in the coming year. It is also a chance for newly implementing districts to learn from and connect with colleagues who have been engaged in this work over the past two years. ESE is pleased to bring together districts, associations, and educator preparation programs to build connections and relationships across the educator pipeline in Massachusetts.

More information and registration (by 5/15) is available at:
http://registration.airprojects.org/MDESEEducatorEvaluationDistrict/

Thursday, May 2, 2013

Resource Round-Up

Wow...what a busy time of year! There is lots of great information on data, assessment, and educator effectiveness related topics coming our way...here are some of the latest resources that you may find valuable:

PARCC Release of Assessment Blueprints and Test Specifications: PARCC has released a set of test specification documents, including assessment blueprints and evidence statement tables, to help educators and the general public better understand the design of the PARCC assessments.


EWIS Implementation Support Pilot: ESE is looking to select approximately six districts to receive regular on-site technical assistance, facilitation, and training to support their use of the Early Warning Indicator System (EWIS) data in the 2013-14 school year. Applications are due on Friday, May 31st.

New Educator Evaluation QRG: Data Collection provides information about the data elements that districts will report as part of EPIMS related to the educator evaluation system.

ESE Newsletters and Update Pages:
Student Assessment Updates (latest update: 4/22)
Educator Evaluation E-Newsletter (latest version: April 2013)
RTTT Newsletter (latest version: Spring 2013)

Upcoming Webinars:
ACCESS for ELLs Score Reporting - May 23, 28, or 29
Assessment Literacy Webinar Series - Archives of Sessions 1-3 (Session 4 will be in July)

Monday, April 29, 2013

MCAS Participation and the Need for Data Quality

As schools gear up to administer the Math and Science MCAS exams next month, I thought it would be a good opportunity for a reminder of the importance of data quality, particularly when it comes to MCAS participation data.

As you may know, one way that a school can be classified as “Level 3” is by having less than 90% MCAS participation for the aggregate or any subgroup. (Schools can similarly be classified as “Level 2” by having less than 95% MCAS participation for the aggregate or any subgroup; see page 5 of the School Leaders’ Guide for additional information.)

Last year, 32 schools (27 of which were high schools) were identified for low MCAS participation statewide - 9 in the "very low / <90%" (Level 3) category and 23 in the "low / <95%" (Level 2) category. In some cases, the classification of a single student brought the participation percentage of a subgroup below 90%, thus designating the school as Level 3. In addition, several schools were preliminarily identified as Level 3 for low participation, but after these schools looked further at their data, they identified errors they had initially missed which brought their participation above the 90% mark. School-wide participation in achievement assessments is important, and it's essential that every student is counted correctly. It’s clear that quality data collection, roster verification, and proper  classification can prevent confusion and misidentification.

Additional guidance related to MCAS participation requirements, including information specific to high school students, can be found in Part II: Student Participation in the the Principal’s Administration Manual. Appendix D of this document also provides information about reporting various types of student non-participation, and how this information is used for MCAS reporting and accountability reporting.

For schools interested in looking back at their participation history (for both aggregate and subgroups), this data can be found in two places on the Profiles pages for that school. On the "Assessment" tab for each year, there is a report entitled "Participation Report." On the "Accountability" tab, clicking "View Detailed 2012 Data" and scrolling to the bottom of the page will bring you to a table of participation data by subgroup.

Monday, April 8, 2013

PARCC Massachusetts Web Site Now Live

The PARCC Massachusetts Web site (through the Massachusetts Department of Higher Education) is now live at http://www.mass.edu/parcc.  The site provides a variety of links, documents, and resources related to the PARCC assessments, Common Core, and College and Career Readiness.

Friday, April 5, 2013

Update on PPI Webinars

I posted yesterday about a set of PPI webinars being offered by DESE in May. I have since learned that the webinar is targeted for Charter, ELT and Innovation Schools due to aspects specific to their performance-based accountability structures.

DESE anticipates holding additional trainings on PPI and the new accountability/assistance leveling system later this spring or summer that will be broadly available to all public schools and districts. I will post information here as it becomes available.

In order to minimize confusion, DESE staff will clear the registration system of all current registrants and ask that you re-register if these trainings are still appropriate for you. ( If you did register, you probably already received an email with a very similar message directly from DESE.)

My apologies for any confusion this may have caused!

Thursday, April 4, 2013

Edwin Analytics Updates

The following information was shared in yesterday's Edwin Analytics webinar (thanks to my colleague Paula O'Sullivan at the Greater Boston DSAC for sharing!)

1. February Biology results are now available (as of 3/25) in the following student-level reports:
  • Student Profile Report (PR600)
  • Assessment History Summary (PE613)
  • Assessment Detail (PE606)
  • Student Item Analysis Report (IT616)
  • MCAS Student Roster (PE617)
2. Competency Determination status updates are now available  (as of 3/25) in the following student-level reports:
  • Student Profile Report (PR600)
  • Assessment History Summary (PE613)
  • Student CD Roster (PE618)
3. March Retest data will be available in mid-late April.

4. The Edwin Analytics team is seeking feedback on the new EWIS and Postsecondary Reports. Please share any feedback you may have with me (and I will pass along) or directly with the Edwin team (Edwin@doe.mass.edu).

5. Internet Explorer 10 is NOT compatible with Cognos. If you use IE and upgraded to 10, there are steps you can take to revert back to IE 9 (please visit the Edwin Analytics Home Page for more information.)

Two other updates shared related to data and technology: 

  • You can find ESE’s School Interoperability Framework (SIF) updates, past newsletters and other materials on the SIF site
  • DESE is planning a state-wide IT directors meeting in Andover on May 16th. The purpose of this meeting to build collaboration among IT directors. More information will be coming.

Tuesday, March 19, 2013

ESE Assessment Literacy Webinar Series

ESE is offering an 8-part webinar series which will run from now through December, focused on assessment literacy in the context of district-determined measures (DDM's) for educator evaluation. These sessions will run 60-90 minutes and are designed for district and/or school teams who will be involved in the identification and selection of district-determined measures. The webinars and related materials will be archived and posted online along with the schedule at http://www.doe.mass.edu/edeval/ddm/webinar.html.

Webinar Session 1, "Introduction: District-Determined Measures and Assessment Literacy" was offered last Thursday, March 14, but will run a second time on Wednesday, March 27, at 4:00 pm. The registration is available online (the registration page also includes the handouts/PowerPoint for Session 1) through this link.

Also, stay tuned to the ESE site for "Technical Guide A" which will be coming this spring to accompany Part VII: Rating Educator Impact on Student Learning Using District-Determined Measures of Student Learning.

Monday, March 11, 2013

PARCC Assessment Update

The Partnership for Assessment of Readiness for College and Careers (PARCC) recently released a guidance document which contains important information on test design, the testing window and approximate testing time, and "rule of thumb" guidance on the number of computer devices needed to administer the PARCC assessments. The direct link to the PDF guidance document is here, or you can visit PARCC's Assessment Administration page for the latest news and updates, including a readiness tool, FAQ, user guide, and more.
.

Monday, March 4, 2013

Additional Guidance on District-Determined Measures

As part of the Commonwealth's new educator evaluation system, districts will be required to identify and administer a district-wide set of student performance measures for each grade/subject that "permit a comparison of student learning gains." Where MCAS student growth percentile (SGP) exists, regulations state that it must be used as one of these measures, but it will be up to districts to identify/develop additional measures of student learning gains for many grades and subjects.

(You may recall a previous blog post about an activity designed to help teachers engage with and reflect on classroom-level growth data...I continue to use this activity with very positive responses from teacher teams!)

ESE has recently released a Quick Reference Guide on District-Determined Measures (along with QRG's on other EdEval topics) which summarizes the regulations and timeline related to the implementation of DDM's for all districts in Massachusetts. Additional guidance on the identification and selection of DDM's and related topics is scheduled for release later this Spring.

Monday, February 25, 2013

Three New Data Sets – Now Available in Profiles

Per the latest Commissioner's Email Update, ESE has recently added the following three new data sets to School/District Profiles.
  • 2012-13 Enrollment: New district and school reports to reflect 2012-13 enrollment include enrollment by grade, race/gender, and selected populations.
  • Attrition: This data set looks at attrition across school years. Users can filter the data by school year, district or school levels, and various subgroups. Clicking on an individual district or school will show all of the subgroups for that entity. To view the data, go to http://profiles.doe.mass.edu/state_report/attrition.aspx.
  • MassCore Completion: This data set reflects the percent of graduates who have completed the MassCore curriculum as reported by districts. While we acknowledge there are data quality issues within this data set, by publishing the data, we expect the quality of the reporting will improve. In future years, ESE will use its own course scheduling data collection to supplement the district-reported data. The data are online at http://profiles.doe.mass.edu/state_report/masscore.aspx
Please note that the links above will take you to the state report showing the data for all districts/schools. From an individual school or district Profiles page, these new reports can all be found under the "Student" tab.

Friday, February 1, 2013

The Education Data Warehouse is now Edwin Analytics

Today's announcement from ESE:

As of today, the Education Data Warehouse (EDW) can be accessed through the Edwin Analytics link on the ESE’s Security Portal.

Edwin Analytics is accessible using the EDW login roles. These roles should be assigned to district and school users in accordance with the rules set forth in the Department’s Data Access Policy for the EDW.

Edwin Analytics includes:
  • All EDW state data (MCAS, Growth, SIMS, EPIMS, SCS) and all personal content saved to a users’ ‘My Folders’, district, and school folders.
     
  • Six new Early Warning Indicator System (EWIS) and Postsecondary Outcomes reports located on the District, School, and Student tabs. (The same 5 district and school level reports are on the District and School tabs and one student level report is on the Student tab.)

Contact for Support:
  • For access to Edwin Analytics, contact your district’s Directory Administrator (DA). If you already have access to the EDW, you automatically have access to Edwin Analytics and the specific reports available for your level of access.
  • For technical support questions, please contact the Data Warehouse Contact person in your district. A list can be found at Profiles: People Search and choose Data Warehouse Contact in the ‘Function’ dropdown.
  • If you are the district’s Data Warehouse contact and need additional technical support, contact the EDW team at datawarehouse@doe.mass.edu or 781-338-6820.
  • For data issues including analysis, please contact the team responsible for data content:
  • For general Edwin questions, please contact edwin@doe.mass.edu 
I am also happy to answer questions or provide guidance, so please don't hesitate to contact me! Your questions help me understand how districts are using this new data resource. I hope to offer some "Edwin Analytics" workshops this spring as well.