63L-2.001: Purpose and Scope
63L-2.002: Definitions
63L-2.003: Frequency of Reviews
63L-2.004: Program Notification and Review Preparation
63L-2.005: Conducting Reviews
63L-2.006: Indicators, Standards and Ratings
63L-2.007: Certified Reviewers
63L-2.008: Challenges to Program Reports
63L-2.009: Failed Standards
63L-2.010: Internal Review Board
63L-2.011: Deemed Status
PURPOSE AND EFFECT: The rule chapter governs the administration of the department’s statewide quality assurance system, which ensures the quality of programs operated by or on behalf of the department to serve youth and families.
SUMMARY: The rule chapter details the quality assurance system, including the review and reporting of program performance.
SUMMARY OF STATEMENT OF ESTIMATED REGULATORY COSTS: No Statement of Estimated Regulatory Cost was prepared.
Any person who wishes to provide information regarding a statement of estimated regulatory costs, or provide a proposal for a lower cost regulatory alternative must do so in writing within 21 days of this notice.
SPECIFIC AUTHORITY: 985.64 FS.
LAW IMPLEMENTED: 985.632 FS.
A HEARING WILL BE HELD AT THE DATE, TIME AND PLACE SHOWN BELOW:
DATE AND TIME: Thursday, June 10, 2010, 2:00 p.m.
PLACE: DJJ Headquarters, 2737 Centerview Drive, General Counsel’s Conference Room 3223, Tallahassee, Florida. For information about participation by telephone, contact John Milla at (850)921-4129
THE PERSON TO BE CONTACTED REGARDING THE PROPOSED RULE IS: John Milla, 2737 Centerview Dr., Ste.3200, Tallahassee, FL 32399-3100, e-mail: john.milla@djj.state.fl.us
THE FULL TEXT OF THE PROPOSED RULE IS:
63L-2.001 Purpose and Scope.
This rule establishes the process and requirements for the department’s statewide quality assurance system, as mandated by subsection 985.632(5), F.S.
Rulemaking Authority 985.632, 985.64 FS. Law Implemented 985.632 FS. History–New__________.
63L-2.002 Definitions.
(1) Assistant secretary – The department employee responsible for the statewide administration of a respective program area as defined in subsection 20.316(2), F.S.
(2) Certified reviewer – A department or provider program manager, supervisor, or other experienced employee who has been trained to participate in reviews and provides technical and professional assistance, expertise, and program knowledge to the review team.
(3) Chief of Quality Assurance – The department employee responsible for the statewide administration of the Bureau of Quality Assurance.
(4) Contract manager/program monitor – A department employee assigned to serve as the liaison between the department and the provider in all matters involving the contract.
(5) Contract provider – Any person or business entity that contractually operates a program on behalf of the department.
(6) Director of Program Accountability – The department employee responsible for the statewide administration of the Office of Program Accountability, which includes the Bureau of Quality Assurance.
(7) Indicator – The basic level of evaluation in the quality assurance process that focuses on a specific aspect of the delinquency service delivered by the program.
(8) Internal review board – The panel empowered to make a recommendation to the Secretary for or against the cancellation of a provider’s contract as a corrective action in accordance with subsection 63L-2.009(4), F.A.C.
(9) Lead reviewer – The review specialist who is responsible for all communication with the program to be reviewed, working through any logistical problems prior to the review, monitoring and time management of the review team, and assembling, briefing, and coordinating the activities of the review team.
(10) Performance rating – A rating category based on numerical values that reflects a program’s performance. Performance is rated for indicators, standards, and the overall program.
(a) Indicator-level performance – Performance rating for the indicator being reviewed, as determined by the review team.
1. Exceptional – The program consistently meets all requirements, and a majority of the time exceeds most of the requirements, using either an innovative approach or exceptional performance that is efficient, effective, and readily apparent. Numerical value: 10.
2. Commendable – The program consistently meets all requirements without exception, or the program has not performed the activity being rated during the review period, and exceeds procedural requirements and demonstrates the capacity to fulfill those requirements. Numerical value: 8.
3. Acceptable – The program consistently meets requirements, although a limited number of exceptions occur that are unrelated to the safety, security, or health of youth, or the program has not performed the activity being rated during the review period, and meets all procedural requirements and demonstrates the capacity to fulfill those requirements. Numerical value: 7.
4. Minimal – The program does not meet requirements, including at least one of the following: an exception that jeopardizes the safety, security, or health of youth; frequent exceptions unrelated to the safety, security, or health of youth; or ineffective completion of the items, documents, or actions necessary to meet requirements. Numerical value: 5.
5. Failed – The items, documentation, or actions necessary to accomplish requirements are missing or are done so poorly that they do not constitute compliance with requirements, or there are frequent exceptions that jeopardize the safety, security, or health of youth. Numerical value: 0.
(b) Standard-level performance – Performance rating for the standard being reviewed, based on the aggregate of indicator-level performance ratings within that standard. The definitions are the same as those used for indicator-level performance ratings.
1. Exceptional – 90% to 100%
2. Commendable – 80% to 89%
3. Acceptable – 70% to 79%
4. Minimal – 60% to 69%
5. Failed – 0% to 59%
(c) Overall program performance – Performance rating for the program being reviewed, based on the aggregate of all indicator-level performance ratings within all standards. The definitions are the same as those used for indicator-level performance ratings.
1. Exceptional – 90% to 100%
2. Commendable – 80% to 89%
3. Acceptable – 70% to 79%
4. Minimal – 60% to 69%
5. Failed – 0% to 59%
(11) Program – An entity that is operated by or on behalf of the department, either directly or through a contract provider, that provides delinquency services to youth.
(12) Program administrator – The department employee responsible for the management of quality assurance operations and staff within a geographical area determined by the Chief of Quality Assurance.
(13) Quality assurance – The system for the assessment of program performance in areas such as management, operations, and service delivery. The Bureau of Quality Assurance is the office within the department responsible for overseeing this process.
(14) Regional director – The department employee responsible for the regional administration of a respective program area as defined in subsection 20.316(2), F.S.
(15) Review specialist – A Bureau of Quality Assurance employee who is responsible for leading or otherwise conducting reviews as assigned by the program administrator.
(16) Review team – A group of individuals, typically composed of a lead reviewer, review specialist(s), certified reviewer(s), and the contract manager/program monitor, who are assembled to review a program.
Rulemaking Authority 985.632, 985.64 FS. Law Implemented 985.632 FS. History–New__________.
63L-2.003 Frequency of Reviews.
(1) All programs in which youth are housed overnight shall be reviewed each fiscal year.
(2) All programs in which youth are not housed overnight shall be reviewed every other fiscal year.
(3) Reviews shall be conducted at any other time at the request of the Secretary.
(4) New programs, as well as existing programs that are transferred to a different contract provider, shall not be eligible for review for a period of nine months following the execution of the initial contract.
Rulemaking Authority 985.632, 985.64 FS. Law Implemented 985.632 FS. History–New__________.
63L-2.004 Program Notification and Review Preparation.
(1) Programs shall receive notification seven (7) calendar days prior to the review, to include:
(a) A personal telephone call by the lead reviewer or designee to the program director/superintendent or administrator in-charge.
(b) A follow-up e-mail from the lead reviewer or designee to the program director/superintendent or administrator in-charge, requesting documentation that must be submitted to the lead reviewer by the end of the same business day, such as personnel and training records, information listed in subsection 63L-2.004(2), F.A.C., and variances or waivers obtained in accordance with subsections 63L-2.006(7) and 63L-2.006(8), F.A.C.
(2) The lead reviewer or designee shall review the following information, if applicable, prior to the review:
(a) The program’s current contract, including amendments;
(b) Prior contract monitoring reports;
(c) Incident reports received by the department within the review period defined by subsection 63L-2.006(2), F.A.C., or paragraph 63L-2.009(2)(b), F.A.C., if applicable;
(d) Accreditation reports, if applicable and available;
(e) Electronic training records for program staff;
(f) Juvenile Justice Information System (JJIS) and other electronic information systems;
(g) Medical Services Profile Form (QA1 July 2010), which is incorporated into this rule and is accessible electronically at http://www.djj.state.fl.us/QA/index.html; and
(h) Mental Health and Substance Abuse Services Profile Form (QA2 July 2010), which is incorporated into this rule and is accessible electronically at http://www.djj.state.fl.us/QA/index.html.
(3) Certified reviewers shall be notified in the following manner:
(a) Before the fiscal year begins, the lead reviewer or designee shall e-mail the certified reviewer and his/her supervisor with the dates that the certified reviewer is to be available to participate in a review.
(b) Thirty (30) calendar days prior to the review, the lead reviewer or designee shall e-mail the certified reviewer reminding him/her of the review dates, and advising whether or not the review requires overnight travel. This letter shall not reveal the name or location of the program being reviewed.
(c) Seven (7) calendar days prior to the review, the lead reviewer or designee shall contact the certified reviewer with the details of the review, including the name and location of the program being reviewed.
Rulemaking Authority 985.632, 985.64 FS. Law Implemented 985.632 FS. History–New__________.
63L-2.005 Conducting Reviews.
(1) Prior to the review, the lead reviewer shall arrange a meeting for review team members to receive assignments and a briefing on the review process.
(2) Shortly after the review team arrives at the program, the lead reviewer shall conduct an entrance conference with the program director/superintendent to discuss the review process.
(3) The review team member(s) assigned to any security- or safety- related standards/indicators shall conduct a tour of the program to determine the current practices. The lead reviewer and other review team members may participate in the program tour. During the review, team members shall visit and observe the areas of the program pertinent to their assignments.
(4) The review team shall meet at least daily to report the status of each member’s progress and share information. Each review team member shall summarize his/her findings with other team members, seek other team members’ observations that may pertain to his/her assignment, or encourage team members to report information that may or may not support a particular thought or observation.
(5) The review team shall conduct daily debriefing sessions with the program director/superintendent and other program staff for the purposes of clarifying findings, answering questions, and requesting additional information, if necessary.
(6) Within twenty-four (24) hours after the final daily debriefing:
(a) The lead reviewer shall discuss the review team’s preliminary findings with the program administrator or designee; and
(b) The program administrator or designee shall send preliminary indicator-level, standard-level, and overall program performance ratings to the program director/superintendent via e-mail.
Rulemaking Authority 985.632, 985.64 FS. Law Implemented 985.632 FS. History–New__________.
63L-2.006 Indicators, Standards, and Ratings.
(1) Standards are developed for each program type within the department and are based on Florida Statute, administrative rule, department policy, and contract language. Standards are incorporated into this rule and are accessible electronically at http://www.djj.state.fl.us/QA/index.html:
(a) Child In Need of Services/Family In Need of Services (CINS/FINS) Standards (July 2010);
(b) Community Supervision Standards (July 2010);
(c) Day Treatment Standards (July 2010);
(d) Detention Standards (July 2010);
(e) Diversion Standards (July 2010);
(f) Practical Academic Cultural Education (PACE) Standards (July 2010); and
(g) Residential Standards (July 2010).
(2) Review team members shall review the program’s written records and files, conduct surveys and interviews, and observe the program’s operations to determine prevailing practice for a period of six (6) months preceding the review. If the program has not performed the activity being rated during the six-month review period, with the approval of the lead reviewer, team members may extend the scope of the review period back to the date of the last review, but no more than twelve (12) months.
(3) The review team shall discuss each indicator and assign the most appropriate performance rating. Professional judgment shall play an important role in determining the performance rating. Review team members must be prepared to explain and provide justification for this determination. The rating of each indicator and standard is open for discussion. The review team may accept the findings, request additional information, or change the rating as a result of the discussion. The lead reviewer maintains final approval authority during this phase of the process.
(4) Review team members shall use the following categories when rating indicator-level performance: Exceptional, Commendable, Acceptable, Minimal, and Failed. Criteria for each performance rating category is defined in paragraph 63L-2.002(10)(a), F.A.C.
(5) Standard-level performance ratings shall be determined by dividing the aggregate of indicator-level ratings within that standard by the number of applicable indicators in that standard multiplied by 10 (the highest possible indicator-level rating). The resulting percentage shall determine the standard-level performance rating. Percentage ranges for standard-level program performance ratings are defined in paragraph 63L-2.002(10)(b), F.A.C.
(6) The overall program performance rating shall be determined by dividing the aggregate of all indicator-level ratings by the number of all applicable indicators multiplied by 10 (the highest possible indicator-level rating). The resulting percentage shall determine the overall program performance rating. Percentage ranges for overall program performance ratings are defined in paragraph 63L-2.002(10)(c), F.A.C.
(7) An indicator that is addressed by the program in a manner that varies from the indicator defined in subsection 63L-2.006(1), F.A.C. shall be rated based on an alternative compliance measure.
(a) An indicator that is based on a program area-specific administrative rule requirement shall be rated based on an alternative compliance measure when the program obtains a variance of the applicable administrative rule.
(b) All other indicators shall be rated based on appropriate alternative compliance measures when:
1. The program specifically meets the criteria for an alternative compliance measure as described in the indicator; or
2. The program obtains a variance of subsection 63L-2.006(1), F.A.C. The program shall be responsible for obtaining a variance in accordance with Section 120.542, F.S., and Chapter 28-104, F.A.C.
(8) An indicator that is not applicable to the program being reviewed shall not be rated and shall not count for or against the program in the rating process.
(a) An indicator that is based on a program area-specific administrative rule requirement shall be non-applicable when the program obtains a waiver of the applicable administrative rule.
(b) All other indicators shall be non-applicable when:
1. The program specifically meets the criteria for a non-applicable rating as described in the indicator; or
2. The program obtains a waiver of subsection 63L-2.006(1), F.A.C. The program shall be responsible for obtaining a waiver in accordance with Section 120.542, F.S., and Chapter 28-104, F.A.C. It is not necessary for a program to obtain a waiver for a discretionary administrative rule requirement.
Rulemaking Authority 985.632, 985.64 FS. Law Implemented 985.632 FS. History–New__________.
63L-2.007 Certified Reviewers.
(1) Each contract provider program shall provide the number of certified reviewers prescribed by the contract to participate in at least one review during the fiscal year.
(2) State-operated programs shall provide the number of certified reviewers identified by the Chief of Quality Assurance and applicable assistant secretary as necessary to conduct reviews as prescribed by Rule 63L-2.003, F.A.C.
(3) Travel, lodging, and meals/per diem for participation on a review shall be at the expense of the certified reviewer’s program.
(4) In order to qualify as a certified reviewer, a state or provider employee must:
(a) Successfully complete the certified reviewer certification training conducted by the Bureau of Quality Assurance, at the expense of the employee’s program; and
(b) Be a manager, supervisor, or other experienced employee possessing at least a Bachelor’s degree from an accredited university and having a minimum of two years of experience in juvenile justice programs. Exceptions shall be allowed upon approval of the Chief of Quality Assurance based on the employee’s current position and years of experience in juvenile justice programs.
Rulemaking Authority 985.632, 985.64 FS. Law Implemented 985.632 FS. History–New__________.
63L-2.008 Challenges to Program Reports.
(1) The program administrator or designee shall e-mail a draft of the program report to the program director/superintendent within twenty (20) calendar days of the conclusion of the review.
(2) The program director/superintendent or designee shall have five (5) working days to review the draft program report and respond with any challenge related to accuracy, wording, or ratings contained in the draft program report. Any challenge and supporting documentation not received by the close of business on the fifth working day shall not be considered.
(3) The program administrator shall attempt to resolve the challenge with the program director/superintendent.
(4) If the program director/superintendent is not satisfied with the program administrator’s resolution, he/she shall have five (5) working days from receipt of the response to refer the challenge to the Chief of Quality Assurance, who shall, in consultation with the Director of Program Accountability and the applicable assistant secretary or designee, make the final decision regarding the challenge.
Rulemaking Authority 985.632, 985.64 FS. Law Implemented 985.632 FS. History–New__________.
63L-2.009 Failed Standards.
(1) This subsection applies to any program that receives a standard-level or overall program performance rating of Failed, as defined by subparagraphs 63L-2.002(10)(b)5. and 63L-2.002(10)(c)5., F.A.C.
(2) A follow-up review of the program shall be conducted within six (6) months of publication of the program report to determine if corrective action taken by the program has resulted in improvements.
(a) The program shall be reviewed on the Failed standard(s) only.
(b) Review team members shall review the program’s written records and files, conduct surveys and interviews, and observe the program’s operations to determine prevailing practice for a period of three (3) months preceding the review. If the program has not performed the activity being rated during the three-month review period, with the approval of the lead reviewer, team members may extend the scope of the review period back to the date of the last review, but no more than six (6) months.
(3) For programs that received a standard-level performance rating of Failed, if the review team determines that corrective action taken by the program has not resulted in improvements to the extent that the program would receive a standard-level performance rating of at least Minimal on a regular review, further corrective action shall be taken as determined by the Secretary or designee.
(4) For programs that received an overall program performance rating of Failed, if the review team determines that corrective action taken by the program has not resulted in improvements to the extent that the program would receive an overall program performance rating of at least Minimal on a regular review, further corrective action shall be taken in accordance with paragraph 985.632(5)(f), F.S.
(5) If cancellation of the provider’s contract is pursued as a corrective action in accordance with subsection 63L-2.009(4), F.A.C., the provider shall have the opportunity to submit information to the internal review board in accordance with subsection 63L-2.010(4), F.A.C.
(6) The next regular review shall be conducted in accordance with Rule 63L-2.003, F.A.C., but no earlier than six months from publication of the program report addendum that documents the findings of the follow-up review conducted in accordance with subsection 63L-2.009(2), F.A.C.
Rulemaking Authority 985.632, 985.64 FS. Law Implemented 985.632 FS. History–New__________.
63L-2.010 Internal Review Board.
(1) The department shall establish an internal review board chaired by the Deputy Secretary. Membership shall include the Chief of Staff, the assistant secretary for the program area reviewed, and a representative of the general counsel’s office.
(2) The purpose of the internal review board is to determine if there are documented extenuating circumstances that contributed to the determination that the program would not receive an overall program performance rating of at least Minimal on a regular review, in accordance with subsection 63L-2.009(4), F.A.C.
(3) The department shall serve the provider with the final report and written notice of their rights and how to initiate an internal review board meeting by registered or certified mail, return receipt requested.
(4) The provider shall have ten (10) working days from receipt of the certified letter to present any documentation of extenuating circumstances to the Deputy Secretary.
(a) If the provider fails to respond within ten (10) working days, the department shall proceed with cancellation of the contract without an internal review board meeting.
(b) If the provider submits documentation of extenuating circumstances that impacted their performance, an internal review board meeting shall be scheduled.
(5) The internal review board shall review and consider the documentation, as well as any other pertinent information.
(a) The provider shall be given an opportunity to present information in-person or via conference call.
(b) The internal review board shall consider all information and make a recommendation to the Secretary within ten (10) working days following the meeting.
(6) The Secretary or designee shall make the final decision regarding any contract action to be taken.
Rulemaking Authority 985.632, 985.64 FS. Law Implemented 985.632 FS. History–New__________.
63L-2.011 Deemed Status.
(1) The department shall recognize programs that have attained the highest levels of performance as measured by the quality assurance process.
(2) The department shall award deemed status to any program in which youth are housed overnight that achieves the following during its review:
(a) An overall program performance rating of at least Commendable, as defined by paragraph 63L-2.002(10)(c), F.A.C.;
(b) A standard-level performance rating of at least Commendable, as defined by paragraph 63L-2.002(10)(b), F.A.C., on all health, mental health/substance abuse, and safety/security standards; and
(c) A standard-level performance rating of at least Acceptable, as defined by paragraph 63L-2.002(10)(b), F.A.C., on all other standards.
(3) The department shall award deemed status to any program in which youth are not housed overnight that achieves the following during its review:
(a) An overall program performance rating of at least Commendable, as defined by paragraph 63L-2.002(10)(c), F.A.C.; and
(b) A standard-level performance rating of at least Acceptable, as defined by paragraph 63L-2.002(10)(b), F.A.C., on all standards,
(4) In the fiscal year following the qualifying review of any program in which youth are housed overnight, there shall be a one-day review that focuses on healthcare, mental health/substance abuse, and safety/security standards. The second fiscal year following the qualifying review, there shall be a regular review.
(5) Deemed status shall have no impact on the reviews of programs in which youth are not housed overnight, as there will be no review during the fiscal year following the qualifying review pursuant to subsection 63L-2.003(2), F.A.C. The second fiscal year following the qualifying review, there shall be a regular review.
(6) Upon request by the Secretary, or during the one-day review, if the review team determines that the program would not receive an overall program performance rating of at least Acceptable on a regular review, the program shall immediately lose its deemed status, and a regular review shall be conducted within ninety (90) days.
Rulemaking Authority 985.632, 985.64 FS. Law Implemented 985.632 FS. History–New__________.