How BQA uses Amazon Bedrock to streamline education quality reporting

Given the value of data today, organizations across industries handle vast amounts of data across multiple formats. Manually reviewing and processing this information is difficult, time-consuming, and subject to potential errors. This is where Intelligent Document Processing (IDP) combined with the power of generative AI comes in as an innovative solution.

Powering IDP is the integration of generative AI, which utilizes large-scale language models (LLMs) and generative techniques to understand and generate human-like text. This integration enables organizations to not only extract data from documents, but also interpret, summarize, and generate insights from the extracted information, enabling more intelligent and automated document processing workflows.

The Education and Training Quality Authority (BQA) plays an important role in improving the quality of education and training services in the Kingdom of Bahrain. BQA reviews the performance of all educational and training institutions, including schools, universities and vocational colleges, thereby promoting the professional advancement of the country’s human capital.

BQA oversees a comprehensive quality assurance process, including setting performance standards and conducting objective reviews of training institutions. This process involves the collection and analysis of extensive documentation, including Self-Evaluation Reports (SERs), supporting evidence, and various media formats from the institutions under review.

The collaboration between BQA and AWS was facilitated through the Cloud Innovation Center (CIC) program, a joint initiative between AWS, Tamkeen, and Bahrain’s leading universities, including Bahrain University of Technology and the University of Bahrain. The CIC program aims to foster innovation within the public sector by providing a collaborative environment where government agencies can work closely with AWS consultants and university students to develop cutting-edge solutions using the latest cloud technologies. The purpose is

As part of the CIC program, BQA built a proof-of-concept solution that leverages AWS services and generative AI capabilities. The main objective of this proof of concept was to test and validate the proposed technology and demonstrate its feasibility and potential to streamline BQA reporting and data management processes.

This post describes how BQA leveraged the power of Amazon Bedrock, Amazon SageMaker JumpStart, and other AWS services to streamline its overall reporting workflow.

Challenge: Streamlining self-assessment reports

BQA has traditionally provided SER templates to education and training institutions as part of the review process. Institutions are required to submit a review portfolio that includes a completed SER and supporting documentation, which may not fully comply with established reporting standards.

The existing process presented several challenges.

• Inaccurate or incomplete submissions – Agencies may provide incomplete or inaccurate information in submitted reports and supporting evidence, which may create gaps in the data needed for a comprehensive review.
• Supporting evidence is missing or insufficient – Supporting materials provided as evidence by institutions often did not substantiate the report’s claims and called into question the evaluation process.
• consumes a lot of time and resources – This process requires significant time and resources to be spent manually reviewing the submission and following up with the institution to request additional information as necessary to correct the submission; As a result, the overall review process slowed down.

These challenges have highlighted the need for a more streamlined and efficient approach to the submission and review process.

Solution overview

The proposed solution uses Amazon Bedrock and Amazon Titan Express models to enable IDP functionality. This architecture seamlessly integrates multiple AWS services with Amazon Bedrock and enables efficient data extraction and comparison.

Amazon Bedrock is a fully managed service that provides access to leading AI startups and Amazon’s high-performance foundational models (FM) through a unified API. A wide range of FMs is available, allowing you to choose the best model for your specific use case.

The following diagram shows the solution architecture.

The solution consists of the following steps:

1. The relevant documents are uploaded and stored in an Amazon Simple Storage Service (Amazon S3) bucket.
2. Event notifications are sent to an Amazon Simple Queue Service (Amazon SQS) queue, which arranges each file for further processing. Amazon SQS acts as a buffer, allowing different components to send and receive messages reliably without being directly tied together, making the system more scalable and fault-tolerant.
3. A text extraction AWS Lambda function is called by the SQS queue to process each queued file and use Amazon Textract to extract the text from the document.
4. The extracted text data is placed into a separate SQS queue for the next processing step.
5. The text summarization Lambda function is called with this new queue containing the extracted text. This function sends a request to SageMaker JumpStart. There, the Meta Llama text generation model is deployed to summarize the content based on the prompts provided.
6. In parallel, the InvokeSageMaker Lambda function is called to perform the comparison and evaluation. Compare the extracted text to the BQA standard on which the model was trained to assess text compliance, quality, and other relevant metrics.
7. Summarized data and evaluation results are stored in Amazon DynamoDB tables
8. Upon request, the InvokeBedrock Lambda function invokes Amazon Bedrock to generate the generated AI summaries and comments. This function builds detailed prompts designed to guide the Amazon Titan Express model in evaluating university submissions.

Rapid engineering with Amazon Bedrock

To harness the power of Amazon Bedrock and ensure that the generated output adheres to your desired structure and formatting requirements, we have developed carefully crafted prompts that follow the following guidelines.

• submission of evidence – Presents evidence submitted by institutions under relevant indicators, providing the model with the necessary context for evaluation
• Evaluation criteria – Outline the specific criteria by which the evidence should be evaluated
• Evaluation procedure – Tell the model:
  • If the evidence is unrelated to the indicator, indicate N/A
  • Evaluate university self-evaluation based on criteria
  • Cite evidence directly from the content and assign a score of 1 to 5 to each comment.
• response format – Bullet-point your answers with a 100-word character limit, focusing on relevant analysis and evidence.

To use this prompt template, create a custom Lambda function in your project. This function should handle retrieving the necessary data such as the indicator name, the evidence submitted by the university, and the rubric criteria. Include a prompt template within your function to dynamically set placeholders. (${indicatorName}, ${JSON.stringify(allContent)}and ${JSON.stringify(c.comment)}) Use the obtained data.

The Amazon Titan Text Express model generates an evaluation response based on the prompt instructions you provide, following the specified format and guidelines. You can process and analyze model responses within your functions and extract compliance scores, related analytics, and evidence.

Below is an example prompt template.

for (const c of comments) {
        const prompt = `
        Below is the evidence submitted by the university under the indicator "${indicatorName}":
        ${JSON.stringify(allContent)}
    
         Analyze and Evaluate the university's eviedence based on the provided rubric criteria:
        ${JSON.stringify(c.comment)}

        - If the evidence does not relate to the indicator, indicate that it is not applicable (N/A) without any additional commentary.
        
       Choose one from the below compliance score based on evidence submitted:
       1. Non-compliant: The comment does not meet the criteria or standards.
        2.Compliant with recommendation: The comment meets the criteria but includes a suggestion or recommendation for improvement.
        3. Compliant: The comment meets the criteria or standards.

        THE END OF THE RESPONSE THERE SHOULD BE EITHER SCORE: (SCORE: COMPLIANT OR NON-COMPLIANT OR COMPLIANT WITH RECOMMENDATION)
        Write your response in concise bullet points, focusing strictly on relevant analysis and evidence.
        **LIMIT YOUR RESPONSE TO 100 WORDS ONLY.**

        `;

        logger.info(`Prompt for comment ${c.commentId}: ${prompt}`);

        const body = JSON.stringify({
          inputText: prompt,
          textGenerationConfig: {
            maxTokenCount: 4096,
            stopSequences: (),
            temperature: 0,
            topP: 0.1,
          },
        });

The following screenshot shows an example of the response that Amazon Bedrock generates.

result

Implementing Amazon Bedrock has provided transformational benefits to institutions. By automating and streamlining the collection and analysis of a wide range of documents, including SERs, supporting evidence, and various media formats, institutions can improve the accuracy and consistency of their reporting processes and prepare them for review processes. can. This not only reduces the time and cost of manual data processing, but also improves compliance with quality expectations, resulting in increased trust and quality for institutions.

For BQA, this implementation will help achieve one of its strategic objectives focused on streamlining its reporting processes, achieving significant improvements across a variety of key metrics and increasing the overall efficiency and effectiveness of its operations. Significantly improved.

Expected key success metrics include:

• 70% less time required to generate accurate, standards-compliant self-assessment reports, leading to increased overall efficiency.
• Reduce the risk of errors and non-compliance in the reporting process and enforce adherence to established guidelines.
• The ability to summarize long submissions into concise bullet points allows BQA reviewers to quickly analyze and understand the most pertinent information, reducing evidence analysis time by 30%.
• More accurate compliance feedback capabilities enable reviewers to effectively assess submissions against established standards and guidelines while achieving 30% operational cost savings through process optimization .
• Increase transparency and communication through seamless interactions, allowing users to easily request additional documentation or clarification.
• Real-time feedback allows institutions to quickly make necessary adjustments. This is particularly helpful in maintaining the accuracy and integrity of your transmissions.
• Enhance decision-making by providing data insights. This helps universities identify areas for improvement and make data-driven decisions to strengthen processes and operations.

The following screenshot shows an example of using Amazon Bedrock to generate a new rating.

conclusion

This post outlined the implementation of Amazon Bedrock at the Bureau for Education and Training Quality (BQA), demonstrating the transformative potential of generative AI to revolutionize quality assurance processes in the education and training sector. If you are interested in delving into the technical details further, the complete code for this implementation is available in the following GitHub repository: If you are interested in running a proof of concept similar to ours, please submit your challenge idea on the Bahrain University of Technology or Bahrain University CIC website.

About the author

Malam Alsagh She is a Cloud Infrastructure Architect at Amazon Web Services (AWS), helping AWS customers accelerate their journey to the cloud. He is currently focused on developing innovative solutions powered by generative AI and machine learning (ML) for the public sector.