One of the most time consuming activities for many organizations with business customers is responding to inquiries about security practices. Most customers have a questionnaire they want completed to help them evaluated how well the company meets their requirements. Completing the questionnaire usually requires an analyst to review each question and provide a response.

In order to provide an accurate response, the analyst needs to be someone who understands the company’s security policies and standards as well as the company’s products and services.

An LLM seems like an ideal tool to automate the generation of responses. I created a proof of concept to have an LLM respond to questions based on a set of policies.

Initially I implemented a RAG approach, which worked well for my test policy documents. When I tested a large, single policy document, it exceeded the context window for the model and the responses would get stuck on the table of contents.

A commercial provider had a similar issue, as shown here:

After some experimentation, I designed an agentic approach that leverages the excellent searching capabilities of grep to find the relevant sections within the appropriate policies. This is then fed to the LLM to generate the response.

More details to come….