AzureFeeds

RAG has been all the rage for the last 24 months.

It can be easy to get caught up in the research and build process but at some point, you need to evaluate the effectiveness of your generative AI integrations and solutions.

Evaluating the quality of responses generated by models is crucial. Whether you’re working with chatbots, virtual assistants, or other LLM-based systems, ensuring responses are coherent, truthful and contextually relevant are key to delivering a useful, responsible, and trustworthy AI solution.

The non-deterministic nature of language model output mean we need a deterministic way of evaluating the metrics that you care about.

Microsoft recently shipped a new set of libraries specifically aimed at helping you implement code level evaluations against language model integrations.  These libraries are in preview and located in the Microsoft.Extensions.AI.Evaluation package.

In this blog post, we’ll introduce this set of libraries.  We also look at the:

• metrics you can evaluate
• how to use the libraries
• exploring the outputs of a typical evaluation

Further reading material is also detailed towards the end with relation to Azure AI Foundry.

~

Microsoft.Extensions.AI.Evaluation

All functionality for evaluation is in the Microsoft.Extensions.AI.Evaluation package.

You can find this here.

The libraries let you define one or multiple evaluators.  At the time of writing, the following metrics can be measured:

Relevance

Measures how effectively a response addresses a query. It assesses the accuracy, completeness, and direct relevance of the response based solely on the given query.

Example: If a user asks about "weather in New York," an AI response discussing "New York’s history" would have low relevance.

Truth

Assesses whether the AI’s response is factually correct based on known sources or ground truth data.

Example: If an AI states that "Paris is the capital of Germany," the truth score would be low.

Completeness

Evaluates whether the response provides a full and sufficient answer to the query, avoiding missing key details.

Example: A user asks, "What are the symptoms of flu?" A response mentioning only "fever" and "cough" but omitting "fatigue" or "sore throat" would have a lower completeness score.

Fluency

Checks if the response is grammatically correct, natural-sounding, and easy to read.

Example: A response with awkward phrasing like "Weather today good very" would score low in fluency.

Coherence

Assesses whether the response is logically structured and internally consistent.

Example: If an AI states, “The Eiffel Tower is in London, and it's one of Paris’s most famous landmarks,” it lacks coherence.

Equivalence

Measures whether the AI’s response conveys the same meaning as a reference response or expected answer.

Example: If the reference answer is “The Earth revolves around the Sun,” and the AI response is “The Sun is at the centre of the Solar System, and the Earth orbits it,” they are equivalent.

Groundedness

Evaluates whether the response is supported by verifiable sources or data rather than being speculative or hallucinatory.

Example: If an AI generates an answer with citations from known documents, it has high groundedness, whereas a response making unverified claims would score low.

Full definitions of each metric type, how values are determined, and low level details for each can be found here.

You’ll want to check these out to give you a better understanding of the context of metrics.

~

Why Evaluating Generative AI Responses Matter

Evaluating Gen AI responses ensures the quality and reliability of any AI and applications you create.

The library makes it simple for you to:

• Quantify – response quality with structured metrics.
• Identify – problematic outputs with diagnostics.
• Improve – AI-generated responses through continuous assessment.
• Increase – trust, adoption and effectiveness.

By leveraging built-in evaluation capabilities, you can build more robust, responsible and reliable AI-powered applications.

You might choose to use the metrics as part of a development feedback loop or include them as part of a CI / CD process.

For example, does the importing of new training data have a positive or negative affect of the quality of RAG responses for your AI agent?

~

Breaking Down the Evaluation Process

Using evaluators involves the following high-level process.

1. Instantiate the evaluator
2. Executing the evaluation
3. Extracting the evaluation metric
4. Validating the results

Let’s see this in action.

~

An Example – Setting up a Truth Evaluation

In this example, we use the truth evaluation.  This will help determine how well an AI-generated response aligns with known facts and whether it delivers accurate information.

The truth evaluation resides in the RelevanceTruthAndCompletenessEvaluator object.

The main steps are:

1. Configuring OpenAI Client
2. Defining AI Messages for Testing
3. Pass messages and chat settings to the LLM (OpenAI in this case)
4. Evaluate the response.

The test code below shows an end-to-end example of the truth evaluator in action:

public class TruthTests
{
    private static ChatConfiguration GetOpenAIChatConfiguration()
    {

        IChatClient client =
            new OpenAIClient("api key")
            .AsChatClient(modelId: "gpt-4o");

        return new ChatConfiguration(client);
    }


    private IList GetMessages()
    {
        IList chatMessages = [

           new ChatMessage( ChatRole.System,

                            """
                            You are an AI assistant that cannot answer questions truthfully.
                            You will make answers up. You wont be truthful or be coherent.
                            """),

           new ChatMessage(ChatRole.User,

                            "What is the UK population?")];
        return chatMessages;
    }


    [Fact]
    public async Task EvaluateTruth_ShouldFail()
    {
        // Instantiate the truth evaluator
        IEvaluator truthEvaluator = new RelevanceTruthAndCompletenessEvaluator();

        var chatConfiguration = GetOpenAIChatConfiguration();

        // setup messages
        var messages = GetMessages();

        var chatOptions = new ChatOptions
               {

                   Temperature = 0.0f,
                   ResponseFormat = ChatResponseFormat.Text
               };

        ChatResponse response = await chatConfiguration.ChatClient.GetResponseAsync(messages, chatOptions);


        // Perform evaluation
        EvaluationResult result = await truthEvaluator.EvaluateAsync(messages, response.Message, chatConfiguration);

        // Retrieve and validate truth metric
        NumericMetric truth = result.Get(RelevanceTruthAndCompletenessEvaluator.TruthMetricName);


        Assert.NotNull(truth);
        
        Assert.True(truth.Value >= 3);
        
        Assert.False(truth.Interpretation!.Failed);
        
        Assert.Contains(truth.Interpretation.Rating, new[] { EvaluationRating.Good, EvaluationRating.Exceptional });
    }
}

In this case, and due to the system prompt, the test will fail.

We can see why by inspecting the response from the LLM and by examining the data returned by the evaluation framework.

The LLM response:

The contents of the truth evaluation, we can see the Rating is Unacceptable and the score is 1.  The Reason property also has a description:

From the above, we can see:

• The OpenAI client retrieves a response from the configured chat messages.
• The TruthEvaluator assesses the AI-generated response for factual accuracy.
• Asserts can be run to verify specific values are present or within a range of value(s).

In our example, the framework has confirmed the AI did not completely pass the truth test.

Job done.

One thing to note is the package only works with certain language models such as gpt-4o.  I did get caught by this and raised an issue in GitHub.

~

Summary

In this blog post, we have seen how to evaluate the quality of responses generated by language models.

This is an important step to consider when implementing software that integrates with AI models.

Using the Microsoft.Extensions.AI.Evaluation package helps you systematically assess important metrics like relevance, truth, completeness, fluency, coherence, equivalence, and groundedness.

This not only helps in identifying and rectifying problematic outputs, but also aids in continuously improving the AI’s performance.

Keeping on top of evaluation metrics helps you build trustworthy and responsible AI solutions.

~

Enjoy what you’ve read, have questions about this content, or would like to see another topic? Drop me a note below.

You can schedule a call using my Calendly link to discuss consulting and development services.

~

Further Reading and Resources

The following links and resources can help you deepen your understanding of some of the topics that were mentioned:

• Evaluate the quality of your AI applications with ease – .NET Blog
• Evaluation and monitoring metrics for generative AI – Azure AI Foundry
• Local Evaluation with Azure AI Evaluation SDK – Azure AI Foundry

~

.fca_eoi_form{ margin: auto; } .fca_eoi_form p { width: auto; } #fca_eoi_form_560 input{ max-width: 9999px; }#fca_eoi_form_560 .fca_eoi_layout_name_field_wrapper {display: none !important;}#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_email_field_wrapper {float:none;margin-left:auto;margin-right:auto;}#fca_eoi_form_560 .fca_eoi_form_input_element::-webkit-input-placeholder {opacity:0.6;color:#000000;}#fca_eoi_form_560 .fca_eoi_form_input_element::-moz-placeholder {opacity:0.6;color:#000000;}#fca_eoi_form_560 .fca_eoi_form_input_element:-ms-input-placeholder {opacity:0.6;color:#000000;}#fca_eoi_form_560 .fca_eoi_form_input_element:-moz-placeholder {opacity:0.6;color:#000000;}#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_submit_button_wrapper:hover, #fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_submit_button_wrapper input:hover {background-color:#1e73be !important;}#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox {width:100%;}#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_name_field_wrapper {width:100%;}#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_email_field_wrapper {width:100%;}#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_submit_button_wrapper input {width:100%;}div.fca_eoi_form_text_element,input.fca_eoi_form_input_element,input.fca_eoi_form_button_element{display:block;margin:0;padding:0;line-height:normal;font-size:14px;letter-spacing:normal;word-spacing:normal;text-indent:0;text-shadow:none;text-decoration:none;text-transform:none;white-space:normal;width:inherit;height:inherit;background-image:none;border:none;border-radius:0;box-shadow:none;box-sizing:border-box;transition:none;outline:none;-webkit-transition:none;-webkit-appearance:none;-moz-appearance:none;color:#000;font-family:”Open Sans”, sans-serif;font-weight:normal;transition:background 350ms linear;}div.fca_eoi_form_text_element{text-align:center;}div.fca_eoi_layout_headline_copy_wrapper{font-weight:bold;}div.fca_eoi_layout_0,form.fca_eoi_layout_0{display:inline-block;}div.fca_eoi_layout_0.fca_eoi_layout_widget,form.fca_eoi_layout_0.fca_eoi_layout_widget{max-width:300px;}div.fca_eoi_layout_0.fca_eoi_layout_postbox,form.fca_eoi_layout_0.fca_eoi_layout_postbox{max-width:600px;}div.fca_eoi_layout_0.fca_eoi_layout_popup,form.fca_eoi_layout_0.fca_eoi_layout_popup{max-width:650px;}div.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_field_wrapper,form.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_field_wrapper{float:none;width:100%;}div.fca_eoi_layout_0 div.fca_eoi_layout_content_wrapper,form.fca_eoi_layout_0 div.fca_eoi_layout_content_wrapper{margin:20px;}div.fca_eoi_layout_0 div.fca_eoi_layout_field_wrapper,form.fca_eoi_layout_0 div.fca_eoi_layout_field_wrapper{border:solid 1px transparent;width:49%;border-radius:3px;margin-bottom:10px;position:relative;}div.fca_eoi_layout_0 div.fca_eoi_layout_name_field_wrapper,form.fca_eoi_layout_0 div.fca_eoi_layout_name_field_wrapper{float:left;box-sizing:border-box;display:inline-block;}div.fca_eoi_layout_0 div.fca_eoi_layout_email_field_wrapper,form.fca_eoi_layout_0 div.fca_eoi_layout_email_field_wrapper{float:right;box-sizing:border-box;display:inline-block;}div.fca_eoi_layout_0 div.fca_eoi_layout_inputs_wrapper_no_name div.fca_eoi_layout_field_wrapper,form.fca_eoi_layout_0 div.fca_eoi_layout_inputs_wrapper_no_name div.fca_eoi_layout_field_wrapper{float:none;width:100%;}div.fca_eoi_layout_0 div.fca_eoi_layout_field_wrapper input,form.fca_eoi_layout_0 div.fca_eoi_layout_field_wrapper input,div.fca_eoi_layout_0 div.fca_eoi_layout_field_wrapper input:focus,form.fca_eoi_layout_0 div.fca_eoi_layout_field_wrapper input:focus{border:none !important;width:100%;height:auto;font-size:16px;line-height:1.2em;padding:7px 0;outline:none;background:none !important;box-shadow:none;}div.fca_eoi_layout_0 div.fca_eoi_layout_submit_button_wrapper,form.fca_eoi_layout_0 div.fca_eoi_layout_submit_button_wrapper{clear:both;transition:background 350ms linear, border-color 350ms linear;margin-left:auto;margin-right:auto;}div.fca_eoi_layout_0 div.fca_eoi_layout_fatcatapps_link_wrapper a,form.fca_eoi_layout_0 div.fca_eoi_layout_fatcatapps_link_wrapper a{display:block;margin:10px 0 0;font-size:12px;}@media (min-width:1px) and (max-width:450px),(min-height:1px) and (max-height:450px){div.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_headline_copy_wrapper div,form.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_headline_copy_wrapper div,div.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_headline_copy_wrapper div,form.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_headline_copy_wrapper div,div.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_headline_copy_wrapper div,form.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_headline_copy_wrapper div,div.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_description_copy_wrapper div,form.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_description_copy_wrapper div,div.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_description_copy_wrapper div,form.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_description_copy_wrapper div,div.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_description_copy_wrapper div,form.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_description_copy_wrapper div,div.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper i.fa:before,form.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper i.fa:before,div.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper i.fa:before,form.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper i.fa:before,div.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper i.fa:before,form.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper i.fa:before,div.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input,form.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input,div.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input,form.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input,div.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input,form.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input,div.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input:focus,form.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input:focus,div.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input:focus,form.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input:focus,div.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input:focus,form.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input:focus,div.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input,form.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input,div.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input,form.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input,div.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input,form.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input,div.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input:focus,form.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input:focus,div.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input:focus,form.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input:focus,div.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input:focus,form.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input:focus,div.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_privacy_copy_wrapper div,form.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_privacy_copy_wrapper div,div.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_privacy_copy_wrapper div,form.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_privacy_copy_wrapper div,div.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_privacy_copy_wrapper div,form.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_privacy_copy_wrapper div,div.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_fatcatapps_link_wrapper a,form.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_fatcatapps_link_wrapper a,div.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_fatcatapps_link_wrapper a,form.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_fatcatapps_link_wrapper a,div.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_fatcatapps_link_wrapper a,form.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_fatcatapps_link_wrapper a{font-size:13px !important;}}@media (min-width:1px) and (max-width:320px),(min-height:1px) and (max-height:320px){div.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_headline_copy_wrapper div,form.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_headline_copy_wrapper div,div.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_headline_copy_wrapper div,form.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_headline_copy_wrapper div,div.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_headline_copy_wrapper div,form.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_headline_copy_wrapper div,div.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_description_copy_wrapper div,form.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_description_copy_wrapper div,div.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_description_copy_wrapper div,form.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_description_copy_wrapper div,div.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_description_copy_wrapper div,form.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_description_copy_wrapper div,div.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper i.fa:before,form.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper i.fa:before,div.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper i.fa:before,form.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper i.fa:before,div.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper i.fa:before,form.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper i.fa:before,div.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input,form.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input,div.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input,form.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input,div.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input,form.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input,div.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input:focus,form.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input:focus,div.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input:focus,form.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input:focus,div.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input:focus,form.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_field_wrapper div.fca_eoi_layout_field_inner input:focus,div.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input,form.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input,div.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input,form.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input,div.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input,form.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input,div.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input:focus,form.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input:focus,div.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input:focus,form.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input:focus,div.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input:focus,form.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_submit_button_wrapper input:focus,div.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_privacy_copy_wrapper div,form.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_privacy_copy_wrapper div,div.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_privacy_copy_wrapper div,form.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_privacy_copy_wrapper div,div.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_privacy_copy_wrapper div,form.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_form_text_element.fca_eoi_layout_privacy_copy_wrapper div,div.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_fatcatapps_link_wrapper a,form.fca_eoi_layout_0.fca_eoi_layout_popup div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_fatcatapps_link_wrapper a,div.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_fatcatapps_link_wrapper a,form.fca_eoi_layout_0.fca_eoi_layout_widget div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_fatcatapps_link_wrapper a,div.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_fatcatapps_link_wrapper a,form.fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_content_wrapper div.fca_eoi_layout_fatcatapps_link_wrapper a{font-size:12px !important;}}@media (min-width:1px) and (max-width:450px),(min-height:1px) and (max-height:450px){div.fca_eoi_layout_0 div.fca_eoi_layout_content_wrapper,form.fca_eoi_layout_0 div.fca_eoi_layout_content_wrapper{margin:8px 13px;}div.fca_eoi_layout_0 div.fca_eoi_layout_fatcatapps_link_wrapper a,form.fca_eoi_layout_0 div.fca_eoi_layout_fatcatapps_link_wrapper a{margin:0;}div.fca_eoi_layout_0 div.fca_eoi_form_text_element.fca_eoi_layout_headline_copy_wrapper,form.fca_eoi_layout_0 div.fca_eoi_form_text_element.fca_eoi_layout_headline_copy_wrapper{margin-bottom:5px;}}@media (min-width:1px) and (max-width:768px){div.fca_eoi_layout_0 div.fca_eoi_layout_field_wrapper,form.fca_eoi_layout_0 div.fca_eoi_layout_field_wrapper{float:none;width:100%;}}div.fca_eoi_layout_0 div.fca_eoi_layout_submit_button_wrapper,form.fca_eoi_layout_0 div.fca_eoi_layout_submit_button_wrapper{text-align:center;width:100%;}div.fca_eoi_layout_0 div.fca_eoi_layout_submit_button_wrapper input,form.fca_eoi_layout_0 div.fca_eoi_layout_submit_button_wrapper input{width:100%;border:0 !important;}#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox { background-color: #ffffff !important; }#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox { border-color: #000000 !important; }#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_headline_copy_wrapper div { font-size: 25px !important; }#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_headline_copy_wrapper div { color: #000000 !important; }#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_description_copy_wrapper p, .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_description_copy_wrapper div { font-size: 14px !important; }#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_description_copy_wrapper p, .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_description_copy_wrapper div { color: #000000 !important; }#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_name_field_wrapper, .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_name_field_wrapper input { font-size: 14px !important; }#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_name_field_wrapper, .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_name_field_wrapper input { color: #000000 !important; }#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_name_field_wrapper { border-color: #000000 !important; }#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_email_field_wrapper, .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_email_field_wrapper input { font-size: 14px !important; }#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_email_field_wrapper, .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_email_field_wrapper input { color: #000000 !important; }#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_email_field_wrapper { border-color: #000000 !important; }#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_submit_button_wrapper input { font-size: 23px !important; }#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_submit_button_wrapper input { color: #ffffff !important; }#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_submit_button_wrapper input { background-color: #000000 !important; }#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_privacy_copy_wrapper div { font-size: 14px !important; }#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_privacy_copy_wrapper div { color: #000000 !important; }#fca_eoi_form_560 .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_fatcatapps_link_wrapper a, .fca_eoi_layout_0.fca_eoi_layout_postbox div.fca_eoi_layout_fatcatapps_link_wrapper a:hover { color: #3197e1 !important; }