Large language model (LLM) observability

Site24x7 currently supports OpenAI. OpenAI observability gives full visibility into your OpenAI implementation.

Gain valuable insights into your OpenAI usage, perform model-wise cost analysis, log prompts and responses, identify errors, and more—all to track the performance and utilization of your OpenAI implementation in real time.

Use case

With the Site24x7 OpenAI observability feature, you can monitor all external calls made from your application to OpenAI, providing full visibility into your application's interaction with OpenAI's services.

For example, consider a customer support chatbot that uses OpenAI to generate responses to user inquiries. When a user asks a question, the chatbot makes a call to OpenAI to retrieve a suitable response. The Site24x7 OpenAI monitoring module captures these requests, tracking metrics such as response times, success rates, and any errors. Additionally, it allows you to gain valuable insights into your OpenAI token usage, perform model-wise cost analysis, log prompts and responses.

This comprehensive monitoring ensures that you can track the performance and utilization of your OpenAI implementation in real time, optimize costs, and maintain a seamless user experience, thereby enhancing customer satisfaction and operational efficiency.

Instructions to add OpenAI observability

To install the OpenAI module, follow the instructions given below:
Node.js | Python

Node.js

1. Open your Node.js application.
2. Access the node package manager (NPM).
3. Install the site24x7 openaimodule using the following command:

   npm install @site24x7/openai-observability --save

4. Add the following statement to the first line of the application's source code, before any other import or require statement.
   • If you are using ES or Typescript:

     import * as site24x7-openai from '@site24x7/openai-observability'

   • If you are using Common JS:

     require('@site24x7/openai-observability')

5. Copy the license key from the Site24x7 portal.
   
   
6. Set the license key as an environment variable as shown below:

   SITE24X7_LICENSE_KEY=<YOUR LICENSE KEY HERE>

7. Restart the application.

Set configuration values as environment variables:

You can set configuration values like sampling factor and capture OpenAI text as environment variables using the following keys:

• SITE24X7_SAMPLING_FACTOR
• SITE24X7_CAPTURE_OPENAI_TEXT

Python

1. Install the site24x7-openaimodule using the following command:

   pip install site24x7-openai-observability --upgrade

   Note

   If you are using a virtual environment, activate it and perform the above command to install the site24x7-openai-observability package into your virtual environment.

2. Add the below code to your application’s start file to integrate the observability agent:

   import site24x7_openai_observability

3. Copy the license key from the Site24x7 portal.
   
   
4. Set the license key as an environment variable as shown below:

   SITE24X7_LICENSE_KEY=<YOUR LICENSE KEY HERE>

5. Restart the application.

Additional configurations:

You can configure the sample factor and capture OpenAI text as environment variables using the following keys:

Performance metrics

To begin:

1. Log in to the Site24x7 web client.
2. Navigate to APM > OpenAI.

The top band displays the average response time, request count, average tokens, and cost.

Average Response Time

This graph depicts the average response time of the requests over the selected time period. The Min is the minimum response time, and the Max is the maximum response time.

HTTP Response Codes

This graph provides a summary of the various HTTP responses that were received. You can hover over each peak to get the count.

Model Usage

Model Usage widget shows the percentage of requests sent from each Model as a separate block. The blocks are color-coded for easy identification. Hover over each block to see the total number of requests.

Cost Per Model

This table gives an overall summary of all the models and their costs.

Model-wise Throughput

Gives visibility into the total number of requests sent from each model over the specified time period. Hovering over the peaks will give you the precise request count for that model.

Errors

Gives an overall summary on the Response Code, Error Message, and Count for the errors that occurred.