Track you LLM prompt engineering¶

Developing effective Large Language Model (LLM) applications involves continuously refining the prompts you feed into foundational models and GenAI agents.

CometLLM is your solution for tracking and managing all your LLM prompt engineering workflows.

Refer to Comet Experiment Management for your experiment tracking when training/fine-tuning LLM and GenAI models. Also, explore the Panels gallery for LLMOps-specific Comet Experiment visualizations such as the Prompt History and Prompt Playground panels.

Prompt engineering logging with CometLLM¶

To log your prompt engineering workflows with CometLLM you need to follow a simple four-step process:

1. Install the comet-llm package.
2. Create an LLM Project.
3. Log your prompts and chains, together with relevant metrics and metadata, to the LLM Project.
4. Analyze your LLM Project performance in the Comet UI.

Below, we provide more details and references for each step.

1. Install the comet-llm package¶

To install the comet-llm package, simply run:

pip install comet-llm

You can find more information in the comet-llm reference page.

2. Create an LLM Project¶

Comet differentiates LLM Projects from standard Comet ML Projects because prompt engineering workflows require novel experiment tracking and managing functionalities with respect to classic ML training and tuning workflows as introduced in the Analyze your LLM Project from Comet UI section.

You can create an LLM Project from Comet UI or from Comet LLM SDK.

From Comet UIFrom Comet LLM SDK

1. Navigate to your Comet workspace home page.

2. Click on the New Project button by the top right corner of the screen.

3. Choose Large Language Models as the Project type, and set the other project fields with your desired values.

4. Click Create.

comet-llm is the Python SDK offered by Comet to power your LLM prompt engineering management.

To create an LLM project programmatically, simply initialize the SDK with your desired project name and workspace using comet_llm.init(). For example:

1 2 3

import comet_llm comet_llm.init(project="llm-example")

The specified project is created as soon as the first prompt or chain is logged.

Once the LLM Project has been created, you can access it interactively from the Projects UI page.

3. Log your prompts and chains to the LLM Project¶

The Comet LLM Project lets you keep track and manage the prompts that you send to your third-party or proprietary LLM applications, especially when doing prompt engineering.

CometLLM differentiates between prompts and chains.

• A prompt is a single LLM model call.

  You can log prompts with comet_llm.log_prompt().

• A chain is a sequence of multiple LLM model calls and other pre-/post-processing steps wrapped inside a span.

  You can log chains with comet_llm.start_chain(), comet_llm.Span(), and comet_llm.end_chain().

You can add any custom metadata to the logged prompts and chains to streamline the analysis of your prompt engineering trials.

The Log your prompts and chains page provides you with all the details, instructions, and examples for you to get started with logging your prompt engineering inside a Comet LLM Project.

4. Analyze your LLM Project from Comet UI¶

The Comet UI lets you analyze and manage LLM projects, with all associated logged prompts and chains, within user-friendly interfaces that you can easily share with your team members.

You can take advantage of Comet UI's user-friendly Prompts table view to seamlessly analyze and manage your LLM prompts and chains. If preferred, you can also search and export your LLM metadata programmatically.

Example Comet LLM Project¶

Click on the button below to explore an example project and get started with LLM Projects in the Comet UI!

Additionally, you can review an example notebook with Comet-powered prompt engineering by clicking on the tutorial below.