API Inference

This module provides a command-line interface for interacting with Llama models through the Llama API.

Overview

The api_inference.py script allows you to:

Connect to Llama's API using your API key
Launch a Gradio web interface for sending prompts to Llama models
Get completions from models like Llama-4-Maverick-17B

Prerequisites

Python 3.8 or higher
A valid Llama API key
Required Python packages:
- gradio
- llama_api_client

Installation

Ensure you have the required packages installed:

pip install gradio llama_api_client

Usage

You can run the script from the command line using:

python api_inference.py [OPTIONS]

Command-line Options

--api-key: Your API key (optional)
- If not provided, the script will look for the appropriate environment variable based on the provider
--provider: API provider to use (optional, default: "Llama")
- Available options: "Llama", "OpenAI"

Setting Up Your API Key

You can provide your API key in one of two ways:

Command-line argument:

python api_inference.py --api-key YOUR_API_KEY --provider Llama

Environment variable: The environment variable name depends on the provider you choose:

# For Llama (default provider)
export LLAMA_API_KEY=YOUR_API_KEY

# For OpenAI
export OPENAI_API_KEY=YOUR_API_KEY

For Windows:

# Command Prompt (example for Llama)
set LLAMA_API_KEY=YOUR_API_KEY

# PowerShell (example for Llama)
$env:LLAMA_API_KEY="YOUR_API_KEY"

Example

Run the script:

# Using Llama (default provider)
python api_inference.py --api-key YOUR_API_KEY

# Using a different provider
python api_inference.py --api-key YOUR_API_KEY --provider OpenAI

The script will launch a Gradio web interface (typically at http://127.0.0.1:7860)
In the interface:
- Enter your prompt in the text box
- The default model is "Llama-4-Maverick-17B-128E-Instruct-FP8" but you can change it
- Click "Submit" to get a response from the model

Troubleshooting

API Key Issues

If you see an error like:

No API key provided and *_API_KEY environment variable not found

Make sure you've either:

Passed the API key using the --api-key argument
Set the appropriate environment variable for your chosen provider (LLAMA_API_KEY)

Advanced Usage

You can modify the script to use different models or customize the Gradio interface as needed.

Implementation Notes

The script uses type hints for better code readability and IDE support:
```
api_key: Optional[str] = args.api_key
```
This line uses the Optional type from the typing module to indicate that api_key can be either a string or None. The Optional type is imported from the typing module at the beginning of the script.

License

[Include license information here]