mod_openai File Documentation 🔗 ↑ TOC

This document provides a comprehensive overview of all C and C++ source files in the mod_openai FreeSWITCH module, explaining their purpose, functionality, and key contributions to the system.

Overview 🔗 ↑ TOC

The mod_openai module is a FreeSWITCH module that enables AI-powered voice conversations, real-time transcription, translation, and text-to-speech capabilities. The module integrates with OpenAI, Azure, and other AI services to provide intelligent voice interaction features.

Core Module Files 🔗 ↑ TOC

mod_openai.c 🔗 ↑ TOC

Lines of Code: 4,623
Purpose: Main FreeSWITCH module implementation and entry point

Key Functionality:

• FreeSWITCH module lifecycle management (load, shutdown, runtime)
• Application interface (openai) for voice AI conversations
• API interface for external control and status queries
• SWML (SignalWire Markup Language) execution and processing
• Action processing for AI function calls
• Session management and call flow control
• Configuration loading and global settings management

Key Functions:

• openai_execute() - Main application execution
• process_action() - Handles AI function calls and SWML execution
• load_globals() - Loads module configuration
• check_commands() - Processes queued commands
• Various cache management functions

ai_utils.c 🔗 ↑ TOC

Lines of Code: 9,690
Purpose: Core AI session utilities and conversation management

Key Functionality:

• AI session lifecycle (create, destroy, manage)
• Conversation array management with token tracking
• Speech detection and audio processing threads
• Text-to-Speech (TTS) integration and management
• Video processing and vision model support
• Real-time streaming response handling
• Billing and usage tracking
• Input/output queue management
• Interruption and barge-in handling

Key Functions:

• ai_session_create() / ai_session_destroy() - Session management
• ai_conversation_*() functions - Conversation handling
• ai_send_text() - Send text to AI models
• launch_threads() / end_threads() - Thread management
• wait_for_speaking() - Speech synchronization
• Token counting and conversation consolidation functions

Communication and Protocol Files 🔗 ↑ TOC

webhook.c 🔗 ↑ TOC

Lines of Code: 1,145
Purpose: HTTP webhook handling and external communication

Key Functionality:

• Debug data posting to external webhooks
• Asynchronous webhook processing with threading
• HTTP client functionality using cURL
• Stream response handling for real-time data
• Error handling and retry logic
• Event firing for relay integration

Key Functions:

• send_debug_data() - Synchronous webhook posting
• send_debug_data_async() - Asynchronous webhook posting
• openai_post_data() - Generic HTTP POST functionality
• fire_relay_event() - Event system integration
• Various cURL callback functions

swaig.c 🔗 ↑ TOC

Lines of Code: 2,767
Purpose: SWAIG (SignalWire AI Gateway) protocol implementation

Key Functionality:

• Inline command parsing (~FN:, ~LN:, ~EM:, ~VL: patterns)
• JSON variable expansion and substitution
• Function call extraction and processing
• Regular expression utilities
• Text formatting and processing
• Emoji detection and removal
• Markdown rendering from JSON
• Phone number formatting integration

Key Functions:

• parse_swaig() - Parse SWAIG inline commands
• expand_jsonvars() - JSON variable expansion
• extract_json() - JSON extraction from text
• remove_emoji() - Emoji processing
• strip_markdown() - Markdown cleaning
• regex_replace_function() - Regex utilities

Media Processing Files 🔗 ↑ TOC

translate_transcribe.c 🔗 ↑ TOC

Lines of Code: 2,996
Purpose: Real-time translation and transcription services

Key Functionality:

• Live speech transcription using Deepgram or Google Speech
• Real-time language translation
• Conversation management for transcription sessions
• Media bug integration for audio capture
• Multi-language support and detection
• AI-powered summarization
• Frame processing and audio handling

Key Functions:

• openai_transcribe_init() - Initialize transcription
• openai_translate_speech_init() - Initialize translation
• live_conversation_*() functions - Conversation management
• transcribe_callback() / translate_callback() - Media processing
• process_frame() - Audio frame processing

tts.c 🔗 ↑ TOC

Lines of Code: 2,280
Purpose: Text-to-Speech engine implementation

Key Functionality:

• REST-based TTS engine interface
• Voice mapping for ElevenLabs and PlayHT services
• Audio file caching and streaming
• Multi-threaded TTS processing
• Volume and emotion parameter handling
• File download and caching mechanisms
• Real-time audio generation

Key Functions:

• tts_rest_speech_open() / tts_rest_speech_close() - TTS session management
• tts_rest_speech_feed_tts() - Feed text for synthesis
• tts_rest_speech_read_tts() - Read synthesized audio
• eleven_labs_voice() / playht_voice() - Voice mapping
• Various caching and file management functions

Utility Files 🔗 ↑ TOC

time.c 🔗 ↑ TOC

Lines of Code: 475
Purpose: Safe time parsing utilities

Key Functionality:

• Parse time strings in multiple formats (1h, 30m, 1:30, etc.)
• Input validation and sanitization
• Overflow protection and bounds checking
• Error reporting with detailed error codes
• Memory-safe string processing

Key Functions:

• parse_time() - Main time parsing function
• get_error_message() - Error code to message conversion
• Various safe parsing helper functions

wallet.c 🔗 ↑ TOC

Lines of Code: 213
Purpose: Billing and usage tracking

Key Functionality:

• Wallet tick generation for billing
• TTS usage tracking and billing
• Resource billing integration with SignalWire
• Usage counting and reporting
• Billing pause/resume functionality

Key Functions:

• start_wallet_ticks() / stop_wallet_ticks() - Billing control
• tts_tick() - TTS usage billing
• check_billing() - Usage validation
• get_wallet_tick_json_args() - Billing data preparation

test.c 🔗 ↑ TOC

Lines of Code: 838
Purpose: Test utilities and validation

Key Functionality:

• Unit tests for time parsing functionality
• JSON variable expansion testing
• Number formatting validation
• Token counting verification
• Integration testing examples

Key Functions:

• test_time() - Time parsing tests
• main() - Test execution and examples
• Various validation and testing utilities

Header Files 🔗 ↑ TOC

mod_openai.h 🔗 ↑ TOC

Lines of Code: 993
Purpose: Main header with structure definitions and prototypes

Key Definitions:

• ai_session_t - Core AI session structure
• openai_application_t - Application configuration structure
• ai_conversation_t - Conversation management structure
• ai_settings_t - AI model settings structure
• Extensive function prototypes and type definitions
• Configuration constants and defaults

swaig.h 🔗 ↑ TOC

Lines of Code: 137
Purpose: SWAIG utilities and Unicode integration

Key Definitions:

• Unicode/ICU integration headers
• Regular expression constants
• Time parsing error codes and types
• Function prototypes for SWAIG utilities
• Text processing constants

C++ Integration Files 🔗 ↑ TOC

tokens.cpp 🔗 ↑ TOC

Lines of Code: 214
Purpose: Multi-language token counting using SentencePiece

Key Functionality:

• SentencePiece processor management for multiple languages
• Token counting for various language models
• Word-based fallback counting
• Language model loading and caching
• Integration with C codebase

Key Functions:

• count_tokens_string() - Count tokens in text
• count_tokens_file() - Count tokens in files
• tokens_init() / tokens_uninit() - Initialization
• Language processor management functions

utf8.cpp 🔗 ↑ TOC

Lines of Code: 486
Purpose: High-performance UTF-8 validation (Google's algorithm)

Key Functionality:

• Structural UTF-8 validation
• SSE4.1 optimized validation for x86 platforms
• ASCII fast-path optimization
• Codepoint boundary detection
• Invalid sequence handling

Key Functions:

• IsStructurallyValid() - UTF-8 validation
• SpanStructurallyValid() - Find valid UTF-8 prefix
• ValidUTF8() - Core validation algorithm
• Various SIMD-optimized helper functions

numbers.cpp 🔗 ↑ TOC

Lines of Code: 58
Purpose: Phone number formatting using Google libphonenumber

Key Functionality:

• Phone number parsing and validation
• Multiple format support (national, international, E164, RFC3966)
• Country code handling
• Invalid number detection
• Integration with text processing pipeline

Key Functions:

• format_number() - Main number formatting function
• Google libphonenumber integration

File Dependencies and Relationships 🔗 ↑ TOC

Core Dependencies: 🔗 ↑ TOC

• mod_openai.c → Uses ai_utils.c, webhook.c, swaig.c
• ai_utils.c → Uses swaig.c, webhook.c, tts.c, wallet.c
• swaig.c → Uses utf8.cpp, numbers.cpp, tokens.cpp
• translate_transcribe.c → Uses ai_utils.c, webhook.c

Header Dependencies: 🔗 ↑ TOC

• mod_openai.h → Included by most C files
• swaig.h → Included by swaig.c and utility files

External Dependencies: 🔗 ↑ TOC

• FreeSWITCH core libraries
• cJSON for JSON processing
• cURL for HTTP communications
• PCRE for regular expressions
• ICU for Unicode handling
• SentencePiece for tokenization
• Google libphonenumber for number formatting

Summary 🔗 ↑ TOC

The mod_openai codebase is a comprehensive implementation of AI-powered voice communication features for FreeSWITCH. The files are well-organized with clear separation of concerns:

• Core module files handle FreeSWITCH integration and session management
• Communication files manage external API calls and webhook processing
• Media processing files handle real-time audio/video and AI model integration
• Utility files provide supporting functionality like time parsing and billing
• C++ files integrate external libraries for specialized processing
• Header files provide structure definitions and API contracts

The architecture supports real-time AI conversations, multi-language processing, advanced text-to-speech synthesis, and comprehensive billing integration, making it a full-featured AI communication platform.