Clever AI Technical Evaluation Report

Executive Summary

This report evaluates two flagship AI-powered products under the Clever AI umbrella, both of which are redefining merchant intelligence and driving measurable business impact:

Clever Forecaster: Provides sales predictions and cashback impact analysis, enabling merchants to make data-driven decisions.
Clever Actionable Insights: Combines data analysis capabilities with Generative AI and Agentic AI to deliver instant, context-aware business recommendations.

By leveraging state-of-the-art AI techniques—including Deep Learning, Generative AI, Retrieval-Augmented Generation (RAG), and Agentic AI—we have:

Validated the near-linear correlation between cashback and sales through data analysis;
Achieved 100% reporting accuracy for real-time business performance metrics;
Achieved 70-80% forecast confidence level in sales forecast;
Achieved 78% usefulness rate for Clever AI actionable insights;
Achieved sub-second AI response times, enabling seamless real-time business intelligence.

These results are reported as at 17/02/2025.

The opportunity now is to scale adoption, fine-tune AI-driven recommendations, and expand AI capabilities into broader merchant ecosystems.

Clever Forecaster

Clever Forecaster is designed to help businesses make reliable sales predictions while evaluating potential impacts of cashback incentives. This feature uses a distilled time series model for sales forecasting, functioning as a specialised AI trained on economic and business data. The model analyses:

Transactional data: Daily transaction volumes and payment amounts.
External factors: Consumer confidence, economic indicators, and competitor trends.
Market context: Industry patterns, seasonality, and special events.

Clever Forecaster applies deep learning techniques, using a variant Transformer-based architecture adapted for multi-horizon forecasting challenges. It inherits several technical components of Temporal Fusion Transformer.

Through regular monitoring and adjustment, the system updates its parameters based on a self-corrective mechanism. This helps maintain forecast reliability over time, allowing the system to adapt to changing market conditions.

Evaluation Metrics

To assess the effectiveness of our forecasting model, we used Symmetric Mean Absolute Percentage Error (SMAPE), a widely recognised metric in time series forecasting. SMAPE is computed as:

To enhance interpretability, we converted SMAPE scores into an intuitive proxy metrics called Confidence:

Model Training Process

Our initial evaluations indicated an error range of 20-30% on a rolling window basis, translating to an confidence range of 70-80% across all merchants. This suggests that our forecasting system provides reasonably reliable predictions with strong generalisation.

To enhance forecasting performance, we applied a structured fine-tuning approach focused on optimizing the final layers of the neural network. Our methodology employs a depth-based strategy that fine-tunes the last five layers while keeping earlier layers frozen, balancing adaptability with stability. The process includes step optimisation with base steps ranging from 5 to 40, guided by an early stopping mechanism to prevent overfitting.

To estimate the impact of cashback on sales, we employed a non-linear regression approach using a comprehensive dataset spanning nearly 1,000 transaction dates and over 100 participating merchants. The analysis covered cashback rates ranging from 0% to 25%, providing a robust foundation for understanding the incentive-response relationship.

Our analysis reveals a clear relationship between cashback incentives and sales performance. The data suggests that sales multipliers demonstrate a positive correlation with cashback rates, with the impact varying across different incentive levels. Specifically:

Initial Growth (Cashback 1-20%): A relatively linear increase in sales multipliers, where each percentage point of cashback generates consistent incremental sales growth, with multipliers scaling to 1.2x – 1.45x.
Stabilisation (Cashback >20%): The sales multiplier trend stabilises at higher cashback levels, suggesting a mature phase of consumer engagement where sales continue to perform strongly at approximately 1.5x baseline levels.

While the data demonstrates the effectiveness of cashback programs as a sales driver, it’s important to note that the observed relationship represents an average effect across diverse merchants and time periods. Individual results may vary based on factors such as industry dynamics, seasonality, and competitive intensity.

Expert-in-the-Loop

While Clever Forecaster can autonomously trigger its own parameters to retrain, our team of account managers, fintech experts, and advisors also continously provides feedback to ensure alignment with real-world conditions.

This collaborative feedback loop allows us to:

Enhance Confidence Through Expert Validation: Account team’ interventions have reduced forecast deviation by 20-25% in scenarios involving seasonal promotions and special events, particularly for merchants with complex promotional calendars.
Incorporate Market Context: When merchants signal significant business changes (e.g., expansion plans, new product lines), expert adjustments have improved forecast accuracy by up to 15% compared to pure algorithmic predictions.
Refine Edge Cases: For merchants with unique business models or irregular patterns, human oversight has helped reduce extreme prediction errors (>50% deviation) by approximately 30%.

By incorporating expert oversight into the forecasting process, we ensure our predictions remain both data-driven and practically applicable, improving their utility for business strategy development while maintaining algorithmic rigor.

Clever Actionable Insights

Clever Actionable Insights is an AI-powered agentic system integrated directly into the merchant dashboard, designed to provide real-time business insights and actionable recommendations.

The recommendation engine consists of two primary components:

RAG-based Data Intelligence Agent: A natural language processing system achieving 100% accuracy in SQL query translation to retrieve real-time data. This agent leverages Retrieval-Augmented Generation (RAG) to enhance its analytical capabilities by combining structured database queries with external contextual knowledge. This agent enables:

Live transaction data streaming for up-to-the-minute insights;
Continuous context updates for merchant activities
Dynamic adaptation to changing business patterns
Instant anomaly and opportunity detection

GenAI-based Business Intelligence Agent: A finetuned analysis system, trained on responses from Hello Clever payment experts, that:

Generates actionable insights from your data in real-time;
Identifies business optimisation opportunities;
Recommends tailored actions to increase sales.

Evaluation

The Business Intelligence Agent undergoes a rigorous, multi-layered evaluation to ensure that its recommendations are relevant, actionable, and aligned with merchant needs. Our approach integrates LLM-as-a-judge historical benchmarking and live merchant feedback to iteratively enhance recommendation quality.

Historical Benchmarking (01/2021- 02/2025)

To assess the reliability and effectiveness of AI-generated insights, we designed a Predictive Validation Framework that evaluates recommendations based on historical transaction data. This framework follows a structured Before-After Validation methodology:

Step 1: Dataset Segmentation
- We selected 100+ merchants operating on the Hello Clever platform between 2021-2025.
- Their transactional, cashback, and conversion data were divided into two periods:
  - Before Period (T₁) – Capturing a defined pre-insight phase of 3-6 months before AI recommendations.
  - After Period (T₂) – Capturing merchant performance in the months following the recommendation window.
Step 2: AI-Generated Insights for T₁ (Simulated Predictions)
- The Business Intelligence Agent was fed transaction and sales patterns from T₁ (without knowledge of T₂).
- It generated retrospective insights and actionable recommendations, mimicking a real-time merchant experience.
Step 3: Relevance Validation Against T₂
- Each AI-generated recommendation from T₁ was tested against real merchant behaviour in T₂. Using LLM-as-a-judge technique, we assigned scores (0-100) to measure the effectiveness of insights across key dimensions:
  - Usefulness (Merchant & Platform Alignment): 78/100
  - Coherence (Communication Style): 75/100
  - Comprehensibility (Language Adaptation): 72/100
  - Actionability (Conversion Focus): 68/100

This Predictive Validation Framework allowed us to test whether AI-generated insights could realistically guide merchants toward better business decisions.

Live Merchant Evaluation

To further refine AI-generated recommendations, we are implementing a live feedback mechanism directly within the merchant dashboard. This ensures that insights evolve based on real-world feedback.

Live Feedback Collection – Merchants can rate insights based on relevance, clarity, and effectiveness.
Qualitative Feedback Loop – Merchants can flag incorrect or redundant suggestions in real time.
Impact Tracking – We analyse whether insights lead to actual improvements in sales and conversion rates.

Our AI dynamically evolves through feedback-informed learning:

Automated A/B Testing with live merchants measures the conversion impact of AI-driven recommendations.
Expert-in-the-loop Refinement ensures recommendations align with real-world trends.
Self-Optimising AI Agents adjust their strategy based on merchant engagement patterns.

By combining historical validation, AI-driven assessment, and live merchant feedback, Clever Actionable Insights continuously improves, ensuring that AI-generated recommendations are both data-driven and merchant-centric.

Our AI principles

At Hello Clever, responsible AI development is at the core of our innovation, ensuring trust, fairness, and security in every merchant interaction. Our commitment is built on three key pillars:

Data Privacy and Security: We implement end-to-end encryption, strict access controls, and detailed audit trails to safeguard merchant data. Regular security assessments, penetration testing, and full compliance with global data protection regulations reinforce our commitment to privacy.
Fair and Unbiased System Design: Our AI treats all merchants equitably, ensuring consistent quality across business sizes and industries. We leverage region-aware recommendations, transparent explanations, and regular bias testing to maintain fairness, accuracy, and accountability.
Cultural and Regional Sensitivity: Our AI is designed to adapt to local business environments, offering multi-language support, recognition of regional holidays, and market-specific insights. By tailoring recommendations to diverse merchant needs, we ensure AI-driven decisions are contextually relevant and globally inclusive.

With these principles, Clever AI not only enhances business intelligence but does so in a way that is secure, ethical, and truly merchant-first.