def advanced_ensemble_prediction(predictions, features, weights=None):
    # Adaptive weighting based on prediction confidence (e.g., VADER score magnitude) and feature heuristics (e.g., word_count, caps_ratio).
    # Outlier detection and filtering using statistical methods (median, standard deviation of predictions).
    # Feature-based confidence adjustment to boost/penalize certain model predictions.
    # Median smoothing for high-variance predictions to stabilize output.

Core data science libraries loaded successfully
Visualization libraries (matplotlib, seaborn, plotly) loaded successfully
Text processing libraries loaded successfully

/Users/777david/Library/Python/3.9/lib/python/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(

Transformer libraries (transformers, torch) loaded successfully. Using device: cpu
Progress tracking (tqdm) loaded successfully
IPython.display loaded successfully
WordCloud library loaded successfully (though ABSA is now primary for text insights)
Requests library loaded successfully

Starting Sentiment Analysis Pipeline with Transformer and ABSA capabilities...
============================================================

Initializing NLTK resources...
Checking required NLTK resources (will download if not present):
  Resource 'VADER sentiment analyzer' (vader_lexicon) is available.
  Resource 'Punkt tokenizer' (punkt) is available.
  Resource 'POS tagger' (averaged_perceptron_tagger) is available.
  Resource 'WordNet corpus' (wordnet) is available.
  Resource 'Stopwords corpus' (stopwords) is available.
  Resource 'Open Multilingual Wordnet 1.4' (omw-1.4) not found. Attempting download...
  Downloading 'Open Multilingual Wordnet 1.4' (omw-1.4)...
  Successfully downloaded 'Open Multilingual Wordnet 1.4' (omw-1.4).
NLTK initialization completed successfully!
Loaded 36 positive words, 31 negative words, 159 domain words, 93 spec words.

Sentiment Analysis Pipeline (Transformer & ABSA Enhanced) - Script Loaded & Ready
======================================================================
This version includes:
 - Transformer Models (e.g., RoBERTa) for overall sentiment.
 - Fine-grained Aspect-Based Sentiment Analysis (ABSA) with rule-based fallback.
 - Worst Prediction Analysis with full review text displayed in a Pandas table.
 - Additional visualizations for rule-based features (distributions, heatmap).
 - Alternative plot for 'Rating Distribution in Top Categories'.
 - Display of Top 50 model predictions with full text in Jupyter output.
 - Display of Top 50 rule-based features with full text preview in Jupyter output.
 - Option to use the full dataset (sample_size=None).
 - Preserved rule-based models and evaluation features.
 - Cross-Validation and Clustering analyses.
 - Interactive Model Comparison Dashboard (Plotly HTML).
 - Comprehensive Markdown Reporting.
======================================================================

To execute in Jupyter:
1. Run CELL 1 (this cell).
2. Sequentially execute CELLs 2-10.
3. Or call `run_pipeline_demo()` for automated run.

If running as standalone script: Uncomment `run_pipeline_demo()` at script end.
Initializing Sentiment Analysis Pipeline with Transformer and ABSA capabilities...
2025-06-20 23:33:31 - INFO - setup_logging - ============================================================
Sentiment Analysis Pipeline Logger Initialized
Results directory: /Users/777david/sentiment_analysis/sentiment_results_transformer_absa_20250620_233331
Log file: /Users/777david/sentiment_analysis/sentiment_results_transformer_absa_20250620_233331/sentiment_analysis_20250620_233331.log
============================================================
2025-06-20 23:33:31 - INFO - __init__ - SentimentAnalysisPipeline instance created.
2025-06-20 23:33:31 - INFO - _initialize_resources - Initializing NLTK-dependent resources.
2025-06-20 23:33:31 - INFO - _initialize_resources - NLTK VADER & stopwords loaded.

[nltk_data] Downloading package omw-1.4 to
[nltk_data]     /Users/777david/nltk_data...
[nltk_data]   Package omw-1.4 is already up-to-date!

    NLTK VADER & stopwords loaded.

Some weights of the model checkpoint at cardiffnlp/twitter-roberta-base-sentiment-latest were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.bias', 'roberta.pooler.dense.weight']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Device set to use cpu

2025-06-20 23:33:32 - INFO - __init__ - Transformer sentiment model 'cardiffnlp/twitter-roberta-base-sentiment-latest' loaded on cpu.
2025-06-20 23:33:32 - INFO - __init__ - NLTK sent_tokenize loaded for ABSA.
2025-06-20 23:33:32 - ERROR - __init__ - Failed to load ABSA Transformer model: Converting from SentencePiece and Tiktoken failed, if a converter for SentencePiece is available, provide a model path with a SentencePiece tokenizer.model file.Currently available slow->fast converters: ['AlbertTokenizer', 'BartTokenizer', 'BarthezTokenizer', 'BertTokenizer', 'BigBirdTokenizer', 'BlenderbotTokenizer', 'CamembertTokenizer', 'CLIPTokenizer', 'CodeGenTokenizer', 'ConvBertTokenizer', 'DebertaTokenizer', 'DebertaV2Tokenizer', 'DistilBertTokenizer', 'DPRReaderTokenizer', 'DPRQuestionEncoderTokenizer', 'DPRContextEncoderTokenizer', 'ElectraTokenizer', 'FNetTokenizer', 'FunnelTokenizer', 'GPT2Tokenizer', 'HerbertTokenizer', 'LayoutLMTokenizer', 'LayoutLMv2Tokenizer', 'LayoutLMv3Tokenizer', 'LayoutXLMTokenizer', 'LongformerTokenizer', 'LEDTokenizer', 'LxmertTokenizer', 'MarkupLMTokenizer', 'MBartTokenizer', 'MBart50Tokenizer', 'MPNetTokenizer', 'MobileBertTokenizer', 'MvpTokenizer', 'NllbTokenizer', 'OpenAIGPTTokenizer', 'PegasusTokenizer', 'Qwen2Tokenizer', 'RealmTokenizer', 'ReformerTokenizer', 'RemBertTokenizer', 'RetriBertTokenizer', 'RobertaTokenizer', 'RoFormerTokenizer', 'SeamlessM4TTokenizer', 'SqueezeBertTokenizer', 'T5Tokenizer', 'UdopTokenizer', 'WhisperTokenizer', 'XLMRobertaTokenizer', 'XLNetTokenizer', 'SplinterTokenizer', 'XGLMTokenizer', 'LlamaTokenizer', 'CodeLlamaTokenizer', 'GemmaTokenizer', 'Phi3Tokenizer']. Will attempt rule-based ABSA fallback.
2025-06-20 23:33:32 - INFO - __init__ - Rule-based ABSA (with TextBlob) will be used as a fallback if needed.
2025-06-20 23:33:32 - INFO - __init__ - Pipeline initialized. Results directory: /Users/777david/sentiment_analysis/sentiment_results_transformer_absa_20250620_233331
Pipeline initialized. Results stored in: /Users/777david/sentiment_analysis/sentiment_results_transformer_absa_20250620_233331

Pipeline instance ('pipeline') created. Results in: /Users/777david/sentiment_analysis/sentiment_results_transformer_absa_20250620_233331
Notebook Cell 3: Using data_source = 'sentiment_data.csv'
Notebook Cell 3: Using sample_size = Full Dataset

2025-06-20 23:33:32 - INFO - load_data - Loading data: sentiment_data.csv, sample: Full Dataset
Loading and cleaning data from: sentiment_data.csv...
  Initial raw shape: (100000, 2)
  Retained 99934 rows after cleaning (dropped 66).
  Using full dataset (99934 reviews).
  Final data shape: (99934, 2)
  Detecting product categories...

Detecting categories:   0%|          | 0/99934 [00:00<?, ?review/s]

  Category distribution:
    Electronics: 53,632 (53.7%)
    Unknown: 22,465 (22.5%)
    Clothing: 16,151 (16.2%)
    Books: 3,217 (3.2%)
    Home: 2,788 (2.8%)
    Beauty: 1,525 (1.5%)
    Food: 156 (0.2%)
Data loading and cleaning complete.
2025-06-20 23:33:35 - INFO - load_data - Loaded 99934 reviews. Ratings: Counter({5: 59761, 4: 20429, 3: 8471, 1: 6387, 2: 4886}). Categories: [('electronics', 53632), ('unknown', 22465), ('clothing', 16151), ('books', 3217), ('home', 2788)]

Data loading successful! Total reviews: 99,934
  Avg rating: 4.22
------------------------------------------------------------

import sys
import os
import re
import time
import json
import math
import logging
import warnings
from datetime import datetime
from functools import lru_cache
from typing import Dict, List, Tuple, Union, Optional, Any, Set
from collections import Counter, defaultdict

# Suppress warnings for cleaner output
warnings.filterwarnings("ignore", category=FutureWarning)
warnings.filterwarnings("ignore", category=UserWarning)
# Suppress specific sklearn warnings as requested
warnings.filterwarnings("ignore", category=RuntimeWarning, module="sklearn.utils.extmath")
warnings.filterwarnings("ignore", category=RuntimeWarning, module="sklearn.cluster._kmeans")
warnings.filterwarnings("ignore", message="Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel")
warnings.filterwarnings("ignore", message="KMeans is known to have a memory leak on Windows with MKL")


# Core data science libraries
try:
    import numpy as np
    import pandas as pd
    from sklearn.metrics import (
        confusion_matrix, f1_score, precision_score, recall_score,
        cohen_kappa_score, mean_absolute_error, mean_squared_error,
        classification_report, silhouette_score
    )
    from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold
    from sklearn.cluster import KMeans
    from sklearn.feature_extraction.text import TfidfVectorizer
    from sklearn.preprocessing import StandardScaler
    print("Core data science libraries loaded successfully")
except ImportError as e:
    print(f"CRITICAL ERROR: Missing core libraries: {e}")
    print("Please install: pip install numpy pandas scikit-learn scipy")
    sys.exit(1)

# Visualization libraries
try:
    import matplotlib.pyplot as plt
    import seaborn as sns
    import plotly.graph_objects as go
    import plotly.express as px
    from plotly.subplots import make_subplots
    plt.style.use('seaborn-v0_8-whitegrid')
    sns.set_palette("husl")
    matplotlib_available = True
    plotly_available = True
    print("Visualization libraries (matplotlib, seaborn, plotly) loaded successfully")
except ImportError:
    try:
        import matplotlib.pyplot as plt
        import seaborn as sns
        plt.style.use('seaborn-v0_8-whitegrid')
        sns.set_palette("husl")
        matplotlib_available = True
        plotly_available = False
        print("Basic visualization libraries loaded (plotly not available)")
    except ImportError:
        print("Visualization libraries not available")
        matplotlib_available = False
        plotly_available = False

# Natural Language Processing libraries
try:
    from textblob import TextBlob
    import emoji as emoji_lib
    textproc_available = True
    print("Text processing libraries loaded successfully")
except ImportError:
    print("TextBlob/Emoji not available - some features will be limited (e.g., rule-based ABSA fallback).")
    textproc_available = False

# Transformer library for advanced NLP models
try:
    import torch
    from transformers import pipeline, AutoTokenizer, AutoModelForSequenceClassification
    transformers_available = True
    TRANSFORMER_DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    print(f"Transformer libraries (transformers, torch) loaded successfully. Using device: {TRANSFORMER_DEVICE}")
except ImportError as e:
    print(f"Transformers/PyTorch not available - BERT/RoBERTa models and advanced ABSA will be disabled: {e}")
    print("Please install: pip install transformers torch torchvision torchaudio sentencepiece")
    transformers_available = False
    TRANSFORMER_DEVICE = "cpu" # Fallback to CPU if torch import fails but transformers is there (less likely)


# Progress tracking
try:
    from tqdm.auto import tqdm
    tqdm_available = True
    print("Progress tracking (tqdm) loaded successfully")
except ImportError:
    print("tqdm not available - using basic progress indication")
    tqdm_available = False

    class tqdm:
        def __init__(self, iterable=None, desc=None, total=None, **kwargs):
            self.iterable = iterable
            self.desc = desc or ""
            self.total = total or (len(iterable) if hasattr(iterable, '__len__') else None)
            self.n = 0
            self.start_time = time.time()
            print(f"{self.desc}: Starting...")

        def __iter__(self):
            for i, item in enumerate(self.iterable):
                yield item
                self.n += 1
                if self.total and self.total > 0:
                    if i % max(1, self.total // 20) == 0 or i == self.total - 1:
                        elapsed_time = time.time() - self.start_time
                        progress_percent = 100 * (i + 1) / self.total
                        print(f"{self.desc}: {i+1}/{self.total} ({progress_percent:.1f}%) - Elapsed: {elapsed_time:.2f}s")
            if self.total:
                print(f"{self.desc}: Completed in {time.time() - self.start_time:.2f}s!")

        def update(self, n=1):
            self.n += n
            if self.total and self.total > 0:
                if self.n % max(1, self.total // 20) == 0 or self.n == self.total:
                    elapsed_time = time.time() - self.start_time
                    progress_percent = 100 * self.n / self.total
                    print(f"{self.desc}: {self.n}/{self.total} ({progress_percent:.1f}%) - Elapsed: {elapsed_time:.2f}s")

        def close(self):
            if self.total and self.n < self.total:
                print(f"{self.desc}: Closed prematurely at {self.n}/{self.total}.")
            elif self.total and self.desc:
                print(f"{self.desc}: Progress bar closed.")
            pass

# Jupyter widgets (still used for `display_section_header` and `safe_display` if available)
try:
    from IPython.display import display, HTML, Markdown, clear_output
    ipywidgets_available = True
    print("IPython.display loaded successfully")
except ImportError:
    print("IPython.display not available")
    ipywidgets_available = False
    def display(x): print(x)
    def HTML(x): return str(x)
    def Markdown(x): return str(x)
    def clear_output(wait=False): pass

# Word cloud library (kept for potential simple use cases, but ABSA is primary)
try:
    from wordcloud import WordCloud, STOPWORDS as WC_STOPWORDS
    wordcloud_available = True
    print("WordCloud library loaded successfully (though ABSA is now primary for text insights)")
except ImportError:
    print("WordCloud library not available")
    wordcloud_available = False

# Request library for downloads
try:
    import requests
    requests_available = True
    print("Requests library loaded successfully")
except ImportError:
    print("Requests library not available")
    requests_available = False

print("\nStarting Sentiment Analysis Pipeline with Transformer and ABSA capabilities...")
print("=" * 60)

#%% === NLTK INITIALIZATION ===

def initialize_nltk_resources():
    global NLTK_INITIALIZED
    try:
        import nltk
        print("\nInitializing NLTK resources...")
        nltk_data_dir = os.path.expanduser('~/nltk_data')
        os.makedirs(nltk_data_dir, exist_ok=True)
        if nltk_data_dir not in nltk.data.path:
            nltk.data.path.append(nltk_data_dir)
        resources = {
            'vader_lexicon': 'VADER sentiment analyzer',
            'punkt': 'Punkt tokenizer',
            'averaged_perceptron_tagger': 'POS tagger',
            'wordnet': 'WordNet corpus',
            'stopwords': 'Stopwords corpus',
            'omw-1.4': 'Open Multilingual Wordnet 1.4'
        }
        print("Checking required NLTK resources (will download if not present):")
        all_downloaded_successfully = True
        for resource, description in resources.items():
            is_available = False
            try:
                if resource == 'vader_lexicon':
                    from nltk.sentiment.vader import SentimentIntensityAnalyzer
                    SentimentIntensityAnalyzer()
                elif resource == 'punkt': nltk.data.find('tokenizers/punkt')
                elif resource == 'averaged_perceptron_tagger': nltk.data.find('taggers/averaged_perceptron_tagger')
                elif resource == 'wordnet':
                    try: nltk.data.find('corpora/wordnet.zip')
                    except LookupError: nltk.data.find('corpora/wordnet/index.verb')
                    from nltk.corpus import wordnet as wn
                    wn.ensure_loaded()
                elif resource == 'stopwords': nltk.data.find('corpora/stopwords')
                elif resource == 'omw-1.4': nltk.data.find('corpora/omw-1.4')
                is_available = True
                print(f"  Resource '{description}' ({resource}) is available.")
            except LookupError: print(f"  Resource '{description}' ({resource}) not found. Attempting download...")
            except Exception as e_check: print(f"  Error during initial check for '{description}' ({resource}): {e_check}. Attempting download...")

            if not is_available:
                try:
                    print(f"  Downloading '{description}' ({resource})...")
                    download_result = nltk.download(resource, quiet=False, download_dir=nltk_data_dir)
                    if download_result:
                        print(f"  Successfully downloaded '{description}' ({resource}).")
                        if resource == 'wordnet':
                            try: from nltk.corpus import wordnet as wn; wn.ensure_loaded(); print(f"  WordNet verified after download.")
                            except Exception as e_verify: print(f"  WordNet verification failed after download: {e_verify}"); all_downloaded_successfully = False
                        elif resource == 'vader_lexicon':
                            try: from nltk.sentiment.vader import SentimentIntensityAnalyzer; SentimentIntensityAnalyzer().polarity_scores("test"); print(f"  VADER verified after download.")
                            except Exception as e_verify: print(f"  VADER verification failed after download: {e_verify}"); all_downloaded_successfully = False
                    else: print(f"  Download reported failure for '{description}' ({resource})."); all_downloaded_successfully = False
                except Exception as e_download: print(f"  Failed to download or verify '{description}' ({resource}): {e_download}"); all_downloaded_successfully = False

        if not all_downloaded_successfully: print("\nWarning: Not all NLTK resources were successfully downloaded or verified.")
        try:
            from nltk.sentiment.vader import SentimentIntensityAnalyzer
            from nltk.tokenize import word_tokenize, sent_tokenize
            from nltk.stem import WordNetLemmatizer
            from nltk.corpus import stopwords, wordnet
            SentimentIntensityAnalyzer().polarity_scores("test")
            word_tokenize("test tokenization"); sent_tokenize("Test. Sentence tokenization.")
            WordNetLemmatizer().lemmatize("tests", pos=wordnet.NOUN)
            set(stopwords.words('english'))
            NLTK_INITIALIZED = True
            print("NLTK initialization completed successfully!")
            return True
        except Exception as e: print(f"NLTK component test failed: {e}"); NLTK_INITIALIZED = False; return False
    except ImportError: print("NLTK library not installed. Install with: pip install nltk"); NLTK_INITIALIZED = False; return False
    except Exception as e: print(f"Unexpected error during NLTK initialization: {e}"); NLTK_INITIALIZED = False; return False
NLTK_INITIALIZED = False
initialize_nltk_resources()

#%% === GLOBAL CONFIGURATION AND CONSTANTS ===
POSITIVE_WORDS = {'excellent', 'amazing', 'wonderful', 'fantastic', 'outstanding', 'brilliant', 'superb', 'great', 'good', 'best', 'perfect', 'love', 'awesome', 'incredible', 'impressive', 'delighted', 'satisfied', 'happy', 'pleased', 'recommend', 'favorite', 'nice', 'easy', 'fast', 'effective', 'reliable', 'quality', 'comfortable', 'beautiful', 'works', 'responsive', 'durable', 'compatible', 'smooth', 'stylish', 'soft'}
NEGATIVE_WORDS = {'terrible', 'awful', 'horrible', 'worst', 'bad', 'poor', 'disappointing', 'hate', 'useless', 'broken', 'defective', 'fail', 'failure', 'problem', 'issue', 'cheap', 'waste', 'regret', 'return', 'refund', 'complaint', 'avoid', 'mediocre', 'junk', 'slow', 'difficult', 'frustrating', 'inferior', 'subpar', 'flawed', 'damaged'}
INTENSIFIERS = {'very', 'extremely', 'absolutely', 'incredibly', 'really', 'truly', 'highly', 'especially', 'particularly', 'exceptionally', 'totally', 'completely', 'utterly', 'thoroughly', 'so', 'too', 'quite', 'remarkably', 'super', 'ultra'}
NEGATION_TERMS = {'not', 'no', "n't", 'never', 'neither', 'nor', 'none', 'nothing', 'without', 'hardly', 'scarcely', 'barely'}
DOMAIN_WORDS = {
    'electronics': {'positive': {'fast', 'responsive', 'reliable', 'quality', 'performance', 'durable', 'compatible', 'battery', 'screen', 'audio', 'video', 'wireless', 'clear', 'sharp', 'bright', 'efficient', 'powerful', 'stable', 'connectivity'}, 'negative': {'broken', 'slow', 'lag', 'crash', 'malfunction', 'defective', 'incompatible', 'overheat', 'noisy', 'dead', 'static', 'blurry', 'dim', 'unstable', 'disconnect', 'freeze', 'error'}},
    'clothing': {'positive': {'comfortable', 'soft', 'quality', 'stylish', 'perfect', 'fit', 'beautiful', 'warm', 'cozy', 'flattering', 'breathable', 'fashionable', 'elegant', 'attractive', 'luxurious', 'smooth'}, 'negative': {'small', 'large', 'tight', 'cheap', 'shrink', 'ripped', 'thin', 'itchy', 'faded', 'loose', 'uncomfortable', 'scratchy', 'rough', 'wrinkled', 'poor', 'worn'}},
    'books': {'positive': {'engaging', 'interesting', 'well-written', 'compelling', 'enjoyed', 'recommend', 'insightful', 'gripping', 'page-turner', 'brilliant', 'informative', 'educational', 'entertaining', 'thoughtful'}, 'negative': {'boring', 'slow', 'predictable', 'disappointed', 'waste', 'poorly', 'confusing', 'shallow', 'unrealistic', 'dragging', 'repetitive', 'tedious', 'dull'}},
    'food': {'positive': {'delicious', 'tasty', 'fresh', 'flavorful', 'yummy', 'healthy', 'organic', 'crispy', 'creamy', 'savory', 'sweet', 'nutritious', 'gourmet', 'satisfying'}, 'negative': {'stale', 'expired', 'tasteless', 'rotten', 'bland', 'sour', 'artificial', 'moldy', 'greasy', 'soggy', 'bitter', 'spoiled', 'disgusting'}},
    'beauty': {'positive': {'effective', 'natural', 'smooth', 'gentle', 'clean', 'glowing', 'hydrating', 'refreshing', 'lightweight', 'absorbs', 'nourishing', 'moisturizing', 'radiant'}, 'negative': {'irritation', 'breakout', 'greasy', 'sticky', 'harsh', 'chemical', 'allergic', 'drying', 'heavy', 'residue', 'burning', 'reaction'}},
    'home': {'positive': {'sturdy', 'functional', 'beautiful', 'organized', 'convenient', 'bright', 'useful', 'powerful', 'quiet', 'durable', 'spacious', 'modern', 'elegant'}, 'negative': {'flimsy', 'wobbly', 'broken', 'damaged', 'leaky', 'loud', 'difficult', 'cheap', 'unstable', 'cramped', 'outdated'}}
}

# FIXED: Split technical specs into truly generic vs category-specific
PRODUCT_SPEC_WORDS = {
    'generic_specs': {'size', 'weight', 'color', 'material', 'dimension', 'design', 'build', 'construction', 'durability', 'assembly', 'installation', 'usability', 'price', 'value', 'quality'},
    'electronics_specs': {'processor', 'ram', 'storage_type', 'gpu', 'display_type', 'refresh_rate', 'port', 'os', 'software', 'capacity', 'speed', 'resolution', 'frequency', 'voltage', 'wattage', 'memory', 'storage', 'connectivity', 'interface', 'compatibility', 'specification', 'feature', 'performance', 'functionality', 'battery', 'screen', 'audio', 'video', 'camera', 'keyboard', 'mouse', 'wireless'},
    'clothing_specs': {'fabric', 'stitching', 'zipper', 'button', 'collar', 'sleeve', 'waist', 'inseam', 'sizing_chart', 'fit', 'comfortable', 'soft', 'warm', 'breathable'},
    'books_specs': {'author', 'genre', 'pages', 'cover_type', 'publisher', 'series', 'edition', 'plot', 'character', 'writing_style'},
    'food_specs': {'ingredient', 'nutrition', 'allergen', 'flavor_profile', 'texture', 'portion_size', 'origin', 'freshness_date'},
    'beauty_specs': {'ingredient_list', 'skin_type', 'hair_type', 'fragrance', 'application', 'spf', 'volume', 'shade'},
    'home_specs': {'material_quality', 'assembly_time', 'warranty', 'maintenance', 'energy_efficiency', 'dimensions_w_h_d'}
}
ALL_DOMAIN_WORDS = set().union(*(cat_data.get('positive', set()) | cat_data.get('negative', set()) for cat_data in DOMAIN_WORDS.values()))
ALL_SPEC_WORDS = set().union(*(spec_set for spec_set in PRODUCT_SPEC_WORDS.values()))
ALL_SENTIMENT_WORDS = POSITIVE_WORDS.union(NEGATIVE_WORDS)
print(f"Loaded {len(POSITIVE_WORDS)} positive words, {len(NEGATIVE_WORDS)} negative words, {len(ALL_DOMAIN_WORDS)} domain words, {len(ALL_SPEC_WORDS)} spec words.")

#%% === UTILITY FUNCTIONS ===
def setup_logging(results_dir: str) -> logging.Logger:
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    log_file = os.path.join(results_dir, f"sentiment_analysis_{timestamp}.log")
    logger = logging.getLogger('sentiment_pipeline')
    if logger.hasHandlers(): logger.handlers.clear()
    formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(funcName)s - %(message)s', datefmt='%Y-%m-%d %H:%M:%S')
    file_handler = logging.FileHandler(log_file, encoding='utf-8'); file_handler.setFormatter(formatter); file_handler.setLevel(logging.INFO)
    console_handler = logging.StreamHandler(sys.stdout); console_handler.setFormatter(formatter); console_handler.setLevel(logging.INFO)
    logger.addHandler(file_handler); logger.addHandler(console_handler); logger.setLevel(logging.INFO)
    logger.info("=" * 60 + f"\nSentiment Analysis Pipeline Logger Initialized\nResults directory: {results_dir}\nLog file: {log_file}\n" + "=" * 60)
    return logger

def display_section_header(title: str, level: int = 1) -> None:
    if ipywidgets_available and is_jupyter_environment():
        colors = ['#2E8B57', '#4169E1', '#FF6347', '#1E90FF'];
        color_idx = min(level-1, len(colors)-1)
        color = colors[color_idx]
        display(HTML(f"""<div style="background: linear-gradient(90deg, {color}22 0%, {color}11 100%); border-left: 4px solid {color}; padding: 15px 20px; margin: 20px 0; border-radius: 5px; box-shadow: 0 2px 4px rgba(0,0,0,0.1);"><h{level} style="color: {color}; margin: 0; font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif; font-weight: 600;">{title}</h{level}></div>"""))
    else: print(f"\n{'='*(60-level*5)}\n{'#'*level} {title}\n{'='*(60-level*5)}")

def safe_display(data) -> None:
    try:
        if isinstance(data, (pd.DataFrame, pd.Series)):
            if ipywidgets_available and is_jupyter_environment():
                if isinstance(data, pd.DataFrame) and len(data) > 0:
                    # Ensure DataFrame passed to Styler has a unique index or reset it.
                    # If 'index' is a column name, it could conflict.
                    df_to_style = data.copy()
                    if 'index' in df_to_style.columns and not df_to_style.index.is_unique:
                        # If 'index' column exists and index is not unique, it might be safer to reset.
                        # This is a heuristic; the Styler error is often about the DataFrame's actual index.
                        pass # Assuming df_to_style has a RangeIndex from pd.DataFrame(list_of_dicts)
                    
                    # Ensure the DataFrame index is unique for Styler
                    if not df_to_style.index.is_unique:
                        df_to_style = df_to_style.reset_index(drop=True)

                    display(df_to_style.style.set_table_styles([
                        {'selector': 'th', 'props': [('background-color', '#f0f2f6'), ('color', '#2c3e50'), ('font-weight', 'bold'), ('padding', '8px'), ('text-align', 'left')]},
                        {'selector': 'td', 'props': [('padding', '8px'), ('text-align', 'left')]},
                        {'selector': 'tr:nth-child(even)', 'props': [('background-color', '#f8f9fa')]}
                    ]).set_properties(**{'border': '1px solid #ddd', 'width': 'auto'}))
                else:
                    display(HTML(data.to_html(max_rows=50, notebook=True)))
            else:
                print(data.to_string(max_rows=100))
        else:
            if ipywidgets_available and is_jupyter_environment():
                display(data)
            else:
                print(data)
    except KeyError as ke:
        # Specifically catch the Styler KeyError if it still happens
        print(f"Display error (likely Styler incompatibility with DataFrame index/columns): {ke}\nAttempting basic print...\n{data.to_string(max_rows=100)}")
    except Exception as e:
        print(f"Display error: {e}\nAttempting basic print...\n{data}")


def is_jupyter_environment() -> bool:
    try:
        from IPython import get_ipython
        if get_ipython() is not None and get_ipython().__class__.__name__ != 'TerminalInteractiveShell': return True
    except ImportError: return False
    return False

@lru_cache(maxsize=10000)
def get_wordnet_pos(treebank_tag: str) -> str:
    if not NLTK_INITIALIZED: return 'n'
    try:
        from nltk.corpus import wordnet
        if treebank_tag.startswith('J'): return wordnet.ADJ
        elif treebank_tag.startswith('V'): return wordnet.VERB
        elif treebank_tag.startswith('N'): return wordnet.NOUN
        elif treebank_tag.startswith('R'): return wordnet.ADV
        else: return wordnet.NOUN
    except Exception: return 'n'

@lru_cache(maxsize=50000)
def lemmatize_word(word: str, pos_tag: str) -> str:
    if not NLTK_INITIALIZED: return word
    try:
        from nltk.stem import WordNetLemmatizer
        return WordNetLemmatizer().lemmatize(word, pos=get_wordnet_pos(pos_tag))
    except Exception: return word

def extract_ngrams(text: str, n: int = 2) -> List[str]:
    if not text or not isinstance(text, str): return []
    words = text.split()
    if len(words) < n: return []
    return [' '.join(words[i:i + n]) for i in range(len(words) - n + 1)]

#%% === ENHANCED TEXT PREPROCESSING ===
class TextPreprocessor:
    def __init__(self, resources: dict = None):
        self.resources = resources or {}
        fallback_stopwords = {"i", "me", "my", "myself", "we", "our", "ours", "ourselves", "you", "your", "yours", "yourself", "yourselves", "he", "him", "his", "himself", "she", "her", "hers", "herself", "it", "its", "itself", "they", "them", "their", "theirs", "themselves", "what", "which", "who", "whom", "this", "that", "these", "those", "am", "is", "are", "was", "were", "be", "been", "being", "have", "has", "had", "having", "do", "does", "did", "doing", "a", "an", "the", "and", "but", "if", "or", "because", "as", "until", "while", "of", "at", "by", "for", "with", "about", "against", "between", "into", "through", "during", "before", "after", "above", "below", "to", "from", "up", "down", "in", "out", "on", "off", "over", "under", "again", "further", "then", "once", "here", "there", "when", "where", "why", "how", "all", "any", "both", "each", "few", "more", "most", "other", "some", "such", "no", "nor", "not", "only", "own", "same", "so", "than", "too", "very", "s", "t", "can", "will", "just", "don", "should", "now"}
        self.stop_words = self.resources.get('stop_words', fallback_stopwords) if NLTK_INITIALIZED and 'stop_words' in self.resources and self.resources['stop_words'] else fallback_stopwords
        self.review_stopwords = {'product', 'item', 'review', 'amazon', 'bought', 'purchase', 'order', 'price', 'money', 'buy', 'get', 'use', 'used', 'using', 'time', 'would', 'could', 'should', 'one', 'two', 'three', 'also', 'even'}
        self.all_stopwords = self.stop_words.union(self.review_stopwords)

    def clean_text(self, text: str) -> str:
        if not isinstance(text, str) or pd.isna(text): return ""
        text = text.lower()
        if textproc_available:
            try: text = emoji_lib.demojize(text, delimiters=(" ", " "))
            except Exception: pass
        text = re.sub(r'http[s]?://\S+', ' ', text)
        text = re.sub(r'@\w+', ' ', text)
        text = re.sub(r'\b[A-Za-z0-9]{20,}\b', ' ', text)
        text = re.sub(r"[^a-z0-9\s'.!?]", " ", text)
        text = re.sub(r'\s+', ' ', text).strip()
        return text

    def tokenize_and_lemmatize(self, text: str) -> List[str]:
        if not text: return []
        if not NLTK_INITIALIZED: return [word for word in text.split() if word not in self.all_stopwords and len(word) > 1 and word.isalpha()]
        try:
            from nltk.tokenize import word_tokenize; from nltk.tag import pos_tag
            text_for_lemma = re.sub(r"[^a-z0-9\s']", " ", text.lower()); text_for_lemma = re.sub(r'\s+', ' ', text_for_lemma).strip()
            tokens = word_tokenize(text_for_lemma)
            try: tagged_tokens = pos_tag(tokens)
            except Exception: tagged_tokens = [(token, 'NN') for token in tokens]
            return [lemmatize_word(word, tag) for word, tag in tagged_tokens if word not in self.all_stopwords and len(word) > 1 and word.isalpha()]
        except Exception: return [word for word in text.split() if word not in self.all_stopwords and len(word) > 1 and word.isalpha()]

    def preprocess_text_for_rule_based(self, text: str) -> str:
        cleaned = self.clean_text(text)
        tokens = self.tokenize_and_lemmatize(cleaned)
        return ' '.join(tokens)

    def preprocess_text_for_transformers(self, text: str) -> str:
        if not isinstance(text, str) or pd.isna(text): return ""
        text = text.lower()
        if textproc_available:
            try: text = emoji_lib.demojize(text, delimiters=(" ", " "))
            except Exception: pass
        text = re.sub(r'http[s]?://\S+', ' ', text)
        text = re.sub(r'@\w+', ' ', text)
        text = re.sub(r'\s+', ' ', text).strip()
        return text

    def batch_preprocess(self, texts: List[str], mode: str = 'rule_based', batch_size: int = 1000) -> List[str]:
        if not texts: return []
        total_texts = len(texts); processed_texts = []
        preprocess_func = self.preprocess_text_for_rule_based if mode == 'rule_based' else self.preprocess_text_for_transformers
        global tqdm; progress_bar_desc = f"Preprocessing texts ({mode} - Batch)"
        p_bar = tqdm(total=total_texts, desc=progress_bar_desc, unit="texts") if tqdm_available else None
        if not p_bar: print(f"{progress_bar_desc}: Starting for {total_texts} texts...")
        for i in range(0, total_texts, batch_size):
            batch = texts[i:min(i + batch_size, total_texts)]
            processed_texts.extend([preprocess_func(text) for text in batch])
            if p_bar: p_bar.update(len(batch))
            elif (i // batch_size) % max(1, (total_texts // batch_size) // 10) == 0 or i + batch_size >= total_texts: print(f"{progress_bar_desc}: {len(processed_texts)}/{total_texts} processed...")
        if p_bar: p_bar.close()
        else: print(f"{progress_bar_desc}: Completed processing {len(processed_texts)} texts!")
        return processed_texts

#%% === ENHANCED FEATURE EXTRACTION (for Rule-Based Models) ===
class FeatureExtractor:
    def __init__(self, resources: dict = None):
        self.resources = resources or {}
        if NLTK_INITIALIZED and 'vader_analyzer' in self.resources and self.resources['vader_analyzer']: self.vader_analyzer = self.resources.get('vader_analyzer')
        else:
            try: from nltk.sentiment.vader import SentimentIntensityAnalyzer; self.vader_analyzer = SentimentIntensityAnalyzer()
            except: self.vader_analyzer = None
    def extract_basic_features(self, text: str) -> Dict[str, Any]:
        raw_words = re.findall(r'\b\w+\b', text.lower()); wc = len(raw_words)
        sc = len(re.findall(r'[.!?]+', text)) if text else 0; sc = 1 if sc == 0 and wc > 0 else sc
        return {'text_length': len(text), 'word_count': wc, 'sentence_count': sc, 'avg_word_length': np.mean([len(w) for w in raw_words if w]) if wc > 0 else 0}
    def extract_sentiment_features(self, text: str) -> Dict[str, Any]:
        features = {'vader_negative': 0.0, 'vader_neutral': 1.0, 'vader_positive': 0.0, 'vader_compound': 0.0, 'textblob_polarity': 0.0, 'textblob_subjectivity': 0.5}
        if self.vader_analyzer:
            try: vs = self.vader_analyzer.polarity_scores(text); features.update({'vader_negative': vs['neg'], 'vader_neutral': vs['neu'], 'vader_positive': vs['pos'], 'vader_compound': vs['compound']})
            except Exception: pass
        if textproc_available:
            try: blob = TextBlob(text); features['textblob_polarity'] = blob.sentiment.polarity; features['textblob_subjectivity'] = blob.sentiment.subjectivity
            except Exception: pass
        return features
    def extract_lexicon_features(self, preprocessed_text_for_rules: str) -> Dict[str, Any]:
        words = set(preprocessed_text_for_rules.lower().split()) if preprocessed_text_for_rules else set()
        pc = len(words.intersection(POSITIVE_WORDS)); nc = len(words.intersection(NEGATIVE_WORDS))
        features = {'positive_word_count': pc, 'negative_word_count': nc, 'intensifier_count': len(words.intersection(INTENSIFIERS)), 'negation_count': len(words.intersection(NEGATION_TERMS))}
        tsc = pc + nc; features['sentiment_ratio'] = (pc / tsc if tsc > 0 else 0.5)
        return features
    def extract_domain_features(self, preprocessed_text_for_rules: str, category: str = 'unknown') -> Dict[str, Any]:
        words = set(preprocessed_text_for_rules.lower().split()) if preprocessed_text_for_rules else set()
        domain_data = DOMAIN_WORDS.get(category, {'positive': set(), 'negative': set()})
        return {'domain_positive_count': len(words.intersection(domain_data.get('positive', set()))), 'domain_negative_count': len(words.intersection(domain_data.get('negative', set())))}
    def extract_linguistic_features(self, raw_text: str) -> Dict[str, Any]:
        features = {'exclamation_count': raw_text.count('!') if raw_text else 0, 'question_count': raw_text.count('?') if raw_text else 0, 'caps_ratio': sum(1 for c in raw_text if c.isupper()) / len(raw_text) if raw_text and len(raw_text) > 0 else 0.0, 'digit_count': sum(1 for c in raw_text if c.isdigit()) if raw_text else 0}
        star_patterns = {'mentions_5_star': r'\b(5|five)[\s-]*stars?\b', 'mentions_4_star': r'\b(4|four)[\s-]*stars?\b', 'mentions_3_star': r'\b(3|three)[\s-]*stars?\b', 'mentions_2_star': r'\b(2|two)[\s-]*stars?\b', 'mentions_1_star': r'\b(1|one)[\s-]*stars?\b'}
        for feat, patt in star_patterns.items(): features[feat] = bool(re.search(patt, raw_text, re.IGNORECASE)) if raw_text else False
        return features
    def extract_ngram_features(self, preprocessed_text_for_rules: str) -> Dict[str, Any]:
        features = {}
        for n in [2, 3]:
            ngrams = extract_ngrams(preprocessed_text_for_rules, n); features[f'{n}gram_count'] = len(ngrams)
            pos_ng, neg_ng = 0,0
            for ngram in ngrams: words = set(ngram.split()); pos_ng += 1 if not words.isdisjoint(POSITIVE_WORDS) else 0; neg_ng += 1 if not words.isdisjoint(NEGATIVE_WORDS) else 0
            features[f'{n}gram_positive_count'] = pos_ng; features[f'{n}gram_negative_count'] = neg_ng
        return features
    def extract_all_features(self, raw_text: str, preprocessed_text_for_rules: str, category: str = 'unknown') -> Dict[str, Any]:
        if not raw_text or pd.isna(raw_text): return self._get_default_features()
        features = {}; features.update(self.extract_basic_features(raw_text)); features.update(self.extract_sentiment_features(raw_text)); features.update(self.extract_linguistic_features(raw_text))
        features.update(self.extract_lexicon_features(preprocessed_text_for_rules)); features.update(self.extract_domain_features(preprocessed_text_for_rules, category)); features.update(self.extract_ngram_features(preprocessed_text_for_rules))
        features['custom_sentiment_score'] = self._compute_custom_sentiment(features)
        return features
    def _get_default_features(self) -> Dict[str, Any]:
        defaults = {'text_length': 0, 'word_count': 0, 'sentence_count': 0, 'avg_word_length': 0.0, 'vader_negative': 0.0, 'vader_neutral': 1.0, 'vader_positive': 0.0, 'vader_compound': 0.0, 'textblob_polarity': 0.0, 'textblob_subjectivity': 0.5, 'positive_word_count': 0, 'negative_word_count': 0, 'intensifier_count': 0, 'negation_count': 0, 'sentiment_ratio': 0.5, 'domain_positive_count': 0, 'domain_negative_count': 0, 'exclamation_count': 0, 'question_count': 0, 'caps_ratio': 0.0, 'digit_count': 0, 'mentions_5_star': False, 'mentions_4_star': False, 'mentions_3_star': False, 'mentions_2_star': False, 'mentions_1_star': False, 'custom_sentiment_score': 0.0}
        for n in [2, 3]: defaults[f'{n}gram_count']=0; defaults[f'{n}gram_positive_count']=0; defaults[f'{n}gram_negative_count']=0
        return defaults
    def _compute_custom_sentiment(self, features: Dict[str, Any]) -> float:
        score = features.get('vader_compound',0.0)*0.3 + features.get('textblob_polarity',0.0)*0.2 + ((features.get('sentiment_ratio',0.5)*2)-1)*0.25
        dp, dn = features.get('domain_positive_count',0), features.get('domain_negative_count',0); dt = dp + dn
        if dt > 0: score += ((dp-dn)/dt)*0.15
        ng_sum, ng_types = 0.0, 0
        for n in [2,3]:
            ngp, ngn = features.get(f'{n}gram_positive_count',0), features.get(f'{n}gram_negative_count',0); ngt = ngp + ngn
            if ngt > 0: ng_sum += (ngp-ngn)/ngt; ng_types += 1
        if ng_types > 0: score += (ng_sum/ng_types)*0.1
        return np.clip(score, -1.0, 1.0)
    def batch_extract_features(self, raw_texts: List[str], preprocessed_texts_for_rules: List[str], categories: List[str] = None, batch_size: int = 1000) -> pd.DataFrame:
        if not raw_texts or not preprocessed_texts_for_rules or len(raw_texts) != len(preprocessed_texts_for_rules): return pd.DataFrame()
        if categories is None: categories = ['unknown'] * len(raw_texts)
        elif len(categories) != len(raw_texts): categories = (list(categories) + ['unknown']*len(raw_texts))[:len(raw_texts)]
        all_features = []; total = len(raw_texts)
        global tqdm; p_bar_desc = "Extracting features (Batch)"; p_bar = tqdm(total=total, desc=p_bar_desc, unit="texts") if tqdm_available else None
        if not p_bar: print(f"{p_bar_desc}: Starting for {total} texts...")
        for i in range(0, total, batch_size):
            b_raw, b_prep, b_cat = raw_texts[i:min(i+batch_size,total)], preprocessed_texts_for_rules[i:min(i+batch_size,total)], categories[i:min(i+batch_size,total)]
            all_features.extend([self.extract_all_features(r,p,c) for r,p,c in zip(b_raw,b_prep,b_cat)])
            if p_bar: p_bar.update(len(b_raw))
            elif (i // batch_size) % max(1, (total // batch_size)//10) == 0 or i+batch_size >= total: print(f"{p_bar_desc}: {len(all_features)}/{total} processed...")
        if p_bar: p_bar.close()
        else: print(f"{p_bar_desc}: Completed for {len(all_features)} texts!")
        return pd.DataFrame(all_features)

#%% === SENTIMENT MODELS (RULE-BASED + TRANSFORMERS) ===
class TransformerSentimentPredictor:
    def __init__(self, model_name_or_path: str = "cardiffnlp/twitter-roberta-base-sentiment-latest", logger=None):
        self.logger = logger or logging.getLogger(__name__); self.model_name = model_name_or_path; self.pipeline = None
        if transformers_available:
            try: self.pipeline = pipeline("sentiment-analysis", model=self.model_name, device=TRANSFORMER_DEVICE, return_all_scores=False); self.logger.info(f"Transformer sentiment model '{self.model_name}' loaded on {TRANSFORMER_DEVICE}.")
            except Exception as e: self.logger.error(f"Failed to load Transformer model '{self.model_name}': {e}")
        else: self.logger.warning("Transformers library not available. TransformerSentimentPredictor will not function.")
    def predict(self, text: str) -> Optional[Dict[str, Any]]:
        if not self.pipeline or not text or pd.isna(text): return None
        try:
            with warnings.catch_warnings(): warnings.simplefilter("ignore"); result = self.pipeline(text, truncation=True, max_length=512)
            return result[0] if result and isinstance(result, list) else None
        except Exception as e: self.logger.error(f"Error during Transformer prediction for '{text[:50]}...': {e}"); return None
    def batch_predict(self, texts: List[str], batch_size: int = 32) -> List[Optional[Dict[str, Any]]]:
        if not self.pipeline or not texts: return [None] * len(texts)
        results = []; p_bar_desc = f"Transformer ({self.model_name.split('/')[-1]}) Batch Predict"; p_bar = tqdm(total=len(texts), desc=p_bar_desc, unit="texts") if tqdm_available and len(texts) > batch_size else None
        for i in range(0, len(texts), batch_size):
            batch_in = texts[i:min(i+batch_size, len(texts))]
            try:
                with warnings.catch_warnings(): warnings.simplefilter("ignore"); batch_out = self.pipeline(batch_in, truncation=True, max_length=512)
                results.extend(batch_out)
            except Exception as e: self.logger.error(f"Error Transformer batch prediction: {e}"); results.extend([None]*len(batch_in))
            if p_bar: p_bar.update(len(batch_in))
        if p_bar: p_bar.close()
        final_results = []; res_idx = 0
        for _ in texts: final_results.append(results[res_idx] if res_idx < len(results) and results[res_idx] is not None else None); res_idx +=1
        return final_results

class SentimentModels:
    """Collection of sentiment analysis models for rating prediction."""

    @staticmethod
    def roberta_sentiment_to_stars(prediction_output: Optional[Dict[str, Any]]) -> int:
        """Convert RoBERTa sentiment model output to star ratings."""
        if prediction_output is None or 'label' not in prediction_output:
            return 3 # Default to neutral if no prediction

        label = prediction_output['label'].lower()

        if 'positive' in label: return 5
        elif 'negative' in label: return 1
        elif 'neutral' in label: return 3
        elif label == "label_2" or label == "2": return 5
        elif label == "label_0" or label == "0": return 1
        elif label == "label_1" or label == "1": return 3
        else: return 3


    @staticmethod
    def vader_to_stars(features: Dict[str, Any], thresholds: List[float] = None) -> int:
        """Convert VADER sentiment scores to star ratings using optimized thresholds."""
        final_thresholds = thresholds if thresholds is not None else [-0.6, -0.2, 0.2, 0.7]

        for star_val in [5, 4, 3, 2, 1]:
            if features.get(f'mentions_{star_val}_star', False):
                return star_val

        compound = features.get('vader_compound', 0.0)

        if compound <= final_thresholds[0]:
            return 1
        elif compound <= final_thresholds[1]:
            return 2
        elif compound < final_thresholds[2]:
            return 3
        elif compound < final_thresholds[3]:
            return 4
        else:
            return 5

    @staticmethod
    def textblob_to_stars(features: Dict[str, Any], thresholds: List[float] = None) -> int:
        """Convert TextBlob sentiment scores to star ratings."""
        final_thresholds = thresholds if thresholds is not None else [-0.5, -0.1, 0.3, 0.6]

        for star_val in [5, 4, 3, 2, 1]:
            if features.get(f'mentions_{star_val}_star', False):
                return star_val

        polarity = features.get('textblob_polarity', 0.0)

        if polarity <= final_thresholds[0]:
            return 1
        elif polarity <= final_thresholds[1]:
            return 2
        elif polarity < final_thresholds[2]:
            return 3
        elif polarity < final_thresholds[3]:
            return 4
        else:
            return 5

    @staticmethod
    def feature_based_to_stars(features: Dict[str, Any]) -> int:
        """Heuristic feature-based model using multiple feature types."""
        for star_val in [5, 4, 3, 2, 1]:
            if features.get(f'mentions_{star_val}_star', False):
                return star_val

        score = 0.0
        score += features.get('vader_compound', 0.0) * 1.5
        score += features.get('textblob_polarity', 0.0) * 0.5
        score += (features.get('positive_word_count', 0) - features.get('negative_word_count', 0)) * 0.2
        score += (features.get('domain_positive_count', 0) - features.get('domain_negative_count', 0)) * 0.3

        if score <= -1.0:
            return 1
        elif score <= -0.25:
            return 2
        elif score < 0.35:
            return 3
        elif score < 1.1:
            return 4
        else:
            return 5

    @staticmethod
    def multidimensional_to_stars(features: Dict[str, Any]) -> int:
        """Advanced multidimensional sentiment analysis model (Valence-Arousal-Dominance inspired)."""
        for star_val in [5, 4, 3, 2, 1]:
            if features.get(f'mentions_{star_val}_star', False):
                return star_val

        valence_score = (features.get('vader_positive', 0.0) - features.get('vader_negative', 0.0) +
                         features.get('textblob_polarity', 0.0) * 0.8 +
                         (features.get('sentiment_ratio', 0.5) - 0.5) * 2)

        arousal_score = (features.get('intensifier_count', 0) * 0.1 +
                         features.get('exclamation_count', 0) * 0.05 +
                         features.get('caps_ratio', 0.0) * 2.0 +
                         (1 - features.get('vader_neutral', 1.0)))

        dominance_score = ((1 - features.get('textblob_subjectivity', 0.5)) * 2.0 -
                           features.get('question_count', 0) * 0.1 +
                           min(features.get('word_count', 0) / 50.0, 1.0))

        context_score = 0.0
        domain_pos = features.get('domain_positive_count', 0)
        domain_neg = features.get('domain_negative_count', 0)
        if (domain_pos + domain_neg) > 0:
            context_score = ((domain_pos - domain_neg) / (domain_pos + domain_neg))

        linguistic_score = 0.0
        if features.get('word_count', 0) > 0:
            linguistic_score += (min(features.get('avg_word_length', 0) / 6.0, 1.0) -
                                 features.get('negation_count', 0) * 0.1)

        final_score = (valence_score * 0.40 + arousal_score * 0.20 + dominance_score * 0.15 +
                       context_score * 0.15 + linguistic_score * 0.10)

        if final_score <= -0.8:
            return 1
        elif final_score <= -0.3:
            return 2
        elif final_score < 0.4:
            return 3
        elif final_score < 0.9:
            return 4
        else:
            return 5

    @staticmethod
    def rule_based_enhanced_to_stars(features: Dict[str, Any]) -> int:
        """Enhanced rule-based model with sophisticated decision trees."""
        for star_val in [5, 4, 3, 2, 1]:
            if features.get(f'mentions_{star_val}_star', False):
                return star_val

        strong_negative = (features.get('negative_word_count', 0) >= 3 or
                           features.get('vader_negative', 0.0) > 0.5 or
                           features.get('textblob_polarity', 0.0) < -0.4)
        strong_positive = (features.get('positive_word_count', 0) >= 3 or
                           features.get('vader_positive', 0.0) > 0.6 or
                           features.get('textblob_polarity', 0.0) > 0.5)
        has_negation = features.get('negation_count', 0) > 0
        high_intensity = (features.get('intensifier_count', 0) > 1 or
                          features.get('exclamation_count', 0) > 1 or
                          features.get('caps_ratio', 0.0) > 0.1)

        domain_sentiment_score = 0.0
        domain_pos = features.get('domain_positive_count', 0)
        domain_neg = features.get('domain_negative_count', 0)
        if (domain_pos + domain_neg) > 0:
            domain_sentiment_score = (domain_pos - domain_neg) / (domain_pos + domain_neg)

        if strong_negative and not has_negation:
            return 1 if high_intensity else 2
        if strong_positive and not has_negation:
            return 5 if high_intensity or domain_sentiment_score > 0.3 else 4

        if has_negation:
            if strong_positive: return 2
            if strong_negative: return 3

        adjusted_score = features.get('vader_compound', 0.0) + domain_sentiment_score * 0.3
        if adjusted_score <= -0.5:
            return 1
        elif adjusted_score <= -0.1:
            return 2
        elif adjusted_score < 0.3:
            return 3
        elif adjusted_score < 0.7:
            return 4
        else:
            return 5

    @staticmethod
    def hybrid_weighted_to_stars(features: Dict[str, Any]) -> int:
        """Hybrid model combining statistical and rule-based approaches."""
        for star_val in [5, 4, 3, 2, 1]:
            if features.get(f'mentions_{star_val}_star', False):
                return star_val

        weights_conf = {
            'vader': min(abs(features.get('vader_compound', 0.0)) * 2.0, 1.0),
            'textblob': min(abs(features.get('textblob_polarity', 0.0)) * 1.5, 1.0),
            'lexicon': min((features.get('positive_word_count', 0) + features.get('negative_word_count', 0)) / 5.0, 1.0),
            'domain': min((features.get('domain_positive_count', 0) + features.get('domain_negative_count', 0)) / 3.0, 1.0)
        }
        length_confidence = min(features.get('word_count', 0) / 100.0, 1.0)

        predictions_with_weights = []
        if weights_conf['vader'] > 0.1:
            predictions_with_weights.append((SentimentModels.vader_to_stars(features), weights_conf['vader']))
        if weights_conf['textblob'] > 0.1:
            predictions_with_weights.append((SentimentModels.textblob_to_stars(features), weights_conf['textblob']))

        feature_model_conf = (weights_conf.get('lexicon',0.0) + weights_conf.get('domain',0.0) + length_confidence) / 3.0
        predictions_with_weights.append((SentimentModels.feature_based_to_stars(features), 0.5 + feature_model_conf * 0.5))
        predictions_with_weights.append((SentimentModels.rule_based_enhanced_to_stars(features), length_confidence * 0.8))

        if predictions_with_weights:
            valid_weighted_preds = [(p, w) for p, w in predictions_with_weights if w > 0]
            if not valid_weighted_preds: return 3

            total_weight = sum(w for _, w in valid_weighted_preds)
            if total_weight > 0:
                weighted_sum = sum(p * w for p, w in valid_weighted_preds)
                final_pred_float = weighted_sum / total_weight
                return max(1, min(5, round(final_pred_float)))
        return 3

    @staticmethod
    def lexicon_ensemble_to_stars(features: Dict[str, Any]) -> int:
        """Ensemble of multiple lexicon-based approaches."""
        for star_val in [5, 4, 3, 2, 1]:
            if features.get(f'mentions_{star_val}_star', False):
                return star_val

        scores = []
        sentiment_ratio_val = features.get('sentiment_ratio', 0.5)
        scores.append(sentiment_ratio_val * 4.0 + 1.0)

        pos_words = features.get('positive_word_count', 0)
        neg_words = features.get('negative_word_count', 0)
        intensifier_count = features.get('intensifier_count', 0)
        adj_pos = pos_words * (1 + intensifier_count * 0.2)
        adj_neg = neg_words * (1 + intensifier_count * 0.2)
        if (adj_pos + adj_neg) > 0:
            scores.append((adj_pos / (adj_pos + adj_neg)) * 4.0 + 1.0)

        domain_pos_words = features.get('domain_positive_count', 0)
        domain_neg_words = features.get('domain_negative_count', 0)
        if (domain_pos_words + domain_neg_words) > 0:
            scores.append((domain_pos_words / (domain_pos_words + domain_neg_words)) * 4.0 + 1.0)

        negation_count = features.get('negation_count', 0)
        base_diff = pos_words - neg_words
        if negation_count > 0 and base_diff != 0:
             base_diff = -base_diff
        scores.append((np.tanh(base_diff * 0.2) + 1.0) * 2.0 + 1.0)

        return max(1, min(5, round(np.mean(scores)))) if scores else 3

    @staticmethod
    def ngram_enhanced_to_stars(features: Dict[str, Any]) -> int:
        """N-gram enhanced sentiment analysis model."""
        for star_val in [5, 4, 3, 2, 1]:
            if features.get(f'mentions_{star_val}_star', False):
                return star_val

        base_score = features.get('vader_compound', 0.0) * 0.5 + \
                     features.get('textblob_polarity', 0.0) * 0.3

        ngram_sentiment_contribution = 0.0
        total_ngram_weight = 0

        for n, weight_multiplier in zip([2, 3], [0.5, 0.5]): # Use 2,3-grams as per feature extraction
            ngram_pos = features.get(f'{n}gram_positive_count', 0)
            ngram_neg = features.get(f'{n}gram_negative_count', 0)

            if ngram_pos + ngram_neg > 0:
                ngram_type_sentiment = (ngram_pos - ngram_neg) / (ngram_pos + ngram_neg)
                ngram_sentiment_contribution += ngram_type_sentiment * weight_multiplier
                total_ngram_weight += weight_multiplier

        if total_ngram_weight > 0:
            normalized_ngram_score = ngram_sentiment_contribution / total_ngram_weight
            final_score = base_score * 0.7 + normalized_ngram_score * 0.3
        else:
            final_score = base_score

        if final_score <= -0.6:
            return 1
        elif final_score <= -0.2:
            return 2
        elif final_score < 0.2:
            return 3
        elif final_score < 0.6:
            return 4
        else:
            return 5

    @staticmethod
    def advanced_ensemble_prediction(predictions: Dict[str, int], features: Dict[str, Any] = None,
                                   weights: Dict[str, float] = None) -> int:
        """Advanced ensemble model with adaptive weighting and outlier detection."""
        if weights is None:
            weights = {
                'vader': 0.15, 'textblob': 0.10, 'feature_based': 0.15,
                'multidimensional': 0.10, 'rule_based_enhanced': 0.08,
                'hybrid_weighted': 0.08, 'lexicon_ensemble': 0.07,
                'ngram_enhanced': 0.07,
                'roberta_sentiment': 0.20
            }

        valid_preds = {k: v for k, v in predictions.items()
                       if pd.notna(v) and k in weights and 1 <= v <= 5}
        if not valid_preds: return 3

        pred_values = list(valid_preds.values())
        if len(pred_values) >= 3:
            median_pred = np.median(pred_values)
            std_pred = np.std(pred_values)
            if std_pred > 0.5:
                filtered_preds_temp = {k: v for k, v in valid_preds.items()
                                       if abs(v - median_pred) <= max(1, 1.5 * std_pred)}
                if len(filtered_preds_temp) >= max(1, len(valid_preds) * 0.5):
                     valid_preds = filtered_preds_temp

        adapted_weights = {k: weights[k] for k in valid_preds if k in weights}

        if features:
            if 'vader' in adapted_weights and \
               (features.get('caps_ratio', 0.0) > 0.05 or features.get('exclamation_count', 0) > 0):
                adapted_weights['vader'] *= 1.2

            domain_words_count = features.get('domain_positive_count', 0) + features.get('domain_negative_count', 0)
            if domain_words_count > 2:
                for m_key in ['multidimensional', 'rule_based_enhanced']:
                    if m_key in adapted_weights: adapted_weights[m_key] *= 1.1

            if 'roberta_sentiment' in adapted_weights and features.get('word_count',0) > 20 and features.get('sentence_count',0) > 1:
                 adapted_weights['roberta_sentiment'] *= 1.2

        total_initial_weight = sum(adapted_weights.values())
        if total_initial_weight > 0:
            adapted_weights = {k: v / total_initial_weight for k, v in adapted_weights.items()}
        else:
            adapted_weights = {k: 1.0/len(valid_preds) for k in valid_preds}

        weighted_sum_final, total_weight_final = 0.0, 0.0
        for model_key, pred_val in valid_preds.items():
            if model_key in adapted_weights:
                weight_val = adapted_weights[model_key]
                weighted_sum_final += pred_val * weight_val
                total_weight_final += weight_val

        if total_weight_final == 0:
            return round(np.mean(list(valid_preds.values()))) if valid_preds else 3

        final_pred_float = weighted_sum_final / total_weight_final

        if len(valid_preds) >= 2:
            pred_std_dev_after_filter = np.std(list(valid_preds.values()))
            if pred_std_dev_after_filter > 1.0:
                 final_pred_float = (final_pred_float * 0.7) + (np.median(list(valid_preds.values())) * 0.3)

        return max(1, min(5, round(final_pred_float)))

#%% === ASPECT-BASED SENTIMENT ANALYSIS ===
class AspectBasedSentimentAnalyzer:
    def __init__(self, results_dir: str, logger: Optional[logging.Logger] = None):
        self.results_dir = results_dir
        self.absa_results_dir = os.path.join(results_dir, 'absa_analysis')
        os.makedirs(self.absa_results_dir, exist_ok=True)
        self.logger = logger or logging.getLogger(__name__)

        self.absa_pipeline = None
        self.use_rule_based_absa_fallback = False # Flag for fallback

        # Try to initialize NLTK sentence tokenizer first, as it's needed by both methods
        self.sent_tokenizer_nltk = None
        if NLTK_INITIALIZED:
            try:
                from nltk.tokenize import sent_tokenize
                self.sent_tokenizer_nltk = sent_tokenize
                self.logger.info("NLTK sent_tokenize loaded for ABSA.")
            except ImportError:
                self.logger.warning("NLTK sent_tokenize not available. ABSA might be impaired.")
        else:
            self.logger.warning("NLTK not initialized. NLTK sent_tokenize not available for ABSA.")

        if transformers_available:
            try:
                model_name = "yangheng/deberta-v3-base-absa-v1.1"
                self.absa_pipeline = pipeline("text-classification", model=model_name, tokenizer=model_name, device=TRANSFORMER_DEVICE, return_all_scores=False)
                self.logger.info(f"ABSA Transformer model '{model_name}' loaded on {TRANSFORMER_DEVICE}.")
            except Exception as e:
                self.logger.error(f"Failed to load ABSA Transformer model: {e}. Will attempt rule-based ABSA fallback.")
                self.use_rule_based_absa_fallback = True
        else:
            self.logger.warning("Transformers library not available. Will attempt rule-based ABSA fallback.")
            self.use_rule_based_absa_fallback = True

        if self.use_rule_based_absa_fallback:
            if not textproc_available:
                self.logger.error("TextBlob not available, cannot use rule-based ABSA fallback. ABSA will be skipped if Transformer also failed.")
            else:
                self.logger.info("Rule-based ABSA (with TextBlob) will be used as a fallback if needed.")

    def get_aspect_keywords_for_category(self, category: str) -> Set[str]:
        """FIXED: Get category-specific aspects without mixing electronics terms into other categories"""
        aspects = set()
        
        # Add category-specific domain words (positive + negative)
        if category != 'unknown' and category in DOMAIN_WORDS: 
            aspects.update(DOMAIN_WORDS[category].get('positive', set()) | DOMAIN_WORDS[category].get('negative', set()))
        
        # Add generic specs that apply to all product categories
        aspects.update(PRODUCT_SPEC_WORDS.get('generic_specs', set()))
        
        # Add category-specific specs only for the matching category
        category_spec_key = f"{category}_specs"
        if category_spec_key in PRODUCT_SPEC_WORDS: 
            aspects.update(PRODUCT_SPEC_WORDS[category_spec_key])
        
        # Filter out very short words that might cause false matches
        return {asp for asp in aspects if len(asp) > 2}

    def analyze_review(self, review_text: str, category: str = 'unknown') -> List[Dict[str, Any]]:
        if not review_text or pd.isna(review_text): return []

        if not self.sent_tokenizer_nltk:
            self.logger.warning("NLTK Sentence tokenizer not available. Cannot perform ABSA for this review.")
            return []

        aspect_sentiments = []
        try:
            sentences = self.sent_tokenizer_nltk(review_text)
        except Exception as e:
            self.logger.error(f"Error tokenizing sentences for review: '{review_text[:50]}...': {e}")
            return []

        candidate_aspects = self.get_aspect_keywords_for_category(category)
        if not candidate_aspects: return []

        current_method_is_transformer = self.absa_pipeline and not self.use_rule_based_absa_fallback
        current_method_is_rule_based = self.use_rule_based_absa_fallback and textproc_available

        if current_method_is_transformer:
            cleaned_sentences_for_transformer = [re.sub(r'\s+', ' ', s.lower().strip()) for s in sentences]
            for sent_idx, sentence in enumerate(cleaned_sentences_for_transformer):
                if not sentence: continue
                mentioned_aspects = {asp for asp in candidate_aspects if re.search(r'\b'+re.escape(asp)+r'\b', sentence)}
                if not mentioned_aspects: continue
                for aspect_term in mentioned_aspects:
                    input_text = f"{sentence} [SEP] {aspect_term}"
                    try:
                        with warnings.catch_warnings(): warnings.simplefilter("ignore"); result = self.absa_pipeline(input_text, truncation=True, max_length=512)
                        if result and isinstance(result,list) and result[0]:
                             aspect_sentiments.append({
                                 'aspect':aspect_term, 'sentiment':result[0]['label'].lower(),
                                 'score':float(result[0]['score']), 'context':sentences[sent_idx].strip(),
                                 'method': 'transformer_deberta'})
                    except Exception as e: self.logger.error(f"Error ABSA Transformer pred for '{aspect_term}' in '{sentence[:50]}...': {e}")

        elif current_method_is_rule_based:
            for sent_idx, sentence_text in enumerate(sentences):
                cleaned_sentence = re.sub(r'\s+', ' ', sentence_text.lower().strip())
                if not cleaned_sentence: continue
                mentioned_aspects = {asp for asp in candidate_aspects if re.search(r'\b' + re.escape(asp) + r'\b', cleaned_sentence)}
                if not mentioned_aspects: continue
                try:
                    blob = TextBlob(sentence_text)
                    polarity = blob.sentiment.polarity
                    sentiment_label = "positive" if polarity > 0.1 else "negative" if polarity < -0.1 else "neutral"
                    for aspect_term in mentioned_aspects:
                        aspect_sentiments.append({
                            'aspect': aspect_term, 'sentiment': sentiment_label,
                            'score': abs(polarity), 'context': sentences[sent_idx].strip(),
                            'method': 'rule-based_textblob'
                        })
                except Exception as e:
                    self.logger.error(f"Error during rule-based ABSA for sentence '{sentence_text[:50]}...': {e}")
        return aspect_sentiments

    def batch_analyze_reviews(self, reviews: List[str], categories: List[str], batch_size: int = 10) -> List[List[Dict[str, Any]]]:
        can_do_transformer_absa = self.absa_pipeline and not self.use_rule_based_absa_fallback
        can_do_rule_based_absa = self.use_rule_based_absa_fallback and textproc_available and self.sent_tokenizer_nltk

        if not (can_do_transformer_absa or can_do_rule_based_absa):
            self.logger.warning("No viable ABSA method (Transformer or rule-based fallback). Skipping batch ABSA.")
            return [[] for _ in reviews]

        all_results = []; p_bar_desc = "ABSA Batch Processing"; p_bar = tqdm(total=len(reviews), desc=p_bar_desc, unit="reviews") if tqdm_available and len(reviews) > batch_size else None
        for i in range(0, len(reviews), batch_size):
            batch_rev, batch_cat = reviews[i:min(i+batch_size,len(reviews))], categories[i:min(i+batch_size,len(reviews))]
            for rev_txt, cat in zip(batch_rev, batch_cat):
                all_results.append(self.analyze_review(rev_txt, cat))
            if p_bar: p_bar.update(len(batch_rev))
        if p_bar: p_bar.close()
        return all_results

    def generate_absa_summary_visualizations(self, all_aspect_data: List[List[Dict[str, Any]]], categories: Optional[List[str]] = None) -> None:
        if not all_aspect_data or not matplotlib_available: self.logger.warning("No ABSA data or matplotlib for visualization."); return
        flat_aspect_list = [item for sublist in all_aspect_data for item in sublist]
        if not flat_aspect_list: self.logger.info("No aspects extracted for ABSA summary."); return
        df_aspects = pd.DataFrame(flat_aspect_list)
        if not df_aspects.empty:
            self._plot_aspect_sentiment_distribution(df_aspects, "Overall", "overall_aspect_sentiments.png")
            if categories and len(categories) == len(all_aspect_data):
                review_indices = [i for i, sublist in enumerate(all_aspect_data) for _ in sublist]
                if review_indices:
                    df_aspects['review_category'] = [categories[i] for i in review_indices]
                    top_n_cats = df_aspects['review_category'].value_counts().nlargest(5).index
                    for cat in top_n_cats:
                        if cat == 'unknown': continue
                        df_cat_aspects = df_aspects[df_aspects['review_category']==cat]
                        if not df_cat_aspects.empty: self._plot_aspect_sentiment_distribution(df_cat_aspects, f"Category: {cat.title()}", f"category_{cat}_aspect_sentiments.png")
            summary_path = os.path.join(self.absa_results_dir, "absa_summary_data.csv")
            try: df_aspects.to_csv(summary_path, index=False); self.logger.info(f"Aggregated ABSA data saved to {summary_path}")
            except Exception as e: self.logger.error(f"Failed to save aggregated ABSA data: {e}")
        self.logger.info("ABSA summary visualizations generated.")

    def _plot_aspect_sentiment_distribution(self, df_aspects: pd.DataFrame, title_prefix: str, filename: str):
        if df_aspects.empty: return
        top_aspects = df_aspects['aspect'].value_counts().nlargest(15).index
        df_top_aspects = df_aspects[df_aspects['aspect'].isin(top_aspects)]
        if df_top_aspects.empty: return
        aspect_sentiment_counts = df_top_aspects.groupby(['aspect', 'sentiment']).size().unstack(fill_value=0)
        for col in ['positive','negative','neutral']:
            if col not in aspect_sentiment_counts.columns: aspect_sentiment_counts[col]=0
        cols_in_order = [c for c in ['positive','neutral','negative'] if c in aspect_sentiment_counts.columns]
        aspect_sentiment_counts = aspect_sentiment_counts[cols_in_order]

        fig, ax = plt.subplots(figsize=(12, max(8, len(top_aspects)*0.5)))
        try:
            aspect_sentiment_counts.plot(kind='barh', stacked=True, color={'positive':'mediumseagreen','neutral':'lightskyblue','negative':'lightcoral'}, ax=ax)
            ax.set_title(f'{title_prefix} - Top Aspect Sentiment Distribution', fontsize=15); ax.set_xlabel('Number of Mentions',fontsize=12); ax.set_ylabel('Aspect Term',fontsize=12)
            ax.legend(title='Sentiment',loc='lower right'); ax.invert_yaxis(); plt.tight_layout()
            save_path = os.path.join(self.absa_results_dir, filename); plt.savefig(save_path, dpi=300)
            if is_jupyter_environment(): plt.show()
            self.logger.info(f"Saved ABSA plot: {save_path}")
        except Exception as e: self.logger.error(f"Error plotting ABSA for {title_prefix}: {e}")
        finally: plt.close(fig)

#%% === WORST PREDICTION ANALYZER ===
class WorstPredictionAnalyzer:
    """Analyzes worst predictions, outputs static DataFrames, and lists full review texts."""
    def __init__(self, results_dir: str, logger: Optional[logging.Logger] = None):
        self.results_dir = results_dir
        os.makedirs(self.results_dir, exist_ok=True)
        self.worst_predictions_data: Dict[str, List[Dict[str, Any]]] = {}
        self.pipeline_ref = None
        self.logger = logger or logging.getLogger(__name__)

    def save_worst_predictions_to_csv(self, model_name: str) -> None:
        if model_name not in self.worst_predictions_data or not self.worst_predictions_data[model_name]: return
        df_worst = pd.DataFrame(self.worst_predictions_data[model_name])
        cols_to_save = ['original_data_index','true_rating','predicted_rating','error','text_preview','vader_compound','textblob_polarity','custom_score','word_count','transformer_label','transformer_score']
        # Adjust 'original_data_index' to 'index' if that's the actual key in current_model_worst_data
        if 'index' in df_worst.columns and 'original_data_index' not in df_worst.columns:
            cols_to_save[0] = 'index' # Use 'index' if 'original_data_index' is missing

        df_worst_filt = df_worst[[col for col in cols_to_save if col in df_worst.columns]]
        safe_name = model_name.lower().replace(' ','_').replace('-','_')
        filename = os.path.join(self.results_dir, f"{safe_name}_worst_predictions.csv")
        try: df_worst_filt.to_csv(filename, index=False, encoding='utf-8'); self.logger.info(f"Worst predictions CSV for {model_name} saved to {filename}")
        except Exception as e: self.logger.error(f"Error saving worst predictions CSV for {model_name}: {e}")

    def plot_worst_prediction_visuals(self, worst_preds_df: pd.DataFrame, model_name: str):
        self.logger.info(f"Visualizations for worst predictions for model '{model_name}' are now replaced by direct listing of full reviews.")
        pass # Visualizations are intentionally not generated as per Request 1

    def analyze_and_display_worst_predictions(self, model_name_single: str,
                                             y_true_model: np.ndarray,
                                             y_pred_model: np.ndarray,
                                             original_texts_for_model: List[str],
                                             features_df_for_model: pd.DataFrame,
                                             valid_indices_mask_for_model: Optional[np.ndarray] = None, # Not directly used if data is pre-filtered
                                             transformer_predictions_df_for_model: Optional[pd.DataFrame] = None,
                                             top_n: int = 50,  # FIXED: Changed from 100 to 50
                                             min_error_threshold: int = 1
                                             ) -> None:
        if len(y_true_model) == 0:
            self.logger.warning(f"\nNo valid predictions to analyze for worst cases in {model_name_single}.")
            return

        errors = np.abs(y_true_model - y_pred_model)
        current_model_worst_data = []
        # These indices are relative to the y_true_model, y_pred_model, etc. passed to this function
        # which are already filtered by valid_mask in ModelEvaluator.evaluate_model
        sorted_indices_relative_to_current_arrays = np.argsort(errors)[::-1]

        for i_worst in range(min(top_n, len(sorted_indices_relative_to_current_arrays))):
            idx_in_current_arrays = sorted_indices_relative_to_current_arrays[i_worst]

            if errors[idx_in_current_arrays] < min_error_threshold:
                break

            text_to_use = original_texts_for_model[idx_in_current_arrays] if idx_in_current_arrays < len(original_texts_for_model) else "N/A Text"
            
            # 'index' here will be the index within the *current filtered arrays*
            item_data = {
                'index': idx_in_current_arrays,
                'true_rating': int(y_true_model[idx_in_current_arrays]),
                'predicted_rating': int(y_pred_model[idx_in_current_arrays]),
                'error': int(errors[idx_in_current_arrays]),
                'text_preview': text_to_use,
            }

            if not features_df_for_model.empty and idx_in_current_arrays < len(features_df_for_model):
                features_row = features_df_for_model.iloc[idx_in_current_arrays]
                item_data.update({
                    'vader_compound': f"{features_row.get('vader_compound', 'N/A'):.3f}" if isinstance(features_row.get('vader_compound'), (float, np.floating)) else features_row.get('vader_compound', 'N/A'),
                    'textblob_polarity': f"{features_row.get('textblob_polarity', 'N/A'):.3f}" if isinstance(features_row.get('textblob_polarity'), (float, np.floating)) else features_row.get('textblob_polarity', 'N/A'),
                    'custom_score': f"{features_row.get('custom_sentiment_score', 'N/A'):.3f}" if isinstance(features_row.get('custom_sentiment_score'), (float, np.floating)) else features_row.get('custom_sentiment_score', 'N/A'),
                    'word_count': features_row.get('word_count', 'N/A')
                })

            if "RoBERTa Sentiment" in model_name_single and \
               transformer_predictions_df_for_model is not None and \
               idx_in_current_arrays < len(transformer_predictions_df_for_model):
                transformer_row = transformer_predictions_df_for_model.iloc[idx_in_current_arrays]
                item_data['transformer_label'] = transformer_row.get('roberta_sentiment_raw_label', 'N/A')
                item_data['transformer_score'] = f"{transformer_row.get('roberta_sentiment_raw_score', 'N/A'):.3f}" if isinstance(transformer_row.get('roberta_sentiment_raw_score'), (float, np.floating)) else transformer_row.get('roberta_sentiment_raw_score', 'N/A')

            current_model_worst_data.append(item_data)

        self.worst_predictions_data[model_name_single] = current_model_worst_data
        self.save_worst_predictions_to_csv(model_name_single) # CSV saving is preserved

        if not current_model_worst_data:
            self.logger.info(f"\nNo predictions found with error >= {min_error_threshold} for {model_name_single}.")
            return

        # USER REQUEST 1 (MODIFIED): Display as Pandas DataFrame table
        print(f"\n--- Top {len(current_model_worst_data)} Worst Predictions for Model: {model_name_single} (Error >= {min_error_threshold}) ---")
        
        df_full_worst_predictions = pd.DataFrame(current_model_worst_data)
        
        # Define column order for display
        cols_order_display = ['index', 'true_rating', 'predicted_rating', 'error', 
                              'vader_compound', 'textblob_polarity', 'custom_score', 'word_count']
        if 'transformer_label' in df_full_worst_predictions.columns:
            cols_order_display.extend(['transformer_label', 'transformer_score'])
        cols_order_display.append('text_preview') # Full text at the end for readability

        # Ensure only existing columns are selected and in the desired order
        df_full_worst_predictions_display = df_full_worst_predictions[[col for col in cols_order_display if col in df_full_worst_predictions.columns]]

        with pd.option_context('display.max_colwidth', None, 'display.width', 1000): # Allow full text display
             safe_display(df_full_worst_predictions_display)
        print(f"--- End of Worst Predictions Table for {model_name_single} ---")
        
        # Visualizations are no longer generated here
        # self.plot_worst_prediction_visuals(pd.DataFrame(current_model_worst_data), model_name_single)

#%% === ENHANCED MODEL EVALUATOR ===
class ModelEvaluator:
    def __init__(self, results_dir: Optional[str] = None, logger: Optional[logging.Logger] = None, pipeline_instance: Optional[Any] = None):
        self.results_dir = results_dir or '.'; self.logger = logger or logging.getLogger(__name__)
        self.pipeline = pipeline_instance
        self.worst_prediction_analyzer = WorstPredictionAnalyzer(os.path.join(self.results_dir, "worst_predictions_analysis"), self.logger)
        if self.pipeline:
            self.worst_prediction_analyzer.pipeline_ref = self.pipeline


    def calculate_metrics(self, y_true: List[int], y_pred: List[int]) -> Dict[str, float]:
        if not y_true or not y_pred: return {}
        y_true_arr, y_pred_arr = np.array(y_true), np.array(y_pred)
        valid_mask = ~np.isnan(y_pred_arr) & ~np.isinf(y_pred_arr)
        if not np.any(valid_mask): return {}
        y_true_filt, y_pred_filt = y_true_arr[valid_mask], y_pred_arr[valid_mask]
        try: y_pred_clamped = np.clip(y_pred_filt.astype(int),1,5)
        except ValueError: self.logger.warning("Could not cast preds to int for clamping."); y_pred_clamped = np.clip(pd.to_numeric(y_pred_filt,errors='coerce').fillna(3).astype(int),1,5)
        if len(y_true_filt)==0: return {}
        accuracy = np.mean(y_true_filt==y_pred_clamped); close_accuracy = np.mean(np.abs(y_true_filt-y_pred_clamped)<=1)
        mae = mean_absolute_error(y_true_filt,y_pred_clamped); rmse = np.sqrt(mean_squared_error(y_true_filt,y_pred_clamped))
        unique_labels = sorted(np.unique(np.concatenate((y_true_filt,y_pred_clamped)))); unique_labels = unique_labels or [1,2,3,4,5]
        f1_macro = f1_score(y_true_filt,y_pred_clamped,average='macro',zero_division=0,labels=unique_labels)
        f1_weighted = f1_score(y_true_filt,y_pred_clamped,average='weighted',zero_division=0,labels=unique_labels)
        precision_macro = precision_score(y_true_filt,y_pred_clamped,average='macro',zero_division=0,labels=unique_labels)
        recall_macro = recall_score(y_true_filt,y_pred_clamped,average='macro',zero_division=0,labels=unique_labels)
        kappa = cohen_kappa_score(y_true_filt,y_pred_clamped,weights=None)
        f1_close = f1_score(np.ones_like(y_true_filt),(np.abs(y_true_filt-y_pred_clamped)<=1).astype(int),zero_division=0,pos_label=1)
        return {'accuracy':accuracy,'close_accuracy':close_accuracy,'mae':mae,'rmse':rmse,'f1_macro':f1_macro,'f1_weighted':f1_weighted,'f1_close':f1_close,'precision_macro':precision_macro,'recall_macro':recall_macro,'kappa':kappa,'sample_size':len(y_true_filt)}

    def plot_confusion_matrix(self, y_true: List[int], y_pred: List[int], model_name: str, metrics: Dict[str, float]) -> None:
        if not matplotlib_available: self.logger.warning(f"Matplotlib unavailable for CM: {model_name}"); return
        y_true_arr, y_pred_arr = np.array(y_true), np.array(y_pred)
        valid_mask = ~np.isnan(y_pred_arr)&~np.isinf(y_pred_arr)
        if not np.any(valid_mask): self.logger.warning(f"No valid preds for CM: {model_name}"); return
        y_true_filt,y_pred_filt = y_true_arr[valid_mask], y_pred_arr[valid_mask]
        try: y_pred_clamped = np.clip(y_pred_filt.astype(int),1,5)
        except ValueError: y_pred_clamped = np.clip(pd.to_numeric(y_pred_filt,errors='coerce').fillna(3).astype(int),1,5)
        if len(y_true_filt)==0: self.logger.warning(f"Zero valid preds for CM: {model_name}"); return
        labels_cm=[1,2,3,4,5]; cm_vals = confusion_matrix(y_true_filt,y_pred_clamped,labels=labels_cm)
        cm_norm = cm_vals.astype('float')/np.maximum(1,cm_vals.sum(axis=1,keepdims=True))
        annot_txt = np.empty_like(cm_vals,dtype=object)
        for i in range(cm_vals.shape[0]):
            for j in range(cm_vals.shape[1]): annot_txt[i,j] = f"{cm_vals[i,j]}\n({cm_norm[i,j]*100:.1f}%)"
        fig_cm=None
        try:
            fig_cm,ax_cm = plt.subplots(figsize=(12,10))
            sns.heatmap(cm_vals,annot=annot_txt,fmt="",cmap='Blues',xticklabels=[f'{i}S' for i in labels_cm],yticklabels=[f'{i}S' for i in labels_cm],cbar_kws={'label':'# Reviews'},linewidths=0.5,linecolor='gray',ax=ax_cm)
            for i_cm in range(len(labels_cm)):
                ax_cm.add_patch(plt.Rectangle((i_cm,i_cm),1,1,fill=False,edgecolor='green',lw=3))
                if i_cm>0: ax_cm.add_patch(plt.Rectangle((i_cm-1,i_cm),1,1,fill=False,edgecolor='orange',lw=2,ls='--'))
                if i_cm<len(labels_cm)-1: ax_cm.add_patch(plt.Rectangle((i_cm+1,i_cm),1,1,fill=False,edgecolor='orange',lw=2,ls='--'))
            from matplotlib.lines import Line2D
            legend_elements = [Line2D([0],[0],color='green',lw=3,label='Correct Prediction'), Line2D([0],[0],color='orange',lw=2,linestyle='--',label='+/-1 Star Error')]
            ax_cm.legend(handles=legend_elements,loc='upper left',bbox_to_anchor=(1.02,1))
            title = f'{model_name} - Confusion Matrix\nAcc: {metrics.get("accuracy",0.0)*100:.1f}% | +/-1 Acc: {metrics.get("close_accuracy",0.0)*100:.1f}% | MAE: {metrics.get("mae",0.0):.3f}'
            ax_cm.set_title(title,fontsize=14,fontweight='bold',pad=20); ax_cm.set_xlabel('Predicted Rating',fontsize=12); ax_cm.set_ylabel('True Rating',fontsize=12)
            plt.tight_layout()
            if self.results_dir: plt.savefig(os.path.join(self.results_dir,f"{model_name.lower().replace(' ','_').replace('-','_')}_confusion_matrix.png"),dpi=300,bbox_inches='tight')
            if is_jupyter_environment(): plt.show()
        except Exception as e_plot: self.logger.error(f"Error plotting CM for {model_name}: {e_plot}",exc_info=True); print(f" Error plotting CM for {model_name}: {e_plot}")
        finally:
            if fig_cm: plt.close(fig_cm)

    def plot_rating_distribution(self, y_true: List[int], y_pred: List[int], model_name: str, metrics: Dict[str, float]) -> None:
        if not matplotlib_available: self.logger.warning(f"Matplotlib unavailable for rating dist: {model_name}"); return
        y_true_arr, y_pred_arr = np.array(y_true), np.array(y_pred)
        valid_mask = ~np.isnan(y_pred_arr)&~np.isinf(y_pred_arr)
        if not np.any(valid_mask): self.logger.warning(f"No valid preds for rating dist plot: {model_name}"); return
        y_true_filt,y_pred_filt = y_true_arr[valid_mask], y_pred_arr[valid_mask]
        try: y_pred_clamped = np.clip(y_pred_filt.astype(int),1,5)
        except ValueError: y_pred_clamped = np.clip(pd.to_numeric(y_pred_filt,errors='coerce').fillna(3).astype(int),1,5)
        if len(y_true_filt)==0: self.logger.warning(f"Zero valid preds for rating dist plot: {model_name}"); return
        rating_levels=list(range(1,6)); true_counts=pd.Series(y_true_filt).value_counts().reindex(rating_levels,fill_value=0); pred_counts=pd.Series(y_pred_clamped).value_counts().reindex(rating_levels,fill_value=0)
        fig_dist=None
        try:
            fig_dist,ax_dist = plt.subplots(figsize=(12,7)); x_indices=np.arange(len(rating_levels)); bar_width=0.35
            rects1 = ax_dist.bar(x_indices-bar_width/2,true_counts.values,bar_width,label='True Ratings',color='deepskyblue',edgecolor='navy')
            rects2 = ax_dist.bar(x_indices+bar_width/2,pred_counts.values,bar_width,label=f'{model_name} Predictions',color='lightcoral',edgecolor='darkred')
            max_count_val = 1
            if not true_counts.empty or not pred_counts.empty:
                 max_count_val = max(true_counts.max() if not true_counts.empty else 0,
                                   pred_counts.max() if not pred_counts.empty else 0, 1)


            for rg in [rects1,rects2]:
                for rect in rg:
                    h=rect.get_height()
                    if h>0: ax_dist.text(rect.get_x()+rect.get_width()/2.,h+max_count_val*0.01,f'{int(h)}',ha='center',va='bottom',fontsize=9)
            ax_dist.set_xlabel('Star Rating',fontsize=12); ax_dist.set_ylabel('Number of Reviews',fontsize=12)
            title = f'{model_name} - Rating Distribution Comparison\n(Sample Size: {metrics.get("sample_size",0):,})'
            ax_dist.set_title(title,fontsize=14,fontweight='bold',pad=15); ax_dist.set_xticks(x_indices); ax_dist.set_xticklabels([f'{i}S' for i in rating_levels])
            ax_dist.legend(fontsize=10); ax_dist.grid(axis='y',alpha=0.5,linestyle='--'); plt.tight_layout()
            if self.results_dir: plt.savefig(os.path.join(self.results_dir,f"{model_name.lower().replace(' ','_').replace('-','_')}_rating_distribution.png"),dpi=300,bbox_inches='tight')
            if is_jupyter_environment(): plt.show()
        except Exception as e_plot: self.logger.error(f"Error plotting rating dist for {model_name}: {e_plot}",exc_info=True); print(f" Error plotting rating dist for {model_name}: {e_plot}")
        finally:
            if fig_dist: plt.close(fig_dist)

    def plot_radar_chart(self, model_results: Dict[str, Dict[str, Any]]) -> None:
        if not matplotlib_available or not model_results: self.logger.warning("Matplotlib or model results unavailable for radar chart."); return
        print("Generating enhanced radar chart for model comparison...")
        metrics_radar=['accuracy','close_accuracy','f1_macro','f1_weighted','f1_close','precision_macro','recall_macro','kappa','mae','rmse']
        labels_radar=['Exact Acc.','+/-1 Acc.','F1 Macro','F1 Wgt.','F1 (+/-1 Acc.)','Precision (M)','Recall (M)','Kappa','Inv. MAE','Inv. RMSE']
        higher_is_better=[True,True,True,True,True,True,True,True,False,False]
        models_list = list(model_results.keys())
        if not models_list: self.logger.warning("No models for radar chart."); return
        data_all_metrics = defaultdict(list)
        for model_name in models_list:
            for metric_name in metrics_radar: data_all_metrics[metric_name].append(model_results[model_name].get('metrics',{}).get(metric_name,np.nan))
        normalized_data = defaultdict(list)
        for idx,metric_name in enumerate(metrics_radar):
            metric_vals = [v for v in data_all_metrics[metric_name] if pd.notna(v)]
            if not metric_vals: normalized_data[metric_name]=[0.5]*len(models_list); continue
            min_v,max_v = min(metric_vals),max(metric_vals)
            for oval in data_all_metrics[metric_name]:
                if pd.isna(oval): nval=0.5
                elif max_v==min_v: nval = 0.5 if min_v==0 else (1.0 if higher_is_better[idx] else 0.0)
                else: nval = (oval-min_v)/(max_v-min_v)
                if metric_name=='kappa': nval=(oval+1.0)/2.0 if pd.notna(oval) else 0.5
                elif not higher_is_better[idx] and pd.notna(oval): nval = 1.0-nval
                normalized_data[metric_name].append(np.clip(nval,0,1))
        plot_vals_list = [[normalized_data[m][i] for m in metrics_radar] for i in range(len(models_list))]
        num_vars = len(metrics_radar); angles = np.linspace(0,2*np.pi,num_vars,endpoint=False).tolist(); angles+=angles[:1]
        fig_radar=None
        try:
            fig_radar,ax_radar = plt.subplots(figsize=(14,14),subplot_kw=dict(projection='polar'))
            colors = sns.color_palette("husl",len(models_list))
            for i,model_name in enumerate(models_list):
                model_plot_vals = plot_vals_list[i]+[plot_vals_list[i][0]]
                ax_radar.plot(angles,model_plot_vals,'o-',linewidth=2,markersize=6,label=model_name,color=colors[i])
                ax_radar.fill(angles,model_plot_vals,alpha=0.25,color=colors[i])
            ax_radar.set_xticks(angles[:-1]); ax_radar.set_xticklabels(labels_radar,fontsize=10)
            ax_radar.set_ylim(0,1); ax_radar.set_yticks(np.arange(0,1.1,0.2)); ax_radar.set_yticklabels([f"{t:.1f}" for t in np.arange(0,1.1,0.2)],fontsize=9)
            ax_radar.grid(True,linestyle='--',alpha=0.7); ax_radar.set_facecolor('#f9f9f9')
            plt.title('Normalized Model Performance Comparison\n(Higher score is better for all axes)',fontsize=16,fontweight='bold',pad=30)
            ax_radar.legend(loc='upper right',bbox_to_anchor=(1.35,1.15),fontsize=11,frameon=True,fancybox=True,shadow=True)
            plt.tight_layout()
            if self.results_dir: plt.savefig(os.path.join(self.results_dir,'model_performance_radar_chart.png'),dpi=300,bbox_inches='tight')
            if is_jupyter_environment(): plt.show()
        except Exception as e_plot: self.logger.error(f"Error plotting radar chart: {e_plot}",exc_info=True); print(f" Error plotting radar chart: {e_plot}")
        finally:
            if fig_radar: plt.close(fig_radar)

    def create_interactive_model_comparison_dashboard(self, model_results: Dict[str, Any]) -> None:
        if not plotly_available or not model_results: print("Plotly or model results unavailable for dashboard."); return
        print("Creating interactive model comparison dashboard...")
        models_list = list(model_results.keys()); metrics_for_dashboard={}
        for model_iter in models_list:
            metrics = model_results[model_iter].get('metrics',{})
            if metrics: metrics_for_dashboard[model_iter]=metrics
        if not metrics_for_dashboard: print("No valid metrics for dashboard."); return
        models_list = list(metrics_for_dashboard.keys())
        fig = make_subplots(rows=2,cols=2,subplot_titles=('Accuracy Metrics','F1 Scores','Error Metrics (Lower is Better)','Overall Ranking (F1 Close Acc.)'),specs=[[{},{}],[{},{"type":"bar"}]])
        acc_metrics = ['accuracy','close_accuracy']
        for i,metric_name in enumerate(acc_metrics):
            values = [metrics_for_dashboard[model].get(metric_name,0) for model in models_list]
            fig.add_trace(go.Bar(x=models_list,y=values,name=metric_name.replace('_',' ').title(),marker_color=px.colors.qualitative.Plotly[i],text=np.round(values,3),textposition='auto'),row=1,col=1)
        f1_metrics = ['f1_macro','f1_weighted','f1_close']
        for i,metric_name in enumerate(f1_metrics):
            values = [metrics_for_dashboard[model].get(metric_name,0) for model in models_list]
            fig.add_trace(go.Bar(x=models_list,y=values,name=metric_name.replace('_',' ').title(),marker_color=px.colors.qualitative.Pastel[i],text=np.round(values,3),textposition='auto'),row=1,col=2)
        error_metrics = ['mae','rmse']
        for i,metric_name in enumerate(error_metrics):
            values = [metrics_for_dashboard[model].get(metric_name,0) for model in models_list]
            fig.add_trace(go.Bar(x=models_list,y=values,name=metric_name.upper(),marker_color=px.colors.qualitative.Set3[i],text=np.round(values,3),textposition='auto'),row=2,col=1)
        f1_close_scores = {model:metrics_for_dashboard[model].get('f1_close',0) for model in models_list}
        sorted_models = sorted(f1_close_scores.keys(),key=lambda m:f1_close_scores[m],reverse=True)
        sorted_vals = [f1_close_scores[m] for m in sorted_models]
        fig.add_trace(go.Bar(x=sorted_models,y=sorted_vals,name='F1 (+/-1 Star) Ranking',marker_color=px.colors.sequential.Plasma,text=np.round(sorted_vals,3),textposition='auto'),row=2,col=2)
        fig.update_layout(height=900,width=1200,title_text="<b>Interactive Model Performance Dashboard</b>",title_x=0.5,title_font_size=20,showlegend=True,legend_title_text='Metrics',bargap=0.15,bargroupgap=0.1,font=dict(family="Arial, sans-serif",size=10))
        fig.update_traces(texttemplate='%{text}',hoverinfo='x+y+name')
        if is_jupyter_environment(): fig.show()
        html_path = os.path.join(self.results_dir,'interactive_model_dashboard.html')
        try: fig.write_html(html_path); print(f"Interactive dashboard saved to: {html_path}"); self.logger.info(f"Interactive dashboard saved: {html_path}")
        except Exception as e: print(f"Error saving dashboard: {e}"); self.logger.error(f"Failed to save dashboard: {e}")

    def evaluate_model(self, y_true: List[int], y_pred: List[int], model_name: str, prediction_column_name_in_final_df: str, confidence_scores: Optional[List[float]]=None, original_texts_for_analysis: Optional[List[str]]=None, features_df_for_analysis: Optional[pd.DataFrame]=None, transformer_predictions_df: Optional[pd.DataFrame]=None) -> Dict[str,Any]:
        self.logger.info(f"Evaluating model: {model_name} (column: {prediction_column_name_in_final_df})..."); print(f"\nEvaluating model: {model_name}...")
        metrics = self.calculate_metrics(y_true,y_pred)
        if not metrics: self.logger.warning(f"No valid preds/metrics for {model_name}. Skipping."); return {}
        self._display_metrics_table(metrics,model_name); self.plot_confusion_matrix(y_true,y_pred,model_name,metrics); self.plot_rating_distribution(y_true,y_pred,model_name,metrics)

        y_true_arr, y_pred_arr = np.array(y_true), np.array(y_pred)
        valid_mask = ~np.isnan(y_pred_arr) & ~np.isinf(y_pred_arr)

        y_true_valid_for_model = y_true_arr[valid_mask]
        y_pred_valid_for_model = y_pred_arr[valid_mask]
        try: y_pred_valid_int_for_model = y_pred_valid_for_model.astype(int)
        except ValueError: y_pred_valid_int_for_model = pd.to_numeric(y_pred_valid_for_model,errors='coerce').fillna(3).astype(int)

        if original_texts_for_analysis and self.pipeline is not None:
            current_model_original_texts = [original_texts_for_analysis[i] for i, keep in enumerate(valid_mask) if keep] if valid_mask is not None else original_texts_for_analysis
            current_model_features_df = features_df_for_analysis[valid_mask].reset_index(drop=True) if features_df_for_analysis is not None and valid_mask is not None and not features_df_for_analysis.empty else pd.DataFrame()
            current_model_transformer_preds_df = transformer_predictions_df[valid_mask].reset_index(drop=True) if transformer_predictions_df is not None and valid_mask is not None and not transformer_predictions_df.empty else pd.DataFrame()

            self.worst_prediction_analyzer.analyze_and_display_worst_predictions(
                model_name_single=model_name,
                y_true_model=y_true_valid_for_model,
                y_pred_model=y_pred_valid_int_for_model,
                original_texts_for_model=current_model_original_texts,
                features_df_for_model=current_model_features_df,
                valid_indices_mask_for_model=None,
                transformer_predictions_df_for_model=current_model_transformer_preds_df,
                top_n=50,  # FIXED: Changed from 100 to 50
                min_error_threshold=1
            )

        if confidence_scores and len(confidence_scores)==len(y_true_arr) and "RoBERTa" not in model_name and "BERT" not in model_name:
            conf_scores_valid = np.array(confidence_scores)[valid_mask]
            if len(conf_scores_valid)==len(y_true_valid_for_model): self._analyze_confidence(y_true_valid_for_model,y_pred_valid_int_for_model,conf_scores_valid,model_name)
            else: self.logger.warning(f"Length mismatch for confidence scores after filtering in {model_name}.")
        elif confidence_scores and ("RoBERTa" in model_name or "BERT" in model_name): self.logger.info(f"Skipping custom confidence score analysis for Transformer model: {model_name}")

        self.logger.info(f"Evaluation for {model_name} complete."); print(f"Evaluation for {model_name} complete.")
        return {'model_name':model_name,'prediction_column_name_in_final_df':prediction_column_name_in_final_df,'metrics':metrics,'predictions':y_pred_valid_int_for_model.tolist() if len(y_pred_valid_int_for_model)>0 else []}

    def _display_metrics_table(self, metrics: Dict[str, float], model_name: str) -> None:
        metrics_df_data = {'Metric':['Accuracy (Exact)','Accuracy (+/-1 Star)','MAE','RMSE','F1 Score (Macro)','F1 Score (Weighted)','F1 Score (+/-1 Star)','Precision (Macro)','Recall (Macro)',"Cohen's Kappa",'Sample Size'], 'Value':[f"{metrics.get(k,0.0):.4f}" if k!='sample_size' else f"{metrics.get(k,0):,}" for k in ['accuracy','close_accuracy','mae','rmse','f1_macro','f1_weighted','f1_close','precision_macro','recall_macro','kappa','sample_size']]}
        df_metrics = pd.DataFrame(metrics_df_data); print(f"\n--- {model_name} Performance Metrics ---"); safe_display(df_metrics); print("------------------------------------")

    def _analyze_confidence(self, y_true_valid: np.ndarray, y_pred_valid: np.ndarray, confidence_scores_valid: np.ndarray, model_name: str) -> None:
        if not matplotlib_available: self.logger.warning(f"Matplotlib unavailable for confidence plot: {model_name}"); return
        if len(y_true_valid)==0: self.logger.warning(f"No valid data for confidence analysis: {model_name}"); return
        bins_conf = np.array([0.0,0.25,0.5,0.75,0.9,1.01]); labels_conf = ['V.Low (0-.25)','Low (.25-.5)','Med (.5-.75)','High (.75-.9)','V.High (.9-1)']
        results,acc_list,close_acc_list,mae_list,counts_list = [],[],[],[],[]
        for i_conf in range(len(bins_conf)-1):
            mask = (confidence_scores_valid>=bins_conf[i_conf])&(confidence_scores_valid<bins_conf[i_conf+1]); count = np.sum(mask); counts_list.append(count)
            if count > 0:
                acc = np.mean(y_true_valid[mask]==y_pred_valid[mask]); close_acc = np.mean(np.abs(y_true_valid[mask]-y_pred_valid[mask])<=1); mae = mean_absolute_error(y_true_valid[mask],y_pred_valid[mask])
                acc_list.append(acc); close_acc_list.append(close_acc); mae_list.append(mae)
                results.append({'Confidence Level':labels_conf[i_conf],'Num Reviews':count,'Accuracy':f"{acc:.3f}",'Close Acc (+/-1S)':f"{close_acc:.3f}",'MAE':f"{mae:.3f}"})
            else:
                for lst in [acc_list,close_acc_list,mae_list]: lst.append(np.nan)
                results.append({'Confidence Level':labels_conf[i_conf],'Num Reviews':0,'Accuracy':"N/A",'Close Acc (+/-1S)':"N/A",'MAE':"N/A"})
        if results: print(f"\n--- Confidence vs. Performance Analysis for {model_name} ---"); safe_display(pd.DataFrame(results)); print("------------------------------------------------------")
        fig_conf=None
        if matplotlib_available and any(c>0 for c in counts_list):
            fig_conf,ax1 = plt.subplots(figsize=(12,7)); x_indices = np.arange(len(labels_conf))
            try:
                ax1.plot(x_indices,acc_list,'o-g',markerfacecolor='lightgreen',label='Exact Accuracy',ms=8); ax1.plot(x_indices,close_acc_list,'s-b',markerfacecolor='skyblue',label='Close Accuracy (+/-1 Star)',ms=8)
                ax1.set_xlabel('Confidence Level Bin',fontsize=12); ax1.set_ylabel('Accuracy Score',color='k',fontsize=12); ax1.tick_params(axis='y',labelcolor='k')
                ax1.set_xticks(x_indices); ax1.set_xticklabels(labels_conf,rotation=30,ha='right'); ax1.grid(True,linestyle=':',alpha=0.6); ax1.set_ylim(0,1.05)
                ax2 = ax1.twinx(); ax2.plot(x_indices,mae_list,'^-r',markerfacecolor='lightcoral',label='MAE (Lower is Better)',ms=8)
                ax2.set_ylabel('Mean Absolute Error (MAE)',color='r',fontsize=12); ax2.tick_params(axis='y',labelcolor='r')
                for i_bin,count_val in enumerate(counts_list):
                    if count_val>0 and not np.isnan(acc_list[i_bin]): ax1.text(x_indices[i_bin],acc_list[i_bin]+0.02,f'n={count_val}',ha='center',va='bottom',fontsize=9,color='gray')
                lines1,labels1 = ax1.get_legend_handles_labels(); lines2,labels2 = ax2.get_legend_handles_labels()
                ax2.legend(lines1+lines2,labels1+labels2,loc='upper center',bbox_to_anchor=(0.5,-0.18),ncol=3,fancybox=True,shadow=True)
                fig_conf.suptitle(f'Model Performance vs. Custom Confidence Score - {model_name}',fontsize=15,fontweight='bold'); plt.tight_layout(rect=[0,0.1,1,0.95])
                if self.results_dir: plt.savefig(os.path.join(self.results_dir,f'{model_name.lower().replace(" ","_").replace("-","_")}_confidence_analysis.png'),dpi=300,bbox_inches='tight')
                if is_jupyter_environment(): plt.show()
            except Exception as e_plot: self.logger.error(f"Error plotting confidence for {model_name}: {e_plot}",exc_info=True); print(f" Error plotting confidence for {model_name}: {e_plot}")
            finally:
                if fig_conf: plt.close(fig_conf)

#%% === DATA LOADING AND UTILITIES ===
def download_file(url: str, destination: str) -> bool:
    if os.path.exists(destination): print(f"File exists: {destination}. Skipping."); return True
    if not requests_available: print("Requests library not available."); return False
    try:
        print(f"Downloading from {url} to {destination}..."); response = requests.get(url,stream=True,timeout=60); response.raise_for_status()
        total_size = int(response.headers.get('content-length',0)); os.makedirs(os.path.dirname(destination),exist_ok=True)
        global tqdm; dl_desc = f"Downloading {os.path.basename(destination)}"
        with open(destination,'wb') as f_out:
            if total_size>0 and tqdm_available:
                with tqdm(total=total_size,unit='B',unit_scale=True,desc=dl_desc,ncols=80) as p_bar:
                    for chunk in response.iter_content(chunk_size=8192):
                        if chunk: f_out.write(chunk); p_bar.update(len(chunk))
            else:
                if not tqdm_available: print(f"{dl_desc}: Starting (size unknown or tqdm unavailable)...")
                for chunk_num,chunk in enumerate(response.iter_content(chunk_size=8192)):
                    if chunk: f_out.write(chunk)
                    if not tqdm_available and chunk_num%100==0 and chunk_num>0: print(f"{dl_desc}: Downloaded {chunk_num*8192/1024/1024:.2f} MB...")
                if not tqdm_available: print(f"{dl_desc}: Download stream finished.")
        print(f"Download successful: {destination}"); return True
    except requests.exceptions.RequestException as e_req: print(f"Download failed (network/HTTP error): {e_req}")
    except Exception as e_gen: print(f"Download failed (unexpected error): {e_gen}")
    if os.path.exists(destination): os.remove(destination)
    return False

def detect_product_category(text: str) -> str:
    if not isinstance(text,str) or not text.strip(): return 'unknown'
    words_in_text = set(re.findall(r'\b\w{3,}\b',text.lower()))
    best_score, detected_cat = 0, 'unknown'
    for cat_key, domain_data in DOMAIN_WORDS.items():
        all_cat_keywords = domain_data.get('positive',set()) | domain_data.get('negative',set())
        cat_spec_key = f"{cat_key}_specs"
        if cat_spec_key in PRODUCT_SPEC_WORDS: all_cat_keywords.update(PRODUCT_SPEC_WORDS[cat_spec_key])
        current_score = len(words_in_text.intersection(all_cat_keywords))
        if current_score > best_score: best_score,detected_cat = current_score,cat_key
    return detected_cat if best_score>=1 else 'unknown'

def load_and_clean_data(file_path: str, sample_size: Optional[int]=None) -> Tuple[List[str],List[int],List[str]]:
    print(f"Loading and cleaning data from: {file_path}...")
    try:
        df = pd.read_csv(file_path,usecols=['overall','reviewText'],dtype={'overall':'str','reviewText':'str'},on_bad_lines='warn')
        print(f"  Initial raw shape: {df.shape}"); initial_rows = len(df)
        df.dropna(subset=['overall','reviewText'],inplace=True); df['overall'] = pd.to_numeric(df['overall'],errors='coerce'); df.dropna(subset=['overall'],inplace=True)
        df = df[df['overall'].between(1,5)]; df['overall'] = df['overall'].astype(int)
        df['reviewText'] = df['reviewText'].astype(str).str.strip(); df = df[df['reviewText'].str.len()>0]
        cleaned_rows = len(df); print(f"  Retained {cleaned_rows} rows after cleaning (dropped {initial_rows-cleaned_rows}).")
        if sample_size is not None and 0<sample_size<len(df): print(f"  Sampling {sample_size} reviews..."); df=df.sample(n=sample_size,random_state=42).reset_index(drop=True)
        elif sample_size is not None and sample_size>=len(df): print(f"  Sample size ({sample_size}) >= available data ({len(df)}). Using all data.")
        elif sample_size is None: print(f"  Using full dataset ({len(df)} reviews).")

        print(f"  Final data shape: {df.shape}"); texts,ratings = df['reviewText'].tolist(),df['overall'].tolist()
        print("  Detecting product categories..."); categories = []
        global tqdm; cat_iter = tqdm(texts,desc="Detecting categories",unit="review") if tqdm_available else texts
        for idx,text_item in enumerate(cat_iter):
            categories.append(detect_product_category(text_item))
            if not tqdm_available and idx>0 and len(texts)>0 and (idx+1)%max(1,len(texts)//10)==0: print(f"Detecting categories: {idx+1}/{len(texts)}...")
        if tqdm_available and isinstance(cat_iter,tqdm): cat_iter.close()
        elif not tqdm_available and len(texts)>0: print("Detecting categories: Completed!")
        cat_counts=Counter(categories); print("\n  Category distribution:"); total_cats=len(categories)
        for cat,count in cat_counts.most_common(): print(f"    {cat.title()}: {count:,} ({(count/total_cats)*100 if total_cats>0 else 0:.1f}%)")
        print("Data loading and cleaning complete."); return texts,ratings,categories
    except FileNotFoundError: print(f"CRITICAL Error: File not found: {file_path}"); raise
    except pd.errors.EmptyDataError: print(f"CRITICAL Error: File empty: {file_path}"); raise
    except KeyError as e: print(f"CRITICAL Error: Missing column: {e}. Ensure 'overall','reviewText' exist."); raise
    except Exception as e: print(f"CRITICAL Error: Unexpected error loading/cleaning: {e}"); raise

#%% === MAIN PIPELINE ===
class SentimentAnalysisPipeline:
    def __init__(self, results_dir: Optional[str]=None):
        print("Initializing Sentiment Analysis Pipeline with Transformer and ABSA capabilities...")
        if results_dir is None: timestamp = datetime.now().strftime("%Y%m%d_%H%M%S"); results_dir = f"sentiment_results_transformer_absa_{timestamp}"
        os.makedirs(results_dir,exist_ok=True); self.results_dir = os.path.abspath(results_dir)
        self.logger = setup_logging(self.results_dir); self.logger.info("SentimentAnalysisPipeline instance created.")
        self.resources = self._initialize_resources(); self.preprocessor = TextPreprocessor(self.resources)
        self.feature_extractor = FeatureExtractor(self.resources); self.evaluator = ModelEvaluator(self.results_dir,self.logger,pipeline_instance=self)
        self.roberta_sentiment_predictor = TransformerSentimentPredictor(logger=self.logger) if transformers_available else None
        self.absa_analyzer = AspectBasedSentimentAnalyzer(self.results_dir,self.logger)
        self.texts,self.ratings,self.categories,self.preprocessed_texts_rules,self.preprocessed_texts_transformers = [],[],[],[],[]
        self.features_df,self.transformer_predictions_df,self.final_df = pd.DataFrame(),pd.DataFrame(),pd.DataFrame()
        self.absa_results_all_reviews,self.evaluation_results,self.models_run_successfully = [],{},[]
        self.logger.info(f"Pipeline initialized. Results directory: {self.results_dir}"); print(f"Pipeline initialized. Results stored in: {self.results_dir}")
    def _initialize_resources(self) -> Dict[str,Any]:
        self.logger.info("Initializing NLTK-dependent resources."); resources={}
        if NLTK_INITIALIZED:
            try:
                from nltk.sentiment.vader import SentimentIntensityAnalyzer; from nltk.corpus import stopwords
                resources['vader_analyzer']=SentimentIntensityAnalyzer(); resources['stop_words']=set(stopwords.words('english'))
                self.logger.info("NLTK VADER & stopwords loaded."); print("    NLTK VADER & stopwords loaded.")
            except Exception as e: self.logger.warning(f"Failed to load VADER/stopwords: {e}"); print(f"    Warning: Could not load VADER/stopwords: {e}"); resources.setdefault('vader_analyzer',None); resources.setdefault('stop_words',set())
        else: self.logger.warning("NLTK not initialized. Using fallback."); print("    Warning: NLTK not initialized. Using fallback."); resources['vader_analyzer']=None; resources['stop_words']=set()
        return resources
    def load_data(self,data_source:str,sample_size:Optional[int]=None)->None:
        display_section_header("STEP 1: Data Loading & Initial Cleaning",1); self.logger.info(f"Loading data: {data_source}, sample: {sample_size if sample_size is not None else 'Full Dataset'}")
        file_path=data_source
        if data_source.startswith('http'):
            dl_fname = f"downloaded_sentiment_data_{datetime.now().strftime('%Y%m%d')}.csv"; data_sdir = os.path.join(self.results_dir,"data"); os.makedirs(data_sdir,exist_ok=True); target_path = os.path.join(data_sdir,dl_fname)
            self.logger.info(f"URL source. Downloading to: {target_path}")
            if not download_file(data_source,target_path): self.logger.error(f"Download failed: {data_source}"); raise Exception(f"Data download failed: {data_source}")
            file_path=target_path; self.logger.info(f"Data downloaded to: {file_path}")
        self.texts,self.ratings,self.categories = load_and_clean_data(file_path,sample_size)
        self.logger.info(f"Loaded {len(self.texts)} reviews. Ratings: {Counter(self.ratings)}. Categories: {Counter(self.categories).most_common(5)}")
        print(f"\nData loading successful! Total reviews: {len(self.texts):,}");
        if self.ratings: print(f"  Avg rating: {np.mean(self.ratings):.2f}"); print("-"*60)
    def preprocess_data(self)->None:
        display_section_header("STEP 2: Text Preprocessing",1); self.logger.info("Initiating text preprocessing for rule-based and Transformer models.")
        if not self.texts: self.logger.error("No data to preprocess."); raise Exception("No data loaded.")
        self.preprocessed_texts_rules = self.preprocessor.batch_preprocess(self.texts,mode='rule_based')
        self.logger.info(f"Preprocessing for rule-based models complete: {len(self.preprocessed_texts_rules)} texts.")
        self.preprocessed_texts_transformers = self.preprocessor.batch_preprocess(self.texts,mode='transformer')
        self.logger.info(f"Preprocessing for Transformer models complete: {len(self.preprocessed_texts_transformers)} texts.")
        print("\nText preprocessing finished!")
        if self.texts and self.preprocessed_texts_rules and self.preprocessed_texts_transformers and len(self.texts) > 0:
            print(f"  Sample - Original: '{self.texts[0][:80]}...'\n  Sample - Processed (Rules): '{self.preprocessed_texts_rules[0][:80]}...'\n  Sample - Processed (Transformers): '{self.preprocessed_texts_transformers[0][:80]}...'")
        print("-"*60)
    def extract_features_for_rule_based_models(self)->None:
        display_section_header("STEP 3A: Feature Extraction (Rule-Based Models)",1); self.logger.info("Extracting features for rule-based models.")
        if not self.texts or not self.preprocessed_texts_rules: self.logger.error("Texts not available for feature extraction."); raise Exception("Texts not available.")
        self.features_df = self.feature_extractor.batch_extract_features(self.texts,self.preprocessed_texts_rules,self.categories)

        csv_path = os.path.join(self.results_dir,'extracted_rule_based_features.csv')
        try:
            self.features_df.to_csv(csv_path,index=False,encoding='utf-8')
            self.logger.info(f"Rule-based features extracted ({self.features_df.shape}). Saved to: {csv_path}")
        except Exception as e:
            self.logger.error(f"Failed to save rule-based features CSV: {e}")
            print(f"  Warning: Could not save features CSV: {e}")

        print(f"\nRule-based feature extraction finished! Shape: {self.features_df.shape}")

        if not self.features_df.empty and self.texts and len(self.texts) == len(self.features_df):
            print(f"  Preview of extracted rule-based features (Top {min(50, len(self.features_df))} rows with full text preview on the left):")

            if isinstance(self.texts, list) and all(isinstance(t, str) or pd.isna(t) for t in self.texts):
                original_texts_series = pd.Series(self.texts, name='full_text_preview', index=self.features_df.index) # Assumes features_df has compatible index
                df_for_display = pd.concat([original_texts_series, self.features_df], axis=1)

                display_limit = min(50, len(df_for_display))  # FIXED: Changed from 300 to 50
                with pd.option_context('display.max_colwidth', None):
                    safe_display(df_for_display.head(display_limit))
            else:
                self.logger.warning("Original texts not in expected list-of-strings format for display with features. Displaying features_df only.")
                safe_display(self.features_df.head(min(50, len(self.features_df))))  # FIXED: Changed from 300 to 50
        elif not self.features_df.empty:
            self.logger.warning("Original texts not available or length mismatch for display with features. Displaying features_df only.")
            safe_display(self.features_df.head(min(50, len(self.features_df))))  # FIXED: Changed from 300 to 50

        print("-"*60)

    def run_absa(self)->None:
        display_section_header("STEP 3B: Aspect-Based Sentiment Analysis (ABSA)",1); self.logger.info("Initiating ABSA.")
        if not self.texts or not self.categories: self.logger.error("Texts/categories missing for ABSA."); raise Exception("Texts/categories missing for ABSA.")

        can_do_transformer_absa = self.absa_analyzer and self.absa_analyzer.absa_pipeline and not self.absa_analyzer.use_rule_based_absa_fallback
        can_do_rule_based_absa = self.absa_analyzer and self.absa_analyzer.use_rule_based_absa_fallback and textproc_available and self.absa_analyzer.sent_tokenizer_nltk

        if not (can_do_transformer_absa or can_do_rule_based_absa):
            self.logger.warning("ABSA analyzer not fully initialized or dependencies missing (Transformer/TextBlob/NLTK sent_tokenize). Skipping ABSA.")
            print("ABSA analyzer not available or configured properly for Transformer or rule-based fallback. Skipping.")
            return

        if can_do_transformer_absa:
            self.logger.info("Using Transformer-based ABSA.")
            print("Using Transformer-based ABSA.")
        elif can_do_rule_based_absa:
            self.logger.info("Using rule-based ABSA fallback (TextBlob & NLTK).")
            print("Using rule-based ABSA fallback (TextBlob & NLTK).")

        self.absa_results_all_reviews = self.absa_analyzer.batch_analyze_reviews(self.texts,self.categories)
        self.logger.info(f"ABSA complete: {len(self.absa_results_all_reviews)} reviews processed.")
        raw_path = os.path.join(self.absa_analyzer.absa_results_dir,'absa_raw_results.json')
        try:
            with open(raw_path,'w') as f: json.dump(self.absa_results_all_reviews,f,indent=2)
            self.logger.info(f"Raw ABSA results saved to {raw_path}")
        except Exception as e: self.logger.error(f"Failed to save raw ABSA: {e}")

        num_aspects_found = sum(len(r) for r in self.absa_results_all_reviews)
        print(f"ABSA complete. {num_aspects_found} aspect-sentiments found.")
        if num_aspects_found > 0:
            for i,res_list in enumerate(self.absa_results_all_reviews):
                if res_list: print(f"    Review {i} example (method: {res_list[0].get('method', 'N/A')}): {res_list[:2]}"); break
        else: print("    No aspects found in processed reviews.")
        print("-"*60)

    def run_models(self)->None:
        display_section_header("STEP 4: Running Sentiment Models",1); self.logger.info("Running models.")
        if self.features_df.empty and (not self.roberta_sentiment_predictor or not self.roberta_sentiment_predictor.pipeline):
            self.logger.error("No features or Transformer. Cannot run models.");
            print("CRITICAL: No features available for rule-based models AND Transformer model could not be initialized. Model execution cannot proceed.")
            raise Exception("No features or Transformer model.")

        self.final_df=pd.DataFrame({'text_index':range(len(self.texts)),'original_text':self.texts,'preprocessed_text_rules':self.preprocessed_texts_rules,'preprocessed_text_transformers':self.preprocessed_texts_transformers,'true_rating':self.ratings,'detected_category':self.categories})
        if not self.features_df.empty: self.final_df=pd.merge(self.final_df,self.features_df,left_index=True,right_index=True,how='left')
        else: self.logger.warning("features_df is empty. Rule-based models might not produce meaningful results if they rely on these features.")

        self.models_run_successfully=[]
        rule_defs = {"VADER":(SentimentModels.vader_to_stars,'vader_stars',lambda:self.resources.get('vader_analyzer') is not None and not self.features_df.empty),"TextBlob":(SentimentModels.textblob_to_stars,'textblob_stars',lambda:textproc_available and not self.features_df.empty),"Feature-Based":(SentimentModels.feature_based_to_stars,'feature_based_stars',lambda:not self.features_df.empty),"Multidimensional":(SentimentModels.multidimensional_to_stars,'multidimensional_stars',lambda:not self.features_df.empty),"Rule-Based Enhanced":(SentimentModels.rule_based_enhanced_to_stars,'rule_based_stars',lambda:not self.features_df.empty),"Hybrid Weighted":(SentimentModels.hybrid_weighted_to_stars,'hybrid_stars',lambda:not self.features_df.empty),"Lexicon Ensemble":(SentimentModels.lexicon_ensemble_to_stars,'lexicon_ensemble_stars',lambda:not self.features_df.empty),"N-gram Enhanced":(SentimentModels.ngram_enhanced_to_stars,'ngram_enhanced_stars',lambda:not self.features_df.empty),}
        for dn,(mf,pcn,avail_fn) in rule_defs.items():
            if not avail_fn(): print(f"  Skipping {dn} (unavailable/features_df empty)."); self.logger.info(f"Skipped {dn}."); continue
            print(f"  Running {dn}..."); self.logger.info(f"Executing {dn}.")
            try: self.final_df[pcn]=self.final_df.apply(lambda r:mf(r.to_dict()),axis=1); self.models_run_successfully.append(dn); print(f"    {dn} completed.")
            except Exception as e: print(f"    ERROR {dn}: {e}"); self.logger.error(f"{dn} failed: {e}",exc_info=True)

        if self.roberta_sentiment_predictor and self.roberta_sentiment_predictor.pipeline:
            dn_trans="RoBERTa Sentiment"; pcn_trans="roberta_sentiment_stars"; print(f"  Running {dn_trans}..."); self.logger.info(f"Executing {dn_trans}.")
            try:
                trans_raw_out = self.roberta_sentiment_predictor.batch_predict(self.preprocessed_texts_transformers)

                if self.transformer_predictions_df.empty or len(self.transformer_predictions_df) != len(trans_raw_out):
                     self.transformer_predictions_df = pd.DataFrame(index=range(len(trans_raw_out)))

                self.transformer_predictions_df['roberta_sentiment_raw_label']=[o['label'] if o else None for o in trans_raw_out]
                self.transformer_predictions_df['roberta_sentiment_raw_score']=[o['score'] if o else None for o in trans_raw_out]
                self.final_df[pcn_trans]=[SentimentModels.roberta_sentiment_to_stars(o) for o in trans_raw_out]
                self.models_run_successfully.append(dn_trans); print(f"    {dn_trans} completed.")
            except Exception as e: print(f"    ERROR {dn_trans}: {e}"); self.logger.error(f"{dn_trans} failed: {e}",exc_info=True)
        else:
            print("  Skipping RoBERTa Sentiment model (predictor not available or pipeline failed to load).")
            self.logger.warning("Skipped RoBERTa Sentiment model as predictor or its pipeline is not available.")

        if len([m for m in self.models_run_successfully if m!="Ensemble"])>=1:
            print("  Running Advanced Ensemble..."); self.logger.info("Executing Advanced Ensemble.")
            ens_map={'vader_stars':'vader','textblob_stars':'textblob','feature_based_stars':'feature_based','multidimensional_stars':'multidimensional','rule_based_stars':'rule_based_enhanced','hybrid_stars':'hybrid_weighted','lexicon_ensemble_stars':'lexicon_ensemble','ngram_enhanced_stars':'ngram_enhanced','roberta_sentiment_stars':'roberta_sentiment'}
            try:
                def apply_ens(row):
                    preds={ek:row[pce] for pce,ek in ens_map.items() if pce in row and pd.notna(row[pce])}
                    feats=row.to_dict() if not self.features_df.empty else {}
                    return SentimentModels.advanced_ensemble_prediction(preds,features=feats)
                self.final_df['ensemble_stars']=self.final_df.apply(apply_ens,axis=1)
                self.models_run_successfully.append('Ensemble'); print("    Advanced Ensemble completed.")
            except Exception as e: print(f"    ERROR Advanced Ensemble: {e}"); self.logger.error(f"Ensemble failed: {e}",exc_info=True)
        else: print("  Skipping Advanced Ensemble (no base models)."); self.logger.info("Skipped Ensemble.")


        if not self.final_df.empty:
            preds_csv_path = os.path.join(self.results_dir,'model_predictions_with_features.csv')
            try:
                self.final_df.to_csv(preds_csv_path,index=False,encoding='utf-8')
                self.logger.info(f"Predictions saved to: {preds_csv_path}")
                print(f"  Model predictions with features saved to: {preds_csv_path}")

                if is_jupyter_environment():
                    display_limit = min(50, len(self.final_df))  # FIXED: Changed from 300 to 50
                    print(f"\n  Preview of 'model_predictions_with_features.csv' (Top {display_limit} rows, including full text):")

                    display_cols = ['original_text', 'true_rating', 'detected_category']
                    prediction_cols = [col for col in self.final_df.columns if '_stars' in col]
                    display_cols.extend(prediction_cols)
                    display_cols_present = [col for col in display_cols if col in self.final_df.columns]

                    with pd.option_context('display.max_colwidth', None):
                        safe_display(self.final_df[display_cols_present].head(display_limit))
            except Exception as e:
                self.logger.error(f"Failed to save predictions CSV or display preview: {e}")
                print(f"  Warning: Could not save predictions CSV or display preview: {e}")

        print(f"\nModel prediction phase finished! Models run: {', '.join(self.models_run_successfully) or 'None'}")
        print(f"  Final DF shape: {self.final_df.shape}"); print("-"*60)


    def evaluate_models(self)->None:
        display_section_header("STEP 5: Model Evaluation & Analysis",1); self.logger.info("Initiating model evaluation.")
        if not hasattr(self,'final_df') or self.final_df.empty: self.logger.error("final_df empty. Cannot evaluate."); raise Exception("final_df empty.")
        model_cols={'VADER':'vader_stars','TextBlob':'textblob_stars','Feature-Based':'feature_based_stars','Multidimensional':'multidimensional_stars','Rule-Based Enhanced':'rule_based_stars','Hybrid Weighted':'hybrid_stars','Lexicon Ensemble':'lexicon_ensemble_stars','N-gram Enhanced':'ngram_enhanced_stars','RoBERTa Sentiment':'roberta_sentiment_stars','Ensemble':'ensemble_stars'}
        self.evaluation_results={}
        for dn,pcn in model_cols.items():
            if dn in self.models_run_successfully and pcn in self.final_df.columns:
                self.logger.info(f"Evaluating model: {dn}")
                eval_res = self.evaluator.evaluate_model(y_true=self.final_df['true_rating'].tolist(),y_pred=self.final_df[pcn].tolist(),model_name=dn,prediction_column_name_in_final_df=pcn,confidence_scores=(self.final_df['custom_sentiment_score'].tolist() if 'custom_sentiment_score' in self.final_df and not self.features_df.empty else None),original_texts_for_analysis=self.final_df['original_text'].tolist(),features_df_for_analysis=self.features_df if not self.features_df.empty else pd.DataFrame(),transformer_predictions_df=self.transformer_predictions_df if not self.transformer_predictions_df.empty else pd.DataFrame())
                if eval_res: self.evaluation_results[dn]=eval_res
                else: self.logger.warning(f"Eval for {dn} no results.")
            elif dn in self.models_run_successfully: self.logger.warning(f"Pred col '{pcn}' not found for {dn}.")

        self.logger.info("Model evaluation phase finished.")

    def generate_visualizations(self)->None:
        display_section_header("STEP 6: Generating Advanced Visualizations",1); self.logger.info("Generating visualizations.")
        if not matplotlib_available: self.logger.warning("Matplotlib unavailable. Skipping visualizations."); return
        try: print("  Generating data distribution plots..."); self.logger.info("Plotting data distributions."); self._plot_data_distribution()
        except Exception as e: self.logger.error(f"Data dist plot failed: {e}",exc_info=True)
        try:
            if not self.features_df.empty:
                print("  Generating rule-based feature analysis plots (Boxplots)..."); self.logger.info("Plotting rule-based feature analysis (Boxplots)."); self._plot_feature_analysis()
                print("  Generating additional rule-based feature analysis plots (Distributions, Heatmap)..."); self.logger.info("Plotting additional rule-based feature analysis (Distributions, Heatmap)."); self._plot_additional_feature_visuals()
            else: print("  Skipping rule-based feature plots (no features).")
        except Exception as e: self.logger.error(f"Rule-based feature plot failed: {e}",exc_info=True)
        try:
            if self.categories: print("  Generating category analysis plots..."); self.logger.info("Plotting category analysis."); self._plot_category_analysis()
            else: print("  Skipping category plots (no categories).")
        except Exception as e: self.logger.error(f"Category plot failed: {e}",exc_info=True)
        try:
            if hasattr(self,'evaluation_results') and self.evaluation_results:
                print("  Generating detailed model comparison plots (including Radar Chart)...");
                self.logger.info("Plotting model comparisons (including Radar).");
                self._plot_model_comparison_details()
            else: print("  Skipping detailed model comparison plots (no eval results).")
        except Exception as e: self.logger.error(f"Model comparison plot failed: {e}",exc_info=True)
        try:
            if self.absa_results_all_reviews and self.absa_analyzer: print("  Generating ABSA summary visualizations..."); self.logger.info("Generating ABSA visualizations."); self.absa_analyzer.generate_absa_summary_visualizations(self.absa_results_all_reviews,self.categories)
            else: print("  Skipping ABSA visualizations (no ABSA results/analyzer).")
        except Exception as e: self.logger.error(f"ABSA viz failed: {e}",exc_info=True)
        print("\nVisualization generation phase finished!"); self.logger.info("Visualization completed."); print("-"*60)

    def _plot_data_distribution(self)->None:
        if not self.texts or not self.ratings: print("  No texts/ratings for data dist."); return
        fig,axes=plt.subplots(2,2,figsize=(18,14)); fig.suptitle('Exploratory Data Analysis: Data Distributions',fontsize=18,fontweight='bold'); ax_flat=axes.flatten()
        try:
            if self.ratings:
                rc=pd.Series(self.ratings).value_counts().sort_index(); colors=sns.color_palette("viridis_r",len(rc)); bars=ax_flat[0].bar(rc.index.astype(str)+"S",rc.values,color=colors,width=0.6)
                ax_flat[0].set_title('Star Rating Distribution',fontsize=14); ax_flat[0].set_xlabel('Star Rating'); ax_flat[0].set_ylabel('Number of Reviews')
                max_rc_val = rc.max() if not rc.empty else 1
                for bar in bars: yv=bar.get_height(); ax_flat[0].text(bar.get_x()+bar.get_width()/2.,yv+max_rc_val*0.01,int(yv),ha='center',va='bottom') if yv>0 else None; ax_flat[0].grid(axis='y',ls='--',alpha=0.7)
        except Exception as e: self.logger.error(f"Rating dist plot error: {e}",exc_info=True); ax_flat[0].text(0.5,0.5,"Error",ha='center')
        try:
            if self.texts:
                tl=[len(str(t)) for t in self.texts if pd.notna(t)]
                if tl: ax_flat[1].hist(tl,bins=50,alpha=0.75,color='lightseagreen',edgecolor='darkcyan'); ax_flat[1].set_title('Review Text Length (Chars)',fontsize=14); ax_flat[1].set_xlabel('# Chars'); ax_flat[1].set_ylabel('Frequency'); ml=np.mean(tl); ax_flat[1].axvline(ml,color='red',ls='dashed',lw=2,label=f'Mean: {ml:.0f}'); ax_flat[1].legend(); ax_flat[1].grid(axis='y',ls='--',alpha=0.7); ax_flat[1].set_xscale('log')
                else: ax_flat[1].text(0.5,0.5,"No text lengths",ha='center')
        except Exception as e: self.logger.error(f"Text length plot error: {e}",exc_info=True); ax_flat[1].text(0.5,0.5,"Error",ha='center')
        try:
            if self.categories:
                cc=pd.Series(self.categories).value_counts().head(10)
                if not cc.empty: colors=sns.color_palette("Spectral",len(cc)); bars=ax_flat[2].barh(cc.index,cc.values,color=colors,height=0.7); ax_flat[2].set_title(f'Top {len(cc)} Categories',fontsize=14); ax_flat[2].set_xlabel('# Reviews'); ax_flat[2].invert_yaxis(); [ax_flat[2].text(val+cc.max()*0.01,i,str(val),va='center') for i,val in enumerate(cc.values)]; ax_flat[2].grid(axis='x',ls='--',alpha=0.7)
                else: ax_flat[2].text(0.5,0.5,"No categories",ha='center')
        except Exception as e: self.logger.error(f"Category dist plot error: {e}",exc_info=True); ax_flat[2].text(0.5,0.5,"Error",ha='center')
        try:
            if self.categories and self.ratings and matplotlib_available:
                dfp = pd.DataFrame({'category': self.categories, 'rating': self.ratings})
                top_cats_series = pd.Series(self.categories).value_counts()
                min_reviews_for_prop = max(20, int(len(self.texts) * 0.001)) if len(self.texts) > 0 else 20
                top_cats = top_cats_series[top_cats_series >= min_reviews_for_prop].nlargest(7).index.tolist()

                if not top_cats and not top_cats_series.empty :
                    top_cats = top_cats_series.nlargest(min(7, len(top_cats_series))).index.tolist()

                if top_cats:
                    dff = dfp[dfp['category'].isin(top_cats)]
                    if not dff.empty:
                        cat_rating_props = dff.groupby('category')['rating'].value_counts(normalize=True).mul(100).unstack(fill_value=0)
                        for r_col in range(1, 6):
                            if r_col not in cat_rating_props.columns: cat_rating_props[r_col] = 0.0

                        cat_rating_props = cat_rating_props.reindex(columns=sorted(cat_rating_props.columns)).reindex(top_cats)

                        rating_colors = sns.color_palette("RdYlGn", n_colors=5)

                        cat_rating_props.plot(kind='barh', stacked=True, ax=ax_flat[3],
                                              color=[rating_colors[i] for i in range(5)], width=0.8)
                        ax_flat[3].set_title('Rating Proportions in Top Categories', fontsize=14)
                        ax_flat[3].set_xlabel('Percentage of Reviews (%)')
                        ax_flat[3].set_ylabel('Category')
                        ax_flat[3].legend(title='Rating', labels=[f'{i+1}-Star' for i in range(5)], bbox_to_anchor=(1.05, 1), loc='upper left')
                        ax_flat[3].set_xlim(0, 100)
                        ax_flat[3].grid(axis='x', ls='--', alpha=0.7)

                        for n, (idx, row) in enumerate(cat_rating_props.iterrows()):
                            cumulative_width = 0
                            for i, val in enumerate(row):
                                if val > 5:
                                    ax_flat[3].text(cumulative_width + val / 2, n, f'{val:.0f}%', ha='center', va='center', fontsize=7, color='black' if val > 30 else 'dimgray')
                                cumulative_width += val
                    else: ax_flat[3].text(0.5,0.5,"No data for top cats proportion plot",ha='center')
                else: ax_flat[3].text(0.5,0.5,"Not enough categories/data for plot",ha='center')
            else: ax_flat[3].text(0.5,0.5,"No categories/ratings or matplotlib",ha='center')
        except Exception as e: self.logger.error(f"Category rating proportion plot error: {e}",exc_info=True); ax_flat[3].text(0.5,0.5,"Error",ha='center')

        plt.tight_layout(rect=[0,0,1,0.96]); plt.savefig(os.path.join(self.results_dir,'data_distributions_analysis.png'),dpi=300,bbox_inches='tight')
        if is_jupyter_environment(): plt.show(); plt.close(fig)


    def _plot_feature_analysis(self)->None:
        if self.features_df.empty or not self.ratings or len(self.features_df)!=len(self.ratings): print(" No rule-based features/ratings for boxplot analysis."); return
        key_feats=['text_length','word_count','avg_word_length','vader_compound','textblob_polarity','textblob_subjectivity','positive_word_count','negative_word_count','custom_sentiment_score','2gram_count','3gram_count']
        avail_feats=[f for f in key_feats if f in self.features_df.columns];
        if not avail_feats: print(" No suitable numeric features for box plotting."); return
        nfeats=len(avail_feats); nc=3; nr=math.ceil(nfeats/float(nc)); fig,axes=plt.subplots(nr,nc,figsize=(nc*6.5,nr*5.5),squeeze=False)
        fig.suptitle('Boxplot Analysis of Key Rule-Based Features vs. True Star Rating',fontsize=18,fontweight='bold'); ax_flat=axes.flatten()
        plot_df=self.features_df[avail_feats].copy(); plot_df['true_rating']=self.ratings
        for i,feat_name in enumerate(avail_feats):
            ax=ax_flat[i]
            try:
                sns.boxplot(x='true_rating',y=feat_name,data=plot_df,ax=ax,palette="coolwarm_r",flierprops={'markerfacecolor':'0.50','markersize':3,'marker':'o'})
                title=feat_name.replace("_"," ").title(); title+= " (Aggregated Score)" if feat_name=='custom_sentiment_score' else ""
                ax.set_title(title,fontsize=14); ax.set_xlabel('True Star Rating',fontsize=12); ax.set_ylabel('Feature Value',fontsize=12); ax.grid(True,linestyle=':',alpha=0.6)
            except Exception as e: self.logger.error(f"Error plotting feature '{feat_name}': {e}",exc_info=True); ax.text(0.5,0.5,f"Error plotting\n{feat_name}",ha='center',va='center',transform=ax.transAxes)
        for j in range(i+1,len(ax_flat)): ax_flat[j].set_visible(False)
        plt.tight_layout(rect=[0,0,1,0.96]); plt.savefig(os.path.join(self.results_dir,'rule_features_vs_rating_boxplots.png'),dpi=300,bbox_inches='tight')
        if is_jupyter_environment(): plt.show(); plt.close(fig)

    def _plot_additional_feature_visuals(self) -> None:
        if self.features_df.empty or not matplotlib_available:
            self.logger.warning("Skipping additional feature visuals: features_df empty or matplotlib unavailable.")
            return

        numerical_features_df = self.features_df.select_dtypes(include=np.number)
        if numerical_features_df.empty:
            self.logger.warning("Skipping additional feature visuals: No numerical features found.")
            return

        key_dist_features = ['vader_compound', 'textblob_polarity', 'custom_sentiment_score',
                               'positive_word_count', 'negative_word_count', 'word_count', 'text_length',
                               'intensifier_count', 'negation_count', 'sentiment_ratio']
        plotted_dist_features = []

        num_key_feats_to_plot = len([f for f in key_dist_features if f in numerical_features_df.columns])
        if num_key_feats_to_plot > 0:
            ncols_dist = min(3, num_key_feats_to_plot)
            nrows_dist = math.ceil(num_key_feats_to_plot / ncols_dist)
            fig_dist, axes_dist = plt.subplots(nrows_dist, ncols_dist, figsize=(ncols_dist * 6, nrows_dist * 5), squeeze=False)
            axes_flat_dist = axes_dist.flatten()
            feat_idx = 0
            for feat_name in key_dist_features:
                if feat_name in numerical_features_df.columns:
                    sns.histplot(numerical_features_df[feat_name], kde=True, ax=axes_flat_dist[feat_idx], bins=min(30, numerical_features_df[feat_name].nunique()))
                    axes_flat_dist[feat_idx].set_title(f'Distribution of {feat_name.replace("_", " ").title()}', fontsize=11)
                    axes_flat_dist[feat_idx].set_xlabel(feat_name, fontsize=10)
                    axes_flat_dist[feat_idx].set_ylabel('Frequency', fontsize=10)
                    plotted_dist_features.append(feat_name)
                    feat_idx += 1

            for i in range(feat_idx, nrows_dist * ncols_dist):
                fig_dist.delaxes(axes_flat_dist[i])

            fig_dist.suptitle('Distribution of Key Numerical Features', fontsize=16, fontweight='bold')
            plt.tight_layout(rect=[0, 0, 1, 0.95])
            dist_plot_path = os.path.join(self.results_dir, 'key_features_distributions.png')
            plt.savefig(dist_plot_path, dpi=300)
            if is_jupyter_environment(): plt.show()
            plt.close(fig_dist)
            self.logger.info(f"Key feature distribution plots saved to {dist_plot_path}")
        else:
            self.logger.info("No key features found for distribution plots.")


        if len(numerical_features_df.columns) > 30:
            if plotted_dist_features:
                 cols_for_corr = [col for col in plotted_dist_features if col in numerical_features_df.columns]

                 if len(cols_for_corr) < 30:
                     remaining_cols = [col for col in numerical_features_df.columns if col not in cols_for_corr]
                     if remaining_cols:
                         try:
                             variances = numerical_features_df[remaining_cols].var().sort_values(ascending=False)
                             cols_for_corr.extend(variances.head(30 - len(cols_for_corr)).index.tolist())
                         except Exception:
                             cols_for_corr.extend(np.random.choice(remaining_cols, size=min(len(remaining_cols), 30 - len(cols_for_corr)), replace=False).tolist())
            else:
                cols_for_corr_all = numerical_features_df.columns.tolist()
                try:
                    variances = numerical_features_df.var().sort_values(ascending=False)
                    cols_for_corr = variances.head(30).index.tolist()
                except Exception:
                    cols_for_corr = np.random.choice(cols_for_corr_all, size=min(len(cols_for_corr_all),30), replace=False).tolist()
            corr_features_df = numerical_features_df[cols_for_corr]
        else:
            corr_features_df = numerical_features_df

        if len(corr_features_df.columns) > 1 :
            correlation_matrix = corr_features_df.corr()
            fig_corr_height = max(8, len(corr_features_df.columns) * 0.4)
            fig_corr_width = max(10, len(corr_features_df.columns) * 0.5)
            fig_corr, ax_corr = plt.subplots(figsize=(fig_corr_width, fig_corr_height))

            sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', fmt=".2f", linewidths=.5, ax=ax_corr, annot_kws={"size": 8 if len(corr_features_df.columns) < 20 else 6})
            ax_corr.set_title('Correlation Heatmap of Numerical Features', fontsize=14, fontweight='bold')
            plt.xticks(rotation=45, ha="right", fontsize=9)
            plt.yticks(rotation=0, fontsize=9)
            plt.tight_layout()
            corr_plot_path = os.path.join(self.results_dir, 'numerical_features_correlation_heatmap.png')
            plt.savefig(corr_plot_path, dpi=300)
            if is_jupyter_environment(): plt.show()
            plt.close(fig_corr)
            self.logger.info(f"Feature correlation heatmap saved to {corr_plot_path}")
        else:
            self.logger.info("Not enough numerical features (need >1) for a correlation heatmap.")


    def _plot_category_analysis(self)->None:
        if not self.categories or not self.ratings or len(self.categories)!=len(self.ratings): print(" No categories/ratings for analysis."); return
        df_cat=pd.DataFrame({'category':self.categories,'rating':self.ratings}); cat_stats=df_cat.groupby('category')['rating'].agg(count='size',avg_rating='mean',std_rating='std').reset_index()
        cat_stats_filt=cat_stats[cat_stats['count']>=50].sort_values('count',ascending=False).head(10)
        if cat_stats_filt.empty: print(" No categories meet min review count (50) for plot."); return
        fig,axes=plt.subplots(1,2,figsize=(20,9)); fig.suptitle('Product Category Performance Analysis (Filtered by Min. 50 Reviews)',fontsize=18,fontweight='bold')
        try:
            sns.barplot(x='count',y='category',data=cat_stats_filt,ax=axes[0],palette="viridis",orient='h',dodge=False)
            axes[0].set_title('Review Count (Top Filtered Categories)',fontsize=14); axes[0].set_xlabel('# Reviews'); axes[0].set_ylabel('Product Category')
            max_cat_count = cat_stats_filt['count'].max() if not cat_stats_filt.empty else 1
            for i,val in enumerate(cat_stats_filt['count']): axes[0].text(val+max_cat_count*0.01,i,f"{val:,}",va='center',fontsize=9)
            axes[0].grid(axis='x',ls='--',alpha=0.7)
        except Exception as e: self.logger.error(f"Category count bar plot error: {e}",exc_info=True)
        try:
            sp=axes[1].scatter(cat_stats_filt['avg_rating'],cat_stats_filt['count'],s=cat_stats_filt['count']*0.3+100,c=cat_stats_filt['avg_rating'],cmap='coolwarm_r',alpha=0.85,edgecolors='black',linewidth=0.7)
            axes[1].set_title('Avg Rating vs. Review Count (Top Filtered Categories)',fontsize=14); axes[1].set_xlabel('Avg Star Rating'); axes[1].set_ylabel('# Reviews (Log Scale)'); axes[1].set_yscale('log'); axes[1].grid(True,ls=':',alpha=0.6)
            for _,row in cat_stats_filt.iterrows(): axes[1].annotate(row['category'],(row['avg_rating'],row['count']),textcoords="offset points",xytext=(0,8),ha='center',fontsize=9,color='dimgray')
            cbar=fig.colorbar(sp,ax=axes[1],label='Average Rating',pad=0.02); cbar.ax.tick_params(labelsize=9)
        except Exception as e: self.logger.error(f"Category scatter plot error: {e}",exc_info=True)
        plt.tight_layout(rect=[0,0,1,0.95]); plt.savefig(os.path.join(self.results_dir,'category_performance_analysis.png'),dpi=300,bbox_inches='tight')
        if is_jupyter_environment(): plt.show(); plt.close(fig)

    def _plot_model_comparison_details(self)->None:
        if not hasattr(self,'evaluation_results') or not self.evaluation_results: print(" No eval results for model comparison."); return

        if self.evaluator:
            self.evaluator.plot_radar_chart(self.evaluation_results)
        else:
            self.logger.warning("ModelEvaluator instance not available, cannot plot radar chart.")

        models_eval = list(self.evaluation_results.keys())
        if not models_eval: print(" No models evaluated; cannot create comparison heatmap/bar charts."); return
        metrics_comp=['accuracy','close_accuracy','mae','rmse','f1_macro','f1_weighted','f1_close','kappa']; disp_names=['Exact Acc.','+/-1 Star Acc.','MAE','RMSE','F1 (Macro)','F1 (Weighted)','F1 (+/-1 Acc.)','Kappa']; higher_better=[True,True,False,False,True,True,True,True]
        comp_data=[{'Model':mn, **{mc:self.evaluation_results[mn].get('metrics',{}).get(mc,np.nan) for mc in metrics_comp}} for mn in models_eval]
        comp_df = pd.DataFrame(comp_data).set_index('Model')
        if comp_df.empty: print(" Comparison DataFrame empty."); return
        heatmap_df=comp_df.copy()
        for idx,mcol in enumerate(metrics_comp):
            valid_v=heatmap_df[mcol].dropna()
            if not valid_v.empty:
                min_v,max_v=valid_v.min(),valid_v.max()
                if max_v>min_v: norm_s=(valid_v-min_v)/(max_v-min_v); heatmap_df[mcol]=1-norm_s if not higher_better[idx] else norm_s
                else: heatmap_df[mcol]=0.5
            else: heatmap_df[mcol]=np.nan
        fig_hm=None
        try:
            fig_hm,ax_hm = plt.subplots(figsize=(16,9)); sns.heatmap(heatmap_df.T,annot=comp_df.T,fmt='.3f',cmap='RdYlGn',center=0.5,linewidths=.5,linecolor='gray',cbar_kws={'label':'Normalized Performance (Higher is Better)'},ax=ax_hm,annot_kws={"size":9})
            ax_hm.set_title('Model Performance Comparison (Annotation: Raw Value; Color: Normalized)',fontsize=16,fontweight='bold',pad=20); ax_hm.set_xticklabels(ax_hm.get_xticklabels(),rotation=45,ha="right",fontsize=10); ax_hm.set_yticklabels(disp_names,rotation=0,fontsize=10)
            plt.tight_layout(); plt.savefig(os.path.join(self.results_dir,'model_comparison_heatmap_detailed.png'),dpi=300,bbox_inches='tight')
            if is_jupyter_environment(): plt.show()
        except Exception as e: self.logger.error(f"Model comparison heatmap error: {e}",exc_info=True)
        finally:
            if fig_hm: plt.close(fig_hm)

        n_metrics_bar=len(metrics_comp); nc_bar=min(3,n_metrics_bar); nr_bar=math.ceil(n_metrics_bar/float(nc_bar))
        fig_bar,axes_bar = plt.subplots(nr_bar,nc_bar,figsize=(nc_bar*6.5,nr_bar*5.5),squeeze=False); axes_bar_flat=axes_bar.flatten()
        fig_bar.suptitle('Detailed Model Performance: Metric-by-Metric Comparison',fontsize=18,fontweight='bold')
        bar_cols=sns.color_palette("Paired",len(models_eval))
        for i_mbar,(mkey,mdisp) in enumerate(zip(metrics_comp,disp_names)):
            ax_curr=axes_bar_flat[i_mbar]
            try:
                mvals=comp_df[mkey].sort_values(ascending=not higher_better[i_mbar])
                bars=ax_curr.bar(mvals.index,mvals.values,color=[bar_cols[models_eval.index(mn)] for mn in mvals.index],alpha=0.9,width=0.6)
                ax_curr.set_title(mdisp,fontsize=14); ax_curr.set_ylabel('Metric Score',fontsize=12); ax_curr.set_xticklabels(mvals.index,rotation=45,ha='right',fontsize=10); ax_curr.grid(axis='y',ls=':',alpha=0.7)
                for bar_item in bars: yv=bar_item.get_height(); ax_curr.text(bar_item.get_x()+bar_item.get_width()/2.,yv,f'{yv:.3f}',ha='center',va='bottom' if yv>=0 else 'top',fontsize=9) if pd.notna(yv) else None
            except Exception as e: self.logger.error(f"Bar plot for '{mdisp}' error: {e}",exc_info=True); ax_curr.text(0.5,0.5,f"Error plotting\n{mdisp}",ha='center',va='center',transform=ax_curr.transAxes)
        for j in range(i_mbar+1,len(axes_bar_flat)): axes_bar_flat[j].set_visible(False)
        plt.tight_layout(rect=[0,0,1,0.95]); plt.savefig(os.path.join(self.results_dir,'model_comparison_barcharts_detailed.png'),dpi=300,bbox_inches='tight')
        if is_jupyter_environment(): plt.show(); plt.close(fig_bar)

    def perform_cross_validation_analysis(self)->None:
        display_section_header("STEP 7: Cross-Validation Analysis",1); self.logger.info("Initiating CV analysis.")
        if self.features_df.empty or not self.ratings: print("No rule-based features/ratings for CV. Skipping."); self.logger.warning("Skipping CV: Features empty or ratings missing."); return
        print("Performing 5-fold CV for selected rule-based models..."); X_cv=self.features_df.select_dtypes(include=[np.number]).fillna(0); y_cv=np.array(self.ratings)
        if len(X_cv)!=len(y_cv): print("CV length mismatch. Skipping."); self.logger.error(f"CV length mismatch: X({len(X_cv)}), y({len(y_cv)})."); return
        cv_summary={}; cv_folds=5; models_cv={'VADER':SentimentModels.vader_to_stars,'TextBlob':SentimentModels.textblob_to_stars,'Feature-Based':SentimentModels.feature_based_to_stars,'N-gram Enhanced':SentimentModels.ngram_enhanced_to_stars}
        skf=StratifiedKFold(n_splits=cv_folds,shuffle=True,random_state=42)
        for model_name,model_fn in models_cv.items():
            print(f"  Cross-validating {model_name}..."); scores={'accuracy':[],'mae':[],'f1_close':[]}
            for fold_n,(train_idx,test_idx) in enumerate(skf.split(X_cv,y_cv)):
                try:
                    test_feats=X_cv.iloc[test_idx]; test_true=y_cv[test_idx]; test_preds=[model_fn(row.to_dict()) for _,row in test_feats.iterrows()]
                    test_preds_arr=np.array(test_preds)
                    scores['accuracy'].append(np.mean(test_true==test_preds_arr)); scores['mae'].append(mean_absolute_error(test_true,test_preds_arr))
                    close_acc_bool=np.abs(test_true-test_preds_arr)<=1; scores['f1_close'].append(f1_score(np.ones_like(test_true),close_acc_bool.astype(int),pos_label=1,zero_division=0))
                except Exception as e: self.logger.error(f"Error CV fold {fold_n+1} for {model_name}: {e}"); scores['accuracy'].append(np.nan); scores['mae'].append(np.nan); scores['f1_close'].append(np.nan)
            if any(not np.isnan(s) for s in scores['accuracy']): cv_summary[model_name]={'accuracy_mean':np.nanmean(scores['accuracy']),'accuracy_std':np.nanstd(scores['accuracy']),'mae_mean':np.nanmean(scores['mae']),'mae_std':np.nanstd(scores['mae']),'f1_close_mean':np.nanmean(scores['f1_close']),'f1_close_std':np.nanstd(scores['f1_close'])}
            else: self.logger.warning(f"No valid scores for {model_name} in CV.")
        if cv_summary:
            cv_df=pd.DataFrame(cv_summary).T; cv_df.columns=['Accuracy (Mean)','Accuracy (Std)','MAE (Mean)','MAE (Std)','F1 Close Acc. (Mean)','F1 Close Acc. (Std)']; cv_df=cv_df.round(4)
            print(f"\n--- Cross-Validation Results ({cv_folds}-Fold) ---"); safe_display(cv_df)
            cv_csv=os.path.join(self.results_dir,'cross_validation_analysis_results.csv')
            try: cv_df.to_csv(cv_csv,encoding='utf-8'); self.logger.info(f"CV results saved: {cv_csv}"); print(f"CV results saved: {cv_csv}")
            except Exception as e: self.logger.error(f"Failed to save CV CSV: {e}"); print(f"  Warning: Could not save CV CSV: {e}")
        else: print("No CV results to display/save."); self.logger.info("CV analysis complete, no results.")
        print("Cross-validation analysis phase complete!"); print("-"*60)

    def perform_clustering_analysis(self)->None:
        display_section_header("STEP 8: Clustering Analysis (on Rule-Based Features)",1); self.logger.info("Initiating clustering analysis.")
        if self.features_df.empty: print("No rule-based features for clustering. Skipping."); self.logger.warning("Skipping clustering: Features empty."); return
        print("Performing clustering on rule-based features..."); X_cluster=self.features_df.select_dtypes(include=[np.number]).fillna(0)
        if X_cluster.empty: print("No numeric features for clustering. Skipping."); self.logger.warning("Clustering skipped: No numeric features."); return
        X_scaled=StandardScaler().fit_transform(X_cluster)
        max_k=min(10,max(2,len(X_scaled)//100));
        if max_k<2: print("Insufficient data for clustering (need at least 2 clusters). Skipping."); self.logger.warning("Clustering skipped: Insufficient data for min 2 clusters."); return
        sil_scores=[]; k_range=range(2,max_k+1)
        print(f"  Determining optimal k in range {k_range} using Silhouette Score...")
        for k_iter in k_range:
            try:

                km=KMeans(n_clusters=k_iter,random_state=42,n_init=10, algorithm='lloyd')
                labels=km.fit_predict(X_scaled)
                if len(np.unique(labels))>1: sil_scores.append(silhouette_score(X_scaled,labels))
                else: sil_scores.append(-1)
            except Exception as e: self.logger.error(f"Error KMeans for k={k_iter}: {e}"); sil_scores.append(-1)

        if not sil_scores or all(s == -1 for s in sil_scores) or max(sil_scores) < -0.5 :
            print("Cannot determine optimal k or all silhouette scores are very low. Clustering aborted.");
            self.logger.error("Clustering aborted: Optimal k determination failed or silhouette scores too low."); return

        opt_k=k_range[np.argmax(sil_scores)]; best_sil=max(sil_scores); print(f"  Optimal k: {opt_k} (Silhouette: {best_sil:.3f})")
        final_km=KMeans(n_clusters=opt_k,random_state=42,n_init=10, algorithm='lloyd'); final_labels=final_km.fit_predict(X_scaled)
        results=[]; ratings_arr=np.array(self.ratings); cats_arr=np.array(self.categories)
        for i_cluster in range(opt_k):
            mask=(final_labels==i_cluster); n_members=np.sum(mask)
            if n_members==0: continue
            ratings_c=ratings_arr[mask]; cats_c=cats_arr[mask]
            results.append({'Cluster_ID':int(i_cluster),'Num_Reviews_in_Cluster':int(n_members),'Average_Rating':float(np.mean(ratings_c)) if len(ratings_c)>0 else np.nan,'Rating_StdDev':float(np.std(ratings_c)) if len(ratings_c)>0 else np.nan,'Top_Category':Counter(cats_c).most_common(1)[0][0] if len(cats_c)>0 else 'N/A'})
        cluster_df=pd.DataFrame(results); print(f"\n--- Clustering Analysis Summary (k={opt_k}) ---"); safe_display(cluster_df)
        output_data={'optimal_k':int(opt_k),'best_silhouette_score':float(best_sil),'cluster_labels_per_review':[int(lbl) for lbl in final_labels.tolist()],'cluster_summary_analysis':results}
        json_path=os.path.join(self.results_dir,'clustering_analysis_results.json')
        try:
            with open(json_path,'w',encoding='utf-8') as f: json.dump(output_data,f,indent=4)
            self.logger.info(f"Clustering results saved: {json_path}"); print(f"Clustering results saved: {json_path}")
        except Exception as e: self.logger.error(f"Failed to save clustering JSON: {e}"); print(f"  Warning: Could not save clustering JSON: {e}")
        print("Clustering analysis phase complete!"); print("-"*60)

    def generate_report(self)->None:
        display_section_header("STEP 9: Generating Analysis Report",1); self.logger.info("Report generation initiated.")
        report_md_content=self._create_detailed_report_markdown()
        report_path=os.path.join(self.results_dir,'sentiment_analysis_report_transformer_absa.md')
        try:
            with open(report_path,'w',encoding='utf-8') as f: f.write(report_md_content)
            self.logger.info(f"Analysis report saved: {report_path}"); print(f"  Analysis report saved: {report_path}")
        except Exception as e: self.logger.error(f"Failed to save report: {e}"); print(f"  Warning: Could not save report: {e}")
        if is_jupyter_environment(): print("\n  Report Snippet (first 500 chars):"); display(Markdown(report_md_content[:500]+"\n\n... (Full report saved)"))
        self.logger.info(f"Report generation finished: {report_path}"); print("\nReport generation finished!"); print("-"*60)

    def _create_detailed_report_markdown(self)->str:
        report_md = f"# Sentiment Analysis Report (Transformer & ABSA Enhanced)\n\n**Date Generated:** {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n**Results Directory:** `{self.results_dir}`\n\n---\n\n"
        report_md += "## Overview of Pipeline Capabilities\n\nThis pipeline incorporates:\n- **Hybrid Modeling**: Rule-based models combined with Transformer-based models (e.g., RoBERTa) for overall sentiment.\n- **Fine-grained Insights**: Aspect-Based Sentiment Analysis (ABSA) for specific feature sentiment. Includes rule-based fallback if Transformer ABSA model fails.\n- **Worst Prediction Analysis**: Pandas DataFrame table with full review text, and CSVs for error analysis.\n- **Robust Evaluation**: Standard metrics, CV for rule-based models, clustering.\n- **Visualization**: Dashboards (Plotly HTML), static plots for data distributions (including alternative category rating plot), feature importance (including additional feature distributions and heatmap), and model comparisons.\n\n---\n\n## 1. Dataset Overview & Preparation\n"
        report_md += f"- **Total Reviews Analyzed:** {len(self.texts):,}\n"
        if self.ratings: report_md += f"- **Avg Star Rating:** {np.mean(self.ratings):.2f}\n- **Rating Distribution:**\n" + "".join([f"  - {r} Stars: {Counter(self.ratings)[r]:,} ({(Counter(self.ratings)[r]/len(self.ratings))*100:.1f}%)\n" for r in sorted(Counter(self.ratings).keys())])
        if self.categories: report_md += "\n- **Product Category Distribution (Top 5):\n" + "".join([f"    - {cat.title()}: {cnt:,} ({(cnt/len(self.categories))*100:.1f}%)\n" for cat,cnt in Counter(self.categories).most_common(5)])
        report_md += "- **Preprocessing**: Rule-Based (cleaning, token/lemma, stopwords), Transformers (minimal cleaning).\n- **Feature Engineering (Rule-Based)**: Basic stats, VADER/TextBlob scores, lexicon counts, domain terms, linguistic cues, n-grams.\n\n"
        report_md += "---\n\n## 2. Sentiment Model Performance (Overall Rating)\n"
        if hasattr(self,'evaluation_results') and self.evaluation_results:
            best_model,best_f1 = None,-1.0
            for mn,rd in self.evaluation_results.items():
                f1c = rd.get('metrics',{}).get('f1_close',-1.0)
                if f1c>best_f1: best_f1,best_model = f1c,mn
            if best_model: report_md += f"**Overall Best Model (F1 +/-1 Star):** **{best_model}** (Score: {best_f1:.4f})\n\n"
            headers = ['Model','Accuracy','Close Acc. (+/-1)','MAE','RMSE','F1 Macro','F1 (+/-1 Acc.)','Kappa','Sample Size']
            report_md += "### Detailed Performance Metrics:\n| " + " | ".join(headers) + " |\n|-" + "-|-".join(["-"*len(h) for h in headers]) + "-|\n"
            key_map = {'Accuracy':'accuracy','Close Acc. (+/-1)':'close_accuracy','MAE':'mae','RMSE':'rmse','F1 Macro':'f1_macro','F1 (+/-1 Acc.)':'f1_close','Kappa':'kappa','Sample Size':'sample_size'}
            for mn,rd in self.evaluation_results.items():
                m=rd.get('metrics',{}); vals=[mn]+[f"{m.get(key_map[h]):.4f}" if key_map[h]!='sample_size' and isinstance(m.get(key_map[h]),(int,float)) else (f"{m.get(key_map[h]):,}" if key_map[h]=='sample_size' else str(m.get(key_map[h],'N/A'))) for h in headers[1:]]
                report_md += "| " + " | ".join(vals) + " |\n"
            report_md += "\n- **Worst Predictions Analysis**: Full list of worst predictions (including full review text) displayed as a Pandas table during evaluation. CSVs also saved in `worst_predictions_analysis/`.\n- **Interactive Dashboard**: `interactive_model_dashboard.html` for visual model comparison.\n- **Model Predictions Preview**: Top 50 rows of `model_predictions_with_features.csv` (including full text) displayed in Jupyter output during `run_models` step.\n\n"  # FIXED: Changed from 300 to 50
        else: report_md += "No model evaluation results for overall rating.\n"
        report_md += "\n---\n\n## 3. Aspect-Based Sentiment Analysis (ABSA)\n"
        absa_method_used = "Transformer (`yangheng/deberta-v3-base-absa-v1.1`)"
        if self.absa_analyzer and self.absa_analyzer.use_rule_based_absa_fallback:
            absa_method_used = "Rule-Based Fallback (TextBlob & NLTK)"
        elif not (self.absa_analyzer and self.absa_analyzer.absa_pipeline):
            absa_method_used = "Not Performed (Transformer model load failed and/or rule-based fallback dependencies not met)"
        report_md += f"ABSA Method: {absa_method_used}.\n\n"

        if self.absa_results_all_reviews:
            report_md += "- **Visualizations**: Sentiment distributions for top aspects in `absa_analysis/`.\n- **Raw Data**: `absa_analysis/absa_raw_results.json`.\n- **Aggregated Data**: `absa_analysis/absa_summary_data.csv`.\n\n"
            flat_aspects = [item for sublist in self.absa_results_all_reviews for item in sublist]
            if flat_aspects:
                df_aspects_rep = pd.DataFrame(flat_aspects); report_md += "**Top 5 Discussed Aspects (Overall):**\n"
                for aspect,count in df_aspects_rep['aspect'].value_counts().nlargest(5).items(): report_md += f"  - **{aspect}**: {count:,} mentions (Positive: {len(df_aspects_rep[(df_aspects_rep['aspect']==aspect)&(df_aspects_rep['sentiment']=='positive')])}, Negative: {len(df_aspects_rep[(df_aspects_rep['aspect']==aspect)&(df_aspects_rep['sentiment']=='negative')])}, Neutral: {len(df_aspects_rep[(df_aspects_rep['aspect']==aspect)&(df_aspects_rep['sentiment']=='neutral')])})\n"
        else: report_md += "ABSA not run or no results generated.\n"
        report_md += "\n---\n\n## 4. Advanced Statistical Analyses\n"
        cv_path = os.path.join(self.results_dir,'cross_validation_analysis_results.csv')
        report_md += "### A. Cross-Validation (CV) Analysis (Rule-Based Models)\n"
        if os.path.exists(cv_path):
            try:
                cv_df = pd.read_csv(cv_path,index_col=0); report_md += "K-fold CV results (Mean +/- Std Dev):\n\n| Model | Accuracy | MAE | F1 Close Acc. |\n|-------|----------|-----|---------------|\n"
                for mn,row in cv_df.iterrows(): report_md += f"| {mn} | {row.get('Accuracy (Mean)',0):.3f} +/- {row.get('Accuracy (Std)',0):.3f} | {row.get('MAE (Mean)',0):.3f} +/- {row.get('MAE (Std)',0):.3f} | {row.get('F1 Close Acc. (Mean)',0):.3f} +/- {row.get('F1 Close Acc. (Std)',0):.3f} |\n"
            except Exception as e: report_md += f"*Could not load CV results: {e}*\n"
        else: report_md += "CV analysis not performed/saved for rule-based models.\n"
        cluster_path = os.path.join(self.results_dir,'clustering_analysis_results.json')
        report_md += "\n### B. Clustering Analysis (on Rule-Based Features)\n"
        if os.path.exists(cluster_path):
            try:
                with open(cluster_path,'r') as f: data=json.load(f)
                report_md += f"K-Means clustering results:\n- **Optimal k:** {data.get('optimal_k','N/A')}\n- **Silhouette Score:** {data.get('best_silhouette_score','N/A'):.3f}\n\n**Cluster Profile Summary:**\n| Cluster ID | Num Reviews | Avg Rating | Top Category |\n|------------|-------------|------------|--------------|\n"
                for item in data.get('cluster_summary_analysis',[]): report_md += f"| {item.get('Cluster_ID')} | {item.get('Num_Reviews_in_Cluster'):,} | {item.get('Average_Rating',0.0):.2f} | {item.get('Top_Category','N/A')} |\n"
            except Exception as e: report_md += f"*Could not load clustering results: {e}*\n"
        else: report_md += "Clustering analysis not performed/saved.\n"
        report_md += "\n---\n\n## 5. Technical Notes\n"
        report_md += f"- **NLTK Status:** {'Initialized' if NLTK_INITIALIZED else 'Not Initialized/Failed'}\n"
        report_md += f"- **Transformer Status:** {'Available' if transformers_available else 'Not Available'}. Device: {TRANSFORMER_DEVICE}\n"
        report_md += f"- **TextBlob/Emoji Status:** {'Available' if textproc_available else 'Not Available (limits rule-based ABSA fallback)'}\n"
        report_md += f"- **Key Python Libraries Used:** Pandas, NumPy, Scikit-learn, Matplotlib, Seaborn, Plotly, NLTK, TextBlob, Transformers, Torch, Requests.\n"
        if hasattr(self,'models_run_successfully'): report_md += f"- **Overall Sentiment Models Executed:** {', '.join(self.models_run_successfully) or 'None'}\n"
        report_md += "- **Overall Sentiment Transformer:** `cardiffnlp/twitter-roberta-base-sentiment-latest`.\n"
        report_md += "\n---\n\n## 6. Directory & Key Files\nResults in: `{self.results_dir}`\n- `sentiment_analysis_report_transformer_absa.md`: This report.\n- `sentiment_analysis_[timestamp].log`: Log file.\n- `extracted_rule_based_features.csv`.\n- `model_predictions_with_features.csv`.\n- `cross_validation_analysis_results.csv`.\n- `clustering_analysis_results.json`.\n- `interactive_model_dashboard.html`.\n- `worst_predictions_analysis/`: Per-model CSVs of worst predictions.\n- `absa_analysis/`: Raw JSON, aggregated CSV, and plots for ABSA.\n- Various PNG plots for data distributions, feature analysis, model comparisons.\n\n---\n\n## Recommendations & Next Steps\n"
        report_md += "1.  **Explore Dashboard & Worst Preds**: Use `interactive_model_dashboard.html` and CSVs in `worst_predictions_analysis/`. Full list of worst predictions displayed as table during evaluation.\n2.  **Scrutinize ABSA Data**: Review `absa_analysis/` outputs for fine-grained insights.\n3.  **Analyze Clusters**: Check `clustering_analysis_results.json` for customer segments.\n\n*Potential Improvements: Advanced ATE for ABSA, fine-tune Transformers, Transformer embeddings as features, advanced ensemble methods, category-specific Transformer models.*\n\n---\n\n*End of Sentiment Analysis Report.*\n"
        return report_md.strip()

    def run_complete_analysis(self, data_source: str, sample_size: Optional[int] = None,
                            include_cross_validation: bool = True,
                            include_clustering: bool = True) -> None:
        self.logger.info(f"Starting complete analysis with Transformers & ABSA. Data source: {data_source}, Sample size: {sample_size if sample_size is not None else 'Full Dataset'}")
        print(f"\n{'='*20} Starting Sentiment Analysis Pipeline Run (Transformer & ABSA Enhanced) {'='*20}")
        overall_start_time = time.time()
        try:
            self.load_data(data_source,sample_size)
            self.preprocess_data()
            self.extract_features_for_rule_based_models()
            self.run_absa()
            self.run_models()
            self.evaluate_models()
            self.generate_visualizations()

            if include_cross_validation:
                self.perform_cross_validation_analysis()
            else: self.logger.info("Cross-validation analysis skipped as per configuration.")

            if include_clustering:
                self.perform_clustering_analysis()
            else: self.logger.info("Clustering analysis skipped as per configuration.")

            self.generate_report()

            total_execution_time = time.time() - overall_start_time
            self.logger.info(f"Pipeline execution completed successfully in {total_execution_time:.2f} seconds.")
            display_section_header("Pipeline Execution Completed Successfully!",1)
            print(f"Total Execution Time: {total_execution_time:.2f} seconds ({total_execution_time/60:.2f} minutes).")
            print(f"Number of Reviews Processed: {len(self.texts):,}")
            print(f"All results, logs, and visualizations saved in directory: {self.results_dir}")

            if hasattr(self,'evaluation_results') and self.evaluation_results:
                best_model_final, best_f1_close_final = None, -1.0
                for model_name_final, eval_data_final in self.evaluation_results.items():
                    f1_close_score_final = eval_data_final.get('metrics',{}).get('f1_close',-1.0)
                    if f1_close_score_final > best_f1_close_final:
                        best_f1_close_final, best_model_final = f1_close_score_final, model_name_final
                if best_model_final:
                    print(f"Best Performing Model (Overall Rating, F1 Score +/-1 Star): **{best_model_final}** (Score: {best_f1_close_final:.4f})")

            print(f"\nKEY CAPABILITIES UTILIZED IN THIS RUN:")
            print(f"   - Rule-Based Sentiment Models with N-gram Features")
            if transformers_available and self.roberta_sentiment_predictor and self.roberta_sentiment_predictor.pipeline :
                print(f"   - Transformer-Based Overall Sentiment Prediction (e.g., RoBERTa)")
            else:
                print(f"   - Transformer-Based Overall Sentiment Prediction (SKIPPED - dependencies not met)")

            absa_transformer_ok = self.absa_analyzer and self.absa_analyzer.absa_pipeline and not self.absa_analyzer.use_rule_based_absa_fallback
            absa_fallback_ok = self.absa_analyzer and self.absa_analyzer.use_rule_based_absa_fallback and textproc_available and self.absa_analyzer.sent_tokenizer_nltk
            if absa_transformer_ok:
                 print(f"   - Fine-grained Aspect-Based Sentiment Analysis (ABSA via Transformer)")
            elif absa_fallback_ok:
                 print(f"   - Fine-grained Aspect-Based Sentiment Analysis (ABSA via Rule-Based Fallback)")
            else:
                 print(f"   - Fine-grained Aspect-Based Sentiment Analysis (ABSA) (SKIPPED - dependencies not met for Transformer or Fallback)")


            print(f"   - Worst Prediction Analysis (Full Text Table Display & CSVs)")
            if include_cross_validation: print(f"   - Cross-Validation Analysis Performed (Rule-Based Models)")
            if include_clustering: print(f"   - Clustering Analysis Performed (on Rule-Based Features)")
            print(f"   - Interactive Model Comparison Dashboard (Plotly HTML)")
            print(f"   - Additional Feature Visualizations (Distributions, Correlation Heatmap)")
            print(f"   - Display of Top 50 Model Predictions with Full Text in Jupyter")  # FIXED: Changed from 300 to 50
            print(f"   - Display of Top 50 Rule-Based Features with Full Text Preview in Jupyter")  # FIXED: Changed from 300 to 50

        except Exception as e_pipeline_main:
            self.logger.critical(f"CRITICAL PIPELINE ERROR: {e_pipeline_main}", exc_info=True)
            print(f"\n{'!'*25} PIPELINE EXECUTION FAILED! {'!'*25}\nError: {e_pipeline_main}")
            print(f"Please check the log file in '{self.results_dir}' for detailed error information.")
        finally:
            print(f"\n{'='*25} End of Pipeline Run {'='*25}")

#%% === DEMONSTRATION ===
def run_pipeline_demo():
    print("\n"+"="*70+"\nSENTIMENT ANALYSIS PIPELINE DEMO (Transformer & ABSA Enhanced)\n"+"="*70+"\n")
    try: pipeline_instance = SentimentAnalysisPipeline()
    except Exception as e: print(f"CRITICAL ERROR: Pipeline init failed: {e}\nDemo cannot continue."); return None
    local_data='sentiment_data.csv'; remote_url='https://raw.githubusercontent.com/VaRkanar/datasets_/master/sentiment_data.csv'
    data_src = local_data if os.path.exists(local_data) else remote_url; src_type = "(local)" if data_src==local_data else "(remote, will download)"
    print(f"Using data source: {data_src} {src_type}")

    sample_sz = None
    print(f"Demo config - Sample: {'Full Dataset' if sample_sz is None else f'{sample_sz:,}'}")
    if sample_sz is None or (isinstance(sample_sz, int) and sample_sz > 50000):
        print("WARNING: Processing a large dataset (or full dataset) with Transformers/ABSA can be very time and memory intensive.")

    run_cv=True; run_cluster=True
    print(f"Analysis features in demo: Cross-Validation={run_cv}, Clustering={run_cluster}\n"+"-"*70)

    pipeline_instance.run_complete_analysis(data_source=data_src,sample_size=sample_sz,include_cross_validation=run_cv,include_clustering=run_cluster)
    print("\n"+"="*70+"\nDEMONSTRATION RUN COMPLETE\n"+"="*70+f"\nOutputs, logs, reports in: '{pipeline_instance.results_dir}'")
    print("\nKey capabilities demonstrated:\n - Rule-Based & Transformer Overall Sentiment\n - ABSA (with Rule-Based Fallback)\n - Worst Prediction Analysis (Full Text Table Display & CSVs)\n - Enhanced Feature Visualizations\n - Top 50 Model Predictions with Full Text Display\n - Top 50 Rule-Based Features with Full Text Preview Display")  # FIXED: Changed from 300 to 50
    if run_cv: print(" - Cross-Validation (Rule-Based)")
    if run_cluster: print(" - Clustering (Rule-Based Features)")
    print(" - Interactive Dashboard (Plotly HTML)\n - Markdown Reporting\n - Alternative Rating Distribution Plot")
    return pipeline_instance

#%% === JUPYTER NOTEBOOK EXECUTION ===

if __name__ == "__main__" or is_jupyter_environment():
    print("\nSentiment Analysis Pipeline (Transformer & ABSA Enhanced) - Script Loaded & Ready")
    print("="*70+"\nThis version includes:\n - Transformer Models (e.g., RoBERTa) for overall sentiment.\n - Fine-grained Aspect-Based Sentiment Analysis (ABSA) with rule-based fallback.\n - Worst Prediction Analysis with full review text displayed in a Pandas table.\n - Additional visualizations for rule-based features (distributions, heatmap).\n - Alternative plot for 'Rating Distribution in Top Categories'.\n - Display of Top 50 model predictions with full text in Jupyter output.\n - Display of Top 50 rule-based features with full text preview in Jupyter output.\n - Option to use the full dataset (sample_size=None).\n - Preserved rule-based models and evaluation features.\n - Cross-Validation and Clustering analyses.\n - Interactive Model Comparison Dashboard (Plotly HTML).\n - Comprehensive Markdown Reporting.\n"+"="*70)  # FIXED: Changed from 300 to 50
    print("\nTo execute in Jupyter:\n1. Run CELL 1 (this cell).\n2. Sequentially execute CELLs 2-10.\n3. Or call `run_pipeline_demo()` for automated run.")
    print("\nIf running as standalone script: Uncomment `run_pipeline_demo()` at script end.")

# CELL 2 (Recommended for Jupyter)
if is_jupyter_environment() or __name__ == "__main__":
    if 'pipeline' not in locals() or not isinstance(pipeline, SentimentAnalysisPipeline):
        pipeline = SentimentAnalysisPipeline()
        print(f"\nPipeline instance ('pipeline') created. Results in: {pipeline.results_dir}")
    else:
        print("\nPipeline instance already exists.")


# CELL 3
DATA_SOURCE_NOTEBOOK = 'sentiment_data.csv'
if not os.path.exists(DATA_SOURCE_NOTEBOOK):
    print(f"Local file '{DATA_SOURCE_NOTEBOOK}' not found. Using remote URL as fallback for notebook execution.")
    DATA_SOURCE_NOTEBOOK = 'https://raw.githubusercontent.com/VaRkanar/datasets_/master/sentiment_data.csv'

SAMPLE_SIZE_NOTEBOOK = None
print(f"Notebook Cell 3: Using data_source = '{DATA_SOURCE_NOTEBOOK}'")
print(f"Notebook Cell 3: Using sample_size = {'Full Dataset' if SAMPLE_SIZE_NOTEBOOK is None else f'{SAMPLE_SIZE_NOTEBOOK:,}'}")


if 'pipeline' in locals() and isinstance(pipeline,SentimentAnalysisPipeline):
    try:
        pipeline.load_data(DATA_SOURCE_NOTEBOOK,SAMPLE_SIZE_NOTEBOOK)
    except Exception as e:
        print(f"Error in Cell 3 (load_data): {e}")
        pipeline.logger.error(f"Error in Cell 3 (load_data): {e}", exc_info=True)
else: print("Warning: Pipeline not initialized. Run CELL 2 first.")

# CELL 4
if 'pipeline' in locals() and hasattr(pipeline,'texts') and pipeline.texts:
    try:
        pipeline.preprocess_data()
    except Exception as e:
        print(f"Error in Cell 4 (preprocess_data): {e}")
        pipeline.logger.error(f"Error in Cell 4 (preprocess_data): {e}", exc_info=True)
else: print("Warning: No data loaded or pipeline not initialized. Run CELLs 2 & 3 first.")

# CELL 5 (Feature Extraction & ABSA)
if 'pipeline' in locals() and hasattr(pipeline,'texts') and pipeline.texts:
    try:
        pipeline.extract_features_for_rule_based_models()
        pipeline.run_absa()
    except Exception as e:
        print(f"Error in Cell 5 (extract_features_for_rule_based_models or run_absa): {e}")
        pipeline.logger.error(f"Error in Cell 5: {e}", exc_info=True)
else: print("Warning: Original text data not available. Run previous cells.")


# CELL 6
if 'pipeline' in locals() and ((hasattr(pipeline,'features_df') and not pipeline.features_df.empty) or \
                                (hasattr(pipeline,'roberta_sentiment_predictor') and pipeline.roberta_sentiment_predictor and pipeline.roberta_sentiment_predictor.pipeline)):
    try:
        pipeline.run_models()
    except Exception as e:
        print(f"Error in Cell 6 (run_models): {e}")
        pipeline.logger.error(f"Error in Cell 6 (run_models): {e}", exc_info=True)
else: print("Warning: Features or Transformer model not ready. Run previous cells.")

# CELL 7
if 'pipeline' in locals() and hasattr(pipeline,'final_df') and not pipeline.final_df.empty:
    try:
        pipeline.evaluate_models()
    except Exception as e:
        print(f"Error in Cell 7 (evaluate_models): {e}")
        pipeline.logger.error(f"Error in Cell 7 (evaluate_models): {e}", exc_info=True)
else: print("Warning: Models not run or final_df not available. Run CELL 6 first.")

# CELL 8
if 'pipeline' in locals() and hasattr(pipeline,'texts') and pipeline.texts:
    try:
        pipeline.generate_visualizations()
    except Exception as e:
        print(f"Error in Cell 8 (generate_visualizations): {e}")
        pipeline.logger.error(f"Error in Cell 8 (generate_visualizations): {e}", exc_info=True)
else: print("Warning: Insufficient data for visualizations.")

# CELL 9 (CV and Clustering)
if 'pipeline' in locals() and hasattr(pipeline,'features_df') and (not pipeline.features_df.empty or pipeline.roberta_sentiment_predictor is not None) :
    try:
        print("\n--- Starting Cross-Validation Analysis (Rule-Based Models) ---"); pipeline.perform_cross_validation_analysis()
        print("\n--- Starting Clustering Analysis (on Rule-Based Features) ---"); pipeline.perform_clustering_analysis()
    except Exception as e:
        print(f"Error in Cell 9 (CV/Clustering): {e}")
        pipeline.logger.error(f"Error in Cell 9 (CV/Clustering): {e}", exc_info=True)
else: print("Warning: Features DataFrame not available for CV or Clustering. Run relevant previous cells.")

# CELL 10
if 'pipeline' in locals():
    try:
        pipeline.generate_report()
    except Exception as e:
        print(f"Error in Cell 10 (generate_report): {e}")
        pipeline.logger.error(f"Error in Cell 10 (generate_report): {e}", exc_info=True)
else: print("Warning: Pipeline instance not available. Cannot generate report.")


# --- Main execution guard for standalone script running ---
if __name__ == "__main__" and not is_jupyter_environment():
    print("\nThis script is best experienced in a Jupyter Notebook or IPython environment.")
    print("To run the full demo in script mode, uncomment the line below.")
    # run_pipeline_demo()
    pass

Rank	Model	±1 Star Acc	F1 Macro	MAE	Exact Acc
#1	Advanced Ensemble (CHAMPION)	82.91%	0.3250	0.7437	48.93%
#2	Feature-Based (RULE-BASED CHAMPION)	81.07%	0.3204	0.7590	53.27%
#3	Multidimensional	82.61%	0.2539	0.7795	44.62%
#4	Hybrid Weighted	81.94%	0.3089	0.8136	45.99%
#5	RoBERTa Sentiment (PRECISION LEADER)	80.50%	0.2871	0.7020	58.05%
#6	N-gram Enhanced	81.51%	0.2830	0.8261	41.57%
#7	VADER	80.25%	0.3053	0.8260	48.15%

Model	Accuracy (Mean ± Std)	MAE (Mean ± Std)	F1 Close Acc. (Mean ± Std)
Feature-Based	51.97% ± 0.22%	0.7735 ± 0.0053	0.8937 ± 0.0015
VADER	46.86% ± 0.16%	0.8415 ± 0.0038	0.8882 ± 0.0016
N-gram Enhanced	40.59% ± 0.27%	0.8291 ± 0.0061	0.8988 ± 0.0017
TextBlob	15.99% ± 0.10%	1.2819 ± 0.0031	0.7309 ± 0.0022

	full_text_preview	text_length	word_count	sentence_count	avg_word_length	vader_negative	vader_neutral	vader_positive	vader_compound	textblob_polarity	textblob_subjectivity	exclamation_count	question_count	caps_ratio	digit_count	mentions_5_star	mentions_4_star	mentions_3_star	mentions_2_star	mentions_1_star	positive_word_count	negative_word_count	intensifier_count	negation_count	sentiment_ratio	domain_positive_count	domain_negative_count	2gram_count	2gram_positive_count	2gram_negative_count	3gram_count	3gram_positive_count	3gram_negative_count	custom_sentiment_score
0	What I recieved is not what is pictured here OR ADVERTIZED, but it is a vidio cable, and it does do the job I needed it for. If you were looking for a high quality cable do not buy this product. I recieved a cheap thin flimsy cable that is not gold plated or anything else they claim here. Bait and Switch? or a mistake in the order !!!!! . I do not know. I will attempt to contact the sender and find out. I contacted the seller and they told me to contact amazon.com so i did, and amazon.com was good enough to give me a $4.00 refund; which is good, because all this cable is woth is $2 to $3. I only needed it to run my wife's kareokee to the TV. If it was for any thing else I would have sent it back for a full refund. I'm surprised that this item is still listed as a high grade cable.	791	167	15	3.586826	0.020000	0.909000	0.071000	0.844900	0.142727	0.616364	5	1	0.035398	5	False	False	False	False	False	2	2	0	0	0.500000	1	2	58	6	6	57	9	9	0.232015
1	Excellent unit and a pretty simple install using the wiring adapter and removal tools I purchased with it. Kenwood quality is evident in the look and sound today and bodes well for the future. If you get the extra stuff its really simple to install--just be sure you have the frame on correctly (I 'd use a sharpie and mark it 'top' and 'front' before you slide it off the radio to bend the tabs out so it will fit snugly in the dash). This one went in my wife's 99 Miata and I'll likely do another in mine because it will be even simpler the next time. She also reports that the CD player is pretty skip-free, which isn't easy in a sports car.	644	130	5	3.853846	0.018000	0.853000	0.129000	0.930700	0.265556	0.547434	0	0	0.017081	2	False	False	False	False	False	3	0	1	0	1.000000	2	0	52	5	0	51	7	0	0.832321
2	I'm enjoying this keyboard, I'm getting another one for my other laptop that I keep at home.I've had this now for almost a year. Yes, I experience the BT lag, too, but I live with it. (Hence the 4 stars instead of 5) *UPDATE* (Now 5 stars--see update below.) Would I buy it again? Well, yes...as I said, I'm buying another. One BT annoyance is that you can pair your device with only one other device.Great layout, excellent size, superb tactile feel. One big advantage with a wireless keyboard and using a note/netbook is that you are no longer at the mercy of how high the desk/table/airline tray is--you can have the comfort (and more healthy) height of typing from your lap. I love it!I love the slim design and the power off button--it fits great in my laptop backpack. It is so light, too. With my 2.8 lbs. Asus netbook, adding this lightweight is really nothing. And because of its size, it fits in any bag, attaché, backpack, etc. easily.*UPDATE*Been using my two MS Mobile 6000's for a year now. Still loving it. Also--I am using it on my iPod Touch! I no longer am taking my netbook or my laptop to school. Just this keyboard and my Touch. Can't beat that.I came across a post that helped me solve any BT wake up issues. Instead of 20 seconds or so of waiting, it only takes about 2 seconds to wake up. Also, since replacing my laptop, I was experiencing seriously annoying disconnections, sometimes requiring deleting it from BT and reinstalling it. This blog post solved it all. In my case, all I needed to do was to clear the checkbox "Allow the computer to turn off this device to save power" in Device Manager and NO MORE PROBLEMS!!! Here's the link:[...]Enjoy!	1691	327	31	3.944954	0.074000	0.742000	0.185000	0.991800	0.261732	0.622540	6	1	0.048492	12	True	True	False	False	False	4	2	1	1	0.666667	2	1	142	11	4	141	14	6	0.526553
3	Overall, this is a fantastic camera that I'm extremely pleased with.However, I bought it to take pictures indoors without flash. I was prepared for lower quality images indoors without a flash, but the Lumix does not do nearly as well as my Canon Elph in identical lighting situations. The flash is good, but if you want a camera to shoot indoors without flash, be prepared for fairly noisy images with a lot of artifacts.	422	76	4	4.460526	0.125000	0.721000	0.154000	0.504700	0.400000	0.633333	0	0	0.021327	0	False	False	False	False	False	4	0	1	1	1.000000	1	1	39	8	0	38	11	0	0.581410
4	These work very well with mySamsung PN64D7000 64-Inch 1080p 600 Hz 3D Plasma HDTV (Black). They come in a nice felt bag inside a zipped hard case. They fit very well even over my glasses (which are 58 mm from side to side FWIW). They charge via a microsub cable (included, in case you don't already have a bunch). I like them more than my Samsung SSG-3050s. I'll buy more of these. One tip -- they don't seem to auto-associate the first time you turn them on. So, first fully charge them, then turn on your TV, then press and hold the power button until it blinks rapidly. Then it will associate and work great.	611	120	9	3.916667	0.012000	0.851000	0.138000	0.935900	0.294697	0.490152	0	0	0.052373	22	False	False	False	False	False	2	0	0	0	1.000000	1	0	53	3	0	52	4	0	0.839709
5	Its much more powerful then I expected!!!!!BIG,DEEP ,WINDOW BREAKING BASS What I really like about it you can fine tune it to your liking. Some subs sound good with movies,and terrible with music, this sub is truly fantastic with both.	236	43	3	4.302326	0.057000	0.555000	0.388000	0.950600	0.196296	0.622222	5	0	0.127119	0	False	False	False	False	False	2	1	2	0	0.666667	1	0	21	3	2	20	4	3	0.574915
6	A lot of computer fans come with the small 3 pin connector and no adapter to plug in to the power supply. And a lot of motherboards do not have any spare fan power pins. Therefore, a small pile of these is handy to have around if you ever have to add or replace certain fans.I found these at two bucks each with no additional shipping charge (Prime subscription) and ordered 5 along with one of these:Cooler Master 120mm Silent Case Fan 4-in-1 Value Pack - (R4-S2S-124K-GP)	475	93	3	4.000000	0.050000	0.847000	0.103000	0.571900	0.062857	0.474286	0	0	0.037895	12	False	False	False	False	False	0	0	0	0	0.500000	0	1	43	0	0	42	0	0	0.034141
7	I finally found a 10 foot lightning cable to replace the old iPhone cable that I travel with. Because so many hotel rooms don't have convenient electric plugs, the long cable had been a lifesaver. This one works fine but it is so thick and heavy that it's not really suitable for travel applications.	300	56	3	4.303571	0.000000	0.927000	0.073000	0.421500	0.023958	0.540625	0	0	0.016667	2	False	False	False	False	False	0	0	1	0	0.500000	0	1	26	0	0	25	0	0	-0.018758
8	VERY POOR QUALITY I USED IT ONCE AND THE CONNECTION BROKE THE FIRST DAY !! IT SHOULD HAVE BEEN ON A SMALL CABLE	111	22	1	3.954545	0.321000	0.679000	0.000000	-0.874300	-0.126458	0.504444	2	0	0.783784	0	False	False	False	False	False	1	1	0	0	0.500000	1	2	7	2	1	6	2	1	-0.304248
9	I got this camera years ago! I still work with it occasionally, but back in it's hayday, I used it for bulb photography with lightning. The plus was using infinity and not autofocus (because the autofocus is horrendous), but the negatives were a maximum 60 second bulb and lots of noise. Of course, back when this camera was $1,000+, it was pretty good. Still, in the daylight it takes beautifully vivid photos. I upgraded the firmware on it just before Konica halted the production of all cameras, which was supposed to fix the autofocus, but didn't make any difference to me. It's been through rain, snow and heat and has still held up. She's probably been through at least 10,000 pictures (lots of lightning photos that contained no lightning, probably every 95 in 100 were deleted). Very robust. Still gets four stars despite it's problems. I'll keep it til it dies!	882	159	11	4.308176	0.017000	0.859000	0.124000	0.960700	0.113889	0.352778	2	0	0.014739	16	False	True	False	False	False	1	1	0	0	0.500000	0	0	70	2	2	69	3	3	0.310988
10	I've been using this for the last 3 years or so. It is excellent -- great range, and I cannot recall last time I changed the battery. I have young kids, they've taken it apart (removed the track ball) million times, smacked it into the ground, thrown it, no problems. Works like a charm. I'm going to get another one for my 2nd HTPC.	333	66	5	3.833333	0.069000	0.745000	0.186000	0.854200	0.316667	0.380556	0	0	0.033033	2	False	False	False	False	False	2	1	0	0	0.666667	1	0	27	3	2	26	4	3	0.570070
11	I purchased this huge headset to use for music and softphone. It sounds great in stereo and 7.1 but the mic is absolutely useless. All the other end hears is a humming background noise. They cant make out what im saying so it has to go....on to the next one.	258	52	6	3.846154	0.078000	0.844000	0.079000	-0.235700	0.115000	0.445000	0	0	0.015504	2	False	False	False	False	False	1	1	1	0	0.500000	0	0	21	2	2	20	3	3	-0.047710
12	This is definitely a replacement for the original jambox. Personally I find that it has a louder and crisper sound as compared to the to the original. The sound for it's size is great. Perfect for a office, bedroom or any setting for maybe 4 people max in a small room. The build is solid and has a good weight to it. The battery life is great. I was able to get over 8 hours of continuous play with the volume at 70%. Overall I'm very pleased with this speaker and highly recommend it.	486	96	8	3.968750	0.000000	0.698000	0.302000	0.983500	0.369375	0.554062	0	0	0.020576	4	False	False	False	False	False	5	0	1	0	1.000000	1	1	39	10	0	38	14	0	0.718925
13	This adapter does its job and does it well. Unlike the Apple MacBook power adapters, this consistently works. I've had the kindle for a few years now and the adapter is durable and doesn't show any signs of wear.	214	41	3	4.097561	0.000000	0.946000	0.054000	0.273200	0.025000	0.175000	0	0	0.028037	0	False	False	False	False	False	1	0	0	0	1.000000	1	0	16	2	0	15	3	0	0.586960
14	I have used one of these mounts for over a year now. This is my second for my second car and second Garmin. THis is a superior mount to the suction mount in my opinion. It is much easier to use and stays where you place it. I have traveled cross country with this and had absolutely no problems with it. It sits properly on my dash; it is easy to mount the Garmin on it and it is easier to read and hear from the distance that it is from me as opposed to the suction mount. I found the suction mount hard to use. It was a very long reach to mount it to my windshield and since I was traveling to California it was also illegal. The bean bag mount hasn't disappointed me yet. It has yet to move from the place that I have set it. I have not had a near accident with it in place so I don't know how it stays in place during emergency maneuvers, but so far I would say it is going to work as well as any other mounting device. I highly recommend this mount.	954	198	12	3.752525	0.046000	0.857000	0.097000	0.842100	-0.008137	0.444706	0	0	0.020964	0	False	False	False	False	False	2	1	2	0	0.666667	1	1	72	4	2	71	5	3	0.363503
15	So far so good with this shoulder strap. I didn't have much faith in this product at first.But after trying it a few times its a Great Strap. I added an extra secure line to the camera just in case.I use this on my backup inexpensive body. I still wouldn't put my expensive body or any of the L lens on it. I rather play it safe than sorry :)Definitely a lot more comfortable over the straps that is provided with the camera. Next step is to buy a reliable Black Rapid Fastenr (FR-3)to replace this cheap one. I will	520	106	8	3.792453	0.035000	0.780000	0.185000	0.950400	0.184314	0.557843	0	0	0.034615	1	False	False	False	False	False	4	1	0	0	0.800000	1	0	43	8	1	42	11	1	0.702538
16	Brought for husband to use. Good fit for his surround sound system. He's happy with the purchase and so am I	109	22	2	3.863636	0.000000	0.651000	0.349000	0.844200	0.575000	0.600000	0	0	0.036697	0	False	False	False	False	False	2	0	0	0	1.000000	1	0	7	3	0	6	4	0	0.868260
17	While originally looking to overhauling my old PC, I was actually shown this case by a fellow Amazon customer after they brought it to my attention, and figured that I'm actually better off having one built from complete scratch.But what really made this case unique, was not only does it light blue once turned on, but having 5 pre-installed fans built into it and also having the option to add 2 more for a total of 7 fans (good if you're a heavy overclocker), which gives an excellent amount of airflow. So for those who like building computers from scratch, I highly recommend getting this case!!!!	603	110	3	4.363636	0.000000	0.850000	0.150000	0.947300	0.277539	0.552500	4	0	0.014925	3	False	False	False	False	False	3	0	2	0	1.000000	1	0	47	6	0	46	9	0	0.839698
18	This keyboard is awesome. I honestly could do without the light that comes on whenever you get close to it, but I guess it's good because it reminds you that you left it on and therefore that could be a battery saver. I have only tested it with my apple products (Macbook Pro, iPhone 5S and iPad air) and it works flawlessly. I have not tested it with Windows but I have heard that it works well there too.	406	80	4	3.987500	0.000000	0.837000	0.163000	0.905200	0.528571	0.742857	0	0	0.029557	1	False	False	False	False	False	2	0	0	1	1.000000	1	0	29	4	0	28	5	0	0.877274
19	I have 16gb of these in my Maximus IV Extrme z and they work great. Windows gave them a 7.9 the highest score on the ratings.	125	27	3	3.592593	0.000000	0.843000	0.157000	0.624900	0.800000	0.750000	0	0	0.048000	4	False	False	False	False	False	1	0	0	0	1.000000	0	0	9	2	0	8	3	0	0.697470
20	I am very happy with this card; good value, I would buy it again and would recommend it to others.	98	20	1	3.800000	0.000000	0.543000	0.457000	0.895500	0.850000	0.800000	0	0	0.020408	0	False	False	False	False	False	3	0	0	0	1.000000	1	0	5	5	0	4	4	0	0.938650
21	This case is perfect for my SD600. My only complaint is that if the wrist-strap is not threaded through the top hole it doesn't close right. Minor issue. It's small, has a belt loop (not a clip, which is not dependable) and it's a hard case so it'll protect the camera in a handbag or pocket. I recommend it.	312	64	5	3.687500	0.058000	0.768000	0.174000	0.806200	0.111756	0.547173	0	0	0.022436	3	False	False	False	False	False	2	2	0	0	0.500000	1	1	23	3	4	22	3	6	0.240402
22	I bought this for my father to replace the crappy one that I snapped in half. I must say that I love this thing. The price is high, but the fact that it feels so substantial, moreso than any other keyboard I have ever used before, and that the backlighting automatically pops up when you are within an inch of the keys, makes it almost feel intuitive. very good wireless keyboard for those who have a seperate mouse and dislike trackpads.	441	81	4	4.320988	0.067000	0.817000	0.116000	0.641500	0.255667	0.492333	0	0	0.013605	0	False	False	False	False	False	2	0	0	0	1.000000	1	0	32	4	0	31	6	0	0.743583
23	Now I hated the stupid mouse that came with my imac and wasn't to unhappy when the little scroll ball finally stopped working, but this mouse is just junk. At least on my imac the dependably of it is less than 50%. More than half of my clicks don't register and the movement is sometimes smooth but mostly choppy. Plus no side scrolling is a downer. I know for $18 what was i expecting right. I can't speak for other computers, but if your looking for a replacement for an imac then DO NOT buy this crappy mouse.	512	101	6	3.980198	0.145000	0.838000	0.016000	-0.922500	-0.173855	0.526465	0	0	0.023438	4	False	False	False	False	False	1	1	0	0	0.500000	0	0	40	2	2	39	3	3	-0.311521
24	I purchased this to charge the batteries for my Motorola Verizon Droid 2X Global phone. It works really well. It was a replacement for another wall charger. I purchased the other and it worked but I lost it so I purchased another and it was the same but the batteries no longer fit. This one is great as the contacts are somewhat reconfigurable and the length can be adjusted with a spring to hold the battery in so no need for a precise fit which was the problem with other battery charger. The other issue is this can apparently automatically sense the polarity of the battery contacts and so you don't have to worry about accidentally putting it in backwards which would be a potential issue for a reconfigurable battery charger like this is. Assuming this lasts, this is a great product and works like it should.	823	148	7	4.466216	0.090000	0.737000	0.173000	0.942300	0.205769	0.453846	0	0	0.017011	1	False	False	False	False	False	1	2	1	0	0.333333	1	0	59	4	6	58	6	9	0.370511
25	This works great with my iphone 5 and other android devices one thing to know though is you'll need up to date bluetooth capabilities for example my macbook air mid2011 will not pair with this device.I would still recommend this for sure to anyone with bluetooth firmware more up to date.	288	53	2	4.433962	0.000000	0.841000	0.159000	0.836000	0.418750	0.628472	0	0	0.006944	5	False	False	False	False	False	2	0	0	0	1.000000	1	0	23	4	0	22	5	0	0.834550
26	Love everything about it. Would have given it a five star but at the time that I purchased it not many compatible apps.	119	23	2	4.130435	0.000000	0.885000	0.115000	0.381800	0.125000	0.550000	0	0	0.025210	0	True	False	False	False	False	2	0	0	0	1.000000	1	0	8	3	0	7	3	0	0.639540
27	We tried this out with both the iPod and the iPhone. It's a nice little alarm clock with an AM/FM radio and a dock for your device.There is an application available that lets you specify bass and treble levels. You can toggle military or U.S. time. You can sync your iPod Touch or iPhone automatically with the clock. You can set two alarms on your phone or iPod, and you can set up the music you want for your alarm. There's a dimmer for the clock, though we didn't find it too obtrusive.It charges your phone, and we like it for that because it looks better than a charger.There's a remote, but it doesn't sync to your device and let you identify and jump to a particular song. There are remotes out now with this feature.It's attractive, the sound quality is decent, and you can move the speakers away from the clock to suit your preferences. We think this would be good for a dorm room.This is a nice item and it does its job.	930	188	15	3.872340	0.034000	0.865000	0.101000	0.917600	0.291987	0.553846	0	0	0.026882	0	False	False	False	False	False	3	0	0	0	1.000000	1	0	70	8	0	69	11	0	0.833677
28	I only have this for a day, but I did some testing on this, importing some photos and a video to my 4th generation iPad. There is a small card packed with the device that lists the problems that you may have, so some of these are "documented bugs". I got one message that this is an unsupported device, the card says to ignore this. It occasionally is difficult to get the device to be recognized, but some tinkering with the sequence of connection and launch of the photo app.eventually brings it up (documented). The imports were all successful. IOS crashed twice while I was testing (not documented). I think that this can be prevented by ending the photo program before you disconnect the device. Since this is an intermittent problem, I am not sure of this fix, but it didn't crash after I tried this. In summary, this is much cheaper than the Apple device and it does work, but with some quirks.	901	166	10	4.289157	0.111000	0.836000	0.053000	-0.873200	-0.008333	0.739815	0	0	0.018868	1	False	False	False	False	False	0	2	0	0	0.000000	1	2	63	0	6	62	0	9	-0.663627
29	I got this because I read that if you shoot video with a DSLR, you need a fast card with fast write times. They were correct. I use this to shoot videos and it works very well. I have since retired it after 2 years of use for an even faster card by the same manufacturer. This card served me very well and I highly recommend these cards to anyone who shoots video from and SLR or from and digital video camera that uses SD cards. Exceptional product.	450	88	6	4.045455	0.055000	0.858000	0.087000	0.417400	0.203333	0.433125	0	0	0.037778	1	False	False	False	False	False	2	0	1	0	1.000000	2	0	38	6	0	37	8	0	0.665887
30	I am very pleased with the NE56R34u laptop. Since it is my first laptop I cannot judge from other models.The first thing I did was upgrade the ram to 8 gigs which made it very speedy. I also upgraded to windows 8 pro and now this baby screams.Even though this has a low-end Pentium processor it works for what I need it to do which is word processing, email, watching videos and light web browsing. Cannot go wrong for the price ( 320.00 ) from Amazon.	452	88	7	4.034091	0.000000	0.907000	0.093000	0.754100	0.160714	0.563095	0	0	0.030973	11	False	False	False	False	False	1	0	0	0	1.000000	1	0	35	1	0	34	1	0	0.758373
31	I already have an Onkyo receiver as my home theater receiver and have been immensely satisfied with it. I decided to go with Onkyo again when setting up speakers above my kitchen cabinets for CD and radio listening when I am in the kitchen cooking or washing dishes, etc. The TX-8222 has proven to be a worthy purchase. The speaker wire posts are screw-on binding posts, not spring clips as in cheaper models by other manufacturers. Definitely makes it easier to connect wire to the back. The 50 watts are more than powerful enough to power a pair of Sony 3-way bookshelf speakers. FM radio reception is pretty good, and the unit remembers your station presets even if the power cord is unplugged from the outlet. My two complaints about the receiver are minor - there is no numerical volume indicator on the display when you raise or lower the volume, which would have been nice, and the volume dial on the receiver moves rather pointlessly when adjusting the volume by remote. For the price though, this model can't be beat if you're looking for a stereo receiver.	1066	192	9	4.468750	0.035000	0.849000	0.116000	0.944000	0.177222	0.565000	0	0	0.016886	7	False	False	False	False	False	3	1	0	0	0.750000	1	0	91	6	2	90	9	3	0.643644
32	Many gaming mousepads come with a travel case, this one doesn't. The mousepad is nice though.	94	17	2	4.352941	0.000000	0.833000	0.167000	0.421500	0.550000	0.750000	0	0	0.021277	0	False	False	False	False	False	1	0	0	0	1.000000	0	0	8	2	0	7	2	0	0.586450
33	The Plantronics GameCom 777 Surround Sound Gaming Headset has a few things going for it, but comfort and ease of set-up are not among them. After working in radio for many years, wearing headphones for hours per day, you learn quickly what does and doesn't work. As far as comfort goes, these simply don't work.Now I'll be the first to admit it can't be easy to design a pair of truly comfortable over the ear headphones (in broadcasting we call them cans). There are many things to consider; from variations in the user's ears and head width, to depth of cushioning, etc.I find the 777s to be too tight and the interior space of each ear piece to small. The result is a headphone that feels bad right from the start and only gets worses the longer you wear it.As far as set-up goes the 777 is a train wreck. No instructions, and a very hit ans miss driver. Sometimes the computer recognizes it, sometimes not.Once installed, the mic on the headset does it's job. but is nothing special.The one truly redeeming factor for this product is the sound. While there is of course no way to create a true surround sound experience with a headphone, I found the 777s to have good full drive, plenty of base and a pretty crisp high-end.	1227	237	14	4.084388	0.077000	0.772000	0.151000	0.963800	0.175916	0.578022	0	0	0.017930	12	False	False	False	False	False	3	1	1	1	0.750000	1	2	111	6	2	110	9	3	0.449323
34	I bought this TV to replace my old Sony tube TV which was defective and had a greenish screen. That TV was only a few years old, too, which doesn't say too much for Sony products. This TV, however, is so beautiful; it is FAR superior to the old tube TV I had. There is just no comparison. The colors are vibrant and the contrast is amazing, especially in HD. It is my first HD TV and I can definitely see why people love these TVs so much.The sound is just okay. I do not have the sound distortion problem resulting from using the QAM tuner and not a cable box with Comcast with this TV. Hopefully, I will never get this problem since it is a major issue with the 32" Samsung TV we bought shortly after this one. You can see what I mean by reading my review on the LN32A330 model. Apparently, most of Samsung's tuners are defective in these TVs and do not work well with cable signals (mainly Comcast) unless there is a cable box. This results in distorted or "robotic" sound on HD channels at regular intervals.I highly recommend this TV ONLY if you use a cable box. I personally wouldn't take a chance with it if you have Comcast or any other cable without a cable box. My 26" doesn't have the sound problem, but it could only be a matter of time before it pops up, which I have heard does happen.	1299	256	15	3.964844	0.063000	0.855000	0.082000	0.715400	0.198978	0.506863	0	0	0.052348	9	False	False	False	False	False	4	3	2	2	0.571429	1	1	105	8	12	104	12	18	0.270130
35	I recently tried a Bioserie Bioplastic Cover for iPod touch 4G which costs $24.95. It kept losing it's shape and spitting out my ipod touch and it only offered protection from scratches.This Amazon basic cover at less than $10 does hold my ipod and it's design will not only protect from scratches on the back but also protect the device if I drop it.All the buttons still work fine and it's easy to access the charging slot.I keep my ipod in my pocket and the cover does not interfere with that.	496	98	6	4.030612	0.091000	0.833000	0.076000	-0.107300	0.085417	0.471875	0	0	0.024194	7	False	False	False	False	False	1	0	0	0	1.000000	0	0	41	2	0	40	3	0	0.334893
36	Works really well. We have 4 different mount configurations, and this has been able to handle them all. The setup makes it easy to get everything setup, then installed.	168	29	3	4.655172	0.000000	0.831000	0.169000	0.647800	0.283333	0.564583	0	0	0.017857	1	False	False	False	False	False	1	0	1	0	1.000000	0	0	13	2	0	12	3	0	0.601007
37	I was happy to be able to acquire this GLASS screen cover. It is about an 1/8 of an inch smaller side to side than the screen on my Nikon 1 J2! Provides solid protection and does not cover up any needing information on the screen. Craig S.	239	49	4	3.816327	0.000000	0.887000	0.113000	0.680000	0.325000	0.556250	1	0	0.050209	4	False	False	False	False	False	1	0	0	0	1.000000	1	0	19	1	0	18	1	0	0.769000
38	We bought this so my wife would have a secondary monitor setup that she could use wherever: home office, work, or wherever. It works great for her on her WIndows laptops (both are Thinkpads). The integrated stand/case is great and the display is even fast enough that web videos (e.g. youtube) are watchable. They're not as good as native display, but the frame rate is fine. The whole setup is reasonably lightweight and portable.I have also used it briefly with my Macs. While Toshiba doesn't include the driver on the included CD rom, if you go to the DisplayLink.com web site, you can download OSX drivers from there.The only minor issue we have with it is that the place for storing the included USB cord in the cover/stand isn't the best design. It works OK, but it's not the most convenient way to store such a cable.	832	157	12	4.133758	0.040000	0.888000	0.072000	0.605200	0.310417	0.520833	0	0	0.031250	0	False	False	False	False	False	4	1	0	0	0.800000	2	0	68	10	2	67	14	3	0.609330
39	Installed Windows 7 Ultimate fast and easy. Probably the easiest intallation Ive ever installed on a hard drive as it was already pre-configured for Win7. The packaging was excellent. Great price btw...	202	33	4	4.969697	0.035000	0.627000	0.338000	0.920100	0.356944	0.787500	0	0	0.039604	2	False	False	False	False	False	4	0	0	0	1.000000	1	0	19	6	0	18	7	0	0.847419
40	The Canon body cap is included with the purchase of your Canon SLR camera. If you happen to lose the cap, you need a new one. Don't risk damage to your camera body that will end up costing more than a replacement cap. The cap protects the camera's interior from stray dust particles and damage while your lens is off the camera. To save some money, look for anoff brand versionat a local camera shop. There are no special parts to this cap.PROS:Protects the interior of your cameraCONS:Expensive for a piece of plastic (but you gotta have protection.)	556	103	7	4.262136	0.046000	0.810000	0.144000	0.854500	0.114448	0.381494	0	0	0.037770	0	False	False	False	False	False	0	0	0	0	0.500000	0	0	50	0	0	49	0	0	0.279240
41	It is working without any problems.I bought it for my friend. He likes it and using everyday.It is good deal.	109	22	4	3.909091	0.000000	0.656000	0.344000	0.836000	0.700000	0.600000	0	0	0.036697	0	False	False	False	False	False	1	1	0	1	0.500000	0	0	7	2	2	6	2	3	0.380800
42	Sounds great but hard to get used to deep insertion. Don't plan on hearing too much outside noise in fact pretty much cuts out the whole world. Would be good if you did running or activities that other buds would fall out. Cord does get tangled in the pocket but no big deal. If you want to up grade your sound on your Ipod this is the way to go.	349	71	5	3.816901	0.087000	0.765000	0.148000	0.714000	0.193939	0.437879	0	0	0.017192	0	False	False	False	False	False	2	0	0	0	1.000000	0	0	31	4	0	30	5	0	0.602988
43	I find that I hardly use this product. The holder clip broke almost immediately and the unit does seat well with my iSkin attached. When it is connected I wasn't happy with the performance. I hoped to use this when I mountain biked, but it just doesn't perform.	261	50	4	4.140000	0.084000	0.843000	0.072000	-0.140300	0.254167	0.770833	0	0	0.030651	0	False	False	False	False	False	1	0	0	1	1.000000	1	0	18	2	0	17	3	0	0.508743
44	EDIT 3/8/2013: After doing a second factory reset and updating the firmware it has a spanking new OS (4.2.1) that still freaks out the same way. So it definitely has to be a hardware issue. Having not treated the tablet roughly and having had it in a case since I started using it I would have to say that I just got a ticking bomb that just went off.If I rest the tablet flat with my hand in the middle or on a table it seems to freak out. If I hold the bottom left corner (when horizontal) it seems to function fine. However I can't hold it like that for long as it isn't comfortable. Once I release that corner it freaks out. Faulty manufacturing has to be the issue here.Asus customer service (after jumping through a few hoops) can't guarantee the will fix or replace it for free so having to ship this device back to the US via FedEx with damage insurance and then paying shipping cost back seems like a big gamble. The money would likely be better spent buying a new a completely different table from a different manufacture in the near future.That said I cannot recommend this or any other Asus product to anyone, especially if they live outside of the United States. The review is left at two stars because I purchased two of these tablets. One for me and my girlfriend and her's seems to still be working fine for now.-------------------------------EDIT 26/2/2013: For the last week the tablet has suddenly decided to launch applications on it's own or even type random letters while I try to use the keyboard to send a message. It's not limited to any one application as it even tries to re-size videos while I am watching them through MX Player. Even after doing a factory restore with limited application icons on the screen it will now try to change the wallpaper, which is suppose to happen when you tap and hold on the home screen. Seeing as I am not doing this it just seems the tablet has started to freak out.On checking forums it seems many others have had this issue after use and also straight out of the box. My tablet has not been dropped or damaged in anyway recently, so there is no known cause for this other than possible hardware issues. It has had the latest firmware update for a while with no issues until now.Thankfully I have not had the cracking screen issues some warned me about, but with the tablet randomly acting up after just a few months of use I can't safely recommend that anyone buys this or any other Asus tablet until this issue can be resolved. Unfortunately in checking with Amazon's return policy I am only able to get a partial refund, which could be nothing much at all.-------------------------------OLD REVIEW:Everyone loves the new iPad(3), but not everyone can afford one. Also some people want to do more with their tablet than Apple allows. You know stuff like connect external devices, transfer files, watch various video formsts and add more storage space. If you fall under that category then the Asus Transformer TF300 is the tablet for you.The TF300 is the budget version of the Asus Transformer TF700, which is their premium tablet. However the most major changes are that the back is made of a very sturdy and ribbed plastic rather than metal and the screen is not as bright as the TF700. the TF700 is 600nits of brightness and the TF300 is 300nits. What this means is that it's difficult to use the TF300 outdoors or anywhere that there will be lots of glare. The screen resolution is also slightly lower on the TF300 at 1280 by 800, but it's no slouch buy any means. The screen quality is beautiful and you can easily use it indoors at half the brightness. However if you feel a strong need to use your tablet outdoors in the harsh daylight you will need to fork over an additional $100 for the TF700.Performance wise you honestly won't have to go any higher than the balanced setting if you are playing simple games, reading books, playing music and surfing the web. However you can switch over to high performance for graphic intense games and other apps but your battery life will take a hit, depending on what you are doing. There is an energy saving mode, which by default the tablet switches to when the battery power gets to around 15%. This option I wouldn't recommend as it can cause the screen to flicker various brightness levels while watching video. Some people have complained about this tablet flickering and honestly this likely is the reason why. Thankfully you can choose not to use that setting and also set the tablet to not go into that mode when the battery is low. Battery life seems to be more than enough to go two days before charging it with light use. Heavy users that game, watch videos use wfi, bluetooth, GPS and connect external devices (like hard drives) will be able to make it through the daylight hours before they need a charge if it's set to balanced power setting. Having not tested high performance I cannot honestly say how long it would last.Currently this tablet comes with the Android Ice Cream Sandwich OS, but once you connect it to a Wifi network it will automatically download Jelly Bean. Thus far under Jelly Bean there have been far less issues with how the tablet performs and the OS is much more user friendly. That said the default web browser can crash and I've experienced this only once so far, but closing the browser app an resuming takes seconds to solve that issue. However I can't say if this is an OS issue, a buggy app or a bad coding on the specific website. That said you will find that some sites will still default to mobile versions, which has nothing to do with the tablet and all to do with web developers mainly focusing on building pages for the iPad experience and not Android. Hopefully they will correct this in the near future.The build quality seems to be very good. It feels sturdy and the material used don't feel cheap. Even the plastic used on the rear of the device feels well crafted and not cheap. Plus the tablet will likely be put into a case for protection so any complaints abut the use of plastic rather than metal are trivial. Much like the folks who complained abut the iPhone 5 metal casing being easy to scratch. Protect you investment with a case and you will be fine.The 8 megapixel rear camera is a decent performer, but don't expect photos and videos to be as crisp as ones from a DSLR or a point and shoot. It's basically great for quick snapshots in fair lighting conditions. If it gets to dark or of there are very bright and dark areas that you are photographing you will see the camera's limitations. That said if you can get a hold of and HDR app you should see better results. There is no LED flash so low light photography is a no go. With the 10" tablet form factor taking photos will seem a bit clunky. 7" is more ideal, but as always a DSLR camera is what you use when you are more serious about getting that winning shot.With the USB attachment you can connect external drives directly to your tablet and transfer files back and forth without the need of a computer. The Android file directory makes it very easy to copy and paste and create new folders. Also if you are the kind of individual who loves to customize your device then the TF300 with it's vanilla Android OS will fit your requirements like a glove.So if you are in the market for a price friendly 10" tablet that doesn't cut corners to get the price down, and delivers quality performance, then the Asus Transformer TF300 is the most ideal solution for you.	7501	1403	67	4.235210	0.073000	0.822000	0.105000	0.995600	0.158319	0.509304	0	0	0.024663	75	False	False	False	True	False	8	8	2	2	0.500000	6	1	611	27	36	610	39	54	0.422279
45	Pros: features and versatility: CD, HD radio, iPod/iPhone control, Bluetooth phone integration/music playback, Aux port, preamp outputs, 9 band equalizer. All features work as advertised. Great sound.Cons: HD radio reception could be better, USB connection on rear rather on front panel. Configuration can be complicated (but well worth the effort). Unit did not include extraction tools mentioned by instructions.First, thoughts on where to purchase: there is the question whether warranty support is jeopardized if an Alpine product is not purchased from an authorized dealer (Crutchfield will tell you this as well). I ordered mine from LegacyStereo through Amazon and had no problem registering the serial number on Alpine's web site. If the serial number is ground off the frame (as was the case for 1 reviewer), send it back ASAP and go with a different supplier!HD radio is a wonderful thing -- CD quality sound and many more (free) programming choices. Unfortunately, it's downfall the requirement of a strong signal. That said, the OEM HD raidio in my 2013 Subaru can maintain HD reception much better than the Alpine unit. But for the most part, it's OK within an urban area.Bluetooth and microphone work great for making/receiving hands-free phone calls as well as streaming music.IMO, sound quality is slightly better using the USB connection rather than BT. Plus, the USB port will continue to charge a connected device even when the Alpine unit is turned off. Having the USB connection on the rear and snaking the USB cable into the cabin could be a challenge for some installations. A front panel connection would be much cleaner and convenient.The HD148 has an almost overwhelming array of configuration options. Allow several hours of quality time with the manual to master them all. After tweaking the 9 band equalizer, crossover points for front and rear speakers and spatial effects, the sound was vastly better than just using the unit out of the box. I have a first generation Miata with speakers in the doors and headrests and now the HD148 coaxes so much more sound out of the speakers than my old head unit did.The HD148 is definitely worth it, and then some, with just a few minor drawbacks.	2218	379	23	4.722955	0.013000	0.852000	0.134000	0.991000	0.310417	0.467708	1	0	0.037872	16	False	False	False	False	False	3	1	0	0	0.750000	1	0	205	12	2	204	18	3	0.705812
46	I thought this was a great headset until I bought the HBS-730. I still use the 700's every week, have no complaints, but the 730's have better bass and a lady talks to you to tell you what's going on with the headset rather than beeping. I tried two other bluetooth headsets and both were pathetic compared to these. They have the right weight balance, and your ears NEVER get tired.	386	75	4	4.026667	0.110000	0.766000	0.123000	0.252100	-0.012755	0.622959	0	0	0.033679	9	False	False	False	False	False	1	1	0	1	0.500000	0	0	27	2	2	26	2	3	0.063079
47	I bought this tablet when my computer was broken and had to use this and I am impressed with how responsive and great this tablet is. It is good for entertainment and work.	172	33	2	4.181818	0.071000	0.575000	0.354000	0.906200	0.525000	0.687500	0	0	0.017442	0	False	False	False	False	False	3	1	0	0	0.750000	1	1	9	5	2	8	6	3	0.539955
48	I thought I'd try this out, because I was worried about my Kindle Fire screen getting scratched. So I pulled it out of its cardboard envelope. But it's so thin, a film really, it wouldn't protect it from any deep scratch anyways. And from what I've read, the Kindle's screen glass is pretty tough. Then I read the back. It's like a decal you adhere to the screen. And there in the fine print: "Crease marks are permanent and will affect performance. ...You may notice rainbow-like patterns...when viewed at a slanted angle. It should not affect viewing or performance." There's that performance thing again. So if it doesn't protect it from deep scratches, could affect performance, and is pretty much a fancy decal, what's the point? Maybe if you have a kid with really grubby fingers that likes to smear jelly... but nah, it's not for me. I decided against risking sticking this decal on my screen and risking performance...for no real benefit.	946	176	17	4.187500	0.136000	0.746000	0.118000	0.204000	0.052315	0.490278	0	1	0.023256	0	False	False	False	False	False	0	0	1	0	0.500000	2	0	77	0	0	76	0	0	0.221663
49	Used it as a substitute for a failed wireless connection to a remote (in house) laser printer. Set up fairly simple, but do not connect via a power strip or extension. Use a wall outlet. Boxes are fairly large, so placing them in outlets that are not too view able is a good idea. 3 lights (some blinking) can also be objectionable.	335	62	5	4.193548	0.036000	0.898000	0.065000	0.401900	0.135714	0.418452	0	0	0.011940	1	False	False	False	False	False	1	0	0	0	1.000000	1	0	29	2	0	28	3	0	0.647713

	index	true_rating	predicted_rating	error	vader_compound	textblob_polarity	custom_score	word_count	text_preview
0	0	1	5	4	0.845	0.143	0.232	167	What I recieved is not what is pictured here OR ADVERTIZED, but it is a vidio cable, and it does do the job I needed it for. If you were looking for a high quality cable do not buy this product. I recieved a cheap thin flimsy cable that is not gold plated or anything else they claim here. Bait and Switch? or a mistake in the order !!!!! . I do not know. I will attempt to contact the sender and find out. I contacted the seller and they told me to contact amazon.com so i did, and amazon.com was good enough to give me a $4.00 refund; which is good, because all this cable is woth is $2 to $3. I only needed it to run my wife's kareokee to the TV. If it was for any thing else I would have sent it back for a full refund. I'm surprised that this item is still listed as a high grade cable.
1	45112	5	1	4	0.843	0.118	0.777	66	After reading the numerous positive reviews took the plunge and ordered this for my new Vizio 60" LED to wall mount. Wall bracket very well made and simple to mount! Very sturdy. One reviewer gave one star because he said vertical brackets were to short, he apparently didn't read they include 2 extensions so it'll fit.Great product getting ready to order another!
2	44391	5	1	4	0.882	0.234	0.614	126	Love the clicking, love the mechanical feel. I grew up on the initial IBM PC's in the 1980's and this is like going back in time. Just feels right. Yes, it is more money but what do you expect for a quality product. Not sure what the guy that rated one-star was concerned about on the warranty. There is a Money-back guarantee for 30 days after the purchase - product needs to be in its original condition and packaging, but there is also a limited one year warranty as described on the daskeyboard website under guarantee heading. The warranty period is based upon the shipping date of the purchased product to the buyer so you should be safe if there is a problem.
3	44425	1	5	4	0.942	0.180	0.519	160	This was a little gift from a friend who doesn't know a micro-card from a microscope. It was very nice of him, and a seemingly great idea. The holder does hold all of my 9 micro cards plus the adapter, BUT as with any micro card, when you get them out, they're so small that you can still drop them. The claim that you'll never lose them again is silly. PLUS there's no way to label the cards for content. The only "good" thing is that this doesn't take up more room than a credit card. Now you can lose all of your micro cards at one time.Every mico card that I have purchased (SanDisk) come with an adapter. There is a case that holds 16 micro-cards in their adapters, and is much more protective and better than this. It may be bigger, but at least I haven't misplaced it yet.
4	44430	5	1	4	-0.681	0.050	-0.544	32	Like it, but ended up using the leather one I got at the same time. No issues with either case as far as magnets causing a problem with hardware or installed apps.
5	44676	5	1	4	-0.790	-0.001	-0.587	75	I have never had to deal with a dead laptop battery because of this sucker! Granted, it is about the same size as my laptop, it puts out enough power to double my already 8 hours of battery on my Acer netbook.It is solid as a rock, doesn't feel cheap at all since it is made of metal and after 6 months of use it still puts out and holds a steady charge.
6	44759	5	1	4	-0.643	0.187	-0.055	173	OK, I admit it... I forgot what the wood finish on my desk looked like as it had not seen daylight in months! Despite having a good sized desk at home, I still manage to put more junk on it than I should. So when I got this keyboard, I was at first scared that I would lose it under papers. It is so small! I have kinda thick hands, so I was worried that it would be painful to use this keyboard. NOPE! I am amazed how natural this keyboard feels. The buttons give good resistance when punched. And did I mention it was small? I went from a ergonomic keyboard to the wired Apple aluminum keyboard when they first came out. That cleared up a lot of room. And this has given even more room (to place more unneeded junk)! Seriously, my mouse pad is bigger than the keyboard. You'll be amazed at the simple beauty of it. I'm very happy with it and think you will be too!
7	44784	5	1	4	0.954	0.553	0.659	113	I am writing this review in response to one of the 1-star reviews that claims this product does not work with the iPhone 2.1 or 2.2 software updates. I purchased the product for my brother as a Christmas gift, and was a little worried that it would not work. My brother and I have been playing around with the battery all day today on our v2.2 iPhone 3Gs... and I can verify that it works PERFECTLY!We have not had any problems working with the battery, and it's nice that you can charge your iPhone even with the battery connected. Great product with a great price from Amazon!
8	44797	1	5	4	0.872	0.064	0.008	250	What a drag. After having my faithful Palm IIIe for seven or eight years, I finally dropped it and it died. I decided to replace it with the low-end Zire 21, no longer available through Amazon, but I saw it at Office Depot for sixty bucks. I checked out the reviews here AFTER I bought my Palm, and was disconcerted by the large number of negative ratings, but I thought I'd see how it worked for me.At first, I was perfectly happy with it -- I mainly use my Palm for alarms and phone numbers, and the Zire 21 handles those things well. Yeah, the screen is small without a backlight, but I could read it just fine.Unfortunately, even though I've had it just over a month, twice now I've turned it on and it's completely reset itself -- it's lost every bit of data I put on it and acts as if it's a brand-new Palm Pilot. I need to do a HotSync to restore my data, and then reset the time and date. So far this has only happened to me when I'm close to my computer, but if it ever happens when I'm on the road and need the data, I'll be very upset.I'm going to see if it's not too late to return this thing and get something better. Too bad that Palm has not lived up to it's earlier promise.
9	44845	1	5	4	0.876	0.170	0.350	220	I love android, I love the fact that I am able to customize my tablet, and do a whole lot of other cool features. The size was perfect for one hand. But too bad, my first N7, came with a dent on the side of the screen, which is hardly notice unless pinned to the light. Luckily I purchased from GameStop and was able to go in and exchange it for a second N7. It was great until last night, where I had left it to charge so I can bring it to school the next morning to work on a project to realize that it did not charge at all. So I came home and try to troubleshoot the problem but it looks like the port was defected and started to malfunction after seven days. So I went back to GameStop and got me a 3rd exchange, and when I got home I googled all the defects that is being encountered wit the N7 and found lots of problems, so I didn't want to take a gamble with the third one and exchanged it for my money back.It was too bad, I really had a great time with the tablet...well at least I'll be ready to buy one once all defects are taken care of.
10	44934	1	5	4	0.814	0.192	0.632	72	Wow. This adapter was awesome for a bit. Until I left it plugged in over night. It still works, but it got so hot that it melted over night. The plastic is all warped and the enclosure of the drive is also melted. If you buy this, you can only use it for a few hours at a time. Do not leave it running for days at a time like I did.
11	44946	5	1	4	-0.650	0.103	0.167	370	The Good1. A no nonsense product...2. No funny software gimmicks....just plug and play3. Rigid protective casing with rubberised corner4. Stays well gripped where ever it is kept.5. Read and write data fast...no need to wait.6. No fans in the product so it is very silent,7. Drive warms up slightly on using but it is OK8. all my laptops detect this drive fast and wihout any glitches.9. really fast shipping, recieved it the next day.The BAD1. The disk rotates always when it is connected to computer...so plugging it always to a computer may be a bad idea.....connect it only when you need it for your drive's long life.2. The Rubber material at both ends are soft but they attracts lot of dust ..and within two days the product look like it is used for 1year.3. Big hassle to carry a power adapter and a USB cable along with this product...so bag is a must for those who wants to carry it around.4. product looks like a solid block (not a "beauty") ...( again i need only a storage space )TipsDrive comes FAT32 formatted... lot of Geeks are worried about the format and formatting to NTFS .....I decided to keep it as FAT32 because1. NTFS is windows specific ..will not work with Mac ...but FAT32 will ...2. Any time you can format your drive to NTFS from FAT32 if you want3. But you cannot go back to FAT32 from NTFS4. files in FAT32 needs more space than NTFS...(i still dont know how is that) ...but i have 1TB of storage space now so am not worried about that ..but may formatt it to NTFS when my drive have only 20% of space left ....5. Some say movies more than 4GB file size cannot be saved in FAT32 formats as 4GB is the max size....but this dont affect me because i am not a movie fan.6. So if you are using this drive not for movies ...FAT32 is the way..no need to format.WishThis product would be "excellent" if there was a carrying case which can fit the Drive, the charger and cables all together
12	96020	5	1	4	-0.913	0.135	-0.138	86	When I realized my AVR wouldn't pass digital audio to a second zone, I was pretty bummed. Then I found this device! I was able to pass my HDMI digital signal through this device to the AVR while also sending the audio to analog inputs on the AVR for the second zone. No problem with any delay or sync problems. No problem with audio quality. I can now push my music library from my HTPC into all the zones and I couldn't be happier.
13	45111	1	5	4	0.717	0.039	-0.169	202	I bought this drive before any user reviews had been posted. I wish I had waited because, as all the other reviewers have said, this drive is just no good. I have a Seagate 200GB external drive and never had any problems with it. However, I needed an additional drive to save dvd files, just the type of file this drive does not like. For the first time in my ownership of Windows computers, I started to get a yellow triangle warning icon. In the event log, I too had a myriad of 50 and 51 write failures. The Seagate tech support technician told me the drive was defective and needed to be returned. Seagate only replaces defective drives with factory refurbished ones, not with new ones. Thank to Amazon, I was able to choose between getting a replacement (with a brand new drive) or a refund. After seeing that I was not alone in my horrible experience with this drive, I chose the refund and, after I receive my credit, will buy another 200GB drive. These seems to work just fine. If you need a lot of space, you are better off buying two Seagate 200s rather than the 400GB drive.
14	45131	5	1	4	-0.922	0.068	0.237	195	I owe the maker of this screen protector a huge apology for my former review. Initially, I was very frustrated with the screen protector because I couldn't get a sticker off after installation on my Nano. I lived with that opaque sticker blocking part of my iPod Nano's screen for quite a while. Eventually, I decided that having no screen protection was preferable to having an obstructed view of the LCD. Imagine my chagrin when I realized the offending sticker so stubbornly affixed....was attached to what was intended to be a temporary top layer (A screen protector for the screen protector? Something to facilitate bubble-free installation?), which peeled off to reveal the ACTUAL iPod Nano 6th Gen screen protector in all its unblemished glory.If you're looking for an inexpensive and effective screen protector, look no further. Just PLEASE spare yourself future shame by being aware that the screen protector itself is sandwiched between two layers that BOTH require removal - one before installation in order to expose the surface that is applied to your Nano's screen, and the other to be taken off after the protector has been installed.
15	44272	1	5	4	0.723	0.099	0.087	58	I have a relatively short run (50'), but I live 46 miles away from the transmitters. I was getting about 50% on the TV signal meter for the poor stations, and in the upper 80's for the good ones. Add this device and got about 5 %less across the board. I had high hopes. Oh well; returned.
16	45152	1	5	4	0.885	0.107	0.637	102	I had this router for about six months. The first few months, it functioned ok, but within the last couple of months, the router would no longer connect after about 24 hours, requiring a reboot in order to connect to the internet. And even then, the speed was atrocious. Images on pages would not even load, java and flash based games would connect and then freeze.I have since replaced it with aCisco-Linksys E2000 Advanced Wireless-N Routerand, within the 24 hours since then, my internet browsing has dramatically increased.Definitely recommend staying away from Belkin and I now love Cisco.
17	45158	1	5	4	0.822	0.022	-0.249	57	Yes it's cheap. That doesn't mean it's worth buying. I ordered two of these. One seems to work fine. The other only had one working earpiece. I had to send it back for a replacement (pain in the but), and I have to hope the replacement doesn't fail too. DAVE says "be forewarned."
18	45269	1	5	4	0.700	0.293	0.619	56	Not only does it not charge 100% ever, it also can't really charge anything cuz of the first reason. Bout the only thing this device is good for is the light which is 1 led, so any flashlight with 1+ led's will blow that away too.Don't buy, there are better out there.
19	45279	1	5	4	0.654	0.128	0.355	98	Easy to set up for my Panasonic Viera 32" set that didn't come with WiFi. So far it has worked without problems.Update: Instead of 5 stars, I have to change my rating to 1 star.After using the Belkin wireless link for a couple months it failed completely. I should have listened to the other Amazon reviewers about its poor reliability - so don't buy it if you are expecting something great for low cost. I have since replaced it with a refurbished Netgear WN3000RP Universal WiFi Range Extender which costs more but has better reviews.
20	95989	1	5	4	0.926	0.331	0.694	92	I guess i did not do enough research on this product because i have one with the old firmware which out of the box, does not work, you need to do a file modification on a ini file to get their own updater to work properly. Amazing how things like this get by QA, which seem's more like none was done...See:[...]And note on newegg here[...]then have fun even trying to register on their forums since they ask you a question i certainly don't know the answer to.
21	45375	1	5	4	0.901	0.274	0.306	122	This SSD generally gets good reviews, but there is a definite subset of users who experienced BSOD's on an irregular basis. Very difficult to diagnose. I only feel comfortable attributing the problem to this SSD because I replaced it with an Intel SSD (X25-M SSDSA2MH120G2K5)and have not had a single problem.I updated the firmware. I looked at every possible driver combination. Replacing this SSD with an Intel was a last-ditch effort to salvage a custom-built Intel i7/ASUS running WIN 7 Pro x64.My suggestion: make sure you investigate whether this SSD has caused problems for your computer, chipset or motherboard. The advantages of speed and a good price fade quickly when you get increasing problems.
22	45386	1	5	4	0.910	0.103	0.560	749	I purchased this receiver hoping I could integrate a variety of digital and analog sources into my cluster of receivers and amps that drive speakers in several rooms and outdoors. The TX-NR509 seemed to offer everything I needed: multiple HDMI inputs and a full complement of analog inputs, subwoofer pre-out, preamp out for surround rear speakers, the highly touted "zone2" output, network capability, and the added bonus capability of using an Android smartphone for remote control and streaming.I spent the better part of two days connecting all the cables, running the setup procedures, and becoming familiar with the receiver's capabilities and limitations. The TX-NR509 does have very sophisticated capability to optimize its output for your listening room. It even comes with a calibrated microphone that you use to support a series of tests that the "audissey" system runs to check your room. After completing the setup procedure I checked out audio and video performance with HDMI input from my satellite receiver--it performed beautifully. Then I connected the ethernet cable; the TX-NR509 immediately connected to my home network, recognized all the compatible devices (basically PCs with Windows Media Player). Using the Onkyo remote I was able to browse for media files and play them on the TX-NR509 just as easily as from any PC keyboard. Next I installed the Onkyo application for my Android phone. It installed in seconds and connected to the TX-NR509 immediately. A moment later I was streaming music to the receiver, controlling volume, and many other functions right from my DroidX. I turned on my outdoor speakers (powered by a secondary amplifier fed by the Onkyo back speaker pre-out), and walked outdoors with my Droid--it continued to function as a remote 50' away from the TX-NR509--a very nice capability for patio listening.As I continued to explore the behavior of my system I began to see the limitations of the TX-NR509. I will list them in order as I discovered them.1) the actual number of independent sources the receiver can handle is far smaller than the number of inputs terminals on the back of the device. There are four basic selector positions for source inputs, "BD/DVD", "VCR/DVR", "CBL/SAT", and "GAME". Each of these has an HDMI jack and a set of analog jacks. If you use both the HDMI input and the analog input for the same selector position, the analog input will be ignored. There are also four digital audio input jacks that similarly compete for selector positions. Bottom line, there are 12 possible audio inputs, but only four of them can be used.2) the TX-NR509 has no up-convert or down-convert capability for audio (digital audio cannot be converted to analog and vice versa). As a consequence, if your source is digital (say HDMI from satellite or internet radio) there is no analog audio signal, so the much ballyhooed "Zone2" output is mute and useless. How could Onkyo overlook such a vital capability in today's digital world? I thought I could overcome this limitation by using the analog audio output of my HDTV (which does have down-convert capability). This trick seemed to work as a means to feed audio from digital sources to my remote speaker systems until I discovered an even more critical flaw in the receiver's design.3) The last straw, the show-stopper for me was my discovery that enabling audio feed over HDMI cable from the TX-NR509 to the TV (an option in the setup menu) automatically disables the pre-amp outputs to the subwoofer and surround rear speakers. Together the latter two limitations prevent me from operating all speaker systems simultaneously at full capability, even though I have plenty of power to do so.Finally, while mulling over my options but still enjoying the wireless streaming capability from my Android to the Onkyo receiver the network card in the receiver apparantly failed completely. All attempts to connect to the network result in a stalled system with the message "initializing" ad nauseum.Tomorrow this device goes back in the box and back to Amazon. I will look for something better.The silver lining in this dark cloud for me it that I discovered that Onkyo's free app for the Android works very well as a streaming device and remote control for Windows Media Player on any networked PC. This opens up some interesting possiblilities.
23	45402	1	5	4	0.725	0.509	0.519	27	Right out of the box it had a static sound over everything! I could not take it! Instead I bought the Altec Lansing iMT228 and love it!
24	45432	5	1	4	-0.894	-0.218	-0.662	41	I ordered the wrong size for a new lenses I had bought off eBay, I use these to protect the lens from stractches, but I ordered the wrong size this works as expected on other lenses I have, no problems here.
25	45437	5	1	4	0.968	0.078	0.806	321	I'm really surprised at the number of 1-star reviews on here and the complaining about the lack of flash. Let's face it -- flash is useful for some things, but as the dominant media plugin it is on its way out. I've found very little that actually needs flash and the sites that do use it, there is mostly an app that can accomplish the same thing but way better.Why not review it for what it is rather than something that you think it is not. Obviously many of these people do not own one. The common thread seems to be "I used a friends".I bought this about a week after it came out and I have used it every day since. I have hundreds of free and paid apps both for fun and work. I recently took it on vacation with my bluetooth keyboard and was able to leave all sorts of things behind. Watching movies, reading books, playing games on the flight -- perfect. After taking pictures on my iPhone I was able to wirelessly send them to my iPad where I could edit and enhance them with the dozens of image apps out there.Every day something new comes out that enhances the experience as developers have jumped on board in a frenzy. Gaming is a totally different experience as the size makes a big difference. Going back to gaming on a phone is just not possible.Battery life is amazingly long. Even with alot of use every day, I can get 2-4 days of use with no charging. It's well put together and slick -- hard to believe it is the first one.And yes, everyone will jump on the bandwagon and try to mimic it -- and if Flash is your one and only destiny that you can't live without, then by all means wait. Otherwise this one is the one.
26	45443	5	1	4	-0.718	-0.033	-0.222	18	This is my third for my family.SOOOOOOOOO much beter than ipod.Only negative is lack of acessories.
27	45533	1	5	4	-0.247	0.127	0.051	198	I bought this for <$6 so wasn't expecting much. I wanted a tripod to do some table top stop motion (just goofing around with the kids). This item performed below my very low expectations.Cons:1) The adhesive holding the plastic to the aluminum simply failed, item fell apart in my hands (and no I wasn't cranking on it)2) Soft plastic easily stripped such that joints no longer tighten making the unit completely useless3) Lightweight, cheap construction4) Bolt holding mini tripod in stripped out, no longer stays inPros:Provides an excellent object lesson on the difference between something "inexpensive" (i.e. a low priced item that functions) and "cheap" (i.e. a poorly manufactured piece of junk). This is it's only redeeming quality.It would cost more to ship back so I just burned $6 on an experiment. Save yourself time and trouble, this item is of zero value to anyone. What I cannot fathom are the 5 star reviews for this product. These represent either a different manufacturing process in the past (possible) or the company itself leaving favorable reviews. The fact that this item is on "clearance" pricing suggests the latter.
28	44364	5	1	4	-0.700	0.101	-0.540	48	I needed a mouse that keeps my wrist from getting sore and to prevent some other problems with my hand. It works much more smoothly than a previous trackball I have used. Some transition was necessary because I had not used a trackball in several years. No complaints.
29	44263	1	5	4	0.982	-0.035	-0.012	180	Wanting to watch Netflix on my TV, I gave this WD Media Player a try, and it was a major disappointment - I'm returning mine. Here's sample of its shortcomings.1) Despite its advertising, it does NOT show in 1080p. The same HD documentary from Netflix that is sharp and clear on my TV via Playstation3/Netflix is blurry and muddy via WD Media Player (which also uses HDMI).2) The WD Media Player advertising video explicitly states it can support wireless via USB-key style adapter. It turns out to be false advertising. WD Media Player does NOT support wireless - you must have wired connection.3) Its User Interface is very slow and often hangs - it seems to have a lot of bugs remaining to be fixed.4) Its remote control is a marvel of user-UNfriendly engineering, with tiny buttons that actually hurt your fingers.Using this device is an exercise in frustration. You'd be better off paying more for a PS3 for a full-featured, well-engineered media console with wireless and blu-ray support.
30	43217	5	1	4	-0.696	-0.138	-0.586	45	I got this as a back up for my Canon G 15. I've not pushed the camera to its limits, but this should serve its purpose. Price was right. If it goes bad, I have to remind myself that is is "aftermarket."
31	43735	5	1	4	-0.809	0.019	-0.739	111	After reading the reviews and comparing the price of different cases, I decided to purchase this cover. I've used it for three days now and I have no issues with it. I was a bit concerned since most of the reviews were before the iPad Mini with retina display model came out, but have no fear, it fits the new mini model too. The only thing I don't like about the case so far is the flimsy stylus holder. I have a stylus, but I don't use it often so I don't carry it with me all the time. For the price you can't go wrong!
32	96151	1	5	4	0.847	-0.010	0.102	79	OK I received this in 2.5 weeks from Hong Kong when i received it it was a very small package, so i figured it was too small for my laptop i had to move it so it could cool the RAM area at least, and it is too slim and thin seems very fragile and flexible. So therefore spend a little bit more money on buying something that ACTUALLY works For Bigger Laptops Like 14" 15" 17" ones.
33	43387	5	1	4	-0.727	0.104	0.153	104	Very nice portable hard drive for the price. It is powered by the USB connection, so no need for power supply or more cables.I use it to store all my pictures and music collection on so I can access them easily by hooking this up to any of the 3 computers in the house. As far as transferring files, just drag and drop. Things don't get any easier than that. I am puzzled that others have complained that no directions come with this drive. Just plug it in to the USB port and treat it like another hard drive on your computer.
34	43451	1	5	4	0.816	0.236	0.431	133	Whoever says this thing is great has got to have no sense of audio quality. FM can sound pretty good if you're near a radio station's tower; I can be standing right next to this transmitter and still get scratchy, static-y audio when I know the audio source is battery-powered, strong, and clean from an mp3 player. Using the antenna extension didn't really help either.For $100+, I expected a higher quality of construction, but it is super-light cheap-feeling plastic for the whole thing. While this may be a plus for portability, even the mini-jack seemed like it had cheap construction (nickel-plating instead of gold-plated). Really disappointed with this, just wish I had the time to return it within the 30-day window.
35	96144	1	5	4	0.991	0.137	0.208	577	I'm sure this is a fine router once it's working but I guess my annoyance threshold is too low. I'm not an expert but not a complete novice either...I've set up and configured more wireless routers and other tech toys for myself and friends and relatives than I can remember. The installation experience with this item was a chore that looked like it could become a hobby and that's not what I was looking for.I opted to use the browser-based wizard setup. There's a little pamphlet included with the router that steps you through using it. Problem is that the instructions and screen shots and progression through the wizard in the pamphlet is different from what you experience in the browser (in my case, Chrome on a Win7 Home laptop from HP - nothing fancy). So when the browser's screen differs from what's in the pamphlet, you kind of have to guess. None of the guesses seemed difficult, but the wizard's responses were confusing. Here are a few examples (by no means all examples):The pamphlet says when you browse to the router's config page, you get a welcome page. You don't. You get prompted to enter the admin password. That's OK, I kept reading and found out there's no password by default.The welcome page appeared a couple of times as I went through the wizard (I went through the wizard a lot of times), but just as often didn't, and I don't recall it ever having exactly the options shown in the pamphlet.The pamphlet shows a page image where you enter the name and password for each of the 2.4Ghz and 5Ghz bands. I never saw that, not once.At the completion of the wizard I did get an encouraging prompt: "Click the save button to save your settings and reboot the router." Unfortunately, the only buttons on the page are Next, Back, Cancel and Connect. No Save button. So I clicked Connect as the next best thing, or so it seemed to me. Bang, back to the login page where I was prompted for the admin password again and the wizard started over.There were other issues, more than I want to take time to describe in detail, but briefly:* seemingly random jumps from anywhere in the wizard back to the login page, way too quickly after the last keystroke to be a timeout IMHO* some parameters are saved during a pass through the wizard pages; others are not (as I could see during my next pass through the wizard); others may or may not be - this is particularly interesting when your new admin password is saved during one pass and a different one (I was thinking "maybe it doesn't like special characters?") is not during a subsequent pass, so your most recent password is invalid and your previous one works* a given wizard page may or may not appear during a pass through the wizardThis is a Vine review so obviously I'm not going to return it but if it were something I'd paid for, it would be on the way back to the good people at Amazon in the morning. As I said at the outset, this is probably a fine piece of work but for me, it's not worth the effort vs what I have in place now.
36	43528	1	5	4	0.701	0.314	0.623	48	Bought this for the 4x 2.5" storage capacity. the bottom 2 slide in slots fit fine, but the top 2 snap in ones do not fit most 2.5" drives. Fits 3.5"s fine but not as nice as other options out there.
37	43537	1	5	4	0.910	-0.003	0.422	212	I was a long time user of the wireless natural keyboard and it broke and I was looking for a replacement. I purchased this keyboard thinking it was going to be a good keyboard and boy was I wrong.First of all if you are familiar with the natural keyboards they are the ones that have the split keyboard and the keys in the center are at least an inch away from eachother in the center. the comfort keyboard is not split in the center its just curved and is supposed to feel the same but it doesnt. I typed about two emails and my hands were already hurting.The mouse is also bad. The buttons are very hard or stiff. I was having to press very hard to get the buttons to work. The scroll button was very hard as well. I had the scroll button programed to use as a back button and it hardly ever registered the function.I had to return this keyboard and instead purchased an upgraded version of my original keyboard which is now the microsoft wireless natural multimedia keyboard with matching wireless intellimouse explorer 2.0. I am so happy with this keyboard and my hands are fine now even after typing this long review.
38	43630	1	5	4	0.762	0.298	0.288	64	Started out skipping, then played through the first movie fine. After that it kept on skipping and wouldn't play the movies properly. I hooked up a dvd player to make sure it wasn't the tv or the cable. Sure enough, everything worked fine on the dvd player. This 3d bluray player must have been discontinued for a reason. It doesn't work.
39	43653	5	1	4	-0.806	-0.833	-0.609	73	my mom drops everything she's ever owned (maybe even me? it would explain a lot). this thing is about all i can see that might give her ipad a chance of living beyond 3 months.it's really ugly and stupid looking, but it works fine.i also took the screen protector off, cause those things are terrible and you can't hit the edges of the screen with that thing on.
40	96131	5	1	4	-0.700	-0.100	-0.580	25	Simple, compact and affordable Windows keyboard, no complaints. I like that I can use commonly used keyboard short cuts to raise or lower the volume
41	43686	5	1	4	-0.942	0.139	-0.555	155	I've been using this for about 1.5 years now, and I like it. The battery has yet to die and I've been able to intuit the instructions easily without needing to consult the manual constantly.Actually, I went to turn it on while I was typing up this review and it's dead. So there you go. About 1.5 years from 2 AAA batteries.It has a port for microphones, a port for earphones, and a usb port for downloading the audio files.I've been using it to record myself instructing new hires with a throat mic and it has worked quite well.My only complaints are:The USB port is not covered and gets dirt and lint in it quite easily.The "on" switch is a toggle, and it can get activated by accident if you aren't careful. I haven't had too much problem with this though.
42	43691	1	5	4	0.966	0.497	0.889	132	Hi, this will be brief.I own a 7D and bought one of these LCD protectors and it shattered after about 1 month. And I take care to protect my camera quite well.I replaced with a Giottos SP8302L Aegis Multi-Coted Screen Protector (fits the 7D perfectly) and what a huge improvement. The Giottos has a twelve (12) layer multi-coating which really improves contrast and glare and best of all it is manufactured from SCHOTT optical grade glass. The best there is. It has a transmission factor of 98%. You just can't beat that. It is literally invisible. The Giottos has been on my 7D and also my 5D Mark II for over a year and a half and it still looks brand new. It is worth every penny. [...]
43	96126	5	1	4	-0.859	0.134	-0.068	223	I haven't gotten much use out of it other than some fiddling around. The crank is really easy to use and the device fits well in my hand (I have small hands so was concerned about it being too large to crank.) Its very light and has a rubberized coating on the outside, making it much nicer to hold than the typical hard plastic. The weather presets worked great and it has good reception. But I expected this as I live in a major city. I was shocked at how nice the speaker is in it! I expected something tinny and sub-par (as its a tiny unit) but it sounds great!My concern lies at the power switch for the lantern function. I have to mash rather [...] the button to turn it on and there's no click to tell me when its on. I'm worried about breaking the switch. I think this is because if the button is depressed somewhat, the light will turn on but not stay on. However, I still feel there should be a click of some sort to avoid the button being pressed too hard.I haven't gotten the chance to test the cell phone charging unit as I don't have the cable. It has to be ordered from the company/another company.
44	96122	1	5	4	0.856	-0.046	0.248	77	Item arrived today... The very first time I tried to use it, to my surprise and amazement, the hot shoe connector section literally fell apart when I gently and carefully slid the locking lever into place!!! This exposed the internal wiring. The quality of materials and workmanship on this item is poor. It gets returned tomorrow. I will be highly unlikely to buy this brand again - I wouldn't trust it not to damage my valued cameras.
45	44245	5	1	4	-0.936	0.083	-0.314	168	This hardshell case covered in nylon does the job of protecting a bluetooth headset, compacting into a container small enough to easily fit a shirt or pants pocket. One would be hard pressed to fit a recharging cord in this case along with the headset, but for thr latter alone, it is more than sufficient. The protection it provides alone more than justifies the purchase, keeping the contents safe from both damage and disfigurement, thus preventing a repeat purchase of another headset due to loss or breakage. Aside from headsets, this case will protect anything small enouth to fit and of value ( gold chains, rings, ) from loss and damage while in a pocket or any other tight enclosure that might contain potentially marring and abrasive ma-terial. Due to this case alone, the jawbone earpeice contained therein will be the last headset purchased due to loss or damage while in a coat, shirt, or pants-pocket, and for this reason alone earns a ' 10 ', or 5 full stars.
46	43782	5	1	4	-0.818	-0.071	-0.259	95	OK here's the thing----this isn't just way cool to have but a necessity! I can attest to that, especially after my computer got infected with a nasty and I had to have it basically undergo surgery. Had it not been for the fact I had everything on this hard drive.....well let's just say BECAUSE I had everything on it I still have my sanity! (and my computer) I swear by this to the point I'm considering buying an extra; just to store my pictures on for truly 'safe-keeping'.
47	43795	5	1	4	-0.916	0.105	-0.604	98	Just received these and put them in my new Mac Mini. I have the newest Mac Mini with Lion pre-installed and no optical drive. Installation took less than a minute and they work just fine. System Profiler shows the full amount (8GB). Sure it's been just 1 day but they are working as they're supposed to. If they fail or have problems I'll come back here to let everyone know. Just can't beat the $53.00 price tag with $10.00 rebate.UPDATE: 4/10/12 - Memory still working well! No problems whatsoever.
48	43805	5	1	4	-0.763	0.110	-0.343	60	I am using this (3 weeks) with a USB3 port and a Linksys ac modem and I am getting fantastic results. No more problems with Netflex. It`s worth the investment.Initially I used this adapter with a Linksys N dual band router and it also fixed the problems with netflex. I just want to avoid problems in the future.
49	43880	1	5	4	0.957	0.067	0.336	374	Not a bad looking display, and it was an easy job to set up. All worked fine using it with my XBox until I tried Netflix. I'd get black screen dropouts, the Netflix app would suddenly close, or claim I was signed out of XBL. Or the TV would claim "No signal!" or "Unsupported". Looking online, it seems like this happens to lots of folks. I also had the same intermittent problems watching TV via Windows Media Center on the XBox.I ended up returning this TV because of these constant HDCP hiccups. Sometimes it would recover by itself in a few seconds. Sometimes selecting another input source and going back fixed it. Sometimes I needed to power cycle the TV, and sometimes I needed to power cycle both the TV and XBox.The best part of this whole experience is that because I spent time researching the problem and trying many different solutions (new cables, software/firmware updates, even swapping out the XBox for a different hardware rev) I went over Amazon's 30-day return window by less than 24 hours. Now they refuse to take it back and give me a refund. Be careful! If you start to see problems, send it back ASAP! Amazon is pretty ruthless with their return window. This is the last time I'll try fixing a purchase instead of sending it back immediately.This TV is a great deal for a 26", ONLY if you don't require HDCP. For any HDMI source (or DVI requiring HDCP) you should definitely find a different display.I bought this one to replace a Samsung that had a less annoying HDCP bug, so probably avoid Samsung as well as Sharp. The replacement for this Sharp is a Panasonic Viera TC-L32U3 32-Inch 1080p LCD (ASIN: B004M8SC92). It has an IPS lcd panel which looks a million times better, is bigger, and was only about $50 more. No HDCP problems with it so far at all. Unfortunately, it looks like Amazon stopped selling this model. The other 32" Panasonic available is only 720p, so be careful!UPDATE: 2 months on the Panasonic, and not a single problem. And it's a gorgeous looking picture! Much MUCH happier.

	Confidence Level	Num Reviews	Accuracy	Close Acc (+/-1S)	MAE
0	V.Low (0-.25)	14236	0.237	0.654	1.257
1	Low (.25-.5)	17800	0.417	0.772	0.943
2	Med (.5-.75)	31608	0.558	0.903	0.584
3	High (.75-.9)	20854	0.732	0.946	0.347
4	V.High (.9-1)	1051	0.883	0.987	0.135

Multi-Model Sentiment Analysis Pipeline

David Kwan's HKU MSc Capstone

(2023, Enhanced in 2025, still under improvement)

A. Core Problem & Business Impact

B. Project Impact Summary

C. Results Summary (Nearly 100,000 Reviews Analysis)

D. Technical Architecture, Implementation, & Deployment

E. Enhanced Evaluation Framework

F. Production Features & Scale Validation

G. Business Applications & Strategic Insights

H. Future Development & Enhancement Roadmap

PROJECT OUTPUT

STEP 1: Data Loading & Initial Cleaning

STEP 2: Text Preprocessing

STEP 3A: Feature Extraction (Rule-Based Models)

STEP 3B: Aspect-Based Sentiment Analysis (ABSA)

STEP 4: Running Sentiment Models

STEP 5: Model Evaluation & Analysis

STEP 6: Generating Advanced Visualizations

STEP 7: Cross-Validation Analysis

STEP 8: Clustering Analysis (on Rule-Based Features)

STEP 9: Generating Analysis Report

Sentiment Analysis Report (Transformer & ABSA Enhanced)¶

Overview of Pipeline Capabilities¶

COMPLETE PYTHON CODE

	Metric	Value
0	Accuracy (Exact)	0.4815
1	Accuracy (+/-1 Star)	0.8025
2	MAE	0.8260
3	RMSE	1.3077
4	F1 Score (Macro)	0.3053
5	F1 Score (Weighted)	0.4882
6	F1 Score (+/-1 Star)	0.8904
7	Precision (Macro)	0.3069
8	Recall (Macro)	0.3060
9	Cohen's Kappa	0.1343
10	Sample Size	99,934

	Metric	Value
0	Accuracy (Exact)	0.1785
1	Accuracy (+/-1 Star)	0.5804
2	MAE	1.2666
3	RMSE	1.4878
4	F1 Score (Macro)	0.1566
5	F1 Score (Weighted)	0.1687
6	F1 Score (+/-1 Star)	0.7345
7	Precision (Macro)	0.2925
8	Recall (Macro)	0.2547
9	Cohen's Kappa	0.0281
10	Sample Size	99,934

	Confidence Level	Num Reviews	Accuracy	Close Acc (+/-1S)	MAE
0	V.Low (0-.25)	14236	0.157	0.481	1.401
1	Low (.25-.5)	17800	0.180	0.555	1.290
2	Med (.5-.75)	31608	0.169	0.601	1.244
3	High (.75-.9)	20854	0.183	0.672	1.151
4	V.High (.9-1)	1051	0.762	0.988	0.255

	Metric	Value
0	Accuracy (Exact)	0.5327
1	Accuracy (+/-1 Star)	0.8107
2	MAE	0.7590
3	RMSE	1.2622
4	F1 Score (Macro)	0.3204
5	F1 Score (Weighted)	0.5183
6	F1 Score (+/-1 Star)	0.8954
7	Precision (Macro)	0.3298
8	Recall (Macro)	0.3188
9	Cohen's Kappa	0.1696
10	Sample Size	99,934

	Confidence Level	Num Reviews	Accuracy	Close Acc (+/-1S)	MAE
0	V.Low (0-.25)	14236	0.233	0.695	1.171
1	Low (.25-.5)	17800	0.483	0.793	0.846
2	Med (.5-.75)	31608	0.664	0.903	0.488
3	High (.75-.9)	20854	0.767	0.944	0.316
4	V.High (.9-1)	1051	0.885	0.987	0.133

	Metric	Value
0	Accuracy (Exact)	0.4462
1	Accuracy (+/-1 Star)	0.8261
2	MAE	0.7795
3	RMSE	1.1616
4	F1 Score (Macro)	0.2539
5	F1 Score (Weighted)	0.4620
6	F1 Score (+/-1 Star)	0.9048
7	Precision (Macro)	0.3710
8	Recall (Macro)	0.2669
9	Cohen's Kappa	0.1314
10	Sample Size	99,934

	Confidence Level	Num Reviews	Accuracy	Close Acc (+/-1S)	MAE
0	V.Low (0-.25)	14236	0.188	0.728	1.159
1	Low (.25-.5)	17800	0.262	0.870	0.943
2	Med (.5-.75)	31608	0.572	0.923	0.549
3	High (.75-.9)	20854	0.767	0.944	0.316
4	V.High (.9-1)	1051	0.885	0.987	0.133

	Metric	Value
0	Accuracy (Exact)	0.4295
1	Accuracy (+/-1 Star)	0.7443
2	MAE	0.9804
3	RMSE	1.4697
4	F1 Score (Macro)	0.2857
5	F1 Score (Weighted)	0.4550
6	F1 Score (+/-1 Star)	0.8534
7	Precision (Macro)	0.2906
8	Recall (Macro)	0.3021
9	Cohen's Kappa	0.1145
10	Sample Size	99,934

	Confidence Level	Num Reviews	Accuracy	Close Acc (+/-1S)	MAE
0	V.Low (0-.25)	14236	0.216	0.653	1.255
1	Low (.25-.5)	17800	0.399	0.727	1.029
2	Med (.5-.75)	31608	0.465	0.816	0.840
3	High (.75-.9)	20854	0.666	0.858	0.573
4	V.High (.9-1)	1051	0.838	0.937	0.276

	Metric	Value
0	Accuracy (Exact)	0.4599
1	Accuracy (+/-1 Star)	0.8194
2	MAE	0.8136
3	RMSE	1.2581
4	F1 Score (Macro)	0.3089
5	F1 Score (Weighted)	0.4785
6	F1 Score (+/-1 Star)	0.9007
7	Precision (Macro)	0.3202
8	Recall (Macro)	0.3103
9	Cohen's Kappa	0.1360
10	Sample Size	99,934

	Accuracy (Mean)	Accuracy (Std)	MAE (Mean)	MAE (Std)	F1 Close Acc. (Mean)	F1 Close Acc. (Std)
VADER	0.468600	0.001600	0.841500	0.003800	0.888200	0.001600
TextBlob	0.159900	0.001000	1.281900	0.003100	0.730900	0.002200
Feature-Based	0.519700	0.002200	0.773500	0.005300	0.893700	0.001500
N-gram Enhanced	0.405900	0.002700	0.829100	0.006100	0.898800	0.001700

	Cluster_ID	Num_Reviews_in_Cluster	Average_Rating	Rating_StdDev	Top_Category
0	0	9888	4.011226	1.209863	electronics
1	1	90046	4.247052	1.179567	electronics

	Confidence Level	Num Reviews	Accuracy	Close Acc (+/-1S)	MAE
0	V.Low (0-.25)	14236	0.224	0.694	1.183
1	Low (.25-.5)	17800	0.419	0.802	0.882
2	Med (.5-.75)	31608	0.530	0.921	0.590
3	High (.75-.9)	20854	0.690	0.948	0.386
4	V.High (.9-1)	1051	0.870	0.987	0.148

	Metric	Value
0	Accuracy (Exact)	0.3433
1	Accuracy (+/-1 Star)	0.7506
2	MAE	0.9894
3	RMSE	1.3575
4	F1 Score (Macro)	0.2538
5	F1 Score (Weighted)	0.3787
6	F1 Score (+/-1 Star)	0.8575
7	Precision (Macro)	0.2959
8	Recall (Macro)	0.2838
9	Cohen's Kappa	0.0893
10	Sample Size	99,934

	Confidence Level	Num Reviews	Accuracy	Close Acc (+/-1S)	MAE
0	V.Low (0-.25)	14236	0.140	0.453	1.517
1	Low (.25-.5)	17800	0.209	0.677	1.172
2	Med (.5-.75)	31608	0.364	0.921	0.748
3	High (.75-.9)	20854	0.660	0.950	0.413
4	V.High (.9-1)	1051	0.841	0.987	0.177

	Metric	Value
0	Accuracy (Exact)	0.4157
1	Accuracy (+/-1 Star)	0.8151
2	MAE	0.8261
3	RMSE	1.1974
4	F1 Score (Macro)	0.2830
5	F1 Score (Weighted)	0.4463
6	F1 Score (+/-1 Star)	0.8981
7	Precision (Macro)	0.3370
8	Recall (Macro)	0.3053
9	Cohen's Kappa	0.1370
10	Sample Size	99,934

	Confidence Level	Num Reviews	Accuracy	Close Acc (+/-1S)	MAE
0	V.Low (0-.25)	14236	0.174	0.603	1.262
1	Low (.25-.5)	17800	0.252	0.848	0.954
2	Med (.5-.75)	31608	0.488	0.935	0.610
3	High (.75-.9)	20854	0.750	0.947	0.329
4	V.High (.9-1)	1051	0.885	0.987	0.133

	Metric	Value
0	Accuracy (Exact)	0.5805
1	Accuracy (+/-1 Star)	0.8050
2	MAE	0.7020
3	RMSE	1.2250
4	F1 Score (Macro)	0.2871
5	F1 Score (Weighted)	0.5151
6	F1 Score (+/-1 Star)	0.8919
7	Precision (Macro)	0.2409
8	Recall (Macro)	0.3810
9	Cohen's Kappa	0.2537
10	Sample Size	99,934

	Metric	Value
0	Accuracy (Exact)	0.4893
1	Accuracy (+/-1 Star)	0.8291
2	MAE	0.7437
3	RMSE	1.1615
4	F1 Score (Macro)	0.3250
5	F1 Score (Weighted)	0.5083
6	F1 Score (+/-1 Star)	0.9065
7	Precision (Macro)	0.3542
8	Recall (Macro)	0.3254
9	Cohen's Kappa	0.1796
10	Sample Size	99,934

	Confidence Level	Num Reviews	Accuracy	Close Acc (+/-1S)	MAE
0	V.Low (0-.25)	14236	0.217	0.689	1.146
1	Low (.25-.5)	17800	0.395	0.845	0.820
2	Med (.5-.75)	31608	0.595	0.924	0.519
3	High (.75-.9)	20854	0.761	0.946	0.319
4	V.High (.9-1)	1051	0.885	0.987	0.133