Have you ever wondered how Snapchat filters know exactly where your eyes and mouth are? Or how your phone can unlock just by looking at your face? The magic behind this is called Computer Vision, a field of Artificial Intelligence that allows computers to “see” and understand digital images.

Today, we are going to build something incredibly fun using Computer Vision: a Real-Time Meme Matcher.

Point your webcam at yourself, make a shocked face, and watch as the app instantly matches you with the “Overly Attached Girlfriend” meme. Smile and raise your hand, and Leonardo DiCaprio raises a glass right back at you.

But this isn’t just a fun project. We are going to build this using Object-Oriented Programming (OOP). OOP is a professional coding style that makes your code clean, organized, and easy to upgrade. By the end of this tutorial, you will have a working AI app and a solid understanding of how professional software is structured.

Let’s dive in!

Prerequisites

Before we start coding, make sure you have the following ready:

  • Python 3.11 or newer installed on your computer.
  • A working webcam.
  • A folder named assets in your project directory containing a few popular meme images (like success_kid.jpg, disaster_girl.jpg, etc.).

You will also need to install a few Python libraries. Open your terminal or command prompt and run:

pip install mediapipe opencv-python numpy

The Theory: How Does It Work?

Before we look at the code, let’s understand the two main concepts powering our application: Computer Vision (Facial Landmarks) and Object-Oriented Programming.

1. Facial Landmarks (How the AI “Sees” You)

We are using a Google library called MediaPipe. When you feed an image to MediaPipe, it places a virtual “mesh” of 478 invisible dots (called landmarks) over your face.

To figure out your expression, we use simple math. For example, how do we know if your mouth is open in surprise?

We measure the vertical distance between the dot on your top lip and the dot on your bottom lip.

If the distance is large, your mouth is open! We do the same for your eyes and eyebrows to calculate “scores” for surprise, smiling, or concern.

2. Object-Oriented Programming (OOP)

Instead of writing one massive, confusing block of code, OOP allows us to break our program into separate components called Classes.

Think of a Class as a blueprint.

For our Meme Matcher, we will create three distinct classes, each with a “Single Responsibility” (a golden rule of coding):

  1. ExpressionAnalyzer (The Brain): Handles the AI math and MediaPipe.
  2. MemeLibrary (The Database): Loads the images and compares the user’s face to the memes.
  3. MemeMatcherApp (The UI): Opens the webcam and draws the pictures on the screen.

Step 1: Building the Brain

Let’s start by creating the class that does all the heavy lifting. Create a file named meme_matcher.py and import the necessary tools. Then, we will define our first class.

import cv2
import numpy as np
import mediapipe as mp
from pathlib import Path
from concurrent.futures import ThreadPoolExecutor
import pickle
import os
import subprocess

class ExpressionAnalyzer:
    """
    The ExpressionAnalyzer class acts as the 'Brain' of our project.
    It encapsulates (hides away) the complex MediaPipe machine learning logic.
    """
    
    # Class Variables: Landmark indices for eyes, eyebrows, and mouth
    LEFT_EYE_UPPER = [159, 145, 158]
    LEFT_EYE_LOWER = [23, 27, 133]
    RIGHT_EYE_UPPER = [386, 374, 385]
    RIGHT_EYE_LOWER = [253, 257, 362]
    LEFT_EYEBROW = [70, 63, 105, 66, 107]
    RIGHT_EYEBROW = [300, 293, 334, 296, 336]
    MOUTH_OUTER = [61, 291, 39, 181, 0, 17, 269, 405]
    MOUTH_INNER = [78, 308, 95, 88]
    NOSE_TIP = 4

    def __init__(self, frame_skip: int = 2):
        self.last_features = None  
        self.frame_counter = 0     
        self.frame_skip = frame_skip 

        # Download the required AI models automatically
        self.face_model_path = self._download_model(
            "face_landmarker.task",
            "[https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/1/face_landmarker.task](https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/1/face_landmarker.task)"
        )
        self.hand_model_path = self._download_model(
            "hand_landmarker.task",
            "[https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/1/hand_landmarker.task](https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/1/hand_landmarker.task)"
        )

        # Initialize MediaPipe objects for both video and images
        
        self.face_mesh_video = self._init_face_landmarker(video_mode=True)
        self.hand_detector_video = self._init_hand_landmarker(video_mode=True)
        self.face_mesh_image = self._init_face_landmarker(video_mode=False)
        self.hand_detector_image = self._init_hand_landmarker(video_mode=False)

Understanding the Brain

In the code above, we define lists of numbers like LEFT_EYE_UPPER. These are the exact dot numbers (out of the 478) that outline the eye.

The __init__ method is a special function called a constructor. Whenever we create an ExpressionAnalyzer, this code runs automatically to set everything up. It downloads the MediaPipe AI models from Google’s servers and loads them into memory so they are ready to process faces.

Next, we add the logic to extract features:

    # ... (Add this inside the ExpressionAnalyzer class) ...

    def extract_features(self, image: np.ndarray, is_static: bool = False) -> dict:
        """Analyzes an image and returns facial/hand features as a dictionary."""
        
        face_landmarker = self.face_mesh_image if is_static else self.face_mesh_video
        
        hand_landmarker = self.hand_detector_image if is_static else self.hand_detector_video

        rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
        
        mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=rgb)

        if is_static:
            face_res = face_landmarker.detect(mp_image)
            hand_res = hand_landmarker.detect(mp_image)
        else:
            self.frame_counter += 1
            if self.frame_counter % self.frame_skip != 0:
                return getattr(self, "last_features", None)
            
            face_res = face_landmarker.detect_for_video(mp_image, self.frame_counter)
            hand_res = hand_landmarker.detect_for_video(mp_image, self.frame_counter)

        if not face_res.face_landmarks:
            return None

        landmarks = face_res.face_landmarks[0]
        landmark_array = np.array([[l.x, l.y] for l in landmarks])
        
        # Calculate the mathematical features
        features = self._compute_features(landmark_array, hand_res)
        self.last_features = features
        return features

    def _compute_features(self, landmark_array: np.ndarray, hand_res) -> dict:
        """Helper function to calculate Eye Aspect Ratio (How open the eye is)"""
        
        def ear(upper, lower):
            vert = np.linalg.norm(landmark_array[upper] - landmark_array[lower], axis=1).mean()
            horiz = np.linalg.norm(landmark_array[upper[0]] - landmark_array[upper[-1]])
            return vert / (horiz + 1e-6) 

        left_ear = ear(self.LEFT_EYE_UPPER, self.LEFT_EYE_LOWER)
        right_ear = ear(self.RIGHT_EYE_UPPER, self.RIGHT_EYE_LOWER)
        avg_ear = (left_ear + right_ear) / 2.0

        # Mouth calculations
        
        mouth_top, mouth_bottom = landmark_array[13], landmark_array[14]
        mouth_height = np.linalg.norm(mouth_top - mouth_bottom)
        mouth_left, mouth_right = landmark_array[61], landmark_array[291]
        mouth_width = np.linalg.norm(mouth_left - mouth_right)
        mouth_ar = mouth_height / (mouth_width + 1e-6)

        # Eyebrow calculations
        
        left_brow_y = landmark_array[self.LEFT_EYEBROW][:, 1].mean()
        right_brow_y = landmark_array[self.RIGHT_EYEBROW][:, 1].mean()
        left_eye_center = landmark_array[self.LEFT_EYE_UPPER + self.LEFT_EYE_LOWER][:, 1].mean()
        right_eye_center = landmark_array[self.RIGHT_EYE_UPPER + self.RIGHT_EYE_LOWER][:, 1].mean()
        
        avg_brow_h = ((left_eye_center - left_brow_y) + (right_eye_center - right_brow_y)) / 2.0

        # Check for hands
        
        num_hands = len(hand_res.hand_landmarks) if hand_res.hand_landmarks else 0
        hand_raised = 1.0 if num_hands > 0 else 0.0

        return {
            'eye_openness': avg_ear,
            'mouth_openness': mouth_ar,
            'eyebrow_height': avg_brow_h,
            'hand_raised': hand_raised,
            'surprise_score': avg_ear * avg_brow_h * mouth_ar,
            'smile_score': (1.0 - mouth_ar),
        }

This section might look heavily mathematical, but it’s just measuring distances! For instance, mouth_height calculates the distance from the top lip to the bottom lip. We bundle all these measurements into a neat little package (a Python dictionary) and return it.

Step 2: Building the Database

Now that our brain can understand expressions, we need a library to hold our memes.

class MemeLibrary:
    """
    Acts as a database for our memes. 
    It 'has-a' relationship with ExpressionAnalyzer (Dependency Injection).
    """
    
    CACHE_FILE = "meme_features_cache.pkl"

    def __init__(self, analyzer: ExpressionAnalyzer, assets_folder: str = "assets", meme_height: int = 480):
        self.analyzer = analyzer 
        self.assets_folder = assets_folder
        self.meme_height = meme_height

        self.memes = []
        self.meme_features = []

        self.feature_keys = ['surprise_score', 'smile_score', 'hand_raised', 'eye_openness', 'mouth_openness', 'eyebrow_height']
        self.feature_weights = np.array([25, 20, 25, 20, 25, 20])
        self.feature_factors = np.array([10, 10, 15, 5, 5, 5])

        self.load_memes()

    def load_memes(self):
        """Loads memes from disk or a cache file to save time."""
        if os.path.exists(self.CACHE_FILE):
            with open(self.CACHE_FILE, "rb") as f:
                self.memes, self.meme_features = pickle.load(f)
            return

        assets_path = Path(self.assets_folder)
        image_files = list(assets_path.glob("*.jpg")) + list(assets_path.glob("*.png"))

        # Analyze multiple memes at the same time
        with ThreadPoolExecutor() as executor:
            results = list(executor.map(self._process_single_meme, image_files))

        for r in results:
            if r:
                meme, features = r
                self.memes.append(meme)
                self.meme_features.append(features)

        with open(self.CACHE_FILE, "wb") as f:
            pickle.dump((self.memes, self.meme_features), f)

    def _process_single_meme(self, img_file: Path) -> tuple:
        img = cv2.imread(str(img_file))
        if img is None: return None
        
        h, w = img.shape[:2]
        scale = self.meme_height / h
        img_resized = cv2.resize(img, (int(w * scale), self.meme_height))
        
        features = self.analyzer.extract_features(img_resized, is_static=True)
        if features is None: return None
            
        return {'image': img_resized, 'name': img_file.stem.replace('_', ' ').title(), 'path': str(img_file)}, features

    def compute_similarity(self, features1: dict, features2: dict) -> float:
        """Mathematical formula to compare two dictionaries of facial features."""
        if features1 is None or features2 is None: return 0.0
        
        vec1 = np.array([features1.get(k, 0) for k in self.feature_keys])
        vec2 = np.array([features2.get(k, 0) for k in self.feature_keys])
        
        diff = np.abs(vec1 - vec2)
        similarity = np.exp(-diff * self.feature_factors)
        return float(np.sum(self.feature_weights * similarity))

    def find_best_match(self, user_features: dict) -> tuple:
        if user_features is None or not self.memes: return None, 0.0
            
        scores = np.array([self.compute_similarity(user_features, mf) for mf in self.meme_features])
        if len(scores) == 0: return None, 0.0
            
        best_idx = int(np.argmax(scores)) 
        return self.memes[best_idx], scores[best_idx]

The Magic of Dependency Injection

Did you notice how the __init__ method takes analyzer: ExpressionAnalyzer as an argument?

This is a concept called Dependency Injection.

Instead of the Library trying to build its own AI model, we just hand it the Brain we already built. This keeps our code completely separate and organized!

The find_best_match function is where the matching happens. It takes the dictionary of your face (how wide your eyes are, etc.) and compares it to the dictionaries of all the memes. The meme with the closest numbers wins!

Step 3: Building the App Controller

With our AI brain and meme database built, it’s time to bring them to life! We need an application class to turn on your webcam, capture the video, and draw the results on your screen.

class MemeMatcherApp:
    """
    The main Application class. 
    It initializes the other classes and contains the main while loop.
    """
    
    def __init__(self, assets_folder="assets"):
        self.analyzer = ExpressionAnalyzer()
        self.library = MemeLibrary(analyzer=self.analyzer, assets_folder=assets_folder)

    def run(self):
        cap = cv2.VideoCapture(0)
        cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
        cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)

        print("\n🎥 Camera started! Press 'q' to quit\n")

        while cap.isOpened():
            ret, frame = cap.read()
            if not ret: break
            frame = cv2.flip(frame, 1) # Mirror effect

            # 1. Ask the Analyzer to look at the webcam frame
            user_features = self.analyzer.extract_features(frame)
            
            # 2. Ask the Library to find the best matching meme
            best_meme, score = self.library.find_best_match(user_features)

            # 3. Handle the User Interface (Displaying the result)
            h, w = frame.shape[:2]
            
            if best_meme:
                meme_img = best_meme['image']
                meme_h, meme_w = meme_img.shape[:2]
                
                scale = h / meme_h
                new_w = int(meme_w * scale)
                meme_resized = cv2.resize(meme_img, (new_w, h))

                display = np.zeros((h, w + new_w, 3), dtype=np.uint8)
                display[:, :w] = frame               
                display[:, w:w + new_w] = meme_resized 

                # Draw UI Text boxes
                cv2.rectangle(display, (5, 5), (200, 45), (0, 0, 0), -1)
                cv2.putText(display, "YOU", (10, 35), cv2.FONT_HERSHEY_SIMPLEX, 1.2, (0, 255, 0), 2)
                
                cv2.rectangle(display, (w + 5, 5), (w + new_w - 5, 75), (0, 0, 0), -1)
                cv2.putText(display, best_meme['name'], (w + 10, 35), cv2.FONT_HERSHEY_SIMPLEX, 0.8, (0, 255, 255), 2)
            else:
                display = frame
                cv2.putText(display, "No face detected!", (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 0, 255), 2)

            cv2.imshow("Meme Matcher - Press Q to quit", display)
            if cv2.waitKey(1) & 0xFF == ord("q"):
                break

        cap.release()
        cv2.destroyAllWindows()

The Infinite Loop

The core of any video application is a while loop. The application reads one picture from your webcam, asks the ExpressionAnalyzer for the features, asks the MemeLibrary for a match, glues the webcam picture and the meme picture together side-by-side using NumPy, and displays it. Then, it repeats this instantly for the next frame!

Step 4: Putting it All Together

Finally, we just need to start the application. At the very bottom of your file, add the entry point:

if __name__ == "__main__":
    print("Meme Matcher Starting...\n")
    # Create the application object and run it
    app = MemeMatcherApp(assets_folder="assets")
    app.run()

Conclusion

Congratulations! You have just built a complex Artificial Intelligence application using advanced Computer Vision techniques.

More importantly, you built it the right way. By structuring your code using Object-Oriented Programming, your project is scalable. Want to add a Graphical User Interface (GUI) with buttons later? You don’t have to touch the math inside the Brain or the Database; you only have to modify the App class.

To see the real magic, download a few distinct meme images, put them in an assets folder next to your script, and run it. Try raising your eyebrows, opening your mouth wide, or throwing up a peace sign.

Happy coding!

Check out all our books that you can read for free from this page https://10xdev.blog/library