Making a game with webcam controls

Making a game with webcam controls

When I was young I was fascinated by the playstation 2 EyeToy. I used to wonder how it worked and the movement detection intrigued me. During my years at university I took a course on computer vision thinking that it might fill me in on the doubts I had on the subject. The course was great on a theoretical point of view and I thought there was no better way to test out the theories than by making a webcam game. Unfortunately at the time I did not have time but I kept the idea close to heart.

So a few years later I finally had the time and motivation to make this project reality. The idea was simple there are enemies attacking you from all the screen edges and you need to move your hand around to defend yourself. Of course, being part of Sleeping Penguinz, I had to make the enemies angry penguins that threw snowballs at you. Now that I had an idea all that was left was implementing the webcam input. In unity we can get the webcam input texture like so:

WebCamTexture webcam;
Texture2D outputBackground;
Texture2D outputMotion;

Color32[] data;
Color32[] preData;
public RawImage webcamBackground;
public RawImage webcamMotion;

IEnumerator Start()
{

    yield return Application.RequestUserAuthorization(UserAuthorization.WebCam | UserAuthorization.Microphone);
    if (Application.HasUserAuthorization(UserAuthorization.WebCam | UserAuthorization.Microphone))
    {
        webcam = new WebCamTexture();
        webcam.Play();

        yield return new WaitUntil( () => webcam.didUpdateThisFrame);

        Debug.Log("webcam size is: " + webcam.width + " " + webcam.height);
        outputBackground = new Texture2D(webcam.width, webcam.height);
        outputMotion = new Texture2D(webcam.width, webcam.height);


        webcamBackground.texture = outputBackground;
        webcamMotion.texture = outputMotion;


        data = new Color32[webcam.width * webcam.height];
        preData = new Color32[webcam.width * webcam.height];
    }
}

void Update()
{
    if (countTimer)
    {
        timer += Time.deltaTime;
        int minutes = (int)(timer / 60);
        int seconds = (int)timer % 60;
        foreach (Text t in timeText) { t.text = minutes.ToString("00") + ":" + seconds.ToString("00"); }
    }
    if (data != null && webcam.didUpdateThisFrame && !gamePaused)
    {
        webcam.GetPixels32(data);
        //backgroud
        outputBackground.SetPixels32(data);
        outputBackground.Apply();
        
        //motion
        Color32[] motion = SubtractImage(data, preData);
        outputMotion.SetPixels32(motion);
        outputMotion.Apply();

        //hit balls
        foreach(Ball ball in balls)
        {
            if(!ball.gameObject.activeSelf) { continue; }
            if (ball.Destroyed()) { continue; }
            if(RectHit(ball.rectTransform, motion))
            {
                ScoreUp(ball.Hit());
            }
        }

        //Array.Copy(data, preData, data.Length);
        webcam.GetPixels32(preData);
    }
}

Vector2 ScreenToWebcam(Vector2 point)
{

    RectTransform webcamRect = webcamBackground.GetComponent<RectTransform>();

    Vector2 screenToWebcam = new Vector2(webcam.width / webcamRect.sizeDelta.x,
                                            webcam.height / webcamRect.sizeDelta.y);

    Vector2 webcamSpacePoint = new Vector2((point.x - webcamRect.anchoredPosition.x)* screenToWebcam.x,
                                    (point.y - webcamRect.anchoredPosition.y)* screenToWebcam.y);

    return webcamSpacePoint;
}

private bool RectHit(RectTransform rect, Color32[] motion)
{
    //anchored bottom left
    Vector2 bl = new Vector2(rect.anchoredPosition.x, rect.anchoredPosition.y);
    Vector2 tr = new Vector2(bl.x+ rect.sizeDelta.x, bl.y+ rect.sizeDelta.y);

    //enemy image in webcam size
    Vector2 blwc = ScreenToWebcam(bl);
    Vector2 trwc = ScreenToWebcam(tr);

    // Debug.Log(bl.ToString() + tr.ToString() + blwc.ToString() + trwc.ToString());

    int xSize = (int)(trwc.x - blwc.x);
    int ySize = (int)(trwc.y - blwc.y);

    int countMotionPixels = 0;
    for(int x = (int)blwc.x; x < trwc.x; x++)
    {
        for (int y = (int)blwc.y; y < trwc.y; y++)
        {
            int index = webcam.width * y + (webcam.width-x);
            //Debug.Log("x: "+x+" y "+y);
            if(index < 0 || index >= motion.Length) { continue; }
            if (motion[index].a != 0) { countMotionPixels++; }
        }
    }

    if (countMotionPixels >= (xSize * ySize) / 3) { return true; }

    return false;
}

Color32[] SubtractImage(Color32[] d1, Color32[] d2)
{
    Color32[] dout = d1;
    for(int i = 0; i<d1.Length; i++)
    {
        
        int r = Mathf.Abs( d1[i].r - d2[i].r );
        int g = Mathf.Abs( d1[i].g - d2[i].g );
        int b = Mathf.Abs( d1[i].b - d2[i].b );
        if (r+g+b > motionThreshold)
        {
            dout[i] = Color.green;
        }
        else
        {
            dout[i].a = 0;
        }
    }
    return dout;
}

Take into account that this code snippet is an extract of my code and might be missing function definitions to be functional. I am sure you can fill in the wholes. If you struggle for a very long time to do so, feel free to contact me and I will see if I can help you out.

By reading the code above you might have figured out the technique I use to detect motion. It is actually the most basic idea and surprisingly it works quite well. The idea is taking the current input frame from the webcam and subtracting the previous from from it. Comparing each pixel of the result to a arbitrary threshold (depends on the camera and lighting of the room, should be set by the player) we can find every spot on the input image that “moved”/changed from the previous input image.

Now that we know how to detect movement on the webcam input we need to map that movement on the screen. To do so I used the Unity UI canvas and scaled down the resolution of the screen to match the webcam pixels one-to-one.

From there I knew the area and position that a snowball took on screen. To detect if the player hits a snowball I compute the number of pixels that “moved”/changed inside that snowball. If the number of pixels that changed is greater than a third (arbitrary number, it could be anything, but from play testing it felt right) of the area of the snowball then it counts as a hit.

As you see the idea is simple and it works fairly well. It took me two or three days to get the webcam movement detection to work as I wanted. Then I started adding more features, penguins and projectiles like bombs, fish and bananas, that have different patterns from snowballs.

All the beautiful art was made by Moulfry. You can find the game here

In the end I passed roughly a month working on Penguin Move and I really enjoyed making this game.

Author face

Santiago Rubio (Sangemdoko)

A electronics and information engineer who works on game development in his free time. He created Sleeping Penguinz to publish the games he makes with his friends and familly.

Recent post