How to Access Android Camera in Unity (III): Customized camera plugin with preview window and UI on it

Overview

   This article will continue the last article and add functionalities such as the camera preview window and buttons to allow user to adjust camera parameters.
  When Unity call this plugin, instead of take a snapshot and leave, the process will stay in the Camera Setting (unless you press the "Return" button so that it'll return to game),  as shown below. In this moment, Unity is temporarily blocked: You can't hear any game sound (unless you invoke it by the android API here), or running game in the background.

Solution

1. At first we need to add one more function "DisplayCameraSettingActivity()" in AndroidCameraAPI.cs, so that it would like:

/*
 * This Class Call the Android Code to Take Picture.
 * Note: there should be a AndroidCameraAPI.jar in \Assets\Plugins\Android
 */
public class AndroidCameraAPI : MonoBehaviour
{
    private AndroidJavaObject androidCameraActivity;

    public void Awake()
    {
        AndroidJavaClass unity = new AndroidJavaClass("com.unity3d.player.UnityPlayer");
        this.androidCameraActivity = unity.GetStatic<AndroidJavaObject>("currentActivity");
    }

    public void TakePhotoFromAndroidAPI(string _pathToSavePhoto, string _fileNameOfSavedPhoto)
    {
        this.androidCameraActivity.Call("TakePhoto",
                                        _pathToSavePhoto, _fileNameOfSavedPhoto+".jpg");
    }

    public void DisplayCameraSettingActivity()
    {
        this.androidCameraActivity.Call("DisplayCameraSettingActivity");
    }
}

2. In the Entrance of Android API (the MainActivity.java):

import android.content.Intent;
import android.hardware.Camera;
import android.os.Bundle;
import android.util.Log;

import com.unity3d.player.UnityPlayerActivity;

public class MainActivity extends UnityPlayerActivity
{
    private Camera androidCamera = null;
    private Camera.Parameters userDefinedCameraParameters = null;

    @Override
    public void onCreate(Bundle _savedInstanceState)
    {
        super.onCreate(_savedInstanceState);

        openCameraSafely();
        setupCameraWithDefaultParameters();
    }
    private void openCameraSafely()
    {
        Log.i("Unity", "Open Camera Safely...");
        try
        {
            this.androidCamera = Camera.open();
        }
        catch(Exception cameraIOException)
        {
            ExceptionManager.SaveErrorCodeToFile(ExceptionManager.CAMERA_ERROR_LOG_FILE_NAME,
                                                "Cannot Connect to Camera,"
                                                + " please Check connection...\n"
                                                + "\tIf this doesn't work, please"
                                                + " Replug the Camera or Reboot this Machine.");

            this.finish();
        }
        Log.i("Unity", "Open Camera Success!");
    }
    private void setupCameraWithDefaultParameters()
    {
        try
        {
            this.userDefinedCameraParameters = this.androidCamera.getParameters();
            this.userDefinedCameraParameters.setExposureCompensation(CameraDataCenter.userDefinedCameraExposureLevel);
            this.userDefinedCameraParameters.setZoom(CameraDataCenter.userDefinedCameraZoomLevel);
            this.userDefinedCameraParameters.setPictureSize(CameraDataCenter.PHOTO_WIDTH,
                                                            CameraDataCenter.PHOTO_HEIGHT);
            this.androidCamera.setParameters(this.userDefinedCameraParameters);
        }
        catch(Exception exception)
        {
            ExceptionManager.SaveErrorCodeToFile(ExceptionManager.CAMERA_ERROR_LOG_FILE_NAME,
                                                "Can't get CameraParameters,"
                                                +" which means the camera might in bad status."
                                                +" Do you access the camera with multithread?");
        }
    }

    public void DisplayCameraSettingActivity()
    {
        Log.i("Unity", "Go to Camera Settings...");
        Intent intentOfCameraSetting = new Intent(this, CameraSettingActivity.class);
        this.androidCamera.release();
        startActivityForResult(intentOfCameraSetting,
                                CameraDataCenter.CAMERA_SETTING_REQUEST_CODE);
    }

    protected void onActivityResult(int _requestCode, int _resultCode, Intent _cameraSettingIntent)
    {
        Log.i("Unity", "MainActivity.onActivityResult().");
        if( (_requestCode == CameraDataCenter.CAMERA_SETTING_REQUEST_CODE)
            &&(_resultCode == RESULT_OK) )
        {
            this.userDefinedCameraParameters.setExposureCompensation(CameraDataCenter.userDefinedCameraExposureLevel);
            this.userDefinedCameraParameters.setZoom(CameraDataCenter.userDefinedCameraZoomLevel);
        }
        this.openCameraSafely();
    }


    public void TakePhoto(final String _targetPathToSavePhoto,
                          final String _targetPhotoFileName)
    {
        Log.i("Unity", "Take Photo...");
        if(this.userDefinedCameraParameters != null)
            this.androidCamera.setParameters(this.userDefinedCameraParameters);
        this.androidCamera.startPreview();

            Camera.PictureCallback savePictureCallBack = new SavePictureCallBack(_targetPathToSavePhoto, _targetPhotoFileName);
            this.androidCamera.takePicture(null, null, savePictureCallBack);

            Log.i("Unity", "Take Photo Success!");
    }

    @Override
    public void onDestroy()
    {
        Log.i("Unity", "CameraActivity is Destroyed!");
        if(this.androidCamera != null)
            this.androidCamera.release();

        super.onDestroy();
    }

}

    You can see that when you call DisplayCameraSettingActivity(), the process will change to CameraSettingActivity which will be demonstrate bellow.
    Note that When you move to another Activity, you should release the lock of android.camera by camera.release() since only one activity can hold the lock of android.camera.
   Also note that when the CameraSettingActivity return (when the user hit the "Return" button on the screen), the function onActivityResult() will be called. That is the time that we require the lock of android.camera back (so that the TakePhoto() can be called latter in Unity). Then the process will return to Unity and continue the game.

3. The CameraSettingActivity.java is shown below:

public class CameraSettingActivity extends Activity
{
    private Camera androidCamera;
    private Camera.Parameters userDefinedCameraParameters = null;
    private CameraPreviewer cameraPreviewer;
    private FrameLayout cameraPreviewFrame;

    private LinearLayout linearLayout;
    private TextView exposureLevelText;
    private TextView zoomLevelText;
    private Button returnButton;
    private final float UI_TEXT_SIZE = 32f;


    @Override
    protected void onCreate(Bundle _savedInstanceState)
    {
        super.onCreate(_savedInstanceState);

        initializeLayoutSetting();

        openCameraSafely();
        setupCameraWithDefaultParameters();
        initializeCameraPreview();

        createReturnButton();
        CreateExposureInspector();
        CreateZoomLevelInspector();
    }
    private void initializeLayoutSetting()
    {
        DisplayMetrics displayedScreen = new DisplayMetrics();
        this.getWindowManager().getDefaultDisplay().getMetrics(displayedScreen);

        this.requestWindowFeature(Window.FEATURE_NO_TITLE);
        this.getWindow().setFlags(WindowManager.LayoutParams.FLAG_FULLSCREEN,
                                  WindowManager.LayoutParams.FLAG_FULLSCREEN);

        this.linearLayout = new LinearLayout(this);
        LayoutParams parametersOfLinearLayout = new LayoutParams(LayoutParams.MATCH_PARENT,
                                                                 LayoutParams.MATCH_PARENT);

        setContentView(this.linearLayout, parametersOfLinearLayout);
    }
    private void openCameraSafely()
    {
        try
        {
            this.androidCamera = Camera.open();
        }
        catch(Exception cameraIOException)
        {
            ExceptionManager.SaveErrorCodeToFile(ExceptionManager.CAMERA_ERROR_LOG_FILE_NAME,
                                                "Cannot Connect to Camera,"
                                                + " please Check connection...\n"
                                                + "\tIf this doesn't work, please"
                                                + " Replug the Camera or Reboot this Machine.");

            this.finish();
        }
    }

    /*
     * UI Related...
     */
    private void createReturnButton()
    {
        this.returnButton = new Button(this);
        this.returnButton.setText("Return");
        this.returnButton.setTextSize(UI_TEXT_SIZE);
        this.returnButton.setOnClickListener(
                new Button.OnClickListener()
                {
                    @Override
                    public void onClick(View _view)
                    {
                        OnReturnButtonClick(_view);
                    }
                }
        );
        this.returnButton.setX(100);
        this.returnButton.setY(600);
        addContentView( this.returnButton, new LayoutParams(150, 75) );
    }

    private void CreateExposureInspector()
    {
        createIncreaseExposureButton(350, 75, 75, 65);
        createCurrentExposureText(500, 85, 400, 70);
        createDecreaseExposureButton(850, 75, 75, 65);
    }
    private void createIncreaseExposureButton(int _left, int _top, int _width, int _height)
    {
        Button increaseExposureButton = new Button(this);
        increaseExposureButton.setText("+");
        increaseExposureButton.setTextSize(UI_TEXT_SIZE);
        increaseExposureButton.setOnClickListener(
                new Button.OnClickListener()
                {
                    @Override
                    public void onClick(View _view)
                    {
                        OnIncreaseExposureButtonClick(_view);
                    }
                }
        );
        increaseExposureButton.setX(_left);
        increaseExposureButton.setY(_top);
        addContentView( increaseExposureButton, new LayoutParams(_width, _height) );
    }
    private void createCurrentExposureText(int _left, int _top, int _width, int _height)
    {
        this.exposureLevelText = new TextView(this);
        this.exposureLevelText.setTextSize(UI_TEXT_SIZE);
        updateExposureLevelTexts();
        this.exposureLevelText.setX(_left);
        this.exposureLevelText.setY(_top);
        addContentView(this.exposureLevelText, new LayoutParams(_width, _height));
    }
    private void createDecreaseExposureButton(int _left, int _top, int _width, int _height)
    {
        Button decreaseExposureButton = new Button(this);
        decreaseExposureButton.setText("-");
        decreaseExposureButton.setTextSize(UI_TEXT_SIZE);
        decreaseExposureButton.setOnClickListener(
                new Button.OnClickListener()
                {
                    @Override
                    public void onClick(View _view)
                    {
                        OnDecreaseExposureButtonClick(_view);
                    }
                }
        );
        decreaseExposureButton.setX(_left);
        decreaseExposureButton.setY(_top);
        addContentView( decreaseExposureButton, new LayoutParams(_width, _height) );
    }
    private void updateExposureLevelTexts()
    {
        int currentExposureLevel = CameraDataCenter.userDefinedCameraExposureLevel;

        if( currentExposureLevel > 0)
            this.exposureLevelText.setText("Exposure Level:  +"+currentExposureLevel);
        else
            this.exposureLevelText.setText("Exposure Level:  "+currentExposureLevel);
    }

    private void CreateZoomLevelInspector()
    {
        createIncreaseZoomLevelButton(350, 150, 75, 65);
        createCurrentZoomLevelText(500, 160, 400, 70);
        createDecreaseZoomLevelButton(850, 150, 75, 65);
    }
    private void createIncreaseZoomLevelButton(int _left, int _top, int _width, int _height)
    {
        Button increaseZoomLevelButton = new Button(this);
        increaseZoomLevelButton.setText("+");
        increaseZoomLevelButton.setTextSize(UI_TEXT_SIZE);
        increaseZoomLevelButton.setOnClickListener(
                new Button.OnClickListener()
                {
                    @Override
                    public void onClick(View _view)
                    {
                        OnIncreaseZoomLeveleButtonClick(_view);
                    }
                }
        );
        increaseZoomLevelButton.setX(_left);
        increaseZoomLevelButton.setY(_top);
        addContentView( increaseZoomLevelButton, new LayoutParams(_width, _height) );
    }
    private void createCurrentZoomLevelText(int _left, int _top, int _width, int _height)
    {
        this.zoomLevelText = new TextView(this);
        this.zoomLevelText.setTextSize(UI_TEXT_SIZE);
        updateZoomLevelTexts();
        this.zoomLevelText.setX(_left);
        this.zoomLevelText.setY(_top);
        addContentView(this.zoomLevelText, new LayoutParams(_width, _height));
    }
    private void createDecreaseZoomLevelButton(int _left, int _top, int _width, int _height)
    {
        Button decreaseZoomLevelButton = new Button(this);
        decreaseZoomLevelButton.setText("-");
        decreaseZoomLevelButton.setTextSize(UI_TEXT_SIZE);
        decreaseZoomLevelButton.setOnClickListener(
                new Button.OnClickListener()
                {
                    @Override
                    public void onClick(View _view)
                    {
                        OnDecreaseZoomLevelButtonClick(_view);
                    }
                }
        );
        decreaseZoomLevelButton.setX(_left);
        decreaseZoomLevelButton.setY(_top);
        addContentView( decreaseZoomLevelButton, new LayoutParams(_width, _height) );
    }
    private void updateZoomLevelTexts()
    {
        int currentZoomLevel = CameraDataCenter.userDefinedCameraZoomLevel;

        if( currentZoomLevel > 0)
            this.zoomLevelText.setText("Zoom Level:  +"+currentZoomLevel);
        else
            this.zoomLevelText.setText("Zoom Level:  "+currentZoomLevel);
    }

    /*
     * Camera Settings Related...
     */
    private void initializeCameraPreview()
    {
        this.cameraPreviewer = new CameraPreviewer(this, this.androidCamera);

        this.cameraPreviewFrame = new FrameLayout(this);
        addContentView(this.cameraPreviewFrame,
                       new LayoutParams(ViewGroup.LayoutParams.FILL_PARENT, LayoutParams.FILL_PARENT)
                      );
        this.cameraPreviewFrame.addView(this.cameraPreviewer);
    }
    private void setupCameraWithDefaultParameters()
    {
        try
        {
            this.userDefinedCameraParameters = this.androidCamera.getParameters();
            this.userDefinedCameraParameters.setPictureSize(CameraDataCenter.PHOTO_WIDTH,
                                                            CameraDataCenter.PHOTO_HEIGHT);
            updateCameraExposureSettings(CameraDataCenter.userDefinedCameraExposureLevel);
            updateCameraZoomSettings(CameraDataCenter.userDefinedCameraZoomLevel);
        }
        catch(Exception exception)
        {
            ExceptionManager.SaveErrorCodeToFile(ExceptionManager.CAMERA_ERROR_LOG_FILE_NAME,
                                                 "Can't get CameraParameters,"
                                                +" which means the camera might in bad status."
                                                +" Do you access the camera with multithread?");
        }
    }
    private void updateCameraExposureSettings(int _userRequiredExposure)
    {
        try
        {
            CameraDataCenter.userDefinedCameraExposureLevel = _userRequiredExposure;
            this.userDefinedCameraParameters.setExposureCompensation(_userRequiredExposure);
            this.androidCamera.setParameters(userDefinedCameraParameters);
        }
        catch(Exception exception)
        {
            ExceptionManager.SaveErrorCodeToFile(ExceptionManager.CAMERA_ERROR_LOG_FILE_NAME,
                                                "Can't Set CameraParameters,"
                                                +"  Are there Invalid Camera Settings "
                                                +"(such as Invalid Picture Size, Zoom Level"
                                                +" or Exposure Level)?");
        }
    }
    private void updateCameraZoomSettings(int _userDefinedZoomLevel)
    {
        try
        {
            CameraDataCenter.userDefinedCameraZoomLevel = _userDefinedZoomLevel;
            this.userDefinedCameraParameters.setZoom(_userDefinedZoomLevel);
            this.androidCamera.setParameters(userDefinedCameraParameters);
        }
        catch(Exception exception)
        {
            ExceptionManager.SaveErrorCodeToFile(ExceptionManager.CAMERA_ERROR_LOG_FILE_NAME,
                                                "Can't Set CameraParameters,"
                                                +"  Are there Invalid Camera Settings "
                                                +"(such as Invalid Picture Size, Zoom Level"
                                                +" or Exposure Level)?");
        }
    }

    /*
     * Following is the Call Back Functions for Button Clicked.
     */
    public void OnIncreaseExposureButtonClick(View _increaseExposureButton)
    {
        int currentExposure = this.userDefinedCameraParameters.getExposureCompensation();
        ++currentExposure;
        int maxExposure = this.userDefinedCameraParameters.getMaxExposureCompensation();
        if( currentExposure > maxExposure )
            currentExposure = maxExposure;
        updateCameraExposureSettings(currentExposure);
        updateExposureLevelTexts();
    }
    public void OnDecreaseExposureButtonClick(View _decreaseExposureButton)
    {
        int currentExposure = this.userDefinedCameraParameters.getExposureCompensation();
        --currentExposure;
        int minExposure = this.userDefinedCameraParameters.getMinExposureCompensation();
        if( currentExposure < minExposure)
            currentExposure = minExposure;
        updateCameraExposureSettings(currentExposure);
        updateExposureLevelTexts();
    }
    public void OnIncreaseZoomLeveleButtonClick(View _increaseZoomButton)
    {
        int currentZoomLevel = this.userDefinedCameraParameters.getZoom();
        ++currentZoomLevel;
        int maxZoomLevel = this.userDefinedCameraParameters.getMaxZoom();
        if( currentZoomLevel > maxZoomLevel )
            currentZoomLevel = maxZoomLevel;
        updateCameraZoomSettings(currentZoomLevel);
        updateZoomLevelTexts();
    }
    public void OnDecreaseZoomLevelButtonClick(View _decreaseZoomeButton)
    {
        int currentZoomLevel = this.userDefinedCameraParameters.getZoom();
        --currentZoomLevel;
        int minZoomLevel = 0;
        if( currentZoomLevel < minZoomLevel)
            currentZoomLevel = minZoomLevel;
        updateCameraZoomSettings(currentZoomLevel);
        updateZoomLevelTexts();
    }

    public void OnReturnButtonClick(View _returnButton)
    {
        this.androidCamera.stopPreview();

        Intent returnIntent = new Intent();
        setResult(RESULT_OK, returnIntent);

        Log.i("Unity", "Release Camera...");
        this.androidCamera.release();
        this.finish();
    }

    @Override
    public void onDestroy()
    {
        Log.i("Unity", "CameraActivity is Destroyed!");
        if(this.androidCamera != null)
            this.androidCamera.release();

        super.onDestroy();
    }
}

   You can see that I create the buttons dynamically in onCreate(), instead of using the R.layout (which is the normal way to create UI widget in android), because there're some issue about the missing of  R.layout when the Unity use android plugins. I guess the reason might be that Unity will replace the R.layout itself... And this is the reason that I don't call the

    setContentView(R.layout.activity_main);

in the onCreate() as we normally do when writing "pure" android app.

4. Since you have two Activities now, you should also change your AndroidManifest.xml to:

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
      package="com.igs.dinosaur"
      android:versionCode="1"
      android:versionName="1.0">
    <uses-sdk android:minSdkVersion="9" />
 
 <uses-permission android:name="android.permission.CAMERA" />
 <uses-feature android:name="android.hardware.camera" />
 <uses-feature android:name="android.hardware.camera.autofocus" />
 <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
 
    <application
  android:icon="@drawable/app_icon"
  android:label="@string/app_name"
  android:debuggable="true">
  
        <activity android:name=".MainActivity"
                  android:label="@string/app_name">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />
                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>
  
  <activity android:name=".CameraSettingActivity">
  </activity>
  
    </application>
</manifest>

5. Finally, here is the CameraPreviewer that will be called by the CameraSettingActivity:

public class CameraPreviewer extends SurfaceView
                            implements SurfaceHolder.Callback
{
    private SurfaceHolder displayedSurface;
    private Camera currentCamera;

    public CameraPreviewer(Context _cameraContext, Camera _camera)
    {
        super(_cameraContext);
        this.currentCamera = _camera;
        this.displayedSurface = this.getHolder();
        this.displayedSurface.addCallback(this);

        //Following line might be required if you use the early version of android
        this.displayedSurface.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
    }

    @Override
    public void surfaceCreated(SurfaceHolder _surfaceHolder)
    {
        try
        {
            this.currentCamera.setPreviewDisplay(_surfaceHolder);
            this.currentCamera.startPreview();
        }
        catch (IOException exception)
        {
            ExceptionManager.SaveErrorCodeToFile(ExceptionManager.CAMERA_ERROR_LOG_FILE_NAME,
                                                 "Camera Preview failed, this might due to"
                                                 +" the SurfaceHolder cannot be obtained...");
        }
    }

    @Override
    public void surfaceChanged(SurfaceHolder _surfaceHolder, int _format, int _width, int _height)
    {
        this.Refresh();
    }

    public void Refresh()
    {
        stopPreviewBeforeRefresh();
        restartPreview();
    }
    private void stopPreviewBeforeRefresh()
    {
        try
        {
            this.currentCamera.stopPreview();
        }
        catch(Exception _exception)
        {
            /*
             * This Exception can be Ignore, since it
             * might be no preview before.
             */
        }
    }
    private void restartPreview()
    {
        try
        {
            this.currentCamera.setPreviewDisplay(this.displayedSurface);
            this.currentCamera.startPreview();
        }
        catch (IOException exception)
        {
            ExceptionManager.SaveErrorCodeToFile(ExceptionManager.CAMERA_ERROR_LOG_FILE_NAME,
                                                 "Camera Preview failed, this might due to"
                                                 + " the SurfaceHolder cannot be obtained...");
        }
    }

    @Override
    public void surfaceDestroyed(SurfaceHolder _surfaceHolder) {}
}

Result

    Now you can adjust the camera parameters by click the buttons and preview the camera, as shown below.

Conclusion

    Here're the advantages and disadvantages of calling the camera in Android API compare to the Unity API:

Advantages:

     As mention in the last article, you can adjust many camera parameters in android API.

Disadvantages:

    When calling the android API, Unity will temporary stopped. So if you want to play music, display UI or detect IO, you should do it in Android API! You can't rely on the easy-to-use Unity anymore.

2016/6/2 updated:
   We finally solve the Unity temporary stopped problem by using the Android Service. The Service is like running things in another thread, but yet its display can cover the original APP.
   Therefore, we extend the Service that can display camera video. When we want to show the WebCam preview window, we call that Service. And it will display the camera video which will cover the display of our game but yet leave the game still running.
    This solution could also be used to solve the temporary stop problem when you want to apply Unity API to display movie in Android. You can extend the Service to display movie so that the game will still running when the movie is playing.