Visual Media Processing Using MATLAB Beginner's Guide

Altering structuring elements from strel to suit our needs

Time for action – ROI refinement using strel

More morphological operations

Manually defining a non-rectangular ROI

Using roipoly to make a mask

Using imfreehand to make a mask

Time for action – making a custom object eraser function

Analyzing objects in an image

Detecting edges in an image

Detecting corners in an image

Detecting circles in an image

Summary

[ ii ]

33

33

35

36

36

39

41

41

42

43

44

45

46

47

48

49

51

53

57

57

60

63

64

64

66

67

71

72

73

74

78

78

78

80

81

84

84

86

87

90

Table of ContentsChapter 4: Working with Color Images

An introduction to color image processing

Basic color image manipulations

Setting a rectangular area to a specified color

Time for action – repainting two areas in a color image

Thresholding color images

Time for action – isolating the red pixels in an image

Achieving color masking

Time for action – color isolation

The importance of different color spaces

Time for action – color space transformation

CIE-Lab* for more efficient color masking

Time for action – color isolation using CIE-Lab*

Fixing illumination issues in RGB color images

Fixing illumination issues in CIE-Lab*

A practical example – red eye reduction

Time for action – writing a function for red eye reduction

Taking advantage of eye circularity

Time for action – automating our function for red eye reduction

Summary

Chapter 5: 2-Dimensional Image Filtering

An introduction to image filtering

Processing neighborhoods of pixels

The basics of convolution

The ugly mathematical truth

Time for action – applying averaging filters in images

Alternatives to convolution

Using imfilter

Creating filters with fspecial

Different ways to blur an image

Time for action – how much blurring is enough

Time to make art using blurring

Time for action – creating the bokeh effect in an image

Removing noise using blurring

Time for action – trying to remove different types of noise

The importance of the median filter

Time for action – removing salt & pepper with medfilt2

Bringing back the details

Time for action – enhancing the edges in our images

[ iii ]

91

92

93

95

96

97

98

101

102

104

104

106

106

110

112

113

114

117

117

122

123

124

124

127

128

129

132

133

133

134

134

137

137

141

142

145

146

147

147

Table of ContentsBrighten up the lights

Time for action – brighten up the lights in our soldier picture

Summary

Chapter 6: Mixing Images for Science or Art

The importance of mixing or combining images

Using multispectral imaging

Loading and manipulating the multispectral images

Time for action – visible spectrum from a multiband image of Rio

Time for action – working with invisible spectrums

Creating composite images

Using imfuse to create a composite image

Using imshowpair to inspect a composite image

Time for action – cloning the seagull

One step beyond – blending selected image regions

Time for action – directing a threatening scene

Creating High Dynamic Range images

Time for action – composing your own HDR images

Stitching images for the creation of panoramas

Time for action – basic approach to panorama stitching

Summary

Chapter 7: Adding Motion – From Static Images to Digital Videos

An introduction to digital videos

The meaning of frames

Interlaced versus progressive

Frame rates and their importance

Calculating number of frames

Some thoughts on choosing frame rates

Loading videos in MATLAB

Loading videos with aviread

Loading videos with mmreader

Loading videos with VideoReader

Choosing which function to use for video reading

Playing back videos in MATLAB

Time for action – reading and playing back a video

Making videos from static images

Time for action – constructing and saving a video

Inspecting a video using montage

Time for action – don't wait for the ball

A tool just for your playback needs – implay

Using the GUI of implay

[ iv ]

149

150

151

153

154

154

155

156

159

162

162

162

162

166

168

172

172

175

175

179

181

182

182

182

183

184

184

185

185

187

188

189

190

191

192

193

195

195

196

197

Table of ContentsUsing implay to play a video file

Using implay to play an image sequence

Creating time-lapse videos

Time for action – time-lapsing a regular video

Saving your time-lapse videos in a gif file

Summary

Chapter 8: Acquiring and Processing Videos

Using MATLAB for digital video recording

The Hardware Browser window

The Information window

The Desktop Help window

The Preview window

The Acquisition Parameters window

The General tab

The Device Properties tab

The Logging tab

The Triggering tab

The Region of Interest tab

The Session Log window

Time for action – capturing a video using a firewire connection

The importance of video compression

Checking the size of an uncompressed video

Checking the size of an MP4 video without any motion

Checking the size of an MP4 video with high motion

Working with uncompressed videos

Working with large videos in postproduction

Time for action – making an edge detection video

Acquiring frames for time-lapse videos

Detecting your acquisition hardware

Creating a video object and acquiring a frame

Time for action – using MATLAB as an intervalometer

Real-time processing of time-lapse videos

Time for action – creating time-lapses with isolated colors

Real-time processing of normal videos

Evaluating real-time capabilities with a simple example

Time for action – adjusting the contrast of the video

Revisiting the contrast adjustment example

Time for action – adding preview in our code

Summary

[ v ]

199

199

199

200

201

202

205

206

206

207

207

207

208

208

208

208

209

209

209

210

214

215

216

217

218

219

219

221

222

222

224

226

226

228

228

229

231

231

234

Table of ContentsChapter 9: Spatiotemporal Video Processing

Basic video processing with MATLAB

Cropping and resizing our video

Time for action – loading, cropping, resizing, and saving a video

Filtering your video frames

Time for action – reducing the blocking effect

Deinterlacing videos in MATLAB

Intra-frame filtering for deinterlacing tasks

Deinterlacing with the Computer Vision System Toolbox

Time for action – deinterlacing a video using the vision toolbox

Deinterlacing with the custom functions

Time for action – deinterlacing with line repetition

Time for action – deinterlacing with the scan line interpolation

Inter-frame filtering for the deinterlacing tasks

Temporal deinterlacing by field merging

Time for action – deinterlacing with field merging

Temporal deinterlacing by field averaging

Time for action – deinterlacing with field averaging

Mixing intra-frame and inter-frame deinterlacing

Vertical and temporal interpolation for deinterlacing

Time for action – vertical and temporal interpolation method

Adding a new dimension to the filters

Spatiotemporal averaging filter

Time for action – implementing a spatiotemporal averaging filter

Using convolution for spatiotemporal averaging

Time for action – spatiotemporal averaging filter with the convn function

Summary

Chapter 10: From Beginner to Expert – Handling Motion and 3-D

Detecting and estimating motion in videos

Detecting motion

Time for action – detecting a moving object in a still scene

Time for action – detecting motion in a complex scene

Estimating the motion

Estimating motion using optical flow

Time for action – tracking people with Horn-Schunck optical flow

Time for action – warping frames using optical flow

Compensating camera motion using feature tracking

Time for action – tracking feature points for motion

compensation of a shaky video

[ vi ]

235

235

236

236

241

241

243

244

244

244

246

246

248

250

250

251

253

253

256

256

256

259

260

260

263

263

265

267

268

268

269

271

273

275

275

280

283

283

Table of ContentsWorking with stereoscopic images

Time for action – creating a 3-D video from left and right videos

Time for action – creating a 3-D video from a regular one

Summary

Appendix: Pop Quiz Answers

Index

288

289

290

292

295

301

[ vii ]

Table of ContentsPreface

Digital visual media has, undoubtedly, become a vital part of our everyday lives. Analog means

of storing and processing information have gradually faded and are nowadays used either

by aficionados of analog media, or for very specialized applications. Capturing and storing

image or video information have rapidly become common, fast, and cheap processes, since

almost everyone can have access to a digital electronic device that can be used for these aims,

whether it is a photographic or video camera, or even a mobile phone. The outburst

of visual media-capturing devices has led to an increase of amateur photographers and

weekend filmmakers, who often have a problem deciding what software to use to process

their stored images or videos. The rule of thumb is that free software solutions often have

limited functionalities or are very complicated, while commercial solutions tend to be very

expensive and sometimes do not provide all the functionalities that a user would hope for.

This book presents a rather uncommon alternative solution that might not be considered

by users who only need an image, or a video editing software, but could certainly appeal

to users who are also students, scientists, or just have easy access to the multifunctional,

high level programming environment, called MATLAB.

What this book covers

Chapter 1, Basic Image Manipulations, introduces you to the environment of MATLAB

and takes you on a tour to its basic tools and functionalities. Then, image importing and

displaying in MATLAB is discussed, followed by a demonstration of the MATLAB GUI for

image manipulation. Basic image transformations are covered, such as rotating/flipping,

resizing, and cropping an image. Finally, different ways of writing an image are presented.

The chapter includes hands-on examples that tie most of the processes covered, together.

Chapter 2, Working with Pixels in Grayscale Images, is based on examples of pixel-based

processing of an image. Several classic processes for image enhancement are discussed,

such as thresholding, local, or global contrast enhancement. The methods presented

use several techniques that gently introduce you to the secrets of MATLAB programming.

A practical example in image enhancement concludes this chapter.

Chapter 3, Morphological Operations and Object Analysis, introduces the basic methods

of morphological image analysis. In it, you will learn of ways to perform binarization

of a grayscale image using the thresholding methods. Edge detection and other

morphological operators are presented and explained, so that you learn how to select

and manipulate particular image regions that interest you the most. You will also learn

the techniques that automatically detect corners, circles, and lines in an image. Several

hands-on examples will vividly demonstrate all these techniques.

Chapter 4, Working with Color Images, extends previous methods to color images. Some

of the processes mentioned for grayscale are now revisited for color image processing.

Different color spaces and their advantages are explained with examples on color

enhancement in MATLAB. You will learn how illumination and color can be separated and

processed independently. The technique for color isolation is explained through a practical

example and finally, some of the methods mentioned previously are used to teach you how

to develop a popular application: red eye correction in your photographs.

Chapter 5, 2-Dimensional Image Filtering, dives into some more complex issues for image

filtering, such as deblurring and sharpening of images. You will get to work on more

sophisticated techniques for image denoising. Some more interesting and fun examples will

let you start enjoying your experience more deeply. We will work on ways to apply some of

the filter locally, to enhance or blur specific image regions.

Chapter 6, Mixing Images for Science or Art, will wake up the artist, or the scientist in

you. You will learn the techniques that mix channels of multispectral images for scientific

visualization. Then, we will present fun, hands-on examples for blending, or stitching images,

to produce artistic results. We will also work on ways to create artistic HDR (High Dynamic

Range) images in MATLAB. Finally, we will present a simple way to create panoramic images.

Chapter 7, Adding Motion – From Static Images to Digital Videos, introduces you to video

processing by building on the previous knowledge you have acquired. The fact, that videos

can be generated by static images, will help you to better comprehend basic ideas. So, after

covering the basics of video frame processing in MATLAB and demonstrating how we can load

and play back videos, we will show how to create a video from static images. The construction

of a time-lapse video is the basic hands-on example we will be working on in this chapter.

Chapter 8, Acquiring and Processing Videos, demonstrates the functionalities of the image

acquisition tool for MATLAB. You will be given step-by-step examples on ways to shoot video

with your camera and use your computer as a Digital Video Recorder, using the special GUI

tool contained in MATLAB. Video compression and basic color video processing techniques

are also demonstrated in this chapter, accompanied by a discussion on performance issues.

Chapter 9, Spatiotemporal Video Processing, introduces you to command line manipulation

and processing of videos. After covering basic video frames manipulations in MATLAB, you

will learn how to deinterlace videos, using intra-frame, inter-frame, or mixed techniques.

Furthermore, spatiotemporal video filtering is presented, with hands-on examples to help

you get the idea.

[ 2 ]

PrefaceChapter 10, From Beginner to Expert – Handling Motion and 3-D, introduces you to methods

of motion detection in videos. Building on basic knowledge, we will get to the point

of creating a simple surveillance system in MATLAB. You will also be taught the basics

of estimating motion using popular optical flow algorithms, included in one of the toolboxes

of MATLAB. You will also be introduced to feature-based image registration for motion

compensation. The working example for this will be video stabilization. Finally, we will

introduce an example of three-dimensional video and cover a very basic and fun example

of turning a regular video to a 3-D one.

What you need for this book

In order to practice what you read in this book, you should have access to a computer with

an installed version of MATLAB. The screenshots you will see in this book are all taken from

MATLAB Version R2012b, which was the most recent one at the time of writing this book.

However, since MATLAB is also a programming language, you will not need to worry about

any differences in the way R2012b looks compared to earlier versions. The great majority

of the things we will cover in this book will be 100 percent compatible with most previous

versions. In the rare cases, when we use a brand new functionality, we will also provide

alternative solutions for previous versions.

The most important thing to make sure of, however, is that you have to find an installed

version of MATLAB that includes at least the two basic toolboxes for image and video

processing: Image Processing Toolbox and Image Acquisition Toolbox. An extra toolbox,

named Computer Vision Toolbox, will also be used for a very small part of video processing.

Toolboxes are collections of ready-made functions for special purposes. For those of you

with a little familiarity with programming, they could be thought of as libraries. The more

toolboxes included in your installation of MATLAB, the more functionalities the environment

will provide for you. Most of our code in this book will be based on basic MATLAB functions

included in all installations and the two toolboxes that were mentioned previously.

Who this book is for

This guide to visual media processing using MATLAB will be very useful to a beginner

programmer who has little or no knowledge of the environment, but would like to use

it as an alternative, or possibly, substitute solution to common image and video editors.

The only thing that you will need to have before starting this book is a basic prior knowledge

of image and video processing to grasp the material covered more easily. Also, some basic

programming experience could come in handy, but is not necessary, since most parts

of the book start from scratch.

[ 3 ]

PrefaceCustomer support

Now that you are the proud owner of a Packt book, we have a number of things to help

you to get the most from your purchase.

Downloading the example code

You can download the example code files for all Packt books you have purchased from

your account at http://www.packtpub.com. If you purchased this book elsewhere,

you can visit http://www.packtpub.com/support and register to have the files

e-mailed directly to you.

Errata

Although we have taken every care to ensure the accuracy of our content, mistakes

do happen. If you find a mistake in one of our books—maybe a mistake in the text or the

code—we would be grateful if you would report this to us. By doing so, you can save other

readers from frustration and help us improve subsequent versions of this book. If you find

any errata, please report them by visiting http://www.packtpub.com/submit-errata,

selecting your book, clicking on the errata submission form link, and entering the details

of your errata. Once your errata are verified, your submission will be accepted and the errata

will be uploaded to our website, or added to any list of existing errata, under the Errata

section of that title.

Piracy

Piracy of copyright material on the Internet is an ongoing problem across all media. At Packt,

we take the protection of our copyright and licenses very seriously. If you come across any

illegal copies of our works, in any form, on the Internet, please provide us with the location

address or website name immediately so that we can pursue a remedy.

Please contact us at [email protected] with a link to the suspected

pirated material.

We appreciate your help in protecting our authors, and our ability to bring you

valuable content.

Questions

You can contact us at [email protected] if you are having a problem with

any aspect of the book, and we will do our best to address it.

[ 6 ]

PrefaceBasic Image Manipulations

1

Since it was first released, MATLAB has been associated with technical

computing and scientific programming. Due to its high uptake in academia

and its large and active community, it has grown to become a versatile and

multifunctional tool, providing solutions in a vast diversity of fields. Its usage

in image and video processing for scientific applications has been popular

for a while, but more recent versions have included processing tools that

are more user-friendly and aimed at a broader spectrum of users. The

usefulness of MATLAB for image processing is rather self-explanatory, since

it is a programming environment specialized for matrix manipulation and

images are nothing more than matrices.

In this chapter, you will be introduced to the very basics of image manipulation using

MATLAB. Prior experience in using MATLAB is not required, since we will be covering

everything from scratch. Some basic understanding of programming would be beneficial.

More specifically, in this chapter we will cover:

‹ The basic details of the MATLAB environment and especially those that will

be used extensively in this book

‹ The various ways of loading, displaying, and saving an image using MATLAB

‹ The most basic image manipulations that can be handled by MATLAB, that is,

rotation, cropping, and resizing

So, let's get started.

The available commands are all the inherent MATLAB functions, most of which help you

manipulate matrices (hence the name: MATrixLABoratory). In this book, we will mainly

be using functions for image manipulation, included in the Image Processing and Image

Acquisition toolboxes.

The Current Folder window

The Current Folder window is basically a file manager, resembling Windows Explorer. You can

navigate your way through the folders in your computer, in order to find files you would like

to use in your work, for example, an image you would like to load. By default, MATLAB can

access the contents of the current folder and a number of folders that have been included

in its path. During installation, all the folders containing the installed toolboxes have been

added to MATLAB's path; hence the functions and files contained in them can be accessed

no matter what the current folder is.

The Details window

The Details window is also informative, displaying information about the file you have

selected in the current folder window. The details are displayed only when the selected

file is recognized by MATLAB.

The Workspace window

The Workspace window is used for a constant view of all variables that are present,

providing information about their names, types, plus minimum and maximum values,

where this is applicable.

The ribbon

The ribbon contains a collection of basic MATLAB processes and resides in the top part of the

environment window. It is based on the latest Graphical User Interface trends, like the ones

used in Microsoft Office software products since the 2007 version. It has three main tabs:

‹ HOME

‹ PLOTS

‹ APPS

These three tabs are briefly described in the following sections.

[ 10 ]

Basic Image ManipulationsThese processes are core functionalities of everyday image manipulation for every amateur

photographer. They provide the foundations for any complex image processing task and will

be used throughout the book. So, congratulations! You have set the first stepping stone to

climb to more sophisticated image processing tasks. The rest of the chapters will guide you

through some more complex image processing that MATLAB offers and will then move on

to video processing. Depending on your needs, you will either be able to use it as a quick

reference for any of the techniques it covers, or you can read through the chapters in

a sequential order, as you would do in a Media Processing course.

The next chapter will introduce you to different ways to work with grayscale image pixels and

manipulate their values. On finishing it, you will be able to enhance and improve the visual

quality of an image. Have fun!

[ 31 ]

Chapter 12

Working with Pixels In Grayscale

Images

Now you have grasped some basic visual media processes that MATLAB has to

offer. You have learned how to import and export images, apply basic geometric

transformations on them, and generally perform tasks that are included in most

basic image editors. In this chapter, you will start building up your MATLAB skills

by taking advantage of ready-made functions that allow editing of pixel values

in an image. You will also start making your own small programs, save them as

scripts or functions, and apply them in practical examples.

In this chapter, we will cover:

‹ How to manipulate one or more pixels in an image using for loops, or indexing

‹ How to perform histogram-based processing using MATLAB

‹ How to write our first scripts and functions for automating more complex processes

So, let's get started!

Accessing image pixels and changing their values

To gain a better understanding of how MATLAB treats images, we have to revisit the way

it stores them in the Workspace window. In the previous chapter, we discussed the origin

of MATLAB and why it is an ideal choice for processing images. So, let's start with a simple

quiz to freshen your memory.

As we can see, both results are identical. To verify this, we could use the MATLAB's isequal

function, which compares two matrices used as input and output. It assigns 1 if they are

equal and 0 if they are not. Let's see how it works, by comparing matrix B to itself:

>> isequal(B,B)

The output of the previous code is as follows:

ans =

1

Indeed, the result was 1. Now, let's make some holes in matrix B and see if they are the

same. Suppose, we want to change the values of the elements residing in the area defined

in the previous examples to 0. Switch to Editor, erase the two lines that have to do with

printing the matrix B, after the two methods add the following lines at the end of your

previous script and then save it as MySecondScript.m:

for pos_r = 2:2+2

for pos_c = 2:2+4

B(pos_r, pos_c)= 0;

end

end

B % print the result of the loop method

clear B;% Erase matrix B from the workspace

B(1:5, 1:10) = 255;% Re-create matrix B

B(2:2+2, 2:2+4) = 0;

B % print the result of the indexing method

This time type the following code in the command line:

>> MySecondScript

[ 38 ]

Working with Pixels in Grayscale ImagesTo select the rectangles to be altered, we had to define the top and bottom row indices

and the left and right column indices. The top-left rectangle was defined in a rather

intuitive manner. We used index 1 for both the top row and the left column. The indices

for the bottom row and the right column were set to 30 and 40 respectively.

The tricky part was selecting the indices that should be used for the bottom-right rectangle.

Again, we knew the height and width, but we should use it with respect to the height and

width of the image. However, altering the width and height values for each new image

would be highly impractical. This is why we used the very convenient index keyword end,

which denoted the maximum valued index for each dimension. When it is used for rows,

it automatically takes the value of the maximum number of rows, and when it is used for

columns it takes the maximum number of columns. In our case we used it in both positions,

to calculate the proper top row index (end-39) and bottom row index (end), and also to

calculate the proper left column index (end-49) and right row index (end).

Thresholding an image

Now that you have learned two different ways to work with image pixels, we will present

another useful and common tool found in image processing software, which is thresholding.

Image thresholding can be defined as the process of creating binary images by setting pixels

with values above a certain threshold to 1 and the rest to 0. It is usually used for separating

the foreground from the background of an image. As we did for the previous examples, we

will show three different ways to implement image thresholding in MATLAB; using for loops,

a special way of indexing, and using a ready-made thresholding MATLAB function.

Image thresholding using for loops

The classic programming way to implement grayscale image thresholding is by using

two nested for loops in a similar fashion to the one used in the previous sections. More

specifically, the following script can be used to threshold my_image.bmp:

img = imread('my_image.bmp'); % Read image

subplot (1,2,1) % Open a figure for 2 images

imshow(img) % Show original image

title ('Original image') % Add title

threshold = 150; % Set threshold level

for pos_r = 1:size(img,1) % For all rows

for pos_c = 1:size(img,2) % For all columns

if img(pos_r,pos_c) > threshold % Check pixel value

img(pos_r,pos...