Image Processing


Published on

Published in: Education
No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Image Processing

  1. 1. Image Processing PDF generated using the open source mwlib toolkit. See for more information. PDF generated at: Thu, 27 May 2010 21:40:25 UTC
  2. 2. Contents Articles Introduction 1 Image 1 Pixel 2 Image processing 9 Digital image processing 11 Optical character recognition 14 Applications 18 Computer vision 18 Optical sorting 25 Augmented Reality 26 Face detection 37 Feature detection (computer vision) 39 Lane departure warning system 44 Non-photorealistic rendering 47 Medical image processing 50 Microscope image processing 57 Morphological image processing 59 Remote sensing 67 Medical imaging 74 References Article Sources and Contributors 81 Image Sources, Licenses and Contributors 83 Article Licenses License 84
  3. 3. 1 Introduction Image An image (from Latin imago) is an artifact, for example a two-dimensional picture, that has a similar appearance to some subject—usually a physical object or a person. Characteristics Images may be two-dimensional, such as a photograph, screen display, Definition of image and imagery, from Thomas and as well as a three-dimensional, such as a statue. They may be Blount's Glossographia Anglicana Nova, 1707. captured by optical devices—such as cameras, mirrors, lenses, telescopes, microscopes, etc. and natural objects and phenomena, such as the human eye or water surfaces. The word image is also used in the broader sense of any two-dimensional figure such as a map, a graph, a pie chart, or an abstract painting. In this wider sense, images can also be rendered manually, such as by drawing, painting, carving, rendered automatically by printing or computer graphics technology, or developed by a combination of methods, especially in a pseudo-photograph. A volatile image is one that exists only for a short period of time. This may be a reflection of an object by a mirror, a projection of a camera obscura, or a scene displayed on a cathode ray tube. A fixed image, also called a hard copy, is one that has been recorded on a material object, such as paper or textile by photography or digital processes. A mental image exists in an individual's mind: something one remembers or imagines. The subject of an image need not be real; it may be an abstract concept, such as a graph, function, or "imaginary" entity. For example, Sigmund Freud claimed to have dreamed purely in aural-images of dialogs. The development of synthetic acoustic technologies and the creation of sound art have led to a consideration The top image is captured using photography. of the possibilities of a sound-image made up of irreducible phonic The bottom image is rendered. Images are produced by capturing or rendering. substance beyond linguistic or musicological analysis. A still image is a single static image, as distinguished from a moving image (see below). This phrase is used in photography, visual media and the computer industry to emphasize that one is not talking about movies, or in very precise or pedantic technical writing such as a standard. A film still is a photograph taken on the set of a movie or television program during production, used for promotional purposes.
  4. 4. Image 2 Moving image A moving image is typically a movie (film), or video, including digital video. It could also be an animated display such as a zoetrope. See also • Cinematography • Mental image • Painting • Photography External links • The B-Z Reaction: The Moving or the Still Image? [1] • FACE: Friends of Active Copyright Education [2] • Library of Congress - Format Descriptions for Still Images [3] References [1] http:/ / www. apple. com/ science/ insidetheimage/ bzreaction/ [2] http:/ / www. csusa. org/ face/ home. htm [3] http:/ / www. digitalpreservation. gov/ formats/ fdd/ still_fdd. shtml Pixel In digital imaging, a pixel (or picture element[1] ) is a single point in a raster image. The pixel is the smallest addressable screen element; it is the smallest unit of picture that can be controlled. Each pixel has its own address. The address of a pixel corresponds to its coordinates. Pixels are normally arranged in a 2-dimensional grid, and are often represented using dots or squares. Each pixel is a sample of an original This example shows an image with a portion image; more samples typically provide more accurate representations greatly enlarged, in which the individual pixels of the original. The intensity of each pixel is variable. In color image are rendered as little squares and can easily be systems, a color is typically represented by three or four component seen. intensities such as red, green, and blue, or cyan, magenta, yellow, and black. In some contexts (such as descriptions of camera sensors), the term pixel is used to refer to a single scalar element of a multi-component representation (more precisely called a photosite in the camera sensor context, although the neologism sensel is also sometimes used to describe the elements of a digital camera's sensor),[2] while in others the term may refer to the entire set of such component intensities for a spatial position. In color systems that use chroma
  5. 5. Pixel 3 subsampling, the multi-component concept of a pixel can become difficult to apply, since the intensity measures for the different color components correspond to different spatial areas in a such a representation. The word pixel is based on a contraction of pix ("pictures") and el (for "element"); similar formations with el  for "element" include the words: voxel[3] and texel.[3] A photograph of sub-pixel display elements on a laptop's LCD screen Etymology The word "pixel" was first published in 1965 by Frederic C. Billingsley of JPL (in Pasadena, CA), to describe the picture elements of video images from space probes to the Moon and Mars. However, Billingsley did not coin the term himself. Instead, he got the word "pixel" from Keith E. McFarland, at the Link Division of General Precision in Palo Alto, who did not know where the word originated. McFarland said simply it was "in use at the time" (circa 1963).[4] The word is a combination of picture and element, via pix. The word pix appeared in Variety magazine headlines in 1932, as an abbreviation for the word pictures, in reference to movies.[5] By 1938, "pix" was being used in reference to still pictures by photojournalists.[4] The concept of a "picture element" dates to the earliest days of television, for example as "Bildpunkt" (the German word for pixel, literally picture point) in the 1888 German patent of Paul Nipkow. According to various etymologies, the earliest publication of the term picture element itself was in Wireless World magazine in 1927,[6] though it had been used earlier in various U.S. patents filed as early as 1911.[7] Some authors explain pixel as picture cell, as early as 1972.[8] In video processing, pel is often used instead of pixel.[9] For example, IBM used it in their Technical Reference for the original PC. Words with similar etymologies • Texel (texture element) and luxel (lux element) are words used to describe a pixel when it is used in specific context (texturing and light mapping respectively) • A voxel is a volume element, the 3D analogue of a 2D pixel. • Surfels (surface elements) have the same naming pattern as pixels, but share more similarities with shrunken triangles than expanded pixels.
  6. 6. Pixel 4 Technical A pixel is generally thought of as the smallest single component of a digital image. The definition is highly context-sensitive. For example, there can be "printed pixels" in a page, or pixels carried by electronic signals, or represented by digital values, or pixels on a display device, or pixels in a digital camera (photosensor elements). This list is not A pixel does not need to be rendered as a small exhaustive, and depending on context, there are several terms that are square. This image shows alternative ways of reconstructing an image from a set of pixel synonymous in particular contexts, such as pel, sample, byte, bit, dot, values, using dots, lines, or smooth filtering. spot, etc. The term "pixels" can be used in the abstract, or as a unit of measure, in particular when using pixels as a measure of resolution, such as: 2400 pixels per inch, 640 pixels per line, or spaced 10 pixels apart. The measures dots per inch (dpi) and pixels per inch (ppi) are sometimes used interchangeably, but have distinct meanings, especially for printer devices, where dpi is a measure of the printer's density of dot (e.g. ink droplet) placement.[10] For example, a high-quality photographic image may be printed with 600 ppi on a 1200 dpi inkjet printer.[11] Even higher dpi numbers, such as the 4800 dpi quoted by printer manufacturers since 2002, do not mean much in terms of achievable resolution.[12] The more pixels used to represent an image, the closer the result can resemble the original. The number of pixels in an image is sometimes called the resolution, though resolution has a more specific definition. Pixel counts can be expressed as a single number, as in a "three-megapixel" digital camera, which has a nominal three million pixels, or as a pair of numbers, as in a "640 by 480 display", which has 640 pixels from side to side and 480 from top to bottom (as in a VGA display), and therefore has a total number of 640 × 480 = 307,200 pixels or 0.3 megapixels. The pixels, or color samples, that form a digitized image (such as a JPEG file used on a web page) may or may not be in one-to-one correspondence with screen pixels, depending on how a computer displays an image. In computing, an image composed of pixels is known as a bitmapped image or a raster image. The word raster originates from television scanning patterns, and has been widely used to describe similar halftone printing and storage techniques. Sampling patterns For convenience, pixels are normally arranged in a regular two-dimensional grid. By using this arrangement, many common operations can be implemented by uniformly applying the same operation to each pixel independently. Other arrangements of pixels are also possible, with some sampling patterns even changing the shape (or kernel) of each pixel across the image. For this reason, care must be taken when acquiring an image on one device and displaying it on another, or when converting image data from one pixel format to another. For example: • LCD screens typically use a staggered grid, where the red, green, and blue components are sampled at slightly different locations. Subpixel rendering is a technology which takes advantage of these differences to improve the rendering of text on LCD Text rendered using Cleartype screens. • Some digital cameras use a Bayer filter, resulting in a regular grid of pixels where the color of each pixel depends on its position on the grid. • A clipmap uses a hierarchical sampling pattern, where the size of the support of each pixel depends on its location within the hierarchy. • Warped grids are used when the underlying geometry is non-planar, such as images of the earth from space.[13] • The use of non-uniform grids is an active research area, attempting to bypass the traditional Nyquist limit.[14]
  7. 7. Pixel 5 • Pixels on computer monitors are normally "square" (this is, having equal horizontal and vertical sampling pitch); pixels in other systems are often "rectangular" (that is, having unequal horizontal and vertical sampling pitch – oblong in shape), as are digital video formats with diverse aspect ratios, such as the anamorphic widescreen formats of the CCIR 601 digital video standard. Display resolution vs. native resolution in computer monitors Computers can use pixels to display an image, often an abstract image that represents a GUI. The resolution of this image is called the display resolution and is determined by the video card of the computer. LCD computer monitors also use pixels to display an image, and have a native resolution. Each pixel is made up of triads, with the number of these triads determining the native resolution. On some CRT monitors, the beam sweep rate may be fixed, resulting in a fixed native resolution. Most CRT monitors do not have a fixed beam sweep rate, meaning they don't have a native resolution at all - instead they have a set of resolutions that are equally well supported. To produce the sharpest images possible on an LCD, the user must ensure the display resolution of the computer matches the native resolution of the monitor. On a CRT with a fixed beam sweep rate, you are required to use the native resolution. On a CRT without this restriction, you may use any resolution supported by the monitor that matches the monitor's physical aspect ratio and it will look fine. If the aspect ratio of the physical display and the selected resolution are different, many different things can happen. On some LCDs, the monitor will stretch or squash the image to fill the entire display. This can result in the image appearing blurry or jagged. On others, the aspect ratio will be maintained while expanding the image to fit the display, resulting in black bars on the top or sides of the image. This can also result in the image appearing blurry or jagged, depending on the native resolution of the display and the selected resolution on the computer. For example, let's take the relatively common case of a full-screen application written assuming a 4:3 aspect ratio running on a 16:10 aspect ratio widescreen display. If the selected resolution were 1600×1200 and you were running it on a 1920×1200 display that maintains aspect ratio while expanding the image to fit, the image would not look blurry, because each pixel in the 1600×1200 image maps to exactly 1 pixel on the 1920×1200 display. If the selected resolution were 1280×960, the display would have to try to stretch the 960 pixels to fill 1200 pixels, which would mean each of the selected resolution pixels needs to take up 1.25 pixels on the physical display. Since this can't be done, the monitor uses some scheme to figure out how to distribute the colors from those 960 pixels to fill the 1200 pixels of the physical display. This mapping results in a blurry or jagged appearance. However, if the selected resolution were 800×600, you would be OK again, because 600 pixels can be expanded to 1200 pixels by having each pixel from the selected resolution take up 2 pixels on the physical display. On yet other LCD monitors, if the selected resolution is less than the native resolution of the monitor, the monitor will display things in the selected resolution, with a black border around the edges. Bits per pixel The number of distinct colors that can be represented by a pixel depends on the number of bits per pixel (bpp). A 1 bpp image uses 1-bit for each pixel, so each pixel can be either on or off. Each additional bit doubles the number of colors available, so a 2 bpp image can have 4 colors, and a 3 bpp image can have 8 colors: • 1 bpp, 21 = 2 colors (monochrome) • 2 bpp, 22 = 4 colors • 3 bpp, 23 = 8 colors ... • 8 bpp, 28 = 256 colors • 16 bpp, 216 = 65,536 colors ("Highcolor" ) • 24 bpp, 224 ≈ 16.8 million colors ("Truecolor")
  8. 8. Pixel 6 For color depths of 15 or more bits per pixel, the depth is normally the sum of the bits allocated to each of the red, green, and blue components. Highcolor, usually meaning 16 bpp, normally has five bits for red and blue, and six bits for green, as the human eye is more sensitive to errors in green than in the other two primary colors. For applications involving transparency, the 16 bits may be divided into five bits each of red, green, and blue, with one bit left for transparency. A 24-bit depth allows 8 bits per component. On some systems, 32-bit depth is available: this means that each 24-bit pixel has an extra 8 bits to describe its opacity (for purposes of combining with another image). Subpixels Many display and image-acquisition systems are, for various reasons, not capable of displaying or sensing the different color channels at the same site. Therefore, the pixel grid is divided into single-color regions that contribute to the displayed or sensed color when viewed at a distance. In some displays, such as LCD, LED, and plasma displays, these single-color regions are separately addressable elements, which have come to be known as subpixels. For example, LCDs typically divide each pixel horizontally into three subpixels. When the square pixel is divided into three subpixels, each subpixel is necessarily rectangular. In the display industry terminology, subpixels are often referred to as pixels, as they are the basic addressable elements in a viewpoint of hardware, and they call pixel circuits rather than subpixel circuits. Geometry of color elements of various CRT and LCD Most digital camera image sensors also use single-color sensor displays; phosphor dots in a color CRT display (top row) bear no relation to pixels or subpixels. regions, for example using the Bayer filter pattern, and in the camera industry these are known as pixels just like in the display industry, not subpixels. For systems with subpixels, two different approaches can be taken: • The subpixels can be ignored, with full-color pixels being treated as the smallest addressable imaging element; or • The subpixels can be included in rendering calculations, which requires more analysis and processing time, but can produce apparently superior images in some cases. This latter approach, referred to as subpixel rendering, uses knowledge of pixel geometry to manipulate the three colored subpixels separately, producing a slight increase in the apparent resolution of color displays. While CRT displays also use red-green-blue masked phosphor areas, dictated by a mesh grid called the shadow mask, it would require a difficult calibration step to be aligned with the displayed pixel raster, and so CRTs do not currently use subpixel rendering. Megapixel A megapixel (MP) is 1 million pixels, and is a term used not only for the number of pixels in an image, but also to express the number of image sensor elements of digital cameras or the number of display elements of digital displays. For example, a camera with an array of 2048×1536 sensor elements is commonly said to have "3.1 megapixels" (2048 × 1536 = 3,145,728). The megapixel count is often used as a figure of merit, though it is only one of the figures that determines camera quality. Digital cameras use photosensitive electronics, either charge-coupled device (CCD) or complementary metal–oxide–semiconductor (CMOS) image sensors, consisting of a large number of single sensor elements, each of which records a measured intensity level. In most digital cameras, the sensor array is covered with a patterned color filter mosaic having red, green, and blue regions in the Bayer filter arrangement, so that each sensor element can
  9. 9. Pixel 7 record the intensity of a single primary color of light. The camera interpolates the color information of neighboring sensor elements, through a process called demosaicing, to create the final image. These sensor elements are often called "pixels", even though they only record 1 channel (only red, or green, or blue) of the final color image. Thus, two of the three color channels for each sensor must be interpolated and a so-called N-megapixel camera that produces an N-megapixel image provides only one-third of the information that an image of the same size could get from a scanner. Thus, certain color contrasts may look fuzzier than others, depending on the allocation of the primary colors (green has twice as many elements as red or blue in the Bayer arrangement). Standard display resolutions The display resolution of a digital television or display device is the number of distinct pixels in each dimension that can be displayed. It can be an ambiguous term especially as the displayed resolution is controlled by all different factors in cathode ray tube (CRT) and flat panel or projection displays using fixed picture-element (pixel) arrays. One use of the term “display resolution” applies to fixed-pixel-array displays such as plasma display panels (PDPs), liquid crystal displays (LCDs), Digital Light Processing (DLP) projectors, or similar technologies, and is simply the physical number of columns and rows of pixels creating the display (e.g., 1920×1200). A consequence of having a fixed grid display is that, for multi-format video inputs, all displays need a "scaling engine" (a digital video processor that includes a memory array) to match the incoming picture format to the display. Note that the use of the word resolution here is misleading. The term “display resolution” is usually used to mean pixel dimensions (e.g., 1920×1200), which does not tell anything about the resolution of the display on which the image is actually formed. In digital measurement the display resolution would be given in pixels per inch. In analog measurement, if the screen is 10 inches high, then the horizontal resolution is measured across a square 10 inches wide. This is typically stated as “xxx lines horizontal resolution, per picture height.” Example: Analog NTSC and PAL TVs can typically display 480 (for NTSC) lines horizontal resolution, per picture height which is equivalent to 640 total lines from left-edge to right-edge. Selected standard display resolutions include: Display standards comparison
  10. 10. Pixel 8 Name Megapixels Width x Height CGA 0.064 320×200 EGA 0.224 640×350 VGA 0.3 640×480 SVGA 0.5 800×600 XGA 0.8 1024×768 SXGA 1.3 1280×1024 UXGA 1.9 1600×1200 WUXGA 2.3 1920×1200 See also • Computer display standard • Gigapixel image • Image resolution • Intrapixel and Interpixel processing • Pixel advertising • Pixel art • Pixel art scaling algorithms • Pixel aspect ratio • Point (typography) • Raster scan • Rasterisation • Vector graphics • Voxel External links • A Pixel Is Not A Little Square [15]: Microsoft Memo by computer graphics pioneer Alvy Ray Smith. • Pixels and Me [16]: Video of a history talk at the Computer History Museum. • Square and non-Square Pixels [17]: Technical info on pixel aspect ratios of modern video standards (480i,576i,1080i,720p), plus software implications. References [1] Rudolf F. Graf (1999). Modern Dictionary of Electronics (http:/ / books. google. com/ books?id=o2I1JWPpdusC& pg=PA569& dq=pixel+ intitle:"Modern+ Dictionary+ of+ Electronics"+ inauthor:graf& lr=& as_brr=0& ei=5ygASM3qHoSgiwH45-GIDA& sig=7tg-LuGdu6Njypaawi2bbkeq8pw). Oxford: Newnes. p. 569. ISBN 0-7506-43315. . [2] Michael Goesele (2004). New Acquisition Techniques for Real Objects and Light Sources in Computer Graphics (http:/ / books. google. com/ books?id=ZTJJ8QzNv1wC& pg=PA33& dq=sensel+ sensor-element& lr=& as_brr=0& ei=t_-4R7WMPKCQtwOV-I2SBQ& sig=nbPybOvZhWwQSt0uS0Rtq60mJlg). Books on Demand. ISBN 3833414898. . [3] Foley, James D.; Andries van Dam, John F. Hughes, Steven K. Feiner (1990). "Spatial-partitioning representations; Surface detail". Computer Graphics Principles and Practice. The Systems Programming Series. Addison-Wesley. ISBN 0-201-12110-7. "These cells are often called voxels (volume elements), in analogy to pixels." [4] Lyon, Richard F. (2006). A brief history of 'pixel' (http:/ / www. foveon. com/ files/ ABriefHistoryofPixel2. pdf). IS&T/SPIE Symposium on Electronic Imaging. [5] "Online Etymology Dictionary" (http:/ / www. etymonline. com/ index. php?search=pixel). .
  11. 11. Pixel 9 [6] "On language; Modem, I'm Odem" (http:/ / query. nytimes. com/ gst/ fullpage. html?res=990CE3DB1E31F931A35757C0A963958260), The New York Times, April 2, 1995. Accessed April 7, 2008. [7] Sinding-Larsen, Alf Transmission of Pictures of Moving Objects (http:/ / www. google. com/ patents?vid=USPAT1175313& id=Df9EAAAAEBAJ& dq="picture+ element"& as_drrb_ap=b), US Patent 1,175,313, issued March 14, 1916. [8] Robert L. Lillestrand (1972). "Techniques for Change Detection". IEEE Trans. Computers C-21 (7). [9] Lewis, Peter H. (February 12, 1989). The Executive Computer; Compaq Sharpens Its Video Option (http:/ / www. nytimes. com/ 1989/ 02/ 12/ business/ the-executive-computer-compaq-sharpens-its-video-option. html). The New York Times. [10] Derek Doeffinger (2005). The Magic of Digital Printing (http:/ / books. google. com/ books?id=s2hIx1amJUcC& pg=PA24& dq=printer+ dots-per-inch+ pixels-per-inch& ei=ilKfSKjyBYm6tQPAu9WdBQ& sig=ACfU3U1img9C-r3_16xNKoYjTubg02GyVA). Lark Books. p. 24. ISBN 1579906893. . [11] "Experiments with Pixels Per Inch (PPI) on Printed Image Sharpness" (http:/ / www. clarkvision. com/ imagedetail/ printer-ppi/ ). July 3, 2005. . [12] Harald Johnson (2002). Mastering Digital Printing (http:/ / books. google. com/ books?id=wto19gxFyfQC& pg=PA40& dq=inkjet+ printer+ 4800+ dpi+ addressability& lr=& as_brr=3& ei=ZI-kSITKAo3sswO7g-2dBQ& sig=ACfU3U2RdAVkFq_0rsmQK6QwKPcDveJcxg#PPA41,M1). Thomson Course Technology. p. 40. ISBN 1929685653. . [13] "Image registration of blurred satellite images" (http:/ / staff. utia. cas. cz/ zitova/ registration. htm). . Retrieved 2008-05-09. [14] "ScienceDirect - Pattern Recognition: Image representation by a new optimal non-uniform morphological sampling:" (http:/ / www. sciencedirect. com/ science/ article/ B6V14-3YXB1T5-8/ 2/ 15b559f2aefbc7bdc54c55df148c2374). . Retrieved 2008-05-09. [15] http:/ / www. alvyray. com/ memos/ 6_pixel. pdf [16] http:/ / video. google. com/ videoplay?docid=-5655850487750051532 [17] http:/ / lurkertech. com/ lg/ pixelaspect/ Image processing In electrical engineering and computer science, image processing is any form of signal processing for which the input is an image, such as photographs or frames of video; the output of image processing can be either an image or a set of characteristics or parameters related to the image. Most image-processing techniques involve treating the image as a two-dimensional signal and applying standard signal-processing techniques to it. Image processing usually refers to digital image processing, but optical and analog image processing are also possible. This Monochrome black/white image article is about general techniques that apply to all of them. The acquisition of images (producing the input image in the first place) is referred to as imaging.
  12. 12. Image processing 10 Typical operations Among many other image processing operations are: • Euclidean geometry transformations such as enlargement, reduction, and rotation • Color corrections such as brightness and contrast adjustments, color mapping, The red, green, and blue color channels of a photograph by Sergei Mikhailovich Prokudin-Gorskii. The fourth image is a composite. color balancing, quantization, or color translation to a different color space • Digital compositing or optical compositing (combination of two or more images). Used in film-making to make a "matte" • Interpolation, demosaicing, and recovery of a full image from a raw image format using a Bayer filter pattern • Image registration, the alignment of two or more images • Image differencing and morphing • Image recognition, for example, extract the text from the image using optical character recognition or checkbox and bubble values using optical mark recognition • Image segmentation • High dynamic range imaging by combining multiple images • Geometric hashing for 2-D object recognition with affine invariance Applications • Computer vision • Optical sorting • Augmented Reality • Face detection • Feature detection • Lane departure warning system • Non-photorealistic rendering • Medical image processing • Microscope image processing • Morphological image processing • Remote sensing See also • Imaging • Photo manipulation • List of image analysis software • Near sets • Multidimensional systems
  13. 13. Image processing 11 Further reading • Tinku Acharya and Ajoy K. Ray. Image Processing - Principles and Applications [1]. Wiley InterScience. • Wilhelm Burger and Mark J. Burge (2007). Digital Image Processing: An Algorithmic Approach Using Java [2]. Springer. ISBN 1846283795 and ISBN 3540309403. • R. Fisher, K Dawson-Howe, A. Fitzgibbon, C. Robertson, E. Trucco (2005). Dictionary of Computer Vision and Image Processing. John Wiley. ISBN 0-470-01526-8. • Bernd Jähne (2002). Digital Image Processing. Springer. ISBN 3-540-67754-2. • Tim Morris (2004). Computer Vision and Image Processing. Palgrave Macmillan. ISBN 0-333-99451-5. • Milan Sonka, Vaclav Hlavac and Roger Boyle (1999). Image Processing, Analysis, and Machine Vision. PWS Publishing. ISBN 0-534-95393-X. External links • Lectures on Image Processing [3], by Alan Peters. Vanderbilt University. Updated 28 April 2008. • EURASIP Journal on Image and Video Processing [4] — Open Access journal on Image Processing • Image processing algorithms, implementations and demonstrations [5] References [1] http:/ / books. google. co. in/ books?id=smBw4-xvfrIC& lpg=PP1& ots=FVFYaOATEF& dq=image%20processing%20ajoy%20ray& pg=PP1#v=onepage& q=& f=false [2] http:/ / www. imagingbook. com/ [3] http:/ / www. archive. org/ details/ Lectures_on_Image_Processing [4] http:/ / www. hindawi. com/ journals/ ivp/ [5] http:/ / www. ipol. im/ Digital image processing Digital image processing is the use of computer algorithms to perform image processing on digital images. As a subfield of digital signal processing, digital image processing has many advantages over analog image processing; it allows a much wider range of algorithms to be applied to the input data, and can avoid problems such as the build-up of noise and signal distortion during processing. Since images are defined over two dimensions (perhaps more) digital image processing can be modeled in the form of Multidimensional Systems. History Many of the techniques of digital image processing, or digital picture processing as it was often called, were developed in the 1960s at the Jet Propulsion Laboratory, MIT, Bell Labs, University of Maryland, and a few other places, with application to satellite imagery, wirephoto standards conversion, medical imaging, videophone, character recognition, and photo enhancement.[1] But the cost of processing was fairly high with the computing equipment of that era. In the 1970s, digital image processing proliferated, when cheaper computers and dedicated hardware became available. Images could then be processed in real time, for some dedicated problems such as television standards conversion. As general-purpose computers became faster, they started to take over the role of dedicated hardware for all but the most specialized and compute-intensive operations. With the fast computers and signal processors available in the 2000s, digital image processing has become the most common form of image processing, and is generally used because it is not only the most versatile method, but also the cheapest.
  14. 14. Digital image processing 12 Digital image processing technology for medical applications was inducted into the Space Foundation Space Technology Hall of Fame in 1994.[2] Tasks Digital image processing allows the use of much more complex algorithms for image processing, and hence can offer both more sophisticated performance at simple tasks, and the implementation of methods which would be impossible by analog means. In particular, digital image processing is the only practical technology for: • Classification • Feature extraction • Pattern recognition • Projection • Multi-scale signal analysis Some techniques which are used in digital image processing include: • Pixelization • Linear filtering • Principal components analysis • Independent component analysis • Hidden Markov models • Partial differential equations • Self-organizing maps • Neural networks • Wavelets Applications Digital camera images Digital cameras generally include dedicated digital image processing chips to convert the raw data from the image sensor into a color-corrected image in a standard image file format. Images from digital cameras often receive further processing to improve their quality, a distinct advantage digital cameras have over film cameras. The digital image processing is typically done by special software programs that can manipulate the images in many ways. Many digital cameras also enable viewing of histograms of images, as an aid for the photographer to better understand the rendered brightness range of each shot. Film Westworld (1973) was the first feature film to use digital image processing to pixellate photography to simulate an android's point of view.[3] See also • Computer graphics • Computer vision • Digitizing • Endrov • GPGPU • ImageJ
  15. 15. Digital image processing 13 • FIJI (software) • Homomorphic filtering • OpenCV • Standard test image • Super-resolution • Multidimensional systems Further reading • Wilhelm Burger and Mark J. Burge (2007). Digital Image Processing: An Algorithmic Approach Using Java [2]. Springer. ISBN 1846283795 and ISBN 3540309403. • R. Fisher, K Dawson-Howe, A. Fitzgibbon, C. Robertson, E. Trucco (2005). Dictionary of Computer Vision and Image Processing. John Wiley. ISBN 0-470-01526-8. • Bernd Jähne (2002). Digital Image Processing. Springer. ISBN 3-540-67754-2. • Tim Morris (2004). Computer Vision and Image Processing. Palgrave Macmillan. ISBN 0-333-99451-5. • Milan Sonka, Vaclav Hlavac and Roger Boyle (1999). Image Processing, Analysis, and Machine Vision. PWS Publishing. ISBN 0-534-95393-X. External links • Tutorial for image processing [4] (contains a Java applet) • Image processing algorithms, implementations and demonstrations [5] References [1] Azriel Rosenfeld, Picture Processing by Computer, New York: Academic Press, 1969 [2] "Space Technology Hall of Fame:Inducted Technologies/1994" (http:/ / www. spacetechhalloffame. org/ inductees_1994_Digital_Image_Processing. html). Space Foundation. 1994. . Retrieved 7 January 2010. [3] A Brief, Early History of Computer Graphics in Film (http:/ / www. beanblossom. in. us/ larryy/ cgi. html), Larry Yaeger, 16 Aug 2002 (last update), retrieved 24 March 2010 [4] http:/ / www. ph. tn. tudelft. nl/ Courses/ FIP/ frames/ fip. html
  16. 16. Optical character recognition 14 Optical character recognition Optical character recognition, usually abbreviated to OCR, is the mechanical or electronic translation of scanned images of handwritten, typewritten or printed text into machine-encoded text. It is widely used to convert books and documents into electronic files, to computerize a record-keeping system in an office, or to publish the text on a website. OCR makes it possible to edit the text, search for a word or phrase, store it more compactly, display or print a copy free of scanning artifacts, and apply techniques such as machine translation, text-to-speech and text mining to it. OCR is a field of research in pattern recognition, artificial intelligence and computer vision. OCR systems require calibration to read a specific font; early versions needed to be programmed with images of each character, and worked on one font at a time. "Intelligent" systems with a high degree of recognition accuracy for most fonts are now common. Some systems are capable of reproducing formatted output that closely approximates the original scanned page including images, columns and other non-textual components. History In 1929 Gustav Tauschek obtained a patent on OCR in Germany, followed by Handel who obtained a US patent on OCR in USA in 1933 (U.S. Patent 1,915,993 [1]). In 1935 Tauschek was also granted a US patent on his method (U.S. Patent 2,026,329 [2]). Tauschek's machine was a mechanical device that used templates and a photodetector. In 1950, David H. Shepard, a cryptanalyst at the Armed Forces Security Agency in the United States, addressed the problem of converting printed messages into machine language for computer processing and built a machine to do this, reported in the Washington Daily News on 27 April 1951 and in the New York Times on 26 December 1953 after his U.S. Patent 2,663,758 [3] was issued. Shepard then founded Intelligent Machines Research Corporation (IMR), which went on to deliver the world's first several OCR systems used in commercial operation. The first commercial system was installed at the Readers Digest in 1955. The second system was sold to the Standard Oil Company for reading credit card imprints for billing purposes. Other systems sold by IMR during the late 1950s included a bill stub reader to the Ohio Bell Telephone Company and a page scanner to the United States Air Force for reading and transmitting by teletype typewritten messages. IBM and others were later licensed on Shepard's OCR patents. In about 1965 Readers Digest and RCA collaborated to build an OCR Document reader designed to digitize the serial numbers on Reader Digest coupons returned from advertisements. The font used on the documents were printed by an RCA Drum printer using the OCR-A font. The reader was connected directly to an RCA 301 computer (one of the first solid state computers). This reader was followed by a specialized document reader installed at TWA where the reader processed Airline Ticket stock. The readers processed document at a rate of 1,500 documents per minute, and checked each document, rejecting those it was not able to process correctly. The product became part of the RCA product line as a reader designed to process "Turn around Documents" such as those Utility and insurance bills returned with payments. The United States Postal Service has been using OCR machines to sort mail since 1965 based on technology devised primarily by the prolific inventor Jacob Rabinow. The first use of OCR in Europe was by the British General Post Office (GPO). In 1965 it began planning an entire banking system, the National Giro, using OCR technology, a process that revolutionized bill payment systems in the UK. Canada Post has been using OCR systems since 1971. OCR systems read the name and address of the addressee at the first mechanized sorting center, and print a routing bar code on the envelope based on the postal code. To avoid confusion with the human-readable address field which can be located anywhere on the letter, special ink (orange in visible light) is used that is clearly visible under ultraviolet light. Envelopes may then be processed with equipment based on simple barcode readers. In 1974 Ray Kurzweil started the company Kurzweil Computer Products, Inc. and led development of the first omni-font optical character recognition system — a computer program capable of recognizing text printed in any
  17. 17. Optical character recognition 15 normal font. He decided that the best application of this technology would be to create a reading machine for the blind, which would allow blind people to have a computer read text to them out loud. This device required the invention of two enabling technologies — the CCD flatbed scanner and the text-to-speech synthesizer. On January 13, 1976 the successful finished product was unveiled during a widely-reported news conference headed by Kurzweil and the leaders of the National Federation of the Blind. In 1978 Kurzweil Computer Products began selling a commercial version of the optical character recognition computer program. LexisNexis was one of the first customers, and bought the program to upload paper legal and news documents onto its nascent online databases. Two years later, Kurzweil sold his company to Xerox, which had an interest in further commercializing paper-to-computer text conversion. Kurzweil Computer Products became a subsidiary of Xerox known as Scansoft, now Nuance Communications. Current state of OCR technology The accurate recognition of Latin-script, typewritten text is now considered largely a solved problem on applications where clear imaging is available such as scanning of printed documents. Typical accuracy rates on these exceed 99%; total accuracy can only be achieved by human review. Other areas—including recognition of hand printing, cursive handwriting, and printed text in other scripts (especially those with a very large number of characters)—are still the subject of active research. Accuracy rates can be measured in several ways, and how they are measured can greatly affect the reported accuracy rate. For example, if word context (basically a lexicon of words) is not used to correct software finding non-existent words, a character error rate of 1% (99% accuracy) may result in an error rate of 5% (95% accuracy) or worse if the measurement is based on whether each whole word was recognized with no incorrect letters[4] . On-line character recognition is sometimes confused with Optical Character Recognition[5] (see Handwriting recognition). OCR is an instance of off-line character recognition, where the system recognizes the fixed static shape of the character, while on-line character recognition instead recognizes the dynamic motion during handwriting. For example, on-line recognition, such as that used for gestures in the Penpoint OS or the Tablet PC can tell whether a horizontal mark was drawn right-to-left, or left-to-right. On-line character recognition is also referred to by other terms such as dynamic character recognition, real-time character recognition, and Intelligent Character Recognition or ICR. On-line systems for recognizing hand-printed text on the fly have become well-known as commercial products in recent years (see Tablet PC history). Among these are the input devices for personal digital assistants such as those running Palm OS. The Apple Newton pioneered this product. The algorithms used in these devices take advantage of the fact that the order, speed, and direction of individual lines segments at input are known. Also, the user can be retrained to use only specific letter shapes. These methods cannot be used in software that scans paper documents, so accurate recognition of hand-printed documents is still largely an open problem. Accuracy rates of 80% to 90% on neat, clean hand-printed characters can be achieved, but that accuracy rate still translates to dozens of errors per page, making the technology useful only in very limited applications. Recognition of cursive text is an active area of research, with recognition rates even lower than that of hand-printed text. Higher rates of recognition of general cursive script will likely not be possible without the use of contextual or grammatical information. For example, recognizing entire words from a dictionary is easier than trying to parse individual characters from script. Reading the Amount line of a cheque (which is always a written-out number) is an example where using a smaller dictionary can increase recognition rates greatly. Knowledge of the grammar of the language being scanned can also help determine if a word is likely to be a verb or a noun, for example, allowing greater accuracy. The shapes of individual cursive characters themselves simply do not contain enough information to accurately (greater than 98%) recognize all handwritten cursive script. It is necessary to understand that OCR technology is a basic technology also used in advanced scanning applications. Due to this, an advanced scanning solution can be unique and patented and not easily copied despite being based on
  18. 18. Optical character recognition 16 this basic OCR technology. For more complex recognition problems, intelligent character recognition systems are generally used, as artificial neural networks can be made indifferent to both affine and non-linear transformations.[6] A technique which is having considerable success in recognising difficult words and character groups within documents generally amenable to computer OCR is to submit them automatically to humans in the reCAPTCHA system. OCR software OCR Software and ICR Software technology are analytical artificial intelligence systems that consider sequences of characters rather than whole words or phrases. Based on the analysis of sequential lines and curves, OCR and ICR make 'best guesses' at characters using database look-up tables to closely associate or match the strings of characters that form words. See also • Automatic number plate recognition • CAPTCHA • Computational linguistics • Computer vision • Digital Mailroom • Machine learning • Music OCR • OCR SDK • OCR Software • Optical mark recognition • Raster to vector • Raymond Kurzweil • Speech recognition • Book scanning • Institutional Repository • Digital Library • OCR-B External links • 17 Things [7] Explanation of basic handwriting recognition principles and history • Unicode OCR - Hex Range: 2440-245F [8] Optical Character Recognition in Unicode • Free Online OCR [9] Free Online OCR References [1] http:/ / www. google. com/ patents?q=1,915,993 [2] http:/ / www. google. com/ patents?q=2,026,329 [3] http:/ / www. google. com/ patents?q=2,663,758 [4] Suen, C.Y., et al (1987-05-29), Future Challenges in Handwriting and Computer Applications (http:/ / users. erols. com/ rwservices/ pens/ biblio88. html#Suen88), 3rd International Symposium on Handwriting and Computer Applications, Montreal, May 29, 1987, , retrieved 2008-10-03 [5] Tappert, Charles C., et al (1990-08), The State of the Art in On-line Handwriting Recognition (http:/ / users. erols. com/ rwservices/ pens/ biblio90. html#Tappert90c), IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol 12 No 8, August 1990, pp 787-ff, , retrieved 2008-10-03
  19. 19. Optical character recognition 17 [6] LeNet-5, Convolutional Neural Networks (http:/ / yann. lecun. com/ exdb/ lenet/ ) [7] http:/ / technology. 17things. com/ what-is-handwriting-recognition. html [8] http:/ / www. unicode. org/ charts/ PDF/ U2440. pdf [9] http:/ / www. free-online-ocr. com
  20. 20. 18 Applications Computer vision Computer vision is the science and technology of machines that see. As a scientific discipline, computer vision is concerned with the theory behind artificial systems that extract information from images. The image data can take many forms, such as video sequences, views from multiple cameras, or multi-dimensional data from a medical scanner. As a technological discipline, computer vision seeks to apply its theories and models to the construction of computer vision systems. Examples of applications of computer vision include systems for: • Controlling processes (e.g., an industrial robot or an autonomous vehicle). • Detecting events (e.g., for visual surveillance or people counting). • Organizing information (e.g., for indexing databases of images and image sequences). • Modeling objects or environments (e.g., industrial inspection, medical image analysis or topographical modeling). • Interaction (e.g., as the input to a device for computer-human interaction). Computer vision is closely related to the study of biological vision. The field of biological vision studies and models the physiological processes behind visual perception in humans and other animals. Computer vision, on the other hand, studies and describes the processes implemented in software and hardware behind artificial vision systems. Interdisciplinary exchange between biological and computer vision has proven fruitful for both fields. Computer vision is, in some ways, the inverse of computer graphics. While computer graphics produces image data from 3D models, computer vision often produces 3D models from image data. There is also a trend towards a combination of the two disciplines, e.g., as explored in augmented reality. Sub-domains of computer vision include scene reconstruction, event detection, video tracking, object recognition, learning, indexing, motion estimation, and image restoration. State of the art Computer vision is a diverse and relatively new field of study. In the early days of computing, it was difficult to process even moderately large sets of image data. It was not until the late 1970s that a more focused study of the field emerged. Computer vision covers a wide range of topics which are often related to other disciplines, and consequently there is no standard formulation of "the computer vision problem". Moreover, there is no standard formulation of how computer vision problems should be solved. Instead, there exists an abundance of methods for solving various well-defined computer vision tasks, where the methods often are very task specific and seldom can be generalized over a wide range of applications. Many of the methods and applications are still in the state of basic research, but more and more methods have found their way into commercial products, where they often constitute a part of a larger system which can solve complex tasks (e.g., in the area of medical images, or quality control and measurements in industrial processes). In most practical computer vision applications, the computers are pre-programmed to solve a particular task, but methods based on learning are now becoming increasingly common.
  21. 21. Computer vision 19 Related fields Much of artificial intelligence deals with autonomous planning or deliberation for robotical systems to navigate through an environment. A detailed understanding of these environments is required to navigate through them. Information about the environment could be provided by a computer vision system, acting as a vision sensor and providing high-level information about the environment and the robot. Artificial intelligence and computer vision share other topics such as pattern recognition and learning techniques. Consequently, computer vision is sometimes seen as a part of the artificial intelligence field or the computer science field in general. Physics is another field that is closely related to computer vision. Computer vision systems rely on image sensors which detect electromagnetic radiation which is typically in the form of either visible or infra-red light. The sensors are designed using solid-state physics. The process by which light propagates and reflects off surfaces is explained using optics. Sophisticated image sensors even require quantum mechanics to provide a complete understanding of the image formation process. Also, various measurement problems in physics can be addressed using computer vision, for example motion in fluids. A third field which plays an important role is neurobiology, specifically the study of the biological vision system. Over the last century, there has been an extensive study of eyes, neurons, and the brain structures devoted to processing of visual stimuli in both humans and various animals. This has led to a coarse, yet complicated, description of how "real" vision systems operate in order to solve certain vision related tasks. These results have led to a subfield within computer vision where artificial systems are designed to mimic the processing and behavior of biological systems, at different levels of complexity. Also, some of the learning-based methods developed within computer vision have their background in biology. Yet another field related to computer vision is signal processing. Many methods for processing of one-variable signals, typically temporal signals, can be extended in a natural way to processing of two-variable signals or multi-variable signals in computer vision. However, because of the specific nature of images there are many methods developed within computer vision which have no counterpart in the processing of one-variable signals. A distinct character of these methods is the fact that they are non-linear which, together with the multi-dimensionality of the signal, defines a subfield in signal processing as a part of computer vision. Beside the above mentioned views on computer vision, many of the related research topics can also be studied from a purely mathematical point of view. For example, many methods in computer vision are based on statistics, optimization or geometry. Finally, a significant part of the field is devoted to the implementation aspect of computer vision; how existing methods can be realized in various combinations of software and hardware, or how these methods can be modified in order to gain processing speed without losing too much performance. The fields most closely related to computer vision are image processing, image analysis and machine vision. There is a significant overlap in the range of techniques and applications that these cover. This implies that the basic techniques that are used and developed in these fields are more or less identical, something which can be interpreted as there is only one field with different names. On the other hand, it appears to be necessary for research groups, scientific journals, conferences and companies to present or market themselves as belonging specifically to one of these fields and, hence, various characterizations which distinguish each of the fields from the others have been presented. The following characterizations appear relevant but should not be taken as universally accepted: • Image processing and image analysis tend to focus on 2D images, how to transform one image to another, e.g., by pixel-wise operations such as contrast enhancement, local operations such as edge extraction or noise removal, or geometrical transformations such as rotating the image. This characterization implies that image processing/analysis neither require assumptions nor produce interpretations about the image content. • Computer vision tends to focus on the 3D scene projected onto one or several images, e.g., how to reconstruct structure or other information about the 3D scene from one or several images. Computer vision often relies on
  22. 22. Computer vision 20 more or less complex assumptions about the scene depicted in an image. • Machine vision tends to focus on applications, mainly in manufacturing, e.g., vision based autonomous robots and systems for vision based inspection or measurement. This implies that image sensor technologies and control theory often are integrated with the processing of image data to control a robot and that real-time processing is emphasized by means of efficient implementations in hardware and software. It also implies that the external conditions such as lighting can be and are often more controlled in machine vision than they are in general computer vision, which can enable the use of different algorithms. • There is also a field called imaging which primarily focus on the process of producing images, but sometimes also deals with processing and analysis of images. For example, medical imaging contains lots of work on the analysis of image data in medical applications. • Finally, pattern recognition is a field which uses various methods to extract information from signals in general, mainly based on statistical approaches. A significant part of this field is devoted to applying these methods to image data. Applications for computer vision One of the most prominent application fields is medical computer vision or medical image processing. This area is characterized by the extraction of information from image data for the purpose of making a medical diagnosis of a patient. Generally, image data is in the form of microscopy images, X-ray images, angiography images, ultrasonic images, and tomography images. An example of information which can be extracted from such image data is detection of tumours, arteriosclerosis or other malign changes. It can also be measurements of organ dimensions, blood flow, etc. This application area also supports medical research by providing new information, e.g., about the structure of the brain, or about the quality of medical treatments. A second application area in computer vision is in industry, sometimes called machine vision, where information is extracted for the purpose of supporting a manufacturing process. One example is quality control where details or final products are being automatically inspected in order to find defects. Another example is measurement of position and orientation of details to be picked up by a robot arm. Machine vision is also heavily used in agricultural process to remove undesirable food stuff from bulk material, a process called optical sorting. Military applications are probably one of the largest areas for computer vision. The obvious examples are detection of enemy soldiers or vehicles and missile guidance. More advanced systems for missile guidance send the missile to an area rather than a specific target, and target selection is made when the missile reaches the area based on locally acquired image data. Modern military concepts, such as "battlefield awareness", imply that various sensors, including image sensors, provide a rich set of information about a combat scene which can be used to support strategic decisions. In this case, automatic processing of the data is used to reduce complexity and to fuse information from multiple sensors to increase reliability.
  23. 23. Computer vision 21 One of the newer application areas is autonomous vehicles, which include submersibles, land-based vehicles (small robots with wheels, cars or trucks), aerial vehicles, and unmanned aerial vehicles (UAV). The level of autonomy ranges from fully autonomous (unmanned) vehicles to vehicles where computer vision based systems support a driver or a pilot in various situations. Fully autonomous vehicles typically use computer vision for navigation, i.e. for knowing where it is, or for producing a map of its environment (SLAM) and for detecting obstacles. It can also be used for detecting certain task specific events, e. g., a UAV looking for forest fires. Examples of supporting systems are Artist's Concept of Rover on Mars, an example of an obstacle warning systems in cars, and systems for autonomous unmanned land-based vehicle. Notice the stereo cameras mounted on top of the Rover. (credit: Maas landing of aircraft. Several car manufacturers have demonstrated Digital LLC) systems for autonomous driving of cars, but this technology has still not reached a level where it can be put on the market. There are ample examples of military autonomous vehicles ranging from advanced missiles, to UAVs for recon missions or missile guidance. Space exploration is already being made with autonomous vehicles using computer vision, e. g., NASA's Mars Exploration Rover and ESA's ExoMars Rover. Other application areas include: • Support of visual effects creation for cinema and broadcast, e.g., camera tracking (matchmoving). • Surveillance. Typical tasks of computer vision Each of the application areas described above employ a range of computer vision tasks; more or less well-defined measurement problems or processing problems, which can be solved using a variety of methods. Some examples of typical computer vision tasks are presented below. Recognition The classical problem in computer vision, image processing, and machine vision is that of determining whether or not the image data contains some specific object, feature, or activity. This task can normally be solved robustly and without effort by a human, but is still not satisfactorily solved in computer vision for the general case: arbitrary objects in arbitrary situations. The existing methods for dealing with this problem can at best solve it only for specific objects, such as simple geometric objects (e.g., polyhedra), human faces, printed or hand-written characters, or vehicles, and in specific situations, typically described in terms of well-defined illumination, background, and pose of the object relative to the camera. Different varieties of the recognition problem are described in the literature: • Object recognition: one or several pre-specified or learned objects or object classes can be recognized, usually together with their 2D positions in the image or 3D poses in the scene. • Identification: An individual instance of an object is recognized. Examples: identification of a specific person's face or fingerprint, or identification of a specific vehicle. • Detection: the image data is scanned for a specific condition. Examples: detection of possible abnormal cells or tissues in medical images or detection of a vehicle in an automatic road toll system. Detection based on relatively simple and fast computations is sometimes used for finding smaller regions of interesting image data which can be further analyzed by more computationally demanding techniques to produce a correct interpretation. Several specialized tasks based on recognition exist, such as:
  24. 24. Computer vision 22 • Content-based image retrieval: finding all images in a larger set of images which have a specific content. The content can be specified in different ways, for example in terms of similarity relative a target image (give me all images similar to image X), or in terms of high-level search criteria given as text input (give me all images which contains many houses, are taken during winter, and have no cars in them). • Pose estimation: estimating the position or orientation of a specific object relative to the camera. An example application for this technique would be assisting a robot arm in retrieving objects from a conveyor belt in an assembly line situation. • Optical character recognition (OCR): identifying characters in images of printed or handwritten text, usually with a view to encoding the text in a format more amenable to editing or indexing (e.g. ASCII). Motion analysis Several tasks relate to motion estimation where an image sequence is processed to produce an estimate of the velocity either at each points in the image or in the 3D scene, or even of the camera that produces the images . Examples of such tasks are: • Egomotion: determining the 3D rigid motion (rotation and translation) of the camera from an image sequence produced by the camera. • Tracking: following the movements of a (usually) smaller set of interest points or objects (e.g., vehicles or humans) in the image sequence. • Optical flow: to determine, for each point in the image, how that point is moving relative to the image plane, i.e., its apparent motion. This motion is a result both of how the corresponding 3D point is moving in the scene and how the camera is moving relative to the scene. Scene reconstruction Given one or (typically) more images of a scene, or a video, scene reconstruction aims at computing a 3D model of the scene. In the simplest case the model can be a set of 3D points. More sophisticated methods produce a complete 3D surface model. Image restoration The aim of image restoration is the removal of noise (sensor noise, motion blur, etc.) from images. The simplest possible approach for noise removal is various types of filters such as low-pass filters or median filters. More sophisticated methods assume a model of how the local image structures look like, a model which distinguishes them from the noise. By first analysing the image data in terms of the local image structures, such as lines or edges, and then controlling the filtering based on local information from the analysis step, a better level of noise removal is usually obtained compared to the simpler approaches. Computer vision systems The organization of a computer vision system is highly application dependent. Some systems are stand-alone applications which solve a specific measurement or detection problem, while others constitute a sub-system of a larger design which, for example, also contains sub-systems for control of mechanical actuators, planning, information databases, man-machine interfaces, etc. The specific implementation of a computer vision system also depends on if its functionality is pre-specified or if some part of it can be learned or modified during operation. There are, however, typical functions which are found in many computer vision systems. • Image acquisition: A digital image is produced by one or several image sensors, which, besides various types of light-sensitive cameras, include range sensors, tomography devices, radar, ultra-sonic cameras, etc. Depending on the type of sensor, the resulting image data is an ordinary 2D image, a 3D volume, or an image sequence. The pixel values typically correspond to light intensity in one or several spectral bands (gray images or colour
  25. 25. Computer vision 23 images), but can also be related to various physical measures, such as depth, absorption or reflectance of sonic or electromagnetic waves, or nuclear magnetic resonance. • Pre-processing: Before a computer vision method can be applied to image data in order to extract some specific piece of information, it is usually necessary to process the data in order to assure that it satisfies certain assumptions implied by the method. Examples are • Re-sampling in order to assure that the image coordinate system is correct. • Noise reduction in order to assure that sensor noise does not introduce false information. • Contrast enhancement to assure that relevant information can be detected. • Scale-space representation to enhance image structures at locally appropriate scales. • Feature extraction: Image features at various levels of complexity are extracted from the image data. Typical examples of such features are • Lines, edges and ridges. • Localized interest points such as corners, blobs or points. More complex features may be related to texture, shape or motion. • Detection/segmentation: At some point in the processing a decision is made about which image points or regions of the image are relevant for further processing. Examples are • Selection of a specific set of interest points • Segmentation of one or multiple image regions which contain a specific object of interest. • High-level processing: At this step the input is typically a small set of data, for example a set of points or an image region which is assumed to contain a specific object. The remaining processing deals with, for example: • Verification that the data satisfy model-based and application specific assumptions. • Estimation of application specific parameters, such as object pose or object size. • Classifying a detected object into different categories. See also • Active vision • Artificial intelligence, Strong AI • Digital image processing • Image processing • List of computer vision topics • Machine learning • Machine vision • Machine vision glossary • Medical imaging • Pattern recognition • Photogrammetry
  26. 26. Computer vision 24 Further reading • Dana H. Ballard and Christopher M. Brown (1982). Computer Vision [1]. Prentice Hall. ISBN 0131653164. • David Marr (1982). Vision. W. H. Freeman and Company. ISBN 0-7167-1284-9. • Azriel Rosenfeld and Avinash Kak (1982). Digital Picture Processing. Academic Press. ISBN 0-12-597301-2. • Berthold Klaus Paul Horn (1986). Robot Vision. MIT Press. ISBN 0-262-08159-8. • Olivier Faugeras (1993). Three-Dimensional Computer Vision, A Geometric Viewpoint. MIT Press. ISBN 0-262-06158-9. • Tony Lindeberg (1994). Scale-Space Theory in Computer Vision [2]. Springer. ISBN 0-7923-9418-6. • James L. Crowley and Henrik I. Christensen (Eds.) (1995). Vision as Process. Springer-Verlag. ISBN 3-540-58143-X and ISBN 0-387-58143-X. • Gösta H. Granlund and Hans Knutsson (1995). Signal Processing for Computer Vision. Kluwer Academic Publisher. ISBN 0-7923-9530-1. • Reinhard Klette, Karsten Schluens and Andreas Koschan (1998). Computer Vision - Three-Dimensional Data from Images [3]. Springer, Singapore. ISBN 981-3083-71-9. • Emanuele Trucco and Alessandro Verri (1998). Introductory Techniques for 3-D Computer Vision. Prentice Hall. ISBN 0132611082. • Milan Sonka, Vaclav Hlavac and Roger Boyle (1999). Image Processing, Analysis, and Machine Vision. PWS Publishing. ISBN 0-534-95393-X. • Bernd Jähne and Horst Haußecker (2000). Computer Vision and Applications, A Guide for Students and Practitioners. Academic Press. ISBN 0-13-085198-1. • Linda G. Shapiro and George C. Stockman (2001). Computer Vision. Prentice Hall. ISBN 0-13-030796-3. • Bernd Jähne (2002). Digital Image Processing. Springer. ISBN 3-540-67754-2. • David A. Forsyth and Jean Ponce (2003). Computer Vision, A Modern Approach. Prentice Hall. ISBN 0-12-379777-2. • Richard Hartley and Andrew Zisserman (2003). Multiple View Geometry in Computer Vision. Cambridge University Press. ISBN 0-521-54051-8. • Gérard Medioni and Sing Bing Kang (2004). Emerging Topics in Computer Vision. Prentice Hall. ISBN 0-13-101366-1. • Tim Morris (2004). Computer Vision and Image Processing. Palgrave Macmillan. ISBN 0-333-99451-5. • E. Roy Davies (2005). Machine Vision : Theory, Algorithms, Practicalities. Morgan Kaufmann. ISBN 0-12-206093-8. • R. Fisher, K Dawson-Howe, A. Fitzgibbon, C. Robertson, E. Trucco (2005). Dictionary of Computer Vision and Image Processing. John Wiley. ISBN 0-470-01526-8. • Nikos Paragios and Yunmei Chen and Olivier Faugeras (2005). Handbook of Mathematical Models in Computer Vision [4]. Springer. ISBN 0-387-26371-3. • Wilhelm Burger and Mark J. Burge (2007). Digital Image Processing: An Algorithmic Approach Using Java [2]. Springer. ISBN 1846283795 and ISBN 3540309403. • Pedram Azad, Tilo Gockel, Rüdiger Dillmann (2008). Computer Vision - Principles and Practice [5]. Elektor International Media BV. ISBN 0905705718.
  27. 27. Computer vision 25 External links General resources • Keith Price's Annotated Computer Vision Bibliography [6] and the Official Mirror Site Keith Price's Annotated Computer Vision Bibliography [7] • USC Iris computer vision conference list [8] • Computer Vision Online [9] • Computer Vision Central [10] • The Computer Vision Genealogy Project [11] References [1] http:/ / homepages. inf. ed. ac. uk/ rbf/ BOOKS/ BANDB/ bandb. htm [2] http:/ / www. nada. kth. se/ ~tony/ book. html [3] http:/ / www. cs. auckland. ac. nz/ ~rklette/ Books/ SpringerCV98/ Springer98. html [4] http:/ / www. mas. ecp. fr/ vision/ Personnel/ nikos/ paragios-chen-faugeras/ [5] http:/ / ivt. sourceforge. net/ book. html [6] http:/ / iris. usc. edu/ Vision-Notes/ bibliography/ contents. html [7] http:/ / www. visionbib. com/ bibliography/ contents. html [8] http:/ / iris. usc. edu/ Information/ Iris-Conferences. html [9] http:/ / www. ComputerVisionOnline. com [10] http:/ / www. ComputerVisionCentral. com [11] http:/ / visiongenealogy. summerhost. info/ Optical sorting Optical Sorting is a process of visually sorting a product though the use of Photodetector (light sensors), Camera[1] , or the standard Mark 1 Human eye ball[2] . In its simplest operation, a machine will simply see how much light is reflected off the object using a simple Photodetector (such as a Photoresistor) and accept or reject the item depending of how reflective it is (light or dark). More sophisticated systems use Image processing to discriminate the colors of the object, often via a controlled spectra of light, even beyond the visible spectrum into the IR and UV range. Shape detection is an evolving ability. The common method of removal is jets of compressed air, but other exist. The term "Optical sorting" also includes manual seeing and manipulating process. Satake Corporation, Best [3], and Sortex [4] are three of several companies that build Optical Sorting machines. References [1] http:/ / www. woodheadpublishing. com/ en/ book. aspx?bookID=614& printer=yes Detecting foreign bodies in food [2] "Image Processing for the Food Industry", E. R. Davis; World Scientific Publishing Co. Pte. Ltd.; London, 2000 [3] http:/ / www. bestnv. com/ [4] http:/ / www. buhlergroup. com/ 17217EN. asp
  28. 28. Augmented Reality 26 Augmented Reality Augmented reality (AR) is a term for a live direct or indirect view of a physical real-world environment whose elements are augmented by virtual computer-generated imagery. It is related to a more general concept called mediated reality in which a view of reality is modified (possibly even diminished rather than augmented) by a computer. As a result, the technology functions by enhancing one’s current perception of reality. Augmented GeoTravel for the iPhone 3GS uses In the case of Augmented Reality, the augmentation is conventionally the compass, the GPS and the accelerometers to in real-time and in semantic context with environmental elements, such show the Augmented Reality view as sports scores on TV during a match. With the help of advanced AR technology (e.g. adding computer vision and object recognition) the information about the surrounding real world of the user becomes interactive and digitally usable. Artificial information about the environment and the objects in it can be stored and retrieved as an information layer on top of the real world view. The term augmented reality is believed to have been coined in 1990 by Thomas Caudell, an employee of Boeing at the time[1] . Augmented reality research explores the application of Wikitude World Browser on the iPhone 3GS uses computer-generated imagery in live-video streams as a way to expand GPS and solid state compass the real-world. Advanced research includes use of head-mounted displays and virtual retinal displays for visualization purposes, and construction of controlled environments containing any number of sensors and actuators. Definition There are two commonly accepted definitions of Augmented Reality today. One was given by Ronald Azuma in 1997 [2] . Azuma's definition says that Augmented Reality AR Tower Defense game on the Nokia N95 • combines real and virtual smartphone (Symbian OS) uses fiduciary markers • is interactive in real time • is registered in 3D Additionally Paul Milgram and Fumio Kishino defined Milgram's Reality-Virtuality Continuum in 1994 [3] . They describe a continuum that spans from the real environment to a pure virtual environment. In between there are Augmented Reality (closer to the real environment) and Augmented Virtuality (is closer to the virtual environment). This continuum has been extended into a two-dimensional plane of "Virtuality" and "Mediality"[4] . Taxonomy of Reality, Virtuality, Mediality. The origin R denotes unmodified reality. A continuum across the Virtuality axis V includes reality augmented with Milgram's Continuum graphics (Augmented Reality), as well as graphics augmented by reality (Augmented Virtuality). However, the taxonomy also
  29. 29. Augmented Reality 27 includes modification of reality or virtuality or any combination of these. The modification is denoted by moving up the mediality axis. Further up this axis, for example, we can find mediated reality, mediated virtuality, or any combination of these. Further up and to the right we have virtual worlds that are responsive to a severely modified version of reality. (at right) Mediated reality generalizes the concepts of mixed reality, etc.. It includes the virtuality reality continuum (mixing) but also, in addition to additive effects, also includes multiplicatave effects (modulation) of (sometimes deliberately) diminished reality. Moreover, it considers, more generally, that reality may be modified in various ways. The mediated reality framework describes devices that deliberately modify reality, as well as devices that accidentally Mediated Reality continuum showing four points: modify it. Augmented Reality, Augmented Virtuality, Mediated Reality, and Mediated Virtuality on the Virtuality and More recently, the term augmented reality has been blurred a bit Mediality axes due to the increased interest of the general public in AR. Examples Commonly known examples of AR are the yellow "first down" lines seen in television broadcasts of American football games using the 1st & Ten system, and the colored trail showing location and direction of the puck in TV broadcasts of ice hockey games. The real-world elements are the football field and players, and the virtual element is the yellow line, which is drawn over the image by computers in real time. Similarly, rugby fields and cricket pitches are branded by their sponsors using Augmented Reality; giant logos are inserted onto the fields when viewed on television. In some cases, the modification of reality goes beyond mere augmentation. For example, advertisements may be blocked out (partially or wholly diminished) and replaced with different advertisements. Such replacement is an example of Mediated reality, a more general concept than AR. Television telecasts of swimming events also often have a virtual line which indicates the position of the current world record holder at that time. Another type of AR application uses projectors and screens to insert objects into the real environment, enhancing museum exhibitions for example. The difference to a simple TV screen for example, is that these objects are related to the environment of the screen or display, and that they often are interactive as well. Many first-person shooter video games simulate the viewpoint of someone using AR systems. In these games the AR can be used to give visual directions to a location, mark the direction and distance of another person who is not in line of sight, give information about equipment such as remaining bullets in a gun, and display a myriad of other images based on whatever the game designers intend. This is also called the head-up display. In some current applications like in cars or airplanes, this is usually a head-up display integrated into the windshield. The F-35 Lightning II has no Head-up display. This is because all targets are tracked by the aircraft's situational awareness and the sensor fusion is presented in the pilot's helmet mounted display. The helmet mounted display provides an augmented reality system that allows the pilot to look through his own aircraft as if it wasn't there.
  30. 30. Augmented Reality 28 History • 1957-62: Morton Heilig, a cinematographer, creates and patents a simulator called Sensorama with visuals, sound, vibration, and smell.[5] • 1966: Ivan Sutherland invents the head-mounted display suggesting it was a window into a virtual world. • 1975: Myron Krueger creates Videoplace that allows users to interact with virtual objects for the first time. • 1989: Jaron Lanier coins the phrase Virtual Reality and creates the first commercial business around virtual worlds. • 1992: Tom Caudell [6] coins the phrase Augmented Reality while at Boeing helping workers assemble cables into aircraft. • 1992: L.B. Rosenberg develops one of the first functioning AR systems, called VIRTUAL FIXTURES, at the U.S. Air Force Armstrong Labs, and demonstrates benefit on human performance.[7] [8] • 1992: Steven Feiner, Blair MacIntyre and Doree Seligmann present first major paper on an AR system prototype, KARMA, at the Graphics Interface conference. Widely cited version of the paper is published in Communications of the ACM next year. • 1999: Hirokazu Kato (加藤 博一) develops ARToolKit at the HITLab and it is demonstrated at SIGGRAPH that year. • 2000: Bruce H. Thomas develops ARQuake, the first outdoor mobile AR game, and is demonstrated in the International Symposium on Wearable Computers. • 2008: Wikitude AR Travel Guide [9] launches on Oct. 20, 2008 with the G1 Android phone. • 2009: Wikitude Drive - AR Navigation System launches on Oct. 28, 2009 for the Android platform. • 2009: AR Toolkit is ported to Adobe Flash (FLARToolkit) by Saqoosha [10], bringing augmented reality to the web browser. • 2010: acrossair [11], AR Browser bringing augmented reality to the mobile user on the iPhone 3Gs for hyper-local search. Technology Hardware The main hardware components for augmented reality are: display, tracking, input devices, and computer. Combination of powerful CPU, camera, accelerometers, GPS and solid state compass are often present in modern smartphones, which make them prospective platforms for augmented reality. Display There are three major display techniques for Augmented Reality: I. Head Mounted Displays II. Handheld Displays III. Spatial Displays Head Mounted Displays A Head Mounted Display (HMD) places images of both the physical world and registered virtual graphical objects over the user's view of the world. The HMD's are either optical see-through or video see-through in nature. An optical see-through display employs half-silver mirror technology to allow views of physical world to pass through the lens and graphical overlay information to be reflected into the user's eyes. The HMD must be tracked with a six degree of freedom sensor.This tracking allows for the computing system to register the virtual information to the physical world. The main advantage of HMD AR is the immersive experience for the user. The graphical information is slaved to the view of the user. The most common products employed are as follows: MicroVision
  31. 31. Augmented Reality 29 Nomad, Sony Glasstron, and I/O Displays. Handheld Displays Handheld Augment Reality employs a small computing device with a display that fits in a user's hand. All handheld AR solutions to date have employed video see-through techniques to overlay the graphical information to the physical world. Initially handheld AR employed sensors such as digital compasses and GPS units for its six degree of freedom tracking sensors. This moved onto the use of fiducial marker systems such as the ARToolKit for tracking. Today vision systems such as SLAM or PTAM are being employed for tracking. Handheld display AR promises to be the first commercial success for AR technologies. The two main advantages of handheld AR is the portable nature of handheld devices and ubiquitous nature of camera phones. Spatial Displays Instead of the user wearing or carrying the display such as with head mounted displays or handheld devices; Spatial Augmented Reality (SAR) makes use of digital projectors to display graphical information onto physical objects. The key difference in SAR is that the display is separated from the users of the system. Because the displays are not associated with each user, SAR scales naturally up to groups of users, thus allowing for collocated collaboration between users. SAR has several advantages over traditional head mounted displays and handheld devices. The user is not required to carry equipment or wear the display over their eyes. This makes spatial AR a good candidate for collaborative work, as the users can see each other’s faces. A system can be used by multiple people at the same time without each having to wear a head mounted display. Spatial AR does not suffer from the limited display resolution of current head mounted displays and portable devices. A projector based display system can simply incorporate more projectors to expand the display area. Where portable devices have a small window into the world for drawing, a SAR system can display on any number of surfaces of an indoor setting at once. The tangible nature of SAR makes this an ideal technology to support design, as SAR supports both a graphical visualisation and passive haptic sensation for the end users. People are able to touch physical objects, and it is this process that provides the passive haptic sensation. [2] [12] [13] [14] Tracking Modern mobile augmented reality systems use one or more of the following tracking technologies: digital cameras and/or other optical sensors, accelerometers, GPS, gyroscopes, solid state compasses, RFID, wireless sensors. Each of these technologies have different levels of accuracy and precision. Most important is the tracking of the pose and position of the user's head for the augmentation of the user's view. The user's hand(s) can tracked or a handheld input device could be tracked to provide a 6DOF interaction technique. Stationary systems can employ 6DOF track systems such as Polhemus, ViCON, A.R.T, or Ascension. Input devices This is a current open research question. Some systems, such as the Tinmith system, employ pinch glove techniques. Another common technique is a wand with a button on it. In case of smartphone, phone itself could be used as 3D pointing device, with 3D position of the phone restored from the camera images. Computer Camera based systems require powerful CPU and considerable amount of RAM for processing camera images. Wearable computing systems employ a laptop in a backpack configuration. For stationary systems a traditional workstation with a powerful graphics card. Sound processing hardware could be included in augmented reality systems.
  32. 32. Augmented Reality 30 Software For consistent merging real-world images from camera and virtual 3D images, virtual images should be attached to real-world locations in visually realistic way. That means a real world coordinate system, independent from the camera, should be restored from camera images. That process is called Image registration and is part of Azuma's definition of Augmented Reality. Augmented reality image registration uses different methods of computer vision, mostly related to video tracking. Many computer vision methods of augmented reality are inherited form similar visual odometry methods. Usually those methods consist of two parts. First interest points, or fiduciary markers, or optical flow detected in the camera images. First stage can use Feature detection methods like Corner detection, Blob detection, Edge detection or thresholding and/or other image processing methods. In the second stage, a real world coordinate system is restored from the data obtained in the first stage. Some methods assume objects with known 3D geometry(or fiduciary markers) present in the scene and make use of those data. In some of those cases all of the scene 3D structure should be precalculated beforehand. If not all of the scene is known beforehand SLAM technique could be used for mapping fiduciary markers/3D models relative positions. If no assumption about 3D geometry of the scene made structure from motion methods are used. Methods used in the second stage include projective(epipolar) geometry, bundle adjustment, rotation representation with exponential map, kalman and particle filters. Applications Current applications Advertising: Marketers started to use AR to promote products via interactive AR applications. For example, at the 2008 LA Auto Show, Nissan unveiled the concept vehicle Cube and presented visitors with a brochure which, when held against a webcam, showed several versions of the vehicle[15] . In August 2009, Best Buy ran a circular with an augmented reality code that allowed users with a webcam to interact with the product in 3D.[16] Support with complex tasks: Complex tasks such as assembly, maintenance, and surgery can be simplified by inserting additional information into the field of view. For example, labels can be displayed on parts of a system to clarify operating instructions for a mechanic who is performing maintenance on the system. [17] .AR can include images of hidden objects, which can be particularly effective for medical diagnostics or surgery. Examples include a virtual X-ray view based on prior tomography or on real time images from ultrasound or open NMR devices. A doctor could observe the fetus inside the mother's womb [18] .See also Mixed reality. Navigation devices: AR can augment the effectiveness of navigation devices for a variety of applications. For example, building navigation can be enhanced for the purpose of maintaining industrial plants. Outdoor navigation can be augmented for military operations or disaster management. Head-up displays or personal display glasses in automobiles can be used to provide navigation hints and traffic information. These types of displays can be useful for airplane pilots, too. Head-up displays are currently used in fighter jets as one of the first AR applications. These include full interactivity, including eye pointing. Industrial Applications: AR can be used to compare the data of digital mock-ups with physical mock-ups for efficiently finding discrepancys between the two sources. It can further be employed to safeguard digital data in combination with existing real prototypes, and thus save or minimize the building of real prototypes and improve the quality of the final product. Military and emergency services: AR can be applied to military and emergency services as wearable systems to provide information such as instructions, maps, enemy locations, and fire cells. Prospecting: In the fields of hydrology, ecology, and geology, AR can be used to display an interactive analysis of terrain characteristics. Users could use, and collaboratively modify and analyze, interactive three-dimensional maps.