To appear in the Proceedings of the SPIE Photonic East 95 - Mobile Robots X Conference , Philadelphia , Pennsylvania , October 1995 . Structured Light Systems for Dent Recognition : Lessons Learned Juan C. Santamar ?? a College of Computing Georgia Institute of Technology Atlanta , Georgia 30332-0280 U.S.A. Ronald C. Arkin College of Computing Georgia Institute of Technology Atlanta , Georgia 30332-0280 U.S.A. ABSTRACT This paper describes the results from a feasibility analysis performed on two different structured light system designs and the image processing algorithms they require for dent detection and localization . The impact of each structured light system is analyzed in terms of their mechanical realization and the complexity of the image processing algorithms required for robust dent detection . The two design alternatives considered consist of projecting vertical or horizontal laser stripes on the drum surface . The first alternative produces straight lines in the image plane and requires scanning the drum surface horizontally , whereas the second alternative produces conic curves on the camera plane and requires scanning the drum surface vertically . That is , the first alternative favors image processing against mechanical realization while the second alternative favors mechanical realization against image processing . The results from simulated and real structured light systems are presented and their major advantages and disadvantages for dent detection are presented . The paper concludes with the lessons learned from experiments with real and simulated structured light system prototypes . 1 INTRODUCTION This paper describes the lessons learned on several hardware and software alternatives in the design of a structured light system3 ,5 to detect dent and bumps -LRB- blisters -RRB- on drums . Savannah River Technology Center -LRB- SRTC -RRB- is developing the Stored Waste Autonomous Mobile Investigator -LRB- SWAMI-II -RRB- to provide inspection and survey of low-level nuclear waste drums . SWAMI-II will help insure the health and safety of workers by monitoring the condition of radioactive , hazardous waste . The SWAMI-II base vehicle is a modified version of the HelpMate mobile robot manufactured by Transitions Research Corporation -LRB- TRC -RRB- . The vehicle will be equipped with a structured light system designed to recognize anomalies such as dents or blisters on a drum 's surface . The drums are stored on pallets on both sides of each aisle . As the robot passes each drum , it will estimate the location of each drum to appropriately aim a structured light system , taking images of it with cameras mounted on the rear of the vehicle . The objective of the dent recognition system is to detect dents and bumps -LRB- blisters -RRB- on a drum 's surface . The requirements of the system are to be able to recognize dents greater than 1 inch -LRB- 2.54 cm -RRB- laterally and vertically , and greater than 1 inch -LRB- 2.54 cm -RRB- deep on the visible portion of each drum . The drums have a black , shiny surface and some paper labels and will be located on pallets on both sides of the robot . The basic design involves a laser and a camera . The laser projects a stripe on the drum 's surface to make anomalies such as dents and blisters become visible as deformations of the laser stripe . Image processing software will analyze stored images off-board . This paper describes the results from a feasibility analysis performed on two structured light system designs and the image processing algorithms they require for dent detection and localization . The two design alternatives considered consist of projecting either vertical or horizontal laser stripes on the drum surface . The first alternative produces straight lines in the image plane and requires scanning the drum surface horizontally , whereas the second alternative produces conic curves on the camera plane and requires scanning the drum surface vertically . That is , the first alternative favors image processing against mechanical realization while the second alternative favors mechanical realization against image processing . In the general case , three different approaches for dent recognition were investigated . All of the approaches are variations on how to process an image of the drum surface illuminated by a laser beam . The first approach consists of zero-detection in the first derivative of the projected laser stripe . Points in which the first derivative evaluates to zero denote possible minima or maxima points on the laser stripe . These extreme points are candidate deformations on the drum 's surface . The second approach consists of detecting outliers from a fitted model of the laser stripe . For example , all those points that lie beyond two standard deviations from the fitted model are candidate deformations on the drum 's surface . The third approach involves spatio-temporal sampling to construct a 3-D model of the drum 's surface . This paper is organized as follows : Section 2 describes the structured light systems and their components as well as the image sampling techniques that are used to gather data . Section 3 describes the image processing algorithms for each of the three different approaches . Section 4 presents results using the alternatives based on images generated by both a simulation package and real-world laser striping prototypes . Section 5 presents an analysis of the results , a discussion of the advantages and disadvantages of each approach , and some recommendations for improving performance . 2 IMAGING HARDWARE 2.1 System design SWAMI-II will be equipped with a mobile structured light system placed on the back of the vehicle . The structured light system will face toward a target drum at one side of the robot . As the robot navigates through the aisle , the structured light system will position itself , always pointing towards the center of the drum . Drums on the other side of the aisle will be scanned similarly during a second pass with the robot facing in the opposite direction . Two structured light systems design were considered : horizontal scanning and vertical scanning . The former projects a vertical laser stripe on the drum surface while scanning the drum horizontally . The later projects a horizontal laser stripe on the drum surface while scanning the drum vertically . The horizontal scanning structured light system was the original design proposed by SRTC but was discarded afterwards due to the mechanical top view drum camera laser a l a l side view cameralaser 2 drum laser 1 l Figure 1 : Horizontal scanning design Figure 2 : Vertical scanning design complexity of its implementation . The horizontal scanning structured light system design consisted of a laser and a camera mounted on a mobile platform . The camera and the laser were each fixed on the opposite extremes of a supporting bar arranged horizontally . The laser beam and the camera axis converge on a point located on an imaginary line perpendicular to the bar that crosses its midpoint . Figure 1 shows the horizontal scanning structured light system . In the horizontal scanning design , as the robot navigates through an aisle , the structured light system moves in such a way as to always point to the center of the drum . The movement of the frame is performed by following the circumference keeping a constant distance between the structured light system and the surface of the drum . Figure 3 demonstrates this procedure . The laser projects an eye-safe beam that produces a laser stripe on the drum surface . Dents and bumps can be recognized by capturing and analyzing images containing this stripe . The vertical scanning design consisted of two lasers and a camera mounted on a mobile platform . The two lasers are each fixed on opposite extremes of a supporting bar and the camera is located on the midpoint . The laser beams and the camera axis are parallel to each other and perpendicular to the drum surface . Figure 2 shows the redesigned structured light system . Using this system , as the robot navigates through an aisle , it will stop at each drum , then the structured light system moves vertically to cover the drum surface . Figure 4 demonstrates this procedure . The lasers projects two eye-safe beams that will produce two laser stripes on the drum surface , one above and one below the center of the image . Dents and bumps can be recognized by capturing and analyzing images containing these stripes . Image capture is the same in both designs . Images are captured at fixed intervals . Each image is thresholded and thinned in order to obtain a line description that occupies less space to store and is easier to process later on . Further details of the image processing depend on the particular approach selected . The next section describes each strategy in detail . 2.2 System components The components used in the prototype systems are the same as the components specified for the current structured light system design . The following lists describes each component . top view drum camera laser q x y trajectory robot q q min max d drum z trajectory robot d side view cameralaser 2 laser 1 z z min max Figure 3 : Sampling in the horizontal scanning design Figure 4 : Sampling in the vertical scanning design Camera : WATEC P/N WAT-203EX Lens : JML P/N CTV 20080/100 Laser : Lasiris P/N SNF-501L-635M-3-45 CCD Size : 1/2 '' format 6:4 ? 4:8 mm Horizontal Resolution : 550 TV Lines Sensitivity : 0.1 Lux Lens : C Mount External Sync -LRB- HD-VD -RRB- Size : 2:2000 ? 2:4400 ? 1:8800 Focal Length : 6.0 mm f / # : 1.2 Format : 1/2 '' CCD Mount : C Manual Iris Filter Diameter : M37 :5 ? :5 Length : 40 mm Diameter : 45 mm Weight : 110 grams Wave Length : 635 nm @ 25 Cffi Power : 3 mW Class : IIa Stability : 0.25 nm/Cffi Fan angle : 45 degrees Modulation Option : Input TTL Frequency range : DC , 6 Hz -LCB- 2 MHz 3 VISUAL DENT DETECTION APPROACHES This section describes in detail several approaches to detect dents on drum surfaces using either the horizontal or vertical scanning structured light system designs . Each approach uses a different image processing technique to detect possible deformations on the drum by analyzing the images captured with the structured light systems described in the previous section . All of the approaches utilize the same input formats and generate the same output . The differences involve the image processing technique used to detect the surface deformations . There are two types of inputs to the dent detection software : configurational and data . The configurational input consists of information regarding the position and orientation of the laser and camera . The software only requires this information once at the beginning of execution . The data input consists of the images acquired by the camera . The software acquires this information periodically using it to detect and localize dents on drums . It is assumed that the structure supporting the laser and the camera always points toward the center of the drum while the structured light system scans the drum surface -LRB- refer to Figures 1 and 2 -RRB- . The input specifications follow : 1 . Frame Structure Information . The following information is required for the configuration of the structured light system : -LRB- a -RRB- l : the distance in meters between the camera and the laser on the frame . Range : -LSB- 0 ; 1 -RRB- -LRB- b -RRB- ff : the angle in degrees between the longitudinal axis of the camera and the supporting frame . Range : -LSB- 0 ; 90 -RSB- -LRB- c -RRB- d : the distance between the middle point of the supporting frame and the center of the drum . Range : -LSB- 0 ; 1 -RRB- 2 . Data Information . The following information is required for each data sample : -LRB- a -RRB- -LSB- In -RSB- : the digitized image at that position . Range : In is an array of pixel values -LSB- 0 ; 255 -RSB- . Ther output delivered by the dent recognition software consists of whether or not there is a dent in the processed image . The position of a detected dent can then be estimated considering the position of the structured light system at the time that image was taken . 3.1 Preprocessing algorithms and common image processing routines The following image processing routines are general and most are used in both of the structured light system designs . Cut : The purpose of this routine is to clip an image with a specified window and produce another image with only the information inside the given window . This function is required only in the vertical scanning structured light system and it is used to separate the top laser stripe from the bottom one . The window depends only on the configuration information and it is the same for each of the scanned images . In this way , each laser stripe can be processed individually and independently from each other . Binarize : The purpose of this function is to contrast the pixels related to the laser beam from the background . The routine takes an image and a threshold value and produces a quantized two-valued image where all the pixels of the original with an intensity above the specified threshold are set to the maximum intensity value and the rest of the pixels are set to zero . Average : The purpose of this routine is to produce a list of points -LRB- x ; y -RRB- that represent the line produced by the projected laser beam on the image . Average scans the image column by column and returns the weighted row average of the for that column1 -LRB- i.e. y -LRB- x -RRB- is the center of mass across all rows at column x -RRB- . Equation 1 shows the formula to estimate the row -LRB- y -RRB- for a given column -LRB- x -RRB- . y -LSB- x -RSB- = 8 > < > : if Ix = r = R X r = 0 r I -LRB- x ; r -RRB- Ix otherwise -LRB- 1 -RRB- where Ix = R X r = 0 I -LRB- x ; r -RRB- , and R is the total number of rows . The result of this routine a thinned line that serves as the best estimator of the laser stripe projected on the drum surface . The thinned line is the basic input for all of the dent detection approaches . Extrapolate : The purpose of this function is to fill up the empty spaces in a thinned line description of a laser stripe . Empty spaces are the result of low intensity spots in the laser stripe that were filtered out by the imaging or binarize operations . The interpolation is performed linearly using the two extreme points in the gap . Smooth : The purpose of this function is to cancel out noise in a thinned line and produce a smoothed version of the laser stripe . The filter used is an unidimensional Gaussian kernel of a specified size S and bandwidth constant 1The direction of the scan depends on the orientation of the laser stripe . Horizontal laser stripes are scanned column by column and vertical laser stripes are scanned row by row . rc noise spot window -LRB- W -RRB- moving average -LRB- -RRB- I -LRB- x , y -RRB- rows columns Figure 5 : Moving-window-average B. Equation 2 shows the formula to calculate the smoothed y for each x. y -LSB- x -RSB- = S = 2 X t = ? S = 2 y0 -LSB- x ? t -RSB- Gaussian -LSB- t -RSB- -LRB- 2 -RRB- where y0 -LSB- ? -RSB- is the periodic expansion of y -LSB- ? -RSB- and Gaussian -LSB- t -RSB- = exp -LRB- ? t2 B -RRB- is the filter that spans from ? S = 2 to S = 2 . Moving-Window-Average : Similar to Average , the purpose of this routine is to produce a list of points -LRB- x ; y -RRB- that represent the line produced by projected laser beam on the image . The difference is that the MovingWindow-Average routine estimates the y coordinate of each point by calculating a weighted average over a small window centered on the moving average of the last N points . In this way , bright spots that are produced by reflection or refraction of the laser beam are filtered out . Figure 3.1 shows the procedure and Equation 3 shows the formula used to estimate the row -LRB- y -RRB- given column -LRB- x -RRB- . This function was created to overcome noisy bright spots on the images taken with the prototype system in the real world . Moving-window-average is computed using : y -LSB- x -RSB- = 8 >> < >> : if Ix = rc + W = 2 X r = rc?W = 2 r I -LRB- x ; r -RRB- Ix otherwise -LRB- 3 -RRB- where Ix = rc + W = 2 X r = rc?W = 2 I -LRB- x ; r -RRB- , which is the center of mass across W rows at column x and centered at row rc , and rc is the moving average of the last N points , which is updated for the following column using the following an exponential smoothing formula : rc N N +1 rc + 1 N +1 y -LSB- x -RSB- . 3.2 Dent detection approaches using vertical laser stripes This section describes two approaches used to detect dents on drum surfaces . Vertical laser stripes result from the horizontal scanning structured light system design . The approaches described here take advantage of the projection of vertical laser stripes over cylindrical surfaces . This simplifies the task of detecting dents because a vertical laser stripe projected over a smooth cylindrical surface produces a vertical line in the image plane . Thus , any deviations from a vertical line are an indication of the presence of an anomaly on the surface . First Derivative Approach : One way to detect deformations on the drum 's surface consists of detecting the maxima and minima points For every captured image I -LRB- x ; y -RRB- : 1 . Binarize the image . 2 . Average binary image to produce thinned line . 3 . Smooth thinned line with a Gaussian filter . 4 . Calculate the first derivative . 5 . Detect all the points that evaluate to zero . 6 . Report a list of all the maxima and minima points above limits . Figure 6 : Zero Detection Algorithm . For every captured image I -LRB- x ; y -RRB- : 1 . Binarize the image . 2 . Calculate column sample mean ? x and sample column standard deviation ps2 3 . Detect all outlying points Ibinary -LRB- x ; y -RRB- = 1 such that jy ? ? xj > kps2 4 . Report a list of all the consecutive outlying points above a limit . Figure 7 : Outliers Detection Algorithm . of the projected laser stripe in the captured image . Minima and maxima points are candidates for estimating the size of the deformations on the drum 's surface . A maximum or a minimum point of a function can be detected by analyzing the first derivative of such a function and detecting the points in which it evaluates to zero . Maxima and minima points are points in which the first derivative is zero . Then , the magnitude of the deviation from the straight line to the extreme point can be used to estimate the deformation on the drum surface . Before estimating the first derivative of the laser stripe in an image , some preprocessing is required . First , the image must be thresholded to discard all information not related to the portion of the drum where the laser is projected -LRB- i.e. , apply Binarize -RRB- . Second , all the horizontal points of the stripes are averaged together to produce a single point per row -LRB- i.e. , apply Average -RRB- . In this way , the original vertical laser stripe is condensed to a vertical line . Third , the line is smoothed with a Gaussian kernel to filter out noise -LRB- i.e. , apply Smooth -RRB- . Finally , the first derivative is estimated and the zero crossing points are detected . One last step verifies the magnitude of the maximum and minimum points to detect valid dents and blisters on the drum surface . The algorithm is shown in Figure 6 . Outliers Approach : As in the previous approach , the purpose of this approach is to detect deformations on the drum 's surface by detecting deviations of the projected laser stripe . However , instead of finding the maximum and minimum points on a preprocessed vertical line , this method concentrates on locating points that are far from the vertical stripe in a statistical sense . The best line fit to the projected laser stripe can be calculated using least squared estimation -LRB- LSE -RRB- .2 Then , all the outlying points can be identified and the dents detected if there exist enough consecutive outlying points . For the simple case of a vertical straight stripe , the mean and the standard deviation of all the points in the thresholded image are calculated . Then , all of the points lying beyond some constant multiple of the sample standard deviation -LRB- ps2 -RRB- are reported as outliers . The algorithm is shown in Figure 7 . 2In this case , the LSE of the best vertical line is simply the sample average -LRB- i.e. , x = ? x -RRB- . For every captured image I -LRB- x ; y -RRB- : 1 . Cut the image into top and bottom sections . 2 . Calculate thinned lines using the Moving-Window-Average routine . 3 . Smooth thinned lines with a Gaussian filter . 4 . Calculate the second derivative . 5 . Detect all outlying points . 6 . Report a list of all the outlying points above limits . Figure 8 : Second-derivative Algorithm . 3.3 Dent detection approaches using horizontal laser stripes This section describes three approaches used to detect dents on drum surfaces . Horizontal laser stripes result from the vertical scanning design of the structured light system . Projecting horizontal laser stripes over cylindrical surfaces produces a curved line on the image plane of the camera -LRB- i.e. , a conic -RRB- . Detecting dents is not as simple as when analyzing vertical stripes because the reference is not a linear function . Using a simple camera model1 it can be shown that the laser stripe is projected on the image plane as a section of an ellipse . However , given the current structured light system design and the field of view of the lens , the camera can cover at most 1/3 of the drum surface .3 Under this configuration , the section of the ellipse can be approximated by a parabola without significant error . The approaches described here take advantage of a parabolic model of the line in the image plane . Thus , any deviations from this model can be considered as an indication of an anomaly on the surface on the drum . Deviations from Second Derivative Approach : Since the reference model is a parabola , a simple way to detect deformations on the drum 's surface consists of detecting when the quadratic coefficient is not constant . That is , taking the second derivative of the line and detecting outliers from its mean . Outlying points only provide information about strong deviations from the reference model -LRB- i.e. , a parabola -RRB- . In order to estimate the magnitude of the deformation on the drum surface it is necessary to calculate the best fitting parabola and consider the difference between the parabola and the line , both evaluated at the outlying points . Before estimating the second derivative of the laser stripe in an image , some preprocessing is required . First , the image is clipped to produce two image sections : one with the top laser stripe and the other with the bottom laser stripe . Second , each of the image sections is preprocessed with the Moving-Window-Average routine described previously to produce thinned lines . Third , the lines are smoothed with a Gaussian kernel to filter out noise . Finally , the second derivative is estimated and the outlying points are detected . One last step verifies the magnitude of the outlying points to detect valid dents and blisters on the drum surface . The algorithm is shown in Figure 8 . Outliers from a Fitted Parabola : As in the previous approach , the purpose of this approach is to detect deformations on the drum 's surface by detecting deviations of the projected laser stripe from its reference model . However , instead of calculating outliers relative to the second derivative , this method concentrates on fitting the best parabola to the given points and detecting outliers from the fitted curve in a statistical sense . The LSE estimation of the best fitting parabola to the projected laser stripe is calculated using the list of 3For this reason the structured light system consists of three cameras and six lasers . Each camera and its respective pair of lasers cover 1/3 of the total surface of the drum . For every captured image I -LRB- x ; y -RRB- : 1 . Cut the image into top and bottom sections . 2 . Calculate thinned lines using the Moving-Window-Average routine . 3 . Estimate parameters of the polynomium p -LRB- x -RRB- = + 1x + 2x2 4 . Detect all outlying points such that jy -LRB- xh -RRB- ? p -LRB- xh -RRB- j > kps2xh where ps2xh = XhT -LRB- s2XT X ? 1 -RRB- Xh 5 . Report a list of all the consecutive outlying points above a limit . Figure 9 : Parabolic Model Outliers Detection Algorithm . points -LRB- x ; y -RRB- from the thinned line .2 Then , all the outlying points are identified and dents reported as detected if there exist enough consecutive outlying points . For the case of a parabola -LRB- i.e. , a second order polynomium -RRB- , the sample standard deviation of a point depends on its offset from the mean and the inherent error in the signal . All the points beyond some constant of its sample standard deviation can be reported as outlying . The algorithm is shown in Figure 9 . The LSE estimation of the parameters is based on a model is of the form : Yi = + 1Xi + 2X2i + ffli where i = 1 ; : : : ; n and n is the number of points . The previous equation can be compactly written in matrix terms as follows : Y = X + ffl where Y = 2 64 y1 ... yn 3 75 ; X = 2 64 1 x1 x21 ... ... ... 1 xn x2n 3 75 ; = 2 4 1 2 3 5 ; and ffl = 2 64 ffl1 ... ffln 3 75 : Given the above formulae , the least squared error estimation of can be calculated as : ? = -LRB- XTX -RRB- ? 1XTY It is important to mention that the previous formulae estimate the parabola that minimizes the cost function Q = Pnk = 1 -LRB- yk ? ^ yk -RRB- . However , if there is a relatively small rotation between the drum and the image plane of the camera as depicted in Figure 10 , then it is required to find at which image plane angle the cost function Q -LRB- ? -RRB- is minimized . The algorithm in Figure 11 uses gradient descent to find the best fitting parabola when the frame of reference is allowed to rotate freely -LRB- i.e. , finds the LSE estimation of the best fitting parabola assuming there is a misalignment with the axis of the drum and the normal vector of the image plane -RRB- . Spatio-Temporal Approaches : This approach uses each projected laser stripe to progressively construct a 3-D model of the drum 's surface .4 This 3-D model is then analyzed to detect dents of different sizes and shapes . Constructing a 3-D model is possible by fitting a parabola to consecutive curves and using it as a reference . With this information , it is possible to estimate the depth of each point on the drum surface given the deviation of its corresponding point on the reference surface . Thus , by combining the information of the points of the laser stripes of images captured at different positions , a three dimensional grid of the drum 's surface can be estimated . Such a grid can then be processed to locate and detect dents and blisters of arbitrary shape . focal point laser stripe drum x y x ' y ' rotated image plane image plane Figure 10 : Camera model with rotated image plane . For a thinned line description -LRB- xi ; yi -RRB- and i = 1 ; : : : ; n : 1 . Standardize all points : -LRB- xi ; yi -RRB- ! -LRB- x0i ; y0i -RRB- where x0i = -LRB- xi ? ? x -RRB- = ps2x and y0i = -LRB- yi ? ? y -RRB- = ps2y 2 . l min angle , r max angle . 3 . while -LRB- r ? l < ffl -RRB- 3.1 . m l + r2 3.2 . Rotate all points an angle m h xri yri i = h cos -LRB- m -RRB- sin -LRB- m -RRB- ? sin -LRB- m -RRB- cos -LRB- m -RRB- ih x0iy0i i 3.3 . Estimate best parabola -LRB- xri ; fyri -RRB- for -LRB- xri ; yri -RRB- 3.4 . Calculate cost : Q Pnk = 1 -LRB- yri ? fyri -RRB- 2 3.5 . If @Q @m = exit 3.6 . If @Q @m > r m 3.7 . If @Q @m < l m Figure 11 : Rotated Regression Algorithm . 1 . For all the images In -LRB- x ; y -RRB- of a drum : 1.1 . Cut the image into top and bottom sections . 1.2 . Calculate thinned lines using the Moving-Window-Average routine . 1.3 . Smooth thinned lines with a Gaussian filter . 1.4 . Estimate reference curve using the k consecutive smoothed lines . 1.5 . Construct three-dimensional grid by subtracting thinned lines and the reference curves . 2 . Scan three-dimensional grid for dents and blisters . 3 . Report a list of all the dents and blisters . Figure 12 : 3D Modeling Algorithm . As in the previous approaches , some preprocessing of the raw image is required to reduce the amount of noise . The preprocessing steps involve estimating the thinned line using the moving window average routine , extrapolating missing points , and smoothing the resultant curve with a Gaussian filter . The algorithm is shown in Figure 12 . 4 RESULTS Experiments with the horizontal scanning structured light system were performed in simulation only because the mechanical realization of a prototype proved too expensive . Experiments with the vertical scanning structured light system were performed with two prototypes : one was assembled at the Georgia Tech Mobile Robot Laboratory -LRB- GT-prototype -RRB- and the other one was assembled by Savannah River Technology Center personnel -LRB- SRTC-prototype -RRB- . Figures 21a and 21b show a picture of the prototypes . Both prototypes have the camera rotated 90 degrees clockwise to take advantage of greater resolution in the horizontal axis of the image plane . This also means that horizontal lines on the drum surface look vertical on the image . The following subsections present the results of the preprocessing routines , the horizontal , and vertical scanning structured light systems . 4.1 Preprocessing Routines All the figures presented here are based on image series taken with the SRTC-prototype . Cut : This routine takes a portion of an image . Figure 13 shows a 512 ? 486 image and the result of a Cut operation with window starting at -LRB- 95 ; 0 -RRB- , 35 columns wide and 486 rows tall -LRB- center -RRB- . The bright spots results from specular reflections produce by the camera 's lens and the shiny drum surface . Binarize : This routine creates a binary image . Figure 13 shows a 35 ? 486 image -LRB- center -RRB- and the result of a Binarize operation with threshold 20 -LRB- right -RRB- . Average : This routine creates a thinned line from an image . Figure 14 the result of an Average operation to the 35 ? 486 image from Figure 13 -LRB- center -RRB- . Smooth : This routine filters noise from a thinned line . Figure 15 shows the result of a Smooth operation of size of 43 pixels and bandwidth constant of 7 pixels applied to the thinned line in Figure 14 . Moving-Window-Average : This routine creates a thinned line from an image . Figure 16 shows the result of a Moving-Window-Average operation with window 10 to the 35 ? 486 image from Figure 13 -LRB- center -RRB- . Figure 13 : Left : the original image . Center : its bottom portion . Right : binarized . Figure 14 : Thinned line . Figure 15 : Smoothed line . Figure 16 : Moving-Window-Average thinned line . Figure 17 : Left : Standard regression . Right : Regression with rotation . Regression : Although this is not a preprocessing routine , an example of the difference between the standard regression and the algorithm in Figure 11 is shown below . Figure 17 shows the fitted parabola to a thinned line . 4.2 Horizontal scanning structured light system All the experiments performed on the horizontal scanning structured light system were simulated . All images were generated using the rayshade environment . The simulated structured light system has the following parameters -LRB- refer to Figures 1 and 3 -RRB- : l = 10cm , ff = 81ffi , d = 80cm , and ? 150ffi < = ? < = ? 30ffi . Images were captured every 5 degrees . The model of the drum contained three dents and three bumps . Figure 18 shows the model of the drum used during these experiments . Figure 18 : Drum Model . First derivative approach : Figure 19 shows the different steps of the zero-detection algorithm of the image at position ? = ? 135ffi . Outliers approach : Figure 20 shows the different steps of the outlier-detection algorithm of the image at position ? = ? 135ffi . 4.3 Vertical scanning structured light system The GT-prototype and the SRTC-prototype were used to perform experiments using the current structured light system design . The GT-prototype consists of one camera and one laser mounted 4.5 inches above the camera . Figure 21 a shows a picture of the prototype . The SRTC-prototype was designed and developed at the Savannah River Technology Center . The SRTC-prototype consists of one camera and two lasers . Figure 21b shows a picture of the prototypes . Deviations in Second Derivative Approach : Figure 22 shows the results for different steps of the second derivative algorithm . The image was gathered using the GT-prototype . There is one dent of depth 1/8 of an inch -LRB- approx . -RRB- in the center . Outliers from fitted parabola : Figure 23 shows the different steps of the algorithm that detects outliers from the best fitted parabola . The image was gathered using the GT-prototype . There is one dent of depth 1/8 of an inch -LRB- approx . -RRB- in the center . Spatio-Temporal Approach : Figure 24 shows the different steps of the spatio-temporal algorithm . Images were gathered using the SRTC-prototype . The processing was performed using 17 images , each one is taken by vertically moving the light system 50 mils down -LRB- i.e. , vertical scanning every 50 mils -RRB- . The top portion of the images cover a total of 5 artificial created bumps -LRB- three to the right and two to the left -RRB- . 5 DISCUSSION AND RECOMMENDATIONS In general , all the approaches presented here can be used to detect dents on drums . However , some approaches are more sensitive to noise and lighting conditions . Approaches that are based on derivatives are very sensitive to noise , but smoothing operations can reduce the variability . There is a compromise between strong filtering to reduce the noise and the desired resolution for dent detection . The more the filter rejects noise the less chance of detecting small dents . Specular reflection is a major problem for line extraction because the images contain spots of bright light near the lines -LRB- see right of Figure 13 -RRB- . These bright spots tend to bend the line during the thinning operation , which can trigger the dent detection algorithm and produce false positives . The Moving-Window-Average routine can help solve this potential problem effectively as shown in Figure 16 . Positioning and misalignment are another source of errors . Figure 17 shows the systematic error that can be introduced by fitting a parabola assuming perfect alignment between the axis of the drum and the image plane . The reason for this is that the regression equations minimize the squared difference in the ordinates axis only . The algorithm in Figure 11 corrects this problem by searching the rotation angle that minimize the squared difference in the ordinate axis . One major problem with the structured light system is the ambient lighting conditions . The experiments performed at Georgia Tech and in Savannah River Laboratories were performed under rigorous lighting constraints . Figure 19 : Zero detection images . Top left is the raw image . Top right is the thresholded image . Middle left is the mean of rows . Middle right is the smoothed line -LRB- expanded -RRB- . Bottom is the first derivative -LRB- expanded -RRB- . The two bumps were detected -LRB- plus two ribs -RRB- . Figure 20 : Outliers images . Top left is the raw image . Top right is the thresholded image . Bottom shows outliers -LRB- k = 2 -RRB- . Two two bumps were detected -LRB- plus two ribs -RRB- . Figure 21 : Prototypes : GT-prototype -LRB- left -RRB- and SRTC-prototype -LRB- right -RRB- . Figure 22 : Deviations in second derivative . Top left is the raw image . Top right is relevant portion . Middle left is the thinned line . Middle right is the smoothed line . Bottom left is the first derivative . Bottom right is the second derivative . The dent is readily detected . outlier region outlier region Figure 23 : Outliers from fitted parabola . Top left is the raw image . Top right is relevant portion . Middle left is the thinned line . Middle right is the smoothed line . Bottom left is the fitted parabola . Bottom right is the difference between the smoothed line and the parabola showing the outlying regions . The dent is readily detected . Figure 24 : Spatio-temporal approach . Top left is the set of original thinned lines . Top right is the set of smoothed lines . Bottom left is the set of fitted parabolas . Bottom right is the difference between the smoothed lines and the parabolas . However , none of the two prototype systems used included a red filter to enhance laser strip contrast and reduce ambient light . It is expected that a red filter can assist in solving this problem , but further evaluation is necessary . The sensitivity of the vertical scanning structured light system design is too low considering the amount of noise and size of the dents . Considering some images taken from the SRTC-prototype , there is a difference between the closest point -LRB- center of image -RRB- and the farthest point -LRB- one of the sides -RRB- of 20 pixels . 55-gallon drums , have a radius of 12 inches approximately . Since the camera covers 1/3 of half the drum surface , the sensitivity of the structured light system is about 0.3 inches/pixels . In the vertical scanning structured light system design , the axis of camera and the laser light plane are parallel . The focal length and the distance between the lens and the drum are the two parameters that determine the sensitivity of the system . Reducing the distance between the drum and the camera or increasing the focal length will increase the sensitivity . However , both alternatives reduce the field of view of the camera . A third alternative consists of projecting the lasers at an angle . This can increase the deformation of the line on irregular surfaces and indirectly increase the sensitivity of the system . 6 ACKNOWLEDGMENTS This research was supported by the Westinghouse-Savannah River Technology Center . Additional funding for the Mobile Robotics Laboratory is provided by ARPA/ONR , the U.S. Army Applied Aviation Directorate , and the NAtional Science Foundation . 7 REFERENCES -LSB- 1 -RSB- R.O. Duda and P.E. Hart . Pattern Classification and Scene Analysis , Chapter 7 , Wiley , New York , 1973 . -LSB- 2 -RSB- J. Neter , W. Wasserman , and M.H. Kutner . Applied Regression Models , Chapters 6 and 9 , Prentice Hall , New Jersey , 1989 . -LSB- 3 -RSB- D. Lake . \ Lighting and Imaging , Part 2 : Making it All Work Right , '' Advanced Imaging , pp. 66 -LCB- 89 , June , 1994 . -LSB- 4 -RSB- A.C. Kak . \ Depth Perception For Robots '' , Handbook of Industrial Robotics , Shimon Y. Nof -LRB- ed . -RRB- , pp. 272 -LCB- 319 , John Wiley & Sons , New York , 1985 . -LSB- 5 -RSB- D. Wei and M. Gini . \ The Use of Taper Light Beam for Object Recognition , '' Robot Vision . -LSB- 6 -RSB- W.J. Wolfe , J. Gunderson , and M.E. Walworth . \ Computer Vision Barrel Inspection , '' SPIE Mobile Robots VIII , Vol . 2058 , pp. 128 -LCB- 133 , 1993 .

O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O B I I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

dent detection
image processing
structured light systems


Scalable Implementation of Synchronization Primitives on Broadcast Rings ? Martin H. Davis , Jr. . Umakishore Ramachandran GIT-CC-93 / 07 January 1993 College of Computing Georgia Institute of Technology Atlanta , GA 30332-0280 fdavism,ramag@cc.gatech.edu Abstract Synchronization is an important aspect of parallel program design . By definition synchronization is an aspect of a program where multiple processors participate . Thus it is important to design and implement hardware primitives that scale well with the size of the parallel machine , both in terms of space and time requirements . The focus of this research is to propose implementation for some well-known synchronization primitives in a broadcast ring network . The key aspects of the implementation are to make local decisions to determine the outcome of the synchronization operations ; and to keep the space overhead per node constant independent of the number of processors participating in such operations . It is also shown that the implementation incurs exactly the minimum amount of communication to perform the synchronization operations . ? This work has been funded in part by NSF PYI Award MIP-9058430 . 1 Introduction The work in this paper focuses on how shared memory style synchronization commands can be efficiently implemented . In order to make our solution general , we have made the implementation of these commands orthogonal to the physical implementation of the shared memory abstraction . In contemplating this problem , we have been interested in how the interconnection network can be structured so as to provide more than just communication services , i.e. , how can the interconnection network 's structure provide support for useful parallel programming commands ? From this consideration we have developed interesting \ local decision '' techniques for implementing locks , barriers , and F&OP synchronization primitives . In Section 2 we describe these implementation techniques . Our implementation incurs exactly the minimum amount of communication to perform the synchronization , by making local decisions for determining the outcome of the synchronization operations . Further , the implementation requires a constant amount of storage space per node independent of the number of processors participating in a synchronization operation . We present some discussion and analysis regarding these issues in Section 3 . Finally , we conclude in Section 4 with questions for future work . 2 Synchronization Primitives There are software algorithms for barriers and spin-lock that minimize the amount of network messages and storage space -LSB- 4 -RSB- for implementing such synchronization operations . However , such algorithms do not reduce the inherent latency for synchronization operations . This latency is two-fold : -LRB- 1 -RRB- the potential wait times at synchronization points due to simultaneous access from parallel processors ; and -LRB- 2 -RRB- the intrinsic overhead for implementing the synchronization . In this section we describe our proposed hardware implementation for some well-known synchronization primitives that reduces this latency . Our implementation techniques for the three synchronization primitives are all local decision monitoring based . That is , each node maintains its own local copy of a data structure , which represents the current state of a particular instance of the synchronization primitive , by monitoring all the nodes ' commands which affect that particular instance . The data structure contains enough information that each node independently decides how it is affected by the current state . The only architectural requirement for our local decision scheme is that the nodes transmit their synchronization commands via what we call a broadcast ring . The broadcast ring has three properties which are illustrated in Fig. 1 . First , the nodes are attached -LRB- at least logically if not physically -RRB- to the medium as a ring . Thus , a node 's message is transmitted unidirectionally to all its downstream neighbors , and a node always hears the messages from all its upstream neighbors . Second , every node 's message is broadcast to every other node , i.e. , each node is connected to the broadcast ring such that it hears all messages -LRB- including its own -RRB- . This connection is equivalent to every node having a connection so that every node -LRB- including itself -RRB- is upstream of the connection . Third , each node hears the broadcast messages in the same order as every other node ; this property is expressed as a \ train '' of messages . In the following subsections we describe these local decision monitoring algorithms for locks , barriers , and F&OP . 2.1 Locks A lock is a mechanism by which a task can access a set of shared data associated with the lock in a controlled and regulated fashion . There are two different modes in which a lock can be requested : read -LRB- or shared , denoted as R/S -RRB- and write -LRB- or exclusive , denoted as W/E -RRB- . By definition , use of the R/S mode guarantees to the task that the data can not change for the duration of the lock . Similarly , by definition , use of the W/E mode guarantees to the task that no other task can access the data , much less change the data . How requests for a particular lock are serviced can be understood by the concept of peer groups , as illustrated in Fig. 2 . A peer group of lock requests is a set of consecutive and compatible requests for a particular lock . Thus , a W/E request forms a peer group of size one . R/S requests made without an intervening W/E request form a peer group of arbitrary size . Although peer groups can be serviced out of order , some rules must exist to prevent starvation of requests . From a performance standpoint , if a R/S peer group is the currently serviced peer group and no other peer groups are pending , then another R/S request joins the current R/S peer group and is granted the lock immediately . Fig. 3 shows the data structure needed for each lock -LRB- note the various fields are grouped into four parts -RRB- . Remember that this data structure is replicated at each node and modified independently by each node . The first group -LRB- LOCK-ID and my -RRB- contains the fields to identify the lock and define the peer group into which the lock request is placed . The second group -LRB- prior -RRB- of fields defines the peer group which is to be serviced immediately before the lock request 's own peer group . The third group -LRB- most-recent -RRB- of fields defines the most recent peer group of lock requests , and the fourth group -LRB- next-recent -RRB- defines the peer group just before the most recent peer group . These four groups of fields , when taken as a whole over all the nodes , implement a distributed queue of lock requests . This queue is replicated at each node ; note , however , that each node stores only the portion of the queue that is necessary to its knowing wherein the queue its own request falls and when its request should be granted . To maintain its portion of the distributed queue , each node follows specific monitoring algorithms -LRB- shown in Appendix A -RRB- when observing all nodes ' lock requests . Recall that the broadcast ring 's properties guarantees that each node sees all the lock command traffic both preceding and succeeding its own command and that each node sees the same sequence of lock commands as every other node . The algorithms can be summarized as follows -LRB- the details can be found in -LSB- 1 -RSB- -RRB- . The first algorithm is for when a node sees a lock request from any node except itself : the node updates the most-recent and next-recent groups . The second algorithm is for when a node sees its own lock request command : the node updates the my fields , the prior , most-recent , and next-recent groups , and determines whether it can immediately acquire the lock . The third and fourth algorithms are needed for when the node sees a R/S or W/E unlock command respectively : the most-recent and next-recent groups are always updated appropriately , and the node 's own outstanding lock request -LRB- if any -RRB- is granted if appropriate . Thus , this implementation provides the logical abstraction of a nearly FCFS queue for lock requests . It is \ nearly '' FCFS because of the broadcast ring 's possible re-ordering of nodes ' requests within one ring cycle . Although the broadcast ring 's unidirectional property compromises true FCFS service , the same property also guarantees forward progress and no starvation of the lock requests . 2.2 Barriers The barrier is a rendezvous point for some set of tasks . The barrier 's semantics are that after initialization of the barrier , the participating tasks are free to reach the barrier at their own pace . A task is in the arrival phase when it reaches the barrier . Barrier completion occurs when all participating tasks have arrived at the barrier . A task must wait at the barrier until it becomes aware of barrier completion , at which time it can proceed past the barrier . Fig. 4 depicts the simple data structure which captures the state of a barrier . Recall that there is one such data structure per barrier , and this data structure is replicated at each node . The ID field uniquely identifies the barrier . The participation field stores how many nodes are participating in the barrier , and the counter field tracks how many nodes have reached the barrier . By maintaining the state of the barrier , each node can deduce when barrier completion occurs . To maintain the state of the barrier , each node follows a simple monitoring algorithm -LRB- shown in Appendix B -RRB- when observing all nodes ' barrier commands . If the observed barrier command is for barrier initialization , then the participation and counter fields are set to the number of participating nodes and zero respectively . If the observed barrier command is that a node has reached the barrier , then the counter field is incremented . When a node itself reaches the barrier , it stalls at the barrier by watching the counter field and waiting for it to equal the participation field . When the two fields are equal , the node realizes that barrier completion has occurred . Note that since each node locally ascertains when barrier completion has occurred , there is no need for an explicit command to notify nodes of barrier completion . If the data structure continues to represent the same logical barrier , then reuse of the barrier hardware is a simple matter of each node resetting its copy of the counter field . If the data structure is to represent a different logical barrier , then an explicit barrier initialization command is required to reset the participation and counter fields . 2.3 Fetch - & - OP The F&OP -LRB- Fetch - & - OP -RRB- primitive is a generalization of the Fetch - & - ADD primitive introduced by Gottlieb -LSB- 2 -RSB- . Give that the OP represents any associative , commutative operator , the semantics is that when a node issues F&OP -LRB- V , e -RRB- , the command returns the old value of V to the node and atomically replaces V with V OP e . An attractive property of the F&OP command is its potential combining capability . This capability can be expressed by saying that the F&OP command must satisfy the serialization principle : if V is a shared variable and many nodes issue F&OP -LRB- V , e -RRB- to the same target V simultaneously , then the effect of the many parallel F&OP commands is exactly what it would be if they had occurred in some -LRB- unspecified -RRB- serial order . That is , the final value of V due to the parallel F&OP -LRB- V , e -RRB- commands is the result of applying all the operators OP and operands e to the original V , and each node receives an intermediate value of V depending upon where its own particular F&OP -LRB- V , e -RRB- command happens to fall in the arbitrary serial order . The data structure associated with each target V is shown in Fig. 5 . The V-current field stores the current value of V , and the V-mine field stores the value of V which the node obtains from having issued the F&OP -LRB- V , e -RRB- command . Before any F&OP -LRB- V , e -RRB- command for a particular target V is issued by a node , the target V must be initialized , i.e. , the V-current field is set to the initial value of V. From this data structure , each node determines the value of V it should receive from its F&OP -LRB- V , e -RRB- command and updates V. To update its data structure , each node follows a simple monitoring algorithm -LRB- shown in Appendix C -RRB- when observing all nodes ' F&OP commands . When the node observes its own command , it sets V-mine equal to V-current -LRB- and returns that value to the CPU -RRB- ; for any node 's command , the node updates V-current . Thus , each node is provided the logical abstraction that its F&OP command is placed somewhere into a serial ordering of all the nodes ' F&OP commands . 3 Analysis and Discussion 3.1 General observations There are several general observations to be made about these implementations . First , each implementation is decentralized . Each node independently maintains its copy of the appropriate data structure by observing the synchronization command traffic . Specifically , for locks , each node observes the stream of lock commands including its own . By observing this stream of commands , the node determines in what peer group its own command falls ; from the stream , the node also deduces when the lock is released by the previous holder -LRB- s -RRB- . Thus , no central entity takes responsibility for queuing lock requests and granting them . For barriers , each node observes the stream of barrier arrivals including its own . By observing this stream , each node independently counts how many nodes have arrived at the barrier ; from this information , each node independently ascertains when barrier completion has occurred . Thus , since no central entity computes barrier completion , no central entity has responsibility for notifying others of barrier completion . For the F&OP command , each node also observes the stream of commands including its own . By observing this stream , each node determines its own command 's place which , in turns , allows the node to compute the value of its F&OP command . Since each node listens to all the F&OP commands , each node also maintains the current value of the F&OP targets , thus avoiding the need for a central entity to maintain the value . The key to this decentralized scheme 's working is that each node hears all nodes ' messages in the same order as every other node . Recall that we are interested in how the interconnection network 's structure provides support for the implementations . Our second observation concerns how the broadcast ring 's property of ordering the messages plays a role in the implementations . In the case of locks , the ordering automatically forms the service order of peer groups ; thus , the interconnection network takes the responsibility for arbitrating among the queue of lock requests . For barriers , the ordering plays no role . For the F&OP command , the ordering automatically creates the -LRB- arbitrary -RRB- serial ordering needed for each node to calculate locally the value of its F&OP -LRB- V , e -RRB- command . The third observation concerns the space requirements . For the lock design , note that no matter how large the distributed queue of lock requests becomes , each node only needs to store four peer groups of information -LRB- per distinct lock -RRB- . For the barrier , each node stores only one group per barrier no matter how many nodes participate in the barrier . For the F&OP , each node also stores only one group per target V no matter how many nodes issue F&OP commands to that target . Thus , for each synchronization primitive implementation , the data structure storage requirements are constant per node per instance of the primitive no matter how many nodes participate in operations on the primitive . The fourth observation regards the number of messages generated for each primitive and how that number compares with the minimum number needed . For either a R/S or W/E lock request , a node generates only one message -LRB- the minimum number -RRB- . When a node releases a lock , only one message is generated -LRB- the minimum number -RRB- . No additional messages are generated for the lock actually to be granted to the next requester since the nodes are monitoring the request and release traffic to determine when they should be granted the lock . For the barrier operation , one message is generated to initialize the barrier , which is the minimum . One message is generated for each node arriving at the barrier , which is the minimum . No additional messages are generated when barrier completion occurs since each node calculates for itself when barrier completion occurs . For the F&OP command , each node generates a message when issuing the command , which is the minimum . No additional messages are needed for the node to calculate the result of its F&OP or keep the the target V updated since each node observes all the command traffic and performs those calculations locally . Together , the third and fourth observations demonstrate that the implementation of these synchronization primitives are scalable in both space and time requirements . 3.2 Implementation-dependent comments In -LSB- 1 -RSB- we assumed an optical fiber implementation of the broadcast ring ; this assumption allows us to make two implementation-dependent observations . First , the previous discussion has implicitly assumed that all of a given synchronization command 's traffic is carried on only one broadcast ring . The high bandwidth of optical fibers -LRB- on the order of ten Terahertz -RRB- makes it possible to have , as the optical technology matures , large numbers -LRB- on the order of a hundred -RRB- of high speed -LRB- Gbit or more -RRB- channels . Thus , if the lock command -LRB- or barrier or F&OP command -RRB- traffic is heavy enough , one might allocate several high speed channels to alleviate the communication bottleneck . The algorithms previously described do not need modification to work properly if the command traffic is split over several channels as long as one restriction is observed . That restriction is that all command traffic pertaining to one instance of a given synchronization command must be carried on one channel . An additional implication of splitting a given type of command traffic over several channels is that the node will need to replicate the hardware that implements the monitoring algorithms . As a second observation -LRB- that assumes using optical fibers -RRB- , we show some minimum and maximum execution times of these synchronization primitives for various system configurations -LRB- the details of the derivation of these times are given in -LSB- 1 -RSB- -RRB- . Tables 1 , 2 , and 3 list these execution times for idle and busy locks , last arrival and simultaneous arrival barriers , and single and multiple F&OP commands respectively . These execution times are listed for small and large system sizes -LRB- 50 and 100 nodes respectively -RRB- and for small and large physical scales -LRB- end-to-end propagation times of 50 nsec , or 10 m , and 500 nsec , or 100 m , respectively -RRB- . An idle lock is one not being held when the node requests it ; a busy lock is one for which the node waits before being granted it . A last arrival barrier is one for which all nodes but one have previously arrived ; a simultaneous arrival barrier is one for which some arbitrary number of nodes arrive together at the barrier . A single F&OP command situation is one in which only one node issues the F&OP for a particular target V ; in the multiple F&OP command situation , some number of nodes P issue the F&OP command to the same target V . These execution times indicate that even for large system sizes and large physical scales , the raw execution times of these three synchronization primitives is on the order of from a few microseconds to a few tens of microseconds . 4 Conclusions We have presented efficient implementation techniques for some well-known synchronization operations that rely on local decisions for determining the outcome of such operations . We have shown that our implementation incurs exactly the minimum number of messages to perform these operations . Further , the space requirement at each node for implementing these algorithms is a constant independent of the number of participating processors . Thus our implementation is scalable in terms of both time and space requirements . These algorithms depend upon the interconnection network being a broadcast ring having three particular properties . These algorithms demonstrate that it is useful to consider how an interconnection network 's structure -LRB- and resulting properties -RRB- can aid the implementation of useful parallel programming constructs . By assuming a particular implementation technology for the broadcast ring -LRB- optical fibers -RRB- , we have also presented some results showing the raw execution performance of these commands to be reasonable over even large system sizes and physical scales . The next research step is to consider the various practical issues and concomitant ramifications of constructing systems using these techniques . There are several different questions to be addressed in this next step . What are the engineering issues in building the broadcast ring in either electronic or optical hardware ? How many nodes can be supported in either technology ? What are the design issues involved in storing the replicated data structures distinct from the architecture 's shared memory ? Should local hardware tables be used for this storage ? Are there existing parallel systems -LRB- e.g. , the KSR machine -LSB- 3 -RSB- -RRB- having a ring-style interconnection network which could utilize these algorithms ? Are there other interconnection network structures which can be utilized in a similar manner ? Finally , how do \ real '' parallel programs perform when using these synchronization primitives ' implementations versus using other -LRB- previously suggested -RRB- software and hardware implementations ? References -LSB- 1 -RSB- M. H. Davis , Jr. . Optical Waveguides in General Purpose Parallel Computers . PhD thesis , College of Computing , Georgia Institute of Technology , Atlanta , GA 30332-0280 , Dec. 1992 . Available as Technical Report GIT-CC-93 / 06 . -LSB- 2 -RSB- A. Gottlieb and C. Kruskal . Coordinating parallel processors : A partial unification . Computer Architecture News , pages 16 -LCB- 24 , Oct. . 1981 . -LSB- 3 -RSB- Kendall Square Research Corp. , Waltham , Massachusetts . Kendall Square Research Technical Summary , 1992 . -LSB- 4 -RSB- J. M. Mellor-Crummey and M. L. Scott . Algorithms for scalable synchronization on shared-memory multiprocessors . ACM Transactions on Computer Systems , 9 -LRB- 1 -RRB- :21 -LCB- 65 , February 1991 . oeoeoe - - - Node 1 U T B Node i U T B Node N U T B fflfflfflffl 1 's msg i 's msg N 's msgfflffl o - oe Figure 1 : The broadcast ring 's properties . P3 P1 P4 P5 P7 P2 P6 P8 P9 R/S req . W/E req . R/S req . W/E req . - - - Figure 2 : Lock request peer groups . LOCK-ID my-type my-id my-status prior-type prior-id prior-count most-recent-type most-recent-id most-recent-count next-recent-type next-recent-id next-recent-count Figure 3 : The data structure associated with each lock . ID participation counter Figure 4 : The data structure which represents the state of each barrier . V-current V-mine Figure 5 : The data structure associated with each target V. Execution Time -LRB- nsec -RRB- Idle Busy N o -LRB- nsec -RRB- min max min max 50 50 70 2555 170 3910 50 500 70 2555 670 4360 500 50 70 25055 170 37660 500 500 70 25055 1070 38110 Table 1 : Minimum and maximum execution times for idle and busy locks . N is the system size , and o is the physical scale . Execution Time -LRB- nsec -RRB- Last Simul N o -LRB- nsec -RRB- min max min max 50 50 70 2605 670 2655 50 500 70 3055 670 3555 500 50 70 25105 6295 25155 500 500 70 25555 6295 26055 Table 2 : Minimum and maximum execution times for last arrival and simultaneous arrival barriers . N is the system size , and o is the physical scale . Execution Time -LRB- nsec -RRB- Single Multiple N o -LRB- nsec -RRB- min max min max 50 50 70 2555 170 2555 50 500 70 2555 670 2555 500 50 70 25055 170 25055 500 500 70 25055 1070 25055 Table 3 : Minimum and maximum execution times for single command and multiple command F&OP . N is the system size , and o is the physical scale . A Lock Monitoring Algorithms if -LRB- COMMAND == request -RRB- : if -LRB- SENDER ! = self -RRB- : / / Just update most-recent and next-recent peer groups . switch REQUEST_TYPE : case w/e : / / Automatically starts new Peer Group . next-recent-type : = most-recent-type ; next-recent-id : = most-recent-id ; next-recent-count : = most-recent-count ; most-recent-type : = w/e ; most-recent-id : = REQUEST-id ; most-recent-count : = 1 ; break ; case r/s : if -LRB- most-recent-type == -LRB- none . OR . w/e -RRB- -RRB- : / / Starts a new Peer Group . next-recent-type : = most-recent-type ; next-recent-id : = next-recent-id ; next-recent-count : = next-recent-count ; most-recent-type : = r/s ; most-recent-id : = REQUEST-id ; most-recent-count : = 1 ; else : / / Since most-recent Peer Group is R/S , this request joins it . most-recent-count + + ; endif . break ; endswitch . endif . endif . Figure 6 : The algorithm which nodes follow for processing Lock Requests from other nodes . if -LRB- COMMAND == request -RRB- : if -LRB- SENDER == self -RRB- : / / Must define my and prior peer groups , as well as update most-recent and next-recent peer groups . switch REQUEST_TYPE : case w/e : / / Automatically starts new Peer Group . / / First create prior group from most-recent group : prior-type : = most-recent-type ; prior-id : = most-recent-id ; prior-count : = most-recent-count ; / / Update next-recent and most-recent groups : next-recent-type : = most-recent-type ; next-recent-id : = most-recent-id ; next-recent-count : = most-recent-count ; most-recent-type : = w/e ; most-recent-id : = REQUEST-id ; most-recent-count : = 1 ; / / Create my group : my-type : = most-recent-type ; my-id : = most-recent-id ; if -LRB- prior-type == none -RRB- : my-status : = granted ; else : my-status : = pending ; endif . break ; First part -LRB- of three -RRB- of Fig. 7 . Figure 7 : The algorithm which nodes follow for processing Lock Requests from themselves . case r/s : if -LRB- most-recent-type == -LRB- none . OR . w/e -RRB- : / / Form new Peer Group : / / First create prior group from most-recent : prior-type : = most-recent-type ; prior-id : = most-recent-id ; prior-count : = most-recent-count ; / / Update next-recent and most-recent groups : next-recent-type : = most-recent-type ; next-recent-id : = most-recent-id ; next-recent-count : = most-recent-count ; most-recent-type : = r/s ; most-recent-id : = REQUEST-id ; most-recent-count : = 1 ; / / Create my group : my-type : = most-recent-type ; my-id : = most-recent-id ; if -LRB- prior-type == none -RRB- : my-status : = granted ; else : my-status : = pending ; endif . Second part -LRB- of three -RRB- of Fig. 7 . else : / / Since most recent Peer Group is R/S , this request / / joins it . / / First create my group : prior-type : = next-recent-type ; prior-id : = next-recent-id ; prior-count : = next-recent-count ; / / Update most recent group : most-recent-count + + ; / / Create my group : my-type : = most-recent-type ; my-id : = most-recent-id ; if -LRB- prior-type == none -RRB- : my-status : = granted ; else : my-status : = pending ; endif . endif . break ; endswitch . endif . endif . Third part -LRB- of three -RRB- of Fig. 7 . if -LRB- COMMAND == w/e-unlock -RRB- : if -LRB- REQUEST-id == next-recent-id -RRB- : next-recent-type : = none ; endif . if -LRB- REQUEST-id == most-recent-id -RRB- : most-recent-type : = none ; endif . if -LRB- REQUEST-id == prior-id -RRB- : prior-type : = none ; if -LRB- my-type == -LRB- w/e . OR . r/s -RRB- : my-status : = granted ; endif . endif . if -LRB- REQUEST-id == my-id -RRB- : / / Change my-type to indicate that node / / no longer holds this lock | there is no / / my-count field to update . my-type : = none ; endif . endif . Figure 8 : The algorithm which nodes follow for processing W/E UNLOCK commands . if -LRB- COMMAND == r/s-unlock -RRB- : if -LRB- REQUEST-id == next-recent-id -RRB- : next-recent-count -- ; if -LRB- next-recent-count == 0 -RRB- : next-recent-type : = none ; endif . endif . if -LRB- REQUEST-id == most-recent-id -RRB- : most-recent-count -- ; if -LRB- most-recent-count == 0 -RRB- : most-recent-type : = none ; endif . endif . if -LRB- REQUEST-id == prior-id -RRB- : prior-count -- ; if -LRB- prior-count == 0 -RRB- : prior-type : = none ; if -LRB- my-type == -LRB- w/e . OR . r/s -RRB- : my_status : = granted ; endif . endif . endif . if -LRB- REQUEST-id == my-id -RRB- : / / Change my-type to indicate that node / / no longer holds this lock | there is no / / my-count field to update . my-type : = none ; endif . endif . Figure 9 : The algorithm which nodes follow for processing R/S UNLOCK commands . B Barrier monitoring algorithms switch command : case initialize : participation : = N ; counter : = 0 ; break ; case reach : counter + + ; break ; endcase ; endswitch . Figure 10 : The monitoring algorithm that each node follows in processing barrier commands . C F&OP monitoring algorithms if -LRB- SENDER == self -RRB- : V-mine : = V-current ; notify CPU that the F&OP has completed by returning V-mine ; endif ; V-current : = -LRB- V-current OP e -RRB- . Figure 11 : The monitoring algorithm each node follows in the hybrid F&OP scheme .

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

broadcast ring interconnect
distributed queues
local decision algorithms
scalable synchronization primitives
shared memory multiprocessor