Search results matching tags 'SQL Server' and 'Data',Data&orTags=0Search results matching tags 'SQL Server' and 'Data'en-USCommunityServer 2.1 SP2 (Build: 61129.1)Data Mining Algorithms – Hierarchical Clustering, 28 Mar 2015 06:33:25 GMT21093a07-8b3d-42db-8cbf-3350fcbf5496:58301Dejan Sarka<p>Clustering is the process of grouping the data into classes or clusters so that objects within a cluster have high similarity in comparison to one another, but are very dissimilar to objects in other clusters. Dissimilarities are assessed based on the attribute values describing the objects.</p> <p>There are a large number of clustering algorithms. The major methods can be classified into the following categories.</p> <ul> <li><strong>Partitioning methods</strong>. A partitioning method constructs K partitions of the data, which satisfy the following requirements: (1) each group must contain at least one object and (2) each object must belong to exactly one group. Given the initial K number of partitions to construct, the method creates initial partitions. It then uses an iterative relocation technique that attempts to improve the partitioning by moving objects from one group to another. There are various kinds of criteria for judging the quality of the partitions. Some most popular include the k-means algorithm, where each cluster is represented by the mean value of the objects in the cluster, and the k-medoids algorithm, where each cluster is represented by one of the objects located near the center of the cluster.</li> <li><strong>Hierarchical methods</strong>. A hierarchical method creates a hierarchical decomposition of the given set of data objects. These methods are agglomerative or divisive. The agglomerative (bottom-up) approach starts with each object forming a separate group. It successively merges the objects or groups close to one another, until all groups are merged into one. The divisive (top-down) approach starts with all the objects in the same cluster. In each successive iteration, a cluster is split up into smaller clusters, until eventually each object is in one cluster or until a termination condition holds.</li> <li><strong>Density-based methods</strong>. Methods based on the distance between objects can find only spherical-shaped clusters and encounter difficulty in discovering clusters of arbitrary shapes. So other methods have been developed based on the notion of density. The general idea is to continue growing the given cluster as long as the density (number of objects or data points) in the “neighborhood” exceeds some threshold; that is, for each data point within a given cluster, the neighborhood of a given radius has to contain at least a minimum number of points.</li> <li><strong>Model-based methods</strong>. Model-based methods hypothesize a model for each of the clusters and find the best fit of the data to the given model. A model-based technique might locate clusters by constructing a density function that reflects the spatial distribution of the data points. Unlike conventional clustering, which primarily identifies groups of like objects, this conceptual clustering goes one step further by also finding characteristic descriptions for each group, where each group represents a concept or a class.</li> </ul> <p>A hierarchical clustering model training typically starts by calculating a <strong>distance matrix</strong> – a matrix with distances between data points in a multidimensional hyperspace, where each input variable defines one dimension of that hyperspace. Distance measure can be a geometrical distance or some other, more complex measure. A <strong>dendrogram</strong> is a tree diagram frequently used to illustrate the arrangement of the clusters produced by hierarchical clustering. Dendrograms are also often used in computational biology to illustrate the clustering of genes or samples. The following set of pictures shows the process of building an agglomerative hierarchical clustering dendrogram.</p> <p><a href=""><img title="image" style="border-top:0px;border-right:0px;border-bottom:0px;border-left:0px;display:inline;" border="0" alt="image" src="" width="202" height="244" /></a> <a href=""><img title="image" style="border-top:0px;border-right:0px;border-bottom:0px;border-left:0px;display:inline;" border="0" alt="image" src="" width="136" height="52" /></a><a href=""><img title="image" style="border-top:0px;border-right:0px;border-bottom:0px;border-left:0px;display:inline;" border="0" alt="image" src="" width="202" height="244" /></a> <a href=""><img title="image" style="border-top:0px;border-right:0px;border-bottom:0px;border-left:0px;display:inline;" border="0" alt="image" src="" width="136" height="52" /></a><a href=""><img title="image" style="border-top:0px;border-right:0px;border-bottom:0px;border-left:0px;display:inline;" border="0" alt="image" src="" width="202" height="244" /></a> </p> <p><a href=""><img title="image" style="border-top:0px;border-right:0px;border-bottom:0px;border-left:0px;display:inline;" border="0" alt="image" src="" width="136" height="52" /></a></p> <p><a href=""><img title="image" style="border-top:0px;border-right:0px;border-bottom:0px;border-left:0px;display:inline;" border="0" alt="image" src="" width="202" height="244" /></a> <a href=""><img title="image" style="border-top:0px;border-right:0px;border-bottom:0px;border-left:0px;display:inline;" border="0" alt="image" src="" width="136" height="52" /></a><a href=""><img title="image" style="border-top:0px;border-right:0px;border-bottom:0px;border-left:0px;display:inline;" border="0" alt="image" src="" width="202" height="244" /></a> <a href=""><img title="image" style="border-top:0px;border-right:0px;border-bottom:0px;border-left:0px;display:inline;" border="0" alt="image" src="" width="136" height="52" /></a><a href=""><img title="image" style="border-top:0px;border-right:0px;border-bottom:0px;border-left:0px;display:inline;" border="0" alt="image" src="" width="202" height="244" /></a> </p> <p>Cluster analysis segments a heterogeneous population into a number of more homogenous subgroups or clusters. Typical usage scenarios include:</p> <ul> <li>Discovering distinct groups of customers</li> <li>Identifying groups of houses in a city</li> <li>In biology, deriving animal and plant taxonomies</li> <li>Can even make predictions once the clusters are built and distribution of a target variable in the clusters is calculated. </li> </ul>PASS SQL Saturday #356 Slovenia Recapitulation, 15 Dec 2014 11:28:11 GMT21093a07-8b3d-42db-8cbf-3350fcbf5496:57402Dejan Sarka<p>So the <a href="">event</a> is over. I think I can say for all three organizers, <a href="">Mladen Prajdić</a>, <a href="">Matija Lah</a>, and <a href="">me</a>, that we are tired now. However, we are extremely satisfied. It was a great event. First few numbers and comparison with <a href="">SQL Saturday #274</a>, the first SQL Saturday Slovenia event that took place last year.</p> <table cellspacing="0" cellpadding="2"> <tr> <td> <p align="center"></p> </td> <td> <p align="center">SQL Saturday #274</p> </td> <td> <p align="center">SQL Saturday #356</p> </td> </tr> <tr> <td> <p align="center">People</p> </td> <td> <p align="center">135</p> </td> <td> <p align="center">220</p> </td> </tr> <tr> <td> <p align="center">Show rate</p> </td> <td> <p align="center">~87%</p> </td> <td> <p align="center">~95%</p> </td> </tr> <tr> <td> <p align="center">Proposed sessions</p> </td> <td> <p align="center">40</p> </td> <td> <p align="center">82</p> </td> </tr> <tr> <td> <p align="center">Selected sessions</p> </td> <td> <p align="center">15</p> </td> <td> <p align="center">24</p> </td> </tr> <tr> <td> <p align="center">Selected speakers</p> </td> <td> <p align="center">14</p> </td> <td> <p align="center">23</p> </td> </tr> <tr> <td> <p align="center">Countries</p> </td> <td> <p align="center">12</p> </td> <td> <p align="center">16</p> </td> </tr> </table> <p>The numbers nearly doubled. We are especially proud of the show rate; with 95%, this is much better than average for a free event, and probably the highest so far for a SQL Saturday. We asked registered attendees to be fair and to unregister if they know they can’t attend the event in order to make room for those from the waiting list. An old Slovenian proverb says “A nice word finds a nice place”, and it works. 36 registered attendees unregistered. Therefore, we have to thank to both, the attendees of the event and those who unregistered.</p> <p>Of course, as always, we also need to thank to all of the speakers, sponsors and volunteers. All volunteers were very helpful; however, I would like to especially point out <a href="">Saša Mašič</a>. Her work goes well beyond simple volunteering. I must mention also the <a href="">FRI</a>, the Faculty of Computer and Information Science, where the event was hosted for free. It is also worth mentioning that we are lucky to live in <a href="">Ljubljana</a>, such a beautiful city with extremely nice inhabitants who like to enjoy good food, hanging around and mingling, and long parties. Because of that we could be sure in advance that both speakers and attendees from other countries would enjoy spending time here also outside the event, that they would feel safe, and get help whenever they would need it.</p> <p>From the organizational perspective, we tried to do our best, and we hope that everything was OK for speakers, sponsors, volunteers, and attendees. Thank you all!</p>SQL Saturday #274 Slovenia Recapitulation, 23 Dec 2013 11:49:50 GMT21093a07-8b3d-42db-8cbf-3350fcbf5496:52222Dejan Sarka<p><font color="#06bd02" size="3"><strong>Pure success!</strong></font></p> <p>I could simply stop here. However, I want to mention again everybody involved in this, and also some who were unfortunately missing.</p> <p>First of all, PASS is the organization that defined SQL Saturdays. And apparently the idea works<img class="wlEmoticon wlEmoticon-smile" style="border-top-style:none;border-left-style:none;border-bottom-style:none;border-right-style:none;" alt="Smile" src="" /></p> <p>I have to thank again to all of the speakers. Coming to share your amazing knowledge is something we really appreciate. The presentations were great, from the technical and other perspectives.</p> <p>Of course, we could not do the event without sponsors. I am not going to enlist all of them again; I will just mention the host, <a href="">pixi* labs</a>, the company that hosted the event and who's&#160; employees helped with all of the organization. In addition, I need to mention <a href="">Vina Kukovec</a>. Boštjan Kukovec, an old member of Slovenian SQL Server and Developers users group, organized free wine tasting after the event. And what a wine it is!</p> <p>Finally, thanks to all attendees for coming. We had approximately 85% show up rate; only 15% or the registered attendees didn’t come. This is an incredible result, worldwide! And from the applause after the raffle, when we closed the vent (and started wine tasting), I conclude that the attendees were very satisfied as well.</p> <p>I want to mention three people that wanted to come, but run out of luck this time. Peter Stegnar from pixi* labs was the one that immediately offered the venue, and permeated with his enthusiasm also other pixi* labs members. Due to family reasons he couldn’t join us to see the results of his help. Tobiasz Janusz Koprowski wanted to speak, organized his trip, looked forward to join us; however, just couple of days before the event he had to cancel because of some urgent work at customer’s site. And what to say about Kevin Boles? He really tried hard to come. Think of it, he was prepared to come from USA! he was already on the airport, when his flight got cancelled due to technical problems. We were in constant touch Friday evening. He managed to change the flight, went to the gate, but was not admitted to the plane. because there was only seven minutes left till take off. Catching next flights would not make any sense anymore, because he would come too late anyway. He really did the best he could do, he just didn’t have enough luck this time. Peter, Tobiasz, and Kevin, thank you for your enthusiasm, we seriously missed you, and we certainly hope we will meet on our next event!</p>How Does the Cloud Change a Database Administrator’s Job?, 29 Jan 2013 15:08:32 GMT21093a07-8b3d-42db-8cbf-3350fcbf5496:47385BuckWoody<p>I recently<a href="" target="_blank"> posted a blog entry on how cloud computing would change the Systems Architect&rsquo;s role in an organization</a>. In a way, the Systems Architect has the easiest transition to a new way of using computing technologies. In fact, that&rsquo;s actually part of the job description.&nbsp;I mentioned that a Systems Architect has three primary vectors to think about for cloud computing, as it applies to what they should do:</p> <ol> <li><span style="color:#0000ff;">Knowledge - Which options are available to solve problems, and what are their strengths and weaknesses.</span></li> <li><span style="color:#0000ff;">Experience - What has the System Architect seen and worked with in the past.</span></li> <li><span style="color:#0000ff;">Coordination - A system design is based on multiple factors, and one person can't make all the choices. There will need to be others involved at every level of the solution, and the Systems Architect will need to know who those people are and how to work with them.</span></li> </ol> <h1>The Database Administrator Role</h1> <p>But a Database Administrator (DBA) is probably one of the harder roles to think about when it comes to cloud computing. First, let&rsquo;s define what a Database Administrator usually thinks about as part of their job:</p> <ul> <li><span style="color:#993300;">Planning, Installing and Configuring a Database Platform</span></li> <li><span style="color:#993300;">Planning, designing and creating databases</span></li> <li><span style="color:#993300;">Planning, designing and implementing High Availability and Disaster Recovery for each database (HADR) based on requirements for its workload</span></li> <li><span style="color:#993300;">Maintaining and monitoring the database platform</span></li> <li><span style="color:#993300;">Implementing performance tuning on the databases based on monitoring</span></li> <li><span style="color:#993300;">Re-balancing workloads across database servers based on monitoring</span></li> <li><span style="color:#993300;">Securing databases platforms and individual databases based on requirements and implementation</span></li> </ul> <p>That&rsquo;s just a short list, and each of those unpacks into a larger set of tasks.</p> <p>The issue is that<em> I&rsquo;ve never actually met a DBA that does all of those things</em>, or <strong>just</strong> all of those things. Many times they do much more, sometimes the systems are so large they specialize on just a few of them.</p> <p>And as you can see from the list, some of these areas are shared with other roles. For instance, in some shops, the DBA plans, purchases, sets up and configures the hardware for database servers. In others that&rsquo;s done<br />by the Infrastructure Team. In some shops the DBA designs databases from software requirements, and in others the developers do that &ndash; or perhaps it&rsquo;s done as a joint effort. The same holds true for database code &ndash; sometimes the<br />DBA does it, other times the developer, and still others it&rsquo;s a shared task.</p> <p>In fact, you could argue that there are few other roles in IT where the roles are so intermixed. Also, the DBA works with software the company develops, and software the company buys. They work with hardware, networking, security and software. There are certain aspects of design and tuning that are outside the purview of some of those things, and inside the others.</p> <p>With all of these variables, simply telling a DBA that they should &ldquo;use the cloud&rdquo; is not the proper approach.</p> <h1>How the Cloud Changes Things</h1> <p>To be sure, the DBA has the same vectors as the Systems Architect. They need to educate themselves on the options within this new option (<span style="color:#0000ff;">Knowledge</span>), try a few test solutions out (<span style="color:#0000ff;">Experience</span>) and of course work with others on various parts of the implementation (<span style="color:#0000ff;">Coordination</span>). But it goes beyond that.</p> <p><a href="" target="_blank">There are three big buckets of cloud computing</a>, dealing with simply using a Virtual Machine (IaaS) to writing code without worrying about the virtualization or even the operating system (PaaS) and using software that&rsquo;s already written and being delivered via an Application Programming Interface (API). Each of these has so many options and configurations that it&rsquo;s often better to think about the problem you&rsquo;re trying to solve rather than all of the technology within a given area - although some of that is certainly necessary anyway.&nbsp;</p> <h2>Database Platform Architecture</h2> <p>I&rsquo;ll start with when the DBA should even consider cloud computing for a solution. Once again, it&rsquo;s not an &ldquo;all or nothing&rdquo; paradigm, where you either run something on premises or in the cloud &ndash; it&rsquo;s often a matter of selecting the right components to solve a problem.&nbsp; In my design sessions with DBA&rsquo;s I break these down into three big areas where they might want to consider the cloud &ndash;and then we talk about how to implement each one:</p> <ol> <li><span style="color:#0000ff;">Audiences</span></li> <li><span style="color:#0000ff;">HADR</span></li> <li><span style="color:#0000ff;">Data Services</span></li> </ol> <h3>Audiences</h3> <p>If the users of your database systems all sit in the same facility, you own the servers and networking, and the application servers are separate from the database server, it doesn&rsquo;t usually make sense to take that database workload and place it on Windows Azure &ndash; or any other cloud provider. The latency alone prevents a satisfactory performance profile, and in some cases won&rsquo;t work at all. It doesn&rsquo;t matter if the cloud solution is cheaper or easier &ndash; if you&rsquo;re moving a lot of data every second between an on-premises system and the cloud it won&rsquo;t work well.</p> <p>However &ndash; if your users are in multiple locations, especially globally, or you have a mix of company and external customer users, it might make sense to evaluate a shared data location. You still need to consider the implications of how much data the application server pushes back and forth, but you may be able to locate both the application server and SQL Server in an IaaS role. Assuming the data sent to the final client will work across public Internet channels, there may be a fit. There are security implications, but unless you have point-to-point connections for your current solution you&rsquo;re faced with the same security questions on both options.</p> <p>Your audience might also be developers looking for a way to quickly spin up a server and then turn it down when they are done, paying for the time and not the hardware or licenses. This is also a prime case for evaluating IaaS. And there are others that you'll find in your own organization as you work through the requirements you have.&nbsp;&nbsp;</p> <p>Resources: Windows Azure Virtual Machines: <a href=""></a>&nbsp;and&nbsp;<span style="color:#993300;">Windows Azure SQL Server Virtual Machines</span>: <a href=""></a></p> <h3>HADR</h3> <p>The next possible place to consider using cloud computing with SQL Server is as a part of your High Availability and Disaster Recovery plans. In fact, this is the most common use I see for cloud computing and the Database Administrator. The key is the Recovery Point Objective (RPO) and Recovery Time Objective (RTO). Based on each application&rsquo;s requirements, you may find that using Windows Azure or even supplementing your current plan is<br />the right place to evaluate options. I&rsquo;ve covered this use-case in more detail in another article.</p> <p><span style="color:#993300;">References: SQL Server High Availability and Disaster Recovery options with Windows Azure</span>: <a href=""></a></p> <h3>Data Services</h3> <p>Windows Azure, along with other cloud providers, offers another way to design, create and consume data. In this use-case, however, the tasks DBA&rsquo;s normally perform for sizing, ordering and configuring a system don&rsquo;t apply.</p> <p>With Windows Azure SQL Databases (the artist formerly known as SQL Azure), you can simply create a database and begin using it. There are places where this fits and others where it doesn&rsquo;t, and there are differences, limitations and enhancements, so it isn&rsquo;t meant as replacement for what you could do with &ldquo;Full-up&rdquo; SQL Server on a Windows Azure Virtual Machine or an on-premises Instance. If a developer needs an Relational Database Management<br />(RDBMS) data store for a web-based application, then this might be a perfect fit.</p> <p>But there is more to data services than Windows Azure SQL Databases. Windows Azure also offers MySQL as a service, RIAK and MongoDB (among others) and even Hadoop for larger distributed data sets. In addition you can use Windows Azure Reporting Services, and also tap into datasets and data functions in the Windows Azure Marketplace.</p> <p>The key for the DBA with this option is that you <em>will</em> have to do a little investigation this time, and potentially without a specific workload in mind this time. I think that&rsquo;s acceptable thing to ask &ndash; DBA&rsquo;s constantly keep up with data processing trends, and most will consider different ways to solve a problem.</p> <p><span style="color:#993300;">References:</span></p> <p><span style="color:#993300;">Windows Azure SQL Databases</span>: <a href="" target="_blank"></a></p> <p><span style="color:#993300;">Windows Azure Reporting Services</span>: <a href="" target="_blank"></a></p> <p><span style="color:#993300;">HDInsight Service (Hadoop on Azure): </span><a href="" target="_blank"></a></p> <p><span style="color:#993300;">MongoDB Offerings on Windows Azure</span>: <a href="" target="_blank"></a></p> <p><span style="color:#993300;">Windows Azure Marketplace</span>: <a href="" target="_blank"></a></p> <p>&nbsp;</p>The Data Scientist, 15 Nov 2011 15:00:18 GMT21093a07-8b3d-42db-8cbf-3350fcbf5496:39814BuckWoody<p>A new term - well, perhaps not that new - has come up and I’m actually very excited about it. The term is Data Scientist, and since it’s new, it’s fairly undefined. I’ll explain what I <em>think</em> it means, and why I’m excited about it.</p> <p>In general, I’ve found the term deals at its most basic with analyzing data. Of course, we all do that, and the term itself in that definition is redundant. There is no science that I know of that does not work with analyzing lots of data. But the term seems to refer to more than the common practices of looking at data visually, putting it in a spreadsheet or report, or even using simple coding to examine data sets. </p> <p>The term Data Scientist (as far as I can make out this early in it’s use) is someone who has a strong understanding of data sources, relevance (statistical and otherwise) and processing methods as well as front-end displays of large sets of complicated data. Some - but not all - Business Intelligence professionals have these skills. In other cases, senior developers, database architects or others fill these needs, but in my experience, many lack the strong mathematical skills needed to make these choices properly. </p> <p>I’ve divided the knowledge base for someone that would wear this title into three large segments. It remains to be seen if a given Data Scientist would be responsible for knowing all these areas or would specialize. There are pretty high requirements on the math side, specifically in graduate-degree level statistics, but in my experience a company will only have a few of these folks, so they are expected to know quite a bit in each of these areas. </p> <p><strong>Persistence</strong></p> <p>The first area is finding, cleaning and storing the data. In some cases, no cleaning is done prior to storage - it’s just identified and the cleansing is done in a later step. This area is where the professional would be able to tell if a particular data set should be stored in a Relational Database Management System (RDBMS), across a set of key/value pair storage (NoSQL) or in a file system like HDFS (part of the Hadoop landscape) or other methods. Or do you examine the stream of data without storing it in another system at all? </p> <p>This is an important decision - it’s a foundation choice that deals not only with a lot of expense of purchasing systems or even using Cloud Computing (PaaS, SaaS or IaaS) to source it, but also the skillsets and other resources needed to care and feed the system for a long time. The Data Scientist sets something into motion that will probably outlast his or her career at a company or organization.</p> <p>Often these choices are made by senior developers, database administrators or architects in a company. But sometimes each of these has a certain bias towards making a decision one way or another. The Data Scientist would examine these choices in light of the data itself, starting perhaps even before the business requirements are created. The business may not even be aware of all the strategic and tactical data sources that they have access to. </p> <p><strong>Processing</strong></p> <p>Once the decision is made to store the data, the next set of decisions are based around how to process the data. An RDBMS scales well to a certain level, and provides a high degree of ACID compliance as well as offering a well-known set-based language to work with this data. In other cases, scale should be spread among multiple nodes (as in the case of Hadoop landscapes or NoSQL offerings) or even across a Cloud provider like Windows Azure Table Storage. In fact, in many cases - most of the ones I’m dealing with lately - the data should be split among multiple types of processing environments. This is a newer idea. Many data professionals simply pick a methodology (RDBMS with Star Schemas, NoSQL, etc.) and put all data there, regardless of its shape, processing needs and so on. </p> <p>A Data Scientist is familiar not only with the various processing methods, but how they work, so that they can choose the right one for a given need. This is a huge time commitment, hence the need for a dedicated title like this one. </p> <p><strong>Presentation</strong></p> <p>This is where the need for a Data Scientist is most often already being filled, sometimes with more or less success. The latest Business Intelligence systems are quite good at allowing you to create amazing graphics - but it’s the data behind the graphics that are the most important component of truly effective displays. </p> <p>This is where the mathematics requirement of the Data Scientist title is the most unforgiving. In fact, someone without a good foundation in statistics is not a good candidate for creating reports. Even a basic level of statistics can be dangerous. Anyone who works in analyzing data will tell you that there are multiple errors possible when data just seems right - and basic statistics bears out that you’re on the right track - that are only solvable when you understanding why the statistical formula works the way it does. </p> <p>And there are lots of ways of presenting data. Sometimes all you need is a “yes” or “no” answer that can only come after heavy analysis work. In that case, a simple e-mail might be all the reporting you need. In others, complex relationships and multiple components require a deep understanding of the various graphical methods of presenting data. Knowing which kind of chart, color, graphic or shape conveys a particular datum best is essential knowledge for the Data Scientist. </p> <p><strong>Why I’m excited</strong></p> <p>I love this area of study. I like math, stats, and computing technologies, but it goes beyond that. I love what data can do - how it can help an organization. I’ve been fortunate enough in my professional career these past two decades to work with lots of folks who perform this role at companies from aerospace to medical firms, from manufacturing to retail. </p> <p>Interestingly, the size of the company really isn’t germane here. I worked with one very small bio-tech (cryogenics) company that worked deeply with analysis of complex interrelated data. </p> <p>So&#160; watch this space. No, I’m not leaving Azure or distributed computing or Microsoft. In fact, I think I’m perfectly situated to investigate this role further. We have a huge set of tools, from RDBMS to Hadoop to allow me to explore. And I’m happy to share what I learn along the way. </p>Big Data and the Cloud - More Hype or a Real Workload?, 18 Oct 2011 13:57:36 GMT21093a07-8b3d-42db-8cbf-3350fcbf5496:39156BuckWoody<p>Last week Microsoft announced several new offerings for “Big Data” - and since I’m a stickler for definitions, I wanted to make sure I understood what that really means. What is “Big Data”? What size hard drive is that? After all, my laptop has 1TB of storage - is my laptop “Big Data”?</p> <p>There are actually a few definitions for this term, most notably those involving the <a href="" target="_blank">“Four V’s” Volume, Velocity, Variety and Variability</a>. Others <a href="" target="_blank">disagree with this</a> definition. I tend to try and get things into their simplest form, so I’m using this definition for myself:</p> <p align="center"><font color="#c0504d" size="3">Big data is defined as a <em>large set </em>of <em>computationally expensive </em>data that is <em>worked on simultaneously</em>.</font> </p> <p>Let me flesh that out a&#160; little. To be sure, “Big Data” has a larger size than say a few megabytes. The reason this is important is that it takes special hardware to be able to move large sets of data around, store it, process it and so on. (<font color="#c0504d">large set</font>)</p> <p>If you store a LOT of data, but only use a small portion of it at a time, that really isn’t super-hard to do. It’s mainly a storage issue at that point. But, if you do need to work with a large portion of the data at one time, then the memory, CPU and transfer components of the system have to adapt to be responsive - new ways to work with that data (game theory, knot-algorithms, map-reduce, etc.) need to be brought into play. (<font color="#c0504d">computationally expensive</font>)</p> <p>Once that data is loaded into the processing area (memory or whatever other mechanism is used) it must be worked on in parallel to come back in a reasonable time. You have two options here - you can scale the system up with more internal hardware (CPU’s, memory and so on) or you can scale it out to have multiple systems work on it at the same time using paradigms such as map/reduce and so on. Actually, when you lay this out in an architecture diagram, scale up or out doesn’t actually change the logical structure of the process - in scale out the network becomes the bus, and the nodes become more RAM and computing power. Of course, there are changes in code for how you stitch the workload back together. (<font color="#c0504d">worked on simultaneously</font>)</p> <p>So back to the original question. Is Big Data, as I have defined it here, a workload for Windows and SQL Azure? Absolutely! In fact, it’s probably one of the main workloads, and I believe it represents the latest, and perhaps also the earliest frontier of computing. Jim <a href="" target="_blank">Gray, a former researcher here at Microsoft and a hero of mine, was working on this very topic.</a> I believe as he did - all computing is simply an interface over data. </p> <p>Microsoft has multiple offerings on the topic of Big Data. In posts that follow from myself and my co-workers, we’ll explore when and where you use each one. Whether you are a data professional or a developer, this is the new frontier - <a href="" target="_blank">don’t wait to educate yourself</a> on how to leverage Big Data for your organization. </p> <p><strong>Hadoop on Windows Azure and SQL Server&#160; </strong>- Microsoft’s <a href="" target="_blank">partnership to include Hadoop workloads on Windows Azure</a> and <a href="" target="_blank">SQL Server/Parallel Data Warehouse (PDW)</a></p> <p><strong>LINQ to HPC </strong>- Microsoft’s High-Performance Computing SKU of <a href="" target="_blank">HPC is now in Azure</a></p> <p><strong>Windows Azure Table Storage </strong>- A <a href="" target="_blank">key/value pair type storage with full partitioning</a> that is immediately consistent, able to handle huge loads of data and works with any REST-compatible language</p> <p>&#160;<strong>Other offerings </strong>- Including the new <a href="" target="_blank">Data Explorer</a>, <a href="" target="_blank">Project Daytona (with a Big Data Toolkit for Scientists and researchers)</a>, <a href="" target="_blank">Power View</a> and more. </p> <p>The era of Big Data is here. And you can use Windows and SQL Azure to bring it to your organization. </p>SQL Azure Use Case: Shared Storage Application, 26 Apr 2011 13:33:50 GMT21093a07-8b3d-42db-8cbf-3350fcbf5496:35207BuckWoody<p><span style="font-size:x-small;"><em><span style="font-size:small;">This is one in a series of posts on when and where to use a distributed architecture design in your organization's computing needs. You can find the main post here: </span><a href=""><span style="font-size:small;"><u><font color="#800080"></font></u></span></a><span style="font-size:small;"> </span></em></span></p> <p><strong><span style="font-size:small;">Description:</span></strong></p> <p><span style="font-size:small;">On-premise data will be a part of computing for quite some time – perhaps permanently. Bandwidth requirements, security, or even financial considerations for large data sets often dictate that relational (on non-relational) systems will be maintained locally in many organizations, especially in enterprise computing. </span></p> <p><span style="font-size:small;">But distributed data systems are useful in many situations. Organizations may wish to store a portion of data off-site, either for sharing the data with other applications (including web-based applications) or as a supplement to a High-Availability and Disaster Recovery (HADR) strategy.</span></p> <span style="font-size:small;"> <p><strong><span style="font-size:small;">Implementation:</span></strong></p> <p><span style="font-size:small;">SQL Azure can be used to add an additional option to an HADR strategy by copying off portions (or all) of an on-premise database system.</span></p> <p><span style="font-size:small;"><a href=""><img style="background-image:none;border-bottom:0px;border-left:0px;padding-left:0px;padding-right:0px;display:inline;border-top:0px;border-right:0px;padding-top:0px;" title="sql-aHADR" border="0" alt="sql-aHADR" src="" width="298" height="181" /></a></span></p> <p><span style="font-size:small;">In this arrangement, on-premise systems remain as they are. Data is replicated using many technologies, such as SQL Server Integration Services (SSIS), scripts, or Microsoft’s Sync Framework to a SQL Azure database. This data can be kept “cold”, meaning that a manual process is required to bring the data back, or as a “warm” standby using connection string management in the application.</span></p> <p><span style="font-size:small;">Recently we architected a solution where a company kept a rolling two-week window of data replicated to SQL Azure using the <a href="" target="_blank">Sync Framework</a>. The application, a compiled EXE running on user’s systems, had a “switch connections” button, that allowed the users to take a laptop to another location, select that option, and continue working from anywhere they had Internet connectivity. This required forethought and planning, and did not replace their primary HADR systems, but it did allow them to continue operations in the case of a severe outage at multiple sites. Since they are an emergency services provider, this gave them the highest redundancy.</span></p> <p><span style="font-size:small;">Another option is to amalgamate data from disparate sources. </span></p> <p><span style="font-size:small;"><a href=""><img style="background-image:none;border-bottom:0px;border-left:0px;padding-left:0px;padding-right:0px;display:inline;border-top:0px;border-right:0px;padding-top:0px;" title="sql-aHyb" border="0" alt="sql-aHyb" src="" width="342" height="134" /></a></span></p> <p><span style="font-size:small;">In this arrangement, two or more data services (one of which is SQL Azure) are accessed by a single program. The program queries each system independently, and using LINQ a single query can work across all of the data, assuming there is some sort of natural or artificial “key” that can join the data sets together. The user programs simply view this single data set as a single data source, unaware of the underlying data sets. This allows great flexibility and agility in the downstream program. The upstream data sources can change as long as the elements are kept consistent.</span></p> <p><span style="font-size:small;">There are performance and security implications to amalgamated data systems, but if architected carefully they provide multiple benefits. A few of of these are that other systems can access the individual data sources, reporting is simplified and standardized, and multiple copies of data are eliminated.</span></p> <span style="font-size:small;"> <p><strong><span style="font-size:small;">Resources:</span></strong></p> <p><span style="font-size:small;">You can read more about the Sync Framework and SQL Azure here: <a href=""></a>&#160;</span></p> <p><span style="font-size:small;">If you are new to LINQ, you can find more resources on it here: <a href=""></a>&#160;</span></p> </span></span>Dealing with Data: Defining the Components to Tune, 02 Sep 2010 12:21:05 GMT21093a07-8b3d-42db-8cbf-3350fcbf5496:28535BuckWoody<p>I've been reading a fascinating article about the Large Hadron Collider, or LHC facility. It's a scientific research facility that houses a particle collider, which generates an incredible amount of data. Their original plan was to stream the data to tape, then sending the data to "islands" closer to the users, offloading the network as quickly as possible. But they found that the network could handle the streaming better than they thought - so they now stream the data directly to the users, saturating the network. It's a new way of thinking about moving the data around.</p> <p>Another interesting data concept is that they filter it before they store it. We're not talking trivial reductions here - they are filtering a petabyte (PB) of data a second to a gigabyte per second! That's incredible. In fact, an overwhelming majority of the CPU power there doesn't go to computing numbers and so on in the scientific exercises - it's used to filter the data. </p> <p>Most of us concern ourselves with data storage. We fret over space, the cost of drives, and backing up. But the LHC staff deals with that as well - but they are more concerned with network and CPU. To be sure, their data profile is different than yours or mine - but there are still things we can learn from their efforts. You can read the whole article yourself here: <a href=";utm_medium=rss&amp;utm_campaign=rss">;utm_medium=rss&amp;utm_campaign=rss</a></p>Permanent storage – a myth or a requirement?, 05 Aug 2010 14:12:31 GMT21093a07-8b3d-42db-8cbf-3350fcbf5496:27647BuckWoody<p>I'm looking over some data retention strategy documents for a company, I noticed that they want to keep some data "forever". Now, beyond the quibble of determining how to store something for eternity, how permanent can data really be? There are atomic (n, not ACID, atomic) ways of storing binary representations, but will the interpretation of those 0's and 1's really be there for more than a few decades? Think about the data stored on punch cards - where is that now?</p> <p>The answer is that it morphed into other storage along the way. If something really needs to be permanent, you need to think more about it's migration than the actual media it's stored on. That's what I'm recommending in this document - tag data sets to be migrated first, or at least considered in the migration planning. It's a simple pivot on storage versus programming, but an important one.</p>There is No Perfect Information, 07 Jul 2010 13:59:59 GMT21093a07-8b3d-42db-8cbf-3350fcbf5496:26807BuckWoody<p>I read a very interesting (and lengthy) article yesterday from Reuter’s (<a href="" target="_blank">you can find the whole thing here</a>) in which the author goes into some detail about people in the government that are self-described “data dogs”. I find it an interesting study in general, since I also value data as well – but the article finally seems to land on a topic that I thought had been put to bed.</p> <p>Several of the folks in the article seem to be under the belief that if we just had a little more data, we’d be able to predict the future with it, at least on an economic front. I don’t agree. Sure, more data is usually a good thing – but there is a limit. It’s different for each situation, but sometimes more data isn’t any better than a little data, especially when you can’t process it all anyway.</p> <p>My science background teaches me that there is no such thing as “perfect” information about any system – it’s a myth. So I was surprised at finding very intelligent people chasing after “perfect” data – I just don’t think we’ll ever have. On the other hand, I could be reading too much into their statements. Perhaps they are just looking for that “good enough” level of data.</p> <p>What do you think? </p>