The Synergy of SharePoint 2013 and Data Virtualization

It’s a powerful combination: Data Virtualization and SharePoint. SharePoint 2013, the most recent release, supports broader-than-ever opportunities for data integration and data-centric implementations. SharePoint has evolved to include its availability as a service in the cloud and a hybrid architecture, along with its fully on premise model.

The availability of these alternate architectures is a compelling argument for adopting SharePoint for many types of applications beyond the historical content management uses. The rich Business Connectivity Services (BCS) provide a range of features that are data-centric, elevating SharePoint to another level as a platform for applications, dashboards, and BI tools.

As implementations mature, though, certain requirements are coming to the forefront. In particular, the reality of most applications is that data needs are not limited to a single data source, or even a single data source type, which is a limitation of not only SharePoint, but of the third-party vendors that provide connectivity. The potential of the built-in powerful BCS is not fully leveraged, and the end result is either limited scope of applications or the need to build a staging database either on premise or in the cloud, the latter of which brings additional security concerns for many customers.

The ideal use of SharePoint leverages BCS to access data live from the original sources, validates the data, presents it to the end user, or application for action or editing, and pushes changes back to the original sources without ever saving it anywhere. End user security ensures that end users only see or edit data to which they have access.

This is what Data Virtualization (DV) brings to the table, and Stone Bond’s Enterprise Enabler is a full-scope integration platform, with data federation at its core. The single Integrated Development Environment (IDE) for development, testing, deployment and monitoring, is built on and fully extensible with Microsoft .Net. Within a few minutes, one can configure and generate data virtualization in all of the data access modes supported by SharePoint, OData,SOAP, REST, ODBC, ADO.Net, and Enterprise Enabler's Custom Connector.

What is Data Virtualization?

“Virtualization” is everywhere but nowhere. The term is virtually ubiquitous. The first use I remember when computers got into the picture was “virtual reality,” when we computationally rendered 3D worlds in 2D, complete with lighting models and all that. The good thing is that all those complicated algorithms are now encapsulated for cool things and used by beginner gamers. In those days, we had to actually calculate every pixel ourselves. But I digress.

First, let’s clarify that “virtualizing data” means putting in the cloud or elsewhere in order to eliminate some of the hassles of its existence and maintenance. That has nothing to do with data virtualization, which is a term that I believe is still evolving.

Data Virtualization, according to Rick van der Lans, who literally wrote the book, is “the technology that offers data consumers a unified, abstracted, and encapsulated view for querying and manipulating data stored in a heterogeneous set of data stores.**”  

As the discipline matures, he is expanding his view, as in his new white paper, Creating an AgileData Integration Platform Using Data Virtualization. Definitely recommended reading. 

The “unified, abstracted, and encapsulated view,” from his original definition is the core concept, in my opinion, of data virtualization. In other words, there is a mechanism to bring together, or “federate,” virtually, data from many sources in a way that is useful. This means that the data is federated without creating a physical or cached staging database, but is aligned, transformed, and made available for use. So, for example, you may have a SharePoint BCS application that needs data from SAP, Oracle, and Salesforce.com. Data virtualization will provide a mechanism to merge all of the data into the form necessary for the end user’s interaction in SharePoint. The data is federated “on the fly” and delivered virtually to a web page, on-demand upon refresh of the screen. Think about the security of the backend data that has been accessed…it never actually moves from its original source! Data virtualization also includes writeback to the sources (with end user security, but that’s for another blog) so that an end user can, for example correct his phone number or address, sending it as an update directly to the backend source. )See more examples at http://tinyurl.com/a3wkffc)

You can see that this description expands the definition to include any sources, not just data stores, although the focus of most data virtualization products is BI, in which case, that limitation it makes sense. The BI view of using data virtualization usually is with respect to federating relational databases for the sole purpose of querying. The tools that were designed assuming that constraint have some difficulty accommodating the expanding definition. 

In addition to evolving from federating data stores to federating any kinds of disparate sources, data virtualization is shedding the concept of “on-demand” only. Now federated data is not just available by web services, ado.net, ODBC, JDBC, etc, but for any type of data integration, such as ETL, EAI, etc. 

In fact, it is the “data virtualization” concept of federation that becomes the kingpin for “Convergence,” as Gartner is wont to say, of all integration modalities in a single toolset, sharing metadata and business rules across all. 

**Rick F. van der Lans, Data virtualization for Business Intelligence Systems, Morgan Kauyfmann, 2012