The BBBT Sessions: WhereScape

Originally founded in 1997 in Auckland, NZ as a data warehouse consulting company, WhereScape has evolved to become a solution provider and —especially in the last five years— a key player in the data management market and especially in the data warehousing and big data spaces.

During a great session with the BBBT, WhereScape showed their “latest and greatest” news and innovations, triggering meaningful discussions and interactions with the analysts of the BBBT.

Here, a summary and commentary of that cool session.

WhereScape at a glance

As mentioned before, during an evolution process that expands for more than 20 years, WhereScape became a solution provider of data infrastructure and automation solutions, it currently offers three main solutions:

• WhereScape 3D. A solution to assist in planning, modeling and designing data infrastructure projects as well as enabling rapid prototyping.
• WhereScape RED. A solution to enable fast-track development, deployment and operation of data infrastructure projects and reduce delivery times, effort, cost and risk of new projects.
• WhereScape Data Vault Express. A solution especially designed to enable the automation of the entire life cycle of a modern database modeling method Data Vault 2.0-based project delivery.

Along with its vast experience in the consulting field and later with its software solutions, WhereScape has become a major player that navigates in the frontline of the data warehouse and data infrastructure automation market.

Is in the context of “automation” that the BBBT session with WhereScape was focused on, centering on the software company’s advances within its data infrastructure software.

Today and especially since the last couple of years, it comes as no surprise that automation is becoming increasingly important for almost all major software technology footprints, including the data warehousing and data management markets.

Why? 
Well, because as Neil Barton —WhereScape’s chief technology officer— explains neatly during the briefing: automation in a data infrastructure can provide great value to an organization which can be translated into many aspects of its operation, from cost, development time and risk reduction, to enabling IT to keep the pace with business needs.
This is especially true when data warehousing has traditionally been done in a complicated and cumbersome way (Figure 1).

Figure 1.  The traditional data warehousing development/maintenance cycle (Image courtesy of WhereScape)

Consequently, WhereScape aims to provide automation and simplification of most, if not all, the process data warehousing process by incorporating not just plain automation of tasks but encapsulation of methodologies, the best practices and industry standards within to enhance simplification, provide time reduction and yet, ensuring compliance with internal and external data management regulations.

One of what I think is a unique aspect of WhereScape is the company’s “holistic” view to automation in which all the data cycle is considered via a metadata driven approach, making it possible for WhereScape to easily provide documentation and lineage and have a full cycle management approach within a single solution (Figure 2), and more so, to bring integration to WhereScape’s own set of solution.

Figure 2. Simplification and automation with WhereScape (Image courtesy of WhereScape)

Interestingly, while not trying to “reinvent the wheel”, and remaining loyal to a well proven life cycle (Figure 3.) WhereScape developed and made this life cycle an intelligent and automated one by aiming to improve all its faces from the discovery to the rest of its stages to reduce time, effort and augment efficiency.

By being metadata-driven and having an integrated structure within all its components, WhereScape  incorporates automation to the entire life cycle process and enable smoother documentation.

This is especially true under the design and operation processes —normally a complex and tedious stages— in which WhereScape introduces, for the first one, an automated model generation engine which allows speeding the common iterative design cycle as well as the ability to manage efficiency by enabling solid dependency management and integrated scheduling and logging to ensure efficient auditing and operation analysis.

Figure 3. WhereScape’s automation life cycle (Image courtesy of WhereScape)

Additionally, to showing efficient automation control and management, WhereScape also shows flexibility by offering support for a set of projects, including:

• Data Warehouses
• Data Marts
• Data Vaults
• Cloud deployments and
• Big data support

Making WhereScape to count as an important emerging player in the data management automation market.

WhereScape at the BBBT

After a summary of what WhereScape is all about, allow me to highlight some of the most relevant aspects of what was a great presentation conducted by Neil Barton, WhereScape’s Chief Technology Officer (CTO).

The briefing gave us good info about what WhereScape has been working in recent months and what is now up to for the future, some of which include:

• Increasing data platform coverage. WhereScape has been working to expand its set of “data infrastructure automation” solutions to a wider number of data platforms, which now includes:
• Amazon Redshift
• Microsoft Azure SQL Data Warehouse
• Snowflake
• EXASOL
• SAP HANA
• PostgreSQL
• Extensive support for Data Vaults.  WhereScape has worked on providing extensive support for Data Vault —a database modeling method designed to provide long-term historical storage of data coming in from multiple operational systems— and more specifically engineered for Data Vault 2.0.
• Support for Data Streaming. The briefing included an interesting discussion about WhereScape’s new support for data streaming a feature aimed to help IT teams to manage hybrid flows of streaming real-time and traditional batch-based data by enabling the design, development and deployment of more advanced data infrastructures.
• Reinforcing Support for Cloud Data Platforms. Within the company’s efforts expand support for data platforms it can be highlighted the continuous support for cloud-based data platforms which includes direct support for cloud-based data management engines including Snowflake, Amazon’s Redshift, and Microsoft SQL Azure.
• Addressing a Hybrid Reality. WhereScape has taken steps to address a reality for companies that now navigate data that exist both on cloud and on-premises sources making it possible to move data between both world and to ease this transition.   

On another key moment during WhereScape’s presentation, WhereScape guided us towards what the company has developed to enable organizations to, via its solutions combined with key partners including StreamSets, include support for streaming IoT integration.

Real-time data flows, or “streaming” data sources, can now be collected from many areas within an organization’s data landscape, including from in-field units sharing sensor-based data, social media feeds to support sentiment analysis, or from internal systems feeds.

Utilizing industry-leading dataflow management technology developed by StreamSets and the proven efficiencies of WhereScape automation solutions WhereScape RED or WhereScape Data Vault Express, WhereScape automation with Streaming minimizes the learning curve for IT teams working with these new data sources and ensures best practice integration of streaming data (Figure 4).

Figure 4. WhereScape’s Streaming IoT Integration (Image courtesy of WhereScape)

Serving as a central element, WhereScape aims to provide all the necessary tools to complete the full IoT cycle from data collection to the provision of insights and information for users, automating the event queuing transformation and storage and administration stages and according to WhereScape:

• New analytic opportunities
• Speed up pipeline development
• Hide complexity of underlying technologies
• Minimize the learning curve for teams
• Integrate with batch-based data
• Ease ongoing management

The briefing portrays a great environment to help us confirm WhereScape’s role in the market as well as what the company is doing to improve all their solutions to adapt to new technologies and needs. A great, clear and concise briefing by WhereScape.

WhereScape: A Tough a Couple of Final Thoughts

From we can see, WhereScape has been interestingly evolving, from being a data warehouse automation solution to become what the company calls: a data infrastructure automation solution.

Interestingly, the company has shown savviness to evolve its solutions so that, via a model driven design, to incorporate not just connectors to new data sources but the necessary tools and design to enable organizations to modernize full data management platforms in complete hybrid environments.

Figure 5. WhereScape’s RED Multiple Data Model Architecture Overview (Image courtesy of WhereScape)

Of course, even though WhereScape has grown a solid and intelligent solution strategy, from a market perspective, gaining market share hasn’t and will not be a walk in the park.

As automation technologies continue to gain steam in all main software areas, a larger number of software companies are now incorporating increasingly sophisticated mechanisms to their data management and analytics solutions thus, making automation another key feature in an already extremely competitive market, such is the case for Oracle and its new Oracle Autonomous Data Warehouse Cloud Service or offerings by other competitors like Attunity Compose or Panoply. 

So, does WhereScape can lead in data infrastructure automation?

Of course, but will need to make it can consistently deliver IT organizations with the means to achieve effective automation within all faces —design, development, and deployment— of, especially, a data warehouse and, generally, a full fledge data infrastructure.

This includes, achieving seamless collection and processing of multiple source types and of course effective complexity cutting.

From the briefing, it seems WhereScape is, if not there, on a right way to achieve some or many of these goals, making it one of those companies up to the challenge for gaining the favor of companies with complex data infrastructures to deal with.

WhereScape is rapidly realigning to serve a new generation of data management systems and needs.