The Top 10 Mistakes in Data Architecture - part 1

The IT Architecture Journal is meant to help both practitioners of IT Architecture as well as those who might hire architects. It is vital that both of these stakeholder group understand not only the core principles behind the practice of Architecture but also where and when Architecture can go wrong. Today, we're going to talk about some of the more common problems or mistakes that arise in the practice of Data Architecture. While there are no concrete rules or really even standard practices in relation to some of the following considerations; we have attempted to clarify where standard practice is already present or perhaps just emerging.

Architecture can help solve puzzles, but should never produce its own

Mistake 1 - Lack of Context
A great deal of Data Architecture being practiced today is totally generic in nature. This is not just a question as to whether the effort may be high-level or not, even higher-level analysis and design can be targeted or contextual in nature. Part of the problem here is the desire by many organizations to standardize best practice based upon industry-defined expectations. This tends to happen most often in relation to the DMBOK (Data Management Book of Knowledge).

Now, there's nothing wrong with either the DMBOK or the desire to standardize approaches to data management or architecture based upon industry lessons learned. What is problematic however is taking much of that information at face value without the realization that the information was deliberately made generic in order to reach the widest possible audience. As with any standard practice or technical standard, it is only as good as its applicability to any specific scenario.

So the question here is what value the purely generic information has; the answer is, as in other cases; it can be an excellent starting point but it should never be the destination. For example, XML or JSON represent different approaches for forming data messages, yet knowledge of the core JSON or XML standards is only valuable in the context of how it might be applied to your own data (which standard to choose, and how it will be used, structured and integrated into your solution).

Far too many Data Architecture projects present the generic information without the follow-on "contextualization." This is a sure path to project failure.

Mistake 2 - A Disconnected Approach
Where does Data Architecture start and where does it end? This is more than an interesting philosophical question. If one stops to think about it for moment, it becomes clear that there is data lurking nearly everywhere in the enterprise - yet much of it - perhaps even the majority of it is not officially managed as part of the "Data Architecture." This can include the following types of data:

• Email and content (including Social Media)
• Metadata
• System logs (for all / any type of systems) 
• Portfolio Management or EA tools
• Smaller DBs or tools not officially managed by the IT department (often business users)
• Mobile and Web App related data 
• Applications (commercial software packages) that have their own databases
• Device data (and / or embedded databases) 

Obviously, there will be situations as a Data Architect where you'll be asked to focus on only one particular 'slice' of the overall data environment for an organization. However, this mistake is more directed to the larger question as to how the enterprise views data architecture and data management. If an organization views portions of its data assets to be outside the enterprise processes associated with data management - then there is a problem. There is also a problem when the management of data is further divided across semi-autonomous groups within the organization. Both of these are very common practices today.

What occurs on these situations is the creation of data silos or islands. The natural consequences of that includes:

• Duplicative data across the enterprise 
• An inability to deploy truly effective analytics
• Significantly higher costs for IT

The easiest way to tear down these walls and connect the data islands is to make the commitment up front that the Data Architecture will include all data meaningful to the organization so that design and management can be aligned with enterprise goals and expectations.

Mistake 3 - Lack of Actionable Governance
Note that we didn't just say 'Governance.' Everyone in the data business loves to talk about Governance, but what does that really amount to? Governance tends to be both a workflow process as well as an associated toolset. If either the tool or the workflow process has limitations or is not present, the Governance effort generally fails. When we say "Actionable," what we mean is very specific; it involves the following elements:

1. Integration of Data Governance with all pertinent system management processes.
2. Integration of Data Governance with security standards and processes.
3. Integration of Data Governance and Portfolio Management process.
4. Integration of Data Governance and Data Architecture.
5. Adherence to clearly defined metrics, tracked on a continual basis. (this can include ALM metrics, integrity metrics as well as performance metrics).
6. The Governance process has clearly defined roles with authority and decision gates built in. 
7. The Governance is closely aligned to both tactical and strategic organizational goals.

This set of elements goes well beyond what many groups consider Governance. Often times this happens because they aren't willing to make the investment in order to support these activities. Ultimately any perceived savings are in fact highly illusory given the well-defined cost impacts of environments suffering from a out of control data sprawl (which always seems to come as a surprise but really shouldn't - recreating the same solution 4 times costs 4 times more, right?).

Mistake 4 - Data Architecture isn't Taken Seriously
As you might imagine, this is a problem not limited to Data Architecture. So, what do we mean when we say Architecture isn't taken seriously? This tends to manifest itself in the following ways:

• No investment at all in architecture
• Limited or one-off investments (periodic reviews but no dedicated architecture practice)
• No investment in architecture tools
• Lack of integration between architecture and other key processes, such as Portfolio Management
• No architecture methodology employed

Architecture is like anything else - you get out of it what you put in it. If the resources and commitment to an effort is minimal, it is likely that the value derived will be less than may have been expected (unless it was understood to be a token effort from the start). In many ways, the Architecture lifecycle is similar to an ALM (application lifecycle management) approach. Architecture is design writ large - typically taking into account all potential aspects of a ecosystem of individual solutions. In this sense, it is very difficult to translate Architecture into Agile methodology except for time-boxing delivery expectations. 

But architecture without any methodology is simply ineffective. 

Mistake 5 - Product Only Focus
This one deserves a bit of caveat. If your project is narrowly focused on deployment of one particular product / technology, then it is obvious that the product architecture will be the primary focus of the effort. However, this problem refers to the larger enterprise. An important value that an IT Architecture effort brings to the table is the ability to abstract capability from the specific technologies that will ultimately manifest that capability. What is this important?

1. Because in a large environment, there will likely be dozens of technologies involved.
2. Because technologies and products change. 
3. Because the Architect is not generally an agent or advocate for any particular technology, but rather an unbiased 'change agent.' This role is also referred to as the "Honest Broker."

So, the mistake that occurs most often in relation to this issue is the idea that one product or technology will provide some sort of "Silver Bullet" solution that will remarkably transform the enterprise. While this does on occasion happen, it does not happen very often. Generally the scope for any Data Architect will be wider than one product or product suite (or even several), and most importantly the true scope is the problem space associated with the organization's goals rather than mere management of specific commercial software packages.

We will cover the next five mistakes in an upcoming post...

copyright 2014,  Stephen Lahanas

#ITarchitectureJournal
#StephenLahanas
#Semantech-Inc

Read More

Security Architecture, Defined

There are some people who don't recognize Security Architecture as its own niche within IT Architecture, but given the number of roles / positions now using the title "Security Architect," perhaps the naysayers have missed out on something. This post will examine what Security Architecture consists of and how it fits within the larger context of IT Architecture.

Back in the old days, security was a lot easier. We didn't have internet connectivity everywhere, few if any viruses, and only a handful of hackers who were mostly in it for the fun. Things have sure changed. What used to be referred as "Information Assurance" or IA has gradually morphed into "Cyber Security." While IA was primarily focused on management of information assets (which included copious amounts of paper locked in vaults), Cyber Security has tentacles that reaches all the up and down every solution stack and into every nook and cranny of the newly dubbed "Internet of Things." (this year's hype king).

Security Architecture is related both the practice of Cyber Security as well as the practice of IT Architecture. We've tentatively classified it as a subset of Solution Architecture, but just like some other areas like SOA, Security Architecture proves to be a bit ubiquitous in both principle and practice. Originally, when those of us worked on Security Architecture projects, we tended to fund them primarily concerned with Data Center design (or consolidation) or more specifically focused on tightening up perimeter security within a Data Center. Now, Security Architecture has become much more expansive, including aspects of UI coding all the way to management of mobile devices (to encrypt data at rest and prevent identity theft for example). So lets move on to some definitions.

Security Architecture, Defined

Security Architecture represents the ability to represent and resolve any IT-related security issues using IT Architecture techniques. These security issues can be internal or external and can either be technology agnostic or technology specific. The solutions developed as a result of Security Architecture analysis and design are often decomposed into "Security Controls," which focused on narrowly defined portions of the overall security landscape. 

Security is still based on core IA principles but those have been expanded
 to include a much wider arena of potential action

In our initial definition, we alluded to something called Security Controls; this is worthy of further examination...

Security Controls, Defined

Security Controls are the individual measures necessary to mitigate specific security threats or concerns. These controls can be part of larger information security standards (NIST, FISMA Cyber Framework, etc) or they can be defined within specific organizations (or a combination of both could be used). The controls consist of guidelines and policies (for example, coding practices), specific configurations for systems, and test cases for security evaluation.

Management of Security Controls isn't necessarily a Security Architecture function, however definition of Security Controls often is (coming after some type of audit that may include review of the existing architecture).

Which brings us back to the larger question of what Security Architecture is and what exactly a Security Architect does. As we alluded to earlier, once upon a time it was common to associate Security Architecture with data center technology (like Intrusion Detection Systems for example) and Information Assurance with control over information or data. Today, Security Architecture spans the entire stack from infrastructure (Cloud or Traditional) through Data, Application up through to UX. It also extends out to other enterprises and to mobile platforms and social media (or other external Cloud based services). Security Architecture is always only as strong as its weakest link - and with the ever-expanding scope of operations the extent of the oversight necessary to mitigate those weak links makes the job of the Security Architect more daunting than ever before.

Security Architecture must facilitate both prevention of threats as well as active oversight of operational integrity. While the Security Architect does not sit in a NOC (Network Operations Center) he or she does often determine how operational analysts will conduct their jobs. This field within IT Architecture is an excellent example why cross-domain expertise is needed for many Architecture roles.

We will be examining a number of specific Security Architecture case studies in future posts here on the IT Architecture Journal.

copyright 2014,  Stephen Lahanas

#ITarchitectureJournal
#StephenLahanas
#Semantech-Inc

Read More

How to Create Product Maps

In some of our articles here you've probably noticed various types of diagrams thrown in as examples. All of these come from real-world project related exercises. When an IT Architect produces a diagram we generally refer to it as an "Artifact." While this may sound like we're talking about relics, it is simply an odd naming convention of our industry. An Artifact can be any sort of deliverable; a Word doc, a spreadsheet or perhaps even a file associated with a specific EA software environment. The common thread across any of these deliverable formats is the notion that the Artifact must be more or focused on one topic (sometimes this isn't the case, but most times it is).

One particularly interesting type of Artifact is the "Product Map." So what exactly is a Product Map? Well, there are two distinct types of Product Map and there are only tangentially related.

Type 1 - A Map showing the relationship between products, comparison of products or elements of a complex product suite.
Type 2 - A Map showing product components and their evolution (past and future roadmap).

So, you might be asking yourself, why is such a thing necessary and what does this have to do with IT Architecture?

Why We Need Product Maps
Believe it or not, the companies that produce these products don't always do a good job at depicting the composition of their products or the relationships between products. And, if we extend our view to a large heterogeneous enterprise environment, there is little incentive for a product company to spend too much time thinking about how their product might relate to their competitors or other products which don't seem to be related to the function their solution performs.

There are a variety of situations where product maps might be needed; they include:

• Representation of "As-Is" architecture. Product maps would be drill-down artifacts that illustrate more of a physical view.
• Product Evaluation and Source Selection efforts.
• Transition Planning exercise (in other words, being able to illustrate migration of one set of tools to another).
• Enterprise Integration Design. Getting different product sets to work with one another or even send messages to other can be challenging. 

The product map differs from other types of architecture artifact in that it depicts some aspect of a larger solution from a specific product perspective or context. The way we might highlight an identity management solution in an agnostic logical view could look far different than a product specific IAM stack (see the diagram below).


IT Architecture & Product Integration 
So how does the IT Architect facilitate Product Integration? Well, at its core this represents a problem solving exercise; in fact one of the more complex ones that the majority of enterprise tend to face on a routine basis. One might think that since so many similar organizations face this type of challenge that there would be a wider set of industry standard approaches to this - but this isn't the case. While many enterprises share the same tools, the exact configuration and inventory will almost certainly vary widely. And as most of you know, even moving from one version of a product to the next can be major hassle in itself.

The IT Architect takes this problem and turns into a design exercise. This begins by documenting all of the pertinent aspects surrounding the product:

• Key features
• Standards support
• Administrative and management considerations
• Customization support (languages, configuration)
• Inter-operability mechanisms (database connectivity, APIs, etc.)
• Version control and version specific variations of the above
• Overlap of features or deprecation of features (either between different versions or options of one product or competing products).

The Architect then defines all of the related design decisions based on a larger set of expectations for how the product might be used. Some products of course merely provide a platform for even more custom development (for example an DBMS like Oracle) - with the larger Oracle stack then there are opportunities for both custom and packaged analytics. The determination as to whether to go one way or the based upon what products are already in the inventory or which ones ought to be included near-term and long-term is a classic example of contextual product mapping.

An example of a product map of a suite of related security products

IT Architecture & Product Design 

Can IT Architecture support product design or development? In a word yes. This represents Type 2 Product Mapping. For example, what if you had an excellent idea for a new mobile app and wanted to manage it in some start-up scenario? There are of course various business considerations for a typical start up, but just as important is management of the product itself and that should start with its architecture.

The architecture will drive many key decisions related to the product's introduction and evolution including:

• Determination of what technologies should be harnessed to actualize the product.
• Development of the product roadmap (including conditional variations based upon contingencies or opportunities).
• Determination of key systems design options and solutions.
• Governance of the product (and all its related infrastructure and code) across its lifecycle.
• Determination of product synergies in the market (the larger ecosystem of related solutions).

We will look at Architecture and Product Design in greater depth in upcoming posts here on the IT Architecture Journal.

Copyright 2014,  Stephen Lahanas

#ITarchitectureJournal
#StephenLahanas
#Semantech-Inc

Read More

Lifecycle Management, Defined

Like many other aspects of Information Technology over the past 15 to 20 years, Lifecycle Management has evolved quite a bit. In fact, it didn't generally even get referred to as "Lifecycle Management" until about ten years ago. It (both the term and the practice) has become more expansive in nature as time has passed. This is an interesting not just due to the evolution which has occurred to the surrounding practice, it is also the one area in IT where technology and business / management comes closest to being blended into one concern.

Lifecycle Management is also the reason that IT Architecture exists the way it does today - without the driving force of various governance processes and mandates, IT Architecture would have likely never quite crossed the threshold from systems or software engineering and design. We'll return to that in a minute.

So time for definitions - and in this case - a hierarchy.

Lifecycle Management, DefinedLifecycle Management is the practice of coordinating all the related activities for any given selection of IT capability (within or across organizations). There is generally a hierarchy for these activities which falls roughly into the following structure; Portfolio, Program, Project. Generally at the Portfolio level, the capability management spans many systems whereas there may be multiple Projects supporting just one piece of software or one system. In an optimal situation, all Lifecycles should be unified based upon shared methodology, principles, goals and supporting tools. 

Many people have long considered Lifecycle Management to be the province of the manager. However, nothing could be further from the truth. For Lifecycle Management to function as intended there has to be a partnership between management and technology resources. This partnering usually occurs on the tech side through the role of IT Architect.

This view is a more traditional enterprise-focused view (mainly waterfall in nature) 

Why Lifecycle Management isn't just about Management
Let's go over a few things that must have technology input to succeed:

1. System (and project) Scope - While there may be a number of specific expectations from business, the most successful systems (and projects) are those in which the tech side helps to define the boundaries of what can realistically be done. This manifests itself at the highest level in portfolio roadmaps and at lower levels through statements of objectives or similar scope documents for projects (which then get passed through governance processes for approval). 
2. Project Estimation - This is a much more complex process than many realize. Good project estimation requires a precise understanding of the complexity of the technical tasks involved. Whether you're talking about building J2EE-based APIs or deploying a WCMS, the tech experts who have been there / done that are invaluable in helping to define a realistic project estimate. Since much of IT is measured based upon estimated cost versus actual expense - this area is disproportionately important. 
3. Risk Assessments - Sometimes this includes Architecture Assessments, other times it is more of a technology strategy type exercise.
4. Technology Selection - Should we Build or Buy, or if we're going to buy, how do we choose? These are the typical questions that get asked, but who can really answer them? The IT Architect, of course - its part of their job description.
5. Solution Decomposition - This is related to project estimation but more detailed. Generally, somewhere within a planning process, there is a preliminary design exercise which goes as far as determining how best to 'break up' elements of the intended solution into digestible pieces. This also involves prioritization - understanding which parts have to come before or follow others due to various technical dependencies or constraints. 
6. Project Oversight - This is a big topic, one we will dedicate other posts to, but for now we'll just introduce it. IT Architects often play a semi-leadership role within individual projects (as a project or solution architect). This is so technical issues can be tracked, managed and mitigated with continuity across all phases of the individual lifecycle. 
7. Enterprise Integration - This is without a doubt the hardest part of the overall set of potential Lifecycle Management related activities. Now, many think of Enterprise Integration as a purely code focused or networking relating topic - believe me it's not. Enterprise Integration is in reality the intersection of potentially overlapping systems, data and expectations. The most important aspect of Enterprise Integration is inter-operability design  (which comes in many, many forms). It's a study in complexity and not for the faint of heart. 

Let's step back for a moment and explore Lifecycle Management from a different angle. In previous posts we've been talking about methodology a bit; Agile, Waterfall and so on, but what does it all mean? The short answer goes something like this:

An IT methodology quantifies the workflow or lifecycle steps necessary to provide consistent support for system development and management. Most lifecycles share the same common elements; a planning phase, design phase, build phase and implementation phase. And yes even Agile has all of these. Within the common framework, there is a massive amount of variation. So, if you ever had to work with Carnegie Mellon's CMMi (waterfall spiral methodology) and then were put on a Scrum project (Agile variant) it would be like moving from Iowa to Italy - a culture shock to be sure. 

In today's terminology, you often hear the term Application Lifecycle Management (ALM) used - it is essentially the same thing as Software Development Lifecycle (SDLC) except perhaps that today an ALM can encompass both Waterfall and Agile whereas before SDLC was more Waterfall focused. So what are Waterfall and Agile, anyway? 

Waterfall - this refers to both purely linear and 'linear iterative' development methodologies, wherein requirements are more fully defined up front and allocated to segments in a somewhat sequential manner. In some cases, code is completed and tested in iterations - other times no capability is delivered until the very end (traditional waterfall).

Agile - Up front design is minimal, few if any requirements are formally development. The focus is the production of working capability with each (very short) iteration. Design in this model becomes iterative in that core assumptions can be readdressed and correct "in flight" (which is why up front requirements become redundant or unnecessary). 

We will be talking a lot more about 'Agile' and 'Waterfall' as we progress on the blog. 

Lifecycle Management has evolved from a set of competing software development methodologies to become a potentially all encompassing set of technology management processes for the enterprise. And it will continue to evolve with the IT Architect maintaining a key role for some time to come.

Copyright 2014,  Stephen Lahanas

#ITarchitectureJournal
#StephenLahanas
#Semantech-Inc

Read More

Aligning Lifecycle Methodologies - Part 1

In a previous post, I contended that requirements are still an important part of most enterprise environments, even those that might be using Agile lifecycle methodologies. However, I didn't specify exactly how those requirements should be captured — or even what constitutes a requirement. This is an important question when determining how to align multiple lifecycle approaches within one organization (the need for which is much more common than most people realize)

This question encompasses two major lifecycle approaches; Agile and Waterfall methodology. Many organizations feel that User Stories (Agile) are sufficient to build both code and project structures around them. Is it out of the question to consider that Use Cases might be used within an Agile approach?

Before answering that, let’s take a look at User Stories, Use Cases and Requirements in context again:

• A User Story can be thought of as a high level or conceptual scenario, or perhaps just a problem. An example might something like this: A user needs to be able to save a report into multiple file formats for downloading, including both .pdf and .xls. Even though we’ve described two different formats, it still represents the same scenario or problem.
• A Use Case, on the other hand, can be considered a functional requirement. The difference between a functional requirement and technical one is the level of detail the document includes about how specific actions are carried out. For example, a use case would capture the process flow, or steps involved in generating the report and the expected outcomes or alternatives. This helps to build test cases later on. A use case is slightly more structured and detailed and is generally written by someone IT or in combination with the IT group. That way, some of the technical expectations are captured. In other words, the use case describes not just behavior, but how that behavior is achieved.
• A Requirement in this context refers to a technical requirement. This is entirely specific to the elements needed to develop and/or deploy the capabilities described in the previous two documents. It allows us to formalize performance constraints, data elements and other pertinent technical details. This is especially important in environments where EA, service or data governance paradigms are in place. If written properly, requirements can then serve as system documentation.

Mapping across lifecycle approaches begins with requirements (however they're captured)

While many organizations practicing Agile bypass requirements, they may in fact still capture the same information after the fact if they actually document their deployed solutions (after the fact). When you’re considering whether or how to employ user stories and use cases, keep these points in mind:

• They are not the same thing.
• The user story ought to come first.
• The use case ought to be derived from the user story.
• Any requirements managed from this process should be embedded within, or otherwise traceable to, a specific use case and user story.
• User stories could be considered either scenarios, high-level processes or problems.
• Every organization approaches this process somewhat differently. For example, in a less Agile environment, a number of other architecture artifacts could be added to the illustration above:
• At the user story level, many people like to capture a data-flow diagram or a context diagram.
• At the use case level, many folks like to add a UML sequence diagram and conceptual data model.
• At the requirement level, many UML diagrams are often added including activity models, state models, component diagrams or deployment diagrams.
• If a formal EA framework is being applied, those UML or ERD diagrams are often replaced or folded into EA Framework views/artifacts.

Getting back to the original question — should use cases be used with Agile lifecycle management? — my recommendation is yes. Here’s why and how.

Why to Do It

1. It ensures some level of systems engineering is included in the Agile design process.
2. It helps to define the tasks associated with each user story and doesn’t add that much extra effort - it also doesn’t require a use case diagram (which in most cases doesn’t convey that much meaning anyway).
3. It ensures that both IT and business stakeholders capture their expectations in formats tailored to their roles and needs.

How to Do It

• Define your use case template and what it must convey.
• Determine whether user stories require more than one use case. It happens occasionally, although in most situations user stories should be in a 1-1 ratio.
• Ensure that someone from IT helps with, or provides much of the detail, in the use case.
• Ensure that use case development is aligned with both the quality and testing paradigms. This is especially important if no requirements are captured.

The majority of Agile development environments are fairly flexible in regards to storing of related attachments, so the biggest impact to employing use cases along with user stories is in the design and development process. This is the first step towards aligning lifecycle methodologies; we will examine this further in several upcoming posts.

Copyright 2014,  Stephen Lahanas

#ITarchitectureJournal
#StephenLahanas
#Semantech-Inc

Read More

Understanding Architecture Accessibility

The topic of Architecture Accessibility may sound a bit abstract upon first glance, but it is in fact one of the most important considerations as to whether an architecture effort will add value or not to an enterprise (or any particular solution lifecycle). It is also directly related (or perhaps a part of) the most important consideration for any architecture - its usability.

Let's step back for a moment and paint the background a bit.

IT Architecture has progressed in fits and starts over the past 20 years or so partially because of the widely varied results or outcomes associated with its practice under various scenarios. This is to be expected when considering that this is a fairly new aspect of Information Technology - it appeared rapidly and has been evolving ever since.

But why has IT Architecture sometimes gotten a bad reputation? The common causes for this include the following:

• A lack of clear expectations as to what the Architecture activity would really provide.
• A lack of connection between different types of Architecture efforts - for example Solution Architecture being managed separately from Enterprise Architecture or Data Modeling or application design not being considered as architecture.
• A lack of integration with governance and portfolio planning and management processes. 
• A lack of detail and / or specific implementation-level guidance.
• A lack of integration between architecture activities and ALMs / SDLCs (Software Development Lifecycles) 

You may have recognized a common thread in most of these problems - that organizations which produce Architecture in a vacuum seldom succeed. A variation of that theme is directly related to our core thesis on architecture accessibly.

Making Architecture accessible requires clear organization and a componentized, targeted approach

What is Architecture Accessibility and Why is it Important?
Accessibility in this context refers both to how people can reach the Architecture but also to how it is structured. So, here is a fundamental consideration - if an Architecture is produced yet few within the larger organization have access to it - how likely is it that the Architecture will achieve its originally intended purpose? Taken a step further, if we examine the structure issue; how likely is it that an Architecture will be exploited efficiently if it is difficult to find the pertinent information within it that any given stakeholder might need?

As Architectures grow in complexity, there tends to be a larger and larger potential audience associated with it. For nearly everyone in that audience, only a portion of the Architecture will be relevant to them. Thus the idea of a monolithic architecture artifact (say, a giant MS Word doc) becomes highly problematic. Use of a formal Enterprise Architecture repository can become equally problematic if added licensing is required for anyone who needs to view the Architecture (which in my view is a highly counter-intuitive strategy on the part of companies which produce such software).

What is at risk if Architecture becomes inaccessible is "Shelfware Syndrome." Architecture like software can provide no value if isn't applied to something - it has to be used. And this is how we begin to make the connection between Architecture Accessibility and Usability.

Architecture Usability, Defined
"Architecture Usability" refers to the ability for the Architecture to communicate the right level of information to specific stakeholder groups. Architecture must generally support multiple stakeholder groups simultaneously. What proves to be 'usable' in one specific organization may be different than in another, but generally it will involve a mix of descriptive (design explanation) and prescriptive (implementation guidance) content. Usable Architecture then is technical design content which is structured such that it can be easily accessed by the right audience at varying levels of detail.

Architecture is a continuum - it provides a mechanism whereby any number of projects and systems can be tracked and managed in context with one another. It represents a sort of meta-integration mechanism if used properly and the best architectures are those which are fully integrated within the organizations they serve. Good Architecture is Democratic as well in that the wider the shared designs are disseminated, the more likely it will be used effectively to guide the enterprise towards a unified set of goals (and some tangible end-state payoffs).

Copyright 2014,  Stephen Lahanas

#ITarchitectureJournal
#StephenLahanas
#Semantech-Inc

Read More

SOA, Redefined

Once upon a time, SOA (short for Services Oriented Architecture) was the king of IT Hype; for years people talked incessantly about it, attempted to extol its virtues and many tried using it within their own enterprise environments. SOA has had mixed success over the years, but that's somewhat hard to measure for a number of reasons, including:

1. There were conflicting definitions for it to begin with - some viewed it more as a practice or design approach, others viewed from a product perspective and still others saw it as a standards focused phenomenon. 
2. What SOA was has morphed quite a bit from where it began - it has changed so much in fact that it can hardly even be referred to in its own context any longer. We'll explain that in a minute. 
3. It was never entirely clear from the initial hype, what the real benefits of SOA would be if adopted. That has changed somewhat since the mid-to-late 2000's but that's only because the solution context evolved.

SOA drives design on the right side of this picture...

So, let's back up for a moment and first look at what SOA at several different points over the past decade. 

In the misty depths of the turn of the new millennium...

• In 2004, innovations in software engineering - notably the notion of Object Oriented programming techniques and languages along with pattern-driven design had begun to change the perception that applications must handle redundant tasks each time they faced them. In other words, the notion of software that could be shared as services through abstracted interfaces was becoming popular. The core design concept was extrapolated outwards to an enterprise scale and the first serious discussions about SOA as a practice or set of architectural principles began in earnest. 
• On a parallel track, the W3C (World Wide Web Consortium) Microsoft and several other companies began plotting out a standard communications protocol for linking software objects or services across the Internet. The standard, later called SOAP (Simple Object Access Protocol) was part of a larger set of W3C standards including XML, XHTML, CSS, DOM, WSDL and UDDI which when taken together would provide standard approaches toward encoding data, passing it and locating the places it was hiding as well as defining the format for how to display it. SOAP was adopted in 2003. 
• Also around this time a set of data management technologies emerged; partially based on EAI (Enterprise Application Integration) and ETL (Extract Transform & Load), that linked SOAP with Services concepts in the context of data movement. This new technology was called the Enterprise Service Bus (ESB).

fast forward to 2006-2008...

• A vast array of products have been restructured to accommodate the design principles which have matured somewhat. These products were formerly application server platforms and other governance management tools among others. The "SOA Stack" had been born.
• Industry hype probably reached its apex around this point. SOA projects were popping up everywhere and the idea of what SOA meant was little clearer.
• That definition focused on key design principles (such as "loose coupling" and "encapsulation" both borrowed from Object Oriented Design) and core elements of the stack (which had just evolved) - so the typical SOA solution would follow the principles and use the stack (starting with an ESB, then applying a UDDI registry, governance tool, business process manager and the appropriate app server / container, such as weblogic or websphere etc.)
• Java figured prominently in the SOA landscape, which only makes sense given that SOA evolved partially from Object Oriented Design techniques and principles. Thus, a great many of the SOA projects chose J2EE as the development language because it was designed to support OOD. 
• Some larger organizations, like the DoD, wanted to take this all further and bundle identity management and infrastructure services along with SOA initiatives. This was the beginning of a transition from consolidated hosting environments to Cloud Services in the Federal ecosystem. 
• On the commercial side, something called Software as a Service began to emerge at this time (it had in fact been around before then but got rebranded as SaaS). It involved the notion of a subscription to get thin client applications served directly from the service provider.  
• Virtualization becomes an interesting way to not waste CPU cycles in the datacenter.

2010

• Dave makes contact and the next generation of HAL computers malfunctions yet again! - sorry couldn't help throwing in the obscure sci-fi reference...
• Smart phones conquer the world. This part is accurate - it did happen and it came as a bit of surprise. It also changed the way we looked at development platforms, delivery platforms and 'object communication.' RESTful services, which had been around for awhile, got a huge boost in large part to the explosion of 'apps' being developed and deployed to IOS and Android devices. SOA would never be the same...
• Let's not forget DropBox, Google Drive and a 100 other Cloud offerings that began appearing around this time. Infrastructure got sexy.
• Oh, and everything turned "Agile."

2012 - 2014

• The longest war in American history still isn't over yet. When it started, enterprise IT was still largely mainframe driven - ten years later - what diversity we find...
• The popularity of Object Oriented Programming has hit the skids so to speak. This is because the scripting languages have overtaken J2EE in popularity as those languages support Agile and mobile platforms better (J2EE is still around of course, but then so is Cobol - the coding is on the wall). 
• People seldom refer to SOA projects anymore - or SOA has become synonymous with application architecture where organizations have more or less adopted the principles. Often times now its being done with REST instead of SOAP.

So, now that our tour of the past decade has brought us full circle back to the present; how would we go about defining SOA in 2014? Wikipedia defines it almost as a single pattern - others - like Thomas Erl - have written very interesting books on it.

Services Oriented Architecture (SOA), Redefined

SOA represents a nexus of design principles and standards for internet-driven applications. There is an implied reliance on specific toolsets that support the resulting services created based upon those principles and standards, but those tools do not define SOA any more than Object Oriented Design is defined by Eclipse. In other words, you could conceivably use any number of other tools to produce a solution that might conform to those principles or standards.  

The notion of enterprise services, Managed Services and Software as a Service (thus much of Cloud Computing) are natural extensions of the core SOA concept, but aren't SOA (anymore than SOA is OOD even though it is largely derived from it). SOA or for that matter OOD can be used to help facilitate parts of these other solutions. 

In the context of our blog here, I like to think of SOA and OOD has subcategories within the practice of Application Architecture (which itself fits within Solution Architecture). Management of enterprise services which may have been created using SOA fits into Infrastructure Architecture. Obviously, when one works on an application service that is intended to have a Cloud or SaaS element to it (or must rely on enterprise services to function), multiple architecture tiers or practice areas will be involved - which is another good reason to have IT Architects who can handle it all, right?

opyright 2014,  Stephen Lahanas

#ITarchitectureJournal
#StephenLahanas
#Semantech-Inc

Read More

Application Architecture, Defined

As part of our initiation of the IT Architecture Journal, we've been trying to help define the landscape - we're continuing that trend today with an introduction to Application Architecture. These definitions are being provided as a foundation - both for IT Architecture education in general but also as part of our larger goal to help harmonize industry terminology and expectations regarding IT Architecture. And of course these articles also serve as a foundation for us as we delve into more detailed or specific topics later. Like so many other aspects of Architecture - there is quite a lot of overlap and complexity connected to Application Architecture.

The definitions we provide here are not meant to be mere academic representations of some well understood a priori knowledge, but rather are based on experience from many years of architecture practice. The goal here is to provide the most pragmatic explanations possible - and to clearly link what might otherwise appear to be conflicting concepts. In other words, we are attempting to demystify IT Architecture.

First, let us address the overlapping or conflicting terms. Application Architecture is often referred to as System Architecture, Systems Engineering or even Software Architecture or Design (or even Software Engineering). All of these terms are roughly analogous and can fit neatly within the scope of Solution Architecture (which is itself a subset of IT Architecture), yet there are aspects of Application Architecture practice which are stretching the bounds of what it encompasses. Even within some of these other terms there lies deeper hierarchies of sub-terms; for example within Software Engineering we'd likely find both Object Oriented Design and GoF (Gang of Four) Pattern Design. When we begin to discuss SOA (Services Oriented Architecture) and SAAS (Software as a Service) then things will get even more complicated as we will see.

Application Architecture would be focused mostly within the Business Tier of this view

Application Architecture, Defined
Application Architecture is the set of design activities and artifacts associated with the production or management of solution logic. Solution logic is the set of business rules and functions that support key processes or workflows associated with any given organization. While Application Architecture requires and / or manipulates data and sits atop an infrastructure it often viewed separately from those elements in organizations where infrastructure as a service or data management organizations are in place. There are usually clear delineations regarding where application responsibility ends and data or infrastructure etc. begin.

Application Architecture generally follows systems and software design guidelines or expectations; so an Application Architect would likely use UML to create Class diagrams to help model data for their application, whereas a Data Architecture would likely use an ERD or Dimensional to do something similar (even though both are working with the same data in that context).

Application Architecture is represented in most of the major EA Frameworks; in TOGAF and FEAF it is referred to as Application Architecture, in Zachman it Systems. In DODAF it is roughly equivalent to Systems and Services Views.

Skills Required
To work as an Application Architect, you'll need the following skills:

• The ability to design and problem solve.
• The ability to use UML notation to capture standard designs.
• The ability to apply standard patterns or create / define your own.
• An understanding of the underlying technologies involved. Object Oriented design in Java is generally quite a bit different than scripting with JavaScript and Perl in a LAMP stack. 
• The ability to understand and harness / exploit an Application Development Lifecycle and associated tools - this is referred to in industry as Application Lifecycle Management (ALM). 
• The ability to translate business needs into business logic and to define integration of that logic within the system environments available. 

A Note about SOA
So, what makes SOA a game-changer for Application Architecture? Basically it has to do with implied integration across other "tiers" of the overall Solution Architecture. SOA brings with it a sort of mega-pattern for design that includes expectations for data access and infrastructure exploitation and thus it is sometimes hard to abstract these non-application elements of the solution. While it's easy enough to abstract all this from the developers (you simply give them the guidance and the self-service development environments) for the architect it's not so simple. We'll be talking about this more soon...

Copyright 2014,  Stephen Lahanas

#ITarchitectureJournal
#StephenLahanas
#Semantech-Inc

Read More

The Difference Between IT Architecture & Subject Matter Expertise

This is a question that comes up very frequently in my line of work, although it isn't always posed as a question. The question in its most generic form goes something like this - "how much specific technical expertise should an Architect have." That's a tough one isn't it, especially if one considers themselves an IT Architect as a opposed to something much more specific (let's say a Hyperion or Informatica Architect).

We've defined IT Architecture in this blog but we haven't covered Subject Matter Expertise in great detail yet. What does it mean to be a SME?

Subject Matter Expertise, Defined
Being a Subject Matter Expert requires an in-depth understanding in one specific topic; that topic can be a field of practice, a theory or some particular product or tool (or set of related products). In the context of IT, that expertise necessarily implies the ability to perform any related tasks (configuration, design or development) at a more or less 'master' level. This further implies that the SME role often actually encompasses several typical roles; Architect, Lead Developer and Administrator.

So, this definition obviously begs a number of interesting questions:

1. Should Architects be expert in one or more technologies?
2. Can Architects become experts in many (or all of) the various technologies they work with?
3. How important is the discipline or practice of architecture as compared to Subject Matter Expertise?
4. Can someone who is only expert in one area then handle or deal with a large heterogeneous environment?

All of these questions are being driven to some extent by the expectations of organizations that would hire Architects or SMEs. Part of the dynamic that's present these days is that many enterprises want both things (the Architect and the SME) but only want to pay for one person to do it - thereby merging the roles. In some of those cases, this desire is somewhat justified in that there may be relatively little design effort required but a great deal of configuration, follow-on development and many coordinated deployments.

But is that really a question of whether one ought to have an SME skill set as opposed to an IT Architecture skill set - or is it a question of whether the assignments and team composition is correct? Let's consider some reactions to all these questions:

• It is not realistic to assume that any one technical resource will become expert in every technology within their particular environment. It is also not necessary.
• Architecture is much more focused on design than deployment - granted it plays an oversight role once things have left the drawing board, but that merely means the Architect is available for other design duties. It is unlikely in most cases that the designer will be the best development resource (either due to higher cost or the fact they spend more time designing than developing).
• Architects and SMEs can work together - as separate roles - quite effectively. There is no reason that both or either role has to be full-time - they can be brought in as necessary.
• It will be necessary at times for Architects to "dive-in" to one specific area or another to solve a problem. Sometimes in doing this the Architect then becomes an expert in that area. In many other times, the Architect understands the common technical standards, the underlying principles and perhaps key features but does not become the expert. That's ok as well, as long as what the Architect is trained to do is solve problems - regardless of the technology involved and that others have similar dedicated roles and skills sets that complement the team.

I've heard folks voice concerns that they want Architects who can implement rather than merely design. This is an interesting situation and perhaps reflects a bit the relative lack of consensus surrounding IT Architecture in industry. If we were to compare ourselves to our brick and mortar brethren, how common would it be for us to expect a building architect to be scaling a high rise to place beams or pour concrete or even to determine the proper layout of pipes and wires. The traditional architect designs with all of those considerations in mind, works with engineers and contractors to solve problems and is ultimately responsible for the outcome of the project. He / she is in an expert in what they do but not an expert in every possible technology involved in building today's high-rises.

IT and building construction are both similar and different at the same time. But the core question of expanding complexity is fundamentally the same. The more complex your environment, the more likely you will need a division of labor, and at one end of the spectrum you will find the Architect who has design command of all the relevant technologies within a unified solution context. At the other end you will have SMEs who know everything there is to know about individual pieces of that puzzle. Occasionally this is one and the same person - but not very often - nor should it be.

Copyright 2014,  Stephen Lahanas

#ITarchitectureJournal
#StephenLahanas
#Semantech-Inc

Read More

Principles of Integrated Design

When people think about design they often visualize the Apple product line or user interface experts coming up with cool graphics to deploy in websites. Design in IT is much more than that though; while they are many individual piece parts of an overall lifecycle that are concerned with specific design tasks, say like the User Interface, the entire lifecycle itself is also driven by the need to create functional systems. This holds true for both hardware and software, as well for complex enterprise initiatives like ERP or even Cyber Security.  If we consider "Design" as a verb - the activity which produces something - then "Architecture" is the noun which design creates. "Integrated Design" represents the ability to conduct design activities within a uniform enterprise context.  Integrated Design is highly analogous to IT Architecture as a practice field; however there are some aspects of design which seem to go beyond the bounds of what most folks considered Architecture.

To get a better sense as to how Integrated Design, is applied let's look at some typical enterprise IT projects.  In projects such as a legacy migration or an ERP transformation you're likely to find the following types of design activities:

• Enterprise Architecture
• Data Modeling
• Data Architecture Design
• Business Process Modeling
• User Interface Design 
• User Experience Design (and yes this is somewhat different as User Experience crosses platforms and technologies these days)
• Application & Service Design
• Infrastructure and / or Network Design

These tasks are generally vital to the success of an overall project, yet many if not most of them happen in design "stovepipes."  The concept of Integrated Design is similar to how some people view Enterprise Architecture except that Integrated Design is less focused on formal frameworks or specific EA tools - in other words it is more likely to provide an all-encompassing umbrella for all types of design activity.  For example, some people might not consider a detailed Network Design or Data Center Design to be part of an Enterprise Architecture although others might - there is no standard definition or boundary. Integrated Design however, is a more flexible, perhaps even Agile way to tackle what is in fact one of the biggest challenges most enterprise face today - complexity.

Design and Architecture often merge in user interfaces or "Digital Experience" solution

The complexity we're referring to is the ability to reconcile and align all of the various designs (and design activities) associated with a complex enterprise or specific enterprise initiative into a single solution utilizing a unified lifecycle management framework.

There are three simple principles behind Integrated Design:

1. All design activities must share the same lifecycle methodology and automation framework. Integrating models or architectures can sometimes be referred to as meta-integration and as the name implies it represents an added layer of complexity to an already complex project. A shared automation framework does not have to be an EA tool.
2. All designs must be targeted to the shared capabilities.  There are finite set of capabilities associated with any given solution. By pegging or mapping each design to its relevant solution capabilities we achieve and maintain automatically a certain level of integration. 
3. All design (on the project or within the solution) is related. While some designs may only be complementary rather than dependent, all designs and thus all later functionality have relationships to one another. Understanding those relationships up front is perhaps the greatest risk reduction technique available in IT today.  These relationships are both broad and narrow in nature (and can extend down to the data element level).

Design is and always will be a creative endeavor; however, if we keep in mind the principles of Integrated Design we can see how that creativity can be merged along with systems engineering to produce innovative and successful solutions.

Copyright 2014,  Stephen Lahanas

#ITarchitectureJournal
#StephenLahanas
#Semantech-Inc

Read More

Business Architecture, Defined

There are a number of disciplines within IT Architecture that tend to overlap, part of this is because the definitions haven't achieved industry wide status (for example, various EA Frameworks define similar functions with different terms and they also often use the same terms differently). Part of the problem in this case though, has to do with the relative ambiguity regarding how and where IT Technology and Business drivers can or should mix. There is also some confusion in relation to where Business Analysis ends (as a practice) and where Business Architecture begins. We will help to answer some of these questions in this post.

Business Architecture has many potential components and drivers

First, let's attempt to apply a standard definition to the practice area:

Business Architecture - This represents the entire set of design activities associated with translating business needs and expectations into technical solutions. Those design activities include the following typical tasks and / or artifacts: Context Maps, Concept Mapping, Process Engineering, Capability Maps, Business Rule Definition, Conceptual Data Modeling, Domain Modeling, Class Definition, Object Definition and Business Glossaries. This list is not exhaustive but it does highlight that Business Architecture includes both analysis and design. It also shows that there are clear intersections with data and application (system) architecture. Business Architecture tends to be the first area of design focus within the overall lifecycle of any given solution or set of solutions.   

So, in reality there are many people who work as Business Analysts that in fact perform Architecture tasks. On the flip side, many architects are asked to define and manage requirements or otherwise work side by side with business stakeholders and analysts to help better understand workflows and data contexts. The lines here are indeed fuzzy as the type of architect asked to do all of this may be referred to variously as a Solution Architect, Data Architect, Project Architect or perhaps even something else. Use of the term "Business Architect" is still relatively uncommon. The reason for this is most likely that many organizations want Architects who can perform those upfront Business design activities but also follow that with more specific types of solution design tasks and artifacts.

Business Architects often use EA tools to perform these types of activities; products like Mega or Troux allow Business Architecture to managed in the context of TOGAF driven enterprise. Other tools can be used to manage governance, application policies, business rules and other types of design artifacts.

Another important aspect of the role of Business Architects is their focus on Lifecycle Management. Now, we will address Lifecycle Management in more detail in another upcoming post, but the key elements involved are Governance and Portfolio coordination. So, Business Architects also often help to oversee Governance processes, help develop Business Cases, estimate project costs and schedules and otherwise support portfolio planning processes.

When we view all of this in the larger context of IT Architecture, it is worth considering the larger question whether all architects ought to have some experience working on the "business side" of solutions in order to better understand why some projects fail and how to better align business goals with IT reality. Another facet of this topic is the notion of what an Architecture Manager does. That is a role that isn't necessarily associated with Business Architecture per se but nevertheless includes quite a number of similar tasks and artifacts. Any IT Architecture manager (regardless of the type of architecture being performed) must be able to manage project planning, cost estimation and other types of team collaboration activities.

We will examine many of the terms and concepts introduced here in greater depth in upcoming posts.

Copyright 2014,  Stephen Lahanas

#ITarchitectureJournal
#StephenLahanas
#Semantech-Inc

Read More

Introducing Design Patterns

Design Patterns are an important part of software engineering today, but what many folks don't realize is how ubiquitous they are becoming within the full spectrum of IT activities. Part of the reason this is the case is due to the flexible nature of Patterns.  Mastering Design Patterns is a critical part now of most IT Architects' roles and from a larger perspective is also becoming increasingly import for most enterprises.

Before discussing how to manage Design Patterns, we first need to define what they are:

Design Pattern - This represents a description and (usually) a visualization of some specific aspect of a larger design solution. Patterns can employ standard or non-standard notation and may or may not be used as Enterprise Architecture (EA) artifacts. The most important characteristic of a Pattern is that it encapsulates at least one significant design principle. Patterns are usually combined to produce comprehensive solutions.  Patterns can also be used as code "templates/stubs" to facilitate rapid development of applications and services. 

Beyond this basic definition, Patterns can mean many things to many different types of IT specialists. For example, Cyber Security experts can manage Patterns as mechanisms for modeling Threat behaviors. Network Engineers can view combinations of Data Center layouts and hosting configurations as Patterns. Data Architects could, if they wished, view prototypical Conceptual Models as Patterns and in fact a Canonical Model could be considered a specific type of Data Pattern applied across multiple enterprises. This sounds a little strange perhaps because many people associate the concept of Patterns with Object Oriented Programming and the original Gang of Four book of OOD patterns. Yet, there is nothing that explicitly states anywhere that Patterns have to be limited to application design (Object-Oriented or otherwise).

The flexibility of the Design pattern is simultaneously both its greatest asset and drawback.  It's an asset because it allows architects, developers and engineers the flexibility to define re-usable solutions for common design challenges. It's a drawback because it tends to promote decentralized management (or perhaps no management at all) of design artifacts and may also lead to more situations where solution elements are designed out of context of one another.  So how do we manage something that is flexible, effective yet often disruptive? Here are five suggestions for how to master Design Patterns:

1. Create your own Design Pattern Taxonomy - Keep in mind that there is no standard approach from industry as to how an organization (or individual) ought to manage patterns (unless of course you wish to blend it with your EA approach and integrate them within one of several EA Frameworks). So what's being recommended here is to create a taxonomy of the types of Design Patterns you think you may need. This might include application patterns, data patterns, UI patterns etc. - whatever makes sense.

2. Come with a Standard approach for Pattern Visualization - Many people like to use UML for this but quite frankly there's some design problems that UML isn't so great in representing. The benefit of using UML is that it should make alignment with your ALM easier (in most cases).

3. Come up with Standard Template for Pattern Description - You'll need to ask yourself here just how much information is required. While this shouldn't become a huge specification you will need to provide enough information so folks not involved in the original design won't have to come back and ask what you really meant.

4. Standardize your own Design Terminology - As we all know there's many ways to describe the same thing. Building a glossary that can support all of the Design Patterns you may create will come in handy for you and your enterprise. (again for those who are already utilizing an EA approach you can probably borrow from that or at least make sure that any Patterns not managed as part of the EA share common terminology).

5. Align your Design Patterns with your ALM / SDLC - This is perhaps the most important of the five recommendations. The majority of enterprises are already working with some sort of Application Lifecycle Management methodology. Often times ad hoc Design Patterns don't get "baked in" to the ALM. This can lead to much confusion and generally makes the overall design approach somewhat less efficient than it might otherwise be.

Design Patterns are meant to accomplish three critical things:

• Increase the speed of the design process.
• Ensure that lessons learned are adopted.
• Reduce overall costs by finding ways for certain capabilities to be reused. 

Actively managing Design Patterns helps to ensure that all three goals can be achieved.

Copyright 2014,  Stephen Lahanas

#ITarchitectureJournal
#StephenLahanas
#Semantech-Inc

Read More

Data Architecture, Defined

Someone asked me what at first sounded like a very straightforward question last year; "what is Data Architecture" - or more precisely, what does it mean to you. Usually, I'm not usually at a loss for words when it comes to expounding upon IT Architecture related topics - but it occurred to me at that moment that my previous understanding of what Data Architecture really represents is or has been a little flawed or perhaps just outdated. So I gave a somewhat convoluted and circumspect answer.

Where does Architecture fit within this picture?

The nature of what's occurring in the Data domain within IT is itself changing - very quickly and somewhat radically. The rise of Big Data and proliferation of User Driven discovery tools represents quite a departure from the previous more deterministic view of how data ought to be organized, processed and harvested. So how does all of this effect Data Architecture as a practice within IT (or more specifically within IT Architecture)?

But before we dive into the implications of the current revolution and its subsequent democratizing of data, we need to step back and look again the more traditional definitions as to what Data Architecture represents. I'll start with a high level summary view:

Traditional Data Architecture can be divided into two main focus areas; 1 - the structure of the data itself and 2 - the systems view of whatever components are utilized to exploit the data contained within the systems. Data in itself is the semantic representation or shorthand of the processes, functions or activities that an organization is involved with. Data has traditionally been subdivided (at least for the past several decades) into two categories; transactional and knowledge-based or analytic (OLTP vs. OLAP). 

Now we'll move to a traditional summary definition of Data Architecture practice:

Data Architecture is the practice of managing both the design of data as well as of the systems which house or exploit that data. As such, this practice area revolves around management of data models and architecture models. Unfortunately, the application of Governance within this practice is sporadic and when it does occur is often split into two views: governance of the data (models) and governance of systems (patterns and configurations). 

So, that seems to be fairly comprehensive; but is it? Where does Business Intelligence fit in - is it part of the data management or system management - is it purely knowledge focused or does it also include transactional data? For that matter, do Data Warehouses only concern themselves with analytic data or can they be used to pass through transactional data to other consumers? And isn't Big Data both transactional and analytic in nature? And BTW- how do you model Big Data solutions either from a systems or data modeling standpoint? Now - we start to begin seeing how things can get confusing.

We also need to take into consideration that there has been an attempt made to standardize some of this from an industry perspective - it's referred to as the Data Management Book of Practice or DMBOK. I think in some ways it's been successful in attempting to lay out an industry taxonomy (much like ITIL did) but not as successful in linking that back into the practice of Data Architecture. The following diagram represents an attempt to map the two together...

There isn't a one to mapping between DMBOK and data architecture practice, but it's close

One of the areas that the DMBOK has fallen short is Big Data; my guess is that they will need to rethink their framework once again relatively soon to accommodate what's happening in the real world. In the diagram above, we have a somewhat idealized view in that we've targeted a unified governance approach for both data modeling and data systems.

Let's take a moment and discuss the challenges presented by the advent of new Big Data and BI technology. We'll start with BI - let's say your organization is using Oracle's BI suite - Oracle Business Intelligence Enterprise Edition (OBIEE). Within OBIEE you have a more or less semantic / metadata management tool called Common Enterprise Information Model (CEIM). It produces a file (or files) that maps out the business functionality of all the reports or dashboards associated with the solution. Where does that fit from an architecture standpoint? It has a modeling like interface but it isn't a 3rd normal form model or even a dimensional model. It represents a proprietary Oracle approach (both as an interface and modeling approach). It allows you to track dimensions, data hierarchies and data structures - so it is a viable architecture management tool for BI (at least for OBIEE instantiations). But some traditional Data Architecture groups would not view this as something the architects would manage - it might handed off to OBIEE administrators. This situation is not unique to Oracle of course, it applies to IBM / Cognos and other BI tools as well and there's a whole new class of tools that are completely driven by end users (rather than structured in advance from an IT group).

Now let's look at Big Data. Many of the Big Data tools require command line interface management and programming in order to create or change core data structures. There is no standard modeling approach for Big Data as it encompasses at least 5 different major approaches (as different say as 3NF is from Dimensional). How does an architecture group manage this? Right now, in most cases it's not managed as data architecture but more as data systems architecture. The problem here is obvious; just as organizations have finally gained some insight into the data they own or manage - a giant new elephant as entered the room. How is that new capability going to impact the rest of the enterprise - how can it be managed effectively?

Back to the original question - what is Data Architecture. I'd like to suggest that the practice of Data Architecture is more than the sum of its traditional activities. Data Architecture is the practice of understanding, managing and properly exploiting data in the context of problems any given organization has to solve. It is not limited by prior classifications or practice but has a consistent mandate to be able to represent and hopefully govern in some fashion data as an asset (internally or shared collaboratively). Data Architecture as we know is going to change quite a bit in the next two years and that's a very good thing.

Copyright 2014,  Stephen Lahanas

#ITarchitectureJournal
#StephenLahanas
#Semantech-Inc

Read More