Uploading files from the file server using Scribe

The Challenge
You need to upload files from a file server into CRM. A good example of this requirement might be if you're migrating from another CRM system where attachments to CRM entities are stored directly on the file system and referenced from the CRM records rather than being stored as a blob directly in the database as is the case with Microsoft Dynamics CRM. When a file is already stored as a blob in the database, the conversion is fairly straight forward. But how do you about uploading files that are stored on the file server as attachments to MSCRM records?



The Solution
Fortunately the solution is also pretty straight forward if you have the option of using Scribe for migration. As a picture is worth a thousand words, I will illustrate it with a screenshot:

In short, all you need to do is to map to the "annotation" entity and ensure that:

• isdocument = 1
• vfAttachmentFileName = path of the file on the file server

And that should pretty much do the trick.

Support for Scribe Insight v6.5.x to be discontinued

Scribe has just announced the end of life for it's Scribe Insight 6.5 product line. Is this something to be worried about if you are still on the 6.5 version? In short, not really. Very often when products get to their "end of life" stage, they tend to be pretty stable platforms. I think Scribe Insight 6.5 falls into this category and therefore I do not consider it "do or die" in terms of upgrading. You probably  haven't had to log a ticket to their support group for quite some time now and chances are you are unlikely to need to do so going forward.

Does this mean I'm recommending against an upgrade to the Scribe 7.x product line? Absolutely not. First of all I'm just trying to prioritize the upgrade requirement amongst the many competing priorities that a company might have. I would therefore not place performing a Scribe upgrade as being a crucial upgrade unless there were some compelling reasons to do so. A compelling reason would obviously be if you have been dealing with a number of support issues. Another compelling reason would be if your company's tolerance levels (whether it be mandated or otherwise) for using an officially "unsupported" product are low. Finally, a compelling reason might be because you want to take advantage of the new features that the new product line has to offer.

Indeed, the last reason mentioned is really a reason for upgrading in and of itself rather than having to deal with end of life support issues. Although perhaps the latter is enough to push you over the edge so to speak...

Officially I do most definitely recommend an upgrade to the latest version of Scribe Insight. This version has been out for quite some time now and therefore you are no longer on the bleeding/cutting edge by doing so. You might even be considered to be a laggard at this stage in the game. That notwithstanding, my reason for this recommendation is most definitely because of the 3rd compelling reason I gave for going for an upgrade namely, to take advantage of the new features that the product line has to offer.

Let me explain. Every new product lines offer capabilities that the previous one did not. The lack of these features may very well impact the way you go about making a design decision. For example, before CRM 4.0 you did not have the benefit of many-to-many relationships and the design approach taken in CRM 3.0 for how you go about representing certain relationships may well have been different than how you would have done so using CRM 4.0. Of course the same applies to CRM 2011. A more relevant example - at some point in time Scribe introduced the Query Publisher into their product offering. Having this feature allowed for more options in terms of integration approaches - something that you would not have been able to even consider in previous versions. The result might be that you end up redesigning some of the pre-existing integrations at some point in the future to take advantage of the new capabilities.

Scribe Insight 7.x introduced a number of new capabilities over it's predecessor. In my opinion, the "game changer" of this release (as it relates to design approach as illustrated above) is the "multi-target" capability. Without going into a whole lot of detail, this feature offers a very oft-requested capability that can simplify and even eliminate some integration threads. Having this capability available may very well change how you go about making your integration design decisions for new DTS's and it is primarily for this reason that I make my upgrade recommendation.

Scribe Online

Scribe Online is Scribe's latest product offering. The current offering is limited to work with Microsoft CRM and basically allows for a CRM Online solution to be replicated locally. The value add of such a solution is perhaps a little less obvious than Scribe's flagship offering: Scribe Insight. Simply put - whenever you are entertaining the notion of building any sort of integration with Microsoft CRM (or for that matter standalone databases, or any of the other Scribe templates such as GP, Salesforce etc.), Scribe Insight is going to have to be one of the options considered for accomplishing this. The benefit of using a tool such as Scribe Insight over say, a custom built solution, is fairly substantial and I've reviewed some of the considerations in a former post.

The Scribe Online Replication Service though has - at least to me - a somewhat less obvious niche.

First and foremost, this offering is not meant to be a backup solution which - in fairness - Scribe has made fairly clear. Although CRM Online cannot be backed up as you could for say an On Premise implementation, I think that it has to be kind of axiomatic of the cloud model that your data is secure (I'm sure Google and SalesForce would agree...). Having said that, it could provide companies with a "sense of comfort" that their online data is also available locally. But I personally would not encourage that type of thinking as it goes against the kind of thinking that is necessary in order to move into the cloud.

I think it can also be argued that Scribe Online's Replication Service is perhaps a little anachronistic in that it kind of defeats the purpose of going online in the first place. The major driver for going to CRM Online in the first place is so you can subscribe to the SaaS model and not have to deal with servers, environments, maintenance etc. But yet - here we have Scribe Online that will require you to have a local server with SQL installed in order for it to work...

If so, I think it is fairly clear that the Scribe Online Replication Service is going to be required in a given number of scenarios. So what are these? Below I will attempt to outline where I think such as service can come in handy which is based both on my own considerations and information garnered from Scribe demos:

• Reporting - As highlighted previously, reporting in CRM 2011 Online is significantly more robust than it was in 4.0. And therefore for most companies - at least initially - you should be able to handle the reporting requirements that you have using the CRM Online model. But what if you have sophisticated reported needs? Or better yet - what if you want to create a data warehouse that you can use for setting up Key Performance Indicators (KPIs) to enable you to slice/dice your data in various ways that you cannot do using your operational database (i.e. CRM Online)? Such reporting is typically a more sophisticated requirement to see trends, perform data mining, etc. in order to analyze your business. In such a case, Scribe Online can be a perfect complement to your CRM Online offering allowing you to bring the data into your local environment where it can be combined with Business Intelligence (BI) tools to deliver the goods. I believe that Scribe may also be looking at partnering with BI vendors that may facilitate the delivery of pre-packaged metrics or something along those lines. Time will tell...

• Data Cleanup/Import etc. - Another use could be if you have a need to clean up (e.g. de-dup) your production data but do not want to do it in the live environment. This requirement combines the capability of Scribe Insight with Scribe Online Replication Services. Using this approach, you could bring your data to a local environment, perform all the clean up in that environment, and when you're done you could use Scribe Insight to update your production data.

These are the major benefits that I can think of right now. And based on this, I believe Scribe Online only has value when using CRM Online as opposed to Partner Hosted. With the latter, although your data is still "online" you typically will have more access to your environment, database etc. than in the pure online model. I think it also bears mentioning that the cost of Scribe's Online Replication Service is fairly nominal at $99/month which makes it that much easier to take the plunge should you wish to experiment.

It's still very early days with regards to Scribe's Online offering and the Replication Service is only the first offering using this platform. It will therefore be interesting to see where this product heads as more capabilities get added. For those who want to know more about the strategic direction of this product, you can review Lou Guercia's (Scribe's president and CEO) interview with IT Business Edge.

On Integration

On numerous occasions I've had the question posed to me whether it is necessary to use a tool such as Scribe for building integrations vs. using CRM web services (or using eConnect for integrating to GP as opposed to Scribe and so on and so forth).

I think the question itself indicates that there is perhaps a misunderstanding of what the term "integration" really is all about.

First of all, by way of disclosure, I highly recommend using an integration tool for tying systems together. Scribe is an excellent example of a tool for building integrations with MSCRM that I fully endorse. The rest of this blog entry will attempt to both answer the question as well as justify this recommendation.

In short, when it comes to integration, it is really not a question of Scribe vs. CRM web services but rather Scribe and CRM web services. In short, there is nothing stopping you from creating an "integration" using the CRM SDK and web services but to do so is reinventing the wheel to a large extent. It needs to be remembered that "integration" is not just the act of making one system talk to another - that is arguably the easiest part of the long term effort. A successful integration strategy has a number of other components, including:

• Loosely tying systems together. We do not want the source application to be tied to the target application in such a way that if the target application is down, the source application is down too. Such would be the case if we were to build a simple “synchronous plugin” integration approach (an "asynchronous plugin" would result in the integration disappearing into the good night when the second system is down).
• Queueing. It is important to be able to queue integrations up such that under load or under the scenario described above where one system is down – items get written to a queue and are integrated as soon as the target application can accept them. 
• Retries. When an integration fails it is important to be able to have an effective automatic as well as manual retry mechanism. Sometimes an integration fails for a condition that will self correct and will succeed on the next retry. For troubleshooting purposes, it must be easy to have a way to retry an operation to get to the root cause. In short an integration must either succeed or if it fails it should fail “noisily” so it can be proactively addressed.
• Alerts. It’s important to be able to set up a proactive alert mechanism if there are issues with integration. This helps to quickly identify and resolve issues as they arise.
• Customizability – It’s important to be able to easily modify an integration for changing business needs. It’s equally important to use a tool that is supported, has easy access to training and/or qualified resources. It also helps for the tool to be a “configuration” rather than “development” tool.
• Console Management. It’s important to have a console that provides a single view of integration activity (especially when there are multiple integrations going on) as well as an ability to stop and start integrations in a simple manner.

A product like Scribe brings all of these capabilities out of the box. Scribe also leverages what are referred to as "adapters" for integrating to products such as MSCRM. And when we refer to the Scribe adapter, we mean the capability of Scribe to automatically expose and talk to all MSCRM web services; similarly when we refer to the Scribe adapter for GP we mean the capability to automatically expose and talk to all GP eConnect routines. It's not a matter of either/or as posed above but rather whether you want to build a robust integration strategy using off the shelf tools that will stand the test of time.