Data Import Options

When discussing data imports to CRM there are 2 distinct scenarios:

Data Migration

Data Migration generally refers to a large scale conversion effort and is typically performed as part of a CRM Go Live effort where data is migrated from the legacy system and is usually quite an involved effort. This exercise is technical in nature, typically performed by IT professionals and requires a great attention to detail. Correspondingly the migration logic is also usually quite involved. Meaning that there can be quite a bit of data transformation as part of the migration exercise. And if migrating from another system, you typically want to be able to connect to the data in the legacy database rather than extracting the data to an intermediate CSV file. So as a general rule of thumb when doing this type of data migration you typically want to be using a tool like Scribe to make it happen.

Importing Data

Importing Data generally refers to a more specific one-time or ongoing requirement to import data into a live CRM environment. For example, the need to import new leads into the CRM database. Typically this is a more straight-forward "one-to-one" exercise (i.e. no data transformation required) and if this is the case, the out of the box "Data Import Wizard" can be used for this function.

One thing to note is that while the "Data Import Wizard" has advanced by leaps and bounds, it still has one chief limitation - it only can import new records. If you're looking to use the wizard to update existing records, I'm afraid you're fresh out of luck. And if that is a requirement you'll once again need to look for a 3rd party tool such as Scribe.

For an excellent general overview of the features and functions of the Data Import Wizard please refer to this post. At the end of the post it lists a number of 3rd party tools that can be used to fill the gap in functionality of the Data Import Wizard should you encounter it. Although none of these products have what I would term a "no-brainer" price. Below is a rough comparison in pricing of the various tools on the market (as of this writing):

1. Scribe: 
• $3000: 15 user license
• $5500: 100 user license
• $1900: 60 day migration license
2. Inaport: 
• $1799: standard (should be sufficient for import function)
• $3499: professional
• $1195: 30 day migration  "professional" version which is apples to apples comparison for the Scribe migrate license)
3. Import Manager (best option if end user interface is required)
• ~$2500
4. Import Tool (attractive pricing) 
1. ~$1300: Full Version
2. ~$325: 60 day migration license

The following tools don't seem to be realistic candidates based on their price point:

1. Starfish 
• $4800/year: Basic Integration
• $1495: 60 day migration license
2. eOne SmartConnect 
• $4500 (not sure if this is a one time or per year fee)
3. QuickBix 
• $7000: 100 user license ($70/user)
4. Jitterbit
• $800/month: Standard Edition
• $2000/month: Professional Edition
• $4000/month: Enterprise Edition

The following tools do not seem to fulfill the requirement:

1. CRM Migrate
• Tool specifically built for migrating SalesForce to CRM
2. CRM Sync: 
• Very little information and no pricing for this tool

CRM Live Purge

So you've successfully conducted your Dynamics Sure Step Proof of Concept and your customer has decided to pull the trigger and implement CRM. Congratulations. You now quickly transition into implementation mode and the first step you need to do is purge your pilot data from your CRM environment. Or... you've successfully migrated your CRM data but you discover through testing that some data was deficient and you need to purge and reimport. Whatever the scenario your requirement right now is that you need to purge your data.

What is the best way of going about this in a CRM Live environment?

Well, technically this post does not only cover CRM Online - it can address purging in any CRM environment. However the reason why I'm addressing the CRM Live environment is because you can of course only purge data via the front end, 3rd party tool (e.g. Scribe) or using some programmatic solution that leverages the CRM APIs.

... And of course you should be doing the same in any environment, but let's just say that pragmatism sometimes sets in. For example, the size of the data makes the purge exercise a very slow one. So while I'm not going to go into it there of course "unsupported" ways of purging a CRM database in an on premise environment that are a thousand times more efficient than going via the API.

Anyway, back to the issue at hand - how do we go about purging? There are a number of ways:

1. Programmatic - Create a small .Net application. This should be pretty straight forward but requires a coded solution and you could run into problems with timeouts for large data sets.
2. 3rd Party e.g. Scribe - Build DTS's to purge the data. Again quite straight forward but assumes you have access to such a tool
3. Front end - you can of course use CRM's front end. However going from entity view to entity view and deleting that way would be highly inefficient both because you are limited to 250 rows per deletion and because you'd need to remember to repeat the same steps each time - assuming you need to purge more than once. So a more efficient way is to use the Bulk Record Deletion Tool. The remainder of this post will discuss using the Bulk Record Deletion as a purge tool.

The benefit of using the Bulk Record Deletion tool is that it essentially documents the steps required for the purge and it can subsequently easily be re-used to perform subsequent data purges. Below are the steps required to facilitate this:

Use the Bulk Deletion Wizard to delete all records from the entity in question (or you can limit this to a subset of records you wish to purge by filtering appropriately). Repeat for each entity you wish to purge.

Name the job appropriately and set a recurrence for 365 days. If you do not set a recurrence you will not be able to reuse the job at a later point in time so this is set only to facilitate reuse.

Now all your purge jobs will appear under the "Recurring Bulk Deletion System Jobs" and when you want to re-run the purge you can highlight all your purge jobs and select More Actions | Modify Recurrence to re-run the jobs immediately.

NB: It goes without saying that you need to create a very conspicuous reminder to cancel your purge jobs once you have cutover to production or else you might be in for a nasty surprise down the line... Use responsibly and at your own discretion.

Web Resource Utility use case

In light of my previous post, I thought I'd give a use case for the Web Resource Utility.

In CRM 4.0 we implemented a customization for contacts such that an image of the contact would be present on the contact form. This was done by referencing files on the file server. Although it's possible to do it the same way in 2011 a better approach is to have the contact images stored as web resources (for example, ease of access for remote users).

The question therefore was how to go about uploading all the files from the file server as web resources. Doing it manually would take way too long... Enter the Web Resource Utility.

Basically, all that is necessary is to point the Web Resource Utility to the folder containing the image files and click upload. Well, at least in theory...

One of the issues faced was that the image files contained spaces in them and the Web Resource Utility did not accept spaces...

One option would be to update the source code of the Web Resource Utility as we of course have access to that. Another option, would be to manipulate the files to remove the spaces using a file multi-renaming tool such as the one shown to replace all cases of " " with "_" (underscore).

And subsequently the files can be uploaded using the Web Resource Utility -

One other slightly painful feature of the Web Resource Utility is the presence of the leading "/" for the imported file names.

Once again, the source code could be updated to remove this but for the time being I've decided to live with this as is. The end result is that we have managed to bulk upload hundreds of images without breaking a sweat.

Web Resource Utility for Bulk Upload

In the old days of 4.0, it was quite a simple matter to add custom icons to custom entities by simply uploading the files in the entity customization area thereby sprucing up your CRM installation by having more different icons for different entities rather than the default drab-looking widget icon.

This is of course still possible in CRM 2011 but now the icons need to be added as web resource which - although an overall better design - it tends to add quite a few more steps to this simple requirement.

Luckily Microsoft has provided the Web Resource Utility as one of the tools that forms part of the CRM SDK which can be compiled and use to bulk import web resources.

If you prefer to avoid the compilation step you can download a compiled version from here.

You can then use this utility to bulk upload web resources into your CRM installation and then just reference them as necessary.

You can of course use this utility to upload any type of web resource although images are probably going to be the most likely.

See the following post for a review of how to connect to a CRM Live instance using this tool.

Importing Customizations in CRM 2011

My previous post dealt with approaches for exporting  customizations from CRM.  Importing the customizations has also changed a little.

In CRM 4.0, the export file was self contained and it could be imported from anywhere. You also had the ability of importing zip as well as xml files. When you imported files such as the Site Map and ISV.config files, the changes took effect immediately.

In CRM 2011, the exported zip contains some additional files such as those shown below:

The only file you want to be editing is the "customizations" file. Once you have finished making your edits (e.g. Site Map modifications) you'll need to zip up the parent folder (SiteMap_0_1 as shown in the screenshot above) and import the zip file as shown below:

Finally, note that unlike 4.0, you'll need to publish for ALL imports before the changes will take effect.