Blog

BI Conference 2010: PowerPivot

Thu, 10 Jun 2010 13:51:43 GMT

I have attended a couple of PowerPivot sessions at this conference 1) because it’s so damn cool and 2) very practically, it is one of the only real new product releases prior to this conference (yes , SQL 2008 R2 is out, and SSRS enhancements are nice and MDS is, well, we can cover that later….).

The sessions that I attended were all very well thought out, the speakers were enthusiastic and very knowledgeable – thought there were a couple I couldn’t get into (unfortunately, an all to common problem at this BI conference).

The big value for me out of these PowerPivot sessions were some of the links and resources that I gleaned out of the presentations, shared here:

Also, for any Twitter late bloomers out there like me, utilize TweetDeck to follow keep up with PowerPivot conversations using the #PowerPivot hashtag – it is a great way to find all the knowledgeable people out there working with, developing and using PowerPivot.

You may call him Master

Tue, 08 Jun 2010 18:37:53 GMT

Congratulations to my friend and colleague Greg LeClair for officially becoming a SQL Server Master! It became official today.

For anyone unfamiliar with the Master’s program, check out the link above – grueling program. Greg joins a select few – if you filter out all of the Microsoft guys, there currently are thirteen SQL Master’s (2005 and 2008) and four Architects in the world – nice crowd to be associated with.

Congrats Greg!

Extracting SharePoint Data using SSIS

Tue, 08 Jun 2010 18:10:13 GMT

!!!! Follow-up to this posted September 19, 2008: SQL 2008 SSIS Adapter for SharePoint from Microsoft Dev team posted to Codeplex !!!!

SharePoint and WSS are fantastic in that almost everything is driven by Web Services. For the data and reporting gurus, our job is a lot easier when it comes to extracting data from the mini-applications and workflows that the business community creates inside a WSS or SharePoint Portal framework – though not completely straight forward, so that is what we’ll discuss today: using SSIS to extract data from WSS lists using the out of the box WSS web services.

In this example, we have a user community that has data in a WSS list that needs to be extracted so it can be reported on. The data in this list represents a collection of information about a business process.

One of our major limitations in using out of the box SSIS features is the Web Service Task. If you look at Books Online, article: http://msdn2.microsoft.com/en-us/library/ms140114.aspx, the first note indicates that the Web Service task supports primitive types only. If you look at the WSDL for our particular method we’ll be using, GetListItems, you can see from the WSDL below that we will have a problem:

To get around this we’ll use a proxy since the Web Service task is the only item currently that can use a web reference. For our first step, we will use the same tool the VS 2005 uses when you add a web reference to a web or windows project: wsdl.exe. We’ll create our class by executing the following from the command line (wsdl.exe should be located in your C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\Bin directory):

Wsdl.exe /language:VB /out:SharePointListService.vb http://SharePointURL/TargetSite/_vti_bin/lists.asmx?wsdl

Note that you will need to do this again for any sub-site that you are targeting for your import as part of the WSDL description is the location of the service.

Our SSIS package will have three simple control flow steps:

- Script Task

- XML Task

- Data Flow Task

Script Task

After you run this command, open your newly created vb file (I normally code C#, but the SSIS script task only uses VB). In your script task, we are going to make a new class reference. Right click on the solution, go to Add > Add Class. You can use the same name as newly created file from wsdl.exe or something that will more appropriately fit given your project. Once you create the new class file, erase any text/code automatically generated and then paste the content of your wsdl.exe outputted file into this code file in your project. Add references to the System.XML and System.Web.Services.

After you have completed this step, you are ready to consume your new class and mimic the Web Service SSIS task.

Here is sample code from the default ScriptMain file. My code in this sample uses the proxy class Lists (generated by wsdl.exe) to call the service, pass the necessary parameters and then save the output to an xml file. Note that for the view name for a list, you have to pass the GUID. It seems that the method was not set to return values using the common view name.

One of the easiest ways to grab the GUID is to go to modify the view you want to use to pull the data from and look in the URL, the portion of the query string after View= has the GUID for this view.

Another way to do this is to use the Web Service Task against the Views web service in WSS/SharePoint, call the method GetViewCollection, pass the List name as a parameter and you can get an XML output with all the views and GUIDs associated with a list item.

After implementing the code above, your Script Task is complete and your output will be an XML file of all the data from the list given the view you hit. An important point to note is to set your rowLimit parameter. The others can be blank or null, but if the rowLimit is not set, then by default you will only get the data that displays in the default view paging size, which is usually 100 records. By setting it to a figure larger than the collection, you will get the entire collection of records.

XML Task

The SSIS XML Source in the Data Flow section of your project does not like XML files that have multiple namespaces, which is almost a guarantee when you are retrieving data from a SharePoint List.

The XML Task is pretty basic in that it strips the extraneous namespaces from the XML file. I got this from Ashvini Sharma’s blog when trying to solve the same problem in hitting RSS feeds for a real estate project. Here is the link to his post:

http://sqljunkies.com/WebLog/ashvinis/archive/2005/05/25/15654.aspx

Right click on the XML Task, choose edit, and then we will set the properties on the task as such:

Under the SecondOperand property, paste the following XSLT:

<?xml version="1.0" encoding="utf-8" ?>

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:output method="xml" indent="no" />

<xsl:template match="/|comment()|processing-instruction()">

<xsl:copy>

<xsl:apply-templates />

</xsl:copy>

</xsl:template>

<xsl:template match="*">

<xsl:element name="{local-name()}">

<xsl:apply-templates select="@*|node()" />

</xsl:element>

</xsl:template>

<xsl:template match="@*">

<xsl:attribute name="{local-name()}">

<xsl:value-of select="." />

</xsl:attribute>

</xsl:template>

</xsl:stylesheet>

I have the FileConnection set to the file that my first task outputted, and the operation saves the results so that I end up with a clean version of my outputted WSS/SharePoint list data.

Data Flow Task

Setting up the import can be one of the more painful tasks, depending on the number of columns you have on the view and list your are hitting. The XML that you will have after pulling the data will have GUIDs for all of the user defined columns in your list, which probably is where most of your data is.

To know exactly how the GUIDs in your XML export map up to columns on your list, you can export the schema for SharePoint. In the Lists web service, hit the method GetList and pass it the name of the List you are using. Your output will have the schema of the list. Here is a sample for one column:

The XName element has the GUID for the column that will show up in the data export. You can use the output from the GetList method to build mappings for your export.

Your Data Flow Task at a minimum will have two objects: an XML Source and your OLE DB Destination. I would recommend putting a Data Conversion object between the two – that way you can save on space on the insert into the database and have a way to redirect failed rows for analysis should you get something really strange in your incoming dataset.

That rounds it out; please feel free to post any questions.

BI Conference 2010: Keynote now online

Tue, 08 Jun 2010 15:33:48 GMT

Just posted earlier this afternoon, Ted Kummert’s keynote speech for the 2010 BI conference can be viewed here.

Other notable speakers during this keynote:

Donald Farmer, Microsoft Principal Program Manager for Microsoft SQL Server
Amir Netz, Microsoft Senior Architect for SQL Server BI

BI conference 2010: PowerPivot and Visual Studio 2010

Tue, 08 Jun 2010 15:14:10 GMT

One of the great topics in today’s keynote session centered around shortly? coming features for PowerPivot – namely:

Integration with Visual Studio 2010
Ability to utilize SSAS 2008 R2 as point of storage for billions of rows, not just hundreds of millions
Pivot Reporter extension for PowerPivot and SharePoint integration
Record Viewer view to help manage wide tables in PowerPivot data source
Lineage and Dependency Tracking and visualizations
KPIs!!

Unfortunately, no committed dates to when we will have these features available, but hopefully we will see the previews soon.

BI conference 2010: a few good sessions

Tue, 08 Jun 2010 08:58:45 GMT

And this was one of them: “Microsoft SQL Server Analysis Services Consolidation Best Practices Considerations”. The SQL CAT team continues to deliver – every conference I have been to, this group at Microsoft always delivers great content.

Now, before I start to rave too much, I will say the title of the session fails a bit; rather, it should have read: “Really cool things we did with Analysis Services and we are SO excited to share”.

The consolidation best practices slide presentation was rather perfunctory and mundane. There were some starts to good thoughts, but none of which we really explored in depth.

The pattern for the slides was all to lead up to the “really cool things” that this team did with Analysis Services and fortunately enough, is available to the public.

Ah, so what is this really cool thing? Simply put, it allows for load balancing of individual SSAS databases across multiple SSAS instances – load balancing at database level, really cool.

Check out the ASLB Toolkit post here on the SQL Cat blog; it contains links to the source and other reference documentation.

ssas processing problems

Mon, 07 Jun 2010 23:20:50 GMT

By now, some of you may have come across this error when processing your dimensions:

File system error: a FileStore error from WriteFile occurred. Physical file: \\?\some path to a file. Logical file:….

At the core of this error is the fact that the asstore, bstore, kstore and string.data files for your dimension is hitting the 4GB limit and it cannot no longer be written to by SSAS.

This is a limit that has to be worked with and there are several ways to manage to that limit. To understand how to work with this limit, it is important to know how you may be running into this limit, and it happens in one of two ways:

1. Exceeding the limit via one of your temporary string stores during processing of your dimension object

2. Exceeding the limit via one of the final string stores of the dimension object

It is important to understand which of these two scenarios is causing your issue as the resolutions for each are different. In the first scenario where it is a temporary store problem, there are several options to streamline the processing of your dimension and get around the this error.

First Option: Check your the value on your dimension for the attribute ProcessingGroup (as show in the image below). By table loads the entire dimension table into memory via a table scan query – if you have a large table then this could be your issue. Changing to ByAttribute will allow the dimension processor to issue distinct queries by attribute and, so long as you do not have an attribute that goes over 4GB (large string values, etc), then you this is your potential resolution.

Third Option: There are a couple of threads out there and one or two posts that mention modifying the PropertyBufferRecordLimitEnabled in the msmdsrv.ini file. This is one potential way of resolving your issue, particularly if, either due to string keys or ways in which your attribute hierarchies are built (or, for shame, not built), then changing this setting to 1 will push down the size of the temporary records buffer, making it less likely that you will hit the 4GB limit for temporary files during processing.

Fourth Option: Check your hierarchies – this is probably the most important! Processing time and memory usage during processing of dimensions depends on the number of numbers and the number of attribute relationships. During processing of a particular attribute, all dependent attributes are held in memory – if you no attribute relationships, then all members must be kept in memory. If you can build cascading relationships, then the dependent pair is processed and can keep therefore keep moving incrementally and efficiently.

Now, if you are facing the second scenario where it is a physical limit of the final output, not the temp processing object, your options are, unfortunately, few.

In this case, only one comes to mind (for an easy resolution):

Your One Option: If you have been processing your dimension using Process Update Occasionally process your dimension by using Process Full to clear out old data in any of the offending files. In a process update, the string data files are appended to versus being overwritten to improve processing time and thus can grow out of hand if frequently processed this way. You can also handle this by doing a Process Clear and forcing a new slate for the dimension files.

If that is not your issue, the unfortunately you will have to look at several things:

1. Your chosen data type for the offending member – unicode types are easy targets, and, in some cases, may have easy resolutions
2. Is this a necessary member? You may find that you do have the option to remove this member from the cube and make it available via drillthrough or report actions on the cube that pull the data from the relational source
3. ROLAP – not the most appealing, but until the 4GB limit is removed, and if that member is necessary, this may be your only choice

Some useful references:

- Denny Lee’s post on the SQL Cat Blog covering ByAttribute or ByTable scenario
- SQL Server 2008 Analysis Services Performance Guide

SSAS tools: mdx studio

Tue, 17 Nov 2009 16:42:46 GMT

So many good tools out there, and yet not all of them get noticed…..

At the SQL Pass conference the week before, I attended one of the classes by the Microsoft CSS groups on common Analysis Services support issues. While not the best presentation format or speaking, the information and thought processes were very relevant and thought out, so all in all – good session.

What surprised me was how few of the attendees were familiar with Mosha’s MDX Studio that the Microsoft speaker presented as a resource – so in the nature of spreading the good word, here is the link:http://www.mdxstudio.com/

What is it? From Mosha’s site:

MDX Studio is tool which helps users of Microsoft Analysis Services to analyze complex MDX expressions, monitor performance characteristics of MDX queries and get insight into how MDX interacts with other UDM features such as attribute relationships. MDX Studio provides unique visualization representation of MDX expression and allows the user to interact with different stages of MDX execution.

Sharepoint server 2010 Insights

Wed, 28 Oct 2009 16:02:29 GMT

You may or may not have heard by now that the business intelligence capability in SharePoint 2010 now has a platform branding called INSIGHTS.

I will blog more about the features in my “down time” at the SQL PASS conference next week, but I wanted to put the links out to everyone so you can start ramping up on what is coming inside SharePoint 2010 with respect to business intelligence.

First and foremost, the official Insights link: http://sharepoint2010.microsoft.com/product/capabilities/Insights/Pages/default.aspx. Lots of information and resource links posted here - -a great way to take a top down look at what the Insights platform is all about.

If you have completely been ignoring the news and the gossip, then you probably have missed the two biggest architectural features in the Insights platform will be PerformancePoint Services and Excel Services support of PowerPivot (formerly Project Gemini). PerformancePoint Services will utilize the new application services architecture that is now part of SharePoint - -this allows for far greater flexibility in deployment and most importantly, security.

PerformancePoint Services links:

http://technet.microsoft.com/en-us/library/ee661741(office.14).aspx

- http://technet.microsoft.com/en-us/library/ee424392(office.14).aspx

Excel Services: http://technet.microsoft.com/en-us/library/ee424401(office.14).aspx

Report Builder 3.0: http://technet.microsoft.com/en-us/library/bb418434(SQL.10).aspx

Technical Diagrams: http://technet.microsoft.com/en-us/library/cc263199(office.14).aspx

So, keep reading and don’t get left behind. If you landed here and you are more interested in the other features of SharePoint and not business intelligence, visit my fellow Catapultian Matt’s post here. I will, of course, encourage you to stay here and become more interested in BI, but I’m not into forcing my opinion in anyone, I’ll leave that to the media.

SSIS Adapter for SharePoint

Mon, 26 Oct 2009 15:41:01 GMT

Kevin Idzi of Microsoft was kind enough to leave a comment on my Extracting SharePoint Data using SSIS post informing me of Integration Services enhancements posted on CodePlex.

I had been delaying posting as I wanted to have a full sample showing it in use, but between work and buying a new house recently, I have not had the time to put together my virtual machine images for a complete demo.

The community sample has the following components available to download (in separate install packages):

XML Destination Pipeline component
Regular Expression Flat File for regular expression based flat file parsing
Delimited Flat File Parser, for files with rows that are missing column fields
Package Generation Sample
Hello World Sample for demonstrating using the UI API
SharePoint List Source and Destination – this component was written by Kevin
WMI Source Component

All of these components are available for download here.

The base requirements for installing these components are .NET 3.5 and SQL 2008 Integration Services.

Once you run the MSI, you will be able to select the two new adapters from the "Choose Toolbox Items" dialogue:

One of the nice features that Kevin included in this adapter is the ability to execute CAML queries against the SharePoint List:

If you do not know how to write CAML or are not good at it like I am, then use the U2U CAML Query Builder written by Patrick Tisseghem; I've got a quick tutorial here: CAML Query Builder.

If you have any feedback about using this SharePoint List Adapter with SSIS, please be sure to leave comments or suggestions on the CodePlex site for Kevin and his team.

SSIS on a cluster

Thu, 15 Oct 2009 15:30:02 GMT

I figured this was worth a quick post: while many of SQL Server’s aspects are cluster aware, SSIS is not. Why? My guess is that there is not a pressing business need for this to be a cluster aware service – while many companies have vital ETL processes to help run company business, the need for failover in this scenario is not as pressing as in transactional systems.

With that said, I had a customer that was testing the failover of one of their main SQL clusters and suddenly noticed that a more than several jobs failed to run. In helping on the issues, I noticed that they had SSIS steps in the job pointing to the cluster name for pulling out the packages form MSDB, and that, more importantly, someone had forgotten to install the SSIS service on the failover node.

Someone had configured SSIS to point to the cluster on the primary node, but must have forgotten to install the service and do the same config change on the failover node – easy enough fix; but also a good education opportunity for the client as they were not familiar with SSIS at all (transitioning to a new environment for them this past year, so most of this is new for them).

SSIS is not recommend to be configured as a cluster resource, but that does not mean it cannot be used on a SQL Server that is clustered – you just have to be aware of what you are doing and the consequences. In my current scenario, we have SSIS configured to work on the cluster environment, but not as a cluster resource (the second link below will show you how).

Here are the references:

Just be sure to weigh the pros and cons as even the simple config change has its own consequences, in particular the section in the first link on disadvantages of SSIS as a cluster resource.

Awesome speaker for our next BI User group!!

Mon, 14 Sep 2009 16:47:42 GMT

I am excited to announce that we have secured a great speaker for our next Central Texas Business Intelligence User Group meeting on October 21st at the Microsoft office in Austin.

Bryan C. Smith, prolific SSAS author and now of Microsoft fame, will be coming down to speak on a yet to be disclosed topic on SQL Server Analysis Services.

If you have not done so, and you live/work in the area, be sure to join our group on LinkedIn or send me an e-mail and I will add you to our distribution list – this is a speaker you do not want to miss (if you love all things SSAS).

On a broader note, we will be upping the frequency of the meetings to monthly after the October meeting.

In that vein, we are looking for some more presenters (Larry Clark (MS) and I love presenting, but I think the group would like to see some new faces). Please e-mail me or post to the LinkedIn group board if you are interested.

Hope to see some new faces there!

Design Tip - Analysis Services Aggregations

Mon, 17 Aug 2009 12:21:15 GMT

I was cleaning up a cube design for a customer, primarily working with the current implementation to resolve some performance issues and make some minor alterations to the dimensions, when I ran across an aggravation in designing aggregations for the partitions I implemented ( there were none before, part of the issue).

I had just finished designing my partitions when I started to design the aggregations for my partitions – when the designer returned this obnoxious message:

0 aggregations have been designed. The optimization level is 0% (0 bytes).

I will mention again that this cube was not my design, and when given the opportunity, everything from the dimensional model to the entire cube will be redesigned, however, this message was a nuisance. Fortunately, I kept my cool and figured out what I was doing wrong. The Aggregation Design Wizard will not allow a resulting aggregation set to exceed a certain percentage of the fact table (I hear it is 30%, but I can't find this documented anywhere).

Here are a few general rules with aggregations that will help avoid this message:

Make sure the partitions you are trying to assign an aggregation design are evenly distributed. In other words, do they have a similar number of facts/rows in them or are they varied? One aggregation design can work for multiple partitions so long as those partitions are similar or close in overall row counts
If you have a large number of attributes in your dimensions related to a particular partition set you are designing for, you may consider changing some of the options on the Review Aggregation Usage screen in the wizard. Attributes that are rarely used can potentially be set to None
On the Specify Object Counts page, check the counts tabulated for each dimension as well as the measure group. On the dimensions, the wizard does not calculate the partition count. This number is the count of that member distinctly appearing inside the specific partition. For example, if your partition broken by year, then the count of dates on your Calendar dimension would be 365, months would be 12 and so on.

In addition to the above, take a look at your individual partitions -- are the individual partitions too large? BIDS Helper suggests a 20 million plus row count per partition is probably not partition with enough detail.

As for my specific problem – due to the "funky" (lack of a better word) design of the fact tables, I did not have reasonable splits in both my row counts and related member counts for my initial partition designs as I had initially suspected, and being in a hurry that evening, I did not update the counts inside the partition design screen before moving over to designing aggregations. I simply assumed I had picked a good way of doing it on the first try!

So, a lesson in patience reiterated yet again, and after getting some sleep, I looked at the issue the next morning with some fresh eyes and found my solution.

Hopefully this little explanation will save some time for someone out there – feel free to let me know if any further explanations are necessary, or feel free to share your aggregation design issue!

Reminder: BI User Group meeting August 13th!

Thu, 06 Aug 2009 16:55:06 GMT

Reminder, we are having our next Central Texas Microsoft Business Intelligence User Group meeting a week from today, August 13^th, 2009 from 6:00pm to 8:00pm at the Microsoft Austin office.

Session Topic: "PerformancePoint Monitoring and Analytics 101"

Description:

In this session we will cover the basics of PerformancePoint: what it is, how it is deployed and sample architecture patterns. We will also cover the basic elements of Dashboard Designer and how developers and business analysts will use this tool to develop charts, grids, scorecards and KPIs for their business to consume.

If you haven't joined the group on LinkedIn yet, then please feel free to do so – it is an easy way to keep up with any updates, meeting announcements and meet others in our profession in the central Texas area. Logo below will take you to the group home page:

Be sure to RSVP if you are coming!

WhitePaper: PerformancePoint in an extranet environment

Thu, 30 Jul 2009 15:09:28 GMT

Great read for when you are stuck in the airport: Publishing PerformancePoint Server in Extranet Scenarios.

This is actually a very detailed document outlining many of the aspects around establishing an extranet deployment, particularly a secure one.

Important Note: We have run across a bug in Server 2008's implementation of Kerberos. This issue will cause a fatal flaw to your implementation with SSAS in the scenario– Dan English provides a greater level of detail in his post earlier this year. Just as a recap, you will run into errors in accessing SSAS where the MDX will get truncated in the Kerberos ticket that gets passed to the SSAS service. The only way around this at the moment is to have the front end servers be Server 2003, or CTP versions of Server 2008 R2.

Note that there is no "official" fix for this at the moment, though one is expected as a cumulative update in August.

Gather stats from SharePoint

Fri, 24 Jul 2009 09:25:20 GMT

I was not actually looking for this, but that does not mean it is not incredibly useful.

I was actually looking for what was available from the PerformancePoint Team blog in the way of updates to the SDK documentation, or heck, an SDK versus just a single page in the MSDN canon, when I came across this code sample posted on code.msdn.microsoft.com (what the real difference between this site and codeplex.com is, I do not know, but it gets confusing having both).

This code sample, titled "Gather information from SharePoint to a database for reporting", does not come with a lot of documentation or instruction, but by odd coincidence, it seems I found this on its publishing day – today, May 11. The code sample consists of two parts, the C# console app project and the database.

To get this tool up and running:

Download the project file for the app and the database
Unless you are in France or have French collation set up on your server or personal machine, I would recommend hand editing the script to remove that language collation……………Or just do what I did and create a database called Stats and used the BAK file included to restore to (the DDL does not have any table create scripts?)
Start up VS 2008 and open the unzipped project. From here you can either compile so you have an executable or debug through it and watch it run

One important note: This app is not yet configured for remote execution, so it has a dependency on being run directly on the SharePoint server in order to gather stats. SQL Server configuration is just a connection string in the app.config, so that can be a different box/remote server.

This handy little app produces quite a few usage statistics on a SharePoint farm, with the most rich being at the site level. Be interesting to see where this goes next, I have no immediate need for it, but I figured it is interesting enough of a find to post and share!

PerformancePoint, Security and Account Access

Tue, 21 Jul 2009 12:04:09 GMT

I was assisting a client clean up an installation of PerformancePoint. They thought they had followed all of the steps, but were perplexed by an error they were getting after deploying the dashboards to SharePoint. Given what was at the bottom of the error, and how few people realized this slight difference in authenticating accounts, I figured it was worth a quick post.

To start, the error received was:

"You do not have permission to see this data. (Domain\UserAccount). Contact the administrator for more details."

My client was perplexed. They had developed the dashboard in Dashboard Designer and previewed it with success on the Monitoring server preview site. This person was an admin of Monitoring server and a site collection admin in SharePoint. The question to me then was – what was going on?

To help explain, below is a diagram of a sample diagram of a PerformancePoint deployment on a farm, which is exactly what this client has in their environment:

The big "gotcha" here for him is based on the usage of NTLM as the authentication method and a lack of understanding on where deployed dashboards actually render and load.

In a non-Kerberos implementation, deployed dashboards will connect to data sources under the identity of the SharePoint service, which was the bit of understanding this user was missing – his perception was that SharePoint rendered output from the Monitoring web service, and thus authentication still happened from the Monitoring Web Service – once this perception was corrected, we added the SharePoint account as a reader on the cube for his dashboards and life went back to normal.

SQL 2008 Intellisense

Sat, 18 Jul 2009 13:26:00 GMT

In my post, Why use aliases?, I spoke in favor of using a certain syntax in your SQL programming, particularly because SQL Server 2008 gives you almost no reason not to with integrated Intellisense.

One little note for those who may have run into a frustration while developing – Intellisense in SQL 2008 does not always refresh timely. This can be particularly annoying if you are in the middle of developing, make some schema changes and then try to write a script – intelllisense will not prompt you with the correct values, or tell you that objects and schemas do not exist.

Only two known ways to resolve this for now:

Main Menu: Edit > Intellisense > Refresh Local Cache
Or, shortcut keys: CTRL + Shift + R

Extending the Data Profiling Task

Sat, 18 Jul 2009 13:23:03 GMT

SQL Server 2008 RTM has been out a while now and yet, I am just now making my way to using the Data Profiling Task, new to SSIS in this release. Time and client demands are always a factor governing when and what I get to look at, however I had read up on the features and was hoping that what I had read was not all the new task could do.

The new task is a good start, but unfortunately, it is a little more limited than I would have hoped. Fortunately though, with the SSIS APIs open to us, we can use the class object as we please.

So first -- what did I find limiting about the Data Profiling Task?

It appears that its intended design was to serve as a checkpoint during loads/transforms and that conditional logic can be applied based upon results; this is incredibly useful. My major concern is with activities that happen before the design phase – and quite directly, before any development begins at all.

Reference this great little tip from Kimball here on data profiling and its necessity according to his framework – profiling is tantamount to understanding the system that you are getting ready to work with in the course of a project.

To get to the heart of what I wanted to see the Data Profiling task do: I want to use it to generate an aggregate view of a given system so that I can view the results and learn more about the data within the database. Many of us have been given a database, custom built or extended far from the original software vendor's spec, with little documentation.

Even for those who are not preparing to move this data for warehousing or reporting, anything that provides a reference to what the data is, how it is stored, how unique, etc. is a boon for developers.

The Data Profiling task can deliver this data -- but only one table at a time?

There are no expression settings or variables that you can use to set which table and where the output goes. For two of the profiling tasks, I can see this as a bit of a problem – both the Candidate Key and Value Inclusion profile do not lend themselves to being easily generated dynamically via iterating over all the tables and views in a database.

But all the others are useful in this aggregated manner, so why, if I am looking at a client database that has 96 tables, must I build 96 tasks via Visual Studio and still not end up with an aggregate report (just 96 xml files)? I would like to be able to run this on the entire database, and start looking at the hotspots or data profile results that are the fat tails in the distribution chart.

My solution is to build the SSIS package programmatically, run it and then read and aggregate the results. I still need to clean up the solution a bit and finish some unit tests, so I will post it soon for everyone to download and enjoy.

My envisioned endgame for this project is for the output to be configurable for user desired thresholds and for that output to render in HTML or Word – still a little ways away from that, but I have the bulk of the SSIS work done. If anyone is interested in contributing, let me know and I will see about getting this on CodePlex.

Notes for anyone who wants to work with this project or attempt this on their own:

This effort makes use of the Microsoft.DataDebugger.DataProfiling class, which is not necessarily a supported API. An example of using this class is on the SSIS Team Blog here.
DLL references necessary for this project come from multiple places. For example, the Microsoft.SqlServer.DataProfilingTask is in ~\Program Files\Microsoft SQL Server\100\DTS\Tasks and Microsoft.SqlServer.DataProfiler is in ~\Program Files\Microsoft SQL Server\100\DTS\Binn
It is a good idea to read up on building a package programmatically on MSDN as there are class objects, such as TaskHost, which are not immediately intuitive as they are not seen when using the designer

Installing SSRS for SharePoint, the fun way

Tue, 07 Jul 2009 13:11:27 GMT

I figured the search engines would do a good job of delivering the answers to these questions, but as I have seen this theme reoccur many times over the past few weeks in the MSDN forums, I am going to write out the steps for everyone (if nothing else, at least I'll just be pasting a link to this versus copying the steps all over again in forum responses).

For many people installing the SSRS plug in for SharePoint/WSS (both SSRS 2005, but more likely 2008), you can get these annoying errors (this error can be found in Root:\Users\YourLogin\AppData\Local\Temp and will be prefixed with RS_SP):

For some of you, this error may become a common nuisance when installing the SSRS plug in for MOSS/WSS 3.0 for both SSRS 2005 and 2008:

******* User does not have permission to add feature to site collection: http://<SharePoint URL here>:36076.

This can be particularly frustrating if you are logged in to the box under the farm admin account and have local admin privileges.

To work around this error, follow these steps:

1. Double check that you have a site collection set up on the farm, not just the web app

2. Log into the server you are installing the Reporting Services plug in to with credentials that are the farm admin for the MOSS install

3. Run the CMD tool with Administrator privileges (also called elevated permission)

4. Navigate to where you have the sharepointrs.msi and type sharepointrs.msi SKIPCA=1

5. Navigate to your Temp% directory to find the unpacked rsCustomAction.exe file

6. In the CMD tool, execute rsCustomAction.exe /i

7. After the install, you will probably need to navigate to Central Administration > Site Actions > Site Settings and under the heading Site Collection Administration click Site Collection Features.

8. From there, click activate on Report Server Integration Feature

9. The settings for SSRS will now have their sub section on the Application Management tab in Central Admin

References:

http://technet.microsoft.com/en-us/library/aa905871.aspx