This week I presented at the Mastering BusinessObjects 2010 conference, hosted in Melbourne, Australia.  The conference was organised by Eventful Management, who did a fantastic job.  International presenters included Timo Elliott, Ingo Hilgefort, Dave Rathbun, Rajeev Kapur and Tom Nather.  You can see some photos of the event by Timo here.

My topic was called ‘SAP BusinessObjects XI 3.1 End-to-End Demonstration’, in which I wanted to do a live demonstration of one dataset being used with most of the BOBJ tools, including Data Integrator, Universe Designer, Web Intelligence, Xcelsius, Live Office, QaaWS and Explorer.

Here is a link to the PowerPoint deck, though there’s not many slides.. (Fletcher_EndtoEndSBO_MBO)

I want to provide a walkthrough of the presentation, and include links to the datasets I used.  This should hopefully allow anyone (who has access to the SAP BusinessObjects XI 3.1 environment) to follow my steps.

To begin with, I introduced the dataset I used.  I downloaded it from http://data.australia.gov.au/, and it shows the Number of Crimes in New South Wales, Australia, broken down by Local Government Authority (LGA), Month/Year (over 12 years), Offence Category and Subcategory.  All in all it represents over 1.7 million records, so not a small dataset.  I also extended the dataset by introducing State Region (multiple LGA’s belong to a Region) in another CSV file.

To start with, the first part of the presentation was to use Data Integrator to import data from two CSV files into a SQL Server database, including de-normalising the data into a dimensional model.  You can download the CSV files from the Datasets page of my blog, under the NSW Crimes heading.  You will also need to create a database (I used SQL Server) to host the tables in the Target Datastore.

Here are the steps I following to build the ETL job to move the data from the CSV files to the SQL Server database.

1. Create a new Flat File Format for the ‘NSW Crimes Data 95-08.csv’ (CSV#1) for extracting data
2. Create a new Flat File Format for the ‘Regions and LGAs.csv’ (CSV#2) for extracting data
3. Create a new database to store the extracted data in, and create a new Datastore (I called mine DEMO_E2E_SBO)
4. Create a new Project and Job
5. Within the new Job, create a new dataflow for staging the main source file, using a template table
6. Run the job to stage the data
7. Import the template table and set to overwrite
8. Add a new dataflow for creating the DIM_DATE table, using the Date_Generation transform
   1. Set the date generation from 1995.01.01 to 2008.12.31, monthly
   2. Add the following output columns to the Query transform
      1. Date_SK (julian of Date)
      2. Year
      3. Quarter
      4. Month
      5. Date
9. Add a new dataflow for DIM_LGA, using the CSV#1 flat file format as a source
   1. Add the following output columns to the Query transform
      1. LGA_SK (use gen_row_number function)
      2. State_Region (leave blank for now)
      3. Local_Government Authority
   2. Group by the LGA column
   3. Add in the CSV#2 as another source, combining in another Query transform, with the following changes
      1. Populate State_Region
      2. Make sure the WHERE clause includes LGA=LGA
10. Create a new dataflow for DIM_OFFENCE_TYPE, using the following columns in the Query transform
    1. Offence_Type_SK (use gen_row_num to populate)
    2. Offence Category
    3. Offence Subcategory
    4. Make sure to GROUP BY Offence Category and Subcategory
11. Run the job to create and load the dimension tables, then import each of the dimension tables and set them to overwrite values
12. Create a dataflow for loading the FACT_CRIMES
13. Add in a pivot transform, using the following settings
    1. Pivot sequence column called ‘Seq_Number’
    2. Non-pivot Columns
       1. LGA
       2. Offence Category
       3. Offence Subcategory
    3. Pivot Columns
       1. All month columns
    4. Data field column called ‘Number_of_Crimes’
    5. Header column called’ Month’
14. Add in a query transform, with settings as follows
    1. Add the following columns
       1. Crime _SK (use gen_row_num to populate)
       2. Date_SK – use this mapping = julian(to_date(’01 ‘ || Pivot.Month,’DD MON YYYY’))
       3. LGA_SK – use lookup_ext function to populate with the LGA_SK from the DIM_LGA table
       4. Offence_Type_SK – use lookup_ext function to populate with the Offence_Type_SK from the DIM_OFFENCE_TYPE table
       5. Number_of_Crimes – direct mapping
15. Run the job
16. Import fact table and set to overwrite, and you’re done!

I’ve also exported the entire Target Datastore to an Access DB (also available on the the Datasets page of my blog), as well as exporting the entire job to an ATL file which you can also download.  I created the job in Data Services XI 3.2, so the ATL file won’t be compatible with versions prior to that.

I hope that someone finds the above of value if they are learning Data Services for the first time.  In the next post I will cover the rest of the presentation, so stay tuned!

Here is a PDF of my presentation to view.

Some other highlights of the conference were:

• Pioneer is due in 2010 (combining Voyager with Bex Analyzer)
• Support for 64bit environments coming soon
• New release of BOE due in 2010 as well (XI 4.0?)
• SAP BWA (in-memory datastore) will be updated to allow hardware acceleration of other data sources, not just SAP BW
• BO Text Analysis (as presented by Plaut) looks very interesting, turns unstructured data (blogs, emails) into structed relational data
• Timo Elliott’s recommendation – don’t just implement software, aim high and look to change the business
• Dave Rathbun presented on universe tips and Web Intelligence essentials, such as calculation contexts (often misunderstood), multiple data providers and sub-queries
• Data Services 3.2 due in Q3 2009, will support 64bit Linux, and next release of DS will support 64bit Windows
• Data Services 4.0 will potentially migrate to use the BOE architecture – fingers crossed on my part

It was a great conference with a variety of users and partners present, and I hope to be there at the MBO 2010 conference.

- Josh

My top vote (a bit wishful I think) was the ‘Salmon Dance’ song by The Chemical Brothers. I make no apologies as I really like the song. But, sadly, it didn’t come number 1. I suppose I should be grateful it at least made it into the list – I think it was around #50…

So, you ask, what was the top 10? Well, let’s have a look:

1. Muse – Knights of Cydonia
2. Silverchair – Straight Lines
3. Kings of Leon – On Call
4. John Butler Trio – Better Than
5. Faker – This Heart Attack
6. Foo Fighters – The Pretender
7. Daft Punk – Harder, Better, Faster, Stronger (Live)
8. Cold War Kids – Hang Me Up To Dry
9. Soko – I’ll Kill Her
10. The Panics – Don’t Fight It

If you want to, listen to the top 10 countdown here or check out the Hottest 100 website.

I was happy Silverchair didn’t win (sorry guys, but wasn’t too fussed) but the Muse song was awesome. They only won it by 13 votes too! The Foo Fighters and Daft Punk songs were okay too, but *sigh*… No Salmon Dance!

It was also just good catching up, good music, and I finally got to crack my first ever home-brew beer – a lager, and it tasted great!

And don’t worry, I promise I’m gonna start posting stuff about work soon!

- Josh