For some months ago I had the opportunity to spend quite a lot of time playing around with Biztalk 2006. During this period I passed the two Biztalk exams (2004 and 2006) and tried to learn as much about Biztalk as absolutely possible. I really think Biztalk is a great product, its certainly one the best products for process integration and integration intensive, system-to-system BPM, but the support for human-centric business processes is not that good, I think. Biztalk is easy to use, has good tools, is powerful and ships with a LOT of functionality, for instance the support for BAM and BI is really impressing. But after playing around with Biztalk for a while I soon realized that one of the drawbacks of the product is the lack of support for working agile. I am a big fan of agile methodologies and I really think that agile methods is the right way to go to create better software. So I started investigating if it was possible to make Biztalk more agile. I ended up making a presentation that I held at the Norwegian .Net User Group late November 2006 where I sum up my experiences and findings with Biztalk and agility. The presentation can be find right here, its in norwegian, so I will now try to sum up some of the main in this blog entry.
First I would like to say something in general about integration projects. It seems to me that integration projects are done less agile than more “normal” application projects. The concepts of test-driven development, iterations, close customer collaboration, simple design and the focus on working/running software instead of documentation dont seem to be used that much when it comes to integration projects. Of course one of the reasons why this is so is because enterprise integration projects often has a greater level of complexity compared to “normal” projects which makes it harder to use agile techniques like test-first and continuous integration. Integration projects by definition have to deal with multiple applications running on multiple platforms in different locations. The systems to be integrated are often some of the most critical systems to the business, there are different technologies and systems with totally different characteristics that should work together, the communication patterns and message exchange patterns are complex and hard to test. To sum up, an integration solution is extremely vulnerable to failure because it ties the enterprise together. But why are these challenges an excuse to not work agile? Should it not be the other way around? These characteristics of integration projects should instead encourage to work more agile to handle all the complexity, uncertainty and risks in a better way. But to be able to do this, the integration products must support some basic things, like unit testing and automated and scripted deployment. Too many integration suites are properitary big black-boxes that you cant dive into and gain control over. Good old Gregor Hophe has written a lot more about this in his two articles: TestDriven EAI and Agile EAI.
A figure that show how you can slice your integration projects into smaller iterations:
Back to Biztalk. One of the main problems with traditional Biztalk development is the the slow and complex process of deploying and testing a solution. You develop some artifacts (schemas, pipelines, orchestrations, etc), you compile, and build and then deploy them to the server. Then you have to manually start all the ports in the orchestrations. For each port you have to specify it, enlist it and start it. This also goes for all the orchestrations. Then you can start your manual testing. This could be something like opening a browser to initiate a http-request, wait for processing, open windows explorer and validate that a file was written to a folder. Then you have to clean up and maybe initialize the state of the application to run a new test run. Some of this problems are solved in Biztalk 2006, but you still have to do the stop/unenlist/undeploy/redeploy/enlist/start/test-cycle which is really time consuming. Its also a big risk that you do something in wrong order which often causes the application to fail. So, there are two big problems related to traditional Biztalk development:
-Manual deployment and building (biztalk applications cannot be run, tested or debugged “in-place”, they have to be deployed to the server for doing this)
-Testing is done manually (this is a time consuming activity and lacks systematism)
So, I searched around on the net and talked to people, and I found some really great tools and software that solves many of these problems.
For solving the problems with the time consuming and manual deployment of applications I have found two available Biztalk deployment frameworks:
-Microsoft Services UK Enterprise Solutions Build Framework (Based on MSBuild)
-Deployment Framework for BizTalk 2006 av Scott Colestock (Based on NAnt)
Because I like using MSBuild I chose to try out the Microsoft Services UK Enterprise Solutions Build Framework. This is a huge library of custom made MSBuild tasks, including a lot of Biztalk deployment tasks. Some of the Biztalk related tasks:
-Application: AddReferences, Create, Delete, Exists, RemoveReferences, Start, Stop
-Assembly: Deploy, Exists, ExportBindings, ImportBindings, Undeploy,
-Orchestration: Bind, Enlist, Exists, Start, Stop, TerminateInstances, UnBind, UnEnlist
-ReceiveLocation: Configure, Exists
-ReceivePort: AddReceiveLocation, Configure, Create, Delete, Exists, RemoveReceiveLocation
-SendPort: Configure, Create, Delete, Enlist, Exists, Start, Stop, UnEnlist
-SendPortGroup: AddSendPort, Configure, Create, Delete, Enlist, Exists, RemoveSendPort, Start, Stop, UnEnlist
As you can see, these tasks can automate almost everything you want to do in a biztalk deployment. Let me show you a few examples (the application I created to test these concepts is based upon the one from here).
Define and set properties for an application in the build script:
<propertygroup>
<solutionname>PurchaseOrderSystem.sln</solutionname>
<buildtype>Release</buildtype>
<buildpath>C:\PurchaseOrderSystem\bin\Deployment\</buildpath>
<btapplicationname>PurchaseOrderSystem</btapplicationname>
<hostname>BizTalkServerApplication</hostname>
</propertygroup>
Stop an application that is already running:
<target Name="StopApplication" Condition="$(AppExists)=='True'" >
<biztalk2006 .Application.Stop
Application="$(BTApplicationName)"
Server="$(BTServerName)"
Database="$(BTServerDatabase)"
Password="$(BTServerPassword)"
UserName="$(BTServerUserName)" />
<message text="App Stopped: $(BTApplicationName)" />
</target>
Create send ports with attributes:
<biztalk2006 .SendPort.Create
Name="SendPort1"
PrimaryTransportAddress="C:\PurchaseOrderSystem\POAckOut\%MessageID%.xml"
PrimaryTransportProtocolName="FILE"
IsTwoWay="False"
SendPipeline="Microsoft.BizTalk.DefaultPipelines.PassThruTransmit, Microsoft.BizTalk.DefaultPipelines, Version=3.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
Application="$(BTApplicationName)"
Server="$(BTServerName)"
Database="$(BTServerDatabase)"
Password="$(BTServerPassword)"
UserName="$(BTServerUserName)" />
Bind ports to orchestrations:
<biztalk2006 .Orchestration.Bind
Name="PurchaseOrderSystem.ProcessPO"
HostName="$(HostName)"
OrchestrationPorts="@(Ports)"
Application="$(BTApplicationName)"
Server="$(BTServerName)"
Database="$(BTServerDatabase)"
Password="$(BTServerPassword)"
UserName="$(BTServerUserName)" />
Start application:
BizTalk2006.Application.Start
Application="$(BTApplicationName)"
Server="$(BTServerName)"
Database="$(BTServerDatabase)"
Password="$(BTServerPassword)"
UserName="$(BTServerUserName)" />
<message text="App Started: $(BTApplicationName)" />
This is just a few examples of what that can be done with this great framework. Thanks a lot to the people at Microsoft UK that has developed these tasks and made them freely available.
So, what about testing? Ever heard of BizUnit? BizUnit is a lovely little framework created by Kevin Smith. Its a framework for creating automated functional tests for Biztalk 2006. You can use both NUnit and VSTS to run your tests, I prefer the latter because its completely integrated into Visual Studio. With BizUnit you make your own TestCases in the same way you do with normal unit testing frameworks. But BizUnit tests are not really unit tests, its doesnt test the code, its more like a black-box testing of different Biztalk artifacts. It automates the entire test process. You define TestCases by using different kinds of test steps that ships along with the framework. There are two main categorizes of test steps:
-TestSteps (ex. FileCreateStep, FileMoveStep, FileValidateStep, HTTPPostStep, HttpRequestResponseStep, MSMQCreateQueueStep, SOAPHTTPRequestResponseStep)
-ValidationSteps (ex. XMLValidationStep, BinaryValidationStep )
These test steps are autonomous and isolated from each other. BizUnit takes care of the data flow and interaction between the test steps. TestCases are defined in XML and consists of three parts:
-Test setup: One or more test steps
-Test execution: One or more test steps
-Test cleanup: : One or more test steps
A simple example of a testcase scenario could be like this:
Here you want to test a simple orderprocess orchestration that read orders from a folder, processes them and writes them out to another folder. Here you can create a TestCase with two TestSteps, a FileCreateStep to initiate the processing, and a FileValidateStep to wait for the output of the process and validate the result message.
This testcase can be defined in XML like this:
<testcase testName="Test_1_ProcessPO">
<testsetup>
</testsetup>
<testexecution>
<teststep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileCreateStep">
<sourcepath>C:\PurchaseOrderSystem\InputFiles\PO_001.xml</sourcepath>
<creationpath>C:\PurchaseOrderSystem\POIn\PO_001.xml</creationpath>
</teststep>
<teststep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileValidateStep">
<timeout>60000</timeout>
<directory>C:\PurchaseOrderSystem\POAckOut</directory>
<searchpattern>*.xml</searchpattern>
<deletefile>true</deletefile>
<validationstep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.XmlValidationStep">
<xmlschemapath>C:\PurchaseOrderSystem\POA.xsd</xmlschemapath>
<xmlschemanamespace>http://PurchaseOrderSystem.POA</xmlschemanamespace>
<xpathlist>
<xpathvalidation query="/*[local-name()='POAck' and namespace-uri()='http://PurchaseOrderSystem.POA']/*[local-name()='PONum' and namespace-uri()='']">PO_001</xpathvalidation>
<xpathvalidation query="/*[local-name()='POAck' and namespace-uri()='http://PurchaseOrderSystem.POA']/*[local-name()='POItemsCount' and namespace-uri()='']">2</xpathvalidation>
</xpathlist>
</validationstep>
</teststep>
<teststep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileValidateStep">
<timeout>60000</timeout>
<directory>C:\PurchaseOrderSystem\POOut</directory>
<searchpattern>*.xml</searchpattern>
<deletefile>true</deletefile>
<validationstep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.XmlValidationStep">
<xmlschemapath>C:\PurchaseOrderSystem\PO.xsd</xmlschemapath>
<xmlschemanamespace>http://PurchaseOrderSystem.PO</xmlschemanamespace>
<xpathlist>
<xpathvalidation query="/*[local-name()='PORequest' and namespace-uri()='http://PurchaseOrderSystem.PO']/*[local-name()='PONum' and namespace-uri()='']">PO_001</xpathvalidation>
<xpathvalidation query="/*[local-name()='PORequest' and namespace-uri()='http://PurchaseOrderSystem.PO']/*[local-name()='POStatus' and namespace-uri()='']">REJECTED</xpathvalidation>
</xpathlist>
</validationstep>
</teststep>
</testexecution>
<!-- Test cleanup: test cases should always leave the system in the state they found it -->
<testcleanup>
</testcleanup>
</testcase>
Finally you can make a TestClass with VSTS that runs the test case:
using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.Services.BizTalkApplicationFramework.BizUnit;
using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace BizTalkUnitTest
{
[TestClass]
public class TestPurchaseOrderProcess
{
public TestPurchaseOrderProcess()
{
}
[ClassInitialize]
static public void SetUp(TestContext context)
{
BizUnit bizUnit = new BizUnit(@"C:\PurchaseOrderSystem\TestCases\Test_Setup.xml", BizUnit.TestGroupPhase.TestGroupSetup);
bizUnit.RunTest();
}
[TestMethod]
public void ValidatePO_Rejected()
{
BizUnit bizUnit = new BizUnit(@"C:\PurchaseOrderSystem\TestCases\Test_1_POSystem.xml");
bizUnit.RunTest();
}
[TestMethod]
public void ValidatePO_Approved()
{
BizUnit bizUnit = new BizUnit(@"C:\PurchaseOrderSystem\TestCases\Test_2_POSystem.xml");
bizUnit.RunTest();
}
}
}
To sum this up, traditional Biztalk development can be quite troublesome and rigid, you have to do a lot of manual work to get things done when you want to test and deploy your solution. But if you automate your build and deployment process and introduces automated functional testing of Biztalk artifacts you can be able to work much more agile and really increase your productivity. You will free up time for actual development and decrease the time spent on activities that dont creates value. BizUnits tests are great to ensure quality in your applications and you can also use them as regression tests to handle changes in your application. But, writing large tests in XML can be very time consuming, so I recommend to use a tool like BizUnit Code Snippets for generating XML when writing tests. Through automated build and deployment with Microsoft UK Build Framework you save a lot of time when deploying application and you reduce the number fails because the process is scripted instead done manually.
I think there are still some challenges involved when trying to work agile with Biztalk 2006. The main point here is refactoring. When working agile, refactoring is something you do all the time. If you have a Biztalk solution that is automated with at deployment framework and also includes a lot of BizUnit tests, then it takes forever to change the names of components and artifacts. You then have to search hundreds of lines of XML to find where to do the changes. This really is time consuming and introduces a lot of bugs. What about the possibility to define both build scripts and Bizunit test cases programmatically instead of using XML? Then refactoring could have been done as with normal code through Visual Studio. It would really have saved a lot of time and boring work, what do you think about this idea?
February 18th, 2007
Bruce Silver has an interesting post named BPM on SOA: What would it look like? - Part 1. Here he talks about whats the differences between BPM and SOA, I will here repeat these points and add my own comments:
Where BPM is top-down, SOA is more bottom-up, utilizing and exposing the already existing infrastructure, hiding the complecity and details of technology from the business layers of the enterprise architecture
BPM is business-driven, a model created by the business drives the implementation, SOA is IT-driven and there are technical architects that defines the scope
A BPM project success is measured by business metrics and KPIs. A SOA is measured by architectural metrics, ease of integration, cost of change, etc
BPM is more project-oriented, where SOA is more enterprise wide infrastructure-oriented
In BPM, what is reused in the process-model, in SOA, services are reused
In BPM business processes are more hierachical, chained and nested
In BPM suites concrete endpoints are used, in SOA and ESBs, abstract endpoints are used for utilizing effective protocol-switching
As Bruce goes on saying, this is how the situation is right now, and its perfectly possible that BPM and SOA will fit more tighter together in the future, when BPM processes starts to utilize services of a SOA. Bruce then goes on argumenting how human tasks should not be part of a SOA, instead placed in BPM layer.
In another, more recently, post from Bruce, he talks about BPEL: “Human-centric BPMers’ lack of love for BPEL is today taken for granted, but who knew there were BPEL-haters out there in the SOA world as well?”. Further he goes on saying “BPEL is really good for orchestrating business services out of fine-grained APIs — how SOA uses orchestration. It’s less good — although it can do it — for orchestrating end-to-end business processes.”. And I think I agree with him on that on, because BPEL has already proved its lack of portability and everyone that has taken a look at the specification knows that the simplicity is no longer a design goal here (the BPEL 2.0 standard are even not backwards compatible, more on BPEL here). So, further I agree with Bruce that BPMN-model should be the best way to model and make portable business processes. Different platforms and vendors then have differents techniques for implementing/converting these models into a executable process language (like BPEL or something else).
November 23rd, 2006
Found it at Infoq, right here. Mark Richards from IBM talkes about 65 minutes about ESBs in a vendorneutral, critical and very informative way, summary:
Mark Richards tells us what an ESB is, its role, what capabilities it provides, and the various ways an ESB can be implemented. He takes a close look at the JBI specification (JSR-208) and explains what impact it will have with the ESB world. This will teach you how to determine your own specific requirements for an ESB and then match these requirements to the product space.
A few special points that I thought was interesting:
-Mark says that the idea of a federated ESB is for extremely large businesses, often like worldwide organizations that require an ESB for each division and department. Federated ESBs can communicate between each other and its possible to implement services that span multiple ESBs. I am really wondering what David Chappell over at Sonic has to say about that 
-Mark also goes on pointing out 10 capabilites that an ESB should have and then maps these capabilites onto 4 architectural components that can be spread out on multiple physical servers. These 4 components are mediator, choreographer, service registry/mapper and rule engine.
-Mark arguments why its best to have the mediator in front of the choreographer, and I really agree, thats really make sense. Then the choreographer acts just as a normal client to the mediator and then you dont have to implement a business process for each service
-What I dont understand in the presentation is the part about choreography vs orchestration. Mark says that choreograpy is about making business processes of the abstract business services while orchestration are binding the physical services together. This I really dont understand, please explain!
-Mark then explains a bit about JBI and explains what many people really dont understand, that JBI is first of all a concern for vendors that offers integration products. They have to write a JBI compatible message router and then write components that offers functionality. JBI standardizes the differents parts of an integration product (for instance an ESB) into Service Engines and Binding Components. This allows for mixing different kind of integration technologies from different vendors into your specific needs in a particular case.
-The last part he talkes about mule and servicemix and gives us a short clue on what their actually doing.
Mule is a simple ligtweight messaging framework that offers a container for managing UMOs. UMOs are POJOS that can do anything and that are connected to the outside world through communication endpoints. Extremely fast and ligweight, but you have to write a lot of code yourself, Marks says. ServiceMix actually is something totally different. ServiceMix is a 100% JBI compliant ESB and therefore provides a implement of a Normalized message router and a lot of different service engines and binding components to plug into the router. Thats why codehaus has both these offers, because the two products offer something different.
October 24th, 2006
Alle presentasjonene fra It-tinget 2006 ligger nå tilgjengelig her, veldig mye interessant! It-tinget er et årlig arrangement i regi av Ergogroup som tar for seg utviklingen innenfor offentlig it og it-politikk. Jeg var ikke tilstede, men har gått igjennom presentasjonene og her er mine kommentarer/oppsummeringer fra noen utvalgte presentasjoner:
Digitalisering af den offentlige sektor - En statusrapport fra Danmark: I Danmark jobbes det med en tilsvarende løsning som MinSide i Norge. Denne har de kalt borger.dk. Mye av den samme funksjonaliteten og målområde som minside, som skal gi innbyggerne et felles grensesnitt til både statlige og kommunale tjenester. Første versjon slippes 1.januar 2007 og heter altså borger.dk, mens andre versjon som blir mer innholdsrik heter borgerportal.dk og kommer i 2008. I Danmark har de også en ekvivalent til Altinn, i portalen virk.dk.
Hva innebærer NAV-reformen og hvilke krav til IKT-strategi? Arbeids- og velferdsdirektør Tor Saglie: Grundig og helhetlig presentasjon av både forretningsmessige, organisatoriske og teknologiske utfordringer med NAV-reformen. www.nav.no ble lansert denne uken. Dette er en ny statlig portal til erstatning for www.aetat.no og www.trygdeetaten.no hvor foreløpig innhold og tjenester fra de tidligere statlige etatene kun er overført, altså ingen nye tjenester, men de gamle samlet i en portal. Man skal blant annet med denne portalen prøve å flytte så mange brukere som mulig med tilhørende tjenestetransaksjoner over på nett. Videre vil strebe etter en it-strategi som kan bidra til å realisere prinsippene i
eNorge 2009 nemlig: Kommunikasjon internt i det offentlige skal skje elektronisk, felles og forpliktende politikk for bruk av åpne standarder, felles og obligatorisk sikkerhetsportal og standard for bruk av PKI.
Direktør Offentlige Løsninger i ErgoGroup, Håvard Larsen, sin presentasjon om å digitalisere offentlig virksomhet: Oppsummerer fra CapGeminis årlige eGovernment-undersøkelse: Norge tilbake på topp med e-forvaltning, blant EUs absolutt flinkeste innen offentlige e-tjenester og Norge på 5. plass foran Danmark og Englan. Østerrike best med 80% av borgertjenestene på nett. Norge får høy score for elektronisk selvangivelse, arbeidsformidling, arbeidsledighetstrygd, studielån og byggetillatelse. Lav score på pass, førerkort og kjøretøyregistrering. Videre består store deler av presentasjonen av en meget bra gjennomgang av 5 fokusområder knyttet til offentlig it: Arbeidsflater, Integrasjon, eHandel, ASP/Drift og eDemokrati.
Spennende presentasjon fra Visma som gikk mye pÃ¥ kundetilfredshet, kunjelojalitet, byttekostnader, monopoler osv som kan anvendes pÃ¥ mange omrÃ¥der bÃ¥de innenfor it og politikk. Hvis byttkostnader for en kunde er svært høye fÃ¥r man en høy lojalitet, men “Hvis kundelojaliteten er basert pÃ¥ kundetilfredshet er man mer robust mot endringer i teknologi, markedsforhold og rammebetingelser”. Videre spørres det “Men, nÃ¥r er kunder sÃ¥ tilfredse at de bidrar til vekst og lønnsomhet?” og at dette kan undersøkes med et enkelt spørsmÃ¥l: “Kjære kunde, er du sÃ¥ fornøyd med vÃ¥re varer og tjenester at du anbefaler oss til venner og bekjent?” og “Bare de kundene som svarer JA er virkelig lojale og bidrar derfor til langsiktige lønnsomhet”. Utifra dette er det altfor mange organiasjoner, virksomheter og bedrifter som har alt for lite kundedialog.
Prosessforbedring i helsevesenet med bruk av Oracle e-Buisness Suite, Erik Johannessen, Seksjonssjef, Seksjon for administrative støttesystemer
IT-avdelingen, Rikshospitalet-Radiumhospitalet HF: Rikshospitalet ønsker effektive informasjonssystemer som understøtter kontinuerlig prosessutvikling. Man ønsker en helhetlig prosessorientering som innebærer: På hvilken måte genereres informasjon, Hvordan brukes informasjonen, Kompetanse og metoder for workflow analyser og kontinuerlig prosessforbedring, Fokus for bedre utnyttelse av ressurser, Etablere måleparameter og avvikshåndtering, Fokus også for �den gode pasientopplevelse�. Vi blir presentert for et meget spennende eksempel som viser prosessflyt ved en sengepost i Lungemedisinsk avdeling. Dette illustrerer Rikshospitalets svært komplekse administrative prosesser. Videre går Johannesen igjennom hvordan de har fått til prosessforbedringer i flere administrative avdelinger, effektivisering av logistikk og veien videre via mer digitalisering.
October 4th, 2006