CSG Winter 2015 – Integration Workshop

Beth Ann –

Integration getting more complex and it’s an area where central IT has always struggled. Demand is rising.

Do you have an integration strategy? 11 do 2 don’t

What’s driving it – Innovation, security, new ERPs, BI/CRM, Other: diverse, distributed cloud systems

Where does integration responsibility reside? –  Central IT / Middleware – 6, distributed – 5, virtual group or hybrid – 2, shotgun – 1

Do you have an esb? WSO2 – 4, Mulesoft – 1, Oracle – 2, None yet – 2.

Biggest app challenges – Workday, mainframe, ERPs, departmental apps, apps that require data transformation

Manage integration performance? Most don’t or have limited perf management

Security? certs, “service” accounts, custom tools, policy, platform (Boomi, MuleSoft), Nginx, IDS, Kibana, encryption firewalls.

Critical staff skill sets: Data architecture, advanced programming, basic programming, business analyst, config management

Starting point: Myriad of point-to-point integrations, not well documented, fragile, difficult to be agile,

Drivers for change: New student system, SAAS solutions gaining traction: Concur, Canvas, Success Factors (used in health system and HR training), MIR3

New Approach – SOA, ESB

Benefits – Flexibility, Scalability, reasonable cost, availability, centralized management and monitoring

ESB selection criteria – Services: pub-sub, synchronous messaging, synch messaging, transaction support, transformations, web service generation, guaranteed delivery

Deployment Environment: Scalability, avaialbility, load balancing, clustering

Governance and Deployment: versioning, deployment, upgrades, support roles

Runtime

Security

Three finalists: WSO2, Mulesoft, Oracle

30 applications in production

Open Data Initiative – sparked by undergraduate assembly resolution to open up access to non-confidential dat sets, implemented as restful APIs. Deployed APIs: map item filter service, map item filter parameters service, courses catalog search service, course section search service, course section search parameters service, dining service, directory search service, directory person details, news/events/map search service, transit data service.

Used by departmental web apps, student developed web apps, and (soon) student developed mobile app

Development and Slumni Relations – Key system is Peoplesoft app, real time sync of key biographic and contact data with IModules online community; high volume; fire and forget

Canvas – Real time enrollments; very well received by students, faculty, and staff.

Lessons learned: Don’t get too far in front of vendors (every release of one product breaks APIs), Mulesoft – support generally good, but long wait times for some fixes; Mulesoft – upgrades more difficult than expected; Mulesfot – development environment wasn’t set up for how they liked to work, had to tune; Instructor – API issues… questionable design choices, bugs, throughput problems, documentation; Instructure – overburdened test environment – ended up paying for a dedicated test system

In the beginning – standalone APIs, buildings/ Campus maps, Project Task Awards check – hard to find, moved around; Other random APIs – SOAP, RPC< rematch, RMI, RESTish

“SOA is Dead; Long Live Services” – Anne Thoimas Mane, Burton Group, January 2009

CAP – Community Academic Profiles – Med School Faculty Profiles. Was Java + Oracle; Dept Drupal Sites that wanted data – not java, not oracle, very common, need profile data. Common answer: RESTful API, JSON, OAuth 2.0.

API “Gang of Four” forms. On campus Apigee workshop validate approach. CAP API using Spring and MongoDB – generate JSON versions on profile changes; OAuth AS using Spring. API gateway using node.js. Minimal feature set

Early 2013 – CAP API design & implementation, client/server only, API gateway implementation.

Early 2014 Planning – API gateway enhancements, OAuth 2.0 AS enhancements (user (3 legged) tokens, SAML integration), Thin client + API for accounts system; developer portal

Mid 2014 – Reorg interrupted things but still moving forward

API Discussions: Real-time bus location; device compliance system; AED locations; environmental health & safety; staff & faculty training; directory data.

Common Concerns: opening up data, lack of desks, APIs and OAuth are new and scary; system load and support; infrastructure development

Still to do: Developer portal; “3 legged” Oauth; New gateway features: caching, token verification, rate limiting; UI for managing tokens

Lot of interest in  SOA and ESB from a lot of people for a very long time.

Software solutions in varying degrees of use.

2013 IT Strategic Plan included a section on Integration – still needed to figure out implementation.

Hired a Director of Systems and Data Integration. Workday, opportunity to redo many integrations. CIO very interested in exposing APIs. Infrastructure becomes “exostructure”.

Systems and data integration – created a rationale, scope, and vision document. Central integration competency center – 2 people located with the DBAs, integration community of practice, training for all app teams on new tools.

Defined a three year time horizon in constraining discussion about tools. Had advocates for Fuse/Service Mix, Dell Boomi, IBM Cast Iron, WebMethods. Determined that WebMethods would work well (they already had it) for short term.  Focused on gaps and ended up selecting Layer7 for API management. Uses Telend for easy ETL work.

Merged IDM with systems and data development. Developers, platform admins.

Yale Data API service – being socialized by web team for reuse of data on sites. Data governance – quietly beginning to get some traction; Workday integrations.

Current business and use cases for better integration – New ERPs, mobile experience, achieving the “connected campus” means breaking the ERP silos, engagement layer will create many more integrations; seamless intuitive user experiences – drivers for better application integration of extension tools.

Mobile experience – students expect realtime rich data. Use cases: registration, mobile access (ID cards) and identity, innovation.

Emerging and future use cases: increased importance of BI and CRM – engagement layer for experience, aggregation of data across multiple points, actionable data and outcomes; Internet of Things; Agility in service adoptions (cloud) means data integration must be agile too.

Challenges: Technology – systems that were never designed to share data; vendor lok in issues, lack of APIs; fragile integrations that break; performance and monitoring. Business process: data elements defined in isolation. Greater complexity: no longer just point to point; new skills for staff; new situational awareness.

Path forward: Gain situational awareness by mapping and inventory of integrations across stacks, understand the gap between use cases and capabilities. Track or adopt iPaaS solution with standardized connectors; Build integration center of excellence, shift away from artisanal handcrafted integrations, unified voice with vendors to push for modern integration capabilities, integration capability needs to be a heavier incluence on priorities for service/system modernization and product service selection; align with data governance.