IT Management – Oren Sreebny

CSG Spring 2018 – Strategically reallocating/restructuring IT resources

Paul Erickson (Nebraska, Lincoln)

Stage Set

Jim Phelps (Washington)

Laundry list of technologies: hyper personalization; AI and ML; IoT; Autonomous systems & robotics; everything on every device, everywhere; virtual and augmented reality; Big data, data driven organizations; hyper connected world

Cultural impacts of digital transformation – will transform all aspects of work and society in the same way that the industrial revolution and electrification transformed work/society; DX will lead to whole new technologies never seen before; There will be whole new classes of jobs, skills, and competencies; This is a period of great disruption; We cannot predict the winners and losers, either technology nor business; we need to be adaptable; we need resources to invest and investigate during this time of change; higher education will be disrupted too; the winners in higher education may not exist yet.

Change Management will be a critical competency. Example of Nordstrom adapting their culture of perfection to an agile IT process.

Are we a commodity provider or a business transformation partner? If it’s a commodity we can expect budgets to keep shrinking and visibility to go lower. If we are business transformation partner it implies whole new sets of skills. Staff must have customer experience and business skills. Teams need to respond quickly to business changes and opportunities. We need to be change leaders. IT needs to be transparent and build trust and actively manage relationships with the business. We need to understand the business well enough to bring opportunities to the table.

Transformation of IT’s stack itself: DevOps and all it implies (full-stack teams, job rotation, joing meetings, “servant leadership”, etc), chaos monkey. Culture shift – learning, trust, collaboration.

“We need to become IDEO consultants for campus, not Gartner consultants” Matthew Rascoff – Duke

Steve Fleagle – Iowa – OneIT

Board hired Deloitte to look at IT improvements. Adopted 4 recommendations. Proposed a 3 year self-implementation plan – 16 projects with a savings target of $3.6m.

Structure – project teams; coleaders from central and distributed units; program office to oversee projects; Steering committee

Communication – web site, kickoff meetings with each college, town hall meetings, IT leader retreats, monthly newsletter mailings, administrative updates.

Nine of 16 projects complete, seven active projects on track for closeout by end of FY18, 27.39 FTE redirected or eliminated. Two biggest efforts were desktop/device management and data center consolidation.

What went well? Executive support: Provost, new president; Clear direction from and access to the Board; Intentional transparency; High staff engagement: license to collaborate; Momentum: strong kickoff and early wins; Structure (program office, steering committee, change management)

What could have gone better? Challenging to convey what Board wanted and articulate parameters; generating campus buy-in was a long, exhausting process (lack of synergy at senior leadership levels. collegiate IT leaders caught between Deans and CIO), Mixed messages about involvement of other regent schools, not enough focus on cultural/political factors – us vs them mentality, when everyone agreed it was easy, changing roles felt threatening and caused resistance).

Lessons: Describe future state clearly or people won’t let go of the past; executive support is critical; mandate is a blessing and a curse (you still need buy-in); people need to know how change impacts them personally; helpful to identify common ground and diferences up front; good project managers are helpful; figure out who has skin in the game and work through concerns one conversation at a time). People really like success stories of how other people have gone through things.

Next Chapter: IT integration with health care IT

Brett Blackman – Nebraska – OneIT

President asked to look at efficiencies in IT across Nebraska, which is 4 campuses. Drivers: scale/efficiencies, improve IT security, improve/maintain services, cut $6m in permanent budget.

Strategy – fromed team structures to review IT organization (March 2017) – combined 360 central IT staff, 200 distributed IT staff. Learned from peers. Communication – be transparent with staff. Implemented new org model (September 2017) – balanced between scaled services and forward facing campus services; 80% of staff had some change to job; community of practice teams.

Five skill areas across the whole consolidated org: IT strategy and planning; Client Services; Security; Enterprise services; Infrastructure. Specific academic and application services exist at the individual campus level.

Outcomes – OneIT is the foundation for $6m in savings – reduced staffing largely through attrition; IT efficiencies; Procurement (joint contracts); Aligned distributed IT (administrative and academic); unifying central ITS campus budgets. Improved security; Improved/mantained services;

Lessons learned: Communication; Culture…change is hard. Enabling leadership at all levels; transition planning

Not everyone believes in collaboration when it means giving up local control.

IT, Procurement, and General Counsel at Nebraska (Paul Erickson)

As IT moves off-premise, more services are governed by contracts – IT has traditionally struggled with Ts&Cs. Not really commodity buy, so Procurement and GC weren’t comfortable with the issues. Had a multi-year effort to articulate ideal Ts&Cs.

Enterprise Architect & Strategy @ UW (Jim Phelps)

Shifting EA practice – was downward facing at tech, now very business architecture focused. How to link strategy management to investment planning to project portfolio management. EA value proposition – help set and then lead vision of change. Creates need for lots of workforce development.

IT Help desk Consolidation & Opportunities for Innovation – Phoebe Johnson (Minnesota)

Help desk Alignment – the who, what, where, and when. In 2012 there were more than 73 IT help desks across the University system.

IT Support Alignment: Cost savings; Great service; Regional zone support.

2 success stories: Liberal Arts Technology & Innovation Services – allows them to invest in relationships with faculty, students, and staff and foster a culture of innovation. College of Food, Agriculture & Natural Resource Sciences – Went from 8 end user support people to 1.5, freeing up people for academic and research tech and app development. Allowed them to focus on online courses.

Technology Advisory Council – Phoebe Johnson (Minnesota)

System-wide membership, deep technology expertise.

Purpose: Provide UMN staff and faculty with expert guidance & advice; reduce institutional cost; avoid redundant technologies; improve alignment with UMN standards, policies & practice. Engage in activities like work in RFP processes.

What we do: expert consultation and support to units; connecting people with similar needs; A portfolio of technologies.

How it works – inspiration, research, assessment, selection, purchase & implementation

Sarah Christen – Cornell

Wanted cloud initiative to partner with the rest of campus, not just central IT. Started talking about developers moving into a “broker” role – but that wasn’t a popular term. Moving from operators/administrators to innovators, developing an infinite number of solutions to meet problems. Creating infrastructure as code, and operations tasks are automated (which takes innovation), highly variable technology stacks – we consult ranter than run. Updates are frequent and automated (but still planned). New products are purchased frequently and need to be integrated into larger solutions.

Central IT as a transformation partner: focus on partnership – don’t reinvent the wheel when central IT can fill the gap. Staff dedicated to helping campus make the transition from the data center to the cloud – documented best practices, refactoring applications, collaboration venues, help/support tickets. Goal is to help with transition and training so the team an support and maintain their services.

Staff transitions: Desktop engineering (Appstream, VDI, CM and JAMF); Storage (starting to backend backups to the cloud, file replications in AWS); DBAs (embracing RDS, exploring new DB platforms, moving away from Oracle).

CSG Winter 2018 – Research and Teaching & Learning IT: Partnering with the Library

This morning’s workshop on partnering between IT and Libraries features Jenn Stringer/Chris Hoffman (Berkeley), Jennifer Sparrow/Joe Salem (Penn State), Diane Butler (Rice), Cliff Lynch (CNI), Louis King (Yale), David Millman (NYU)

The morning is starting off with some thoughts from Cliff Lynch (CNI):

Reminders of some things many haven’t lived through: In the early 90s there was a call not only for collaboration between IT and Libraries, but serious talk of merging. It was tried at a few institutions, like Columbia University. The takeaway was that it’s fairly crazy at large institutions. The mission expansion of each has been in differing rather than overlapping areas. But it’s been successful at a number of liberal arts organizations.

When CNI was founded it was totally viewed as a collaboration between the CIO and the head of the Library at member institutions. In the early 2000s that makeup was changing. The representation was the head of the library and someone doing research or academic computing, or doing digital work in the libraries. Led to increasing disengagement of the CIOs. Starting around 2000 started putting on executive roundtables with the intent of re-engaging the CIOs. It was fairly easy in the first few years to come up with topics in that sweet spot, but it got harder. If you look back from 1990 – 2005 you see that Libraries had low levels of technical expertise. At the same time libraries had developed some internal expertise in technologies important for digital humanities, data curation, etc, where there is now more competence than in the central IT org, which has structured its mission around infrastructure, compliance, etc. Libraries continue to rely on IT for fundamental infrastructure.

If you look at the landscape, how much IT capability is native to the library, and how much replicates or compliments the expertise in IT. This is hugely inconsistent. If you polled the CSG campuses you’d be surprised at the degree of variation in organic IT expertise in the library.

Collaborations involving library have become much more multilateral rather than bilateral with IT – involving partners like University Presses, Museums, research data management, digital scholarship centers (often involving academic school or department), geospatial centers, maker spaces. \

Don’t forget collaboration on institutional policies. Data governance, privacy and reuse of student data and analytics, responsibility of university to preserving scholarly products. Had a recent roundtable looking at policy implications of adoption of widespread cloud platforms.

This area does not lend itself to checklists.

UC Berkeley – Chris Hoffman

A history of good intentions – Museum Informatics Project – Housed in Library, Digital collections and DAMS. Complicating factors: Sustainability, budget cuts, grant funding; priorities; loss of key champions; culture.

Collectionspace – managing collections for museums.

Research Data Management – an impetus for change. New drivers (DMP requirements), new change leaders, new models for partnership. Benchmarking justified need. Broad definition of research data – all digital parts of a research project. Priority to nurture collaboration between IT and Library. Co-funded a position for program manager. Campus-wide perspective, investing in understanding and bridging cultures.

What’s next? More challenging tests to partnerships, RDM 2.0, Visualization and makerspaces, more fundamental technologies? (archival storage, virtual teaching and research environments); strategic alignment?

NYU – Stratos Efstathiadis, David Millman, David Ackerman

Research technology works closely with LIbraries.

Data Services – estab. 2008. 11 FTE Consultation and instructional support for scholars using quantitative, qualitative, survey design, and geospatial software and methods. Joint service of IT and Libraries.

Digital Library Technology Services – estab ca. 2000. Digital content publication and preservation. New services to support current scholarly communication. R&D to develop new services and partnerships, 19 FTE.

Research Data Management Services – estab 2015. 2 FTE. Promulgate beset practices in data organization, curation, description, publication, compliance, preservation planning, and sharing.

Research Cloud Services – new collaboration build on other preexisting services. Inteconnected research storage environment. REimagine a spectrum of cloud storage from dynamic to published final products. Provide backbone for researchers but also Libraries collections and workflows.

Yale – Louis King

Considerable history at Yale in working in digital transformation space.

Office of Digital Assets and Infrastructure – Sept 2008. Work closely with Library and ITS. Focus on Digital Assets & Infrastructure. Take advantage of disciplinary approach of libraries and technical capacity of IT.

Looking for ways to gain efficiencies and lower overhead for people who want to manage digital content.

Had some substantial initial success, but changes: Initial provost sponsor left Yale, 2009 financial crash, VP retired, two library director transitions, transition in IT director, emerging digital systems in Library.

Late 2012 relaunch as Yale Digital Collections Center, but closed in 2015. But it catalyzed momentum towards digital transformation at Yale. Established the foundation for many successful current and future collaborations.

Rice University – Diane Butler

Library and IT have been partners for a long time. For a very short time, the organizations were merged. Research IT and library have been partners since 2012 and informally even further back. Began iwth library providing the service and IT providing the core infrastructure but has morphed into a collaborative partnership.

Areas of collaboration: Data Management (through Library). Provide consultation, including creating DMP, describing and organizing data, storing data, and sharing data. Training, Access to resources such as platform for sharing and preserving publications and small-to-medium datasets. Still an area for work as faculty aren’t very engaged.

Digital Scholarship: Service provided by library with IT providing infrastructure. Preserving scholarship, navigating copyright and open access, managing and visualizing data, digitizing materials, consultation, etc. Research IT has history in supporting engineering and sciences, but not so much in humanities.

Digital Humanities: Imagine Rio Project. Most successful collaborative project to date. An architecture and history professors joining together to imagine Rio de Janeiro. Searchable atlas of social and urban evolution of Rio.

Positive outcomes: Research IT had not supported Humanities or qualitative social sciences previously. Success of project has brought in more funding. Research IT now has 2 facilitators that are working with faculty in those disciplines.

At Rice the board has come up with some base funding for research computing, so that all of the work doesn’t have to be funded by grants.

Penn State – Joe Salem and Jennifer Sparrow

Strong history of working with libraries, IT, and student services on accessibility issues. Thinking about spaces in place and how to leverage institutional spaces. Built a “blue box” classroom.

Worked on the Dreamery – a co-learning space for bringing emerging technologies onto campus.

Driving strategic initiatives: Collaborative, technology-infused space. Inherited a space called the Knowledge Commons. Includes a corner with staffing from both Libraries and Academic Tech. Service partnership profile has grown from just a focus on media, to overall platform for supporting students. Work on curricular support together – open educational resources and portable content. Instructional design is a focus.

Learning Spaces committee – Provide leadership in innovative instruction.

What makes the partnership work … or not? What does each side bring to the table?

Chris – Berkeley

Visualization service at Berkeley. HearstCAVE: Connected virtual spaces over the Pacific Research Platform around preserving archaeology preservation. Thinking about how it connects with data science.

Markerspaces at UCB – pockets of excellence and experimentation. Jacobs Institute for Design Innovation. Talking with library and ETS to look at space.

Hooking the two together in a Center for Connected Learning.

Research Data Management at Yale
Much Ado about Something: Complex funder requirements; reliable verficiation of results; reuse of data in new research.

What are the responsibilities and rights of the University and faculty regarding research data? They put out a Yale Research Data & Materials Policy. Developed over 2-2 years with collaboration across the university. There is significant collaboration in support of that policy – Library and IT collaboration: Research Data Strategic Initiaitive Group, Research Data Consultation Group, Yale Center for Research Computing.

Recommendation: Research Data Service Unit; REports within LIbrary – Assessment, coordination, outreach and communication. Federated support model for all research data support services – research technology, data management, metadata, outreach & communications, customer relations, education and training, research data administrative analytics.

NYU – David Millman

Bottom-up requirements – survey local researchers: IT/Lib complementary styles, contacts. Survey peers: IT’Lib coordinated.

Executive review: Dean, AVP-level

NYU – research repository service identification. Umbrella of services – – researc lifecycle. Creation, manipulation, publication, etc. Holistic — customer focus. 1. HPC storage. 2 – medium” performance storage (CIFS, NFS); 3 – “published” sotrage – preserved, curated, citable.

IT/Library crossover strategy questions: business of universities: long-term preservation of scholarship. Any updates on our participation in digital preservation facilities? Some of our colleagues have recommended highly distributed protocols for better preservation. How do we approach this?

CSG Winter 2018 – Much Ado About GDPR

We’re in sunny, warm LA for the Winter CSG meeting, hosted by USC. Last night, Asbed coordinated a group to go out for tacos at http://chichenitzarestaurant.com/ , which was excellent!

This morning we’re starting off with a workshop on GDPR, featuring: Sharif Nijim (Notre Dame), Jim Behm (Michigan), Paul Erickson (Nebraska), Alan Crosswell (Columbia), and Kitty Bridges (NYU)

GDPR = General Data Protection Regulation – 127 days until enforcement on May 25

Membership survey :
87% think GDPR is an institutional risk
58% identified as beginners in GDPR
70% either don’t know or don’t think their institution will be compliant
41% have engaged outside counsel
50% General Counsel and IT partnership to lead compliance initiative.

What is GDPR? Alan Crosswell.

EU regulation on personal data protection, applicable to people, products, or services. Replaces old regulations dating back to 1995. Covers: personal data (relating to people). Examples: IP address, genetic data, health data, research data, video surveillance. Who is covered? EU individuals or any company that offers products/services to EU individuals or collects/processes their personal data (includes non-EU citizens located in EU).

Requirements: Identify personal data; data protection by design; individual rights on data usage (transparency, right to data erasure, right to data portability, etc); obtain proper consent (opt-in); withdrawn consent and the right to be forgotten (opt-out); breach notification; designate data have to designate a protection / privacy officer (DPO).

What does it mean for a student to have the right to be forgotten?

Penalty: Failing to report breaches within 72 hours maximum of 20 million euros or 4% of organizational annual revenue – whichever is greater.

Preparing for GDPR – key steps: Promote awareness; discover PII you hold; implement data protection by design; identify legal basis for processing personal data; review procedures for communicating personal data, individual data rights, data consent, guardian consent for minor’s data, data breach detection, response, and notificaton; designate data protection / privacy officer.

Definitions:
EU Indivdidaul – physically located in an EU member state, both EU citizens and non-EU citizens.
Personal Data – relating to identified natural person. name, ID number, location data, online identifier, address, email, passport, cookies, drivers license, etc
Consent: freely given, unambiguous indication of data subject’s wishes of subject’s wishes.

Question: does this include firewall logs? General agreement that it does.

Comment: This is subject to legal jurisdiction, and the thought that this is generally applicable to everything we do might not be correct.

GDPR Scenarios

Recruiting: NYU recruiter holding open house in Paris for EU people to find out about NYU. Recruiter gathers name, interests, and hands over wifi credentials. Need to give an explicit consent form, saying which elements are collected, what they’ll be used for, and how long they’ll be retained. Has to be provided in the native language. (Is your admissions prospecting software aware of and planning on how to handle GDPR? That’s institutional data – it’s an indemnification issue. What kind of contract language do you have?).

Admissions: Need name, national ID, country of origin, addresses, high school transcripts, etc. to make effective admissions decision. Also use that information for research (see Unizen). How is consent for data tracked through the various systems? (Common App Organization GDPR adjustment ETA? – “early spring”).

Question – has anyone reached out to European universities on what they’re doing to prepare for GDPR?

Matriculation – example of alleged assault from student abroad. What happens if student exercises GDPR rights to not share data back to the US? Could contracts with partners abroad be affected if we don’t behave according to GDPR? Example of LMS vendor that is spinning up version of LMS in the EU specifically for GDPR – do we keep our data there for EU citizens?

Research – What about information about researchers kept on servers? Do legal federations with agreements help us? GEANT did a study on GDPR impact on Edugain. Emerging attribute release agreements help with GDPR compliance. GEANT is submitting a new code of conduct for GDPR – a way of publishing attributes in an open and transparent way. Coming out later this year. Transparency, documentation, and incident response are critical pieces.

Alumni and Benefactors – What data are collected and where is it? What if they want to be removed? Compliance might be viewed as a revenue issue. There is a notion in GDPR of “legitimate interest” but that isn’t a blanket clause.

Comment: We should follow advice of counsel on how to approach GDPR. It may not be worth a lot of worry at this point about how much this impacts us. Just because it’s over the Internet doesn’t make it different than any other issue between countries and how citizens are treated. We all need to decide what our risk posture will be.

How many campuses operate summer camps with people under 16 from EU countries?

If institutions are backing away from collecting citizenship data (from concern about undocumented people), does that impede our effort to comply?

Educause and GDPR: Trying to curate best resources – see page at: https://library.educause.edu/topics/policy-and-law/eu-general-data-protection-regulation-gdpr

Good to start with JISC resources. https://www.jisc.ac.uk/gdpr

Territoriality – we higher ed institutions generally have enough business links that we should surmise that GDPR might apply in some way.

Educause is working with other US higher ed groups (NACUA, etc) on GDPR guidance. It’s slow going, and all organizations are struggling with what advice to give members.

Notre Dame – Initial meeting with General Counsel (8/2017); Elevated to information governance committee (9/2017); Assigned to IT by institutional risk committee (10/2017); Compliance questionnaire circulated (11/2017); Questionnaire data aggregated and analyzed (1/2018) – Hard to collect data across the institution – will need help from general counsel in complying with collection. The vision is that data stewards will be accountable for the data in their areas. Impossible to collect every last piece of data, but important to show due diligence and have a process for dealing with issues.

NYU – Have hired external counsel – issuing questionnaires. In data collection mode, focused on central administrative entities. Don’t yet know what the institutional posture will be. General Counsel will advise. Will likely think of this as responsibility of business offices, who have been involved in discussions. IT is a key partner, knowing how things are connected together. First thing to focus on: Documenting identity data; movements of data between systems; prioritizing what to worry about first (biggest risk). Especially tricky areas are warehousing, analytics, and logs. Logs: operational logs (IP addresses, MAC addresses, authentication logs, DB logs, application logs) used for troubleshooting and trending. Can they even be made anonymous? Audit logs – understanding who has access, understanding really how long things need to be kept in identifiable form.

Nebrasksa: Bringing together multiple conversations around GDPR – General Counsel coordinating. Work in progress – expect to at least have posture before deadline. NACUA webinar was very helpful. Distance ed group started early on. Good test of relationships across campus – IT as implementer. Research group is interested in GDPR to help guide data governance. Indemnifaction – example of SaaS contract where vendor struck out “global standards.”

UMich – Started this past summer – taking a “cautious approach”. Concern about the extent to which regulations will apply to US institutions. Group led by General Counsel with representation across campus. Counsel has hired a consultant to help guide campus through the process. There’s enough gray areas that it’s unlikely that campuses will be held accountable in May. For state institutions, it may be the state that is accountable, not the institution. Might not be the case at Michigan.

Rice – Chief Compliance Officer leads a working group with the CISO. Creating an institutional web site for information.

UVa and Va Tech – very early in process. Conversations with General Counsel. State AG’s office has hired counsel who should be issuing guidance for state higher ed.

Ron – IT is the only organization that touches every other organization in multiple domains – so it falls to us to be of service.

Minnesota – Counsel leading effort, still assessing impact and how much needs to be done.

Iowa – In due diligence approach, with Counsel taking lead. Will be naming a privacy officer. Creating a plan for operations that take place in the EU, which is a relatively small set.

CMU – very early on. Taking gap analysis approach.

Sharif – taking the approach that much institutional data is “legitimate interest” vs. asking specific consent. But that still requires transparency. How far does legitimate interest go?

Maybe this worry is overblown (like we did with CALEA)? It’s primarily targeting Googles and Facebooks, not higher ed.

Should we be reviewing cloud contracts for how GDPR is or isn’t covered? Could Educause help come up with a checklist for review? To what extent does it affect Net+ contracts? (e.g. LMS). We could have an area on the CSG site for sharing information.

We may be likely to see something analogous in the US, so this won’t be wasted effort. Much of what we need to do for GDPR are just good enterprise data practices.

It’s not an IT project, it’s about institutional risk. Should be part of that regular assessment process.

CSG Winter 2017 – Recommendations/guides for updating IT skill portfolio

Paul Erickson – Nebraska

Framing the issue – don’t have the skill sets or expertise for a cloud world. Run, grow transform – on-prem, traditional focus on “run” – how to change a working environment and shift investment/resources (how do you change an engine while the car is running?)?

What skill sets are we missing? Process management (granting/revoking; provisioning; integration; authorizations/permissions, vendor coordination, managing interdependencies); Integration; Product/Service management; Client relationship management.

People who contributed in the past might not have the skills to take us into the future. How do we offer them opportunities to grow that honor their contributions and allow them to grow? Adapt and evolve in an environment of continuous change.

Identify ideal employee skills. Help those who are great technologists make the transition.

Denise Cunningham – Columbia

Head of HR for technology division.

One reason people resist change is because they focus on what the have to give up instead of what they have to gain. Important to keep this at front of mind.

A framework for Organizational Performance & Change: Burke-Litwin Model

burkelitwinmodel

External environment (e.g. the cloud) impacts the organization. The spine of the model – external environment influences leadership, which influences management practice, then work unit climate, then motivation, then individual.

Focus on Work Unit Climate: What it feels like to work here; nature of our interaction with each other; interpersonal relations in the group; what we focus on and consider important.

What factors influence Work Unit Climate? Leadership and management practices. Work unit climate is the most direct factor in performance.

There’s a learning climate or a performance climate. Learning: emphasis on improving skills and abilities; stresses process and learning; motivated to increase competence and change. Performance: emphasis is on demonstrating skills; stresses outcomes and results; people are afraid to make mistakes or change.

Goals: Learning: quality, trying new things original ideas; effort. Performance: following standard procedures; high performance standards; getting task done on time.

Feedback: Learning climate: supportive/coaching role; improving work quality; two-way feedback, questions encouraged. Performance climate: evaluative role; level of competence compared to other employees, one-way feedback, questions discouraged.

When implementing change employees want to hear about it from their manager.

There is no correlation between strong individual contributors and leaders.

Changing organizational culture can take 12-18 months. Or two years in higher education. Can’t do it at all without leadership being a part of it.

When people say they’re going to get in trouble, that can be a rationale for not changing. How do you make sure new staff don’t become part of a dysfunctional culture. Ask questions at hiring about the core values. Zappo’s does this well. Build the values into the performance appraisal.

CSG Fall 2016: Large scale research and instructional computing in the Clouds

leaguefront

We’re at the University of Michigan in Ann Arbor for the fall CSG Meeting in the Michigan League. Fall semester is in full swing here.

Mark McCahill from Duke kicks off the workshop with an introduction on when and why the cloud might be a good fit.

The cloud is good for unpredictable loads due to the capability to elastically expand and shrink. Wisconsin example of spinning up 50-100k Condor cores in AWS. http://research.cs.wisc.edu/htcondor/HTCondorWeek2016/presentations/WedHover_provisioning.pdf

Research-specific, purpose-built clouds like Open Science Grid and XSEDE.

Is there enough demand on campus today to develop in-house expertise managing complex application stacks? e.g. should staff help researchers write hadoop applications?

Technical issues include integration with local resources like storage, monitoring, or authentication. That’s easier if you extend the data center network to the cloud, but what about network latency and bandwidth? There are issues around private IP address space, software licensing models, HPC job scheduling, slow connections, billing. Dynamic provisioning of reproducible compute environments for researchers takes more than VMs. Are research computing staff prepared for a more DevOps mindset?

New green field deployments are easier than enhancing existing resources.

Researchers will need to understand cost optimization in the cloud if they’re doing large scale work. That may be a place where central IT can help consult.

AWS Educate Starter – less credits than Educate, but students don’t need a credit card.

Case Studies: Duke large scale research & instructional cloud computing

MOOC course (Managing Big Data with MySQL) that wanted to provide 10k students with access to a million row MySQL database. Ended up with over 50k students enrolled.

Architecting for the cloud: Plan to migrate the workload – cloud provider choice will change over time. Incremental scaling with building-block design. Plan for intermittent failures – during provisioning and runtime. Failure of one VM should not affect others.

Wrote a Ruby on Rails app that runs on premise that maps user to their assigned Docker container and redirect them to the proper container host/port. Docker containers running Jupyter notebooks. Read-only access to MySQL for students. Each VM runs 140 Jupyter notebook containers + 1 MySQL instance. In worst case scenario only 140 users can be affected by a runaway SQL query. Containers restarted once/day to clear sessions.

At this scale (50-60 servers) – 1-2% failure rates. Be prepared for provisioning anomalies. Putting Jupyter notebooks into git made it easy to distribute new versions as content was revised. Hit a peak of ~7400 concurrent users. Added a policy of reclaiming containers that had not been visited for 90 days.

Spring 2016 – $100k of Azure compute credits expiring June 30. Compute cluster had all the possible research software on all the nodes, through NFS mounts in the data center. To extend it to Azure have to put a VPN tunnel in private address space. Provision Centos linux VMs then make repeated Puppet runs to get things set up, then mount NFS over the tunnel. SLURM started seeing nodes fail and then come back to life. Needed deeper monitoring that knows more than just nodes being up or down. The default VPN link into Azure maxes out at 100-200 Mbps, so they throttle the Azure VMs at the OS level so they don’t do more than 10 Mbps each. They limit the number of VMs in an Azure subscription to 20 and run workloads that do more compute and less IO. Provisioned each VM at 16 core with 112 GB RAM. Started seeing failures because there were no more A11 nodes available in the Azure East data center – unclear if/when there will be more nodes there. Other regions add latency. Ended up $96k used in one month. 80 nodes (16 cores and 112 GB RAM) in 4 groups of 20 nodes in several data centers. VPN tunnel for each subscription group.

(One school putting their Peoplesoft HR system in the cloud.)

Stratos Efstathiadis – NYU

– Experiences from running Big Data and Machine Learning courses on public clouds – Grad courses provided by NYU departments and centers. Popula courses with large number of students requiring substatial computing resources (GPUs, Hadoop, Spark, etc).

They have substantial resources on premise. Scheduled tutorialson R,MapReduce, Hive, Spark etc. Consultations with faculty, work closely with TAs. Why cloud? Timing of resources, ability to separate resources (courses vs. research), access to specific computing architectures, students need to learn the cloud.

Need a systamatic approach; Use case: Deep Learning class from the Center of Data Science. 40 student teams that needed access to NVidia K80 GPU boards. Each team must have access to identical resources to compete. Instructors must be able to assign resources and control. Required 50 AWS g2.2xlarge instances. Issues: Discounts/vouchers are stated per student, not teams. Need to enforce usage caps at various levels so instructor-imposed caps are not exceeded. Daily email notifications to instructors, TAs and teams providing current costs and details. Students were charged for a full hour every time they spun up an instance. AWS costs were estimated ~ $65k per class. On-prem solution was $200k.

Use case: Spatial data repository for searching, discovering, previewing and downloading GIS spatial data. First generation was locally hosted – difficult to update, not scalable, couldn’t collaborate with other institutions; lack of in-house expertise; no single sign on. Decided to go to the cloud.

Use case: HPC disaster recovery
Datasets were available a few days after Sandy, but where to analyze them? Worked with other institutions to get access to HPC, but challenges included copying large volumes of data and different user environments and configurations. Started using MIT’s Star (Software Tools for Academics and Researchers), could also use AWS cfnCluster. Set up a Globus endpoint on S3 to copy data. Software licensing is a challenge – e.g. Matlab. Worked things out with Mathworks. Currently they’re syncing environments between NY and Abu Dhabi campus, but they’re investigating the cloud – looking at star/cfnCluster approach, but also might do a container based approach with Docker.

CSG Winter 2014 – Digital Campus: IT, Media Management, Innovation, pt. 1: Vision

It was nice to make the change from -10 degrees in Chicago to 70 in San Diego! We’re being hosted for the Winter meeting by the kind folks from UC San Diego.

Kelly Doney from Georgetown kicks off the full-day workshop on the Digital Campus. With all the digital changes we see on campus, are these all a natural evolution of the role of CIO or is there something new? We’ll hear from Ron Kramer (Notre Dame) who has the title of Chief Digital Officer in addition to CIO. Is the role of the CIO changing? Is there a role for a Chief Digital Officer?

Survey results (15 institutions responding) – Only 2 institutions have a Digital strategy, but many are thinking about it. CIO is sharing responsibility for driving the digital strategy or directly responsible in 12 of the 15 schools. MIT and Notre Dame have a CDO, at Notre Dame it’s the same person as the CIO. At MIT, the CIO is not directly involved in the institutional digital strategy. What is the primary reason the campus does not have a CDO – Mostly because digital issues are handled adequately by multiple offices across campus, and the need to centralize ind a CDO is not pervasive or possible. Conclusions – a formal digital strategy seems rare. Digital can be and is handled by the CIO. Very few feel the need to appoint a CDO.

Ron Kramer, Notre Dame

What is the role of the CIO and IT in driving the campus “digital strategy” and creating the “digital campus” of the future?

Is this a time where we can take advantage of things going on around us to serve the universities better?

“Anything we render as digital can be open to measurement, learning, and ultimately iteration – nothing is finished and nothing is final”. So what is our role? Does IT become part of the conversation, influence outcomes, or take a leadership role? The IT organizations are the only ones that touch all the parts of the University, so that’s why it became the focal point of this effort at Notre Dame. Is the “digital phenomena” different enough to take special note? Is the core of what we do being disrupted by “digital”?

What’s in the CDO portfolio? Some possible examples: Operating the classroom; Evolving the Library; Creating and managing learning materials; Enabling digital publishing; Orchestrating the mobile ecosystem; Enhancing campus life when mobile enabled; Producing and managing video (going to build a $30 million digital media center at Notre Dame); Building new production facilities; Archiving, curating & preserving digital assets; Supporting research and data analytics; Delivering a demanding fan experience; Supporting increasing monitoring and tracking; Guiding architecture and design.

Lots of corporations are appointing CDOs, which came about largely from the marketing role. What is the role in higher ed of the marketing communications role? One school noted that the conversation is driven by the tactical (not strategic) orientation of the public affairs and communications role.

Panel –

Lisa Davis, Georgetown (CIO and VP for Information Services). Working closely with the COO and the Provost to figure out what a digital campus looks like and how they will evolve Georgetown. Looking at using digital tools and content to drive revenue back into the university to move the institution forward.

Tracey Futhey – Duke. 0s and 1s have been our life since we started our jobs, so what’s this new mantra? Historically we thought of partners as being the business people for enterprise systems, or the faculty for research computing, today the list of partners is further expanded, but that doesn’t mean there’s a need for a different strategy or title. But it is different than looking at IT as just the plumber role, but many IT organizations have already made that evolution.

Scott Midkiff – VP of IT and CIO at VA Tech. The combination of strategy and operations is important to have the leverage and resources to achieve new goals. Partnerships are important as we move new strategies forward. There are evolutions, but there are always areas where we have to be thinking about strategies, but might not need a new role or title.

There are lots of organizations at our universities that have been fairly siloed and digital has the capability of transforming the ways they do business – having those conversations is what’s important.

Ron – If there is going to be a CDO, wouldn’t we want it to be the CIO?

At Harvard there’s a CDO, which grew out of the Communications office, is very tied to the campaign and external public digital communications.

Adobe was talking to 9 different units at Notre Dame, poised to sell each of them Digital Publisher licenses. Ron was able to coordinate that conversation. They also had 5 different units trying to build production studios – there was advantage of having the CDO role to be in that conversation and helping to coordinate.

CSG Fall 2012 – Driving Out Technical Debt

Sharif Nijim from Notre Dame is leading a discussion on driving out technical debt.

Technical debt is a concept that when you make technical compromises you build up technical debt that you then have to pay off later. At Stanford it’s being used to talk about not keeping bench-depth of staff in various areas, or deploying systems without the companion expertise in the tool. Allows you trade off quick delivery for long-term payoff.

Kitty notes that there is technical debt in all of the legacy services we keep hanging on to as we get thinner and thinner on the ground. Michael notes that it’s not just technology but also in skill sets, expertise and other areas. Laxmi brings up the concept of varying levels of risk in debt – not always the same.

Ilee – With ERP systems we create backlogs of requests from user community that we can’t get to. With old technologies the cost per unit change is higher – that’s why you want to retire technical debt by moving to new technologies. Want to incorporate new features that take care of some of that backlog. At USC they had systems that were 25 years old – was getting increasingly harder to find people who could maintain the software, and it took a long time to get things done. They decided to go to Kuali for finance and research administration, for HR they decided to go to Workday, still working with Kuali group on Student system. The timeline started in 2009, and will end in 2016/17. They think that the cost per unit change will go down as well as gaining improved business processes. They’re automating processes that used to be manual, allowing more debt to be retired. Created a data warehouse with Cognos that also allows the University to do analytics.

One way of looking at that is replacing older, riskier debt, with newer higher-quality debt. Will also allow of the retirement of debt in some distributed areas as shadow systems are eliminated.

I asked how one measures this technical debt – Teri-Lynn says that Garnet must have a method because they have done estimates of the size of technical debt overall, but she hadn’t been able to find any methodology documented.

Mark notes that there’s debt you take on intentionally or debt that you back into, and looking at it as balancing the risk in your portfolio is a way of understanding it. With financial debt we know what’s on our books, but we don’t know with technical debt.

Michael makes the point that not all debt is bad – Kitty likens this to “I’ll gladly pay you Tuesday for a hamburger today.” Tim says that as we try to get to yes in satisfying requests we are taking on more debt. Bruce says that as we make tactical decisions we have to be conscious of keeping things narrow enough to not let them creep into more debt. Kitty talks about the maturation of service management and service owners not always realizing the amount of technical debt that particular technologies are accumulating. Bruce notes that this is because people are attached to the technology, not the capability.

Steve notes that sometimes retiring technical debt can come at the cost of incurring political debt.

It would be good if we had periodic reviews of debt. We could categorize the debt the same way we do with risk management. Tim asks if we could catalog the debt perhaps as an addendum to our service catalog. Kitty replies that many of us do risk management, and incorporating this as a concept of risk management could help.

Tim talks about Harvard’s approach – they’ve identified a person to evangelize the concept of technical debt, provide that level of awareness to the business. He’ll follow the approach of building risk statements into the services. It’s in a formative state.

In the chat room the question is raised as to whether we’re building “cloud debt” as we move services to the cloud.

CSG Fall 2012 – Balancing Central and Distributed Services

Bernie Gulacheck from Minnesota is leading a discussion on Central and Distributed Services. This is not a new topic, but the context has changed. We’ve seen the delivery of technology services change over the years. In the late ’80s and early ’90s distributed service units in Libraries, Administration, Academic Computing, were amalgamated into central IT units. Then the conversation shifted to the current landscape of distributed technology units and a central unit. The model along service continuum was often new technology emerging in the distributed units and then later being centralized for economies of scale. The cloud shifts this dynamic, where both central and distributed units can shift or bring new services into being in the cloud.

We’d like to believe that each unit that manages its own technology services is focused on its mission so as to create complementary and not duplicative services – sometimes that’s the case, sometimes it isn’t. What are the elements that facilitate this model? One comment is that what works is transparency – letting the deans and administrators know what is being offered centrally and going through the services each school is offering to see where there is duplication. The service catalog was very important in making this happen. Making that visible allows the conversation about efficiency and making sure that the quality of central services is acceptable to the schools.

Cornell has a structure where the distributed technology leaders also report in to an associate CIO in the central office – they are learning how to build the trust and efficiency in the group. They are building a brand of IT@Cornell that encompasses the entire concept, and that’s starting to work. The services organization is trying to lower cost and maximize efficiencies in order to provide the best service possible to demonstrate the utility value of central services so they don’t have to be duplicated locally.

Kitty notes that we have to be conscious that some services can only be delivered by the person who sits right next to the user – need to know the people, who’s got grant deadlines, etc. Also it’s a challenge for us to make core services easy enough to use.

Bernie notes that often the cloud services are superior to what we can offer, but the factors preventing us from moving in that direction are some of the same factors that prevent the distributed units from moving services to the center.

Elazar notes that trust is a key factor – they’re rolling out a new desktop support environment that will cover the whole institution, and it’s the same with consolidating data centers. Ron Kraemer notes that little things count – like referring to services as the “OIT” data center instead of the “Notre Dame” data center.

Tom says that the ability to present central services as something that distributed units can just use, much as they do the cloud, is important.

Sometimes the actual consolidation of services, even when everyone agrees it makes sense, can be perceived as threatening people’s jobs, which makes it hard to make progress.

Tracy notes that the more you can include people from the units within the central organization as much as possible can help build the relationships. Also you have to build a story and stick to it that gives people hope and a sense of purpose – where is the evolution of their position?

The concept of the say/do ratio is important – ideally would be 1:1.

Developing soft skills in the organization is important.

Bill notes that they started something called the Stanford Technical Leaders Program, where they brought in MOR to help build skills with 13 technical people from the central unit and 13 distributed people from around the campus. Last year they put on an un-conference, and registration fills within minutes after they open the web site. Once it gets to management it’s a failure – want to build the soft skills and the relationships.

It’s important to be honest that jobs are shifting and new skills will be needed – it won’t always be possible to retrain people, and in some cases groups will shrink.

At Brown they looked and found that they’re 49% central and 51% distributed, and in many cases the distributed people are being paid better than in central IT.

Tom notes that governance has helped, but that inputs from distributed units hasn’t always come through those administrative processes. Being able to prioritize and schedule work realistically is important.

Bill talks about “getting beyond polite.” He was told that that his (Bill’s) presence in the room was too loud, and without him in the room the discussion gets more down to earth.

I noted that often people ask for the help of the central unit in solving problems but we don’t have the capacity to deliver help in a timely manner. Bernie then asks what happens when we build services that have been requested by the distributed technology services but the units then opt out and complain about cost increases? Chuck has found that an effective technique is to let the unit lead the project and be responsible for end-to-end including announcement can be effective. Ilee says that making sure that the distributed units are involved in the definition of the services and that having a way to communicate with the deans is important.

Where we’re still in hot water is where we’ve over-promised and underestimated the complexity of replacing local services with central services, which burns our goodwill chips. We don’t want to stifle innovation in the units.

There are often pressures on the CIO to optimize cost in IT, but deans and other leaders can be hesitant to have conversations about the steps necessary to achieve those savings.

It might be possible to give schools score cards about where they are in comparison to each other and central units – has to be done independently (e.g. by the finance unit). Can help deans make decisions on how to allocate resources.

Having visibility into all the IT requests can help people understand what is happening and alert people to potential duplications of effort.

At one institution they don’t use the word “distributed” but use “federated”.

One person notes that if you have distributed people also report in to the central unit that you let the units off the hook a bit – can be a double-edged sword.

CSG Fall 2012 – Projecting Infrastructure into the Cloud

Tom Barton from Chicago and Michael Gettes from Carnegie Mellon are leading a discussion on Projecting Infrastructure into the Cloud.

Identity Federation & Attribute Release – Federated Access anyone? Release directory info! In InCommon identity providers get into the federation, but not always service providers. – get your SP into the Federation. At CMU they release directory information – For everyone- eduPersonPrincipalName (which for them is an email address), and eduPersonScopedAffiliation. For non-students: givenName, surname, commonName, email. Allows for very quick integration of cloud providers. Will this work for others? Ken notes that projects such as Vivo have lots of data with no access control.

Contracts – we spend lots of time on compliance and security, but not on functionality and defining the relationship. CMU and PSU are requiring their vendors to join InCommon. One comment is that vendors are increasingly resistant to joining InCommon.

There’s a bunch of discussion about things that are beyond identity – how do we deprovision users, how do we communicate limitations, where things are easier or harder in the cloud. Kitty notes that in some contract negotiations with cloud vendors they are requiring targets about load and latency testing from different points in the world.

CSG Fall 2012 – Future of the IT Organization pt. 2

Now we’re having a series of point-counterpoint arguments about some issues around the future of IT organizations in higher ed.

The first is about the pace of change. Bernie Gulachek from Minnesota is arguing the point that change in higher education comes slowly and will continue to do so. Applications are up and enrollment is being managed, and until that is threatened our institutions have little impetus for change. Our institutions are not architected for change and we have a hard time affecting outcomes. We’ve had lots of opportunities for change in the last 10-15 years, but we’ve made conscious decisions to not bring about change. We’re now talking about commodity solutions in the cloud because we weren’t able to come together and create our own solutions collaboratively over the last ten years. Change will come incrementally.

Ted Dodds from Cornell takes the position that we’re in denial – people believe we can duck and cover and the old ways will serve us into the future. The notion that somehow we can protect and insulate our community from the change doesn’t make sense. What he’s seeing for the first time ever is that academic leadership is provoked and engaged by MOOCs. These will undeniable change and affect the business models of our institutions. If we as IT leaders aren’t out in front of and participating in this change we won’t survive.

Charlie Leonhart from Georgetown and Bill Clebsch from Stanford are arguing about efficiency in IT organizations. Charlie starts by stating that the era of big IT is over. When he walks around campus he hears complaints about “you’re too big, you’re too fat, you’re too slow, you have too many people, you’re in my way – you guys suck.” We have basic facts – decimated budgets, needs to cut spending. We require bold leadership. Five point plan – cut spending, do more with less, cut staff positions, let’s virtualize. Get rid of desktop devices. Encourage innovation that drive down costs. Build strategic partnerships. Reduce central services and let users vote with dollars.

Bill starts saying there are 3 tsunamis: Research computation and big data (not building wet labs anymore, but building to support cyberscience); online learning; cloud and mobility. All this takes more money. The dollar savings aren’t there – but advantages in scaling, speed, and response to user. Mobility is a cost, not a cost saver. This is becoming a larger part of our lives – we can’t stop spending more. This is a part of the critical mission of the University – can’t stop spending now.

Elazar thinks we can do both – achieve efficiencies in our organization and use the savings to invest.

Elazar Harel from UCSF and Ron Kraermer are arguing about Bring Your Own Device. Elazar starts by saying that he believes that everybody should be able to bring whatever devices they want and they should work well in our environments. It also means BYOA (apps), BYOC (connectors), BYOP (printers), etc. When you think about this, there are some solutions – we need to be in the cloud with our infrastructure and applications.

Ron asks what business are we in? We’re in the business of delivering an optimal educational experience. We can compromise that technology by trying to support everything. What he plans to do at Notre Dame is lower the cost of textbooks by delivering them on standard devices (iPads), and they’ll train faculty on those devices to develop and deliver content. iPads will dominate the education market, and they can use it to improve the experience.

There’s a general discussion of the emerging roles of vendor relationship managers, service and product managers and technical integrators. The crowd isn’t clear yet as to how much or how quickly the old roles will go away.