Citrix’s Singapore office once had a concern with a type of internal document, and how it was affecting people’s time.

According to Tan Sze-Tong, the cloud and virtualisation company’s Regional Channel Strategy and Operations Director for the Asia-Pacific and Japan region, a lot of its partner business plans lacked quality, completeness, and substance.

In addition, the Singapore office’s people managers were spending considerable time having to correct these partner business plans, and go back and forth with sales representatives on updating said plans.

This was a concern because 90% of Citrix’s business is driven through its solution partners.

“Partner sales enablement and account management are very important to Citrix,” Tan emphasised. “Part of being an effective partner sales rep is to be able to prepare and execute on the joint partner business plans. This is something we focus a lot on as part of our sales management training.”

The state of the partner business plans and the time spent on them were really counter-productive, he added.

“There were many document versions being sent on email, no proper tracking in place, managers were not spending their time effectively, and partner sales reps were not really learning from this process,” Tan remarked.

Automate and educate

Consequently, Citrix looked for solutions and found Singapore-based edtech firm Noodle Factory.

While mainly serving higher education, K-12, and corporate learning institutions, Noodle Factory’s AI-powered learning platform (fondly called Walter) helped to automate Citrix Singapore’s repetitive processes and build foundational knowledge.

“For example, we have so many different partner programs in place – the platform is great for reinforcing the details of the different programs so that sales reps are equipped with this knowledge,” shared Tan.

The AI also helped to automate business plan evaluations that were previously done by the people managers.

“The AI integration was great in ensuring that sales plans covered all the key areas, that there were sufficient activities, and that the focus was aligned to Citrix’s key sales objectives,” he noted.

“Of course, the AI would not necessarily be able to understand the nuances within different partner companies – but this is where the managers can provide value in their interactions with their sales reps; and having the AI to automate the evaluation and feedback on structured parts of the plans freed up the managers to be able to have more of such valuable discussions with their sales reps,” Tan continued.

Selling points

Noodle Factory, however, isn’t the only company with an AI-based platform, as there are similar tools out there. Citrix went with Noodle Factory because the edtech firm uses a “very practical approach” to using AI.

“Rather than trying to use AI to create better sales reps, the use of AI is really focused on automating repetitive tasks and building structured knowledge within training and development – these are areas where there is little value for managers and trainers to be involved directly,” Tan explained.

“The AI does not ‘take over’; it assists so that humans have more time to focus on areas where there is value in human-to-human interactions,” he observed.

Speaking on what sets Noodle Factory apart from competitors, Dr Jim Wagstaff, its Chief Learning Officer and Co-Founder, shared some of the feedback they received.

“Firstly, our platform is designed from the ground up with content-rich learning environments in mind. Secondly, we are content and ‘channel’ agnostic. This means that clients can utilise their own content that they have spent so much time, money, and effort creating. Plus they can choose the messaging channel or channels that make the most sense for their users, for example, Microsoft Teams, Slack, a widget on a website, etc,” he said.

“Specifically for Citrix, they were looking for a way to provide real-time, personalised coaching to their partner account managers about a very specific skill that is core to the role-developing account plans with Citrix channel partners,” Wagstaff pointed out.

“The feedback that we have heard from Citrix is that this represents an innovative use of AI to simultaneously provide real-time feedback and coaching, effectively automating a process that is important, but repetitive – something that can be handled efficiently through the use of AI,” he added.

Citrix’s use of Noodle Factory’s platform started out at the Singapore office and among its APAC and Japan leaders. The organisation then adopted the platform at a worldwide level once it was launched.

How AI understands

One critical factor that helped the platform’s overall effectiveness was the natural language processing (NLP) technology it used. According to Wagstaff, Noodle Factory developed its NLP algorithms using transfer learning to train it to understand and engage around the conversational use of language.

“Since we are content-agnostic, it is important that our NLP algorithms engage learners based on data (i.e. text-based content) that the algorithms are exposed to,” he said.

“When we first began our development process on the platform, we were privileged to work with AI Singapore and a number of experts from the National University of Singapore. During this early development stage, we trained the algorithms on the nuances of the English language—including slang and how to handle unusual phrasing of questions, and humans were used to establish a baseline level of accuracy,” Wagstaff shared.

“In those early days, human accuracy on handling user questions was around 86% while the accuracy of the algorithms were at around 76%. Now, four years later, the accuracy of the algorithm in responding to user questions and conducting assessments has improved to around 83%. Our goal is always to be better than our human baseline and we are getting tantalisingly close,” he added.

One thing to remember about an NLP algorithm, or any flavour of machine learning, said Wagstaff, is that it is only as good as the training data that the algorithm is exposed to, and how the algorithm handles the input and output on a quantitative level.

“The rule of thumb with these types of algorithms continues to be ‘The more clean data you expose the algorithm to, the more accurate your algorithm will be’. To this end, the more conversations our users have with our algorithms within the context of the data sets that the algorithms have access to, the more meaningful and accurate these conversations or activists become,” he remarked.

Plans for the future

Now that its concerns with productivity and business plans are resolved, what’s next for Citrix?

“Over the course of the pandemic, we’ve really accelerated our ability to enable companies to support a distributed workforce and hybrid remote work. It’s even more important these days that companies can securely and efficiently support their workers when they work from anywhere, from any device at any time,” Tan said.

“Some of the most exciting areas we are developing are in the area of application delivery, flexible digital workspaces through our desktop-as-a-service offering, security, and analytics – just to name a few,” he shared.

“Just like our customers, we’ve had to navigate all these challenges of distributed work and virtual and cloud application delivery – so, similarly, the technologies we plan to adopt would need to support our internal teams in these areas as well,” Tan added.

As for Noodle Factory’s roadmap, Wagstaff said it’s influenced by their interaction with clients.

“A few of the things that you will see over the next several months include the ability to go beyond simply one-on-one engagement with learners by introducing a social learning element. This reflects the desire for a more collaborative approach to learning via small groups or project teams. The Noodle Factory platform will essentially be there to support the team learning approach as an assistant or, in some cases, an AI guide to the topic or assignment at hand,” he said.

Along with that, Noodle Factory plans to introduce sentiment analysis capabilities to pick up early signals that learners might be struggling or potentially burning out.

“By picking up on these early signals, managers can be brought into the loop to provide a ‘human touch’ via support and coaching of learners,” Wagstaff explained.

“Another area that we are hard at work on is the ability to engage learners on reflective types of questions or assessments. Currently we really focus on very factual, concept-driven engagements with learners, but by moving from Level 1 (knowledge acquisition) to Level 2 (knowledge application) in our ability to navigate and support and orientation towards Bloom’s taxonomy of learning, this will allow us to dramatically expand the types of subjects and assessments in which we engage learners,” he concluded.