CSUSB Advising Podcast

Ep. 99 - From Questions to Answers: Using the Ask Cody Chatbot

Season 1 Episode 99

In Ep. 99, academic advisor Matt Markin chats with Michael Casadonte, the director of digital transformation at CSUSB. Michael discusses CSUSB's chatbot, Ask Cody, which uses generative AI technology in its responses. What are the benefits to students? Can Ask Cody handle questions about campus resources, processes, and important dates? Are the answers accurate? Find out here! 

Subscribe to the CSUSB Advising Podcast on Apple, Spotify, and more!

Follow us on social media:

Instagram - @csusbadvising
Tik Tok - @csusbadvising
YouTube - @csusbadvising

https://csusbadvising.buzzsprout.com/

Matt Markin  
Hey, and welcome to another episode of the CSUSB Advising Podcast. My name is Matt Markin, an academic advisor at Cal State San Bernardino, and this is episode 99 on today's episode, we're learning more about chat bots and the benefits chat bots have for you as a student at CSUSB and to help us learn more, let's welcome our guest, Michael Casadonte, the director of digital transformation. Michael, welcome.

Michael Casadonte  
Thank you. Matt, glad to be here.

Matt Markin  
Before we kind of dive in talking about chat bots, tell us a little bit about yourself and what you do at CSUSB?

Michael Casadonte  
Well, I've been at CSUSB since October of 1998 and I was, I was hired as the first instructional designer here. But years passed and opportunities came, and the Office of Digital transformation was created about six years ago, and I was fortunate enough to to lead that group. And so in digital transformation, we oversee the websites on campus, the campus portal, the mobile app, we oversee the digital map of the university. We have a hand in things like the university calendar and the campus directory and things like that. And about seven years ago, we started building a chat bot, which has grown into ask Cody, which is what we're here to talk about.

Matt Markin  
Yeah, I think that's a great transition to that. So what exactly is a chat bot? What exactly is Ask Cody?

Michael Casadonte  
So a chat bot is a means of communicating with with the machine that has been informed by instructions and code to perform certain tasks. A conversational chat bot such as Cody, as well as GPT, is known as a conversational chat bot, and it's intended to respond as though you are talking to it. So it is not like a search engine that delivers results. It is conversational in nature, and so you're not chatting with a person, but it's meant to feel like you are.

Matt Markin  
And let's say, you know, I'm a student, you know? What can you tell me about how I as a student can access a chat bot? Is it? You know, it's accessible on mobile devices, desktop computers, laptops?

Michael Casadonte  
Sure. Well, Ask Cody was created really with students in mind, to work as an assistant, to be able to answer the multitude of questions that students have when they come to university. So it again, I said it was seven years old. It started on the financial aid website. That was the only place you could get it, and then slowly expand it out to other offices as we worked with those offices, and so to name a few, it's on the advising website, admissions, registrar, orientation, the Career Center, parking and the department that I work in, which is its information technology services. But starting a little over a year ago, we placed it within the mycoyote portal, and suddenly, the traffic more than doubled when we did that, and so the interactions are slightly over 50% from inside the mycoyote portal. So it's available on about a dozen websites, including the University Calendar and the campus directory, as well as working within myCoyote.

Matt Markin  
And I imagine if it's available on various websites, it's available 24/7, a student at any time, can just go on there and ask a question?

Michael Casadonte  
Yes. That is one of the benefits of a chat bot. It probably should have mentioned it when I described what a chat bot is, because they are all meant to be 24/7 and available anytime. There's no reason why they shouldn't be working, not to mention that they can answer questions simultaneously, so a multitude of people could be asking questions all at the same time.

Matt Markin  
Definitely a lot of benefits a student could get from using the chat bot.

Michael Casadonte  
Absolutely one of the benefits that I wanted to make sure that I mentioned was when you when a student asks a question inside of the portal, inside of myCoyote, we know a little bit about the person when they come in. We we do grab their name and and use it in the welcome message. After that. We don't really use their name, but we know their status. So we know whether they're an applicant, whether they're an undergraduate student, a graduate student, an international student, maybe a Palm Desert campus student. And I was just looking at an interaction today where the bot responded. With since you're an applicant, and then continue with its answer. And that's exactly what we want it to do, because anyone who's worked here for a while, or anyone who's been a student for a while knows that answers from people on campus usually start with, Well, are you an undergraduate or grad? You know, some questions about yourself, and so we want the chat bot to know as much about the end user as possible, so that it can deliver an answer that's catered to them, because that, as we know the the answer is very different depending on who you are.

Matt Markin  
Very true. Yeah. So that kind of helps that it's not just going to give a generic answer to a question that could be wrong. It's trying to really understand a little bit more information, background information about the students, so it can hopefully accurately give an answer.

Michael Casadonte  
Yeah. I mean, if, if you use it on the website, then it doesn't know anything about you. It It can't. And so the you know, the answers in will be longer, right? Because it will say, Well, if you're this, here's the answer. If this is the case, then here's the answer. And so it's just simply longer. And so one of the benefits, I guess, of knowing the role that someone the roles that someone might be in is that you get a shorter, custom, personalized answer. 

Matt Markin  
So question because you said it's on, like, about a dozen websites. So I know, like for advising, we have the chat bot on there, but if, if a student asked a financial aid related type question on the advising website, chat bot, would it answer the same way, or does it direct it to the chat bot for financial aid?

Michael Casadonte  
Yes, it it can handle any question about any topic from any particular place. The only nuance difference would be, if you're logged into myCoyote, it might know a few things about you, and then custom deliver an answer. So I'll admit, back in seven years ago, when we built the bot there, we actually built a different bot for every office, and that became unmanageable, and so we had a means of kicking them over to another department to help answer a question. But that's not the case anymore. Just you're you're free to ask, and you can ask questions on topics that are handled by other offices that don't even have the chat bot on their website. So it's very open to just about any topic that you can imagine, as long as it relates to CSUSB.

Matt Markin  
Very cool. Now, how accurate, I guess, would you say the information is that's provided by the chat bot?

Michael Casadonte  
So that's a really good question and an important one to be able to answer, so I appreciate it. The accuracy of the chat bot, really, we kind of have to delve into a little bit about how it derives its answers. So I'm going to talk a little bit about structured data first structured data are data that is I'll give examples like the campus directory. The campus directory is full of information about the people who work here, faculty and staff. They have first names and last names and titles and locations and emails and all of that sort of stuff. And the chat bot ingests all of the data as it relates to campus directory every single day, and that's structured data. So there's a imagine a spreadsheet or a table that has columns with unique column names, and with that, the chat bot has, I'll say, an easier time delivering an answer if it is going to use the structured data to deliver the answer. So accuracy, accuracy when structured data is used, like the university calendar and the campus directory is very, very high, as long as the source is accurate, so the campus directory is pulls from our student and faculty staff database of information. So as long as that's accurate, the answer is accurate, and the same with the university calendar, which is a bunch of date important dates for the university, but it's not very complex. It's just there's a date and there's a year and and there's a title to that entry. So then I'll move on to unstructured data. Unstructured data is when, when it can't be when the answer can't be derived from a structured data source, it'll go to unstructured data, which I'll call the campus website. So we index the entire campus, CSUSB, Campus website domain, and then. Use that when necessary, to use the artificial intelligence tools to pull the answer from our website. So what is the accuracy of the answer depends on the accuracy of the website. We know that there's some older information out there on the website that may that the bot may have a difficult time finding and well, the dated, the, sorry, the dated information, it may find that and not realize that it's out of date. So for example, if somebody doesn't put a year for something on their website, the bot won't know what year it applies to, and if it was four years ago, it may deliver an answer like it's upcoming. And that's that's problematic. So it really does depend. It has delivered some inaccurate information. We hope that sometimes it says it's not quite sure about the answer it's delivering, and it may do that. I should mention that if it can't find an answer to a question on CSUSB website, it then goes to the third level, which is to pretty much perform a Google search, or to just search the world for a particular answer to your question it. Think of it as a person who is determined to try to give you an answer. 

Matt Markin  
So it's not going to get tired. It's going to try to find some answer for for the question that the students asking?

Michael Casadonte  
It is not going to get tired. In fact, one of the questions that someone asked, it likely a student was asked to help with their programming homework, and the bot said, I can't help you with your programming homework, but I can recommend a programming class that you can take to help you your program. So it is really fixated on CSUSB and directing you back to everything that we offer. 

Matt Markin  
I think that's a great example. And I guess going along that those lines, can you provide some other examples of other types of questions that students have asked?

Michael Casadonte  
Sure, this is pretty easy for me to do, because I now compile a what students asked last week report that I send out, and so it breaks it down into different categories. So the way I want to present it to you is there are where, when, why, what, how and who questions. Just, you know, the standard types, but we do deal with a lot of questions about where something is, where a building is, where an office is, and we do have a campus map that it may refer to and link to to find show you exactly where those things are. A lot of questions about when something occurs in that in that invokes the university calendar structured data to give them an answer, we get a lot of those questions, when is spring break? When is the last day to add drop a class? When are final exams? Things like that. All of those things are in the university calendar. But as you would, as you would expect, students struggle with how this university works. And so we are a series of processes here on campus. The process to do something is this, and a lot of the questions that students ask are about some process or another to complete the task. For example, change my major. There's quite a few questions about, do I have to see an advisor to change my major? Actually, there's a lot of questions that are followed by, you know, should I see an advisor to do this thing? In most cases, the answer is 100% yes, you should do that. So which is, which is good to see? So that's we get questions all over the all over the map. I, in fact, I want to tell you a story about a conversation I read from a student who was asking about their financial aid refund and when it would be directly deposited. I guess they had set it up to be directly deposited into their bank, and they were asking when that might occur, and Cody responded with, it should take about three business days, but it could take the up to 10 business days. Okay, great. Well, I noticed in in looking at the system we built up to look at conversations that the student came back the next day just to tell Cody that it had got is their direct deposit in their bank account, to which Cody just said, Well, congratulations, thanks. Have a you know, good luck this semester, they didn't ask a question. 

Matt Markin  
Well, they got the help by asking the question, and then hey, wanted to say, thanks for that information. I got I got the deposit. 

Michael Casadonte  
Yeah, isn't that great? 

Matt Markin  
It is. I know. I feel I know the answer to this, but I'm just going to ask it, do you feel that the chat bot is going to replace communicating with an academic advisor or a department?

Michael Casadonte  
I don't think it's close at all to replacing an advisor or an academic department. Do you know, due to several reasons, one, the the relationship that students probably should develop with another human being whose job it is to be an advisor, I think having been a student myself, that relationship is really important and can be very comforting. So the reason why I say it's a long way away is I guess that suggests that it might someday be able to do that. And how would it? How would it do that? It the information that the chat bot doesn't know is all of the pathways that a student could take to get their degree in any particular subject. There's there are road maps, for sure. Those road maps are just examples of one, one way to get to the final destination of obtaining your degree, but the chat bot is not aware of all the elements of a roadmap. It might be aware of the document and be able to share that with someone who is curious about what courses they need to take to achieve a degree, somebody who's shopping around, but a student who's in the thick of it, of taking classes it was on the road, the chat bot isn't not, isn't going to have insight into that, and just simply won't be able to answer it. And maybe someday, if we decide to feed it that it's it going back to structured data, it will have to be extremely structured data in order to make an attempt to say you need to take this class before this class, right? It doesn't, it doesn't know about prerequisites and things like that, right? 

Matt Markin  
So it seems like the chat bot is a great resource for students to use. It's a nice companion, especially after eight to five when most offices are not open, but it definitely would be something that students should still communicate with their advisor, still go to a department, but still be able to utilize the chat bot. And in this case, the more questions that the student asks through the chat bot, the more reliable it might become?

Michael Casadonte  
Well it, I know that a lot of people think that it works that way and it this one does not. It doesn't build off of the questions and the answers that it gave. It does, behind the scenes, deliver a confidence level to us, literally, a number that represents its confidence in the answer that it gave. And every single interaction is evaluated by human beings to determine many things, whether or not something went wrong, whether a link was correct, what department the authoritative source of of that answer should be, which, within the context of a conversation with a single individual might be several departments.

Matt Markin  
But if anything like the questions that are being asked, it is being reviewed by someone, and so it is being there are updates that are being made?

Michael Casadonte  
Well, I want to talk about the websites, the indexing of websites on campus, which, because we're the same department that that oversees the websites. That's not a challenge to do, but through looking at some of these interactions and looking the answer, looking at the answers that it gives, we meet regularly with the departments that are most connected to the chat bot and are most involved with delivering the answers. We work with them to look for inaccurate answers that are delivered, and then look to the source of that inaccurate answer, which is in most cases, the website, so that that information can be removed or updated so that it is accurate. So look at it this way, the chat bot is, turns out to also be a tool to determine and find any inaccurate information on the websites at CSUSB, so that we can turn that around and change that.

Matt Markin  
Well, I'll say in other ways. You know, you send us a weekly report of some of the questions that are being asked on the chat bot. And I know, for me, in the office, I work with academic advising, we've looked at some of what are some of the common questions being asked each week, and then we're developing short, little Instagram TikTok videos that we can also post on our social media for that.

Michael Casadonte  
That's great. That's how we use it as well. We look at the questions that are being asked. And so one of the things that spawned from looking at those questions was and it was already in the works, but I wrote a survey that ASI ended up sending out to all the students, asking them which important dates out of these, I think there were about 10 which of these they'd be interested in receiving a push notification about and and the frequency with which they should be sent. So we're working to send those push notifications out, say seven days before the last day to add drop or census to let them know that date is up and coming. Well, we think one of the effects of those push notifications about important dates will be they will not ask the chat bot that those questions as much anymore. We're not in the business of discouraging them from using the chat bot, but we're utilizing the questions that they are asking to act on certain things and change things and communicate better with students.

Matt Markin  
Love it. Oh, Michael, that's all the questions I have. Thanks so much for being on the podcast today talking about the Ask Cody chat bot, great conversation.

Michael Casadonte  
Sure. Thanks for having me. 

People on this episode